Agentic Systems
From a clever demo to a dependable agent.
Design, build, and evaluate AI agents that use tools, handle uncertainty, and know when to ask a human.
An agent is more than a model in a loop. It needs a defined task, tools it can trust, a stopping rule, and a way to recognize failure. Build these pieces around one practical project, then measure whether the system actually helps.
Your learning path.
01A useful task before an agentWeek 1 · ≈4 hours+
Define the job, success criteria, and the smallest useful workflow — then decide, on evidence, whether an agent is the right shape at all.
Objectives
- Distinguish tasks that need an agent from tasks that need a workflow.
- Write a task definition with measurable success criteria.
- Establish a non-agent baseline to measure any later improvement against.
Topics
- Agents versus scripts, workflows, and chat interfaces
- The autonomy ladder: where control is traded for capability
- Task decomposition and operating boundaries
- Failure budgets and a baseline without an agent
- Cost, latency and variance as first-class design constraints
Concepts
Control flow ownership
Whether the sequence of steps was written by you or chosen by the model — the only definition of “agentic” that survives contact with production.
Failure budget
The rate and severity of wrong answers the task can absorb before the system stops being worth running.
Baseline
The simplest solution that works, measured, so that any added autonomy has something to prove itself against.
Exercise
Write a one-page agent brief with five acceptance criteria.
Deliverable
Agent brief, baseline measurement, and an explicit autonomy-level decision with its justification.
02The agent loopWeek 1 · ≈4 hours+
Connect model decisions to tools without losing control of execution.
Objectives
- Implement a bounded decide–act–observe loop.
- Express tool selection as structured output rather than parsed prose.
- Define stop conditions, step budgets and fallback behaviour before the first run.
Topics
- Messages, structured outputs, and tool selection
- Input schemas, result handling, and explicit state
- Step limits, token budgets, wall-clock limits and stop conditions
- Deterministic fallbacks when the loop cannot converge
- Reading a trace: what the agent decided and why
Concepts
Stop condition
The explicit rule that ends the loop. An agent without one is an outage with good intentions.
Structured output
A schema-constrained response that names the action and its arguments, removing prose parsing from the critical path.
Trace
The recorded sequence of decisions, tool calls, costs and timings for one run — the only way to debug after the fact.
Exercise
Build a bounded loop with two tools and a hard execution limit.
Deliverable
A running loop with budgets configured and a readable trace for one successful and one abandoned run.
Lesson material is being prepared for this module.
03Tools that are safe to useWeek 2 · ≈5 hours+
Treat every tool as a small API with a clear contract and a hostile client.
Objectives
- Write a tool contract covering schema, permissions, timeouts and error semantics.
- Make write operations idempotent and safe to retry.
- Place approval gates where they are justified, and nowhere else.
Topics
- Validation that rejects rather than coerces
- Permissions, least privilege and blast radius
- Timeouts, retries, idempotency, and duplicate actions
- Typed errors the model can act on
- Human approval gates for external writes
Concepts
Idempotency key
A caller-supplied identifier that makes a repeated write produce one effect — the control that prevents the second charge.
Least privilege
Each tool is scoped more narrowly than the agent, so a redirected agent still cannot reach beyond the union of its tools.
Fail closed
When permission or state is uncertain, the safe default is refusal plus escalation, never optimistic action.
Exercise
Add validation and an approval boundary to a write-capable tool.
Deliverable
A hardened tool contract, plus a short written justification for each approval gate you added or declined to add.
Lesson material is being prepared for this module.
04Context, retrieval, and memoryWeek 2 · ≈4 hours+
Give the agent relevant evidence without accumulating noise.
Objectives
- Separate session state, task state and persistent memory deliberately.
- Attribute answers to retrieved sources.
- Manage a context budget and resolve conflicting evidence.
Topics
- Session state versus persistent memory
- Retrieval, source attribution, and freshness
- Chunking, ranking and the cost of irrelevant context
- Context budgets and handling conflicting evidence
- When memory becomes a liability rather than a feature
Concepts
Context budget
A deliberate allocation of the window across instructions, task state and evidence — decided, not accumulated.
Source attribution
Binding each claim to the retrieved passage that supports it, which makes hallucination visible instead of plausible.
Staleness
Retrieved evidence that was true once. Freshness is a property the retrieval layer must assert, not assume.
Exercise
Add a small knowledge source and cite the evidence behind an answer.
Deliverable
A retrieval-backed answer with attributions, plus a documented policy for conflicting and stale sources.
Lesson material is being prepared for this module.
05Reliability under pressureWeek 3 · ≈5 hours+
Explore what happens when the happy path ends — deliberately, before production does it for you.
Objectives
- Design so that a fully persuaded agent still cannot exceed its permissions.
- Distinguish failure from unknown outcome and handle each correctly.
- Make partial completion visible rather than narrated away.
Topics
- Prompt injection through tool results and documents
- Instruction/data separation as a design property, not a prompt trick
- Unavailable services, timeouts and partial completion
- Uncertainty, escalation, and meaningful error messages
- Running a structured failure drill
Concepts
Prompt injection
Instructions smuggled into content the agent reads, attempting to redirect its behaviour from inside the data.
Unknown outcome
A timeout is not a failure. Querying state before retrying is the difference between one refund and two.
Silent partial completion
Three of five steps succeeded and the summary says “done”. Success must be asserted by the system, not narrated by the model.
Exercise
Run a failure drill with malicious input and an unavailable tool.
Deliverable
A failure-drill report: what broke, what held, and which control you added in response to each.
Lesson material is being prepared for this module.
06Evaluation and observabilityWeek 3 · ≈4 hours+
Measure performance on tasks, not impressive-looking responses.
Objectives
- Build a fixed evaluation set including ambiguous, adversarial and budget-limited cases.
- Score outcomes with a written rubric rather than an impression.
- Compare a change against a baseline and report the result honestly.
Topics
- Test datasets, expected behaviors, and regression checks
- Designing cases that can actually fail
- Tracing decisions, tool calls, cost, and latency
- Rubrics, inter-rater agreement and scoring drift
- Comparing improvements against the baseline
Concepts
Evaluation set
A fixed collection of tasks with expected outcomes, used to compare versions of the system rather than impressions of it.
Regression case
A test added the day a bug was found, so the same failure cannot return unnoticed.
Rubric
The written scoring rule that makes two reviewers reach the same verdict on the same run.
Exercise
Create a ten-case evaluation set with a written scoring rubric.
Deliverable
An evaluation suite with a baseline result and a before-and-after comparison for one deliberate change.
Lesson material is being prepared for this module.
07A system someone else can runWeek 4 · ≈5 hours+
Package the project and make its limitations explicit.
Objectives
- Document scope, boundaries and known limitations in operator language.
- Provide monitoring, a kill switch and a rollback path.
- Hand over a system a colleague can operate without you.
Topics
- Configuration, secrets, and deployment boundaries
- Monitoring, alerting thresholds and cost controls
- Rollback, kill switches and operational ownership
- A clear runbook and human escalation procedure
- Writing down what the system must never do
Concepts
Runbook
The document that lets someone who was not there diagnose and stop the system on a Monday morning.
Kill switch
A single, tested control that halts the agent — and a named person permitted to use it.
Operational ownership
The named team accountable for the system’s behaviour once it is no longer a project.
Exercise
Complete the capstone, evaluation report, and operator guide.
Deliverable
Capstone system, evaluation report, runbook and handover notes.
Lesson material is being prepared for this module.
Who this is for +
- Developers ready to move beyond single-prompt applications.
- Technical builders who want to automate a bounded business workflow.
- Engineers evaluating when agents are worth the complexity.
Before you begin +
- Comfort with basic Python or JavaScript, functions, and HTTP APIs.
- Familiarity with Git and running a small local application.
- A laptop and access to a language-model API for practical exercises.
What you will learn +
- Choose between a deterministic workflow and an autonomous agent.
- Write tool contracts with validated inputs and explicit permissions.
- Manage state, short-term memory, and retrieved context.
- Introduce approvals before sensitive or irreversible actions.
- Evaluate task success, cost, latency, and failure modes.
- Deliver an agent with a runbook and a repeatable evaluation suite.
Applied assignments +
Agent brief
A task definition, scope boundaries, and acceptance criteria.
Working agent
A bounded tool-using workflow with validation and approval gates.
Failure & evaluation report
Ten test cases, failure analysis, and before-and-after measurements.
Your capstone project +
Build a research-to-action assistant
Capstone brief +
Create an assistant that consults a supplied knowledge base, drafts a proposed action, and waits for approval before executing it. Submit the code, tool contracts, evaluation cases, and a concise runbook.
Assessment +
A 60-minute scenario exam: choose an architecture, identify unsafe tool behavior, diagnose a failed run, and explain your evaluation strategy.
What you will produce +
- Agent task brief
- Tool-contract checklist
- Evaluation scorecard
- Operational runbook
The method, explained +
Kicker
THE METHOD, EXPLAINED
What actually makes a system agentic?
An agent is a system that is given a goal rather than a procedure, chooses its own next action, acts on the world through tools, and decides when it is finished. Everything else is a workflow with a language model in it — which is frequently the better design.
- The useful distinction is not how clever the model is but who holds the control flow. In a workflow, you wrote the steps and the model fills in the judgment at fixed points. In an agent, the model chooses the steps and you constrain the space it may choose within. The second is more capable and strictly harder to operate: every additional degree of freedom is an additional failure mode you now own.
- That is why this course treats autonomy as a dial rather than a destination. Most production systems that are described as agents are, correctly, mostly deterministic workflows with one or two genuinely agentic segments. Knowing where to place those segments — and where to refuse them — is the engineering skill.
- The parts that make an agent dependable are unglamorous: tool contracts with validated inputs and least-privilege permissions, explicit stop conditions, approval gates in front of irreversible actions, traces you can read after the fact, and an evaluation set that tells you whether last week’s change helped. Demos skip all five. Systems cannot.
Principles
Earn the autonomy
Start from the deterministic workflow that solves the task and introduce autonomy only where the branching genuinely cannot be enumerated.
Tools are APIs, not conveniences
Every tool gets a schema, validation, a permission scope, a timeout, and an idempotency story. The model will call it in ways you did not anticipate.
Bound the loop
Step limits, budget limits, wall-clock limits and explicit stop conditions. An agent without a stopping rule is an outage with good intentions.
Gate the irreversible
Reads are cheap to get wrong. Writes, payments, deletions and outbound messages get a human approval boundary until the evidence says otherwise.
Treat tool output as untrusted input
Retrieved documents and API responses are an injection surface. Content that arrives through a tool must never be able to escalate what the agent is allowed to do.
Measure tasks, not transcripts
A good-sounding answer is not a completed task. Score against task outcomes, cost, latency and failure class, on a fixed evaluation set, against a baseline.
Loop caption
The agent loop with its control points. Every box outside the model is something you own and can constrain.
When heading
When an agent is the right shape — and when it is not
When yes
- The path through the task genuinely varies and cannot be enumerated in advance.
- The task needs to gather evidence before it can decide what to do.
- Partial progress is useful and a human can take over mid-task.
- Failure is visible, recoverable and cheap relative to the value of success.
When no
- The steps are known — write the workflow; it is cheaper, faster and testable.
- The action is irreversible and the cost of a wrong call is severe.
- You cannot yet describe what a successful run looks like.
- You have no way to observe what the system did after it did it.
Misconceptions
“More autonomy means more capability.”
More autonomy means more variance. Capability comes from good tools, good context and a well-scoped task; autonomy just decides who sequences them.
“The agent can check its own work.”
Self-critique catches some classes of error and is systematically blind to others — particularly ones that follow from a misunderstanding it is still holding. Independent checks earn their keep.
“Evaluation can wait until it works.”
Without a fixed evaluation set you cannot tell improvement from luck, and every prompt change becomes an unfalsifiable opinion.
“Multi-agent solves hard problems.”
Splitting a task across agents multiplies coordination cost and failure surface. It helps when the sub-tasks are genuinely independent, and hurts otherwise.
Course glossary +
Agent loop
The cycle of model decision, tool call, observation and re-decision, bounded by explicit stop conditions.
Tool contract
The schema, permissions, timeout and error semantics of a single action the agent may take.
Approval gate
A mandatory human decision inserted before an irreversible or externally visible action.
Prompt injection
Instructions smuggled into content the agent reads — a retrieved document, a web page, an API response — attempting to redirect its behaviour.
Evaluation set
A fixed collection of tasks with expected outcomes, used to compare versions of the system rather than impressions of it.
Trace
The recorded sequence of decisions, tool calls, costs and timings for a single run — the only way to debug behaviour after the fact.
Compare the approaches +
Every method is driven by something
The “-driven development” family is usually presented as a list of competing camps. It is more useful to read it as a single question asked repeatedly: which artefact is allowed to decide what gets built? Each method answers differently, and the answers compose more often than they compete.
| Method | What drives the work | Primary artefact | Strongest at | Costs you |
|---|---|---|---|---|
| VIBE — Vibe coding (no method) | An informal prompt and whatever comes back | Working code, no written intent | Throwaway prototypes and personal tools | No definition of correct; silent architectural drift |
| TDD — Test-Driven Development | A failing test written before the code | A unit test suite | Tight feedback, regression safety, design pressure on interfaces | Describes units well and product intent poorly |
| BDD — Behaviour-Driven Development | A shared example of behaviour in business language | Given–When–Then scenarios | A common language across product, QA and engineering | Scenario suites rot quickly without ownership |
| ATDD — Acceptance-Test-Driven Development | An acceptance criterion agreed before work starts | Automated acceptance tests | Customer-visible definition of done | Needs real discipline to stay automated and current |
| DDD — Domain-Driven Design | The domain model and its ubiquitous language | Bounded contexts and a shared model | Complex domains, integration boundaries, long-lived systems | Heavy for small systems; easy to cargo-cult |
| MDD — Model-Driven Development | A formal model that code is generated from | A UML or domain-specific-language model | Regulated, repetitive or multi-target generation | Tooling lock-in and painful round-tripping |
| CDD — Contract-First / API-First | An interface contract agreed before either side is built | An OpenAPI, protobuf or schema definition | Parallel work across teams; verifiable integration | Says nothing about behaviour behind the interface |
| TyDD — Type-Driven Development | The type system and its signatures | Types that make illegal states unrepresentable | Correctness by construction in expressive languages | Bounded by what your type system can actually say |
| DocDD — Documentation-Driven Development | The user-facing document written first | README, changelog or API documentation | Outside-in thinking; catching bad ergonomics early | Documentation drifts the moment attention moves on |
| SDD — Spec-Driven Development | A written specification of behaviour, constraints and non-goals | The specification plus a requirement-to-evidence matrix | Directing AI agents; keeping human judgment on what and why | Specification maintenance; over-specifying wastes effort |
Note
These are layers, not rivals. A mature spec-driven workflow typically states acceptance criteria in BDD-style examples, expresses interfaces as contracts, and verifies the result with tests that a TDD practitioner would recognise. What spec-driven development adds is the part the others predate: an artefact precise enough to hand to a non-human implementer, and a traceable answer to “how do we know it did what we asked?”
Closing heading
Where spec-driven development sits
Closing
- Ahead of TDD, not instead of it. TDD starts once you know what the unit should do; the specification is where that knowledge is decided, argued over and recorded.
- Above the contract. An API contract fixes the shape of an interface. The specification fixes what happens behind it, including what must never happen.
- Beside the domain model. Domain-driven design gives you the language to write a precise specification in. The specification is where that language gets used to settle a concrete behaviour.
- Against vibe coding — but only on work that will outlive the afternoon. The method earns its cost where correctness is contested, expensive or durable.
Teaching perspective +
This course uses a fictional AI teaching persona, not a real instructor.
Name
Mira Voss
Role
Agent systems & evaluation
Bio
Mira is the teaching persona for Agentic Systems. Her course perspective brings together bounded autonomy, reliable tools, and practical evaluation.
Approach
Build a small system you can explain, measure it, and improve it one decision at a time.