Job-driven agent design

As an agent gains capabilities, traditional evals become difficult to manage. Starting from defined jobs is a more sustainable approach as an agent grows, and has the added benefit of guiding agent refinement.

Agent evals increasingly measure task completion against a resulting state rather than a prescribed trajectory. The basic idea isn't new: define a job as a state change first, then optimize the system that produces it. What I've found useful is making that state change the guiding abstraction for the agent design process.

Job-first design is how we build agents at Tonic. It gives our agents a stable target while we optimize their models, tools, and architecture. It also lets us estimate usage costs with enough accuracy to commit to pricing before building an agent.

How do we test agent routers?

More tools means more possible decisions. Giving every request access to every capability becomes costly for the model to reason over.

One way to mitigate this is to dynamically resolve capabilities at runtime. Instead of giving every request every possible tool, the router determines which capabilities should be available for the task.

But as we built increasingly complex routers, we ran into another problem: how do we ensure the right capabilities are resolved at the right time?

We could assert that a given input should resolve to a particular tool, but this couples the eval to the implementation. If we replace a tool, change the router, or discover a better path to the same outcome, the eval fails even when the agent still does the right thing.

We needed something more stable to test against, which lead us to jobs.

Defining jobs as state changes

The first step in defining a job is to identify the desired state change: what is true before the agent acts, and what should be true after.

The agent's job is to make that transition. How it gets there is an implementation detail unless the path itself is part of the product requirement.

Instead of testing whether the router selected a particular tool, we test whether the agent produced the state required by the job.

The resulting state becomes a stable contract. Models, tools, prompts, routing strategies, and other parts of the architecture can change underneath it.

This avoids rewriting top-level evals whenever the implementation changes. It also leaves room for us to optimize against a fixed target. We can find the smallest model that reliably produces the desired state, compare routing strategies against the same jobs, or fine-tune models for a particular use.

Evals at different levels

Testing the resulting state doesn't mean we stop testing individual components.

A router still needs to resolve the right capabilities. Retrieval still needs to return the right information. Tools still need to behave correctly.

Most evals tells us whether one part of the system behaved as expected; a job eval tells us whether the system accomplished its high-level goal.

Jobs organize user flows

Once we started defining jobs, we quickly discovered how they're useful for guiding an agent's development roadmap and product scope.

Within a state change, there are different ways the user and agent might need to interact. The agent might need to clarify something, develop an idea further, retrieve information, or ask for approval before continuing.

These become user flows.

The job tells us what ultimately needs to be true. The flows describe the different ways the agent might need to get there. We can ask which capabilities each flow actually requires, then those flows can then have their own targeted evals.

The design starts to work backward from the outcome.

Using this approach to design my personal agent

To illustrate how job-first design works in practice, I'll use my personal agent, Pearl, as a case study. You can find the repo here.

I used ChatGPT mostly as a thinking partner with research capabilities and a writing editor. For coding, I used Cursor, which has more robust agentic capabilities. I had Cursor connected to GitHub so it can manage my repos, Vercel for deployments, and Neon for direct database connections. I wanted a single agent for developing my thoughts and then executing on them, and I wanted to have more control over my data and over how the agent understands my intent.

I started with three jobs:

The jobs are the transformations. The states are the objects on either side.

That forced me to define what those states actually mean.

My agent is chat-based, so an initial state usually begins with a message. That message might include a PDF, link, or image. A follow-up also includes conversation history and context like user memory. We'll then classify and route inputs to the right job based on defined objects.

For thinking, I define a well-reasoned idea as one where the important concepts are sufficiently precise and the claims follow from them. We'll define an object to validate an input as a "well-reasoned idea" and configure the agent to block moving into the Writing job until the Thinking job is completed.

Research adds an evidence requirement. Important external claims in a cited answer should be grounded in sources other than the user. Our "well-reasoned idea" object might require research, and the Research job then informs choices about data integrations because its output object, "cited answer," must have sources.

Writing adds a different constraint. An audience-aware communication needs to translate the underlying idea into a form appropriate for a particular reader, medium, and intended effect. For an "audience-aware communication," we'll need to define objects for each type, and configure tooling to reason between defined objects. For example, a blog post will have formatting validation and maybe an LLM scorer that grades its quality. But I also want our agent to model the audience, and then look at both the blog post object and the audience profile object, editing the blog post to make references that are more likely to resonate with the target audience.

Testing states instead of paths

Consider this vague input:

French is weirdly orderly.

Maybe I mean French grammar feels systematic. Maybe French vocabulary feels more discrete than English. Maybe I've already explained what I mean in another conversation.

The right path depends on context.

The agent might ask me to clarify. It might recover what I meant from memory. It might research whether the intuition corresponds to anything in linguistics. Or it might already have enough context to develop the thought directly.

Prescribing one of those paths would make the eval less useful.

What I actually care about is the resulting idea: did the agent resolve the important ambiguity, make "orderly" sufficiently precise, and avoid turning my intuition into an unsupported claim?

If it did, the job succeeded.

The trajectory only matters when the trajectory itself is part of the requirement.

Designing from the state change

Following this reasoning gives us a hierarchy for agent design:

Each layer constrains the one below it, which keeps architecture downstream of the behavior we actually want.

Once the desired state is defined, we can swap models, change routing strategies, add or remove tools, or restructure the agent without changing the thing we're ultimately optimizing for.

This is the part of job-driven design I've found most useful. Agent implementations change constantly. Models improve. Tools get replaced. Routing strategies evolve.

The job gives us something stable to optimize against.

Define the job first, then let everything evolve underneath it.