What Are Deep Agents?

For deep agents to succeed, their evaluations need to be task aware. There are many datapoints that represent different tools, constraints, and definitions of success, so we have to customize the checks that define what we care about for each task. In some cases, we might put more emphasis on output verification. In others, what tools the agent calls, the sequence it follows, and the policies it optimizes for might be what you emphasize.

In practice, the best way to create these tests is to start from first principles and think through what you care about, how you can quantify it, and what concrete labels or signals are returned by the environment to help with the quantification. Once you come up with measurements, create small example tests and gradually iterate.

Observability is key when evaluating agent behavior. Single-step evaluations help diagnose which step in the agent’s reasoning process failed, full-turn evaluations help determine what user queries the agent has trouble addressing, and multi-turn evaluation can determine where across a full conversation the agent broke. Across all three levels of evaluation, there are certain principles that apply: 

• Practice eval-driven development: Define evals for planned capabilities before building them, not after.
• Build datasets from real failures: Synthetic coverage is useful, but production failures can catch what you didn’t anticipate.
• Start with tracing, not scoring: Manually review agent traces first—what you observe informs which metrics actually matter.

As you migrate agents into production, consider starting from the minimal viable setup an agent requires to succeed in your environment. Agents can be connected to a lot of capabilities and tools, but starting with the most basic setup enables you to test your agent observability, discern what capabilities are actually essential for success on your tasks, and balance potentially compute-expensive tradeoffs.

Due to the nature of production environments and the importance of reliability and other key factors, it is further recommended that, where appropriate, you analyze what verified and vetted components are offered by the ecosystem rather than defaulting to building bespoke features or tools.

When you are routing to multiple specialized deep agents, you should use a top-level orchestrator to classify the incoming task, select the best specialist agent, and forward the request along with any needed tools, policies, and context. By working to measure intent and capability, this layer matches multiple expert agents as needed to fulfill the user’s task or query.

The routing decision itself can be based on different factors such as intent, domain, required tools, complexity, latency targets, or policy constraints. This works as long as each agent is described by a clear capability profile, including what tasks it handles well and what tells it has access to. If the orchestrator’s confidence is low, the system can fall back to a more general agent or ask for clarification.