Code- or UI-defined agent evaluations with controlled dependencies, repeated trials, trajectory scoring, and experiment comparison.
Best for: teams that need to simulate multi-step tasks with stubbed services and test whether tool choices, arguments, and outcomes remain reliable across trials.
- Its current testing guide recommends stubbing external dependencies with controlled state so test runs stay reproducible and cannot issue real refunds or messages.
- Supports focused decision checks, end-to-end trajectory evaluations, repeated trials, and case-level experiment comparison.
- Connects reviewed production failures back into the next evaluation dataset and CI regression suite.
- Teams still define the scenario state, scorer, pass threshold, and release decision.
- A passing simulated trajectory is pre-release evidence, not proof that the same behavior improved after release.
