Eval harness

The eval harness (zeroclaw eval run, crate crates/zeroclaw-eval) runs agent evaluation cases (JSON trace fixtures) through the real agent loop and grades each run against declarative expectations. It is how ZeroClaw guards agent-loop behavior (tool dispatch, multi-turn ordering, response formatting, refusals) against regression.

It is distinct from [agent.eval], the in-loop response-quality scorer. The harness is configured under [eval] and invoked as a CLI subcommand.

Modes

Suite taxonomy

Suites are directories of *.json fixtures (see evals/README.md):

• evals/regression/: must stay at 100% pass. Gated in CI via crates/zeroclaw-eval/tests/regression_suite.rs; a failure blocks merge. This is the default [eval].suite_dir.
• evals/capability/ (planned): hard tasks with a low pass rate; tracked over time, never gated.
• evals/live/ (planned): cases executed against a real provider; never run in CI by default.

Running

# Replay the default regression suite:
zeroclaw eval run

# Point at a specific suite, emit machine-readable JSON:
zeroclaw eval run --suite evals/regression --format json

--suite overrides [eval].suite_dir; --mode overrides [eval].mode. Suite loading is non-recursive: only direct *.json children of the suite directory are cases.

Exit-code contract

zeroclaw eval run exits 0 iff every case passed, and 1 otherwise (any failed check or run error). This is the CI gate: the process exit code is the signal. The same decision is exposed as the pure function SuiteReport::exit_code() so it can be tested at its real boundary.

Case format

Each fixture is an LlmTrace: a model_name, a list of conversation turns (each with a user_input and scripted response steps), and declarative expects. A case is either positive (a behavior that must happen) or negative (a behavior that must NOT happen, e.g. tools_not_used, response_not_contains, max_tool_calls: 0). See evals/README.md for the authoring rules, including the two-experts test and the privacy requirement that fixtures use placeholder identities only.

Expectations that grade the dispatch boundary rather than scripted text (tool_arguments_contain, tool_results_contain, and exact_tool_calls) are documented in crates/zeroclaw-eval/README.md. Reach for them whenever a case claims that a value round-tripped through a tool or that a specific number of tool calls happened; expectations over the final response alone cannot show either, because the replay provider scripts that response itself. The recorded-call list is the canonical dispatch fact. Tool names and aggregate success are derived from it at grading time, so richer boundary evidence does not create a second independently mutable tool-call summary.

Fixture loading fails closed, because a required gate must not certify a case that cannot fail. LlmTrace::from_file() rejects a fixture that declares no conversation turns (the replay would drive the agent zero times and grade its expectations against an empty run), one whose expectation block is omitted or empty, one holding a zero-length entry in a string-backed expectation family (response_contains, response_not_contains, response_matches, tools_used, tools_not_used), and one carrying an unknown top-level or expectation key. The dispatch-boundary families fail closed on the same principle: an empty tool or needle, a vacuous min_tool_calls: 0, and count bounds that contradict each other are all rejected at load. Every rejection names the offending fixture and field.

Admission cannot see one remaining form of vacuity: an assertion that a run producing nothing already satisfies, such as a lone max_tool_calls: 0. The gated suite test grades every committed fixture against an empty run and requires at least one failed check, so a case that certifies no behavior cannot join the required gate.