The evaluator

Standard interface · name interface-evaluator · v0.1.0-draft · 2026-08-31 · portable

The exchanges for checking a deployed agent against its package. Three: evaluate this, what is the test, and what did you find.

This is the primary interface of the agent evaluator role. Implementing it alone does not grant the role, because the role also carries its derivation rule, its grading limits, its conformance test, and the record its holders sign.

1. Why this exists

Anything that checks agents has to be askable in a known way, or every owner scripts a different harness for every checker. This interface publishes the asking side of that job, so a caller who has never met a particular evaluator can still request an evaluation, read what will be checked before it runs, and get the verdict.

2. The exchanges

  1. Evaluate this. A deployed agent and the package it was built from. The answer is a graded verdict: each case, its result, the records audit, and whether the agent conforms.
  2. What is the test? The test derived from a package, without running it: the cases that would be put and the records that would be audited.
  3. What did you find? The verdict for one past evaluation.

3. The contract

  1. Every exchange is answered or refused with a reason. Silence is not an answer.
  2. The test can be read before it runs. An implementer answers "what is the test" without touching the agent under test, so a caller can see what will be checked first.
  3. A verdict never grades quality. An answer reports what passed, what failed, what was inconclusive, and what is flagged for human review. Asked to score how good an agent's output was, an implementer declines.
  4. No document. The signed evaluation record belongs to the agent evaluator role, not to this interface. An implementer that does not hold the role answers these exchanges and signs nothing.

4. Conformance

Behavioural. A harness asks all three exchanges. Each is answered or refused with a reason. The test is readable before anything runs, and its cases trace to the package. A request to score output quality is declined. A verdict answer distinguishes pass, fail, inconclusive, and flagged.

v0.1.0-draft (2026-08-31): first draft, extracted from the agent evaluator role, which previously carried these exchanges inline.