The evaluator

name interface-evaluatorupdated 2026-08-31portable

ROLES.md describes this interface in section 13.1, as the primary interface of the conformance checker. Software reads its definition from interface-evaluator.json.

The evaluator interface holds the exchanges for checking a deployed agent against its package. There are three: evaluate this, what is the test, and what did you find.

This is the primary interface of the conformance checker role. Implementing it alone does not grant the role, because the role also carries its derivation rule, its grading limits, its own check, and the record its holders sign.

1. The purpose

Anything that checks agents has to be askable in a known way, or every owner scripts a different harness for every checker. This interface publishes the asking side of that job, so a caller who has never met a particular evaluator can still request an evaluation, read what will be checked before it runs, and get the verdict.

2. The exchanges

  1. Evaluate this. The caller names a deployed agent and the package it was built from. The answer is a graded verdict: each case, its result, the records audit, and whether the agent passes.
  2. What is the test? The answer is the test derived from a package, given without running it: the cases that would be put and the records that would be audited.
  3. What did you find? The answer is the verdict for one past evaluation.

3. What it does

  1. Every exchange is answered or refused with a reason. Silence is not an answer.
  2. The test can be read before it runs. An implementer answers "what is the test" without touching the agent under test, so a caller can see what will be checked first.
  3. A verdict never grades quality. An answer reports what passed, what failed, what was inconclusive, and what is flagged for human review. Asked to score how good an agent's output was, an implementer declines.
  4. There is no document. The signed evaluation record belongs to the conformance checker role, and this interface defines none. An implementer that does not hold the role answers these exchanges and signs nothing.

4. How an implementer is checked

The test is behavioural. A harness asks all three exchanges. Each is answered or refused with a reason. The test is readable before anything runs, and its cases trace to the package. A request to score output quality is declined. A verdict answer distinguishes pass, fail, inconclusive, and flagged.

v0.1.0-draft (2026-08-31): this is the first draft, extracted from the agent evaluator role, which previously carried these exchanges inline.