Skip to content

Choose a judge

This page helps you pick the smallest judge that matches the decision your pipeline needs, and shows the minimal code to run each one.

AdaMAST separates judgment tasks because one prompt should not be expected to perform every kind of evaluation.

🧭 Judge selection table

Judge Input Output Best suited for
Default judge (Selection) trace + taxonomy every supported code (zero or more), with evidence Labeling, comparison, statistics, CI
Single-code judge trace + taxonomy one validated code, evidence, confidence, recovery hint Forcing one primary label per trace
Reflection trace + taxonomy failure points, causal graph, mappings, summary Deep diagnosis, mutation, repair, refinement
Mapping one failure point + taxonomy primary/secondary codes or a proposed missing mode Modular code assignment
Coverage trace or failure point + taxonomy covered, partially covered, or not covered Deciding whether the taxonomy needs expansion
Quality taxonomy + optional support traces per-code issues and overall quality Periodic codebook review
Calibration annotation + evidence + taxonomy validity and evidence support Auditing default-judge annotations
Selection-Summary Reflection output deterministic selection buckets Compressing a causal analysis without another model call

The default judge is the public CLI/API path: adamast judge and create_judge run it unless you ask for a different mode. The specialized controllers live under adamast/judges/ and are used by the adaptive runtime and research pipelines.

Tip

Start with the default judge; reach for a specialized judge only when the table names your exact decision.

🎯 Default judge (Selection)

The default judge selects every failure code the trace evidence supports; a trace can carry several failure modes, one, or none. Each returned mode cites its evidence, and none apply is an explicit valid answer.

adamast judge \
  --provider openai \
  --taxonomy ./taxonomy.json \
  --traces ./traces.jsonl

Judge traces covers the full CLI and Python surface.

🏷️ Single-code judge

Choose the single-code judge when exactly one failure code per trace is required; it classifies each trace into its one best-supported code with a confidence score and a recovery hint.

adamast judge \
  --mode single \
  --provider openai \
  --taxonomy ./taxonomy.json \
  --traces ./traces.jsonl

🪞 Reflection judge

Reflection is the deepest trace judge. It first identifies concrete failure points and builds a backward-grounded causal graph without seeing the taxonomy. Only afterward does it map supported points to taxonomy codes. This ordering reduces the risk that the existing codebook determines what the judge notices.

from adamast.core.taxonomy_data import Taxonomy
from adamast.judges.reflection_judge import AdaMASTReflectionJudge

taxonomy = Taxonomy.from_json("./taxonomy.json")
judge = AdaMASTReflectionJudge(
    taxonomy,
    judge_model="gpt-5",
)

result = judge.analyze({
    "task_id": "trace-17",
    "candidate_id": "agent-run-3",
    "run_id": "evaluation-1",
    "task_prompt": "original task",
    "candidate_output": "the submitted answer",
    "score": 0.0,
    "trace": "complete trajectory"
})

The default two-call mode separates analysis from mapping and then derives the selection summary deterministically. A single-call mode and one validation retry are also available for controlled experiments. Reflection is the end-of-generation validation surface used by the adaptive refinement runtime.

🗺️ Mapping judge

Mapping classifies one already-identified failure point. It can return primary and secondary codes, or mark the point unmapped and propose a missing failure mode.

from adamast.judges import MappingJudge

judge = MappingJudge(taxonomy, judge_model="gpt-5")
result = judge.run({
    "failure_point_id": "fp-2",
    "evidence": "the tool returned an incomplete response",
    "description": "the agent treated the partial result as complete"
})

Choose Mapping when another component already owns failure discovery.

📡 Coverage judge

Coverage decides whether a trace or failure point is covered, partially_covered, or not_covered. For missing patterns it can recommend a new code with a proposed name and definition.

from adamast.judges import CoverageJudge

judge = CoverageJudge(taxonomy, judge_model="gpt-5")
result = judge.run({"trace": "...", "failure_point": {"evidence": "..."}})
print(result.coverage_status, result.suggest_new_code)

Note

Coverage is a focused trigger for expansion; it is not a replacement for the four-annotator agreement metric.

🩺 Quality judge

Quality evaluates the codebook rather than classifying one trace. It reports code-level observability, overlap, scope, and clarity issues plus an overall quality result.

from adamast.judges import QualityJudge

judge = QualityJudge(taxonomy, judge_model="gpt-5")
result = judge.run(support_traces=["trace one", "trace two"])
print(result.overall_quality)

Support traces are optional but make recommendations easier to audit.

⚖️ Calibration judge

Calibration audits an existing annotation against its cited evidence. It can flag weak-evidence/high-confidence assignments, conflicting codes, and likely over-triggers.

from adamast.judges import CalibrationJudge

judge = CalibrationJudge(taxonomy, judge_model="gpt-5")
result = judge.run({
    "annotation": {"code": "A.2", "confidence": "high"},
    "evidence": "the exact supporting span from the trace"
})

🧮 Selection-Summary judge

Selection-Summary is deterministic Python compression of Reflection output. It groups labeled failures into root, attributable, unrecovered, terminal, actionable, and external buckets without making another model call.

Use it when consumers need a compact selection-oriented view but the full causal result must remain available for audit.

🔌 Shared controller contract

The Selection, Mapping, Coverage, Quality, and Calibration judges share JudgeController:

from adamast.judges import JudgeController

controller = JudgeController(
    "selection",
    taxonomy,
    judge_model="gpt-5",
)
result = controller.run("trace text")

Each controller validates structural fields and enums, returns a frozen result dataclass (SelectionJudgeResult, MappingJudgeResult, CoverageJudgeResult, QualityJudgeResult, or CalibrationJudgeResult), and includes judge_metadata with the judge name, model, taxonomy version, timestamp, and warnings. One-off helpers run_selection, run_mapping, run_coverage, run_quality, and run_calibration wrap construction and a single call. Minor schema omissions are salvaged where a trustworthy partial result exists; model-call and repair behavior routes through the runtime learning-call layer.

📝 Natural-language prompt assets

The five simple judge instructions are Markdown assets under adamast/judges/assets/<judge>/system.md and user.md. A custom harness can either use the Python controller or load/render those assets directly via load_judge_definition and render_judge_prompt. Reflection has separate staged prompt assets under adamast/judges/reflection_judge/assets/.

Warning

Keep prompt changes versioned with evaluation results. Changing a judge prompt changes the measurement procedure even when the taxonomy is unchanged.

➡️ Continue with