Skip to content

Generate a taxonomy

On this page you turn a folder of completed traces into a validated failure taxonomy: one command, no agent integration required.

This is the simplest AdaMAST workflow: provide completed traces, select a model provider, and receive a standalone failure taxonomy with the full inter-annotator agreement layer (independent annotators must be able to agree on the codes before the result counts).

Run this first

Commands on this page read the bundled examples. Create them in the directory you are working from with python -m adamast.examples.

🚀 Generate from the CLI

  1. Install AdaMAST from the documentation home.
  2. Prepare a supported JSON or JSONL source; see Prepare traces.
  3. Run adamast validate and inspect the normalized form when importing a new trace format.
  4. Configure one provider as described in Providers and models.
  5. Run:
adamast generate \
  --model gpt-5-nano \
  --traces adamast-examples/traces.jsonl \
  --output ./taxonomy-run

That path comes from python -m adamast.examples, so the command runs as written; substitute your own traces once it works. OpenAI is the default provider, so --provider is only needed to choose a different one.

The --traces value may be one accepted file or a directory. The output must be a directory because AdaMAST writes the public taxonomy, a manifest, a browser view, normalized inputs, and intermediate agreement artifacts.

Make it yours

I want to… Do this
Open the browser field guide after generation add --view (details in Outputs and field guide)
Tune the acceptance gate --max-rounds, --kappa-target, --coverage-floor (see "Configure the gate")
Call it from code instead generate_taxonomy(...) (next section)

🐍 Generate from Python

from adamast import generate_taxonomy

taxonomy = generate_taxonomy(
    "adamast-examples/traces.jsonl",
    "./taxonomy-run",
    provider="openai",
    model="gpt-5-nano",
    open_viewer=True,
)

print(taxonomy["status"])
print(len(taxonomy["codes"]))

The function returns the same dictionary written to taxonomy.json.

Advanced entry points exist for custom pipelines: prepare_taxonomy_for_agreement adapts a layered draft for the agreement program, and build_public_taxonomy assembles the final taxonomy.json from its results. Most workflows only need generate_taxonomy.

🔬 What generation does

1. Normalize traces

Every accepted source is converted to canonical AdaMAST JSONL and recorded with a trace report. This keeps provider prompts independent from the original benchmark or harness format.

2. Draft the taxonomy

The draft engine analyzes the domain, agent roles, and observable failure patterns. It produces three layers of failure codes:

Category Meaning
A System or execution failures that can affect any role
B Role-specific quality failures
C Domain reasoning or cross-role failures

3. Run agreement refinement

Four independent annotators apply the draft to sampled traces. AdaMAST reconciles discovered errors, deliberates over disagreements, rewrites weak definitions, and measures both agreement and coverage. See the Agreement gate for the full decision rule.

4. Publish with an explicit status

The public taxonomy receives one of two statuses:

  • accepted when macro Fleiss kappa and error coverage both meet their targets;
  • review_required when artifacts were produced but the configured gate was not satisfied.

Note

AdaMAST never changes review_required to accepted silently.

Read the certification in context

The manifest's acceptance block reports the final kappa together with its 95% bootstrap confidence interval (final_kappa_ci95) and the number of reconciled errors it was computed over (final_kappa_n_subjects). A kappa from a handful of errors is a noisy estimate; a wide interval means the acceptance verdict rests on limited evidence, so add traces or rounds before treating the taxonomy as certified.

Two further properties keep the certification honest:

  • The published taxonomy is always the exact version the reported metrics measured. A refinement is never applied after the last measurement round.
  • All four annotators run on the same model in separate contexts, so the kappa certifies single-model self-consistency in applying the codes — a meaningful ambiguity check, but weaker than agreement across independent human raters. The manifest records this as annotator_model_diversity: "single-model".

When the annotators deliberated disambiguation rules or anchor examples on the way to agreement, those ship inside taxonomy.json under codebook. The measured agreement was achieved with that codebook available; keep it alongside the codes when you hand the taxonomy to another judge or team.

🎛️ Configure the gate

adamast generate \
  --provider anthropic \
  --model YOUR_MODEL_ID \
  --traces adamast-examples/traces.jsonl \
  --output ./taxonomy-run \
  --max-rounds 5 \
  --kappa-target 0.75 \
  --coverage-floor 0.70

Use --no-early-stop when an experiment requires every configured round even after the stopping conditions are stable.

Warning

Raising a target makes acceptance stricter; it does not automatically make the taxonomy better.

🚥 Interpret the exit code

Exit code Meaning
0 Generation completed and the taxonomy was accepted
3 Generation completed but the taxonomy requires review
2 Input, provider configuration, or pipeline execution failed

Automation should inspect both the process exit code and taxonomy.json.status.

Warning

Do not feed a review_required taxonomy into production judging unless the caller explicitly accepts that risk.

When a fixed one-shot taxonomy stops being enough, move on to the adaptive runtime, which regenerates and refines as traces accumulate.

➡️ Continue with