Skip to content

Prepare traces

On this page you turn the agent logs you already have into traces AdaMAST accepts, and check them locally before any model calls are made.

Generation and the judge use the same loader. Give AdaMAST one .json or .jsonl file, or a directory containing those files, and it normalizes every accepted record to one stable shape before any model call.

🧾 What counts as a trace

A trace is one recorded agent execution: the sequence of steps the agent took while attempting a single task. Concretely, that is the messages, tool calls, and tool results of the run, each with a role and its content, plus the task itself and, when known, how the run ended. One task (or one episode) per trace record; unrelated tasks belong in separate records.

Traces deliberately do not have to arrive in one particular format. The loader auto-detects seven record shapes, from native AdaMAST records and chat-message lists to tau-bench results and Codex CLI sessions (see the shapes table), and converts each of them to the same canonical record. Content matters more than shape: any format is acceptable as long as it preserves the attempted task, the agent's intermediate steps, and the observations that let a careful reader diagnose what went wrong.

A minimal acceptable trace, in the native AdaMAST shape, is a single JSONL line:

{"problem_id": "demo-1", "task": "Return the final calculation", "raw_trajectory": "[USER]\nWhat is 17 * 8?\n\n[ASSISTANT]\n17 * 8 = 126.", "metadata": {"system": "demo-calculator"}}

Save that line as trace.jsonl and run adamast validate trace.jsonl: the report shows one trace in format adamast with no empty trajectories, and the file is ready for adamast generate or adamast judge. The raw_trajectory string is free-form; the [USER]/[ASSISTANT] labels shown here are simply how AdaMAST renders chat messages, not required markup.

✅ The three-step path

  1. Collect traces into one .json or .jsonl file, or a directory of those files. Most existing formats load as-is; see the shapes table below.
  2. Validate: adamast validate ./my-traces
  3. Optional: normalize and read the result yourself: adamast normalize ./my-traces --output ./traces.normalized.jsonl

Make it yours

  • Point step 2 at a single file or a whole directory; both work.
  • Keep your existing export format; the shapes table below shows what the loader already recognizes.
  • Choose where the normalized copy goes with --output.

🔍 Validate before generation

adamast validate ./my-traces

Example report:

{
  "trace_count": 24,
  "files": ["/home/user/my-traces/run.jsonl"],
  "formats": {"messages": 24},
  "empty_trajectories": 0
}

Note

Validation is local and makes no model calls.

Fix missing or empty trajectories before starting generation; an empty trajectory cannot provide useful evidence for drafting or agreement.

🧐 Normalize for inspection

adamast normalize ./my-traces --output ./traces.normalized.jsonl

The normalized file is deterministic UTF-8 JSONL. Inspect it to verify that roles, tool activity, task text, and outcome signals survived import as intended.

Tip

You never have to run this step: both adamast generate and adamast judge perform this normalization automatically for file-based input. Run it when you want to see exactly what the models will see.

The same loaders are importable from Python: load_traces(path) returns the normalized records, load_trace_bundle(path) adds the validation report, and write_normalized_jsonl(traces, path) writes the canonical JSONL file.

🧾 The canonical record

When you control trace export, use this format:

{
  "problem_id": "trace-17",
  "task": "Optional original task or user request",
  "raw_trajectory": "The complete agent, model, and tool trajectory",
  "metadata": {
    "system": "my-agent",
    "split": "evaluation"
  }
}

problem_id and raw_trajectory are the important fields. task may be empty, and metadata may contain any JSON object useful to your experiment. Use a stable, unique problem_id so annotations and artifacts can be traced back to their source.

📦 Accepted containers

JSON

A JSON file may contain one trace object, an array of trace objects, or an envelope with a traces array:

{
  "traces": [
    {
      "problem_id": "trace-1",
      "raw_trajectory": "..."
    }
  ]
}

JSONL

In JSONL, each line is a complete JSON object. Blank lines are ignored.

{"problem_id":"trace-1","task":"...","raw_trajectory":"...","metadata":{}}
{"problem_id":"trace-2","task":"...","raw_trajectory":"...","metadata":{}}

Directory

When the source is a directory, AdaMAST recursively reads every .json and .jsonl file in sorted path order. Other file types are ignored.

🔄 Accepted record shapes

You usually do not need to convert to the canonical record; the loader recognizes these shapes on its own:

Source shape Required signal Normalization behavior
AdaMAST native raw_trajectory Preserves the trajectory and metadata
String trajectory trajectory or trace as a string Moves the string to raw_trajectory
Chat messages messages, or trajectory as a list Renders roles, content, names, and tool calls into a readable trajectory
MAD / MAST-Data envelope mas_name plus trace.trajectory Preserves MAS, model, benchmark, and source trace identifiers in metadata
tau-bench traj, task_id, and reward Renders messages and appends the success/failure outcome
Codex CLI session JSONL events such as session_meta, turn_context, and response_item Combines messages, tool calls, and tool outputs into one trace
Generic event log every JSONL item has event Reads supported run, prompt, response, output, and final-answer events

For ordinary records, trace_id, problem_id, or id can identify the trace. task or prompt can supply the task text. If none of raw_trajectory, trajectory, messages, or trace.trajectory is present, validation fails instead of guessing.

🧼 Trace quality checklist

  • Include the attempted task, not only the final answer.
  • Preserve the observations and tool results that support a failure diagnosis.
  • Keep successful as well as failed trajectories when they clarify boundaries between failure modes.
  • Remove secrets and private data before the trace reaches a model provider.
  • Keep benchmark labels or oracle outcomes out of the trajectory when they would leak the answer to the judge.
  • Split unrelated tasks into separate trace records.

Note

Runtime integrations use a related episode-level trace contract; see Traces and learning after completing the standalone workflow.

➡️ Continue with