The Config

bench.config.ts is the published contract. It declares what the experiment varies, what it measures, and what those measurements mean. Nothing in it executes — execution lives in tests/.

import { defineBench } from "@axon/types"

export default defineBench({
    description: "Measures how reliably agents navigate an unfamiliar repository.",
    factors: [ ... ],
    dimensions: [ ... ],
    measurements: [ ... ],
    artifacts: [ ... ],
    workspace: { template: "./template", retain: "failed" },
    trials: 3,
})

Factors — what the experiment varies

A factor is one variable with a set of values. The runner takes the Cartesian product of all factors and runs every combination — each combination is a cell.

factors: [
    {
        id: "agent",
        label: "Agent",
        kind: "agent",
        role: "subject",
        values: [
            { id: "navigator",  value: { ref: "../navigator" } },
            { id: "generalist", value: { ref: "../generalist" } },
        ],
    },
    {
        id: "model",
        label: "Model",
        kind: "model",
        role: "control",
        values: [
            { id: "sonnet", value: { engine: { provider: "axon", model: "claude-sonnet-4-6" } } },
            { id: "gpt",    value: { engine: { provider: "openrouter", model: "openai/gpt-5" } } },
        ],
    },
],

Two agents × two models = four cells. Each cell boots the normal Axon() harness with that combination applied and runs the full test suite against it.

role states what each factor is for in the experiment:

• subject — the thing being measured. The benchmark's results are claims about it.
• control — varied deliberately or pinned, to isolate the subject's effect.
• input — the material the subject works on: datasets, fixtures, task sets.

A factor with a single value stays in the matrix as a pinned control — it's part of the record of what was held constant.

kind states what the value is — an agent, a model, a dataset — which is what lets the runner apply it to the harness in the right place.

Dimensions — how results slice

Dimensions are tags that test cases attach to their observations — difficulty, language, category. They don't multiply the matrix; they label what was measured so results can be sliced after the fact.

dimensions: [
    { id: "difficulty", label: "Difficulty", cardinality: "low",
      value: { kind: "category", values: ["easy", "medium", "hard"] } },
],

A factor changes what runs. A dimension describes what ran.

Measurements — what the numbers mean

Every observation a test emits must be declared here. The declaration carries the meaning: type, domain, unit, direction, aggregation.

measurements: [
    {
        id: "success",
        label: "Task success",
        description: "Whether the agent returned the exact requested file.",
        value: { kind: "boolean" },
        objective: "maximize",
        aggregate: "rate",
        required: true,
    },
    {
        id: "files_examined",
        label: "Files examined",
        value: { kind: "number", integer: true, unit: "files", domain: { min: 0 } },
        objective: "minimize",
        aggregate: "mean",
    },
    {
        id: "confidence",
        label: "Confidence over time",
        value: { kind: "number", domain: { min: 0, max: 1 } },
        objective: "maximize",
        aggregate: "last",
        grain: "step",
    },
],

• value — boolean, number (with domain and unit), or category.
• objective — which direction is better. A benchmark where nobody can tell whether 12 is good is not a benchmark.
• aggregate — how raw observations reduce: rate, mean, last, and friends.
• grain — most measurements are per-case; grain: "step" records a series within a case (confidence over time, cost per tick).
• required — a case that never observes this measurement is incomplete, and the runner treats it as such.

This is the half that makes results comparable. The test says what happened; the config says what that means.

Artifacts — the evidence

Declared outputs that tests attach — the agent's final answer, a produced file, a transcript. The harness hashes, persists, and correlates them; a published result carries its evidence with it.

artifacts: [
    { id: "answer", label: "Final answer", role: "output", mediaTypes: ["application/json"] },
],

Workspace — where the subject operates

Each run gets a fresh workspace copied from template/ — the repository to navigate, the files to edit, the mess to clean up. Subjects never share state across runs.

workspace: { template: "./template", retain: "failed" },

retain controls what survives: keep failed runs' workspaces for inspection, discard the rest.

Trials

trials: 3,

Agents are stochastic; one run is an anecdote. trials repeats every cell, and the declared aggregations reduce across them.