Benchmarks

"How good is this agent?" has never had a real answer. Not because measurement is hard, but because agent was never a standardized thing — every framework's agent is a different shape, so no result transfers, no comparison holds, and every eval is a one-off script someone wrote and nobody can rerun.

Axon agents are a standard shape. That makes them benchmarkable — and it makes benchmarks portable, publishable, and comparable, the same as agents and modules.

A benchmark is a controlled experiment

The idea is old-fashioned science: hold everything constant, vary one thing, measure what changes.

A benchmark declares factors — the variables of the experiment. Each factor has a set of values; the runner expands them into a matrix and runs every combination. Swap the model while holding the agent fixed, and you're benchmarking models. Swap the agent while holding the model fixed, and you're benchmarking agents. The harness is the same Axon() runtime used everywhere else — each cell of the matrix applies its combination to the normal boot, so the thing you measure is the thing that runs in production.

Any variable in the stack can be isolated this way: the model, the agent, the config — each is just a factor with values.

A benchmark is a folder

Like everything else in Axon:

my-bench/
├── bench.config.ts     # the contract — factors, measurements, dimensions
├── tests/              # execution — standard Bun test files
│   └── navigation.bench.ts
├── template/           # copied into a fresh workspace for each run
└── README.md           # methodology and limitations

The split matters. Test files contain execution. The config gives the resulting observations stable meaning — what each measurement is, its domain and unit, whether higher is better, how values aggregate. Two people running the same benchmark on different machines produce results that mean the same thing, because the meaning lives in the published contract, not in the script.

Publishable, like everything else

Benchmarks live on the registry alongside agents and modules. Publish one and anyone can run it; browse the registry and see how agents score against benchmarks the community can inspect and rerun. An agent's claim to quality stops being a README assertion and starts being a reproducible result.

Where to go

The Config — factors, dimensions, measurements, and artifacts: the contract in full.

Writing Tests — the execution side: bench.observe(), evidence, and where Bun's test semantics end and scoring begins.