fixtures/

fixtures/ is for everything the benchmark needs and the agent must not have.

my-bench/
└── fixtures/
    ├── expected/
    │   └── parser.ts       # the reference solution
    ├── rubric.md           # grading criteria for a judge
    └── seed.sql            # data for setup()

Expected outputs, grading rubrics, judge prompts, seed data, comparison baselines. Read by setup() and by your tests; never copied into the world the agent runs in.

Why it is a separate directory

Two reasons, and both are about keeping results honest.

The agent could read the answer. A reference solution sitting in the workspace is findable by anything with a filesystem tool. A benchmark that leaks its own answer key is measuring retrieval, not capability — and it would do so silently, producing high scores that look like good news.

Reproducibility. The workspace is copied and the copy is the agent's root, so its reachable filesystem is bounded and identical every run. Fixtures are read from the benchmark directory by the harness, which means they cannot become part of what the subject depends on. Move the benchmark, restructure the repo, clone it onto another machine — the agent's world is unchanged.

Using them

From setup(), which runs after the workspace copy and before the agent boots:

export default defineBench<Schema>({
    async setup() {
        await seedDatabase(await Bun.file("./fixtures/seed.sql").text())
    },
})

From a test, for grading:

it("matches the reference implementation", async ({ workspace }) => {
    const { axon } = await Axon()
    await axon.request("Implement parseLine in src/parser.ts")

    const expected = await Bun.file("./fixtures/expected/parser.ts").text()
    const actual = await workspace.read("src/parser.ts")

    observe("exact_match", actual.trim() === expected.trim())
})

The test can read both sides. The agent can only read one.

Judges

When a benchmark grades with a model rather than an assertion, the rubric belongs here:

const rubric = await Bun.file("./fixtures/rubric.md").text()

Two things matter about judges beyond where the rubric lives. The judge must be held constant across the matrix — if the assessor varies alongside the subject, the measuring instrument is changing with the thing being measured, and the numbers compare nothing. And its token spend is recorded separately from the subject's, so the cost of grading never contaminates the cost of the work.

What does not belong here

Anything the task genuinely requires. If the agent needs a config file, a dataset, or a dependency to do its job, that is part of its world and belongs in workspace/.

The test is simple: would a person doing this task by hand need it? If yes, it is the workspace. If it only exists to check their work, it is a fixture.