[{"data":1,"prerenderedAt":684},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fv2\u002Fbench\u002Fresults":3},{"id":4,"title":5,"body":6,"description":676,"extension":677,"meta":678,"navigation":679,"path":680,"seo":681,"stem":682,"__hash__":683},"docs\u002Fdocs\u002Fv2\u002Fbench\u002Fresults.md","Results",{"type":7,"value":8,"toc":668},"minimark",[9,13,17,20,56,59,64,165,168,172,175,253,256,268,272,275,281,355,358,364,470,473,477,480,514,517,524,531,535,538,571,589,592,596,610,646,661,664],[10,11,5],"h1",{"id":12},"results",[14,15,16],"p",{},"A run executes every test in every cell, persists what happened as events, and then\nderives its result by replaying that log.",[14,18,19],{},"The order matters. Execution writes evidence; the result is a reading of the evidence. Any\nresult can be rebuilt later without re-running anything:",[21,22,27],"pre",{"className":23,"code":24,"language":25,"meta":26,"style":26},"language-bash shiki shiki-themes dark-plus","axon bench result \u003Crun-id>\n","bash","",[28,29,30],"code",{"__ignoreMap":26},[31,32,35,39,43,46,50,53],"span",{"class":33,"line":34},"line",1,[31,36,38],{"class":37},"sCudf","axon",[31,40,42],{"class":41},"sKc5r"," bench",[31,44,45],{"class":41}," result",[31,47,49],{"class":48},"sTNBD"," \u003C",[31,51,52],{"class":41},"run-i",[31,54,55],{"class":48},"d>\n",[14,57,58],{},"Same log, same projection, same numbers — six months later, on a different machine.",[60,61,63],"h2",{"id":62},"what-a-run-produces","What a run produces",[21,65,69],{"className":66,"code":67,"language":68,"meta":26,"style":26},"language-ts shiki shiki-themes dark-plus","{\n    manifest,        \u002F\u002F what was run: schema, cells, pins, harness identity\n    trials,          \u002F\u002F one record per test × cell × trial\n    observations,    \u002F\u002F every value you recorded\n    artifacts,       \u002F\u002F content-addressed evidence\n    sessions,        \u002F\u002F the full event stream per trial\n    faults,          \u002F\u002F harness failures, typed\n    coverage,        \u002F\u002F expected vs executed vs completed, and what is missing\n}\n","ts",[28,70,71,76,90,102,114,126,137,148,159],{"__ignoreMap":26},[31,72,73],{"class":33,"line":34},[31,74,75],{"class":48},"{\n",[31,77,79,83,86],{"class":33,"line":78},2,[31,80,82],{"class":81},"s3F5K","    manifest",[31,84,85],{"class":48},",        ",[31,87,89],{"class":88},"sOLPB","\u002F\u002F what was run: schema, cells, pins, harness identity\n",[31,91,93,96,99],{"class":33,"line":92},3,[31,94,95],{"class":81},"    trials",[31,97,98],{"class":48},",          ",[31,100,101],{"class":88},"\u002F\u002F one record per test × cell × trial\n",[31,103,105,108,111],{"class":33,"line":104},4,[31,106,107],{"class":81},"    observations",[31,109,110],{"class":48},",    ",[31,112,113],{"class":88},"\u002F\u002F every value you recorded\n",[31,115,117,120,123],{"class":33,"line":116},5,[31,118,119],{"class":81},"    artifacts",[31,121,122],{"class":48},",       ",[31,124,125],{"class":88},"\u002F\u002F content-addressed evidence\n",[31,127,129,132,134],{"class":33,"line":128},6,[31,130,131],{"class":81},"    sessions",[31,133,85],{"class":48},[31,135,136],{"class":88},"\u002F\u002F the full event stream per trial\n",[31,138,140,143,145],{"class":33,"line":139},7,[31,141,142],{"class":81},"    faults",[31,144,98],{"class":48},[31,146,147],{"class":88},"\u002F\u002F harness failures, typed\n",[31,149,151,154,156],{"class":33,"line":150},8,[31,152,153],{"class":81},"    coverage",[31,155,85],{"class":48},[31,157,158],{"class":88},"\u002F\u002F expected vs executed vs completed, and what is missing\n",[31,160,162],{"class":33,"line":161},9,[31,163,164],{"class":48},"}\n",[14,166,167],{},"No aggregates. No charts. Both are projections over this record, and storing a projection\nnext to the evidence for it is how the two drift apart.",[60,169,171],{"id":170},"physics-come-free","Physics come free",[14,173,174],{},"Every trial records what it cost, derived from the session log rather than declared by\nyou:",[176,177,178,189],"table",{},[179,180,181],"thead",{},[182,183,184,187],"tr",{},[185,186],"th",{},[185,188],{},[190,191,192,203,213,223,233,243],"tbody",{},[182,193,194,200],{},[195,196,197],"td",{},[28,198,199],{},"durationMs",[195,201,202],{},"wall clock for the trial",[182,204,205,210],{},[195,206,207],{},[28,208,209],{},"tokens",[195,211,212],{},"input and output, summed across every engine call",[182,214,215,220],{},[195,216,217],{},[28,218,219],{},"costUsd",[195,221,222],{},"real spend, when the provider prices the call",[182,224,225,230],{},[195,226,227],{},[28,228,229],{},"engineCalls",[195,231,232],{},"how many times the model was consulted",[182,234,235,240],{},[195,236,237],{},[28,238,239],{},"toolCalls",[195,241,242],{},"how many times it reached for the world",[182,244,245,250],{},[195,246,247],{},[28,248,249],{},"errors",[195,251,252],{},"engine and run failures",[14,254,255],{},"You declare none of this. It falls out of the events the runtime already emits, which\nmeans it is available for every benchmark ever written, including ones whose authors never\nthought about cost.",[14,257,258,259,263,264,267],{},"Physics are split between ",[260,261,262],"strong",{},"subject"," and ",[260,265,266],{},"assessor",". When a benchmark uses a model to\njudge another model's output, the judge's token spend is recorded separately — otherwise\nthe cost of measuring would be indistinguishable from the cost of the thing measured.",[60,269,271],{"id":270},"failure-and-low-scores-are-different-facts","Failure and low scores are different facts",[14,273,274],{},"A trial that failed is not a trial that scored badly, and the record keeps them apart.",[14,276,277,280],{},[260,278,279],{},"Faults"," are harness failures, and they are typed:",[176,282,283,293],{},[179,284,285],{},[182,286,287,290],{},[185,288,289],{},"Code",[185,291,292],{},"Meaning",[190,294,295,305,315,325,335,345],{},[182,296,297,302],{},[195,298,299],{},[28,300,301],{},"boot",[195,303,304],{},"the agent never started",[182,306,307,312],{},[195,308,309],{},[28,310,311],{},"timeout",[195,313,314],{},"the trial exceeded its budget",[182,316,317,322],{},[195,318,319],{},[28,320,321],{},"budget",[195,323,324],{},"the run exhausted its spend limit",[182,326,327,332],{},[195,328,329],{},[28,330,331],{},"cancelled",[195,333,334],{},"stopped deliberately",[182,336,337,342],{},[195,338,339],{},[28,340,341],{},"process",[195,343,344],{},"the subprocess died",[182,346,347,352],{},[195,348,349],{},[28,350,351],{},"protocol",[195,353,354],{},"the harness and the runtime disagreed",[14,356,357],{},"A fault excludes the trial from scoring. It is not a zero, because a zero is a claim about\nthe agent and a fault is a statement about the experiment.",[14,359,360,363],{},[260,361,362],{},"Coverage"," tracks the gap between what should have happened and what did:",[21,365,367],{"className":66,"code":366,"language":68,"meta":26,"style":26},"{\n    expectedTrials: 12,\n    executedTrials: 12,\n    completedTrials: 11,\n    expectedMeasurements: 24,\n    observedMeasurements: 21,\n    missing: { fault: 2, not_emitted: 1 },\n}\n",[28,368,369,373,389,400,412,424,436,466],{"__ignoreMap":26},[31,370,371],{"class":33,"line":34},[31,372,75],{"class":48},[31,374,375,379,382,386],{"class":33,"line":78},[31,376,378],{"class":377},"sqDPy","    expectedTrials",[31,380,381],{"class":48},": ",[31,383,385],{"class":384},"sDGX8","12",[31,387,388],{"class":48},",\n",[31,390,391,394,396,398],{"class":33,"line":92},[31,392,393],{"class":377},"    executedTrials",[31,395,381],{"class":48},[31,397,385],{"class":384},[31,399,388],{"class":48},[31,401,402,405,407,410],{"class":33,"line":104},[31,403,404],{"class":377},"    completedTrials",[31,406,381],{"class":48},[31,408,409],{"class":384},"11",[31,411,388],{"class":48},[31,413,414,417,419,422],{"class":33,"line":116},[31,415,416],{"class":377},"    expectedMeasurements",[31,418,381],{"class":48},[31,420,421],{"class":384},"24",[31,423,388],{"class":48},[31,425,426,429,431,434],{"class":33,"line":128},[31,427,428],{"class":377},"    observedMeasurements",[31,430,381],{"class":48},[31,432,433],{"class":384},"21",[31,435,388],{"class":48},[31,437,438,441,444,447,449,452,455,458,460,463],{"class":33,"line":139},[31,439,440],{"class":377},"    missing",[31,442,443],{"class":48},": { ",[31,445,446],{"class":377},"fault",[31,448,381],{"class":48},[31,450,451],{"class":384},"2",[31,453,454],{"class":48},", ",[31,456,457],{"class":377},"not_emitted",[31,459,381],{"class":48},[31,461,462],{"class":384},"1",[31,464,465],{"class":48}," },\n",[31,467,468],{"class":33,"line":150},[31,469,164],{"class":48},[14,471,472],{},"Most published benchmark numbers cannot distinguish \"scored zero\" from \"never ran\". This\none can, and the distinction is in the artifact rather than in a footnote someone wrote\nafterwards.",[60,474,476],{"id":475},"the-manifest-is-what-makes-a-result-comparable","The manifest is what makes a result comparable",[14,478,479],{},"A result without a manifest is a number with no provenance. The manifest records:",[481,482,483,490,496,502,508],"ul",{},[484,485,486,489],"li",{},[260,487,488],{},"The benchmark"," — name, version, content hash",[484,491,492,495],{},[260,493,494],{},"The schema"," — every measurement declared, hashed",[484,497,498,501],{},[260,499,500],{},"The cells"," — every axis, including the ones held constant, each pinned to an\nimmutable version or content hash",[484,503,504,507],{},[260,505,506],{},"The workspace"," — hash, file count, byte count",[484,509,510,513],{},[260,511,512],{},"The harness"," — Axon version, Bun version, protocol version",[14,515,516],{},"Held constants are in there for the same reason as varied ones. A score means nothing\nwithout knowing what did not move alongside what did.",[14,518,519,520,523],{},"Pins are immutable identities, never mutable references. A cell that says\n",[28,521,522],{},"model=claude-sonnet-4.6"," records the resolved engine, not the tag that pointed at it, so\na result stays interpretable after the tag has moved on.",[14,525,526,527,530],{},"Secrets never enter the manifest. Values are pinned by hash, and an ",[28,528,529],{},"env"," axis records\nthat an environment varied without recording what was in it.",[60,532,534],{"id":533},"retaining-the-world","Retaining the world",[14,536,537],{},"When a trial fails, the workspace it failed in is worth more than any number about it.",[21,539,541],{"className":66,"code":540,"language":68,"meta":26,"style":26},"workspace: { source: \".\u002Fworkspace\", retain: \"failed\" }\n",[28,542,543],{"__ignoreMap":26},[31,544,545,548,550,553,555,558,560,563,565,568],{"class":33,"line":34},[31,546,547],{"class":377},"workspace",[31,549,443],{"class":48},[31,551,552],{"class":377},"source",[31,554,381],{"class":48},[31,556,557],{"class":41},"\".\u002Fworkspace\"",[31,559,454],{"class":48},[31,561,562],{"class":377},"retain",[31,564,381],{"class":48},[31,566,567],{"class":41},"\"failed\"",[31,569,570],{"class":48}," }\n",[14,572,573,576,577,580,581,584,585,588],{},[28,574,575],{},"failed"," keeps the world of any trial that did not complete, ",[28,578,579],{},"always"," keeps every one, and\n",[28,582,583],{},"never"," keeps none. The failed world sits under ",[28,586,587],{},".bench\u002F"," exactly as the agent left it —\nthe half-applied patch, the file it should not have touched, the dependency it installed.",[14,590,591],{},"Workspace changes are captured as part of the trial record too, so a diff is available\neven when the world itself was not retained.",[60,593,595],{"id":594},"reading-a-run","Reading a run",[21,597,599],{"className":23,"code":598,"language":25,"meta":26,"style":26},"axon bench run\n",[28,600,601],{"__ignoreMap":26},[31,602,603,605,607],{"class":33,"line":34},[31,604,38],{"class":37},[31,606,42],{"class":41},[31,608,609],{"class":41}," run\n",[21,611,613],{"className":23,"code":612,"language":25,"meta":26,"style":26},"Benchmark run 01JQZX8K2M4N6P8R\n  4 passed, 2 failed, 12 observations\n",[28,614,615,626],{"__ignoreMap":26},[31,616,617,620,623],{"class":33,"line":34},[31,618,619],{"class":37},"Benchmark",[31,621,622],{"class":41}," run",[31,624,625],{"class":41}," 01JQZX8K2M4N6P8R\n",[31,627,628,631,634,637,640,643],{"class":33,"line":78},[31,629,630],{"class":37},"  4",[31,632,633],{"class":41}," passed,",[31,635,636],{"class":384}," 2",[31,638,639],{"class":41}," failed,",[31,641,642],{"class":384}," 12",[31,644,645],{"class":41}," observations\n",[21,647,649],{"className":23,"code":648,"language":25,"meta":26,"style":26},"axon bench result 01JQZX8K2M4N6P8R\n",[28,650,651],{"__ignoreMap":26},[31,652,653,655,657,659],{"class":33,"line":34},[31,654,38],{"class":37},[31,656,42],{"class":41},[31,658,45],{"class":41},[31,660,625],{"class":41},[14,662,663],{},"The full record as JSON — every trial, every observation, every fault, with the manifest\nthat makes it mean something.",[665,666,667],"style",{},"html pre.shiki code .sCudf, html code.shiki .sCudf{--shiki-default:#DCDCAA}html pre.shiki code .sKc5r, html code.shiki .sKc5r{--shiki-default:#CE9178}html pre.shiki code .sTNBD, html code.shiki .sTNBD{--shiki-default:#D4D4D4}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s3F5K, html code.shiki .s3F5K{--shiki-default:#9CDCFE}html pre.shiki code .sOLPB, html code.shiki .sOLPB{--shiki-default:#6A9955}html pre.shiki code .sqDPy, html code.shiki .sqDPy{--shiki-default:#C8C8C8}html pre.shiki code .sDGX8, html code.shiki .sDGX8{--shiki-default:#B5CEA8}",{"title":26,"searchDepth":78,"depth":78,"links":669},[670,671,672,673,674,675],{"id":62,"depth":78,"text":63},{"id":170,"depth":78,"text":171},{"id":270,"depth":78,"text":271},{"id":475,"depth":78,"text":476},{"id":533,"depth":78,"text":534},{"id":594,"depth":78,"text":595},"What a run produces, why physics come free, and why aggregates are derived rather than stored.","md",{},true,"\u002Fdocs\u002Fv2\u002Fbench\u002Fresults",{"title":5,"description":676},"docs\u002Fv2\u002Fbench\u002Fresults","Segm735WVrj2UkVZq3bQkf0_0unNb1bG-BTREpIa6zI",1785237008191]