[{"data":1,"prerenderedAt":602},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fv2\u002Fbench\u002Fmeasuring":3},{"id":4,"title":5,"body":6,"description":595,"extension":596,"meta":597,"navigation":315,"path":598,"seo":599,"stem":600,"__hash__":601},"docs\u002Fdocs\u002Fv2\u002Fbench\u002Fmeasuring.md","Measuring",{"type":7,"value":8,"toc":586},"minimark",[9,13,22,25,28,33,44,84,92,96,106,181,184,190,234,240,244,254,268,272,275,405,408,411,475,481,484,488,495,501,507,510,514,527,530,534,541,575,582],[10,11,5],"h1",{"id":12},"measuring",[14,15,16,17,21],"p",{},"A test suite says: ",[18,19,20],"em",{},"these assertions must hold, or I am unhappy."," It encodes something you\nalready know should be true, and stopping at the first violation is the correct response.",[14,23,24],{},"A benchmark cannot say that. It exists because you do not know what will happen — that is\nthe entire reason to run it. Halting on the first surprise throws away the data you built\nthe experiment to collect.",[14,26,27],{},"So there are two verbs.",[29,30,32],"h2",{"id":31},"expect-is-for-the-harness","expect is for the harness",[14,34,35,39,40,43],{},[36,37,38],"code",{},"expect"," asks whether the ",[18,41,42],{},"scenario"," held. The workspace copied, the agent booted, the\nfile the task refers to is actually there.",[45,46,51],"pre",{"className":47,"code":48,"language":49,"meta":50,"style":50},"language-ts shiki shiki-themes dark-plus","expect(workspace).toHaveFile(\"src\u002Fparser.ts\")\n","ts","",[36,52,53],{"__ignoreMap":50},[54,55,58,61,65,69,72,75,77,81],"span",{"class":56,"line":57},"line",1,[54,59,38],{"class":60},"sCudf",[54,62,64],{"class":63},"sTNBD","(",[54,66,68],{"class":67},"s3F5K","workspace",[54,70,71],{"class":63},").",[54,73,74],{"class":60},"toHaveFile",[54,76,64],{"class":63},[54,78,80],{"class":79},"sKc5r","\"src\u002Fparser.ts\"",[54,82,83],{"class":63},")\n",[14,85,86,87,91],{},"If that fails, this run measured nothing. The world was not what the scenario assumed, so\nthe trial is ",[88,89,90],"strong",{},"excluded"," — not scored zero. A zero would be a claim about the agent; an\nexclusion is the truth, which is that nothing was tested.",[29,93,95],{"id":94},"observe-is-for-the-subject","observe is for the subject",[14,97,98,101,102,105],{},[36,99,100],{},"observe"," records how the ",[18,103,104],{},"subject"," did. It never fails and never halts.",[45,107,109],{"className":47,"code":108,"language":49,"meta":50,"style":50},"observe(\"resolved\", await workspace.tests.pass())\nobserve(\"collateral\", (await workspace.changed()).length - 1)\n",[36,110,111,144],{"__ignoreMap":50},[54,112,113,115,117,120,123,127,130,133,136,138,141],{"class":56,"line":57},[54,114,100],{"class":60},[54,116,64],{"class":63},[54,118,119],{"class":79},"\"resolved\"",[54,121,122],{"class":63},", ",[54,124,126],{"class":125},"sYbnZ","await",[54,128,129],{"class":67}," workspace",[54,131,132],{"class":63},".",[54,134,135],{"class":67},"tests",[54,137,132],{"class":63},[54,139,140],{"class":60},"pass",[54,142,143],{"class":63},"())\n",[54,145,147,149,151,154,157,159,161,163,166,169,172,175,179],{"class":56,"line":146},2,[54,148,100],{"class":60},[54,150,64],{"class":63},[54,152,153],{"class":79},"\"collateral\"",[54,155,156],{"class":63},", (",[54,158,126],{"class":125},[54,160,129],{"class":67},[54,162,132],{"class":63},[54,164,165],{"class":60},"changed",[54,167,168],{"class":63},"()).",[54,170,171],{"class":67},"length",[54,173,174],{"class":63}," - ",[54,176,178],{"class":177},"sDGX8","1",[54,180,83],{"class":63},[14,182,183],{},"A model that fails every single trial is a finding. It is very often the finding. A\nbenchmark that reported itself broken because the thing it measured performed badly would\nbe useless for the exact comparisons it exists to make.",[14,185,186,187,189],{},"There is a second reason they cannot merge. ",[36,188,38],{}," throws on first failure:",[45,191,193],{"className":47,"code":192,"language":49,"meta":50,"style":50},"expect(workspace).toPassTests()          \u002F\u002F fails, throws\nexpect(workspace).toHaveChangedOnly([])  \u002F\u002F never runs\n",[36,194,195,215],{"__ignoreMap":50},[54,196,197,199,201,203,205,208,211],{"class":56,"line":57},[54,198,38],{"class":60},[54,200,64],{"class":63},[54,202,68],{"class":67},[54,204,71],{"class":63},[54,206,207],{"class":60},"toPassTests",[54,209,210],{"class":63},"()          ",[54,212,214],{"class":213},"sOLPB","\u002F\u002F fails, throws\n",[54,216,217,219,221,223,225,228,231],{"class":56,"line":146},[54,218,38],{"class":60},[54,220,64],{"class":63},[54,222,68],{"class":67},[54,224,71],{"class":63},[54,226,227],{"class":60},"toHaveChangedOnly",[54,229,230],{"class":63},"([])  ",[54,232,233],{"class":213},"\u002F\u002F never runs\n",[14,235,236,237,239],{},"You would lose every measurement after the first one — precisely on the runs where you\nmost want them. ",[36,238,100],{}," collects all of them, always.",[29,241,243],{"id":242},"the-rule","The rule",[14,245,246],{},[88,247,248,250,251,253],{},[36,249,38],{}," guards the experiment. ",[36,252,100],{}," records the result.",[14,255,256,257,259,260,262,263,265,266,132],{},"If you find yourself wanting ",[36,258,38],{}," to not-fail, you wanted ",[36,261,100],{},". If you find\nyourself checking that the agent did well, that is ",[36,264,100],{},". If you are checking that the\nbenchmark itself is functioning, that is ",[36,267,38],{},[29,269,271],{"id":270},"declaring-what-can-be-observed","Declaring what can be observed",[14,273,274],{},"What a benchmark records is a TypeScript type:",[45,276,278],{"className":47,"code":277,"language":49,"meta":50,"style":50},"type Schema = {\n    \u002F** Has the agent resolved the bug? *\u002F\n    resolved: boolean\n\n    \u002F** Files edited beyond the target. @objective minimize *\u002F\n    collateral: number\n\n    \u002F** Strategy the agent took. *\u002F\n    approach: \"refactor\" | \"patch\" | \"rewrite\"\n}\n\nexport default defineBench\u003CSchema>({ ... })\n",[36,279,280,293,298,310,317,329,340,345,351,373,379,384],{"__ignoreMap":50},[54,281,282,286,290],{"class":56,"line":57},[54,283,285],{"class":284},"scz_3","type",[54,287,289],{"class":288},"sNl3T"," Schema",[54,291,292],{"class":63}," = {\n",[54,294,295],{"class":56,"line":146},[54,296,297],{"class":213},"    \u002F** Has the agent resolved the bug? *\u002F\n",[54,299,301,304,307],{"class":56,"line":300},3,[54,302,303],{"class":67},"    resolved",[54,305,306],{"class":63},": ",[54,308,309],{"class":288},"boolean\n",[54,311,313],{"class":56,"line":312},4,[54,314,316],{"emptyLinePlaceholder":315},true,"\n",[54,318,320,323,326],{"class":56,"line":319},5,[54,321,322],{"class":213},"    \u002F** Files edited beyond the target. ",[54,324,325],{"class":284},"@objective",[54,327,328],{"class":213}," minimize *\u002F\n",[54,330,332,335,337],{"class":56,"line":331},6,[54,333,334],{"class":67},"    collateral",[54,336,306],{"class":63},[54,338,339],{"class":288},"number\n",[54,341,343],{"class":56,"line":342},7,[54,344,316],{"emptyLinePlaceholder":315},[54,346,348],{"class":56,"line":347},8,[54,349,350],{"class":213},"    \u002F** Strategy the agent took. *\u002F\n",[54,352,354,357,359,362,365,368,370],{"class":56,"line":353},9,[54,355,356],{"class":67},"    approach",[54,358,306],{"class":63},[54,360,361],{"class":79},"\"refactor\"",[54,363,364],{"class":63}," | ",[54,366,367],{"class":79},"\"patch\"",[54,369,364],{"class":63},[54,371,372],{"class":79},"\"rewrite\"\n",[54,374,376],{"class":56,"line":375},10,[54,377,378],{"class":63},"}\n",[54,380,382],{"class":56,"line":381},11,[54,383,316],{"emptyLinePlaceholder":315},[54,385,387,390,393,396,399,402],{"class":56,"line":386},12,[54,388,389],{"class":125},"export",[54,391,392],{"class":125}," default",[54,394,395],{"class":60}," defineBench",[54,397,398],{"class":63},"\u003C",[54,400,401],{"class":288},"Schema",[54,403,404],{"class":63},">({ ... })\n",[14,406,407],{},"A type, not a config block, because a type is the right tool for the job. A doc comment is\nwhere a description naturally goes. A union is how anyone would write a fixed set of\ncategories. And the moment you model types in JSON, you are building a worse TypeScript\ninside your config file.",[14,409,410],{},"Everything is derived from it:",[412,413,414,427],"table",{},[415,416,417],"thead",{},[418,419,420,424],"tr",{},[421,422,423],"th",{},"From",[421,425,426],{},"To",[428,429,430,439,447,455,465],"tbody",{},[418,431,432,436],{},[433,434,435],"td",{},"property name",[433,437,438],{},"the measurement id",[418,440,441,444],{},[433,442,443],{},"property type",[433,445,446],{},"boolean, number, or category",[418,448,449,452],{},[433,450,451],{},"doc comment",[433,453,454],{},"what it means to a reader",[418,456,457,462],{},[433,458,459],{},[36,460,461],{},"@objective minimize",[433,463,464],{},"which direction is better",[418,466,467,472],{},[433,468,469],{},[36,470,471],{},"@unit ms",[433,473,474],{},"display suffix",[14,476,477,480],{},[36,478,479],{},"observe(\"resolved\", true)"," is checked against this. A typo in the id is a compile error,\nnot a silent gap discovered later when the leaderboard has a hole in it.",[14,482,483],{},"Aggregation defaults from the type — booleans aggregate as a rate, numbers as a mean,\ncategories as counts — and can be overridden per measurement when the default is wrong.",[29,485,487],{"id":486},"why-the-schema-is-not-only-a-type","Why the schema is not only a type",[14,489,490,491,494],{},"Types vanish at compile time. This one is extracted during ",[36,492,493],{},"axon prepare"," and written into\nthe run manifest, because two things need it at runtime:",[14,496,497,500],{},[88,498,499],{},"Coverage."," A benchmark can only report \"expected four measurements, observed three\" if\nit knows the fourth exists. A measurement that never fired is invisible unless it was\ndeclared.",[14,502,503,506],{},[88,504,505],{},"Comparability."," Results from different people enter a shared aggregate only if their\nschemas match. The manifest carries the schema hash; a benchmark that changed what it\nmeasures is a different benchmark, and results across the change should not be silently\naveraged.",[14,508,509],{},"You write a type. The system keeps a value.",[29,511,513],{"id":512},"objectives-and-only-where-needed","Objectives, and only where needed",[14,515,516,518,519,522,523,526],{},[36,517,461],{}," is on ",[36,520,521],{},"collateral"," and not on ",[36,524,525],{},"resolved",", because more resolved bugs\nbeing better is obvious and fewer touched files being better is not.",[14,528,529],{},"Direction is the one thing that genuinely cannot be inferred. Everything else — the type,\nthe id, how to aggregate — follows from what you already wrote.",[29,531,533],{"id":532},"evidence-beyond-numbers","Evidence beyond numbers",[14,535,536,537,540],{},"Some things are not scalars. ",[36,538,539],{},"attach"," stores them as content-addressed artifacts on the\nrun:",[45,542,544],{"className":47,"code":543,"language":49,"meta":50,"style":50},"await bench.attach(\"diff\", await workspace.diff())\n",[36,545,546],{"__ignoreMap":50},[54,547,548,550,553,555,557,559,562,564,566,568,570,573],{"class":56,"line":57},[54,549,126],{"class":125},[54,551,552],{"class":67}," bench",[54,554,132],{"class":63},[54,556,539],{"class":60},[54,558,64],{"class":63},[54,560,561],{"class":79},"\"diff\"",[54,563,122],{"class":63},[54,565,126],{"class":125},[54,567,129],{"class":67},[54,569,132],{"class":63},[54,571,572],{"class":60},"diff",[54,574,143],{"class":63},[14,576,577,578,581],{},"Deduplicated by hash, referenced from the trial that produced it, and available when\nsomeone asks ",[18,579,580],{},"why"," a cell scored the way it did. A number tells you which model won; the\ndiff tells you what it actually did.",[583,584,585],"style",{},"html pre.shiki code .sCudf, html code.shiki .sCudf{--shiki-default:#DCDCAA}html pre.shiki code .sTNBD, html code.shiki .sTNBD{--shiki-default:#D4D4D4}html pre.shiki code .s3F5K, html code.shiki .s3F5K{--shiki-default:#9CDCFE}html pre.shiki code .sKc5r, html code.shiki .sKc5r{--shiki-default:#CE9178}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sYbnZ, html code.shiki .sYbnZ{--shiki-default:#C586C0}html pre.shiki code .sDGX8, html code.shiki .sDGX8{--shiki-default:#B5CEA8}html pre.shiki code .sOLPB, html code.shiki .sOLPB{--shiki-default:#6A9955}html pre.shiki code .scz_3, html code.shiki .scz_3{--shiki-default:#569CD6}html pre.shiki code .sNl3T, html code.shiki .sNl3T{--shiki-default:#4EC9B0}",{"title":50,"searchDepth":146,"depth":146,"links":587},[588,589,590,591,592,593,594],{"id":31,"depth":146,"text":32},{"id":94,"depth":146,"text":95},{"id":242,"depth":146,"text":243},{"id":270,"depth":146,"text":271},{"id":486,"depth":146,"text":487},{"id":512,"depth":146,"text":513},{"id":532,"depth":146,"text":533},"expect versus observe, and declaring what a benchmark records as a TypeScript type.","md",{},"\u002Fdocs\u002Fv2\u002Fbench\u002Fmeasuring",{"title":5,"description":595},"docs\u002Fv2\u002Fbench\u002Fmeasuring","E9C9sc9iRTWFOA6gycbxhXXhZr5JhU4xtU_AIti0xNg",1785237008123]