[{"data":1,"prerenderedAt":440},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fv2\u002Fbench\u002Fworkspace":3},{"id":4,"title":5,"body":6,"description":432,"extension":433,"meta":434,"navigation":435,"path":436,"seo":437,"stem":438,"__hash__":439},"docs\u002Fdocs\u002Fv2\u002Fbench\u002Fworkspace.md","workspace\u002F",{"type":7,"value":8,"toc":426},"minimark",[9,13,20,96,99,102,107,169,179,183,190,238,247,251,254,301,308,311,401,408,412,415,422],[10,11,5],"h1",{"id":12},"workspace",[14,15,16,19],"p",{},[17,18,5],"code",{}," is the world the agent works in. A repository with a bug in it, a directory of\ndocuments, a project scaffold — whatever the task is about.",[21,22,27],"pre",{"className":23,"code":24,"language":25,"meta":26,"style":26},"language-bash shiki shiki-themes dark-plus","my-bench\u002F\n└── workspace\u002F\n    ├── src\u002F\n    │   └── parser.ts\n    ├── tests\u002F\n    │   └── parser.test.ts\n    └── package.json\n","bash","",[17,28,29,38,48,57,69,77,87],{"__ignoreMap":26},[30,31,34],"span",{"class":32,"line":33},"line",1,[30,35,37],{"class":36},"sCudf","my-bench\u002F\n",[30,39,41,44],{"class":32,"line":40},2,[30,42,43],{"class":36},"└──",[30,45,47],{"class":46},"sKc5r"," workspace\u002F\n",[30,49,51,54],{"class":32,"line":50},3,[30,52,53],{"class":36},"    ├──",[30,55,56],{"class":46}," src\u002F\n",[30,58,60,63,66],{"class":32,"line":59},4,[30,61,62],{"class":36},"    │",[30,64,65],{"class":46},"   └──",[30,67,68],{"class":46}," parser.ts\n",[30,70,72,74],{"class":32,"line":71},5,[30,73,53],{"class":36},[30,75,76],{"class":46}," tests\u002F\n",[30,78,80,82,84],{"class":32,"line":79},6,[30,81,62],{"class":36},[30,83,65],{"class":46},[30,85,86],{"class":46}," parser.test.ts\n",[30,88,90,93],{"class":32,"line":89},7,[30,91,92],{"class":36},"    └──",[30,94,95],{"class":46}," package.json\n",[14,97,98],{},"It is copied into a fresh directory for every run, and the agent boots with that copy as\nits root. The directory here is never mutated: it is the source of truth, and every trial\ngets an identical clone of it.",[14,100,101],{},"That is what makes trials comparable. Six runs of the same test against the same workspace\nstart from byte-identical worlds, so any difference in outcome came from the variation\nunder test rather than from residue left by the run before.",[103,104,106],"h2",{"id":105},"declaring-it","Declaring it",[21,108,112],{"className":109,"code":110,"language":111,"meta":26,"style":26},"language-ts shiki shiki-themes dark-plus","export default defineBench\u003CSchema>({\n    workspace: { source: \".\u002Fworkspace\", retain: \"failed\" },\n})\n","ts",[17,113,114,137,164],{"__ignoreMap":26},[30,115,116,120,123,126,130,134],{"class":32,"line":33},[30,117,119],{"class":118},"sYbnZ","export",[30,121,122],{"class":118}," default",[30,124,125],{"class":36}," defineBench",[30,127,129],{"class":128},"sTNBD","\u003C",[30,131,133],{"class":132},"sNl3T","Schema",[30,135,136],{"class":128},">({\n",[30,138,139,143,146,149,152,155,158,161],{"class":32,"line":40},[30,140,142],{"class":141},"s3F5K","    workspace:",[30,144,145],{"class":128}," { ",[30,147,148],{"class":141},"source:",[30,150,151],{"class":46}," \".\u002Fworkspace\"",[30,153,154],{"class":128},", ",[30,156,157],{"class":141},"retain:",[30,159,160],{"class":46}," \"failed\"",[30,162,163],{"class":128}," },\n",[30,165,166],{"class":32,"line":50},[30,167,168],{"class":128},"})\n",[14,170,171,174,175,178],{},[17,172,173],{},"source"," defaults to ",[17,176,177],{},".\u002Fworkspace",", so a benchmark that uses the conventional layout can\nomit it entirely. A benchmark that needs no world at all declares none, and the agent gets\nan empty directory.",[103,180,182],{"id":181},"retention","Retention",[14,184,185,186,189],{},"Runs are cheap to lose and expensive to reproduce. ",[17,187,188],{},"retain"," decides which worlds survive:",[191,192,193,204],"table",{},[194,195,196],"thead",{},[197,198,199,202],"tr",{},[200,201],"th",{},[200,203],{},[205,206,207,218,228],"tbody",{},[197,208,209,215],{},[210,211,212],"td",{},[17,213,214],{},"never",[210,216,217],{},"discard every workspace",[197,219,220,225],{},[210,221,222],{},[17,223,224],{},"failed",[210,226,227],{},"keep the world of any trial that did not complete",[197,229,230,235],{},[210,231,232],{},[17,233,234],{},"always",[210,236,237],{},"keep all of them",[14,239,240,242,243,246],{},[17,241,224],{}," is the default and usually the right one. When a trial fails, the world it failed\nin tells you more than any number about it — the half-applied patch, the file it should not\nhave touched, the dependency it installed. It sits under ",[17,244,245],{},".bench\u002F"," exactly as the agent\nleft it.",[103,248,250],{"id":249},"captured-changes","Captured changes",[14,252,253],{},"Whatever the retention policy, what changed is recorded on the trial:",[21,255,257],{"className":109,"code":256,"language":111,"meta":26,"style":26},"observe(\"collateral\", (await workspace.changed()).length - 1)\n",[17,258,259],{"__ignoreMap":26},[30,260,261,264,267,270,273,276,279,282,285,288,291,294,298],{"class":32,"line":33},[30,262,263],{"class":36},"observe",[30,265,266],{"class":128},"(",[30,268,269],{"class":46},"\"collateral\"",[30,271,272],{"class":128},", (",[30,274,275],{"class":118},"await",[30,277,278],{"class":141}," workspace",[30,280,281],{"class":128},".",[30,283,284],{"class":36},"changed",[30,286,287],{"class":128},"()).",[30,289,290],{"class":141},"length",[30,292,293],{"class":128}," - ",[30,295,297],{"class":296},"sDGX8","1",[30,299,300],{"class":128},")\n",[14,302,303,304,307],{},"The harness snapshots the world before the agent boots and diffs it afterward, which is why\n",[17,305,306],{},"changed()"," can answer a question the test itself never had the information to answer.",[14,309,310],{},"Capture is tunable when the defaults get in the way:",[21,312,314],{"className":109,"code":313,"language":111,"meta":26,"style":26},"workspace: {\n    source: \".\u002Fworkspace\",\n    capture: {\n        ignore: [\".git\", \"node_modules\", \".axon\"],\n        maxBytes: 10 * 1024 * 1024,\n    },\n}\n",[17,315,316,324,338,345,369,391,396],{"__ignoreMap":26},[30,317,318,321],{"class":32,"line":33},[30,319,12],{"class":320},"sqDPy",[30,322,323],{"class":128},": {\n",[30,325,326,329,332,335],{"class":32,"line":40},[30,327,328],{"class":320},"    source",[30,330,331],{"class":128},": ",[30,333,334],{"class":46},"\".\u002Fworkspace\"",[30,336,337],{"class":128},",\n",[30,339,340,343],{"class":32,"line":50},[30,341,342],{"class":320},"    capture",[30,344,323],{"class":128},[30,346,347,350,353,356,358,361,363,366],{"class":32,"line":59},[30,348,349],{"class":320},"        ignore",[30,351,352],{"class":128},": [",[30,354,355],{"class":46},"\".git\"",[30,357,154],{"class":128},[30,359,360],{"class":46},"\"node_modules\"",[30,362,154],{"class":128},[30,364,365],{"class":46},"\".axon\"",[30,367,368],{"class":128},"],\n",[30,370,371,374,376,379,382,385,387,389],{"class":32,"line":71},[30,372,373],{"class":320},"        maxBytes",[30,375,331],{"class":128},[30,377,378],{"class":296},"10",[30,380,381],{"class":128}," * ",[30,383,384],{"class":296},"1024",[30,386,381],{"class":128},[30,388,384],{"class":296},[30,390,337],{"class":128},[30,392,393],{"class":32,"line":79},[30,394,395],{"class":128},"    },\n",[30,397,398],{"class":32,"line":89},[30,399,400],{"class":128},"}\n",[14,402,403,404,407],{},"The ignore list exists because ",[17,405,406],{},"node_modules"," appearing in a diff is noise, not signal, and\na benchmark whose install step dominates its change record is measuring the wrong thing.",[103,409,411],{"id":410},"what-cannot-be-in-here","What cannot be in here",[14,413,414],{},"Symlinks are refused before a run starts, not followed. A symlink in a workspace is a path\nout of the world — the whole point of the copy is that the agent's reachable filesystem is\nbounded and identical every time, and a link pointing at the benchmark's own directory\nbreaks both properties at once.",[14,416,417,418,421],{},"The same reasoning covers ",[17,419,420],{},"fixtures\u002F",". Anything the agent must not see — expected outputs,\njudge rubrics, grading data — belongs there instead, one level up and out of reach.",[423,424,425],"style",{},"html pre.shiki code .sCudf, html code.shiki .sCudf{--shiki-default:#DCDCAA}html pre.shiki code .sKc5r, html code.shiki .sKc5r{--shiki-default:#CE9178}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sYbnZ, html code.shiki .sYbnZ{--shiki-default:#C586C0}html pre.shiki code .sTNBD, html code.shiki .sTNBD{--shiki-default:#D4D4D4}html pre.shiki code .sNl3T, html code.shiki .sNl3T{--shiki-default:#4EC9B0}html pre.shiki code .s3F5K, html code.shiki .s3F5K{--shiki-default:#9CDCFE}html pre.shiki code .sDGX8, html code.shiki .sDGX8{--shiki-default:#B5CEA8}html pre.shiki code .sqDPy, html code.shiki .sqDPy{--shiki-default:#C8C8C8}",{"title":26,"searchDepth":40,"depth":40,"links":427},[428,429,430,431],{"id":105,"depth":40,"text":106},{"id":181,"depth":40,"text":182},{"id":249,"depth":40,"text":250},{"id":410,"depth":40,"text":411},"The agent's world — copied fresh for every run, and the only thing it can see.","md",{},true,"\u002Fdocs\u002Fv2\u002Fbench\u002Fworkspace",{"title":5,"description":432},"docs\u002Fv2\u002Fbench\u002Fworkspace","NhwXEwgs0xQIKuHI_avsMXUe0EDnhh6EYGIC411eNyE",1785237008173]