[{"data":1,"prerenderedAt":525},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fv2\u002Fbench\u002Fmatrix":3},{"id":4,"title":5,"body":6,"description":517,"extension":518,"meta":519,"navigation":520,"path":521,"seo":522,"stem":523,"__hash__":524},"docs\u002Fdocs\u002Fv2\u002Fbench\u002Fmatrix.md","The Matrix",{"type":7,"value":8,"toc":509},"minimark",[9,13,21,98,101,106,114,176,179,220,229,232,236,239,287,294,297,304,308,311,344,351,362,369,373,403,406,413,417,420,423,427,430,461,467,470,502,505],[10,11,5],"h1",{"id":12},"the-matrix",[14,15,16,20],"p",{},[17,18,19],"code",{},"matrix"," declares what varies. Every key is a place in the agent; every value is a\nvariation to try there.",[22,23,28],"pre",{"className":24,"code":25,"language":26,"meta":27,"style":27},"language-ts shiki shiki-themes dark-plus","matrix: {\n    model: [\n        OpenRouter({ model: \"anthropic\u002Fclaude-sonnet-4.6\" }),\n        OpenRouter({ model: \"openai\u002Fgpt-5\" }),\n    ],\n}\n","ts","",[17,29,30,42,51,72,86,92],{"__ignoreMap":27},[31,32,35,38],"span",{"class":33,"line":34},"line",1,[31,36,19],{"class":37},"sqDPy",[31,39,41],{"class":40},"sTNBD",": {\n",[31,43,45,48],{"class":33,"line":44},2,[31,46,47],{"class":37},"    model",[31,49,50],{"class":40},": [\n",[31,52,54,58,61,65,69],{"class":33,"line":53},3,[31,55,57],{"class":56},"sCudf","        OpenRouter",[31,59,60],{"class":40},"({ ",[31,62,64],{"class":63},"s3F5K","model:",[31,66,68],{"class":67},"sKc5r"," \"anthropic\u002Fclaude-sonnet-4.6\"",[31,70,71],{"class":40}," }),\n",[31,73,75,77,79,81,84],{"class":33,"line":74},4,[31,76,57],{"class":56},[31,78,60],{"class":40},[31,80,64],{"class":63},[31,82,83],{"class":67}," \"openai\u002Fgpt-5\"",[31,85,71],{"class":40},[31,87,89],{"class":33,"line":88},5,[31,90,91],{"class":40},"    ],\n",[31,93,95],{"class":33,"line":94},6,[31,96,97],{"class":40},"}\n",[14,99,100],{},"Two variations, one key: two cells. Everything else about the agent is identical across\nthem, which is what makes any difference in the results attributable to the model.",[102,103,105],"h2",{"id":104},"the-key-is-the-binding-point","The key is the binding point",[14,107,108,109,113],{},"There is no separate declaration of what a variation ",[110,111,112],"em",{},"is",". The key says where it lands in\nthe assembled agent, and the runtime applies it during the normal boot.",[115,116,117,130],"table",{},[118,119,120],"thead",{},[121,122,123,127],"tr",{},[124,125,126],"th",{},"Key",[124,128,129],{},"What it replaces",[131,132,133,144,154,166],"tbody",{},[121,134,135,141],{},[136,137,138],"td",{},[17,139,140],{},"agent",[136,142,143],{},"which agent project is loaded at all",[121,145,146,151],{},[136,147,148],{},[17,149,150],{},"cognet",[136,152,153],{},"the brain bundled into that agent",[121,155,156,161],{},[136,157,158],{},[17,159,160],{},"model",[136,162,163],{},[17,164,165],{},"config.engine",[121,167,168,173],{},[136,169,170],{},[17,171,172],{},"env",[136,174,175],{},"environment variables, merged",[14,177,178],{},"Four names, because four cover almost everything people want to compare. Anything else is\nreachable by path:",[22,180,182],{"className":24,"code":181,"language":26,"meta":27,"style":27},"matrix: {\n    \"config.engine.temperature\": [0, 0.7, 1.0],\n}\n",[17,183,184,190,216],{"__ignoreMap":27},[31,185,186,188],{"class":33,"line":34},[31,187,19],{"class":37},[31,189,41],{"class":40},[31,191,192,195,198,202,205,208,210,213],{"class":33,"line":44},[31,193,194],{"class":67},"    \"config.engine.temperature\"",[31,196,197],{"class":40},": [",[31,199,201],{"class":200},"sDGX8","0",[31,203,204],{"class":40},", ",[31,206,207],{"class":200},"0.7",[31,209,204],{"class":40},[31,211,212],{"class":200},"1.0",[31,214,215],{"class":40},"],\n",[31,217,218],{"class":33,"line":53},[31,219,97],{"class":40},[14,221,222,223,225,226,228],{},"A dotted path addresses any location in the blueprint, and the values are merged there.\nThe named keys exist for ergonomics — ",[17,224,160],{}," is easier to read than\n",[17,227,165],{}," — not because the runtime treats them specially.",[14,230,231],{},"This is why the named set stays small. The escape hatch means a new kind of comparison\nnever needs a new key, and a vocabulary that only grows is a vocabulary that eventually\nmeans nothing.",[102,233,235],{"id":234},"grids-are-a-choice-you-make","Grids are a choice you make",[14,237,238],{},"Two keys multiply:",[22,240,242],{"className":24,"code":241,"language":26,"meta":27,"style":27},"matrix: {\n    model: [sonnet, gpt5],\n    cognet: [\"..\u002Fzero\", \"..\u002Fexperimental\"],\n}\n",[17,243,244,250,266,283],{"__ignoreMap":27},[31,245,246,248],{"class":33,"line":34},[31,247,19],{"class":37},[31,249,41],{"class":40},[31,251,252,254,256,259,261,264],{"class":33,"line":44},[31,253,47],{"class":37},[31,255,197],{"class":40},[31,257,258],{"class":63},"sonnet",[31,260,204],{"class":40},[31,262,263],{"class":63},"gpt5",[31,265,215],{"class":40},[31,267,268,271,273,276,278,281],{"class":33,"line":53},[31,269,270],{"class":37},"    cognet",[31,272,197],{"class":40},[31,274,275],{"class":67},"\"..\u002Fzero\"",[31,277,204],{"class":40},[31,279,280],{"class":67},"\"..\u002Fexperimental\"",[31,282,215],{"class":40},[31,284,285],{"class":33,"line":74},[31,286,97],{"class":40},[14,288,289,290,293],{},"Four cells. Add ",[17,291,292],{},"trials: 5"," and that is twenty runs of every test. Add a third key with\nthree values and it is sixty.",[14,295,296],{},"The growth is not hidden and not capped. A grid is sometimes exactly right — you may\ngenuinely need to know whether a cognet's advantage holds across models — but it is a\ndecision with a cost, and the cost belongs in front of the person choosing to pay it.",[14,298,299,303],{},[300,301,302],"strong",{},"One key is the shape to reach for."," Changing a single variable is what makes a result\nsay something about that variable. Two models across two cognets tells you which\ncombination won; it does not cleanly tell you why.",[102,305,307],{"id":306},"variations-are-values-not-names","Variations are values, not names",[14,309,310],{},"You never write ids. They are derived from the variation itself:",[22,312,314],{"className":24,"code":313,"language":26,"meta":27,"style":27},"matrix: {\n    model: [OpenRouter({ model: \"anthropic\u002Fclaude-sonnet-4.6\" })],\n}\n",[17,315,316,322,340],{"__ignoreMap":27},[31,317,318,320],{"class":33,"line":34},[31,319,19],{"class":37},[31,321,41],{"class":40},[31,323,324,326,328,331,333,335,337],{"class":33,"line":44},[31,325,47],{"class":37},[31,327,197],{"class":40},[31,329,330],{"class":56},"OpenRouter",[31,332,60],{"class":40},[31,334,64],{"class":63},[31,336,68],{"class":67},[31,338,339],{"class":40}," })],\n",[31,341,342],{"class":33,"line":53},[31,343,97],{"class":40},[14,345,346,347,350],{},"becomes the cell ",[17,348,349],{},"model=anthropic-claude-sonnet-4.6",".",[14,352,353,354,357,358,361],{},"Derived from the value rather than the position, because position is not stable. If ids\nwere ",[17,355,356],{},"model[0]"," and ",[17,359,360],{},"model[1]",", reordering the list would silently repoint every cell id\nand results from an earlier run would stop lining up with the new ones — a comparison\nthat looks fine and is wrong.",[14,363,364,365,368],{},"When two variations derive the same id — two ",[17,366,367],{},"Mock()"," engines with different behaviour,\nsay — they are suffixed rather than rejected, since a pair the id cannot distinguish is\nstill a pair worth running.",[102,370,372],{"id":371},"a-single-value-is-still-an-axis","A single value is still an axis",[22,374,375],{"className":24,"code":313,"language":26,"meta":27,"style":27},[17,376,377,383,399],{"__ignoreMap":27},[31,378,379,381],{"class":33,"line":34},[31,380,19],{"class":37},[31,382,41],{"class":40},[31,384,385,387,389,391,393,395,397],{"class":33,"line":44},[31,386,47],{"class":37},[31,388,197],{"class":40},[31,390,330],{"class":56},[31,392,60],{"class":40},[31,394,64],{"class":63},[31,396,68],{"class":67},[31,398,339],{"class":40},[31,400,401],{"class":33,"line":53},[31,402,97],{"class":40},[14,404,405],{},"One cell, and the model is recorded in the manifest as a pinned constant.",[14,407,408,409,412],{},"Held constants are part of the result on purpose. A number is only comparable if you know\nwhat did ",[110,410,411],{},"not"," vary alongside what did — a score against Sonnet 4.6 and a score against\nGPT-5 are not the same measurement, even when the benchmark, the workspace, and every\ntest are identical.",[102,414,416],{"id":415},"what-the-matrix-cannot-vary","What the matrix cannot vary",[14,418,419],{},"A dataset. If a benchmark runs over 2,000 task instances, those instances are the\npopulation you are measuring over — not variations you are comparing. Adding them as an\naxis would multiply cells by 2,000 and produce a Cartesian product where you wanted a\nsample.",[14,421,422],{},"That distinction is real and the shape for it is still being settled. Today a scenario\nthat needs many cases iterates them inside the test.",[102,424,426],{"id":425},"where-variations-reach-the-agent","Where variations reach the agent",[14,428,429],{},"Nothing in a test says which cell it is running in:",[22,431,433],{"className":24,"code":432,"language":26,"meta":27,"style":27},"const { axon } = await Axon()\n",[17,434,435],{"__ignoreMap":27},[31,436,437,441,444,448,451,455,458],{"class":33,"line":34},[31,438,440],{"class":439},"scz_3","const",[31,442,443],{"class":40}," { ",[31,445,447],{"class":446},"s9McN","axon",[31,449,450],{"class":40}," } = ",[31,452,454],{"class":453},"sYbnZ","await",[31,456,457],{"class":56}," Axon",[31,459,460],{"class":40},"()\n",[14,462,463,466],{},[17,464,465],{},"Axon()"," boots the subject with this cell's variation already applied. The scenario does\nnot branch on it, and could not sensibly do so — a test that behaves differently\ndepending on which model it got is no longer measuring the same thing across cells.",[14,468,469],{},"If you do need the current variation — for a label, or a workspace path — it is available:",[22,471,473],{"className":24,"code":472,"language":26,"meta":27,"style":27},"const engine = bench.axis(\"model\")\n",[17,474,475],{"__ignoreMap":27},[31,476,477,479,482,485,488,490,493,496,499],{"class":33,"line":34},[31,478,440],{"class":439},[31,480,481],{"class":446}," engine",[31,483,484],{"class":40}," = ",[31,486,487],{"class":63},"bench",[31,489,350],{"class":40},[31,491,492],{"class":56},"axis",[31,494,495],{"class":40},"(",[31,497,498],{"class":67},"\"model\"",[31,500,501],{"class":40},")\n",[14,503,504],{},"Typed from your own matrix, so it narrows to exactly the variations you declared.",[506,507,508],"style",{},"html pre.shiki code .sqDPy, html code.shiki .sqDPy{--shiki-default:#C8C8C8}html pre.shiki code .sTNBD, html code.shiki .sTNBD{--shiki-default:#D4D4D4}html pre.shiki code .sCudf, html code.shiki .sCudf{--shiki-default:#DCDCAA}html pre.shiki code .s3F5K, html code.shiki .s3F5K{--shiki-default:#9CDCFE}html pre.shiki code .sKc5r, html code.shiki .sKc5r{--shiki-default:#CE9178}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sDGX8, html code.shiki .sDGX8{--shiki-default:#B5CEA8}html pre.shiki code .scz_3, html code.shiki .scz_3{--shiki-default:#569CD6}html pre.shiki code .s9McN, html code.shiki .s9McN{--shiki-default:#4FC1FF}html pre.shiki code .sYbnZ, html code.shiki .sYbnZ{--shiki-default:#C586C0}",{"title":27,"searchDepth":44,"depth":44,"links":510},[511,512,513,514,515,516],{"id":104,"depth":44,"text":105},{"id":234,"depth":44,"text":235},{"id":306,"depth":44,"text":307},{"id":371,"depth":44,"text":372},{"id":415,"depth":44,"text":416},{"id":425,"depth":44,"text":426},"Binding points, variations, and why one variable at a time is the default.","md",{},true,"\u002Fdocs\u002Fv2\u002Fbench\u002Fmatrix",{"title":5,"description":517},"docs\u002Fv2\u002Fbench\u002Fmatrix","OIkQd1mmsV7Eh-K9PErUKtyU26U5XxlvDd-M8bIDE8E",1785237008185]