Claude Fable 5.2 max≈$18.21 / task · 81.1 ptsthe best there is
Execute itcode & commands
Kimi K3.1 Pro max≈$3.54 / task · ≈70.3 ptsthe best there isKimi K3.1 Pro max≈70.3range 67.4–73.3Gemini 4 Carbon high≈69.7range 66.8–72.6The ranges overlap, so either could lead.
← cheaper
10,000× cheaper
1,000× cheaper
100× cheaper
10× cheaper
full price
too weakjust oktoo weak
Figure it out. Price midway between list and fallback rates; the fallback's share isn't published. Under 60 points I wouldn't bother. The cheapest way over that bar: Muse Ember xhigh at ≈$0.038 (estimated).
Execute it. Close call: Kimi K3.1 Pro max ≈70.3 (67.4–73.3) against Gemini 4 Carbon high ≈69.7 (66.8–72.6); the ranges overlap. Score 60% measured, the rest predicted. Under 40 points I wouldn't bother. The cheapest way over that bar: GPT-6.1 Terra low at ≈$0.22 (estimated).
Cheapest pair in both lanes
past redMuse Ember xhigh + GPT-6.1 Terra low18× cheaper
past amberGPT-6.1 Sol high + Muse Ember xhigh12× cheaper
a modelthe picktoo weakjust okclose call ranges overlap≈ estimated
Every configuration
quality against cost · open ↓close ↑
See the trade-off
Higher is better, left is cheaper. The dashed line joins the rungs on the rail above; the ring is the pick.
Benchmark frontiers
Figure score against cost per task, 132 configurations, log cost axis. Higher is better; left is cheaper.Scored on Briefcase analytical quality, AA-LCR 1.1, AA-Omniscience index; the flags mark the pick.
Conditional tier envelope $0.019–$0.093 per mix task; midpoint scenario, not expected price.
Conditional tier envelope $0.044–$0.22 per mix task; midpoint scenario, not expected price.
Conditional tier envelope $0.072–$0.36 per mix task; midpoint scenario, not expected price.
Conditional tier envelope $0.11–$0.57 per mix task; midpoint scenario, not expected price.
Conditional tier envelope $0.21–$1.05 per mix task; midpoint scenario, not expected price.
Hollow mark = estimated score or cost · pale dashed line = Pareto frontier · coloured line = one model's effort rungs · red = too weak, under the owner's 60-point floor · amber = just ok, 60–70 · the flags mark the pick · data: November 11, 2026
Models you can use
52 of 52 included · open ↓close ↑
Turn off what you can’t use; the rail recomputes. Open a model to drop single efforts.
6/6 efforts · edit
5/5 efforts · edit
6/6 efforts · edit
6/6 efforts · edit
6/6 efforts · edit
7/7 efforts · edit
7/7 efforts · edit
7/7 efforts · edit
5/5 efforts · edit
1/1 effort · edit
5/5 efforts · edit
5/5 efforts · edit
5/5 efforts · edit
5/5 efforts · edit
5/5 efforts · edit
5/5 efforts · edit
5/5 efforts · edit
1/1 effort · edit
3/3 efforts · edit
3/3 efforts · edit
2/2 efforts · edit
1/1 effort · edit
3/3 efforts · edit
3/3 efforts · edit
4/4 efforts · edit
3/3 efforts · edit
3/3 efforts · edit
3/3 efforts · edit
3/3 efforts · edit
3/3 efforts · edit
2/2 efforts · edit
1/1 effort · edit
2/2 efforts · edit
1/1 effort · edit
1/1 effort · edit
4/4 efforts · edit
1/1 effort · edit
2/2 efforts · edit
2/2 efforts · edit
2/2 efforts · edit
2/2 efforts · edit
1/1 effort · edit
1/1 effort · edit
1/1 effort · edit
3/3 efforts · edit
1/1 effort · edit
1/1 effort · edit
1/1 effort · edit
2/2 efforts · edit
1/1 effort · edit
1/1 effort · edit
1/1 effort · edit
How this is measured
sources, weights and limits · open ↓close ↑
How the rail is builtundominated rungs on a log job-share axis
The rail measures a whole job, not one benchmark task. Each dot is a model and effort that no cheaper comparable configuration beats on the mix score. A dot sits at what that rung contributes to the job: its lane's share of the job, times what the rung costs against the best rung in its own lane. The right end of the rail is the job with both lanes at their best, so the ticks read 1, .1, .01 — fractions of that job, not dollars. Equal distances still mean equal cost ratios.
Why not one dollar figure for both lanes? Because a benchmark task is not a job, and the two lanes' tasks are not the same size. Figure combines practical analysis, document reasoning and factual reliability. Execute combines repository work, terminal tasks, professional deliverables and app workflows. Their benchmark task sizes differ. Charging both lanes the same dollar ceiling treats one of each as an equal purchase — which it is not, when a real job runs one lane once and the other for hours.
"Your job is" sets the split. Mostly figuring puts 10% of the job in Execute, Balanced puts half, Mostly executing puts 90% there. The line is then one threshold both lanes' contributions sit under, and it converts back to a dollar ceiling per lane before anything is chosen — the lane carrying most of the job gets squeezed first, which is the point. Each answer still shows its own benchmark dollars per task and the date of the capture it came from.
Those three numbers are chosen, not measured. They are round positions, with the two "mostly" options symmetric around Balanced. There is no retained source behind 10, 50 and 90, and none is claimed: they exist so a reader can say roughly what their week looks like, and a spinner would promise a precision nobody here has. Pick the one nearest your own and read the answer it gives.
The author's own jobs are a separate thing, and they are measured. Across 16 of his Claude Code jobs that ran both a figuring phase and an executing phase, 2026-09-03 to 09-08, executing took 5 to 94 percent of those two phases' API-equivalent cost at that day's published rates — median 84 percent, 75 percent weighted by dollars, over $347 of attributed spend. The largest such job spent $63 of $77 executing. That is one person's five days on one harness. It is not a bill, it is not a measurement of your work, and the width of that range is exactly why the split is a control rather than a constant.
Those runs were also almost all cache reads: 98.4 percent of input-side tokens across the Claude Code sessions, and 96.1 percent of input tokens across 258 Codex sessions in the same window. For work shaped like that a model's cache-read price orders the bill far more than its input price does, and a benchmark task bounded at a few minutes cannot show you that. Opus 5 reads cache at $0.50 per million tokens, Fable 5.1 at $0.25, Sonnet 5 at $0.20, Haiku 4.5 at $0.10 (Anthropic pricing, read 2026-09-08).
The rail carries the undominated rungs only. Cheaper and stronger steps are the dots themselves: every Pareto step for both lanes is on the line, and dragging to one selects it. A ceiling below every rung produces no recommendation for that lane and names the cheapest comparable option instead.
How alternatives qualifyscreening rules and evidence limits
A model does not need every benchmark to earn consideration. One strong, relevant evaluation can qualify a scoped alternative. Figure looks across practical analysis, reasoning, factual reliability and document work. Execute looks across coding-agent work, professional outputs and tool workflows. A specialist remains a specialist; its best result does not establish broad capability.
Within your available models and effort settings, each benchmark admits results within 5 score points of its leader, or 100 Elo on native Elo boards. It also admits undominated quality–cost options within 20 points, or 400 Elo, of the leader. “Undominated” means no measured option is both at least as good and at least as cheap, with one strict advantage. Missing cost cannot earn a value claim. These margins are chosen screening rules, not uncertainty estimates or proof of a meaningful lead.
Each track shows its mix and relevant supporting evidence. Figure also carries hard-question and document-reading results; Execute carries FrontierCode, Zapier, SciCode, Briefcase rubric checks and other agent-work results. Supporting scores do not enter the mix. Every qualifying model remains accessible; the first six are a compact alphabetical preview. Open a model's row to see efforts, native scores, source, date, harness and fallback status. Several results from one publisher are not independent corroboration.
The shortlist uses native Briefcase and GDPval Elo and the published AA-Omniscience index. Both mixes use the fixed conversions below; the native evidence stays visible beside them. Native and composite units must not be compared directly. FrontierCode does not say whether Claude Code rerouted a Claude run to a fallback model. Its Claude results count only for models Claude Code never reroutes. Kimi's Zapier effort is undisclosed. Such scoped results can support consideration, but cannot fill an exact mix gap.
How the recommendation is pickedquality, spending and uncertainty
Start with quality, then decide how much to spend. Quality first selects the highest score in the declared mix among configurations with enough measured or explicitly estimated inputs. When a source does not name the effort it ran, the row is read as the vendor's default, the setting you get by naming none, and every surface labels it default (assumed). Equal scores prefer lower cost, then a stable configuration identity. Estimates can keep useful models in the comparison, but when the pick or an alternative under the same ceiling is estimated and their ranges overlap, the answer is marked close call, which opens a note naming both configurations. The range does not include all uncertainty in the measured benchmarks.
There is one mix for Figure it out and one for Execute it. The provider chips filter models you can access: Codex is OpenAI-only; Claude Code is Anthropic-only. Each benchmark keeps the harness that measured it. The composite does not claim to measure the model inside either product.
The rail sets a logarithmic ceiling in shares of one job: equal distances mean equal cost ratios, and the position converts to a dollar ceiling per lane through that lane's share of the job. Each kind of work then answers with its highest score within its own ceiling. Every Pareto step is a dot on the rail, so dragging past one selects it. A ceiling too low produces an explicit empty recommendation with the cheapest available alternative. Releasing the rail at its top removes the ceiling — Quality first — without changing model or effort exclusions.
The floors are the author's judgment, not a measurement. Below 60 points for Figure it out, or 40 for Execute it, he would not run that lane at all; from there up to 70 and 50 a configuration is acceptable but not strong. The rail and the Every configuration chart shade those stretches red and amber, and an answer that lands in one says so. The floors never change the pick — it is still the highest score under your ceiling — and no source stands behind the numbers.
Cheaper models that are not on the frontier still appear where their evidence earns it: the shortlist under Why this choice states each qualifying model’s measured scope, system, and unknown or above-ceiling costs. A component strength is not a broad recommendation.
The audit tests the policy, not the probability of being right. Open Method sensitivity & winning margins to see whether the pick survives half or double Elo slopes, wider group weights, or a benchmark left out. Estimates stay fixed while the weights move, and scenario counts are not confidence percentages. A stable result can still be wrong for your task: the scenarios cannot supply real-world outcomes that were never measured. The Execute leader and its limits explains what the Execute pick with no budget limit rests on, for All models and Claude Code.
Why this choicemeasurements, method & access
132 of 145 configurations can set this lane’s recommendation, 65 of them with estimates.
Why 13 configurations cannot set this recommendation
Only 30% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Token accounting is not published for this benchmark
AA-LCR 1.1: Comparable API pricing is not published
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
How the estimate is made
30% of score weight measured. Score ≈54.1; missing-input range 37.2–62.1 points. API cost ≈$0.54 per mix task; range $0.063–$4.70.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 123 configurations. Held-out mean absolute error 12.64 points; benchmark-mean baseline 14.62.
A higher or lower effort of the same model was measured, so the prediction is held at Gemini 3.7 Flash high’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $1.65 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.14 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.82 log dollars; benchmark-mean baseline 0.95.
Missing measurement: Comparable API pricing is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 57.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0077 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Gemini 3.7 Flash medium · Benchmark-specific harnesses
Only 30% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Token accounting is not published for this benchmark
AA-LCR 1.1: Comparable API pricing is not published
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
How the estimate is made
30% of score weight measured. Score ≈56.5; missing-input range 39.3–65.7 points. API cost ≈$0.61 per mix task; range $0.073–$5.08.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 123 configurations. Held-out mean absolute error 12.64 points; benchmark-mean baseline 14.62.
A higher or lower effort of the same model was measured, so the prediction is held at Gemini 3.7 Flash high’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $1.85 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.16 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.82 log dollars; benchmark-mean baseline 0.95.
Missing measurement: Comparable API pricing is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 59.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0084 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Only 30% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Token accounting is not published for this benchmark
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
How the estimate is made
30% of score weight measured. Score ≈55.3; missing-input range 36.9–65.0 points. API cost ≈$1.82 per mix task; range $0.38–$12.02.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 121 configurations. Held-out mean absolute error 12.68 points; benchmark-mean baseline 14.76.
A higher or lower effort of the same model was measured, so the prediction is held at GPT-5.5 medium’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $5.56 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 98 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.37.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 58.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GPT-5.5 Pro xhigh · Benchmark-specific harnesses
Score unavailable · Comparable cost unavailable
Only 0% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Token accounting is not published for this benchmark
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Token accounting is not published for this benchmark
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
Only 30% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Token accounting is not published for this benchmark
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
How the estimate is made
30% of score weight measured. Score ≈40.2; missing-input range 23.2–48.2 points. API cost ≈$0.68 per mix task; range $0.13–$4.98.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 122 configurations. Held-out mean absolute error 12.52 points; benchmark-mean baseline 14.16.
A higher or lower effort of the same model was measured, so the prediction is held at Gemini 3.5 Flash high’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $2.09 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 99 configurations. Held-out mean absolute error 1.00 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 46.2 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0096 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Only 30% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Token accounting is not published for this benchmark
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
How the estimate is made
30% of score weight measured. Score ≈47.9; missing-input range 29.5–57.8 points. API cost ≈$1.80 per mix task; range $0.38–$11.87.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 121 configurations. Held-out mean absolute error 12.68 points; benchmark-mean baseline 14.76.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $5.49 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 98 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.37.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 47.8 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality: Comparable API pricing is not published
AA-LCR 1.1: Comparable API pricing is not published
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Token accounting is not published for this benchmark
How the estimate is made
60% of score weight measured. Score ≈52.3; missing-input range 42.5–62.0 points. API cost unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 56.8 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Claude Fable 5 high · Benchmark-specific harnesses
Fallback behavior is not disclosed; exact system cannot be established
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
How the estimate is made
0% of score weight measured. Score ≈62.1; missing-input range 46.4–73.1 points. API cost ≈$3.45 per mix task; range $1.31–$9.12.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $10.95 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.51 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 57.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.033 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Fallback behavior is not disclosed; exact system cannot be established
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
How the estimate is made
0% of score weight measured. Score ≈56.0; missing-input range 41.7–68.7 points. API cost ≈$1.23 per mix task; range $0.46–$3.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $3.83 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.25 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 48.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.015 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Claude Fable 5 medium · Benchmark-specific harnesses
Fallback behavior is not disclosed; exact system cannot be established
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
How the estimate is made
0% of score weight measured. Score ≈59.8; missing-input range 44.3–71.7 points. API cost ≈$2.15 per mix task; range $0.93–$5.06.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $6.76 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.36 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 54.1 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Claude Fable 5 xhigh · Benchmark-specific harnesses
Fallback behavior is not disclosed; exact system cannot be established
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
How the estimate is made
0% of score weight measured. Score ≈63.1; missing-input range 47.4–73.7 points. API cost ≈$4.64 per mix task; range $1.52–$14.16.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $14.78 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.63 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 59.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.041 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Gemini 3.1 Pro Preview high · Benchmark-specific harnesses
Only 0% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
How the estimate is made
0% of score weight measured. Score ≈39.7; missing-input range 28.6–50.8 points. API cost ≈$0.64 per mix task; range $0.077–$5.42.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $1.98 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.16 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 41.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0089 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Only 0% of score weight measured; at least 50% required for a recommendation
Briefcase analytical quality: No published measurement at this exact configuration
Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
AA-LCR 1.1: No published measurement at this exact configuration
AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
AA-Omniscience index: No published measurement at this exact configuration
AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
How the estimate is made
0% of score weight measured. Score ≈54.7; missing-input range 40.4–69.1 points. API cost ≈$1.22 per mix task; range $0.45–$3.37.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $3.80 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.24 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 47.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.015 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Evidence behind the chosen configuration
The weights are declared policy, not a validated prediction of your work. Collaborative framing and planning remain unmeasured.
Score & native measurements
81.08 points · Figure it out. Benchmark proxies do not guarantee correctness.
Supporting evidence, outside the score: Briefcase presentation 1706.03 Elo · 2026-11-11. Same exact model and effort; AA benchmark harness.
How the estimate is made
100% of score weight measured. Score 81.1; no score component estimated. API cost ≈$18.21 per mix task; range $18.13–$18.28.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-LCR 1.1 · cost: $1.05 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Claude Haiku 5.5 max Figure: $0.21–$1.05 per mix task, a conditional tier envelope. The rail uses its $0.63 midpoint scenario, not an expected price.
At this ceiling: lower-tier scenario → Claude Fable 5.2 max; midpoint → Claude Fable 5.2 max; upper-tier scenario → Claude Fable 5.2 max. The budget choice stays the same at these three scenarios.
Tier assumptions and all measured efforts
Only request-tier allocation varies, using captured AA tokens, cache-hit rate, five-minute cache writes and task counts. Every request above 100,000 prompt tokens, including cached tokens, costs five times as much. This envelope does not guarantee an actual bill or establish a confidence interval. Other billing modifiers are outside it. Estimates may exceed your ceiling. All tier rows move together in these three readings; they do not cover every independent allocation.
Claude Haiku 5.5 low: $0.019–$0.093 per Figure mix task; 52.9 points measured.
Claude Haiku 5.5 medium: $0.044–$0.22 per Figure mix task; 60.3 points measured.
Claude Haiku 5.5 high: $0.072–$0.36 per Figure mix task; 62.1 points measured.
Claude Haiku 5.5 xhigh: $0.11–$0.57 per Figure mix task; 64.5 points measured.
Claude Haiku 5.5 max: $0.21–$1.05 per Figure mix task; 68.1 points measured.
Execute predicts its missing DeepSWE component separately; its total bill has no established bound.
Access: subscription, API or your hardware
Reviewed 2026-11-11. Availability does not reproduce a benchmark’s agent, effort or fallback configuration.
Subscription Verified route
Max, Ultra, premium Team and premium legacy seat-based Enterprise seats include Fable models within up to 50% of the shared weekly allowance. Pro, standard Team and standard Enterprise seats use purchased usage credits from the start. A launch offer includes Fable on Pro and Team standard seats through 2026-11-16 23:59 PT.
Plan-dependent inclusion or credits; the 50% is not extra capacity. Usage-based Enterprise uses API rates. Launch-week Pro inclusion ends 16 November 2026.
Claude Code requires 2.1.321 or newer. App fallback is automatic; availability alone does not reproduce the benchmark fallback system. Mythos 5.2 shares the same specs and price but is limited-availability verification-program access and is not this model.
claude-fable-5-2, released 2026-11-09. Adaptive thinking always on; default effort high.
$8 input / $40 output USD per million tokens; 5-minute cache write $10, 1-hour $16, cache read $0.20. Batch is 50% off. The $16 / $80 fast setting is not the measured configuration.
API fallback must be configured. Cost and quality depend on the exact fallback model/settings, not only the primary model. Effort: low through max; default high. Mythos 5.2 is a separate limited-availability id. The $16 / $80 fast setting is a different price and is unmeasured.
These are declared scenarios, not confidence percentages. They vary Elo slope by half/double, group weights by 0.5/1/2, omit benchmarks/groups. Your setup and exclusions apply. Priced results apply your spending ceiling; quality-only results ignore costs.
Fixed cohorts keep the baseline eligible configurations; expanded cohorts admit configurations newly scoreable under each composition. Validated estimates stay fixed while weights vary. Native margins below include only measured pairs; these scenarios do not test estimation uncertainty. Several components share Artificial Analysis, model judges and agent tools; they are not independent confirmations. Benchmark-specific harness and fallback limits remain.
Baseline: 132 priced configurations from 45 models. Claude Fable 5.2 max leads Claude Opus 5.5 max by 1.668 points.
Briefcase analytical quality: native margin -110.5800 at 30.0% weight.
AA-LCR 1.1: native margin -0.0133 at 30.0% weight.
AA-Omniscience index: native margin +18.6333 at 40.0% weight.
Scenario
Fixed · quality only
Fixed · priced
Expanded · quality only
Expanded · priced
Declared profile
Claude Fable 5.2 max133 configurations · 46 models
Claude Fable 5.2 max132 configurations · 45 models
Claude Fable 5.2 max133 configurations · 46 models
Claude Fable 5.2 max132 configurations · 45 models
Elo slope 0.5×
Claude Fable 5.2 max133 configurations · 46 models
Claude Fable 5.2 max132 configurations · 45 models
Claude Fable 5.2 max133 configurations · 46 models
Claude Fable 5.2 max132 configurations · 45 models
Elo slope 2×
Claude Fable 5.2 max133 configurations · 46 models
Claude Fable 5.2 max132 configurations · 45 models
Claude Fable 5.2 max133 configurations · 46 models
Claude Fable 5.2 max132 configurations · 45 models
CritPt ↗xhigh effort · 31.14 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
GDP.pdf all-pass ↗low effort · 28.00 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-LCR 1.1 ↗max effort · 85.33 points · $1.58 / benchmark taskNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-LCR 1.1 ↗high effort · 83.67 points · $1.48 / benchmark taskNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-LCR 1.1 ↗medium effort · 84.67 points · $1.48 / benchmark taskNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
Claude Fable 5.2Practical analysis · Scientific reasoning +2Comparable profile option · One publisher · Effort / access caveat+
At least one qualifying effort has complete, priced profile evidence. Anthropic API model claude-fable-5-2, released November 9, adaptive thinking always on, default effort high. AA measures low through max with Default Fallback. Costs retain AA's captured $8 / $40 list rate (cache-read $0.20). The $16 / $80 fast setting and Mythos 5.2 are not this configuration. Claude Code requires 2.1.321 or newer. Direct DeepSWE is unmeasured.
CritPt ↗max effort · 35.43 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗xhigh effort · 35.71 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗high effort · 32.86 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗medium effort · 30.86 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
HLE ↗max effort · 74.28 points · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-Omniscience index ↗max effort · 66.77 points · $0.22 / benchmark taskNear the available leader · Quality–cost frontier · AA benchmark harness · fallback enabled · 2026-11-11
AA-Omniscience index ↗xhigh effort · 64.77 points · $0.033 / benchmark taskNear the available leader · Quality–cost frontier · AA benchmark harness · fallback enabled · 2026-11-11
AA-Omniscience index ↗high effort · 64.03 points · $0.020 / benchmark taskNear the available leader · Quality–cost frontier · AA benchmark harness · fallback enabled · 2026-11-11
AA-Omniscience index ↗medium effort · 63.50 points · $0.015 / benchmark taskNear the available leader · Quality–cost frontier · AA benchmark harness · fallback enabled · 2026-11-11
AA-Omniscience index ↗low effort · 62.93 points · $0.014 / benchmark taskNear the available leader · Quality–cost frontier · AA benchmark harness · fallback enabled · 2026-11-11
Claude Haiku 5.5Financial analysisComparable profile option · One publisher · Effort / access caveat+
At least one qualifying effort has complete, priced profile evidence. AA now measures low through max as standalone efforts, matching the documented absence of server-side fallback. Lower-tier tariffs are restored, but the exact-row caveat says 5x long-context pricing is omitted. Aggregate tokens do not establish request-tier allocations; native benchmark bills stay missing. Figure uses a separately marked conditional tier envelope and non-expected midpoint; Execute predicts missing DeepSWE separately and has no established total-bill bound. AA flags pre-release AutomationBench refusals pending reevaluation. Direct DeepSWE is unmeasured; fallback-unknown Claude Code and fallback-labelled Zapier results join nothing.
AA-AnalystAgent pass@1 ↗max effort · 68.25 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · single model · 2026-11-11
Claude Opus 5.5Practical analysis · Scientific reasoning +2Comparable profile option · One publisher · Effort / access caveat+
At least one qualifying effort has complete, priced profile evidence. AA measures low through max with default safety fallback. AA publishes a fallback tariff for every effort and, among the benchmarks used here, fallback fractions for AA-Omniscience only: its cost is a direct bill, and every other Opus 5.5 AA cost is a cost bound shown at its midpoint. No matching direct DeepSWE result exists; its cost is predicted from the AA-Omniscience bill.
Briefcase analytical quality ↗max effort · 2125.59 Elo · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
Briefcase analytical quality ↗xhigh effort · 2080.87 Elo · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗max effort · 31.71 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗xhigh effort · 31.71 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗high effort · 30.86 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
GDP.pdf all-pass ↗high effort · 28.80 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-LCR 1.1 ↗max effort · 84.67 points · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-LCR 1.1 ↗xhigh effort · 84.67 points · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
AA-LCR 1.1 ↗medium effort · 84.33 points · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
Claude Sonnet 5.5Financial analysis · Practical analysis +1Comparable profile option · One publisher · Effort / access caveat+
At least one qualifying effort has complete, priced profile evidence. AA measures low through max with default safety fallback, which Anthropic routes to Sonnet 5, whose input/output list prices match. AA captures the $0.10 cache-read rate, reduced October 7. AA publishes no fallback route or fractions, so costs are primary-tariff scenarios. FrontierCode Claude Code runs do not disclose fallback use and join nothing. No matching direct DeepSWE result exists.
AA-AnalystAgent pass@1 ↗max effort · 69.50 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
Briefcase analytical quality ↗max effort · 2100.10 Elo · Fallback token fractions are not published for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗max effort · 31.43 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
CritPt ↗xhigh effort · 31.14 points · Comparable task accounting is not captured for this benchmarkNear the available leader · AA benchmark harness · fallback enabled · 2026-11-11
At least one qualifying effort has complete, priced profile evidence. AA measured max effort and thinking disabled on the September release. Other native reasoning settings are unmeasured; earlier V4 Flash and V4 Pro measurements do not transfer, and neither do these to DeepSeek-V4.1-Flash-1017, which the deepseek-flash alias has served since 2026-10-17. AA prints that build as separate max, high and non-reasoning rows, stored as DS V4.1 Flash 1017.
AA-LCR 1.1 ↗max effort · 84.00 points · $0.031 / benchmark taskNear the available leader · Quality–cost frontier · AA benchmark harness · single model · 2026-11-11
How options qualify & what this search misses
The screening rules are under How alternatives qualify. No cost means no value claim, and a strength is not an endorsement of the whole model.
On 2026-11-11 this search examined 734 AA configurations, 134 FrontierCode entries and 134 Zapier entries. Coverage and public availability remain incomplete; this is a source-backed shortlist, not proof that every great model was found.
Access reference for all 52 maintained modelsAccess: subscription, API or your hardware
Reviewed 2026-09-10. Availability does not reproduce a benchmark’s agent, effort or fallback configuration.
Subscription Verified route
Plus and Business Standard include limited Astra usage; Pro and Business Premium can use their full existing Work/Codex allowance on Astra.
Plan subscription plus optional credits; task/model-dependent allowance.
Work and Codex share usage; Chat has separate allowances. Optional purchased credits extend usage. Inclusion is not unlimited usage or zero API task cost.
$10 input / $50 output USD per million tokens; standard API billing.
API efforts: low, medium, high, xhigh, max. Harness menus may differ. Long-context surcharges and caching terms are on the model page. Ultra is a separate multiagent system; benchmark scores do not transfer to it.
Use your allowance wellchat, agent and when to switch
Practical guidance for each kind of work, from the configuration the rail names. There is no measured quota optimum.
Use your allowance well.
Settle the decision
Decide what success means with the model that has the useful context.
Your agentClaude Fable 5.2 · max · Estimated
Start here when the investigation needs files, tests or tools.
Limits & when to switch
On Claude Pro and Max, Chat and Claude Code share usage limits. Switching between them adds no separate allowance. Max and Team include monthly Claude Console API credits, which do not cover Claude Code or app extra usage or change those limits.
Skip a separate chat when the plan is settled or the decision needs repository tools. Reconsider the model or effort when the design is unresolved or fixes keep failing; carry the useful context with you.
Move the rail to inspect the trade-off. API savings do not measure subscription savings, and the benchmarks do not identify when to escalate.
Selected model access: Max, Ultra, premium Team and premium legacy seat-based Enterprise seats include Fable models within up to 50% of the shared weekly allowance. Pro, standard Team and standard Enterprise seats use purchased usage credits from the start. A launch offer includes Fable on Pro and Team standard seats through 2026-11-16 23:59 PT. Check access & evidence below.
Decide what success means with the model that has the useful context.
Your agentKimi K3.1 Pro · max · Estimated
Bring the decision. Build, verify and fix in the same task.
Limits & when to switch
Check whether your chat and agent tools share an allowance before moving work between them.
Skip a separate chat when the plan is settled or the decision needs repository tools. Reconsider the model or effort when the design is unresolved or fixes keep failing; carry the useful context with you.
Move the rail to inspect the trade-off. API savings do not measure subscription savings, and the benchmarks do not identify when to escalate.
Selected model access: Kimi Code: k3.1-pro on Moderato / Plus plans and above; up to 1M context on Allegretto / Pro and above. Check access & evidence below.
Product facts checked 2026-10-08.
Practical guidance · no measured quota optimum.
Subscription, API or self-hostingwhat the cost numbers mean
Benchmark API cost is a comparison unit, not your bill. Subscription plans may include an allowance, require credits from the first request, restrict supported tools or leave exact versions undisclosed. Max and Team now include monthly Claude Console API credits; these do not cover Claude Code or app extra usage or change their limits. The selected model’s access disclosure and the full access reference keep those distinctions separate, with official sources and verification dates.
OpenAI’s October system card distinguishes October Chat Sol/Luna checkpoints from September versions in Work and Codex. The Chat variants do not inherit the measurements here; official help and launch pages also disagree about Chat availability and rollout scope.
Downloadable weights establish a self-hosting route, not laptop suitability or savings. The reference gives sourced serving examples and distinguishes author formats, community quantizations and related hosted variants. For example, hosted Qwen Flash is based on Flash-Next with additional production features; it is not silently treated as the same measured weights configuration. Local hardware, electricity and operations costs are not estimated here.
What turning the effort dial up costsmeasured steps and regressions
Turning the dial up buys the model more thinking. You pay for that thinking in tokens, so it is a spending dial first and a quality dial second.
That makes it measurable. Every row here is one model at one setting, scored on DeepSWE v1.1 · mini-swe-agent, Terminal-Bench 4.0 · AA harness, GDPval-AA v2.1 and AutomationBench-AA partial credit and billed with the same job weights. The ladder compares complete measured settings; a missing setting is skipped, not estimated.
GPT-5.6 Terra shows it most starkly. Among models with at least four complete settings, the last step costs about 10× what the first did: $0.020 a point at the bottom, $0.20 at the top. Measured November 11, 2026.
GPT-5.6 Terra — each step of the reasoning dial, on the execution lane
Step
Score
Points bought
Added cost
$ / point
low → medium
23.30
+5.70
$0.1113
$0.0195
medium → high
32.77
+9.47
$0.3694
$0.0390
high → xhigh
39.45
+6.68
$0.7733
$0.1158
xhigh → max
53.98
+14.53
$2.8868
$0.1987
No number here tells you where to stop. The board prices a point at each step. What a wrong answer costs you is the other half, and only you have it.
Higher does not always mean better, either. 2 measured configurations score lower at a higher setting:
GPT-6 Astra · xhigh → max — 0.25 points down, $2.33 more per task. Paying more for a worse result.
Grok 4.6 · high → xhigh — 0.60 points down, $0.76 more per task. Paying more for a worse result.
These measured component scores do not include benchmark error bars. Read that as a reason to check your own setting, not as a law about the model. Either way, “turn it to max” is a guess rather than advice.
Never pick the setting on its own. A cheap model turned up and a costly one turned down are the same kind of thing here — which is why the card compares configurations rather than ranking model names.
What each task is measured onsources, weights and missing evidence
Two owned mixes define the work. Their weights are reasoned policy from a two-week sample of the author's own agent work (213 sessions, 2026-09-03 to 2026-09-16), changed as little as that sample allowed. They are not scientifically established constants. Publisher composite indexes cannot change these formulas or decide which effort settings appear.
Execute it: 40% direct DeepSWE v1.1, 35% AA Terminal-Bench 4.0, 15% AutomationBench-AA partial credit, 10% GDPval-AA v2.1.
What scores the Execute lane
Direct DeepSWE v1.1 measures repository work in mini-swe-agent, using pass@1 over scored attempts on its 113-task set. Provider, verifier and network errors are excluded by the publisher; context-window failures and agent timeouts count as failures. Its source publishes mean cost over scored attempts. Its publisher prices Astra's five costs at current API tariffs.
AA's Terminal-Bench 4.0 measures command-line work in AA's own scaffold. GDPval-AA v2.1 compares professional deliverables. AutomationBench-AA contributes the fraction of objectives completed, including partial credit; it is not the fraction of whole workflows completed. These are separate source runs from the AA Coding Agent Index.
Together, repository and terminal work carry 75% of Execute, while app workflows and documents retain 25%. Each component keeps its exact model, effort, benchmark version and harness. The mix is a workload proxy across those settings, not a measurement of one universal agent. Changing the provider filter changes access, never the benchmark harness or its score.
Fixed conversions preserve differences without depending on the current roster. Both mixes use:
Rate p → 100 × pPublished Omniscience O → (O + 100) / 2Native Briefcase/GDPval E → (E − 500) / 20
Elo is not clipped: 1000 becomes 25 points, 1500 becomes 50 and 2500 becomes 100; higher or negative values remain possible. Twenty Elo per point is a declared cross-benchmark policy, not an empirically established exchange rate. 1000 Elo is not asserted to mean human-level performance. Measured scores do not change when another model is added or filtered. Estimates are re-fitted when the measurements change; UI filters do not re-fit them.
The November 11 capture uses GDPval-AA v2.1, Briefcase v1.1 and AA-Omniscience v1.1 (3,000 questions). Artificial Analysis Intelligence Index 5.0.1, which differs from 5.0 only in regrading CritPt v2, does not score or gate either mix; AA-LCR v1.1 and GDPval-AA v2.1 still do. AA re-fits both Elo scales within those versions; all current scores use one capture's results together. Differences from earlier captures are not evidence of model improvement. The weights and conversions above stay fixed.
Costs use the same weights as quality. AA bills use benchmark token counts, cache rates, prices and published fallback fractions divided by task count. When fallback pricing is unavailable, a separately marked estimate uses the primary model tariff; it is not a verified fallback-inclusive bill. Briefcase analytical quality uses the full Briefcase task bill. Direct DeepSWE costs are already per scored attempt. The resulting cost prices the declared benchmark mix, not the reader's next job or subscription quota.
Missing results and estimates
A missing measurement stays missing in the source. The estimator tries each missing component against available components in the exact configuration. Quality uses a straight line or a curve that levels off near the top, whichever predicts held-out models better for that pair of benchmarks; cost uses a straight line in log dollars. A predictor score beyond every trained configuration is read at the highest or lowest trained value. A predicted score never goes past the best or below the worst measured score on its benchmark. It never sits above the same model's measured higher effort, or below its measured lower effort, by more than the largest such inversion measured on that benchmark between reasoning efforts; a non-reasoning setting, which is not simply a lower effort, allows the larger of that and the largest inversion measured between non-reasoning and reasoning settings. A prediction held at any of those edges says so. Each training model receives equal total weight. Validation hides all efforts of a model together, and the predictor must beat a model-balanced benchmark-mean baseline across at least six other models. Predictions never train on other predictions or borrow an adjacent effort as a measurement. Sparse rows can show predictions, but a recommendation still needs at least half its score weight directly measured.
An Estimated label and hollow chart point identify a configuration with estimated quality or cost. The evidence disclosure shows the measured share, missing components and estimate basis. Statistical predictions include a predictor, held-out error and range. That range adds weighted component error allowances, each the model-balanced 95th-percentile held-out error among the third of measured configurations whose predictor scores are nearest, widened for predictors outside the training range. With each model left out in turn, those allowances covered 90% of held-out configurations on average across benchmark pairs: 89% in each board's bottom quarter and 91% in its top quarter. A single allowance for the whole board covered 84% in the bottom quarter. Coverage varies by pair: for DeepSWE predicted from Terminal-Bench 4.0 it is 85% overall, with a top-quarter allowance a third as wide as the whole-board one. It is an empirical missing-input range, not a confidence interval. Small measured-score differences also need not matter in practice.
Known token accounting with captured fallback tariffs but missing fractions supplies cost bounds from the cheaper and dearer rates in each token category. Their midpoint is a display convention, not an expected bill. If AA publishes a per-benchmark cost but no fallback tariff, its captured tokens priced at the primary tariff supply a scenario, with the actual fallback-inclusive bill and range unknown. Where a separate benchmark's cost is missing, a validated transfer may use that source-derived scenario as its predictor; the resulting estimate has no defensible bill range. Cost and quality estimates are separate; either may remain unavailable. The rail uses the displayed central estimates, so an estimated cost can exceed its indicative ceiling.
Fallback-enabled systems never inherit standalone or fallback-unknown measurements. A prediction about a system absent from the training overlap remains an explicit extrapolation. Generic model names do not establish dated releases. If the evidence is too sparse or a predictor fails validation, the row remains visible without a composite.
A measurement replaces an estimate when it arrives, with its error allowance, at the weekly refresh. An older benchmark version's result never fills a gap in the current one, and the weights stay fixed.
What this composition costs you
Different harnesses still matter. Direct DeepSWE and AA evaluate different scaffolds, tools and tasks. Owning the mix restores useful component coverage but cannot isolate model quality from those environments or predict performance inside Codex or Claude Code. The evidence table retains the source of each number so that trade-off stays inspectable.
Source quality is uneven. DeepSWE uses original engineering tasks and behavior-based verification, but covers a limited repository and language mix. AA's deliverable and analysis comparisons depend on model judges. AutomationBench's partial-credit objectives are easier to satisfy than complete workflows. FrontierCode, Zapier and HLE remain supporting evidence; they do not gate the mixes.
What the Figure lane's own sources are worth
The Figure mix emphasizes analysis, document reasoning and factual reliability. Its sources have their own limits:
HLE is supporting evidence, outside the mix. It has contested answers. FutureHouse examined 321 text-only chemistry and biology questions and found that 29% ± 3.7 had answers contradicted by peer-reviewed literature (FutureHouse, 23 July 2025). The HLE team's own follow-up on that page put roughly 18% of a reviewed subset as problematic under three-expert review and committed to rolling revisions. The finding covers that subset, not the whole exam.
AA-Omniscience's questions are machine-written. This capture uses AA-Omniscience v1.1, 3,000 questions. Artificial Analysis states the questions were generated by an LLM-based question-generation agent from authoritative academic and industry sources (AA, read 2026-10-28). The sources are human; the questions are not. An earlier public description named 6,000 questions; that count is not this version.
AA-LCR v1.1 corrected its own answers. AA's methodology records that v1.1 added a grading system prompt, corrected 16 answer keys and changed the grader, and that scores are not directly comparable with v1.0 (AA methodology, read 2026-09-08).
Briefcase is Artificial Analysis's own private benchmark. Its 91 tasks across four held-out scenarios were written by outside experts from firms including Google, McKinsey and BCG, but AA owns and runs it, only a fifth demonstration scenario is public, and AA's methodology table lists one repeat per task — no averaging over attempts (AA-Briefcase, read 2026-09-08). Its analytical-quality Elo is a preference ranking, which is not a correctness measure.
These are benchmark proxies, not success probabilities. Collaborative problem framing and planning remain unmeasured in Figure; selective escalation and supervision remain unmeasured in Execute. Agent harnesses, tools, task sets and judges differ. Shared AA provenance is not independent corroboration, and small numerical gaps need not matter in real work.
Data and limitscoverage, effort names, raw rows and sources
Which models are included
The comparison uses AA model evidence captured November 11, 2026, and direct DeepSWE v1.1, re-read that day; DeepSWE's source payload is unchanged since its September 22 generation. AA also prints a Daybreak Blue Sol max row with cybersecurity results only; it supplies no component of either mix and does not inherit the standard Sol measurements. The table includes the exact effort settings those sources publish for catalogued models, including rows that cannot form a full mix. It excludes rows AA estimates rather than measures, and the settings named below. The shortlist also preserves independent benchmark observations. Every result keeps its source effort, harness and system; measurements do not transfer across them.
An older model stays while something here depends on it: a frontier step, a route, a qualifying shortlist strength or value option, or a substitute for a named gap. Replacement within its vendor line starts that check; age alone never removes it. Opus 4.8 and Sonnet 5 left on September 29, replaced by Opus 5.5 and Sonnet 5.5. Mistral Large 4 Preview left on November 6, replaced by Mistral Large 4. GPT-5.4 Pro left on November 11, displaced by GPT-5.5 Pro; CritPt xhigh no longer qualifies as a shortlist strength after Fable 5.2 raised the ceiling.
Astra has no non-reasoning row in AA's capture, and current OpenAI documentation does not support that configuration. Muse Spark 1.3 max is publicly available and included; missing costs remain visible. Meta now documents native xhigh and Standard-tier Muse Spark 1.3 max; Contributor variants and Muse Code multi-agent runs do not inherit these measurements. The first-party DeepSeek Flash and Flash Vision aliases now serve V4.1 Flash, whose results cannot be inferred from the measured older versions; on October 17 DeepSeek upgraded V4.1 Flash to V4.1-Flash-1017, which AA prints as its own max, high and thinking-disabled rows, compared below as DeepSeek V4.1 Flash 1017, so the V4.1 Flash rows describe the September release rather than what the alias returns today. V4.1 Flash has its own measured max and thinking-disabled rows. GLM-5.3 is measured at low and max; its high setting is a named gap. AA prints partial results for Agnes 2.5 Pro Beta, with an estimated overall index and no Briefcase, AutomationBench or Terminal-Bench 4.0 result, and none for Agnes 3.0 Flash, which is not compared. Agnes 2.5 Pro Beta is not compared either: its AA-LCR, 83.0, is beaten by GPT-6 Luna max at 83.3, and its one shortlist observation is on a benchmark neither mix reads. Every measured non-reasoning row of a catalogued model is included like any other effort, including GPT-5.6 Luna, Sol and Terra, GPT-5.5, Qwen3.8 27B and DeepSeek V4 Pro 0813. SWE-2 has scoped FrontierCode evidence without a complete lane mix. Step 5 Preview keeps an undisclosed AA effort setting, so it is read as its vendor's default and labeled default (assumed). Qwen3.8 Max 0902 is stored at xhigh, which AA now names and Alibaba documents as the default. Qwen3.8-Flash-Next names no effort, so that row is read as the vendor default and labeled default (assumed). Step’s scoped task costs use AA’s captured $0.10 per million cached-input tokens; StepFun’s pricing page now lists the same rate; its October 8 version listed $0.05. Grok 4.7 low, high and xhigh have AA task costs alongside their measured Figure scores; xAI's reasoning guide also lists medium, which is unmeasured. MiMo-V2.6-Pro and MiMo-V2.6-Flash each have measured Figure evidence with thinking enabled, their native default. Grok 4.7 and both MiMo models lack a matching direct DeepSWE result; Execute uses separately labelled estimates where validation permits. Pro and Flash results do not transfer to each other, and UltraSpeed, thinking-disabled use and MiMo's open RL weights are unmeasured.
GPT-6 Sol and Luna each have six exact AA settings, with non-reasoning kept distinct from the five reasoning efforts. Neither has a matching direct DeepSWE result; the Execute lane labels any validated prediction separately. Opus 5.5 has five AA settings with its default safety fallback, and its Figure scores are measured. AA's fallback table lists a fallback tariff for each of the five efforts and, among the benchmarks used here, per-benchmark fractions for AA-Omniscience only. Its Omniscience cost is therefore a direct bill, and every other Opus 5.5 AA cost is a cost bound shown at its midpoint: the primary tariff is the low end and the fallback tariff the high end, so every Opus 5.5 Figure cost is a cost bound too. Opus 5.5 has no matching direct DeepSWE result: its Execute score predicts the missing 40%, and its direct DeepSWE task cost is predicted from its own AA-Omniscience bill, so its Execute costs carry a range. At max that bill is above every trained configuration, so the DeepSWE cost is held at the highest cost among the configurations it was trained on, $11.84, and the range is wide.
GPT-6.1 Sol has five exact AA settings, low to max, released September 29. OpenAI's model page lists those five efforts and excludes none and minimal; it mentions no fallback routing, so the rows are treated as standalone. It prices at $2 input, $0.10 cached input, $2.50 cache write and $10 output per million tokens. Its Figure scores are fully measured: max scores 72.2 at $0.76 a task, against Astra max at 71.4 and $3.19, which makes it the Codex Figure pick with no budget limit. The margin is 0.82 points, between two measured scores, so no estimate is involved; 0 of the 23 alternative weightings in the method sensitivity table put Astra max ahead. It has no matching direct DeepSWE result, and figures from other sources do not substitute for one, so its Execute score predicts the missing 40%: at max it reads about 63.8 (range 61.1 to 66.6) at about $1.50 (range $1.01 to $2.93), under the measured Astra xhigh at 65.9. GPT-6 Sol and GPT-5.6 Sol are superseded by it but stay, since a scoped shortlist strength or value option depends on each.
GPT-6.1 Terra has six exact AA settings, none through max, released October 26. OpenAI's model page lists those six efforts, with medium the API default, and excludes minimal; it mentions no fallback routing, so the rows are treated as standalone. It prices at $1 input, $0.05 cached input, $1.25 cache write and $5 output per million tokens. Figure is fully measured: max scores 65.6 at $0.29 a task, behind Sol 6.1 max at 72.2 and $0.76, so max is not a rung; medium is on Codex Figure, and low is too. Direct DeepSWE has no Terra 6.1 row, so Execute predicts the missing 40%: at max it reads about 62.3 (range 59.5 to 65.0) at about $1.25 (range $0.82 to $2.54). On All models Execute only low remains; medium and high sit on the Codex Execute frontier; max does not. GPT-5.6 Terra is superseded by it but stays, since a scoped shortlist strength or value option depends on it.
Grok 4.7 low scores 67.8 on Figure at $0.70 a task, against 70.3 at $1.99 for high. DeepSeek V4 Pro 0813's non-reasoning row, none in the chart, scores 36.0 at $0.15. AA now prices both none and max at DeepSeek's peak rates, $1.32 input and $3.96 output per million tokens. Off-peak rates are half those prices; the chart retains the captured peak basis.
Claude Haiku 5.5, released October 7, has five measured AA efforts, low to max; Anthropic's effort guide makes medium the API default. On October 8 AA removed Default Fallback from the same five row IDs, matching Anthropic's fallback guide: Haiku has no server-side fallback. Its Figure score at max is 68.1. AA restored lower-tier tariffs, but its row-specific caveat says the fivefold long-context pricing is not yet reflected. A separate release note says costs reflect tiered pricing; the captured payload still has only aggregate tokens and lower-tier tariffs, with no request-tier allocations.
Native bills stay missing, but the comparison includes a marked tier scenario. For Haiku max, the standard Figure mix gives $0.21–$1.05 per task, conditional on AA's captured tokens, cache-hit rate, five-minute cache writes and task counts. The rail uses the $0.63 midpoint scenario, not an expected price or an inferred premium share. The comparison shows whether the budget choice changes at the lower endpoint, midpoint and upper endpoint; these readings do not cover every possible request allocation. Other billing modifiers are outside the envelope. Anthropic's model page lists $0.10 input, $0.01 cached input and $0.50 output per million tokens through 100,000 prompt tokens, with fivefold rates above that threshold. AA also flags elevated pre-release AutomationBench refusals and plans reevaluation; the comparison retains the published scores. No effort has a matching direct DeepSWE measurement. Execute predicts that missing component using the existing model-held-out estimator, trained only on other models' measured results. Its cost is seeded from the marked AA tier scenario; the source accounting and transfer errors are not jointly bounded, so the total Execute bill has no established bound.
Mistral released Mistral Large 4 with its weights on October 29 (release post). The release post and the model card say mistral-large-4 now serves that build, which replaces the October 6 public preview; the preview's measurement does not transfer, and the API's model page still describes the preview. It takes low, medium or high, with high the recommended default (reasoning guide), and the release post lists $0.70 input, $0.07 cached input and $2.10 output per million tokens, which are AA's captured rates. AA measures high and medium; low is unmeasured. Neither measured setting has an AA-LCR result, so Figure predicts that share: high reads about 59.2 (range 57.4 to 61.0) at about $0.37 (range $0.34 to $0.55) a task, behind MiMo-V2.6-Pro at 64.3 and $0.13. Execute predicts the missing DeepSWE and GDPval-AA shares and reads about 49.3 (range 44.7 to 53.9) at about $2.86 (range $2.29 to $3.95), behind MiMo-V2.6-Flash at 49.9 and about $0.33, itself an estimate. Neither lane puts either setting on a frontier. The weights, and a separate NVFP4 checkpoint, are published under Mistral's own open-weight license; neither inherits the API's scores or costs.
Moonshot released Kimi K3.1 Pro on October 23 as kimi-k3.1-pro (quickstart). It always thinks, takes low, high or max with max the default, and lists $2 input, $0.20 cached input and $10 output per million tokens, which are AA's captured rates. Moonshot published the weights on November 2 under its own license; self-hosted costs are not priced here. AA records the effort it ran as max, which is Moonshot's default; low and high do not inherit it. Figure is fully measured at 74.0 and $2.25 a task, between Opus 5.5 medium at 73.6 and about $1.96 and high at 75.3 and about $2.76. Direct DeepSWE has no Kimi K3.1 Pro row, so Execute predicts the missing 40% and reads about 70.3 (range 67.4 to 73.3) at about $3.54 (range $2.95 to $4.63), above Sonnet 5.5 max at 68.8 and about $7.27; the two score ranges overlap. Its Terminal-Bench 4.0 score, 67.2%, is above every trained configuration, so the prediction is held at the edge of the measured range and flagged. On that estimate it leads All models Execute at about half Sonnet 5.5 max's cost. Kimi K3 stays: a scoped shortlist strength still depends on it.
MiniMax released MiniMax-M3.1-Flash on October 19 (launch post). It always thinks, takes low, medium, high, xhigh or max with max the default (API reference), and lists $0.15 input, $0.015 cached input and $0.60 output per million tokens, which are AA's captured rates. AA records the effort it ran as max, which is MiniMax's default; it is MiniMax's only model here. Figure is fully measured at 59.8 and $0.16 a task, behind MiMo-V2.6-Pro at 64.3 and $0.13. Direct DeepSWE has no row, so Execute predicts the missing 40% and reads about 51.9 (range 48.5 to 55.3) at about $0.87 (range $0.54 to $1.88), behind Sol 6.1 medium, which scores 59.6 at about $0.67. Its strongest result is AutomationBench-AA partial credit, 72.5 at $0.066 a task, which qualifies it as a scoped value option on the shortlist. MiniMax's own DeepSWE v1.1 figure of 73.7% is a vendor statement and is not scored; the direct board lists no MiniMax result. Self-hosted weights are not verified.
DeepSeek upgraded V4.1 Flash to V4.1-Flash-1017 on October 17; deepseek-flash serves it, and the retired deepseek-v4-flash names are still accepted and routed to it, at the Flash prices AA captured; AA's costs use the peak rates, and off-peak is half. Its weights are published under MIT, and a self-hosted copy inherits none of these results. AA measures max, high, the API default, and thinking disabled, none in the chart; low is unmeasured and the September release's results do not transfer to this build. At max Figure is fully measured at 64.0 and $0.21 a task. Execute predicts the missing DeepSWE share and reads about 55.9 (range 52.5 to 59.4) at about $0.86 (range $0.53 to $1.84). Its GDPval-AA result, 62.3 converted points at $0.35 a task, qualifies it as a scoped value option. high has no AA-LCR or GDPval-AA result: Figure reads about 58.5 (range 56.4 to 60.2) at about $0.13 (range $0.11 to $0.21) and Execute about 52.7 (range 48.1 to 57.3) at about $0.78 (range $0.46 to $1.71). Thinking disabled scores 52.8 on Figure, fully measured, at $0.25, and about 39.0 (range 29.1 to 48.9) on Execute at about $0.57 (range $0.31 to $1.33). DeepSeek's own DeepSWE v1.1 figure of 77.0 is a vendor statement and is not scored; the direct board lists no result for this build. No setting is on an All models frontier; with OpenAI models excluded, neither is. Figure max sits below MiMo-V2.6-Pro and the September V4.1 Flash max; Muse Ember takes the other Figure rows and the Execute steps.
Gemini 4 Argon reached general availability in the Gemini API on October 13 (changelog); Google announced it on September 30 for vetted cyber defenders only, and the model id is gemini-4-argon. Google's thinking guide lists low, medium and high, with medium the default. AA measures High only, on a row dated September 30, before general availability, and does not say how it ran the model, so the comparison assumes the row is the model Google now serves. Argon is compared at high only. medium, the default, is a named gap; FrontierCode and Zapier also print medium, supporting evidence that scores neither mix. Costs keep AA's captured rates of $4 input, $0.20 cached input and $20 output per million tokens; AA prints no cache-write tariff. Google's pricing page still lists $2 / $0.10 / $10 through November 30 and doubles all three from December 1; this capture already stores the doubled rates. Figure is fully measured at 68.9 and $3.65 a task; the November 6 capture records fewer of Argon's input tokens as cache hits than the October 28 one did, at unchanged rates, so its costs rose. Direct DeepSWE has no Argon row, so Execute predicts the missing 40% and reads about 66.2 (range 63.5 to 69.0) at about $7.38. That is a higher estimated score than Astra xhigh's measured 65.9, at an estimated $7.38 (range $5.94 to $12.06) above Astra's measured $4.34, and the score range includes Astra's figure. Neither is a rung on the All models Execute frontier, because Kimi K3.1 Pro max scores more for less; Astra xhigh stays the Codex Execute pick. With Anthropic and Moonshot models excluded, Carbon high leads All models Execute on its estimate and on measured components alone; Astra xhigh leads when every missing DeepSWE result is set to the measured average. Against Argon's estimate, Sonnet 5.5 max stays ahead only while its DeepSWE result is above 66.3. Google's own DeepSWE v1.1 figure of 77.9% is a vendor statement and is not scored; the direct board lists no Argon result.
Meta released Muse Wonder on October 12 as muse-wonder-1.0. Meta's model list calls it a preview in the Model API, at $4 input, $0.20 cached input and $20 output per million tokens. The reasoning page lists minimal through max, rejects none, and says max on this model does not select Muse Wonder Full. AA measures max and high. At max Figure is fully measured at 71.2 and $2.76 a task, behind Sol 6.1 max at 72.2 and $0.76; Execute predicts the missing DeepSWE share and reads about 63.6 (range 60.8 to 66.3) at about $6.74 (range $5.38 to $11.17). high has no AA-LCR or GDPval-AA result: Figure reads about 70.0 (range 68.0 to 71.2) at about $1.79 (range $1.73 to $2.09) and Execute about 57.7 (range 54.3 to 60.8) at about $4.74 (range $3.89 to $6.34). Neither setting is on an All models frontier; with OpenAI models excluded, high is a Figure rung. Muse Wonder Full, which AA also prints at Max, is for Muse Code Power and Maximum subscribers only, with no API and so no comparable task price. It is a named absence, not a row: its AA results do not transfer to Muse Wonder, and Meta's own DeepSWE claims for the Muse Wonder line, which cover the standard model as well, are not scored.
Thinking Machines released Inkling 2 Small on November 5 (release post). Tinker samples it as thinkingmachines/Inkling-2-Small, with a 64K context, and as a 256K variant, at $0.30 input, $0.03 cached input and $1.00 output per million tokens, which are AA's captured rates; serverless inference, in beta with a waitlist for production use, serves only a 256K NVFP4 build. AA measured a release candidate and does not say which endpoint, context or precision it called. The 64K id cannot hold AA-LCR's inputs, which need at least 128K, so the row did not come from it, and whether it describes the 256K variant or the NVFP4 build is not published. Effort is a number from 0 to just under 1, with named presets from none to xhigh and high, 0.9, the default (effort guide). AA does not say which effort it measured, so the row is read as high and labeled default (assumed). The release post says its own evaluations ran at xhigh, but it prints its GDPval-AA, AA-LCR, AA-Omniscience and EnterpriseOps-Gym results as AA's, so it does not say which effort AA ran either. Figure is fully measured at 65.2 and $0.32 a task, behind the fully measured GPT-6.1 Sol medium at 68.6 and $0.20; it is a Figure rung only with OpenAI and DeepSeek models both excluded. Direct DeepSWE has no row, so Execute predicts the missing 40% and reads about 42.0 (range 32.8 to 51.3) at about $1.76 (range $1.39 to $2.43), behind GPT-6.1 Terra low at 42.7 and about $0.22, itself an estimate. Its AnalystAgent and EnterpriseOps-Gym results are scoped shortlist strengths. Its own DeepSWE v1.1 figure of 72.1% is a vendor statement and is not scored. Weights are closed; the release post says they will follow under Apache 2.0.
Tencent made Hy4 generally available on November 4 as hy4 on TokenHub, its OpenAI-compatible API, at a list price of $0.834 input, $0.042 cached input and $2.501 output per million tokens, which are AA's captured rates. Until December 31, 2026, Tencent's billing terms settle Hy4 at half its list price, while TokenHub's page headlines 75% off without stating terms; the costs here stay at list. AA prints one Hy4 row, a pre-release build dated November 3 at hy4's list rate, which hy4-preview shares. Nothing published ties that build, or a serving precision, to either id, so the row is compared as a related variant: its results are not verified to transfer to hy4. Tencent's deep reasoning guide sets hy4 to high by default and also offers no_think; AA does not name the effort it ran, so the row is read as high and labeled default (assumed), and no_think is unmeasured. It has no AA-LCR result, so Figure predicts that share and reads about 60.8 (range 58.8 to 62.8) at about $0.44 (range $0.41 to $0.64) a task, behind GPT-6.1 Sol low at 63.9 and $0.11. GDPval-AA and direct DeepSWE are missing too, so Execute predicts half its score: about 56.2 (range 52.5 to 59.9) at about $1.35 (range $0.86 to $2.79), behind GPT-6.1 Sol medium at 59.6 and about $0.67, itself an estimate. Muse Ember's cheaper Execute rungs now sit below Sol 6.1 medium, so Hy4 is not an Execute rung even with OpenAI models excluded. Tencent has not published Hy4's weights.
Anthropic released Claude Fable 5.2 on November 9 as claude-fable-5-2. Adaptive thinking is always on; effort is low through max with high the API default (overview). It prices at $8 input, $0.20 cached input and $40 output per million tokens, which are AA's captured rates; the $16 / $80 fast setting is unmeasured, and Mythos 5.2 is a separate limited-availability id. AA measures all five efforts with Default Fallback. Figure max scores 81.1 at about $18.21 (range $18.13 to $18.28) a task, against Opus 5.5 max at 79.4 and about $9.27, which makes it the All models and Claude Code Figure pick with no budget limit. Direct DeepSWE has no Fable 5.2 row, so Execute predicts the missing 40%: at max it reads about 68.0 (range 65.3 to 70.8) at about $17.05 (range $14.24 to $26.01). medium is on All models Execute, between Sol 6.1 max and Kimi K3.1 Pro max. Claude Code requires 2.1.321 or newer. A launch offer includes Fable on Pro and Team standard seats through 2026-11-16 23:59 PT. Fable 5.1 stays: a scoped shortlist strength still depends on it.
Google made Gemini 4 Carbon generally available in the Gemini API on November 10 (changelog); the model id is gemini-4-carbon. Google's thinking guide lists low, medium and high, with medium the default. AA measures all three, on rows dated October 28. Costs keep AA's captured rates of $4 input, $0.20 cached input and $20 output per million tokens, the rate Google lists from December 1; Google's pricing page still lists $2 / $0.10 / $10 through November 30. Figure is fully measured at 72.1 and $2.73 a task at high. Direct DeepSWE has no Carbon row, so Execute predicts the missing 40% and reads about 69.7 (range 66.8 to 72.6) at about $5.90 (range $4.67 to $9.89). That is the All models Execute runner-up, at more than Kimi K3.1 Pro max costs for a lower score, so it is not a rung. medium has no AA-LCR result: Figure reads about 70.9 (range 68.9 to 71.5) at about $2.21 (range $2.14 to $2.56). low likewise: Figure reads about 68.3 (range 66.2 to 69.1) at about $2.24 (range $2.18 to $2.60). Neither setting is on an All models frontier.
Meta released Muse Ember on November 9 as muse-ember-1.0. Meta's docs list it in Standard only, at $0.10 input, $0.004 cached input and $0.50 output per million tokens up to 200K input tokens, and $0.30 / $0.012 / $1.50 for the whole request above that; AA's captured rates are the $0.10 / $0.50 tier. The reasoning page lists minimal through max, with max the default. AA measures max and xhigh. At max Figure reads about 63.9 (range 61.9 to 66.0) at about $0.050 (range $0.042 to $0.093) a task; Execute predicts half its score: about 58.4 (range 55.0 to 61.9) at about $0.36 (range $0.19 to $0.89). Both settings sit on the All models Figure and Execute frontiers, between Luna 6 xhigh and MiMo-V2.6-Pro on Figure and between Terra 6.1 low and Sol 6.1 medium on Execute. With OpenAI models excluded, Muse Ember max is an Execute rung, between Muse Ember xhigh and Fable 5.2 low. minimal, low, medium and high are unmeasured. Contributor is not available. Direct DeepSWE is unmeasured.
Ling 3.0 Flash VL was a scoped document-reasoning value option while AA priced it. Its author card publishes separate MIT weights; the vendor changelog dates public API access to September 4, while AA dates the row September 10. AA leaves effort undisclosed, so the row is read as the vendor default and labeled default (assumed). AA withdrew its price fields on October 14, so its native task costs are unavailable; the official price page is not substituted, and the measured scores stay without a cost. It has no matching direct DeepSWE result.
Ling 3.1 Flash, JT-4.1 Flash 236B A21B and Solar Mini 4 have AA rows but no result that qualifies for the shortlist, so none is catalogued. Ling 3.1 Flash 1019 (October 22) has every core Figure component measured but is not compared: on Figure it scores 64.1 at $1.18 a task, beaten on score and cost by MiMo-V2.6-Pro, 64.3 at $0.13 a task; on Execute, with DeepSWE predicted, it is beaten by Sol 6.1 medium, which scores 59.6 at about $0.67, and still is with DeepSWE set to the best measured 74.1. Nemotron 3.5 Bolt E4B has no published component scores. Nanbeige4.3-33B (October 25) publishes Apache 2.0 weights but has no API price or hosted endpoint, so it has no task cost and is not compared.
Sparse predictions appear for other models, but rows with less than half the score weight measured cannot become recommendations. GPT-5.5 Pro xhigh has no exact component evidence to seed a model-specific estimate. The shortlist states evidence limits beside each model.
The Execute leader and its limits
With no budget limit, as of the November 11 capture, Kimi K3.1 Pro max leads Execute for All models, at about 70.3 and about $3.54 (range $2.95 to $4.63) a task, and Sonnet 5.5 max leads for Claude Code, at about 68.8 and about $7.27. The Kimi K3.1 Pro lead rests on the predicted DeepSWE result. Its score range overlaps the range for Gemini 4 Carbon high, so the answer says close call. On the measured 60% of the score alone it still leads. Terminal-Bench 4.0 is 67.2% against 65.2% on Carbon and GDPval-AA 1861 against 1733 Elo; AutomationBench is 73.3% against 77.8% on Carbon. Neither has a matching direct DeepSWE result, and both are predicted 72.6 on it, just under the best measured 74.1. Both Terminal-Bench 4.0 scores are above every trained configuration, so each is read at the highest trained score, 59.6, and both predictions are flagged.
For Claude Code, Sonnet 5.5 max's range overlaps the range for Fable 5.2 max, so that lead is not settled either, and the answer says close call. On the measured 60% alone Sonnet 5.5 still leads. Terminal-Bench 4.0 is 63.6% against 59.6% on Fable 5.2 and GDPval-AA 1838 against 1831 Elo; AutomationBench is 71.8% against 76.4% on Fable 5.2. Fable 5.2 max also has no direct DeepSWE result and is predicted 72.6. Like every fallback-enabled system, each prediction is also an extrapolation from systems measured without a fallback. Sonnet 5.5 comes to about $7.27 per Execute task and Fable 5.2 to about $17.05 (range $14.24 to $26.01).
For All models the lead also rests on that prediction. The fully measured Astra xhigh scores 65.9; Kimi K3.1 Pro stays ahead of it only while its DeepSWE result is above 61.5. Of the configurations with a measured DeepSWE result, the five with the highest Terminal-Bench 4.0 scores, Astra at medium to max and Opus 5 max, reach 72.8 to 74.1, so the lead is plausible but not measured. The leaders were also scored four ways: as published, with a plain straight line and no bounds, with every missing DeepSWE result set to the measured average of 56.7, and on measured components alone. Only the measured average changes the All models leader, to Astra xhigh; the Claude Code leader is the same in all four. For Codex, Astra xhigh leads Execute with a margin of 0.2 points over Astra max. The range for GPT-6.1 Sol max includes Astra xhigh's score, and the plain straight line, which has no bounds, would put GPT-6.1 Sol max first at 67.3, so whether Astra xhigh stays ahead depends on the estimate for GPT-6.1 Sol's missing DeepSWE result.
Sonnet 5.5 has five AA settings with the default safety fallback. Anthropic's Sonnet 5.5 release notes say the fallback sends requests Sonnet 5.5 refuses on cybersecurity or AI-development grounds to Sonnet 5, whose input and output list prices match. Anthropic's platform release notes date Sonnet 5.5's cache-read reduction from $0.20 to $0.10 per million tokens to October 7; AA's capture uses the new rate. AA publishes no route or token split for those retries, so how many tokens the retries use, and whether AA's token counts include them, are unknown; every Sonnet 5.5 AA cost is therefore a primary-tariff scenario without a range. At max, Sonnet 5.5 comes to about $7.27 per Execute task and Opus 5.5 to about $12.65 (range $6.74 to $31.54). Both include an estimated cost for the missing DeepSWE share: Sonnet 5.5's is predicted from its own Terminal-Bench 4.0 cost, a primary-tariff scenario, and Opus 5.5's from its own AA-Omniscience bill, held at the edge of the measured range, with the rest of its AA costs at the midpoint of their cost bounds. The gap between them says little, and neither is a measured Execute bill.
Supporting benchmarks and missing results
Additional evidence can qualify a scoped shortlist alternative without changing the broad score. Figure includes GDP.pdf, CritPt, AnalystAgent and ITBench. Execute includes FrontierCode, Zapier, SciCode, EnterpriseOps-Gym and APEX-Agents. Supporting observations never replace a missing core component. Coverage varies across exact configurations; not measured is a missing result, not poor performance.
AA-Briefcase separately reports analytical quality and presentation. A low presentation score does not arithmetically lower its separate analytical-quality Elo. Its rubric, analytical Elo, and overall Elo answer different questions; none is silently substituted for another here.
Where the numbers come from
Artificial Analysis, direct DeepSWE, Cognition, and Zapier supply the measurements and accounting. Missing measurements or task accounting remain visible; a track can be selected only when its mix is measured or completed by a validated estimate.
Effort dials, model by model
GPT-6 Astra has measured low, medium, high, xhigh, and max configurations here. OpenAI's model guide excludes none; Pro is a separate class.
GPT-6.1 Sol has measured low, medium, high, xhigh, and max configurations here, with medium the API default. OpenAI's model guide excludes none and minimal.
GPT-6.1 Terra has measured none, low, medium, high, xhigh, and max configurations here, with medium the API default. OpenAI's model page excludes minimal.
Claude's effort documentation distinguishes model-specific effort settings. The measured Fable configurations include the source's fallback behavior; the direct bill accounts for those fractions.
Gemini Flash uses low, medium, and high thinking levels here. Each is independently measured. Gemini 4 Argon uses the same three levels, with medium the default; only high is measured, and medium is a named gap. Gemini 4 Carbon uses the same three; AA reports all three.
Claude Fable 5.2 has measured low through max here, with high the API default. The measured configurations include the source's fallback behavior.
Muse Wonder documents minimal through max and rejects none; AA reports max and high.
Muse Ember documents minimal through max, with max the default; AA reports max and xhigh.
Mistral Large 4 accepts low, medium and high, with high the recommended default; AA reports high and medium.
Inkling 2 Small takes an effort number with presets from none to xhigh, high the default. AA does not disclose its effort, so the one row is read as high, the default, and labeled default (assumed); the vendor's own evaluations ran at xhigh.
Hy4 defaults to high and also offers no_think. AA does not disclose its effort, so the one row is read as high and labeled default (assumed); no_think is unmeasured.
MiniMax M3.1 Flash always thinks and accepts low, medium, high, xhigh and max, with max the default. AA records the effort it ran as max.
Kimi K3.1 Pro always thinks and accepts low, high and max, with max the default. AA records the effort it ran as max.
DeepSeek V4.1 Flash exposes public API low, high (default), and max. medium and xhigh map to high; ultra maps to max; none disables thinking. Max and thinking disabled are measured here, on the September release; the alias has served V4.1-Flash-1017 since October 17, and that build has its own max, high and thinking-disabled rows.
Earlier DeepSeek versions and GLM document low / high / max; an unmeasured setting stays a named gap.
Qwen3.8-Flash-Next distinguishes its effort dial from disabling thinking. AA names no effort on the stored row, so it is read as the vendor default and labeled default (assumed).
What the prices do not include
Benchmark API cost is not your invoice. It includes the evaluated runs' recorded API usage, but does not price your extra retries, human review, latency, subscriptions, or the cost of a wrong answer. An API cost ratio is not a subscription-quota ratio. The rail prices a benchmark task mix as shares of a job; it does not forecast the cost of your next job, and the job split it uses is a control you set, not a measurement of you.
145 catalogued configurations · November 11, 2026 · ordered by Figure score. Each lane uses its declared job weights. ≈ marks an estimated score or cost; row details give its basis, any available range, measured gaps and recommendation eligibility. These scores are benchmark proxies, not probabilities of success.
Model · effort
Figure
$ / figure task
Execute
$ / execute task
Claude Fable 5.2 · maxEvidence and eligibility
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 81.1; no score component estimated. API cost ≈$18.21 per mix task; range $18.13–$18.28.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-LCR 1.1 · cost: $1.05 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈68.0; missing-input range 65.3–70.8 points. API cost ≈$17.05 per mix task; range $14.24–$26.01.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $28.19 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
81.08
≈$18.2067
≈68.01
≈$17.0538
Claude Opus 5.5 · maxEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 79.4; no score component estimated. API cost ≈$9.27 per mix task; range $6.59–$11.96.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $29.90 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.76 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈67.2; missing-input range 64.4–69.9 points. API cost ≈$12.65 per mix task; range $6.74–$31.54.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-Omniscience index. 8 training models, 17 configurations. Held-out mean absolute error 0.57 log dollars; benchmark-mean baseline 0.86.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; The fitted curve passes the highest measured cost, so the prediction is held there; Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $18.40 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $12.14 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $1.71 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
79.41
≈$9.2736
≈67.16
≈$12.6463
Claude Fable 5.2 · xhighEvidence and eligibility
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 79.2; no score component estimated. API cost ≈$8.87 per mix task; range $8.81–$8.94.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-LCR 1.1 · cost: $0.97 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈68.0; missing-input range 65.2–70.8 points. API cost ≈$11.14 per mix task; range $9.08–$17.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $10.03 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
79.16
≈$8.8735
≈67.98
≈$11.1389
Claude Fable 5.2 · highEvidence and eligibility
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 78.0; no score component estimated. API cost ≈$4.94 per mix task; range $4.87–$5.01.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-LCR 1.1 · cost: $0.96 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈66.7; missing-input range 63.9–69.4 points. API cost ≈$7.49 per mix task; range $5.93–$12.27.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $4.41 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
77.98
≈$4.9412
≈66.69
≈$7.4867
Claude Opus 5.5 · xhighEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 77.7; no score component estimated. API cost ≈$5.40 per mix task; range $3.87–$6.93.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $17.30 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.67 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈66.3; missing-input range 63.6–69.1 points. API cost ≈$6.41 per mix task; range $4.43–$8.84.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-Omniscience index. 8 training models, 17 configurations. Held-out mean absolute error 0.57 log dollars; benchmark-mean baseline 0.86.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $12.27 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $5.62 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $1.07 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
77.74
≈$5.3979
≈66.31
≈$6.4105
Claude Fable 5.1 · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute estimate
60% of score weight measured. Score ≈62.2; missing-input range 59.5–65.0 points. API cost ≈$10.41 per mix task; range $8.70–$15.97.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
76.72
$7.4199
≈62.22
≈$10.4120
Claude Fable 5.2 · mediumEvidence and eligibility
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 76.0; no score component estimated. API cost ≈$2.34 per mix task; range $2.28–$2.41.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-LCR 1.1 · cost: $0.95 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈63.9; missing-input range 61.2–66.7 points. API cost ≈$3.45 per mix task; range $2.83–$4.58.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.49 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
75.99
≈$2.3421
≈63.94
≈$3.4505
Claude Opus 5.5 · highEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 75.3; no score component estimated. API cost ≈$2.76 per mix task; range $2.06–$3.46.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $8.52 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.67 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈64.3; missing-input range 61.5–67.0 points. API cost ≈$3.76 per mix task; range $2.55–$5.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-Omniscience index. 8 training models, 17 configurations. Held-out mean absolute error 0.57 log dollars; benchmark-mean baseline 0.86.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $7.09 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.96 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.85 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
75.29
≈$2.7588
≈64.29
≈$3.7587
Claude Fable 5.2 · lowEvidence and eligibility
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 75.1; no score component estimated. API cost ≈$1.97 per mix task; range $1.91–$2.04.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-LCR 1.1 · cost: $0.95 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈63.4; missing-input range 60.7–66.1 points. API cost ≈$2.96 per mix task; range $2.40–$3.98.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.07 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
75.06
≈$1.9749
≈63.40
≈$2.9609
Claude Fable 5.1 · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute estimate
60% of score weight measured. Score ≈63.0; missing-input range 60.3–65.8 points. API cost ≈$8.72 per mix task; range $7.17–$13.75.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
74.97
$5.8269
≈63.03
≈$8.7224
Claude Sonnet 5.5 · maxEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Figure · AA-Omniscience index: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 74.9; no score component estimated. API cost ≈$6.30 per mix task; fallback-inclusive bill and range unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $20.47 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.34 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.15 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈68.8; missing-input range 65.9–71.6 points. API cost ≈$7.27 per mix task; range unknown, because part of the cost is an accounting scenario.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s primary-tariff scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $12.65 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $6.84 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $1.03 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
74.89
≈$6.3047
≈68.76
≈$7.2746
Kimi K3.1 Pro · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈70.3; missing-input range 67.4–73.3 points. API cost ≈$3.54 per mix task; range $2.95–$4.63.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
74.01
$2.2476
≈70.34
≈$3.5352
Claude Opus 5.5 · mediumEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 73.6; no score component estimated. API cost ≈$1.96 per mix task; range $1.50–$2.42.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $5.86 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.66 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈61.8; missing-input range 59.1–64.6 points. API cost ≈$3.05 per mix task; range $2.04–$4.45.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-Omniscience index. 8 training models, 17 configurations. Held-out mean absolute error 0.57 log dollars; benchmark-mean baseline 0.86.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $5.63 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.05 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.77 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
73.62
≈$1.9592
≈61.84
≈$3.0549
Claude Fable 5.1 · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute estimate
60% of score weight measured. Score ≈61.0; missing-input range 58.3–63.8 points. API cost ≈$6.54 per mix task; range $5.22–$10.83.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
72.96
$3.8732
≈61.02
≈$6.5420
Claude Opus 5 · max
72.34
$5.5770
61.24
$12.3602
GPT-6.1 Sol · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈63.8; missing-input range 61.1–66.6 points. API cost ≈$1.50 per mix task; range $1.01–$2.93.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
72.23
$0.7609
≈63.82
≈$1.4955
Gemini 4 Carbon · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈69.7; missing-input range 66.8–72.6 points. API cost ≈$5.90 per mix task; range $4.67–$9.89.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
72.08
$2.7327
≈69.67
≈$5.8971
Claude Fable 5.1 · mediumEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute estimate
60% of score weight measured. Score ≈57.7; missing-input range 55.0–60.5 points. API cost ≈$5.28 per mix task; range $4.12–$9.04.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
71.60
$3.0230
≈57.71
≈$5.2810
GPT-6 Astra · max
71.41
$3.1894
65.68
$6.6658
Muse Wonder · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈63.6; missing-input range 60.8–66.3 points. API cost ≈$6.74 per mix task; range $5.38–$11.17.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
71.16
$2.7552
≈63.57
≈$6.7417
Gemini 4 Carbon · mediumEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈70.9; missing-input range 68.9–71.5 points. API cost ≈$2.21 per mix task; range $2.14–$2.56.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 42 training models, 122 configurations. Held-out mean absolute error 3.99 points; benchmark-mean baseline 4.35.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.27 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 100 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈67.3; missing-input range 63.8–70.8 points. API cost ≈$5.27 per mix task; range $3.99–$9.23.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 42 training models, 121 configurations. Held-out mean absolute error 3.45 points; benchmark-mean baseline 8.44.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $2.72 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 98 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.26.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈70.95
≈$2.2104
≈67.28
≈$5.2686
GPT-6 Astra · xhigh
70.75
$2.2807
65.93
$4.3358
Grok 4.7 · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈52.1; missing-input range 48.7–55.5 points. API cost ≈$7.04 per mix task; range $5.66–$11.51.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
70.67
$2.4925
≈52.10
≈$7.0408
Claude Sonnet 5.5 · xhighEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Figure · AA-Omniscience index: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 70.6; no score component estimated. API cost ≈$2.19 per mix task; fallback-inclusive bill and range unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $6.98 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.30 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0088 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈64.9; missing-input range 62.2–67.7 points. API cost ≈$3.75 per mix task; range unknown, because part of the cost is an accounting scenario.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s primary-tariff scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $6.08 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.95 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.42 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
70.62
≈$2.1853
≈64.94
≈$3.7520
GPT-6 Astra · high
70.50
$1.7282
63.38
$3.4234
Grok 4.7 · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈51.3; missing-input range 47.9–54.7 points. API cost ≈$4.79 per mix task; range $4.06–$6.15.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
70.31
$1.9946
≈51.33
≈$4.7874
GPT-6.1 Sol · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈63.0; missing-input range 60.3–65.8 points. API cost ≈$0.99 per mix task; range $0.63–$2.05.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
70.24
$0.3733
≈63.04
≈$0.9870
GPT-6.1 Sol · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈61.6; missing-input range 58.8–64.3 points. API cost ≈$0.84 per mix task; range $0.53–$1.79.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
70.18
$0.3105
≈61.58
≈$0.8439
Muse Wonder · highEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈70.0; missing-input range 68.0–71.2 points. API cost ≈$1.79 per mix task; range $1.73–$2.09.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 42 training models, 122 configurations. Held-out mean absolute error 4.08 points; benchmark-mean baseline 4.33.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.25 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 100 configurations. Held-out mean absolute error 0.87 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈57.7; missing-input range 54.3–60.8 points. API cost ≈$4.74 per mix task; range $3.89–$6.34.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 42 training models, 121 configurations. Held-out mean absolute error 3.57 points; benchmark-mean baseline 8.40.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $2.20 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 98 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.26.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈70.01
≈$1.7869
≈57.72
≈$4.7387
Claude Fable 5.1 · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute estimate
60% of score weight measured. Score ≈55.1; missing-input range 52.4–57.9 points. API cost ≈$4.38 per mix task; range $3.68–$5.70.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
69.49
$2.4603
≈55.14
≈$4.3799
GPT-6 Astra · medium
69.28
$1.4723
61.12
$3.1395
Kimi K3 · maxEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈69.2; missing-input range 60.6–78.3 points. API cost ≈$2.12 per mix task; range $2.11–$2.19.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 63.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.21 points; benchmark-mean baseline 10.04.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.017 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.37.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈69.17
≈$2.1199
45.78
$3.9523
Gemini 4 Argon · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈66.2; missing-input range 63.5–69.0 points. API cost ≈$7.38 per mix task; range $5.94–$12.06.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
68.88
$3.6458
≈66.23
≈$7.3776
GPT-6.1 Sol · mediumEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈59.6; missing-input range 56.9–62.4 points. API cost ≈$0.67 per mix task; range $0.40–$1.48.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
68.58
$0.1972
≈59.61
≈$0.6750
Gemini 4 Carbon · lowEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈68.3; missing-input range 66.2–69.1 points. API cost ≈$2.24 per mix task; range $2.18–$2.60.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 42 training models, 122 configurations. Held-out mean absolute error 3.99 points; benchmark-mean baseline 4.35.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.28 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 100 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈65.7; missing-input range 62.0–69.5 points. API cost ≈$5.05 per mix task; range $3.79–$8.96.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 42 training models, 121 configurations. Held-out mean absolute error 3.45 points; benchmark-mean baseline 8.44.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $2.77 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 98 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.26.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈68.31
≈$2.2432
≈65.72
≈$5.0461
Claude Haiku 5.5 · maxEvidence and eligibility
Figure · Briefcase analytical quality: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-LCR 1.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-Omniscience index: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · GDPval-AA v2.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · AutomationBench-AA partial credit: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure estimate
100% of score weight measured. Score 68.1; no score component estimated. API cost ≈$0.63 per mix task; conditional tier envelope $0.21–$1.05.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $2.02 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.058 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.018 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈50.3; missing-input range 46.8–53.7 points. API cost ≈$1.52 per mix task; range unknown, because part of the cost is an accounting scenario.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it. The missing benchmark cost is predicted separately; the total bill has no established bound.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s conditional request-tier midpoint scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $1.96 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.67 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.13 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈$1.5212scenario + prediction · bill range unknown
Claude Opus 5 · xhighEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈68.1; missing-input range 59.9–75.4 points. API cost ≈$4.52 per mix task; range $4.51–$4.65.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 57.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.14 points; benchmark-mean baseline 10.01.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.035 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈68.07
≈$4.5182
59.49
$10.3722
Grok 4.7 · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈45.3; missing-input range 41.9–48.7 points. API cost ≈$3.18 per mix task; range $2.61–$4.25.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
67.79
$0.6963
≈45.33
≈$3.1789
Claude Sonnet 5.5 · highEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Figure · AA-Omniscience index: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 67.7; no score component estimated. API cost ≈$1.07 per mix task; fallback-inclusive bill and range unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $3.27 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.29 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0057 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈58.0; missing-input range 55.3–60.8 points. API cost ≈$1.68 per mix task; range unknown, because part of the cost is an accounting scenario.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s primary-tariff scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $2.19 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.69 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.31 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
67.74
≈$1.0717
≈58.04
≈$1.6774
Qwen3.8 Max 0902 · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈56.1; missing-input range 53.4–58.9 points. API cost ≈$9.52 per mix task; range $7.81–$15.06.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
67.73
$5.9450
≈56.11
≈$9.5186
Muse Spark 1.3 · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈54.1; missing-input range 50.7–57.5 points. API cost ≈$3.48 per mix task; range $2.87–$4.61.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
67.37
$1.0321
≈54.09
≈$3.4771
Grok 4.6 · xhigh
66.57
$1.5307
48.53
$4.8498
DeepSeek V4.1 Flash · maxEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈66.1; missing-input range 56.9–75.2 points. API cost ≈$0.21 per mix task; range $0.21–$0.23.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 60.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0025 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈52.5; missing-input range 49.1–55.9 points. API cost ≈$1.19 per mix task; range $0.77–$2.44.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈66.06
≈$0.2082
≈52.48
≈$1.1861
GPT-6 Sol · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈58.2; missing-input range 55.4–60.9 points. API cost ≈$2.68 per mix task; range $2.17–$3.63.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
66.04
$0.8562
≈58.16
≈$2.6783
Claude Opus 5.5 · lowEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 66.0; no score component estimated. API cost ≈$0.63 per mix task; range $0.52–$0.73.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $1.42 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.66 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈50.3; missing-input range 46.9–53.7 points. API cost ≈$1.81 per mix task; range $1.05–$3.55.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-Omniscience index. 8 training models, 17 configurations. Held-out mean absolute error 0.57 log dollars; benchmark-mean baseline 0.86.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $2.88 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.25 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.59 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
66.01
≈$0.6253
≈50.29
≈$1.8122
Claude Opus 5 · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈65.9; missing-input range 56.1–73.6 points. API cost ≈$3.36 per mix task; range $3.35–$3.49.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 56.9 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.14 points; benchmark-mean baseline 10.01.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.035 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈65.94
≈$3.3602
58.77
$7.4928
Claude Fable 5 · maxEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈65.7; missing-input range 56.5–74.9 points. API cost ≈$7.19 per mix task; range $7.16–$7.42.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 59.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.062 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈56.9; missing-input range 55.6–59.7 points. API cost ≈$13.31 per mix task; range $8.26–$22.49.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 13 training models, 40 configurations. Held-out mean absolute error 0.50 log dollars; benchmark-mean baseline 0.78.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $26.18 per benchmark task, estimated.
Midpoint of per-token-category primary/fallback allocation bounds; not an expected bill or confidence interval.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈65.72
≈$7.1856
≈56.94
≈$13.3102
GPT-6.1 Terra · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈62.3; missing-input range 59.5–65.0 points. API cost ≈$1.25 per mix task; range $0.82–$2.54.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
65.55
$0.2901
≈62.26
≈$1.2475
Muse Spark 1.3 · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈45.8; missing-input range 42.4–49.2 points. API cost ≈$3.16 per mix task; range $2.59–$4.23.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
65.55
$0.8674
≈45.83
≈$3.1631
GPT-6 Astra · low
65.52
$0.7207
54.87
$1.6604
Step 5 Preview · default (assumed)Evidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈52.6; missing-input range 49.2–56.0 points. API cost ≈$3.20 per mix task; range $2.62–$4.28.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
65.39
$0.7479
≈52.61
≈$3.1977
Inkling 2 Small · default (assumed)Evidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈42.0; missing-input range 32.8–51.3 points. API cost ≈$1.76 per mix task; range $1.39–$2.43.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
65.23
$0.3181
≈42.03
≈$1.7632
Claude Opus 5 · mediumEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈65.0; missing-input range 55.6–71.9 points. API cost ≈$1.81 per mix task; range $1.80–$1.94.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.14 points; benchmark-mean baseline 10.01.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.034 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈64.96
≈$1.8098
52.73
$4.8090
GPT-5.6 Sol · maxEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈64.8; missing-input range 55.7–74.0 points. API cost ≈$1.33 per mix task; range $1.33–$1.41.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 60.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.020 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈64.83
≈$1.3346
57.63
$6.6141
Claude Haiku 5.5 · xhighEvidence and eligibility
Figure · Briefcase analytical quality: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-LCR 1.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-Omniscience index: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · GDPval-AA v2.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · AutomationBench-AA partial credit: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure estimate
100% of score weight measured. Score 64.5; no score component estimated. API cost ≈$0.34 per mix task; conditional tier envelope $0.11–$0.57.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $1.10 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.046 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0021 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈48.3; missing-input range 44.9–51.7 points. API cost ≈$1.15 per mix task; range unknown, because part of the cost is an accounting scenario.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it. The missing benchmark cost is predicted separately; the total bill has no established bound.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s conditional request-tier midpoint scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $1.38 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.28 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.083 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈$1.1485scenario + prediction · bill range unknown
Grok 4.6 · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈64.4; missing-input range 56.4–69.1 points. API cost ≈$1.25 per mix task; range $1.25–$1.29.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 57.9 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.37 points; benchmark-mean baseline 10.20.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.36.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈64.44
≈$1.2529
49.13
$4.0936
MiMo-V2.6-Pro · reasoningEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈54.9; missing-input range 52.2–57.6 points. API cost ≈$0.52 per mix task; range $0.29–$1.19.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
64.33
$0.1305
≈54.90
≈$0.5170
DeepSeek V4 Flash Vision · maxEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈64.1; missing-input range 54.6–73.6 points. API cost ≈$0.85 per mix task; range $0.85–$0.88.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0035 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈41.3; missing-input range 32.0–50.5 points. API cost ≈$3.03 per mix task; range $2.47–$4.08.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈64.10
≈$0.8523
≈41.26
≈$3.0312
DeepSeek V4.1 Flash 1017 · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈55.9; missing-input range 52.5–59.4 points. API cost ≈$0.86 per mix task; range $0.53–$1.84.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
63.97
$0.2140
≈55.95
≈$0.8625
GLM-5.3 · max
63.96
$1.7474
57.37
$4.7146
Muse Ember · maxEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈63.9; missing-input range 61.9–66.0 points. API cost ≈$0.050 per mix task; range $0.042–$0.093.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 43 training models, 123 configurations. Held-out mean absolute error 4.00 points; benchmark-mean baseline 4.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.029 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 101 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈58.4; missing-input range 55.0–61.9 points. API cost ≈$0.36 per mix task; range $0.19–$0.89.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 43 training models, 122 configurations. Held-out mean absolute error 3.51 points; benchmark-mean baseline 8.48.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.071 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 99 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈63.95
≈$0.0497
≈58.43
≈$0.3650
GPT-6.1 Sol · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈50.4; missing-input range 47.0–53.8 points. API cost ≈$0.49 per mix task; range $0.27–$1.12.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
63.90
$0.1107
≈50.43
≈$0.4863
Claude Sonnet 5.5 · mediumEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Figure · AA-Omniscience index: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 63.8; no score component estimated. API cost ≈$0.51 per mix task; fallback-inclusive bill and range unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $1.39 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.29 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0031 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈50.4; missing-input range 47.0–53.8 points. API cost ≈$1.29 per mix task; range unknown, because part of the cost is an accounting scenario.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s primary-tariff scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $1.58 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.32 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.26 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
63.76
≈$0.5064
≈50.37
≈$1.2942
Grok 4.6 · mediumEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈63.7; missing-input range 53.9–68.2 points. API cost ≈$0.89 per mix task; range $0.89–$0.93.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.37 points; benchmark-mean baseline 10.20.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.36.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈63.65
≈$0.8907
46.66
$3.3272
Muse Ember · xhighEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈63.5; missing-input range 61.5–65.6 points. API cost ≈$0.038 per mix task; range $0.032–$0.081.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 43 training models, 123 configurations. Held-out mean absolute error 4.00 points; benchmark-mean baseline 4.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.025 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 101 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈55.9; missing-input range 52.4–59.3 points. API cost ≈$0.32 per mix task; range $0.16–$0.80.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 43 training models, 122 configurations. Held-out mean absolute error 3.51 points; benchmark-mean baseline 8.48.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.055 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 99 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈63.55
≈$0.0382
≈55.88
≈$0.3240
GPT-6.1 Terra · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈58.4; missing-input range 55.7–61.1 points. API cost ≈$0.70 per mix task; range $0.42–$1.54.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
63.55
$0.1541
≈58.40
≈$0.6973
Claude Fable 5 · xhighEvidence and eligibility
Figure excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Execute excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Artificial Analysis has not published a model row for this configuration
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Artificial Analysis has not published a model row for this configuration
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Artificial Analysis has not published a model row for this configuration
Figure estimate
0% of score weight measured. Score ≈63.1; missing-input range 47.4–73.7 points. API cost ≈$4.64 per mix task; range $1.52–$14.16.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $14.78 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.63 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 59.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.041 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
40% of score weight measured. Score ≈51.9; missing-input range 42.8–58.2 points. API cost ≈$13.03 per mix task; range $8.20–$26.17.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 10.88 points; benchmark-mean baseline 14.45.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $19.29 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; The fitted curve passes the highest measured cost, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 43 configurations. Held-out mean absolute error 4.48 points; benchmark-mean baseline 7.08.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $6.33 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 5.87 points; benchmark-mean baseline 10.18.
A higher or lower effort of the same model was measured, so the prediction is held at Fable 5 max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (1.9 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $1.85 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; The fitted curve passes the highest measured cost, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈63.11
≈$4.6377
≈51.85
≈$13.0269
GPT-5.6 Sol · xhighEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈62.9; missing-input range 53.7–72.1 points. API cost ≈$1.05 per mix task; range $1.04–$1.12.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 59.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.020 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈62.91
≈$1.0483
50.64
$3.3859
GPT-6 Sol · xhighEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈62.3; missing-input range 52.8–66.0 points. API cost ≈$0.40 per mix task; range $0.40–$0.44.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.6 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.47 points; benchmark-mean baseline 10.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.02 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈52.3; missing-input range 48.9–55.7 points. API cost ≈$1.53 per mix task; range $1.03–$3.00.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈62.33
≈$0.4016
≈52.30
≈$1.5256
GPT-6.1 Terra · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈56.1; missing-input range 53.3–58.8 points. API cost ≈$0.53 per mix task; range $0.30–$1.20.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
62.21
$0.1126
≈56.07
≈$0.5264
Claude Fable 5 · highEvidence and eligibility
Figure excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Execute excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Artificial Analysis has not published a model row for this configuration
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Artificial Analysis has not published a model row for this configuration
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Artificial Analysis has not published a model row for this configuration
Figure estimate
0% of score weight measured. Score ≈62.1; missing-input range 46.4–73.1 points. API cost ≈$3.45 per mix task; range $1.31–$9.12.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $10.95 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.51 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 57.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.033 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
40% of score weight measured. Score ≈50.6; missing-input range 42.5–56.7 points. API cost ≈$10.75 per mix task; range $6.74–$20.08.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 10.88 points; benchmark-mean baseline 14.45.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $18.14 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 43 configurations. Held-out mean absolute error 4.48 points; benchmark-mean baseline 7.08.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $4.70 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 5.87 points; benchmark-mean baseline 10.18.
A higher or lower effort of the same model was measured, so the prediction is held at Fable 5 max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (1.9 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $1.74 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈62.14
≈$3.4525
≈50.64
≈$10.7514
Claude Haiku 5.5 · highEvidence and eligibility
Figure · Briefcase analytical quality: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-LCR 1.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-Omniscience index: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · GDPval-AA v2.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · AutomationBench-AA partial credit: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure estimate
100% of score weight measured. Score 62.1; no score component estimated. API cost ≈$0.21 per mix task; conditional tier envelope $0.072–$0.36.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $0.67 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.045 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0012 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈44.0; missing-input range 40.5–47.4 points. API cost ≈$0.90 per mix task; range unknown, because part of the cost is an accounting scenario.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it. The missing benchmark cost is predicted separately; the total bill has no established bound.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s conditional request-tier midpoint scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $0.99 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.13 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.068 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈$0.8971scenario + prediction · bill range unknown
GPT-6 Sol · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈62.0; missing-input range 52.8–65.1 points. API cost ≈$0.25 per mix task; range $0.25–$0.29.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 60.1 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.47 points; benchmark-mean baseline 10.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.02 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈50.0; missing-input range 46.5–53.4 points. API cost ≈$1.32 per mix task; range $0.87–$2.66.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈61.97
≈$0.2534
≈49.95
≈$1.3190
Qwen3.8-Flash-Next · default (assumed)Evidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈50.0; missing-input range 46.6–53.4 points. API cost ≈$0.83 per mix task; range $0.51–$1.81.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
61.81
$0.4875
≈50.02
≈$0.8334
GPT-5.6 Sol · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈61.1; missing-input range 51.9–70.3 points. API cost ≈$0.74 per mix task; range $0.74–$0.82.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.9 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.020 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈61.12
≈$0.7426
48.36
$2.3884
GLM-5.3-Flash · max
60.88
$0.3009
51.65
$0.5424
Claude Sonnet 5.5 · lowEvidence and eligibility
Figure · Briefcase analytical quality: Fallback token fractions are not published for this benchmark
Figure · AA-LCR 1.1: Fallback token fractions are not published for this benchmark
Figure · AA-Omniscience index: Fallback token fractions are not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: AA fallback-enabled system has no matching direct DeepSWE measurement
Execute · Terminal-Bench 4.0 · AA harness: Fallback token fractions are not published for this benchmark
Execute · GDPval-AA v2.1: Fallback token fractions are not published for this benchmark
Execute · AutomationBench-AA partial credit: Fallback token fractions are not published for this benchmark
Figure estimate
100% of score weight measured. Score 60.9; no score component estimated. API cost ≈$0.33 per mix task; fallback-inclusive bill and range unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $0.80 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.29 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0029 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈44.6; missing-input range 41.2–48.0 points. API cost ≈$1.11 per mix task; range unknown, because part of the cost is an accounting scenario.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s primary-tariff scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Fallback-enabled system absent from measured DeepSWE training overlap. The held-out test does not establish accuracy here.
Missing measurement: AA fallback-enabled system has no matching direct DeepSWE measurement. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $1.28 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.21 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.24 per benchmark task, estimated.
Artificial Analysis benchmark tokens priced at the primary model tariff. AA publishes no fallback route or billed attempts for this system; the actual fallback-inclusive bill and its range are unknown.
Missing measurement: Fallback token fractions are not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
60.87
≈$0.3293
≈44.59
≈$1.1084
Hy4 · default (assumed)Evidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈60.8; missing-input range 58.8–62.8 points. API cost ≈$0.44 per mix task; range $0.41–$0.64.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 43 training models, 123 configurations. Held-out mean absolute error 4.00 points; benchmark-mean baseline 4.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.11 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 101 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈56.2; missing-input range 52.5–59.9 points. API cost ≈$1.35 per mix task; range $0.86–$2.79.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 43 training models, 122 configurations. Held-out mean absolute error 3.51 points; benchmark-mean baseline 8.48.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.58 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 99 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈60.82
≈$0.4418
≈56.17
≈$1.3524
Claude Haiku 5.5 · mediumEvidence and eligibility
Figure · Briefcase analytical quality: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-LCR 1.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-Omniscience index: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · GDPval-AA v2.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · AutomationBench-AA partial credit: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure estimate
100% of score weight measured. Score 60.3; no score component estimated. API cost ≈$0.13 per mix task; conditional tier envelope $0.044–$0.22.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $0.40 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.045 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.00094 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈39.0; missing-input range 35.6–42.4 points. API cost ≈$0.61 per mix task; range unknown, because part of the cost is an accounting scenario.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it. The missing benchmark cost is predicted separately; the total bill has no established bound.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s conditional request-tier midpoint scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $0.58 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.062 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.057 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈$0.6144scenario + prediction · bill range unknown
GPT-6.1 Terra · mediumEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈51.9; missing-input range 48.5–55.3 points. API cost ≈$0.39 per mix task; range $0.21–$0.92.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
60.32
$0.0843
≈51.87
≈$0.3887
Grok 4.6 · lowEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈60.3; missing-input range 52.3–64.9 points. API cost ≈$0.27 per mix task; range $0.27–$0.31.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.1 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.37 points; benchmark-mean baseline 10.20.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.36.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈60.30
≈$0.2692
30.33
$1.1439
Claude Fable 5 · mediumEvidence and eligibility
Figure excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Execute excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Artificial Analysis has not published a model row for this configuration
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Artificial Analysis has not published a model row for this configuration
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Artificial Analysis has not published a model row for this configuration
Figure estimate
0% of score weight measured. Score ≈59.8; missing-input range 44.3–71.7 points. API cost ≈$2.15 per mix task; range $0.93–$5.06.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $6.76 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.36 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 54.1 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
40% of score weight measured. Score ≈47.7; missing-input range 39.3–53.9 points. API cost ≈$6.63 per mix task; range $4.40–$11.39.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 10.88 points; benchmark-mean baseline 14.45.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $10.65 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 43 configurations. Held-out mean absolute error 4.48 points; benchmark-mean baseline 7.08.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $2.90 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 5.87 points; benchmark-mean baseline 10.18.
A higher or lower effort of the same model was measured, so the prediction is held at Fable 5 max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (1.9 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $1.16 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈59.83
≈$2.1468
≈47.72
≈$6.6269
Claude Opus 5 · lowEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈59.8; missing-input range 50.2–66.9 points. API cost ≈$0.77 per mix task; range $0.75–$0.90.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.14 points; benchmark-mean baseline 10.01.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.034 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.04 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈59.78
≈$0.7671
44.26
$2.6600
MiniMax M3.1 Flash · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈51.9; missing-input range 48.5–55.3 points. API cost ≈$0.87 per mix task; range $0.54–$1.88.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
59.76
$0.1644
≈51.89
≈$0.8733
GPT-5.5 · xhighEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈59.7; missing-input range 50.5–68.8 points. API cost ≈$1.37 per mix task; range $1.36–$1.47.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 60.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.025 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈59.66
≈$1.3736
43.33
$6.3331
Muse Spark 1.2 · xhighEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈59.3; missing-input range 51.4–67.3 points. API cost ≈$0.51 per mix task; range $0.51–$0.54.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 57.2 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0081 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈59.33
≈$0.5149
35.41
$3.0244
Mistral Large 4 · highEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈59.2; missing-input range 57.4–61.0 points. API cost ≈$0.37 per mix task; range $0.34–$0.55.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 43 training models, 123 configurations. Held-out mean absolute error 4.00 points; benchmark-mean baseline 4.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.098 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 101 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈49.3; missing-input range 44.7–53.9 points. API cost ≈$2.86 per mix task; range $2.29–$3.95.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 43 training models, 122 configurations. Held-out mean absolute error 3.51 points; benchmark-mean baseline 8.48.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.50 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 99 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈59.23
≈$0.3694
≈49.31
≈$2.8627
Qwen3.8 27B · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈34.9; missing-input range 24.7–45.0 points. API cost ≈$2.94 per mix task; range $2.39–$3.95.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
59.17
$0.9963
≈34.87
≈$2.9373
Gemini 3.8 Flash · medium
59.02
$1.6324
49.09
$6.1673
GPT-5.5 · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈58.8; missing-input range 49.7–68.0 points. API cost ≈$1.05 per mix task; range $1.04–$1.14.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 60.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.024 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈58.83
≈$1.0497
39.77
$4.1247
GPT-6 Sol · mediumEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈58.6; missing-input range 49.4–62.1 points. API cost ≈$0.16 per mix task; range $0.16–$0.20.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 59.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.47 points; benchmark-mean baseline 10.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.02 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈45.7; missing-input range 42.3–49.1 points. API cost ≈$1.01 per mix task; range $0.64–$2.12.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈58.60
≈$0.1605
≈45.69
≈$1.0077
Gemini 3.8 Flash · high
58.59
$2.2918
50.14
$7.8667
DeepSeek V4.1 Flash 1017 · highEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈58.5; missing-input range 56.4–60.2 points. API cost ≈$0.13 per mix task; range $0.11–$0.21.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 42 training models, 121 configurations. Held-out mean absolute error 3.72 points; benchmark-mean baseline 3.95.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.054 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 99 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈52.7; missing-input range 48.1–57.3 points. API cost ≈$0.78 per mix task; range $0.46–$1.71.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 42 training models, 121 configurations. Held-out mean absolute error 3.49 points; benchmark-mean baseline 8.43.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.18 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 98 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.28.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈58.51
≈$0.1283
≈52.68
≈$0.7759
GPT-5.6 Sol · mediumEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈58.2; missing-input range 50.2–66.2 points. API cost ≈$0.41 per mix task; range $0.41–$0.49.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 58.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.019 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈58.19
≈$0.4131
41.91
$1.4701
MiMo-V2.6-Flash · reasoningEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈49.9; missing-input range 46.5–53.3 points. API cost ≈$0.33 per mix task; range $0.17–$0.82.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
57.42
$0.0514
≈49.94
≈$0.3334
GPT-6.1 Terra · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈42.7; missing-input range 39.3–46.1 points. API cost ≈$0.22 per mix task; range $0.076–$1.21.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
56.49
$0.0542
≈42.73
≈$0.2154
GPT-6 Luna · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈41.9; missing-input range 32.7–51.1 points. API cost ≈$0.34 per mix task; range $0.18–$0.83.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
56.49
$0.0550
≈41.90
≈$0.3434
Gemini 3.7 Flash · mediumEvidence and eligibility
Figure excluded from recommendations: Only 30% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Figure · AA-LCR 1.1: Comparable API pricing is not published
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · GDPval-AA v2.1: Comparable API pricing is not published
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Figure estimate
30% of score weight measured. Score ≈56.5; missing-input range 39.3–65.7 points. API cost ≈$0.61 per mix task; range $0.073–$5.08.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 123 configurations. Held-out mean absolute error 12.64 points; benchmark-mean baseline 14.62.
A higher or lower effort of the same model was measured, so the prediction is held at Gemini 3.7 Flash high’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $1.85 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.16 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.82 log dollars; benchmark-mean baseline 0.95.
Missing measurement: Comparable API pricing is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 59.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0084 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈44.4; missing-input range 37.2–46.3 points. API cost ≈$1.84 per mix task; range $0.96–$7.93.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 42 training models, 120 configurations. Held-out mean absolute error 11.02 points; benchmark-mean baseline 16.94.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $2.55 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.80 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Comparable API pricing is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 5.92 points; benchmark-mean baseline 10.39.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.38 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈56.48
≈$0.6059
≈44.42
≈$1.8415
GPT-5.5 · mediumEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈56.3; missing-input range 47.1–65.5 points. API cost ≈$0.63 per mix task; range $0.63–$0.72.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 59.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.024 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈56.28
≈$0.6342
33.09
$2.1564
Claude Fable 5 · lowEvidence and eligibility
Figure excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Execute excluded from recommendations: Fallback behavior is not disclosed; exact system cannot be established.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Artificial Analysis has not published a model row for this configuration
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Artificial Analysis has not published a model row for this configuration
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Artificial Analysis has not published a model row for this configuration
Figure estimate
0% of score weight measured. Score ≈56.0; missing-input range 41.7–68.7 points. API cost ≈$1.23 per mix task; range $0.46–$3.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $3.83 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.25 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 48.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.015 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
40% of score weight measured. Score ≈42.3; missing-input range 34.8–48.3 points. API cost ≈$3.77 per mix task; range $2.29–$8.05.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 10.88 points; benchmark-mean baseline 14.45.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $5.70 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 43 configurations. Held-out mean absolute error 4.48 points; benchmark-mean baseline 7.08.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.65 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 5.87 points; benchmark-mean baseline 10.18.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.71 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈56.02
≈$1.2298
≈42.29
≈$3.7685
DeepSeek V4 Pro 0813 · maxEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈43.5; missing-input range 40.1–46.9 points. API cost ≈$2.48 per mix task; range $1.99–$3.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
55.90
$0.3450
≈43.46
≈$2.4829
GPT-5.6 Terra · max
55.65
$0.9433
53.98
$4.4476
Gemini 3.7 Flash · highEvidence and eligibility
Figure · Briefcase analytical quality: Cache-hit accounting is not published
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · Terminal-Bench 4.0 · AA harness: Cache-hit accounting is not published
Execute · GDPval-AA v2.1: Cache-hit accounting is not published
Execute · AutomationBench-AA partial credit: Cache-hit accounting is not published
Figure estimate
60% of score weight measured. Score ≈55.6; missing-input range 46.5–64.8 points. API cost ≈$0.42 per mix task; range $0.064–$4.34.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $1.32 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 37 training models, 101 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Cache-hit accounting is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 58.9 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0058 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
100% of score weight measured. Score 44.7; no score component estimated. API cost ≈$2.00 per mix task; range $1.04–$8.68.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Terminal-Bench 4.0 · AA harness · cost: $2.80 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Cache-hit accounting is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.87 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Cache-hit accounting is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.41 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Cache-hit accounting is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈55.63
≈$0.4248
44.70
≈$2.0008
GPT-5.5 · lowEvidence and eligibility
Figure excluded from recommendations: Only 30% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Figure estimate
30% of score weight measured. Score ≈55.3; missing-input range 36.9–65.0 points. API cost ≈$1.82 per mix task; range $0.38–$12.02.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 121 configurations. Held-out mean absolute error 12.68 points; benchmark-mean baseline 14.76.
A higher or lower effort of the same model was measured, so the prediction is held at GPT-5.5 medium’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $5.56 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 98 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.37.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 58.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈19.6; missing-input range 16.4–23.0 points. API cost ≈$0.80 per mix task; range $0.57–$1.85.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 42 training models, 118 configurations. Held-out mean absolute error 11.04 points; benchmark-mean baseline 16.85.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $0.77 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 35 training models, 95 configurations. Held-out mean absolute error 0.44 log dollars; benchmark-mean baseline 1.29.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 37 configurations. Held-out mean absolute error 5.50 points; benchmark-mean baseline 10.05.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.22 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 12 training models, 36 configurations. Held-out mean absolute error 0.71 log dollars; benchmark-mean baseline 0.91.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈55.30
≈$1.8214
≈19.64
≈$0.8040
GPT-5.6 Luna · xhighEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈55.1; missing-input range 46.2–55.1 points. API cost ≈$0.071 per mix task; range $0.070–$0.085.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 48.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 8.87 points; benchmark-mean baseline 9.78.
A higher or lower effort of the same model was measured, so the prediction is held at Luna max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (7.2 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0014 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.38.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈55.11
≈$0.0706
34.91
$0.7374
Mistral Large 4 · mediumEvidence and eligibility
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Figure estimate
70% of score weight measured. Score ≈55.1; missing-input range 48.4–61.1 points. API cost ≈$0.15 per mix task; range $0.14–$0.23.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 43 training models, 123 configurations. Held-out mean absolute error 4.00 points; benchmark-mean baseline 4.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.057 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 101 configurations. Held-out mean absolute error 0.86 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈47.2; missing-input range 42.6–51.8 points. API cost ≈$1.97 per mix task; range $1.54–$2.78.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 43 training models, 122 configurations. Held-out mean absolute error 3.51 points; benchmark-mean baseline 8.48.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.21 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 37 training models, 99 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈55.09
≈$0.1508
≈47.23
≈$1.9718
Qwen3.8 Max · xhighEvidence and eligibility
Figure excluded from recommendations: Only 0% of score weight measured; at least 50% required for a recommendation.
Execute excluded from recommendations: Only 40% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Artificial Analysis has not published a model row for this configuration
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Artificial Analysis has not published a model row for this configuration
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Artificial Analysis has not published a model row for this configuration
Figure estimate
0% of score weight measured. Score ≈54.7; missing-input range 40.4–69.1 points. API cost ≈$1.22 per mix task; range $0.45–$3.37.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $3.80 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.24 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 47.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.015 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
40% of score weight measured. Score ≈40.4; missing-input range 32.9–47.8 points. API cost ≈$3.73 per mix task; range $2.27–$7.97.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 10.88 points; benchmark-mean baseline 14.45.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $5.64 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 43 configurations. Held-out mean absolute error 4.48 points; benchmark-mean baseline 7.08.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $1.63 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 5.87 points; benchmark-mean baseline 10.18.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.71 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈54.72
≈$1.2189
≈40.37
≈$3.7349
GPT-5.6 Luna · max
54.35
$0.1289
43.31
$1.5332
Gemini 3.7 Flash · lowEvidence and eligibility
Figure excluded from recommendations: Only 30% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Figure · AA-LCR 1.1: Comparable API pricing is not published
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · GDPval-AA v2.1: Comparable API pricing is not published
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Figure estimate
30% of score weight measured. Score ≈54.1; missing-input range 37.2–62.1 points. API cost ≈$0.54 per mix task; range $0.063–$4.70.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 123 configurations. Held-out mean absolute error 12.64 points; benchmark-mean baseline 14.62.
A higher or lower effort of the same model was measured, so the prediction is held at Gemini 3.7 Flash high’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $1.65 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.14 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.82 log dollars; benchmark-mean baseline 0.95.
Missing measurement: Comparable API pricing is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 57.0 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0077 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
50% of score weight measured. Score ≈37.7; missing-input range 30.8–41.1 points. API cost ≈$1.64 per mix task; range $0.84–$8.44.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 42 training models, 120 configurations. Held-out mean absolute error 11.02 points; benchmark-mean baseline 16.94.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $2.24 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.71 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Comparable API pricing is not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 5.92 points; benchmark-mean baseline 10.39.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.35 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈54.10
≈$0.5403
≈37.66
≈$1.6411
GPT-5.6 Terra · xhigh
53.88
$0.6082
39.45
$1.5608
Gemini 3.8 Flash · lowEvidence and eligibility
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
100% of score weight measured. Score 53.4; no score component estimated. API cost ≈$0.38 per mix task; range $0.057–$4.07.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $1.19 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 99 configurations. Held-out mean absolute error 1.00 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈36.8; missing-input range 28.0–42.4 points. API cost ≈$9.24 per mix task; range $7.75–$12.34.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 13 training models, 38 configurations. Held-out mean absolute error 6.97 points; benchmark-mean baseline 11.99.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 12 training models, 37 configurations. Held-out mean absolute error 0.46 log dollars; benchmark-mean baseline 0.82.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
53.41
≈$0.3835
≈36.76
≈$9.2432
GPT-5.6 Sol · lowEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈53.1; missing-input range 43.4–62.8 points. API cost ≈$0.25 per mix task; range $0.24–$0.32.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 56.6 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.019 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈53.10
≈$0.2478
28.71
$0.7105
GPT-5.6 Luna · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈53.0; missing-input range 45.2–53.0 points. API cost ≈$0.041 per mix task; range $0.041–$0.055.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 48.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 8.87 points; benchmark-mean baseline 9.78.
A higher or lower effort of the same model was measured, so the prediction is held at Luna max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (7.2 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0014 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.38.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈52.98
≈$0.0414
28.11
$0.3613
Claude Haiku 5.5 · lowEvidence and eligibility
Figure · Briefcase analytical quality: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-LCR 1.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure · AA-Omniscience index: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · GDPval-AA v2.1: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Execute · AutomationBench-AA partial credit: AA cost accounting omits higher long-context pricing; request-tier allocations are not published
Figure estimate
100% of score weight measured. Score 52.9; no score component estimated. API cost ≈$0.056 per mix task; conditional tier envelope $0.019–$0.093.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Briefcase analytical quality · cost: $0.14 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.044 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.00076 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈35.8; missing-input range 26.5–45.0 points. API cost ≈$0.44 per mix task; range unknown, because part of the cost is an accounting scenario.
Request-tier scenario: the midpoint sets the displayed position; it is not an expected price. The tier envelope assumes the captured tokens, cache-hit rate, five-minute cache writes and task counts. Other billing modifiers are outside it. The missing benchmark cost is predicted separately; the total bill has no established bound.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Predicted from the named predictor benchmark’s conditional request-tier midpoint scenario, not a measured bill. Regression trains on measured component pairs with whole models held out. The source accounting and transfer errors are not jointly bounded; the actual bill and range are unknown.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $0.36 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.031 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.046 per benchmark task, estimated.
Midpoint of all-base and all-premium request-tier scenarios above 100,000 prompt tokens, including cached tokens. Conditional on captured AA tokens, cache-hit rate, five-minute cache writes and scored-task denominators; not an expected bill, guaranteed actual-bill bound or confidence interval. Request-tier allocations and other billing modifiers are unknown. Tariffs: https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-10-08).
Missing measurement: AA cost accounting omits higher long-context pricing; request-tier allocations are not published. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈$0.4402scenario + prediction · bill range unknown
DeepSeek V4.1 Flash 1017 · noneEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Execute estimate
50% of score weight measured. Score ≈39.0; missing-input range 29.1–48.9 points. API cost ≈$0.57 per mix task; range $0.31–$1.33.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 42 training models, 121 configurations. Held-out mean absolute error 3.49 points; benchmark-mean baseline 8.43.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.37 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 98 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.28.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
52.80
$0.2535
≈39.00
≈$0.5709
GPT-6 Luna · xhighEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈37.2; missing-input range 28.0–46.5 points. API cost ≈$0.30 per mix task; range $0.15–$0.75.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
52.72
$0.0280
≈37.24
≈$0.3020
GPT-6 Sol · lowEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈52.6; missing-input range 44.6–56.8 points. API cost ≈$0.099 per mix task; range $0.096–$0.14.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 57.4 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.47 points; benchmark-mean baseline 10.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.02 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈38.2; missing-input range 28.9–47.4 points. API cost ≈$0.57 per mix task; range $0.33–$1.29.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈52.60
≈$0.0993
≈38.18
≈$0.5701
Ling 3.0 Flash VL · default (assumed)Evidence and eligibility
Figure · Briefcase analytical quality: Comparable API pricing is not published
Figure · AA-LCR 1.1: Comparable API pricing is not published
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: Comparable API pricing is not published
Execute · GDPval-AA v2.1: Comparable API pricing is not published
Execute · AutomationBench-AA partial credit: Comparable API pricing is not published
Figure estimate
60% of score weight measured. Score ≈52.3; missing-input range 42.5–62.0 points. API cost unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 56.8 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈7.4; missing-input range 5.7–17.5 points. API cost unknown.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈52.26
Unpriced
≈7.36
Unpriced
Qwen3.8 27B · mediumEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈32.8; missing-input range 22.7–43.0 points. API cost ≈$3.85 per mix task; range $3.18–$5.11.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
52.19
$0.9658
≈32.82
≈$3.8500
Qwen3.8 27B · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈26.5; missing-input range 16.4–36.7 points. API cost ≈$4.03 per mix task; range $3.34–$5.32.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
51.81
$0.6995
≈26.54
≈$4.0274
GLM-5.3 · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈52.2; missing-input range 48.1–52.7 points. API cost ≈$3.71 per mix task; range $2.95–$5.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 13 training models, 39 configurations. Held-out mean absolute error 7.11 points; benchmark-mean baseline 12.19.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 12 training models, 38 configurations. Held-out mean absolute error 0.58 log dollars; benchmark-mean baseline 0.81.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
51.73
$0.2627
≈52.21
≈$3.7137
GPT-6 Luna · highEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈33.0; missing-input range 22.9–43.2 points. API cost ≈$0.22 per mix task; range $0.073–$1.29.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
51.38
$0.0218
≈33.04
≈$0.2211
GPT-5.6 Terra · high
51.00
$0.3191
32.77
$0.7875
Kimi K3 · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈39.1; missing-input range 30.6–43.3 points. API cost ≈$5.06 per mix task; range $3.93–$8.58.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 13 training models, 39 configurations. Held-out mean absolute error 6.98 points; benchmark-mean baseline 12.22.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 12 training models, 38 configurations. Held-out mean absolute error 0.55 log dollars; benchmark-mean baseline 0.80.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
50.88
$0.2521
≈39.14
≈$5.0615
GPT-6.1 Terra · noneEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈48.2; missing-input range 44.8–51.6 points. API cost ≈$0.47 per mix task; range $0.26–$1.10.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
50.86
$0.0945
≈48.23
≈$0.4709
GPT-6 Luna · mediumEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈27.6; missing-input range 17.4–37.8 points. API cost ≈$0.16 per mix task; range $0.048–$0.97.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
49.18
$0.0130
≈27.59
≈$0.1605
GPT-5.5 · noneEvidence and eligibility
Figure excluded from recommendations: Only 30% of score weight measured; at least 50% required for a recommendation.
Execute excluded from recommendations: Only 10% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Figure estimate
30% of score weight measured. Score ≈47.9; missing-input range 29.5–57.8 points. API cost ≈$1.80 per mix task; range $0.38–$11.87.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 121 configurations. Held-out mean absolute error 12.68 points; benchmark-mean baseline 14.76.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $5.49 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 98 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.37.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 47.8 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
10% of score weight measured. Score ≈5.3; missing-input range 2.4–24.5 points. API cost ≈$0.54 per mix task; range $0.15–$2.07.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 13 training models, 39 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 12.40.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 12 training models, 36 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.80.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 42 training models, 118 configurations. Held-out mean absolute error 11.04 points; benchmark-mean baseline 16.85.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $0.58 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 35 training models, 95 configurations. Held-out mean absolute error 0.44 log dollars; benchmark-mean baseline 1.29.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 42 training models, 119 configurations. Held-out mean absolute error 8.23 points; benchmark-mean baseline 11.40.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.077 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 36 training models, 97 configurations. Held-out mean absolute error 0.61 log dollars; benchmark-mean baseline 1.15.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈47.88
≈$1.7991
≈5.34
≈$0.5406
GPT-6 Sol · noneEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈47.8; missing-input range 39.8–55.8 points. API cost ≈$0.17 per mix task; range $0.17–$0.21.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 47.9 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 9.47 points; benchmark-mean baseline 10.25.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.02 log dollars; benchmark-mean baseline 1.35.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈38.4; missing-input range 29.2–47.7 points. API cost ≈$1.49 per mix task; range $0.99–$2.98.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈47.81
≈$0.1731
≈38.43
≈$1.4914
Gemini 3.5 Flash · mediumEvidence and eligibility
Execute excluded from recommendations: Only 0% of score weight measured; at least 50% required for a recommendation.
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈47.3; missing-input range 39.4–55.3 points. API cost ≈$2.59 per mix task; range $2.59–$2.63.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 54.3 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
0% of score weight measured. Score ≈10.8; missing-input range 2.0–28.7 points. API cost ≈$6.46 per mix task; range $3.10–$13.80.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality, on a curve that levels off near the top. 13 training models, 39 configurations. Held-out mean absolute error 8.19 points; benchmark-mean baseline 9.83.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 12 training models, 38 configurations. Held-out mean absolute error 0.52 log dollars; benchmark-mean baseline 0.82.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 45 training models, 130 configurations. Held-out mean absolute error 10.74 points; benchmark-mean baseline 16.01.
The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $10.91 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 39 training models, 111 configurations. Held-out mean absolute error 0.46 log dollars; benchmark-mean baseline 1.26.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality, on a curve that levels off near the top. 42 training models, 121 configurations. Held-out mean absolute error 3.39 points; benchmark-mean baseline 8.26.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $3.20 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 36 training models, 98 configurations. Held-out mean absolute error 0.22 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality, on a curve that levels off near the top. 45 training models, 131 configurations. Held-out mean absolute error 8.41 points; benchmark-mean baseline 10.95.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.86 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Briefcase analytical quality. 40 training models, 113 configurations. Held-out mean absolute error 0.66 log dollars; benchmark-mean baseline 1.19.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈47.32
≈$2.5890
≈10.84
≈$6.4555
GPT-5.6 Luna · mediumEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈46.7; missing-input range 38.9–46.7 points. API cost ≈$0.015 per mix task; range $0.015–$0.029.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 48.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 8.87 points; benchmark-mean baseline 9.78.
A higher or lower effort of the same model was measured, so the prediction is held at Luna max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (7.2 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0014 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.38.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈46.66
≈$0.0151
11.34
$0.1012
Gemini 3.5 Flash · highEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈46.7; missing-input range 38.7–54.6 points. API cost ≈$2.36 per mix task; range $2.36–$2.40.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 53.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.011 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈46.65
≈$2.3607
26.59
$6.6802
GPT-5.6 Terra · medium
46.52
$0.1744
23.30
$0.4181
Qwen3.8 27B · noneEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈6.4; missing-input range 4.8–16.6 points. API cost ≈$5.21 per mix task; range $4.03–$9.07.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
44.74
$2.7150
≈6.44
≈$5.2140
GPT-5.6 Terra · low
44.68
$0.1487
17.60
$0.3068
GPT-5.6 Sol · noneEvidence and eligibility
Execute excluded from recommendations: Only 25% of score weight measured; at least 50% required for a recommendation.
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈44.2; missing-input range 36.3–52.2 points. API cost ≈$0.26 per mix task; range $0.25–$0.33.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 46.8 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.019 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
25% of score weight measured. Score ≈16.8; missing-input range 7.1–25.4 points. API cost ≈$0.84 per mix task; range $0.30–$2.85.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AutomationBench-AA partial credit, on a curve that levels off near the top. 13 training models, 35 configurations. Held-out mean absolute error 6.56 points; benchmark-mean baseline 12.06.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 12 training models, 35 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.80.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1, on a curve that levels off near the top. 42 training models, 116 configurations. Held-out mean absolute error 10.97 points; benchmark-mean baseline 16.98.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $0.93 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: GDPval-AA v2.1. 35 training models, 93 configurations. Held-out mean absolute error 0.43 log dollars; benchmark-mean baseline 1.29.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈44.22
≈$0.2594
≈16.78
≈$0.8351
GPT-5.6 Luna · lowEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Figure estimate
60% of score weight measured. Score ≈41.9; missing-input range 34.1–41.9 points. API cost ≈$0.010 per mix task; range $0.0095–$0.024.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 48.7 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 8.87 points; benchmark-mean baseline 9.78.
A higher or lower effort of the same model was measured, so the prediction is held at Luna max’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (7.2 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0014 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.38.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈41.87
≈$0.0101
4.88
$0.0365
DeepSeek V4.1 Flash · noneEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈41.3; missing-input range 33.3–49.2 points. API cost ≈$0.065 per mix task; range $0.064–$0.091.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 42.4 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0023 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈34.4; missing-input range 24.3–44.6 points. API cost ≈$1.24 per mix task; range $0.79–$2.56.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈41.27
≈$0.0646
≈34.42
≈$1.2379
GPT-6 Luna · lowEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈6.2; missing-input range 4.5–16.3 points. API cost ≈$0.054 per mix task; range $0.0079–$0.55.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
40.87
$0.0048
≈6.17
≈$0.0539
Gemini 3.5 Flash · minimalEvidence and eligibility
Figure excluded from recommendations: Only 30% of score weight measured; at least 50% required for a recommendation.
Execute excluded from recommendations: Only 0% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Figure estimate
30% of score weight measured. Score ≈40.2; missing-input range 23.2–48.2 points. API cost ≈$0.68 per mix task; range $0.13–$4.98.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 122 configurations. Held-out mean absolute error 12.52 points; benchmark-mean baseline 14.16.
A higher or lower effort of the same model was measured, so the prediction is held at Gemini 3.5 Flash high’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (0.4 points). The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $2.09 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 99 configurations. Held-out mean absolute error 1.00 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 46.2 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 34 training models, 88 configurations. Held-out mean absolute error 9.20 points; benchmark-mean baseline 10.03.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0096 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 30 training models, 68 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.34.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
0% of score weight measured. Score ≈21.0; missing-input range 1.8–29.2 points. API cost ≈$2.20 per mix task; range $0.35–$16.51.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1, on a curve that levels off near the top. 13 training models, 42 configurations. Held-out mean absolute error 8.45 points; benchmark-mean baseline 10.88.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 13 training models, 40 configurations. Held-out mean absolute error 0.62 log dollars; benchmark-mean baseline 0.76.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 121 configurations. Held-out mean absolute error 15.24 points; benchmark-mean baseline 16.92.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $3.26 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 35 training models, 98 configurations. Held-out mean absolute error 0.93 log dollars; benchmark-mean baseline 1.28.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1, on a curve that levels off near the top. 42 training models, 125 configurations. Held-out mean absolute error 7.12 points; benchmark-mean baseline 8.48.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.87 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 101 configurations. Held-out mean absolute error 0.92 log dollars; benchmark-mean baseline 1.27.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 42 training models, 122 configurations. Held-out mean absolute error 10.71 points; benchmark-mean baseline 11.31.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.34 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 36 training models, 100 configurations. Held-out mean absolute error 0.46 log dollars; benchmark-mean baseline 1.18.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈40.18
≈$0.6811
≈21.04
≈$2.1977
Gemini 3.1 Pro Preview · highEvidence and eligibility
Figure excluded from recommendations: Only 0% of score weight measured; at least 50% required for a recommendation.
Execute excluded from recommendations: Only 40% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Artificial Analysis has not published a model row for this configuration
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Artificial Analysis has not published a model row for this configuration
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Artificial Analysis has not published a model row for this configuration
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Artificial Analysis has not published a model row for this configuration
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Artificial Analysis has not published a model row for this configuration
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Artificial Analysis has not published a model row for this configuration
Figure estimate
0% of score weight measured. Score ≈39.7; missing-input range 28.6–50.8 points. API cost ≈$0.64 per mix task; range $0.077–$5.42.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 11.70 points; benchmark-mean baseline 15.30.
The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Briefcase analytical quality · cost: $1.98 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.80 log dollars; benchmark-mean baseline 1.12.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 43 configurations. Held-out mean absolute error 2.44 points; benchmark-mean baseline 2.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-LCR 1.1 · cost: $0.16 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 41 configurations. Held-out mean absolute error 0.79 log dollars; benchmark-mean baseline 0.94.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · score: 41.5 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 8 training models, 17 configurations. Held-out mean absolute error 7.44 points; benchmark-mean baseline 10.80.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened; The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0089 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 8 training models, 17 configurations. Held-out mean absolute error 0.51 log dollars; benchmark-mean baseline 0.86.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
40% of score weight measured. Score ≈10.7; missing-input range 7.8–17.2 points. API cost ≈$1.97 per mix task; range $1.02–$8.52.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 10.88 points; benchmark-mean baseline 14.45.
The fitted curve passes the lowest measured score, so the prediction is held there. The held-out test does not establish accuracy here.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Terminal-Bench 4.0 · AA harness · cost: $2.75 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.83 log dollars; benchmark-mean baseline 1.20.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent, on a curve that levels off near the top. 14 training models, 43 configurations. Held-out mean absolute error 4.48 points; benchmark-mean baseline 7.08.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
GDPval-AA v2.1 · cost: $0.85 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 40 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 1.13.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 14 training models, 40 configurations. Held-out mean absolute error 5.87 points; benchmark-mean baseline 10.18.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AutomationBench-AA partial credit · cost: $0.41 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: DeepSWE v1.1 · mini-swe-agent. 13 training models, 39 configurations. Held-out mean absolute error 0.70 log dollars; benchmark-mean baseline 0.90.
Missing measurement: Artificial Analysis has not published a model row for this configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈39.67
≈$0.6445
≈10.73
≈$1.9659
GPT-5.6 Terra · noneEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈13.0; missing-input range 5.1–15.6 points. API cost ≈$0.37 per mix task; range $0.19–$1.06.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 13 training models, 35 configurations. Held-out mean absolute error 6.79 points; benchmark-mean baseline 11.08.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 12 training models, 34 configurations. Held-out mean absolute error 0.56 log dollars; benchmark-mean baseline 0.73.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
36.35
$0.1518
≈13.02
≈$0.3748
DeepSeek V4 Pro 0813 · noneEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈22.5; missing-input range 12.4–32.7 points. API cost ≈$1.83 per mix task; range $1.44–$2.51.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
36.00
$0.1468
≈22.55
≈$1.8277
GPT-6 Luna · noneEvidence and eligibility
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute estimate
60% of score weight measured. Score ≈18.6; missing-input range 8.4–28.8 points. API cost ≈$0.12 per mix task; range $0.033–$0.75.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 14 training models, 40 configurations. Held-out mean absolute error 6.73 points; benchmark-mean baseline 11.75.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 13 training models, 39 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.77.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
32.52
$0.0091
≈18.61
≈$0.1209
GPT-5.6 Luna · noneEvidence and eligibility
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Figure estimate
60% of score weight measured. Score ≈27.1; missing-input range 19.3–34.8 points. API cost ≈$0.010 per mix task; range $0.0098–$0.024.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
AA-Omniscience index · score: 33.2 converted points, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 33 training models, 87 configurations. Held-out mean absolute error 8.87 points; benchmark-mean baseline 9.78.
Missing measurement: No published measurement at this exact configuration. The weekly refresh uses a compatible measured value as soon as the source supplies one.
AA-Omniscience index · cost: $0.0014 per benchmark task, estimated.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: AA-LCR 1.1. 29 training models, 67 configurations. Held-out mean absolute error 1.01 log dollars; benchmark-mean baseline 1.38.
Missing measurement: Token accounting is not published for this benchmark. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Execute estimate
60% of score weight measured. Score ≈4.8; missing-input range 3.2–4.8 points. API cost ≈$0.22 per mix task; range $0.023–$3.15.
Available ranges describe missing-input uncertainty, not all benchmark uncertainty or a confidence interval. Estimates set the displayed position; overlapping score ranges qualify the recommendation. Estimated costs may exceed a spending ceiling.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness, on a curve that levels off near the top. 13 training models, 35 configurations. Held-out mean absolute error 5.77 points; benchmark-mean baseline 9.75.
A higher or lower effort of the same model was measured, so the prediction is held at Luna low’s measured score, loosened by the largest inversion measured on this benchmark between reasoning efforts (2.3 points). The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
Equal model weights; all efforts of each test model held out. The curve (straight or saturating) is whichever predicts held-out models better, held inside the measured ranges. Range uses the model-balanced 95th-percentile held-out error among the nearest third of measured configurations, which covered about 90% of held-out configurations, widened for extrapolation; not a confidence interval.
Predictor: Terminal-Bench 4.0 · AA harness. 12 training models, 34 configurations. Held-out mean absolute error 0.53 log dollars; benchmark-mean baseline 0.66.
The predictor value is outside every measured configuration, so the prediction is held at the edge of the measured range and its error allowance is widened. The held-out test does not establish accuracy here.
Missing measurement: No exact model and effort result in direct DeepSWE v1.1. The weekly refresh uses a compatible measured value as soon as the source supplies one.
≈27.08
≈$0.0103
≈4.77
≈$0.2169
GPT-5.5 Pro · xhighEvidence and eligibility
Figure excluded from recommendations: Only 0% of score weight measured; at least 50% required for a recommendation.
Execute excluded from recommendations: Only 0% of score weight measured; at least 50% required for a recommendation.
Figure · Briefcase analytical quality: No published measurement at this exact configuration
Figure · Briefcase analytical quality: Token accounting is not published for this benchmark
Figure · AA-LCR 1.1: No published measurement at this exact configuration
Figure · AA-LCR 1.1: Token accounting is not published for this benchmark
Figure · AA-Omniscience index: No published measurement at this exact configuration
Figure · AA-Omniscience index: Token accounting is not published for this benchmark
Execute · DeepSWE v1.1 · mini-swe-agent: No exact model and effort result in direct DeepSWE v1.1
Execute · Terminal-Bench 4.0 · AA harness: No published measurement at this exact configuration
Execute · Terminal-Bench 4.0 · AA harness: Token accounting is not published for this benchmark
Execute · GDPval-AA v2.1: No published measurement at this exact configuration
Execute · GDPval-AA v2.1: Token accounting is not published for this benchmark
Execute · AutomationBench-AA partial credit: No published measurement at this exact configuration
Execute · AutomationBench-AA partial credit: Token accounting is not published for this benchmark
Unscored
Unpriced
Unscored
Unpriced
Data: November 11, 2026 · Artificial Analysis, Datacurve · supporting: Cognition, ZapierPrices here compare setups against each other. Your actual bill will differ.