PokéBench leaderboard

Results

scenario	model	success	stop reasons	turns	cost ($)	tiles explored	idle rate	exclusions (this row)	replays
s1_exit_pallet	gemini	100% (3 seed(s))	success:3	28	0.167115	18	11%	-	0 · 1 · 2
s1_exit_pallet	gemini-lite	0% (3 seed(s))	max_turns:3	100	0.06173	6	72%	-	0 · 1 · 2
s1_exit_pallet	gpt	100% (3 seed(s))	success:3	42	0.2594	22	9%	-	0 · 1 · 2
s1_exit_pallet	haiku reasoning off	0% (3 seed(s))	max_tokens:3	116	0.486947	5	95%	-	0 · 1 · 2
s1_exit_pallet	qwen-local	0% (3 seed(s))	max_turns:3	100	0	21	64%	-	0 · 1 · 2
s1_exit_pallet	sonnet reasoning off	0% (3 seed(s))	max_turns:3	100	1.25145	4	96%	-	0 · 1 · 2
s2_viridian_pokecenter	gemini	67% (3 seed(s))	max_turns:1, success:2	91	0.549521	36	55%	-	0 · 1 · 2
s2_viridian_pokecenter	gemini-lite	0% (3 seed(s))	max_turns:3	100	0.069226	10	84%	-	0 · 1 · 2
s2_viridian_pokecenter	gpt	0% (3 seed(s))	max_turns:3	100	0.590656	27	57%	-	0 · 1 · 2
s2_viridian_pokecenter	haiku reasoning off	0% (3 seed(s))	max_usd:3	128	0.501602	5	96%	-	0 · 1 · 2
s2_viridian_pokecenter	qwen-local	0% (3 seed(s))	max_turns:3	100	0	9	84%	-	0 · 1 · 2
s2_viridian_pokecenter	sonnet reasoning off	0% (3 seed(s))	max_turns:3	100	1.16717	9	89%	-	0 · 1 · 2
s3_viridian_forest	gemini	ceiling probe 3 seed(s) probed.	max_turns:3	100	0.556252	23	77%	-	0 · 1 · 2
s3_viridian_forest	gemini-lite	ceiling probe 3 seed(s) probed.	max_turns:3	100	0.065041	6	85%	-	0 · 1 · 2
s3_viridian_forest	gpt	ceiling probe 3 seed(s) probed.	max_turns:3	100	0.622188	18	65%	-	0 · 1 · 2
s3_viridian_forest	haiku reasoning off	ceiling probe 3 seed(s) probed.	max_usd:3	223	1.00276	1	100%	-	0 · 1 · 2
s3_viridian_forest	qwen-local	ceiling probe 3 seed(s) probed.	max_turns:3	100	0	18	69%	-	0 · 1 · 2
s3_viridian_forest	sonnet reasoning off	ceiling probe 3 seed(s) probed.	max_turns:3	100	1.4077	14	69%	-	0 · 1 · 2
s4_viridian_mart	gemini	100% (3 seed(s))	success:3	9	0.036513	2	80%	-	0 · 1 · 2
s4_viridian_mart	gemini-lite	100% (3 seed(s))	success:3	7	0.003833	3	67%	-	0 · 1 · 2
s4_viridian_mart	gpt	100% (3 seed(s)) 1 seed excluded	success:3	6	0.026125	1	100%	cap_mismatch: stopped on 'max_usd' during a turn-matched sweep — the row would carry fewer turns for a budget/hardware reason, not a capability one	0 · 1 · 2 +1 more attempt attempt 1
s4_viridian_mart	qwen-local	0% (3 seed(s))	max_turns:3	100	0	11	77%	-	0 · 1 · 2
s4_viridian_mart	sonnet reasoning off	100% (3 seed(s)) 9 seeds excluded	success:3	11	0.08367	2	90%	cap_mismatch: stopped on 'max_usd' during a turn-matched sweep — the row would carry fewer turns for a budget/hardware reason, not a capability one cap_mismatch: stopped on 'max_usd' during a turn-matched sweep — the row would carry fewer turns for a budget/hardware reason, not a capability one cap_mismatch: stopped on 'max_usd' during a turn-matched sweep — the row would carry fewer turns for a budget/hardware reason, not a capability one cap_mismatch: stopped on 'max_tokens' during a turn-matched sweep — the row would carry fewer turns for a budget/hardware reason, not a capability one cap_mismatch: stopped on 'max_tokens' during a turn-matched sweep — the row would carry fewer turns for a budget/hardware reason, not a capability one incomplete: no summary.json — the run never finished (this is the tell that exposed the 429 sweep incident) incomplete: no summary.json — the run never finished (this is the tell that exposed the 429 sweep incident) incomplete: no summary.json — the run never finished (this is the tell that exposed the 429 sweep incident) incomplete: no summary.json — the run never finished (this is the tell that exposed the 429 sweep incident)	0 · 1 · 2 +9 more attempts attempt 1 · attempt 2 · attempt 3 · attempt 4 · attempt 5 · attempt 6 · attempt 7 · attempt 8 · attempt 9

reasoning off beside a model name means reasoning OFF — not a like-for-like comparison for that row.

ceiling probe in the success column of every s3_viridian_forest row means ceiling probe, not a ranking result — no scripted oracle exists for this scenario, the success predicate has never fired in any traced run, and winnability under the current anchor state is unverified.

How this table was produced / reproduce it offline

No ROM, network, or API key required:

python -m uv run pokebench score $(grep -v '^#' results_traces.txt) --out results.json

python -m uv run pokebench score $(grep -v '^#' results_traces.txt | grep -v sweep_s4) \
--out results.json

python -m uv run pokebench score $(grep -v '^#' results_traces.txt | grep sweep_s4) \
--turn-matched --out results.json

Full provenance, schema history, and regeneration notes: results_traces.txt in the repo.

9 seeds were excluded during curation, here's why

Every row below already reports seeds_excluded per its own (model, scenario, tier) group -- these 9 seeds were curated out as invalid evidence before scoring even ran, so no row's own exclusion count reflects them.

4 seeds excluded — Sonnet with reasoning ON. The harness's shared max_output_tokens=1024 is also the thinking budget, so these hit the ceiling on 39% of turns and emitted NO tool call on 37%. They measure the ceiling, not the model.
2 run directories
```
runs/bench/s1_exit_pallet-sonnet-t0/20260721-092407/seed{0,1,2}
runs/bench/s2_viridian_pokecenter-sonnet-t0/20260721-103105/seed0
```
2 seeds excluded — Killed at 31 and 1 turns on 2026-07-28 once the truncation above was diagnosed mid-run (they would have reproduced it). $0.60 spent.
2 run directories
```
runs/bench/s2_viridian_pokecenter-sonnet-t0/20260728-101506/_aborted_outputcap_seed0
runs/bench/s3_viridian_forest-sonnet-t0/20260728-102030/_aborted_outputcap_seed0
```
1 seed excluded — Died on the wall-clock cap at 86 turns, before that cap was lifted for the local leg.
1 run directory
```
runs/bench/s1_exit_pallet-qwen-local-t0/20260721-091812/_discarded_wallclock_seed2
```

2 seeds excluded — Incomplete — no summary.json.

2 run directories

runs/bench/s2_viridian_pokecenter-sonnet-t0/20260721-103105/seed1
runs/bench/s3_viridian_forest-sonnet-t0/20260720-125035/seed1

Separately, 3 earlier single-seed run group(s) were superseded by the later M2 sweep -- replaced, not invalid evidence; not counted in the total above.

Full provenance, verbatim: results_traces.txt in the repo.