This page has the full results for the fresh tasks: 8 offline tasks, each built around a Solana fact newer than the models’ training data. Use it when you want the per-model numbers behind the headline, the matched comparisons, or the token counts for Oh My Sol (OMS) and the other harnesses.
No internet for any harness. OMS uses its built-in Solana knowledge.
A run passes when the hidden check on its final workspace passes. Of the 720 rows recorded, 668 runs are counted. The rest are online cells, a Claude Code-through-API footnote cell, and 4 pi runs that lost their login. Intervals are 95% Wilson intervals. The methodology covers how the tasks were designed and graded.

Pass rate

Wall p50 and p90 are the median and 90th-percentile wall-clock time per run, in seconds. GPT-6.1 Sol has 96 runs each for OMS and Codex CLI, spread over 4 batches, and 68 for pi after 4 pi runs were excluded. Every other cell has 24 runs, which is 8 tasks times 3. OMS takes about twice as long on the clock as the CLI harnesses. The intervals for the 24-run cells are wide and overlap, so treat the gaps between them with care.

Matched comparisons

A matched comparison lines up runs with the same task, run index and batch, so the two harnesses are compared on the same piece of work. “OMS-only” means OMS passed and the other harness failed, and “Other-only” means the reverse. The p-value is an exact two-sided sign test on the pairs where exactly one harness passed. If the two harnesses were equally good, those pairs would split roughly evenly, and the p-value tells you how unlikely the observed split would be in that case. Each comparison has two rows. The second row drops fresh-anchor-error-decode, which only OMS solves, through its decode_error tool, so you can see how much of the gap rests on that one task. Among the comparisons with 24 pairs, only Sonnet is below 0.05 in both rows. The GPT-6.1 Sol comparison with Codex CLI is also below 0.05 in both rows, and its p-value is much smaller when all tasks are included.

Per task, GPT-6.1 Sol, 12 runs each

Each cell is the number of passes out of 12 runs. OMS missed fresh-create-account-deposit in 3 of 12 runs because it closed the task with the old rent rate. See The independent checker.

Tokens per run

Input is fresh input plus cache read plus cache write. “Per solved task” is the total over all counted runs divided by passes, which means the tokens a harness spends on failed runs still count against it. Claude Code reports one model call per run, so call counts are not comparable across harnesses. For the Claude cells, fresh input is near zero because Anthropic reports most input as cache read or cache write.