Benchmarks overview

This page gives you the headline numbers for Oh My Sol (OMS), a short guide to reading them, and the limits that come with them. Start here if you want to know how OMS compares with other coding harnesses before you open the detail pages. Two sets of runs are reported. In both, OMS is compared with other coding harnesses that were run as shipped. The two sets ran under different conditions, and a result needs its conditions to mean anything, so every number carries one of these two labels:

Headline results

The p-values come from two-sided permutation tests on the difference of means. Such a test reshuffles which runs belong to which harness and asks how often a gap at least as large as the real one, in either direction, turns up by chance. A small p-value means chance rarely produces a gap that big. The fresh tasks also ran on three other models:

How to read the numbers

Pass rates are given as passes out of runs, with a 95% Wilson interval: the range the true pass rate plausibly sits in. A wide interval means few runs, and most cells have only 24 runs, so expect wide intervals. Matched comparisons pair runs of the same task and run index, so both harnesses are judged on the same work. The sign-test p-value looks only at pairs where exactly one harness passed, because a pair where both passed or both failed says nothing about which harness is better. Solana Bench scores count unique (program id, first instruction byte) pairs from successful transactions. Scores are compared with a permutation test on means, using same-batch runs only. OMS is slower than its rivals. On the fresh tasks it takes about twice as long on the clock as the CLI harnesses, and in Solana Bench it is the slowest of the three.

What the results do not show

These limits come straight from the data. Read them before you quote a headline number.

• One task, fresh-anchor-error-decode, is passed only by OMS, which solves it with its error decoding tool. Because one task can move the totals, the detail pages report comparisons both with and without it.
• Claude Code has only 2 runs per variant in the round-16 Solana Bench comparison. Treat it as a reference, not a test.
• OMS’s mean was above Codex’s in every one of the 8 batches. Runs from different batches are still pooled within each harness, though, so a batch effect is possible.
• OMS stopped below 50 submissions in 4 of its 24 headline Solana Bench runs.
• Two Codex CLI swap runs were excluded from the headline for infrastructure failures. Including them gives p = 0.0024 instead of 0.0101. Runs with fewer infrastructure failures stay in.
• Token counts reach us through a different route for each harness, so no token-saving claim is made.
• Older OMS builds lost to Codex CLI in earlier Solana Bench series. See Solana Bench.
• When the Solana Foundation environment was run privately, an OMS variant showed no measurable gain.
• No red-team benchmark was run, so these numbers support no safety claim.
• Neither Solana Bench run is comparable to the Solana Foundation’s published tables.

Pages