Web on for every harness; same-batch runs on one VM.
What the score measures
In each run, one agent gets one goal file, the same for every harness. It has at most 50submit-tx calls and 30 minutes on a fresh Surfpool 0.10.4 mainnet fork. The agent never holds a key. Instead, the environment signs with a fresh fork-only wallet.
The score counts pairs. A pair is the program id of a top-level instruction plus the first byte of its instruction data. Each pair counts once per run, only transactions that executed successfully count, and instructions with empty data count nothing. Scores are reported in three forms:
- Filtered score (the headline). The count of pairs after three exclusions: Memo and Memo v1, SPL Noop and Metaplex Noop, and any program the run deployed or upgraded itself.
- Raw. All pairs from successful transactions, with no exclusion.
- Strict. Defined only for the Foundation environment run. It counts each arbitrary-data program once, so a program that accepts any instruction data cannot inflate the score.
basic funds the wallet with SOL, and swap adds USDC and USDT.
Round-16 comparison
OMS buildbench/r16 (commit 9b2baad) and Codex CLI ran GPT-6.1 Sol on the Codex subscription. Claude Code ran Claude Opus 5.5 on a Claude Code subscription. All three ran in one container image on one benchmark VM, interleaved run by run, on 2026-11-14 from 05:19Z to 07:39Z. Interleaving keeps the harnesses on the same machine over the same window, and only these same-batch runs are compared.
OMS has the highest mean in every row. Its lowest swap run (280) is above the Codex swap mean over all runs (268.6), but below Codex’s best runs (463). OMS stopped below 50 submissions in 4 of its 24 headline runs, and those runs are included at their score.
Significance
Each test is a two-sided permutation test on the difference of means. It pools both harnesses’ runs, splits them into two groups of the original sizes, and asks how often a split gives a gap at least as large as the real one, in either direction. A small p-value means a random split rarely produces a gap that big.
The basic comparisons are exact enumerations of every split. The Codex swap comparisons are sampled with 200,000 random splits and seed 20261114, so their p-values move in the third decimal between runs of the test. The Claude Code swap comparison has only 153 distinct splits, so its exact p is 9 of 153, 0.0588, which is not significant at 0.05. Claude Code has 2 runs per variant, so treat it as a reference rather than a test.
Exclusions
If the mainnet fork fails during a run, that is an infrastructure failure, not a result. In practice only the twoduelf Codex swap runs, with 13 “Node is unhealthy” failures each, were excluded from the headline, as agreed before the analysis. Runs with fewer infrastructure failures stay in, which favors Codex. A run is never excluded because of its score.
Tokens per run
Input is fresh plus cache read plus cache write. Each figure is a mean over the runs with complete token data.
OMS used about 25% (basic) and 27% (swap) fewer total input tokens than Codex CLI, but about three times more fresh input. The harnesses count tokens through different routes, so no token-saving claim should be drawn from this.
OMS is also the slowest of the three. Median wall time in minutes, for basic and then swap, was 27.7 and 27.8 for OMS, 24.2 and 22.4 for Codex CLI, and 14.0 and 15.5 for Claude Code.
Earlier series
These series used different OMS builds, so they are never pooled with round 16. All of them used GPT-6.1 Sol with web on, and in each row OMS had the lower median.
The sf swap medians include runs with 179 pairs from an arbitrary-data program the filter did not recognize. The pack records adjusted ledger values for these runs, but the result files do not hold them.
The round history shows how OMS got to round 16. Rounds 3 to 6 were near the Codex reference, round 7 regressed, rounds 11 and 12 fixed crashes after a bare 400 response, and round 16 was the first build ahead of Codex in same-batch runs.
The Solana Foundation environment, run privately
This run used the Foundation’s archived environment at its last commit (54c51ca, 2025-08-27), with GPT-6.1 Sol through Codex CLI 0.167.0 on a subscription. On 2026-11-10 it compared their unmodified loop with an OMS variant, 5 runs per cell. The test is an exact permutation test on the difference of medians over 252 splits.
There was no measurable gain for the OMS variant in that environment. One baseline basic run scored 1,497 filtered but only 95 strict, because its score came from programs that accept any instruction data. That run is why the strict score exists.