This page explains how the Oh My Sol (OMS) benchmarks were set up: how the tasks were built and chosen, which versions ran and when, and how a pass or a score was decided. Read it when you need to judge whether a number on the other benchmark pages means what it appears to mean. The offline runs and the Solana Bench runs had different conditions, so each number carries the label for its set. Every offline number carries this label: No internet for any harness. OMS uses its built-in Solana knowledge. Every Solana Bench number carries this one: Web on for every harness; same-batch runs on one VM.

Benchmarks in the pack

The pack holds six benchmarks. The last column gives the run counts and, where there is one, the outcome.

How the fresh tasks were built

  1. The original tasks were saturated. The 12 original tasks passed in every harness, so they could not tell harnesses apart and are now smoke fixtures only. A 20-task hard tier on GPT-6.1 Sol also showed parity.
  2. Fresh tier. Each task is built around a Solana fact newer than the model’s training data, with dated evidence for every fact. Two examples are the rent-exempt minimum change to 5,080 lamports per byte, and transaction format v1 active on mainnet from 2026-09-15 with Agave 4.2.2. The dates were checked on the public web on 2026-11-08.
  3. Authoring and review. One lab’s model wrote each task and the other lab’s model reviewed it, over 1 to 2 rounds. Tasks were frozen in git before their first scored run.
  4. Curation rule. A fresh task counts only if the target model gets its fact wrong offline in at least 2 of 3 tries, which shows the model does not already know the fact. The probe model was gpt-6.1-sol, and every task kept had 3 of 3 wrong or abstained answers. Two tasks did not make it. fresh-gated-burn was withdrawn because its source (2026-05-08) fails the post-June criterion. fresh-token-account-deposit was excluded after a defect: its hidden case expects a 165-byte mint with no account-type byte to be accepted, which the task text never states, and all 15 GPT-6.1 Sol runs failed it in every harness.
  5. Verification before use. The reference patch must pass the visible and hidden checks, and the starter must fail both. Every decoy, which implements the old behavior, must pass the visible check and fail the hidden one. That is why the visible test alone is not evidence of a pass.
  6. What the tier measures. The installed packages already ship the new APIs, so a harness can win a task by finding the current API offline in node_modules. The reference material is one source among others. No task was changed after results existed.
Hidden checks, reference.patch and decoy.patch are not published.

The eight scored tasks

Harness, model and tool versions

Where the sources do not record a version, the table says so rather than guessing.

Run dates

Fresh run records carry no timestamp, so the dates here are file modification times on the benchmark VM. The fresh v2 runs on that VM span 2026-11-09 10:25Z to 2026-11-10 07:02Z. Two GPT-6.1 Sol batches and part of the GPT-6 Astra cells ran on a second VM whose files are not available, so their dates are not available either. The first attempts that were re-pinned after an infrastructure error (72 of 72 failed with a task runtime pin mismatch) are not counted. Solana Bench run start times are in the raw file. The round-16 headline runs span 2026-11-14 05:19Z to 07:39Z, and the Foundation-environment private run spans 2026-11-10 09:58 to 12:28.

Reference material used by OMS

The offline fresh-task cells used a reference snapshot dated 2026-11-08. The online head-to-head and the Foundation-environment run used a build stamped 2026-11-09T21:43:48Z. For round 16, the exact reference version for that date is not available.

How pass or fail was checked

  • A pass needs the hidden check on the final workspace to pass. The visible test ships with the task and also passes for the old behavior, so it cannot decide a pass on its own: 704 of 720 rows pass it, and only 558 pass the hidden check.
  • One scoring rule for every harness. The final workspace is scored whatever stopped the run, whether that was OMS closing the item, a CLI exiting, or a wall cap. Runs with infrastructure failures and 4 pi runs that lost their login are not counted. Runs cut short by a provider stream cut are scored by their workspace.
  • Independent grader. The grader is host-owned and not part of any harness. It uses two containers. A visible container runs the pinned visible verifier on a copy of the workspace. A private container receives only the declared input files, the digest-checked hidden checks and trusted dependencies. Both are offline, read-only and non-root, with capabilities dropped. Hidden checks call contender code only through a helper that runs each call in a separate Node process with the permission model, which lets it read only /grade/src and /grade/node_modules, spawn nothing and write nothing. A pass needs exactly one per-grade nonce line, at least one passing test and no failures. If a task canary turns up in the exported workspace, the transcript, the proxy log or the visible container output, the run is marked invalid.
  • Offline grading. The hidden check never touches the network.
  • False completions. A false completion is a run that declared itself done but fails the hidden check. OMS’s own checker closed 3 of 96 GPT-6.1 Sol runs falsely, all on fresh-create-account-deposit at the old rent rate (6,960 instead of 5,080 lamports per byte).
  • Sample sizes. GPT-6.1 Sol has 96 runs for OMS and Codex CLI. Every other cell has 24 per harness (8 tasks times 3), except pi with 68 on Sol. Intervals are Wilson 95%. Matched comparisons use an exact two-sided sign test.

How Solana Bench was scored

The Solana Bench page covers scoring, exclusions and tests in full. In short, every harness gets one GOAL.md, with its sha256 recorded in each result. Each run has at most 50 submissions and 30 minutes on a fresh Surfpool 0.10.4 mainnet fork that is not slot-pinned, with a fork-only signing wallet. Cells ran interleaved in one container image, and comparisons use same-batch runs only.