---
title: "Methodology"
description: "How the tasks were built and selected, which versions ran, when, and how pass and fail were decided."
url: https://barestate.xyz/docs/benchmarks/methodology
---

# Methodology

> How the tasks were built and selected, which versions ran, when, and how pass and fail were decided.

This page explains how the Oh My Sol (OMS) benchmarks were set up: how the tasks were built and chosen, which versions ran and when, and how a pass or a score was decided. Read it when you need to judge whether a number on the other benchmark pages means what it appears to mean.

The offline runs and the Solana Bench runs had different conditions, so each number carries the label for its set. Every offline number carries this label: **No internet for any harness. OMS uses its built-in Solana knowledge.** Every Solana Bench number carries this one: **Web on for every harness; same-batch runs on one VM.**

## Benchmarks in the pack

The pack holds six benchmarks. The last column gives the run counts and, where there is one, the outcome.

| Benchmark | What it measures | Condition | Runs |
|---|---|---|---|
| Fresh Solana tasks v2 (headline) | Pass rate on 8 tasks that need Solana facts newer than the models' training data | Offline, no web for any harness | 668 counted runs, 720 rows recorded |
| Pilot, real program tasks | 5 program tasks, GPT-6.1 Sol, OMS against Codex CLI | Offline | 15 runs per harness. OMS 15/15 and Codex 15/15, a tie at the ceiling |
| Online head-to-head | The same 8 fresh tasks with web | Web on, same Solana MCP | 24 runs per cell. OMS 21/24, Codex 24/24, not significant |
| Solana Bench, our harness | Unique (program id, first byte) pairs in at most 50 submissions and 30 minutes | Web on, same-batch runs | 189 runs with a result file, 48 in the round-16 headline set |
| Solana Bench, Foundation environment, private run | The same scoring idea in their loop | Their environment, GPT-6.1 Sol through Codex CLI | 30 scored runs |
| Red team (wallet safety) | Never run as a benchmark. The six scenarios were not frozen | Not applicable | No safety claim can be made |

## How the fresh tasks were built

1. **The original tasks were saturated.** The 12 original tasks passed in every harness, so they could not tell harnesses apart and are now smoke fixtures only. A 20-task hard tier on GPT-6.1 Sol also showed parity.
2. **Fresh tier.** Each task is built around a Solana fact newer than the model's training data, with dated evidence for every fact. Two examples are the rent-exempt minimum change to 5,080 lamports per byte, and transaction format v1 active on mainnet from 2026-09-15 with Agave 4.2.2. The dates were checked on the public web on 2026-11-08.
3. **Authoring and review.** One lab's model wrote each task and the other lab's model reviewed it, over 1 to 2 rounds. Tasks were frozen in git before their first scored run.
4. **Curation rule.** A fresh task counts only if the target model gets its fact wrong offline in at least 2 of 3 tries, which shows the model does not already know the fact. The probe model was `gpt-6.1-sol`, and every task kept had 3 of 3 wrong or abstained answers. Two tasks did not make it. `fresh-gated-burn` was withdrawn because its source (2026-05-08) fails the post-June criterion. `fresh-token-account-deposit` was excluded after a defect: its hidden case expects a 165-byte mint with no account-type byte to be accepted, which the task text never states, and all 15 GPT-6.1 Sol runs failed it in every harness.
5. **Verification before use.** The reference patch must pass the visible and hidden checks, and the starter must fail both. Every decoy, which implements the old behavior, must pass the visible check and fail the hidden one. That is why the visible test alone is not evidence of a pass.
6. **What the tier measures.** The installed packages already ship the new APIs, so a harness can win a task by finding the current API offline in `node_modules`. The reference material is one source among others. No task was changed after results existed.

Hidden checks, `reference.patch` and `decoy.patch` are not published.

### The eight scored tasks

| Task | One line |
|---|---|
| `fresh-anchor-error-decode` | Explain an Anchor program failure from the bare transaction error, without logs (anchor-lang 1.2.2 error codes) |
| `fresh-atomic-payout` | Pay up to 60 recipients in one atomic transaction the cluster accepts (transaction v1 size limit) |
| `fresh-budget-migration` | Move a legacy or v0 message's compute-budget instructions into the new-format message header |
| `fresh-confidential-keys` | Derive account-scoped Token-2022 confidential-transfer test keys through Surfpool 1.7.0's derivation RPC |
| `fresh-create-account-deposit` | Plan a System CreateAccount with the exact rent-exempt deposit at the current rate |
| `fresh-geyser-budget` | Summarize message version and resource budget from a decoded Yellowstone stream message |
| `fresh-large-message` | Compile one large atomic message in the current large-transaction format supported by Kit 8.6 |
| `fresh-wire-budget` | Read version, budget and layout from a serialized transaction in any format Kit 8.6 supports |

## Harness, model and tool versions

Where the sources do not record a version, the table says so rather than guessing.

| Item | Value |
|---|---|
| OMS build, fresh tasks, GPT-6.1 Sol | Commit 161143b for batches 1 to 3, main 7a83e91 for batch 4. The build changed between batches with fairness and speed fixes |
| OMS build, fresh tasks, GPT-6 Astra | b16e6f0 |
| OMS build, fresh tasks, Claude cells | Not available. The sources state no commit |
| OMS build, round-16 Solana Bench | Commit 9b2baad, tag `bench/r16-duelc-basic319-swap443` |
| Codex CLI | 0.167.0, stated for the 2026-11-10 series. Not recorded per run for the fresh runs or the round-16 runs |
| Claude Code | 2.1.319, stated for the 2026-11-10 series. Not recorded per run for the fresh runs or the round-16 runs |
| pi | Not available |
| Models | `gpt-6.1-sol` and `gpt-6-astra` (OpenAI, Codex sign-in). `claude-opus-5-5` and `claude-sonnet-5-5` (Anthropic) |
| Reasoning effort | GPT-6.1 Sol "low" in the Foundation-environment runs. Not available for other cells |
| Surfpool | 0.10.4 for Solana Bench. 1.7.0 for the fresh-task verifiers |
| Fresh task runtime | A pinned `node` image, Kit 8.6, locked read-only dependencies |
| Routes | GPT cells on a Codex subscription. Claude Code on a Claude Code subscription. OMS on the Claude models through the Anthropic API with a metering proxy. OMS on GPT through the Codex subscription |

## Run dates

Fresh run records carry no timestamp, so the dates here are file modification times on the benchmark VM. The fresh v2 runs on that VM span 2026-11-09 10:25Z to 2026-11-10 07:02Z. Two GPT-6.1 Sol batches and part of the GPT-6 Astra cells ran on a second VM whose files are not available, so their dates are not available either. The first attempts that were re-pinned after an infrastructure error (72 of 72 failed with a task runtime pin mismatch) are not counted.

Solana Bench run start times are in the raw file. The round-16 headline runs span 2026-11-14 05:19Z to 07:39Z, and the Foundation-environment private run spans 2026-11-10 09:58 to 12:28.

## Reference material used by OMS

The offline fresh-task cells used a reference snapshot dated 2026-11-08. The online head-to-head and the Foundation-environment run used a build stamped 2026-11-09T21:43:48Z. For round 16, the exact reference version for that date is not available.

## How pass or fail was checked

- **A pass** needs the hidden check on the final workspace to pass. The visible test ships with the task and also passes for the old behavior, so it cannot decide a pass on its own: 704 of 720 rows pass it, and only 558 pass the hidden check.
- **One scoring rule for every harness.** The final workspace is scored whatever stopped the run, whether that was OMS closing the item, a CLI exiting, or a wall cap. Runs with infrastructure failures and 4 pi runs that lost their login are not counted. Runs cut short by a provider stream cut are scored by their workspace.
- **Independent grader.** The grader is host-owned and not part of any harness. It uses two containers. A visible container runs the pinned visible verifier on a copy of the workspace. A private container receives only the declared input files, the digest-checked hidden checks and trusted dependencies. Both are offline, read-only and non-root, with capabilities dropped. Hidden checks call contender code only through a helper that runs each call in a separate Node process with the permission model, which lets it read only `/grade/src` and `/grade/node_modules`, spawn nothing and write nothing. A pass needs exactly one per-grade nonce line, at least one passing test and no failures. If a task canary turns up in the exported workspace, the transcript, the proxy log or the visible container output, the run is marked invalid.
- **Offline grading.** The hidden check never touches the network.
- **False completions.** A false completion is a run that declared itself done but fails the hidden check. OMS's own checker closed 3 of 96 GPT-6.1 Sol runs falsely, all on `fresh-create-account-deposit` at the old rent rate (6,960 instead of 5,080 lamports per byte).
- **Sample sizes.** GPT-6.1 Sol has 96 runs for OMS and Codex CLI. Every other cell has 24 per harness (8 tasks times 3), except pi with 68 on Sol. Intervals are Wilson 95%. Matched comparisons use an exact two-sided sign test.

## How Solana Bench was scored

The [Solana Bench](https://barestate.xyz/docs/benchmarks/solana-bench) page covers scoring, exclusions and tests in full. In short, every harness gets one `GOAL.md`, with its sha256 recorded in each result. Each run has at most 50 submissions and 30 minutes on a fresh Surfpool 0.10.4 mainnet fork that is not slot-pinned, with a fork-only signing wallet. Cells ran interleaved in one container image, and comparisons use same-batch runs only.