The model can propose that a requirement is done, but it can’t close one. A separate evaluator does that, and only when its checks pass against a fresh offline Surfpool chain. That split is the point: the thing writing the code never grades its own work.
A requirement moves from active to proposed when the model calls propose_done. The model has no closing tool. The independent checker runs on a fresh offline Surfpool chain. When its checks pass, the requirement is closed as verified. When they fail, it is reopened and goes back to active. After 3 reopens it is blocked.A requirement moves from active to proposed when the model calls propose_done. The model has no closing tool. The independent checker runs on a fresh offline Surfpool chain. When its checks pass, the requirement is closed as verified. When they fail, it is reopened and goes back to active. After 3 reopens it is blocked.

The model can only propose. Only the checker can close a requirement, and three failed checks block it.

What “verified” means

A requirement is verified when four things hold. The pin verifies, meaning the pinned contract, verifier files and evaluator config still match their recorded digests. The requirement is in the proposed state. The evidence is bound to the pinned requirement, config, complete code inputs, verifier and snapshot bytes. And the run passed in offline-snapshot mode. A run in forked mode is labeled as such and never closes a requirement. Forked mode isn’t reachable from the terminal app anyway.

What the check does

Every run starts a fresh offline Surfpool from the pinned snapshot, then evaluates each done condition: The code under test is a saved copy of the inputs, not your live worktree, so edits you make mid-check don’t change the result.

Why the model cannot game it

When a task is created, OMS digests the contract (task.json), the verifier files and the evaluator config, and writes a pin file outside the workspace. The model only receives the code root. On every decision OMS recomputes the digests, and a mismatch makes the evaluator refuse. The runner also refuses to run if the code, verifier, snapshot or specification digest differs from the saved one. The file guards block the model from editing pinned files in the first place. See File-read guards.

A failing check

A failing run moves the requirement back to active and counts a reopen. At the default cap of 3 reopens, the requirement is blocked.

What it does not guarantee

Verified means the pinned checks passed. It doesn’t mean the code is correct in ways the checks never look at. The benchmark methodology records one concrete case. OMS’s own verifier closed 3 of 96 GPT-6.1 Sol runs falsely. All three were on fresh-create-account-deposit, where the code used the old rent rate (6,960 instead of 5,080 lamports per byte). The verifier passed those runs because its checks didn’t catch the old behavior. See Methodology.

Check it yourself

/verify runs the independent offline verification with no model turn. /status shows requirements, proof and budget.

Tests behind these claims

The pack lists these tests: “only a proposed item can be evaluated”, “mainnet-forked evidence is labeled and cannot close an item”, “a verifier edit after pinning makes the evaluator refuse”, “editing task.json to close an item is refused”, “failing runs are bounded by the reopen cap and then block the item”, and “the model-visible tool set has propose_done and no closing tool”.