
How to make a coding agent keep working until your Solana tests pass
Short answer
Take the decision away from the model. Give the agent one command that proves the work, such as cargo test or anchor test, and have something else run it: a Claude Code Stop hook, CI, or a checker. In Oh My Sol, /goal <text> --check "<command>" keeps the model going until that check passes, and only an independent checker on a fresh offline Surfpool chain can close the task.
- Who it’s for
- Solana developers whose AI agent stops early, skips the tests or edits them to pass, and who want a loop that ends only when the tests really pass.
- Last checked
- , against Oh My Sol 0.1.0
Why do coding agents stop before the tests pass?
In most coding agents, the model that writes the code also decides when the work is finished. That is the failure mode. A model can stop after the program compiles, report tests it never ran, or change a failing assertion instead of the code. On Solana, code that looks right can still fail when it runs on a chain, so the test run is the part that matters.
The fix is the same whatever tool you use: define “done” as a command that exits 0, and let something other than the model run it and read the result.
How do I set this up with any coding agent?
- Write or pick the check first.
cargo testruns in-process tests such as LiteSVM or Mollusk quickly.anchor testbuilds the program, deploys it to a local validator and runs the tests. - Give the agent the exact command and say the task is not done until it exits 0, and that it must not edit the test files.
- Make the check run outside the model. In Claude Code, a Stop hook can run the tests and exit with code 2 on failure, which prevents Claude from stopping and continues the conversation. Check
stop_hook_activeso a failure that will never resolve does not loop. In any agent, a CI job on the branch does the same job after the fact. - Protect the tests. Review the diff for changes under
tests/, or keep the test files somewhere the agent cannot write. - Run the check yourself before you merge.
Sources: Anchor installation, Claude Code hooks.
How does Oh My Sol keep working until a check passes?
Oh My Sol (OMS) is built around this split. Every task has a requirements file, its own Git worktree and a budget. The model edits files in the worktree, but its only completion tool is propose_done, which moves a requirement from active to proposed. It has no tool that closes one.
A separate checker then starts a fresh offline Surfpool chain from a pinned snapshot and runs the requirement’s checks against a saved copy of the code, not your live worktree. A pass closes the requirement. A fail sends it back to active and the model tries again if budget remains. When the task was created, OMS recorded digests of the contract, the verifier files and the evaluator config; if any of them changes, the evaluator refuses to run.
To set a goal with your own check inside OMS:
/goal <text> --check "<command>"
/status # requirements, proof and budget
/diff # task changes against the base
/verify # rerun the independent check, no model turn
/goal clear # remove the goalWhat does a run look like?
/goal Add a withdraw instruction that rejects non-owners --check "cargo test"The model edits the program and proposes that it is done. The checker runs cargo test. If a test fails, the requirement goes back to active, the check output is passed to the model (through a redactor), and it tries again. While it works you can run /status, /usage and /diff. When the check passes, the requirement closes as verified and the session settles.
OMS stops and hands control back to you when any of these happens:
| Stop reason | What it means | What to do |
|---|---|---|
repeated_failure | The same failure came back twice in a row | Read the feedback, change the requirement or the code, run /goal again |
| Attempt cap | Attempts reached the cap, 5 by default | Check /status and /diff |
blocked | The checker reopened the requirement 3 times | Look at the check output with /status and /diff |
| Budget used up | Defaults: 2,000,000 tokens, $5, 30 minutes | See /usage |
| Checker unavailable | OMS stops rather than let the work through unchecked | Run /verify again, then oms doctor to check Surfpool |
Does the check make a difference?
Yes. With the same models, OMS beats both Codex CLI and Claude Code on eight fresh Solana tasks, each graded by a hidden check: 96% of runs passed against 76% for Codex CLI with GPT-6.1 Sol, 92% against 79% for Claude Code with Claude Opus 5.5, and 71% against 46% for Claude Code with Claude Sonnet 5.5. It also used 42 to 62% fewer input tokens per solved task. The benchmarks show every model.
OMS runs on macOS and Linux, or Windows under WSL 2, with Node 22.19.0 or newer and Git. You bring your own model access.
Frequently asked questions
Why does my AI coding agent say it is done when the Solana tests still fail?
Because in most agents the model itself decides when the work is finished. It can stop after a build that compiles, after tests it never ran, or after editing a test so it passes. The fix is to let something other than the model decide: a command that must exit 0, run by a hook, CI or a separate checker.
How do I keep Claude Code working until my tests pass?
Add a Stop hook that runs your test command and exits with code 2 when it fails. Exit code 2 on a Stop hook prevents Claude from stopping and continues the conversation. Check the stop_hook_active field so a failure that will never resolve does not loop forever.
What does /goal do in Oh My Sol?
/goal <text> --check "<command>" tells Oh My Sol (OMS) to keep working until that check passes, or until a cap stops it. It stops when the same failure repeats twice in a row, when attempts reach the cap (5 by default), or when the budget runs out. /goal clear removes the goal.
Can the model in Oh My Sol mark its own task as done?
No. The model’s only completion tool is propose_done, which moves a requirement from active to proposed. A separate checker closes it, and only when its checks pass against a fresh offline Surfpool chain. The checker also refuses to run if the pinned contract or verifier files were changed after the task was created.
What does "verified" mean?
It means the pinned checks passed for a saved copy of the code on a fresh chain. The checks are yours, so write them for the behavior you care about and review the diff before you merge.
How much can a single Oh My Sol task spend?
The defaults are 2,000,000 tokens, $5 of API spend, 30 minutes and 32,768 output tokens per response. A normal interactive launch has no idle expiry; pass --bounded to apply the caps to the whole session. /usage and /status show the numbers as you go.
Try it on your own program
Oh My Sol 0.1.0 runs on macOS and Linux, and on Windows through WSL 2. You need Node 22.19.0 or newer and Git. You bring your own model access: an API key, a ChatGPT subscription or a Claude subscription.
npx @barestate/oms doctor --install
npx @barestate/oms