Same task, same prompt byte for byte, same model (claude-sonnet-5). Fourteen scored builds. The only thing that changes is what sits between the model and merge-ready.

What happened

The task is a small booking app: SQLite, Express, a form. Real enough to have real failure modes - ownership checks, date validation, a booking that must be unique, user text rendered back to the page.

Bare Claude Code is good at this. Its best run scored 110 of 120 in two minutes, so the code itself is about as good either way. The difference is consistency. Four runs on the identical prompt did not agree - two shipped stored XSS, three accepted 2026-13-45 as a date - and with no gate, every one of those builds was handed over as done.

Two early Vibestrate rounds wrote the same XSS. Neither reached merge-ready: the review named it, the gate held, both runs ended blocked. That is the claim this page makes. The model still makes the mistakes. Something stands between the mistake and your main branch, and on the runs where it mattered, it did its job.

One control, disclosed rather than buried: bare Claude, given four engineering standards pasted into its prompt, also shipped no XSS. Telling the model helps, in either arm. What the standards did not fix, in any bare run, is the sharpest result in the benchmark: every bare build, and the hand-written reference implementation used to validate the judge, served the whole company’s booking history and per-person charges to a request with no identity at all. Of the fourteen scored builds, two refused that request. Both were Vibestrate runs, on a flow that puts an independent security panel between the code and the verdict.

The scoreboard, all of it

Fourteen scored builds; every score file is in the benchmark repo, including the iteration rounds. secure-build is not a built-in: it was written for this benchmark in the product’s own flow format - a security panel, a fix pass, a re-arbitration, a verify gate. It can be written in an afternoon, and it does not ship in the box.

Where Vibestrate fell short, and what changed

The default flow’s 94 is on this page on purpose, and so is the fact that two early Vibestrate builds wrote the same XSS the bare runs wrote. The cause was a real product defect this benchmark found: project policies - rules like “never render untrusted data as HTML” - only reached the reviewer, never the code-writing seat. A rule the writer never sees can only catch the defect after it exists, at the cost of a full review-and-fix round trip.

We fixed it and ran the round again, because “fixed” is a claim. Same prompt bytes, same model, same four one-line policies as the round that scored 97. The only change was the product, and we committed the prediction to the repo before the run. Result: 110 of 120, approved first pass, in 7 minutes. The fix loop never ran, because the rules reached the writer and the XSS, the date bug and the missing database constraint were never written at all.

The billing endpoint is what is left. No one-line policy names it, because the brief never defines who “admin” is, so the default flow ships it, the way every bare run and the reference author did. Catching it took the security panel, and the panel caught it in two of its five runs. So: rules you write are enforced before the code is written, and the defects nobody named still need a reviewer standing in the right place, sometimes a human.

Round two: SWE-bench Verified, judged by the official harness

The booking app above is our task and our rubric, so round two removes us from the judging entirely. SWE-bench Verified is the standard benchmark for coding agents: 500 real GitHub issues from twelve open-source projects (our six drawn instances landed in django, sympy, pytest and sphinx), where the ground truth is the maintainers’ own fix and the judge is the repository’s own test suite, run by the official swebench harness in Docker. A patch resolves an instance only if the tests the maintainers wrote for the real fix flip from failing to passing while the rest of the suite holds. Nothing of ours grades anything.

We drew six instances by a rule committed before the draw: sort all 500 by the SHA-256 of their id and take the first six. Both arms got the same repository at the same commit and the same issue text, byte for byte. Bare Claude Code got the issue piped straight in; Vibestrate ran it as a task with its stock configuration, disclosed in full below. We submitted every patch to the harness, including the ones Vibestrate itself declined to certify.

Six instances, reported instance by instance - six is what one machine and one day buys, and we are not printing a percentage from it. Failed means the maintainers’ test for the real fix still fails (13798: test_latex_basic; 18211: test_issue_18188). Neither arm broke a single previously-passing test on any instance. Dollar figures are the product’s ledger estimate at API list prices, not a bill; on a subscription plan the same run counts against that plan’s usage limit instead. Bare runs were not instrumented.

The first two columns are identical, instance for instance: the same four resolved, the same two failed, down to the same failing test. Claude writes the code in both arms, and on resolution the harness found no difference.

The third column is where they differ. Bare Claude’s closing message told you nothing: some runs ended asking to run tests, some warned tests never ran, on correct and broken patches alike, and its one flat “it’s been verified working” claim sits on a patch that fails the maintainers’ tests. Vibestrate’s verify gate passed one patch, a correct one. Its reviewer and verifier refused to certify the other five, because no run managed to execute the repos’ test suites, and Vibestrate refuses to certify code whose tests never ran. Both broken patches sit inside that refused set. No patch that fails the maintainers’ tests ever passed Vibestrate’s gate. The cost, in the same breath: three of the five refusals were over-caution on patches that turned out correct. The gate never claimed to know which patches were wrong. It declined to vouch for what it could not check, and on the two instances where that proof was missing and mattered, the most confident “done” in either arm was bare Claude’s, on a broken patch.

The exact configuration, in full

The Vibestrate arm is stock: vibe init --yes plus three setters, the built-in default flow, the scaffolded default crew. No project policies, no validation commands, no custom rules beyond what init writes for everyone.

vibe init --yes
vibe config set providers.claude '{"type":"claude-code","command":"claude","args":["-p"],"input":"stdin","settings":{"safeMode":true}}'
vibe config set profiles.claude-balanced.model sonnet
vibe config set profiles.claude-balanced.power medium
vibe run "$(cat ISSUE.md)" --flow default --crew default --unattended

The default flow: Plan → Architecture → Implement → Validate → Review, looping back through Fix and Re-validate up to twice (three review passes in all), then an independent Verify gate decides merge-readiness. The default crew seats six roles - Planner and Architect read-only, Implementer and Fixer with code-write, Reviewer and Verifier read-only - every seat on the same profile: Claude Code, sonnet, medium effort. The bare arm runs the identical model and effort: claude -p --safe-mode --model sonnet --permission-mode acceptEdits --effort medium.

What this round disclosed about us

A correction, added after this page first went up. We originally described those five refusals as environmental - a vetoed network install here, a missing test command there. That was not the whole story, and the larger part was our fault. Vibestrate’s permission model let a seat be marked as allowed to run commands while never actually granting it: the headless CLI auto-approves file edits but not shell commands, so a command outside the host’s own allow rules waited for an approval nobody could give, and the call simply hung. One reviewer seat made 39 attempts and collected 21 “this command requires approval” results while its own profile said it had shell access. So on those runs the gate was not being careful by design. It was refusing to certify work it had been stopped from checking.

That defect is fixed. We have not re-run SWE-bench Verified since, so the table above stands as measured and we are not restating it. On a separate leg that we did re-run - a chess move generator scored by exact node counts - the same fix moved certification from one run in three to three in three, with zero approval denials, while the code itself scored a perfect 53 of 53 on every run before and after. The gate stopped refusing correct work; it did not make the code better.

• The benchmark found a real Vibestrate defect: the task field was capped at 2,000 characters, so two of the six issues were refused before any agent ran. We did not truncate the issues or work around it - we fixed the product (the cap is now GitHub’s own 65,536 issue-body limit, behind one shared schema) and re-ran only those two instances on the fixed build. That is the third first-contact defect this benchmark program has found and fixed.
• Unattended runs cannot answer approval requests, so “refused to certify” is the expected terminal state when tests cannot run; a human in the loop could have approved the dependency install and gotten a certified-or-rejected verdict instead. sympy‑17318’s run passed its own verify and was held only by the no-auto-merge rule - Vibestrate never merges on its own.
• Evaluation ran the official x86_64 harness images under emulation on an arm64 machine - identically for both arms. The first instance (django‑16100) ran on both arms before the submit-everything rule was written down; the rule was committed before the other five ran.
• The draw rule, per-run predictions, amendments, patches, harness reports and per-instance logs are all committed in the benchmark repo.

Round three: the Supervisor caught a reviewer that could not see the code

Four steps ran. On the fourth, the Supervisor noticed what the reviewer had not been given, and added it.

The Supervisor’s recorded reason. It answers with the name of an artifact and the system supplies the bytes, so it cannot put a word into a prompt that no step produced. It is never asked what to remove.

It ran on every seated step and acted on the one where something was missing. On the others it said so and added nothing. This was one run, and the score was 110 of 120 with or without it. What it shows is the Supervisor spotting a gap, not a better score.

Same prompt, same model, same judge, both arms with a shell

Three default-flow runs against three bare runs. We granted both arms the identical shell rules Vibestrate’s seats receive, so neither is writing code it cannot run.

Both arms served everyone’s billing to anonymous callers in all six runs - that check is not a differentiator here and we do not count it as one. Each run started from the same committed base; every score file and every trace is in the benchmark repo.

How the scoring worked

• The 100-point core rubric and the judge were committed before either path ran. Neither path is told a test suite exists. Two checks were added later - stored XSS after round 1, the billing read after round 2 - and applied retroactively and identically to every build of both paths, all re-scored on the final judge.
• Nothing is graded by a model. Every point is an HTTP status code, a row count, or the schema.
• The judge is itself tested: a deliberately correct reference scores 110 of 120, and a four-bug mutant scored 85 of the 110 points that existed when it ran, failing exactly the four checks the bugs break.
• Both paths get the same prompt bytes (checksummed), the same model, the same permission mode for the code-writing turn, the same host isolation. One asymmetry, disclosed: Vibestrate injects the 48-line rules.md its init scaffolds; the bare path never saw it.
• Two harness bugs and one discarded run (the host slept mid-response) are disclosed in the write-up. We rewrote this page after an adversarial fact-check of its first draft.

Claude writes the code either way, and its best bare run nearly tops the rubric. Across four identical runs it had two bad days. The difference between the two columns is whether anything was standing there when it did.