Proof, not preference
Muence builds coding environments where the reward is a program executing in a sandbox, not a human or a model opinion. A patch scores only when the attack dies and nothing else breaks.
1,000
verified tasks
64.9%
best model: Claude Opus 4.8
11.1%
worst: GPT-OSS 120B
265
tasks no model solves
How a task scores
Code can run and still be dangerously wrong.
Training rewards output that works. Security is usually absent from that loop, so models learn to ship a feature even when the feature leaves an exploit behind. Every check here is a program, so there is nothing to argue with.
- 01The app boots and passes its own spec test.
- 02A real exploit runs against it and prints PWNED.
- 03The model's patch has to make that exploit fail.
- 04And the spec test has to still pass afterwards.
Two ways to fail, and they mean different things.
A model can leave the hole open, or seal it and break the app. Scoring the two separately is what turns a leaderboard into a diagnosis, and it is visible for every one of the 7,000 trials.
What training on it does
A 14B goes from 6% to 78% on tasks it never saw.
Fine-tuning Qwen3-14B on 800 of the tasks and scoring it on the 200 held out moves it 72 points. Every model below is scored on those same held-out tasks, because the fine-tune cannot be measured on the rest — they are its training data.
Secure-patch rate · 200 held-out tasks