We asked a security model to find a real bug.
It matched the filename.
A vendor released small, open-weight models that point at the files likely to contain a vulnerability — running on your own machine, so source never leaves the building. Good idea. We did not want an opinion about it. We wanted an operating envelope: where can this be trusted, and where can it not.
The one case that explains the rest
We asked it to find a path-traversal flaw. It pointed at the file called
pathconverter.py. The bug was in b64.py.
Asked to find a path flaw, it reached for the file with “path” in its name. It matched the filename, not the code.
How the test was built
Ground truth we had not written: published security advisories with their real fix commits. Check the repository out at the commit before the fix, and the files that fix touched become the answer key.
The prompt carried the vulnerability class and nothing else. Advisory summaries name the affected module, and passing those in measures reading comprehension rather than localisation.
What came back
8 real vulnerabilities, 8 different projects
5 produced a verdict at all
1 put the correct file first
0 ever said "there is nothing here"That last line matters more than any percentage. If a tool cannot report “clean”, an inconclusive run and a clean run look identical — and you cannot tell its one hit from its four confident misses without already knowing the answer. On clean repositories, with a bigger budget, even when explicitly reminded the option existed, it never used it. Under pressure to find something absent, it invented a plausible filename and asked for it twenty-two times.
Before we blamed the model
Three times, “the model is weak” turned out to be our own harness. The worst: a parser bug meant zero tool calls ever reached the sandbox, while the transcript showed nineteen. It looked exactly like a small model stuck in a loop. It was a regular expression.
Stop at the transcript, and we publish “this model is useless.”
Read the counters, not the transcript.
What we are not claiming
We exposed nothing. The vendor published a comparable score themselves — we reproduced their honesty rather than catching them out.
Limits belong beside the numbers, not in a footnote: eight cases is not a rate, our corpus favours small localised fixes, our harness is stricter than the vendor's own tooling, and we tested the smallest model in the family.
Why we bothered
The market for securing AI agents filled up fast — vendors, scanners, academic work, an OWASP class for poisoned agent memory. Plenty of tools that inspect what an agent consumes.
Almost nobody publishes the boring, decisive artifact: the operating envelope. Where the thing can be trusted, where it cannot, and how you would know. That is the difference between deploying a model and governing one.
Better to find out in a lab on a Sunday than in front of a customer on a Tuesday.
Attack-first. Falsify. Measure. Then decide.