Notes from the Ethereum Foundation’s Protocol Security team on running coordinated AI agents against real protocol code, including how we organize the work, what holds up under scrutiny, and what client teams and security researchers can take from it. This post stands on its own; later posts will go deeper on individual clients.
What we’ve been running, and what surprised us
On the Ethereum Foundation’s Protocol Security team, we’ve been running coordinated AI agents against the kinds of systems the network depends on, like systems software, cryptographic code, and contracts that have to be right. The agents found real bugs. One is now public: a remotely-triggerable panic in libp2p’s gossipsub, a core part of the peer-to-peer layer Ethereum consensus clients run on, fixed and disclosed as CVE-2026-34219 with credit to the team.
Agents finding bugs wasn’t the surprise. The surprise was how little of the work went into finding them, and how much went into telling the real bugs from the ones that just looked real.
This post is for client teams and security researchers who want to do the same thing. It covers how we organize the agents, the bar a candidate has to clear before it counts as a finding, and the habits that keep the results trustworthy.
Teams elsewhere are converging on the same recipe. Anthropic’s Frontier Red Team built an agent that writes property-based tests and found real bugs across the Python ecosystem. Cloudflare ran a frontier model through a security-research harness against their own systems. Everyone lands on the same loop: point a capable model at a codebase, let it search, and triage what comes back. So the real question is how to do this without drowning in confident-sounding noise.
One caveat up front: tooling for agent-driven audits moves fast, and any specific setup is out of date in a few weeks. So this post is deliberately about the methods, which are persistent, rather than the tooling. Disclosure is its own topic and will probably be its own post.
An agent pointed at a codebase is a search tool, a lot like a fuzzer. The difference is what comes back. A fuzzer hands you a crash and a stack trace. An agent hands you a lot more, including a write-up (call chain, impact claim, suggested severity) and the artifacts to back it, like a proof-of-concept you can run against the real code.
All of that makes the result easy to read and easy to trust, the running proof-of-concept most of all. So don’t count how many candidates an agent produces. Count how many turn out to be real.
How the work is organized
We run many agents in parallel against one target. They coordinate through the repository itself, with shared state in version control and no central process handing out work. An agent writes down a claim where the others can see it, does the work, and commits.
We got this approach from Anthropic’s writeup on building a C compiler with a fleet of agentswhich coordinates the same way. There’s no central coordinator to build or maintain, and less that can go wrong.
The roles are generated by the work that’s discovered:
Recon turns an attack surface into concrete, testable hypotheses. Not “audit the decoder” but “this field is trusted past this point; here’s the property it should keep, the way it might break, and the proof that would settle it.”Hunting takes one hypothesis, traces the code path, and tries to build a reproducer.Gap-filling looks at what was accepted and what was rejected, writes the next batch of hypotheses, and tracks coverage so the agents don’t keep going over the same ground.Validation re-checks each candidate independently, removes duplicates, and decides.
We didn’t invent this pipeline. Cloudflare describes the same stages, recon, parallel hunting, independent validation, deduplication, reporting, and their writeup helped shape ours.
Here’s what a candidate looks like before it counts as a finding:
target: component and entry point an attacker can actually reach
invariant: the property that must hold
mechanism: the specific way it might be made to break
success: observable proof: a panic, a stall, an accepted-invalid input
reproducer: a self-contained artifact that runs against the real code
dedup: a key, so two agents don’t chase the same thing
The schema is there for a reason. It forces a specific, testable claim and a clear definition of done. An agent that has to write down an observable proof can’t fall back on “this looks risky.”
Reproducible or it didn’t happen
One rule matters more than any other. A candidate isn’t a finding until there’s a self-contained artifact that reproduces the failure against the real code, and that runs for someone who didn’t write it.
The reproducer doesn’t read the write-up, and it doesn’t care how confident the model sounded. It either runs or it doesn’t.
Most of its value is in the false positives it catches. Three of them come up over and over, and each one is the agent getting a pass for the wrong reason:
A panic that only happens in a debug build. Compile and run it the way the software actually ships, and the value just wraps around. Nothing crashes. It looks like a crash, but it isn’t one.A reproducer that builds some internal value by hand, one no real input could ever produce, because every path an attacker controls rejects it earlier. The bug only “reproduces” against a function that nothing reachable calls that way.In formal-verification work, a proof that goes through but doesn’t mean what you wanted. The statement is trivially true regardless of what the code does, or it’s weaker than the property you meant to capture. The verifier is satisfied, but the theorem doesn’t constrain the behavior you actually cared about.
None of this is new. It’s the same thing as a test that passes because it doesn’t actually check anything. What’s new is the volume. An agent writes the useless version as fast as the real one, and just as confidently. So the check has to be automatic. You can’t count on the agent to catch itself.
Signal-to-noise is most of the work
Most candidates are wrong, duplicate, or out of scope. That’s not a problem with the method; that’s how it works. The goal is to reject the wrong ones fast and back the real ones with proof that’s hard to argue with.
Every candidate that survives gets two independent checks. Can a real attacker actually reach it in a normal configuration? And what does it cost the attacker to pull off, compared to what it costs the network if it works? A bug that any single peer can trigger is very different from one that needs special access or a huge amount of resources.
Everything gets checked against a running list of what’s already known, fixed, or rejected. Without that, the agents keep rediscovering the same closed issue and reporting it again and again.
Acceptance rates vary a lot from target to target, and that variation is useful on its own. Run this against mature, heavily audited code and almost nothing survives, which is still worth knowing. “We looked hard and found nothing” is a real result. Run it against less-explored code, or against formally verified code, where a machine-checked proof covers a model and the deployed bytecode is only assumed to match it, and more gets through.
We’re not the only ones who found that the triage is the hard part. Cloudflare’s main takeaway was that a narrow scope beats broad scanning. Anthropic’s property-based-testing agent generated something like a thousand candidate reports, then used ranking and expert review to get down to a top tier that held up about 86 percent of the time. The generation was the easy part. I’m not going to publish our own numbers here; tied to a specific target, they’d say more about the target than about the method.
What the agents are good at, and where they mislead
There’s hype in both directions, so here’s a plain list of what the agents do well and where they mislead.
Good atMisleading atReading the spec and the code togetherCall chains that look reachable but aren’tStating and checking a real invariantGaming the success check (a pass for the wrong reason).Drafting a reproducer from a one-line ideaInflating severity to match how dramatic the write-up soundsSuggesting a root cause before you’ve lookedBugs that span a sequence of valid steps
The split isn’t even steady from one task to the next. Stanislav Fort, testing a range of models on real vulnerabilities, calls this a jagged frontieror a model that recovers a full exploit chain on one codebase can fail basic data-flow tracing on another. You can’t assume one good result means the next will hold up, which is another reason every candidate gets checked on its own.
The last row is the important one. A single agent session is good at one-shot reasoning and bad at bugs that span a sequence of steps, where each step is valid and only the order is wrong. For those, the agent isn’t the search tool. Its job is to suggest which sequences are worth running through a stateful test harness. Used that way, it works well. Used as a replacement for the harness, it misses the most expensive bugs there are, the ones that only show up across a sequence.
Keeping it honest
A few habits do most of the work of making agent findings trustworthy, and none of them are complicated.
Provenance on every artifact: what produced it, with what context, against which revision. A finding should be something you can re-run months later.Determinism where it counts: one environment, one way to build and run, so “reproduces” means the same thing on every machine, not just the one where it was found.Norms, not scripts: tell agents what matters, the invariants and the bar for a real finding, instead of a numbered procedure. Over-scripted agents break the same way over-specified tests do, they keep following the steps after the steps stop making sense. A study of repository context files found the same thing: the extra requirements lowered task success and raised cost by over 20%, and the authors recommend keeping context to the minimal requirements.A person makes the final call: agents suggest. They don’t decide what’s real, what’s a duplicate of a known issue, or what gets disclosed and when.
The bottleneck moved
AI didn’t replace the security researcher. It moved the work. The time that used to go into coming up with and chasing down hypotheses now goes into judging them at scale, including building the oracle, running the triage, keeping the list of known issues, and handling disclosure.
The bottleneck didn’t go away. It moved from finding bugs to trusting the results, which is a better place for it, because that’s where human judgment actually matters. But it’s still a bottleneck, and ignoring that is how you end up shipping a wrong “it’s fine.”
The practices that make this work aren’t new. Reproducible failures, real oracles, and careful triage are the same practices that turned fuzzing from a research topic into standard practice over the last fifteen years. The tools are new. The practices aren’t.
How fast the tools keep changing is an open question. Nicholas Carlini, careful and once a skeptic himself, argues the exponential case is worth taking seriouslyeven while he keeps wide error bars on it. If the generation side climbs that fast, the judgment side has to climb with it, or the gap between what gets produced and what actually gets verified only widens.
For the systems Ethereum depends on, that’s the part that matters. Agents let us cover far more ground than we could by hand. In exchange, they ask for more careful judgment, across a much bigger pile of confident-sounding claims. That’s a trade worth making, as long as you remember that the judgment is the real product.
GIPHY App Key not set. Please check settings