Building an LLM Vulnerability Scanner That Doesn't Drown You in False Positives

Every automated smart contract scanner fails the same way. Slither hands you two hundred findings and maybe four of them matter. Point an LLM at the same codebase and it will describe, with total confidence, a reentrancy bug in a function that makes no external calls. Neither tool knows what the protocol is supposed to do.

The scanner I built runs both, then spends real compute trying to disprove its own output before anything reaches me. That last stage does most of the work.

The shape of the system

┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│  Frontend    │────▶│  Backend     │────▶│  Temporal    │
│  React/Vite  │◀────│  FastAPI     │◀────│  Workflows   │
│  :3001       │ SSE │  :8001       │     │  :7233       │
└─────────────┘     └──────┬───────┘     └──────┬───────┘
                           │                     │
                    ┌──────▼───────┐     ┌──────▼──────────┐
                    │  SQLite      │     │  LLM provider   │
                    │  WAL mode    │     │  DeepSeek/Kimi  │
                    └──────────────┘     └─────────────────┘

FastAPI over 52 routes. SQLite in WAL mode across 16 tables. Tree-sitter handles AST work in seven languages, and Temporal sits underneath all of it. There are two isolated worker pools: web3-scan with 40 activities, web3-validation with 20. The split exists because a validation run that pins the CPU should never stall a scan that has been going for forty minutes.

Temporal is not decoration here. A deep scan on a large protocol runs for hours across hundreds of LLM calls. Without durable execution, one worker crash at minute 200 means starting from zero. With it, the workflow resumes at the activity that died.

The whole thing runs on your own machine with your own API key, so source and findings never leave the laptop. For audit work that is not negotiable.

Knowledge base first, findings second

The design choice people find odd: adding a codebase does not scan it. It builds a knowledge base.

KB-Creation runs recon, sink discovery, and briefing generation, and produces exactly zero findings. It populates kb_codebase, kb_chunk, and kb_sink, along with the antipattern tags that every downstream pipeline reads. Only after that do you launch a vulnerability scan.

The ordering came out of watching cold scans fail. An LLM handed a raw Solidity file has no idea which contract holds the funds, which function is the privileged entry point, or what the protocol is supposed to do at all. It pattern-matches on syntax and fills in the rest. Give it a recon manifest with protocol type, deployment target, entry points, and unprotected functions, and the same model starts reasoning about invariants instead of guessing at grep hits.

Five pipelines, different blind spots

Each pipeline is a separate Temporal workflow. They share the knowledge base and cover for each other.

SAST Hunter starts deterministic. The RECON phase is a regex parser with no LLM in the loop at all, pulling protocol type, risk level, entry points, and unprotected functions into a PipelineManifest. Sink discovery then runs Tree-sitter AST analysis plus regex to find data-flow sinks, and has an LLM validate them. Briefings turn recon output and sink clusters into investigation themes. Only in phase four does the executor dispatch targeted prompts, one per vulnerability class across all 26: arithmetic, reentrancy, access control, MEV, precision loss, signature replay, oracle manipulation, proxy architecture, bridge security, and the rest.

Anti-Pattern is the pipeline I trust most, because none of it came from theory. The 120 patterns are derived from roughly 1,001 findings across 32 Immunefi competitions and about 132 bug bounty writeups between 2021 and 2025. A regex scanner scores each file per category and emits a confidence-weighted tag, then an LLM executor runs category-specific detection instructions against whatever matched. These are bugs that paid out.

Critical fans out one executor per high-priority area, and each one receives a rolling summary of everything found so far. Cross-area deduplication falls out of that for free, since the later executors stop re-reporting what the earlier ones already caught. Temporal heartbeats keep the long runs alive.

Solana needed its own track

EVM assumptions produce garbage on Solana. Different execution model, different bug classes.

The first problem was not analysis at all. It was detection. Solana repositories routinely misdetect as TypeScript, because the client SDK outweighs the .rs sources by file count. The scanner would then route the whole codebase through EVM context and flag reentrancy patterns in a runtime that does not work that way.

So Solana gets an explicit track. KB-Creation (Solana) runs an x-ray pre-audit methodology adapted for Anchor and forces the chain language. The Solana Security Scan reuses the Critical workflow with SOLANA_AREAS in place of the EVM set. Six focus areas, each injecting only its slice of a 139-rule corpus:

A seventh pass, novel, runs last, receives every prior finding, and injects no corpus at all. Its only job is to look for what the rule-driven areas structurally could not see.

The adversarial validator

Findings do not go straight to the dashboard. They go through validation, which runs on a separate worker pool with a separate provider and model, and whose prompt is written to refute rather than confirm.

The asymmetry is deliberate. A model asked “is this a real bug?” says yes far too often, because agreement is the low-energy path. A model asked to construct the argument that a finding is a false positive will go read the modifier chain and find the onlyOwner that the finding ignored.

Using a different model for validation than for discovery matters too. Same model in both roles, and you mostly get the discovery model agreeing with itself.

The result is that findings surviving validation are worth opening. The ones that do not, I never see.

What this cost to learn

Every pipeline front-loads regex and AST work before spending a token. It is faster, it is free, and it produces the structured context that makes the expensive part accurate.

Model routing follows the task rather than a preference. KB-Creation, SAST, and validation use a configurable provider. Antipattern and Critical are pinned to a specific model in code, because that is what worked. Validation runs a different provider entirely, by design.

Cost is a design constraint and not an afterthought. Scans run about $0.80 to $1 each, shown live in the dashboard. That number is what makes running five pipelines and an adversarial validator reasonable instead of absurd.

The scanner also does not find bugs for me. It reduces a 50,000-line protocol to fifteen places worth an afternoon. The judgment is still mine, applied to much less surface.