Most security scanner benchmarks test against synthetic code where the authors already know the answer. We wanted something harder: point VulnHunter at a real open-source project with a known vulnerability and give it absolutely nothing — no CVE ID, no file path, no hint about what to look for.
It found the vulnerability. It also found 98 other issues the project's maintainers hadn't patched yet.
The Blind Test
Target: pyLoad, a popular open-source download manager written in Python with Flask. 559 Python files, actively maintained.
CVE: CVE-2025-55156 — a SQL injection in the /json/add_package API endpoint, rated High (CVSS 7.8). The vulnerable code string-interpolates user-supplied URLs directly into a SQL IN clause.
Setup: We checked out the commit immediately before the fix (a123accec, parent of 134edcdf6e) and ran VulnHunter's static scanner with no configuration, no hints, and no LLM — just the rule engines.
The Vulnerable Code
Here's what VulnHunter found at file_database.py:271:
# CVE-2025-55156 — VulnHunter flagged this as VH101 (SQL injection)
def update_link_info(self, data):
self.c.executemany(
"UPDATE links SET name=?, size=?, status=? WHERE url=? AND status IN (1,2,3,14)",
data,
)
# The safe executemany above uses parameterized queries.
# But then the code does this:
statuses = "','".join(x[3] for x in data)
self.c.execute(f"SELECT id FROM links WHERE url IN ('{statuses}')")
The statuses variable is built by joining URL values from the data tuple directly into a SQL string. A URL containing a single quote breaks out of the IN clause and injects arbitrary SQL.
An attacker adding a package with a URL like ') UNION SELECT password FROM users-- would execute:
SELECT id FROM links WHERE url IN ('') UNION SELECT password FROM users--')
The fix (commit 134edcdf6e) switched to parameterized queries:
urls = [x[3] for x in data]
placeholders = ','.join('?' * len(urls))
self.c.execute(f"SELECT id FROM links WHERE url IN ({placeholders})", urls)
Full Results
VulnHunter reported 99 findings across 17 vulnerability categories:
| Severity | Count |
|---|---|
| Critical | 1 |
| High | 22 |
| Medium | 39 |
| Low | 37 |
The scanner identified the CVE as finding #10 with HIGH confidence, correctly classifying it as SQL injection (VH101). It also found a critical code injection in the captcha handler, missing authentication on the /render/ endpoint, open redirects in the login flow, path traversal in file upload APIs, SSRF in captcha resolution, and 12 hardcoded credentials.
Out of 99 findings, the scanning agent identified only 2 false positives (2% FP rate) — both were parameterized queries where the f-string generates placeholder counts, not values.
How the Detection Engine Works
VulnHunter runs three detection engines in parallel, then deduplicates overlapping findings.
AST Pattern Rules (VH100–VH126)
The first engine walks each Python function's AST looking for known-vulnerable patterns. 27 rules cover:
- Injection: SQL (VH101), command (VH100), template (VH103), eval (VH111)
- Crypto: weak hashes (VH112), weak ciphers (VH116), hardcoded IVs (VH124), JWT none algorithm (VH121)
- Auth/Config: CSRF disabled (VH122), CORS wildcard (VH123), insecure framework settings (VH125), missing auth heuristic (VH115)
- Data handling: path traversal (VH106/107), insecure cookies (VH114), mass assignment (VH117), timing attacks (VH118)
- Info disclosure: debug mode (VH110), stack traces in responses (VH126), insecure randomness (VH113), insecure permissions (VH119)
Each rule resolves import aliases through _qualified_name(), so import requests as r; r.get(url) is recognized the same as requests.get(url).
Example — JWT none algorithm (VH121):
import jwt
# VH121 flags all of these:
token = jwt.encode(payload, key, algorithm="none")
data = jwt.decode(token, key, algorithms=["none"])
data = jwt.decode(token, key, options={"verify_signature": False})
Interprocedural Taint Flow (VHNF100–VHNF116)
The second engine builds an intermediate representation of the program and tracks data flow from sources (HTTP request parameters, environment variables, file reads) through transformations to sinks. 17 sink types cover command execution, SQL, code eval, deserialization, path access, HTML response, redirects, LDAP, XPath, session fixation, header injection, log injection, and regex injection.
This engine catches vulnerabilities the AST rules miss — like a tainted value that flows through three function calls before reaching subprocess.Popen.
Secret Detection (VHSEC100–VHSEC103)
The third engine scans for hardcoded credentials: provider-specific API key patterns, private keys, generic high-entropy strings, and weak credential assignments like SECRET_KEY = "dev".
Supply-Chain Analysis via ca9
VulnHunter integrates ca9 for dependency security. ca9 parses requirements.txt, pyproject.toml, Pipfile.lock, and poetry.lock, then checks packages against malware feeds, the OSV database, and supply-chain policies. It flags malicious packages, dependency confusion risk, mutable sources, missing integrity hashes, and suspicious new releases.
Competitive Benchmark
We tested VulnHunter against Semgrep and Bandit on 63 planted vulnerabilities across 6 synthetic Flask repositories covering injection, crypto, auth, file handling, DoS, and logic categories.
| Scanner | Detected | Detection Rate |
|---|---|---|
| VulnHunter | 60/63 | 95.2% |
| Semgrep | 37/63 | 58.7% |
| Bandit | 32/63 | 50.8% |
VulnHunter's advantage comes from three areas:
- Import alias resolution — Semgrep and Bandit miss
import requests as r; r.get(url, verify=False)because they match on raw source names. VulnHunter resolves aliases through the module's import bindings. - Interprocedural flow — the native flow engine tracks taint across function boundaries, catching indirect injection paths that pattern matchers miss.
- Dual-engine deduplication — when both the AST engine and the flow engine detect the same sink, findings merge into a single "multi-engine" result with higher confidence.
The three VulnHunter misses were: broken access control requiring business logic understanding, taint through ORM model attributes (a known limitation of static taint analysis), and a timing attack variant where the comparison target isn't visibly a digest.
Multi-Model Architecture
VulnHunter supports LLM-assisted investigation for deeper analysis beyond static rules.
Default provider: Grok (grok-4.7) via xAI's API. Used for the investigation and verification pipeline in full scan mode.
Anthropic multi-model pipeline: When using --model anthropic, VulnHunter assigns different Claude models to different roles:
- Controller (Opus 4.6): orchestrates the investigation, decides which leads to pursue
- Worker (Opus 5.5): performs deep code analysis on individual findings
- Verifier (Mythos 5.1): independently verifies findings in a separate conversation context for cross-model validation
This multi-model approach means the verifier has no access to the investigator's reasoning, providing genuinely independent confirmation of each finding.
Quick mode (--mode quick) runs only the static engines with no API key required — this is what produced the blind CVE test results above.
What's New
The latest release adds 12 new detection capabilities:
6 new AST rules:
- VH121 — JWT none algorithm and unverified signatures (critical)
- VH122 — CSRF protection disabled,
@csrf_exempt(high) - VH123 — CORS wildcard misconfiguration, escalates when credentials are allowed (high)
- VH124 — Hardcoded IV/nonce in AES-CBC and AES-GCM (high)
- VH125 — Insecure Django/FastAPI configuration:
DEBUG=True,ALLOWED_HOSTS=["*"], weak session cookies (medium) - VH126 — Stack traces and exception details in HTTP responses (medium)
6 existing rule improvements:
- httpx
AsyncClient/Clientinstances recognized as SSRF and TLS bypass sinks random.choice()join pattern flagged for insecure token generation- Non-digest plaintext secret comparisons flagged as timing attacks
FileResponseadded as a path traversal sink for FastAPI and Django- Django
.extra(where=[...])added as a SQL injection sink - Weak and short secret detection for patterns like
SECRET_KEY = "dev"
Example — CORS wildcard with credentials (VH123):
from fastapi.middleware.cors import CORSMiddleware
# VH123 flags this as HIGH, escalated to CRITICAL with credentials
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True, # escalation: wildcard + credentials
)
Try It
# Static scan (free, no API key)
pip install vulnhunter
vulnhunter scan <your-repo> --mode quick
# Full LLM-assisted scan with Grok
export XAI_API_KEY=your-key
vulnhunter scan <your-repo>
# Anthropic multi-model pipeline
export ANTHROPIC_API_KEY=your-key
vulnhunter scan <your-repo> --model anthropic
Output formats include text, JSON, and SARIF for CI/CD integration.