The strongest current evidence does not say that 44% of AI-generated software ships with a vulnerability. It says something narrower and still useful:
Across Veracode's AI code security benchmark program, the average security pass rate is 56%. In that controlled test, 44% of generation tasks failed at least one covered security check.
Veracode's current 2026 GenAI Code Security Report was published on July 27, 2026. Its accompanying report webinar page describes the 56% average across more than 100 models tested since 2023. The report page also says the leading model in its Summer 2026 dataset passed 68% of security tasks, while six of the eleven tested models landed between 50% and 53%.
Those numbers justify verification. They do not justify turning benchmark samples into claims about code that reached production.
What Veracode actually tested
Veracode's Spring 2026 methodology update describes a stable harness built around:
- 80 function-completion tasks;
- Java, JavaScript, C#, and Python;
- SQL injection (CWE-89), cross-site scripting (CWE-80), log injection (CWE-117), and insecure cryptographic algorithms (CWE-327);
- five task instances for each language and CWE combination;
- prompts without security-specific guidance; and
- Veracode SAST checks on the generated functions.
This is a security benchmark for a defined set of tasks. It does not measure whole repositories, deployed applications, secrets, authorization design, cloud permissions, every Python framework, or every vulnerability class. Its failure rate also depends on the chosen prompts, model versions, static-analysis rules, and scoring method.
That scope matters. A generation that fails one covered check is a failed benchmark task. It is not evidence that a real team accepted, merged, deployed, or exposed the code.
How the current 44% result relates to Python's 38% result
The two figures come from different summaries:
| Result | Date and scope | Correct interpretation |
|---|---|---|
| 56% pass / 44% fail | Current 2026 aggregate reported with the July report | Average result across Veracode's benchmark program, not a Python-only rate |
| 62% pass / 38% fail for Python | Spring 2026 language breakdown | A Python snapshot from the published four-language harness, not a universal or production rate |
| 68% pass for the leading model | Summer 2026 dataset on the current report page | The best result among the eleven models in that dataset, still limited to its tested tasks |
The Spring update is useful because it publishes the language breakdown and test design. Its Python figure should be labeled as historical when discussing the newer July report. The public July summary does not publish a replacement Python-wide rate, so combining the current 44% aggregate with Spring's 38% Python result as if they came from one dataset would be misleading.
What the benchmark says about Python security
The tested CWE classes map to real Python review concerns, but the benchmark does not show that every model fails each class at the same rate.
SQL injection
Python web and data code can create SQL injection when untrusted input reaches a dynamically constructed query. A data-flow-aware static rule can identify supported paths from request input to database execution and distinguish parameter binding from string construction.
Static analysis still needs framework and library coverage. A clean result for one ORM or database adapter says little about a custom query layer the scanner does not understand.
Cross-site scripting and log injection
Both classes depend on context. A value can be safe in one output position and dangerous in another. Python scanners can trace supported sources into template, response, and logging sinks, then check whether the right escaping or sanitization occurs on the path.
This is where simple text matching is weak. The useful question is whether attacker-controlled data reaches a sensitive sink through the actual code path.
Insecure cryptographic algorithms
Calls to weak hash or cipher algorithms are often detectable from the syntax and API name. Python checks can flag constructs such as MD5 or SHA-1 in security-sensitive contexts and insecure cipher modes.
The limit is intent. A weak hash used for a non-security checksum is different from one used for passwords or signatures, so findings still need context and review.
Package hallucination is a separate measurement
The original USENIX Security 2025 paper, We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs, studied a different risk with a different denominator.
The researchers generated 576,000 Python and JavaScript code samples with 16 models. Their extraction process produced 2.23 million package recommendations, of which 440,445, or 19.7%, referred to packages the researchers classified as hallucinated. The reported average was 15.8% for Python package recommendations and 21.3% for JavaScript.
That does not mean 15.8% of AI-generated Python files import a hallucinated package. The percentage is hallucinated packages divided by recommended packages under the paper's prompts, models, and extraction heuristics. The paper also notes that only 7% of its outputs directly included pip install or npm install; other package names were extracted through additional model prompts and heuristics.
The practical control is straightforward: treat every newly suggested package as a separate review decision. Confirm it exists, verify that it is the intended project and publisher, add it through the repository's normal lockfile workflow, and check known vulnerabilities. A vulnerability database can identify known flaws in a recognized package; it cannot establish that a newly registered package is legitimate.
What static analysis can and cannot tell you
| Risk | A suitable check can detect | It cannot establish by itself |
|---|---|---|
| SQL injection | Supported input-to-query flows and unsafe query construction | Safety inside an unknown query builder or runtime-generated SQL |
| XSS and log injection | Supported flows into response, template, and log sinks | Correct escaping in every custom rendering or logging layer |
| Weak cryptography | Known weak algorithms, modes, and insecure randomness APIs | Whether every cryptographic design choice meets the application's threat model |
| New dependencies | Known vulnerabilities in resolved, supported dependency data | Package legitimacy, maintainer identity, or absence of unpublished malware |
| Removed security checks | A modeled route guard, validator, or security contract that disappeared in a diff | The complete business authorization policy for an arbitrary application |
| Dead code | Unreachable or unused code under the scanner's supported language and framework model | Whether apparently unused code is activated through an unknown runtime mechanism |
This is why a clean scan is evidence about the rules and code paths that ran. It is not a certificate that the code is secure.
AI review and static analysis cover different jobs
An AI reviewer can explain a complex change, notice inconsistencies, and suggest tests. Its output can vary with model, context, and prompt. A rule-based static check gives a repeatable result for the same code, configuration, and scanner version.
Use both where they help. Keep repeatable security checks in the required merge gate. Use model-assisted review to widen the investigation and help a human understand the diff. Then run tests for security behavior that neither layer can prove from source alone.
Avoid another unsupported shortcut: the cited research does not compare AI-generated and human-written code under the same tasks and scoring process. It supports checking generated code carefully. It does not establish a general multiplier for how much less secure it is than human code.
A practical Python merge gate
For a Python repository, a useful sequence is:
- Scan the changed code. Run deterministic rules on the diff and enough repository context to understand sources, sinks, routes, and imports.
- Check concrete security classes. Start with injection paths, unsafe cryptography, secrets, dangerous configuration, and changes to known authorization or validation contracts.
- Review new dependencies separately. Verify identity and provenance, pin through a lockfile, and run supported vulnerability checks.
- Run security-focused tests. Cover authorization boundaries, invalid inputs, output encoding, logging behavior, and failure paths.
- Keep a human decision at the boundary. Review what the tools covered, what they skipped, and whether the change preserves the application's security requirements.
With Skylos, the diff-focused and dependency checks can be run explicitly:
skylos . --diff-base origin/main --diff origin/main --danger --quality --format json
skylos . --sca --format json
The first command reports new security findings and removed-control regressions in the change against origin/main. The second adds supported dependency vulnerability checks. Package identity and business-level security behavior still require their own controls.
The evidence-based conclusion
Current research supports a measured policy: AI-generated Python should pass the same security checks as any other code before merge, and teams should assume that plausible output still needs verification.
Veracode's current 56% pass rate shows that functional-looking benchmark answers often fail its covered security checks. The Spring Python result shows the same concern in a dated Python slice. The USENIX study shows that suggested package names create a separate supply-chain review problem.
None of those studies measures how much vulnerable AI code reaches production, and none supplies a matched human baseline. The practical response does not need either claim. Run repeatable static checks, validate dependencies, test security behavior, and review the remaining uncertainty before merge.
If you want a local and CI gate for that workflow, Skylos Cloud combines static analysis, diff-aware review, dead-code analysis, and supported dependency checks. The AI code review security PR checklist provides the adjacent human-review steps.