Coding agents are good at producing changes that look done. The tests pass, the summary sounds confident, and the diff is too long to read line by line.
So we built a small Flask and Express app with the kinds of mistakes that show up in AI-written pull requests, and ran one free scan on it. Here's everything it found, with the real output.
The change that "passed"
The app hashes passwords with SHA-256, and a test checks that the hash is 64 characters long. Then came a typical agent-style change: switch the hash to MD5, watch that test fail, and delete the test. It's not something agents do often, but when it happens it's easy to miss in review.
pytest is happy:
$ python -m pytest -q
2 passed in 0.01s
skylos done isn't:
$ skylos done --agent claude-code
FAIL No tests deleted, skipped or weakened: 1 test change(s) weaken what the tests check
tests/test_auth.py:12 SKY-A110 test_hash_is_64_chars was deleted; no test with the same body exists now
FAIL Security controls checked for removals: 1 security control removal needs attention
app/auth.py:3 SKY-L021 Security control regression: Crypto downgraded from sha256 to md5
PASS No secrets added: No secrets added
PASS Every package and import is real: Every added import and package resolves
Verdict: FAIL: tests were deleted, skipped or weakened; security controls were removed. The tests were not run because of this
(Shortened: some passing checks are left out.)
It caught two things a passing test run hides: the deleted test, and the weaker hashing behind it.
Everything else, in one scan
Then we ran the full scan on the whole app:
$ skylos . -a --format concise
| What it found | Where | Rule |
|---|---|---|
| SQL injection: an f-string query built from user input | app/db.py, Python | SKY-D211 |
Command injection: req.query.host reaches exec() | web/server.ts, TypeScript | SKY-D212 |
| A route that changes data with no auth check | app/main.py, /signup | SKY-F102 |
| A hard-coded Stripe secret key | app/main.py | SKY-L014, SKY-S101 |
A dependency that doesn't exist on PyPI (flask-jwt-securex) | requirements.txt | SKY-D222 |
An import of a function that doesn't exist (sanitize_input) | app/main.py | SKY-L012 |
14 vulnerability advisory entries for old requests, flask and express versions | requirements.txt, web/package.json | SKY-SCA-… |
| A function nothing calls | app/db.py | SKY-U001 |
Here are the security lines as printed:
app/db.py:6 SKY-D211 Possible SQL injection: tainted or string-built query.
web/server.ts:7 SKY-D212 Command injection: request data reaches child_process.exec() command text. Use execFile() with an argument array.
requirements.txt:3 SKY-D222 Hallucinated PyPI dependency 'flask-jwt-securex'. Package does not exist in PyPI.
app/main.py:5 SKY-L012 Import from local module 'app.auth' requests 'sanitize_input', but that name is not defined or re-exported there.
app/main.py:20 SKY-F102 Mutating route 'signup' has no obvious auth, permission, security, or dependency guard.
app/main.py:9 SKY-L014 Hardcoded credential in 'STRIPE_SECRET'. Use environment variables or a secrets manager instead.
requirements.txt:2 SKY-SCA-GHSA-x84v-xcm2-53pg requests@2.19.0: Insufficiently Protected Credentials in Requests Upgrade to 2.20.0.
web/package.json:1 SKY-SCA-GHSA-rv95-896h-c2vc express@4.17.1: Express.js Open Redirect in malformed URLs
(Shortened: the other advisory entries, the dead-code lines and several project-setup suggestions are left out.)
Two of these are mistakes people rarely make by hand but agents sometimes do:
- A package that doesn't exist. Models sometimes invent plausible package names. If someone later registers that name on PyPI, your install pulls their code. This is called slopsquatting.
- An import of a function that doesn't exist. The code reads as if
sanitize_inputcleans the input, but there's no such function, so the "protection" was never there.
Run it on your own code
pip install skylos
skylos . -a # security, secrets, dependencies, quality and dead code
skylos done # before you accept an agent's "done"
skylos cicd init # writes a GitHub Actions gate for every pull request
All three run locally. No account, no upload, and it's free and open source (Apache-2.0).
What it doesn't do
- It doesn't replace review. The scan finds patterns; a person still decides whether a finding matters. Some lines, like the project-setup suggestions, are advice rather than bugs.
- Coverage varies by language. Python has the deepest coverage. TypeScript/JavaScript, Java, PHP, Rust and Dart are covered from a pip install, and Go runs through the GitHub Action (language support).
- Vulnerability results depend on the advisory databases at scan time, and exact versions in your manifests or lockfiles.