Benchmark · Skylos 5.3.0 ·
Skylos vs Vulture, Ruff, Pylint, Bandit and Semgrep
Highest overall score in dead code and security, on 43 open-source Python projects and 14,834 findings checked against the code.
Dead code
Overall score (F1)- Skylos0.45
- Pylint0.38
- Ruff0.37
- Vulture0.26
- deadcode0.26
- CytoScnPy0.24
Higher score than Pylint on 30 and than Ruff on 31 of 43 projects.
Security
Overall score (F1)- Skylos0.31
- Semgrep CE0.14
- Bandit0.07
107 real issues found. Semgrep CE 30, Bandit 40.
What we scanned · 43 projects · 745,317 lines of Python · 26 blind, never scanned by Skylos before
- labmlai/annotated_deep_learning_paper_implementations
- django/asgiref
- wbond/asn1crypto
- MagicStack/asyncpg
- authlib/authlib
- OpenBMB/ChatDev
- CloakHQ/CloakBrowser
- docker/docker-py
- fishaudio/fish-speech
- googleapis/google-api-python-client
- Pythagora-io/gpt-pilot
- microsoft/graphrag
- hankcs/HanLP
- p-e-w/heretic
- httplib2/httplib2
- khoj-ai/khoj
- google/langextract
- soxoj/maigret
- ocrmypdf/OCRmyPDF
- PDFMathTranslate/PDFMathTranslate
- pre-commit/pre-commit
- rapidfuzz/RapidFuzz
- ScrapeGraphAI/Scrapegraph-ai
- searxng/searxng
- google-research/timesfm
- ipython/traitlets
- python-attrs/attrs
- pallets/click
- fastapi/full-stack-fastapi-template
- pallets/flask
- encode/httpx
- locustio/locust
- NanmiCoder/MediaCrawler
- miguelgrinberg/microblog
- FoundationAgents/OpenManus
- pydantic/pydantic
- danielgatis/rembg
- psf/requests
- Textualize/rich
- encode/starlette
- tqdm/tqdm
- ultralytics/yolov5
- soimort/you-get
Head to head
Dead code6 tools · 13,241 findings checked
- Right
- Kept on purpose
- Wrong
Skylos 5.3.0
3,036 findings
F10.45
40%47%Found8960.45Highest overall score.
Pylint 4.1.3
2,608 findings
F10.38
38%33%29%Found6680.38Same accuracy, but only checks unused imports, variables, arguments and private members. Skylos finds more.
Ruff 0.17.3
734 findings
F10.37
63%30%Found4490.37More accurate, but Skylos finds 2.0× more.
Vulture 2.16
5,124 findings
F10.26
17%38%45%Found8350.2617% right vs 40% for Skylos. Skylos finds more.
deadcode 2.4.1
4,948 findings
F10.26
18%36%47%Found8100.2618% right vs 40% for Skylos. Skylos finds more.
CytoScnPy 1.3.0
10,059 findings
F10.24
14%47%39%Found1,3010.24Finds more, but 39% of it is wrong (Skylos: 13%).
Security3 tools · 1,593 findings checked
- Real issue
- Noise
- Wrong
Skylos 5.3.0
562 findings
F10.31
19%52%29%Found1070.31Highest overall score.
Semgrep CE 1.183.0
287 findings
F10.14
44%45%Found300.1410% real vs 19% for Skylos. Skylos finds 3.6× more.
Bandit 1.9.4
1,031 findings
F10.07
80%16%Found400.074% real vs 19% for Skylos. Skylos finds 2.7× more.
76 of Skylos's 107 real issues are undeclared dependencies, a supply-chain check Bandit and Semgrep CE do not run.
Never seen vs seen before
Earlier rounds used some of these projects, so the never-seen ones are the fair test. Skylos scores highest on both.
Dead code · F1
Never seen · 26 projects
- Skylos0.40
- Pylint0.34
- Ruff0.29
- Vulture0.25
- deadcode0.25
- CytoScnPy0.23
Seen before · 17 projects
- Skylos0.67
- Pylint0.48
- Ruff0.51
- Vulture0.28
- deadcode0.27
- CytoScnPy0.28
Security · F1
Never seen · 25 projects
- Skylos0.30
- Semgrep CE0.11
- Bandit0.07
Seen before · 17 projects
- Skylos0.32
- Semgrep CE0.22
- Bandit0.07
Is it luck?
Two checks: how many projects Skylos wins, and how often it stays ahead when we re-draw the projects 4,000 times.
Dead code
vs Pylint
Projects won30 of 43 p 0.002
Re-draws won99%
vs Ruff
Projects won31 of 43 p < 0.001
Re-draws won98%
vs Vulture
Projects won32 of 43 p < 0.001
Re-draws won100%
vs deadcode
Projects won31 of 43 p < 0.001
Re-draws won100%
vs CytoScnPy
Projects won32 of 43 p < 0.001
Re-draws won100%
Each dot is one project where at least one tool reported something. Teal: Skylos scored higher there. Gray: the other tool did. Hollow: tied.
Security
vs Semgrep CE
Projects won13 of 42 p 0.17
Re-draws won99%
vs Bandit
Projects won17 of 42 p 0.005
Re-draws won100%
Each dot is one project where at least one tool reported something. Teal: Skylos scored higher there. Gray: the other tool did. Hollow: tied.
Every finding, repo by repo
Each one links to the exact line we scanned.
labmlai/annotated_deep_learning_paper_implementations @ 2be1800 · blind: first time Skylos scanned it
Loading findings…
Appendix: full statistics and methodConfidence intervals, significance tests, per-repository results, repositories and method.
A1. Precision, recall and F1 with 95% intervals
Up and to the right is better. Curves mark equal F1. Hover a tool for exact values.
Intervals resample whole repositories, so one large project cannot dominate.
- Skylos0.45
- Pylint0.38
- Ruff0.37
- Vulture0.26
- deadcode0.26
- CytoScnPy0.24
A2. Every finding, by verdict
Precision is the share of a tool's findings that were right. Recall is the share of all confirmed dead code found by any tool that this tool reported. F1 is their harmonic mean. "Rename only" findings are true, but the fix is renaming a leftover variable to _; they count as right, not as deletable dead code.
| Tool | Findings | Dead code | Rename only | Kept on purpose | Wrong | Precision [95% CI] | Recall [95% CI] | F1 [95% CI] |
|---|---|---|---|---|---|---|---|---|
| Skylos 5.3.0 | 3,036 | 896 | 317 | 1,407 | 397 | 40% [38–42%] | 52% [50–55%] | 0.45 [0.37–0.64] |
| Pylint 4.1.3 | 2,608 | 668 | 321 | 862 | 757 | 38% [36–40%] | 39% [37–41%] | 0.38 [0.28–0.49] |
| Ruff 0.17.3 | 734 | 449 | 12 | 53 | 220 | 63% [59–66%] | 26% [24–28%] | 0.37 [0.21–0.49] |
| Vulture 2.16 | 5,124 | 835 | 54 | 1,936 | 2,278 | 17% [16–18%] | 49% [46–51%] | 0.26 [0.21–0.32] |
| deadcode 2.4.1 | 4,948 | 810 | 54 | 1,756 | 2,307 | 18% [17–19%] | 47% [45–50%] | 0.26 [0.21–0.32] |
| CytoScnPy 1.3.0 | 10,059 | 1,301 | 125 | 4,715 | 3,888 | 14% [14–15%] | 76% [74–78%] | 0.24 [0.17–0.31] |
A3. Significance tests
Each comparison pairs Skylos with one tool on the same findings and the same ground truth. F1 intervals and p-values come from 4,000 bootstrap resamples of whole repositories, which accounts for findings in one project being related.
Right of zero favours Skylos. A teal interval clears zero; gray means the difference is not established. p-values are Holm-adjusted for the 5 comparisons.
Switch the metric to compare precision or recall instead. The table below has every test.
| Skylos vs | Precision difference · Fisher p | Effect size (Cohen's h) | Found only by Skylos / only by tool · McNemar p | F1 difference · bootstrap p | Repos where Skylos F1 is higher / lower / tied |
|---|---|---|---|---|---|
| Ruff | -22.6 pts · < 0.001 | -0.46 | 498 / 51 · < 0.001 | +0.085 · 0.057 | 31 / 3 / 9 · sign test < 0.001 |
| Pylint | +2.3 pts · 0.084 | 0.05 | 449 / 221 · < 0.001 | +0.070 · 0.057 | 30 / 10 / 3 · sign test 0.002 |
| Vulture | +22.8 pts · < 0.001 | 0.51 | 446 / 385 · 0.037 | +0.198 · < 0.001 | 32 / 7 / 4 · sign test < 0.001 |
| CytoScnPy | +26.0 pts · < 0.001 | 0.60 | 179 / 584 · < 0.001 | +0.215 · < 0.001 | 32 / 8 / 3 · sign test < 0.001 |
| deadcode | +22.7 pts · < 0.001 | 0.51 | 444 / 358 · 0.005 | +0.199 · < 0.001 | 31 / 8 / 4 · sign test < 0.001 |
p below 0.05 means the difference is very unlikely to be chance. Every p-value is Holm-adjusted: corrected for making several comparisons at once, so one lucky result cannot pass as significant.
A4. Repository by repository
F1 for each tool in each project.
| Repository | Skylos | Ruff | Pylint | Vulture | CytoScnPy | deadcode |
|---|---|---|---|---|---|---|
| blind | 0.28 | 0.21 | 0.68 | 0.16 | 0.33 | 0.16 |
| blind | 0.67 | – | 0.00 | 0.00 | 0.00 | 0.00 |
| blind | 0.44 | – | 0.24 | 0.13 | 0.03 | 0.07 |
| blind | 0.53 | 0.17 | 0.23 | 0.40 | 0.15 | 0.36 |
| blind | 0.65 | – | 0.00 | 0.22 | 0.02 | 0.23 |
| blind | 0.59 | 0.53 | 0.54 | 0.50 | 0.58 | 0.51 |
| blind | 0.60 | 0.47 | 0.61 | 0.20 | 0.41 | 0.25 |
| blind | 0.54 | 0.00 | 0.22 | 0.14 | 0.09 | 0.13 |
| blind | 0.83 | 0.70 | 0.63 | 0.42 | 0.52 | 0.43 |
| blind | 0.72 | 0.53 | 0.65 | 0.36 | 0.23 | 0.36 |
| blind | 0.33 | – | 0.19 | 0.36 | 0.17 | 0.37 |
| blind | 0.59 | – | 0.05 | 0.38 | 0.34 | 0.40 |
| blind | 0.33 | 0.35 | 0.34 | 0.22 | 0.25 | 0.21 |
| blind | 0.50 | – | 0.00 | 0.10 | 0.07 | 0.11 |
| blind | 0.76 | 0.91 | 0.28 | 0.00 | 0.11 | 0.00 |
| blind | 0.50 | – | 0.26 | 0.18 | 0.36 | 0.19 |
| blind | 0.42 | 0.11 | 0.11 | 0.71 | 0.26 | 0.70 |
| blind | 0.58 | 0.52 | 0.61 | 0.32 | 0.41 | 0.32 |
| blind | 0.27 | – | 0.05 | 0.25 | 0.21 | 0.29 |
| blind | 0.76 | 0.27 | 0.08 | 0.26 | 0.38 | 0.26 |
| blind | – | – | 0.08 | 0.00 | 0.03 | 0.00 |
| blind | 0.00 | – | 0.00 | 0.17 | 0.06 | 0.20 |
| blind | 0.73 | 0.48 | 0.44 | 0.28 | 0.46 | 0.26 |
| blind | 0.38 | 0.14 | 0.09 | 0.25 | 0.06 | 0.26 |
| blind | 0.32 | 0.48 | 0.65 | 0.17 | 0.44 | 0.17 |
| blind | 0.00 | – | 0.13 | 0.07 | 0.00 | 0.09 |
| 0.00 | – | 0.27 | 0.00 | 0.03 | 0.00 | |
| 0.74 | – | 0.07 | 0.48 | 0.07 | 0.47 | |
| 0.33 | 0.33 | – | 0.03 | 0.05 | 0.03 | |
| 0.67 | – | 0.13 | 0.19 | 0.08 | 0.25 | |
| 0.00 | – | 0.18 | 0.00 | 0.04 | 0.00 | |
| 0.32 | – | 0.29 | 0.19 | 0.11 | 0.19 | |
| 0.80 | 0.65 | 0.65 | 0.34 | 0.60 | 0.33 | |
| 0.67 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | |
| 0.42 | 0.04 | 0.08 | 0.64 | 0.41 | 0.66 | |
| 0.24 | – | 0.08 | 0.11 | 0.11 | 0.12 | |
| 0.75 | 0.73 | 0.30 | 0.33 | 0.08 | 0.33 | |
| 0.00 | 0.00 | 0.18 | 0.09 | 0.07 | 0.08 | |
| 0.27 | 0.09 | 0.00 | 0.15 | 0.10 | 0.14 | |
| 0.00 | – | 0.00 | 0.11 | 0.00 | 0.11 | |
| n/a | n/a | n/a | n/a | n/a | n/a | |
| 0.27 | – | 0.25 | 0.24 | 0.33 | 0.24 | |
| 0.75 | 0.71 | 0.69 | 0.51 | 0.50 | 0.51 |
Darker is higher; bold marks the best tool in each repository; a dash means the tool reported nothing there; n/a means nothing in that repository was confirmed, so recall is undefined. Select a repository to list its findings.
A5. The 43 repositories
Each one pinned to the commit we scanned. Blind repositories had never been run through Skylos before this benchmark.
A6. How we measured
Projects
The 26 blind repositories were picked by a fixed rule before any tool ran: 16 in two rounds down the most-starred Python repositories on GitHub, in star order, and 10 libraries down the most-downloaded PyPI packages, from rank 201. Every round skipped anything we had cloned, benchmarked or mentioned before, and kept repositories pushed in 2026 with packaging metadata at the root, 5,000 to 60,000 lines of Python and a test folder. The other 17 are reference repositories from our earlier benchmarks.
What counts
Shipped Python code only: not tests, docs or examples. Findings from all tools are pooled, so the same symbol on the same line is judged once, without knowing which tool reported it. Every verdict records its evidence: the line that uses the code, or why nothing does.
- Dead code: nothing uses it; a maintainer would accept removing it.
- Rename only: an unpacked or loop variable that is never read.
- Kept on purpose: unreferenced but deliberately kept, such as public API, compatibility aliases or signatures a framework requires.
- Wrong: the code is used. Deleting it would break something.
- Security: real issue (worth fixing in this project), noise (true but harmless here) or wrong (the claim is false).
Statistics
- Precision and recall: Wilson score 95% intervals.
- F1 and all differences: 4,000 bootstrap resamples of whole repositories, so related findings in one project are not treated as independent.
- Precision differences: Fisher's exact test, with Cohen's h as effect size.
- Recall differences: exact McNemar test on the same ground truth.
- Repository wins: two-sided sign test.
- All p-values are Holm-adjusted across the comparisons with Skylos in each test.
- Recall is measured against everything any tool found and the review confirmed, the usual pooled ground truth.
Tools and commands
- Skylos 5.3.0:
skylos . --json - Ruff 0.17.3:
ruff check --select F401,F841 . - Pylint 4.1.3:
pylint --disable=all --enable=unused-import,unused-variable,unused-argument,unused-private-member - Vulture 2.16:
vulture . - CytoScnPy 1.3.0:
cytoscnpy . --json - deadcode 2.4.1:
deadcode . - Bandit 1.9.4:
bandit -r . (B101 assert_used skipped, as most projects do) - Semgrep CE 1.183.0:
semgrep scan --config p/default (rules in the security category)
Each tool ran with its default settings on the same checkout, the latest release on the run date. Skylos counts only what it lists by default. Engine: Skylos 5.3.0, commit f4bff656.
Skylos was improved on some reference repositories in earlier rounds, so the blind set is the fairer test. Choose Blind repos at the top to see it on its own. Looking for the fixture regression checks? Regression evidence