Skip to main content

Benchmark · Skylos 5.3.0 ·

Skylos vs Vulture, Ruff, Pylint, Bandit and Semgrep

Highest overall score in dead code and security, on 43 open-source Python projects and 14,834 findings checked against the code.

Dead code

Overall score (F1)
  • Skylos0.45
  • Pylint0.38
  • Ruff0.37
  • Vulture0.26
  • deadcode0.26
  • CytoScnPy0.24

Higher score than Pylint on 30 and than Ruff on 31 of 43 projects.

Security

Overall score (F1)
  • Skylos0.31
  • Semgrep CE0.14
  • Bandit0.07

107 real issues found. Semgrep CE 30, Bandit 40.

Head to head

Dead code6 tools · 13,241 findings checked

  • Right
  • Kept on purpose
  • Wrong
  • Skylos 5.3.0

    3,036 findings

    F10.45

    Found896

    Highest overall score.

  • Pylint 4.1.3

    2,608 findings

    F10.38

    Found668

    Same accuracy, but only checks unused imports, variables, arguments and private members. Skylos finds more.

  • Ruff 0.17.3

    734 findings

    F10.37

    Found449

    More accurate, but Skylos finds 2.0× more.

  • Vulture 2.16

    5,124 findings

    F10.26

    Found835

    17% right vs 40% for Skylos. Skylos finds more.

  • deadcode 2.4.1

    4,948 findings

    F10.26

    Found810

    18% right vs 40% for Skylos. Skylos finds more.

  • CytoScnPy 1.3.0

    10,059 findings

    F10.24

    Found1,301

    Finds more, but 39% of it is wrong (Skylos: 13%).

Security3 tools · 1,593 findings checked

  • Real issue
  • Noise
  • Wrong
  • Skylos 5.3.0

    562 findings

    F10.31

    Found107

    Highest overall score.

  • Semgrep CE 1.183.0

    287 findings

    F10.14

    Found30

    10% real vs 19% for Skylos. Skylos finds 3.6× more.

  • Bandit 1.9.4

    1,031 findings

    F10.07

    Found40

    4% real vs 19% for Skylos. Skylos finds 2.7× more.

76 of Skylos's 107 real issues are undeclared dependencies, a supply-chain check Bandit and Semgrep CE do not run.

Never seen vs seen before

Earlier rounds used some of these projects, so the never-seen ones are the fair test. Skylos scores highest on both.

Dead code · F1

Never seen · 26 projects

  • Skylos0.40
  • Pylint0.34
  • Ruff0.29
  • Vulture0.25
  • deadcode0.25
  • CytoScnPy0.23

Seen before · 17 projects

  • Skylos0.67
  • Pylint0.48
  • Ruff0.51
  • Vulture0.28
  • deadcode0.27
  • CytoScnPy0.28

Security · F1

Never seen · 25 projects

  • Skylos0.30
  • Semgrep CE0.11
  • Bandit0.07

Seen before · 17 projects

  • Skylos0.32
  • Semgrep CE0.22
  • Bandit0.07

Is it luck?

Two checks: how many projects Skylos wins, and how often it stays ahead when we re-draw the projects 4,000 times.

Dead code

  • vs Pylint

    Projects won30 of 43 p 0.002

    Re-draws won99%

  • vs Ruff

    Projects won31 of 43 p < 0.001

    Re-draws won98%

  • vs Vulture

    Projects won32 of 43 p < 0.001

    Re-draws won100%

  • vs deadcode

    Projects won31 of 43 p < 0.001

    Re-draws won100%

  • vs CytoScnPy

    Projects won32 of 43 p < 0.001

    Re-draws won100%

Each dot is one project where at least one tool reported something. Teal: Skylos scored higher there. Gray: the other tool did. Hollow: tied.

Security

  • vs Semgrep CE

    Projects won13 of 42 p 0.17

    Re-draws won99%

  • vs Bandit

    Projects won17 of 42 p 0.005

    Re-draws won100%

Each dot is one project where at least one tool reported something. Teal: Skylos scored higher there. Gray: the other tool did. Hollow: tied.

Every finding, repo by repo

Each one links to the exact line we scanned.

labmlai/annotated_deep_learning_paper_implementations @ 2be1800 · blind: first time Skylos scanned it

Loading findings…

Appendix: full statistics and methodConfidence intervals, significance tests, per-repository results, repositories and method.

A1. Precision, recall and F1 with 95% intervals

Precision against recall, with 95% intervals

Up and to the right is better. Curves mark equal F1. Hover a tool for exact values.

0%0%25%25%50%50%75%75%100%100%F1 0.2F1 0.4F1 0.6F1 0.8Recall: share of confirmed dead code foundPrecision: findings that were rightSkylosRuffPylintVultureCytoScnPydeadcode
F1 by tool, with 95% intervals

Intervals resample whole repositories, so one large project cannot dominate.

  • Skylos0.45
  • Pylint0.38
  • Ruff0.37
  • Vulture0.26
  • deadcode0.26
  • CytoScnPy0.24

A2. Every finding, by verdict

Precision is the share of a tool's findings that were right. Recall is the share of all confirmed dead code found by any tool that this tool reported. F1 is their harmonic mean. "Rename only" findings are true, but the fix is renaming a leftover variable to _; they count as right, not as deletable dead code.

Findings of every tool by verdict, with precision, recall and F1
ToolFindingsDead codeRename onlyKept on purposeWrongPrecision [95% CI]Recall [95% CI]F1 [95% CI]
Skylos 5.3.03,0368963171,40739740% [38–42%]52% [50–55%]0.45 [0.37–0.64]
Pylint 4.1.32,60866832186275738% [36–40%]39% [37–41%]0.38 [0.28–0.49]
Ruff 0.17.3734449125322063% [59–66%]26% [24–28%]0.37 [0.21–0.49]
Vulture 2.165,124835541,9362,27817% [16–18%]49% [46–51%]0.26 [0.21–0.32]
deadcode 2.4.14,948810541,7562,30718% [17–19%]47% [45–50%]0.26 [0.21–0.32]
CytoScnPy 1.3.010,0591,3011254,7153,88814% [14–15%]76% [74–78%]0.24 [0.17–0.31]

A3. Significance tests

Each comparison pairs Skylos with one tool on the same findings and the same ground truth. F1 intervals and p-values come from 4,000 bootstrap resamples of whole repositories, which accounts for findings in one project being related.

Skylos minus each tool, with 95% intervals

Right of zero favours Skylos. A teal interval clears zero; gray means the difference is not established. p-values are Holm-adjusted for the 5 comparisons.

-0.20.0+0.2p (Holm)vs RuffRuff: difference 0.085, 95% interval 0.001 to 0.287, p 0.0570.057vs PylintPylint: difference 0.070, 95% interval 0.005 to 0.247, p 0.0570.057vs VultureVulture: difference 0.198, 95% interval 0.133 to 0.353, p < 0.001< 0.001vs CytoScnPyCytoScnPy: difference 0.215, 95% interval 0.132 to 0.396, p < 0.001< 0.001vs deadcodedeadcode: difference 0.199, 95% interval 0.135 to 0.352, p < 0.001< 0.001

Switch the metric to compare precision or recall instead. The table below has every test.

Statistical tests comparing Skylos with each tool
Skylos vsPrecision difference · Fisher pEffect size (Cohen's h)Found only by Skylos / only by tool · McNemar pF1 difference · bootstrap pRepos where Skylos F1 is higher / lower / tied
Ruff-22.6 pts · < 0.001-0.46498 / 51 · < 0.001+0.085 · 0.05731 / 3 / 9 · sign test < 0.001
Pylint+2.3 pts · 0.0840.05449 / 221 · < 0.001+0.070 · 0.05730 / 10 / 3 · sign test 0.002
Vulture+22.8 pts · < 0.0010.51446 / 385 · 0.037+0.198 · < 0.00132 / 7 / 4 · sign test < 0.001
CytoScnPy+26.0 pts · < 0.0010.60179 / 584 · < 0.001+0.215 · < 0.00132 / 8 / 3 · sign test < 0.001
deadcode+22.7 pts · < 0.0010.51444 / 358 · 0.005+0.199 · < 0.00131 / 8 / 4 · sign test < 0.001

p below 0.05 means the difference is very unlikely to be chance. Every p-value is Holm-adjusted: corrected for making several comparisons at once, so one lucky result cannot pass as significant.

A4. Repository by repository

F1 for each tool in each project.

f1 for each tool in each repository
RepositorySkylosRuffPylintVultureCytoScnPydeadcode
blind0.280.210.680.160.330.16
blind0.67–0.000.000.000.00
blind0.44–0.240.130.030.07
blind0.530.170.230.400.150.36
blind0.65–0.000.220.020.23
blind0.590.530.540.500.580.51
blind0.600.470.610.200.410.25
blind0.540.000.220.140.090.13
blind0.830.700.630.420.520.43
blind0.720.530.650.360.230.36
blind0.33–0.190.360.170.37
blind0.59–0.050.380.340.40
blind0.330.350.340.220.250.21
blind0.50–0.000.100.070.11
blind0.760.910.280.000.110.00
blind0.50–0.260.180.360.19
blind0.420.110.110.710.260.70
blind0.580.520.610.320.410.32
blind0.27–0.050.250.210.29
blind0.760.270.080.260.380.26
blind––0.080.000.030.00
blind0.00–0.000.170.060.20
blind0.730.480.440.280.460.26
blind0.380.140.090.250.060.26
blind0.320.480.650.170.440.17
blind0.00–0.130.070.000.09
0.00–0.270.000.030.00
0.74–0.070.480.070.47
0.330.33–0.030.050.03
0.67–0.130.190.080.25
0.00–0.180.000.040.00
0.32–0.290.190.110.19
0.800.650.650.340.600.33
0.670.000.000.000.000.00
0.420.040.080.640.410.66
0.24–0.080.110.110.12
0.750.730.300.330.080.33
0.000.000.180.090.070.08
0.270.090.000.150.100.14
0.00–0.000.110.000.11
n/an/an/an/an/an/a
0.27–0.250.240.330.24
0.750.710.690.510.500.51

Darker is higher; bold marks the best tool in each repository; a dash means the tool reported nothing there; n/a means nothing in that repository was confirmed, so recall is undefined. Select a repository to list its findings.

A5. The 43 repositories

Each one pinned to the commit we scanned. Blind repositories had never been run through Skylos before this benchmark.

Repositories in the benchmark
RepositorySetPython linesDead-code findingsConfirmed dead codeSecurity findingsReal issuesCommit
labmlai/annotated_deep_learning_paper_implementationsBlind39,5133866862132be1800
django/asgirefBlind1,9879410003dcd19
wbond/asn1cryptoBlind20,0311545280a767d2d
MagicStack/asyncpgBlind11,1351671127013b0191
authlib/authlibBlind23,26646611500df5fb3e
OpenBMB/ChatDevBlind29,056234102562464a681
CloakHQ/CloakBrowserBlind9,7725313500b860957
docker/docker-pyBlind14,2162227100e734fd5
fishaudio/fish-speechBlind10,116913736100f8a12e
googleapis/google-api-python-clientBlind13,25310925310c1bdb24
Pythagora-io/gpt-pilotBlind18,550479532009b763fd
microsoft/graphragBlind39,71440994209c600ff8
hankcs/HanLPBlind57,0163,7754451525281e8ef
p-e-w/hereticBlind7,075391191ecde93f
httplib2/httplib2Blind2,40565651dbbba89
khoj-ai/khojBlind34,76071810210340ae229ca
google/langextractBlind16,85073182503493510
soxoj/maigretBlind11,59813839352b8747b9
ocrmypdf/OCRmyPDFBlind25,40733832490b457e9e
PDFMathTranslate/PDFMathTranslateBlind6,26699133243bdb146
pre-commit/pre-commitBlind7,16814922808cb5845
rapidfuzz/RapidFuzzBlind8,8673712500932842
ScrapeGraphAI/Scrapegraph-aiBlind16,62390195514194055e
searxng/searxngBlind53,2981,153421174ffa88e3
google-research/timesfmBlind16,55813936541e51928e
ipython/traitletsBlind8,722132270a576f24
python-attrs/attrsReference7,48385280c8bb9bf
pallets/clickReference11,136156924204ef3a6
fastapi/full-stack-fastapi-templateReference1,73673110b1125d8
pallets/flaskReference9,4991445902ac8988
encode/httpxReference8,827108150b5addb6
locustio/locustReference12,64914992822054782
NanmiCoder/MediaCrawlerReference25,0764401675118f58399
miguelgrinberg/microblogReference1,8431381152a975ef6
FoundationAgents/OpenManusReference11,429112441673309bf4
pydantic/pydanticReference59,2564101922008d49ca
danielgatis/rembgReference4,7321305142cd8f366
psf/requestsReference5,634143240ef439eb
Textualize/richReference39,043311202009d8f9a3
encode/starletteReference8,4881801407b2948f
tqdm/tqdmReference4,48361011075bdb6c
ultralytics/yolov5Reference16,1371571813111af08ba1
soimort/you-getReference14,6446352221344049548f

A6. How we measured

Projects

The 26 blind repositories were picked by a fixed rule before any tool ran: 16 in two rounds down the most-starred Python repositories on GitHub, in star order, and 10 libraries down the most-downloaded PyPI packages, from rank 201. Every round skipped anything we had cloned, benchmarked or mentioned before, and kept repositories pushed in 2026 with packaging metadata at the root, 5,000 to 60,000 lines of Python and a test folder. The other 17 are reference repositories from our earlier benchmarks.

What counts

Shipped Python code only: not tests, docs or examples. Findings from all tools are pooled, so the same symbol on the same line is judged once, without knowing which tool reported it. Every verdict records its evidence: the line that uses the code, or why nothing does.

  • Dead code: nothing uses it; a maintainer would accept removing it.
  • Rename only: an unpacked or loop variable that is never read.
  • Kept on purpose: unreferenced but deliberately kept, such as public API, compatibility aliases or signatures a framework requires.
  • Wrong: the code is used. Deleting it would break something.
  • Security: real issue (worth fixing in this project), noise (true but harmless here) or wrong (the claim is false).

Statistics

  • Precision and recall: Wilson score 95% intervals.
  • F1 and all differences: 4,000 bootstrap resamples of whole repositories, so related findings in one project are not treated as independent.
  • Precision differences: Fisher's exact test, with Cohen's h as effect size.
  • Recall differences: exact McNemar test on the same ground truth.
  • Repository wins: two-sided sign test.
  • All p-values are Holm-adjusted across the comparisons with Skylos in each test.
  • Recall is measured against everything any tool found and the review confirmed, the usual pooled ground truth.

Tools and commands

  • Skylos 5.3.0: skylos . --json
  • Ruff 0.17.3: ruff check --select F401,F841 .
  • Pylint 4.1.3: pylint --disable=all --enable=unused-import,unused-variable,unused-argument,unused-private-member
  • Vulture 2.16: vulture .
  • CytoScnPy 1.3.0: cytoscnpy . --json
  • deadcode 2.4.1: deadcode .
  • Bandit 1.9.4: bandit -r . (B101 assert_used skipped, as most projects do)
  • Semgrep CE 1.183.0: semgrep scan --config p/default (rules in the security category)

Each tool ran with its default settings on the same checkout, the latest release on the run date. Skylos counts only what it lists by default. Engine: Skylos 5.3.0, commit f4bff656.

Skylos was improved on some reference repositories in earlier rounds, so the blind set is the fairer test. Choose Blind repos at the top to see it on its own. Looking for the fixture regression checks? Regression evidence