Benchmarks · public evidence
Results you can inspect.
Every number below is recomputed from one file you can download, hash and rerun.
- Cases completed
- 109 / 109
- Expected issues found
- 129 / 129
- Clean code left quiet
- 147 / 147
Evidence file SHA-256
7c913c45e94155f75da72240f62286c2c6c8a081b528873a40950a591ed5229aProject-authored regression tests. Not an independent accuracy score. What this covers
Regression results
109 cases across three complete manifests. Counts are recomputed from the downloadable output.
| Suite | Cases completed | Positive checks met | Negative controls clear | Unlabelled in suite | Other detectors | Errors / excluded |
|---|---|---|---|---|---|---|
| Dead code | 21 / 21 | 47 / 47 | 77 / 77 | 0 | 0 | 0 / 0 |
| Security | 68 / 68 | 40 / 40 | 35 / 35 | 1 | 53 | 0 / 0 |
| Code quality | 20 / 20 | 42 / 42 | 35 / 35 | 63 | 37 | 0 / 0 |
Scroll the table horizontally to see every result.
Positive checks ask for a declared symbol or rule to be reported. Negative controls ask for it to stay quiet. Ratios include completed cases only. A finding can satisfy more than one rule or symbol check. Unlabelled findings are in the suite's detector categories but outside its declared labels; they need review. Output from other detectors is counted separately. Neither count is automatically treated as false positives.
Every case, including failures
Each fixture link opens the exact public source commit. Complete native output and labels are in the evidence download.
Dead code 21 cases · 0 failed checks · 0 errors
| Fixture | Status | Checks met | Unlabelled in suite | Other detectors |
|---|---|---|---|---|
| basic-unused-symbolspython | Checks met | 8 / 8 | 0 | 0 |
| fastapi-route-usedpython | Checks met | 3 / 3 | 0 | 0 |
| dynamic-registry-usedpython | Checks met | 4 / 4 | 0 | 0 |
| local-registration-decorator-entrypointspython | Checks met | 7 / 7 | 0 | 0 |
| explicit-same-name-reexportspython | Checks met | 4 / 4 | 0 | 0 |
| annotation-metadata-import-scopepython | Checks met | 5 / 5 | 0 | 0 |
| sqlalchemy-mixed-modelspython | Checks met | 9 / 9 | 0 | 0 |
| flask-cli-blueprintpython | Checks met | 5 / 5 | 0 | 0 |
| package-service-layerspython | Checks met | 9 / 9 | 0 | 0 |
| django-management-commandpython | Checks met | 8 / 8 | 0 | 0 |
| celery-task-entrypointspython | Checks met | 5 / 5 | 0 | 0 |
| pytest-fixture-entrypointspython | Checks met | 3 / 3 | 0 | 0 |
| pydantic-validator-entrypointspython | Checks met | 10 / 10 | 0 | 0 |
| alembic-migration-entrypointspython | Checks met | 6 / 6 | 0 | 0 |
| importlib-plugin-loadingpython | Checks met | 8 / 8 | 0 | 0 |
| package-entrypointspython | Checks met | 2 / 2 | 0 | 0 |
| go-service-entrypointsgo | Checks met | 4 / 4 | 0 | 0 |
| java-service-entrypointsjava | Checks met | 6 / 6 | 0 | 0 |
| java-fxml-controller-callbacksjava | Checks met | 4 / 4 | 0 | 0 |
| typescript-vscode-pseudoterminal-callbackstypescript | Checks met | 9 / 9 | 0 | 0 |
| ts-js-package-reachabilitytypescript, javascript | Checks met | 5 / 5 | 0 | 0 |
Security 68 cases · 0 failed checks · 0 errors
Native reachability evidence: 2 abstentions and 0 rescued definitions. These are not additional passing checks.
Code quality 20 cases · 0 failed checks · 0 errors
Native reachability evidence: 1 abstentions and 0 rescued definitions. These are not additional passing checks.
| Fixture | Status | Checks met | Unlabelled in suite | Other detectors |
|---|---|---|---|---|
| complexity-hotspotpython | Checks met | 3 / 3 | 4 | 2 |
| long-functionpython | Checks met | 3 / 3 | 4 | 2 |
| argument-overloadpython | Checks met | 3 / 3 | 4 | 2 |
| inconsistent-returnpython | Checks met | 2 / 2 | 4 | 1 |
| empty-error-handlerpython | Checks met | 1 / 1 | 4 | 1 |
| type-annotation-gapspython | Checks met | 4 / 4 | 4 | 2 |
| type-annotation-coverage-onlypython | Checks met | 2 / 2 | 4 | 0 |
| framework-route-policypython | Checks met | 4 / 4 | 5 | 0 |
| opaque-identifierpython | Checks met | 5 / 5 | 4 | 3 |
| repo-policy-missing-typecheckerpython | Checks met | 5 / 5 | 0 | 1 |
| clean-modulepython | Checks met | 7 / 7 | 4 | 1 |
| typescript-async-loop-duplicate-conditiontypescript | Checks met | 5 / 5 | 2 | 4 |
| typescript-type-safety-unsafetypescript | Checks met | 3 / 3 | 2 | 4 |
| typescript-type-safety-cleantypescript | Checks met | 3 / 3 | 2 | 4 |
| typescript-quality-signals-unsafetypescript | Checks met | 10 / 10 | 2 | 4 |
| typescript-quality-signals-cleantypescript | Checks met | 7 / 7 | 2 | 3 |
| python-mutable-alias-unsafepython | Checks met | 1 / 1 | 4 | 1 |
| python-mutable-alias-cleanpython | Checks met | 1 / 1 | 4 | 1 |
| go-service-quality-hotspotgo | Checks met | 4 / 4 | 2 | 0 |
| java-service-quality-hotspotjava | Checks met | 4 / 4 | 2 | 1 |
What these results establish
A regression check, with its limits
The corpus contains small, project-authored examples, including patterns distilled from other projects. Skylos has been developed against these examples. They are not independently labelled production repositories or a blind holdout.
Security and quality expectations cover selected rule IDs and symbols, not every finding in the output. This page therefore shows expectation counts instead of presenting them as real-world precision or recall.
Recorded inputs and outputs
The bundle includes 144 tracked fixture and manifest files, original expectations, native scanner reports, analysis errors, effective scan options, dependency versions, and SHA-256 inventories. Every manifest case is accounted for.
These are static source scans with all advice included, without LLM review or dependency-CVE lookups. Source-only fixtures have no Git history: the published profile does not request a before-and-after security regression comparison. Timing is recorded, but it is not a performance score.
A checksum detects changed bytes. It does not prove that a label is correct, that a maintainer ran the scanner honestly, or that the study is independent. Inspecting the public source and rerunning the published procedure are separate checks.
Older studies and their limitations are described in the source benchmark notes. Historical percentages are not mixed into this run.
Verify the files. Rerun the scanner.
1. Verify this published bundle
Download all three files into the same directory. This checks the published digest, embedded input inventories, reproduction script and scoring arithmetic without running fixture code.
node verify.mjs regression-20261022.json --sha256 7c913c45e94155f75da72240f62286c2c6c8a081b528873a40950a591ed5229a --runner reproduce.pyThe command checks the SHA-256 shown at the top. The site build uses the same pinned digest and fails if the evidence is missing, malformed or altered.
2. Produce a fresh run
Download the runner and inspect it first. Use the recorded Python and dependency versions for closer reproduction. A new run will have different timestamps and temporary paths; compare case outcomes, not whole-file checksums.
Reproduction runnergit clone https://github.com/duriantaco/skylos.git
cd skylos
git checkout 0e4055c70e3d8f1b0409227e4f16f6c514b3b5a4
python -m venv .venv
. .venv/bin/activate
python -m pip install -e .
python /path/to/reproduce.py --source . --output /tmp/skylos-replay.jsonThe source commit is pinned, the checkout must be clean, and each case runs in a fresh scanner process. Fixture programs are not executed. No API keys or paid model calls are required.
Environment and scan limitations
- Python
- 3.14.4 (main, Apr 7 2026, 13:13:20) [Clang 15.0.0 (clang-1500.1.0.2.5)]
- Platform
- macOS-13.7.3-arm64-arm-64bit-Mach-O
- Development regression fixtures; no independent holdout or production accuracy claim
- Security and quality labels are rule-level presence/absence expectations, not exhaustive finding labels
- Unmatched findings remain in nativeReport and are not automatically classified as false positives
- Timing is informational, one run on one machine; it is not included in correctness totals
- No security or quality benchmark covers every Skylos rule, secret detector or SCA provider
- Hash checks detect artifact changes; they do not establish independent label correctness
- Profile differs from scripts/security_benchmark.py: changed_files is omitted because isolated fixtures have no Git comparison history; its original profile produced security_regression_unavailable ConfigError in a diagnostic run
- all_advice=True retains raw advice checks; this report does not measure the terminal default noise level
- The Python network audit hook does not sandbox native subprocesses; Go proxy, checksum service and toolchain downloads are disabled explicitly
- Reproduction compares expected-case outcomes, not identical report bytes; temporary paths, timings and randomized Python hashing can vary