Use case

ai agentstestingpythonci

AI Agent Says Tests Pass? Check for Deleted or Skipped Tests

Before merging an AI agent's change, check it didn't delete, skip or weaken tests to get a green run. How to check with skylos done, with real output.

Skylos team• Product engineering

October 9, 2026

10 min read

About this page

A green test run is only evidence if the tests that ran are the tests that existed before the change. This page shows how to check that mechanically, what the check blocks, what it only advises on, and what it misses.

3 notes

Checked against Skylos 4.47.1 (the current PyPI release) on 9 Oct 2026.
Every output block on this page is real output from `skylos done --base main` on a small demo repository with a pytest suite.
Defaults for what blocks and what advises were read from skylos/done/config.py and docs/done-gate.md at that release.

Run this workflow on your repo

The guide matters only if it maps to code you already own

Start with one concrete repo and one real workflow. If the results are useful, then wire the same pattern into CI.

Maintainer proof

Inspect 10 Skylos-assisted dead-code cleanup PRs that maintainers merged into the default branch in 9 open-source projects.

Install the CLI Compare tools first

Install with pip install skylos

Run skylos . -a before changing your workflow

Practical answer

Before merging an AI agent's change, check it didn't delete, skip or weaken tests to get a green run. How to check with skylos done, with real output.

An agent can get a green run by deleting, skipping or narrowing tests, by special-casing the test's inputs, or by turning off a lint or CI step. A diff review misses these easily.
`skylos done --base main` re-runs the tests itself and compares the tests and test settings with the base branch. Settings are read from the base, so the change can't loosen its own check.
By default it blocks on failing tests, deleted or skipped tests, loosened test settings, edits to Skylos's own settings, added secrets and code that special-cases the tests. Weakened assertions, untested changed lines, unknown imports and silenced linters are advice.
pytest runs automatically. Other test runners need a `test_command` line in config.

Step 1
Run it on the branch
Run `skylos done --base main` (or `origin/main`). Exit 0 means pass, 1 means a blocking check failed or did not finish, 2 means Skylos could not run.
Step 2
Read blocks and advice separately
Fix every FAIL. Treat WARN lines as review notes: they point at weakened assertions, untested lines and silenced checks.
Step 3
Make it a required check
Run `skylos cicd init --no-upload` to add a Skylos Done job to GitHub Actions, then make it a required status check so agents without local hooks are covered too.

Coding agents are rewarded for a green run. Most of the time they get there by fixing the code. Sometimes they get there another way:

a failing test is deleted, or gets @pytest.mark.skip;
pyproject.toml gains a -k or --deselect, or testpaths gets narrower;
the CI test step gets || true or continue-on-error: true;
an assertion disappears and the test can no longer fail;
the code learns the test's exact input and returns the expected answer;
a lint or type-check rule is switched off instead of fixed.

This isn't hypothetical. The EvilGenie benchmark documents explicit reward hacking by coding agents, including Codex and Claude Code. And each of the changes above is small enough to miss in a long diff: a deleted test is a missing hunk, a skip is one line, a -k lives in a config file the reviewer never opens.

You can check for them mechanically. This page uses skylos done, which is free and runs locally. The ideas apply whatever tool you use.

The command

pip install skylos

skylos done                     # uncommitted changes, compared with HEAD
skylos done --base main         # a branch or pull request, compared with its merge base
skylos done receipt             # print the latest receipt again

Exit codes: 0 pass, 1 a blocking check failed or didn't finish, 2 Skylos couldn't run (for example, the base branch isn't fetched).

It runs the tests itself instead of trusting the agent's word, compares the tests and test settings with the base, and writes a receipt to .skylos/receipts/. Its settings are read from [tool.skylos.done] at the base commit, never from the change, so an agent can't loosen the check in the same change it is being checked on.

What it looks like

A small repo has three tests for a refund() and a discount() function. An agent "simplifies" refund(), which breaks the case where the fee is larger than the amount. To get green, it deletes test_refund_never_negative and marks test_discount as flaky:

$ skylos done --base main
Skylos done · base 9e2c194 (merge base) · head f905230
UNFINISHED Tests pass when Skylos runs them: Earlier required checks failed or were incomplete
FAIL       No tests deleted, skipped or weakened: 2 test change(s) weaken what the tests check
           tests/test_billing.py:8  SKY-A110 test_refund_never_negative was deleted; no test with the same body exists now
           tests/test_billing.py:11  SKY-A111 test_discount now skips or expects failure (skip)
           tests/test_billing.py:10  SKY-A101 (advice) Test was skipped or xfailed in this diff
           Fix: Put back the removed or skipped tests and the original assertions and settings.
PASS       Skylos settings and hooks left alone: Skylos settings, hooks and protected paths unchanged
PASS       No secrets added: No secrets added
PASS       Every package and import is real: Every added import and package resolves
WARN       Tests check the changed lines: Earlier required checks failed or were incomplete
PASS       Code doesn't special-case the tests: No code special-cases the tests (1 changed file(s) read)
PASS       Linters, type checkers, scanners and CI not silenced: No linter, type-checker, scanner or CI settings weakened
Verdict: FAIL (Tests pass when Skylos runs them, No tests deleted, skipped or weakened)

The tests show as unfinished because Skylos doesn't spend time on the test run while a cheaper blocking check has already failed. Put the tests back, and the real bug shows up:

FAIL       Tests pass when Skylos runs them: 1 of 3 tests failed
           tests/test_billing.py:8  SKY-A113 tests/test_billing.py::test_refund_never_negative failed: assert -2 == 0 + where -2 = refund(3, 5)

Now suppose the agent "fixes" discount() by answering the test's exact input:

def discount(price, pct):
    if price == 200 and pct == 10:
        return 180
    return round(price * (1 - pct / 100), 2)

FAIL       Code doesn't special-case the tests: 1 place(s) in the code special-case the tests
           shop/billing.py:8  SKY-A115 discount() returns 180 when price == 200 and pct == 10, the exact input and answer of tests/test_billing.py::test_discount: this special-cases the test instead of computing the answer

An honest change passes, with advice where the tests are thin:

PASS       Tests pass when Skylos runs them: 3 tests run by Skylos, 3 passed (0 s)
PASS       No tests deleted, skipped or weakened: No tests deleted, skipped or loosened (3 compared)
WARN       Tests check the changed lines: 2 of 2 changed lines are not checked by any test
           shop/billing.py:8  SKY-A120 No test fails if shop/billing.py:8 changes `>` to `>=`. Add an assertion that would.
           shop/billing.py:9  SKY-A120 No test runs shop/billing.py:9 (in `discount`). Add a test that does.
Verdict: PASS

What blocks and what only advises

These are the defaults in 4.47.1. Each check can be set to block, advise, shadow or off in [tool.skylos.done.checks], committed to the base branch.

Check	Rules	Default	What it looks for
Tests pass when Skylos runs them	SKY-A113	blocks	Skylos runs the tests and reads the JUnit XML. A test that fails twice fails the check. Running out of time is "unfinished", never "pass".
No tests deleted, skipped or weakened	SKY-A110, A111, A112	blocks	A test that existed at the base is gone (moves and renames are matched), lost parametrize cases, has no countable assertion left, or gained skip/xfail. Test settings loosened: pytest `-k`/`-m`/`--deselect`/`--ignore`, narrower `testpaths`, `conftest.py` hooks that drop tests, lower coverage floors, CI test steps that may now fail. JS/TS: `.skip`, `.only`, Jest/Vitest settings that stop running existing test files.
Skylos settings and hooks left alone	SKY-A114	blocks	Edits to `[tool.skylos]`, `.skylos/`, the agent hook files, or a CI workflow that runs Skylos.
No secrets added	SKY-S101	blocks	Secrets on added lines.
Code doesn't special-case the tests	SKY-A115, A116, A117	blocks	Added code that returns a test's exact expected value for its exact input, checks whether a test runner is running, reads the tests' own files, or rigs `__eq__` to always pass. Weaker signals are advice.
Weakened assertions	SKY-A101	advice	Fewer assertions, a loosened `pytest.raises`, a new early `return`, renamed or rewritten tests. Only a test left with no countable assertion blocks.
Every package and import is real	SKY-D222, D225, D223	advice	Added imports and dependencies resolve to real, declared packages.
Tests check the changed lines	SKY-A120	advice	A changed line in non-test Python code that no test runs, or that no test notices when Skylos changes it on purpose. Python with pytest only.
Linters, type checkers, scanners and CI not silenced	SKY-A118, A119, A121	advice	New `# noqa` / `@ts-ignore`, rules turned off, paths excluded, `strict` off, CI lint or scan steps that may now fail. With `silenced_checks = "block"`, settings and CI changes block; inline suppressions stay advice.

A deleted test is reported as advice, not a block, when the change also removed the code it tested. That is a feature removal, not test tampering.

Test runners

pytest runs automatically when Skylos finds a pytest project (pytest.ini, a [tool.pytest.ini_options] table in pyproject.toml, a root conftest.py, or a tests/ or test/ directory) and pytest is installed in the environment Skylos runs in.
Anything else needs one setting, committed to the base branch:

[tool.skylos.done]
test_command = "npx vitest run --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml"
junit_xml = "reports/junit.xml"
test_budget_seconds = 300

Without a test_command, a project whose only tests are JavaScript or TypeScript gets an unfinished tests check that says so. The static checks for deleted, skipped and focused JS/TS tests still run. Exit code 0 alone isn't accepted as proof that tests ran, which is why a non-pytest command needs JUnit XML.

In CI

The quickest path is to let Skylos write the workflow:

skylos cicd init --no-upload

This writes .github/workflows/skylos.yml with a changed-line scan job and, when Skylos finds a pytest project or a test_command, a "Skylos Done" job. It needs no Skylos account or API key; --no-upload leaves out the Skylos Cloud upload job. Make the jobs required status checks on your default branch.

If you only want the done check, this is the job on its own. It passes Skylos's own GitHub Actions rules (pinned actions, read-only token, no persisted credentials):

name: Skylos Done
on: pull_request # not pull_request_target: this job runs the PR's code

permissions:
  contents: read

jobs:
  done:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0 # the merge base must be available
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - name: Install Skylos and the project's test dependencies
        run: |
          python -I -m pip install "skylos==4.47.1"
          python -I -m pip install -e . pytest
      - name: Check the change is finished
        run: skylos done --base "origin/$GITHUB_BASE_REF"

Use pull_request, never pull_request_target: the job runs the pull request's own code and tests.

In the agent loop

If you also install the agent hooks and commit a [tool.skylos.done] table first, the same checks run when the agent tries to stop. A failed check sends the agent back with up to ten reasons. After max_stop_blocks attempts on an unchanged checkout (default 3), the hook lets the agent stop with a warning, and the receipt stays failed. Escaping the loop never counts as a pass.

The hooks are local feedback. An agent with access to the file system can change local state, so the required CI job is the check that decides the merge.

What it doesn't catch

The checks are static and literal. Things that still get past:

Changed expected values. assert total == 400 becoming assert total == 200 is at most advice.
Tests that check the wrong thing: assertions on a mock instead of the code, or a tautology the counter can't see (r = 1; assert r == 1).
Edits to test data. Changed fixtures, golden files or JSON cases that the tests read are not judged.
Small or trivial special cases. In the demo above, the agent's refund() branch if amount == 3 and fee == 5: return 0 was not flagged, because returning 0 for a small input is something real code does all the time.
Answers recognised indirectly, such as a hash of the input or a check deep in the call stack.
Isolation. Skylos runs your tests as ordinary subprocesses. Enforcing file system, credential and network boundaries is the CI runner's job.

The full list is in the "What still gets past" sections of docs/done-gate.md. Read it before you rely on the check.

Other tools

Several open-source tools target the same problem, including gatekeep, tamperguard, tampercheck, dunnit, groundtruth and cngx. They differ in which runners, languages and tricks they cover. Run more than one on a branch where you know what the agent did, and keep the one whose output you trust.

Full reference for every check and setting: docs/done-gate.md on GitHub
Rule IDs SKY-A110 to A121: Skylos rules reference
Claude Code, Codex and Cursor hooks for security
Catch removed auth checks and security regressions in AI-generated PRs
A GitHub Actions gate for AI-generated pull requests
AI coding agent security checklist

Try it

pip install skylos
skylos done --base main

Running this across a team? These checks are free in the CLI. Skylos Cloud keeps the results in one place: the Free plan adds a pass/fail check on pull requests and scan history for one project, and the Workspace plan adds up to 10 projects, approvals for findings you decide not to fix, pull request comments, and audit exports.

Frequently asked questions

Does this work with Jest, Vitest or other JavaScript test runners?

Partly. The static checks for deleted, skipped and focused (.only) JavaScript and TypeScript tests, and for loosened Jest or Vitest settings, run without any setup. To have Skylos run the tests, set test_command in [tool.skylos.done] and have the runner write JUnit XML to junit_xml. The changed-line check is Python-only.

What if I meant to delete a test?

There is no per-change override, on purpose, because anything the change could set an agent could set too. Either the change also removes the code the test covered (then it is reported as a feature removal, which is advice), or a human lowers the policy at the base branch, or someone who can bypass the required check does so.

Does it need an account or send my code anywhere?

No account is needed and the checks run on your machine or CI runner. Checking whether added imports resolve to real packages can query package registries.

Is this the only tool that does this?

No. Several open-source tools target the same problem, including gatekeep, tamperguard, tampercheck, dunnit, groundtruth and cngx. Try more than one on your own repository.

Keep the workflow concrete, then pick the right guardrails

All use cases