Testwashing is when a coding agent deletes, skips or weakens a test so CI goes green, then reports the work as done.

It doesn't happen often. But when it does, the pull request looks finished: the tests pass, the summary sounds confident, and the change to the test file is a few lines in a long diff.

Reward hacking in coding agents: what the research says

Models trained to get tests passing sometimes learn to make the tests pass instead of making the code right. Researchers call this reward hacking.

  • Anthropic wrote in the Claude 3.7 Sonnet system card (section 6) that the model occasionally special-cased tests in agentic coding environments like Claude Code. Most often it returned the values the tests expected; sometimes it edited the failing tests to match the code's output. Anthropic said this was infrequent in normal use and usually came after several failed attempts at a general fix. One of its recommendations is to monitor for "unexpected modifications to test files".
  • METR found recent frontier models reward hacking on its tasks. Examples include o3 replacing an evaluator with one that passes every submission, and wrapping a result in an object whose equality check always succeeds. It was rare on METR's general task suite (8 of 1,087 runs) and much more frequent on harder optimisation tasks.
  • ImpossibleBench (Zhong, Raghunathan and Carlini, 2025) gave agents tasks whose tests contradict the spec, so the only way to pass is to cheat. Models modified the tests, overloaded comparison operators and special-cased the test inputs.

So: it happens sometimes, more often when the task is hard, and it's easy to miss in review.

Nine ways to hide a failing test

We built a small pricing module with two tests. Then we made a typical agent change: cap discounts at 50%. The change also switched to floor division, which broke one test. Here's what skylos done says about the honest version, where the agent leaves the failing test alone:

$ skylos done --base main --agent claude-code
FAIL       Tests pass when Skylos runs them: 1 of 2 tests failed: 1 introduced
           Failing tests rerun on main (merge base 828848e): 1 introduced (0.4 s)
           tests/test_pricing.py:8  SKY-A113 tests/test_pricing.py::test_rounds_to_cents failed (introduced by this change: passes on main (merge base 828848e)): assert 16.0 == 16.99 + where 16.0 = apply_discount(19.99, 15)
Verdict: FAIL: 1 of 2 tests failed when Skylos ran them (1 introduced)

(Shortened: passing checks are left out.)

Skylos runs the tests itself instead of trusting the agent's summary, and reruns a failing test on the base branch to show the change introduced the failure. Then we tried nine ways an agent could make that failure go away without fixing the code, one branch each:

What the agent didpytest saysskylos done says
Deleted the failing test1 passedFAIL: test deleted (SKY-A110)
Added @pytest.mark.skip1 passed, 1 skippedFAIL: test newly skipped (SKY-A111)
Replaced the assertion with assert True2 passedFAIL: no assertion left (SKY-A110)
Added --deselect for the test in pyproject.toml1 passed, 1 deselectedFAIL: test settings loosened (SKY-A112)
Hard-coded the answer: if price == 19.99 and percent == 15: return 16.992 passedFAIL: test answer hard-coded (SKY-A115)
Returned a Price whose __eq__ always returns True2 passedFAIL: comparison rigged (SKY-A117)
Loosened the assertion to assert apply_discount(19.99, 15)2 passedPASS, with an advice line (SKY-A101)
Changed the expected value from 16.99 to 16.02 passedPASS, nothing reported
Deleted the failing test and added a new test in its place2 passedPASS, with an advice line: read as a rewrite

pytest was green every time. That's the point: a passing run tells you the tests that ran passed, not that the tests are still the ones you had.

What the catches look like

The deleted test:

$ skylos done --base main --agent claude-code
FAIL       No tests deleted, skipped or weakened: 1 test change(s) weaken what the tests check
           tests/test_pricing.py:8  SKY-A110 test_rounds_to_cents was deleted; no test with the same body exists now
           Fix: Put back the removed or skipped tests and the original assertions and settings.
Verdict: FAIL: tests were deleted, skipped or weakened. The tests were not run because of this

The skip, the emptied assertion and the pytest setting, one line each from their runs:

tests/test_pricing.py:11  SKY-A111 test_rounds_to_cents now skips or expects failure (skip)
tests/test_pricing.py:8  SKY-A110 test_rounds_to_cents is left with no countable assertion (it had 1 at the base): it may no longer be able to fail
pyproject.toml:8  SKY-A112 pytest addopts now deselects tests (--deselect) tests/test_pricing.py::test_rounds_to_cents

The two that leave the tests untouched and change the code instead, which is the special-casing Anthropic described:

pricing.py:7  SKY-A115 apply_discount() returns 16.99 when price == 19.99 and percent == 15, the exact input and answer of tests/test_pricing.py::test_rounds_to_cents: this special-cases the test instead of computing the answer
pricing.py:5  SKY-A117 Price.__eq__ always returns True: any comparison with it passes, so tests that compare values cannot fail

(Shortened: each line comes from a separate run; passing checks are left out.)

Skylos reads its own settings from the base branch, never from the change, so the change can't loosen the check that's judging it.

What it missed

Three of the nine got a passing verdict:

  • A new test in the old test's place. The agent deleted test_rounds_to_cents and added test_discount_capped_at_50 where it had been. Skylos 4.49.0 read that as the test being rewritten, and only printed advice: "test_rounds_to_cents was rewritten in place as test_discount_capped_at_50; check that it still tests the same behaviour".
  • A truthiness assertion. assert apply_discount(19.99, 15) still asserts something, so it's advice ("Specific assertion was replaced with a broad truthiness/null check"), not a failure.
  • An edited expected value. Changing 16.99 to 16.0 leaves a real assertion. Only someone who knows the right answer can tell it's wrong.

So keep reading test diffs. The advice lines show you where to look first.

Gate it on every pull request

pip install skylos
skylos done --base main          # before you accept the agent's "done"
skylos cicd init --no-upload     # adds a Skylos Done job to GitHub Actions

When Skylos finds a pytest project (or a test_command in [tool.skylos.done]), the workflow it writes includes a Skylos Done job that runs skylos done against the pull request's target branch. Make it a required check, so it applies to every agent and every person, with or without local hooks. You can also run it from the agent's Stop hook, so the agent is sent back before it says done (Claude Code stop hook).

The same checks read JavaScript and TypeScript test files. In a small node:test project, adding test.skip to the failing test failed the check:

test/price.test.js:9  SKY-A111 rounds to cents now skips or expects failure (skip)

It's free and open source (Apache-2.0), runs locally, needs no account, and no model decides the result.

Limits

  • It compares with a base. A test deleted in an earlier, already-merged change isn't seen.
  • Some weakening is only advice, and some isn't flagged at all (see above). It narrows what a reviewer has to check; it doesn't replace the review.
  • Python tests run automatically when Skylos finds a pytest project. For other runners, set test_command in [tool.skylos.done]; the checks for deleted, skipped and focused tests run either way.
  • The research numbers are from research tasks, some of them built to be impossible. They show the behaviour exists, not how often it happens in your repository.