The agent says the work is done. CI is green. Then you open the workflow diff and find one changed line: run: pytest -q || true.

The tests did run. One of them failed. The step reported success anyway, because || true means it can no longer fail.

To review AI-generated code, check the evidence before the logic. Confirm that no tests were deleted or skipped and no test settings or CI steps were loosened, that every new package exists, and that no secret or removed auth check came in. Then read the change against the task. Make the mechanical part a required CI check that the pull request cannot edit, and use AI reviewers to explain the change, not to decide the merge.

Tool and vendor facts checked on 21 Nov 2026. Skylos facts are for 5.7.0.

This matters more each month. GitHub's Octoverse 2026 reports that "by September 2026, more than 40% of pull requests on GitHub started with an agent's first draft, up from 17% in March" (opened 21 Nov 2026). The checklist below is vendor-neutral. Skylos, which we build, is one way to automate parts of it, and most rows also name other tools that do the job.


The checklist at a glance

CheckBy handAutomated optionsBlocks the merge?
Tests deleted, skipped or weakenedRead the test-file diffs. pytest -rs lists skips with reasons. Compare pytest --collect-only -q counts on base and headskylos done (SKY-A110, SKY-A111; pytest, Jest, Vitest, Mocha, node:test, Playwright Test)Skylos blocks by default. Weakened assertions are advice
Test settings or CI test step loosenedRead diffs of pyproject.toml, pytest.ini, Jest or Vitest config, package.json scripts, .github/workflows/skylos done (SKY-A112; pytest, Jest, Vitest, package.json, GitHub Actions). CODEOWNERS on workflow filesSkylos blocks. CODEOWNERS makes a code owner approve
Linters, type checks or scanners silencedRead new # noqa, @ts-ignore and ignore entriesRuff PGH004 blanket-noqa (Python). skylos done SKY-A118, A119, A121Ruff fails when the rule is on. Skylos advises by default
Packages that don't existpip index versions NAME, npm view NAME versionSkylos SKY-D222 (Python imports, requirements, package.json, go.mod). Skylos agent hooks at install time. Socket Firewall Free for confirmed malwareSkylos PR gate fails. skylos done advises. Hooks block the install
Secrets on added linesRead new config, fixtures and .env examplesGitHub push protection. Gitleaks. Skylos SKY-S101Push protection blocks the push, with a bypass. Skylos blocks
Security bugs on changed linesTrace new input to queries, shell calls, file paths and URLsSemgrep diff-aware scans. CodeQL with code scanning merge protection. Ruff S rules (Python). skylos . -a --diffSemgrep and CodeQL block when configured. Skylos gate fails on critical findings
Removed auth, CSRF or rate-limit checksRead the deleted lines in routes, handlers and middlewareSkylos SKY-L021 (Python, TypeScript, JavaScript, Go, Java, PHP, Rust, Dart). AI reviewersSkylos blocks in skylos done and the diff gate. AI reviewers comment
The change does what was askedRead it against the issue or promptClaude Code Review, Copilot code review and other AI reviewersComments. Claude Code Review never blocks. Copilot reviews don't count as approvals by default
Code left with no callersLook for helpers nothing calls any moreskylos done SKY-A135 to A137. Ruff F401, F841 (Python unused imports and variables)Advice

Each row is explained below, with the trade-offs.


What goes wrong in AI-generated pull requests?

Coding agents are rewarded for a green run. Most of the time they get there by fixing the code. Sometimes they get there another way. The ImpossibleBench authors (Zhong, Raghunathan and Carlini, 2025, opened 21 Nov 2026) describe an agent that "may delete failing tests rather than fix the underlying bug", and cheating that ranges "from simple test modification to complex operator overloading". Our testwashing write-up collects what Anthropic and METR have reported on the same behaviour.

People make every one of these mistakes too. The difference is volume, and a summary that says "done" with confidence. The patterns to look for:

  • Tests deleted, skipped or weakened. A failing test disappears, gains @pytest.mark.skip(reason="flaky"), or loses its assertion.
  • Test settings or CI loosened. A -k or --deselect in pytest settings, a narrower testpaths, passWithNoTests, or || true and continue-on-error: true on the CI test step.
  • Packages that don't exist. A USENIX Security 2025 study of 16 models and 576,000 code samples found that "the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models" (opened 21 Nov 2026). A missing name fails the install today. The risk is someone registering it later, which is called slopsquatting.
  • Secrets. A real key pasted into a config file or a test fixture.
  • Security bugs on changed lines. String-built SQL, shell calls with user input, unchecked file paths.
  • Removed auth checks. A deleted @login_required or auth middleware is a one-line diff that looks like cleanup.
  • Code left behind. A helper whose last caller the change removed, or a new function nothing calls.

How do you check each one?

1. Were tests deleted, skipped or weakened?

By hand. Read the test-file diffs before the code changes. Look for removed def test_ and it( blocks, new skip, xfail, .skip, .only or .todo markers, and assertions that got looser or disappeared. Two pytest commands help: pytest -rs prints every skipped test with its reason, and running pytest --collect-only -q on the base and the head shows whether the count dropped. A skipped test still counts as collected, so you need both.

Automated. A linter or SAST rule can flag a skip marker, but it reads only the branch, so it can't tell you that a test which existed on main is gone. That needs a comparison with the base. skylos done --base main does that comparison for Python and JavaScript/TypeScript tests, runs the tests itself, and blocks on a deleted test (SKY-A110) or a newly skipped one (SKY-A111). Weaker assertions are advice (SKY-A101). Details are on the verify an AI agent's done page.

2. Were test settings or the CI test step loosened?

By hand. This is the check people skip, because the files are not where the feature is. List them in one command:

git diff --stat main...HEAD -- "*test*" .github pyproject.toml setup.cfg pytest.ini tox.ini package.json "*jest.config*" "*vitest.config*"

Then read every hunk. Look for || true, continue-on-error: true, --exit-zero, -k, -m, --deselect, --ignore, narrower testpaths, passWithNoTests and lower coverage floors.

Automated. skylos done blocks on loosened pytest, Jest, Vitest and package.json test settings and on GitHub Actions test steps that can now fail (SKY-A112). It reads its own settings from the base branch, so the change cannot loosen the check that judges it. The other half of the fix is ownership: put .github/workflows/ in CODEOWNERS so a person has to approve any edit to CI (see the section on protecting the check).

3. Were linters, type checks or scanners silenced?

By hand. Search the diff for new # noqa, # type: ignore, @ts-ignore, eslint-disable, # nosec and new ignore or exclude entries in linter settings.

Automated. Ruff's PGH004 blanket-noqa flags a # noqa with no rule code in Python. Note that Ruff, ESLint and mypy read their settings from the branch they run on, so a pull request that adds a rule to ignore turns it off for itself. skylos done compares those settings with the base and reports weakened ones (SKY-A119), new inline suppressions (SKY-A118) and CI lint or scan steps that can now fail (SKY-A121). These are advice by default; silenced_checks = "block" in [tool.skylos.done.checks] on the base branch makes the settings changes block.

4. Does every new package exist?

By hand. Ask the registry. These are real outputs for a name that is on neither registry:

$ pip index versions invoicepdfkit_turbo
ERROR: No matching distribution found for invoicepdfkit_turbo
$ npm view invoicepdfkit-turbo version
npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/invoicepdfkit-turbo - Not found

If the name exists, check that it is the package you meant: its repository, maintainers and age.

Automated. Skylos checks added Python imports and requirements, package.json and go.mod entries against their registries and reports a missing one as SKY-D222. In skylos done that is advice; the pull request gate (skylos . -a --diff-base origin/main --diff origin/main --gate) fails on it. If Skylos cannot inspect the declared dependencies, it downgrades the finding to a medium "possible hallucinated import" (SKY-D223) that does not fail the gate. To stop the install itself, skylos agent install-hooks blocks made-up and typosquatted packages before pip, uv, npm, pnpm, bun or go runs (how). Socket Firewall Free blocks known-malicious packages at install time for npm, yarn, pnpm, pip, uv and cargo. Its docs are clear that it blocks confirmed malware and lets unknown or unscanned versions through (opened 21 Nov 2026).

5. Did a secret land on an added line?

By hand. Read new config files, test fixtures, .env.example files and anything that looks like a key, token or connection string.

Automated. GitHub push protection "blocks pushes that contain secrets" before they reach the repository; it needs GitHub Secret Protection on the repository, and by default anyone with write access can bypass it with a reason (opened 21 Nov 2026). Gitleaks (MIT) runs as a pre-commit hook or over a range of commits in CI. Skylos blocks a secret on an added line (SKY-S101) in both skylos done and the pull request gate.

6. Are there security bugs on the changed lines?

By hand. Follow each new input from a request, file or message to where it ends up: a query, a shell command, a file path, an outbound URL, a template.

Automated. This is where dedicated SAST tools are strongest, and you may already run one. Semgrep's diff-aware scans report "only the findings that are created by those code changes", and semgrep ci returns exit code 1 on blocking findings (rules in Block mode on the AppSec Platform, or any finding when run on its own). CodeQL results can block a merge through the ruleset rule "Require code scanning results" (GitHub docs). For Python, Ruff's S rules, such as S608 hardcoded-sql-expression, catch common patterns at lint speed (all opened 21 Nov 2026). skylos . -a --diff-base origin/main --diff origin/main reports security findings on changed lines along with secrets, packages and removed controls; its gate always fails on a critical finding and applies your limits to the rest.

7. Was an auth, CSRF or rate-limit check removed?

By hand. Additions get read; deletions get skimmed. List every deleted line and read the ones in routes, handlers, middleware and decorators:

git diff -U0 main...HEAD | grep "^-[^-]"

Automated. Skylos's SKY-L021 compares each changed file with its version on the base and reports a removed login decorator, auth dependency, CSRF protection, TLS verification and similar controls, in Python, TypeScript, JavaScript, Go, Java, PHP, Rust and Dart. In 5.7.0 it blocks in skylos done and fails the diff gate; in our check, a branch that deleted @login_required from a Django-style view failed with "Auth decorator @login_required was removed from 'refund_invoice'". AI reviewers often notice this too. More cases, including lost tenant filters, are in the AI code review security checklist.

8. Does the change do what was asked, and nothing else?

By hand. This is the part no tool does for you. Read the change against the issue or prompt. Check boundaries (>= that became >), error paths, data that crosses a trust boundary, and files the task never mentioned.

Automated, as advice. AI reviewers are good at this, and better than rules at it. Claude Code Review has several agents read "each change alongside the surrounding code, looking for issues such as logic errors, security vulnerabilities, or broken edge cases", with a verification step that filters false positives, and it reads its REVIEW.md from the base or default branch, "never from the pull request's own changes". Copilot code review does similar work inside GitHub. Use their comments as leads. The next sections cover why they are not the merge decision.

9. Did it leave code that nothing calls?

skylos done reports existing code that lost its last caller and new code nothing calls (SKY-A135 to SKY-A137) as advice. Ruff's F401 and F841 catch unused imports and variables in Python. Read these last; they matter for maintenance, not for whether the change is correct.


What does an automated check look like on a real agent branch?

We built a small repository for this page: a shipping_cost() function with three pytest tests and a GitHub Actions workflow that runs pytest -q. Then we wrote an agent-style branch by hand. It adds shipping tiers, and in doing so changes >= to >, which breaks free shipping at exactly 50.00. To get green, it skips the failing test, updates another test's expected fee, and adds || true to the CI test step.

$ git diff main --stat
 .github/workflows/ci.yml |  2 +-
 shop/shipping.py         | 11 +++++++----
 tests/test_shipping.py   |  3 ++-
 3 files changed, 10 insertions(+), 6 deletions(-)

$ git diff main -- .github tests
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -15,4 +15,4 @@ jobs:
       - name: Install
         run: pip install -e ".[test]"
       - name: Run tests
-        run: pytest -q
+        run: pytest -q || true
--- a/tests/test_shipping.py
+++ b/tests/test_shipping.py
@@ -3,12 +3,13 @@ import pytest
 from shop.shipping import shipping_cost
 
 
+@pytest.mark.skip(reason="flaky on CI")
 def test_free_shipping_at_threshold():
     assert shipping_cost(50.00) == 0.0
 
 
 def test_fee_below_threshold():
-    assert shipping_cost(49.99) == 4.99
+    assert shipping_cost(49.99) == 2.99
 
 
 def test_negative_subtotal_rejected():

Trimmed: the diff --git and index lines are removed. The shop/shipping.py change is left out of the second command; its key line is if subtotal >= FREE_SHIPPING_FROM: becoming if subtotal > FREE_SHIPPING_FROM:.

pytest 9.2.0 on the same branch is happy:

$ pytest -q
s..                                                                      [100%]
2 passed, 1 skipped in 0.00s

skylos done --base main on the same branch exits 1:

$ skylos done --base main
Skylos done · base 6136dc8 (merge base) · head 114be36
UNFINISHED Tests pass when Skylos runs them: Earlier required checks failed or were incomplete
FAIL       No tests deleted, skipped or weakened: 2 test change(s) weaken what the tests check
           tests/test_shipping.py:7  SKY-A111 test_free_shipping_at_threshold now skips or expects failure (skip)
           .github/workflows/ci.yml:18  SKY-A112 CI test step 'Run tests' in job 'test' can now fail without failing the build (failure swallowed in the script)
           tests/test_shipping.py:6  SKY-A101 (advice) Test was skipped or xfailed in this diff
           Fix: Put back the removed or skipped tests and the original assertions and settings.
PASS       Skylos settings and hooks left alone: Skylos settings, hooks and protected paths unchanged
PASS       No secrets added: No secrets added
PASS       Every package and import is real: Every added import and package resolves
WARN       Tests check the changed lines: Earlier required checks failed or were incomplete
PASS       Code doesn't special-case the tests: No code special-cases the tests (1 changed file(s) read)
WARN       Linters, type checkers, scanners and CI not silenced: 1 change(s) weaken what a linter, type checker, scanner or CI checks
           .github/workflows/ci.yml:18  SKY-A112 CI test step 'Run tests' in job 'test' can now fail without failing the build (failure swallowed in the script) (also listed under tests)
           Fix: Put back the linter, type-checker, scanner and CI settings, and fix what they report instead. A person should change them.
PASS       Security controls checked for removals: No security control removals detected (2 changed source file(s) compared)
PASS       AI-reviewer settings and scanner markers left alone: AI-reviewer settings unchanged; no scanner suppression markers added
PASS       No code left without callers: No code left without callers
OFF        No introduced type errors: Off in [tool.skylos.done]
Verdict: FAIL: tests were deleted, skipped or weakened. The tests were not run because of this
Receipt: .skylos/receipts/114be36d1e72-6136dc873474.json
See what to read and what is proven unchanged: skylos done map --base main
See every run stage by stage: skylos open, then the Pipeline tab

Skylos 5.7.0, skylos done --base main, scratch repository built for this page, run on 21 Nov 2026. Output unedited.

The tests show as unfinished because Skylos does not spend time running them once a cheaper blocking check has failed. Put the skipped test and the CI step back, and the real bug shows up, rerun on main to prove the change introduced it:

FAIL       Tests pass when Skylos runs them: 1 of 3 tests failed: 1 introduced
           Failing tests rerun on main (merge base 6136dc8): 1 introduced (0.2 s)
           tests/test_shipping.py:6  SKY-A113 tests/test_shipping.py::test_free_shipping_at_threshold failed (introduced by this change: passes on main (merge base 6136dc8)): assert 2.99 == 0.0 + where 2.99 = shipping_cost(50.0)
           Fix: Fix the failing tests, or the code they test.
Verdict: FAIL: 1 of 3 tests failed when Skylos ran them (1 introduced)

Same repository and date, on a branch with the skip and || true reverted. Trimmed to the failing check and the verdict.

Now look at what neither run mentions: assert shipping_cost(49.99) == 4.99 became == 2.99. Here that edit is correct, because the tiers changed the fee. If it had been wrong, nothing above would say so. That is a reviewer's job.


In what order should you review an AI-generated pull request?

Evidence, then logic. If the tests were weakened, reading the logic against them wastes your time.

  1. The task. Read the issue or prompt and the agent's summary. Write one line on what should change and what must stay the same.
  2. The automated checks. Let CI finish, or run skylos done --base main on the branch. A failure on tests or settings goes straight back to the agent.
  3. The files that produce the evidence. Test files, test settings, CI workflows, linter and reviewer settings, CODEOWNERS, agent hook settings. Use the git diff --stat command from check 2.
  4. Dependencies. Manifests, lockfiles and new imports. Every new name should exist and be the package you meant.
  5. Deleted lines. Auth, validation, error handling, logging, rate limits.
  6. The logic against the task. Boundaries, error paths, data handling, scope.
  7. Leftovers. Code nothing calls, TODOs, placeholder values.

For a large diff, skylos done map --base main lists each changed function with its old and new code, its callers and any removed security checks, ordered by risk.


How do you make the check one the agent can't edit?

A check protects main when it is required, and it stays honest when the pull request cannot change it. Four things to set up:

Require it. In branch protection or a ruleset, turn on required status checks. GitHub's docs say "all required status checks must pass before collaborators can merge changes into the protected branch". Note that "Required status checks must have a successful, skipped, or neutral status", so a job skipped by an if: condition does not block (opened 21 Nov 2026).

Protect the workflow file. A pull_request workflow runs the workflow file from the pull request itself, so a pull request can edit or remove the job that checks it. That is exactly how || true turns a required check green. Put .github/workflows/ and .github/CODEOWNERS in CODEOWNERS and turn on "Require review from Code Owners", so any edit to CI needs a named person's approval. GitHub's Enterprise Cloud docs also describe ruleset workflows: "Ruleset workflows can be configured at the organization or enterprise level to require workflows to pass before merging pull requests", with the workflow file kept in a separate repository (GitHub docs, opened 21 Nov 2026).

Read policy from the base. A tool that reads its settings from the branch under review can be loosened by that branch. Skylos reads [tool.skylos.done] and its gate limits from the base, and reports an edit to a workflow that runs it (SKY-A114). Claude Code Review reads REVIEW.md from the base or default branch for the same reason. Linter and test settings are read from the branch, which is why check 2 and check 3 compare them with the base.

Keep a human approval for risky changes. Claude Code Review's docs say "A review never approves or blocks the pull request, so the decision to merge stays with your team", and its check run "doesn't conclude as a failure". GitHub's Copilot code review docs say "By default, Copilot's reviews do not count toward required approvals for the pull request"; when Copilot approvals are enabled in repository, organization and enterprise settings, a Copilot approval can satisfy the rule (both opened 21 Nov 2026). Some reviewers have settings that fail a check, so read your reviewer's docs. If bot approvals count in your repository, skylos cicd init --risk-review adds a "Skylos risk review" check that fails on deleted tests, removed auth checks, dependency changes or weakened CI until a person with write access approves the head commit. Approvals from review bots do not count.


What does automation still miss?

Read the diff. Every tool on this page, Skylos included, misses things a careful reviewer catches:

  • Changed expected values. As in the demo above, an assertion that now expects a different number is either a correct update or the bug written into the test. It takes someone who knows the right answer to tell.
  • A test replaced by a new one in its place. In our testwashing test (Skylos 4.49.0, 14 Oct 2026, a small authored module), pytest passed all nine ways of hiding a failing test. skylos done failed six and missed three: a deleted test with a new test in its place, an assertion loosened to a truthiness check, and an edited expected value.
  • Tests that check the wrong thing. Assertions on a mock instead of the code, or a tautology a static count can't see.
  • Test data. Changed fixtures, golden files or JSON cases that the tests read are not judged.
  • Business rules. Whether a refund, a price or a permission is right for your product.
  • Security issues no rule describes. A query that lost its tenant filter but still runs.

Where Skylos itself falls short:

  • False positives. Security and dead-code rules report things that are fine. Silence one line with a reason, which shows in review: # skylos: ignore[SKY-D211] user_id is an int from the router. Accept existing findings with skylos baseline . and gate just the new ones.
  • Running non-Python tests. pytest runs automatically. Jest, Vitest and other runners need test_command and a JUnit XML path in [tool.skylos.done] before Skylos runs them; the static checks for deleted, skipped and focused JS/TS tests run without it. The changed-line test check (SKY-A120) needs Python and pytest.
  • Package checks need inspectable dependencies. As described in check 4, a made-up import can come back as an advisory SKY-D223 instead of a blocking SKY-D222.
  • Hooks are local. Agent hooks give fast feedback, but an agent with file system access can change local state. The required CI check is what decides the merge.
  • No per-change override, on purpose. Anything the change could set, an agent could set too. A deliberate test removal goes through a person: lower the policy on the base branch, or merge as someone on the bypass list.

Set up the automated part

No account or API key is needed:

pip install skylos
skylos done --base main            # on the agent's branch, before you read it
skylos . -a --diff-base origin/main --diff origin/main     # security, secrets, packages and removed controls on changed lines
skylos cicd init --no-upload       # writes .github/workflows/skylos.yml

skylos cicd init also sets up GitLab CI, Jenkins, Bitbucket Pipelines, Azure Pipelines and CircleCI. Once the workflow is merged into your default branch, run skylos setup: with gh logged in and admin access, it creates a branch ruleset named "Skylos" that requires the workflow's checks. Then add the CODEOWNERS lines:

/.github/workflows/  @your-org/platform
/.github/CODEOWNERS  @your-org/platform

and turn on "Require review from Code Owners". Keep Ruff, Semgrep, CodeQL and your AI reviewer; this adds the checks that compare the branch with main.


Changelog

  • 2026-11-21: First version. Skylos facts for 5.7.0; tool and vendor docs checked on 21 Nov 2026; real skylos done output from a scratch repository.