To protect your codebase from AI coding agents, let the agent propose a patch and have CI check it against rules the repository owner controls. Give the agent the evidence it needs to fix a failure. Keep approval of changes to those rules with a maintainer.
Consider a shop that accepts refunds through day 30. We changed the cutoff from <= 30 to < 30 and moved a skip from one test to another. Pytest and Vitest each reported one pass and one skip, both before and after the change. A customer asking for a refund on the last eligible day would now be turned away. What should a reviewer look at when the dashboard looks exactly the same?
What should an agentic CI pipeline let an agent do?
In agentic CI, an AI agent can inspect a failed run, explain it and propose a repair. Suppose a reviewer asks the agent to fix the refund check. It can read the log and push a commit to its branch. A separate job checks that commit, and a maintainer owns the rule that decides whether it can merge. GitHub's agentic workflows start agents with permission to read the repository. The workflow author can authorize specific writes, such as opening a pull request, through a feature called Safe Outputs. GitHub's explanation of agentic CI
| When this happens | What the agent can do | What the CI owner controls |
|---|---|---|
| A build fails | Read the failed job and suggest which change caused it. | Which logs the agent can read and which credentials the job receives. |
| The agent has a fix | Push a commit to a working branch or propose a PR. | The protected branch, required checks and who may bypass them. |
| A patch changes a test or workflow | Name the affected test or setting and explain why it should change. | A comparison with the trusted base and the reviewer who can accept the change. |
| A fix is ready to release | Link the run that verifies the fix. | Deployment credentials and the production approval step. |
How do you protect the merge on GitHub?
- Run the PR with limited permissions. Start the checking job with
permissions: { contents: read }. Run proposed code on an isolated runner without deployment secrets. GitHub token permissions limit calls to GitHub. The runner determines which other files, credentials and services the code can reach. GitHub Actions secure-use guidance - Protect the definition of a passing build. Require a pull request and passing status checks on the default branch. Assign owners to the workflow files, their helper scripts and the
CODEOWNERSfile itself, then enable required code-owner review. A change to the script that runs the tests deserves the same attention as a change to the workflow that calls it. GitHub rulesets · GitHub code owners - Make the result specific enough to act on. Ask the checking job which existing test stopped running, what behavior it covered and which commit it compared. For the refund change, “day 30 is eligible” gives the payments owner a decision to make. “One test skipped” gives them a count they have seen before.
Check the condition on the required job, too. GitHub treats a job skipped by a job-level if as successful, even when branch protection requires it. Try a PR that changes a test or its configuration and confirm the checking job actually runs. GitHub's job condition behavior
Apply the same idea to the promises your service makes. For a billing service, keep a test that one workspace cannot read another workspace's invoice. For a release pipeline, route a change that grants the publishing job a new permission to the release owner. The agent can propose either edit. The invoice test or the permissions diff gives the reviewer evidence for that particular decision.
How can a broken patch pass both pytest and Vitest?
We wrote the same two tests in Python and TypeScript. On main, the test for day 30 runs and passes. The test for day 31 already has a skip. On the PR branch, we changed the cutoff, enabled the day-31 test and skipped day 30. Here is what happened when we ran each command.
| Run | On the base | On the PR | After restoring the day-30 test on the PR |
|---|---|---|---|
python -m pytest -q -rs | 1 passed, 1 skipped | 1 passed, 1 skipped | 1 failed, 1 passed |
vitest run | 1 passed, 1 skipped | 1 passed, 1 skipped | 1 failed, 1 passed |
Each runner correctly reported the tests selected for that run. In pytest, -rs also prints the skip reason. Our PR reused the reason, “pending policy review.” To spot the regression, a reviewer needed the identity of the test that lost coverage and the policy that made it matter. Pytest skip reporting · Vitest test filtering
Reproduce the Python example in a scratch Git repository
Start in a scratch directory with Python 3.12 and Git installed. The environment setting keeps compiled test files out of this run.
python3.12 -m venv refund-guide-venv
. refund-guide-venv/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python -m pip install "pytest==9.1.1" "skylos==5.7.0"
git init -b main refund-example
cd refund-example
mkdir tests
Create refunds.py.
def accepts_refund(age_days):
return age_days <= 30
Create tests/test_refunds.py.
import pytest
from refunds import accepts_refund
def test_day_30_is_eligible():
assert accepts_refund(30)
@pytest.mark.skip(reason="pending policy review")
def test_day_31_is_ineligible():
assert not accepts_refund(31)
Commit the two files, run the base suite and make a branch.
git add refunds.py tests/test_refunds.py
git commit -m "Cover the inclusive refund deadline"
python -m pytest -q -rs
git switch -c change-refund-cutoff
On that branch, change <= 30 to < 30. Move the skip decorator so it sits immediately above test_day_30_is_eligible. Run the commands below. Pytest exits 0 and Skylos exits 1. Remove the decorator and rerun pytest to see the failing assertion on day 30.
python -m pytest -q -rs
skylos done --base main
What should the required check tell the reviewer?
In each repository we ran skylos done --base main. On a PR runner, fetch the base branch and use its ref, for example skylos done --base origin/main. The command compares the change with its Git merge base. These are the finding lines from the two local runs with Skylos 5.7.0 on 20 November 2026.
tests/test_refunds.py:7 SKY-A111 test_day_30_is_eligible now skips or expects failure (skip)
refunds.test.ts:4 SKY-A111 day 30 is eligible now skips or expects failure (skip)
The new skip blocked both runs before Skylos ran the tests. We ran pytest and Vitest separately for the table. The finding establishes that an existing test now skips. The payments owner can confirm whether day 30 still belongs inside the window. If the policy changed, update the test to express the newly agreed deadline and have that owner review it. Skylos Done checks and limits
How do you roll this out on a real codebase?
Start with a PR whose expected behavior you know. Add the CI job and leave it optional in branch protection while you evaluate it. Pick a test around a limit your team cares about and check that the report names the right test. A boundary that has never had a test can still break. This particular check needs an existing test and a skip pattern it recognizes.
For the next ten PRs, have the CI owner record the finding, whether it changed the review decision and any time spent on a false alarm. Then keep the check if it consistently identifies changes worth holding, narrow it if one class of finding creates the noise, or remove it if the evidence never helps a reviewer. Stop the trial early if the setup or review cost already outweighs the benefit.
You can try Skylos Done locally. If the team wants a shared record, Skylos Cloud can put a pass/fail check on GitHub pull requests and keep scan history. Run the Done workflow · Skylos Cloud workspace governance
For controls to set before giving agents repository access, see the AI coding agent security checklist. For the GitHub Actions workflow, see the pull-request gate setup. For other ways a patch can weaken a test suite, see the testwashing examples.
Change history
20 November 2026. Ran the refund example in Python and TypeScript and checked the merge controls against GitHub's documentation.