Before you ship a vibe-coded Python app: keep secrets out of code, never build SQL or shell commands from user input, never use user input as a template or a URL, keep TLS on and debug off, use secrets for tokens, install only real packages, and check auth on every route. A scanner can check most of that for you; the rest needs a test.

Most vibe coding checklists, such as Replit's, cover hosting, HTTPS and platform settings. This one covers the Python code the assistant writes. To keep it honest, we wrote a small Flask app with the patterns assistants commonly produce and scanned it.

The vibe-coded app

This is app/main.py, the only module in the test app:

import os
import random
import sqlite3
import string

import requests
from flask import Flask, render_template_string, request

app = Flask(__name__)
app.config["SECRET_KEY"] = "dev"
STRIPE_KEY = "sk_live_..."   # a realistic-looking test key in the real file


@app.route("/user")
def get_user():
    user_id = request.args.get("id")
    db = sqlite3.connect("app.db")
    return str(db.execute(f"SELECT * FROM users WHERE id = '{user_id}'").fetchall())


@app.route("/convert")
def convert():
    name = request.args.get("file")
    os.system(f"convert {name} out.pdf")
    return "ok"


@app.route("/hi")
def hi():
    template = request.args.get("t", "Hi {{ name }}")
    return render_template_string(template, name="User")


@app.route("/fetch")
def fetch():
    url = request.args.get("url")
    return requests.get(url, verify=False).text


def reset_token():
    return "".join(random.choices(string.ascii_letters, k=32))


if __name__ == "__main__":
    app.run(debug=True)

skylos . --danger --secrets --format concise printed:

app/main.py:40  SKY-U001  unused function: reset_token
app/main.py:11  SKY-U003  unused variable: STRIPE_KEY
app/main.py:24  SKY-D203  Use of os.system()
app/main.py:37  SKY-D210  requests call with verify=False
app/main.py:41  SKY-D250  Weak random source; use secrets or os.urandom for security-sensitive values
app/main.py:45  SKY-D346  Flask debug mode enabled
app/main.py:18  SKY-D211  Possible SQL injection: tainted or string-built query.
app/main.py:24  SKY-D212  Possible command injection (os.system): tainted input.
app/main.py:37  SKY-D216  Possible SSRF: tainted URL passed to HTTP client.
app/main.py:31  SKY-D349  Server-side template injection: untrusted input (request data `request.args.get`) is compiled as template source by render_template_string(). Pass it as a context variable instead.
app/main.py:11  SKY-S101  Potential stripe secret detected

Adding --quality also reported the requests.get() call without a timeout (SKY-L031). With --gate, the same scan exited 1.

Note what's not there: line 10, app.config["SECRET_KEY"] = "dev", and the fact that none of the routes checks who is calling. Those are the manual items below.

The checklist

#CheckHow to check itOur scan
1No API keys, tokens or passwords in codeSecret scanner; load from environment or a secret managerFlagged (SKY-S101)
2No SQL built from stringsData-flow scan; use parameters (db.execute(q, (user_id,)))Flagged (SKY-D211)
3No shell commands with user inputData-flow scan; use subprocess.run([...]) without shell=TrueFlagged (SKY-D212, D203)
4User input is never a templateData-flow scan; pass input as a variable to a fixed templateFlagged (SKY-D349)
5User-supplied URLs aren't fetched blindlyData-flow scan; allowlist hostsFlagged (SKY-D216)
6TLS verification on, timeouts setScan for verify=False; set timeout= on every requestFlagged (SKY-D210, L031)
7Debug mode off in productionScan for debug=True; read it from the environmentFlagged (SKY-D346)
8Tokens from secrets, not randomScan; use secrets.token_urlsafe(32)Flagged (SKY-D250)
9Every package exists and is declaredskylos . --select SKY-D222,SKY-D223 --strict; detailsNot in this file; tested separately
10Framework secret keys come from configManual: SECRET_KEY must not be a literal like "dev"Not flagged
11Every route checks who is askingManual: tests for unauthenticated, unauthorized and cross-user requestsNot flagged (can't be inferred)
12No auth check removed in a changeskylos . --diff-base origin/main --danger --quality (SKY-L021); detailsNot applicable to a new file

The fixes, in code

import os
import secrets
import sqlite3
import subprocess
from urllib.parse import urlparse

import requests
from flask import Flask, abort, render_template, request

app = Flask(__name__)
app.config["SECRET_KEY"] = os.environ["SECRET_KEY"]       # fails fast if missing
STRIPE_KEY = os.environ.get("STRIPE_KEY")

ALLOWED_HOSTS = {"api.example.com"}


@app.route("/user")
def get_user():
    user_id = request.args.get("id")
    with sqlite3.connect("app.db") as db:
        return str(db.execute("SELECT * FROM users WHERE id = ?", (user_id,)).fetchall())


@app.route("/convert")
def convert():
    name = request.args.get("file", "")
    subprocess.run(["convert", name, "out.pdf"], check=True)
    return "ok"


@app.route("/hi")
def hi():
    return render_template("hi.html", name=request.args.get("name", "User"))


@app.route("/fetch")
def fetch():
    url = request.args.get("url", "")
    if urlparse(url).hostname not in ALLOWED_HOSTS:
        abort(400)
    return requests.get(url, timeout=10).text


def reset_token():
    return secrets.token_urlsafe(32)


if __name__ == "__main__":
    app.run(debug=os.environ.get("FLASK_DEBUG") == "1")

Scanning the fixed file left one finding, and the gate still exited 1:

app/main.py:41  SKY-D216  Possible SSRF: tainted URL passed to HTTP client.

Skylos doesn't recognize a hand-written host allowlist as a guard. Once you've reviewed it, record the decision on the line, and the gate passed:

    return requests.get(url, timeout=10).text  # skylos: ignore[SKY-D216] host checked against ALLOWED_HOSTS

Two things the fixes don't do. The convert route still passes user input to a program as an argument, which the scan didn't flag; validate the file name against what your app expects. And none of this adds authentication; that is item 11.

Item 11: test access control, because no scanner can

A scanner doesn't know who should see invoice 42. Tests do. For every route that reads or changes user data, write the three tests an assistant usually skips:

def test_owner_can_read_invoice(client, user, user_invoice):
    r = client.get(f"/invoice/{user_invoice.id}", headers=user.auth)
    assert r.status_code == 200


def test_other_user_cannot_read_invoice(client, other_user, user_invoice):
    r = client.get(f"/invoice/{user_invoice.id}", headers=other_user.auth)
    assert r.status_code in (403, 404)


def test_anonymous_cannot_read_invoice(client, user_invoice):
    r = client.get(f"/invoice/{user_invoice.id}")
    assert r.status_code == 401

OWASP's Top 10 lists broken access control as A01:2025.

Two failure modes that are new with AI

Made-up packages. Assistants import and install packages that don't exist; attackers register the popular ones (slopsquatting). Check imports with skylos . --select SKY-D222,SKY-D223 --strict, and if an agent runs installs for you, skylos agent install-hooks denies installs of names that don't exist before they run.

Tests deleted to get a green run. An agent told to make the tests pass can delete or skip the failing one. skylos done --base main re-runs the tests and fails if tests were deleted, skipped or emptied of assertions (how).

Gate it

Run the scan on every pull request so the next AI change faces the same checks:

name: Security Gate
on: [pull_request]

permissions:
  contents: read

jobs:
  scan:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - run: python -I -m pip install "skylos==5.1.1"
      - name: Security gate
        run: skylos . --danger --secrets --gate

--gate uses the default policy: among other limits, it fails on any critical security finding, any secret, or more than five high-severity security findings, and the limits are configurable under [tool.skylos.gate]. It does not fail on AI-defect findings unless you set max_ai_defects = 0, so add the package check from item 9 as its own step if you want that to block. For a fuller workflow with annotations and review comments, run skylos cicd init --no-upload.

Suppressions sit in the diff next to the code, so a reviewer sees each one. skylos done also reports suppressions added in a change (SKY-A118, advice by default).

What this checklist doesn't cover

  • Hosting and platform settings: HTTPS, database row-level security, environment separation, backups. Use your platform's checklist.
  • Business logic: who may refund, cancel or export what.
  • Dependencies you already trust that later ship a vulnerability; run skylos . -a (includes dependency checks) or your SCA tool on a schedule.