How to Run a Security Review on Every Pull Request

Quick answer
Start with the classes you will act on: secrets, authorization, injection, and dependency risk.
Run the review on the diff, before a human is asked, and post findings as comments with file and line.
Require evidence per finding and verify before blocking anything.
Block only agreed classes at agreed severity; everything else is advisory.
Track precision and time-to-triage, or the check becomes noise people route around.

What is the minimum viable setup?

A job that runs on pull requests, reads the diff, and reports findings. Use the tool your platform already supports, or a model-based review with a narrow instruction, and keep secret scanning separate because it is exact and cheap.

Post inline comments where possible. A finding without a location costs the reviewer a search, and the search is where attention goes.

# .github/workflows/security-review.yml
name: security-review
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Secret scan (exact, blocking)
        run: |
          git diff origin/${{ github.base_ref }}...HEAD | grep -nE 'AKIA[0-9A-Z]{16}|BEGIN (RSA|OPENSSH) PRIVATE KEY' && exit 1 || true
      - name: Review findings (advisory)
        run: echo "Run the security review action here and post comments"

What should be blocking?

Exact checks: a detected secret, a new dependency with a known critical advisory, a policy violation such as a world-readable bucket. These have low false-positive rates and a clear remediation.

Model-based findings should be advisory until verified. A finding that blocks a merge before anyone confirms it creates pressure to weaken the gate, and the gate then protects nothing.

How do you keep it useful?

Review the review. Once a week, check how many findings were confirmed and how long triage took. If most findings are dismissed, narrow the instruction rather than adding suppressions.

Keep a short list of finding classes the team has agreed to fix. A check with a published scope is trusted; one that reports whatever the model noticed is ignored.

Where does the human fit?

On high-risk paths, always. The review filters mechanical classes; the human checks design, absence, and whether the change belongs. Use code owners for the paths where a missed defect is expensive.

Make the review output part of the pull request record so the human can see what was already checked. That saves time and makes responsibility explicit.

Where this bites vibecoders
The tempting setup is to turn on every rule at once and block on all of them. The first false positive then teaches everyone to bypass the gate. Start with exact checks that block, and let the model-based review advise.

Where AI coding assistants get this wrong

Blocking on unverified model findings.
Running the review after human approval.
Posting a summary with no file or line.
Letting the check comment on style instead of security classes.
Ignoring precision until the team stops reading findings.
Granting the workflow write permissions it does not need.

Checklist

Choose the classes to report.
Run on the diff, before human review.
Keep secret scanning exact and blocking.
Make model findings advisory until verified.
Post inline comments with file and line.
Review precision weekly.

FAQ

Should the security workflow have write access?

Only to comment on the pull request. Do not give it contents write, packages write, or organisation secrets it does not need. A review job is a high-value target precisely because it runs on every change.

What about pull requests from forks?

Treat them as untrusted: no secrets, no write token. Many review tools work read-only on the diff, which is sufficient. Never run a model-based review with credentials that a fork author could influence.

How long should it take?

Fast enough that nobody merges around it. A check that adds several minutes to every small change will be bypassed. Run it in parallel with tests and keep its scope to the diff.

Do I need this on internal tools?

Yes, if they touch real data or credentials. Internal tools often have weaker review because “only employees use them”, which is exactly the assumption that produces lateral movement after a phishing incident.