What Is an Automated Security Review (and What Does It Miss)?

Quick answer
An automated security review examines changed code for vulnerability patterns and returns findings for a human to verify.
It is strongest on pattern-shaped defects: injection, missing authorization checks, secrets, unsafe deserialisation, and known-vulnerable dependencies.
It is weakest on absences: a missing rate limit, an unenforced tenant boundary, a feature flag left on, a migration that will lock a table.
It reads the diff, so it rarely understands behaviour across services or data flows outside the change.
Treat it as a filter that reduces review load, not as an approval authority.

How does an automated security review work?

There are two families. Static analysis matches code against rules or data-flow models for known vulnerability classes. Model-based review reads the diff with an instruction to look for specific classes and explains each finding in prose. Many setups combine both.

Both produce findings, not verdicts. A finding needs a trigger and a consequence before it means anything: which input reaches which sink, and what an attacker gains. Without that, it is a hypothesis.

What does it actually catch well?

Injection through string-built queries, missing authorization on a new route compared with sibling routes, secrets in a diff, unsafe deserialisation, weak crypto choices, and dependencies with known advisories. These are local patterns with a clear signal.

It is also good at consistency checks inside one change: a new endpoint that skips the middleware every other endpoint uses is exactly the kind of defect a diff-aware reviewer notices quickly.

What does it miss?

Anything that is not in the diff. A missing test, a missing migration, a missing tenant filter on an existing query, or a permission that was never added. Absence has no line to comment on, so it falls to the human reviewer.

It also misses cross-service behaviour: whether this write breaks an invariant in another system, whether the retry is idempotent against the real API, whether the queue consumer will process the message twice. Those require knowledge outside the change.

How should a team use it?

Run it before human review, keep its scope narrow and its output evidence-based, and triage findings by verification cost. Track precision. A tool that reports many false findings trains reviewers to dismiss it, which is worse than not running it.

Never let it be the only reviewer on a high-risk change. The value is that a human spends attention on design and absence because the mechanical classes were already filtered.

Where this bites vibecoders
A green security check creates confidence that is not warranted for an AI-built feature, because the dangerous part is usually what was never written: the authorization check, the limit, the audit log. Read the diff for what is missing, not only for what the tool flagged.

Where AI coding assistants get this wrong

Treating a clean security review as approval.
Ignoring findings because the tool has false positives.
Assuming it evaluates code outside the diff.
Letting it review only style-adjacent concerns.
Skipping triage and closing findings unread.
Using it on high-risk changes without a human reviewer.

Checklist

Define the classes it should report.
Require a trigger and a consequence per finding.
Triage by verification cost.
Track confirmed findings and false positives.
Keep a human on high-risk paths.
Review the diff for absences it cannot see.

FAQ

Is this the same as SAST?

SAST is static analysis with rules and data-flow models. An automated security review can include SAST and model-based reading. The distinction matters for expectations: rules are precise and narrow, model-based review is broader and less certain.

Can it replace a penetration test?

No. A pen test exercises the running system and finds logic and configuration issues that a diff review cannot see. An automated review is a change-level filter, not an assessment of the deployed application.

How do I measure whether it is useful?

Count confirmed findings that would otherwise have shipped, and count false positives that consumed time. A useful tool improves the first number without making the second one painful. Review the ratio quarterly.

Should findings block a merge?

Only for classes the team has agreed are blocking and only after verification. An unverified finding that blocks a merge teaches people to bypass the check. Route high-confidence, high-severity findings to a required reviewer instead.