How to Use an AI Code Reviewer Without Trusting It

Quick answer
Give the reviewer a narrow job: find specific defect classes, not “review this code”.
Require a file, a line, and a concrete failure scenario for every finding, so it can be checked rather than believed.
Triage by verification cost: run the suggested reproduction, and close findings that cannot be reproduced.
Keep approval with a human; an AI reviewer that can block or approve becomes a check people learn to game.
Measure precision, not volume. A reviewer that reports twenty findings of which two are real trains the team to ignore it.

What should an AI reviewer look for?

The classes it is actually good at: missing input validation on a new endpoint, a query built from string concatenation, a secret in a diff, a missing authorization check compared with a similar handler, an unhandled error path. These are pattern-shaped and verifiable.

Be explicit about what to ignore: formatting, naming preferences, and architectural taste. A reviewer with no scope produces a long list of opinions that costs more to triage than it saves.

# Review instruction

Review only the added or changed lines in this diff. Report findings in these classes:
1. Authorization: a handler that reads or writes another user's record without an ownership check.
2. Injection: a query or shell command built from unvalidated input.
3. Secrets: a credential, token, or connection string in the diff.
4. Error handling: a failure path that silently succeeds.

For each finding: file, line, the input that triggers it, and the consequence.
Do not report style, naming, or test coverage.

How do you keep the findings honest?

Require a trigger and a consequence for every finding. “This could be a problem” is not reviewable; “a request with another user’s id returns their record because the query filters only by id” is.

Then verify a sample. Run the reproduction, or read the two lines it cites. A reviewer whose findings mostly fail verification should be retuned or removed, not trusted less quietly.

Where does it sit in the pipeline?

Run it before a human is requested, alongside tests and linters, so the human review starts from a cleaner diff. Post findings as comments rather than as a failing status, unless the project has agreed that a class of finding blocks merge.

Keep it off the approval path. A bot that can approve transfers responsibility to a system that cannot be accountable, and a bot that can block creates pressure to weaken the rule.

How do you know whether it is working?

Track how many findings are confirmed, dismissed, and missed. A useful reviewer finds defects that would otherwise reach production and reports few false alarms. Both numbers matter; either one alone rewards the wrong behaviour.

Also track where it does not look. An AI reviewer reads the diff, so it cannot see the missing test, the absent migration, or the feature flag that was never added. Those gaps are for the human pass.

Where this bites vibecoders
The tempting setup is a bot that comments on every pull request and a team that resolves its findings by clicking. That is worse than no reviewer, because it creates a record of review that did not happen. Verify findings or turn it off.

Where AI coding assistants get this wrong

Asking for a general review and getting a list of style opinions.
Accepting findings without a trigger or a consequence.
Letting the bot approve or block merges.
Ignoring precision; a noisy reviewer is ignored entirely.
Assuming the reviewer sees what is missing, not only what changed.
Resolving comments without checking the resolution.

Checklist

Define the defect classes to report.
Exclude style and naming from its scope.
Require file, line, trigger, and consequence.
Verify a sample of findings every week.
Keep approval with a human.
Track confirmed findings and false positives.

FAQ

Can an AI reviewer replace a human reviewer?

No. It can filter a class of mechanical defects before a human reads, which makes the human pass cheaper. It cannot take responsibility, and it cannot see the absence of a change, which is where many serious defects live.

Why does it report so many false positives?

Usually because the instruction is broad and the model fills the space with plausible concerns. Narrow the scope, require evidence, and give it examples of findings you considered valid. Precision improves with instruction, not with volume.

Should findings be posted as comments or a summary?

Inline comments when the tool supports them, because the reviewer needs the location. A summary is easier to skim but easy to dismiss. Either way, link each finding to a file and line so verification is one click.

Does it help with agent-generated pull requests?

Yes, as a first filter on volume, provided a human still reviews the risky paths. The combination that works is automated checks, AI review for known defect classes, and human review routed by risk.