AI code review: what it catches, what it misses and how to set it up
AI code review is an automated review of a code change by a language model, usually run in CI when a pull request opens. It reliably catches violations of written rules, missing checks, common bugs and inconsistencies with nearby code, and it misses product intent, architecture tradeoffs and anything that depends on knowledge outside the repository. Set it up with a short rules file, a CI job with read-only permissions, severity levels and a person who still approves every merge.

What AI code review is
AI code review puts a language model in the position of a first reviewer. A developer opens a pull request. A job in the CI pipeline gives the model the diff, access to the surrounding code and the project’s written rules. The model posts comments on the lines it has concerns about. A human reviewer then reads the change with those comments already in place.
It sits beside two older tools and differs from both.
- A linter applies fixed patterns. It is exact and cheap, and it cannot judge anything that is not a pattern.
- Static analysis follows data and control flow by formal rules. It finds certain bug classes with high precision.
- A model reads code the way a person does. It can apply a rule written in plain language, such as “every new database table must have an access policy”, and it can notice that a function contradicts its own comment. It is also the only one of the three that can be wrong in unpredictable ways.
The three work together. Keep the linter and the tests, and add the model for what they cannot express.
What it catches
| Category | Examples |
|---|---|
| Written rule violations | A new endpoint without the required permission check, a migration that breaks the naming rule, a direct database call where the project requires a service layer |
| Missing checks | No handling for an empty result, no validation on user input, an unchecked error return |
| Common bugs | Off-by-one errors, wrong comparison, a variable shadowed by accident, a promise that is not awaited |
| Inconsistency | A function that does something different from its name or comment, a change in one place that was not made in its twin |
| Security basics | A secret committed to the repository, a query built by string concatenation, an access rule that is too wide |
| Test gaps | New behaviour with no test, a test that asserts nothing |
| Readability | Dead code, unclear names, duplicated blocks |
The first row is the most valuable. A model applies a written rule to every pull request with the same attention, at any hour. Human reviewers apply rules well when they are fresh and less well on the fifth review of the day.
What it misses
| Blind spot | Why |
|---|---|
| Product intent | The reviewer does not know what the feature is for unless someone wrote it down |
| Architecture tradeoffs | It can describe options, but it does not carry the team’s history or constraints |
| Cross-system effects | A change that breaks another service is invisible if that service is in another repository |
| Performance under load | It can spot an obvious slow query, not behaviour at production volume |
| Unwritten business rules | A discount rule that lives in a manager’s head cannot be checked |
| Its own errors | It sometimes reports a problem that does not exist, with full confidence |
Two consequences follow. A person must still approve every merge. And the quality of the review depends on how much of the team’s knowledge is written where the reviewer can read it.
How to set it up
1. Write the rules first
The reviewer is only as specific as its instructions. Start with a short file in the repository that states the rules a senior engineer would enforce. Ten to twenty rules are enough to begin. Good rules are concrete and checkable:
- Every new table has row-level access policies.
- Server functions that change data verify the caller’s role.
- No query is built by concatenating user input.
- Money is stored as integers in the smallest currency unit.
Rules written from real incidents are the most useful. If a pattern caused a production problem once, write it down so the reviewer checks for it on every change. Our post on Claude Code best practices covers how to structure this kind of project file.
2. Run it in CI on every pull request
There are two common ways to run a review with Claude, according to Anthropic’s Claude Code documentation.
- The Claude Code GitHub Action. The documentation describes it as a GitHub Action that runs Claude Code inside your repository’s workflows. With a prompt in the workflow file it runs automatically on events such as a pull request being opened or updated, and the documented review example posts findings as inline comments on the pull request. You control the prompt, the model and the triggers.
- The managed Code Review product. The same documentation describes it as automatic review on every pull request without writing a workflow.
Other providers and code hosts offer comparable options. The setup choices are the same in each case.
3. Limit what the job can do
A review job needs to read code and write comments. It does not need to push commits or change settings. Give the workflow the narrowest permissions that work, store the API key as a secret, and never place it in the repository. Anthropic’s documentation makes the same points: grant the workflow only the permissions it needs, and keep credentials in GitHub Secrets.
One detail for public repositories: the documentation notes that GitHub withholds secrets from runs triggered by pull requests from forks, so the review runs only on pull requests from branches in the same repository.
4. Control the noise
A reviewer that comments on everything gets ignored within two weeks. Three settings prevent that.
- Severity levels. Ask for each finding to be labelled, for example blocking, should fix, or note. Decide which level fails the check.
- Scope. Tell the reviewer what not to comment on: formatting the linter already handles, generated files, vendored code.
- A feedback loop. When the team dismisses a type of finding repeatedly, change the rules file so the reviewer stops raising it.
5. Watch the cost
Each review consumes model usage and CI minutes. Anthropic’s documentation lists the levers: a concise project rules file, a limit on the number of turns per run, workflow timeouts and concurrency controls. Skipping draft pull requests and trivial changes also helps. Track cost per pull request from the first week.
6. Keep people in charge
Set branch protection so that a human approval is required to merge. The model’s review is an input. It is not an approval.
Setup checklist
- Rules file in the repository, written from real incidents and conventions
- Review job triggered when a pull request is opened or updated
- Read-only code permissions and permission to comment, nothing more
- API key stored as a secret, with a spending limit at the provider
- Severity labels and a decision on which level blocks the merge
- Exclusions for generated and vendored files
- Human approval required by branch protection
- A named owner who tunes the rules each month
- Cost per pull request and share of accepted findings tracked
- Data terms of the model provider reviewed for private code
Review after the fact and guards before the fact
Pull request review is one layer. Two other layers catch different things.
A scheduled review. A daily job reviews everything that was merged in the last day, or one area of the codebase in depth. It finds problems that slipped through under time pressure and problems that only appear when several changes are read together. Shai runs a daily LLM code review in CI on the marketplace platform he builds and operates.
Guard hooks. A hook is a script that runs before a coding agent executes a command or writes a file, and it can block the action. A guard blocks known-bad patterns at the moment they are written, which is earlier and cheaper than catching them in review. On the same platform, guard hooks block two SQL patterns that caused real incidents. The mechanism is explained in Claude Code hooks.
| Layer | When it runs | Best at |
|---|---|---|
| Guard hook | Before the code is written or the command runs | Blocking a short list of known dangerous patterns |
| Linter and tests | On every commit | Exact, mechanical checks |
| AI review on the pull request | When the pull request opens or updates | Rules in plain language, missing checks, inconsistency |
| Human review | Before merge | Intent, design, tradeoffs |
| Scheduled AI review | Daily | What the earlier layers let through |
How to tell whether it works
Measure four things for a month.
- Accepted findings. The share of AI comments that led to a code change. A low share means the rules are too vague or too noisy.
- Escaped defects. Bugs found after merge that a written rule should have caught. Each one becomes a new rule.
- Time to first review. How long a pull request waits before any feedback. This usually drops, because the model responds in minutes.
- Cost per pull request. Model usage plus CI minutes.
Where AI review matters most
Teams that write a large share of their code with AI agents need review more, not less. Agents produce code quickly, and they repeat the same mistakes when no rule stops them. The typical problems, such as missing access rules and exposed keys, are described in vibe coding security risks. An automated reviewer with rules for exactly those problems is the practical control.
What to do next
Collect the last ten bugs that reached production and write one checkable rule for each. Put those rules in a file in the repository and run a review job against your open pull requests for two weeks. Count how many findings the team accepted.
If you want the rules, the CI integration, the noise control and the team training done for you, see our AI code review page. It is delivered as a 4 to 8 week program at a fixed price quoted after a 30-minute scoping call, and the first one-hour consultation is free.