GoldenrozBook a call

AI code review: what it catches, what it misses and how to set it up

AI code review is an automated review of a code change by a language model, usually run in CI when a pull request opens. It reliably catches violations of written rules, missing checks, common bugs and inconsistencies with nearby code, and it misses product intent, architecture tradeoffs and anything that depends on knowledge outside the repository. Set it up with a short rules file, a CI job with read-only permissions, severity levels and a person who still approves every merge.

A gold sphere connected by thin rods to six small navy spheres.

What AI code review is

AI code review puts a language model in the position of a first reviewer. A developer opens a pull request. A job in the CI pipeline gives the model the diff, access to the surrounding code and the project’s written rules. The model posts comments on the lines it has concerns about. A human reviewer then reads the change with those comments already in place.

It sits beside two older tools and differs from both.

  • A linter applies fixed patterns. It is exact and cheap, and it cannot judge anything that is not a pattern.
  • Static analysis follows data and control flow by formal rules. It finds certain bug classes with high precision.
  • A model reads code the way a person does. It can apply a rule written in plain language, such as “every new database table must have an access policy”, and it can notice that a function contradicts its own comment. It is also the only one of the three that can be wrong in unpredictable ways.

The three work together. Keep the linter and the tests, and add the model for what they cannot express.

What it catches

Category Examples
Written rule violations A new endpoint without the required permission check, a migration that breaks the naming rule, a direct database call where the project requires a service layer
Missing checks No handling for an empty result, no validation on user input, an unchecked error return
Common bugs Off-by-one errors, wrong comparison, a variable shadowed by accident, a promise that is not awaited
Inconsistency A function that does something different from its name or comment, a change in one place that was not made in its twin
Security basics A secret committed to the repository, a query built by string concatenation, an access rule that is too wide
Test gaps New behaviour with no test, a test that asserts nothing
Readability Dead code, unclear names, duplicated blocks

The first row is the most valuable. A model applies a written rule to every pull request with the same attention, at any hour. Human reviewers apply rules well when they are fresh and less well on the fifth review of the day.

What it misses

Blind spot Why
Product intent The reviewer does not know what the feature is for unless someone wrote it down
Architecture tradeoffs It can describe options, but it does not carry the team’s history or constraints
Cross-system effects A change that breaks another service is invisible if that service is in another repository
Performance under load It can spot an obvious slow query, not behaviour at production volume
Unwritten business rules A discount rule that lives in a manager’s head cannot be checked
Its own errors It sometimes reports a problem that does not exist, with full confidence

Two consequences follow. A person must still approve every merge. And the quality of the review depends on how much of the team’s knowledge is written where the reviewer can read it.

How to set it up

1. Write the rules first

The reviewer is only as specific as its instructions. Start with a short file in the repository that states the rules a senior engineer would enforce. Ten to twenty rules are enough to begin. Good rules are concrete and checkable:

  • Every new table has row-level access policies.
  • Server functions that change data verify the caller’s role.
  • No query is built by concatenating user input.
  • Money is stored as integers in the smallest currency unit.

Rules written from real incidents are the most useful. If a pattern caused a production problem once, write it down so the reviewer checks for it on every change. Our post on Claude Code best practices covers how to structure this kind of project file.

2. Run it in CI on every pull request

There are two common ways to run a review with Claude, according to Anthropic’s Claude Code documentation.

  • The Claude Code GitHub Action. The documentation describes it as a GitHub Action that runs Claude Code inside your repository’s workflows. With a prompt in the workflow file it runs automatically on events such as a pull request being opened or updated, and the documented review example posts findings as inline comments on the pull request. You control the prompt, the model and the triggers.
  • The managed Code Review product. The same documentation describes it as automatic review on every pull request without writing a workflow.

Other providers and code hosts offer comparable options. The setup choices are the same in each case.

3. Limit what the job can do

A review job needs to read code and write comments. It does not need to push commits or change settings. Give the workflow the narrowest permissions that work, store the API key as a secret, and never place it in the repository. Anthropic’s documentation makes the same points: grant the workflow only the permissions it needs, and keep credentials in GitHub Secrets.

One detail for public repositories: the documentation notes that GitHub withholds secrets from runs triggered by pull requests from forks, so the review runs only on pull requests from branches in the same repository.

4. Control the noise

A reviewer that comments on everything gets ignored within two weeks. Three settings prevent that.

  • Severity levels. Ask for each finding to be labelled, for example blocking, should fix, or note. Decide which level fails the check.
  • Scope. Tell the reviewer what not to comment on: formatting the linter already handles, generated files, vendored code.
  • A feedback loop. When the team dismisses a type of finding repeatedly, change the rules file so the reviewer stops raising it.

5. Watch the cost

Each review consumes model usage and CI minutes. Anthropic’s documentation lists the levers: a concise project rules file, a limit on the number of turns per run, workflow timeouts and concurrency controls. Skipping draft pull requests and trivial changes also helps. Track cost per pull request from the first week.

6. Keep people in charge

Set branch protection so that a human approval is required to merge. The model’s review is an input. It is not an approval.

Setup checklist

  • Rules file in the repository, written from real incidents and conventions
  • Review job triggered when a pull request is opened or updated
  • Read-only code permissions and permission to comment, nothing more
  • API key stored as a secret, with a spending limit at the provider
  • Severity labels and a decision on which level blocks the merge
  • Exclusions for generated and vendored files
  • Human approval required by branch protection
  • A named owner who tunes the rules each month
  • Cost per pull request and share of accepted findings tracked
  • Data terms of the model provider reviewed for private code

Review after the fact and guards before the fact

Pull request review is one layer. Two other layers catch different things.

A scheduled review. A daily job reviews everything that was merged in the last day, or one area of the codebase in depth. It finds problems that slipped through under time pressure and problems that only appear when several changes are read together. Shai runs a daily LLM code review in CI on the marketplace platform he builds and operates.

Guard hooks. A hook is a script that runs before a coding agent executes a command or writes a file, and it can block the action. A guard blocks known-bad patterns at the moment they are written, which is earlier and cheaper than catching them in review. On the same platform, guard hooks block two SQL patterns that caused real incidents. The mechanism is explained in Claude Code hooks.

Layer When it runs Best at
Guard hook Before the code is written or the command runs Blocking a short list of known dangerous patterns
Linter and tests On every commit Exact, mechanical checks
AI review on the pull request When the pull request opens or updates Rules in plain language, missing checks, inconsistency
Human review Before merge Intent, design, tradeoffs
Scheduled AI review Daily What the earlier layers let through

How to tell whether it works

Measure four things for a month.

  • Accepted findings. The share of AI comments that led to a code change. A low share means the rules are too vague or too noisy.
  • Escaped defects. Bugs found after merge that a written rule should have caught. Each one becomes a new rule.
  • Time to first review. How long a pull request waits before any feedback. This usually drops, because the model responds in minutes.
  • Cost per pull request. Model usage plus CI minutes.

Where AI review matters most

Teams that write a large share of their code with AI agents need review more, not less. Agents produce code quickly, and they repeat the same mistakes when no rule stops them. The typical problems, such as missing access rules and exposed keys, are described in vibe coding security risks. An automated reviewer with rules for exactly those problems is the practical control.

What to do next

Collect the last ten bugs that reached production and write one checkable rule for each. Put those rules in a file in the repository and run a review job against your open pull requests for two weeks. Count how many findings the team accepted.

If you want the rules, the CI integration, the noise control and the team training done for you, see our AI code review page. It is delivered as a 4 to 8 week program at a fixed price quoted after a 30-minute scoping call, and the first one-hour consultation is free.

Questions buyers ask

What is AI code review?

It is a review of a code change by a language model. The model reads the diff and the surrounding code, checks them against written project rules and posts comments on the pull request. A person still approves and merges.

Can AI code review replace human code review?

No. It is fast and consistent on rule violations, missing checks and common bugs. It does not know why a feature exists, what was agreed with a customer or which tradeoff the team chose last quarter. Use it as a first pass so human reviewers spend their time on design and intent.

What does AI code review miss?

Wrong requirements, architecture decisions, performance under real load, problems that span services the reviewer cannot see, and business rules that are not written down. It can also report problems that do not exist, so findings need a person's judgment.

Does our code leave the company during AI code review?

The diff and the context the reviewer reads are sent to the model provider under your agreement with that provider. Check the data terms of your plan, including retention and training use, before enabling review on private repositories. Some providers also offer access through cloud platforms you already have contracts with.

How much does it cost to set up AI code review?

At Goldenroz, AI code review is set up as part of a 4 to 8 week program for engineering teams, at a fixed price quoted after a 30-minute scoping call. Seats, CI minutes and model usage are paid to the providers from your own accounts.

Written by

CTO and technology partner

Shai leads the technology side: business software, SaaS and mobile apps, payments and escrow systems, Postgres security, AI agents built with Claude Code and MCP, and IT and infrastructure projects with senior infrastructure engineers.

Keep reading

Talk to the person who will do the work

One hour, no sales team. You talk to Daniel or Shai, we look at your account or your product, and you leave with a clear next step.

Or send the details

Book a 1-hour consultationWhatsApp us