AI Code Review: Workflows, Wins, and Pitfalls

The Pull Request Nobody Wanted to Review

Every team has that PR. Fourteen files changed, a migration, two new API endpoints, a refactor of the auth middleware that "just happened while I was in there", and a diff that clocks in at 900 lines. It lands on a Friday afternoon. The reviewer leaves a comment saying "LGTM, looks good" on line 12 and merges it. Three weeks later a subtle race condition ships to production.

This is the failure mode AI code review is actually solving, and it is worth being precise about that. The pitch from every vendor is "catch bugs automatically," but the deeper value is that AI review is tireless and consistent. It will look at the boring file. It will check the migration for an index on a foreign key column even if nobody on the team thinks to ask. It does not have meetings, does not have a stack of other PRs waiting, and does not feel social pressure to approve the code written by the senior who pushed back last time.

That said, AI code review also hallucinates, misses context, generates noise that trains reviewers to ignore alerts, and has well-defined failure modes that are easy to walk into if you deploy it naively. This post covers the real workflows, what actually works, what fails, and how to design a setup that makes the human reviewers better rather than replacing them with a false sense of safety.

What "AI Code Review" Actually Means in 2025

The term covers several different products with meaningfully different architectures:

Static-analysis wrappers like DeepSource, SonarQube with AI rules, and CodeClimate run AST-level or dataflow checks augmented by ML models trained on known bug patterns. They are deterministic given the same code, fast, and relatively low noise. Their ceiling is also low: they catch what their training set covered.

LLM-based PR reviewers like GitHub Copilot Code Review, Coderabbit, Cursor's review mode, and a growing list of newer tools send your diff (sometimes with repo context) to a language model and ask it to comment. Output varies per run. They catch a broader class of issues including logic errors, unclear naming, missing error handling in patterns the static tool never saw. They also hallucinate.

Inline editor review like Copilot Chat, Claude in the editor, or Codeium's explain mode are interactive: a developer asks "what's wrong with this function" and gets an answer. This is closer to pairing than CI gating.

CI-integrated review agents are the newest category: tools that check out the branch, run the tests, read the full file not just the diff, and post structured comments with confidence scores and fix suggestions. Coderabbit's "agentic" mode and some self-hosted Claude-based pipelines fall here.

For the rest of this post the focus is on the LLM-based PR reviewer category since that is where most engineering teams are making decisions right now and where the tradeoffs are least understood.

Designing a Workflow That Does Not Create Review Fatigue

The single biggest failure mode in AI code review is noise. If the tool posts ten comments per PR and six of them are wrong or irrelevant, reviewers learn to ignore all ten. At that point you have spent money to make your review process worse.

The fix is not to lower the comment volume to zero; it is to structure the pipeline so that different signals go to different places.

Here is a practical architecture:

name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize, ready_for_review]

jobs:
  ai-review:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate structured diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD \
            --unified=5 \
            --no-color \
            > /tmp/pr_diff.txt

      - name: Run AI review
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: node scripts/ai-review.mjs

The script that calls the model matters more than the workflow file. Here is a minimal but well-structured version:

// scripts/ai-review.mjs
import Anthropic from "@anthropic-ai/sdk";
import { readFileSync } from "fs";
import { execSync } from "child_process";

const client = new Anthropic();
const diff = readFileSync("/tmp/pr_diff.txt", "utf8");

const SYSTEM = `You are a code reviewer. Your job is to find real bugs, security issues, 
and correctness problems. Do NOT comment on style, formatting, naming preferences, 
or things that are "nice to have". Only report issues that could cause incorrect 
behaviour, data loss, security vulnerabilities, or broken tests.

For each issue, respond with a JSON array of objects with this shape:
{
  "file": "path/to/file.ts",
  "line": 42,
  "severity": "error" | "warning",
  "category": "bug" | "security" | "correctness" | "performance",
  "comment": "One clear sentence describing the problem.",
  "suggestion": "Optional: what to do instead."
}

If you find no real issues, return an empty array [].`;

const response = await client.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 4096,
  system: SYSTEM,
  messages: [
    {
      role: "user",
      content: `Review this pull request diff:\n\n\`\`\`diff\n${diff}\n\`\`\``,
    },
  ],
});

const raw = response.content[0].text;
// Extract JSON robustly even if the model adds prose around it
const match = raw.match(/\[[\s\S]*\]/);
const findings = match ? JSON.parse(match[0]) : [];

// Post only findings with severity "error" as blocking PR comments
// Post "warning" findings as a summary comment, not inline
const errors = findings.filter((f) => f.severity === "error");
const warnings = findings.filter((f) => f.severity === "warning");

if (errors.length === 0 && warnings.length === 0) {
  console.log("No findings. Exiting.");
  process.exit(0);
}

// Post inline comments for errors via GitHub API
for (const finding of errors) {
  await postInlineComment(finding);
}

// Post a single summary comment for warnings
if (warnings.length > 0) {
  await postSummaryComment(warnings);
}

The key decisions here: structured output so you can filter by severity, inline comments only for actual errors, and warnings bundled into a single summary comment rather than individual noise. This keeps the review thread readable.

What AI Review Catches Well

LLM-based review genuinely excels at a few categories that humans consistently miss under time pressure.

Off-by-one errors and boundary conditions. Humans read loops idiomatically. "For i from 0 to length" parses as fine even when the condition is i <= length instead of i < length. The model does not skim.

def chunk_list(items: list, size: int) -> list[list]:
    result = []
    for i in range(0, len(items), size):
        # Bug: i + size should not need + 1 but the slice is already exclusive
        # However if someone refactors to manual indexing this breaks
        result.append(items[i:i + size])
    return result

A good prompt catches that size=0 has no guard and will raise a confusing ValueError from range rather than a meaningful error from the function itself.

Missing error handling at async boundaries. This is one of the highest-value catches I have seen in practice:

// Code in the diff
async function fetchUserData(userId: string) {
  const response = await fetch(`/api/users/${userId}`);
  const data = await response.json();
  return data;
}

The model will flag that response.ok is not checked before calling .json(), that .json() itself can throw if the response body is not valid JSON, and that neither error surfaces a meaningful message to the caller. A human reviewer who has written this exact pattern themselves may not notice it.

SQL injection and similar injection patterns in new ORM usage. When a developer introduces a new data access pattern mid-PR, the model sees it fresh.

def get_user_by_email(email: str, db: Session):
    # Vulnerable: f-string interpolation into a raw query
    return db.execute(f"SELECT * FROM users WHERE email = '{email}'").fetchone()
    # Should use: db.execute(text("SELECT * FROM users WHERE email = :email"), {"email": email})

Inconsistent return types. TypeScript type narrowing means a function that returns string | null in one path and undefined in another will compile but cause bugs at callsites that only guard for null.

Where AI Review Fails: The Honest Failure Modes

Hallucinated bugs. The model will sometimes report a bug that does not exist. The most common pattern is misreading the call stack: seeing a function that looks like it could be called with the wrong argument, without knowing that all callers are typed and validated upstream.

I spent twenty minutes investigating a "null dereference" finding once before I traced the model's logic and found it had assumed an optional chain was missing when it was on the line above the diff hunk. The model did not have enough context about the surrounding file.

This is actually a strong argument for giving the model the full file, not just the diff. Most CI integrations default to the diff for cost reasons, but for files under 500 lines the extra context dramatically reduces false positives.

Domain logic that requires business context. If the review rule is "discount codes cannot be applied to subscription products", no model trained on generic code will know that. It will approve code that violates the rule unless you explicitly encode the constraint in your prompt or in the codebase as a comment or test.

// The model has no idea this is wrong for your business
function applyDiscount(cart: Cart, code: DiscountCode): Cart {
  // Bug: subscription items should be excluded per business rule
  return {
    ...cart,
    items: cart.items.map((item) => ({
      ...item,
      price: item.price * (1 - code.discountRate),
    })),
  };
}

This is not a knock on AI review. A new human reviewer who does not know the business rule would miss it too. The solution is the same in both cases: encode the rule as a test or a comment that is visible in the diff.

Race conditions and distributed systems bugs. These require holding a mental model of concurrent execution that does not fit neatly into a diff-based review. The model can spot obvious mutex misuse or an obvious check-then-act pattern, but subtle ordering bugs across services are out of reach.

Linter fatigue from incorrect style suggestions. If you do not constrain the model's output to correctness issues, it will suggest changing forEach to for...of for performance reasons that do not apply, or flag a ternary as "hard to read" when it is perfectly idiomatic in your codebase. The system prompt discipline described earlier is essential.

The Gotcha That Cost Our Team Two Weeks

This is worth its own section because I see teams make this mistake repeatedly.

We integrated an LLM-based reviewer into CI. It posted comments. Developers started seeing comments and, because they were coming from an automated system, unconsciously gave them lower priority than human reviewer comments. Over a couple of months the review culture shifted slightly: human reviewers began to assume the AI had covered the basics, so they spent less time on correctness and more time on architecture and design. Reasonable on the surface.

The problem: the AI was covering "the basics" inconsistently. A file it had reviewed three weeks ago might have a different response today. Two PRs in the same week with similar patterns might get flagged in one and not the other. Nobody had characterised the tool's actual recall rate on the class of bugs that mattered.

When a data integrity bug shipped, the post-mortem showed the AI had reviewed the PR and found no issues. The team had implicitly treated that as a safety signal. It was not one.

The lesson: AI review is a first-pass filter, not a safety net. It should make human review more targeted, not replace the judgment that the human reviewer brings. The CI status check should probably not block merge; it should surface information. The human reviewer's approval should still be required and still carry weight.

Context Engineering: Making the Model Actually Useful

The quality of AI review output correlates strongly with how well you structure the context you send. A raw diff sent with a generic prompt is much worse than a structured request that tells the model what kind of codebase this is, what patterns are in use, and what it should focus on.

A good system prompt for a TypeScript/Node backend might include:

You are reviewing a TypeScript Node.js backend service using:
- Express 4.x for routing
- Prisma 5.x as the ORM (use parameterised queries, never raw string interpolation)
- Zod for runtime validation at all API boundaries
- Jest for testing

Known patterns in this codebase:
- All async route handlers are wrapped in asyncHandler(); you do NOT need to flag missing try/catch in route handlers
- Database access always goes through service layer classes in src/services/
- Auth is handled by middleware before routes; do not flag missing auth checks on routes under /api/

Focus on:
1. Prisma queries that bypass type safety or use raw SQL without parameterisation
2. Zod schemas that are missing fields present in the corresponding Prisma model
3. Service methods that do not handle Prisma's P2025 (record not found) error
4. Any place where user-supplied data reaches a system call, file path, or external URL without sanitisation

Do not comment on: naming conventions, import ordering, comment style, or any pattern explicitly annotated with // reviewed: intentional

This kind of prompt takes an hour to write and probably halves your false-positive rate. It is the highest-leverage investment in the setup.

Practical "How to Apply This" Checklist

If you are setting up AI code review for the first time or auditing an existing setup, work through these:

1. Define scope before you integrate. Decide which file patterns, directories, and change types the tool should review. Exclude auto-generated files, migrations you do not want re-reviewed, and vendor code.

2. Write a codebase-specific system prompt. Generic prompts produce generic (noisy) output. Spend an hour on this before you spend any time debugging false positives.

3. Separate severity levels and route them differently. Errors as inline comments. Warnings as a single summary. Informational findings as a collapsed section or not at all.

4. Measure noise rate for two weeks. For every AI comment in your PRs, have a developer mark it as valid finding, false positive, or style/opinion. Tune the prompt based on this. Aim for a false-positive rate under 20% before you expand scope.

5. Never use AI approval as a merge gate. Use it as a filter to direct human attention, not as a pass/fail checkpoint.

6. Give the model file context, not just the diff. For files under 1000 lines, attach the full file alongside the diff. The cost is small and the accuracy gain is real.

7. Add a "reviewed: intentional" comment convention. Any pattern that you know the AI will flag but that is correct for your codebase gets this comment. The prompt is configured to skip these. This prevents the same false positive appearing on every PR that touches a particular pattern.

8. Re-evaluate every quarter. Models improve. Prompt techniques improve. A setup that was correctly calibrated six months ago may need adjustment because the model's defaults have shifted.

What Human Reviewers Should Still Own

AI review does not change what experienced engineers bring to a review. What it changes is how they allocate time. With the correctness filter running, a human reviewer should be spending their time on:

• Architecture and extensibility. Does this new abstraction fit the system or is it going to become a maintenance burden in six months?
• Product and business logic. Does this code do what the ticket described? Are there edge cases the ticket did not cover?
• Team knowledge transfer. Is this code understandable to a junior joining in three months? Does it need more documentation?
• Test quality. Are the tests actually testing the right things, or are they testing implementation details that will break every refactor?
• Systemic patterns. Is this the third PR that has the same shape of bug, and should we fix the root cause rather than each instance?

None of those improve with AI review. They get more time because the AI has handled the scan for typos and missing null checks.

Key Takeaways

• AI code review is a first-pass filter, not a safety net. Treating it as a gate creates false confidence.
• Noise kills adoption faster than missed bugs. Invest in prompt engineering before worrying about coverage.
• Give the model file context alongside the diff. The accuracy improvement outweighs the cost for most files.
• Measure your false-positive rate actively for the first month. Tune before expanding scope.
• Domain logic that requires business context will not be caught without explicit prompt engineering or encoded tests.
• Human reviewers should use the time freed by AI filtering to do the work only humans can do: architecture judgment, knowledge transfer, and product sense.
• The hallucination failure mode is real. Build your workflow so a wrong AI comment cannot cause harm; route errors to human judgment, not to merge blockers.
• Revisit your prompt and tool choice every quarter. This space is moving fast enough that a decision made in early 2025 may need revisiting by late 2025.

The teams getting real value from AI code review are not the ones who plugged in a tool and hoped for the best. They are the ones who treated the integration as an engineering problem: scoped it carefully, measured the output, tuned the signal, and kept humans in the loop for the decisions that actually matter.