Writing Tests With AI: Where It Helps and Where It Hurts

The test that passed but should not have

A couple of years ago I was reviewing a pull request where almost every function had a new test. Coverage looked good. CI was green. The author had used an AI assistant to generate the suite in about twenty minutes, which was fast enough to seem impressive.

Then I looked closely at one test:

it("should return the correct total", () => {
  const result = calculateTotal([10, 20, 30]);
  expect(result).toBe(60);
});

Reasonable. Then the next one:

it("should handle empty array", () => {
  const result = calculateTotal([]);
  expect(result).toBe(0);
});

Fine. Then:

it("should handle negative numbers", () => {
  const result = calculateTotal([-5, 10]);
  expect(result).toBe(5);
});

All passing. The problem: calculateTotal had a bug. It silently converted strings to numbers, so calculateTotal(["10", "20"]) returned 30 instead of throwing or returning NaN. No test covered that. The AI had generated tests that proved the happy path worked, and nothing else. It had no idea what the failure modes were, because it had no runtime context and no knowledge of how this function would actually be called.

That pull request is a decent summary of everything useful to know about AI and test writing. The tools can do a lot. But they optimise for plausibility, not correctness, and the difference matters.

What AI tools are actually doing when they write tests

To use these tools well, it helps to understand what is happening under the hood.

When you feed a function to an AI model and ask it to write tests, the model does a few things:

1. It parses the function signature and any JSDoc or type annotations.
2. It looks at the function body to infer intent.
3. It generates test cases that would be representative based on similar code it has seen in training data.

What it cannot do: run the code, call the function, observe actual output, trace through your runtime environment, read your database schema, understand your business rules, or know what edge cases have bitten you in production.

This means AI-generated tests are a prior, not a proof. They express what the AI thinks the function should do, based on how it reads the code. That prior is often useful. It is also systematically biased toward the happy path and toward the assumptions encoded in the function body itself.

The most dangerous result is a test that mirrors the implementation. If the implementation has a bug, the test frequently has the same bug, because the AI inferred both from the same source.

Where AI genuinely helps: the mechanics of test writing

With that caveat stated, AI tools are genuinely good at a specific class of test-writing work, and dismissing them entirely would be wasteful.

Boilerplate removal

Setting up a test file for a new module is tedious. Imports, describe blocks, beforeEach setup, mock configuration, typed fixture data. An AI assistant can produce a plausible skeleton in seconds.

For a Node.js service using Vitest and a Prisma client, you might type:

Generate a Vitest test file for UserService, which takes a PrismaClient in the constructor and exposes createUser(data: CreateUserInput): Promise<User>. Mock PrismaClient with vi.mock.

And get back something like:

import { describe, it, expect, vi, beforeEach } from "vitest";
import { UserService } from "./user-service";
import type { PrismaClient } from "@prisma/client";

const mockPrisma = {
  user: {
    create: vi.fn(),
    findUnique: vi.fn(),
  },
} as unknown as PrismaClient;

describe("UserService", () => {
  let service: UserService;

  beforeEach(() => {
    vi.clearAllMocks();
    service = new UserService(mockPrisma);
  });

  it("should create a user", async () => {
    const input = { email: "a@b.com", name: "Alice" };
    const expected = { id: "1", ...input };
    mockPrisma.user.create = vi.fn().mockResolvedValue(expected);

    const result = await service.createUser(input);

    expect(mockPrisma.user.create).toHaveBeenCalledWith({ data: input });
    expect(result).toEqual(expected);
  });
});

This is correct, typesafe, and would have taken a few minutes to write by hand. The AI produced it in ten seconds. The pattern is reliable for standard dependency injection setups where the shape of the service is clear from its constructor signature and public methods.

Generating parametrised test cases

Another area where AI earns its keep: generating a large set of input/output pairs for pure functions, especially ones with mathematical or string-manipulation logic.

// Ask: "Give me 15 test cases for a function that validates UK postcodes"

const cases = [
  { input: "SW1A 1AA", valid: true },
  { input: "EC1A 1BB", valid: true },
  { input: "W1A 0AX", valid: true },
  { input: "M1 1AE", valid: true },
  { input: "B1 1BB", valid: true },
  { input: "DN55 1PT", valid: true },
  { input: "CR2 6XH", valid: true },
  { input: "SW1A1AA", valid: false },   // no space
  { input: "INVALID", valid: false },
  { input: "1234 567", valid: false },
  { input: "", valid: false },
  { input: "sw1a 1aa", valid: true },   // lowercase, if normalised
  { input: "SW1A  1AA", valid: false }, // double space
  { input: "ZZ9 9ZZ", valid: false },   // reserved
  { input: "GIR 0AA", valid: true },    // special case: Girobank
];

describe.each(cases)("validatePostcode($input)", ({ input, valid }) => {
  it(`returns ${valid}`, () => {
    expect(validatePostcode(input)).toBe(valid);
  });
});

You would not have thought of all fifteen cases quickly. The AI covers the obvious ones fast and sometimes surfaces genuinely tricky edge cases, like the Girobank postcode above, which many developers do not know about. You still need to verify each case against the actual spec (the Royal Mail PAF standard in this case), but the AI saves you the initial brainstorm.

Writing snapshot and integration test shells

For React component tests with Testing Library, AI can produce a plausible shell that you then adjust:

import { render, screen, fireEvent } from "@testing-library/react";
import { JobCard } from "./JobCard";

const mockJob = {
  id: "job-1",
  title: "Senior Engineer",
  company: "Acme Corp",
  location: "London",
  postedAt: new Date("2026-06-01"),
};

describe("JobCard", () => {
  it("renders the job title and company", () => {
    render(<JobCard job={mockJob} onApply={vi.fn()} />);
    expect(screen.getByText("Senior Engineer")).toBeInTheDocument();
    expect(screen.getByText("Acme Corp")).toBeInTheDocument();
  });

  it("calls onApply when the apply button is clicked", () => {
    const onApply = vi.fn();
    render(<JobCard job={mockJob} onApply={onApply} />);
    fireEvent.click(screen.getByRole("button", { name: /apply/i }));
    expect(onApply).toHaveBeenCalledWith("job-1");
  });
});

Again, this is competent and would be a reasonable starting point. The AI does not know the exact prop interface of JobCard unless you provide it, so you have to feed it either the component source or the TypeScript interface. With that context, the output is usually close to correct.

Where AI fails: the four failure modes

1. Mirroring implementation bugs

This is the one from the introduction, and it is the most serious. If a function has a logical error, the AI generates a test that embeds the same assumption. The test passes, the bug ships.

The root cause is that the AI infers expected output by reading the implementation, not by running it. For pure transformations this is usually harmless. For anything involving business logic, it is a significant risk.

The mitigation: never generate tests from the implementation alone. Write at least the core assertions yourself, against a spec or against your own understanding of what the function should return. Use AI to generate the boilerplate around your assertions, not the assertions themselves.

2. Mocks that verify nothing

AI-generated mocks sometimes check that a function was called without checking what it was called with. This produces tests that pass regardless of whether the correct arguments were passed.

// Bad: generated by AI, checks call count but not arguments
expect(mockEmailService.send).toHaveBeenCalled();

// Better: what you actually want
expect(mockEmailService.send).toHaveBeenCalledWith({
  to: "user@example.com",
  subject: "Welcome to Ace Hired",
  template: "welcome",
});

The distinction matters. The first test passes even if you hardcoded the wrong recipient or the wrong template. The second test catches that. AI tools consistently generate the weaker form because it requires less context about the arguments that matter.

3. Missing failure paths

AI tests cover the happy path and sometimes obvious error cases. They rarely cover:

• Network timeouts that return after a partial operation.
• Concurrent modification: two requests that arrive simultaneously and modify the same record.
• Downstream service that returns a valid HTTP 200 with an unexpected body shape.
• Third-party SDK that throws a non-standard error class.
• Pagination edge cases: last page returns fewer items than the page size.

These are the cases that cause production incidents. They require knowledge of your infrastructure and call patterns, which the AI does not have. You have to write these yourself.

4. Overfitting to the current implementation

If you refactor a function without changing its behaviour, your tests should still pass. But AI-generated tests frequently assert on internal details, like the exact sequence of mock calls or the intermediate state of a closure, rather than on observable outputs.

// Bad: testing internal structure, breaks on refactor
expect(mockCache.get).toHaveBeenCalledTimes(1);
expect(mockCache.set).toHaveBeenCalledTimes(1);
expect(mockDb.query).toHaveBeenCalledTimes(1);

// Better: testing observable behaviour
const result = await getUser("user-1");
expect(result).toEqual({ id: "user-1", name: "Alice" });

// Second call should hit cache
const cached = await getUser("user-1");
expect(mockDb.query).toHaveBeenCalledTimes(1); // still only 1 DB call

The second version says what you care about: a cache hit means no extra DB round-trip. The first version says how many times each internal function ran in the current implementation. If you change the caching strategy, the first version breaks even when behaviour is identical.

A real gotcha: the "all green" false confidence trap

Here is the honest lesson from using these tools across several projects.

After running AI-generated suites for the first time and seeing 80-plus percent coverage with all green, there is a temptation to consider the testing done. This is wrong, and the reason is subtle.

Coverage measures whether lines were executed, not whether they were tested correctly. A test can execute a line without asserting anything meaningful about its output. AI-generated tests frequently hit this failure mode because they generate assertions that match the current output of the code, whatever that output is, including buggy output.

The practical consequence: a codebase with 85% AI-generated coverage and a codebase with 50% hand-written, assertion-heavy coverage can differ enormously in how many bugs those tests catch. Coverage percentage is a weak signal. Assertion quality is the real metric, and it is not measured by any standard tooling.

To counter this, consider mutation testing. Tools like Stryker (JavaScript/TypeScript) and mutmut (Python) deliberately introduce bugs into your source code and check whether your tests catch them. If a mutation survives, you have a test gap regardless of what the coverage number says.

npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner


npx stryker run

Stryker reports a mutation score. An AI-generated test suite often scores 40 to 55% on first run. A thoughtfully written suite targeting critical paths typically scores 75 to 85%. The difference is in how carefully the assertions were designed.

Integrating AI into a sensible test workflow

Rather than using AI to write complete test files and then reviewing them, a better workflow layers AI assistance at specific steps.

Step 1: Write the spec assertions yourself first.

Before opening any AI tool, write the core assertions for your function. What inputs do you care about? What must the output be? What errors must be raised?

// Written by hand, before any AI involvement
it("should throw if the email is already registered", async () => {
  mockPrisma.user.findUnique = vi.fn().mockResolvedValue({ id: "existing" });
  await expect(service.createUser({ email: "taken@example.com" })).rejects.toThrow(
    "Email already in use"
  );
});

Step 2: Ask AI to generate the boilerplate and additional positive cases.

With the core assertions written, ask the AI to fill in the setup code, the fixture data, and a few additional happy-path cases. Now the AI is working within constraints you set, not setting them for you.

Step 3: Ask AI to suggest edge cases, then evaluate each one manually.

Given this function signature and these existing tests, what edge cases might I be missing?

This is where AI input is most valuable and least risky. It is brainstorming, not code generation. You review each suggestion and decide whether it represents a real risk.

Step 4: Run mutation tests after any substantial new suite.

If mutation testing is too slow for your entire codebase, run it scoped to the module you just tested.

npx stryker run --files src/user-service.ts

Any surviving mutant is worth a look. Some will be trivially safe; others will reveal a real gap.

Choosing the right tool for the job

Not all AI assistants are equally useful for test writing. As of mid-2026, the meaningful differences are:

GitHub Copilot in-editor (GPT-4o based): Best for boilerplate completion as you type. It uses your open file as context, so it picks up your mock setup and generates consistent follow-on tests. Weakest at catching failure modes because it does not look at your broader codebase.

Cursor with codebase indexing: Significantly better context. It can read your schema, your types, and your existing tests across the repo. This reduces the "wrong mock shape" failure mode considerably. Still does not execute code.

Claude via CLI or API (Sonnet or Opus): Best for long-context analysis: feeding it an entire module, its dependencies, and asking for a comprehensive test plan. Useful for the "what am I missing" step because of larger context windows. Not integrated into your editor, so higher friction for incremental test generation.

Specialized tools (CodiumAI / TestPilot): These are purpose-built for test generation and often do better than general-purpose AI on assertion quality because they are trained specifically on test patterns. Worth evaluating if your team writes tests at scale.

None of these tools run your code. That remains the fundamental constraint across all of them.

Practical takeaways

Use AI for setup, not for correctness. Boilerplate, fixture generation, parametrised case lists. These are the areas where AI saves real time without meaningful risk.

Write your own assertions for anything load-bearing. If a test failure would block a deployment, the assertion needs to come from your understanding of the requirements, not the AI's reading of the implementation.

Treat all AI-generated mocks with suspicion. Check whether they are verifying call arguments or just call counts. Tighten them before committing.

Run mutation testing after any AI-generated suite. Coverage numbers are misleading. Mutation score is a better proxy for how useful your tests actually are.

Use AI to brainstorm edge cases, not to close them. Ask it what might go wrong. Then decide which risks are real, write the test yourself, and run it against the actual code.

The "all green" feeling after AI-generated tests is a warning sign, not a reward. If a brand-new suite passes immediately on a module you have not tested before, that almost certainly means the tests are asserting on whatever the current implementation does, bugs included. Introduce a deliberate bug and verify the suite catches it. If it does not, the suite is providing false confidence.

AI tools change the economics of writing tests in ways that are genuinely useful. They do not change what a good test is. A good test fails when the code is wrong and passes when the code is correct. That property still requires a human who understands what "correct" means for the system they are building.