AI engineer interview questions

As asked

Design a RAG feature that answers customer support questions using internal documentation. How do you keep answers grounded and know when the model should refuse?

Sample answer outline

A good answer separates retrieval quality from generation quality. Chunk documents around semantic boundaries, attach metadata such as product, version, and freshness, and retrieve with hybrid search plus reranking for high precision. The generation prompt should require citations to retrieved passages and should refuse when evidence is missing or contradictory. Evaluation needs golden questions, adversarial no-answer cases, citation accuracy checks, and human review for high-impact categories. Candidates often focus on vector databases while ignoring stale docs, permissions, and the model's tendency to fill gaps.

Expect these follow-ups

• How do you prevent a user from retrieving documents they cannot access?
• What metric tells you retrieval is the bottleneck?
• How do you handle two documents that disagree?

As asked

A support assistant needs conversation history, policy docs, customer metadata, and tool results, but the context window is regularly exceeded. How do you redesign the prompt flow?

Sample answer outline

The answer should treat the context window as a scarce product resource. Keep durable instructions short, summarise long conversation history with provenance, retrieve only task-relevant policy chunks, and pass customer metadata as a compact structured block rather than prose. Tool results should be schema-shaped and pruned to fields the next step needs. Add tests for truncation boundaries because prompt changes can silently push out important context. Candidates often rely on larger context windows instead of deciding what information is actually needed for the task.

Expect these follow-ups

• How do you know a summary has not lost a critical fact?
• What context should never be summarised?
• How would you debug a failure caused by truncation?

As asked

Write a TypeScript tool definition for creating a support ticket, then explain how you would make the schema safe for agent use.

Sample answer outline

The schema should make required fields explicit, restrict enums, and avoid accepting vague free-form blobs for important state. Tool descriptions should say when to call the tool and when not to call it, but the server must still validate permissions and business rules. A good answer includes idempotency keys or deduplication because agents may retry after timeouts. The tool should return a small structured result the agent can reason over, not an entire database record. Candidates often make tools too broad, which increases accidental side effects and makes eval failures hard to diagnose.

Reference implementation (typescript)

import { z } from "zod";

export const createSupportTicketTool = {
  name: "create_support_ticket",
  description:
    "Create one support ticket after the user has described a concrete problem. Do not call for general questions.",
  inputSchema: z.object({
    customerId: z.string().min(1),
    idempotencyKey: z.string().uuid(),
    severity: z.enum(["low", "normal", "high", "urgent"]),
    subject: z.string().min(5).max(120),
    description: z.string().min(20).max(4000),
    orderId: z.string().optional(),
  }),
  async execute(input: unknown) {
    const args = this.inputSchema.parse(input);
    return {
      ticketId: await createTicket(args),
      status: "created" as const,
    };
  },
};

Expect these follow-ups

• How do you version this tool without breaking old agents?
• Where does human approval fit?
• What should the tool return on validation failure?

As asked

You are not training a model from scratch, you are building a feature on top of a foundation model with prompting, retrieval, and maybe light fine-tuning. How does bias-variance thinking still apply to the choices you make, and how do you tell whether your system is underfitting or overfitting the task?

Sample answer outline

Translate the concept into the applied-AI setting. Underfitting shows up as a system that is too generic: weak prompts, no retrieval, and a model that gives plausible but off-target answers because it lacks the task context. Overfitting shows up as a system tuned so tightly to a handful of examples that it fails on real, varied inputs: brittle few-shot prompts, a fine-tune on a tiny biased dataset, or retrieval that only works for the queries you tested. The applied controls are the modern analogues of regularisation and capacity: richer context and retrieval reduce bias, while a held-out evaluation set, diverse examples, and resisting over-tuning to a demo reduce variance. The AI-engineer signal is measuring this with an evaluation harness on representative inputs rather than eyeballing a few prompts, and knowing when a prompt change is genuinely better versus fitted to your test cases.

Expect these follow-ups

• How do you build an evaluation set that reveals overfitting to your examples?
• When does adding retrieval reduce bias versus just adding noise?
• Why can a fine-tune on a small dataset make a model worse on real inputs?

As asked

You need an LLM to extract invoice fields into JSON for downstream automation. How do you make the output reliable enough for production?

Sample answer outline

Define a strict schema first, including nullable fields, enums, currency handling, and confidence or evidence fields. Use model-native structured output or function calling where available, then validate every response with a real parser before accepting it. Add deterministic post-processing only for well-defined normalisation, not for fixing arbitrary model text. The pipeline should route low-confidence, invalid, or high-value invoices to human review and should store the original document and model version for audit. Strong candidates discuss evaluation by field-level precision and recall rather than a vague accuracy score.

Expect these follow-ups

• How do you handle a field that is visible in the invoice but ambiguous?
• What changes if the invoice may be handwritten?
• How do you version the schema without breaking downstream consumers?

As asked

A PM says prompt B sounds better than prompt A for an AI writing feature. How do you decide which one to ship?

Sample answer outline

Start by defining the user outcome, such as acceptance rate, edit distance after generation, task completion, or user-rated usefulness. Offline evals should use a representative dataset with blind human preference labels and rubric-based checks for factuality, tone, policy, and formatting. Online tests need guardrails: latency, cost, complaint rate, regeneration rate, and downstream retention if the feature is material. A strong answer explains why subjective taste alone is not enough but also why automatic judge scores need calibration. Candidates trip up by optimising for win rate against a narrow prompt set and missing product impact.

Expect these follow-ups

• When would you trust an LLM-as-judge?
• How do you avoid overfitting the prompt to the eval set?
• What if prompt B wins quality but doubles latency?