AI engineer system design interview questions

As asked

Design a production RAG system for a 10-million-document internal knowledge base. The system must support sub-2-second p95 end-to-end latency, access control at the document level, and real-time index updates as documents change.

Sample answer outline

A strong design includes an ingestion pipeline (document parser, chunker, embedder, metadata extractor), a vector store (Qdrant or Weaviate with payload-indexed access control fields), a BM25 index alongside for hybrid search, a query pipeline (query rewriting, parallel dense + sparse retrieval, reranking, prompt assembly, LLM call), and a streaming API with prompt caching. The candidate addresses ACL by embedding user permissions as filter metadata at query time, and real-time updates via a change-data-capture pipeline that re-chunks and re-embeds only modified sections.

Expect these follow-ups

• How do you handle a document update that changes 10% of its content?
• Walk me through what happens when a user's access rights change and they had cached results.

As asked

You need an LLM to extract invoice fields into JSON for downstream automation. How do you make the output reliable enough for production?

Sample answer outline

Define a strict schema first, including nullable fields, enums, currency handling, and confidence or evidence fields. Use model-native structured output or function calling where available, then validate every response with a real parser before accepting it. Add deterministic post-processing only for well-defined normalisation, not for fixing arbitrary model text. The pipeline should route low-confidence, invalid, or high-value invoices to human review and should store the original document and model version for audit. Strong candidates discuss evaluation by field-level precision and recall rather than a vague accuracy score.

Expect these follow-ups

• How do you handle a field that is visible in the invoice but ambiguous?
• What changes if the invoice may be handwritten?
• How do you version the schema without breaking downstream consumers?

As asked

Design a centralized LLM gateway that your organization's teams use. It should support routing requests to different models (GPT-4o, Claude, Gemini), enforce cost budgets per team, provide observability, and handle provider outages gracefully.

Sample answer outline

Key components: an API layer accepting a model-agnostic request format, a routing engine that selects provider based on model tag or quality/cost policy, per-team budget tracking in Redis with atomic decrement, a fallback chain when a provider is down, unified logging of every request/response for observability, and a prompt cache keyed on (model, normalized prompt hash). The candidate discusses OpenAI-compatible proxy vs custom protocol, and how to handle streaming across multiple provider formats.

Expect these follow-ups

• How do you normalize streaming responses from different providers that use different SSE formats?
• What happens to in-flight requests when the gateway restarts?

As asked

Design the memory architecture for a personal assistant agent that must remember facts about the user across sessions, prioritize recent or important memories, and not exceed the context window at inference time.

Sample answer outline

A strong design separates memory tiers: working memory (current session in context), episodic memory (past sessions as embeddings in a vector store, retrieved by relevance), and semantic memory (extracted facts in a structured store, retrieved by entity or category). The agent writes to episodic and semantic memory after each session via extraction prompts. At query time, it retrieves the most relevant memories and packs them into a budget-limited context. The candidate discusses forgetting strategies: recency weighting, importance scoring, and explicit user deletion.

Expect these follow-ups

• How do you handle contradictory memories, e.g., the user said they are vegetarian but later ordered meat?
• How do you prevent the memory extraction step from hallucinating false memories?

As asked

Design a system that automatically evaluates every change to your prompts or model, catches regressions before they reach production, and gives the team a dashboard to compare versions.

Sample answer outline

The design includes a versioned eval dataset in a data store (with train/validation/test splits), a CI job triggered on prompt or model config changes, parallel evaluation runs using the new and previous version, automated metrics (faithfulness, answer relevance, exact match for structured outputs), LLM-as-judge for open-ended quality, a comparison report highlighting regressions, a human review queue for flagged cases, and a dashboard tracking metric trends over time. The candidate notes that the eval dataset must be continuously updated from production samples.

Expect these follow-ups

• How do you handle the eval set becoming stale as the user distribution shifts?
• How do you gate a prompt change that improves one metric but regresses another?

As asked

Your LLM product processes user documents that may contain PII. Design a pipeline that detects and scrubs PII before the document is sent to a third-party LLM API, then re-inserts it in the response where needed.

Sample answer outline

The design uses a fast NER model (spaCy or a dedicated PII model like Presidio) to detect PII spans, replaces each span with a reversible placeholder (e.g., [PERSON_1]), maintains a session-scoped mapping from placeholder to original value, sends the scrubbed document to the LLM, and post-processes the response by replacing any placeholder that appears in the output back to the original. The candidate discusses regex fallbacks for structured PII (email, phone, SSN), the latency of the NER step, and audit logging.

Expect these follow-ups

• How do you handle PII that the NER model misses?
• What happens when the LLM generates new PII in its response that was not in the original document?

As asked

Design a conversational product search assistant for an e-commerce site. Users refine their search over multiple turns, and the assistant must remember context, translate natural language into filters, and explain its reasoning.

Sample answer outline

The design maintains a session state with accumulated filters extracted by the LLM from each turn, uses a structured product DB with a well-defined filter schema, has the LLM convert natural language into a typed filter object (function calling), executes the query and returns results, and feeds back the result count so the LLM can suggest refinements. The candidate discusses handling contradictory filters (user says under $50, then asks for premium), and the fallback when no results match.

Expect these follow-ups

• How do you handle ambiguous product attributes like 'a good laptop for school'?
• How does your session state survive a page refresh?

As asked

Design a RAG system that can answer questions over a corpus of PDFs containing text, tables, and embedded images. Users ask questions like 'what does the revenue chart on page 12 of the Q3 report show?'

Sample answer outline

A strong design handles three content modalities: text is chunked and embedded with a text encoder, tables are extracted (via pdfplumber or Camelot) and serialized to markdown or JSON before embedding, and images are described using a vision LLM (generating a text caption) and the caption is embedded. At retrieval time, the query retrieves from a unified index, and at generation time the original image or table is included in the multimodal prompt to the LLM. The candidate discusses bounding box metadata for provenance linking.

Expect these follow-ups

• How do you handle charts where the extracted description loses numeric precision?
• How does your pipeline handle scanned PDFs with no text layer?

As asked

Design an agent system that can answer questions about live financial data: price feeds, news, and order book snapshots arriving at high frequency. The agent must cite its sources and answer within 3 seconds.

Sample answer outline

The design includes real-time ingestion feeding a sliding-window vector store and a separate structured DB for numeric data, specialized tools for each data type (price tool returns latest OHLCV, news tool runs similarity search, order book tool returns aggregated depth), an orchestrator that decomposes the user question into parallel tool calls, a synthesis prompt that cites tool results, and a streaming response under 3 seconds achieved via parallel tool execution and a cached system prompt. The candidate discusses how to avoid stale data in the vector index.

Expect these follow-ups

• How do you prevent the agent from citing a news item that was retracted?
• What do you do if the price tool times out mid-execution?

As asked

Design a pipeline to continuously fine-tune an open-source LLM on new domain data, evaluate it, and deploy it to production with zero downtime.

Sample answer outline

The design includes a data flywheel (production feedback tagged and stored, cleaned via a quality filter), a training pipeline (LoRA/QLoRA for efficiency, triggered on data threshold or schedule), automated evals against a held-out benchmark and a production eval set, blue-green deployment using a serving framework like vLLM or TGI behind a router, canary traffic ramp with automatic rollback on metric regression, and adapter management allowing hot-swap without server restart. The candidate notes the importance of training/serving parity.

Expect these follow-ups

• How do you version and store LoRA adapters alongside base model versions?
• What monitoring do you set up for the serving infrastructure after deployment?