LLM ops engineer system design interview questions

As asked

Design an internal LLM gateway used by ten product teams. It must support provider fallback, prompt versioning, cost controls, and audit logging.

Sample answer outline

The gateway should be a policy and observability layer, not just an HTTP proxy. It needs typed request contracts, prompt and model version identifiers, tenant-level budgets, retries with idempotency, and provider fallback rules that are explicit about quality and compliance tradeoffs. Audit logs should capture inputs, outputs, metadata, and redaction state according to data policy. The answer should include latency budgets and failure modes, such as a fallback model producing different JSON shape or safety behaviour. Strong candidates avoid automatic fallback for high-risk flows unless the evals prove equivalence.

Expect these follow-ups

• How do you prevent fallback from masking a provider outage?
• What data do you store when prompts contain personal data?
• How do teams override defaults without bypassing governance?

As asked

Design the end-to-end serving infrastructure to handle 10,000 requests per minute for a 70B parameter model, with a p99 latency SLO of 5 seconds for responses under 200 output tokens. Include load balancing, GPU allocation, autoscaling, and observability.

Sample answer outline

A strong answer sizes the GPU fleet based on token throughput math (roughly 2 to 4 H100 nodes at 8 GPUs each depending on concurrency), uses a load balancer that routes based on KV cache availability rather than round-robin, deploys vLLM with tensor parallelism within a node and multiple nodes behind the load balancer, implements autoscaling triggered by queue depth, and has a full observability stack (GPU utilization, KV cache utilization, TTFT p99, throughput per node) with alerting before SLO breach.

Expect these follow-ups

• How would the design change if 20% of requests are RAG queries with 16K input context?
• What failure mode would you be most worried about at this scale?

As asked

A browser-use agent keeps searching, clicking, and returning to the same page without completing the task. How do you debug and fix the orchestration?

Sample answer outline

Start with a trace of observations, model messages, tool calls, screenshots or DOM summaries, and stop reasons for each step. Look for missing state, ambiguous success criteria, poor page abstraction, tool errors hidden as normal observations, or prompts that reward continuing rather than deciding. Fixes include explicit task state, step budgets, progress checks, better tool return values, and a final-answer condition that is tested in evals. The agent should fail closed with a useful reason rather than loop until a timeout. Candidates often tune the prompt before proving which observation caused the bad decision.

Expect these follow-ups

• What fields belong in an agent trace?
• How do you distinguish recoverable exploration from a loop?
• What eval would catch this before release?

As asked

Design the promotion process from trained model artefact to production deployment for a model registry used by five ML teams.

Sample answer outline

A good design treats the registry as more than a file store: it records artefacts, code version, data lineage, metrics, owners, approval state, and rollback pointers. Promotion gates should include offline evaluation, bias or safety checks where relevant, reproducibility metadata, and compatibility with the serving runtime. Strong answers describe staged deployment through shadow, canary, and full production rather than a single approve button. The candidate should also discuss permissions so researchers can register models but only authorised owners can promote them. The common trap is relying on manual screenshots of metrics instead of immutable records.

Expect these follow-ups

• What metadata is mandatory before a model can be registered?
• How do you roll back if the old model depends on an old feature definition?
• Who should be allowed to override a failed gate?

As asked

Design a self-hosted LLM platform that serves 20 internal teams, each with its own rate limits, model access controls, and cost attribution. Teams should not be able to starve each other. How do you design the routing, isolation, and billing layers?

Sample answer outline

A control plane issues API keys per team with embedded tier metadata. The serving gateway validates keys, enforces per-team rate limits (token bucket in Redis), tags requests with team_id for tracing, and routes to shared or dedicated GPU pools depending on team SLA tier. A billing pipeline reads the usage events from a message queue (Kafka or SQS), aggregates by team_id and model, and writes to a cost database queried by a dashboard. Isolation between teams is at the rate-limit layer, not the GPU layer, unless a team has a dedicated pool.

Expect these follow-ups

• What guarantees can you actually give about latency isolation between teams sharing GPU pools?
• How do you handle a team that needs a custom fine-tuned model not available on the shared pool?

As asked

Design a retrieval-augmented generation pipeline that embeds user queries, retrieves relevant chunks from a vector database, and generates answers, all under a 3-second end-to-end p99 latency at 5,000 requests per minute. What are the latency budget allocations and the failure modes to design around?

Sample answer outline

Latency budget: embedding (roughly 20ms), ANN search in vector DB (10 to 30ms), context assembly (5ms), LLM prefill and 100-token decode (500ms to 2s depending on model size). To hit 3s p99, the LLM is the dominant risk. Design decisions: cache embeddings for repeated queries, use a fast ANN index (HNSW in Qdrant or pgvector), set a hard cap on retrieved chunks to control context length, and use a smaller 13B model for retrieval-augmented answers. Failure modes: vector DB outage (fall back to no-retrieval generation), embedding service latency spike (short circuit to generation), LLM queue depth spike (shed load with 503).

Expect these follow-ups

• How would you design the embedding cache to handle near-duplicate queries that are semantically identical but phrased differently?
• What changes if the knowledge base updates in real time and cache invalidation matters?

As asked

Design a system that runs automatically on every model update, catches quality regressions before they reach production, and reports results in under 30 minutes. How do you balance coverage versus speed?

Sample answer outline

A fast eval tier (5 minutes) runs a stratified 500-question sample from internal golden set and a subset of MMLU, gated on a statistical significance test. If the fast tier passes, a full eval tier (25 minutes) runs the complete benchmark suite and shadow-traffic comparison. Results are posted to the PR as structured JSON with trend charts. The CI job is a GitHub Actions or Jenkins pipeline that spins up the model on a transient GPU runner, runs inference, and tears down. Eval score history is stored in a time-series database for trend detection.

Expect these follow-ups

• How do you handle the case where the fast tier passes but the full eval reveals a regression in a low-frequency task?
• What would you do if eval jobs are taking 2 hours because the GPU runner pool is saturated?

As asked

A team trains dozens of LoRA fine-tunes per week across multiple base models. Design the model registry and artifact store that tracks lineage, enables reproducibility, and integrates with the serving deployment pipeline.

Sample answer outline

The registry stores metadata (base model ID, training data version, hyperparameters, eval scores, Git commit) as records in a database (Postgres) linked to artifact storage (S3 or GCS) where the actual weights live. Each fine-tune gets a deterministic content hash as its primary ID to enable deduplication. The serving pipeline queries the registry to resolve a logical alias (e.g., customer-X-latest) to a specific artifact URI and loads it. Lineage is tracked by linking the fine-tune record to its parent base model record. MLflow or a custom registry both work; the key is making the API queryable by base model, eval score range, and training data version.

Expect these follow-ups

• How would you handle a scenario where an artifact in S3 is accidentally deleted but the registry still references it?
• What audit trail would you maintain to satisfy a customer asking which version of the model processed their data?

As asked

You want to run a statistically valid A/B test between model A and model B on live traffic, measuring both quality (LLM-as-judge) and downstream business metrics. Walk me through the experiment design, traffic splitting, sample size calculation, and guard rails.

Sample answer outline

Traffic splitting is done at the user session level (not request level) using a hash of the user ID for consistent assignment. Sample size is calculated using a power analysis on the expected effect size from offline eval results, aiming for 80% power at 5% significance. Quality is measured by a judge model scoring model outputs on a 1 to 5 scale, logged asynchronously after each request. Guard rails auto-stop the experiment and roll back to model A if error rate or latency exceeds predefined thresholds, regardless of quality results. Results are reported with confidence intervals, not just point estimates.

Expect these follow-ups

• How do you handle users who interact with both model A and model B across sessions due to logging into different devices?
• What would make you stop the A/B test early even before reaching statistical significance?

As asked

Design an intelligent routing system that classifies incoming queries and routes them to the cheapest model capable of handling them, with automatic escalation if the small model's output fails a quality gate. What are the latency and correctness tradeoffs?

Sample answer outline

A fast classifier (fine-tuned small model or heuristic on prompt features) assigns queries to small (7B), medium (13B), or large (70B) models. Simple factual queries go to 7B; reasoning and coding go to 70B. An optional quality gate runs a quick judge check on the 7B output and escalates to 13B or 70B if confidence is below threshold. The key tradeoffs: the classifier and quality gate add latency (20 to 50ms each), and escalation doubles the effective cost of borderline queries. The system must also handle the case where the classifier is wrong more often than the cost savings justify.

Expect these follow-ups

• How would you measure the accuracy of your complexity classifier without ground truth labels?
• At what percentage of escalation does the cost savings from small-model routing disappear?

As asked

Many API requests repeat similar prompts. Design a semantic caching layer that intercepts requests, finds semantically similar past requests using embedding search, and returns a cached response when similarity exceeds a threshold. What are the correctness risks?

Sample answer outline

The cache proxy embeds the incoming prompt, searches a vector store (Redis with VSS or Qdrant) for nearest neighbors, and returns a cached response if cosine similarity exceeds a tuned threshold (typically 0.97+). Correctness risks: semantically similar prompts can have meaningfully different correct answers (e.g., 'what is the date today' vs 'what was the date yesterday'), so time-sensitive or personalized prompts must be excluded. The cache must also store the full request context (system prompt hash, temperature, model version) as part of the cache key to avoid returning stale or wrong-model responses.

Expect these follow-ups

• How would you measure the false-positive rate of the cache in production without exposing users to wrong answers?
• What threshold would you set for a coding assistant versus a general QA assistant and why?

As asked

You have 8 vLLM nodes and want to share KV cache across them so that a request to node 2 can reuse prefilled cache from node 1. Design the distributed KV cache architecture, including the lookup protocol, the transfer mechanism, and the tradeoffs versus just scaling out prefix caching per node.

Sample answer outline

A distributed KV cache requires a shared index (Redis or a custom coordinator) that maps prefix hash to the node and block ID holding that cache. On a miss, the serving node requests the KV blocks from the cache node over RDMA or TCP (GPU-to-GPU transfer via NCCL or direct P2P if on NVLink fabric). The main tradeoffs versus per-node prefix caching: cross-node transfer adds 1 to 5ms latency and bandwidth overhead, the coordinator is a new SPOF, and consistency is hard (the cache node may have evicted the block). For most deployments, the per-node hit rate from routing (sticking a user to the same node) is sufficient and simpler.

Expect these follow-ups

• How does the routing layer need to change to maximize per-node hit rates without full distributed cache?
• What would you use as the transport layer for GPU-to-GPU KV block transfer to minimize latency?