ML engineer system design interview questions

As asked

Design the end-to-end architecture for a recommendation system that must surface the best 20 items from a catalog of 50 million, with sub-100ms total latency.

Sample answer outline

Stage 1 retrieval uses approximate nearest neighbor search with FAISS or ScaNN over user and item embeddings, returning 1,000 candidates in under 10ms. Stage 2 ranking uses a deeper model with interaction features computed on the 1,000 candidates, running in under 50ms on GPU. The retrieval index is rebuilt nightly and served from memory. User embeddings are precomputed and stored in a feature store. The serving graph materializes fresh context features per request and feeds them to both stages.

Expect these follow-ups

• How do you ensure fresh embeddings for a new user who signed up 10 minutes ago?
• What would you change in this design to support diversity constraints in the final 20 results?

As asked

Design the backend system that powers TikTok's For You Page. A user opens the app and within 200ms sees a ranked, personalized feed of short videos. Walk me through every major component from the client request to the videos appearing on screen.

Sample answer outline

A strong answer covers the candidate pipeline (retrieval from a candidate pool using collaborative filtering and content signals), the ranking model (a two-tower or multi-task neural model scoring candidates), a pre-computation layer that generates candidate sets offline for every user, and a serving layer that merges pre-ranked candidates with real-time signals at request time to meet the 200ms SLA. The candidate should also address cold start for new users and new videos, diversity injection to avoid filter bubbles, and how the feedback loop (watch time, likes, shares) flows back into retraining.

Expect these follow-ups

• How do you handle the cold start problem for a brand new creator who just posted their first video?
• How do you A/B test a new ranking feature without degrading the overall user experience?

As asked

Design a recommendation system that generates a personalized homepage for each of Netflix's 200 million subscribers. The page must load in under 200 milliseconds. Walk me through the offline training pipeline, the online serving layer, and how you handle cold-start for a new subscriber.

Sample answer outline

A strong answer separates the offline batch pipeline (collaborative filtering or two-tower neural models trained on viewing history) from the online serving layer (pre-computed embeddings cached in Redis, real-time context like time of day and device blended at request time). The candidate should cover feature stores, the ranking layer that merges multiple candidate generators, A/B experimentation on ranking, and a cold-start strategy using popularity signals and onboarding questions.

Expect these follow-ups

• How do you handle the explore-exploit tradeoff so you don't just keep recommending what someone already watched?
• A new title launches. How does it get recommended before it has viewing history?

As asked

Design the backend system for a conversational AI product similar to ChatGPT. Users can have many conversations, each with many messages. The system must support real-time streaming responses, be highly available, handle millions of concurrent users, and keep costs manageable. Walk me through your architecture.

Sample answer outline

A strong answer covers: stateless API servers behind a load balancer, conversation history stored in a durable database (Postgres or DynamoDB) with a conversation ID as the partition key, streaming responses via SSE from the model inference layer through to the client, a separate inference cluster with autoscaling, async logging of prompts and responses for billing and safety review, and caching strategies for repeated system prompts. They should also address the challenge of sticky sessions for streaming connections and how to handle model timeouts gracefully.

Expect these follow-ups

• How do you handle the situation where the model inference server dies mid-stream?
• How would you implement conversation sharing, where a user can share a link to a specific conversation with someone else?

As asked

Design the end-to-end system behind a feature like Discover Weekly: a personalized playlist of 30 songs generated once a week for roughly 600 million users. Walk me through the data pipeline, the model serving layer, and how you would keep recommendations fresh and relevant without regenerating everything from scratch each week.

Sample answer outline

A strong answer separates offline training from online serving and explains the batch pipeline that computes user embeddings from listening history. It covers the candidate generation stage (collaborative filtering, content embeddings), the ranking stage, and how diversity constraints prevent filter-bubble playlists. It discusses the cold-start problem for new users and new tracks, and addresses how to store and serve 600 million playlists efficiently using object storage plus a CDN-like precompute strategy.

Expect these follow-ups

• How would you handle a user who listens only in one genre versus someone with very eclectic taste?
• Discover Weekly refreshes weekly. How would you A/B test a change to the recommendation model without users noticing?

As asked

Design a real-time fraud detection system for a payments platform processing 10,000 transactions per second.

Sample answer outline

Pipeline: transaction event → enrichment (account history, device fingerprint, geo) → feature lookup from the online feature store → model inference (target: <50ms) → decision (approve / decline / step-up auth) → write back for offline retraining. Discuss feature freshness (some features are velocity-based: count in last 5 min), model versioning and shadow deployment, rules engine alongside the model for known-bad patterns, the feedback loop from charge-backs into labels, and the cold-start problem for new merchants/users.

Expect these follow-ups

• How do you measure model performance when labels arrive weeks later from chargebacks?
• How do you A/B test a fraud model when one variant blocks fraud and the other doesn't?
• How would you handle adversarial attackers probing the model?

As asked

Design a feature store that serves pre-computed features for a real-time ranking system handling 50,000 requests per second with a p99 latency budget of 10 milliseconds.

Sample answer outline

The online store is a distributed Redis cluster with consistent hashing. Feature values are serialized as dense float arrays (not JSON) to minimize parse time. A read-through cache at the serving tier reduces Redis load. Materialization pipelines run on Spark or Flink and push feature updates with configurable freshness windows. Feature versioning is tracked in a metadata catalog. Monitoring covers p99 latency per feature group, cache hit rate, and feature freshness lag.

Expect these follow-ups

• How do you handle a feature that must be computed in real time from the incoming request context?
• What happens to serving when the Redis cluster has a partial failure during a rolling upgrade?

As asked

Design a serving platform that allows dozens of ML teams to independently deploy, version, and scale their models without stepping on each other, with cost attribution per team.

Sample answer outline

The platform provides a model registry, container-based deployment via Kubernetes with namespace isolation per team, and an autoscaler that watches GPU utilization and request queue depth. Each team deploys through a CI/CD pipeline that validates model artifacts and enforces resource quotas. A sidecar handles metrics, logging, and cost attribution by tagging GPU-hours and request counts per model ID. A shared Triton cluster handles generic deployments while teams with special requirements get dedicated nodes.

Expect these follow-ups

• How do you handle a team whose model has a memory leak that gradually consumes GPU memory?
• What isolation mechanisms prevent a noisy-neighbor model from affecting another team's latency?

As asked

Design a framework that allows the ML team to safely run A/B experiments on production ranking models, with proper randomization, statistical analysis, and guardrail metrics.

Sample answer outline

User assignment uses deterministic hashing of the user ID and experiment ID to ensure consistent bucketing across sessions. The assignment service is a lightweight sidecar. Logging captures treatment assignments alongside predictions and outcomes in an event stream. The analysis layer computes primary and guardrail metrics with t-tests or bootstrap CI, and the system auto-stops experiments where a guardrail is violated. The experiment config is feature-flagged and can be paused without a code deploy.

Expect these follow-ups

• How do you handle novelty effects in an experiment where users behave differently in week 1 than week 4?
• What is Switchback testing and when would you use it over standard user-level A/B?

As asked

Design a monitoring system for a suite of 50 production ML models that detects data drift, prediction drift, and performance degradation, and pages the on-call engineer within 5 minutes of a problem.

Sample answer outline

Each model emits a prediction event with features and output to a Kafka topic. A streaming processor running Flink or Spark Streaming computes rolling distribution statistics per feature and prediction output, comparing against a reference window using PSI. When drift exceeds a configurable threshold, an alert fires to PagerDuty. Label-based performance metrics run in a slower batch pipeline once ground truth labels arrive. The monitoring UI shows heatmaps of feature drift over time per model.

Expect these follow-ups

• How would you reduce alert fatigue when many features drift simultaneously during a seasonal shift?
• What would a self-healing response to detected drift look like without human intervention?

As asked

Design a production inference system for a 70B parameter language model that must serve 1,000 concurrent users with a time-to-first-token under 500ms.

Sample answer outline

The 70B model requires tensor parallelism across at least 4 A100 80GB GPUs per replica. vLLM handles continuous batching and KV cache management with PagedAttention. Multiple replicas sit behind a load balancer that routes based on request queue depth. Prefill and decode are separated onto different replicas to avoid head-of-line blocking from long prompts. Autoscaling adds replicas based on queue depth with a minimum warm replica count to avoid cold start latency. INT8 quantization can double effective throughput at acceptable quality loss.

Expect these follow-ups

• How do you handle streaming responses where the user sees tokens as they are generated?
• What changes in this design for a mixture-of-experts 70B model versus a dense model?

As asked

Design a pipeline that automatically retrains a ranking model weekly, evaluates it against the current production model, and promotes it only if it passes quality gates.

Sample answer outline

An Airflow DAG triggers weekly, pulling the latest labeled data from the feature store, running training on the Kubernetes cluster, logging metrics to MLflow, and running offline evaluation against a held-out set. A comparison step checks whether the challenger beats the champion on AUC and key business metrics by a minimum margin. If gates pass, the pipeline creates a model registry entry tagged as candidate, runs a shadow deployment for 48 hours, then auto-promotes if no regressions appear. Failure at any gate sends a Slack notification and halts promotion.

Expect these follow-ups

• How do you prevent the pipeline from promoting a model that overfits the most recent week's data?
• What would you monitor during the 48-hour shadow period before promotion?

As asked

Design a serving system for a 70-billion parameter language model that needs to handle 500 requests per second with a p99 latency under 2 seconds. You have access to AMD MI300X GPUs. Walk me through your architecture.

Sample answer outline

A strong answer starts with capacity planning: a 70B bfloat16 model is about 140 GB, which fits in a single MI300X with its 192 GB HBM3. It then covers batching strategy (continuous batching vs static batching), a load balancer that routes to healthy GPU nodes, KV cache management for long-context requests, and auto-scaling triggers. The candidate should mention vLLM's paged attention or equivalent for AMD, model quantization (AWQ or GPTQ at int4) to increase throughput, and monitoring: GPU utilization, queue depth, and time-to-first-token as latency decomposition.

Expect these follow-ups

• How does paged attention reduce KV cache memory fragmentation compared to static allocation?
• At 500 RPS with an average 500-token output, how many MI300X nodes do you estimate needing, and show your math?

As asked

A customer wants to build an agent that uses Claude to autonomously manage their cloud infrastructure by reading logs, querying a database, and calling infrastructure APIs. Design the system architecture and describe the specific safeguards you would require before letting this agent take any write action.

Sample answer outline

A strong answer covers the multi-step agent loop: Claude receives context, reasons about needed tools, emits structured tool calls, the orchestrator executes them, results are fed back. For safeguards the answer should include a human-in-the-loop confirmation step before write/delete actions, a dry-run mode for destructive operations, rate limits on API calls, a permission model that restricts which resources the agent can touch, and an audit log of every action taken. The candidate should flag that agentic systems compound errors across steps and that the blast radius of a mistake is much larger than a single turn.

Expect these follow-ups

• How do you handle a situation where the agent gets stuck in a loop making the same failing tool call?
• How do you prevent the agent from being manipulated via injected content in the logs it reads?

As asked

You are building a feature that needs to personalize content recommendations based on a user's app usage patterns. How do you design it so that the personalization is meaningful while respecting Apple's privacy guidelines and avoiding sending raw behavioral data to a server?

Sample answer outline

The answer should center on on-device model inference: compute the embedding or feature vector locally, store only the aggregated preference signal (not raw events), and send only the final recommendation request or nothing at all if the ranking runs on device. Differential privacy can be applied to any aggregate that does leave the device. The candidate should mention App Tracking Transparency, SKAdNetwork for advertising use cases, and the principle of data minimization: collect only what is needed and keep it only as long as necessary. A strong answer distinguishes between personalization that requires a server call and personalization that genuinely can stay local.

Expect these follow-ups

• How do you A/B test an on-device recommendation model without logging individual user behavior?
• What changes if the user disables personalization in Settings?

As asked

ByteDance needs to moderate user-generated content across TikTok at billions of pieces of content per day. Design an automated content moderation system that identifies policy-violating content with high recall and manageable false positive rate.

Sample answer outline

A strong design separates pre-upload scanning (hash matching against known bad content databases like PhotoDNA), async post-upload ML classification (multi-modal models for video, audio, and text), and a human review queue for high-confidence borderline cases. The candidate should address the tradeoff between recall and precision, how to prioritize the human review queue by potential harm severity, how false positives affect creator trust, and how to handle appeals. Geo-specific policy variants and cultural context awareness are differentiators.

Expect these follow-ups

• How do you handle new categories of violating content that your model has never seen before?
• How do you measure the quality of your moderation system in production without seeing ground truth for every decision?

As asked

A machine learning platform team wants a feature store that can serve the same feature definitions for both offline model training from historical data and online real-time inference at low latency. Design the architecture, covering how you handle consistency between offline and online stores.

Sample answer outline

A strong answer describes two stores with a shared feature definition layer. The offline store (Delta Lake or similar) holds historical feature snapshots for point-in-time correct training set generation. The online store (Redis, DynamoDB, or Databricks Online Tables) holds the latest feature values for low-latency lookup. A materialization job computes features from raw data and writes to both stores on a schedule. Feature definitions are registered once in the feature catalog (Databricks Feature Store in this context), and training pipelines use a time-travel query to avoid label leakage. The candidate should discuss the challenge of maintaining consistency: the online store will lag the offline store by the materialization interval.

Expect these follow-ups

• How do you prevent training-serving skew when the online and offline stores compute features in different frameworks?
• How would you handle a feature that requires a sliding window aggregation, like the sum of purchases in the last 7 days?

As asked

When a customer places an order, DoorDash needs to assign a Dasher. The naive approach is first-available, but that ignores Dasher proximity, multi-order batching, restaurant prep time, and Dasher acceptance rates. How would you design the dispatch algorithm, and what data inputs does it need?

Sample answer outline

Strong answers frame this as an optimization problem with multiple objectives (minimize delivery time, maximize Dasher utilization, reduce unassigned orders) and explain why a greedy heuristic often beats ML models in latency-constrained dispatch. Candidates should mention the signal inputs (Dasher GPS, restaurant historical prep time, traffic ETA from a maps API, Dasher historical acceptance rate), batching logic for nearby orders, and a fallback when no Dasher is within a threshold radius.

Expect these follow-ups

• How does your algorithm handle a sudden surge where demand outpaces supply by 3x?
• How would you evaluate whether a new dispatch model is better than the current one in production?

As asked

Design a system that can index hundreds of millions of YouTube videos and return relevant search results and personalized recommendations within 100ms. Describe the data flow from a video upload to it appearing in search results, and separately, how recommendations are generated for a logged-in user.

Sample answer outline

A strong answer separates indexing (offline, async) from serving (online, latency-critical). Indexing involves extracting signals from title, description, captions, and engagement metrics and writing to an inverted index. Serving requires a candidate retrieval stage (ANN or inverted index lookup) followed by a ranking model using user and video features. For recommendations, the candidate should describe embedding-based retrieval and a two-tower model. Google interviewers appreciate candidates who mention the gap between indexing and serving latency and how eventual consistency is acceptable here.

Expect these follow-ups

• How do you prevent the system from only recommending popular videos and starving new content?
• How would you design the A/B testing layer on top of your recommendation system?

As asked

Intel's data center customers want to stream CPU and PCIe telemetry (power, temperature, utilisation) from thousands of Xeon servers to a cloud platform for real-time anomaly detection and dashboards. Design the end-to-end ingestion and processing system. Assume 10,000 servers each emitting 100 metrics every second.

Sample answer outline

A strong answer starts with back-of-envelope math: 10,000 x 100 = 1M metrics/sec, roughly 50-100MB/s depending on encoding. The candidate should propose an agent on each server (collectd/telegraf), a message bus (Kafka or Pulsar with partitioning by server ID), a stream processor (Flink or Kafka Streams) for anomaly detection, a time-series DB (InfluxDB, Prometheus remote write, or TimescaleDB) for storage, and a dashboard layer (Grafana). Discussion should include backpressure handling, schema evolution (Protobuf or Avro), and retention vs. cost tradeoffs.

Expect these follow-ups

• How would you handle a server cluster that sends bursts during a power event?
• If latency for anomaly alerts must be under 5 seconds end-to-end, what does that change in your design?

As asked

Design the system that generates 'People You May Know' recommendations for LinkedIn members. The system should surface relevant second-degree connections and people from similar companies or schools. How would you architect this at LinkedIn's scale?

Sample answer outline

The candidate should cover offline graph computation (mutual connections count, shared employer/school signals), an online serving layer that retrieves pre-computed candidates and re-ranks them in real time, deduplication against existing connections, and freshness constraints. They should discuss how to handle cold-start for new members and privacy constraints (blocked users, hidden profiles).

Expect these follow-ups

• How would you evaluate whether your recommendations are actually good?
• What signals beyond mutual connections would you add to improve relevance?

As asked

Netflix runs hundreds of simultaneous A/B tests across its product, from recommendation algorithms to UI layout to encoding parameters. Design the infrastructure that assigns users to test cells, logs exposure events, computes statistical significance, and prevents users from seeing conflicting experiments.

Sample answer outline

A strong answer covers a hashing-based assignment service that deterministically maps user IDs to experiment cells without network round-trips, a namespace collision system that prevents the same user from being in two mutually exclusive tests, server-side and client-side exposure logging at the moment of fork rather than at page load, and a statistics layer that computes confidence intervals and guards against peeking by requiring fixed-horizon or sequential analysis. The candidate should discuss how engineering teams integrate the SDK without slowing request paths.

Expect these follow-ups

• How do you handle a user who clears cookies or switches devices mid-experiment?
• An experiment shows 2% improvement in engagement but 0.1% increase in video errors. How do you decide whether to ship it?

As asked

You are designing a training system with eight A100 GPUs. When would you rely on NVLink versus PCIe for inter-GPU communication, and how does your choice affect the communication strategy in a data-parallel training job?

Sample answer outline

A strong answer explains that NVLink provides up to 600 GB/s bidirectional bandwidth between GPUs on the same node, versus around 64 GB/s for PCIe 4.0 x16, and is therefore critical for all-reduce operations in data-parallel training where gradient tensors can be tens of gigabytes. Within an NVLink domain such as an NVSwitch-connected DGX A100, all-reduce can run entirely on NVLink. For multi-node, InfiniBand with NCCL handles inter-node traffic. The choice affects whether ring-all-reduce, tree-all-reduce, or hierarchical strategies are efficient. The candidate should mention that NCCL automatically detects the topology and selects the optimal algorithm, and that NVLink also enables peer-to-peer memory access which reduces copy overhead.

Expect these follow-ups

• How does NCCL decide between ring and tree all-reduce and when is each better?
• What changes if you switch from data parallelism to tensor parallelism in terms of communication patterns?

As asked

Design the infrastructure to train a 70-billion parameter language model across 512 A100 GPUs. Cover how you would partition the model, handle communication, manage failures during a multi-day run, and monitor training health.

Sample answer outline

A strong answer covers: (1) 3D parallelism with tensor parallelism within a node over NVLink, pipeline parallelism across nodes, and data parallelism at the outermost level; (2) using Megatron-LM or DeepSpeed for the parallelism implementation; (3) gradient checkpointing and activation recomputation to fit in HBM2e; (4) mixed-precision training with BF16 on Ampere; (5) checkpointing to a distributed filesystem like Lustre or GPFS every N steps to tolerate node failures, and elastic restart capabilities; (6) monitoring via loss curves, gradient norms, and MFU (model FLOPS utilization) as a health signal, with DCGM for GPU health; (7) NCCL timeouts and watchdogs for straggler detection.

Expect these follow-ups

• How do you handle a node failure mid-pipeline without restarting the whole job?
• What is MFU and what range is acceptable for a well-tuned training run?

As asked

Design an inference service that can serve a 13-billion parameter LLM at 1000 requests per second with a p99 latency under 2 seconds on a fleet of A10G GPUs. Walk me through your architecture, batching strategy, and how you would handle latency vs throughput tradeoffs.

Sample answer outline

A strong answer covers: (1) continuous batching with a KV-cache-aware scheduler like vLLM's PagedAttention to maximize GPU utilization without head-of-line blocking; (2) quantizing the model to INT8 or FP8 with TRT-LLM to fit more of the model in 24 GB of VRAM per A10G; (3) a load balancer that routes by session affinity to hit warm KV caches; (4) speculative decoding with a small draft model to reduce per-step latency; (5) a request queue with max-wait time to form micro-batches without blowing the p99 SLA; (6) auto-scaling based on GPU memory utilization and queue depth. The candidate should discuss the tradeoff between throughput and latency explicitly: larger batches improve GPU utilization but raise first-token latency.

Expect these follow-ups

• How does PagedAttention differ from standard KV-cache management and why does it improve throughput?
• What happens to your architecture if the model must serve 128k-token context windows?