MLOps engineer system design interview questions

As asked

Design a feature store for a ride-sharing platform that needs to serve real-time features like driver location recency and surge pricing zone to an ETA model at under 5ms p99. The platform has 50 million rides per day.

Sample answer outline

A strong answer separates the online store (Redis Cluster for sub-millisecond point lookups keyed by entity ID) from the offline store (Hive or BigQuery for historical training data). It describes a dual-write pattern where the streaming pipeline (Kafka plus Flink) writes to both stores simultaneously, discusses point-in-time joins for training data, and handles the co-location of the feature server close to inference to minimize network hops. It should address entity freshness TTLs and schema evolution.

Expect these follow-ups

• How do you handle the case where a feature pipeline is delayed and online features are stale?
• How would you scale the Redis tier to handle 500,000 lookups per second?

As asked

Design a CI/CD pipeline for a team that trains and deploys 10 different ML models, where a code change to a shared feature library should trigger retraining of all affected models and gate deployment on evaluation metrics.

Sample answer outline

A strong answer describes a monorepo or dependency graph that maps library changes to affected model training jobs, a CI trigger that launches parallel training pipelines per affected model (Kubeflow or Airflow), evaluation gates that compare new model metrics against a baseline in the model registry, and a promotion step that only runs if all gates pass. Deployment should be canary per model, with rollback triggered by serving metrics degradation. GitOps (Argo CD) for Kubernetes manifests keeps the deployment state auditable.

Expect these follow-ups

• How do you prevent a single flaky model evaluation from blocking the deployment of all 10 models?
• How do you manage the different retraining schedules and data dependencies of 10 models in the same pipeline?

As asked

Design the promotion process from trained model artefact to production deployment for a model registry used by five ML teams.

Sample answer outline

A good design treats the registry as more than a file store: it records artefacts, code version, data lineage, metrics, owners, approval state, and rollback pointers. Promotion gates should include offline evaluation, bias or safety checks where relevant, reproducibility metadata, and compatibility with the serving runtime. Strong answers describe staged deployment through shadow, canary, and full production rather than a single approve button. The candidate should also discuss permissions so researchers can register models but only authorised owners can promote them. The common trap is relying on manual screenshots of metrics instead of immutable records.

Expect these follow-ups

• What metadata is mandatory before a model can be registered?
• How do you roll back if the old model depends on an old feature definition?
• Who should be allowed to override a failed gate?

As asked

Design a monitoring platform that tracks data drift, prediction drift, and business metric impact across 100 production models, surfaces alerts to model owners, and stores enough history to investigate incidents.

Sample answer outline

A strong answer describes a centralized log ingestion layer (Kafka) that collects prediction logs from all model servers, a batch compute layer (Spark or dbt) that computes drift statistics hourly against reference distributions stored in object storage, a time-series metrics store (Prometheus or InfluxDB) for trend visualization, and an alerting layer with per-model thresholds and owner routing. The design should discuss handling label delay for business metrics and preventing alert storm when a shared upstream data source fails.

Expect these follow-ups

• How do you detect when drift in model A is actually caused by a shared upstream data pipeline failure rather than a genuine distribution shift?
• What does the reference distribution represent, and how often do you update it?

As asked

Design a platform that allows 30 data scientists to run, track, compare, and share ML experiments, with support for large artifact storage, metric visualization, and reproducibility over a 2-year history.

Sample answer outline

A strong answer describes a centralized MLflow or Weights and Biases server with a PostgreSQL backend for run metadata, S3 for artifact storage with lifecycle policies for old runs, tag and search APIs for filtering experiments, and RBAC for team isolation. Reproducibility requires logging the exact Docker image digest, Git commit hash, and dataset version with each run. Compute costs are managed by setting max artifact retention and archiving runs older than 6 months to cheaper storage.

Expect these follow-ups

• How do you prevent experiment tracking metadata from becoming the bottleneck as run volume grows?
• How do you handle the case where two data scientists accidentally run the same experiment and want to compare?

As asked

Design a streaming feature pipeline that computes 5-minute rolling aggregates of user clickstream events and materializes them to an online feature store for a recommendation model. The event volume is 100,000 events per second.

Sample answer outline

A strong answer describes Kafka as the event bus (partitioned by user_id for ordered processing per user), Flink with a tumbling or sliding window aggregate operator computing counts and sums over 5-minute windows, exactly-once semantics via Flink checkpointing and Kafka transactions, and a sink to Redis with TTL set to slightly above the window duration. It should address late-arriving events (watermarks and allowed lateness), backpressure handling, and the failover scenario where Flink recovers from a checkpoint.

Expect these follow-ups

• What is the difference between a Flink tumbling window and a sliding window for this use case?
• How do you handle the cold-start problem for a new user who has no clickstream history?

As asked

Design a batch inference system that scores 1 billion customer records daily using a gradient boosting model, stores results in a data warehouse, and ensures the pipeline completes within a 4-hour window.

Sample answer outline

A strong answer describes reading records from Parquet partitions in S3 or GCS via Spark, distributing inference across 100+ executors with the model broadcast as a binary variable to each executor, writing predictions back to partitioned Parquet before loading to BigQuery or Snowflake. It should estimate that 1 billion records at 1ms per record requires 1 million core-seconds, so roughly 500 executors over 30 minutes. It addresses the practical size limit for Spark broadcasts (driver heap constrains this; most teams treat anything above 1 to 2 GB as too large for a reliable broadcast), data skew in partitions, and checkpoint strategy for failure recovery within the window.

Expect these follow-ups

• How do you handle a model file that is 2GB and too large to broadcast in Spark?
• What SLA breach mitigation strategy do you use if the pipeline is running 2 hours behind at the midpoint?

As asked

Your team is deploying a 70B parameter LLM for a customer-facing chat application. Design the serving infrastructure to handle 500 concurrent users with a time-to-first-token of under 2 seconds.

Sample answer outline

A strong answer covers tensor parallelism across 8 A100 80GB GPUs to fit the 70B model (roughly 140GB in FP16), using vLLM with PagedAttention for memory-efficient KV cache management and continuous batching to maximize GPU utilization. It describes a load balancer routing to a pool of GPU nodes, autoscaling based on queue depth or pending token count, and a request queue to handle burst traffic. Quantization (INT8 or AWQ) can reduce memory footprint to fit on fewer GPUs at a latency cost.

Expect these follow-ups

• What is PagedAttention and why does it improve GPU memory utilization compared to fixed KV cache allocation?
• How do you monitor GPU memory fragmentation in a vLLM deployment?

As asked

You need to serve a fraud detection model globally with a p99 latency SLA of 100ms from any region, while ensuring all model versions are consistent across regions. How do you design the serving infrastructure?

Sample answer outline

A strong answer describes deploying model servers in multiple regions (e.g., us-east, eu-west, ap-southeast), using a CDN or Anycast DNS for request routing to the nearest region, and a centralized model registry (S3 with cross-region replication) that pushes model artifacts to regional object stores. Deployment orchestration ensures all regions flip to a new version within a deployment window before traffic shifts. It should discuss the data residency implications for feature stores in regions with strict data sovereignty laws.

Expect these follow-ups

• How do you handle the case where a model deployment fails in one region but succeeds in others?
• What is the latency budget breakdown between network, feature retrieval, and inference for a 100ms SLA?

As asked

A financial firm needs a model governance catalog that tracks every production model, its training data lineage, evaluation results, approval chain, and audit logs for regulators. Design the data model and system architecture.

Sample answer outline

A strong answer models entities: Model (version, name, owner, risk tier), TrainingRun (data sources, metrics, git commit), DeploymentEvent (timestamp, environment, approver), AuditLog (immutable append-only event stream). The system provides a REST API for queries and a UI for model owners and risk officers, with role-based access control. An immutable audit trail uses an append-only Postgres table or a log service like AWS CloudTrail. Integration with the model registry auto-populates metadata on model promotion.

Expect these follow-ups

• How do you handle models that use third-party API predictions as features? Where does that lineage live?
• What does SR 11-7 (the Fed's model risk management guidance) require that most MLflow setups don't provide out of the box?