MLOps engineer role-specific interview questions

As asked

Describe how you manage model stage transitions in MLflow's model registry, from staging to production, including the checks and automation you put around that workflow.

Sample answer outline

A strong answer covers how MLflow 1.x used named lifecycle stages (Staging, Production, Archived) accessed via the MlflowClient API or the UI, and that MLflow 2.x deprecated these fixed stages in favour of model aliases (e.g., @champion, @challenger) and model tags, which offer more flexibility. For legacy setups, it covers what automated gates run before promoting to Production: metric thresholds, challenger vs champion A/B results, data validation checks, and approval gates in CI. It should mention webhook integrations or event hooks that trigger downstream pipelines when a model alias is updated or a stage transition occurs, and note that teams still on MLflow 1.x stages should plan migration to aliases.

Expect these follow-ups

• How do you roll back a production model in MLflow if inference latency spikes after promotion?
• What do you store in a run's logged artifacts beyond the model weights itself?

As asked

A data scientist hands you a PyTorch training job that needs to run on Kubernetes. Walk me through how you configure resource requests and limits for GPU, CPU, and memory, and what can go wrong if you get them wrong.

Sample answer outline

A strong answer covers requesting nvidia.com/gpu as an extended resource (limits equal requests for GPUs, no partial GPU allocation by default), setting CPU and memory requests based on data loading worker count and model size, and why setting memory limit much higher than request causes OOMKilled pods under pressure. It should mention using DCGM metrics to verify actual GPU utilization, and node taints plus tolerations to restrict GPU nodes to training workloads.

Expect these follow-ups

• How does NVIDIA's device plugin expose MIG slices to Kubernetes pods?
• What happens to a training job pod when the node it runs on runs out of memory?

As asked

Walk me through how you would do a canary deployment of a new recommendation model version, including what metrics you watch, how long you hold the canary, and what triggers a rollback.

Sample answer outline

A strong answer describes routing 5 to 10 percent of traffic to the new model version, monitoring both technical metrics (latency p99, error rate, feature lookup failures) and business metrics (click-through rate, conversion), using statistical tests to compare the two slices, and setting a minimum window of 24 to 48 hours to capture daily traffic patterns. Rollback triggers should include automated rules for latency spikes above a threshold and manual override for business metric degradation.

Expect these follow-ups

• How do you handle the case where the canary serves a non-representative user segment?
• What infrastructure changes are needed in your serving layer to split traffic between two model versions?

As asked

Explain the difference between data drift and concept drift, give a concrete example of each in a credit scoring context, and describe how your monitoring strategy differs for each.

Sample answer outline

Data drift means the input feature distribution P(X) changes, for example average applicant income rising 20% after an economic upturn, while the true relationship between features and default risk is unchanged. Concept drift means the conditional P(Y|X) changes, for example a regulatory change that alters what constitutes a default. Data drift is detectable by monitoring input features with PSI or KS tests; concept drift requires labeled outcome data and is only detectable with a lag equal to the label delay.

Expect these follow-ups

• What is virtual concept drift, and how does it differ from real concept drift?
• How do you monitor for concept drift when your label delay is 90 days?

As asked

Your team has dozens of drift dashboards and nobody trusts the alerts. How would you redesign model drift monitoring?

Sample answer outline

Start by separating input drift, prediction drift, label drift, and business metric degradation because each has a different owner and response. Alert only on monitored signals with an agreed action, such as retraining, feature pipeline investigation, or business review. Use population slices for important cohorts rather than only global averages, since failures often hide in small but valuable segments. Strong candidates discuss delayed labels, seasonality, alert thresholds, and annotation of known events such as campaigns or outages. A weak answer adds more charts instead of reducing noise and tying alerts to playbooks.

Expect these follow-ups

• What drift signal would you page on?
• How do you monitor a model whose labels arrive 30 days later?
• When is drift expected and harmless?

As asked

Walk me through what training-serving skew actually means in the context of a feature store, and how you would detect and prevent it in a production system.

Sample answer outline

A strong answer distinguishes the offline store (batch, historical, used for training) from the online store (low-latency, current, used at inference time) and explains how diverging transformation logic or stale values cause skew. It covers point-in-time correct joins for training, feature logging at inference time, and monitoring distribution drift between training features and live traffic features.

Expect these follow-ups

• How would you backfill an online store from your offline store without causing skew?
• What does point-in-time correctness mean in Feast or Tecton, and what goes wrong if you skip it?

As asked

How does Kubeflow Pipelines decide whether to use a cached execution for a component, and when would you want to disable or invalidate that cache?

Sample answer outline

A strong answer explains that KFP computes a cache key from the component image digest, the input arguments, and the pipeline SDK version, and stores the result in a metadata store backed by ML Metadata (MLMD). Disabling the cache matters when upstream data changes without changing the input signature (e.g., the same date partition path points to new data), or when components have non-deterministic external side effects.

Expect these follow-ups

• How does MLMD store artifact lineage, and how do you query it programmatically?
• What is the difference between a KFP V1 and V2 pipeline artifact URI?

As asked

You are setting up automated drift monitoring for 200 numeric features going into a fraud detection model. Compare using Population Stability Index against Kolmogorov-Smirnov test, and tell me which you would pick and why.

Sample answer outline

A strong answer explains PSI buckets the reference and current distributions and measures the weighted log-ratio (0 to 0.1 stable, 0.1 to 0.2 moderate shift, above 0.2 significant), making it interpretable and threshold-friendly for business SLOs. KS is a nonparametric two-sample test that is more sensitive to distributional shape changes but gives a p-value that is harder to act on operationally. The candidate should note PSI requires discretization choices and fails on heavy-tailed distributions; KS is distribution-free but sensitive to sample size.

Expect these follow-ups

• How do you handle categorical features in drift monitoring, since PSI and KS are designed for continuous data?
• At what cadence would you compute these statistics, and how do you avoid alert fatigue?

As asked

How would you use Great Expectations to validate training data before a model retraining run kicks off, and what happens when a validation suite fails mid-pipeline?

Sample answer outline

A strong answer describes creating an Expectation Suite that checks schema, null rates, value ranges, and distribution statistics on the training dataset, running it as a pipeline step that checkpoints results to a Data Docs store, and failing the pipeline with a non-zero exit code when critical expectations fail. It should distinguish hard failures (null rate above 10% in a required feature) from warnings (distribution shift flagged but pipeline continues).

Expect these follow-ups

• How do you generate an initial expectation suite automatically from a reference dataset?
• Where would you store the expectation suite and Data Docs in a team environment?

As asked

A data scientist reports that rerunning the same training code on the same data gives different model weights each time. Walk me through every layer you would inspect to find and eliminate sources of non-determinism.

Sample answer outline

A strong answer covers: Python random seed, NumPy seed, PyTorch or TensorFlow global seed and CUDA deterministic flags (torch.use_deterministic_algorithms), non-deterministic CUDA ops like atomics in scatter operations, non-deterministic data loader shuffling with a fixed worker seed, floating point accumulation order changes across different GPU counts, and dependency version pinning in Docker images. It should mention that full determinism on GPU often sacrifices performance.

Expect these follow-ups

• What is the CUBLAS_WORKSPACE_CONFIG environment variable and when do you need it?
• How do you log and track all these seed values in your experiment tracker?

As asked

Your team is starting from scratch and needs an orchestration layer for ML pipelines. Compare Apache Airflow and Kubeflow Pipelines on the dimensions that matter most for ML workloads. What would make you pick one over the other?

Sample answer outline

A strong answer notes that Airflow is a general-purpose scheduler with a Python DAG API and good ecosystem integrations but lacks native ML artifact tracking and caching; KFP is ML-specific, integrates with MLMD, has built-in caching, and generates containerized steps automatically. Airflow wins when the team already uses it for data engineering or when ML steps are minority workloads; KFP wins when the primary use case is recurring model training with experiment tracking and lineage requirements.

Expect these follow-ups

• How does Airflow's KubernetesPodOperator compare to KFP's native Kubernetes execution?
• What does Prefect or Argo Workflows offer that neither Airflow nor KFP does?

As asked

Explain how DVC tracks large dataset and model versions without storing them in Git, and walk me through what happens under the hood when you run dvc push and dvc pull.

Sample answer outline

A strong answer explains that DVC stores a .dvc file in Git containing the MD5 hash of the tracked file or directory, while the actual data lives in a content-addressed cache under .dvc/cache or a remote (S3, GCS, Azure Blob). On push, DVC computes hashes for files not already in the remote and uploads them; on pull, it downloads by hash and reconstructs the workspace. It should mention the cache-type symlink default and why it matters for performance on large datasets.

Expect these follow-ups

• How do you share a DVC remote between multiple team members with proper access control?
• What is dvc run and how does it enable pipeline stage caching?

As asked

Your online feature store lookup is adding 40ms to inference latency, violating your SLA. Walk me through the architectural and configuration changes you would investigate.

Sample answer outline

A strong answer covers: switching from Redis Cluster to a local Redis Replica for the inference service to avoid cross-AZ network hops, batching feature lookups for all entities in one request, pre-serializing feature vectors on write (avoid per-request deserialization), checking connection pool exhaustion in the client, and potentially embedding the feature store client in-process rather than via HTTP. It should also mention whether the store is actually reading from the wrong tier (e.g., hitting the offline store by mistake).

Expect these follow-ups

• How does Feast's feature server implement caching, and what are the tradeoffs?
• Would you consider a local in-process cache at the model server? What consistency problems does that introduce?

As asked

Describe how you would set up shadow mode for a new model version before exposing it to any real users, including how requests are duplicated and how you evaluate the shadow model.

Sample answer outline

A strong answer describes mirroring live traffic to the shadow model asynchronously (so shadow latency does not affect real users), logging shadow predictions to a separate sink, and comparing prediction distributions between champion and shadow offline. It should note that shadow mode catches runtime errors, schema mismatches, and prediction distribution shifts before real user impact, but it cannot measure actual business metric impact since shadow predictions are discarded.

Expect these follow-ups

• How do you mirror traffic without adding latency to the live path in Envoy or Istio?
• What would make you skip shadow mode and go straight to canary?

As asked

How do you expose custom ML metrics, like prediction confidence distribution or class imbalance in live predictions, from a Python model server to Prometheus?

Sample answer outline

A strong answer describes using the prometheus_client library to register Histogram or Summary metrics for confidence scores and Counter metrics for per-class prediction counts, exposing them via a /metrics HTTP endpoint, and scraping with a Prometheus ServiceMonitor in Kubernetes. It should mention the cardinality problem of labeling by predicted class when the label space is large, and how to use recording rules to pre-aggregate expensive queries.

Expect these follow-ups

• What is the difference between a Prometheus Histogram and Summary, and which would you use for inference latency?
• How do you correlate a spike in a business dashboard with a specific model version change?

As asked

You need to package a model server so it can be deployed to dev, staging, and production Kubernetes clusters with different resource quotas and model versions. How do you structure the Helm chart and values files?

Sample answer outline

A strong answer covers a base values.yaml with defaults, per-environment overlay files (values.dev.yaml, values.prod.yaml), templating resource requests and model artifact URIs as values, using Helm hooks for pre-upgrade readiness checks, and a named release per environment. It should mention using Helm's lookup function or external secrets operator to pull model registry URIs at deploy time rather than hardcoding them.

Expect these follow-ups

• How do you pin an exact model version in the Helm values and make it auditable in Git?
• What is the difference between Helm and Kustomize for this use case?

As asked

Compare event-driven model retraining triggered by detected drift against a fixed weekly retraining schedule. When is each appropriate, and what are the failure modes of each approach?

Sample answer outline

A strong answer notes that event-driven retraining reacts faster to genuine concept drift but risks training on noisy or anomalous data (a data pipeline outage looks like drift), can trigger retraining in bursts that overload compute, and requires a reliable drift signal with low false-positive rate. Scheduled retraining is predictable and easier to operate but wastes compute when there is no drift and is slow to respond to sudden distribution shifts. Hybrid approaches (scheduled as a baseline, triggered immediately on severe drift) are common.

Expect these follow-ups

• How do you prevent a cascade of triggered retraining jobs when drift is detected simultaneously across many features?
• What minimum dataset size do you require before allowing a triggered retraining to proceed?

As asked

You need to run inference on 500 million records daily using a PyTorch model. Compare running batch inference with Spark UDFs against using Ray for distributed inference, and explain which you would choose and why.

Sample answer outline

A strong answer notes that Spark UDFs for PyTorch inference suffer from Python serialization overhead, lack GPU sharing within an executor, and PyTorch models must be serialized per task or loaded with a broadcast variable, making GPU utilization poor. Ray Data with Ray actors loads the model once per worker, supports GPU batch processing natively, and integrates with Ray's object store for zero-copy data passing. Spark wins if the rest of the pipeline is already in Spark and model inference is a small fraction of compute; Ray wins for GPU-heavy inference at scale.

Expect these follow-ups

• How do you handle Pandas/Arrow interop between Ray Data and your upstream Spark pipeline?
• What is the maximum batch size you can pass through a Spark UDF when the model is a 7B parameter transformer?

As asked

Training jobs on Kubernetes need access to a private S3 bucket, a feature store API key, and a model registry token. Walk me through how you manage these secrets securely without hardcoding them in pipeline code or Docker images.

Sample answer outline

A strong answer covers using Kubernetes Secrets mounted as environment variables or files, backed by an external secrets manager (HashiCorp Vault or AWS Secrets Manager) via the External Secrets Operator or Vault Agent Injector. For S3, it prefers IRSA (IAM Roles for Service Accounts) over access keys entirely. It should mention never baking secrets into Docker layers, scanning images for secret leakage in CI, and rotating secrets without restarting pods by using refreshInterval in External Secrets.

Expect these follow-ups

• What is IRSA and how does it bind a Kubernetes service account to an AWS IAM role?
• How do you audit which training jobs accessed which secrets, and when?

As asked

Your team spends $50,000 per month on on-demand GPU instances for training. How would you migrate training workloads to spot instances, and what fault tolerance mechanisms do you put in place?

Sample answer outline

A strong answer covers checkpointing model state every N steps to durable storage (S3 or GCS), implementing checkpoint-restart logic in training code, using Kubernetes pod disruption budgets and node termination handlers to detect preemption events and trigger a clean checkpoint before the pod is killed, and using mixed on-demand and spot node groups with Kubernetes Cluster Autoscaler. It should note that spot interruption rates vary by instance type and region, and diversifying across instance families reduces interruption risk.

Expect these follow-ups

• How does AWS's spot instance interruption notice work, and how much time does it give you to checkpoint?
• How do you handle distributed multi-node training (PyTorch DDP) when one node is preempted mid-run?

As asked

You deploy a churn prediction model. Actual churn outcomes are known 30 days after prediction. How do you architect the system to collect ground truth and feed it back into model evaluation and retraining?

Sample answer outline

A strong answer describes logging every prediction with a prediction ID and the feature snapshot at inference time to an event store, then joining outcomes 30 days later via a scheduled Spark or dbt job that matches customer IDs to churn events, producing a labeled dataset. This labeled dataset feeds automated model evaluation jobs that compute AUC and calibration on recent cohorts and write results to the experiment tracker. The retraining pipeline reads from this joined table rather than raw event logs.

Expect these follow-ups

• How do you handle selection bias when only a subset of customers have visible outcomes (e.g., customers who churned before the 30-day window)?
• What schema changes would break the prediction-to-label join, and how do you guard against them?

As asked

You have three models: a TensorFlow SavedModel, a PyTorch TorchScript model, and a custom ensemble that chains both with a preprocessing step. How do you configure Triton to serve all three, and what backend does each use?

Sample answer outline

A strong answer names the tensorflow backend for the SavedModel, the pytorch_libtorch backend for TorchScript, and the ensemble scheduler for chaining them via a config.pbtxt that defines the ensemble pipeline with routing and tensor names. It should mention the Python backend for the preprocessing step, discuss batch size configuration per model, and note that ensemble models do not run on GPU directly but route tensors between constituent models.

Expect these follow-ups

• How does Triton's dynamic batching work, and what latency vs throughput tradeoff does it involve?
• How would you A/B test two versions of the same model in Triton without downtime?

As asked

Explain how ML Metadata stores artifact lineage, what types of nodes exist in its graph, and how you would use it to trace which training data version produced a specific deployed model.

Sample answer outline

A strong answer describes MLMD's three node types: Artifact (files or values produced or consumed), Execution (pipeline steps), and Context (grouping such as pipeline run or experiment), and the two edge types: Event (linking Artifact to Execution as input or output) and Attribution/Association (linking to Context). Tracing a deployed model back to its training data means traversing the execution graph upstream from the model artifact through training and data preprocessing executions to the dataset artifact.

Expect these follow-ups

• How does MLMD relate to Vertex AI's managed lineage tracking?
• What are the performance limits of MLMD at scale, and how have you worked around them?

As asked

You are serving a transformer model with Ray Serve and traffic spikes 10x during peak hours. Walk me through how you configure autoscaling and what knobs you tune to avoid cold-start latency.

Sample answer outline

A strong answer covers Ray Serve's autoscaling config fields: target_num_ongoing_requests_per_replica as the primary trigger, min_replicas and max_replicas bounds, and upscale_delay_s to prevent thrashing. Avoiding cold start means setting min_replicas above zero, pre-loading model weights in the constructor rather than the handler, and using a warm pool of replicas. It should also mention Ray's object store for sharing large model state between replicas and the tradeoff with GPU memory.

Expect these follow-ups

• How does Ray Serve handle a request that arrives while scaling is in progress?
• What is the difference between Ray Serve and Seldon Core for model serving at scale?

As asked

A team trains a Spark ML Pipeline that includes a StringIndexer, a StandardScaler, and a GBTClassifier. How do you save that pipeline and serve it outside of Spark in a low-latency REST API?

Sample answer outline

A strong answer notes that Spark ML pipelines save to Parquet-backed metadata directories, which are Spark-dependent and cannot be loaded outside Spark. To serve outside Spark, the candidate should mention exporting the fitted transformers to a portable format: using MLeap to serialize the pipeline to a MLeap bundle, or replicating the preprocessing logic in Python/Java with fitted parameters extracted manually. It should note that MLeap supports the most common transformers but has gaps, and that feature computation in Spark offline vs a Python server reintroduces skew risk.

Expect these follow-ups

• How does MLeap differ from ONNX for exporting Spark ML models?
• What would you do if a custom Spark Transformer has no MLeap equivalent?