Question 1

Describe how you manage model stage transitions in MLflow's model registry, from staging to production, including the checks and automation you put around that workflow.

Accepted Answer

A strong answer covers how MLflow 1.x used named lifecycle stages (Staging, Production, Archived) accessed via the MlflowClient API or the UI, and that MLflow 2.x deprecated these fixed stages in favour of model aliases (e.g., @champion, @challenger) and model tags, which offer more flexibility. For legacy setups, it covers what automated gates run before promoting to Production: metric thresholds, challenger vs champion A/B results, data validation checks, and approval gates in CI. It should mention webhook integrations or event hooks that trigger downstream pipelines when a model alias is updated or a stage transition occurs, and note that teams still on MLflow 1.x stages should plan migration to aliases.

Question 2

A data scientist hands you a PyTorch training job that needs to run on Kubernetes. Walk me through how you configure resource requests and limits for GPU, CPU, and memory, and what can go wrong if you get them wrong.

Accepted Answer

A strong answer covers requesting nvidia.com/gpu as an extended resource (limits equal requests for GPUs, no partial GPU allocation by default), setting CPU and memory requests based on data loading worker count and model size, and why setting memory limit much higher than request causes OOMKilled pods under pressure. It should mention using DCGM metrics to verify actual GPU utilization, and node taints plus tolerations to restrict GPU nodes to training workloads.

Question 3

Walk me through how you would do a canary deployment of a new recommendation model version, including what metrics you watch, how long you hold the canary, and what triggers a rollback.

Accepted Answer

A strong answer describes routing 5 to 10 percent of traffic to the new model version, monitoring both technical metrics (latency p99, error rate, feature lookup failures) and business metrics (click-through rate, conversion), using statistical tests to compare the two slices, and setting a minimum window of 24 to 48 hours to capture daily traffic patterns. Rollback triggers should include automated rules for latency spikes above a threshold and manual override for business metric degradation.

Question 4

Explain the difference between data drift and concept drift, give a concrete example of each in a credit scoring context, and describe how your monitoring strategy differs for each.

Accepted Answer

Data drift means the input feature distribution P(X) changes, for example average applicant income rising 20% after an economic upturn, while the true relationship between features and default risk is unchanged. Concept drift means the conditional P(Y|X) changes, for example a regulatory change that alters what constitutes a default. Data drift is detectable by monitoring input features with PSI or KS tests; concept drift requires labeled outcome data and is only detectable with a lag equal to the label delay.

Question 5

Your team has dozens of drift dashboards and nobody trusts the alerts. How would you redesign model drift monitoring?

Accepted Answer

Start by separating input drift, prediction drift, label drift, and business metric degradation because each has a different owner and response. Alert only on monitored signals with an agreed action, such as retraining, feature pipeline investigation, or business review. Use population slices for important cohorts rather than only global averages, since failures often hide in small but valuable segments. Strong candidates discuss delayed labels, seasonality, alert thresholds, and annotation of known events such as campaigns or outages. A weak answer adds more charts instead of reducing noise and tying alerts to playbooks.

Question 6

Walk me through what training-serving skew actually means in the context of a feature store, and how you would detect and prevent it in a production system.

Accepted Answer

A strong answer distinguishes the offline store (batch, historical, used for training) from the online store (low-latency, current, used at inference time) and explains how diverging transformation logic or stale values cause skew. It covers point-in-time correct joins for training, feature logging at inference time, and monitoring distribution drift between training features and live traffic features.

Question 7

How does Kubeflow Pipelines decide whether to use a cached execution for a component, and when would you want to disable or invalidate that cache?

Accepted Answer

A strong answer explains that KFP computes a cache key from the component image digest, the input arguments, and the pipeline SDK version, and stores the result in a metadata store backed by ML Metadata (MLMD). Disabling the cache matters when upstream data changes without changing the input signature (e.g., the same date partition path points to new data), or when components have non-deterministic external side effects.

Question 8

You are setting up automated drift monitoring for 200 numeric features going into a fraud detection model. Compare using Population Stability Index against Kolmogorov-Smirnov test, and tell me which you would pick and why.

Accepted Answer

A strong answer explains PSI buckets the reference and current distributions and measures the weighted log-ratio (0 to 0.1 stable, 0.1 to 0.2 moderate shift, above 0.2 significant), making it interpretable and threshold-friendly for business SLOs. KS is a nonparametric two-sample test that is more sensitive to distributional shape changes but gives a p-value that is harder to act on operationally. The candidate should note PSI requires discretization choices and fails on heavy-tailed distributions; KS is distribution-free but sensitive to sample size.

Question 9

How would you use Great Expectations to validate training data before a model retraining run kicks off, and what happens when a validation suite fails mid-pipeline?

Accepted Answer

A strong answer describes creating an Expectation Suite that checks schema, null rates, value ranges, and distribution statistics on the training dataset, running it as a pipeline step that checkpoints results to a Data Docs store, and failing the pipeline with a non-zero exit code when critical expectations fail. It should distinguish hard failures (null rate above 10% in a required feature) from warnings (distribution shift flagged but pipeline continues).

Question 10

A data scientist reports that rerunning the same training code on the same data gives different model weights each time. Walk me through every layer you would inspect to find and eliminate sources of non-determinism.

Accepted Answer

A strong answer covers: Python random seed, NumPy seed, PyTorch or TensorFlow global seed and CUDA deterministic flags (torch.use_deterministic_algorithms), non-deterministic CUDA ops like atomics in scatter operations, non-deterministic data loader shuffling with a fixed worker seed, floating point accumulation order changes across different GPU counts, and dependency version pinning in Docker images. It should mention that full determinism on GPU often sacrifices performance.

Question 11

Your team is starting from scratch and needs an orchestration layer for ML pipelines. Compare Apache Airflow and Kubeflow Pipelines on the dimensions that matter most for ML workloads. What would make you pick one over the other?

Accepted Answer

A strong answer notes that Airflow is a general-purpose scheduler with a Python DAG API and good ecosystem integrations but lacks native ML artifact tracking and caching; KFP is ML-specific, integrates with MLMD, has built-in caching, and generates containerized steps automatically. Airflow wins when the team already uses it for data engineering or when ML steps are minority workloads; KFP wins when the primary use case is recurring model training with experiment tracking and lineage requirements.

Question 12

Explain how DVC tracks large dataset and model versions without storing them in Git, and walk me through what happens under the hood when you run dvc push and dvc pull.

Accepted Answer

A strong answer explains that DVC stores a .dvc file in Git containing the MD5 hash of the tracked file or directory, while the actual data lives in a content-addressed cache under .dvc/cache or a remote (S3, GCS, Azure Blob). On push, DVC computes hashes for files not already in the remote and uploads them; on pull, it downloads by hash and reconstructs the workspace. It should mention the cache-type symlink default and why it matters for performance on large datasets.

Question 13

Your online feature store lookup is adding 40ms to inference latency, violating your SLA. Walk me through the architectural and configuration changes you would investigate.

Accepted Answer

A strong answer covers: switching from Redis Cluster to a local Redis Replica for the inference service to avoid cross-AZ network hops, batching feature lookups for all entities in one request, pre-serializing feature vectors on write (avoid per-request deserialization), checking connection pool exhaustion in the client, and potentially embedding the feature store client in-process rather than via HTTP. It should also mention whether the store is actually reading from the wrong tier (e.g., hitting the offline store by mistake).

Question 14

Describe how you would set up shadow mode for a new model version before exposing it to any real users, including how requests are duplicated and how you evaluate the shadow model.

Accepted Answer

A strong answer describes mirroring live traffic to the shadow model asynchronously (so shadow latency does not affect real users), logging shadow predictions to a separate sink, and comparing prediction distributions between champion and shadow offline. It should note that shadow mode catches runtime errors, schema mismatches, and prediction distribution shifts before real user impact, but it cannot measure actual business metric impact since shadow predictions are discarded.

Question 15

How do you expose custom ML metrics, like prediction confidence distribution or class imbalance in live predictions, from a Python model server to Prometheus?

Accepted Answer

A strong answer describes using the prometheus_client library to register Histogram or Summary metrics for confidence scores and Counter metrics for per-class prediction counts, exposing them via a /metrics HTTP endpoint, and scraping with a Prometheus ServiceMonitor in Kubernetes. It should mention the cardinality problem of labeling by predicted class when the label space is large, and how to use recording rules to pre-aggregate expensive queries.

Question 16

You need to package a model server so it can be deployed to dev, staging, and production Kubernetes clusters with different resource quotas and model versions. How do you structure the Helm chart and values files?

Accepted Answer

A strong answer covers a base values.yaml with defaults, per-environment overlay files (values.dev.yaml, values.prod.yaml), templating resource requests and model artifact URIs as values, using Helm hooks for pre-upgrade readiness checks, and a named release per environment. It should mention using Helm's lookup function or external secrets operator to pull model registry URIs at deploy time rather than hardcoding them.

Question 17

Compare event-driven model retraining triggered by detected drift against a fixed weekly retraining schedule. When is each appropriate, and what are the failure modes of each approach?

Accepted Answer

A strong answer notes that event-driven retraining reacts faster to genuine concept drift but risks training on noisy or anomalous data (a data pipeline outage looks like drift), can trigger retraining in bursts that overload compute, and requires a reliable drift signal with low false-positive rate. Scheduled retraining is predictable and easier to operate but wastes compute when there is no drift and is slow to respond to sudden distribution shifts. Hybrid approaches (scheduled as a baseline, triggered immediately on severe drift) are common.

Question 18

You need to run inference on 500 million records daily using a PyTorch model. Compare running batch inference with Spark UDFs against using Ray for distributed inference, and explain which you would choose and why.

Accepted Answer

A strong answer notes that Spark UDFs for PyTorch inference suffer from Python serialization overhead, lack GPU sharing within an executor, and PyTorch models must be serialized per task or loaded with a broadcast variable, making GPU utilization poor. Ray Data with Ray actors loads the model once per worker, supports GPU batch processing natively, and integrates with Ray's object store for zero-copy data passing. Spark wins if the rest of the pipeline is already in Spark and model inference is a small fraction of compute; Ray wins for GPU-heavy inference at scale.

Question 19

Training jobs on Kubernetes need access to a private S3 bucket, a feature store API key, and a model registry token. Walk me through how you manage these secrets securely without hardcoding them in pipeline code or Docker images.

Accepted Answer

A strong answer covers using Kubernetes Secrets mounted as environment variables or files, backed by an external secrets manager (HashiCorp Vault or AWS Secrets Manager) via the External Secrets Operator or Vault Agent Injector. For S3, it prefers IRSA (IAM Roles for Service Accounts) over access keys entirely. It should mention never baking secrets into Docker layers, scanning images for secret leakage in CI, and rotating secrets without restarting pods by using refreshInterval in External Secrets.

Question 20

Your team spends $50,000 per month on on-demand GPU instances for training. How would you migrate training workloads to spot instances, and what fault tolerance mechanisms do you put in place?

Accepted Answer

A strong answer covers checkpointing model state every N steps to durable storage (S3 or GCS), implementing checkpoint-restart logic in training code, using Kubernetes pod disruption budgets and node termination handlers to detect preemption events and trigger a clean checkpoint before the pod is killed, and using mixed on-demand and spot node groups with Kubernetes Cluster Autoscaler. It should note that spot interruption rates vary by instance type and region, and diversifying across instance families reduces interruption risk.

Question 21

You deploy a churn prediction model. Actual churn outcomes are known 30 days after prediction. How do you architect the system to collect ground truth and feed it back into model evaluation and retraining?

Accepted Answer

A strong answer describes logging every prediction with a prediction ID and the feature snapshot at inference time to an event store, then joining outcomes 30 days later via a scheduled Spark or dbt job that matches customer IDs to churn events, producing a labeled dataset. This labeled dataset feeds automated model evaluation jobs that compute AUC and calibration on recent cohorts and write results to the experiment tracker. The retraining pipeline reads from this joined table rather than raw event logs.

Question 22

You have three models: a TensorFlow SavedModel, a PyTorch TorchScript model, and a custom ensemble that chains both with a preprocessing step. How do you configure Triton to serve all three, and what backend does each use?

Accepted Answer

A strong answer names the tensorflow backend for the SavedModel, the pytorch_libtorch backend for TorchScript, and the ensemble scheduler for chaining them via a config.pbtxt that defines the ensemble pipeline with routing and tensor names. It should mention the Python backend for the preprocessing step, discuss batch size configuration per model, and note that ensemble models do not run on GPU directly but route tensors between constituent models.

Question 23

Explain how ML Metadata stores artifact lineage, what types of nodes exist in its graph, and how you would use it to trace which training data version produced a specific deployed model.

Accepted Answer

A strong answer describes MLMD's three node types: Artifact (files or values produced or consumed), Execution (pipeline steps), and Context (grouping such as pipeline run or experiment), and the two edge types: Event (linking Artifact to Execution as input or output) and Attribution/Association (linking to Context). Tracing a deployed model back to its training data means traversing the execution graph upstream from the model artifact through training and data preprocessing executions to the dataset artifact.

Question 24

You are serving a transformer model with Ray Serve and traffic spikes 10x during peak hours. Walk me through how you configure autoscaling and what knobs you tune to avoid cold-start latency.

Accepted Answer

A strong answer covers Ray Serve's autoscaling config fields: target_num_ongoing_requests_per_replica as the primary trigger, min_replicas and max_replicas bounds, and upscale_delay_s to prevent thrashing. Avoiding cold start means setting min_replicas above zero, pre-loading model weights in the constructor rather than the handler, and using a warm pool of replicas. It should also mention Ray's object store for sharing large model state between replicas and the tradeoff with GPU memory.

Question 25

A team trains a Spark ML Pipeline that includes a StringIndexer, a StandardScaler, and a GBTClassifier. How do you save that pipeline and serve it outside of Spark in a low-latency REST API?

Accepted Answer

A strong answer notes that Spark ML pipelines save to Parquet-backed metadata directories, which are Spark-dependent and cannot be loaded outside Spark. To serve outside Spark, the candidate should mention exporting the fitted transformers to a portable format: using MLeap to serialize the pipeline to a MLeap bundle, or replicating the preprocessing logic in Python/Java with fitted parameters extracted manually. It should note that MLeap supports the most common transformers but has gaps, and that feature computation in Spark offline vs a Python server reintroduces skew risk.

Question 26

A compliance team requires per-prediction SHAP feature attributions logged for every inference in production. Walk me through the performance and architectural challenges of doing this at 1000 requests per second.

Accepted Answer

A strong answer covers that SHAP TreeExplainer on tree models is fast (milliseconds per sample) but KernelExplainer on black-box models is orders of magnitude slower. At 1000 RPS, synchronous SHAP would likely violate latency SLAs, so it should be computed asynchronously: log the input features and prediction, publish to a Kafka topic, and run a SHAP consumer service that writes explanations to a store. Alternatively, approximate SHAP using background datasets of 100 samples instead of the full training set, and cache explanations for repeated inputs.

Question 27

Walk me through the Istio VirtualService and DestinationRule configuration you would write to send 90% of traffic to model-v1 and 10% to model-v2 of the same service.

Accepted Answer

A strong answer writes out the VirtualService with two route entries with weight 90 and 10, each routing to a separate subset defined in the DestinationRule. The DestinationRule uses label selectors (e.g., version: v1 and version: v2) to map subsets to the Kubernetes pods. It should mention that sticky sessions via consistent hashing on a header can be added to ensure users always hit the same version within a session.

Question 28

What should a model card include for a customer-facing credit decision model, and how do you keep it synchronized with the actual model in production as it is retrained?

Accepted Answer

A strong answer covers: intended use, out-of-scope uses, training data demographics and date range, evaluation metrics disaggregated by protected groups, known limitations, fairness assessments, and responsible use guidance. Keeping it synchronized means auto-generating quantitative sections (metrics, fairness scores) from the evaluation pipeline on each retraining run and storing the card artifact in the model registry alongside the weights, linked by model version.

Question 29

A PyTorch model with dynamic control flow and custom autograd functions fails to export to ONNX. What are the most common failure modes, and how do you work around them?

Accepted Answer

A strong answer lists: dynamic control flow (if statements based on tensor values) that torch.onnx.export cannot trace statically, custom autograd Functions without a symbolic method registered, in-place ops on non-leaf tensors, and ops with no ONNX opset equivalent. Workarounds include using torch.jit.script instead of trace for dynamic control flow, registering a custom ONNX symbolic, refactoring in-place ops to their out-of-place equivalents, and checking the opset version (17+ has broader coverage).

Question 30

Explain how SageMaker multi-model endpoints work under the hood, when they improve cost versus dedicated endpoints, and what the cold-start tradeoff looks like.

Accepted Answer

A strong answer explains that a multi-model endpoint runs a single container that implements the SageMaker multi-model server API (DescribeModel, InvokeEndpoint with TargetModel), dynamically loading models from S3 into memory on first request and evicting least-recently-used models when memory is tight. Cost improves when many models have sparse, non-overlapping traffic (e.g., per-tenant models). Cold start occurs when a model is not in memory and must be downloaded from S3 and loaded, adding seconds of latency to the first request.

Questions

MLflow model registry stage transitionsRole-specificmediumVery common

As asked

Sample answer outline

Expect these follow-ups

Setting GPU resource requests in Kubernetes for training jobsRole-specificmediumVery common

As asked

Sample answer outline

Expect these follow-ups

Canary deployment strategy for an ML modelRole-specificmediumVery common

As asked

Sample answer outline

Expect these follow-ups

Distinguishing data drift from concept driftRole-specificmediumVery common

As asked

Sample answer outline

Expect these follow-ups

Make drift monitoring actionableRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Explain online vs offline skew in feature storesRole-specifichardCommon

As asked

Sample answer outline

Expect these follow-ups

Kubeflow pipeline caching internalsRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

PSI vs KS test for feature drift monitoringRole-specifichardCommon

As asked

Sample answer outline

Expect these follow-ups

Integrating Great Expectations into an ML pipelineRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Achieving reproducible ML training runsRole-specifichardCommon

As asked

Sample answer outline

Expect these follow-ups

Choosing Airflow vs Kubeflow Pipelines for ML workflowsRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

DVC data and model versioning internalsRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Achieving sub-10ms feature retrieval in a feature storeRole-specifichardCommon

As asked

Sample answer outline

Expect these follow-ups

Shadow mode deployment for ML model validationRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Exporting custom model metrics to PrometheusRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Structuring a Helm chart for a model serving deploymentRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Triggering continuous retraining vs scheduled retrainingRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Batch inference at scale: Spark vs RayRole-specifichardCommon

As asked

Sample answer outline

Expect these follow-ups

Secret management for ML training pipelinesRole-specificmediumCommon

As asked

Sample answer outline

Expect these follow-ups

Using spot/preemptible instances for ML training cost reductionRole-specificmediumCommon

As asked

Sample answer outline