MLOps engineer at N26

As asked

A researcher says yesterday's experiment improved benchmark accuracy by 3 percent, but nobody can reproduce it. What do you check first?

Sample answer outline

Start with the experiment record: exact code commit, model config, data snapshot, preprocessing version, random seeds, hardware, and library versions. Then check whether the reported metric came from the intended split and whether the evaluation script changed between runs. Strong answers discuss determinism limits on GPUs while still insisting on reproducible artefacts and comparable results. The candidate should propose a minimal rerun and then a full rerun under tracked configuration. A common failure is focusing only on the seed while ignoring data leakage or an untracked preprocessing change.

Expect these follow-ups

• Which parts of a GPU training run may remain non-deterministic?
• How would you store data snapshots without copying terabytes every time?
• What would make you distrust the original result?

As asked

A fraud model performs well offline but starts making poor decisions in production. How would you investigate training-serving skew?

Sample answer outline

Compare feature values used during offline training with the values produced by the online serving path for the same entity and timestamp. Look for point-in-time leakage, stale online features, different default values, mismatched transformations, and schema evolution that did not reach both paths. Strong answers include replaying production events through the offline pipeline and logging model inputs at serving time with privacy controls. The candidate should separate skew from drift, because the first is a pipeline consistency bug and the second is a world-change problem. A common miss is checking only model metrics and not the raw feature distributions.

Expect these follow-ups

• What feature would you inspect first in a velocity-based fraud model?
• How do you log online features without creating a privacy problem?
• How would you prevent this class of issue in CI?

As asked

Describe how you manage model stage transitions in MLflow's model registry, from staging to production, including the checks and automation you put around that workflow.

Sample answer outline

A strong answer covers how MLflow 1.x used named lifecycle stages (Staging, Production, Archived) accessed via the MlflowClient API or the UI, and that MLflow 2.x deprecated these fixed stages in favour of model aliases (e.g., @champion, @challenger) and model tags, which offer more flexibility. For legacy setups, it covers what automated gates run before promoting to Production: metric thresholds, challenger vs champion A/B results, data validation checks, and approval gates in CI. It should mention webhook integrations or event hooks that trigger downstream pipelines when a model alias is updated or a stage transition occurs, and note that teams still on MLflow 1.x stages should plan migration to aliases.

Expect these follow-ups

• How do you roll back a production model in MLflow if inference latency spikes after promotion?
• What do you store in a run's logged artifacts beyond the model weights itself?

As asked

A data scientist hands you a PyTorch training job that needs to run on Kubernetes. Walk me through how you configure resource requests and limits for GPU, CPU, and memory, and what can go wrong if you get them wrong.

Sample answer outline

A strong answer covers requesting nvidia.com/gpu as an extended resource (limits equal requests for GPUs, no partial GPU allocation by default), setting CPU and memory requests based on data loading worker count and model size, and why setting memory limit much higher than request causes OOMKilled pods under pressure. It should mention using DCGM metrics to verify actual GPU utilization, and node taints plus tolerations to restrict GPU nodes to training workloads.

Expect these follow-ups

• How does NVIDIA's device plugin expose MIG slices to Kubernetes pods?
• What happens to a training job pod when the node it runs on runs out of memory?

As asked

Walk me through how you would do a canary deployment of a new recommendation model version, including what metrics you watch, how long you hold the canary, and what triggers a rollback.

Sample answer outline

A strong answer describes routing 5 to 10 percent of traffic to the new model version, monitoring both technical metrics (latency p99, error rate, feature lookup failures) and business metrics (click-through rate, conversion), using statistical tests to compare the two slices, and setting a minimum window of 24 to 48 hours to capture daily traffic patterns. Rollback triggers should include automated rules for latency spikes above a threshold and manual override for business metric degradation.

Expect these follow-ups

• How do you handle the case where the canary serves a non-representative user segment?
• What infrastructure changes are needed in your serving layer to split traffic between two model versions?

As asked

Explain the difference between data drift and concept drift, give a concrete example of each in a credit scoring context, and describe how your monitoring strategy differs for each.

Sample answer outline

Data drift means the input feature distribution P(X) changes, for example average applicant income rising 20% after an economic upturn, while the true relationship between features and default risk is unchanged. Concept drift means the conditional P(Y|X) changes, for example a regulatory change that alters what constitutes a default. Data drift is detectable by monitoring input features with PSI or KS tests; concept drift requires labeled outcome data and is only detectable with a lag equal to the label delay.

Expect these follow-ups

• What is virtual concept drift, and how does it differ from real concept drift?
• How do you monitor for concept drift when your label delay is 90 days?