MLOps engineer machine learning interview questions

As asked

A researcher says yesterday's experiment improved benchmark accuracy by 3 percent, but nobody can reproduce it. What do you check first?

Sample answer outline

Start with the experiment record: exact code commit, model config, data snapshot, preprocessing version, random seeds, hardware, and library versions. Then check whether the reported metric came from the intended split and whether the evaluation script changed between runs. Strong answers discuss determinism limits on GPUs while still insisting on reproducible artefacts and comparable results. The candidate should propose a minimal rerun and then a full rerun under tracked configuration. A common failure is focusing only on the seed while ignoring data leakage or an untracked preprocessing change.

Expect these follow-ups

• Which parts of a GPU training run may remain non-deterministic?
• How would you store data snapshots without copying terabytes every time?
• What would make you distrust the original result?

As asked

A fraud model performs well offline but starts making poor decisions in production. How would you investigate training-serving skew?

Sample answer outline

Compare feature values used during offline training with the values produced by the online serving path for the same entity and timestamp. Look for point-in-time leakage, stale online features, different default values, mismatched transformations, and schema evolution that did not reach both paths. Strong answers include replaying production events through the offline pipeline and logging model inputs at serving time with privacy controls. The candidate should separate skew from drift, because the first is a pipeline consistency bug and the second is a world-change problem. A common miss is checking only model metrics and not the raw feature distributions.

Expect these follow-ups

• What feature would you inspect first in a velocity-based fraud model?
• How do you log online features without creating a privacy problem?
• How would you prevent this class of issue in CI?