ML engineer machine learning interview questions

As asked

Explain the bias-variance tradeoff. Use a concrete worked example, not just the definitions.

Sample answer outline

Bias is error from oversimplifying assumptions; variance is error from over-fitting to training noise. Concrete example: predicting house prices. A constant model (predict the mean) has very high bias but zero variance. A degree-15 polynomial on 100 points has very low bias but huge variance - it tracks the training data perfectly and fails on new data. The optimum is somewhere in between. Mitigation: regularisation (L1/L2), cross-validation to estimate the curve, ensembling to average out variance.

Expect these follow-ups

• How does L2 regularisation specifically reduce variance?
• Where does the bias-variance decomposition come from mathematically?
• Why does bagging reduce variance but boosting reduce bias?

As asked

Neural networks are popular - but when is a neural network the wrong tool for the job?

Sample answer outline

Wrong tool when: data is small (under ~10k examples - gradient boosted trees usually win on tabular data, full stop); interpretability is required (regulated industries, medical decisions, anything that needs SHAP-style attribution that holds up to scrutiny); inference latency is tight and the model would not fit; features are mostly categorical and high-cardinality; the cost of false positives is asymmetric and you need calibrated probabilities. XGBoost, LightGBM, and CatBoost on engineered features are still state of the art for most tabular problems.

Expect these follow-ups

• Why do gradient boosted trees beat neural networks on tabular data?
• How would you explain a tree ensemble's predictions to a regulator?
• When do neural networks make sense for tabular data?

As asked

You need to serve a 70B-parameter open-source LLM at under 100ms p95 latency for a single-token completion. Walk me through how you would approach it.

Sample answer outline

Hardware: A100/H100 or H200 GPUs; the model in 16-bit needs ~140GB so multi-GPU tensor-parallel or quantised to int8/int4 to fit one GPU. Software: vLLM or TGI with paged attention. Latency tricks: prefill batching, speculative decoding with a small draft model, KV-cache reuse for common prompts. Discuss the prefill vs decode tradeoff (prefill is compute-bound, decode is memory-bandwidth-bound). Geo-distributed deployment for global p95. For a single-token completion the prefill dominates - heavily quantised models with FlashAttention can hit <50ms on H100.

Expect these follow-ups

• What is speculative decoding and when does it win?
• How does paged attention change throughput vs latency?
• When does int4 quantisation hurt quality enough to matter?

As asked

Design a feature store that serves features to both training jobs and online inference, with consistent semantics between batch and realtime.

Sample answer outline

Two storage tiers: offline (a columnar warehouse like BigQuery or Iceberg, used by training jobs) and online (a low-latency KV like Redis or DynamoDB, used by serving). The core invariant: a feature should produce the same value at training time and serving time, otherwise you get training-serving skew. Solutions: define features as code (e.g. SQL or DSL), compute them once and write to both tiers; use point-in-time joins for backfills so you don't leak future data; tag features with freshness SLAs and version them.

Expect these follow-ups

• How do you handle a feature whose definition changes?
• What is point-in-time correct join and why does it matter?
• How do you prevent training-serving skew when offline and online run different code paths?

As asked

Explain overfitting and underfitting in plain terms, and tell me how you would notice that a model is overfitting. Keep it concrete.

Sample answer outline

Frame the tradeoff through its everyday symptoms rather than the formal decomposition. Underfitting is when a model is too simple to capture the pattern, so it does poorly on both the training data and new data. Overfitting is when a model memorises the training data, including its noise, so it looks great on training but does badly on data it has not seen. You spot overfitting by a large gap between training accuracy and validation accuracy. Mention the simplest fixes an earlier-career engineer reaches for: get more data, use a simpler model, or add regularisation. A concrete picture, such as a wiggly curve threading every training point, lands better than equations here.

Expect these follow-ups

• What does a big gap between training and validation scores tell you?
• Name one quick way to reduce overfitting.
• Why does more data usually help with overfitting?

As asked

Derive the bias-variance decomposition of expected squared error, then explain why the classical tradeoff intuition is complicated by phenomena like double descent in heavily over-parameterised models. Where does the textbook picture mislead you?

Sample answer outline

Start from expected test error at a point and decompose it into irreducible noise, squared bias, and variance, showing the cross term vanishes under the expectation over training sets. That is the classical U-shaped story: increasing capacity lowers bias and raises variance, with an optimum in the middle. The senior nuance is that this picture was built in the under-parameterised regime. In modern over-parameterised models, test error can fall again past the interpolation threshold, the double descent phenomenon, because among the many zero-training-error solutions, optimisation and implicit regularisation favour low-norm, well-generalising ones. So variance does not simply explode with capacity. The strong answer keeps the decomposition rigorous while being honest that capacity alone is a poor predictor of generalisation once you can interpolate the data.

Expect these follow-ups

• Why does the cross term drop out of the decomposition?
• What is the interpolation threshold and what happens beyond it?
• How does implicit regularisation in gradient descent fit this story?

As asked

You are not training a model from scratch, you are building a feature on top of a foundation model with prompting, retrieval, and maybe light fine-tuning. How does bias-variance thinking still apply to the choices you make, and how do you tell whether your system is underfitting or overfitting the task?

Sample answer outline

Translate the concept into the applied-AI setting. Underfitting shows up as a system that is too generic: weak prompts, no retrieval, and a model that gives plausible but off-target answers because it lacks the task context. Overfitting shows up as a system tuned so tightly to a handful of examples that it fails on real, varied inputs: brittle few-shot prompts, a fine-tune on a tiny biased dataset, or retrieval that only works for the queries you tested. The applied controls are the modern analogues of regularisation and capacity: richer context and retrieval reduce bias, while a held-out evaluation set, diverse examples, and resisting over-tuning to a demo reduce variance. The AI-engineer signal is measuring this with an evaluation harness on representative inputs rather than eyeballing a few prompts, and knowing when a prompt change is genuinely better versus fitted to your test cases.

Expect these follow-ups

• How do you build an evaluation set that reveals overfitting to your examples?
• When does adding retrieval reduce bias versus just adding noise?
• Why can a fine-tune on a small dataset make a model worse on real inputs?

As asked

Your model scores beautifully offline but falls apart in production. You suspect data leakage. Explain what leakage is, the common ways it sneaks in, and how you would build the pipeline so it cannot happen.

Sample answer outline

Leakage is when information that would not be available at prediction time leaks into training, so the model looks great offline and collapses in the real world. The classic forms: fitting a scaler or encoder on the full dataset before splitting, so test statistics bleed into training; including a feature that is a proxy for the label or is only populated after the outcome is known; and time leakage, where a row uses information from the future relative to when the prediction would actually be made. The structural fixes are to split first and fit every transform inside the training fold only, then apply it to validation and test, ideally wrapped in a pipeline so the discipline is enforced rather than remembered. For time-based problems, use point-in-time correct joins and time-ordered splits so you never train on the future. The strong answer treats the suspiciously high offline score as a symptom to investigate, not a result to trust.

Expect these follow-ups

• Why does fitting a scaler before the train-test split leak information?
• How does a point-in-time join prevent time leakage?
• What is your first check when offline metrics look too good?

As asked

You have shipped a model to production. Labels arrive days or weeks later, if at all. How do you know whether the model is still healthy, and what do you alert on when you cannot just watch accuracy in real time?

Sample answer outline

Because ground-truth labels are delayed, you cannot rely on live accuracy, so you monitor proxies that move before performance visibly degrades. Watch the input side for data drift: feature distributions shifting away from training, a spike in missing values, or new categorical values the model never saw. Watch the output side for prediction drift: the score distribution sliding, or the rate of a given class changing without a business reason. Keep operational monitoring too, latency and error rates, since a broken feature pipeline often shows up there first. When labels do land, compute true performance on that lagged window and track it over time, and run periodic backtests. Alert on drift thresholds and on pipeline anomalies rather than waiting for accuracy to crater. The mature answer separates the fast signals you can watch now from the slow ground truth, and closes the loop by feeding late labels back for evaluation and retraining.

Expect these follow-ups

• What is the difference between data drift and prediction drift?
• How do you evaluate true performance once delayed labels arrive?
• Which signal usually moves first when a feature pipeline breaks?

As asked

You are building a classifier where the positive class is one percent of the data, say fraud or a rare disease. Why is accuracy the wrong metric, and how do you actually evaluate and tune the model?

Sample answer outline

On a one percent positive rate a model that predicts negative for everyone scores 99 percent accuracy and is useless, so accuracy hides total failure on the class you care about. Evaluate with metrics that focus on the positive class: precision and recall, and the precision-recall curve, which is more informative than the ROC curve under heavy imbalance because it does not get flattered by the huge number of true negatives. Which way you trade precision against recall is a business decision, missing a fraud case versus annoying a good customer, so tune the decision threshold to that cost rather than defaulting to 0.5. For training, options include class weighting or resampling, but be honest that resampling changes the base rate and you must calibrate or correct probabilities afterward if you need them to mean something. The signal is choosing metrics and a threshold that reflect the real cost of each error, not a single headline number.

Expect these follow-ups

• Why prefer a precision-recall curve over ROC under heavy imbalance?
• How do you choose the decision threshold rather than defaulting to 0.5?
• What does resampling do to your predicted probabilities?

As asked

Your recommendation system works well for established users and items, but it has nothing useful to show a brand new user or to surface a freshly added item. How do you handle cold start on both sides?

Sample answer outline

Cold start is two problems. For a new user with no interaction history, fall back to signals you do have: popularity and trending items, anything from onboarding such as stated interests, and contextual cues like location or referral source, then shift to personalised collaborative signals as interactions accumulate. For a new item with no interactions, lean on content features, its attributes, text, or embeddings, so it can be matched to users by similarity to items they already like, rather than waiting for engagement data it does not have yet. A hybrid that blends content-based and collaborative signals degrades gracefully at the edges instead of returning nothing. Add deliberate exploration so new items get a fair chance to be shown and gather the feedback they need, otherwise popular items starve everything new. The strong answer names both sides explicitly and treats cold start as a smooth handoff from content and context to behaviour, not a switch that flips once.

Expect these follow-ups

• What signals do you use for a user with zero interaction history?
• How do content features help a brand new item get recommended?
• Why does a recommender need deliberate exploration?

As asked

Given a raw dataset of user events with timestamps, how do you think about turning it into features for a model that predicts churn? Show me how you reason about it rather than listing transformations.

Sample answer outline

Start from the prediction and the user, not the columns. For churn the useful signals are about engagement over time, so derive recency, frequency, and trend features: how long since the last meaningful action, how often the user acts in a window, and crucially whether activity is rising or falling, since a declining trend predicts churn better than a static count. Aggregate raw events into windows like last 7 and last 30 days so the model sees behaviour, not individual rows, and be strict that every window only uses data available before the prediction point to avoid leakage. Encode categoricals thoughtfully, watching cardinality, and capture lifecycle context like tenure. Throughout, prefer features a domain person would expect to matter and that you can explain, and validate that a feature actually improves held-out performance rather than adding everything and hoping. The signal is reasoning from the user behaviour you are trying to capture, with leakage discipline baked in, not reciting a transformation checklist.

Expect these follow-ups

• Why is the trend of activity often more predictive than its level?
• How do you make sure windowed features do not leak the future?
• How do you decide whether a new feature is actually worth keeping?