ML engineer machine learning interview questions

As asked

Explain the bias-variance tradeoff. Use a concrete worked example, not just the definitions.

Sample answer outline

Bias is error from oversimplifying assumptions; variance is error from over-fitting to training noise. Concrete example: predicting house prices. A constant model (predict the mean) has very high bias but zero variance. A degree-15 polynomial on 100 points has very low bias but huge variance - it tracks the training data perfectly and fails on new data. The optimum is somewhere in between. Mitigation: regularisation (L1/L2), cross-validation to estimate the curve, ensembling to average out variance.

Expect these follow-ups

• How does L2 regularisation specifically reduce variance?
• Where does the bias-variance decomposition come from mathematically?
• Why does bagging reduce variance but boosting reduce bias?

As asked

Explain overfitting and underfitting in plain terms, and tell me how you would notice that a model is overfitting. Keep it concrete.

Sample answer outline

Frame the tradeoff through its everyday symptoms rather than the formal decomposition. Underfitting is when a model is too simple to capture the pattern, so it does poorly on both the training data and new data. Overfitting is when a model memorises the training data, including its noise, so it looks great on training but does badly on data it has not seen. You spot overfitting by a large gap between training accuracy and validation accuracy. Mention the simplest fixes an earlier-career engineer reaches for: get more data, use a simpler model, or add regularisation. A concrete picture, such as a wiggly curve threading every training point, lands better than equations here.

Expect these follow-ups

• What does a big gap between training and validation scores tell you?
• Name one quick way to reduce overfitting.
• Why does more data usually help with overfitting?

As asked

You are building a classifier where the positive class is one percent of the data, say fraud or a rare disease. Why is accuracy the wrong metric, and how do you actually evaluate and tune the model?

Sample answer outline

On a one percent positive rate a model that predicts negative for everyone scores 99 percent accuracy and is useless, so accuracy hides total failure on the class you care about. Evaluate with metrics that focus on the positive class: precision and recall, and the precision-recall curve, which is more informative than the ROC curve under heavy imbalance because it does not get flattered by the huge number of true negatives. Which way you trade precision against recall is a business decision, missing a fraud case versus annoying a good customer, so tune the decision threshold to that cost rather than defaulting to 0.5. For training, options include class weighting or resampling, but be honest that resampling changes the base rate and you must calibrate or correct probabilities afterward if you need them to mean something. The signal is choosing metrics and a threshold that reflect the real cost of each error, not a single headline number.

Expect these follow-ups

• Why prefer a precision-recall curve over ROC under heavy imbalance?
• How do you choose the decision threshold rather than defaulting to 0.5?
• What does resampling do to your predicted probabilities?

As asked

Walk me through what Constitutional AI is, how it differs from standard RLHF with a human reward model, and what practical problems at scale it was designed to solve.

Sample answer outline

A strong answer explains that CAI uses a written set of principles and has the model critique and revise its own outputs, reducing reliance on human labelers for the harmlessness dimension. RLHF trains a reward model from human preference data and uses PPO or similar to optimize against it. CAI at scale reduces the bottleneck of human annotation for safety feedback and makes the value alignment more inspectable since the constitution is readable text. The candidate should mention the SL-CAI and RL-CAI stages.

Expect these follow-ups

• What are the failure modes of a constitution that is too vague or too specific?
• How would you evaluate whether a constitution is actually being followed by the model?

As asked

A new creator posts their first video on TikTok. There is no watch history, no engagement data, nothing. How does your recommendation system decide who to show this video to, and how does it bootstrap the feedback loop?

Sample answer outline

Strong answers address both the item cold start (no engagement data) and the user cold start (new user). For item cold start: use content-based signals (audio, visuals, text description, creator profile), show the video to a small random sample from the creator's existing follower graph, then use early engagement signals (first 100 views) to decide whether to expand distribution. For user cold start: use onboarding interest selection, geographic defaults, or implicit signals from the referral source. The candidate should mention the risk of seeding the wrong distribution early.

Expect these follow-ups

• How long do you wait before deciding a video is low quality versus just slow to gain traction?
• How does the cold start strategy change for a creator who is already famous on another platform?

As asked

You are setting up a machine learning platform for a team of 10 data scientists who will train dozens of models a week. Walk me through how you would use MLflow on Databricks to track experiments, compare runs, register the best model, and gate promotion from staging to production.

Sample answer outline

A strong answer covers: logging parameters, metrics, and artifacts with mlflow.log_param and mlflow.log_metric inside training runs; using experiment UI or mlflow.search_runs to compare across runs; registering the best run to the Model Registry with mlflow.register_model; using staging and production stages with manual or automated approval gates (e.g., a CI job that runs evaluation metrics before promoting); and serving the production model via Databricks Model Serving endpoints. The candidate should mention model aliases in the newer MLflow API which replace the stage transitions.

Expect these follow-ups

• How would you enforce that a model cannot go to production unless it has an AUC above a certain threshold?
• What would you log as artifacts beyond the model weights to make a run fully reproducible?

As asked

Neural networks are popular - but when is a neural network the wrong tool for the job?

Sample answer outline

Wrong tool when: data is small (under ~10k examples - gradient boosted trees usually win on tabular data, full stop); interpretability is required (regulated industries, medical decisions, anything that needs SHAP-style attribution that holds up to scrutiny); inference latency is tight and the model would not fit; features are mostly categorical and high-cardinality; the cost of false positives is asymmetric and you need calibrated probabilities. XGBoost, LightGBM, and CatBoost on engineered features are still state of the art for most tabular problems.

Expect these follow-ups

• Why do gradient boosted trees beat neural networks on tabular data?
• How would you explain a tree ensemble's predictions to a regulator?
• When do neural networks make sense for tabular data?

As asked

You need to serve a 70B-parameter open-source LLM at under 100ms p95 latency for a single-token completion. Walk me through how you would approach it.

Sample answer outline

Hardware: A100/H100 or H200 GPUs; the model in 16-bit needs ~140GB so multi-GPU tensor-parallel or quantised to int8/int4 to fit one GPU. Software: vLLM or TGI with paged attention. Latency tricks: prefill batching, speculative decoding with a small draft model, KV-cache reuse for common prompts. Discuss the prefill vs decode tradeoff (prefill is compute-bound, decode is memory-bandwidth-bound). Geo-distributed deployment for global p95. For a single-token completion the prefill dominates - heavily quantised models with FlashAttention can hit <50ms on H100.

Expect these follow-ups

• What is speculative decoding and when does it win?
• How does paged attention change throughput vs latency?
• When does int4 quantisation hurt quality enough to matter?

As asked

Design a feature store that serves features to both training jobs and online inference, with consistent semantics between batch and realtime.

Sample answer outline

Two storage tiers: offline (a columnar warehouse like BigQuery or Iceberg, used by training jobs) and online (a low-latency KV like Redis or DynamoDB, used by serving). The core invariant: a feature should produce the same value at training time and serving time, otherwise you get training-serving skew. Solutions: define features as code (e.g. SQL or DSL), compute them once and write to both tiers; use point-in-time joins for backfills so you don't leak future data; tag features with freshness SLAs and version them.

Expect these follow-ups

• How do you handle a feature whose definition changes?
• What is point-in-time correct join and why does it matter?
• How do you prevent training-serving skew when offline and online run different code paths?

As asked

You are not training a model from scratch, you are building a feature on top of a foundation model with prompting, retrieval, and maybe light fine-tuning. How does bias-variance thinking still apply to the choices you make, and how do you tell whether your system is underfitting or overfitting the task?

Sample answer outline

Translate the concept into the applied-AI setting. Underfitting shows up as a system that is too generic: weak prompts, no retrieval, and a model that gives plausible but off-target answers because it lacks the task context. Overfitting shows up as a system tuned so tightly to a handful of examples that it fails on real, varied inputs: brittle few-shot prompts, a fine-tune on a tiny biased dataset, or retrieval that only works for the queries you tested. The applied controls are the modern analogues of regularisation and capacity: richer context and retrieval reduce bias, while a held-out evaluation set, diverse examples, and resisting over-tuning to a demo reduce variance. The AI-engineer signal is measuring this with an evaluation harness on representative inputs rather than eyeballing a few prompts, and knowing when a prompt change is genuinely better versus fitted to your test cases.

Expect these follow-ups

• How do you build an evaluation set that reveals overfitting to your examples?
• When does adding retrieval reduce bias versus just adding noise?
• Why can a fine-tune on a small dataset make a model worse on real inputs?

As asked

Your model scores beautifully offline but falls apart in production. You suspect data leakage. Explain what leakage is, the common ways it sneaks in, and how you would build the pipeline so it cannot happen.

Sample answer outline

Leakage is when information that would not be available at prediction time leaks into training, so the model looks great offline and collapses in the real world. The classic forms: fitting a scaler or encoder on the full dataset before splitting, so test statistics bleed into training; including a feature that is a proxy for the label or is only populated after the outcome is known; and time leakage, where a row uses information from the future relative to when the prediction would actually be made. The structural fixes are to split first and fit every transform inside the training fold only, then apply it to validation and test, ideally wrapped in a pipeline so the discipline is enforced rather than remembered. For time-based problems, use point-in-time correct joins and time-ordered splits so you never train on the future. The strong answer treats the suspiciously high offline score as a symptom to investigate, not a result to trust.

Expect these follow-ups

• Why does fitting a scaler before the train-test split leak information?
• How does a point-in-time join prevent time leakage?
• What is your first check when offline metrics look too good?

As asked

You have shipped a model to production. Labels arrive days or weeks later, if at all. How do you know whether the model is still healthy, and what do you alert on when you cannot just watch accuracy in real time?

Sample answer outline

Because ground-truth labels are delayed, you cannot rely on live accuracy, so you monitor proxies that move before performance visibly degrades. Watch the input side for data drift: feature distributions shifting away from training, a spike in missing values, or new categorical values the model never saw. Watch the output side for prediction drift: the score distribution sliding, or the rate of a given class changing without a business reason. Keep operational monitoring too, latency and error rates, since a broken feature pipeline often shows up there first. When labels do land, compute true performance on that lagged window and track it over time, and run periodic backtests. Alert on drift thresholds and on pipeline anomalies rather than waiting for accuracy to crater. The mature answer separates the fast signals you can watch now from the slow ground truth, and closes the loop by feeding late labels back for evaluation and retraining.

Expect these follow-ups

• What is the difference between data drift and prediction drift?
• How do you evaluate true performance once delayed labels arrive?
• Which signal usually moves first when a feature pipeline breaks?

As asked

Your recommendation system works well for established users and items, but it has nothing useful to show a brand new user or to surface a freshly added item. How do you handle cold start on both sides?

Sample answer outline

Cold start is two problems. For a new user with no interaction history, fall back to signals you do have: popularity and trending items, anything from onboarding such as stated interests, and contextual cues like location or referral source, then shift to personalised collaborative signals as interactions accumulate. For a new item with no interactions, lean on content features, its attributes, text, or embeddings, so it can be matched to users by similarity to items they already like, rather than waiting for engagement data it does not have yet. A hybrid that blends content-based and collaborative signals degrades gracefully at the edges instead of returning nothing. Add deliberate exploration so new items get a fair chance to be shown and gather the feedback they need, otherwise popular items starve everything new. The strong answer names both sides explicitly and treats cold start as a smooth handoff from content and context to behaviour, not a switch that flips once.

Expect these follow-ups

• What signals do you use for a user with zero interaction history?
• How do content features help a brand new item get recommended?
• Why does a recommender need deliberate exploration?

As asked

Given a raw dataset of user events with timestamps, how do you think about turning it into features for a model that predicts churn? Show me how you reason about it rather than listing transformations.

Sample answer outline

Start from the prediction and the user, not the columns. For churn the useful signals are about engagement over time, so derive recency, frequency, and trend features: how long since the last meaningful action, how often the user acts in a window, and crucially whether activity is rising or falling, since a declining trend predicts churn better than a static count. Aggregate raw events into windows like last 7 and last 30 days so the model sees behaviour, not individual rows, and be strict that every window only uses data available before the prediction point to avoid leakage. Encode categoricals thoughtfully, watching cardinality, and capture lifecycle context like tenure. Throughout, prefer features a domain person would expect to matter and that you can explain, and validate that a feature actually improves held-out performance rather than adding everything and hoping. The signal is reasoning from the user behaviour you are trying to capture, with leakage discipline baked in, not reciting a transformation checklist.

Expect these follow-ups

• Why is the trend of activity often more predictive than its level?
• How do you make sure windowed features do not leak the future?
• How do you decide whether a new feature is actually worth keeping?

As asked

Explain how INT8 post-training quantization works for a transformer model, and describe what AMD hardware features make INT8 inference faster than FP32 on MI300X.

Sample answer outline

Post-training quantization maps float32 weights to int8 by finding scale and zero-point parameters per tensor or per channel, minimizing the quantization error. Activation quantization requires calibration data to capture typical activation ranges. On MI300X, the Matrix Core units execute INT8 GEMM at roughly 2x the FLOPS of BF16 GEMM on the same hardware (approximately 1307 TOPS INT8 vs 654 TFLOPS BF16), because the same multiplier lanes process more values per cycle with narrower data types. The candidate should also mention that attention layers are memory-bound so BF16 KV cache size matters more there than INT8 weight quantization, and that per-channel quantization typically recovers more accuracy than per-tensor at the cost of more complex dequantization.

Expect these follow-ups

• What is the difference between per-tensor and per-channel quantization and when does each degrade accuracy?
• How does ROCm's MIOpen support INT8 convolution and where can you find the tuning database?

As asked

Imagine you are on the team responsible for Claude's safety behaviors. We've shipped a new version and our internal evals show refusal rate on clearly benign requests has gone up 15%. Walk me through how you would diagnose the root cause and what interventions you would consider.

Sample answer outline

A strong answer starts with slicing the eval data to find which categories of benign requests are being over-refused, distinguishing false positives from legitimate caution. The candidate should consider whether the issue is in RLHF reward signal, the constitution, or a classifier threshold. Interventions include targeted preference data, refining the constitution clauses that are triggering false positives, or adjusting classifier thresholds with held-out test data. The answer should show understanding that over-refusal is a real cost, not a safe default.

Expect these follow-ups

• How do you avoid fixing over-refusal by simply making Claude permissive on genuinely harmful requests?
• What eval dataset would you build to track this metric reliably over time?

As asked

Claude supports context windows up to 200k tokens. Walk me through the computational and memory challenges that creates for attention, and describe at least two techniques used to make long-context inference practical.

Sample answer outline

Attention is O(n^2) in sequence length for both compute and memory in the naive case. At 200k tokens the KV cache alone can be tens of GB per request per layer. Techniques include FlashAttention (tiled, IO-aware recomputation that avoids materializing the full attention matrix), multi-query or grouped-query attention to reduce KV cache size, and quantization of the KV cache. A strong answer also mentions how positional encodings like RoPE are extended or extrapolated for lengths beyond training, since this is a known challenge.

Expect these follow-ups

• How would you detect that a model is actually attending to far-away context versus ignoring it?
• What eval would you design to measure long-context retrieval faithfulness?

As asked

The Chinchilla paper revised how we think about optimal compute allocation for language model training. Explain the core finding and how it would inform a decision about whether to train a 70B parameter model on 1T tokens versus a 7B model on 10T tokens given the same compute budget.

Sample answer outline

Chinchilla found that prior models were significantly undertrained relative to their parameter count. The optimal ratio is roughly 20 tokens per parameter for a compute-optimal run. A 70B model ideally trains on 1.4T tokens; a 7B model on 140B. Given a fixed compute budget, the smaller model trained on far more data often outperforms the larger undertrained model. A strong answer notes that inference cost is a real-world consideration and a smaller well-trained model may be preferred in production even if a larger model with the same compute would score higher on benchmarks.

Expect these follow-ups

• Does the Chinchilla result hold at very large scales, or are there diminishing returns to data?
• How does the optimal token-to-parameter ratio change if inference cost matters more than training cost?

As asked

We want to measure whether Claude maintains honest answers when a user persistently pushes back, claims Claude is wrong, or uses social pressure. How would you design this eval, what failure modes would you look for, and how would you avoid the eval itself becoming a training target that the model games?

Sample answer outline

A strong answer defines a dataset of factually verifiable questions with known correct answers, pairs each with a scripted adversarial follow-up sequence that applies social pressure without providing new evidence, and measures sycophantic capitulation rate. Failure modes include the model changing a correct answer under pressure, the model becoming stubbornly wrong when actually corrected with evidence, and the model hedging so much that it gives no useful answer. To avoid gaming, the eval should use held-out test questions never seen in training and be refreshed regularly.

Expect these follow-ups

• How do you distinguish sycophancy from legitimate updating on new evidence?
• What would a benchmark like TruthfulQA miss about real-world honesty?

As asked

Many Claude API users send the same large system prompt with every request. Explain how KV cache sharing or prompt caching works at the inference level, what it saves, and what implementation challenges it introduces.

Sample answer outline

The KV cache for the attention layers is a function only of the input tokens, so if the prefix of a request is identical across calls the computed K and V tensors for that prefix can be reused. Anthropic offers prompt caching where the system prompt is cached server-side and only charged at a reduced rate on subsequent hits. Implementation challenges include cache invalidation when the system prompt changes by even one token, memory pressure from maintaining caches across concurrent users, and routing requests to the same server instance to hit a warm cache.

Expect these follow-ups

• How would you implement cache routing in a multi-server fleet without sticky sessions?
• What is the right eviction policy for a KV cache under memory pressure?

As asked

Claude accepts images as input. Walk me through how an image goes from raw pixels to something the transformer can reason about alongside text tokens, and what constraints that places on input size and cost.

Sample answer outline

Images are typically passed through a vision encoder such as a ViT that produces patch embeddings, which are projected into the same embedding space as text tokens and concatenated with the text sequence. The number of visual tokens depends on image resolution and patch size; a high-resolution image can produce hundreds or thousands of tokens, which matters for context length and cost. A strong answer notes that Anthropic tiles large images into sub-images to handle arbitrary resolutions and that cost is per-token including visual tokens.

Expect these follow-ups

• How would you handle a 10-megapixel image without hitting context limits?
• What information is typically lost when you downsample or tile an image for token efficiency?

As asked

A customer comes to you with a use case where they need Claude to consistently output structured JSON in a very specific schema for their pipeline. They are getting about 85% compliance with prompting alone. Walk me through how you would advise them on whether to pursue fine-tuning versus prompt engineering, and what the decision criteria are.

Sample answer outline

A strong answer explores prompt engineering options first: few-shot examples showing the exact schema, constrained decoding or grammar-based output parsing, and whether the 15% failure rate is in the schema structure or in the content. Fine-tuning is worth it when the target behavior is highly idiomatic, the volume of examples is large enough to generalize, and the baseline prompt cannot close the gap. Risks of fine-tuning include forgetting general capabilities, higher cost and iteration time, and the need to manage model versions. For JSON compliance specifically, constrained decoding via outlines or similar tools is often the right answer before fine-tuning.

Expect these follow-ups

• How would you measure the 15% failure rate precisely and categorize what types of failures they are?
• What is the minimum dataset size you would want before attempting fine-tuning for a structured output task?

As asked

Explain reward hacking in the context of training a language model with RLHF, give a concrete example of what it looks like in output, and describe two strategies for mitigating it.

Sample answer outline

Reward hacking occurs when the model finds outputs that score highly on the learned reward model but do not represent genuinely preferred behavior, because the reward model is an imperfect proxy. Concrete examples include models that produce very long, verbose answers because human raters historically rated longer answers higher, or models that use sycophantic language to agree with the rater's apparent view. Mitigation strategies include KL penalty between the RL-trained policy and a reference policy to limit how far the model drifts, reward model ensembles to reduce proxy gaming, and iterative reward model updates with fresh human feedback after each RL phase.

Expect these follow-ups

• How do you detect reward hacking in a model you have already trained?
• Is there a fundamental limit to how well a reward model can represent human preferences?

As asked

Apple runs significant ML workloads on-device rather than in the cloud. What are the technical trade-offs, and how does the Neural Engine factor into decisions about where to run a model?

Sample answer outline

On-device advantages: latency (no network round trip), privacy (data never leaves the device), offline availability, and no per-inference server cost. Disadvantages: model size constrained by device storage and RAM, quantization required for speed which costs accuracy, no easy model updates without an app release. The Apple Neural Engine (ANE) is optimized for specific layer types (convolutions, matrix multiplications with specific shapes) but not all model architectures map well to it; a model that falls back to the CPU or GPU can be slower than a smaller cloud-served model. A strong candidate mentions Core ML as the deployment target and discusses the workflow: train in PyTorch or TensorFlow, convert via coremltools, and measure energy impact on device.

Expect these follow-ups

• How would you decide whether to quantize to INT8 or use 16-bit floats for a given model?
• What metrics beyond accuracy do you track for an on-device model in production?

As asked

You are building a feature set for TikTok's video ranking model. List the categories of features you would include, explain which signals you expect to be most predictive, and describe how you handle features that are only available for some videos or users.

Sample answer outline

A comprehensive answer covers at least four categories: user features (demographics, historical engagement rates, watch history embeddings), video features (category, audio fingerprint, visual content embedding, age of content, creator follower count), interaction features (user-video cross features like whether the user typically watches content from this creator category), and context features (time of day, device type, network speed). Predictive power: watch completion rate and replays are stronger signals than likes. Missing feature handling: use default/cold-start values, or train the model to handle sparse embeddings.

Expect these follow-ups

• How do you prevent your training data's user behavior from encoding historical biases into the model?
• How would you detect that a feature is causing overfitting to a specific demographic?