Data scientist machine learning interview questions

As asked

Explain the bias-variance tradeoff. Use a concrete worked example, not just the definitions.

Sample answer outline

Bias is error from oversimplifying assumptions; variance is error from over-fitting to training noise. Concrete example: predicting house prices. A constant model (predict the mean) has very high bias but zero variance. A degree-15 polynomial on 100 points has very low bias but huge variance - it tracks the training data perfectly and fails on new data. The optimum is somewhere in between. Mitigation: regularisation (L1/L2), cross-validation to estimate the curve, ensembling to average out variance.

Expect these follow-ups

• How does L2 regularisation specifically reduce variance?
• Where does the bias-variance decomposition come from mathematically?
• Why does bagging reduce variance but boosting reduce bias?

As asked

Explain overfitting and underfitting in plain terms, and tell me how you would notice that a model is overfitting. Keep it concrete.

Sample answer outline

Frame the tradeoff through its everyday symptoms rather than the formal decomposition. Underfitting is when a model is too simple to capture the pattern, so it does poorly on both the training data and new data. Overfitting is when a model memorises the training data, including its noise, so it looks great on training but does badly on data it has not seen. You spot overfitting by a large gap between training accuracy and validation accuracy. Mention the simplest fixes an earlier-career engineer reaches for: get more data, use a simpler model, or add regularisation. A concrete picture, such as a wiggly curve threading every training point, lands better than equations here.

Expect these follow-ups

• What does a big gap between training and validation scores tell you?
• Name one quick way to reduce overfitting.
• Why does more data usually help with overfitting?

As asked

You are building a classifier where the positive class is one percent of the data, say fraud or a rare disease. Why is accuracy the wrong metric, and how do you actually evaluate and tune the model?

Sample answer outline

On a one percent positive rate a model that predicts negative for everyone scores 99 percent accuracy and is useless, so accuracy hides total failure on the class you care about. Evaluate with metrics that focus on the positive class: precision and recall, and the precision-recall curve, which is more informative than the ROC curve under heavy imbalance because it does not get flattered by the huge number of true negatives. Which way you trade precision against recall is a business decision, missing a fraud case versus annoying a good customer, so tune the decision threshold to that cost rather than defaulting to 0.5. For training, options include class weighting or resampling, but be honest that resampling changes the base rate and you must calibrate or correct probabilities afterward if you need them to mean something. The signal is choosing metrics and a threshold that reflect the real cost of each error, not a single headline number.

Expect these follow-ups

• Why prefer a precision-recall curve over ROC under heavy imbalance?
• How do you choose the decision threshold rather than defaulting to 0.5?
• What does resampling do to your predicted probabilities?

As asked

You have a categorical feature 'city' with 3,000 unique values and you are building an XGBoost churn model. What encoding strategies would you consider, and what are the tradeoffs of each?

Sample answer outline

The candidate should cover target encoding (mean of label per category) with leave-one-out or smoothing to prevent leakage, hashing tricks for memory efficiency, and embeddings if a neural branch is acceptable. They should flag that one-hot encoding is impractical at 3,000 cardinality. Key risks to mention: target leakage through the training fold for target encoding and hash collisions.

Expect these follow-ups

• How would you implement target encoding in a scikit-learn pipeline without leaking labels?
• When would you prefer an embedding layer over target encoding?

As asked

You are building a fraud detection model where 0.1% of transactions are fraudulent. You train a logistic regression and get 99.9% accuracy. What is wrong with that number, and how do you actually build and evaluate a useful model here?

Sample answer outline

The candidate should immediately flag that predicting all negatives gives 99.9% accuracy, making accuracy useless. Evaluation should shift to precision-recall curves, F-beta score (weighting recall more), and AUC-PR. Techniques: class-weight adjustment, SMOTE or other oversampling on training data only, threshold tuning post-training, and cost-sensitive learning. Emphasis on never oversampling before a train-test split.

Expect these follow-ups

• Why is AUC-ROC a poor choice for this problem, and when is it misleading?
• How do you choose the classification threshold after training?

As asked

Your model has a training RMSE of 0.04 and a validation RMSE of 0.28. What does this tell you, and what concrete steps would you take to improve generalization?

Sample answer outline

The large gap between training and validation error is a hallmark of high variance (overfitting). Steps to address: add regularization, reduce model complexity, get more training data, increase dropout if using neural networks, and use ensemble methods like bagging. The candidate should also verify the validation set is representative and that no leakage caused artificially low training error.

Expect these follow-ups

• How would you diagnose if the gap is due to data leakage versus genuine overfitting?

As asked

We are building a model to predict the number of days until a customer churns. You report MAE and RMSE and your manager asks which one to trust. Walk me through the tradeoff and when each is more informative.

Sample answer outline

MAE treats all errors equally and is robust to outliers; RMSE penalizes large errors quadratically, making it sensitive to big misses. If underestimating churn for high-value customers is especially costly, RMSE is more informative because those large errors dominate. If the distribution of churn times is heavy-tailed, report both. The candidate should also mention MAPE as a percentage-based alternative and its failure when true values are near zero.

Expect these follow-ups

• When would you use MAPE instead of MAE, and what is its failure mode?

As asked

You train two classifiers. Model A has AUC-ROC of 0.82 and Model B has 0.79. Your team wants to ship Model A. What does this metric actually measure, and what would make you reconsider?

Sample answer outline

AUC-ROC is the probability that a randomly drawn positive example scores higher than a randomly drawn negative example. It is threshold-agnostic and measures rank-order discrimination rather than calibrated probabilities. However, AUC-ROC can be misleadingly high on severely imbalanced datasets because the ROC curve is insensitive to the large true-negative count. If the dataset has a 1% positive rate, a model can achieve a high AUC while performing poorly on the minority class. The candidate should suggest AUC-PR for imbalanced problems and always calibrate probabilities before using them for decision-making.

Expect these follow-ups

• What does a perfectly calibrated model mean, and how do you check calibration in practice?

As asked

A new creator posts their first video on TikTok. There is no watch history, no engagement data, nothing. How does your recommendation system decide who to show this video to, and how does it bootstrap the feedback loop?

Sample answer outline

Strong answers address both the item cold start (no engagement data) and the user cold start (new user). For item cold start: use content-based signals (audio, visuals, text description, creator profile), show the video to a small random sample from the creator's existing follower graph, then use early engagement signals (first 100 views) to decide whether to expand distribution. For user cold start: use onboarding interest selection, geographic defaults, or implicit signals from the referral source. The candidate should mention the risk of seeding the wrong distribution early.

Expect these follow-ups

• How long do you wait before deciding a video is low quality versus just slow to gain traction?
• How does the cold start strategy change for a creator who is already famous on another platform?

As asked

You are setting up a machine learning platform for a team of 10 data scientists who will train dozens of models a week. Walk me through how you would use MLflow on Databricks to track experiments, compare runs, register the best model, and gate promotion from staging to production.

Sample answer outline

A strong answer covers: logging parameters, metrics, and artifacts with mlflow.log_param and mlflow.log_metric inside training runs; using experiment UI or mlflow.search_runs to compare across runs; registering the best run to the Model Registry with mlflow.register_model; using staging and production stages with manual or automated approval gates (e.g., a CI job that runs evaluation metrics before promoting); and serving the production model via Databricks Model Serving endpoints. The candidate should mention model aliases in the newer MLflow API which replace the stage transitions.

Expect these follow-ups

• How would you enforce that a model cannot go to production unless it has an AUC above a certain threshold?
• What would you log as artifacts beyond the model weights to make a run fully reproducible?

As asked

Neural networks are popular - but when is a neural network the wrong tool for the job?

Sample answer outline

Wrong tool when: data is small (under ~10k examples - gradient boosted trees usually win on tabular data, full stop); interpretability is required (regulated industries, medical decisions, anything that needs SHAP-style attribution that holds up to scrutiny); inference latency is tight and the model would not fit; features are mostly categorical and high-cardinality; the cost of false positives is asymmetric and you need calibrated probabilities. XGBoost, LightGBM, and CatBoost on engineered features are still state of the art for most tabular problems.

Expect these follow-ups

• Why do gradient boosted trees beat neural networks on tabular data?
• How would you explain a tree ensemble's predictions to a regulator?
• When do neural networks make sense for tabular data?

As asked

Design a feature store that serves features to both training jobs and online inference, with consistent semantics between batch and realtime.

Sample answer outline

Two storage tiers: offline (a columnar warehouse like BigQuery or Iceberg, used by training jobs) and online (a low-latency KV like Redis or DynamoDB, used by serving). The core invariant: a feature should produce the same value at training time and serving time, otherwise you get training-serving skew. Solutions: define features as code (e.g. SQL or DSL), compute them once and write to both tiers; use point-in-time joins for backfills so you don't leak future data; tag features with freshness SLAs and version them.

Expect these follow-ups

• How do you handle a feature whose definition changes?
• What is point-in-time correct join and why does it matter?
• How do you prevent training-serving skew when offline and online run different code paths?

As asked

You are not training a model from scratch, you are building a feature on top of a foundation model with prompting, retrieval, and maybe light fine-tuning. How does bias-variance thinking still apply to the choices you make, and how do you tell whether your system is underfitting or overfitting the task?

Sample answer outline

Translate the concept into the applied-AI setting. Underfitting shows up as a system that is too generic: weak prompts, no retrieval, and a model that gives plausible but off-target answers because it lacks the task context. Overfitting shows up as a system tuned so tightly to a handful of examples that it fails on real, varied inputs: brittle few-shot prompts, a fine-tune on a tiny biased dataset, or retrieval that only works for the queries you tested. The applied controls are the modern analogues of regularisation and capacity: richer context and retrieval reduce bias, while a held-out evaluation set, diverse examples, and resisting over-tuning to a demo reduce variance. The AI-engineer signal is measuring this with an evaluation harness on representative inputs rather than eyeballing a few prompts, and knowing when a prompt change is genuinely better versus fitted to your test cases.

Expect these follow-ups

• How do you build an evaluation set that reveals overfitting to your examples?
• When does adding retrieval reduce bias versus just adding noise?
• Why can a fine-tune on a small dataset make a model worse on real inputs?

As asked

Your model scores beautifully offline but falls apart in production. You suspect data leakage. Explain what leakage is, the common ways it sneaks in, and how you would build the pipeline so it cannot happen.

Sample answer outline

Leakage is when information that would not be available at prediction time leaks into training, so the model looks great offline and collapses in the real world. The classic forms: fitting a scaler or encoder on the full dataset before splitting, so test statistics bleed into training; including a feature that is a proxy for the label or is only populated after the outcome is known; and time leakage, where a row uses information from the future relative to when the prediction would actually be made. The structural fixes are to split first and fit every transform inside the training fold only, then apply it to validation and test, ideally wrapped in a pipeline so the discipline is enforced rather than remembered. For time-based problems, use point-in-time correct joins and time-ordered splits so you never train on the future. The strong answer treats the suspiciously high offline score as a symptom to investigate, not a result to trust.

Expect these follow-ups

• Why does fitting a scaler before the train-test split leak information?
• How does a point-in-time join prevent time leakage?
• What is your first check when offline metrics look too good?

As asked

Your recommendation system works well for established users and items, but it has nothing useful to show a brand new user or to surface a freshly added item. How do you handle cold start on both sides?

Sample answer outline

Cold start is two problems. For a new user with no interaction history, fall back to signals you do have: popularity and trending items, anything from onboarding such as stated interests, and contextual cues like location or referral source, then shift to personalised collaborative signals as interactions accumulate. For a new item with no interactions, lean on content features, its attributes, text, or embeddings, so it can be matched to users by similarity to items they already like, rather than waiting for engagement data it does not have yet. A hybrid that blends content-based and collaborative signals degrades gracefully at the edges instead of returning nothing. Add deliberate exploration so new items get a fair chance to be shown and gather the feedback they need, otherwise popular items starve everything new. The strong answer names both sides explicitly and treats cold start as a smooth handoff from content and context to behaviour, not a switch that flips once.

Expect these follow-ups

• What signals do you use for a user with zero interaction history?
• How do content features help a brand new item get recommended?
• Why does a recommender need deliberate exploration?

As asked

Given a raw dataset of user events with timestamps, how do you think about turning it into features for a model that predicts churn? Show me how you reason about it rather than listing transformations.

Sample answer outline

Start from the prediction and the user, not the columns. For churn the useful signals are about engagement over time, so derive recency, frequency, and trend features: how long since the last meaningful action, how often the user acts in a window, and crucially whether activity is rising or falling, since a declining trend predicts churn better than a static count. Aggregate raw events into windows like last 7 and last 30 days so the model sees behaviour, not individual rows, and be strict that every window only uses data available before the prediction point to avoid leakage. Encode categoricals thoughtfully, watching cardinality, and capture lifecycle context like tenure. Throughout, prefer features a domain person would expect to matter and that you can explain, and validate that a feature actually improves held-out performance rather than adding everything and hoping. The signal is reasoning from the user behaviour you are trying to capture, with leakage discipline baked in, not reciting a transformation checklist.

Expect these follow-ups

• Why is the trend of activity often more predictive than its level?
• How do you make sure windowed features do not leak the future?
• How do you decide whether a new feature is actually worth keeping?

As asked

You have 500 candidate features for a linear model. Explain how L1 and L2 regularization behave differently when it comes to feature selection, and in what situations you would prefer each one.

Sample answer outline

L1 (Lasso) drives some coefficients to exactly zero because its penalty has a corner at zero, naturally performing feature selection. L2 (Ridge) shrinks coefficients toward zero but rarely to exactly zero, preferring to spread weight across correlated features. Elastic Net combines both. L1 is preferred when true sparsity is assumed; L2 when all features likely contribute and multicollinearity is present. The candidate should also mention standardization is required before applying either.

Expect these follow-ups

• What happens to correlated features under Lasso versus Ridge?

As asked

Explain how gradient boosted decision trees (like XGBoost or LightGBM) fit each tree in the sequence. Specifically, what is each tree fitted to, and how does the learning rate interact with the number of trees?

Sample answer outline

Each tree is fitted to the pseudo-residuals, which are the negative gradients of the loss function with respect to the current ensemble prediction. For squared error loss these are the raw residuals; for log-loss they are the difference between label and predicted probability. The learning rate scales each tree's contribution, so a smaller rate requires more trees but generalizes better. The candidate should mention early stopping on a validation set to find the optimal number of trees.

Expect these follow-ups

• How does LightGBM's leaf-wise growth strategy differ from XGBoost's level-wise, and what are the implications?

As asked

You are building a model to forecast next week's sales using two years of weekly data. A colleague suggests 5-fold cross-validation. Why is this problematic, and what would you do instead?

Sample answer outline

Standard k-fold randomly shuffles folds, so the model trains on future data to predict the past, which causes data leakage and overestimates generalization performance. The correct approach is time-series cross-validation (expanding window or rolling window): each fold's validation set is always strictly after the training set. The candidate should also mention that seasonality patterns may require full year-long validation windows to avoid optimistic estimates.

Expect these follow-ups

• How does expanding-window CV differ from rolling-window CV, and when would you prefer each?

As asked

A churn model has been in production for six months. The business reports that model-flagged users are churning at a lower rate than expected, but actual churn has risen. What could explain this and how do you investigate?

Sample answer outline

Two problems may be happening together: concept drift (the relationship between features and churn has changed, e.g., due to a product change) and intervention feedback bias (the model's own predictions trigger retention campaigns that lower churn in the flagged group). Investigation steps: monitor feature distribution shift with PSI or KS tests, compare recent population scores to training-time distribution, retrain on recent data, and set up a holdout that receives no intervention to measure true model performance.

Expect these follow-ups

• What is Population Stability Index (PSI) and what threshold triggers a retraining?

As asked

You build a collaborative filtering recommendation model and compute offline precision@10 on a held-out test set. It improves by 8%. You ship it and online CTR drops by 2%. Why might this happen?

Sample answer outline

Offline metrics measured on logged data suffer from exposure bias: the test set only contains items the previous system showed, so precision@10 only measures recall of past recommendations, not true user preferences. The new model may recommend items that users would actually prefer but that never appeared in the logged data. The candidate should mention debiasing approaches like inverse propensity scoring, and argue that online A/B tests are the ground truth for recommendation systems.

Expect these follow-ups

• How would you use inverse propensity scoring to debias offline evaluation?

As asked

You have embeddings of 1 million products in 256 dimensions and want to explore clusters visually. When would you use PCA versus t-SNE or UMAP, and what are the interpretive limits of each?

Sample answer outline

PCA is linear, fast at scale, and preserves global variance, but compresses complex manifold structure poorly. t-SNE preserves local neighborhood structure and is excellent for visualizing clusters, but distances between clusters are not meaningful and it does not scale well past 100k points without approximations. UMAP is faster than t-SNE, scales better, and preserves more global structure. For 1 million points: PCA first to 50 dims, then UMAP on the reduced representation.

Expect these follow-ups

• Why can two runs of t-SNE on the same data produce different cluster layouts?

As asked

You are tuning a LightGBM model with 15 hyperparameters and limited compute. Grid search is out of the question. Walk me through your strategy, which parameters matter most, and how you avoid overfitting to the validation set during tuning.

Sample answer outline

Use Bayesian optimization (Optuna or Hyperopt) which samples efficiently by building a surrogate model of the objective. Priority parameters: num_leaves, min_data_in_leaf (tree complexity), learning_rate (always tune last), and regularization terms (lambda_l1, lambda_l2). Use cross-validation inside each trial, not a single val fold, to avoid tuning to noise. The candidate should mention using a final held-out test set that was never touched during tuning.

Expect these follow-ups

• What is the risk of using the same validation set for both early stopping and hyperparameter selection?

As asked

A business stakeholder wants to know why the model predicted a specific customer is high churn risk. Explain what SHAP values are, how they are computed conceptually, and one practical limit when using TreeExplainer on a LightGBM model.

Sample answer outline

SHAP values are grounded in cooperative game theory: the Shapley value of a feature is its average marginal contribution across all possible orderings of features. TreeExplainer computes exact Shapley values for tree models in polynomial time by traversing the tree and computing expected outputs over subsets. A practical limit: SHAP measures feature attribution relative to a background dataset (the training distribution), so if the background distribution is not representative, local explanations can be misleading.

Expect these follow-ups

• How do SHAP interaction values differ from main SHAP values, and when would you compute them?

As asked

You train a model to predict loan default and get 0.97 AUC on the test set. Your colleague is suspicious. Name three concrete ways data leakage could produce this result and how you would detect each one.

Sample answer outline

Three common leakage types: (1) future information in features, such as a field populated only after default occurs (detect: feature audit against data dictionary, check which fields are available at prediction time); (2) target leakage, where a feature is a proxy for the label populated after the fact; (3) train-test contamination, such as normalizing or encoding with statistics computed over the full dataset before splitting. The candidate should advocate for a strict chronological split and pipeline-based preprocessing.

Expect these follow-ups

• How would you use a feature importance analysis to flag suspected leaky features?