ML engineer at Datadog

As asked

Walk me through the biggest production incident you've personally been on the front line of. What happened, what did you do, and what changed afterwards?

Sample answer outline

Use the STAR structure but lean into the technical specifics. Briefly describe the impact (users affected, revenue, duration). Walk through detection (what alerted you and how), diagnosis (what you ruled out, the dead-end paths), mitigation (the actual fix), and the postmortem outcome (what changed in the system or process). Interviewers look for ownership, calm under pressure, root-cause thinking (not 'we restarted it'), and the discipline to convert pain into permanent fixes.

Expect these follow-ups

• What would you do differently if you saw the same symptoms tomorrow?
• Whose fault was it? How did you handle that conversation?
• Did the postmortem actions actually land, or did they slip?

As asked

Explain the bias-variance tradeoff. Use a concrete worked example, not just the definitions.

Sample answer outline

Bias is error from oversimplifying assumptions; variance is error from over-fitting to training noise. Concrete example: predicting house prices. A constant model (predict the mean) has very high bias but zero variance. A degree-15 polynomial on 100 points has very low bias but huge variance - it tracks the training data perfectly and fails on new data. The optimum is somewhere in between. Mitigation: regularisation (L1/L2), cross-validation to estimate the curve, ensembling to average out variance.

Expect these follow-ups

• How does L2 regularisation specifically reduce variance?
• Where does the bias-variance decomposition come from mathematically?
• Why does bagging reduce variance but boosting reduce bias?

As asked

Tell me about the biggest incident you handled that lived in the platform layer: a broken deploy pipeline, an infrastructure change gone wrong, a certificate expiry, or a cluster failure. Walk me through detection, mitigation, and what changed in the platform afterwards.

Sample answer outline

Frame it through a platform lens, where the blast radius is every team that depends on you. Describe the impact across consumers, not just one service. Detection: what alerted you, and whether it was your monitoring or a downstream team that noticed first. Mitigation: the rollback or break-glass procedure, and whether it existed before the incident or had to be improvised. The strong answer ends with platform-level prevention: a guardrail in the pipeline, a pre-deploy check, an expiry alert, automated rollback. Interviewers listen for ownership of shared infrastructure and the discipline to turn one painful event into a control that protects every team.

Expect these follow-ups

• Did a self-service guardrail exist, or did you have to build one after?
• How did you communicate with the many teams affected at once?
• What pipeline or infrastructure check would have caught this earlier?

As asked

Explain overfitting and underfitting in plain terms, and tell me how you would notice that a model is overfitting. Keep it concrete.

Sample answer outline

Frame the tradeoff through its everyday symptoms rather than the formal decomposition. Underfitting is when a model is too simple to capture the pattern, so it does poorly on both the training data and new data. Overfitting is when a model memorises the training data, including its noise, so it looks great on training but does badly on data it has not seen. You spot overfitting by a large gap between training accuracy and validation accuracy. Mention the simplest fixes an earlier-career engineer reaches for: get more data, use a simpler model, or add regularisation. A concrete picture, such as a wiggly curve threading every training point, lands better than equations here.

Expect these follow-ups

• What does a big gap between training and validation scores tell you?
• Name one quick way to reduce overfitting.
• Why does more data usually help with overfitting?

As asked

You are building a classifier where the positive class is one percent of the data, say fraud or a rare disease. Why is accuracy the wrong metric, and how do you actually evaluate and tune the model?

Sample answer outline

On a one percent positive rate a model that predicts negative for everyone scores 99 percent accuracy and is useless, so accuracy hides total failure on the class you care about. Evaluate with metrics that focus on the positive class: precision and recall, and the precision-recall curve, which is more informative than the ROC curve under heavy imbalance because it does not get flattered by the huge number of true negatives. Which way you trade precision against recall is a business decision, missing a fraud case versus annoying a good customer, so tune the decision threshold to that cost rather than defaulting to 0.5. For training, options include class weighting or resampling, but be honest that resampling changes the base rate and you must calibrate or correct probabilities afterward if you need them to mean something. The signal is choosing metrics and a threshold that reflect the real cost of each error, not a single headline number.

Expect these follow-ups

• Why prefer a precision-recall curve over ROC under heavy imbalance?
• How do you choose the decision threshold rather than defaulting to 0.5?
• What does resampling do to your predicted probabilities?

As asked

Walk me through what happens when you call loss.backward() in PyTorch. How does autograd build the computation graph, and what are the memory implications of retaining it?

Sample answer outline

A strong answer explains that PyTorch builds a dynamic computation graph during the forward pass, attaching grad_fn nodes to tensors. backward() traverses the graph in reverse, applying the chain rule at each node. retain_graph=True is needed for multiple backward passes but prevents freeing intermediate activations, which can double GPU memory usage in practice.

Expect these follow-ups

• What happens if you call backward() twice without retain_graph=True?
• How does gradient accumulation work and why is it used in large batch training?