Data scientist at Datadog

As asked

Explain what a p-value is to a non-technical product manager in under two minutes.

Sample answer outline

A p-value answers: 'if the new variant was actually the same as control, how often would I see data this surprising by pure chance?' A small p-value (typically <0.05) means: this result would be rare if nothing was really different, so we have evidence something IS different. Critically, the p-value does NOT tell you the probability the new variant is better, or by how much. For business decisions, pair it with effect size and confidence interval. Warn against p-hacking and peeking.

Expect these follow-ups

• What does a confidence interval add that the p-value doesn't?
• When would you accept p=0.10 as good enough?
• Why is peeking at p-values dangerous?

As asked

Explain the bias-variance tradeoff. Use a concrete worked example, not just the definitions.

Sample answer outline

Bias is error from oversimplifying assumptions; variance is error from over-fitting to training noise. Concrete example: predicting house prices. A constant model (predict the mean) has very high bias but zero variance. A degree-15 polynomial on 100 points has very low bias but huge variance - it tracks the training data perfectly and fails on new data. The optimum is somewhere in between. Mitigation: regularisation (L1/L2), cross-validation to estimate the curve, ensembling to average out variance.

Expect these follow-ups

• How does L2 regularisation specifically reduce variance?
• Where does the bias-variance decomposition come from mathematically?
• Why does bagging reduce variance but boosting reduce bias?

As asked

In plain language, what is a p-value, and what does a small one tell you? Imagine you are explaining it to a teammate who has not studied statistics.

Sample answer outline

Keep it intuitive and avoid jargon. A p-value answers one question: if there were really no difference between the two things you are comparing, how surprising would your data be by chance alone? A small p-value means your result would be unlikely if nothing was going on, so it is evidence that something real is happening. The one misconception to head off, even at this level, is that a p-value is not the probability that your idea is correct, and a result can be statistically significant while being too small to care about. A clear, honest, non-overclaiming explanation is the whole point.

Expect these follow-ups

• Does a small p-value mean the effect is large?
• What threshold do people commonly use, and is it special?
• What does the p-value not tell you?

As asked

Explain overfitting and underfitting in plain terms, and tell me how you would notice that a model is overfitting. Keep it concrete.

Sample answer outline

Frame the tradeoff through its everyday symptoms rather than the formal decomposition. Underfitting is when a model is too simple to capture the pattern, so it does poorly on both the training data and new data. Overfitting is when a model memorises the training data, including its noise, so it looks great on training but does badly on data it has not seen. You spot overfitting by a large gap between training accuracy and validation accuracy. Mention the simplest fixes an earlier-career engineer reaches for: get more data, use a simpler model, or add regularisation. A concrete picture, such as a wiggly curve threading every training point, lands better than equations here.

Expect these follow-ups

• What does a big gap between training and validation scores tell you?
• Name one quick way to reduce overfitting.
• Why does more data usually help with overfitting?

As asked

You are building a classifier where the positive class is one percent of the data, say fraud or a rare disease. Why is accuracy the wrong metric, and how do you actually evaluate and tune the model?

Sample answer outline

On a one percent positive rate a model that predicts negative for everyone scores 99 percent accuracy and is useless, so accuracy hides total failure on the class you care about. Evaluate with metrics that focus on the positive class: precision and recall, and the precision-recall curve, which is more informative than the ROC curve under heavy imbalance because it does not get flattered by the huge number of true negatives. Which way you trade precision against recall is a business decision, missing a fraud case versus annoying a good customer, so tune the decision threshold to that cost rather than defaulting to 0.5. For training, options include class weighting or resampling, but be honest that resampling changes the base rate and you must calibrate or correct probabilities afterward if you need them to mean something. The signal is choosing metrics and a threshold that reflect the real cost of each error, not a single headline number.

Expect these follow-ups

• Why prefer a precision-recall curve over ROC under heavy imbalance?
• How do you choose the decision threshold rather than defaulting to 0.5?
• What does resampling do to your predicted probabilities?

As asked

Before launching an A/B test, a PM asks how long it needs to run. Walk me through how you size it: what inputs you need, what the minimum detectable effect means, and the traps that make people stop too early.

Sample answer outline

Sizing comes from four inputs: the baseline rate of the metric, the minimum detectable effect you care about, the significance level, and the desired power. The minimum detectable effect is the smallest true change worth detecting, and smaller effects need dramatically larger samples, which is the lever most people underestimate. From those you compute the sample per arm, then divide by daily eligible traffic to get duration, and round up to whole weeks so weekday and weekend behaviour are represented rather than over-weighting whichever days you happened to catch. The trap is peeking and stopping the moment significance appears, which inflates false positives because you gave yourself many chances to cross the line; commit to the planned horizon, or use a sequential method designed for valid early stopping. Also resist declaring a flat result a win for the control too soon if the test was underpowered to begin with. The signal is treating duration as the output of a power calculation, not a guess.

Expect these follow-ups

• Why do smaller detectable effects need so much more traffic?
• Why round the duration to whole weeks?
• What is wrong with stopping as soon as the result looks significant?