Data scientist statistics interview questions

As asked

Explain what a p-value is to a non-technical product manager in under two minutes.

Sample answer outline

A p-value answers: 'if the new variant was actually the same as control, how often would I see data this surprising by pure chance?' A small p-value (typically <0.05) means: this result would be rare if nothing was really different, so we have evidence something IS different. Critically, the p-value does NOT tell you the probability the new variant is better, or by how much. For business decisions, pair it with effect size and confidence interval. Warn against p-hacking and peeking.

Expect these follow-ups

• What does a confidence interval add that the p-value doesn't?
• When would you accept p=0.10 as good enough?
• Why is peeking at p-values dangerous?

As asked

You want to measure the impact of a new feature, but you cannot randomise users to it. What approaches do you consider, and how do you decide?

Sample answer outline

Quasi-experiments are for when an A/B test is impossible (legal, technical, network effects, you only have one product to ship). Options: difference-in-differences (compare pre/post change across treated vs control groups), regression discontinuity (when treatment is assigned by a cutoff), synthetic control (construct a fake control from a weighted sum of similar units), instrumental variables (when you have a natural source of exogenous variation). Each requires assumptions to make causal claims, and you should be explicit about which assumption could break.

Expect these follow-ups

• How do you handle network effects in an A/B test of a social feature?
• When does difference-in-differences fail?
• How do you communicate uncertainty from a quasi-experiment to executives?

As asked

In plain language, what is a p-value, and what does a small one tell you? Imagine you are explaining it to a teammate who has not studied statistics.

Sample answer outline

Keep it intuitive and avoid jargon. A p-value answers one question: if there were really no difference between the two things you are comparing, how surprising would your data be by chance alone? A small p-value means your result would be unlikely if nothing was going on, so it is evidence that something real is happening. The one misconception to head off, even at this level, is that a p-value is not the probability that your idea is correct, and a result can be statistically significant while being too small to care about. A clear, honest, non-overclaiming explanation is the whole point.

Expect these follow-ups

• Does a small p-value mean the effect is large?
• What threshold do people commonly use, and is it special?
• What does the p-value not tell you?

As asked

A stakeholder wants to call experiments as soon as p drops below 0.05, run many metrics at once, and rerun tests that did not reach significance. Explain precisely why each of these inflates false positives, and what you would put in place instead.

Sample answer outline

This probes whether the candidate understands the failure modes, not just the definition. Peeking and stopping at the first significant moment inflates the false-positive rate far above the nominal alpha because you get many chances to cross the threshold; the fix is a pre-registered sample size or a sequential testing method with always-valid p-values or alpha spending. Testing many metrics multiplies the family-wise error rate; control it with a correction such as Benjamini-Hochberg or by designating one primary metric. Rerunning failed tests is selection bias dressed up as rigour. The strong answer reframes the whole programme around effect size and confidence intervals, a fixed decision rule agreed before the test, and guardrail metrics, treating the p-value as one input rather than the verdict.

Expect these follow-ups

• Why does optional stopping inflate the false-positive rate?
• When would you use sequential testing instead of a fixed horizon?
• How do you control error across many metrics without killing power?

As asked

A data scientist tells you a test came back with p = 0.03 and a 1.5 percent lift. As the product manager deciding whether to ship, how do you interpret that, and what questions do you ask before you commit?

Sample answer outline

A PM does not need to derive the maths but must read results without being fooled. The p-value of 0.03 says the result is unlikely if the feature did nothing, so there is reasonable evidence of a real effect, but it says nothing about size or business value. The 1.5 percent lift is the part that matters for the ship decision, so ask about the confidence interval around it: a lift of 1.5 percent that could plausibly be near zero is a different decision from one tightly bounded above one percent. Ask whether the test ran its planned duration or was stopped early, whether guardrail metrics held, and whether the effect is worth the cost and complexity to ship and maintain. The product signal is connecting statistical evidence to a value judgement, not treating significance as an automatic green light.

Expect these follow-ups

• Why do you care more about the confidence interval than the p-value here?
• What guardrail metrics would you insist on before shipping?
• When would you not ship a statistically significant result?

As asked

A teammate is using plain k-fold cross-validation for everything. Explain when that gives you an honestly biased estimate, and what validation scheme you would use for time series and for grouped data.

Sample answer outline

Plain k-fold assumes rows are independent and identically distributed, and quietly lies when they are not. With time series it shuffles future and past together, so the model trains on the future to predict the past and the estimate is optimistic; use a forward-chaining scheme where each fold trains on the past and validates on the next slice, mirroring how the model will actually run. With grouped data, say multiple rows per customer, random folds put the same customer in both train and validation, leaking identity and inflating the score; use grouped folds that keep all of a group on one side. For classification with rare classes, stratify so each fold preserves the class balance and your estimate is stable. The point to land is that cross-validation is only honest when the splits respect the dependence structure of the data, so the validation setup must mimic the real prediction setting.

Expect these follow-ups

• Why does shuffled k-fold give an optimistic score on time series?
• What goes wrong when the same group lands in train and validation?
• When do you need stratified folds?

As asked

A dataset you need to model has columns with a lot of missing values. Walk me through how you decide what to do, and why dropping every row with a gap or just filling with the mean can quietly bias your results.

Sample answer outline

The first question is why the data is missing, because the mechanism drives the right response. If values are missing completely at random, dropping rows loses power but does not bias; if missingness depends on observed variables, you can model it; if it depends on the unobserved value itself, for example high earners declining to report income, any naive fill introduces bias and the missingness itself is informative. Dropping every incomplete row is dangerous when missingness is not random, because you systematically remove a subset and skew the sample. Mean imputation is convenient but shrinks variance, distorts correlations, and pretends to a certainty you do not have. Better options depending on the situation: model-based or multiple imputation that preserves uncertainty, and adding a missing-indicator feature so the model can use the fact that a value was absent, which often carries signal. The strong answer starts from the missingness mechanism rather than jumping to a default fill.

Expect these follow-ups

• How does the missingness mechanism change what you should do?
• Why does mean imputation distort the data even when it looks harmless?
• When is the fact that a value is missing itself a useful feature?

As asked

Before launching an A/B test, a PM asks how long it needs to run. Walk me through how you size it: what inputs you need, what the minimum detectable effect means, and the traps that make people stop too early.

Sample answer outline

Sizing comes from four inputs: the baseline rate of the metric, the minimum detectable effect you care about, the significance level, and the desired power. The minimum detectable effect is the smallest true change worth detecting, and smaller effects need dramatically larger samples, which is the lever most people underestimate. From those you compute the sample per arm, then divide by daily eligible traffic to get duration, and round up to whole weeks so weekday and weekend behaviour are represented rather than over-weighting whichever days you happened to catch. The trap is peeking and stopping the moment significance appears, which inflates false positives because you gave yourself many chances to cross the line; commit to the planned horizon, or use a sequential method designed for valid early stopping. Also resist declaring a flat result a win for the control too soon if the test was underpowered to begin with. The signal is treating duration as the output of a power calculation, not a guess.

Expect these follow-ups

• Why do smaller detectable effects need so much more traffic?
• Why round the duration to whole weeks?
• What is wrong with stopping as soon as the result looks significant?