Data scientist statistics interview questions

As asked

Explain what a p-value is to a non-technical product manager in under two minutes.

Sample answer outline

A p-value answers: 'if the new variant was actually the same as control, how often would I see data this surprising by pure chance?' A small p-value (typically <0.05) means: this result would be rare if nothing was really different, so we have evidence something IS different. Critically, the p-value does NOT tell you the probability the new variant is better, or by how much. For business decisions, pair it with effect size and confidence interval. Warn against p-hacking and peeking.

Expect these follow-ups

• What does a confidence interval add that the p-value doesn't?
• When would you accept p=0.10 as good enough?
• Why is peeking at p-values dangerous?

As asked

In plain language, what is a p-value, and what does a small one tell you? Imagine you are explaining it to a teammate who has not studied statistics.

Sample answer outline

Keep it intuitive and avoid jargon. A p-value answers one question: if there were really no difference between the two things you are comparing, how surprising would your data be by chance alone? A small p-value means your result would be unlikely if nothing was going on, so it is evidence that something real is happening. The one misconception to head off, even at this level, is that a p-value is not the probability that your idea is correct, and a result can be statistically significant while being too small to care about. A clear, honest, non-overclaiming explanation is the whole point.

Expect these follow-ups

• Does a small p-value mean the effect is large?
• What threshold do people commonly use, and is it special?
• What does the p-value not tell you?

As asked

Before launching an A/B test, a PM asks how long it needs to run. Walk me through how you size it: what inputs you need, what the minimum detectable effect means, and the traps that make people stop too early.

Sample answer outline

Sizing comes from four inputs: the baseline rate of the metric, the minimum detectable effect you care about, the significance level, and the desired power. The minimum detectable effect is the smallest true change worth detecting, and smaller effects need dramatically larger samples, which is the lever most people underestimate. From those you compute the sample per arm, then divide by daily eligible traffic to get duration, and round up to whole weeks so weekday and weekend behavior are represented rather than over-weighting whichever days you happened to catch. The trap is peeking and stopping the moment significance appears, which inflates false positives because you gave yourself many chances to cross the line; commit to the planned horizon, or use a sequential method designed for valid early stopping. Also resist declaring a flat result a win for the control too soon if the test was underpowered to begin with. The signal is treating duration as the output of a power calculation, not a guess.

Expect these follow-ups

• Why do smaller detectable effects need so much more traffic?
• Why round the duration to whole weeks?
• What is wrong with stopping as soon as the result looks significant?

As asked

You run an A/B test on a checkout flow and get a p-value of 0.03. Your manager says the result is significant and wants to ship. Walk me through exactly what that p-value tells you and what it does not tell you before you make a decision.

Sample answer outline

A strong answer explains that p=0.03 means there is a 3% probability of observing a result at least as extreme as this one under the null hypothesis, not a 97% probability that the variant is better. The candidate should mention type I error rate, the need to pre-register alpha, multiple comparisons, and the importance of effect size and confidence intervals alongside the p-value.

Expect these follow-ups

• How would your answer change if you had run 20 metrics in this test?
• What does it mean if the confidence interval for the lift includes zero but the p-value is below 0.05?

As asked

Before we launch an A/B test, how do you determine the minimum sample size needed? Walk me through the inputs, the formula, and the practical gotchas you watch for on a product with noisy metrics.

Sample answer outline

The candidate should name the four inputs: baseline conversion rate, minimum detectable effect, desired significance level (alpha), and statistical power (1 minus beta). They should explain the inverse relationship between MDE and required sample size and mention that variance of the metric drives everything. Gotchas: novelty effect, weekday vs weekend traffic, variance inflation for ratio metrics, and two-tailed vs one-tailed tests.

Expect these follow-ups

• How do you handle a metric whose variance is much higher than expected mid-experiment?
• When would you use a one-tailed test and what is the risk?

As asked

You want to measure the impact of a new feature, but you cannot randomise users to it. What approaches do you consider, and how do you decide?

Sample answer outline

Quasi-experiments are for when an A/B test is impossible (legal, technical, network effects, you only have one product to ship). Options: difference-in-differences (compare pre/post change across treated vs control groups), regression discontinuity (when treatment is assigned by a cutoff), synthetic control (construct a fake control from a weighted sum of similar units), instrumental variables (when you have a natural source of exogenous variation). Each requires assumptions to make causal claims, and you should be explicit about which assumption could break.

Expect these follow-ups

• How do you handle network effects in an A/B test of a social feature?
• When does difference-in-differences fail?
• How do you communicate uncertainty from a quasi-experiment to executives?

As asked

A stakeholder wants to call experiments as soon as p drops below 0.05, run many metrics at once, and rerun tests that did not reach significance. Explain precisely why each of these inflates false positives, and what you would put in place instead.

Sample answer outline

This probes whether the candidate understands the failure modes, not just the definition. Peeking and stopping at the first significant moment inflates the false-positive rate far above the nominal alpha because you get many chances to cross the threshold; the fix is a pre-registered sample size or a sequential testing method with always-valid p-values or alpha spending. Testing many metrics multiplies the family-wise error rate; control it with a correction such as Benjamini-Hochberg or by designating one primary metric. Rerunning failed tests is selection bias dressed up as rigour. The strong answer reframes the whole programme around effect size and confidence intervals, a fixed decision rule agreed before the test, and guardrail metrics, treating the p-value as one input rather than the verdict.

Expect these follow-ups

• Why does optional stopping inflate the false-positive rate?
• When would you use sequential testing instead of a fixed horizon?
• How do you control error across many metrics without killing power?

As asked

A data scientist tells you a test came back with p = 0.03 and a 1.5 percent lift. As the product manager deciding whether to ship, how do you interpret that, and what questions do you ask before you commit?

Sample answer outline

A PM does not need to derive the maths but must read results without being fooled. The p-value of 0.03 says the result is unlikely if the feature did nothing, so there is reasonable evidence of a real effect, but it says nothing about size or business value. The 1.5 percent lift is the part that matters for the ship decision, so ask about the confidence interval around it: a lift of 1.5 percent that could plausibly be near zero is a different decision from one tightly bounded above one percent. Ask whether the test ran its planned duration or was stopped early, whether guardrail metrics held, and whether the effect is worth the cost and complexity to ship and maintain. The product signal is connecting statistical evidence to a value judgement, not treating significance as an automatic green light.

Expect these follow-ups

• Why do you care more about the confidence interval than the p-value here?
• What guardrail metrics would you insist on before shipping?
• When would you not ship a statistically significant result?

As asked

A teammate is using plain k-fold cross-validation for everything. Explain when that gives you an honestly biased estimate, and what validation scheme you would use for time series and for grouped data.

Sample answer outline

Plain k-fold assumes rows are independent and identically distributed, and quietly lies when they are not. With time series it shuffles future and past together, so the model trains on the future to predict the past and the estimate is optimistic; use a forward-chaining scheme where each fold trains on the past and validates on the next slice, mirroring how the model will actually run. With grouped data, say multiple rows per customer, random folds put the same customer in both train and validation, leaking identity and inflating the score; use grouped folds that keep all of a group on one side. For classification with rare classes, stratify so each fold preserves the class balance and your estimate is stable. The point to land is that cross-validation is only honest when the splits respect the dependence structure of the data, so the validation setup must mimic the real prediction setting.

Expect these follow-ups

• Why does shuffled k-fold give an optimistic score on time series?
• What goes wrong when the same group lands in train and validation?
• When do you need stratified folds?

As asked

A dataset you need to model has columns with a lot of missing values. Walk me through how you decide what to do, and why dropping every row with a gap or just filling with the mean can quietly bias your results.

Sample answer outline

The first question is why the data is missing, because the mechanism drives the right response. If values are missing completely at random, dropping rows loses power but does not bias; if missingness depends on observed variables, you can model it; if it depends on the unobserved value itself, for example high earners declining to report income, any naive fill introduces bias and the missingness itself is informative. Dropping every incomplete row is dangerous when missingness is not random, because you systematically remove a subset and skew the sample. Mean imputation is convenient but shrinks variance, distorts correlations, and pretends to a certainty you do not have. Better options depending on the situation: model-based or multiple imputation that preserves uncertainty, and adding a missing-indicator feature so the model can use the fact that a value was absent, which often carries signal. The strong answer starts from the missingness mechanism rather than jumping to a default fill.

Expect these follow-ups

• How does the missingness mechanism change what you should do?
• Why does mean imputation distort the data even when it looks harmless?
• When is the fact that a value is missing itself a useful feature?

As asked

Your experimentation platform runs 50 tests a week. A colleague points out that at alpha=0.05 you expect about 2.5 false positives per week by chance. How do you address this at both the individual test level and the platform level?

Sample answer outline

At the test level: Bonferroni correction, Benjamini-Hochberg FDR control, or pre-specifying a primary metric. At the platform level: raising the significance threshold globally, requiring replication for high-stakes decisions, or using a Bayesian framework with informative priors. The candidate should distinguish between FWER and FDR control and note the power cost of each approach.

Expect these follow-ups

• What is the difference between controlling FWER and FDR, and when would you choose one over the other?

As asked

We cannot randomize users to a new pricing tier because it would be legally complex. We have 12 months of observational data. How would you estimate the causal effect of the higher tier on retention?

Sample answer outline

The candidate should outline at least two methods: propensity score matching or inverse probability weighting to control for selection bias, and instrumental variables or difference-in-differences if a natural quasi-experiment exists. They need to articulate the key identification assumptions for each method (e.g., unconfoundedness for PSM, common trends for DiD) and acknowledge that unobserved confounders remain the core threat.

Expect these follow-ups

• How would you test the common-trends assumption in a diff-in-diff design?
• What is the overlap condition in propensity score matching and how do you check it?

As asked

We operate a two-sided marketplace. When we run a standard user-level A/B test, treatment users and control users interact with the same supply pool. Why is this a problem and how do you design around it?

Sample answer outline

This is the SUTVA (Stable Unit Treatment Value Assumption) violation: the treatment affects control units through shared supply, inflating or deflating measured effects. Solutions include cluster-based randomization (geo or market level), switchback experiments (alternating time windows), or ego-network clustering. The candidate should acknowledge the tradeoff: larger clusters reduce interference but reduce statistical power significantly.

Expect these follow-ups

• How would you estimate the variance of a geo-level experiment where you only have 50 markets?

As asked

We launched a UI redesign and the experiment showed a 15% lift in engagement in week one but only 3% in week three. What is happening and how do you distinguish a novelty effect from a genuine sustained improvement?

Sample answer outline

The novelty effect is when users engage more simply because something is new, not because it is better. To distinguish it from a real effect, segment the analysis by user tenure in the experiment (did the lift decay over time?), compare new vs returning users, and look at long-run data from the holdout. The candidate should recommend running the experiment longer or using a pre-exposure period before measuring.

Expect these follow-ups

• How do you decide how long an experiment needs to run to avoid novelty effects?

As asked

Your team is debating switching from frequentist to Bayesian A/B testing. What are the practical differences in how you run and interpret an experiment under each framework, and where does each one break down?

Sample answer outline

Frequentist: fixed sample size, binary reject-or-not at end, well-understood error rates but no peeking. Bayesian: compute posterior over effect size, can express probability that variant is better, enables early stopping with explicit loss thresholds, but prior choice matters and can be gamed. Frequentist breaks down when teams peek repeatedly; Bayesian breaks down when priors are mis-specified or when communicating 'credible intervals' vs confidence intervals confuses stakeholders.

Expect these follow-ups

• How do you set a stopping criterion in a Bayesian test without inflating false positive rates?

As asked

An analysis shows that users who use Feature X have lower conversion than users who do not, but when you break it down by user segment, Feature X users convert better in every segment. How is this possible and what do you report?

Sample answer outline

This is Simpson's Paradox, caused by a confounding variable (user segment) that is correlated with both Feature X adoption and conversion rate. The segment-level analysis is more meaningful because it controls for the confounder. The candidate should explain that the aggregate result is misleading when groups have different sizes and different baseline rates, and recommend reporting the segment-controlled numbers with a causal explanation of the confounder.

Expect these follow-ups

• How would you use propensity score weighting to get a single aggregate estimate that corrects for this confounder?

As asked

You want to detect a 1% lift in revenue per user but would need 4 million users to achieve 80% power with the raw metric. A colleague suggests CUPED. Explain how CUPED works and what it buys you.

Sample answer outline

CUPED (Controlled-experiment Using Pre-Experiment Data) regresses out variance explained by pre-experiment covariates (such as prior week revenue). The adjusted metric is Y minus theta times X_pre, where theta minimizes the residual variance. This can reduce variance by 50% or more, halving the required sample size. The candidate should note that the covariate must be measured before experiment assignment to avoid contamination.

Expect these follow-ups

• What is the statistical condition under which CUPED yields a valid unbiased estimate?

As asked

You use difference-in-differences to measure the impact of a policy change rolled out to one city. What assumptions must hold for the DiD estimate to be valid, and how would you test the most critical one?

Sample answer outline

The core assumption is parallel trends: absent the treatment, the treated and control groups would have followed the same trajectory. To test it, plot pre-treatment trends for both groups and run a placebo test using prior periods as fake treatment windows. Secondary assumptions: no spillovers (SUTVA), stable composition of groups, and no other simultaneous interventions. The candidate should describe event-study plots as a standard robustness check.

Expect these follow-ups

• How would you handle a staggered rollout where different units receive treatment at different times?