Product manager statistics interview questions

As asked

Before launching an A/B test, a PM asks how long it needs to run. Walk me through how you size it: what inputs you need, what the minimum detectable effect means, and the traps that make people stop too early.

Sample answer outline

Sizing comes from four inputs: the baseline rate of the metric, the minimum detectable effect you care about, the significance level, and the desired power. The minimum detectable effect is the smallest true change worth detecting, and smaller effects need dramatically larger samples, which is the lever most people underestimate. From those you compute the sample per arm, then divide by daily eligible traffic to get duration, and round up to whole weeks so weekday and weekend behaviour are represented rather than over-weighting whichever days you happened to catch. The trap is peeking and stopping the moment significance appears, which inflates false positives because you gave yourself many chances to cross the line; commit to the planned horizon, or use a sequential method designed for valid early stopping. Also resist declaring a flat result a win for the control too soon if the test was underpowered to begin with. The signal is treating duration as the output of a power calculation, not a guess.

Expect these follow-ups

• Why do smaller detectable effects need so much more traffic?
• Why round the duration to whole weeks?
• What is wrong with stopping as soon as the result looks significant?

As asked

A data scientist tells you a test came back with p = 0.03 and a 1.5 percent lift. As the product manager deciding whether to ship, how do you interpret that, and what questions do you ask before you commit?

Sample answer outline

A PM does not need to derive the maths but must read results without being fooled. The p-value of 0.03 says the result is unlikely if the feature did nothing, so there is reasonable evidence of a real effect, but it says nothing about size or business value. The 1.5 percent lift is the part that matters for the ship decision, so ask about the confidence interval around it: a lift of 1.5 percent that could plausibly be near zero is a different decision from one tightly bounded above one percent. Ask whether the test ran its planned duration or was stopped early, whether guardrail metrics held, and whether the effect is worth the cost and complexity to ship and maintain. The product signal is connecting statistical evidence to a value judgement, not treating significance as an automatic green light.

Expect these follow-ups

• Why do you care more about the confidence interval than the p-value here?
• What guardrail metrics would you insist on before shipping?
• When would you not ship a statistically significant result?