Data analyst analytics interview questions

As asked

An experiment shows the variant beat control on clicks but the result is not statistically significant. The team wants to ship it. What do you say?

Sample answer outline

Separate the observed lift from the evidence for it: a positive point estimate that is not significant means the data is consistent with no real effect. Check sample size and whether the test reached its planned power, look at the guardrail metrics, and ask whether the team peeked early. Recommend either running longer to reach the planned sample or shipping with eyes open if the downside is low, but be clear about the risk.

Expect these follow-ups

• How would you explain not significant without saying it failed?
• What guardrail metrics would you insist on checking first?
• How does peeking inflate the false positive rate?

As asked

Walk me through how you would write SQL to produce a monthly signup cohort retention table, showing what share of each cohort is still active in later months.

Sample answer outline

Define the cohort as the month of first activity per user, then join activity events back and compute the month offset between activity and cohort start. Pivot or group by cohort month and offset, dividing active users by the cohort size for the retention percentage. Discuss the choice of active definition, handling users with gaps, and why absolute counts plus percentages both belong in the output.

Expect these follow-ups

• How do you define active for this product?
• How would you show this so a non-technical stakeholder reads it instantly?
• What confounds a cohort comparison across very different acquisition channels?

As asked

Your average order value jumps because of a handful of very large orders. How do you decide whether to keep, cap, or exclude them?

Sample answer outline

First investigate the cause: a data entry error, a genuine bulk buyer, or a test transaction behave very differently. Use the median and percentiles alongside the mean so one whale does not distort the story. Decide based on the question being asked, since a fraud review wants the outliers but a typical customer view may winsorize or report them separately. Never delete quietly; document the rule and show the figure both ways.

Expect these follow-ups

• When does the mean mislead and the median tell the truth?
• How would you flag suspected test transactions systematically?
• How do you communicate the impact of excluding outliers honestly?

As asked

A test shows no overall effect, but mobile users responded strongly while desktop users got slightly worse. How do you interpret and act on that?

Sample answer outline

Treat the segment finding as a hypothesis, not a conclusion, because slicing after the fact invites false positives through multiple comparisons. Check whether mobile was a pre-registered segment, whether the sample in each slice is large enough, and watch for Simpson's paradox in the pooled number. Recommend a follow up test targeted at mobile rather than shipping the segment cut as if it were proven.

Expect these follow-ups

• Why is post hoc segmentation risky?
• How would Simpson's paradox show up in this data?
• How would you design a confirmatory test for the mobile finding?

As asked

Sales says revenue was 4.2 million, finance says 3.9 million, and both pulled from your data. How do you resolve it?

Sample answer outline

Trace each number to its definition: timing of recognition, refunds and discounts, currency conversion, and which transactions are in scope usually explain the gap. Reconcile line by line back to a shared source of truth, then agree a single canonical definition and document it. The deliverable is not just the right number but a metric definition everyone signs up to so the disagreement does not recur.

Expect these follow-ups

• Where do revenue definitions most often diverge?
• How do you get two teams to agree on one definition?
• How do you prevent this once you have reconciled it?