Data scientist analytics interview questions

As asked

We are building a B2C subscription app. Product wants a single north-star metric to guide the roadmap. How do you go about choosing it, and what guardrail metrics would you pair it with?

Sample answer outline

A strong answer ties the north-star to the value the product delivers to users, not just company revenue, for example weekly active paying subscribers or a task-completion rate. The candidate should explain how to stress-test the metric against gaming and lagging effects, and propose guardrail metrics that catch harm to other dimensions: revenue per user, NPS, and support ticket rate.

Expect these follow-ups

• How would you handle it if your north-star metric improved but revenue fell?
• What is a counter-metric and when should you make one mandatory in experiments?

As asked

TikTok's daily active users dropped 8% overnight with no scheduled change. You have access to all the data. Walk me through exactly how you investigate this.

Sample answer outline

A strong answer structures the investigation in layers: first rule out a measurement error (logging bug, pipeline delay), then segment by platform (iOS vs Android drop differently?), region, user segment (new vs returning), and traffic source. Next, check whether the definition of DAU changed (did a deduplication fix land?). Then look at funnel steps: is the drop in opens, in continued usage, or in logins? Finally check external factors: app store outages, competitor launches, regional news events. Avoid jumping to a root cause before ruling out simpler explanations.

Expect these follow-ups

• You determine the drop is real and concentrated on Android in Indonesia. What do you do next?
• How do you communicate your findings and uncertainty to leadership before you have a confirmed root cause?

As asked

You ran an A/B test on a new feature in Google Search. The experiment group shows a 0.5% increase in click-through rate with a p-value of 0.03. Your team wants to ship it. What questions do you ask before saying yes?

Sample answer outline

A strong answer goes beyond p-value: Was the sample size pre-determined or did the team peek and stop early (inflating false positives)? What is the practical significance of 0.5%? Is there novelty effect (did engagement spike because the feature is new)? What is the confidence interval, not just the point estimate? Are there segment breakdowns showing the lift is uniform, or does it only help one demographic? Did any guardrail metrics (latency, ad revenue, user satisfaction) move negatively?

Expect these follow-ups

• What would you do if the lift was consistent across all segments but the p-value was 0.08?
• How does the multiple comparisons problem affect this if you tested 10 variants simultaneously?

As asked

You ship a new social-graph feature in a content app. How do you measure whether it worked?

Sample answer outline

Set the north star (does the feature increase the thing the product cares about - daily active, retention, content created). Set proximate metrics that should move first (adoption of the feature, follow graph growth, comments per session). Set guardrail metrics that must not regress (load times, reports of abuse, churn of cohorts who don't use the feature). Plan the A/B test: power, MDE, exposure unit, duration. Plan the read-out: who decides, against what threshold, on what timeline. The strong answer separates leading from lagging indicators and is explicit about confounders.

Expect these follow-ups

• What is a metric you would actively NOT use?
• How do you handle network effects in the experiment?
• What if adoption is high but retention is flat?

As asked

Your team does not have a clear north star metric and different people optimise for different things. How do you choose one, what makes a good north star versus a bad one, and what are the dangers of a single metric?

Sample answer outline

A good north star captures the value the product delivers to users in a way that, when it goes up, the business genuinely benefits, so it sits between a vanity number and pure revenue. It should be a leading indicator you can influence with the product, reflect real user value rather than mere activity, and be understandable enough to align a team. Bad north stars are things like raw signups or total page views, which can rise while the product gets worse, or revenue alone, which is lagging and tempts short-term extraction. The danger of any single metric is that teams optimise it literally and find perverse shortcuts, so it must travel with guardrails, things that must not regress such as retention, quality, and trust, and it should be decomposed into the input metrics teams actually move. The signal is choosing a metric that proxies durable value and pairing it with guardrails, rather than pretending one number is the whole truth.

Expect these follow-ups

• Why are signups or page views poor north star candidates?
• What guardrail metrics would you attach, and why?
• How do you stop a team from gaming the single metric?

As asked

Your usability tests suggest users hate a new onboarding step, but analytics shows users who complete it retain better. How do you make a recommendation?

Sample answer outline

Treat the evidence as answering different questions: usability tests reveal friction and comprehension, while analytics may be affected by selection bias because only motivated users complete the step. Segment the data, check whether the step causes retention or merely filters for intent, and inspect where users drop. The recommendation might be to preserve the valuable commitment or education while redesigning the friction. A senior answer does not pick the evidence source they prefer; it names the uncertainty and proposes the next decision-quality test.

Expect these follow-ups

• How would you test whether the onboarding step causes retention?
• What would you tell a PM who wants to remove it immediately?
• How do you present uncertainty without sounding indecisive?

As asked

Airbnb wants to test a simplified checkout flow that removes one step. How would you design the experiment, choose your metrics, and decide when to ship?

Sample answer outline

Strong answers define a primary metric (booking completion rate), guardrail metrics (revenue per booking, cancellation rate, support contacts), and calculate sample size from expected effect size and desired power. They discuss randomization unit (user-level vs. session-level and why user-level avoids novelty effects), minimum detectable effect, and the risk of Simpsons Paradox if segments are unbalanced. They mention checking for network effects and know when to stop an experiment early.

Expect these follow-ups

• The test shows a statistically significant uplift in completions but a drop in revenue per booking. What do you recommend?
• How do you handle the fact that some users in the control group will eventually see the new flow if you ship it?

As asked

You are running an A/B test on a new ranking feature for the TikTok feed. After two weeks, the treatment group shows a 3% lift in watch time but a 2% drop in next-day retention. How do you interpret this and what do you do?

Sample answer outline

A strong answer recognizes the tension between a short-term engagement metric (watch time) and a longer-term health metric (retention), and does not simply ship because watch time went up. The candidate should check for novelty effects (does the retention drop stabilize after week 3?), segment the results by user cohort (are new users being burned out?), and verify that the watch time increase is not from auto-play inflation or low-quality binge patterns. They should recommend holding the experiment longer and checking 7-day retention before deciding.

Expect these follow-ups

• What is the minimum detectable effect size you would plan for when designing this experiment?
• How do you detect if the treatment group and control group are interfering with each other in a social product?

As asked

DoorDash is testing a simplified one-tap reorder flow for returning customers. How would you design the A/B test, what metric would you use as the primary success metric, and how would you guard against novelty effects, interference between groups, and the test running too long?

Sample answer outline

A strong answer defines the randomization unit as consumer ID (not session, to prevent within-user contamination), names repeat order rate within 30 days as the primary metric with order conversion rate as a guardrail, and calculates sample size from a minimum detectable effect using a power calculation. They should mention the novelty bias (early adopters behave differently), propose a 2-week minimum runtime to capture weekly patterns, and explain why you should not peek at results before the pre-specified end date.

Expect these follow-ups

• Your test shows a statistically significant lift in conversion but a small drop in average order value. What do you do?
• How do you handle the fact that the reorder flow only applies to users who have ordered at least once?

As asked

Lyft wants to test a redesigned driver onboarding flow that reduces the number of steps by 30 percent. Walk me through how you would design the A/B test: what metric is your primary success metric, what guardrail metrics would you monitor, how long you would run the experiment, and how you would decide when to ship.

Sample answer outline

Strong answers choose driver activation rate (completed first trip within 7 days) as the primary metric, add guardrail metrics like background check completion rate and compliance document upload rate, use a power calculation to set sample size and duration, and address novelty effects by running the test for at least two weeks. They should flag that driver supply has marketplace-level effects that make this harder to isolate than a typical consumer experiment.

Expect these follow-ups

• How would you handle it if the primary metric improves but a guardrail metric degrades?
• How do you account for the fact that drivers in the control and treatment groups may interact with each other through the same marketplace?

As asked

Netflix has started enforcing account sharing limits. Walk me through how you would measure the impact of this change on subscriber retention, revenue, and engagement, and how you would separate the causal effect from seasonal or market trends.

Sample answer outline

A strong answer designs a difference-in-differences analysis using markets where the policy rolled out early as the treatment group and markets where it has not yet rolled out as the control. The candidate should name the key metrics (subscriber churn rate, new subscriber acquisition among former shared-account users, ARPU, engagement hours per account) and explain how to detect selection bias in the rollout (Netflix may have chosen low-churn markets first). They should also discuss short-term versus long-term revenue tradeoffs and subscriber lifetime value.

Expect these follow-ups

• How do you measure the impact on users who were sharing a password but never convert to paid accounts? Are they a lost customer or never a real customer?
• What would make you recommend pausing the rollout before completing it globally?

As asked

Netflix monitors video quality for every play session. What metrics would you define to capture the quality of a streaming session from the subscriber's point of view, how would you instrument them on the client, and how would you use them to detect a CDN degradation event before subscribers call support?

Sample answer outline

Key metrics include: rebuffering ratio (seconds buffering per hour of viewing), start-play time (time from press-play to first frame), average bitrate delivered versus requested, video error rate, and audio-video sync drift. On the client, these are sampled at fixed intervals and batch-uploaded with the session ID. For CDN detection, the candidate should describe cohort analysis by ISP and geographic region to identify correlated quality drops that isolate a specific CDN edge node rather than the content or device.

Expect these follow-ups

• A subscriber reports choppy video but your metrics for their session look normal. How do you investigate?
• How do you separate a subscriber's home network problem from a CDN problem using only the session telemetry?

As asked

You want to A/B test a new lane-change assist behavior that you plan to roll out via OTA. How do you design the experiment, assign vehicles to treatment versus control, define your metrics, and decide when you have enough data to call the test?

Sample answer outline

A strong answer covers: randomizing at the vehicle level (not trip level) to avoid within-vehicle contamination, stratifying by vehicle model and driving environment (urban versus highway) since the feature behaves differently in each, choosing primary metrics (successful lane-change rate, disengagement rate) and guardrail metrics (safety incident rate, user disable rate), computing required sample size via power analysis before starting, and using a sequential testing approach to allow early stopping without inflating the false-positive rate. The candidate should discuss why driver behavior change (novelty effect) can confound early results.

Expect these follow-ups

• How do you prevent drivers from opting out of the experiment in ways that bias your results?
• The test shows a statistically significant improvement in lane-change success but also a small (non-significant) increase in near-miss events. What do you do?

As asked

Uber's growth team wants to test whether offering new riders a 20% discount on their first three trips increases 90-day retention. Walk me through how you would design this experiment, including randomization, metrics, and statistical analysis.

Sample answer outline

A strong answer covers the randomization unit (rider ID, not trip, to avoid the same rider getting mixed treatments), holdout sizing using power analysis at the expected effect size, the primary metric (90-day retention or trips in 90 days), guardrail metrics (margin, fraud rate), duration to run before reading results, and the statistical test (two-proportion z-test for retention, t-test for trips). The candidate should flag the novelty effect risk and the need to wait for users to complete their trial trips.

Expect these follow-ups

• You see a significant lift in trips taken during the promotion but no lift in 90-day retention. How do you explain this and what does it mean for the decision?
• How do you handle spillover if a treated rider refers a control-group rider and they both share a trip together?

As asked

Lots of teams use DAU/MAU as a proxy for retention. What metric would you actively avoid using, and why?

Sample answer outline

DAU/MAU itself has known failure modes: it conflates frequency with depth, it's easy to game with notifications, and it's not cohort-aware - a flat DAU/MAU can hide aggressive churn covered by acquisition. Avoid: total session count (rewards superficial poking), time spent (rewards friction, not value), feature adoption in isolation (a single-feature spike doesn't mean health). Prefer: cohorted retention curves (day 1, day 7, day 30), L28 (days active in last 28), and an action-based engagement metric tied to the value the product delivers.

Expect these follow-ups

• When IS DAU/MAU the right metric?
• How do you measure retention for a product used once a quarter?
• What is L28 and why is it better than 28-day MAU?

As asked

DoorDash's analytics pipeline ingests order completion events from Dashers' mobile apps. Some events arrive 10 to 30 minutes late due to offline sync. Your daily revenue report runs at midnight. How do you design the pipeline to produce accurate daily revenue numbers without delaying the report indefinitely?

Sample answer outline

Strong answers use watermarks in a stream processing system (Apache Flink or Spark Structured Streaming) to define how long to wait for late data before closing a time window. They publish a preliminary midnight report, then a corrected report at 2 AM after absorbing most late arrivals, and keep an audit log of corrections. Alternatively, they store raw events with both event_time and process_time and query the data warehouse using event_time to produce retroactively correct numbers. DoorDash uses this pattern internally, publishing near-real-time metrics while allowing a correction window for offline-synced Dasher events. The candidate should distinguish between reporting latency and data accuracy tradeoffs and know that a correction delta approach (subtract the old number, add the corrected one) is safer than overwriting aggregates in place.

Expect these follow-ups

• How late can an event arrive before you decide to exclude it from the corrected report?
• How do you communicate to stakeholders that a published metric may change after correction?

As asked

Lyft is launching in a new mid-size city. Six months after launch, your team wants to know whether the launch is 'successful.' What metrics would you define as success criteria before launch, and how would you interpret them if they showed mixed results, for example strong rider growth but low driver retention?

Sample answer outline

Strong answers define a North Star metric (monthly active trips per capita), supporting metrics on both supply side (driver retention, driver earnings vs local wage) and demand side (rider repeat rate, wait time, NPS), and marketplace health metrics (fill rate, cancellation rate). For the mixed-results scenario they reason through whether low driver retention is a structural problem (market too small) or an addressable one (incentive design, app experience).

Expect these follow-ups

• At what point would you recommend Lyft pull out of a city rather than keep investing?
• How do you isolate Lyft's launch effect from macroeconomic trends in that city?

As asked

A major concert at a 70,000 seat stadium ends at 11pm on a Saturday. How would you forecast driver supply and demand for that 45-minute window after the event ends, and what actions would Uber take based on that forecast?

Sample answer outline

A strong answer covers the historical data inputs (previous similar events in that city, typical Saturday night driver supply curve, expected demand multiplier from comparable events), the forecasting approach (regression model or similar events analogy), confidence intervals, and the operational levers: targeted driver notifications to incentivize supply into the area before 11pm, early surge signal to attract drivers, and pre-positioned availability messaging in the driver app. The candidate should discuss how to measure forecast accuracy after the event.

Expect these follow-ups

• Your forecast said 2,000 drivers would be available but only 800 showed up. What went wrong and how do you improve the next forecast?
• How would you quantify the revenue impact of getting this forecast right versus wrong?