Product manager analytics interview questions

As asked

TikTok's daily active users dropped 8% overnight with no scheduled change. You have access to all the data. Walk me through exactly how you investigate this.

Sample answer outline

A strong answer structures the investigation in layers: first rule out a measurement error (logging bug, pipeline delay), then segment by platform (iOS vs Android drop differently?), region, user segment (new vs returning), and traffic source. Next, check whether the definition of DAU changed (did a deduplication fix land?). Then look at funnel steps: is the drop in opens, in continued usage, or in logins? Finally check external factors: app store outages, competitor launches, regional news events. Avoid jumping to a root cause before ruling out simpler explanations.

Expect these follow-ups

• You determine the drop is real and concentrated on Android in Indonesia. What do you do next?
• How do you communicate your findings and uncertainty to leadership before you have a confirmed root cause?

As asked

You ran an A/B test on a new feature in Google Search. The experiment group shows a 0.5% increase in click-through rate with a p-value of 0.03. Your team wants to ship it. What questions do you ask before saying yes?

Sample answer outline

A strong answer goes beyond p-value: Was the sample size pre-determined or did the team peek and stop early (inflating false positives)? What is the practical significance of 0.5%? Is there novelty effect (did engagement spike because the feature is new)? What is the confidence interval, not just the point estimate? Are there segment breakdowns showing the lift is uniform, or does it only help one demographic? Did any guardrail metrics (latency, ad revenue, user satisfaction) move negatively?

Expect these follow-ups

• What would you do if the lift was consistent across all segments but the p-value was 0.08?
• How does the multiple comparisons problem affect this if you tested 10 variants simultaneously?

As asked

You ship a new social-graph feature in a content app. How do you measure whether it worked?

Sample answer outline

Set the north star (does the feature increase the thing the product cares about - daily active, retention, content created). Set proximate metrics that should move first (adoption of the feature, follow graph growth, comments per session). Set guardrail metrics that must not regress (load times, reports of abuse, churn of cohorts who don't use the feature). Plan the A/B test: power, MDE, exposure unit, duration. Plan the read-out: who decides, against what threshold, on what timeline. The strong answer separates leading from lagging indicators and is explicit about confounders.

Expect these follow-ups

• What is a metric you would actively NOT use?
• How do you handle network effects in the experiment?
• What if adoption is high but retention is flat?

As asked

Your team does not have a clear north star metric and different people optimise for different things. How do you choose one, what makes a good north star versus a bad one, and what are the dangers of a single metric?

Sample answer outline

A good north star captures the value the product delivers to users in a way that, when it goes up, the business genuinely benefits, so it sits between a vanity number and pure revenue. It should be a leading indicator you can influence with the product, reflect real user value rather than mere activity, and be understandable enough to align a team. Bad north stars are things like raw signups or total page views, which can rise while the product gets worse, or revenue alone, which is lagging and tempts short-term extraction. The danger of any single metric is that teams optimise it literally and find perverse shortcuts, so it must travel with guardrails, things that must not regress such as retention, quality, and trust, and it should be decomposed into the input metrics teams actually move. The signal is choosing a metric that proxies durable value and pairing it with guardrails, rather than pretending one number is the whole truth.

Expect these follow-ups

• Why are signups or page views poor north star candidates?
• What guardrail metrics would you attach, and why?
• How do you stop a team from gaming the single metric?

As asked

You are running an A/B test on a new ranking feature for the TikTok feed. After two weeks, the treatment group shows a 3% lift in watch time but a 2% drop in next-day retention. How do you interpret this and what do you do?

Sample answer outline

A strong answer recognizes the tension between a short-term engagement metric (watch time) and a longer-term health metric (retention), and does not simply ship because watch time went up. The candidate should check for novelty effects (does the retention drop stabilize after week 3?), segment the results by user cohort (are new users being burned out?), and verify that the watch time increase is not from auto-play inflation or low-quality binge patterns. They should recommend holding the experiment longer and checking 7-day retention before deciding.

Expect these follow-ups

• What is the minimum detectable effect size you would plan for when designing this experiment?
• How do you detect if the treatment group and control group are interfering with each other in a social product?

As asked

DoorDash is testing a simplified one-tap reorder flow for returning customers. How would you design the A/B test, what metric would you use as the primary success metric, and how would you guard against novelty effects, interference between groups, and the test running too long?

Sample answer outline

A strong answer defines the randomization unit as consumer ID (not session, to prevent within-user contamination), names repeat order rate within 30 days as the primary metric with order conversion rate as a guardrail, and calculates sample size from a minimum detectable effect using a power calculation. They should mention the novelty bias (early adopters behave differently), propose a 2-week minimum runtime to capture weekly patterns, and explain why you should not peek at results before the pre-specified end date.

Expect these follow-ups

• Your test shows a statistically significant lift in conversion but a small drop in average order value. What do you do?
• How do you handle the fact that the reorder flow only applies to users who have ordered at least once?

As asked

Netflix has started enforcing account sharing limits. Walk me through how you would measure the impact of this change on subscriber retention, revenue, and engagement, and how you would separate the causal effect from seasonal or market trends.

Sample answer outline

A strong answer designs a difference-in-differences analysis using markets where the policy rolled out early as the treatment group and markets where it has not yet rolled out as the control. The candidate should name the key metrics (subscriber churn rate, new subscriber acquisition among former shared-account users, ARPU, engagement hours per account) and explain how to detect selection bias in the rollout (Netflix may have chosen low-churn markets first). They should also discuss short-term versus long-term revenue tradeoffs and subscriber lifetime value.

Expect these follow-ups

• How do you measure the impact on users who were sharing a password but never convert to paid accounts? Are they a lost customer or never a real customer?
• What would make you recommend pausing the rollout before completing it globally?

As asked

Lots of teams use DAU/MAU as a proxy for retention. What metric would you actively avoid using, and why?

Sample answer outline

DAU/MAU itself has known failure modes: it conflates frequency with depth, it's easy to game with notifications, and it's not cohort-aware - a flat DAU/MAU can hide aggressive churn covered by acquisition. Avoid: total session count (rewards superficial poking), time spent (rewards friction, not value), feature adoption in isolation (a single-feature spike doesn't mean health). Prefer: cohorted retention curves (day 1, day 7, day 30), L28 (days active in last 28), and an action-based engagement metric tied to the value the product delivers.

Expect these follow-ups

• When IS DAU/MAU the right metric?
• How do you measure retention for a product used once a quarter?
• What is L28 and why is it better than 28-day MAU?

As asked

Lyft is launching in a new mid-size city. Six months after launch, your team wants to know whether the launch is 'successful.' What metrics would you define as success criteria before launch, and how would you interpret them if they showed mixed results, for example strong rider growth but low driver retention?

Sample answer outline

Strong answers define a North Star metric (monthly active trips per capita), supporting metrics on both supply side (driver retention, driver earnings vs local wage) and demand side (rider repeat rate, wait time, NPS), and marketplace health metrics (fill rate, cancellation rate). For the mixed-results scenario they reason through whether low driver retention is a structural problem (market too small) or an addressable one (incentive design, app experience).

Expect these follow-ups

• At what point would you recommend Lyft pull out of a city rather than keep investing?
• How do you isolate Lyft's launch effect from macroeconomic trends in that city?

As asked

A major concert at a 70,000 seat stadium ends at 11pm on a Saturday. How would you forecast driver supply and demand for that 45-minute window after the event ends, and what actions would Uber take based on that forecast?

Sample answer outline

A strong answer covers the historical data inputs (previous similar events in that city, typical Saturday night driver supply curve, expected demand multiplier from comparable events), the forecasting approach (regression model or similar events analogy), confidence intervals, and the operational levers: targeted driver notifications to incentivize supply into the area before 11pm, early surge signal to attract drivers, and pre-positioned availability messaging in the driver app. The candidate should discuss how to measure forecast accuracy after the event.

Expect these follow-ups

• Your forecast said 2,000 drivers would be available but only 800 showed up. What went wrong and how do you improve the next forecast?
• How would you quantify the revenue impact of getting this forecast right versus wrong?