Backend engineer analytics interview questions

As asked

Airbnb wants to test a simplified checkout flow that removes one step. How would you design the experiment, choose your metrics, and decide when to ship?

Sample answer outline

Strong answers define a primary metric (booking completion rate), guardrail metrics (revenue per booking, cancellation rate, support contacts), and calculate sample size from expected effect size and desired power. They discuss randomization unit (user-level vs. session-level and why user-level avoids novelty effects), minimum detectable effect, and the risk of Simpsons Paradox if segments are unbalanced. They mention checking for network effects and know when to stop an experiment early.

Expect these follow-ups

• The test shows a statistically significant uplift in completions but a drop in revenue per booking. What do you recommend?
• How do you handle the fact that some users in the control group will eventually see the new flow if you ship it?

As asked

Lyft wants to test a redesigned driver onboarding flow that reduces the number of steps by 30 percent. Walk me through how you would design the A/B test: what metric is your primary success metric, what guardrail metrics would you monitor, how long you would run the experiment, and how you would decide when to ship.

Sample answer outline

Strong answers choose driver activation rate (completed first trip within 7 days) as the primary metric, add guardrail metrics like background check completion rate and compliance document upload rate, use a power calculation to set sample size and duration, and address novelty effects by running the test for at least two weeks. They should flag that driver supply has marketplace-level effects that make this harder to isolate than a typical consumer experiment.

Expect these follow-ups

• How would you handle it if the primary metric improves but a guardrail metric degrades?
• How do you account for the fact that drivers in the control and treatment groups may interact with each other through the same marketplace?

As asked

Netflix monitors video quality for every play session. What metrics would you define to capture the quality of a streaming session from the subscriber's point of view, how would you instrument them on the client, and how would you use them to detect a CDN degradation event before subscribers call support?

Sample answer outline

Key metrics include: rebuffering ratio (seconds buffering per hour of viewing), start-play time (time from press-play to first frame), average bitrate delivered versus requested, video error rate, and audio-video sync drift. On the client, these are sampled at fixed intervals and batch-uploaded with the session ID. For CDN detection, the candidate should describe cohort analysis by ISP and geographic region to identify correlated quality drops that isolate a specific CDN edge node rather than the content or device.

Expect these follow-ups

• A subscriber reports choppy video but your metrics for their session look normal. How do you investigate?
• How do you separate a subscriber's home network problem from a CDN problem using only the session telemetry?

As asked

DoorDash's analytics pipeline ingests order completion events from Dashers' mobile apps. Some events arrive 10 to 30 minutes late due to offline sync. Your daily revenue report runs at midnight. How do you design the pipeline to produce accurate daily revenue numbers without delaying the report indefinitely?

Sample answer outline

Strong answers use watermarks in a stream processing system (Apache Flink or Spark Structured Streaming) to define how long to wait for late data before closing a time window. They publish a preliminary midnight report, then a corrected report at 2 AM after absorbing most late arrivals, and keep an audit log of corrections. Alternatively, they store raw events with both event_time and process_time and query the data warehouse using event_time to produce retroactively correct numbers. DoorDash uses this pattern internally, publishing near-real-time metrics while allowing a correction window for offline-synced Dasher events. The candidate should distinguish between reporting latency and data accuracy tradeoffs and know that a correction delta approach (subtract the old number, add the corrected one) is safer than overwriting aggregates in place.

Expect these follow-ups

• How late can an event arrive before you decide to exclude it from the corrected report?
• How do you communicate to stakeholders that a published metric may change after correction?