As asked
Airbnb wants to test a simplified checkout flow that removes one step. How would you design the experiment, choose your metrics, and decide when to ship?
Sample answer outline
Strong answers define a primary metric (booking completion rate), guardrail metrics (revenue per booking, cancellation rate, support contacts), and calculate sample size from expected effect size and desired power. They discuss randomization unit (user-level vs. session-level and why user-level avoids novelty effects), minimum detectable effect, and the risk of Simpsons Paradox if segments are unbalanced. They mention checking for network effects and know when to stop an experiment early.
Expect these follow-ups
- The test shows a statistically significant uplift in completions but a drop in revenue per booking. What do you recommend?
- How do you handle the fact that some users in the control group will eventually see the new flow if you ship it?
company:airbnbanalyticsab-testingstatisticsproduct-senseexperimentation