ML engineer product sense interview questions

As asked

The Instagram feed ranking team wants to improve time-spent as a metric. You are running an experiment with a new ranking signal. How do you design the experiment, choose metrics, and decide whether to ship?

Sample answer outline

A strong answer frames the guardrail metrics (likes, comments, shares, follows) alongside the primary metric (time-spent), explains why time-spent alone can be gamed by addictive bad content, and designs an A/B test with holdout sizing based on minimum detectable effect. The candidate should mention how to handle network effects in feed experiments (users on the same social graph can contaminate control and treatment), and what Novelty effect means for ramp duration.

Expect these follow-ups

• Your experiment shows time-spent up 3% but shares down 5%. Do you ship?
• How do you measure long-term user satisfaction when your experiment only runs for two weeks?

As asked

ChatGPT has a memory feature that lets it remember facts about you across conversations. If you were the engineer responsible for this feature, what would you measure to know if it is working well, and what would you change to make it better?

Sample answer outline

A strong answer starts with metrics: recall accuracy when relevant memories exist, precision when memories are injected, user satisfaction signals (do users feel understood vs surveilled), and memory staleness. Then improvements: surfacing what the model has remembered and letting users edit it, smarter deduplication, handling contradictory memories, privacy controls and geographic compliance, and selective injection (not always injecting all memories into every prompt).

Expect these follow-ups

• How would you handle the case where a stored memory becomes outdated or wrong?
• How do you balance personalization with the cost of injecting memory into every prompt?

As asked

Your squad wants to ship a new personalization feature that changes how the Home tab orders content for users. Walk me through how you would design an A/B test, what metric you would use as the primary success metric, what guardrail metrics you would watch, and how long you would run the experiment before making a call.

Sample answer outline

A strong answer starts by picking a north-star metric tied to Spotify's business, such as weekly active listening minutes or track completion rate, and explains why that is better than CTR alone. It describes how to randomly split users, what sample size is needed to detect a meaningful lift (power calculation), and what guardrail metrics (like subscription cancellations or support tickets) would block a rollout even if the primary metric is green. It addresses novelty effects and how long to run to avoid them.

Expect these follow-ups

• How do you handle the fact that users in the control and treatment groups talk to each other and influence listening behavior?
• What would you do if the experiment shows a positive lift for premium users but a negative one for free users?

As asked

A major concert at SoFi Stadium in Los Angeles is happening Saturday night, and Lyft expects 50,000 ride requests in a 90-minute window after the event ends. How would you forecast how many drivers you need in the area, and what mechanisms would you use to get them there?

Sample answer outline

Strong answers describe using historical data from similar events and comparable venues, a time-series demand model tuned on event attendance and typical post-event request patterns, and supply mechanisms like targeted push notifications to nearby drivers, a driver bonus for being in the area, and preemptive surge pricing to incentivize supply. They should note that supply incentives need to be balanced to avoid over-paying and bleeding margin.

Expect these follow-ups

• How would you know in real time whether your supply forecast was accurate enough to adjust the incentive during the event?
• What would you do if your model predicted 30 percent more demand than actually arrived?