Vibe coder or product engineer product sense interview questions

As asked

You want to A/B test a new checkout flow. Walk me through how you set up the experiment: how you decide sample size, how long to run it, what metric you use as the primary success metric, and what common mistakes invalidate results.

Sample answer outline

Set the primary metric before running the test, not after, to avoid p-hacking. Calculate the required sample size based on the baseline conversion rate, the minimum detectable effect, and your desired power (typically 80%) and significance level (typically 5%). With 1000 daily visits and a 5% lift threshold, that might require 2 to 4 weeks. Common mistakes: peeking at results before the predetermined end date and stopping early when significance is reached, segmenting results post-hoc for groups that were not pre-specified, running multiple simultaneous experiments that interact with each other, and novelty effects from users who notice the change and behave differently temporarily.

Expect these follow-ups

• How do you handle a situation where the A/B test shows no significant effect but you strongly believe the new flow is better?

As asked

Walk me through what makes a good north-star metric versus a vanity metric. Give an example of a bad north-star metric and explain why it is bad, then give a better alternative for the same product.

Sample answer outline

A good north-star metric correlates with long-term retention and revenue, is hard to game without actually delivering value, and is something the team can influence directly. Page views is a bad north-star for a document collaboration tool because you can inflate it by breaking navigation or showing popups. A better metric is weekly documents edited per user, which reflects genuine engagement with the core action the product is built around. A strong answer also distinguishes leading versus lagging indicators: the north star may be a leading indicator of revenue rather than revenue itself, because it responds faster to product changes.

Expect these follow-ups

• How do you choose between two metrics that both seem like good north stars but optimize for different behaviors?

As asked

You built a customer-facing prototype with AI assistance and users love it. How do you decide what must change before it becomes production software?

Sample answer outline

Start by separating validated product value from implementation quality. The candidate should assess security, auth, data correctness, observability, performance, accessibility, error recovery, testing, and maintainability. Strong answers include a rewrite-or-harden decision based on code shape, product urgency, and the cost of carrying prototype assumptions. They should preserve the parts that created user value while removing brittle shortcuts such as local-only state, hardcoded IDs, or hidden manual steps. The trap is either shipping the prototype unchanged or rebuilding it so slowly that the product learning decays.

Expect these follow-ups

• What prototype shortcut is acceptable for a pilot but not for production?
• How do you communicate the hardening work to a non-technical founder?
• When is a rewrite cheaper than incremental cleanup?