Generative model engineer interview questions

As asked

A product team wants a generative assistant to answer support questions in the company's tone using private documentation. When would you fine-tune, use RAG, or rely on prompting?

Sample answer outline

Use RAG when the main requirement is fresh private knowledge and citations, because fine-tuning is a poor way to memorise changing facts. Use fine-tuning when you need a consistent behaviour, format, style, or task policy that prompting cannot reliably maintain at scale. Prompting alone is suitable when the task is simple, low-risk, and changes frequently. Strong answers mention evaluation before and after each approach, plus the operational costs of data curation, retraining, retrieval quality, and latency. The common error is treating fine-tuning as a magic upgrade rather than a targeted behavioural intervention.

Expect these follow-ups

• What failure tells you the retriever is the problem?
• What data would you need for a useful fine-tune?
• How would you measure whether tone improved without making factuality worse?

As asked

A scientist proposes a new attention variant that improves long-context performance. How would you design the ablation study?

Sample answer outline

Define the hypothesis and isolate the moving part, otherwise the experiment becomes a comparison of entire stacks. Keep training data, token budget, optimiser, batch size, context length, and evaluation scripts constant unless the ablation explicitly tests one of them. Use tasks that reflect the claimed win, such as retrieval over long context, multi-hop reasoning, and latency or memory measurements at increasing sequence lengths. Strong candidates discuss confidence intervals, repeated seeds for noisy runs, and negative results that still teach something. The common mistake is celebrating one benchmark win while hiding compute cost or regressions on shorter-context tasks.

Expect these follow-ups

• How many seeds are enough for this experiment?
• What if the method wins only at a much higher compute budget?
• How do you report a promising but inconclusive result?

As asked

How would you build a synthetic data pipeline for instruction tuning without filling the training set with low-quality or duplicated examples?

Sample answer outline

Start with a target capability map so synthetic examples are generated for known gaps rather than volume for its own sake. Use strong prompts, multiple generators if possible, and filters for duplication, toxicity, format validity, factuality, and answer diversity. Human review should focus on calibration and hard categories rather than checking every example manually. Strong candidates discuss contamination, train-test leakage, model collapse from self-generated data, and weighting synthetic data below high-quality human data. The common trip-up is optimising token count instead of marginal training value.

Expect these follow-ups

• How do you detect near-duplicate examples at scale?
• When would synthetic data hurt model quality?
• How do you decide the mix of human and synthetic examples?

As asked

A team wants to ship an image generation feature for ecommerce product mockups. How would you evaluate the model before launch?

Sample answer outline

Define product-specific success criteria: prompt adherence, visual quality, brand safety, text rendering, object consistency, diversity, and editability. Use automated metrics only as weak signals, because FID-style metrics do not capture whether a generated product mockup is usable. Build human review rubrics and compare against realistic baselines such as stock imagery, templates, or a previous model. Strong answers include safety filters for protected brands, misleading product claims, and generated text in images. The trap is evaluating only aesthetic quality and missing commercial usability.

Expect these follow-ups

• How would you test consistency across a set of related product images?
• What categories should be blocked before launch?
• How would you measure whether users prefer the generated mockups?

As asked

Explain how KV caching and batching affect throughput and latency in transformer inference.

Sample answer outline

KV cache stores prior key and value tensors so decoding each next token does not recompute attention over the full prefix. It improves decode efficiency but consumes memory proportional to batch size, sequence length, layers, and hidden dimensions. Batching improves GPU utilisation by running multiple requests together, but excessive batching adds queueing delay and can hurt p95 latency. Strong answers distinguish prefill, which is compute-heavy and parallel, from decode, which is often memory-bandwidth constrained. The common mistake is talking about tokens per second without considering time to first token and tail latency.

Expect these follow-ups

• Why can long prompts reduce the number of concurrent requests per GPU?
• What is continuous batching?
• How would prefix caching change product latency?

As asked

A multimodal generation feature is too slow for interactive use. How would you reduce latency without destroying output quality?

Sample answer outline

Break latency into queueing, pre-processing, model inference, post-processing, and network time before changing the model. Candidate optimisations include smaller distilled models, quantisation, caching repeated inputs, batching where it does not hurt interactivity, progressive previews, and asynchronous high-quality refinement. Strong answers evaluate quality with task-specific tests after every optimisation rather than assuming faster is acceptable. They also mention capacity planning because queueing delay can dominate model time under bursty demand. The common mistake is jumping straight to a smaller model without measuring the real bottleneck.

Expect these follow-ups

• When does batching improve throughput but hurt p95 latency?
• How would you expose progressive generation in the product?
• What quality metric would block a latency optimisation?