Generative model engineer interview questions

As asked

A product team wants a generative assistant to answer support questions in the company's tone using private documentation. When would you fine-tune, use RAG, or rely on prompting?

Sample answer outline

Use RAG when the main requirement is fresh private knowledge and citations, because fine-tuning is a poor way to memorise changing facts. Use fine-tuning when you need a consistent behaviour, format, style, or task policy that prompting cannot reliably maintain at scale. Prompting alone is suitable when the task is simple, low-risk, and changes frequently. Strong answers mention evaluation before and after each approach, plus the operational costs of data curation, retraining, retrieval quality, and latency. The common error is treating fine-tuning as a magic upgrade rather than a targeted behavioural intervention.

Expect these follow-ups

• What failure tells you the retriever is the problem?
• What data would you need for a useful fine-tune?
• How would you measure whether tone improved without making factuality worse?

As asked

Explain the math behind DDPM. What is the forward process doing to the data, what distribution does it converge to, and how does the reverse process reconstruct samples? Where does the neural network actually sit in this picture?

Sample answer outline

The forward process is a fixed Markov chain that gradually adds Gaussian noise over T steps following a variance schedule, driving the data distribution toward a standard normal. The reverse process learns to denoise step by step, and the network is trained to predict the noise (or the score) added at each step. A strong answer mentions the reparameterization that lets you jump to any noise level in one shot, which is what the L_simple loss exploits.

Expect these follow-ups

• Why does predicting the noise work better than predicting the original image directly in practice?
• What happens if your variance schedule is too aggressive early in training?

As asked

How does classifier-free guidance work in diffusion models, and why does turning up the guidance scale improve prompt alignment but hurt sample diversity? What is actually happening to the score estimate?

Sample answer outline

CFG trains a single model for both conditional and unconditional generation by randomly dropping the conditioning signal. At inference, the score is extrapolated beyond the conditional direction: score = unconditional + w * (conditional - unconditional). Higher w sharpens the conditional direction, which reduces the effective entropy of the output distribution and collapses toward the mode, trading diversity for fidelity.

Expect these follow-ups

• How does PAG (perturbed attention guidance) try to get around the diversity problem?
• Why can high guidance scales sometimes cause oversaturation in pixel space?

As asked

Explain how LoRA fine-tuning works at the weight level. Why is it valid to represent the weight update as a product of two low-rank matrices, and how do rank and alpha hyperparameters interact in practice?

Sample answer outline

LoRA freezes the pretrained weights and adds a bypass W = W0 + BA where B is d x r and A is r x d, with r much smaller than d. The hypothesis is that the intrinsic dimensionality of the fine-tuning task is low, so a rank-r update captures most of the signal. Alpha scales the contribution: effective_lr scales as alpha/r, so increasing alpha without changing rank amplifies the LoRA contribution. A strong answer notes that merging BA into W0 at inference eliminates latency overhead.

Expect these follow-ups

• When would you choose QLoRA over standard LoRA, and what are the memory savings?
• Which weight matrices in a transformer do you apply LoRA to and why does that choice matter?

As asked

Describe the KV cache in transformer generation. How does memory consumption scale with sequence length and batch size, and what practical problems does this cause when you try to run long-context inference at scale?

Sample answer outline

During autoregressive decoding, keys and values for all previous tokens are cached to avoid recomputing them. Memory grows as O(layers x heads x seq_len x head_dim x batch_size), so a 4096-token sequence with a large batch can consume tens of gigabytes just for the cache. This forces tradeoffs between batch size and max sequence length, and motivates techniques like paged attention (vLLM), sliding window attention, and multi-query attention.

Expect these follow-ups

• How does paged attention in vLLM reduce fragmentation compared to a contiguous KV cache?
• What is GQA and how does it reduce the per-token cache footprint?

As asked

Explain how gradient checkpointing reduces memory during training of large transformer or UNet models. What is the exact compute cost you are paying, and when is it not worth using?

Sample answer outline

Checkpointing discards intermediate activations during the forward pass and recomputes them during the backward pass as needed. This reduces activation memory from O(layers) to O(sqrt(layers)) with optimal checkpoint placement, at the cost of roughly one additional forward pass worth of compute, so throughput drops around 20 to 30 percent. It is not worth using when the bottleneck is GPU utilization rather than memory, or when you already fit comfortably with a large batch and do not need to increase it further.

Expect these follow-ups

• How does selective gradient checkpointing, where you only checkpoint some layers, help you find a better tradeoff?
• Does gradient checkpointing interact with mixed precision in any tricky way?