AI engineer domain knowledge interview questions

As asked

Explain how scaled dot-product attention computes its output. Why is the scaling factor of the square root of the key dimension necessary, and what would happen without it?

Sample answer outline

The candidate explains Q*K^T / sqrt(d_k) produces attention logits, softmax normalizes to attention weights, and those weights multiply V to produce the output. Without scaling, the variance of the dot products grows proportionally to d_k (the key dimension), which pushes logits into regions where softmax saturates and gradients vanish, making training unstable. Dividing by sqrt(d_k) returns the variance to roughly 1. They optionally mention multi-head attention and how it allows attending to different representation subspaces in parallel.

Expect these follow-ups

• What is the computational complexity of self-attention with respect to sequence length?
• How does flash attention reduce memory without changing the result?

As asked

Models with 128k and 1M token context windows exist. Under what conditions would you still choose a RAG pipeline over just stuffing everything into the context?

Sample answer outline

The candidate mentions cost (long contexts are expensive per query), latency (prefill time grows with context), lost-in-the-middle degradation for very long contexts, real-time or frequently updated corpora where you cannot pre-load everything, corpora larger than any context window (millions of documents), and access control (you only retrieve what the user is allowed to see). RAG wins on scalability and access control; long context wins on simplicity and recall when the corpus fits.

Expect these follow-ups

• What changes in your RAG design if your context window goes from 4k to 128k?
• When would you combine RAG with a long context window?

As asked

Explain what temperature, top-p sampling, and top-k sampling do to the model's output distribution. When do you use each and what happens if temperature is set to zero?

Sample answer outline

Temperature scales the logits before softmax: high temperature flattens the distribution toward uniform (more random), low temperature sharpens it (more greedy). Top-k restricts sampling to the k highest-probability tokens before re-normalizing. Top-p (nucleus sampling) restricts to the smallest set of tokens whose cumulative probability exceeds p. At temperature 0 the model is greedy and deterministic. The candidate recommends low temperature for factual, structured tasks and higher temperature for creative tasks.

Expect these follow-ups

• What is the difference between greedy decoding and beam search?
• Why might temperature=0 still produce different outputs on different runs?

As asked

What is chain-of-thought prompting, why does it improve performance on complex tasks, and when does it fail to help?

Sample answer outline

Chain-of-thought asks the model to generate intermediate reasoning steps before the final answer. It helps because the model's next-token prediction benefits from attending to its own prior reasoning, decomposing multi-step problems into simpler ones. It fails for tasks that do not benefit from decomposition (simple lookup, classification with short answers), can introduce incorrect intermediate steps that compound into wrong final answers, and adds latency and cost proportional to the reasoning length.

Expect these follow-ups

• What is zero-shot chain-of-thought versus few-shot chain-of-thought?
• When would you use a scratchpad approach instead of visible chain-of-thought?

As asked

Explain BPE tokenization. Why does the same English word sometimes get tokenized into different numbers of tokens, and why does this matter for LLM applications?

Sample answer outline

BPE starts with a character-level vocabulary and iteratively merges the most frequent byte or character pair into a new token, building a vocabulary of subword units. The same word tokenizes differently depending on its position (leading space token), capitalization, and how common it is in the training corpus. This matters for applications because token count drives cost and context limits, and rare words or names fragment into many tokens making generation less reliable.

Expect these follow-ups

• Why do programming languages sometimes tokenize poorly relative to English text?
• How does tokenization affect math reasoning in LLMs?

As asked

Explain the LoRA fine-tuning method. What is being trained, what stays frozen, and why does this drastically reduce the number of trainable parameters?

Sample answer outline

LoRA freezes the original weight matrix W and adds two low-rank matrices: A of shape (d_in, r) and B of shape (r, d_out). The adapted forward pass computes h = x @ W + x @ A @ B * (alpha / r), where alpha is a scaling hyperparameter. Since r is much smaller than d_in or d_out (for example r=16 vs d=4096), the number of trainable parameters drops from d_in * d_out to (d_in + d_out) * r. At inference time the adapter can be merged into the base weights for zero latency overhead. The candidate mentions rank r, the alpha scaling factor, and which weight matrices to target (typically the attention projection matrices) as key hyperparameters.

Expect these follow-ups

• What is QLoRA and what problem does it solve?
• How do you choose the rank r for a fine-tuning task?

As asked

What are the key properties of a well-trained embedding space, and how do violations of those properties cause problems in a production RAG system?

Sample answer outline

A strong answer covers: semantic similarity should correlate with geometric proximity (violation causes irrelevant retrieval), the space should generalize to out-of-distribution queries (violation causes retrieval gaps for new topic areas), embeddings should be stable across model versions (violation breaks A/B tests and requires full reindexing), and the embedding should capture the relevant task similarity (legal similarity differs from news similarity). The candidate connects each property to a concrete retrieval failure.

Expect these follow-ups

• How do you detect embedding space quality degradation in production?
• What is anisotropy in embedding spaces and why does it matter?

As asked

Your vector DB offers both cosine similarity and dot product as distance metrics. For a retrieval use case, when does the choice matter and which do you pick?

Sample answer outline

The candidate explains that for L2-normalized embeddings the two are equivalent, so the choice only matters when vectors are not normalized. Dot product allows vector magnitude to influence the score, which is useful when magnitude encodes relevance strength (e.g., term frequency). For semantic search with normalized embeddings, cosine similarity is conventional and removes magnitude bias. Most embedding models output approximately normalized vectors, so the two are nearly identical in practice.

Expect these follow-ups

• When would you explicitly want magnitude to affect the similarity score?
• What is the asymmetric inner product used in ColBERT?

As asked

Name and explain the four core RAGAS metrics. For each one, tell me what it catches that the others do not.

Sample answer outline

Faithfulness measures whether the answer is grounded in the retrieved context (catches hallucination). Answer Relevance measures whether the answer is responsive to the question (catches off-topic or evasive answers). Context Precision measures whether the retrieved chunks are relevant to the question (catches over-retrieval). Context Recall measures whether the relevant information exists in the retrieved chunks (catches under-retrieval). A strong candidate can give a concrete example of a failure that only one metric would detect.

Expect these follow-ups

• How do you use these metrics to diagnose whether the problem is in retrieval or generation?
• What are the limitations of using an LLM to compute these metrics?

As asked

Explain what the key-value cache in a transformer LLM is, why it exists, and what happens to memory and throughput when you scale batch size or sequence length.

Sample answer outline

The KV cache stores the key and value activations for all past tokens so the model does not recompute them for each new token during autoregressive generation. Memory grows linearly with sequence length and batch size: KV cache size = 2 * num_layers * num_heads * head_dim * seq_len * batch_size * dtype_bytes. This is often the binding memory constraint in serving, which is why systems like vLLM use paged KV cache (PagedAttention) to reduce fragmentation.

Expect these follow-ups

• What is PagedAttention and how does it help throughput?
• How does prefix caching in the KV cache relate to prompt caching at the API level?