AI research engineer at Cohere

As asked

A researcher says yesterday's experiment improved benchmark accuracy by 3 percent, but nobody can reproduce it. What do you check first?

Sample answer outline

Start with the experiment record: exact code commit, model config, data snapshot, preprocessing version, random seeds, hardware, and library versions. Then check whether the reported metric came from the intended split and whether the evaluation script changed between runs. Strong answers discuss determinism limits on GPUs while still insisting on reproducible artefacts and comparable results. The candidate should propose a minimal rerun and then a full rerun under tracked configuration. A common failure is focusing only on the seed while ignoring data leakage or an untracked preprocessing change.

Expect these follow-ups

• Which parts of a GPU training run may remain non-deterministic?
• How would you store data snapshots without copying terabytes every time?
• What would make you distrust the original result?

As asked

A fraud model performs well offline but starts making poor decisions in production. How would you investigate training-serving skew?

Sample answer outline

Compare feature values used during offline training with the values produced by the online serving path for the same entity and timestamp. Look for point-in-time leakage, stale online features, different default values, mismatched transformations, and schema evolution that did not reach both paths. Strong answers include replaying production events through the offline pipeline and logging model inputs at serving time with privacy controls. The candidate should separate skew from drift, because the first is a pipeline consistency bug and the second is a world-change problem. A common miss is checking only model metrics and not the raw feature distributions.

Expect these follow-ups

• What feature would you inspect first in a velocity-based fraud model?
• How do you log online features without creating a privacy problem?
• How would you prevent this class of issue in CI?

As asked

Design a shared GPU cluster for research training jobs, batch fine-tunes, and low-latency inference. How do you schedule work fairly and efficiently?

Sample answer outline

Separate workload classes because inference and long training jobs have different latency and preemption tolerance. Use quotas, priority classes, gang scheduling for distributed training, and preemptible queues for exploratory work. Strong answers discuss GPU fragmentation, topology awareness, checkpointing, fair-share policies, and visibility into queue time by team. Inference should usually live in reserved pools with autoscaling and admission control rather than competing directly with training. The common failure is maximising utilisation while making urgent production workloads wait behind low-priority experiments.

Expect these follow-ups

• How do you handle a training job that needs eight GPUs on the same node group?
• What metrics tell you the scheduler policy is unfair?
• When would you split research and production into separate clusters?

As asked

Standard self-attention has O(n^2) time and memory complexity in sequence length. Walk me through exactly where that quadratic term comes from, and describe at least two concrete algorithmic approaches that reduce it.

Sample answer outline

The quadratic cost comes from computing all n^2 query-key dot products to form the attention matrix, then multiplying by values. FlashAttention avoids materializing the full n x n matrix by tiling the computation in SRAM and fusing the softmax, achieving O(n) memory while keeping exact attention. Sparse attention methods (Longformer sliding window, BigBird random plus global patterns) reduce compute by restricting which token pairs attend to each other, trading exact for approximate attention.

Expect these follow-ups

• How does FlashAttention 2 differ from the original FlashAttention in terms of the tiling strategy?
• What is the causal masking change required in FlashAttention for decoder-only models?

As asked

Explain what the key-value cache is during autoregressive decoding. Given a model with 32 layers, 32 attention heads, head dimension 128, and sequence length 4096, how much GPU memory does the KV cache consume per request in float16?

Sample answer outline

The KV cache stores the key and value projections for all past tokens so they do not need to be recomputed at each new generation step. Memory is 2 (K and V) times layers times heads times head_dim times sequence_length times bytes_per_element: 2 * 32 * 32 * 128 * 4096 * 2 bytes = 2 GB per request in float16. At scale this becomes the primary bottleneck, motivating techniques like grouped-query attention (GQA) which reduces the number of KV heads, and PagedAttention which manages the cache in non-contiguous blocks.

Expect these follow-ups

• How does grouped-query attention differ from multi-query attention in terms of the KV cache size reduction?
• What does PagedAttention in vLLM do differently from a naive KV cache allocation?

As asked

You are training a large transformer and run out of GPU memory. A colleague suggests gradient checkpointing. Explain what it does, what the exact tradeoff is, and where you would place checkpoints in the network to get the best tradeoff.

Sample answer outline

Gradient checkpointing discards intermediate activations during the forward pass instead of storing them, then recomputes them during backpropagation when they are needed for gradients. The cost is one extra forward pass per checkpointed segment, so runtime increases by roughly 30-35% while memory drops significantly. The optimal placement recomputes about every sqrt(n) layers, which gives O(sqrt(n)) memory instead of O(n), balancing memory savings against recompute time. In practice checkpointing is placed at transformer block boundaries because each block's input is a clean recompute boundary.

Expect these follow-ups

• How does selective activation checkpointing differ from checkpointing every block?
• What happens to the memory profile if you checkpoint too aggressively, for example every single layer?