AI research engineer interview questions

As asked

A researcher says yesterday's experiment improved benchmark accuracy by 3 percent, but nobody can reproduce it. What do you check first?

Sample answer outline

Start with the experiment record: exact code commit, model config, data snapshot, preprocessing version, random seeds, hardware, and library versions. Then check whether the reported metric came from the intended split and whether the evaluation script changed between runs. Strong answers discuss determinism limits on GPUs while still insisting on reproducible artefacts and comparable results. The candidate should propose a minimal rerun and then a full rerun under tracked configuration. A common failure is focusing only on the seed while ignoring data leakage or an untracked preprocessing change.

Expect these follow-ups

• Which parts of a GPU training run may remain non-deterministic?
• How would you store data snapshots without copying terabytes every time?
• What would make you distrust the original result?

As asked

A fraud model performs well offline but starts making poor decisions in production. How would you investigate training-serving skew?

Sample answer outline

Compare feature values used during offline training with the values produced by the online serving path for the same entity and timestamp. Look for point-in-time leakage, stale online features, different default values, mismatched transformations, and schema evolution that did not reach both paths. Strong answers include replaying production events through the offline pipeline and logging model inputs at serving time with privacy controls. The candidate should separate skew from drift, because the first is a pipeline consistency bug and the second is a world-change problem. A common miss is checking only model metrics and not the raw feature distributions.

Expect these follow-ups

• What feature would you inspect first in a velocity-based fraud model?
• How do you log online features without creating a privacy problem?
• How would you prevent this class of issue in CI?

As asked

Design a shared GPU cluster for research training jobs, batch fine-tunes, and low-latency inference. How do you schedule work fairly and efficiently?

Sample answer outline

Separate workload classes because inference and long training jobs have different latency and preemption tolerance. Use quotas, priority classes, gang scheduling for distributed training, and preemptible queues for exploratory work. Strong answers discuss GPU fragmentation, topology awareness, checkpointing, fair-share policies, and visibility into queue time by team. Inference should usually live in reserved pools with autoscaling and admission control rather than competing directly with training. The common failure is maximising utilisation while making urgent production workloads wait behind low-priority experiments.

Expect these follow-ups

• How do you handle a training job that needs eight GPUs on the same node group?
• What metrics tell you the scheduler policy is unfair?
• When would you split research and production into separate clusters?

As asked

A scientist proposes a new attention variant that improves long-context performance. How would you design the ablation study?

Sample answer outline

Define the hypothesis and isolate the moving part, otherwise the experiment becomes a comparison of entire stacks. Keep training data, token budget, optimiser, batch size, context length, and evaluation scripts constant unless the ablation explicitly tests one of them. Use tasks that reflect the claimed win, such as retrieval over long context, multi-hop reasoning, and latency or memory measurements at increasing sequence lengths. Strong candidates discuss confidence intervals, repeated seeds for noisy runs, and negative results that still teach something. The common mistake is celebrating one benchmark win while hiding compute cost or regressions on shorter-context tasks.

Expect these follow-ups

• How many seeds are enough for this experiment?
• What if the method wins only at a much higher compute budget?
• How do you report a promising but inconclusive result?

As asked

Your model scores unusually well on a public benchmark. How would you investigate possible data contamination?

Sample answer outline

First check whether benchmark examples, paraphrases, or answer keys appear in pretraining, fine-tuning, retrieval corpora, or synthetic data generation prompts. Use exact matching, fuzzy matching, embedding similarity, and n-gram overlap because contamination is often transformed rather than copied. A strong answer distinguishes accidental exposure from memorisation and from legitimate generalisation. The candidate should propose a clean holdout or newly authored evaluation to validate the result. Interviewers want to hear that high scores are treated as a debugging signal, not only as good news.

Expect these follow-ups

• How do you handle contamination discovered after a paper is submitted?
• What if you cannot inspect the full pretraining corpus?
• How would you design a benchmark that ages better against contamination?

As asked

How would you build a synthetic data pipeline for instruction tuning without filling the training set with low-quality or duplicated examples?

Sample answer outline

Start with a target capability map so synthetic examples are generated for known gaps rather than volume for its own sake. Use strong prompts, multiple generators if possible, and filters for duplication, toxicity, format validity, factuality, and answer diversity. Human review should focus on calibration and hard categories rather than checking every example manually. Strong candidates discuss contamination, train-test leakage, model collapse from self-generated data, and weighting synthetic data below high-quality human data. The common trip-up is optimising token count instead of marginal training value.

Expect these follow-ups

• How do you detect near-duplicate examples at scale?
• When would synthetic data hurt model quality?
• How do you decide the mix of human and synthetic examples?