Generative model engineer system design interview questions

As asked

You need to serve a 7B parameter instruction-tuned LLM for a production application with a p99 latency target of under 2 seconds for 200-token responses. Design the inference infrastructure, including model serving, batching strategy, scaling, and observability.

Sample answer outline

A strong design uses vLLM or TGI with continuous batching on A10G or A100 GPUs, quantized to INT8 or FP8 to increase throughput per GPU. At 100k req/day (~1.15 req/s average) a single well-optimized GPU likely suffices with autoscaling headroom. Load balancing across 2 to 3 replicas handles burst traffic. Observability: per-request TTFT, TPS, queue depth, GPU utilization, and KV cache hit rate. A CDN or Redis cache for repeated identical prompts reduces load. Streaming responses via SSE minimizes perceived latency even before generation completes.

Expect these follow-ups

• How does your design change if the p99 latency target drops to 500ms?
• How would you handle model versioning and safe rollouts when deploying a new fine-tuned version?

As asked

Walk me through how you would design the end-to-end training pipeline for a 1 billion parameter text-to-image diffusion model. Cover data ingestion, preprocessing, storage, the training cluster setup, checkpointing, and how you monitor for training instabilities.

Sample answer outline

A strong design covers: data pipeline (image-text pairs from LAION or similar, filtered by CLIP score and NSFW classifiers, stored as WebDataset tar shards on object storage for streaming); distributed training with FSDP or DeepSpeed ZeRO-3 across 64 to 256 A100s; gradient accumulation to reach effective batch sizes of 2048+; EMA of weights maintained alongside training weights; checkpoint every 5k steps to object storage with async upload; monitoring via wandb with loss, gradient norm, EMA divergence, and sample grids at regular intervals; training data leakage checks against evaluation sets.

Expect these follow-ups

• How would you handle a node failure mid-training with checkpointing overhead in mind?
• What does your evaluation loop look like and how often do you run FID during training?

As asked

Before shipping a new fine-tuned LLM to production, you need a comprehensive evaluation harness. What evaluations would you run, how would you automate them, and how would you make the results comparable across model versions over time?

Sample answer outline

A strong harness covers: reference benchmarks (MMLU, HellaSwag, HumanEval for code) for capability regression checks; task-specific evals on held-out examples from the fine-tuning domain; LLM-as-judge evaluations for open-ended quality dimensions like helpfulness and coherence; red-team / adversarial prompt sets for safety regression; latency and throughput benchmarks at fixed load. All results are stored in a versioned database keyed by model checkpoint hash, enabling longitudinal comparison. A CI/CD trigger runs evals automatically on every new checkpoint.

Expect these follow-ups

• How do you prevent your eval set from leaking into training data over time as you iterate?
• What is your process when a new model passes automated evals but gets bad feedback from early human testers?

As asked

Design the full RLHF pipeline from collecting human preference data to deploying the aligned model. Include the annotation interface, reward model training, RL fine-tuning, and quality gates before deployment.

Sample answer outline

Pipeline stages: SFT checkpoint from instruction tuning; annotation UI that shows human raters two model responses side-by-side for ranking, with guidelines and inter-annotator agreement monitoring; reward model trained on pairwise preferences with held-out eval on a separate annotation batch; PPO or DPO fine-tuning of the SFT model guided by the reward model, with KL penalty relative to SFT to prevent drift; automated evaluation against prior model on safety, helpfulness, and capability benchmarks; canary deployment with traffic splitting and rollback trigger on regression.

Expect these follow-ups

• How do you handle annotator disagreement when inter-annotator agreement is below 70% on a prompt category?
• What is your strategy for ensuring the reward model generalizes to prompts not in the training distribution?

As asked

You are building the safety layer for a public text-to-image API. Design a pipeline that filters harmful prompts before generation and checks generated images before returning them, at a latency budget of 100ms total overhead.

Sample answer outline

Pre-generation: fast keyword and pattern matcher for obvious policy violations (<1ms), followed by a lightweight text classifier (DistilBERT-scale) for nuanced harmful intent (~10ms). Post-generation: a multi-label image safety classifier checking for NSFW, violence, and minors (<30ms on GPU). Both classifiers run as separate microservices to scale independently. Borderline cases are logged and routed to human review queue asynchronously. Thresholds are configurable per API tier. A strong answer discusses false positive rate tradeoffs and the appeal process for incorrectly blocked content.

Expect these follow-ups

• How do you handle adversarial prompt injection attempts that try to bypass the text classifier?
• What is your strategy for updating classifiers without downtime as new harmful categories emerge?

As asked

A multimodal generation feature is too slow for interactive use. How would you reduce latency without destroying output quality?

Sample answer outline

Break latency into queueing, pre-processing, model inference, post-processing, and network time before changing the model. Candidate optimisations include smaller distilled models, quantisation, caching repeated inputs, batching where it does not hurt interactivity, progressive previews, and asynchronous high-quality refinement. Strong answers evaluate quality with task-specific tests after every optimisation rather than assuming faster is acceptable. They also mention capacity planning because queueing delay can dominate model time under bursty demand. The common mistake is jumping straight to a smaller model without measuring the real bottleneck.

Expect these follow-ups

• When does batching improve throughput but hurt p95 latency?
• How would you expose progressive generation in the product?
• What quality metric would block a latency optimisation?

As asked

Your company wants to let product teams fine-tune foundation models on their own domain data without requiring ML expertise. Design a platform that handles data ingestion, experiment tracking, hyperparameter selection, training orchestration, and model serving of the resulting adapter.

Sample answer outline

A strong design includes: a data upload API with automatic format validation and PII scanning; a default LoRA recipe with sensible hyperparameter defaults (rank 16, alpha 32, cosine LR schedule) and optional Optuna-based sweeps; training jobs submitted to a queue backed by Kubernetes with GPU node pools; MLflow or W&B for experiment tracking linked to job IDs; automated evaluation against a held-out validation split with configurable metrics; model registry storing the LoRA adapter alongside the base model pointer; and a serving layer that loads the adapter at request time or merges it for low-latency serving.

Expect these follow-ups

• How do you prevent teams from accidentally fine-tuning away the base model's safety behaviors?
• What is your strategy for multi-tenant isolation if two teams are training simultaneously on the same cluster?

As asked

You are building the data pipeline for a large-scale image-text model trained on 1 billion image-text pairs. Design the pipeline from raw web crawl to training-ready shards, covering filtering, deduplication, quality scoring, storage format, and streaming to the training cluster.

Sample answer outline

A strong design: crawl produces HTML with image URLs; extract alt-text and captions; download images with distributed workers (Datatrove or img2dataset); filter by resolution (min 256x256), aspect ratio, file size, and NSFW score; deduplicate with MinHash LSH; score image-text alignment with CLIP and filter below threshold 0.28; encode images to JPEG at training resolution; pack into WebDataset .tar shards of 100MB each stored on S3 or GCS; stream shards to the training cluster with WebDataset's IterableDataset, shuffling shard order each epoch.

Expect these follow-ups

• How would you handle the long tail of image domains that are underrepresented in CLIP's embedding space?
• What metadata would you store alongside each shard to enable data ablation experiments later?

As asked

Your team maintains multiple base models and dozens of LoRA adapters trained by different teams. Design a model registry that tracks lineage, enables rollback, supports adapter composition, and integrates with your serving infrastructure.

Sample answer outline

The registry stores base model artifacts keyed by content hash with associated metadata (architecture config, training run ID, eval scores). Adapters reference their parent base model hash, the fine-tuning dataset version, and their own eval results. Composition is tracked as a DAG so you can trace any merged adapter back to its training data and base model. Serving integration pulls the latest stable adapter for a given base + task combination based on eval score thresholds. Rollback is a promotion operation that changes which adapter version is tagged as stable.

Expect these follow-ups

• How do you handle the case where a base model is deprecated but many adapters depend on it?
• What database schema would you use to store the lineage DAG efficiently?

As asked

You manage a cluster of 128 A100 GPUs shared across 20 research teams. Design a scheduler that maximizes GPU utilization, supports preemption for high-priority jobs, handles gang scheduling for multi-GPU runs, and gives teams fair access over time.

Sample answer outline

A strong design uses Kubernetes with the Volcano or Kueue scheduler for batch ML workloads. Gang scheduling ensures all requested GPUs are allocated atomically before a job starts, preventing deadlocks. Fair-share with a time-windowed quota tracks each team's historical consumption and penalizes over-quota submissions. Preemption checkpoints running low-priority jobs asynchronously (saving to object storage) and frees their GPUs for high-priority jobs. A job queue dashboard shows estimated wait times and resource pressure per team.

Expect these follow-ups

• How do you handle the case where a 64-GPU job is waiting but only 32 GPUs are free because a 32-GPU job is not checkpointable?
• What monitoring signals tell you the cluster is underutilized due to scheduling inefficiency vs genuinely idle?