AI infrastructure engineer system design interview questions

As asked

Design an internal LLM gateway used by ten product teams. It must support provider fallback, prompt versioning, cost controls, and audit logging.

Sample answer outline

The gateway should be a policy and observability layer, not just an HTTP proxy. It needs typed request contracts, prompt and model version identifiers, tenant-level budgets, retries with idempotency, and provider fallback rules that are explicit about quality and compliance tradeoffs. Audit logs should capture inputs, outputs, metadata, and redaction state according to data policy. The answer should include latency budgets and failure modes, such as a fallback model producing different JSON shape or safety behaviour. Strong candidates avoid automatic fallback for high-risk flows unless the evals prove equivalence.

Expect these follow-ups

• How do you prevent fallback from masking a provider outage?
• What data do you store when prompts contain personal data?
• How do teams override defaults without bypassing governance?

As asked

A multimodal generation feature is too slow for interactive use. How would you reduce latency without destroying output quality?

Sample answer outline

Break latency into queueing, pre-processing, model inference, post-processing, and network time before changing the model. Candidate optimisations include smaller distilled models, quantisation, caching repeated inputs, batching where it does not hurt interactivity, progressive previews, and asynchronous high-quality refinement. Strong answers evaluate quality with task-specific tests after every optimisation rather than assuming faster is acceptable. They also mention capacity planning because queueing delay can dominate model time under bursty demand. The common mistake is jumping straight to a smaller model without measuring the real bottleneck.

Expect these follow-ups

• When does batching improve throughput but hurt p95 latency?
• How would you expose progressive generation in the product?
• What quality metric would block a latency optimisation?

As asked

Design a shared GPU cluster for research training jobs, batch fine-tunes, and low-latency inference. How do you schedule work fairly and efficiently?

Sample answer outline

Separate workload classes because inference and long training jobs have different latency and preemption tolerance. Use quotas, priority classes, gang scheduling for distributed training, and preemptible queues for exploratory work. Strong answers discuss GPU fragmentation, topology awareness, checkpointing, fair-share policies, and visibility into queue time by team. Inference should usually live in reserved pools with autoscaling and admission control rather than competing directly with training. The common failure is maximising utilisation while making urgent production workloads wait behind low-priority experiments.

Expect these follow-ups

• How do you handle a training job that needs eight GPUs on the same node group?
• What metrics tell you the scheduler policy is unfair?
• When would you split research and production into separate clusters?