AI infrastructure engineer at Asana

As asked

Design an internal LLM gateway used by ten product teams. It must support provider fallback, prompt versioning, cost controls, and audit logging.

Sample answer outline

The gateway should be a policy and observability layer, not just an HTTP proxy. It needs typed request contracts, prompt and model version identifiers, tenant-level budgets, retries with idempotency, and provider fallback rules that are explicit about quality and compliance tradeoffs. Audit logs should capture inputs, outputs, metadata, and redaction state according to data policy. The answer should include latency budgets and failure modes, such as a fallback model producing different JSON shape or safety behaviour. Strong candidates avoid automatic fallback for high-risk flows unless the evals prove equivalence.

Expect these follow-ups

• How do you prevent fallback from masking a provider outage?
• What data do you store when prompts contain personal data?
• How do teams override defaults without bypassing governance?

As asked

Design a shared GPU cluster for research training jobs, batch fine-tunes, and low-latency inference. How do you schedule work fairly and efficiently?

Sample answer outline

Separate workload classes because inference and long training jobs have different latency and preemption tolerance. Use quotas, priority classes, gang scheduling for distributed training, and preemptible queues for exploratory work. Strong answers discuss GPU fragmentation, topology awareness, checkpointing, fair-share policies, and visibility into queue time by team. Inference should usually live in reserved pools with autoscaling and admission control rather than competing directly with training. The common failure is maximising utilisation while making urgent production workloads wait behind low-priority experiments.

Expect these follow-ups

• How do you handle a training job that needs eight GPUs on the same node group?
• What metrics tell you the scheduler policy is unfair?
• When would you split research and production into separate clusters?

As asked

Walk me through the three sharding strategies in PyTorch FSDP: FULL_SHARD, SHARD_GRAD_OP, and NO_SHARD. When would you pick each one, and what are the memory and communication tradeoffs at scale?

Sample answer outline

FULL_SHARD shards parameters, gradients, and optimizer state across all ranks, giving the lowest peak memory but requiring two all-gather passes per forward and one reduce-scatter per backward. SHARD_GRAD_OP keeps parameters unsharded during the forward pass, trading memory for fewer collective calls. NO_SHARD is DDP. A strong answer covers the tradeoff between peak GPU memory and all-reduce/all-gather volume, and explains when gradient checkpointing interacts with FSDP prefetching.

Expect these follow-ups

• How does FSDP's auto_wrap_policy interact with transformer layer boundaries, and what happens if you wrap at the wrong granularity?
• How do you profile the all-gather overhead in FSDP and what tools do you use?

As asked

You have a training loop where you want to overlap CUDA kernel execution with CPU-side data preprocessing. Walk me through how you'd structure streams and events to do this correctly, and what bugs commonly appear when developers get the synchronization wrong.

Sample answer outline

Create a separate CUDA stream for compute and another for data transfer; use cudaEvent_t to signal when a transfer is complete before the compute stream proceeds. Common bugs: forgetting to record the event on the correct stream, calling cudaStreamSynchronize on the wrong stream blocking the CPU, and assuming the default stream is non-blocking. A strong answer also covers how torch.cuda.Stream and stream context managers map onto these primitives.

Expect these follow-ups

• What is the difference between cudaStreamSynchronize and cudaDeviceSynchronize in terms of performance impact?

As asked

Describe what DeepSpeed ZeRO eliminates at each stage. Given a 70B parameter model, how would you estimate peak memory per GPU at ZeRO-3 with mixed precision, and what communication overhead does ZeRO-3 add over ZeRO-1?

Sample answer outline

ZeRO-1 shards optimizer states, ZeRO-2 adds gradient sharding, ZeRO-3 shards parameters as well. For a 70B model in bf16 with Adam, ZeRO-3 on N GPUs keeps roughly (70B * 16 bytes) / N bytes of optimizer state plus activation memory. ZeRO-3 adds parameter gather collectives on every forward/backward pass, roughly doubling communication volume compared to ZeRO-1. Mention ZeRO-Infinity for NVMe offload and the latency implications.

Expect these follow-ups

• How does ZeRO-3 interact with activation checkpointing, and can you combine them safely?

As asked

How does activation (gradient) checkpointing reduce memory, and what is the theoretical recompute cost as a fraction of total training FLOPs? How do you decide which layers to checkpoint in a transformer, and what goes wrong if you checkpoint too aggressively?

Sample answer outline

Checkpointing stores only a subset of activations and recomputes the rest during the backward pass, reducing activation memory from O(L) to O(sqrt(L)) with optimal placement. The recompute adds roughly one extra forward pass worth of FLOPs. In transformers, checkpointing at the attention or MLP boundary is standard. Checkpointing too aggressively increases recompute overhead disproportionately and can also interfere with FSDP's prefetching if the recompute triggers unexpected collective calls.

Expect these follow-ups

• How does selective activation recomputation in Megatron-LM differ from full per-layer checkpointing?