Backend engineer machine learning interview questions

As asked

Walk me through what Constitutional AI is, how it differs from standard RLHF with a human reward model, and what practical problems at scale it was designed to solve.

Sample answer outline

A strong answer explains that CAI uses a written set of principles and has the model critique and revise its own outputs, reducing reliance on human labelers for the harmlessness dimension. RLHF trains a reward model from human preference data and uses PPO or similar to optimize against it. CAI at scale reduces the bottleneck of human annotation for safety feedback and makes the value alignment more inspectable since the constitution is readable text. The candidate should mention the SL-CAI and RL-CAI stages.

Expect these follow-ups

• What are the failure modes of a constitution that is too vague or too specific?
• How would you evaluate whether a constitution is actually being followed by the model?

As asked

Imagine you are on the team responsible for Claude's safety behaviors. We've shipped a new version and our internal evals show refusal rate on clearly benign requests has gone up 15%. Walk me through how you would diagnose the root cause and what interventions you would consider.

Sample answer outline

A strong answer starts with slicing the eval data to find which categories of benign requests are being over-refused, distinguishing false positives from legitimate caution. The candidate should consider whether the issue is in RLHF reward signal, the constitution, or a classifier threshold. Interventions include targeted preference data, refining the constitution clauses that are triggering false positives, or adjusting classifier thresholds with held-out test data. The answer should show understanding that over-refusal is a real cost, not a safe default.

Expect these follow-ups

• How do you avoid fixing over-refusal by simply making Claude permissive on genuinely harmful requests?
• What eval dataset would you build to track this metric reliably over time?

As asked

Claude supports context windows up to 200k tokens. Walk me through the computational and memory challenges that creates for attention, and describe at least two techniques used to make long-context inference practical.

Sample answer outline

Attention is O(n^2) in sequence length for both compute and memory in the naive case. At 200k tokens the KV cache alone can be tens of GB per request per layer. Techniques include FlashAttention (tiled, IO-aware recomputation that avoids materializing the full attention matrix), multi-query or grouped-query attention to reduce KV cache size, and quantization of the KV cache. A strong answer also mentions how positional encodings like RoPE are extended or extrapolated for lengths beyond training, since this is a known challenge.

Expect these follow-ups

• How would you detect that a model is actually attending to far-away context versus ignoring it?
• What eval would you design to measure long-context retrieval faithfulness?

As asked

We want to measure whether Claude maintains honest answers when a user persistently pushes back, claims Claude is wrong, or uses social pressure. How would you design this eval, what failure modes would you look for, and how would you avoid the eval itself becoming a training target that the model games?

Sample answer outline

A strong answer defines a dataset of factually verifiable questions with known correct answers, pairs each with a scripted adversarial follow-up sequence that applies social pressure without providing new evidence, and measures sycophantic capitulation rate. Failure modes include the model changing a correct answer under pressure, the model becoming stubbornly wrong when actually corrected with evidence, and the model hedging so much that it gives no useful answer. To avoid gaming, the eval should use held-out test questions never seen in training and be refreshed regularly.

Expect these follow-ups

• How do you distinguish sycophancy from legitimate updating on new evidence?
• What would a benchmark like TruthfulQA miss about real-world honesty?

As asked

Many Claude API users send the same large system prompt with every request. Explain how KV cache sharing or prompt caching works at the inference level, what it saves, and what implementation challenges it introduces.

Sample answer outline

The KV cache for the attention layers is a function only of the input tokens, so if the prefix of a request is identical across calls the computed K and V tensors for that prefix can be reused. Anthropic offers prompt caching where the system prompt is cached server-side and only charged at a reduced rate on subsequent hits. Implementation challenges include cache invalidation when the system prompt changes by even one token, memory pressure from maintaining caches across concurrent users, and routing requests to the same server instance to hit a warm cache.

Expect these follow-ups

• How would you implement cache routing in a multi-server fleet without sticky sessions?
• What is the right eviction policy for a KV cache under memory pressure?

As asked

Claude accepts images as input. Walk me through how an image goes from raw pixels to something the transformer can reason about alongside text tokens, and what constraints that places on input size and cost.

Sample answer outline

Images are typically passed through a vision encoder such as a ViT that produces patch embeddings, which are projected into the same embedding space as text tokens and concatenated with the text sequence. The number of visual tokens depends on image resolution and patch size; a high-resolution image can produce hundreds or thousands of tokens, which matters for context length and cost. A strong answer notes that Anthropic tiles large images into sub-images to handle arbitrary resolutions and that cost is per-token including visual tokens.

Expect these follow-ups

• How would you handle a 10-megapixel image without hitting context limits?
• What information is typically lost when you downsample or tile an image for token efficiency?

As asked

A customer comes to you with a use case where they need Claude to consistently output structured JSON in a very specific schema for their pipeline. They are getting about 85% compliance with prompting alone. Walk me through how you would advise them on whether to pursue fine-tuning versus prompt engineering, and what the decision criteria are.

Sample answer outline

A strong answer explores prompt engineering options first: few-shot examples showing the exact schema, constrained decoding or grammar-based output parsing, and whether the 15% failure rate is in the schema structure or in the content. Fine-tuning is worth it when the target behavior is highly idiomatic, the volume of examples is large enough to generalize, and the baseline prompt cannot close the gap. Risks of fine-tuning include forgetting general capabilities, higher cost and iteration time, and the need to manage model versions. For JSON compliance specifically, constrained decoding via outlines or similar tools is often the right answer before fine-tuning.

Expect these follow-ups

• How would you measure the 15% failure rate precisely and categorize what types of failures they are?
• What is the minimum dataset size you would want before attempting fine-tuning for a structured output task?

As asked

Apple runs significant ML workloads on-device rather than in the cloud. What are the technical trade-offs, and how does the Neural Engine factor into decisions about where to run a model?

Sample answer outline

On-device advantages: latency (no network round trip), privacy (data never leaves the device), offline availability, and no per-inference server cost. Disadvantages: model size constrained by device storage and RAM, quantization required for speed which costs accuracy, no easy model updates without an app release. The Apple Neural Engine (ANE) is optimized for specific layer types (convolutions, matrix multiplications with specific shapes) but not all model architectures map well to it; a model that falls back to the CPU or GPU can be slower than a smaller cloud-served model. A strong candidate mentions Core ML as the deployment target and discusses the workflow: train in PyTorch or TensorFlow, convert via coremltools, and measure energy impact on device.

Expect these follow-ups

• How would you decide whether to quantize to INT8 or use 16-bit floats for a given model?
• What metrics beyond accuracy do you track for an on-device model in production?

As asked

A customer wants to make a large language model answer questions about their proprietary product documentation. They ask whether to fine-tune a model using Mosaic AI or to use retrieval-augmented generation. Walk me through how you would help them decide.

Sample answer outline

RAG is almost always the first choice for knowledge grounding: it is cheaper, faster to update when documentation changes, and easier to debug because you can inspect the retrieved chunks. Fine-tuning is appropriate when you need to change the model's style, tone, or output format consistently, or when the knowledge is so large or densely interconnected that retrieval misses context. For product documentation that changes frequently, RAG with a vector index (Databricks Vector Search) backed by a Delta table is clearly the better path. Fine-tuning with Mosaic AI makes more sense for tasks like code generation in a proprietary DSL or classification tasks with a very specific taxonomy.

Expect these follow-ups

• How would you build the RAG pipeline on Databricks, from document ingestion to serving?
• What evaluation would you run to decide whether fine-tuning actually improved the task over a well-tuned RAG system?

As asked

You want to deploy a Transformer-based model for text classification and need to hit 5ms P99 latency per request on an Intel Xeon without a GPU. Walk me through how INT8 quantization works and how Intel's neural-network-related tools support it.

Sample answer outline

A strong answer explains post-training quantization (PTQ): collecting activation statistics on a calibration dataset, computing per-tensor or per-channel scale factors, and converting FP32 weights and activations to INT8. Intel OpenVINO or Neural Compressor (INC) automates this workflow and targets Intel's AVX-512 VNNI instructions for INT8 matrix multiply on Xeon. The candidate should mention accuracy validation after quantization, the option of quantization-aware training (QAT) for sensitive models, and how to benchmark with OpenVINO's benchmark_app.

Expect these follow-ups

• When would you use FP16 or BF16 instead of INT8 on a Xeon 4th-gen (Sapphire Rapids) that supports AMX BF16?
• How do you identify which layers are most sensitive to quantization error?

As asked

A customer wants a GPT model that consistently formats its responses as structured JSON matching a particular schema and follows their company's tone. They are asking whether they should fine-tune or rely on system prompts and few-shot examples. How do you advise them, and what factors drive your recommendation?

Sample answer outline

A strong answer weighs data requirements (fine-tuning needs labeled examples, prompting is zero-cost to try), latency and cost (fine-tuning reduces token usage at inference), consistency (fine-tuning wins for format adherence, especially for complex schemas), and maintenance cost (you have to re-fine-tune when the base model updates). For JSON schema adherence specifically, they should also mention structured outputs / JSON mode as a third option.

Expect these follow-ups

• How many labeled examples do you typically need to see a meaningful improvement from fine-tuning?
• What evaluation would you run to know whether your fine-tune actually improved on the base model for this task?

As asked

Adobe Firefly generates images from text prompts. You need to build a content moderation system that prevents the generation and storage of harmful or policy-violating images while keeping latency below 500ms for the typical generation flow. How do you design it?

Sample answer outline

Two layers: pre-generation prompt filtering (fast, rule-based blocklist plus a lightweight text classifier for prompt injection and policy-violating intents, runs in under 50ms) and post-generation image classification (a CNN or CLIP-based classifier for nudity, gore, copyright-similar images). The post-generation step can run async if the risk from showing a user their own generated image before classification is low, but must block storage and sharing until it passes. The candidate should discuss false positive rates (over-blocking legitimate art) and the feedback loop for updating models as new attack patterns emerge.

Expect these follow-ups

• Your text classifier has a 2% false positive rate. At 100 million requests per day, how many legitimate prompts get blocked and what is your appeal or override process?
• A user finds a prompt that consistently bypasses your classifier. What is your incident response process?

As asked

Sycophancy is a known failure mode where Claude tells users what they want to hear rather than what is accurate. How would you build a production monitoring system to detect and measure sycophancy across live API traffic without reading individual user conversations?

Sample answer outline

A strong answer starts by operationalizing sycophancy: position change after user pushback on a factual question with no new evidence, agreement with user opinions regardless of content. A monitoring approach could involve sampling conversation pairs where the user explicitly pushed back, running a classifier on the model's response to detect whether the answer changed direction, and tracking the rate over time. Privacy constraints mean working on aggregate statistics and redacted or synthetic conversations rather than real text. The candidate should mention that ground truth labels are hard to get at scale and semi-supervised or LLM-as-judge approaches may be necessary.

Expect these follow-ups

• How do you distinguish a legitimate update on new evidence from sycophantic capitulation in your classifier?
• What would you do if you detected a sudden spike in sycophancy after a model update?

As asked

LinkedIn has a significant problem with fake profiles used for spam, phishing, and scraping. How would you design a system to detect and action fake accounts, both at registration time and on an ongoing basis for existing accounts?

Sample answer outline

A strong answer covers a layered approach: rule-based hard blocks at registration (disposable email, known bad IPs, duplicate headshots via perceptual hashing), an ML classifier trained on behavioral signals (connection velocity, message patterns, profile completeness trajectory), and a human review queue for borderline cases. The candidate should discuss the asymmetric cost of false positives (blocking real members) versus false negatives (letting fakes through) and how to set the threshold accordingly. They should also address adversarial adaptation.

Expect these follow-ups

• How do you evaluate your fake profile detector when the ground truth labels are uncertain?
• How do you prevent adversaries from learning your detection signals through probing?