AI engineer at GitHub

As asked

Design a RAG feature that answers customer support questions using internal documentation. How do you keep answers grounded and know when the model should refuse?

Sample answer outline

A good answer separates retrieval quality from generation quality. Chunk documents around semantic boundaries, attach metadata such as product, version, and freshness, and retrieve with hybrid search plus reranking for high precision. The generation prompt should require citations to retrieved passages and should refuse when evidence is missing or contradictory. Evaluation needs golden questions, adversarial no-answer cases, citation accuracy checks, and human review for high-impact categories. Candidates often focus on vector databases while ignoring stale docs, permissions, and the model's tendency to fill gaps.

Expect these follow-ups

• How do you prevent a user from retrieving documents they cannot access?
• What metric tells you retrieval is the bottleneck?
• How do you handle two documents that disagree?

As asked

A support assistant needs conversation history, policy docs, customer metadata, and tool results, but the context window is regularly exceeded. How do you redesign the prompt flow?

Sample answer outline

The answer should treat the context window as a scarce product resource. Keep durable instructions short, summarise long conversation history with provenance, retrieve only task-relevant policy chunks, and pass customer metadata as a compact structured block rather than prose. Tool results should be schema-shaped and pruned to fields the next step needs. Add tests for truncation boundaries because prompt changes can silently push out important context. Candidates often rely on larger context windows instead of deciding what information is actually needed for the task.

Expect these follow-ups

• How do you know a summary has not lost a critical fact?
• What context should never be summarised?
• How would you debug a failure caused by truncation?

As asked

Write a TypeScript tool definition for creating a support ticket, then explain how you would make the schema safe for agent use.

Sample answer outline

The schema should make required fields explicit, restrict enums, and avoid accepting vague free-form blobs for important state. Tool descriptions should say when to call the tool and when not to call it, but the server must still validate permissions and business rules. A good answer includes idempotency keys or deduplication because agents may retry after timeouts. The tool should return a small structured result the agent can reason over, not an entire database record. Candidates often make tools too broad, which increases accidental side effects and makes eval failures hard to diagnose.

Reference implementation (typescript)

import { z } from "zod";

export const createSupportTicketTool = {
  name: "create_support_ticket",
  description:
    "Create one support ticket after the user has described a concrete problem. Do not call for general questions.",
  inputSchema: z.object({
    customerId: z.string().min(1),
    idempotencyKey: z.string().uuid(),
    severity: z.enum(["low", "normal", "high", "urgent"]),
    subject: z.string().min(5).max(120),
    description: z.string().min(20).max(4000),
    orderId: z.string().optional(),
  }),
  async execute(input: unknown) {
    const args = this.inputSchema.parse(input);
    return {
      ticketId: await createTicket(args),
      status: "created" as const,
    };
  },
};

Expect these follow-ups

• How do you version this tool without breaking old agents?
• Where does human approval fit?
• What should the tool return on validation failure?

As asked

Walk me through the most common failure modes you have seen in RAG pipelines. For each one, how do you diagnose it and what do you actually change in the pipeline to fix it?

Sample answer outline

Strong answers cover at least four distinct failures: low recall from poor chunking or embedding mismatch, low precision from over-retrieval, lost-in-the-middle where the LLM ignores middle context, and stale index causing factual drift. For each they describe a concrete diagnostic signal (retrieval MRR, LLM faithfulness score, timestamp checks) and the specific fix (re-chunking strategy, MMR reranking, context compression, incremental indexing).

Expect these follow-ups

• How do you measure retrieval quality independently of generation quality?
• When would you use a reranker versus tuning the embedding model?

As asked

You are building a semantic search feature. How do you select an embedding model, and how do you actually measure whether it is good enough for your domain before committing to it?

Sample answer outline

Strong answers mention MTEB benchmarks as a starting point but immediately note that MTEB does not reflect domain-specific distributions. The candidate describes building a domain eval set of query-passage pairs, measuring nDCG or MRR, comparing multiple models (OpenAI text-embedding-3, Cohere embed-v3, BGE, e5-large), and optionally fine-tuning on domain data. They also cover dimension and latency as constraints.

Expect these follow-ups

• When would you fine-tune an embedding model versus just prompting differently?
• How do you handle multilingual content in your embedding pipeline?

As asked

In a production QA system backed by a document corpus, what techniques do you use to reduce hallucination, and how do you measure whether your changes are actually working?

Sample answer outline

Good answers mention grounding the prompt with retrieved context and explicit instructions to cite or refuse, lowering temperature for factual tasks, using chain-of-thought to force reasoning before answering, adding a faithfulness check via a second LLM call or an NLI model, and tracking faithfulness/groundedness metrics (like RAGAS) in a regression suite. The best answers distinguish hallucination types: fabrication versus faithful-but-wrong retrieval.

Expect these follow-ups

• How does your approach change if the user question has no answer in the corpus?
• What trade-off do you accept when you increase citation strictness?