Backend engineer domain knowledge interview questions

As asked

Walk me through the memory hierarchy in a modern AMD GPU, from the farthest off-chip memory to the closest on-chip storage a shader sees. How does each level affect kernel performance?

Sample answer outline

A strong answer covers HBM2e or HBM3 at the board level (high bandwidth but high latency), the L2 cache shared across all compute units, L1/texture cache per compute unit, Local Data Share (LDS, AMD's name for shared memory) inside each compute unit, and vector/scalar register files inside each SIMD lane. The candidate should quantify typical bandwidth numbers for RDNA3 or CDNA2, explain cache line sizes, and name the access patterns that cause bank conflicts in LDS.

Expect these follow-ups

• What is the LDS bank width on CDNA2 and how do you avoid bank conflicts?
• How does the infinity fabric connect the GPU and CPU in an APU or MI300?

As asked

Claude's behavior is shaped by three parties: Anthropic, operators who access the API to build products, and end users. Explain how Claude is supposed to handle conflicts between instructions from each of these parties, and give a concrete example of a conflict and the expected resolution.

Sample answer outline

Anthropic's values and guidelines take highest precedence and are baked into the model via training. Operators can customize behavior within Anthropic's policies via the system prompt, such as restricting topics or enabling adult content on appropriate platforms. Users can adjust within what the operator permits. A conflict example: an operator says 'never discuss competitor products' but a user asks about them. Claude should follow the operator restriction since it is a plausible business reason, not an attempt to harm the user. Claude should not follow operator instructions that actively harm users or deceive them in damaging ways.

Expect these follow-ups

• What should Claude do if an operator instructs it to lie to users about being an AI?
• How does this hierarchy change in the API when there is no system prompt at all?

As asked

You have a Spark job joining a 10TB fact table with a 200MB dimension table and the query plan shows a sort-merge join. Explain why a broadcast hash join would be more efficient here, how Spark decides which join strategy to use, and how you would force the broadcast join if Spark does not choose it automatically.

Sample answer outline

A broadcast join sends the entire small table to every executor, avoiding a shuffle of the large table entirely. Spark chooses broadcast joins when the smaller table is below spark.sql.autoBroadcastJoinThreshold (default 10MB). At 200MB, Databricks may not broadcast automatically. The candidate should mention the broadcast() hint in DataFrame API or SQL: SELECT /*+ BROADCAST(dim) */ ... or df.join(broadcast(dim_df), ...). They should note the risk: broadcasting too large a table causes OOM on executors. The tradeoff between shuffle cost (sort-merge) and memory cost (broadcast) should be explicit.

Expect these follow-ups

• How do you diagnose in the Spark UI whether a join became a sort-merge instead of a broadcast?
• What happens if you broadcast a table that is too large for executor memory?

As asked

You are onboarding a backend engineer who has never used Datadog before. Explain the difference between metrics, distributed traces, and logs, and give a concrete example of when you would reach for each one to debug a latency problem in a microservices API.

Sample answer outline

A strong answer explains that metrics are numeric aggregations good for alerting on trends, traces capture the causal chain across service boundaries to find which hop is slow, and logs carry the raw event detail for understanding why a specific request failed. The debugging example should show the three being used in sequence: alert from a p99 latency metric, trace to find the slow downstream call, log on that service to read the specific error.

Expect these follow-ups

• How does Datadog's APM connect traces to the underlying infrastructure metrics?
• What is cardinality and why does it matter when choosing what to put in a metric tag?

As asked

You have a CUDA kernel that uses warp-level primitives like __shfl_down_sync and warp shuffle reductions. Walk me through how you would port that kernel to HIP for AMD GPUs, and what differences you would watch for.

Sample answer outline

A strong answer explains that AMD's wavefront is 64 lanes wide on CDNA hardware (MI series), not 32 like an NVIDIA warp, which means warp-level primitives need a mask or size change. In HIP, __shfl_down_sync becomes __shfl_down and the implicit warp size assumption of 32 must be replaced with warpSize or the wavefront size constant. The candidate should mention hipify-perl or hipify-clang for automated translation and flag the cases those tools miss, such as warp-size-dependent logic and texture fetch intrinsics. They should note that RDNA3 consumer GPUs use 32-lane wave32 as the DEFAULT mode (unlike CDNA, which defaults to wave64), so code targeting both GPU families must query or configure the wavefront size explicitly rather than assuming either value.

Expect these follow-ups

• What is the default wavefront size on CDNA2 MI250 and how does that affect occupancy calculations?
• How do you handle code that branches on the CUDA architecture enum?

As asked

Explain compute unit occupancy on AMD GPUs. How do register usage, LDS usage, and wavefront count interact, and how would you diagnose and improve low occupancy on a matrix multiplication kernel?

Sample answer outline

A strong answer defines occupancy as the ratio of active wavefronts to the maximum the hardware can support per compute unit. It explains that each register file is a fixed-size pool shared across wavefronts: more registers per thread means fewer wavefronts fit. LDS is similarly partitioned. The candidate should mention using rocprof or Omniperf to read actual occupancy, using __attribute__((amdgpu_num_vgpr)) hints to cap register usage, and explain the occupancy-latency hiding tradeoff where high occupancy hides memory latency but too many wavefronts fighting for cache can hurt performance.

Expect these follow-ups

• At what point does increasing occupancy stop helping and start hurting on a memory-bound kernel?
• How does the AMD CDNA2 compute unit differ structurally from RDNA3 in a way that affects occupancy tuning?

As asked

What is the key architectural difference between the MI300X and previous MI200-series cards, and why does that difference matter for large model inference?

Sample answer outline

A strong answer explains that MI300X is a 3D-stacked APU-class design with 192 GB of HBM3 memory unified between compute dies, which eliminates the CPU-to-GPU PCIe bottleneck for model weight transfers during inference. The candidate should contrast this with MI250X which uses two GCDs connected by Infinity Fabric with separate HBM stacks, explaining how MI300X's unified memory pool simplifies serving large LLMs without tensor parallelism across multiple cards. Mention of 5.2 TB/s aggregate HBM3 bandwidth and how that affects attention layer throughput is a differentiator.

Expect these follow-ups

• How does the MI300X memory capacity compare to an H100 SXM5, and what class of models does that enable in a single card?
• What ROCm version first provided stable FlashAttention support for MI300X?

As asked

You have a HIP kernel running slower than the roofline model predicts. Describe your profiling workflow using AMD's tooling to identify the bottleneck, from first suspicion to a confirmed fix.

Sample answer outline

A strong answer walks through rocprof for a first-pass timeline and hardware counter collection, then Omniperf for a detailed roofline analysis that separates compute-bound from memory-bound bottlenecks. The candidate should mention specific Omniperf panels (L2 cache efficiency, LDS utilization, vector ALU utilization), explain how to read the roofline chart to see which arithmetic intensity puts the kernel in which bound region, and describe the fix pattern for each: loop unrolling for compute-bound, coalescing or prefetching for memory-bound, LDS padding for bank conflicts.

Expect these follow-ups

• How do you collect per-wavefront metrics without serializing all wavefronts and destroying the timing signal?
• What is the difference between rocprof v1 and v2 in terms of counter sets you can collect simultaneously?

As asked

Enterprise customers embed Claude in their products and pass untrusted user content in the prompt. Describe a realistic prompt injection attack against such a system and at least three layers of defense you would build into the platform.

Sample answer outline

A strong answer describes the attack concretely: malicious text in user-supplied input instructs the model to ignore the system prompt and exfiltrate or reveal sensitive data. Defenses include structural separation of trusted system instructions from untrusted user content using clearly delimited sections, model-level training to treat differently-sourced content with appropriate skepticism, output classifiers that detect anomalous responses like data exfiltration patterns, and application-level monitoring for unusual tool call or action patterns. The candidate should note there is no single silver bullet and defense-in-depth is the right framing.

Expect these follow-ups

• How does this change if the model has access to external tools like code execution or web browsing?
• How would you test your defenses without exposing real customer data?

As asked

Claude's extended thinking feature lets you allocate a token budget for the model to reason before answering. How would you advise a customer on choosing that budget, and what engineering trade-offs does a large thinking budget create for the serving infrastructure?

Sample answer outline

Budget choice depends on task complexity: simple factual questions get no benefit from large budgets; hard multi-step reasoning or coding tasks may saturate at a certain budget and see diminishing returns. Infra trade-offs include longer time-to-first-token for the user, higher GPU memory occupancy per request blocking other requests, and higher cost per call. A strong answer also notes the budget must be chosen per use case rather than globally, and that Anthropic's evals can help identify where returns flatten.

Expect these follow-ups

• How would you benchmark whether a given task benefits from more thinking tokens?
• What serving strategy do you use when thinking-heavy requests coexist with quick-turnaround requests on the same fleet?

As asked

When Anthropic prepares a model card or responsible scaling policy evaluation before a major Claude release, what categories of harm do they assess, and how do those assessments influence the release decision?

Sample answer outline

Anthropic's model cards and the Responsible Scaling Policy cover uplift to CBRN weapons, cyberoffense capability, persuasion and deception at scale, and autonomy risks from agentic systems. Assessments use red-teaming, structured threat modeling, and third-party auditors. The RSP defines safety levels and stipulates what mitigations must be in place before a model can be deployed at a given capability tier. A strong answer shows the candidate has read the actual RSP or model cards rather than speaking in generalities.

Expect these follow-ups

• If a model passes all current evals but you have a strong intuition it is more capable than the evals suggest, how would you argue for more cautious deployment?
• What does it mean for a mitigation to be 'sufficient' under the RSP?

As asked

Anthropic is about to ship a new capability that lets Claude autonomously browse the web. You are on the red team. Describe your methodology for finding failure modes, what threat models you would focus on, and how you would prioritize findings for the release decision.

Sample answer outline

A strong answer starts with threat modeling: what new attack surfaces does web browsing open? Prompt injection via malicious web pages is the first-order risk. Other threats include data exfiltration, click fraud, account compromise if credentials are accessible, and the model being manipulated into taking harmful actions in the real world. Methodology includes structured attack scenarios, open-ended exploration by diverse red teamers including domain experts, and automated probing with adversarial page content. Findings are prioritized by impact times likelihood, with any critical path to CBRN uplift or mass harm blocking release.

Expect these follow-ups

• How do you red-team for risks that have never happened before and that you cannot enumerate in advance?
• What is the minimum bar of safety evidence you would require before approving web browsing capability for general release?

As asked

Imagine you are onboarding a new backend engineer to the Jira Software codebase. Walk me through the core data model: what are the primary entities, how do issues, projects, workflows, and boards relate, and where are the key constraints that affect query performance at Atlassian's scale?

Sample answer outline

A strong answer covers: Projects contain Issues; Issues have Types, Statuses, and Priorities; Workflows define valid transitions between Statuses; Boards are views on Issues filtered by JQL. At scale, the candidate should mention that Issues is the highest-cardinality table, that index design on project/status/assignee is critical, and that JQL translates to SQL with multi-tenant isolation per project.

Expect these follow-ups

• How would you shard this data model if a single tenant had 50 million issues?
• Where would you put a caching layer and what would you cache?

As asked

You have a Cisco router receiving the same prefix from three BGP peers. Walk me through every attribute the router evaluates, in the order it evaluates them, to pick the best path. What is the first thing that can eliminate a candidate and what is the final tiebreaker?

Sample answer outline

The complete order is: highest weight (Cisco-proprietary, local to the router only, never advertised to peers), highest local preference, locally originated routes (via network or redistribute), shortest AS path, lowest origin type (IGP < EGP < incomplete), lowest MED (compared only between paths from the same AS by default, behavior changes with bgp always-compare-med), eBGP over iBGP, lowest IGP metric to the BGP next hop, oldest eBGP route (this step only applies when bgp bestpath compare-routerid is NOT configured; with that command the age check is skipped), lowest BGP router ID, and finally lowest neighbor IP address as the final tiebreaker. A strong answer notes that weight is Cisco-specific and not sent to any peer, and that the oldest-route step and router-ID step are mutually exclusive depending on whether bgp bestpath compare-routerid is set.

Expect these follow-ups

• What is the difference between local preference and MED and who controls each?
• When would you want to manipulate AS path length to influence inbound traffic?

As asked

You are troubleshooting an OSPF topology where a remote subnet is not showing up in routing tables. Which LSA types would you check and why? Walk me through what each LSA type represents and where in the topology it gets flooded.

Sample answer outline

LSA types 1 (router) and 2 (network) are flooded within a single area. Type 3 (summary) is generated by the ABR and crosses area boundaries. Type 4 (ASBR summary) tells other areas where to find the ASBR. Type 5 (external) carries external routes and is flooded across the entire OSPF domain except stub and NSSA areas. Type 7 (NSSA external) is generated inside a Not-So-Stubby Area by an ASBR and is converted to a Type 5 by the ABR when leaving the NSSA. Missing a Type 7 LSA is a common cause of routes vanishing at the NSSA boundary. For troubleshooting a missing subnet, a strong answer starts with 'show ip ospf database' to check whether the LSA exists at all, then checks for area boundary filtering, route summarization, stub or NSSA area configuration, and whether the ABR is correctly translating Type 7 to Type 5.

Expect these follow-ups

• What is the difference between a stub area and a totally stubby area?
• Why might a Type 3 LSA exist in the database but not install a route in the RIB?

As asked

A customer is asking whether they should deploy Cisco Meraki MX firewalls and MS switches or go with the Catalyst and Firepower stack. How do you help them decide? What questions do you ask, and what are the real tradeoffs?

Sample answer outline

Meraki wins on cloud-managed simplicity, zero-touch provisioning, and single-pane-of-glass visibility across locations, making it ideal for distributed businesses with limited IT staff. Catalyst and Firepower win on depth of control, on-premises management for regulated industries, advanced routing protocols, and custom policy. Key questions to ask: Does the customer have on-prem networking staff? Do they need MPLS or advanced QoS? Are they in a highly regulated environment where cloud management is a concern? A strong answer also notes Cisco's push to converge these through Cisco+ and cloud-managed Catalyst.

Expect these follow-ups

• How does licensing differ between the two, and how does that affect TCO?
• What would you tell a customer who wants Meraki but operates in healthcare?

As asked

A customer asks you to explain what Cisco's Zero Trust Network Access (ZTNA) offering actually does, how Cisco Duo integrates with it, and how it differs from their existing VPN. They are specifically concerned about whether ZTNA can enforce device posture checks before granting access.

Sample answer outline

Cisco ZTNA (delivered via Cisco Secure Access) grants access per-application rather than per-network, so a user connecting to an HR app cannot automatically reach the finance server on the same subnet. Duo provides the identity and device posture layer: it enforces MFA and can check device health (OS version, disk encryption, endpoint agent status) before granting access, which VPN cannot do at the application level. Unlike VPN, which creates a network tunnel and then relies on ACLs, ZTNA mediates every connection through a cloud proxy and re-evaluates trust continuously. The candidate should mention Duo's Device Trust certificates and how they integrate with Cisco's ISE for on-prem posture.

Expect these follow-ups

• How does ZTNA handle legacy applications that cannot be fronted by a proxy?
• What happens if a user's device fails a posture check mid-session?

As asked

Cloudflare announces the same IP prefixes from every one of its data centers using Anycast. Explain how that routing works at the BGP level, why it makes Cloudflare resilient to DDoS attacks, and what the failure mode is when a PoP goes down.

Sample answer outline

A strong answer explains that Anycast assigns the same IP to multiple BGP origins and the internet's routing protocol naturally directs each client to the topologically nearest PoP. Under a volumetric DDoS, attack traffic is spread across all PoPs rather than hitting a single target, diluting the attack. When a PoP goes down, BGP withdraws that prefix announcement and routers reconverge to the next nearest PoP within seconds to minutes. Candidates should mention that BGP convergence time is the main SLA risk and that Cloudflare's Magic Transit product applies the same model for customer IP ranges.

Expect these follow-ups

• What is the difference between Anycast and GeoDNS for traffic steering?
• How does Cloudflare handle the case where two PoPs are equidistant from a client?

As asked

When a user visits a Cloudflare-proxied site, TLS terminates at the Cloudflare edge, not at the origin server. What are the security and privacy implications of this for the website owner and for the end user, and how does Cloudflare address those concerns architecturally?

Sample answer outline

The candidate should acknowledge that Cloudflare can see plaintext traffic between the edge and the browser, which requires a trust relationship. For the origin, the connection from Cloudflare to origin can use origin certificates or full strict TLS. Privacy-sensitive answers mention Cloudflare's Keyless SSL (private key stays on origin) and Geo Key Manager (keys stored only in specific jurisdictions). Strong candidates also mention that this model is what enables Cloudflare to inspect, cache, and transform traffic, which is the source of the product's value.

Expect these follow-ups

• What is Keyless SSL and what problem does it solve for highly regulated customers?
• How does TLS 1.3 change the handshake latency story at the edge?

As asked

Cloudflare R2 is object storage compatible with the S3 API but charges zero egress fees. Explain how Cloudflare makes R2 financially viable at zero egress, what the architectural implications are for how data is served, and what the trade-offs are compared to S3.

Sample answer outline

R2 is financially viable because data served from R2 exits through Cloudflare's network, which Cloudflare owns and pays for at peering costs much lower than AWS's rack-rate egress fees. Cloudflare can afford to not charge egress because it drives lock-in and broader platform adoption. Architecturally, R2 serves objects from the same network as Workers, so internal data access has low latency. Trade-offs include smaller ecosystem maturity vs S3, no transfer acceleration, fewer storage classes, and limited region controls (data jurisdiction is a newer feature). S3 wins on global feature set and third-party integrations.

Expect these follow-ups

• How would you use R2 with Workers to serve a media-heavy application efficiently?
• What consistency model does R2 provide for concurrent writes to the same key?

As asked

Cloudflare Access is a zero-trust network access product that replaces traditional VPNs. Explain the architectural differences between a traditional VPN approach and Cloudflare Access, and describe the authentication flow a user goes through to reach an internal application.

Sample answer outline

Traditional VPNs grant network-level access once authenticated, meaning a compromised client can reach anything on the internal network. Cloudflare Access enforces per-application policies: each internal app is exposed via a Cloudflare Tunnel (formerly Argo Tunnel) that establishes an outbound connection to Cloudflare, so no inbound firewall ports are opened. Users authenticate via an identity provider (Okta, Google, Azure AD) and Access issues a signed JWT. The Worker on the Cloudflare edge validates the JWT on every request before proxying to the origin. This is the zero trust model: verify identity and device posture per request, not per network join.

Expect these follow-ups

• How does Cloudflare Tunnel work technically, and why is an outbound-only connection model more secure than opening firewall ports?
• How would you enforce device posture (e.g., endpoint is running a certain MDM) as part of an Access policy?

As asked

Databricks ships an execution engine called Photon alongside the standard Spark JVM execution engine. Explain what Photon is, why it exists, and in what workloads it gives the most benefit.

Sample answer outline

Photon is a C++ vectorized query execution engine that Databricks built to exploit CPU SIMD instructions and avoid JVM overhead for analytical SQL queries. It is most beneficial on scan-heavy, aggregation-heavy SQL workloads on Parquet or Delta files because columnar vectorization saturates modern CPUs far better than the row-at-a-time JVM Spark code path. It is less helpful for jobs dominated by Python UDFs or complex graph processing. Candidates should note that Photon integrates transparently through the Catalyst planner and falls back to Spark JVM for unsupported operations.

Expect these follow-ups

• In which situations would you expect Photon to give no improvement or even a slowdown?
• How does Photon relate to Apache Arrow and the broader move toward columnar formats in the data ecosystem?

As asked

A customer wants a single governance layer across multiple Databricks workspaces in different cloud regions. They also want to know which notebooks and jobs read from or write to a given Delta table. How does Unity Catalog address both of those requirements?

Sample answer outline

Unity Catalog provides a three-level namespace (catalog, schema, table) that is shared across all workspaces attached to the same metastore. Permissions are granted once at the catalog or schema level and apply everywhere. For lineage, Unity Catalog captures column-level and table-level lineage automatically from Spark, SQL, and Delta operations, storing which queries, notebooks, jobs, and users read or wrote each table. This lineage is queryable through the Unity Catalog UI and API without any manual tagging by engineers.

Expect these follow-ups

• How would you migrate a workspace from the legacy Hive metastore to Unity Catalog without breaking existing jobs?
• What access control model does Unity Catalog use, and how does it differ from table ACLs in the legacy metastore?

As asked

A data engineering team asks you whether to use Delta Live Tables or write their own Spark Structured Streaming pipeline. What are the concrete tradeoffs and when would each be the right choice?

Sample answer outline

DLT manages pipeline infrastructure, automatic checkpointing, error recovery, data quality expectations, and lineage automatically, reducing operational burden. It is best for teams that want declarative pipelines with enforced SLAs and quality rules and do not need fine-grained control over streaming state management. Hand-written Structured Streaming is better when you need custom state operations (flatMapGroupsWithState), non-Delta sinks, or very specific checkpoint and trigger control. DLT has higher abstraction cost: debugging is harder and the programming model is more constrained.

Expect these follow-ups

• How do DLT expectations work, and what happens to records that fail them?
• Can you mix DLT tables and regular Delta tables in the same Databricks workflow?

As asked

A Databricks SQL query goes from your SQL string to actual tasks running on executors. Walk me through the stages of the Catalyst optimizer: what transformations happen at each stage and what decisions are made along the way.

Sample answer outline

Catalyst works in four stages. Parsing turns the SQL string into an unresolved logical plan. Analysis resolves column names and types against the catalog, producing a resolved logical plan. Logical optimization applies rule-based rewrites: predicate pushdown, constant folding, column pruning. Physical planning converts the logical plan to one or more physical plans using strategies (e.g., choosing broadcast vs. sort-merge join based on table sizes), then the cost model selects the best. Code generation (Janino/WholeStageCodeGen) compiles the physical plan to JVM bytecode or hands it to Photon.

Expect these follow-ups

• Where in this pipeline does Adaptive Query Execution intervene, and what statistics does it use?
• How would you add a custom Catalyst rule to, for example, rewrite all calls to a deprecated UDF to a new one?