Forward-deployed engineer role-specific interview questions

As asked

A customer calls saying their integration stopped returning data an hour ago. No errors in their logs. Walk me through exactly how you diagnose this, starting from zero context.

Sample answer outline

Strong answers work outward from the symptom: check the customer's request shape, then the API gateway logs, then the upstream service logs, correlating by trace ID or timestamp. Candidates should mention reproducing the exact request locally or via curl, verifying auth tokens haven't expired, and checking whether a recent deploy or config change correlates with the failure window.

Expect these follow-ups

• How would your approach differ if the customer is in a timezone 8 hours ahead and you can't get them on a call?
• What do you add to the API response to make this kind of debugging easier next time?

As asked

You are building the outbound webhook system for a B2B product. A customer's endpoint goes down for 45 minutes. What does your retry strategy look like, and how do you prevent thundering herd when their server comes back up?

Sample answer outline

A strong answer covers exponential backoff with jitter, a max retry window (commonly 24 to 72 hours), dead-letter queuing for events that exhaust retries, and a status endpoint or notification so the customer knows what they missed. Thundering herd prevention means staggering retries across customers, not replaying all backlog events at once when the endpoint recovers.

Expect these follow-ups

• How do you handle idempotency so replays don't cause double-processing on the customer side?
• What do you tell the customer if 3 hours of events are lost because their endpoint was down longer than your retry window?

As asked

You are writing a data pipeline that polls a third-party API using OAuth2. Access tokens expire in 1 hour. How do you handle token refresh without breaking the pipeline or racing across multiple workers?

Sample answer outline

Candidates should describe proactive refresh (refresh before expiry, not on 401) using the expires_in field, storing the refresh token securely (not in plain env vars), and using a distributed lock or a single token-manager process to avoid multiple workers simultaneously refreshing. Strong answers also handle the case where the refresh token itself has expired, triggering a re-auth flow with alerting.

Expect these follow-ups

• How do you store the refresh token if this pipeline runs in a Kubernetes job with ephemeral containers?

As asked

A customer emails you: 'Your API works in Postman but fails in our browser app with a CORS error.' What are the specific steps you take to diagnose and fix this?

Sample answer outline

The candidate should distinguish simple vs. preflighted requests (POST with JSON triggers preflight). Diagnosis: inspect the browser DevTools Network tab for the OPTIONS preflight and check the response headers. The fix is ensuring Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers match what the customer's frontend sends. Wildcards don't work with credentials. Strong candidates also mention that Postman bypasses CORS because it is not a browser.

Expect these follow-ups

• How do you set this up so internal dev, staging, and production all behave correctly without hard-coding origins?

As asked

Your integration scripts need to store third-party API keys, database passwords, and OAuth client secrets. What is the right way to manage these across local dev, staging, and production environments?

Sample answer outline

Credentials belong in a secrets manager (AWS Secrets Manager, HashiCorp Vault, or at minimum encrypted environment variables injected at runtime) not in source code, .env files committed to git, or CI logs. For local dev, a .env.local file that is git-ignored is acceptable if it is never the source of truth for production values. Rotation should be possible without redeployment. Strong candidates mention least-privilege access per environment and auditing who accessed which secret.

Expect these follow-ups

• How do you detect if someone accidentally commits a secret to the repo?

As asked

A customer triggers a report generation that takes 3 to 10 minutes. The HTTP request timeout is 30 seconds. How do you architect this so the customer can track progress and retrieve the result?

Sample answer outline

Accept the request synchronously and immediately return a job ID with a 202 Accepted. The actual work runs in a background worker (queue-based or a separate process). Expose a GET /jobs/:id endpoint that returns the current status (queued, processing, done, failed) and the result URL or error detail when finished. Optionally send a webhook when done. Strong candidates mention job TTL, storing partial progress, and handling the case where the worker crashes mid-job.

Expect these follow-ups

• What is the tradeoff between polling versus webhooks for the customer, and when would you offer both?

As asked

You are building a new integration with a customer's ERP system. What logging and tracing do you add from day one, and why is structured logging specifically important in this context?

Sample answer outline

Structured logging (JSON with consistent fields) lets you filter and aggregate across millions of log lines in a log aggregator (Datadog, CloudWatch Insights, etc.) by customer ID, operation type, or status without parsing unstructured text. For an ERP integration, log the customer_id, operation (create_order, sync_product), duration, external request/response status codes, and any error details. Trace IDs correlate a single sync operation across multiple service calls. Strong candidates mention not logging PII or raw credentials.

Expect these follow-ups

• A customer reports a sync failure that happened 3 days ago. How good does your logging need to be to answer their question?

As asked

It is 2 PM on a Thursday. A customer's sales team can't access your platform because the Salesforce SSO integration is returning 500 errors. The Salesforce admin is out of office. What do you do?

Sample answer outline

Immediately check if this is isolated to this customer or all Salesforce users. If isolated, look at recent changes to the customer's Salesforce connected app or your OAuth config. Check if Salesforce's own status page shows anything. Contact the customer's secondary technical contact if the admin is OOO. As a workaround, enable an email/password login bypass if available so users can still access the platform. Set an escalation timer: if unresolved in 1 hour, bring in the manager.

Expect these follow-ups

• What do you tell the customer's VP of Sales who emails asking why their team has been locked out for 30 minutes?

As asked

A customer uploads a 2 GB CSV with 8 million rows. Your Node.js endpoint receives it. How do you process it without running out of memory, and how do you report progress back to the customer?

Sample answer outline

The key insight is streaming with a CSV parser like csv-parse in stream mode, piping directly from the request body, and processing in batches (e.g., 500 rows) before writing to the database. Progress can be tracked by counting processed rows and updating a job record in the DB, which a polling or SSE endpoint exposes to the customer. Candidates should also mention input validation per row and what to do with invalid rows (skip with a report, halt, etc.).

Expect these follow-ups

• What happens if the database write for batch 4000 fails mid-import? How do you roll back or resume?
• How does your approach change if you need to import this from S3 instead of a direct upload?

As asked

Your integration calls a partner API that has a limit of 100 requests per minute per customer. You have 200 customers, some of whom burst during business hours. How do you architect the request layer to stay within limits for every customer?

Sample answer outline

Strong candidates separate per-customer quota tracking (using Redis token bucket or sliding window) from a shared worker pool. They queue requests per customer, consume the queue at a rate capped by the token bucket, and surface 429 errors to the caller with a Retry-After header rather than silently dropping. They also mention exponential backoff on the partner API side and alerting when a customer consistently approaches their limit.

Expect these follow-ups

• How does this design change if the partner's rate limit resets on the hour boundary (not a rolling window)?

As asked

You are receiving JSON payloads from 12 different customers, each of whom sends slightly different field names and date formats for the same concept. How do you build an ingestion layer that normalizes this without maintaining 12 separate code paths?

Sample answer outline

Good answers describe a per-customer mapping config (stored in DB or JSON files) that declares field aliases and transformation rules, applied by a generic normalization engine. JSON Schema validates the incoming payload after mapping. Candidates should mention how they handle new customers (onboarding config), how they surface mapping errors back to the customer, and version control for the configs so rollback is possible.

Expect these follow-ups

• How do you test a new customer's mapping before they go live in production?

As asked

A security review requires rotating all customer API keys within 48 hours. Customers are actively using these keys to call your APIs. How do you execute the rotation without breaking any live integrations?

Sample answer outline

Zero-downtime rotation requires accepting both the old and new key for an overlap window. Steps: generate new key, store both in the validation layer, notify the customer (email plus in-product banner with deadline), give a 7 to 14 day overlap, then revoke the old key after confirming the new one is in active use (detectable via access logs). The overlap period and forced expiry date are the key design decisions to discuss.

Expect these follow-ups

• What if a customer never acknowledges the rotation and their old key is revoked, breaking their production integration?
• How do you audit that every customer completed the rotation?

As asked

A new enterprise customer wants all their data from the past 6 months migrated into your platform before going live. The source is their existing SaaS tool's API, which returns at most 100 records per page and rate limits at 60 requests per minute. How do you plan and execute this backfill?

Sample answer outline

Calculate the total pages needed and the time required at the rate limit ceiling. Build a resumable backfill script that tracks a cursor or page offset in a checkpoint file or DB, so a crash doesn't restart from zero. Respect the rate limit with a token bucket or simple sleep. Run during off-peak hours if the source allows. Validate record counts after each chunk. Strong answers also discuss communicating progress to the customer and doing a dry run with a small date range first.

Expect these follow-ups

• What do you do if the source API doesn't support date-range filtering and can only paginate from the beginning?

As asked

You need to write a one-time script that migrates 2 million customer records to a new schema. It will run in production. What properties does your script need to have, and what makes a migration script 'production-safe'?

Sample answer outline

Production-safe means: idempotent (safe to run twice without corruption), batched (never one giant transaction that locks the table), resumable via a last-processed cursor, dry-run mode that prints what would change without writing, and progress logging with an ETA. The script should also have a killswitch (e.g., check a feature flag or file before each batch) and a rollback plan (either a reverse migration or a backup taken before the run).

Expect these follow-ups

• How do you verify the migration was correct after it runs on all 2 million rows?

As asked

You need to change the shape of a response that 30 customers currently depend on. How do you evolve the API without breaking existing integrations, and what does your versioning strategy look like?

Sample answer outline

Options are URL versioning (/v2/...), header versioning (Accept: application/vnd.api+json; version=2), or additive evolution (new fields alongside old). For a breaking change, URL versioning is most explicit and easiest for customers to understand. Key steps: document the old behavior, give a deprecation timeline (typically 6 to 12 months for B2B), provide a migration guide, track usage of v1 per customer so you know who to contact, and set an end-of-life date. Additive changes should never require a version bump.

Expect these follow-ups

• How do you maintain two live versions of the same endpoint in a single codebase without code duplication?

As asked

A sales engineer promises a prospect a working demo of a new integration in 2 business days. You have never used the third-party API before. How do you approach this, and what corners do you cut knowingly?

Sample answer outline

Read the third-party docs and auth model first (30 min). Spike the auth and one happy-path call before building anything else. Cut: error handling beyond the happy path, tests beyond a smoke test, logging infrastructure, production secrets management (use a test key). Document what is hardcoded or skipped in a README so the handoff to production is explicit. Communicate scope to the sales engineer: what it does vs. what a production version would need.

Expect these follow-ups

• The demo goes well and the prospect signs. Now the sales engineer says 'ship this to production next week.' What do you say?

As asked

Your service had a 2-hour outage that breached a customer's SLA. You are the engineer who fixed it. The customer's technical lead wants a root cause analysis. What do you write and how do you present it?

Sample answer outline

A customer-facing RCA has: a timeline (when the issue started, when it was detected, when it was resolved), root cause in plain language (not internal jargon), immediate fixes already applied, and follow-up action items with owners and dates. Avoid blame language. Strong candidates mention acknowledging the SLA breach explicitly, any credit owed, and scheduling a call if the customer wants to discuss further. The tone should be accountable and clear, not defensive.

Expect these follow-ups

• How do you decide what to include vs. omit when the root cause is an embarrassing human mistake?

As asked

You are about to deploy a change to how your platform connects to Salesforce. It affects all customers using the Salesforce integration. How do you use feature flags to roll it out safely?

Sample answer outline

Gate the new code path behind a flag at the customer/org level, not globally. Enable it first for internal test accounts, then for a small cohort of consenting customers who can tolerate issues, then progressively broader. Monitor errors and Salesforce sync failure rates per flag state. Have a kill switch that instantly reverts all customers to the old path. Strong candidates also discuss how to clean up the flag (and the old code) after a successful full rollout.

Expect these follow-ups

• How do you manage feature flags at scale when you have 50 active flags across 3 integrations?

As asked

You are consuming events from a customer's Kafka topic to sync their order data into your system. Messages can arrive out of order and the topic may replay events on consumer restart. How do you design the consumer to handle this correctly?

Sample answer outline

Handle out-of-order by using a last-write-wins strategy keyed on (entity_id, event_sequence_number) or event timestamp, so an older replayed event does not overwrite a newer state. Handle replays with idempotent writes (upsert on the entity's natural key). Commit offsets only after a successful write. Dead-letter queue for messages that fail after retries. Strong candidates also mention consumer group management, partition assignment, and what happens if processing is slower than message arrival rate.

Expect these follow-ups

• How do you test your consumer's behavior under out-of-order and replayed messages before going live with the customer?

As asked

Your GraphQL API has a field resolvers pattern. A query that asks for 20 users with their orders triggers 21 database queries. How does this happen structurally in GraphQL, and what is the standard fix?

Sample answer outline

GraphQL resolves fields independently, so the orders resolver runs once per user, each making its own DB call. The fix is DataLoader: batch all order lookups for the 20 user IDs into one DB query within a single event loop tick, then distribute results back to each resolver via the cache. Strong candidates explain the batching and per-request cache semantics of DataLoader and why the cache must be per-request to avoid cross-tenant data leaks.

Expect these follow-ups

• If the DataLoader cache must be per-request, how do you ensure it is not accidentally shared across concurrent requests in a Node.js server?

As asked

Your endpoint creates an invoice and triggers an external payment. A network timeout means the customer's client retries. How do you use idempotency keys to prevent double-charging?

Sample answer outline

The client sends an Idempotency-Key header (a UUID they generate). Your server stores the key alongside the result of the first successful execution. On a retry with the same key, return the stored result without re-executing. The idempotency key record should be stored transactionally with the invoice creation so you never store a key for a partially-created resource. Expire keys after a reasonable window (24 hours). Strong candidates note the difference between idempotency (same result) and at-most-once (no retry at all).

Expect these follow-ups

• What do you return if the same idempotency key is used with a different request body?

As asked

On a call with a key customer, their lead developer says they need your API to support bulk record deletion in a single request. Your API only supports deleting one record at a time. They say this is a dealbreaker for their integration. What do you do?

Sample answer outline

First, understand the underlying need: are they deleting a few records or millions? The answer changes the solution. Short-term: offer a client-side script that loops with parallel requests up to the API rate limit (which often solves the performance concern). Medium-term: take this to the product team as a validated request with a specific customer attached. Don't promise a delivery date you can't guarantee. Be honest that you can't commit to a roadmap item on the spot.

Expect these follow-ups

• How do you handle it if the product team says bulk deletion is not on the roadmap for the next 12 months?

As asked

You run a data migration script in production. 30 seconds in, you notice the 'rows deleted' counter is climbing much faster than expected. You expected 5,000 rows; it has already deleted 50,000. What do you do right now?

Sample answer outline

Kill the script immediately (Ctrl-C or kill the process) before deleting more. Check if the script is running inside a transaction (if yes, you can rollback; if no, the deletes are already committed). Assess what was actually deleted from the database logs or a quick count. Alert your team immediately. If in a transaction, rollback. If not, check if you have a pre-run backup and initiate the restore process. Then understand what happened: was there a missing WHERE clause, wrong DB pointed to, or a logic error in the filter?

Expect these follow-ups

• What process changes do you put in place so this can never happen again?

As asked

Your ingestion pipeline crashes every night at midnight because one customer is sending a malformed record in their nightly export. The pipeline processes all customers, so it is blocking everyone else. What do you do immediately and longer term?

Sample answer outline

Immediate fix: isolate this customer's records into their own processing path or add a guard that skips and logs the malformed record rather than crashing. This unblocks all other customers. Then: notify the customer of the specific malformed field with an example, validate all inputs at ingestion rather than failing mid-pipeline, and add a test for this class of bad data to prevent regression.

Expect these follow-ups

• The customer says the malformed data is 'correct on their side' and refuses to fix it. What do you do?

As asked

A sales rep told a customer that a prototype integration you built in 2 days is 'production ready' with a 99.9% uptime SLA. The customer is signing a contract based on this. You know the prototype has no error handling, no tests, and uses hardcoded credentials. What do you do?

Sample answer outline

Escalate immediately to your manager and the sales rep before the contract is signed. Be specific about what 'not production ready' means: list the gaps (no retry, no monitoring, hardcoded creds, no tests). Propose a realistic timeline to make it production-grade. The sales team needs to either re-set the customer's expectation or delay the contract date. Never sign off on production-level SLAs you cannot technically back. Document your concerns in writing.

Expect these follow-ups

• Your manager says 'just ship it and we'll fix it after the deal closes.' How do you respond?