Forward-deployed engineer system design interview questions

As asked

A customer needs your product connected to three internal systems: one REST API, one nightly CSV export, and one database replica they will not let you query directly. How would you design the integration?

Sample answer outline

The answer should normalise around contracts, ownership, and failure recovery rather than just moving data. For the REST API, define pagination, rate limits, retries, and idempotency. For CSV, use a landing bucket, schema validation, checksums, and clear handling for late or malformed files. For the restricted database, negotiate a supported export, change-data-capture feed, or customer-hosted connector rather than unsafe direct access. Strong candidates discuss audit logs, secrets handling, backfills, reconciliation, and how to make the integration observable for both teams.

Expect these follow-ups

• How do you handle a breaking schema change from the customer?
• What would you put in the integration runbook?
• Where do you draw the line between product work and customer-specific glue?

As asked

Design a system that keeps contact records in sync between your platform's Postgres database and a customer's Salesforce org. Changes can happen on either side, and the two systems use different schemas.

Sample answer outline

Core components: a bidirectional event stream (platform emits change events via DB triggers or CDC, Salesforce emits via outbound messaging or Change Data Capture), a sync worker that consumes both streams, a mapping layer that translates schemas, and a conflict resolution policy (e.g., last-write-wins by modified_at timestamp). Store an external_id on both sides to maintain the link. Discuss handling Salesforce's governor limits, deduplication of echoed events, and the initial bulk sync bootstrapping.

Expect these follow-ups

• How do you prevent an infinite echo loop where a change on side A syncs to side B and then side B sends the same change back to side A?
• How do you handle schema differences when a Salesforce admin adds a new custom field that your system doesn't know about?

As asked

You need to build an API gateway layer in front of your microservices that enforces per-customer rate limits, authenticates API keys, and logs every request for audit purposes. Design this component.

Sample answer outline

The gateway validates the API key on every request (in-memory cache of key-to-customer mapping, refreshed from DB every few minutes). Rate limiting uses Redis with a sliding window counter per customer. Audit logging writes to an append-only store (Kafka topic or S3 via Kinesis Firehose) asynchronously so it does not add latency to the request. The gateway is stateless and horizontally scalable. Discuss key revocation propagation (cache invalidation), what to log (method, path, status, customer, latency, but not request body unless you have a reason), and how to handle the gateway itself failing.

Expect these follow-ups

• How do you ensure audit logs cannot be tampered with if a customer disputes an API call happened?

As asked

A customer wants to stream their event data (10,000 events/sec peak) into your analytics platform in near real time. Design the ingestion pipeline from their client SDKs through to the queryable data store.

Sample answer outline

Client SDKs batch and send events over HTTPS to an ingest API. The ingest API acknowledges quickly and pushes to a Kafka topic (partitioned by customer_id for ordering). A stream processor (Flink or a simpler consumer) validates, enriches, and writes to the analytical store (ClickHouse or BigQuery for columnar aggregations). Discuss: schema evolution, dead-letter queues for malformed events, back-pressure handling, customer data isolation in the store, and how you test the pipeline under peak load.

Expect these follow-ups

• A customer sends 100x their expected volume (a bug in their SDK). How does your system protect itself without dropping legitimate traffic from other customers?

As asked

You need to publish an SDK in Python and JavaScript that wraps your REST API. It will be used by customer developers to build integrations. What principles guide the design, and what does the SDK do beyond wrapping HTTP calls?

Sample answer outline

Beyond HTTP wrapping: automatic retry with backoff, pagination handled transparently (return an iterator), type hints and IntelliSense support via generated types from the OpenAPI spec, auth handled in one place (the client constructor), meaningful error types (not just status codes), and a test mode that records and replays HTTP interactions. SDK versioning should match the API version it targets. Strong candidates mention keeping the SDK thin (business logic belongs in the API, not the SDK) and generating the SDK from the OpenAPI spec to avoid drift.

Expect these follow-ups

• A customer using your Python SDK reports that it blocks the event loop in their async application. How do you fix that without breaking synchronous users?

As asked

Each customer needs to configure scheduled jobs that extract data from a source, transform it, and load it into your platform on their own cron schedule. Some customers run 50+ jobs on different schedules. Design this system.

Sample answer outline

A scheduler service stores job definitions (cron expression, customer_id, source config, transform config) in the DB. A tick process fires every minute, queries for jobs due to run, and enqueues each as an independent task. Workers pick up tasks and execute the ETL in isolation. Each job run is logged with start/end time, status, rows processed, and errors for the customer to see. Horizontal scaling requires that the tick process uses DB-level locking (SELECT FOR UPDATE SKIP LOCKED) to avoid double-firing. Discuss per-customer concurrency limits, job timeouts, and alerting on consecutive failures.

Expect these follow-ups

• How do you ensure a single stuck job (e.g., waiting on a slow source API) doesn't starve other customers' jobs from running?

As asked

You run dozens of integrations with third-party tools on behalf of customers. Design the observability stack so your team can quickly answer: which integration is failing, for which customer, how often, and why.

Sample answer outline

Structured logs with customer_id, integration_type, operation, duration, and error_code as indexed fields. Metrics: success rate, p95 latency, and error count per integration per customer, plus third-party API error rates (their outages vs. your bugs). Traces: a trace ID per sync operation that correlates all log entries. Dashboards: one per integration showing health over time, with per-customer drill-down. Alerting: customer-level alerts when error rate crosses a threshold, and integration-wide alerts when a third party is having an outage (many customers fail simultaneously).

Expect these follow-ups

• How do you distinguish between a third-party API outage and a bug you introduced?

As asked

Your product needs to offer an embeddable widget (like a chat bubble or analytics dashboard) that customers add to their own websites with a single script tag. Design the architecture from the widget loader through to the data layer.

Sample answer outline

The script tag loads a thin bootstrapper from your CDN that injects an iframe (for CSS isolation) or shadow DOM widget. Authentication is via a customer-specific public key passed as a data attribute, not a secret. Data flows from the widget to your API over HTTPS with CORS configured per customer origin. A versioned bundle URL ensures customers don't break on updates (opt-in upgrades). CDN caching with a long TTL on the versioned bundle; a short TTL on the 'latest' alias. Discuss CSP compliance and what happens if the customer's site blocks iframes.

Expect these follow-ups

• A customer's Content Security Policy is blocking your widget. How do you help them configure it to allow your widget without weakening their CSP globally?

As asked

Your platform integrates with 15 different third-party tools (Slack, Salesforce, HubSpot, etc.). Customers configure these integrations via a UI, not code. Design the data model and runtime that stores and executes these configurations.

Sample answer outline

A configuration is a per-customer, per-integration JSON document that describes: auth credentials (encrypted at rest), field mappings, trigger conditions, and action steps. A config validator runs when the customer saves to catch obvious errors. A runtime worker interprets and executes the config document by dispatching to integration-specific adapters. Versioning configs (store all versions) allows rollback. Test mode executes the config against a sandbox without real side effects. Discuss how to handle breaking changes when an integration's API changes.

Expect these follow-ups

• How do you debug a customer's failing integration rule if the failure only happens with their specific data shapes?

As asked

Customers must be able to export all of their data from your platform within 30 days (GDPR Article 20). Some customers have millions of records. Design the data export system.

Sample answer outline

Customer requests an export via API or UI. A job is enqueued and runs asynchronously. The job queries data in batches, writes to CSV or JSON Lines, streams to S3, and generates a pre-signed download URL with an expiry (e.g., 7 days). Notify the customer when ready. Data is scoped strictly to the requesting customer (verified by tenant_id). Audit logs record who requested and when. Strong candidates mention: including all data the customer owns (not just one table), excluding other tenants' data even if the schema allows it, and deleting the export from S3 after the download window.

Expect these follow-ups

• A customer submits 10 export requests in 10 minutes. How do you protect your system?

As asked

When a key event occurs in your platform, you need to notify each customer via their configured channels: email, Slack, webhook, and/or SMS. Some customers have thousands of sub-accounts that all need notifying. Design the fanout.

Sample answer outline

The event producer publishes to a central topic. A fanout service looks up the customer's notification preferences and creates one job per channel per customer. Each channel has its own worker queue so a slow email provider doesn't block webhook delivery. Per-customer rate limits prevent flooding a customer's Slack channel. Dead-letter queues and retry policies are per-channel. For customers with thousands of sub-accounts, the fanout itself is parallelized. Strong candidates discuss idempotent delivery and duplicate suppression.

Expect these follow-ups

• A Slack API outage lasts 3 hours. How do you handle the backlog and ensure no notifications are permanently lost?