Data platform engineer interview questions

As asked

You own a daily ETL job that loads orders from an upstream API into the warehouse. Yesterday's job failed halfway through and will be retried. How do you make the pipeline idempotent?

Sample answer outline

A strong answer defines the target invariant first: re-running the job for the same logical window should produce the same final table state, not duplicate rows or partial totals. Use a stable business key such as order_id, write into a staging table, validate counts and required columns, then merge into the target inside a transaction or partition swap. Track high-water marks separately from successful commits so a failed run does not advance the cursor. Candidates often trip up by relying only on append-only loads or by deleting a partition before they know the replacement data is complete.

Expect these follow-ups

• How would your answer change if the source sends updates to old orders?
• What metadata would you store for each pipeline run?
• How do you recover if the warehouse supports no multi-statement transaction?

As asked

Your company has 40 data teams writing Airflow or Dagster jobs in different styles. How would you build a shared orchestration platform without becoming a ticket queue?

Sample answer outline

The platform should provide paved roads: project templates, deployment automation, secrets access, standard retries, alerting, and lineage out of the box. Teams should own their DAG code while the platform owns runtime reliability, conventions, and guardrails. Multi-tenancy needs isolation for dependencies, quotas, credentials, and noisy workloads. Adoption is won by making the easiest path the best path, not by banning all custom jobs on day one. Candidates often miss the product-management side of platform work: documentation, migration support, and measuring whether teams are actually faster.

Expect these follow-ups

• What should be centralised and what should remain team-owned?
• How do you prevent one team's job from starving everyone else?
• What platform metrics would you report after six months?

As asked

Users update their profile and immediately see old data because reads go to replicas. How would you fix this?

Sample answer outline

First confirm the symptom is replication lag by measuring primary commit time, replica replay lag, and the request path. Common fixes include read-your-writes routing to the primary for a short window after a write, session stickiness, or using a version token that waits until a replica has caught up. Do not route all reads permanently to the primary unless the capacity tradeoff is acceptable. Add lag-aware load balancing and alerts so unhealthy replicas stop serving freshness-sensitive reads. Strong answers define which reads require consistency and which can tolerate stale data.

Expect these follow-ups

• How would you expose read consistency requirements in an application API?
• What happens if replica lag is caused by one expensive analytical query?
• How does this differ in eventually consistent NoSQL stores?

As asked

You need to backfill five years of transaction data into a new warehouse model while the daily pipeline keeps running. Walk me through the plan.

Sample answer outline

Separate the historical backfill from the incremental daily path, but make them use the same transformation logic where possible. Chunk by a natural partition such as transaction_date, run in bounded parallelism, and record per-partition status so failed chunks can be retried. Keep the target isolated until validation passes, then cut over views or downstream consumers. Compare row counts, sums, null rates, and sampled record-level values against the old source of truth. The common mistake is treating the backfill as a one-off script with no resume, validation, or cutover plan.

Expect these follow-ups

• How do you handle late-arriving records during the backfill window?
• What validation would convince finance to trust the new model?
• How do you avoid overwhelming the source database?

As asked

You are standardising a data lake used by batch analytics, streaming ingestion, and ML training. How would you choose between Iceberg, Delta, and plain Parquet?

Sample answer outline

Plain Parquet is a file format, not a full table-management layer, so it leaves you to solve schema evolution, atomic commits, deletes, and snapshot isolation yourself. Iceberg and Delta add table metadata, partition evolution, time travel, and safer concurrent writes. The right choice depends on engine support, governance needs, cloud constraints, and whether the organisation can operate the catalogue cleanly. Discuss migration path and interoperability instead of treating the choice as a benchmark contest. A senior answer also covers compaction, small files, retention, and rollback.

Expect these follow-ups

• How do small files hurt query and training performance?
• What does partition evolution buy you?
• How would you migrate existing Parquet data without breaking consumers?

As asked

Product engineering teams keep changing event payloads and breaking analytics and ML pipelines. How would you introduce data contracts?

Sample answer outline

Start with the highest-value event streams and define owners, schemas, compatibility rules, and review paths. Put validation close to the producer so bad events are caught before they enter the lake, while still storing rejected samples for debugging. Use schema registry or contract tests in CI, and make breaking changes explicit with versioning and deprecation windows. The cultural part matters: producers need to understand which downstream decisions depend on their data. Candidates trip up when they propose contracts as a data-team document rather than an enforceable engineering interface.

Expect these follow-ups

• How do you handle a producer that needs to ship a breaking change urgently?
• Where should validation run: client, service, stream, or warehouse?
• How do you stop contracts becoming stale?