Data Quality, Testing & Observability
Every data engineering interview loop eventually asks some version of "how do you know your data is correct?" — and it is one of the fastest ways to separate candidates who have operated a pipeline in production from candidates who have only built one. A candidate who has never been paged for a broken dashboard answers with a list of tools: "we'd use dbt tests, and maybe Great Expectations." A candidate who has actually owned a pipeline answers with a system: checks placed at specific layers for specific reasons, a monitoring strategy that catches the failures schema checks structurally cannot, and a process for what happens in the fifteen minutes after someone says "the revenue number looks wrong" — because in production, "the data was wrong" is never the end of the story. It's the beginning of a triage.
The distinction interviewers are probing for is this: data quality is not a single check you either have or don't have, it's a set of properties (accuracy, completeness, timeliness, consistency, uniqueness) that fail independently, for different reasons, at different points in a pipeline, and are caught by different tools. A NOT NULL constraint catches a completeness failure — it says nothing about whether a populated value is correct. A schema check catches a producer renaming a column — it says nothing about a producer silently sending zero rows because an upstream job failed, since the schema is still perfectly valid, just empty. This is the gap that separates strong answers from weak ones: strong answers name the specific failure mode, the specific layer that would have caught it, and the reason a different-looking check would have missed it entirely.
This subject builds that system layer by layer: what quality actually means (dimensions), where to enforce it (source, pipeline, warehouse), how to keep producers and consumers from silently drifting apart (schema contracts), how to catch failures that pass every schema check (freshness, volume, and distribution monitoring), which tool fits which layer (dbt tests vs. dedicated observability platforms), and what to do when something still breaks in production. It builds on the pipeline design in etl-vs-elt-and-pipeline-design, the orchestration layer in airflow-and-workflow-orchestration, and the transformation layer in dbt-and-analytics-engineering — quality is not a bolt-on to those systems, it has to be designed into each of them.