There is a particular kind of data incident that does real damage. Not the pipeline that crashes at 3am and pages someone — that one is annoying and gets fixed by breakfast. The dangerous one is the pipeline that keeps running, keeps reporting success, and quietly stops loading yesterday's records.
Nobody notices for three weeks. In the meantime, every dashboard built on that table is subtly wrong, and every decision made from those dashboards was made on incomplete information. By the time someone spots it, you cannot easily identify which decisions were affected.
An outage is an inconvenience. A silent failure is a correctness problem wearing an availability costume.
Design for loud failure
The goal is not pipelines that never break. Everything breaks eventually — an API changes, a schema drifts, a source system has an outage. The goal is that when something breaks, you find out in minutes from a system rather than in weeks from a confused stakeholder.
That means three things, none of them exotic: tests that assert what correct looks like, freshness checks that alert when data stops arriving, and volume anomaly detection that catches the load that succeeded but moved a tenth of the usual rows.
Tests are not optional infrastructure
Software engineers stopped arguing about whether to write tests roughly two decades ago. Data engineering is still catching up, largely because the failure modes are less immediately visible.
The baseline is unglamorous: uniqueness on primary keys, referential integrity across joins, not-null on columns that should never be null, accepted-value checks on enumerations, and range assertions on numeric fields. Then the business-rule tests that are specific to you — order totals reconciling to line items, no future-dated transactions, active accounts having an owner.
These add seconds to a pipeline run. They catch the class of error that otherwise reaches a dashboard and gets acted upon.
Idempotency is a correctness property
A pipeline should produce the same result whether it runs once or five times. This sounds obvious and is routinely violated, usually by append-only loads that duplicate rows on retry.
The consequence shows up during incident recovery, which is exactly the worst moment. Something failed, someone re-runs the job to fix it, and now the numbers are inflated in a way that is tedious to unwind. Idempotent loads — merge on a key rather than blind append — remove an entire category of 2am judgement calls.
Lineage is what makes debugging finite
When a figure in a dashboard looks wrong, the question is where it came from. Without lineage, answering that means reading transformation code and following joins by hand, which is slow and unreliable.
Column-level lineage turns a multi-hour investigation into a few clicks. It also makes impact analysis possible in the other direction — before changing an upstream schema, you can see exactly what downstream will break.
Streaming when it earns it
The default 2026 stack is fairly settled: Snowflake or Databricks for storage, dbt for transformation, Airflow or Dagster for orchestration, Fivetran or dlt for ingestion, and Great Expectations or Monte Carlo for quality. Streaming sits alongside this rather than replacing it.
Before adding Kafka to an architecture, it is worth asking a blunt question: what decision changes if this data is fifteen minutes old rather than fifteen seconds? For fraud detection and operational alerting, plenty. For most reporting and analytics, nothing at all.
Streaming carries real cost — operational complexity, harder debugging, more expensive failure modes. It is the right choice when latency genuinely drives a decision, and an expensive affectation when it does not.
The unglamorous conclusion
Good data engineering is mostly boring on purpose. Tested transformations, idempotent loads, freshness monitoring, documented lineage. None of it makes an interesting demo.
What it produces is a data platform people stop arguing with — where the response to an unexpected number is to investigate it rather than to distrust the whole system.