TL;DR: Change data capture pipelines built on Debezium and Kafka Connect fail in a small number of predictable ways — replication slot pressure that forces an unplanned resnapshot, schema changes that break downstream consumers, and connector lag that goes unnoticed until it's a multi-hour backlog. None of these are exotic edge cases; they're the default outcome of running CDC without specific operational guardrails. This article covers the failure modes and the concrete configuration and monitoring choices that prevent them from becoming incidents.

Why CDC fails differently than a normal data pipeline

A batch ETL job that fails just... fails, and you rerun it. A CDC pipeline reading a database's write-ahead log (WAL) or binlog has a harder constraint: the source database is only willing to retain that log for a finite window, and if your connector falls behind long enough for the database to reclaim the log segments your connector still needs, there is no "just rerun it" — the connector's resume position no longer exists. At that point, recovery moves from "restart the connector" to "resnapshot the affected tables," which on a large table can mean hours of downstream staleness and real load on the source database exactly when you can least afford it.

This is the structural reason CDC needs more deliberate failure planning than most pipeline types: the cost of falling behind isn't linear, it's a cliff.

Failure mode 1: replication slot / log retention pressure

On PostgreSQL, Debezium's logical decoding relies on a replication slot, which tells Postgres "don't discard WAL segments this consumer hasn't read yet." If the Debezium connector stops consuming — crashes, gets stuck, or is simply undeployed without being properly dropped — the replication slot keeps accumulating retained WAL indefinitely, because Postgres has no way to know the consumer isn't coming back. Left unmonitored, this is a two-sided failure: either the WAL fills the disk and takes down the source database (the worse outcome), or someone notices and drops the slot, discarding the connector's resume position and forcing a resnapshot anyway.

What actually prevents this:

  • Monitor slot lag directly, not just connector health. Query pg_replication_slots for restart_lsn distance from current WAL position — this reveals slot pressure well before it becomes a disk-space emergency, and it's a signal Debezium's own connector-status endpoint won't surface on its own.
  • Alert on absolute WAL retention, not just a percentage, since a slowly-growing slot on a high-write-volume database can consume disk space fast enough that a daily check interval is too coarse.
  • Never leave an orphaned replication slot. If you're decommissioning a connector, drop its slot explicitly as part of that process — an unattended CDC pipeline is one of the more common causes of an unrelated, confusing "why is Postgres disk full" incident days or weeks later.
  • Size log retention with connector downtime in mind, not just steady-state throughput. If your deployment process involves connector restarts that take minutes, and your WAL retention window is sized for zero downtime, a routine deploy becomes a slot-pressure incident.

The equivalent on MySQL is binlog retention (expire_logs_days / binlog_expire_logs_seconds) and on MongoDB, oplog window — same underlying problem, same principle: the connector's resume position is only valid as long as the source's log retention hasn't rotated past it.

Failure mode 2: snapshot behavior under load

The initial snapshot — reading existing table contents before switching to streaming — is the highest-risk phase of a CDC pipeline's lifecycle, because it's a full-table read competing with production traffic, and because a poorly-sized snapshot can exhaust connector memory on large tables.

Incremental snapshots (Debezium's newer signaling-based mechanism) address the two biggest problems with the older blocking snapshot approach: they don't require stopping streaming to run, and they can be triggered ad hoc for a specific table by inserting a row into a signaling table — meaning you can add a new table to an existing pipeline, or re-snapshot one table after a data-quality issue, without re-snapshotting the entire database or restarting the connector. This matters operationally because the alternative — a full connector restart to pick up one new table — reintroduces the WAL-retention risk from failure mode 1 for the duration of the restart.

Practical snapshot guardrails:

  • Chunk large-table snapshots explicitly (incremental.snapshot.chunk.size in Debezium) rather than accepting the default, which may not be sized appropriately for your table's row width.
  • Run snapshots during lower-traffic windows where possible — the snapshot read still puts load on the source database even with Debezium's watermarking approach to avoid inconsistency.
  • Treat snapshot completion as a monitored event, not a fire-and-forget step — a snapshot that stalls partway through is a silent failure mode if nothing alerts on snapshot progress specifically.

Failure mode 3: schema changes breaking downstream consumers

A CDC pipeline is only as stable as the assumption that a column added, renamed, or retyped on the source table doesn't silently break every downstream consumer of the Kafka topic. Debezium propagates schema changes as part of its event stream, but propagation isn't the same as safe propagation — a downstream consumer with a rigid Avro reader schema can still fail hard on a change Debezium happily passed through.

The outbox pattern closes this gap at the source. Rather than capturing changes to arbitrary application tables (whose schemas evolve for reasons unrelated to the event contract), the outbox pattern has the application explicitly write structured events to a dedicated outbox table, serialized and validated against a schema registry before the database write succeeds. If the event doesn't conform to the registered schema, serialization fails and the transaction rolls back — meaning invalid events can never enter the outbox table in the first place, and Kafka stops being where schema problems get discovered and becomes a channel that only ever carries valid, contract-conforming events.

text
Application code
      │
      ├── writes business data (normal tables)
      └── writes event to outbox table
              (schema-validated via Avro + Schema Registry
               BEFORE the transaction commits)
                      │
                      ▼
         Debezium outbox event router
         (captures only the outbox table,
          transforms rows into clean Kafka events)
                      │
                      ▼
                Kafka topic
         (guaranteed schema-conformant —
          invalid events never reached the DB)

For CDC on tables you don't control the schema evolution process for (a legacy application table, a third-party system), the more defensive posture is tolerant reader design on the consumer side: treat new fields as optional/ignorable rather than failing on unrecognized fields, and use a schema registry compatibility mode (typically BACKWARD or FORWARD depending on who needs to evolve independently) that actually matches how your producers and consumers deploy relative to each other.

Failure mode 4: connector lag that goes unnoticed

The most operationally dangerous CDC failure isn't a crash — a crashed connector at least generates an alert. It's a connector that's technically running but steadily falling behind, because "still running" and "keeping up" are different health signals, and most default monitoring only checks the former.

What to track, specifically, beyond basic connector task state:

  • Source-side lag — how far behind the current WAL/binlog position the connector's read position is, measured in both time and log volume, not just "is the task RUNNING."
  • Kafka-side consumer lag on any downstream consumer of the CDC topics — a healthy connector writing into a topic nobody is reading fast enough is a different failure than the connector itself lagging.
  • Dead-letter queue volume and growth rate. Kafka Connect's DLQ is just another Kafka topic that failed records route to when error tolerance is configured to not halt the connector — a DLQ that's silently accumulating messages is a data-loss risk hiding behind an apparently-healthy connector, since by design the connector keeps running rather than stopping on a message it can't process.
  • Task rebalance frequency. Kafka Connect's distributed mode handles worker failure and task rebalancing automatically, which is good for availability but bad for visibility if nobody's watching — frequent rebalances usually indicate an underlying resource or connectivity problem worth investigating before it becomes an outage.

A reasonable default alerting posture: alert on source lag exceeding a threshold tied to your log retention window (with real margin — if WAL retention is 24 hours, alerting at 20 hours of lag is too late to react), alert on any DLQ growth rather than requiring it to cross a volume threshold, and alert on replication slot pressure independently of connector-reported health, since a connector can report itself healthy while the thing it depends on (log retention) is approaching a cliff.

A pre-production checklist

  • [ ] Replication slot / binlog retention sized with connector restart and deployment downtime factored in, not just steady-state throughput
  • [ ] Alerting on slot/log lag as an absolute measure, independent of connector self-reported health
  • [ ] Incremental snapshot chunking sized per-table, not left at framework defaults for large tables
  • [ ] Outbox pattern (or equivalent schema-registry-enforced contract) for any table whose schema changes for reasons unrelated to the downstream event contract
  • [ ] DLQ volume alerting configured — zero-tolerance on growth, not a volume threshold
  • [ ] Documented, tested procedure for dropping an orphaned replication slot as part of connector decommissioning
  • [ ] Runbook for "connector fell behind past log retention" that doesn't start with someone improvising a resnapshot strategy at 2am

Closing thought

None of the four failure modes above are exotic — they're the predictable consequence of running a system whose core promise ("keep up with every change, forever, without gaps") has a hard dependency on a finite resource (log retention) that most teams don't monitor directly. The fix isn't more sophisticated tooling than Debezium and Kafka Connect already provide; it's treating slot lag, snapshot progress, schema contract enforcement, and DLQ growth as first-class monitored signals from day one, rather than discovering their absence during an incident.