TL;DR: Schema evolution is manageable when you own both ends of the pipe. It becomes a governance problem, not just a technical one, the moment your consumers depend on a schema controlled by a team, a vendor, or a partner organization you have no authority over. The fix is layering technical compatibility rules (backward/forward/full, enforced by a schema registry) on top of an explicit contract and ownership model — because the registry alone can't stop a producer team from shipping a breaking change; it can only tell you after the fact that they did.
Two different problems wearing one name
"Schema evolution" usually gets discussed as a purely technical problem: pick Avro or Protobuf, configure your schema registry's compatibility mode, follow the rules for adding optional fields, done. That's the easy half of the problem, and it's genuinely solved by existing tooling.
The hard half shows up when the producer isn't your team. Maybe it's a different org within the same company with its own roadmap and incentives. Maybe it's a third-party vendor's webhook payload. Maybe it's a partner's API you integrate against. In all these cases, the technical compatibility mechanism (the registry, the schema rules) is necessary but not sufficient — because a schema registry enforces compatibility on schemas registered through it. It can't force a producer team to register their schema at all, and it can't stop a producer team with deploy authority over their own service from being unaware the registry, or the downstream consumer, exists.
The technical layer: compatibility modes, correctly understood
Get this part right first, because it's the foundation everything else sits on.
Compatibility modes (Confluent Schema Registry and equivalents use this vocabulary):
- BACKWARD — a new schema can read data written with the old schema. Consumers upgrade first. This is the default and most common choice because it lets you replay a topic from the beginning with the newest consumer code.
- FORWARD — an old schema can read data written with the new schema. Producers upgrade first.
- FULL — both directions hold; producers and consumers can upgrade independently, in either order.
- NONE — no compatibility checking (rarely the right default; usually a sign compatibility enforcement was never set up rather than a deliberate choice).
Format-specific rules that actually make evolution safe:
- Avro enforces backward compatibility only when new fields are optional or carry a default value — Avro's specification is explicit about this, which is a large part of why it remains a common choice for schema-registry-backed pipelines.
- Protobuf uses numbered field tags as the actual identifier, not field names. Fields can be added or deprecated but tag numbers must never be reused — reusing a tag number for a different field is the single most common way Protobuf schema evolution silently corrupts data, because old binary payloads get misinterpreted against the new schema.
Practical default: start with BACKWARD compatibility, add only optional fields with sensible defaults, and only move to FULL or add stricter transitive checks if you actually hit version-skew problems in production. Defaulting straight to FULL compatibility often over-constrains what producers can do without a corresponding benefit.
Where the technical layer stops protecting you
None of the above prevents a producer team from:
- Renaming a field instead of deprecating the old one and adding a new one (technically "removing and adding" a field, which most compatibility modes will actually catch — but a field repurposed to mean something semantically different while keeping the same name and type passes every automated check and breaks every downstream assumption).
- Changing what a field's value means without changing its type or name — a status code that used to mean "pending" now means "queued," same string, same schema, completely different meaning.
- Deploying a producer change without registering it against the schema registry at all, if the registry isn't a hard gate in their deploy pipeline (and if it's not their pipeline, you have no authority to make it one).
- Deprecating an entire event type or endpoint on their own roadmap timeline, which is a breaking change no compatibility mode is designed to catch, because from the registry's perspective nothing was submitted that violated a rule — the producer just stopped producing.
This is the actual argument for treating schema evolution as a data contract problem, not just a compatibility-mode configuration problem: the contract is where you encode expectations the schema format itself can't express — semantic meaning, deprecation timelines, SLAs on notice periods for breaking changes, and who has authority to approve one.
Designing the governance layer for producers you don't control
Explicit contract ownership, even across organizational boundaries. Every schema needs a named owner and a named consumer list — not because it's bureaucratic, but because the alternative is discovering who depends on a field only after you've broken them. If the producer is external (a vendor, a partner), this ownership record lives on your side even if you can't enforce it on theirs; it's your map of blast radius, not a mechanism.
A version-negotiated contract change process, not a surprise deploy. When a producer needs a genuinely breaking change, the healthy pattern is: propose the new schema version, run it in parallel with the old version for a defined window, let consumers migrate at their own pace, and only retire the old version after confirmed migration — not after an announced or, worse, unannounced date. For producers you don't control, the leverage you have is contractual (a vendor SLA, an internal API governance policy) rather than technical, so this needs to be a negotiated agreement, not an assumption.
Consumer-side defensive parsing, always. Regardless of what the contract says, downstream consumers should be defensive by default: ignore unknown fields rather than failing on them (this is standard advice for both Avro and Protobuf consumers, but bears repeating because it's the single cheapest mitigation available), validate the shape of critical fields at ingestion rather than trusting the schema registry alone, and alert on unexpected-but-not-technically-invalid values (the "pending" vs "queued" semantic drift case) rather than assuming schema validity implies semantic validity.
A dead-letter and quarantine path for schema violations, not a pipeline crash. When an unregistered producer change does slip through, the failure mode should be "these records get quarantined for inspection" — not "the entire pipeline halts" or, worse, "malformed records get silently written downstream and corrupt a table or a model." CDC pipelines in particular need this: a downstream consumer reading old-format events while an upstream producer has already moved on will otherwise either drop events or misparse them without visible failure.
Contract testing at the data layer, mirroring what API teams already do for request/response contracts. The same principle behind consumer-driven contract testing for APIs applies to event schemas: consumers can define, in an automated test, the exact fields and shapes they depend on, and that test can run against a producer's actual schema registrations in CI — giving you an automated, continuously-verified signal that's independent of whether the producer team read your Slack message about the upcoming change.
Failure modes, summarized
| Failure mode | Why the technical layer alone doesn't catch it | Mitigation |
|---|---|---|
| Field repurposed, same name/type | Compatibility checkers validate structure, not semantics | Explicit contract documenting field meaning; alert on value-distribution shifts |
| Producer never registers with the schema registry | Registry only governs what's submitted to it | Governance requirement (contractual or organizational) that registration is mandatory before production traffic |
| Event type deprecated on producer's own timeline | No schema rule was technically violated | Negotiated deprecation windows written into the data contract, with a hard consumer migration deadline |
| Silent data corruption downstream | No visible pipeline failure | Defensive consumer parsing, dead-letter/quarantine paths, anomaly alerting on distributions |
| Tag/field-number reuse in Protobuf | Passes if reviewers miss it; not always registry-enforced | Lint/CI check specifically for tag reuse, independent of registry compatibility mode |
Sources: How to Handle Schema Evolution in Kafka, Schema Evolution & Compatibility Types — Confluent Documentation, Data Contracts Explained — Atlan, Data Contracts in the Real World — Varigence
Syslabs' engineering team designs resilient data pipelines and contracts as part of our data engineering and analytics work.