TL;DR

A schema registry stops a producer from emitting a message that violates a registered shape — it is necessary but not sufficient. A data contract is the superset: it adds semantic guarantees (value ranges, null rates, freshness SLAs), ownership, and versioning policy, and it is enforced in CI/CD before code merges, not just at the Kafka broker at runtime. If you already run Kafka or Pulsar with Avro/Protobuf, you have half a data contract system for free — the framework below tells you what to build on top of it, and when a schema registry alone is genuinely enough.

The problem these two things are both trying to solve

Every data engineering team eventually hits the same failure: a producer team ships an "innocent" change — a renamed field, a status enum that gained a new value, a timestamp that switched from Unix epoch to ISO 8601 — and three pipelines downstream break silently. Nobody on the producer side knew the field was load-bearing for a revenue dashboard. Nobody on the consumer side had a way to say "don't do that" before it happened.

This is fundamentally an interface problem, not a data problem. An API is a contract the moment external code depends on your response shape — the same logic applies to a Kafka topic, a CDC stream, or a nightly export to S3. The question is where you enforce that contract and how much of the meaning of the data it covers, not just its shape.

Schema registries: what they actually guarantee

A schema registry (Confluent Schema Registry, AWS Glue Schema Registry, Redpanda's built-in registry) sits between producers and a topic and does one job well: it rejects a message the instant its structure violates a registered schema. That's it. No malformed payload ever lands in the topic, so no consumer ever has to defend against it.

Compatibility modes control how a schema is allowed to change over time:

ModeGuaranteeUpgrade order
BACKWARDNew schema can read data written with the old schemaConsumers upgrade before producers
FORWARDOld schema can read data written with the new schemaProducers upgrade before consumers
FULLBoth directions holdEither order, independently
*_TRANSITIVE variantsChecked against every prior version, not just the last oneSame as above, with more safety margin

BACKWARD is Confluent's default because it lets you rewind a consumer group to the start of a topic without breaking. In practice, most teams that get burned by "flexible" registries were running NONE compatibility, or had disabled the check entirely for a service in a hurry — a self-inflicted wound the registry was specifically built to prevent.

What a schema registry does not do:

  • Validate that a status field only contains the five values your logic branches on — it only validates the field is a string.
  • Enforce that user_id is never null, or that 99% of events arrive within 15 minutes of the event timestamp.
  • Tell you who owns the producer, what SLA they've committed to, or what happens when they break it.
  • Say anything about batch data sitting in a warehouse table, an S3 export, or a CDC stream your team doesn't control the producer for — schema registries are a streaming-platform concept.

Data contracts: the superset, and why 2026 is different from 2019

Data contracts as a concept have existed for years and mostly lived in wikis — aspirational documents no pipeline actually enforced. Two things changed that: the emergence of a real specification (the Open Data Contract Standard, ODCS, reached v3.1 in December 2025), and CI/CD tooling that treats a contract file the same way you'd treat a test suite.

A data contract typically expresses, in a version-controlled YAML or JSON file next to the producer's code:

yaml
apiVersion: v3.1.0
kind: DataContract
id: orders.order_created
version: 2.3.0
owner: payments-team
schema:
  - field: order_id
    type: string
    required: true
  - field: status
    type: string
    enum: [pending, confirmed, shipped, cancelled, refunded]
    required: true
  - field: amount_cents
    type: integer
    minimum: 0
quality:
  - rule: freshness
    max_lag_minutes: 15
  - rule: null_rate
    field: order_id
    max_percent: 0
sla:
  availability: 99.9
  support_channel: "#payments-data-oncall"

The contract is checked in two places: at PR time (a CI job diffs the proposed schema against the contract and fails the build if it's a breaking change under the declared compatibility policy), and at runtime (a lightweight validator, often built on the same libraries as Great Expectations or Soda, samples production data against the quality rules and pages the owning team, not the consumer, when it drifts).

This is the real shift from the registry model: the producer team owns the failure, because the contract lives in their repo, and their PR is what breaks CI — not a downstream analytics engineer discovering a null spike three days later in a stale dashboard.

GoCardless's public writeup on this is a good real-world reference: their contracts are written in a Jsonette-based format, merged to Git by the data owner, and used to automatically provision the BigQuery/PubSub resources and access policies for that dataset — the contract isn't just a validation gate, it's the deployment artifact.

Decision framework

Use this if you're deciding what to build:

1. Is the data moving through a broker you control (Kafka, Pulsar, Kinesis)? If yes, a schema registry is table stakes — implement it if you haven't, with FULL_TRANSITIVE compatibility for anything more than one consumer. This is a low-cost, high-value guardrail regardless of what else you build.

2. Does correctness depend on values, not just shape? If your pipeline logic branches on an enum, assumes a field is never null, or depends on freshness (e.g., fraud scoring, financial reconciliation), a schema registry gives you false confidence — the message can be perfectly well-typed and still wrong. You need contract-level quality rules.

3. Do producer and consumer sit in different teams, orgs, or trust boundaries? Same team, same repo: a shared type definition and a code review are often enough — you don't need contract infrastructure for two engineers who sit next to each other. Different teams, especially with unclear ownership: you need the ownership and SLA fields a contract provides, because "who do I page" is exactly the question that gets asked at 2am.

4. Is the data batch, not streaming? Schema registries don't apply to a nightly Parquet export or a CDC snapshot landing in a lakehouse table. If this is your situation, look at contract tooling that layers onto dbt (dbt's own contract feature, or Great Expectations/Soda checks wired into your orchestrator) rather than trying to force a streaming-specific tool onto batch data.

5. Can you afford the enforcement infrastructure? Contracts checked only in a wiki are worse than no contract — they create false assurance. If you can't commit to CI enforcement and a runtime validator with actual paging, a schema registry alone, used strictly, is a more honest scope than an unenforced "contract."

In practice, mature organizations don't treat these as competing choices — a schema registry becomes one enforcement mechanism inside a broader data contract, not a replacement for it. Teams running Protobuf contracts enforced through Kafka's schema registry (a pattern used at companies like Convoy) get both: structural safety at the broker, semantic safety in CI.

Failure modes to design against

The silent schema-registry bypass. A well-meaning engineer disables compatibility checks temporarily to unblock a deploy and forgets to re-enable them. Mitigate with a linter/CI check on the registry's compatibility setting itself, not just on schemas.

Contract drift between the YAML and the actual producer code. If the contract file isn't generated from or validated against the producer's actual serialization code, they diverge within a quarter. Prefer contract tooling that can lint the contract against a live schema (Protobuf descriptor, Avro schema, or an OpenAPI-style JSON Schema) rather than trusting hand-maintained YAML.

Quality rules that are too strict for real-world data. A null_rate: 0% rule on a field that legitimately has 0.1% nulls due to an upstream legacy system will page someone every night for a condition nobody can fix. Start quality thresholds from observed production behavior, not aspirational business requirements, and tighten over time.

Ownership rot. Contracts list an owning team, but teams get reorged and Slack channels get archived. Treat the owner and support_channel fields as data that itself needs a freshness check — a quarterly automated audit that pings the listed channel and flags contracts with no response.

Consumer-side over-reliance. Once contracts exist, consumers stop defensive coding entirely — no null checks, no schema tolerance — because "the contract guarantees it." Contracts reduce, but do not eliminate, the need for defensive parsing on the consumer side; a contract violation should degrade gracefully, not crash the pipeline.

A pragmatic rollout path

  1. Turn on FULL_TRANSITIVE compatibility on your existing schema registry for any topic with more than one consumer team. This is a one-day change with immediate payoff.
  2. Pick the two or three most-broken producer/consumer pairs from your last quarter's incident log — not your most important pipelines, your most broken ones — and write contracts for those first.
  3. Wire contract validation into the producer's CI pipeline as a required check, not an advisory one. An unenforced contract is a wiki page with extra syntax.
  4. Add a lightweight runtime quality check (row count, null rate, freshness) on a schedule, alerting the producer's on-call channel, not the consumer's.
  5. Only after that loop is stable, formalize on a standard like ODCS so contracts are portable across tooling and new hires can read them without a tutorial.

Conclusion

Schema registries and data contracts aren't competing tools — they're two layers of the same guarantee, aimed at different failure classes. A registry stops a malformed message from ever reaching a topic; a contract stops a semantically wrong but well-typed message from ever reaching a decision. Start with the registry if you're on a streaming platform and don't have one yet — it's a day of work with an immediate payoff. Layer a contract on top the moment ownership crosses a team boundary or correctness depends on values rather than shape, and enforce it in CI, not in a document nobody opens after the kickoff meeting.

Sources