TL;DR: Don't encrypt application data directly against a managed KMS — it's rate-limited, latency-bound, and priced per call. Use envelope encryption: KMS protects a small data key, and that data key does the actual bulk encryption locally. The part most teams get wrong isn't the encryption itself, it's key rotation — specifically, rotating a key before every place that depends on the old one can actually read it, and treating re-encryption as an afterthought instead of a deliberate, throttled background process.

Encryption key management sounds like a solved problem — call KMS, encrypt your data, done. In practice, almost every production incident in this space traces back to one of two mistakes: encrypting data directly against a managed key service in a way that doesn't scale, or rotating keys without a plan for what happens to data that's still encrypted under the old one. This piece covers the architecture that avoids both: envelope encryption as the default pattern, and the specific failure modes that show up when key rotation isn't designed as carefully as the encryption itself.

Why you don't encrypt data directly against KMS

A managed key service like AWS KMS, GCP Cloud KMS, or Azure Key Vault is built to protect key material, not to be a bulk encryption engine. Its APIs are rate-limited, charge per call, and cap payload size — calling Encrypt directly on every row of application data means every encryption operation is a network round trip to a service with real throughput ceilings. At any meaningful data volume, this becomes a latency bottleneck and, past a certain call rate, starts hitting API limits that fail unpredictably under load — exactly when you can least afford encryption to become the bottleneck.

The envelope encryption pattern

Envelope encryption solves this by using KMS for exactly one thing — protecting a small key — and doing the actual bulk encryption locally, with fast symmetric cryptography:

  1. Call KMS's GenerateDataKey API. It returns two things: a plaintext data key (for immediate local use) and the same key encrypted under your KMS master key (for storage).
  2. Use the plaintext data key to encrypt your actual payload locally with a standard algorithm — AES-256-GCM is the common choice — producing ciphertext.
  3. Store the ciphertext alongside the encrypted (wrapped) data key. Discard the plaintext data key from memory immediately.
  4. To decrypt later: call KMS's Decrypt API on the wrapped data key to get the plaintext key back, then decrypt the payload locally with it.
text
Encrypt:  KMS.GenerateDataKey() → {plaintext_DEK, encrypted_DEK}
          ciphertext = AES-256-GCM(payload, plaintext_DEK)
          store(ciphertext, encrypted_DEK)
          discard(plaintext_DEK)

Decrypt:  plaintext_DEK = KMS.Decrypt(encrypted_DEK)
          payload = AES-256-GCM-decrypt(ciphertext, plaintext_DEK)

The security property this buys: if an attacker steals your database, they get ciphertext and wrapped data keys — useless without access to the KMS master key (the "key-encryption key," or KEK, as distinct from the data-encryption key, or DEK) to unwrap them. Who can even call Decrypt on that master key in the first place is a separate but equally important question — one that comes down to identity and access management on the KMS key policy itself, not just the encryption scheme. And the performance property: only the small data key, not the bulk payload, ever crosses the network to KMS. Nearly every managed KMS's own SDK guidance (AWS, GCP, Azure) converges on this two-tier DEK/KEK pattern for exactly these reasons.

Data key caching: the next performance lever

Even with envelope encryption, calling GenerateDataKey for every single encryption operation still means one KMS API call per operation — which is fine at moderate volume but becomes the limiting factor at high throughput (event-driven systems processing thousands of records per second, for instance).

The standard mitigation is a caching cryptographic materials manager — the AWS Encryption SDK ships one built in — that reuses a plaintext data key across multiple encryption operations instead of requesting a new one each time, bounded by explicit security thresholds: max age, max number of messages encrypted under the same key, max total bytes. This is a real security/performance tradeoff, not a free optimization — a cached key used across more operations means a larger blast radius if that specific plaintext key is ever compromised in memory, which is exactly why the thresholds exist and shouldn't be set arbitrarily high just to minimize KMS calls. Teams report meaningful cost reduction (one documented case cut KMS costs by 77%) from disciplined caching without loosening security guarantees, by tuning cache limits per data sensitivity tier rather than applying one global cache policy everywhere.

Where key rotation actually goes wrong

Rotation itself — telling KMS to start using a new key version for new encryption operations — is the easy part; most managed KMS platforms support this as effectively a config change. The failure modes show up in what rotation doesn't automatically handle:

Premature revocation before propagation completes. In a horizontally scaled service, different instances pick up the new key at different times (config refresh intervals, rolling deploys, caching layers). If the old key is revoked or disabled before every instance has finished any pending work that depends on it, requests start failing unpredictably — the classic documented case is a certificate rotation where half a fleet of load balancers got the new cert and half didn't, and traffic to the stale half broke.

Confusing "rotate" with "re-encrypt." Rotating a KMS key changes what key new encryption operations use — it does not retroactively re-encrypt data that's already encrypted under the previous key version. That data remains readable only as long as the old key version stays available for decryption. If a service generating encryption keys restarts and starts fresh with a new key while treating the old one as disposable, everything encrypted under the old key over the preceding months becomes unreadable — this is a documented real-world failure pattern, not a hypothetical.

Skipping verification. Rotation should be tested by actually decrypting a sample of data with the new key path before treating the rotation as complete, not just confirming the API call succeeded. If rotation tests fail, it usually means some subset of encrypted data can't be decrypted through the expected path — better to find that during a controlled test than during an incident.

Re-encryption: the part that has to be deliberate

Every team we've seen run this well treats it as a written key rotation runbook, not tribal knowledge. Once a key is rotated, you're typically running with data now split across two (or more) key generations — old data under the old key, new data under the new one. Full re-encryption of everything onto the new key is often desirable for compliance or blast-radius reasons, but doing it carelessly is its own outage risk: bulk re-encryption is I/O- and CPU-intensive, and an unthrottled background job that decrypts and re-encrypts an entire dataset can saturate the database or the KMS API rate limit and take down the system it's supposed to be protecting.

Three re-encryption strategies, in order of operational safety:

  • Lazy, access-based re-encryption: re-encrypt a record with the new key the next time it's read or written for a normal application reason, rather than proactively. Low operational risk, but "cold" data that's never accessed may stay on the old key indefinitely — a problem if the old key needs to be fully retired by a compliance deadline.
  • Throttled background re-encryption: a dedicated job re-encrypts records in small batches on a rate-limited schedule, explicitly bounded to a fraction of normal database and KMS capacity. This is the standard approach when a hard deadline for full re-encryption exists.
  • Hybrid: lazy re-encryption as the default, with a throttled background sweep for records that haven't been touched within some window, so cold data eventually migrates without needing full-dataset throughput.

Whichever strategy is chosen, keep the old key version available and decryptable until re-encryption is verifiably complete — not just "probably complete." Retiring or destroying an old key before confirming zero remaining references to it is close to unrecoverable.

A key rotation runbook that avoids the common failures

  1. Generate the new key version; do not disable the old one yet.
  2. Roll out the new key to all services gradually, confirming propagation (not just deployment) across every instance before proceeding.
  3. Switch new encryption operations to the new key version.
  4. Begin re-encryption of existing data using a lazy or throttled strategy appropriate to the data's access pattern and compliance deadline.
  5. Monitor decrypt failures and re-encryption progress explicitly — don't assume silence means success.
  6. Only after re-encryption is verified complete (sampled decryption checks against the new key path, not just a completed job log) should the old key version be scheduled for retirement, and even then with a grace/waiting period before actual deletion.

Encryption key management is one of the recurring gaps we find when auditing security architecture for clients — the encryption itself is rarely the weak point; the rotation and re-encryption plan almost always is.

Sources