Right-to-Erasure Architecture: Deleting Personal Data Across Databases, Backups and Data Lakes

TL;DR: A DELETE statement is the start of erasure, not the end. Rows linger as dead tuples in your database, as older snapshots in your lakehouse, and as copies in backups, caches, search indexes and downstream vendors. Treat erasure as a distributed workflow with a system-of-record map, per-store deletion mechanics, an explicit backup policy, and evidence you can show a regulator. This guide walks through each layer, with the trade-offs and the traps, for teams working under the UK GDPR, the EU GDPR and India's DPDP Act.

Why erasure is now an engineering problem, not just a legal one

For years the right to erasure ("right to be forgotten") lived in privacy policies and ticket queues. Regulators are now looking at how it works in practice. The European Data Protection Board (EDPB) ran its 2025 Coordinated Enforcement Framework on the right to erasure: according to the EDPB's announcement of 18 February 2026, 32 data protection authorities took part, 764 controllers responded, and nine authorities opened or continued formal investigations while 23 ran fact-finding exercises. The EDPB reported seven recurring challenges. The ones it names explicitly are the ones engineers recognise immediately: missing internal procedures for handling requests, reliance on inefficient anonymisation techniques instead of deletion, inconsistent practice, and difficulty determining retention periods and deleting personal data in the context of back-ups (EDPB, 2026).

That list is a fair description of most production estates. A customer record is rarely in one place. It sits in an operational database, gets replicated to a read replica, is exported nightly to object storage, lands in a lakehouse, is indexed in a search cluster, appears in logs, and is shared with processors. "Delete the user" therefore has to mean something precise for every one of those stores.

This article assumes you already know the legal basics. For the details of exemptions, see Article 17(3) of the GDPR (GDPR text) and your own counsel. We focus on the architecture. Nothing here is legal advice; decisions about what must be erased, and when, belong to your data protection officer and legal team.

What the regulators actually require of your deletion design

Three points from primary sources shape the design.

Timing. The UK ICO's guidance says you must respond to an erasure request "without undue delay and at the latest within one month", extendable by a further two months if the request is complex or the individual has made a number of requests, provided you tell the individual within the first month and explain why (ICO). EU GDPR uses the same one-month structure. Your workflow needs a clock that starts at receipt, not at "ticket triaged".

Backups. The ICO is explicit that if a valid request is received and no exemption applies, you have to take steps to ensure erasure from backup systems as well as live systems. It then adds the practical qualifier: "the key issue is to put the backup data 'beyond use', even if it cannot be immediately overwritten." The conditions it describes are that you do not use the backup data for any other purpose, that it is held until replaced on an established schedule, and that you are clear with individuals about what will happen to their data, including in backups (ICO). That is UK guidance. For EU deployments, confirm the position of your lead supervisory authority, because the EDPB specifically flagged backups as a difficulty.

Recipients. If you have disclosed the data to others, you must contact each recipient and tell them about the erasure, unless that proves impossible or involves disproportionate effort, and you must tell the individual who those recipients are if asked (ICO). Architecturally, that means you need a record of where data was sent.

Step one: build the personal data map before you build the pipeline

Every erasure design fails the same way: someone discovers a copy nobody listed. Start with an inventory that answers, for each store:

• What personal data does it hold, and under which identifiers (user ID, email, device ID, order number)?
• Is it a system of record or a derived copy?
• Who owns it, and what is the deletion mechanism?
• What is the longest period before a deleted row is physically gone, including snapshots and backups?
• Is it replicated or exported elsewhere?

The system-of-record versus derived-copy distinction is the most useful single idea in this article. You erase from the system of record, then propagate to derived stores through an event, and each derived store reports completion back. Derived stores are where the surprises live: analytics exports, ML feature stores, support-tool mirrors, cached API responses and log pipelines.

Identifier choice matters too. If every store keys personal data on one stable internal subject ID, the erasure workflow can fan out by that ID. If the same person is keyed by email in one system and phone number in another, you need an identity resolution step first, and you should write down how confident it must be before deletion runs.

The reference architecture

The pattern that holds up in practice is an orchestrated workflow with an auditable ledger, not a script that issues deletes in a loop.

flowchart TD
    A[Erasure request received<br/>clock starts] --> B[Verify identity<br/>resolve subject ID]
    B --> C{Exemption or<br/>retention duty?}
    C -- Yes --> D[Restrict and document<br/>retain only what is required]
    C -- No --> E[Write erasure ledger entry<br/>status: accepted]
    E --> F[Publish erasure event<br/>subject_id, request_id]
    F --> G1[OLTP databases<br/>delete + physical purge]
    F --> G2[Lakehouse tables<br/>delete + snapshot expiry]
    F --> G3[Search, cache, logs]
    F --> G4[Processors and vendors<br/>notify recipients]
    G1 --> H[Each store acks completion<br/>or reports residual window]
    G2 --> H
    G3 --> H
    G4 --> H
    H --> I[Backups: record beyond-use<br/>and expiry date]
    I --> J[Close request, respond<br/>to individual, keep evidence]

Three design rules make this work.

Idempotent deletes. Events will be redelivered. Every consumer must treat "delete subject X" as safe to run twice.

A ledger, not a log line. Persist per-store status (pending, deleted, residual_until=<date>, failed). This is the evidence that answers a regulator or a data subject six months later. Keep in mind the ledger itself must not become a new copy of the personal data you just erased: store the request ID, subject ID and timestamps, not names or email addresses. Also note that even a hash of an email address is generally still personal data, so decide deliberately how you will recognise a returning subject when you restore from a backup.

Failure is visible. A store that cannot confirm deletion should hold the request open and alert someone. Silent partial erasure is worse than a loud failure because it looks like compliance.

Layer 1: relational databases

In PostgreSQL, a DELETE does not physically remove the row immediately. The PostgreSQL documentation says that "tuples that are deleted or obsoleted by an update are not physically removed from their table; they remain present until a VACUUM is done" (PostgreSQL docs). The standard VACUUM "removes dead row versions in tables and indexes and marks the space available for future reuse" (PostgreSQL routine vacuuming), but the extra space is generally not returned to the operating system; it is kept for reuse in the same table. VACUUM FULL rewrites the table into a new file with no dead space, but takes an ACCESS EXCLUSIVE lock and needs extra disk for the new copy.

In practice this means:

BEGIN;
DELETE FROM customer_profile WHERE subject_id = $1;
DELETE FROM customer_address WHERE subject_id = $1;
COMMIT;

-- Reclaims dead tuples for reuse; does not take an exclusive lock.
VACUUM (VERBOSE) customer_profile;
VACUUM (VERBOSE) customer_address;

Autovacuum will normally do this for you on its own schedule, which is usually sufficient for ordinary operation. The point for erasure is that you should know your schedule and be able to state the window. Reaching for VACUUM FULL on every request is rarely sensible given the locking cost; most teams document the autovacuum behaviour and reserve table rewrites for exceptional cases. Whether dead tuples in the table files meet your "erased" standard is a risk decision to make with your DPO, and you should be able to explain it.

Other relational concerns:

• Foreign keys and cascades. Decide per table whether to cascade delete, null out, or replace with an irreversible placeholder. Orders may need to be retained for accounting law while the customer's name and contact details are removed. Retain what a legal obligation requires and nothing more.
• Read replicas and logical replication. Deletes replicate, but replicas have their own vacuum behaviour and lag.
• Write-ahead logs and point-in-time recovery. These contain the pre-delete data until archived segments expire. Treat them as part of the backup policy below.
• Soft deletes. A deleted_at flag is not erasure. If you use soft deletes for product reasons, add a scheduled hard-delete job and make the erasure workflow call it directly.

Layer 2: data lakes and lakehouses

Lakehouse table formats make deletes possible but not instant, because their strength (immutable files and time travel) works against erasure.

Delta Lake

Databricks' GDPR guidance describes the sequence. You delete with a normal statement, for example DELETE FROM bronze.users WHERE user_id = 5, or a MERGE against a control table with WHEN MATCHED THEN DELETE. The deleted data, however, "is still retained in cloud storage", so you must run VACUUM on Delta tables to "permanently remove previous versions of data". The page notes that VACUUM removes files no longer referenced by the table and older than the retention threshold, and that by default this limits time travel to 7 days. It also states that for tables with deletion vectors enabled, you must also run REORG TABLE ... APPLY (PURGE) to permanently delete the underlying records (Databricks documentation).

-- 1. Logical delete for a batch of subjects from a control table
MERGE INTO bronze.users AS t
USING gdpr_control_table AS c
  ON t.user_id = c.user_id
WHEN MATCHED THEN DELETE;

-- 2. If deletion vectors are enabled, rewrite files to drop the rows physically
REORG TABLE bronze.users APPLY (PURGE);

-- 3. Remove old file versions that still contain the data
VACUUM bronze.users;

Two notes. First, check your platform's current documentation for retention-check behaviour before shortening retention below the default. Second, the same page points out that upstream sources (Kafka topics, landing files, source databases) also have to be cleaned, and that streaming tables that read from a table you delete from need handling (it mentions skipChangeCommits or a full refresh). Batching deletes, for example processing the control table once a day, is far cheaper than running one rewrite per request and fits comfortably inside the one-month deadline.

Apache Iceberg

Iceberg's maintenance documentation explains the underlying principle: "data files are not deleted until they are no longer referenced by a snapshot that may be used for time travel or rollback", and "regularly expiring snapshots deletes unused data files" (Iceberg maintenance docs). So a row-level delete is followed by snapshot expiry. In Spark, the documented procedure takes arguments such as table, older_than (default five days ago) and retain_last (default 1); snapshots still referenced by branches or tags are not removed (Iceberg Spark procedures).

-- Delete the subject's rows (creates a new snapshot)
DELETE FROM catalog.db.users WHERE user_id = 5;

-- Expire snapshots older than a cut-off so unreferenced data files can be removed
CALL catalog.system.expire_snapshots(
  table => 'db.users',
  older_than => TIMESTAMP '2026-11-01 00:00:00',
  retain_last => 1
);

-- Clean up files that no metadata references (preview first)
CALL catalog.system.remove_orphan_files(table => 'db.users', dry_run => true);

Row-level deletes in Iceberg are recorded as delete files rather than rewriting data files immediately. The format specification describes position deletes (version 2 tables) and equality deletes (versions 2 and 3), and version 3 adds deletion vectors. The maintenance page also lists operations for compacting data files, rewriting position delete files and removing dangling delete files. Until delete files are compacted into rewritten data files and old snapshots expire, the original bytes can still exist. Plan for both: expiry for time-travel history, and compaction for the files themselves. Check the exact procedure arguments against the version of Iceberg and your engine, since they vary.

Also remember branches and tags: a long-lived tag pins a snapshot, and with it, the data you thought you deleted.

Object storage and bucket versioning

If your data lake sits on versioned object storage, a delete adds a delete marker and leaves the old version in place. Lifecycle rules must expire non-current versions on a schedule consistent with your stated erasure window. Check your cloud provider's documentation for the exact behaviour of the store you use.

Layer 3: backups, snapshots and disaster recovery

Backups are where theory and operations collide. Immutable backups exist so that deleted data can be recovered. Erasure exists so that deleted data stays gone. You reconcile the two with an explicit, documented policy, not with a heroic attempt to edit backups.

The ICO's position gives a workable structure for the UK: erase from live systems promptly, put the backup copy beyond use, let it expire on a defined schedule, and tell the individual what will happen (ICO). The consequences for engineering are specific:

1. Short, fixed backup retention. The backup lifecycle is your effective residual-data window. A 35-day cycle is easier to defend than an open-ended archive. Retention that exceeds what disaster recovery needs is hard to justify.
2. A restore-time re-erasure step. If you ever restore a backup, the erasure ledger must be replayed before the restored system serves traffic. Otherwise a disaster recovery event silently resurrects people who were deleted. This is the most commonly missed control, and it is why the ledger must outlive the data.
3. Restricted restore access. "Beyond use" is an operational claim. Back it with access controls and audit logs showing who can restore and why.
4. Test it. Run a restore drill that includes replaying the ledger, and record the result.

Crypto-shredding as an option

For datasets that are too costly to rewrite, or backups you cannot edit, some teams encrypt each subject's data with a per-subject data key and delete the key to render the ciphertext unreadable. This is a design choice with trade-offs, not a free pass. The sources reviewed for this article do not endorse it as equivalent to deletion, and the EDPB specifically called out reliance on inefficient anonymisation as a problem, so discuss it with your DPO and document your reasoning before relying on it. Keys must themselves be destroyed everywhere, including key backups.

A minimal illustration of the pattern, using AES-GCM from the Python cryptography library:

import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

# subject_id -> 256-bit data key; in production this lives in a KMS/HSM-backed
# key service, never in application memory or the same store as the data.
key_store: dict[str, bytes] = {}

def encrypt_for_subject(subject_id: str, plaintext: bytes) -> bytes:
    key = key_store.setdefault(subject_id, AESGCM.generate_key(bit_length=256))
    nonce = os.urandom(12)  # unique per encryption
    return nonce + AESGCM(key).encrypt(nonce, plaintext, subject_id.encode())

def decrypt_for_subject(subject_id: str, blob: bytes) -> bytes:
    key = key_store[subject_id]  # raises KeyError after erasure
    nonce, ciphertext = blob[:12], blob[12:]
    return AESGCM(key).decrypt(nonce, ciphertext, subject_id.encode())

def erase_subject(subject_id: str) -> None:
    key_store.pop(subject_id, None)  # ciphertext everywhere is now unreadable

Limitations to state plainly: it only protects data that was encrypted under the subject key from the start, it does not remove metadata or unencrypted copies, and it shifts risk onto key management.

Layer 4: everything else that copies personal data

• Search indexes (Elasticsearch, OpenSearch and similar): deleted documents are typically removed during segment merges, not instantly. Document your merge behaviour or force merges where appropriate.
• Caches and CDNs: purge by subject-keyed cache tags and keep TTLs short on personalised responses.
• Logs and observability: the best control is not writing personal data into logs. Where you must, set retention on the log store. Note that under India's DPDP Rules, certain logs and associated personal data must be retained for a minimum of one year for specified purposes, so your log policy may need to reconcile two obligations.
• Machine learning artefacts: training sets, feature stores and embeddings. Deleting a row from a feature store does not alter a model already trained on it; record what your policy is and what lawful basis supports it.
• Processors and third-party tools: CRM, email, support and analytics vendors. Maintain a register of recipients and automate vendor deletion through their APIs where available, with ledger entries for each.

India's DPDP Act: what changes in the design

If you also serve India, the DPDP Act and Rules shape the same pipeline. The statutory text (as published on dpdpa.com; verify against the official Gazette) says in Section 8(7) that a data fiduciary shall, unless retention is necessary for compliance with law, erase personal data when the data principal withdraws consent or as soon as it is reasonable to assume the purpose is no longer served, whichever is earlier, and cause its processors to do the same. Section 12(3) says that on an erasure request the data fiduciary shall erase the data unless retention is necessary for the specified purpose or for compliance with law (DPDP Act s.8, s.12).

Rule 8 of the DPDP Rules 2025 adds mechanics worth building in: at least 48 hours before the end of the applicable time period for erasure, the fiduciary must inform the data principal that the data will be erased, and a minimum one-year retention applies to certain processing logs and associated traffic data (Rule 8). The Third Schedule sets a three-year inactivity period for specific classes of large e-commerce, online gaming and social media entities. The Gazette text says Rules 5 to 16 come into force eighteen months after publication; the notification is dated 13 November 2025, which points to roughly mid-May 2027, but confirm the dates with counsel.

Design implications: a scheduled, inactivity-driven erasure job with a pre-erasure notice step (not just a request-driven one), a processor-propagation step for Section 8(7)(b), and a retention-exception table recording which legal duty justifies keeping each record.

Trade-offs at a glance

Common mistakes

1. Deleting from the primary and calling it done. The row survives in replicas, exports, indexes and vendor tools.
2. Treating soft delete as erasure. A flag does not remove data.
3. No restore-time replay. Disaster recovery brings erased people back.
4. Hashing and calling it anonymous. A hashed email is usually still linkable.
5. Putting personal data in the erasure ledger. The audit trail becomes a new copy.
6. Unbounded backup or snapshot retention. The residual window becomes indefinite.
7. Forgetting derived data. Feature stores, caches, search indexes and reports.
8. No clock. Requests sit in a queue past the one-month deadline.
9. Ignoring legitimate retention. Over-deleting records you are legally required to keep is also a failure.

Implementation checklist

• [ ] Personal data map with system-of-record versus derived-copy labels and owners
• [ ] One stable subject ID, with documented identity resolution
• [ ] Erasure workflow with a deadline clock starting at receipt and extension handling
• [ ] Exemption and retention-duty check with recorded reasoning
• [ ] Event-driven fan-out with idempotent consumers
• [ ] Per-store deletion runbooks: vacuum behaviour, snapshot expiry, deletion vector purge, index merges, cache purge
• [ ] Documented residual window per store, including snapshots and WAL archives
• [ ] Backup retention policy, beyond-use controls and restore-time ledger replay
• [ ] Recipient register and vendor deletion calls
• [ ] Erasure ledger holding statuses only, no personal data
• [ ] Alerts on failed or stalled erasure tasks
• [ ] Periodic test: erase a synthetic subject and verify across every store, then restore a backup and verify replay
• [ ] DPDP: pre-erasure notice, processor propagation and log-retention exceptions where applicable

How Syslabs can help

Syslabs designs and builds data platforms, integrations and compliance-aware engineering for teams selling into the UK, EU and India. If you want a second pair of eyes on your personal data map, your backup and restore policy, or an erasure workflow that actually reaches your lakehouse, we can help scope it with you.

Book a conversation with our engineers: https://www.syslabs.in/book/meeting

References

• European Data Protection Board (2026, 18 February). EDPB identifies challenges hindering the full implementation of the right to erasure. https://www.edpb.europa.eu/news/edpb-identifies-challenges-hindering-the-full-implementation-of-the-right-to-erasure_en
• Information Commissioner's Office. Right to erasure. https://ico.org.uk/for-organisations/guide-to-dp/guide-to-the-uk-gdpr/individual-rights/right-to-erasure
• Databricks. Prepare your data for GDPR compliance. https://docs.databricks.com/aws/en/ldp/gdpr
• Apache Iceberg. Maintenance. https://github.com/apache/iceberg/blob/main/docs/docs/maintenance.md
• Apache Iceberg. Spark procedures. https://github.com/apache/iceberg/blob/main/docs/docs/spark-procedures.md
• PostgreSQL Global Development Group. VACUUM. https://www.postgresql.org/docs/current/sql-vacuum.html
• PostgreSQL Global Development Group. Routine vacuuming. https://www.postgresql.org/docs/current/routine-vacuuming.html
• Digital Personal Data Protection Act 2023, sections 8 and 12 (text via dpdpa.com): https://dpdpa.com/dpdpa2023/chapter-2/section8.html and https://dpdpa.com/dpdpa2023/chapter-3/section12.html
• Digital Personal Data Protection Rules 2025, Rule 8 (text via dpdpa.com): https://dpdpa.com/dpdparules/rule8.html
• Regulation (EU) 2016/679, Article 17: https://gdpr-info.eu/art-17-gdpr/