> ## Documentation Index
> Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Kafka encryption, masking and tokenization

> Choose how to protect sensitive Kafka data with Conduktor Gateway: full payload encryption, field level encryption, masking, tokenization and crypto shredding. Integrates with Vault, AWS KMS, Azure, and GCP.

<Badge stroke color="blue" icon="sparkle" size="lg">Enterprise</Badge>

## Choose how to protect your data

Gateway protects data as it passes through the proxy, so your applications don't change. You have five options, and regulated deployments usually combine two.

| Option                                 | What it gives you                                                                                    | Fits when                                                                                                         | Watch out for                                                                                                             |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| [Masking](#masking)                    | Sensitive fields replaced with a masking character on consume                                        | You replicate production data into lower environments. The copy that lands there has no key and can't be reversed | The record in Kafka is unchanged. Anyone reading outside the policy, or reading the backing cluster, sees the clear value |
| [Full payload encryption](#encryption) | The whole record is unreadable without the key                                                       | One operational decision covers everything leaving your boundary, with no per project work                        | Nothing reading the backing cluster can process the data, including a stream processor wired to your brokers              |
| [Field level encryption](#encryption)  | Only sensitive fields are encrypted, and you control who decrypts                                    | One team needs the card number, another has to be kept away from the customer identity                            | Every project has to declare its sensitive fields                                                                         |
| [Tokenization](#tokenization)          | A deterministic token with no mathematical link to the value, which you can count, group and join on | An analytics team reports on a PII field without seeing its values                                                | Clear text reaching the same place as the tokens compromises the dataset. Needs Vault Enterprise                          |
| [Crypto shredding](#crypto-shredding)  | A key per customer, derived from the record                                                          | You have to honor a right to be forgotten request, which Kafka can't satisfy by deleting a record                 | Deleting a key is permanent, and every record has to carry the identifier the key ID is derived from                      |

These five control who can read your data. To control what gets written instead, so consumers can reject records that didn't come through your infrastructure, see [verify message integrity](/guide/use-cases/kafka-message-signing).

## Masking

Masking replaces sensitive fields with a masking character as the message is consumed. You choose the fields, the character, and whether to mask the whole value or only the first or last few characters, so a card number can keep its last four digits.

Masking applies on read, so the record in Kafka is unchanged. A client reading outside the policy scope, or connecting to the backing cluster directly, still sees the clear value. Masking controls who sees what, it doesn't protect data at rest.

That makes it the tool for producing safe copies rather than for securing a cluster. Consume a production topic through Gateway with masking on, produce the result into a lower environment, and the copy that lands there has no key and can't be reversed.

See the [data masking Interceptor reference](/guide/reference/data-security#data-masking-interceptor). Console also has [data masking policies](/guide/conduktor-in-production/admin/data-masking), which hide values in the Console UI only.

## Encryption

<Tooltip tip="A Kafka proxy that deploys extensible plugins for encryption, filtering, and data processing.">Gateway</Tooltip> encrypts on produce, before the record reaches the broker, so data is already encrypted when it enters Kafka. <Tooltip tip="Conduktor Interceptors are Gateway plugins that transform and manipulate data.">Interceptors</Tooltip> decrypt on consume, for clients authorized to read.

Encryption is non-deterministic: the same value encrypted twice gives two different results. That's what makes it strong, and it's also why an encrypted field can't be counted or joined on. If a team has to work with the values rather than just move them, see [tokenization](#tokenization).

<Info>
  **From our blog:** [Stop building Kafka encryption libraries](https://www.conduktor.io/blog/stop-building-kafka-encryption-libraries) Why client-side encryption sprawls and how proxy-layer encryption centralizes control.
</Info>

Watch field-level encryption end to end, from encrypt on produce through key rotation and controlled decryption:

<iframe className="w-full aspect-video rounded-xl" src="https://www.youtube.com/embed/qWLNev-9vZQ" title="Field-level encryption end to end" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowFullScreen />

### Choose what to encrypt

Full payload encryption covers the record key, the value or the headers, on structured or unstructured messages. Field level encryption covers the fields you name in Avro, Protobuf and JSON.

Pair them. Field level applies where a project declared its fields, full payload catches the rest, so a project that ships without declaring anything still can't put readable PII outside your boundary. The fallback costs you readability, since nothing that skips Gateway can process the record.

Name the fields two ways:

* [List-based](/guide/reference/data-security#list-based): from a list in the Interceptor configuration. Works on any data format, and the platform team owns the list.
* [Schema-based](/guide/reference/data-security#schema-based): from tags in the record schema, so the team that owns the data owns the choice. It only covers data with a schema you control, a weaker fit when much of your data is schemaless JSON, or when Kafka Connect, ksqlDB or a third party product infers the schema.

Set the [algorithm](/guide/reference/data-security#supported-algorithms) and [KMS provider](/guide/reference/data-security#kms-integration) per field.

Encryption and decryption are transparent to your applications. Two details are worth knowing, since they show up when you inspect a topic:

* Gateway writes the key reference into the record header, which is how decryption finds the right key later.
* An Avro record stays Avro, so it keeps its link to its schema in the backing topic.

See the [encryption](/guide/reference/data-security#encryption-configuration) and [decryption](/guide/reference/data-security#decryption-configuration) reference for the configuration.

## Tokenization

Tokenization replaces a value with an unrelated token and keeps the mapping in Vault, so an analytics team can work with a field without the values ever reaching Kafka. The same value always produces the same token, which is what makes counting, grouping and joining possible.

What makes a token a token, rather than ciphertext, is that it carries no mathematical link to the original value. You can't decrypt it. The only way back is to ask Vault, which holds the mapping.

That determinism has a cost: anyone holding a clear value and its token learns every record carrying that value, so tokenization only holds while clear text stays out of the same environment.

Tokenization uses the encryption Interceptors with a `vault-transform://` key, and needs the Transform Secrets Engine in Vault Enterprise. No other KMS provider supports it.

Keys work differently here. Rather than the envelope model in [key management](#key-management), Gateway calls a Transform role you define in Vault, and that role holds the transformation. Vault owns the key material and the token mapping, so rotation and access are governed by your Vault policy rather than by Gateway. See the [tokenization reference](/guide/reference/data-security#tokenization).

## Crypto shredding

Kafka can't delete an individual record, so retention is the only native way to remove data. Crypto shredding gets you there through the keys instead: derive the key ID from a property of the record, such as the customer ID, so each customer's data is encrypted under its own key. Delete that key and the data becomes permanently unreadable, wherever it sits and whatever the topic retention is.

Gateway holds one master key (KEK) in your external KMS, and generates a key per customer itself. Each one is stored encrypted by that master key (as an EDEK) in a compacted Kafka topic, the Encryption Keys Store. So you get per-customer granularity without a key per customer in your KMS, which is what makes this affordable at scale, and the store is safe because an EDEK is worthless without the KEK.

Tombstoning an EDEK in that topic makes the associated data permanently undecryptable through Gateway. There's no undo, so treat the Encryption Keys Store as production state: it decides what remains readable.

Two things to plan for. The key ID is derived from the record, so every record you want to shred individually has to carry the identifier you derive it from, such as a customer ID. And that identifier has to be present at produce time, since Gateway builds the key ID as it encrypts.

This is how organizations honor a right to be forgotten request under GDPR and CCPA. See the [crypto shredding tutorial](/guide/tutorials/crypto-shredding).

## Key management

This applies to [encryption](#encryption) and [crypto shredding](#crypto-shredding), which share the same key model. [Tokenization](#tokenization) uses a Vault Transform role instead, and [message integrity](/guide/use-cases/kafka-message-signing) uses signing keys from Vault KV v2.

Gateway uses envelope encryption:

|   Term   | Definition                                                                                                                      |
| :------: | ------------------------------------------------------------------------------------------------------------------------------- |
|  **KMS** | **Key Management Service**: manages and stores cryptographic keys, including the **KEK**                                        |
|  **KEK** | **Key Encryption Key**: stored in the **KMS**, encrypts the **DEK**. The Interceptor never sees it                              |
|  **DEK** | **Data Encryption Key**: generated by the Interceptor, encrypts the data                                                        |
| **EDEK** | **Encrypted Data Encryption Key**: the **DEK** encrypted by the **KEK**, so the **DEK** stays secure when stored or transmitted |

To **encrypt**, Gateway:

1. Generates a **DEK**.
2. Sends it to the **KMS**, which encrypts it with the **KEK** and returns the **EDEK**.
3. Caches the **DEK** and **EDEK** in memory for a configurable Time to Live (TTL), which keeps KMS calls down.
4. Encrypts the data with the **DEK**.
5. Sends the **EDEK** to the backing Kafka alongside the encrypted data.

To **decrypt**, Gateway:

1. Retrieves the **EDEK** stored with the encrypted data.
2. Sends it to the **KMS**, which returns the **DEK**. Gateway skips this call when the **DEK** is still cached.
3. Decrypts the data with the **DEK**.

<img src="https://mintcdn.com/conduktor/YlJn6qrEZm6kp3UE/images/encryption.png?fit=max&auto=format&n=YlJn6qrEZm6kp3UE&q=85&s=7c5caa7e3c57f46613a03fb57f62127b" alt="envelope encryption" width="1379" height="766" data-path="images/encryption.png" />

### Key rotation security

Gateway supports multiple encryption algorithms with `AES128_GCM` as the default.

When using AES-GCM algorithms, be aware that the same DEK (Data Encryption Key) should not be used to encrypt more than approximately 4 billion (2³²) messages due to security limitations.

For high-traffic scenarios, consider:

* using alternative algorithms like `XCHACHA20_POLY1305` which supports \~2⁸⁰ messages
* implementing DEK rotation by using unique `keySecretId` values
* rotating your KEK (Key Encryption Key) through your KMS provider according to your security policies

<Warning>
  **KEK rotation is your responsibility**: KEK rotation has to be performed by you through your KMS provider (AWS KMS, Azure Key Vault, HashiCorp Vault, GCP KMS).

  Gateway does not automatically rotate KEKs. [Find out about key rotation](/guide/reference/data-security#key-rotation).
</Warning>

## Related resources

* [Configure the KMS](/guide/reference/data-security#kms-integration)
* [Encryption reference](/guide/reference/data-security#encryption-interceptors)
* [Encryption configuration examples](/guide/tutorials/configure-encryption)
* [Tokenization reference](/guide/reference/data-security#tokenization)
* [Data masking reference](/guide/reference/data-security#data-masking-interceptor)
* [Crypto shredding tutorial](/guide/tutorials/crypto-shredding)
* [Console data masking policies](/guide/conduktor-in-production/admin/data-masking)
* [Verify message integrity](/guide/use-cases/kafka-message-signing)
* [Give us feedback/request a feature](https://conduktor.io/roadmap) <Icon icon="up-right-from-square" />

**From our blog:**

* [Designing Kafka applications for data privacy compliance](https://www.conduktor.io/blog/designing-kafka-applications-for-data-privacy-compliance): why privacy enforcement belongs in one layer rather than in every application.
* [Crypto shredding in Kafka](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance): how key deletion makes data permanently unreadable without touching the records.
* [The hidden pitfalls of Kafka's schemaless data](https://conduktor.io/blog/the-hidden-pitfalls-of-kafka-s-schemaless-data): what you lose when the data carries no schema, which is what limits schema-based encryption.
* [FedRAMP High for Kafka without replatforming](https://www.conduktor.io/blog/fedramp-high-for-kafka-without-replatforming) and [Kafka and HIPAA 2026](https://www.conduktor.io/blog/kafka-hipaa-compliance): meeting encryption requirements on an existing cluster.