Skip to main content
Enterprise

​
Choose how to protect your data

Gateway protects data as it passes through the proxy, so your applications don’t change. You have five options, and regulated deployments usually combine two. These five control who can read your data. To control what gets written instead, so consumers can reject records that didn’t come through your infrastructure, see verify message integrity.

​
Masking

Masking replaces sensitive fields with a masking character as the message is consumed. You choose the fields, the character, and whether to mask the whole value or only the first or last few characters, so a card number can keep its last four digits. Masking applies on read, so the record in Kafka is unchanged. A client reading outside the policy scope, or connecting to the backing cluster directly, still sees the clear value. Masking controls who sees what, it doesn’t protect data at rest. That makes it the tool for producing safe copies rather than for securing a cluster. Consume a production topic through Gateway with masking on, produce the result into a lower environment, and the copy that lands there has no key and can’t be reversed. See the data masking Interceptor reference. Console also has data masking policies, which hide values in the Console UI only.

​
Encryption

encrypts on produce, before the record reaches the broker, so data is already encrypted when it enters Kafka. decrypt on consume, for clients authorized to read. Encryption is non-deterministic: the same value encrypted twice gives two different results. That’s what makes it strong, and it’s also why an encrypted field can’t be counted or joined on. If a team has to work with the values rather than just move them, see tokenization.
From our blog: Stop building Kafka encryption libraries Why client-side encryption sprawls and how proxy-layer encryption centralizes control.
Watch field-level encryption end to end, from encrypt on produce through key rotation and controlled decryption:

​
Choose what to encrypt

Full payload encryption covers the record key, the value or the headers, on structured or unstructured messages. Field level encryption covers the fields you name in Avro, Protobuf and JSON. Pair them. Field level applies where a project declared its fields, full payload catches the rest, so a project that ships without declaring anything still can’t put readable PII outside your boundary. The fallback costs you readability, since nothing that skips Gateway can process the record. Name the fields two ways:
  • List-based: from a list in the Interceptor configuration. Works on any data format, and the platform team owns the list.
  • Schema-based: from tags in the record schema, so the team that owns the data owns the choice. It only covers data with a schema you control, a weaker fit when much of your data is schemaless JSON, or when Kafka Connect, ksqlDB or a third party product infers the schema.
Set the algorithm and KMS provider per field. Encryption and decryption are transparent to your applications. Two details are worth knowing, since they show up when you inspect a topic:
  • Gateway writes the key reference into the record header, which is how decryption finds the right key later.
  • An Avro record stays Avro, so it keeps its link to its schema in the backing topic.
See the encryption and decryption reference for the configuration.

​
Tokenization

Tokenization replaces a value with an unrelated token and keeps the mapping in Vault, so an analytics team can work with a field without the values ever reaching Kafka. The same value always produces the same token, which is what makes counting, grouping and joining possible. What makes a token a token, rather than ciphertext, is that it carries no mathematical link to the original value. You can’t decrypt it. The only way back is to ask Vault, which holds the mapping. That determinism has a cost: anyone holding a clear value and its token learns every record carrying that value, so tokenization only holds while clear text stays out of the same environment. Tokenization uses the encryption Interceptors with a vault-transform:// key, and needs the Transform Secrets Engine in Vault Enterprise. No other KMS provider supports it. Keys work differently here. Rather than the envelope model in key management, Gateway calls a Transform role you define in Vault, and that role holds the transformation. Vault owns the key material and the token mapping, so rotation and access are governed by your Vault policy rather than by Gateway. See the tokenization reference.

​
Crypto shredding

Kafka can’t delete an individual record, so retention is the only native way to remove data. Crypto shredding gets you there through the keys instead: derive the key ID from a property of the record, such as the customer ID, so each customer’s data is encrypted under its own key. Delete that key and the data becomes permanently unreadable, wherever it sits and whatever the topic retention is. Gateway holds one master key (KEK) in your external KMS, and generates a key per customer itself. Each one is stored encrypted by that master key (as an EDEK) in a compacted Kafka topic, the Encryption Keys Store. So you get per-customer granularity without a key per customer in your KMS, which is what makes this affordable at scale, and the store is safe because an EDEK is worthless without the KEK. Tombstoning an EDEK in that topic makes the associated data permanently undecryptable through Gateway. There’s no undo, so treat the Encryption Keys Store as production state: it decides what remains readable. Two things to plan for. The key ID is derived from the record, so every record you want to shred individually has to carry the identifier you derive it from, such as a customer ID. And that identifier has to be present at produce time, since Gateway builds the key ID as it encrypts. This is how organizations honor a right to be forgotten request under GDPR and CCPA. See the crypto shredding tutorial.

​
Key management

This applies to encryption and crypto shredding, which share the same key model. Tokenization uses a Vault Transform role instead, and message integrity uses signing keys from Vault KV v2. Gateway uses envelope encryption: To encrypt, Gateway:
  1. Generates a DEK.
  2. Sends it to the KMS, which encrypts it with the KEK and returns the EDEK.
  3. Caches the DEK and EDEK in memory for a configurable Time to Live (TTL), which keeps KMS calls down.
  4. Encrypts the data with the DEK.
  5. Sends the EDEK to the backing Kafka alongside the encrypted data.
To decrypt, Gateway:
  1. Retrieves the EDEK stored with the encrypted data.
  2. Sends it to the KMS, which returns the DEK. Gateway skips this call when the DEK is still cached.
  3. Decrypts the data with the DEK.
envelope encryption

​
Key rotation security

Gateway supports multiple encryption algorithms with AES128_GCM as the default. When using AES-GCM algorithms, be aware that the same DEK (Data Encryption Key) should not be used to encrypt more than approximately 4 billion (2³²) messages due to security limitations. For high-traffic scenarios, consider:
  • using alternative algorithms like XCHACHA20_POLY1305 which supports ~2⁸⁰ messages
  • implementing DEK rotation by using unique keySecretId values
  • rotating your KEK (Key Encryption Key) through your KMS provider according to your security policies
KEK rotation is your responsibility: KEK rotation has to be performed by you through your KMS provider (AWS KMS, Azure Key Vault, HashiCorp Vault, GCP KMS).Gateway does not automatically rotate KEKs. Find out about key rotation.
From our blog: