Skip to main content
Enterprise

​
Why sign Kafka records

Encryption controls who can read your data. Signing answers a different question: did this record come from a producer you trust, and has it changed since? That question matters when something other than your own network can reach the cluster. A managed Kafka service with a public control plane is the common case: the cluster itself is private, but the provider’s interface isn’t, so a record can in principle land in a topic without passing through your own infrastructure. Signing gives consumers a way to tell the two apart. Records produced through carry a signature. Records that appeared some other way don’t, and consumers can drop them.

​
How it works

Two plugins work as a pair:
  • ProduceIntegrityPolicyPlugin signs each record on produce, using HMAC-SHA256 through Google Tink.
  • FetchIntegrityPolicyPlugin verifies the signature on fetch, and drops or keeps the record according to your policy.
Gateway stores the signature in the conduktor.integrity.signature header as JSON with two fields: k, the key URI with its version, and s, the base64-encoded MAC (Message Authentication Code). After a record verifies, Gateway removes that header before handing the record to the consumer, so applications never deal with it.

​
What the signature covers

The MAC isn’t computed over the value alone. Gateway builds a canonical form of the record and signs that, covering:
  • the topic and the partition
  • the record key and the value
  • every header except the signature header itself
  • the record timestamp
The canonical form is serialized with a fixed schema, so no two different records can produce the same bytes by concatenation, and headers are sorted by key, so the order your producer sets them in doesn’t matter. Two consequences follow. Changing anything in that list invalidates the signature, not just the value, so an altered key, an added header or a rewritten timestamp all fail verification. And because the topic and partition are covered, a record copied elsewhere doesn’t verify even byte for byte: the signature binds the record to where it was produced.

​
What happens to records that don’t verify

Gateway drops a record whose signature is missing, malformed or invalid, and emits an audit event naming the reason: missing_signature, malformed_signature, verification_failed:INVALID_SIGNATURE or verification_failed:UNKNOWN_KEY. Audit events need the audit feature enabled through GATEWAY_FEATURE_FLAGS_AUDIT. Records with no signature at all are the interesting case, and you choose what happens with missingSignaturePolicy:
  • SKIP, the default: drop the record and audit it.
  • ALLOW: keep the record in the response and audit it.
Start with ALLOW when you introduce signing to a topic that already has unsigned data or producers that don’t go through Gateway yet. The audit events tell you what would have been dropped, and you switch to SKIP once the stream is clean. Verification failures are different from operational failures. If Vault is unreachable on a cache miss, Gateway propagates the error to the Kafka client rather than dropping the record, so a key store outage doesn’t look like a stream of tampered records.

​
Ordering

The integrity Interceptors are always the outermost layer. Signing runs last on produce, after every other Interceptor has transformed the record, and verification runs first on fetch, before any transformation. So the signature always covers the payload as it was actually written, and it’s checked before anything touches it. Gateway enforces this itself. You don’t set priority values for integrity Interceptors, and Gateway rejects a configuration where two Interceptors of the same type have overlapping scopes.

​
Key management

Signing keys live in your HashiCorp Vault KV v2 instance, and Gateway reads and caches them. KV v2 versions each secret, and Gateway uses that to make rotation safe:
  • On produce, Gateway signs with the latest version at the path you configured, and picks up a new value once the cache entry expires.
  • Each signed record stores its key version in the signature header.
  • On fetch, Gateway reads that version from the header and fetches that exact version to verify, so records signed before a rotation still verify after it.
The consequence to plan for: keep older secret versions readable in Vault until you no longer have to verify records signed with them, which usually means until those records are consumed or past your retention period. Deleting an old version turns every record signed with it into verification_failed:UNKNOWN_KEY.

​
What signing doesn’t cover

A signature covers one record as it was produced. Anything that reads a record and writes a new one produces unsigned output, so a stream processor, a connector or an ETL job that transforms your data breaks the chain. The record downstream is a genuinely new record, not a modified one. This isn’t only about transformation. Since the signature binds the record to its topic and partition, a tool that copies or mirrors records into another topic breaks verification too, even when it changes nothing. If whatever writes the new record produces through Gateway, its output gets signed in its own right and the chain continues. If it’s wired directly to the brokers, its output has no signature, and a consumer set to SKIP drops it. So decide per topic rather than per cluster: signing fits topics whose producers you route through Gateway, which includes using the cluster as a transport between two points you control. From our blog: