How to send large messages in Apache Kafka

Apache Kafka has default message size limits that can be configured to handle larger payloads, but there are important considerations and best practices to follow.

Default message size limits

By default, Kafka has the following message size limits:

• Producer: 1MB (max.request.size)
• Broker: 1MB (message.max.bytes)
• Topic: Inherits from broker setting (max.message.bytes)
• Consumer: 1MB (max.partition.fetch.bytes)

Configuring Kafka for large messages

To send messages larger than 1MB, you need to configure multiple components:

Producer configuration

# Set maximum request size for producer
max.request.size=10485760  # 10MB
# Increase buffer memory if needed
buffer.memory=67108864     # 64MB

Broker configuration

# Set maximum message size for broker
message.max.bytes=10485760          # 10MB
# Set maximum replica fetch size
replica.fetch.max.bytes=10485760    # 10MB
# Set maximum response size
socket.receive.buffer.bytes=1048576 # 1MB
socket.send.buffer.bytes=1048576    # 1MB

Topic configuration

kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name large-topic \
  --add-config max.message.bytes=10485760

Consumer configuration

# Set maximum fetch size for consumer
max.partition.fetch.bytes=10485760  # 10MB
fetch.max.bytes=52428800            # 50MB

Performance implications

Sending large messages in Kafka has several performance implications:

Memory usage

• Larger messages consume more memory on brokers, producers, and consumers
• Can lead to increased garbage collection pressure
• May require tuning JVM heap sizes

Network bandwidth

• Large messages consume more network bandwidth
• Can lead to network congestion and timeouts
• May require adjusting network buffer sizes

Disk I/O

• Larger messages result in more disk I/O operations
• Can impact log compaction performance
• May require faster storage systems

Throughput impact

• Large messages generally reduce overall throughput
• Kafka is optimized for high-throughput, small messages
• Consider message batching strategies

Alternative approaches

Instead of sending large messages directly, consider these alternatives:

1. External storage pattern

Store large payloads in external systems and send only references:

{
  "id": "message-123",
  "timestamp": "2023-01-01T00:00:00Z",
  "data_location": "s3://bucket/path/to/large-file.json",
  "metadata": {
    "size": 50000000,
    "checksum": "abc123"
  }
}

Benefits:

• Keeps Kafka messages small and fast
• Allows for separate scaling of storage and messaging
• Enables efficient caching strategies

2. Message splitting

Break large messages into smaller chunks:

{
  "message_id": "msg-123",
  "chunk_id": "chunk-1",
  "total_chunks": 5,
  "chunk_data": "...",
  "sequence": 1
}

Benefits:

• Works within default Kafka limits
• Allows for parallel processing
• Provides better error recovery

3. Compression

Enable compression to reduce message sizes:

# Producer compression
compression.type=snappy  # or gzip, lz4, zstd

Benefits:

• Reduces network bandwidth usage
• Decreases storage requirements
• Often improves throughput

Best practices

Recommendations for large messages

1. Avoid large messages when possible - Kafka is optimized for small, high-throughput messages
2. Use external storage - Store large payloads externally and reference them in Kafka messages
3. Enable compression - Always enable compression for large messages
4. Monitor memory usage - Ensure adequate heap sizing for all components
5. Test thoroughly - Verify performance impact in your specific environment

Configuration checklist

When configuring for large messages, ensure all these settings are aligned:

• ✅ Producer max.request.size
• ✅ Broker message.max.bytes
• ✅ Topic max.message.bytes
• ✅ Consumer max.partition.fetch.bytes
• ✅ Consumer fetch.max.bytes
• ✅ Broker replica.fetch.max.bytes

Monitoring considerations

Monitor these metrics when working with large messages:

• Memory usage on brokers, producers, and consumers
• Network bandwidth utilization
• Disk I/O patterns and latency
• Garbage collection frequency and duration
• Message throughput and latency

Performance impact

Large messages can significantly impact Kafka performance. Always test in a staging environment that mirrors your production setup before deploying large message configurations.