Kafka consumers can be configured to read from the closest replica rather than always reading from the leader, which can improve performance and reduce cross-datacenter network traffic in geographically distributed deployments.
How replica reading works
By default, Kafka consumers always read from the partition leader. However, starting with Kafka 2.4, consumers can be configured to read from follower replicas that are “close” to the consumer.
Default behavior (leader-only reads)
- All consumers read from partition leaders
- Followers only replicate data, never serve reads
- Simple and consistent behavior
- May result in cross-datacenter traffic
Closest replica reading
- Consumers can read from geographically closest replicas
- Reduces network latency and cross-datacenter bandwidth
- Requires proper rack awareness configuration
- Maintains consistency guarantees
Configuration
Enable closest replica reading
# Consumer configuration
client.rack=us-west-2a
# This tells Kafka which rack/availability zone the consumer is in
# Kafka will prefer replicas in the same rack when available
Broker configuration for rack awareness
# Broker configuration (server.properties)
broker.rack=us-west-2a
# Each broker should be configured with its rack/AZ
# This enables Kafka to make intelligent replica placement decisions
Topic configuration
When creating topics, consider replica placement:
# Create topic with rack-aware replica assignment
kafka-topics --bootstrap-server localhost:9092 \
--create --topic my-topic \
--partitions 6 \
--replication-factor 3 \
--config min.insync.replicas=2
Benefits
Reduced latency
- Consumers read from local replicas instead of remote leaders
- Eliminates cross-datacenter read traffic
- Lower response times for geographically distributed applications
Cost savings
- Reduces expensive cross-region data transfer
- Particularly beneficial in cloud environments with data transfer charges
- Optimizes bandwidth usage in WAN scenarios
Improved availability
- Continues reading even if cross-datacenter links are degraded
- Better resilience in network partition scenarios
- Maintains read availability during datacenter issues
Consistency considerations
Read-after-write consistency
With closest replica reading, you may encounter scenarios where:
- Producer writes to leader in datacenter A
- Consumer reads from follower in datacenter B
- Replication lag may cause temporary inconsistency
Mitigation strategies
# Ensure minimum in-sync replicas for writes
min.insync.replicas=2
# Use appropriate acks setting
acks=all
# Configure consumer to handle potential inconsistencies
Use cases
Multi-region deployments
Region US-West:
- Brokers with rack=us-west
- Consumers with client.rack=us-west
- Reads stay local to us-west replicas
Region EU-Central:
- Brokers with rack=eu-central
- Consumers with client.rack=eu-central
- Reads stay local to eu-central replicas
Availability zone optimization
AZ-1: broker.rack=az-1, client.rack=az-1
AZ-2: broker.rack=az-2, client.rack=az-2
AZ-3: broker.rack=az-3, client.rack=az-3
Each AZ reads from local replicas when possible
Monitoring and observability
Key metrics to monitor
- Replica read rates: Track reads from leaders vs followers
- Cross-datacenter traffic: Monitor reduction in inter-region bandwidth
- Consumer lag by replica: Ensure follower replicas are keeping up
- Replication lag: Monitor lag between leader and follower replicas
JMX metrics
kafka.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*,partition=*
Configuration examples
Cloud deployment (AWS)
# Broker configuration
broker.rack=${aws.availability.zone}
# Consumer configuration
client.rack=us-west-2a
# Additional consumer settings for optimal performance
fetch.min.bytes=1048576
fetch.max.wait.ms=500
On-premises multi-datacenter
# Broker configuration
broker.rack=datacenter-1
# Consumer configuration
client.rack=datacenter-1
# Network optimization
socket.receive.buffer.bytes=65536
fetch.max.bytes=52428800
Best practices
Deployment guidelines
- Configure rack awareness on all brokers and consumers
- Ensure sufficient replicas in each rack/region
- Monitor replication lag to avoid stale reads
- Test failover scenarios to ensure proper behavior
# Optimize for closest replica reading
client.rack=your-rack-id
fetch.min.bytes=1024
max.poll.records=500
# Monitor and tune based on your specific latency requirements
Consistency requirements
- Strong consistency needs: Consider leader-only reads
- Eventually consistent OK: Closest replica reading works well
- Mixed requirements: Use different consumer groups with different configurations
Replication lag impactWhen reading from follower replicas, consumers may see slightly stale data due to replication lag. Ensure your application can tolerate this eventual consistency model.
Gradual rolloutConsider implementing closest replica reading gradually:
- Start with non-critical consumer groups
- Monitor metrics and consistency behavior
- Expand to more critical workloads as confidence builds
Troubleshooting
Common issues
- Missing rack configuration: Ensure both brokers and consumers have rack settings
- Insufficient replicas: Need replicas in consumer’s rack for local reads
- High replication lag: Follower replicas falling behind leader
- Network configuration: Ensure proper connectivity between racks/regions
Verification steps
# Check broker rack configuration
kafka-configs --bootstrap-server localhost:9092 --describe --entity-type brokers
# Monitor consumer metrics
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group your-group
# Check topic replica distribution
kafka-topics --bootstrap-server localhost:9092 --describe --topic your-topic
