In Chapter 19, we learned message queues.
Now we move to a closely related but broader concept:
Distributed event streaming.
A traditional work queue is often modeled as:
Producer
|
v
Queue
|
v
ConsumerA distributed event-streaming platform such as Apache Kafka is better understood as a durable, partitioned, append-only event log:
Producer
|
v
Topic
|
+---- Partition 0
+---- Partition 1
+---- Partition 2
|
v
ConsumersThe important difference is that events can remain available after one consumer processes them.
Multiple independent consumer groups can read the same event stream at their own pace.
This enables:
Apache Kafka is a distributed event-streaming platform.
A simplified mental model is:
Kafka Cluster
|
+---- Topic A
| |
| +---- Partition 0
| +---- Partition 1
|
+---- Topic B
|
+---- Partition 0
+---- Partition 1Producers append records to topics.
Consumers read records from topics.
Kafka stores records for a configured retention period.
A traditional work queue often emphasizes:
message
↓
one worker
↓
acknowledge
↓
remove/completeKafka emphasizes:
append event
↓
retain event
↓
consumers track positionFor example:
OrderCreated
OrderPaid
OrderShippedKafka can retain these events so different consumers can independently read them.
Suppose an e-commerce system creates an order.
Several systems need to react:
Order Service
|
v
OrderCreated
|
+----→ Inventory
+----→ Analytics
+----→ Notification
+----→ Recommendation
+----→ Fraud DetectionWithout an event stream, the Order Service might have to directly call every service.
That creates coupling.
With an event stream, the producer publishes an event and consumers subscribe independently.
The most important terms are:
| Concept | Meaning |
|---|---|
| Broker | Kafka server |
| Cluster | Group of Kafka brokers |
| Topic | Named event stream |
| Partition | Ordered log within a topic |
| Record | Individual message/event |
| Producer | Writes records |
| Consumer | Reads records |
| Consumer group | Consumers cooperating on a workload |
| Offset | Position of a record within a partition |
| Replication | Copies partition data across brokers |
| Leader | Broker currently serving writes/reads for a partition |
| Follower | Replica of a partition |
| ISR | In-sync replicas |
| Retention | How long Kafka keeps records |
These concepts form the foundation of Kafka.
A broker is a Kafka server.
A Kafka cluster may contain multiple brokers:
Kafka Cluster
Broker 1
Broker 2
Broker 3
Broker 4Different partitions are distributed across brokers.
This provides:
A topic is a named stream of records.
Examples:
orders
payments
user-events
notifications
logs
click-eventsConceptually:
Topic: orders
OrderCreated
OrderCreated
OrderPaid
OrderCreated
OrderShippedA topic is divided into partitions for scalability.
A partition is an ordered append-only log.
Example:
Partition 0
offset 0 → event A
offset 1 → event B
offset 2 → event C
offset 3 → event DThe order is guaranteed within this partition.
Kafka does not automatically provide one global order across all partitions of a topic.
Suppose a topic receives:
1 million events/secOne server may not be sufficient.
Split the topic:
Topic
|
+---- P0
+---- P1
+---- P2
+---- P3Different partitions can be stored and processed in parallel.
This is one of Kafka's fundamental scalability mechanisms.
A producer can choose a key.
Example:
key = userIdThe producer can use the key to determine the partition.
Conceptually:
partition = hash(key) % partitionCountThen:
User 101 → Partition 2
User 101 → Partition 2
User 101 → Partition 2This can preserve ordering for that user's events.
Suppose:
User 101
Login
UpdateProfile
LogoutIf all three records go to the same partition:
P2
Login
UpdateProfile
Logouttheir relative order is preserved.
At the same time:
User 202 → P0
User 303 → P1can be processed concurrently.
This is often better than forcing global ordering.
If all events go to one partition:
P0
1
2
3
4
5global ordering is straightforward.
But throughput and parallelism are limited.
Therefore:
more partitions
→ more parallelism
→ weaker global orderingThis is a key Kafka trade-off.
An offset identifies a record's position within a partition.
Example:
Partition 0
offset 0 → A
offset 1 → B
offset 2 → C
offset 3 → D
offset 4 → EA consumer can maintain:
current offset = 3This means it has progressed to a particular position in the partition.
Suppose the consumer has processed:
0
1
2and the next record is:
3The consumer can continue from that position.
This is different from a traditional queue where the broker often tracks completion by removing or acknowledging the message.
Kafka's log remains.
The consumer tracks its position.
A consumer group is a set of consumers working together.
Example:
Topic
|
+---- P0
+---- P1
+---- P2
+---- P3
Consumer Group A
|
+---- Consumer 1
+---- Consumer 2
+---- Consumer 3
+---- Consumer 4Each partition is assigned to a consumer in the group.
Suppose:
4 partitions
4 consumersA possible assignment:
C1 → P0
C2 → P1
C3 → P2
C4 → P3Now processing can happen in parallel.
Suppose:
4 partitions
8 consumersOnly four consumers can actively own partitions at a time within that consumer group.
The others may remain idle.
Therefore:
Maximum useful parallelism within a consumer group is constrained by partition count.
This is one of Kafka's most powerful ideas.
Suppose:
Topic: ordersConsumer group A:
InventoryConsumer group B:
AnalyticsConsumer group C:
NotificationsAll groups can independently consume the same events.
orders topic
|
+----------+----------+
| | |
v v v
Inventory Analytics Notifications
Group Group GroupThis provides fan-out.
Kafka can also behave like a work queue.
Suppose:
Topic
|
+---- P0
+---- P1
+---- P2One consumer group:
Worker 1
Worker 2
Worker 3Each partition is processed by one consumer in the group.
So the same Kafka topic can support different consumption models depending on how groups are configured.
Consumer lag measures how far behind a consumer group is.
Conceptually:
latest available offset
-
consumer processed position
=
lagExample:
Latest offset = 10,000
Consumer position = 9,700
Lag = 300If lag continuously increases, consumers cannot keep up.
A system may appear healthy at the broker level while consumers are falling behind.
For real-time applications, lag may directly translate into user-visible delay.
Examples:
fraud detection delayed
analytics delayed
notifications delayed
search index delayedMonitor both:
lag amount
+
lag growth rateA Kafka producer writes records.
Example event:
{
"eventType": "OrderCreated",
"orderId": "ORD-123",
"userId": "U-42",
"amount": 4999
}The producer chooses:
Partition selection may depend on the key.
Sending every record individually can be inefficient.
A producer can batch multiple records:
Event 1
Event 2
Event 3
Event 4
|
v
one batch
|
v
KafkaBenefits:
Trade-off:
Kafka producers can compress batches.
Common compression choices include:
Compression can reduce:
but consumes CPU.
Choose based on workload and infrastructure.
The producer needs to know how reliably the broker accepted a record.
Conceptually:
Producer
|
| record
v
Broker
|
| acknowledgement
v
ProducerDifferent acknowledgement settings trade off latency, durability, and failure behavior.
Kafka can replicate partitions.
Example:
Partition 0
Broker 1 → Leader
Broker 2 → Replica
Broker 3 → ReplicaReplication factor:
3means there are three copies according to the configured replication model.
If one broker fails, another replica may take over.
For a replicated partition:
Partition 0
Broker 1
Leader
Broker 2
Follower
Broker 3
FollowerThe leader handles the normal partition operations.
Followers replicate the leader's data.
ISR means:
In-Sync Replicas.
These are replicas considered sufficiently caught up with the partition leader according to Kafka's replication rules.
Example:
P0
Leader → Broker 1
ISR → Broker 1, Broker 2, Broker 3If Broker 3 falls too far behind, it may leave the ISR.
ISR → Broker 1, Broker 2ISR membership affects durability and failover decisions.
A replica that is not sufficiently synchronized should not be treated the same as a healthy in-sync replica.
This is especially important when discussing:
Suppose:
Broker 1 = Leader
Broker 2 = Follower
Broker 3 = FollowerBroker 1 crashes.
Kafka can elect another eligible replica as leader.
Conceptually:
Broker 1
X
Broker 2
|
v
New LeaderClients refresh metadata and continue using the new leader.
A system must choose how much acknowledgement and replication it requires.
More durability generally means:
more replicas / stronger acknowledgement
→ potentially higher latencyLower latency settings may accept more risk under failures.
System design is about choosing based on business requirements.
Kafka does not necessarily delete a record immediately after a consumer reads it.
Instead, records remain according to retention policies.
Retention can be based on:
timeor:
storage sizeExample:
retain for 7 daysAfter retention expires, old data can be removed.
Suppose:
Consumer A processed events yesterday.A new consumer starts today.
If the events are still retained, it can read older records.
This enables:
Replay is one of the major differences between a durable event log and a simple transient work queue.
A consumer may need to choose where to begin reading.
Possible concepts include:
earliest
latest
specific offsetFor example:
start from earliestmeans reading retained historical records.
Starting from the latest position means processing new events going forward.
Suppose analytics logic was incorrect.
Events:
Day 1
Day 2
Day 3
...After fixing the consumer:
Historical Kafka events
|
v
Corrected consumer
|
v
Rebuild analyticsThis can avoid requiring the original producer to regenerate every event.
A powerful mental model:
append
append
append
append
appendThe partition is an ordered log.
Consumers move through the log using offsets.
P0:
0 → A
1 → B
2 → C
3 → D
4 → EThis log abstraction is central to Kafka.
A useful distinction:
Something that happened.
OrderCreated
PaymentCompleted
UserRegisteredInstruction to perform something.
CreateOrder
SendEmail
ChargeCardEvent streams are especially natural for facts about things that have already happened.
A good event should be explicit.
Example:
{
"eventId": "evt_123",
"eventType": "OrderCreated",
"eventVersion": 2,
"occurredAt": "2026-10-01T10:00:00Z",
"producer": "order-service",
"payload": {
"orderId": "ORD-123",
"userId": "U-42"
}
}Useful metadata:
Consumers and producers are deployed independently.
Suppose version 1 contains:
{
"userId": "123"
}Version 2 adds:
{
"userId": "123",
"country": "IN"
}Adding an optional field is often easier to roll out safely than removing or changing the meaning of an existing field.
Event schemas should be designed for compatibility.
Large Kafka ecosystems often use a schema registry or an equivalent schema-management system.
Conceptually:
Producer
|
v
Schema Registry
|
v
Kafka
|
v
ConsumerIt can help manage:
Common serialization approaches include JSON, Avro, Protobuf, and JSON Schema.
When consumers join or leave a group, Kafka may redistribute partitions.
Example:
Before:
C1 → P0, P1
C2 → P2, P3C2 leaves.
Partitions may be reassigned:
C1 → P0, P1, P2, P3This redistribution is called rebalancing.
Rebalancing can temporarily affect processing.
Frequent consumer membership changes may cause:
Therefore consumers should be stable and correctly configured.
Consumers need to persist their progress.
Conceptually:
Processed offset = 500The consumer commits that position.
If the consumer crashes, it can resume from a known committed position.
There is an important trade-off.
If the consumer commits before processing:
commit
|
process
|
crashthe message may not be processed again.
Potentially:
message lost from consumer's workIf it commits after processing:
process
|
commit
|
crash before committhe message may be processed again.
Therefore:
commit after successful processing
+
idempotent processingis a common reliability strategy.
Consumers often fetch multiple records.
Example:
Batch:
A
B
C
D
EThe consumer processes them and commits progress.
Batching improves throughput but creates failure considerations.
If record D fails:
A ✓
B ✓
C ✓
D ✗
E ?The consumer must have a clear strategy for retry and commit boundaries.
Kafka supports transactional capabilities that can help coordinate certain producer operations.
Transactions can be useful when a workflow needs atomicity across a set of Kafka writes.
For example:
read input
|
process
|
write outputA transactional design can help prevent partially published output from becoming visible inappropriately.
However, transactions do not magically create a global transaction across arbitrary external databases and services.
Kafka supports mechanisms for stronger processing guarantees, but system design should distinguish:
broker semantics
consumer processing
external side effectsSuppose:
Kafka event
|
v
Consumer
|
v
External payment APIKafka's transaction guarantees cannot automatically make the external payment API exactly-once.
For external side effects, you may still need:
A producer may retry after an uncertain network outcome.
Without protection:
send event
|
v
broker stores
|
X
ACK lost
|
v
producer retriesThe same logical event might be written twice.
Idempotent producer mechanisms help prevent certain duplicate writes caused by producer retries.
Suppose:
Order DB
KafkaThe application wants:
save order
+
publish OrderCreatedA failure between those operations can cause inconsistency.
The transactional outbox pattern remains useful:
DB transaction
|
+---- Order
|
+---- Outbox event
|
v
Outbox publisher
|
v
KafkaThis separates database atomicity from Kafka publishing.
Change Data Capture (CDC) captures database changes.
Conceptually:
Database
|
v
CDC Tool
|
v
Kafka
|
+---- Analytics
+---- Search
+---- Data Lake
+---- Other ServicesThis can reduce tightly coupled application-level integration.
Event sourcing stores business state as a sequence of events.
Instead of only storing:
Account balance = 500the system stores:
Deposit 1000
Withdraw 300
Withdraw 200Current state can be derived from the event history.
Conceptually:
Events
|
v
Projection
|
v
Current StateKafka can be used as part of event-driven architectures involving event sourcing, but Kafka itself does not automatically make a system event-sourced.
CQRS means:
Command Query Responsibility Segregation.
Separate:
write modelfrom:
read modelExample:
Commands
|
v
Write Service
|
v
Events
|
v
Kafka
|
+----→ Search Projection
|
+----→ Analytics Projection
|
+----→ Read ModelDifferent consumers build specialized read models.
Kafka Streams is a stream-processing approach for applications that process Kafka data.
Examples:
orders
|
v
filter
|
v
aggregate
|
v
sales-by-regionOperations may include:
The important idea is:
event stream
→ continuous computation
→ derived stream/stateSuppose events arrive:
purchase
purchase
purchase
purchaseA streaming application can continuously compute:
sales per minuterather than waiting for a nightly batch job.
This enables:
Some analytics need time windows.
Example:
How many purchases occurred
during the last 5 minutes?A streaming system can use a time window:
10:00–10:05
10:05–10:10
10:10–10:15Windowing introduces additional questions around:
These are advanced stream-processing concepts.
When the event actually occurred.
event.occurredAtWhen the consumer processed it.
worker.now()They can differ significantly.
Example:
Event occurred: 10:00
Arrived: 10:05
Processed: 10:06Real-time analytics often needs to reason about event time.
Suppose a purchase happened at:
10:01but arrives at:
10:07A window for 10
–10 may already have been calculated.The system needs a strategy for late data.
Possible approaches include:
Applications can emit logs/events:
Service A
Service B
Service C
Service D
|
v
Kafka
|
v
Log Processing
|
+---- Search
+---- Storage
+---- AlertingKafka can act as a durable buffer between applications and downstream log-processing systems.
Example:
Applications
|
v
Kafka
|
+---- Stream Processor
|
+---- Data Warehouse
|
+---- Data LakeKafka becomes the event backbone.
Different consumers can process the same source stream for different purposes.
Example:
Product DB
|
v
ProductChanged
|
v
Kafka
|
v
Search Indexer
|
v
Search EngineThe database remains the system of record.
Kafka transports changes.
The search engine becomes a derived read model.
Order Service
|
v
Kafka
|
v
Notification Consumer
|
+---- Email
+---- SMS
+---- PushThis allows notification processing to scale independently.
Transactions
|
v
Kafka
|
v
Fraud Stream Processor
|
+---- rules
+---- aggregates
+---- model scoring
|
v
Risk DecisionStreaming enables decisions shortly after events arrive.
A broker fails.
Potential response:
replica becomes leaderConsumer crashes.
Potential response:
partition reassignedProducer retries.
Potential issue:
duplicate sendUse idempotent producer mechanisms where appropriate.
Message may be retried depending on consumer design.
Need:
idempotency
retry
DLQ strategySuppose:
P0 leader → Broker 1
P0 replica → Broker 2
P0 replica → Broker 3Broker 1 fails.
An eligible replica can become leader.
After recovery, the original broker can catch up.
Replication and leader-election behavior are critical to availability.
If replicas fall behind or become unavailable, a partition can become under-replicated.
Monitor:
under-replicated partitions
offline partitions
ISR shrink/expansionPersistent under-replication is an operational warning sign.
Kafka can scale through:
more brokers
more partitions
consumer parallelism
producer batching
compression
storage capacity
network capacityBut scaling one dimension can expose another bottleneck.
For example:
more partitions
→ more consumer parallelism
→ more metadata / operational overheadPartition count is therefore an architectural decision.
Consider:
target throughput
consumer parallelism
expected growth
ordering requirements
broker capacity
retention volumeA common mistake is choosing too few partitions initially and discovering later that consumer parallelism is insufficient.
But excessively high partition counts also create operational overhead.
A good key should generally provide:
Bad key:
countryif:
India = 80% trafficPotential result:
one hot partitionBetter key might be:
userId
accountId
orderIddepending on business requirements.
Example:
Partition 0 → 80,000 events/sec
Partition 1 → 5,000
Partition 2 → 5,000
Partition 3 → 5,000Total throughput may look acceptable, but Partition 0 becomes the bottleneck.
This is why average throughput can hide partition-level problems.
Suppose all events for one extremely active account must remain ordered.
Using:
accountIdas the key preserves order but may create a hot partition.
Possible solutions depend on requirements.
If strict ordering is not required across all events, the workload can sometimes be further partitioned.
If strict ordering is mandatory, the hot key may remain a fundamental limit.
This is a business requirement vs scalability trade-off.
Production Kafka deployments should consider:
authentication
authorization
TLS
encryption
network isolation
secret management
audit logging
topic-level permissions
consumer/producer permissionsDo not allow every application unrestricted access to every topic.
Use least privilege.
Monitor at least:
CPU
memory
disk
network
request latencythroughput
partition size
replication health
under-replicationsend rate
error rate
retry rate
latency
batch sizelag
processing rate
errors
rebalance frequency
commit latencyA useful alert may consider:
lag > threshold
AND
lag increasingbecause a static lag of 1,000 may be acceptable for one workload but dangerous for another.
Business SLA matters.
For example:
fraud detection:
seconds matter
daily analytics:
minutes or hours may be acceptableKafka cost includes:
broker compute
disk
network
replication
retention
cross-region traffic
operations
monitoringReplication factor 3 means additional storage and network overhead.
Long retention also increases storage requirements.
Compression can reduce storage and network usage.
Global systems may need regional Kafka clusters.
Example:
Region A
Kafka A
|
v
Region B
Kafka BPossible architectures include:
Cross-region replication introduces:
Design according to business requirements.
Kafka is not a replacement for a general-purpose transactional database.
Database:
query current state
update records
transactions
constraints
relationshipsKafka:
append events
stream events
retain history
replay
fan-outThey often work together:
Application
|
+----→ Database
|
+----→ KafkaBoth can support stream-like workloads, but they have different architectural characteristics.
Kafka is commonly chosen for:
Redis Streams may be useful when:
Choose based on requirements rather than brand preference.
A simplified comparison:
| Concern | Traditional Queue | Kafka |
|---|---|---|
| Main abstraction | Work queue | Distributed log |
| Consumption | Often removes/completes work | Consumers track offsets |
| Replay | Often limited | Natural with retention |
| Fan-out | Usually needs pub/sub | Consumer groups |
| Partitioning | Depends on system | Core scalability mechanism |
| Ordering | Queue-specific | Per-partition |
| Retention | Often shorter | Configurable durable retention |
| Streaming | Possible | Core use case |
Actual guarantees vary by technology and configuration.
Thinking Kafka is just a faster queue.
Assuming topic-wide ordering.
Ignoring partition-key design.
Adding consumers beyond partition parallelism and expecting unlimited scaling.
Ignoring consumer lag.
Assuming Kafka transactions make external APIs exactly-once.
Using Kafka as a replacement for every database.
Putting giant payloads directly into Kafka.
Ignoring schema evolution.
Ignoring rebalancing behavior.
Choosing too many or too few partitions without capacity planning.
Ignoring replication and under-replicated partitions.
When asked:
Design a Kafka-based architecture.
Start with:
What events?
How many events/sec?
Average event size?
Peak traffic?
How long should events be retained?
How much ordering is required?
How many consumers?
Need replay?
Need real-time processing?
Need multiple independent consumers?
What are the durability requirements?
Single region or multi-region?Then design:
Producers
|
v
Kafka Cluster
|
+---- Topic
| |
| +---- Partitions
|
+---- Consumer Groups
|
+---- WorkersThen discuss:
partition key
replication
acks
offsets
consumer lag
rebalancing
retry
DLQ
idempotency
schema evolution
security
monitoringPartitions provide horizontal scalability and parallelism. They allow a topic's data to be distributed across brokers and processed concurrently by consumers. Ordering is maintained within a partition, not automatically across the entire topic.
Kafka preserves record order within a partition. To preserve order for a business entity, use a stable partition key such as user ID or order ID so related events are routed to the same partition.
Consumer lag is the difference between the latest available position in a partition and the consumer group's processed/committed position. Growing lag indicates that consumers are falling behind the incoming workload.
Kafka retains records according to configured retention policies, while consumers track their own offsets. A consumer can therefore start from an earlier retained position and process historical events again.
A consumer group is a set of consumers that cooperate to process partitions of a topic. Within a group, a partition is normally assigned to one consumer at a time, enabling horizontal work distribution.
Kafka can provide:
decoupling
buffering
asynchronous processing
replay
fan-out
independent consumer scalingDirect HTTP calls may still be better when:
the caller needs an immediate responseThe decision depends on communication semantics.
Build a small event-driven order platform.
Order API
|
v
Kafka
|
+---- Inventory Consumer
|
+---- Notification Consumer
|
+---- Analytics ConsumerImplement events:
OrderCreated
OrderPaid
OrderCancelled
OrderShippedUse:
orderIdas the partition key so one order's lifecycle remains ordered.
Implement:
POST /ordersPublish:
OrderCreatedConsumes:
OrderCreatedConsumes:
OrderCreated
OrderShipped
OrderCancelledConsumes all order events.
Store:
orders
order_events
analyticsin appropriate persistence layers.
Add:
consumer groups
multiple partitions
consumer lag monitoring
retry topic
dead-letter topic
idempotency
schema version
outbox pattern
event replay
metrics
distributed tracingThen simulate:
consumer crash
broker failure
duplicate event
slow consumer
traffic spike
hot partition
schema changeObserve how the architecture behaves.
Before calling a Kafka architecture production-ready:
[ ] Partition strategy
[ ] Partition key defined
[ ] Capacity planned
[ ] Replication configured
[ ] Durability requirements defined
[ ] Producer acknowledgement strategy
[ ] Idempotent producer where needed
[ ] Consumer groups designed
[ ] Offset strategy
[ ] Consumer lag monitoring
[ ] Retry strategy
[ ] DLQ strategy
[ ] Idempotent consumers
[ ] Schema evolution
[ ] Replay strategy
[ ] Retention policy
[ ] Security
[ ] Encryption
[ ] Monitoring
[ ] Alerting
[ ] Backup/recovery plan
[ ] Multi-region requirements
[ ] Cost modelThink about Kafka as:
Kafka Cluster
|
+---------+---------+
| | |
v v v
Topic Topic Topic
|
+----+----+----+
| | | |
v v v v
P0 P1 P2 P3
| | | |
+----+----+----+
|
Consumer Groups
/ | \
v v v
Analytics Inventory NotificationsThe most important relationship is:
Topic
↓
Partitions
↓
Offsets
↓
Consumer Groups
↓
Independent processingThe key concepts from this chapter are:
The core principle is:
Partition for scale.
Key for ordering.
Offset for position.
Replication for durability.
Consumer groups for parallelism.
Retention for replay.
Idempotency for correctness.You should now be able to explain:
In the next chapter we will move from Kafka mechanics to the architecture style built around events.
We will learn:
Mark it as complete to track your mastery.