In the previous chapters, we learned:
Message Queues
↓
Kafka
↓
Event StreamingNow we move one level higher.
The technology is not the architecture.
Kafka, RabbitMQ, SQS, Pub/Sub, Redis Streams, and similar systems are infrastructure.
Event-Driven Architecture (EDA) is an architectural style in which components communicate and coordinate using events.
The central idea is:
A system publishes facts about things that happened, and other components react to those facts.
Example:
Order Service
|
| OrderCreated
v
Event Broker
|
+------→ Inventory
|
+------→ Payment
|
+------→ Notification
|
+------→ AnalyticsThis architecture can reduce direct coupling between services.
An event represents something that has already happened.
Examples:
UserRegistered
OrderCreated
PaymentCompleted
PaymentFailed
ItemShipped
VideoUploaded
InvoiceGeneratedNotice the naming:
OrderCreatedrather than:
CreateOrderThe first describes a fact.
The second describes an instruction.
This distinction is extremely important.
A fact:
OrderCreatedMeaning:
An order was created.
An instruction:
CreateOrderMeaning:
Please create an order.
A request for information:
GetOrderMeaning:
Please give me order information.
Mental model:
Command → Do something
Event → Something happened
Query → Give me informationImagine an Order Service directly calling:
Inventory
Payment
Notification
Analytics
Recommendation
FraudArchitecture:
Inventory
^
|
Payment ← Order → Notification
|
v
AnalyticsThe Order Service becomes aware of many downstream systems.
This creates coupling.
With events:
Order Service
|
v
OrderCreated
|
+----→ Inventory
+----→ Payment
+----→ Notification
+----→ AnalyticsThe producer only needs to publish the event.
New consumers can be added without necessarily changing the original producer.
A typical event-driven architecture contains:
Event Producer
|
v
Event Broker
|
v
Event ConsumerAdditional components may include:
Schema Registry
Dead-Letter Queue
Retry System
Event Store
Stream Processor
Monitoring
TracingThe producer creates an event.
Example:
Order ServiceAfter successfully creating an order:
OrderCreatedThe producer should publish meaningful business facts rather than exposing internal implementation details.
A consumer reacts to an event.
Example:
OrderCreated
|
v
Inventory ServiceThe Inventory Service might reserve inventory.
Another consumer:
OrderCreated
|
v
Notification Servicemay send an email.
Consumers should generally not assume they are the only consumer.
The broker transports and often stores events.
Examples of technologies include:
Kafka
RabbitMQ
Amazon SQS/SNS
Google Pub/Sub
Azure Service Bus
Redis StreamsThe correct technology depends on:
A typical flow:
User
|
v
Order API
|
v
Order Service
|
+----→ Database
|
+----→ OrderCreated
|
v
Broker
|
+------+------+
| | |
v v v
Inventory Payment NotificationEach consumer can evolve independently.
The simplest event style is event notification.
Example:
OrderCreatedThe event tells consumers:
Something happened.
The consumer may then query another service for additional information.
For example:
OrderCreated
|
v
Inventory Service
|
v
GET order detailsThis is simple, but it can introduce additional network calls.
Instead of sending only:
OrderCreatedthe event can carry relevant data:
{
"eventType": "OrderCreated",
"orderId": "ORD-123",
"userId": "U-42",
"items": [
{
"productId": "P-100",
"quantity": 2
}
],
"total": 4999
}Consumers may not need to call the Order Service.
Benefits:
Trade-off:
| Approach | Event Contains | Consumer Dependency |
|---|---|---|
| Notification | Minimal fact | May call producer |
| State transfer | Relevant state | More independent |
Use the smallest event that satisfies the consumer requirements without creating unnecessary coupling.
Choreography means services react to events independently.
Example:
OrderCreated
|
+----→ Inventory
|
+----→ Payment
|
+----→ NotificationThere is no central coordinator telling every service what to do.
Each service decides how to react.
Order workflow:
OrderCreated
|
v
Inventory
|
| InventoryReserved
v
Payment
|
| PaymentCompleted
v
Shipping
|
| ShipmentCreated
v
NotificationThe workflow emerges from event reactions.
Benefits:
However, choreography becomes difficult when workflows become highly complex.
Imagine:
Order
↓
Inventory
↓
Payment
↓
Fraud
↓
Shipping
↓
Notification
↓
Loyalty
↓
AnalyticsWith many events and services, understanding the complete workflow becomes difficult.
Potential problems:
This is sometimes called a distributed business workflow becoming hard to reason about.
Orchestration introduces a central workflow coordinator.
Example:
Order Orchestrator
/ | \
v v v
Inventory Payment ShippingThe orchestrator decides what happens next.
For example:
1. Reserve inventory
2. Charge payment
3. Create shipment
4. Send confirmation| Choreography | Orchestration |
|---|---|
| Services react to events | Coordinator controls workflow |
| Decentralized | Central workflow logic |
| Less central coupling | Coordinator knows participants |
| Can become hard to trace | Easier workflow visibility |
| Good for independent reactions | Good for complex workflows |
Neither is universally superior.
The right choice depends on workflow complexity and ownership.
Order Orchestrator
|
+------------+------------+
| | |
v v v
Inventory Payment Shipping
| | |
+------------+------------+
|
v
CompletedThe orchestrator can handle:
A Saga coordinates a distributed business transaction across multiple services.
Why?
Because a normal database transaction cannot easily span:
Order DB
Inventory DB
Payment DB
Shipping DBA Saga breaks the business transaction into local transactions.
Example:
Create Order
↓
Reserve Inventory
↓
Charge Payment
↓
Create ShipmentIf something fails, compensating actions may undo previous business effects.
Suppose:
Order Created ✓
Inventory Reserved ✓
Payment Charged ✓
Shipping Failed ✗A compensation could be:
Release Inventory
Refund Payment
Cancel OrderThe important idea is:
A Saga uses a sequence of local transactions and compensating actions rather than one global database transaction.
In choreography-based Saga:
OrderCreated
|
v
Inventory
|
InventoryReserved
|
v
Payment
|
PaymentCompleted
|
v
ShippingEach service reacts to events.
Failure events trigger compensating reactions.
In orchestration-based Saga:
Saga Orchestrator
|
+---------+---------+
| | |
v v v
Inventory Payment ShippingThe orchestrator controls the sequence and compensation.
This can be easier to understand for complex workflows.
Sagas do not provide the same atomicity as a single database transaction.
There can be periods of intermediate state.
Example:
Payment = SUCCESS
Shipping = PENDINGDuring that time, the system is temporarily inconsistent from a global perspective.
Therefore business workflows must be designed to tolerate intermediate states.
Event-driven systems often use eventual consistency.
Example:
Order DB
|
v
OrderCreated
|
v
Search IndexThere may be a short delay:
Order exists
but
search index does not show it yetEventually:
event processed
→ search updatedThis is eventual consistency.
Strong consistency aims for:
write
↓
read
↓
immediately sees latest stateEventual consistency allows:
write
↓
temporary stale state
↓
event processing
↓
latest state appearsThe choice depends on business requirements.
Examples:
Where it may be harder to accept:
The exact consistency requirement must be derived from the business operation.
Events can be duplicated.
Example:
OrderCreated
OrderCreatedA consumer must not accidentally:
charge customer twiceUse:
eventId
+
idempotency record
+
transactional business operationExample:
Event ID = evt_123Consumer checks:
Has evt_123 already been processed?If yes:
skipIf no:
process
record evt_123The check and business state update should be designed carefully to avoid races.
Suppose:
Consumer processes event
|
v
Database commit succeeds
|
X
ACK/offset commit fails
|
v
Event delivered againThis is normal in at-least-once systems.
The correct response is not:
Assume duplicates never happen.
Instead:
Make the business operation safe when duplicates happen.
Some workflows require order.
Example:
OrderCreated
OrderPaid
OrderShipped
OrderDeliveredAn event arriving out of order can cause incorrect state.
Possible approaches:
Events evolve.
Version 1:
{
"orderId": "123",
"amount": 500
}Version 2:
{
"orderId": "123",
"currency": "INR",
"amount": 500
}A robust event architecture supports compatible evolution.
Possible strategies:
add optional fieldsor:
OrderCreated.v2depending on the ecosystem.
A new consumer may need to read older events.
A new producer may need to coexist with old consumers.
Therefore event contracts should be treated as APIs.
Consider:
Producer
|
v
Event Contract
|
v
Many ConsumersChanging the event schema can affect many systems.
Large organizations may need:
Events become shared contracts between teams.
A common architectural failure is creating too many events.
Example:
UserNameCharacterChanged
UserNameValidated
UserNameNormalized
UserNameStored
UserProfileUpdatedNot every internal state transition needs to become a public event.
Prefer meaningful business events.
Too coarse:
OrderUpdatedConsumers may not know what changed.
Too fine:
OrderItemQuantityFieldChangedcan create excessive event volume.
A good event represents a meaningful business fact.
Good:
OrderCreated
PaymentCompleted
ShipmentDispatched
UserRegisteredPotentially confusing:
ProcessOrder
DoPayment
UpdateSomethingUse past-tense factual naming for events.
Every important event should have a clear owner.
For example:
Order Service
owns OrderCreatedOther services consume it.
Avoid multiple services independently claiming ownership of the same source-of-truth event.
Suppose:
Order DB
Search Index
Cache
Analytics DBWhich one is authoritative?
Usually one system should be the source of truth.
Example:
Order DB = source of truth
Search = derived
Cache = derived
Analytics = derivedEvents propagate changes to derived systems.
Example:
Order DB
|
| OrderCreated
v
Kafka
|
+----→ Search Index
+----→ Analytics DB
+----→ Cache UpdaterThese consumers build derived state.
If derived state becomes corrupted, event replay may allow rebuilding it.
Traditional CRUD:
Orders table
----------------
order_id
status
amountThe current state is stored directly.
Event sourcing instead stores events:
OrderCreated
PaymentCompleted
OrderShipped
OrderDeliveredThe current state is derived from the event history.
Example:
Events:
OrderCreated
OrderPaid
OrderShippedProjection:
status = SHIPPEDThe event stream becomes the historical record.
Potential benefits:
But it introduces complexity.
Challenges include:
Do not choose event sourcing merely because Kafka is available.
CQRS separates:
Command sidefrom:
Query sideExample:
Commands
|
v
Write Model
|
v
Events
|
v
Broker
|
+---------+---------+
| | |
v v v
Search Read DB AnalyticsQueries can use optimized read models.
Suppose writes require:
normalized relational modelbut reads require:
fast denormalized viewCQRS allows separate models.
Example:
Write DB:
normalized
Read DB:
precomputedEvents connect them.
These concepts are related but different.
You can have:
CQRS without Event Sourcingand:
Event Sourcing without full CQRSThey are often combined, but they are not synonyms.
Event-driven architecture can reduce synchronous dependencies between microservices.
Instead of:
A → B → C → Dyou may have:
A → Event
|
+→ B
+→ C
+→ DBut event-driven architecture does not automatically mean:
microservicesEDA can exist in modular monoliths and other architectures too.
Real systems often use both.
Example:
Client
|
| synchronous
v
Order API
|
+---- synchronous → Order DB
|
+---- asynchronous → Event Broker
|
+→ Notifications
+→ Analytics
+→ SearchDo not force everything through events.
Use synchronous calls when an immediate response is required.
Use asynchronous events when decoupling and eventual processing are appropriate.
Consider food delivery.
OrderCreated
|
v
Restaurant Service
|
RestaurantAccepted
|
v
Payment Service
|
PaymentCompleted
|
v
Delivery Service
|
DriverAssigned
|
v
OrderPickedUp
|
v
OrderDeliveredEach event represents a business state transition.
Suppose:
PaymentCompleted
|
v
Delivery Service
|
X
failThe payment should not necessarily be repeated.
The system can retry the delivery operation independently.
This demonstrates a key benefit of decoupled processing.
Consumers should classify failures.
Transient:
timeout
temporary dependency outage
rate limitRetry:
exponential backoff
+
jitterPermanent:
invalid event
unsupported schema
invalid business stateMove to:
DLQor a controlled error workflow.
A DLQ can contain:
event
error
consumer
attempt count
timestamp
trace IDExample:
{
"eventId": "evt_123",
"consumer": "shipping-service",
"attempts": 5,
"error": "Invalid address",
"failedAt": "2026-10-01T10:00:00Z"
}This makes operational debugging easier.
A poison event can repeatedly crash a consumer.
Example:
invalid schemaIf the consumer retries indefinitely:
consumer
↓
failure
↓
retry
↓
failure
↓
retryThe entire partition may be blocked depending on the processing design.
Use controlled retries and dead-letter strategies.
Suppose ordered processing requires:
Event 1
Event 2
Event 3If Event 2 fails permanently:
Event 1 ✓
Event 2 ✗
Event 3 ?You cannot simply process Event 3 if ordering is required.
This creates a trade-off between:
strict orderingand:
availability / throughputThe business requirement determines the correct strategy.
Users may see temporary differences.
Example:
Order API says:
Order created
Search page:
Order not foundThis can be acceptable if the UI communicates processing state appropriately.
Possible patterns:
processing
pending
syncingGood system design considers the user experience of eventual consistency rather than hiding it.
An event-driven system can be difficult to debug.
Example:
HTTP request
|
v
Order Service
|
v
Kafka
|
v
Payment Consumer
|
v
Payment APIUse:
traceId
correlationId
eventIdto connect the workflow.
Monitor:
publish rate
consume rate
event failures
event agelag
processing latency
retry count
DLQ countOrderCreated → PaymentCompleted latency
PaymentCompleted → ShipmentCreated latencyBusiness-level metrics are often more useful than infrastructure metrics alone.
Suppose:
PaymentCompleted → ShipmentCreatedmust happen within:
30 secondsMonitor the actual workflow latency.
A system can have:
Kafka healthy
CPU healthy
memory healthywhile still violating the business SLA because a consumer is slow.
Security considerations include:
Do not publish secrets or unnecessary personal data into long-retention event streams.
Event streams can retain historical data.
Suppose an event contains:
user email
phone
addressand retention is:
7 yearsA deletion request may be difficult if personal data is embedded throughout immutable historical events.
Therefore privacy requirements must influence event design.
Possible strategies:
Replay is powerful but dangerous.
Suppose replaying:
PaymentCompletedcauses:
charge cardagain.
That would be incorrect.
Therefore consumers should distinguish between:
rebuildable projectionsand:
irreversible external side effectsReplay should be designed carefully.
A projection consumer might safely replay:
OrderCreated
OrderPaid
OrderShippedto rebuild:
search indexBut a payment consumer should not blindly replay historical events and charge customers again.
This distinction is essential in event-driven architectures.
Possible strategies:
Add optional fields.
OrderCreated.v1
OrderCreated.v2OrderCreated
OrderCreatedWithPricingThe organization should have a consistent approach.
Because many consumers depend on events, producer changes can break downstream services.
Contract tests can validate:
producer schema
vs
consumer expectationsThis is especially important in large microservice environments.
A common architecture:
API Gateway
|
v
Order Service
|
+------+------+
| |
v v
Order DB Event Bus
|
+---------------+---------------+
| | |
v v v
Inventory Payment Notification
| | |
v v v
Database Provider ProviderEach service owns its own state.
Events communicate changes.
In a microservice architecture:
Order Service → Order DB
Payment Service → Payment DB
Inventory Service → Inventory DBDo not casually allow every service to directly modify another service's database.
Instead:
Service A
|
v
Event
|
v
Service Bor use an explicit API where synchronous interaction is required.
Suppose:
Order DB
Payment DB
Inventory DBYou cannot assume a single local transaction covers all three.
Instead:
Order transaction
↓
Event
↓
Inventory transaction
↓
Event
↓
Payment transactionThis introduces eventual consistency and potentially Saga-style compensation.
Start:
Create OrderThen:
Reserve Inventory
↓
Charge Payment
↓
Create ShipmentFailure:
Shipment failsCompensate:
Refund Payment
↓
Release Inventory
↓
Cancel OrderThis is a classic distributed workflow.
Saga Orchestrator
|
Create Order
|
v
Reserve Inventory
|
v
Charge Payment
|
v
Create Shipment
|
v
DoneIf payment fails:
Release Inventory
|
v
Cancel OrderThe orchestrator maintains workflow state.
OrderCreated
|
v
Inventory
|
InventoryReserved
|
v
Payment
|
PaymentCompleted
|
v
ShippingFailure events propagate compensation.
This can be elegant for simple workflows but harder to reason about as the workflow grows.
Example:
Business Event
|
v
Notification Event
|
v
Notification Service
|
+---- Email
+---- SMS
+---- PushEach delivery channel can have:
Product Service
|
ProductUpdated
|
v
Event Broker
|
v
Search Indexer
|
v
Search EngineIf the search index is lost:
replay product events
|
v
rebuild indexThis is a powerful use of event retention.
Services
|
v
Event Bus
|
+---- Stream Processor
|
+---- Data Warehouse
|
+---- Data LakeThe same event stream can serve multiple analytical systems.
At very large organizational scale, many event domains may exist:
Orders
Payments
Customers
Inventory
ShippingEach domain can publish events.
Consumers subscribe to events relevant to their needs.
Governance becomes increasingly important:
ownership
schemas
security
discoverability
retention
versioningTurning every internal method call into an event.
Using events where synchronous response is required.
No clear event ownership.
No schema/versioning strategy.
Assuming event delivery is exactly once.
No idempotency.
Using event replay for irreversible side effects without safeguards.
Building overly complex choreography.
Using Saga without defining compensation.
Assuming eventual consistency is acceptable everywhere.
Publishing sensitive information unnecessarily.
Ignoring observability across asynchronous boundaries.
When asked:
Design an event-driven system.
Start with requirements:
What events?
Who produces them?
Who consumes them?
Does the caller need an immediate response?
What throughput?
What ordering?
What durability?
What retention?
Can events be duplicated?
Need replay?
Need workflow orchestration?
Need compensation?
What consistency is required?Then define:
Producer
↓
Event Broker
↓
ConsumersThen discuss:
event contracts
partitioning
ordering
retries
DLQ
idempotency
Saga
observability
securityAnswer:
An event represents a fact that has already happened, while a command is an instruction asking a component to perform an action.
Example:
OrderCreated → event
CreateOrder → commandAnswer:
In choreography, services independently react to events and the workflow emerges from those reactions. In orchestration, a central coordinator explicitly controls the workflow, including sequencing, retries, and compensation.
Answer:
A Saga is a distributed transaction pattern in which a business operation is split into local transactions, with compensating actions used when later steps fail.
Answer:
Because asynchronous events introduce processing delays between the source system and derived consumers. The source can commit immediately while other systems update asynchronously, resulting in temporary inconsistency that converges over time.
Answer:
Because asynchronous delivery commonly permits retries and duplicate events. Idempotent consumers prevent repeated delivery from creating repeated business effects.
Answer:
Kafka and databases solve different primary problems. Kafka is optimized for durable event streams, partitioned logs, replay, and fan-out, while databases provide querying, transactions, constraints, and current application state. Production systems often use both.
Build an event-driven e-commerce workflow.
Order Service
Inventory Service
Payment Service
Shipping Service
Notification ServiceUse:
Kafkaor another event broker.
Implement:
OrderCreated
InventoryReserved
InventoryReservationFailed
PaymentCompleted
PaymentFailed
ShipmentCreated
ShipmentFailed
OrderCompleted
OrderCancelledNormal:
OrderCreated
↓
InventoryReserved
↓
PaymentCompleted
↓
ShipmentCreated
↓
OrderCompletedFailure:
OrderCreated
↓
InventoryReserved
↓
PaymentFailed
↓
ReleaseInventory
↓
OrderCancelledImplement:
event schemas
event IDs
idempotency
consumer groups
retries
DLQ
workflow state
trace IDs
metricsAlso simulate:
duplicate event
consumer crash
payment timeout
inventory failure
out-of-order event
slow consumer
schema version changeAdd an orchestration service:
Saga OrchestratorTrack:
sagaId
orderId
currentStep
status
attempts
failureReason
createdAt
updatedAtImplement compensation.
This transforms the project from a simple event demo into a realistic distributed workflow.
[ ] Event definitions
[ ] Event ownership
[ ] Event schema
[ ] Versioning
[ ] Compatibility rules
[ ] Producer reliability
[ ] Consumer idempotency
[ ] Retry strategy
[ ] DLQ
[ ] Ordering strategy
[ ] Partition strategy
[ ] Event retention
[ ] Replay strategy
[ ] Saga where required
[ ] Compensation logic
[ ] Source-of-truth definition
[ ] Eventual consistency requirements
[ ] Observability
[ ] Distributed tracing
[ ] Security
[ ] Privacy
[ ] Capacity planning
[ ] Failure testingRemember these layers:
ARCHITECTURE
|
v
Event-Driven Design
|
+-----------+-----------+
| |
v v
Choreography Orchestration
| |
+-----------+-----------+
|
v
Event Broker
|
+-----------+-----------+
| | |
v v v
Consumer Consumer ConsumerAnd for distributed workflows:
Local Transaction
↓
Event
↓
Local Transaction
↓
Event
↓
Local TransactionIf a later step fails:
Compensating ActionThe most important concepts are:
The central principle is:
Publish meaningful business facts.
Keep consumers independently useful.
Assume duplicates and failures.
Define consistency explicitly.
Use orchestration when workflows become complex.
Use compensation for distributed transactions.You should now be able to explain:
In the next chapter we will step back from queues and events and study the deeper principles behind distributed systems.
We will learn:
Mark it as complete to track your mastery.