In a traditional application, a transaction is often simple:
BEGIN
update table A
update table B
COMMITA single database can provide ACID guarantees across those operations.
But modern systems often look like this:
Order Service
│
├────► Order DB
│
├────► Payment Service
│
├────► Inventory Service
│
└────► Notification ServiceNow one business operation spans:
This creates the problem of the distributed transaction.
The central question becomes:
How do we keep a multi-step business operation correct when the steps can fail independently?
Consider an e-commerce checkout:
1. Create order
2. Reserve inventory
3. Charge payment
4. Create shipment
5. Send confirmationSuppose:
Order created ✓
Inventory reserved ✓
Payment charged ✓
Shipment creation ✗What should happen?
We now need to decide:
This is the core difficulty.
Everything happens inside one database:
Application
│
▼
Database
┌─────────────┐
│ BEGIN │
│ UPDATE A │
│ UPDATE B │
│ COMMIT │
└─────────────┘The database manages atomicity.
The operation spans multiple resources:
Service A ──► DB A
│
├───────► Service B ──► DB B
│
└───────► Service C ──► DB CNow there is no single local transaction manager that automatically controls every resource.
Suppose:
Payment = ₹1,000
Inventory = 1 itemWorkflow:
Reserve item
↓
Charge paymentIf payment succeeds but inventory reservation fails:
Payment → SUCCESS
Inventory → FAILUREWe have an inconsistent business state.
The system needs a recovery strategy.
Traditional ACID means:
Distributed systems make these guarantees harder because operations can fail independently.
For example:
Service A → SUCCESS
Network → FAILURE
Service B → UNKNOWNThe coordinator may not know whether Service B completed the operation.
This is one of the most difficult distributed transaction scenarios.
Imagine:
Client
│
▼
Payment Service
│
▼
Charge ₹1,000
│
▼
Payment succeeds
│
X
│
Response lostThe client receives:
TIMEOUTBut what actually happened?
Possibilities:
A. Payment never happened
B. Payment succeeded
C. Payment is still processingA blind retry could cause:
Charge 1 → SUCCESS
Charge 2 → SUCCESSTherefore distributed systems need:
There are several major approaches:
Distributed Transactions
│
├── Two-Phase Commit
│
├── Three-Phase Commit
│
├── Saga
│
├── Transactional Outbox
│
├── Idempotent workflows
│
└── Application-specific compensationThese approaches solve different problems.
Two-Phase Commit is a distributed transaction protocol.
It uses:
Example:
Coordinator
/ | \
/ | \
▼ ▼ ▼
DB A DB B DB CThe coordinator manages the transaction.
The coordinator asks every participant:
"Can you commit?"Example:
Coordinator
│
├──► DB A: PREPARE
│
├──► DB B: PREPARE
│
└──► DB C: PREPAREEach participant checks whether it can safely commit.
If successful:
DB A → READY
DB B → READY
DB C → READYThe participants persist enough state to commit later.
If every participant says:
READYthe coordinator sends:
COMMITto all participants.
Coordinator
│
├──► DB A: COMMIT
├──► DB B: COMMIT
└──► DB C: COMMITThe transaction completes.
Suppose:
DB A → READY
DB B → READY
DB C → FAILThe coordinator cannot safely commit the whole transaction.
It sends:
ABORTto participants.
Coordinator
│
├──► DB A: ABORT
├──► DB B: ABORT
└──► DB C: ABORTThe transaction is rolled back.
A simplified model:
START
│
▼
PREPARE
│
┌────────┴────────┐
│ │
ALL YES ANY NO
│ │
▼ ▼
COMMIT ABORT
│ │
▼ ▼
DONE DONEThis is the basic idea behind two-phase commit.
2PC can provide a strong atomicity guarantee across participating resources.
For example:
DB A
DB B
DB Ccan be coordinated so that the transaction either:
all commitor:
all abortThis is useful for certain workloads that genuinely require distributed atomicity.
2PC can block.
Suppose:
Coordinator
│
├── A → READY
├── B → READY
└── C → READYThe coordinator has decided to commit but crashes before participants receive the final decision.
Participants may be left in a prepared state:
DB A → waiting
DB B → waiting
DB C → waitingThey cannot safely make an independent decision without violating atomicity.
This is a major weakness.
Consider:
Participants:
A
B
C
Coordinator:
XIf the coordinator fails at the wrong point:
A → PREPARED
B → PREPARED
C → PREPAREDParticipants may need to wait for recovery or a mechanism that can recover the transaction decision.
This is one reason 2PC introduces coordination overhead and failure-handling complexity.
Suppose:
Coordinator
X
ParticipantsThe coordinator cannot communicate with some participants.
A participant may know:
"I prepared."but not know:
"Did the global transaction commit?"The protocol therefore has to prioritize atomicity over simply allowing participants to independently continue.
2PC can add:
This becomes especially expensive across regions.
Example:
India
│
│ WAN
▼
Europe
│
│ WAN
▼
USCross-region coordination can significantly increase latency.
Some distributed transaction implementations hold resources while a transaction is prepared.
Conceptually:
Transaction starts
↓
Resources reserved
↓
Wait for global decision
↓
Commit / abort
↓
Release resourcesIf the coordinator is unavailable for a long time:
Resources remain occupiedThis can reduce throughput.
Three-Phase Commit (3PC) extends the basic idea of 2PC.
A simplified conceptual flow is:
1. CanCommit?
2. PreCommit
3. DoCommitCompared with 2PC, 3PC introduces an additional intermediate phase intended to reduce certain blocking situations under assumptions about failures and timing.
However:
3PC does not magically make distributed transactions easy or eliminate all failure problems.
It also introduces additional communication and protocol complexity.
Modern microservices frequently avoid global ACID transactions.
Instead, they use a Saga.
A Saga breaks one business transaction into a sequence of local transactions.
Example:
Create Order
↓
Reserve Inventory
↓
Charge Payment
↓
Create ShipmentEach service commits its own local transaction.
If a later step fails, the system executes compensating actions.
Suppose:
Step 1:
Create order ✓
Step 2:
Reserve inventory ✓
Step 3:
Charge payment ✓
Step 4:
Create shipment ✗The system may compensate:
Refund payment
↓
Release inventory
↓
Cancel orderThis is not a database rollback.
It is a business-level compensation.
This distinction is critical.
Transaction
↓
ROLLBACK
↓
Database returns to previous statePayment charged
↓
Refund paymentThe refund is a new business operation.
It is not literally undoing the original database transaction.
Consider:
Order Service
↓
Create Order
↓
Inventory Service
↓
Reserve Item
↓
Payment Service
↓
Charge Customer
↓
Shipping Service
↓
Create ShipmentCompensating actions:
Create Order
↓
Cancel Order
Reserve Item
↓
Release Item
Charge Payment
↓
Refund Payment
Create Shipment
↓
Cancel ShipmentEach step needs a clearly defined compensation where applicable.
In choreography, services react to events.
Example:
Order Service
│
▼
OrderCreated
│
▼
Inventory Service
│
▼
InventoryReserved
│
▼
Payment Service
│
▼
PaymentCompletedThere is no central orchestrator.
Each service knows how to react to events.
Choreography can become difficult when there are many services.
Example:
Order
↓
Inventory
↓
Payment
↓
Shipping
↓
Notification
↓
AnalyticsWith many event relationships, it can become difficult to understand:
This can create a "distributed spaghetti" problem.
In orchestration, a central workflow component coordinates the Saga.
Saga Orchestrator
/ | \
/ | \
▼ ▼ ▼
Order Inventory PaymentThe orchestrator says:
1. Create order
2. Reserve inventory
3. Charge payment
4. Create shipmentIf payment fails:
Orchestrator
↓
Release inventory
↓
Cancel order| Aspect | Choreography | Orchestration |
|---|---|---|
| Central coordinator | No | Yes |
| Communication | Events | Commands/workflow |
| Coupling | Distributed | Central workflow |
| Visibility | Can become difficult | Usually easier |
| Large workflows | Can become complex | Often easier to manage |
| Failure handling | Distributed | Centralized workflow logic |
| Risk | Event spaghetti | Orchestrator complexity |
Neither is universally better.
The workflow shape and organizational requirements matter.
A Saga can be modeled explicitly:
START
│
▼
ORDER_CREATED
│
▼
INVENTORY_RESERVED
│
▼
PAYMENT_COMPLETED
│
▼
SHIPMENT_CREATED
│
▼
COMPLETEDFailure:
PAYMENT_FAILED
│
▼
RELEASE_INVENTORY
│
▼
CANCEL_ORDER
│
▼
FAILEDExplicit state makes recovery easier.
Do not keep critical Saga state only in memory.
Bad:
Orchestrator RAM
↓
workflow stateIf the orchestrator crashes:
state lostBetter:
Orchestrator
│
▼
Durable Workflow DBExample state:
order_id = 123
state = PAYMENT_COMPLETED
inventory = RESERVED
payment = SUCCESS
shipment = PENDINGThe workflow can resume after a crash.
Another important pattern is the Transactional Outbox.
Problem:
Service
│
├── write database
│
└── publish eventWhat if:
DB write → SUCCESS
Event publish → FAILURENow the database contains the new state but other services never receive the event.
Store the business change and outgoing event in the same local database transaction.
BEGIN
UPDATE orders
INSERT INTO outbox_events
COMMITNow both are committed atomically.
Example:
Orders table
----------------
order_id | status
123 | CREATED
Outbox table
-----------------------------
id | event
1 | OrderCreatedA separate publisher reads the outbox:
Database
│
▼
Outbox Table
│
▼
Publisher
│
▼
Message Broker
│
▼
ConsumersIf publishing fails:
RetryThe event remains durable.
Without Outbox:
DB update ✓
Event publish ✗With Outbox:
DB update ✓
Outbox insert ✓
↓
Publisher retries
↓
Event eventually publishedThis provides reliable handoff between local database state and asynchronous messaging.
A publisher might do:
Publish event
↓
Broker accepts
↓
Publisher crashesThe publisher may not know whether the broker accepted the message.
It may retry:
Event → delivered twiceTherefore consumers should be idempotent.
This leads to an important principle:
At-least-once delivery + idempotent processing is often more practical than trying to build perfect exactly-once behavior across the entire system.
An operation is idempotent when repeating it produces the same intended final effect.
Example:
PUT /orders/123/status
status = CONFIRMEDRepeated execution can still result in:
CONFIRMEDFor payment APIs, use an idempotency key:
idempotency_key = abc123First request:
abc123 → processRetry:
abc123 → return existing resultEvery Saga step should ideally be safe to retry.
Example:
ReserveInventory(order_id=123)If called twice:
First → reservation created
Second → existing reservation returnedNot:
Reserve 1 item
Reserve another itemunless that is explicitly intended.
A common interview statement is:
"We need exactly-once processing."
Be careful.
Across distributed systems, exactly-once end-to-end behavior is difficult because:
A practical design often uses:
At-least-once delivery
+
Idempotent consumers
+
Deduplication
+
Durable stateto achieve an effectively-once business outcome.
Consumers can maintain processed IDs:
processed_events
event_id
---------
abc
def
ghiWhen event abc arrives again:
if abc already processed:
skip business operation
else:
process
record abcThe deduplication record itself must be stored safely.
Suppose:
Order created ✓
Inventory reserved ✓
Payment charged ✓
Shipping failed ✗Compensation:
Refund payment
↓
Release inventory
↓
Cancel orderNow suppose refund itself fails:
Refund ✗The system must retry the compensation.
This demonstrates:
Compensation is itself a distributed operation that can fail.
Therefore Saga design requires durable state, retries, idempotency, and monitoring.
Some actions cannot be perfectly undone.
For example:
Email sentYou cannot truly "unsend" an email in the general case.
Instead:
Email sent
↓
Send correction emailSimilarly:
Physical shipment dispatchedmay require:
Return processrather than a database rollback.
This is why business workflows must be designed around real-world reversibility.
Sometimes a system needs exclusive access to a resource.
Example:
Only one worker should process:
order_id = 123A distributed lock may be used:
Worker A
│
▼
Lock(order:123)
│
▼
ProcessBut distributed locks have their own failure problems.
A lease is a lock with an expiration time.
Example:
Lock acquired
TTL = 30 secondsIf the worker crashes:
30 seconds
↓
lease expires
↓
another worker can proceedThis prevents a permanently held lock.
Suppose:
Worker A gets lock
token = 10A pauses due to a long garbage collection or network issue.
The lease expires.
Worker B gets:
token = 11Worker A wakes up and continues.
Now both workers appear active.
A fencing token allows downstream storage to reject stale workers:
Worker A → token 10 → reject
Worker B → token 11 → acceptThis is an important advanced distributed-systems pattern.
2PC can make sense when:
Examples may include certain tightly controlled transactional systems.
But it is usually not the first choice for loosely coupled internet-scale microservices.
Saga is often appropriate when:
Example:
Order
↓
Inventory
↓
Payment
↓
Shipping| Property | 2PC | Saga |
|---|---|---|
| Atomic global commit | Yes, by protocol | No global rollback |
| Transactions | Distributed transaction | Local transactions |
| Compensation | Usually not primary mechanism | Core mechanism |
| Blocking | Possible | Workflow-based |
| Availability | Can suffer | Often better |
| Latency | Coordination-heavy | Can be asynchronous |
| Failure handling | Protocol-driven | Business-driven |
| Microservices fit | Limited | Common |
| Complexity | Infrastructure/protocol | Workflow/business logic |
The choice depends on requirements.
Consider:
Checkout
│
▼
Order Service
│
▼
Payment Service
│
▼
Inventory Service
│
▼
Shipping ServiceA robust workflow might use:
Local DB transactions
+
Outbox
+
Message broker
+
Saga orchestration
+
Idempotency
+
CompensationThis avoids trying to turn every service into one giant distributed database transaction.
API
│
▼
Order Service
│ │
│ └────► Order DB
│
└────► Outbox
│
▼
Message Bus
/ | \
/ | \
▼ ▼ ▼
Inventory Payment Shipping
│ │ │
DB DB DBEach service owns its database.
Events coordinate the workflow.
Suppose:
Payment succeededbut:
Order service crashedAfter restart:
Read durable workflow state
↓
Detect payment success
↓
Continue workflowThis is why workflow state must be durable.
Retries should use:
Exponential backoff
+
Jitter
+
Maximum attempts
+
Dead-letter handlingExample:
1 sec
2 sec
4 sec
8 sec
16 secwith random jitter.
Do not retry forever without limits.
Bad:
Payment request
↓
timeout
↓
retry
↓
charge againBetter:
Payment request
idempotency_key = payment_123
timeout
↓
retry with same key
↓
existing resultThis is one of the most important production patterns for distributed workflows.
Suppose:
Service A → Service BA waits:
10 secondsthen gets:
timeoutA must not automatically assume:
B definitely failedThe operation may have succeeded but the response may have been lost.
Therefore use:
A reconciliation process periodically checks for inconsistent workflow states.
Example:
Order:
PAID
Payment:
SUCCESS
Inventory:
RESERVED
Shipping:
MISSINGReconciliation identifies:
Shipping should be createdand retries the missing step.
This is extremely useful for long-running business workflows.
Some workflows may last:
seconds
minutes
hours
daysExamples:
A database transaction cannot remain open for hours.
Instead use:
Durable workflow state
+
events
+
timeouts
+
compensation
+
retriesConsider:
Order
↓
Fraud review
↓
Human approval
↓
Payment
↓
ShippingThis can take hours.
A Saga-style workflow can persist:
state = WAITING_FOR_REVIEWWhen the reviewer approves:
state = APPROVEDThe workflow continues.
Keep local transactions short.
Avoid distributed transactions unless truly necessary.
Make every retryable operation idempotent.
Persist workflow state.
Use durable events.
Design compensating actions explicitly.
Assume responses can be lost.
Assume messages can be duplicated.
Monitor stuck workflows.
Build reconciliation for long-running systems.
Saga is just a distributed rollback.
A Saga is a sequence of local transactions coordinated into a business workflow.
Compensation is a new business action, not a magical rollback.
Outbox guarantees exactly-once delivery.
No.
Outbox provides reliable persistence of the event alongside the local transaction.
Publishing may still result in duplicate delivery.
Consumers should be idempotent.
Retry means the previous request failed.
Not necessarily.
A timeout means the caller does not know the outcome.
The operation may have succeeded.
2PC completely eliminates failures.
No.
2PC coordinates atomic commit but introduces blocking, coordination, latency, and recovery complexity.
Compensation always restores the exact previous state.
Not necessarily.
Some actions cannot be perfectly reversed.
Compensation should be defined at the business level.
Two-Phase Commit is a distributed transaction protocol involving a coordinator and participants. In the prepare phase, participants indicate whether they can commit. If all participants are prepared, the coordinator sends commit; otherwise it sends abort. The protocol can provide atomic commit but can introduce blocking and coordination overhead.
Possible reasons include:
A Saga breaks a distributed business transaction into a sequence of local transactions. If a later step fails, compensating actions are executed for earlier completed steps.
Services react to events independently.
A central workflow component directs the steps.
Use the model that keeps workflow ownership, failure handling, and operational visibility understandable.
Because distributed systems can retry operations after timeouts or duplicate messages.
Without idempotency:
Retry
↓
Duplicate side effectWith idempotency:
Retry
↓
Same logical resultTransactional Outbox stores the business state change and the outgoing event in the same local database transaction.
A separate publisher later sends the event to the broker.
This prevents the classic:
DB commit succeeded
event publish failedinconsistency window.
A strong answer:
End-to-end exactly-once behavior is difficult in distributed systems because requests, messages, acknowledgements, and processes can fail independently. A practical design often uses at-least-once delivery, idempotent processing, deduplication, durable state, and reconciliation to achieve an effectively-once business outcome.
Requirements:
Design:
Checkout API
│
▼
Order Service
│ │
│ └── Order DB
│
▼
Outbox
│
▼
Message Bus
/ | \
▼ ▼ ▼
Inventory Payment Shipping
│ │ │
DB DB DBDefine:
ORDER_CREATED
INVENTORY_RESERVED
PAYMENT_COMPLETED
SHIPMENT_CREATED
COMPLETEDINVENTORY_FAILED
PAYMENT_FAILED
SHIPPING_FAILED
COMPENSATING
CANCELLEDPayment → Refund
Inventory → Release
Order → CancelOutbox
Idempotency
Retries
DLQ
Reconciliation
Durable workflow stateSuppose this happens:
Order created ✓
Inventory reserved ✓
Payment request sent ✓
Payment timed out ?You do not know whether payment succeeded.
Design the system so that a retry does not double-charge the customer.
A strong solution should include:
Payment ID / Idempotency Key
↓
Payment Service
↓
Durable payment record
↓
Status lookup
↓
Retry safelyThis is a classic distributed-systems interview problem.
Before deploying a distributed workflow, verify:
Think about distributed transactions like this:
Business Transaction
│
┌────────────┴────────────┐
│ │
Global ACID approach Local transactions
│ │
2PC Saga
│ │
Coordination-heavy Compensation-basedAround either approach, production systems usually need:
Idempotency
+
Retries
+
Durable state
+
Outbox/events
+
Timeouts
+
Reconciliation
+
ObservabilityThe key lesson:
Distributed transaction design is not only about committing data. It is about defining correct behavior when any step, machine, message, or network call can fail independently.
You learned:
Next we will go deeper into how distributed systems coordinate multiple nodes.
We will cover:
Mark it as complete to track your mastery.