In the previous chapter, we learned High Availability.
High availability asks:
How do we keep the service available when components fail?
Fault tolerance goes one level deeper:
How should the system continue operating correctly when failures occur?
A fault-tolerant system assumes that failures are normal possibilities.
Examples:
Server crashes
Database becomes unavailable
Network packet is lost
Service becomes slow
Message is duplicated
Message arrives late
A dependency times out
A zone disappears
A process becomes unhealthyThe goal is not to pretend failures do not exist.
The goal is:
Failure
↓
Detection
↓
Isolation
↓
Fallback / Recovery
↓
Continue safelyThese concepts overlap but are not identical.
Focuses on:
Keeping service availableFocuses on:
Continuing correct operation despite faultsA system can have multiple servers for availability but still behave incorrectly when one server becomes stale.
Therefore:
HA
+
Correct failure handling
+
Recovery
=
Strong fault-tolerant architectureA fault is a problem in a component that can cause incorrect behavior.
Examples:
Hardware failure
Software bug
Network failure
Configuration error
Database failure
Human error
Dependency failureA fault may produce a visible failure.
These terms are useful.
The underlying problem.
Database process crashesThe incorrect internal state caused by the fault.
Application loses database connectionThe externally visible incorrect behavior.
User receives HTTP 500Conceptually:
Fault
↓
Error
↓
FailureNot every fault necessarily becomes a visible failure.
Distributed systems encounter many failure types:
Understanding the type of failure helps determine the correct recovery mechanism.
A crash failure occurs when a process stops functioning.
Example:
Server A
↓
CRASHThe server no longer processes requests.
A basic recovery strategy:
Detect
↓
Remove from traffic
↓
Restart / replaceThis is one of the easiest failure types to reason about.
An omission failure occurs when a system fails to send or receive something it should have.
Example:
A ───── message ─────► B
X
lost/droppedThe message never arrives.
This can occur due to:
Retries and acknowledgements can help, but retries introduce duplicate-processing risks.
A timing failure occurs when an operation happens outside the expected time bound.
Example:
Expected:
response < 200 ms
Actual:
response = 10 secondsThe service may technically be functioning, but from the application's perspective it failed its timing requirement.
Timeouts are therefore critical.
Networks can:
This means distributed applications must never assume:
Request sent = request receivedor:
Timeout = operation failedPartial failure is one of the defining challenges of distributed systems.
A centralized application may fail:
Everything failsA distributed system can fail partially:
Service A ✓
Service B ✓
Service C ✗
Service D ✓The application must decide:
Continue?
Retry?
Fallback?
Degrade?
Reject?This is much harder than handling total failure.
Suppose:
Checkout
│
├── Payment ✓
├── Inventory ✓
├── Recommendations ✗
└── Analytics ✗Should checkout fail?
If recommendations and analytics are non-critical:
Checkout → continueThis is fault isolation.
A key principle:
Prevent one component's failure from spreading to unrelated components.
Example:
Service A
│
X
Dependency fails
Without isolation:
A → B → C → entire system fails
With isolation:
A → fallback
B → unaffected
C → unaffectedCommon mechanisms:
Never wait forever for a dependency.
Bad:
request
↓
wait indefinitelyBetter:
request
↓
timeout
↓
fallback / retry / failureTimeouts prevent a slow dependency from consuming resources indefinitely.
Suppose a dependency normally responds in:
50 mswith occasional:
150 msA timeout of:
1 secondmay be reasonable for that operation.
But timeout selection should consider:
Do not blindly choose a timeout such as:
30 secondsfor every service.
Instead of every service choosing its own independent timeout, propagate a request deadline.
Example:
User request
deadline = 2 seconds
Service A
remaining = 1.8 sec
Service B
remaining = 1.2 sec
Service C
remaining = 0.4 secThis prevents downstream operations from continuing after the overall request can no longer succeed.
Retries are useful for transient failures.
Example:
Request
↓
timeout
↓
retry
↓
successBut retries are dangerous when uncontrolled.
Suppose:
10,000 clientsall retry a failing service.
Traffic can become:
Original traffic
+
Retries
+
Retry retriesThe dependency may become even more overloaded.
This is a feedback loop.
Instead of retrying immediately:
retry after:
1s
2s
4s
8sThis gives the dependency time to recover.
If every client waits exactly:
1 secondthey may retry simultaneously.
Use randomness:
0.8s
1.1s
1.4s
0.9sThis spreads retry traffic over time.
A practical retry strategy often combines:
bounded retries
+
exponential backoff
+
jitter
+
deadlineRetries can create duplicates.
Example:
POST /paymentFirst request:
Payment succeedsResponse is lost.
Client retries:
Payment succeeds againPotentially:
₹1,000 charged twiceUse an idempotency key:
payment_request_id = abc123The service records the result.
Retry:
abc123returns the original result instead of creating another charge.
A circuit breaker prevents repeated calls to an unhealthy dependency.
States:
CLOSED
│
│ failures exceed threshold
▼
OPEN
│
│ wait
▼
HALF-OPEN
│
├── success → CLOSED
│
└── failure → OPENThis protects both sides.
Without circuit breaker:
Service A
│
├──► B fails
├──► B fails
├──► B fails
├──► B fails
└──► B failsWith circuit breaker:
A
│
├──► B fails
├──► B fails
│
└── circuit OPEN
↓
fail fastA can use a fallback instead.
Bulkheads isolate resources.
Suppose a service has:
100 worker threadsDependency A consumes all 100.
Then requests to:
Dependency Balso fail.
Instead:
Pool A = 50
Pool B = 50If A becomes unhealthy:
Pool A exhausted
Pool B still availableThis limits blast radius.
Bulkheads can isolate:
Example:
Payment dependency → 30 connections
Search dependency → 20 connections
Profile dependency → 20 connections
Other work → 30 connectionsOne dependency cannot consume the entire connection pool.
Backpressure prevents producers from overwhelming consumers.
Example:
Producer
│
▼
Queue
│
▼
ConsumerProducer:
10,000 events/secConsumer:
2,000 events/secWithout controls:
Queue → grows indefinitelyWith backpressure:
Throttle producer
or
shed lower-priority workWhen the system reaches unsafe capacity:
Reject some workrather than allowing:
Everything → failExample:
Critical checkout → accept
Recommendations → reject/defer
Analytics → delayLoad shedding is a deliberate availability strategy.
A system should define what happens when optional dependencies fail.
Example:
E-commerce
Payment → critical
Inventory → critical
Search → important
Recommendations → optional
Reviews → optionalIf recommendations fail:
Use trending productsIf reviews fail:
Hide reviewsThe core purchase flow remains functional.
Replication creates redundant copies.
Example:
Primary
/ \
R1 R2If primary fails:
R1 / R2can potentially provide recovery.
Replication strategies include:
Each has different consistency and failure characteristics.
Consider:
Server A ─┐
Server B ─┼── same availability zone
Server C ─┘If the zone fails:
A ✗
B ✗
C ✗All replicas are lost simultaneously.
Fault tolerance requires considering independent failure domains.
Typical hierarchy:
Process
↓
Server
↓
Rack
↓
Availability Zone
↓
Region
↓
ProviderA resilient architecture decides which levels it must survive.
A self-healing system automatically responds to failures.
Example:
Instance crashes
↓
Health check detects failure
↓
Remove instance
↓
Launch replacement
↓
Register replacement
↓
Traffic resumesThis reduces MTTR.
But automation must be safe.
Poor automation can create:
Failure
↓
Automatic restart
↓
Restart loop
↓
More resource consumption
↓
System overloadTherefore self-healing needs:
Use retries for:
Transient failureUse fail-fast behavior for:
Permanent failure
Invalid request
Dependency clearly unavailable
Deadline exceededExample:
HTTP 400
→ do not retry
Temporary network timeout
→ retry may be appropriateRetries consume capacity.
Suppose:
Normal traffic = 10,000 req/secIf each request can retry 3 times, worst-case work can become much larger.
Therefore define:
Maximum retries
Maximum retry time
Maximum total request deadlineRetries should be part of capacity planning.
A fallback provides an alternate response when the preferred path fails.
Example:
Primary recommendation service
↓
failure
↓
Trending productsAnother example:
Live exchange rate
↓
failure
↓
recent cached rateFallbacks should be explicitly designed, not accidentally created.
Sometimes stale data is better than no data.
Example:
Current product description unavailable
↓
Use cached descriptionThis is useful when:
staleness is acceptableIt should not be used for data where stale values can cause serious harm.
Queues decouple producers and consumers.
Example:
API
↓
Queue
↓
WorkerIf workers temporarily fail:
Messages remain queuedWhen workers recover:
Process backlogThis improves resilience for asynchronous workloads.
Some messages repeatedly fail.
Instead of retrying forever:
Queue
↓
retry
↓
retry
↓
retry
↓
DLQA Dead-Letter Queue allows operators to inspect problematic messages.
A poison message is a message that repeatedly causes processing failure.
Example:
Malformed payloadRetrying forever wastes resources.
Better:
attempt 1 → fail
attempt 2 → fail
attempt 3 → fail
↓
DLQThen investigate or repair the message.
Fault tolerance is not only about service availability.
Suppose:
Database value becomes corruptedReplication can spread the corrupted value.
Therefore systems may need:
Replication and backup solve different problems.
Provides additional copies for availability.
Primary → ReplicaProvides historical recovery.
Database
↓
Snapshot
↓
Historical copyIf bad data is replicated:
Primary corrupted
↓
Replica corruptedA backup can provide an older clean state.
Suppose backups run:
every 1 hourA failure may cause up to approximately:
1 hourof data loss depending on the backup and recovery design.
Therefore backup frequency influences RPO.
Suppose recovery requires:
30 minutesThen:
RTO ≈ 30 minutesTo reduce RTO, use:
A service should not simply terminate all active requests during deployment.
Instead:
Stop accepting new traffic
↓
Finish active requests
↓
Close connections
↓
ShutdownThis reduces user-visible failures.
During failover or deployment:
Load Balancer
│
X
Stop new requestsExisting requests are allowed to complete.
This is called connection draining.
It prevents sudden termination of active work.
Fault injection deliberately creates failures.
Examples:
Kill process
Add 500 ms latency
Drop packets
Fill disk
Exhaust connections
Return HTTP 500
Disconnect databaseThe goal is to test assumptions.
Chaos engineering is a structured approach to testing resilience.
Process:
Hypothesis
↓
Controlled experiment
↓
Inject failure
↓
Observe system
↓
Compare against expected behavior
↓
ImproveExample hypothesis:
If one application instance fails, user-facing availability remains within the SLO.
A good experiment should:
Do not randomly destroy production systems without safeguards.
Kill 1 instanceAdd latency
Drop packetsSimulate connection failuresDelay consumersRemove an availability zoneEach tests a different failure assumption.
A production resilience test might look like:
Scenario:
One app server fails
Expected:
Traffic moves to healthy instances
No significant user impact
SLO remains satisfiedThen:
Scenario:
Database primary fails
Expected:
Replica promotion
Connections reconnect
Writes resumeThen:
Scenario:
Payment service unavailable
Expected:
Checkout handles failure safely
No duplicate payment| Failure | Detection | Response |
|---|---|---|
| App crash | Health check | Replace instance |
| DB failure | Connection/health monitoring | Failover |
| Network timeout | Timeout | Retry/fallback |
| Dependency overload | Metrics | Circuit breaker |
| Queue backlog | Queue depth | Scale/throttle |
| Poison message | Retry count | DLQ |
| Region failure | Global health | Traffic failover |
| Data corruption | Validation/monitoring | Restore backup |
Consider:
Client
│
▼
Payment API
│
├── Idempotency
│
├── Durable transaction record
│
└── Payment ProviderFailure:
Payment provider timeoutDo not blindly retry the charge.
Instead:
Check idempotency key
↓
Query payment status
↓
Retry safely if necessaryThis prevents duplicate charges.
Order Service
│
├── Order DB
│
└── Outbox
│
▼
Message Bus
/ \
▼ ▼
Inventory PaymentFailures are handled with:
Retries
Idempotency
Saga
Compensation
DLQ
ReconciliationA robust API layer may use:
Client
↓
Load Balancer
↓
API Servers
↓
Timeouts
↓
Circuit Breakers
↓
DependenciesAdd:
Rate limiting
Bulkheads
Caching
Fallbacks
ObservabilityA storage system may use:
Data
│
├── Replica A
├── Replica B
└── Replica CWith:
Quorum
Checksums
Versioning
Repair
SnapshotsIf one replica fails:
Read from healthy replicasThen repair:
Healthy state
↓
Replica repair
↓
Restored redundancySome replicated systems periodically compare replicas and repair differences.
Conceptually:
Replica A
↕
compare
↕
Replica BDifferences are detected and repaired.
This helps replicas converge after failures.
A replicated storage system can follow:
Replica failure
↓
Detect
↓
Continue with remaining replicas
↓
Create replacement
↓
Copy missing data
↓
Restore redundancyThe system should also monitor:
repair progressbecause a system with one remaining replica is more vulnerable to another failure.
Suppose:
4 serverseach supports:
25,000 req/secTotal:
100,000 req/secIf one fails:
75,000 req/secIf normal traffic is:
80,000 req/secthe system is overloaded after a failure.
Therefore fault tolerance requires:
Enough spare capacity to continue operating under expected failures.
You can define expected failure assumptions:
Survive:
1 application server failure
1 cache node failure
1 availability zone failureThen design capacity and redundancy accordingly.
This is more concrete than saying:
"Make it fault tolerant."A useful layered model:
Application
↓
Timeouts
↓
Retries
↓
Circuit Breakers
↓
Queues
↓
Replication
↓
Failover
↓
Backups
↓
Disaster RecoveryEach layer protects against different failure classes.
More resilience mechanisms also create complexity.
For example:
Retry
+
Queue
+
Circuit breaker
+
Cache
+
Fallback
+
Replicationcan create unexpected interactions.
A good design uses the smallest set of mechanisms needed to meet the reliability requirement.
You cannot operate a fault-tolerant system without visibility.
Monitor:
Important metrics include:
Mean time to recovery.
Failure detected
↓
Traffic restoredHow often automated recovery succeeds.
How much additional traffic retries generate.
These metrics reveal whether resilience mechanisms are actually helping.
Fault tolerance means the system never fails.
No.
Fault tolerance means the system is designed to continue safely despite defined failures.
More retries make a system more reliable.
Not necessarily.
Excessive retries can create retry storms and make an outage worse.
Replication is the same as backup.
No.
Replication primarily supports availability and redundancy.
Backups provide historical recovery from corruption, deletion, or other failures.
A timeout means the operation failed.
No.
A timeout means the caller did not receive the expected response in time.
The operation may have succeeded.
Circuit breakers fix the failed dependency.
No.
They protect the caller and broader system from repeatedly depending on an unhealthy component.
Chaos engineering means randomly breaking production.
No.
Good chaos experiments are controlled, hypothesis-driven, observable, and bounded by safety mechanisms.
High availability focuses on keeping a service accessible despite failures. Fault tolerance focuses more broadly on continuing correct operation when faults occur. HA is one important outcome of fault-tolerant design, but fault tolerance also includes safe handling of partial failures, retries, corruption, recovery, and failure isolation.
A strong answer includes:
Timeouts
Circuit breakers
Retries with backoff
Idempotency
Bulkheads
Fallbacks
Queues
Load shedding
Replication
Graceful degradation
Observability
RecoveryThen explain which mechanism protects which dependency.
Because retries can amplify traffic during an outage.
For example:
Dependency slows
↓
Requests timeout
↓
Clients retry
↓
More traffic
↓
Dependency becomes slower
↓
More retriesThis feedback loop can cause cascading failure.
Possible answer:
Detect failure
↓
Stop sending traffic
↓
Fail over according to DB architecture
↓
Reconnect clients
↓
Verify consistency
↓
Restore redundancy
↓
Monitor recoveryThen discuss:
Mention:
Timeouts
+
Circuit breakers
+
Bulkheads
+
Backpressure
+
Rate limiting
+
Load shedding
+
Bounded queues
+
Graceful degradation
+
Retry limitsRequirements:
Design:
Client
│
▼
Load Balancer
│
┌─────┴─────┐
▼ ▼
Payment A Payment B
│ │
└─────┬─────┘
│
Payment DB
│
Outbox/Events
│
▼
Payment ProviderInclude:
Idempotency
Timeouts
Retries
Status reconciliation
Circuit breaker
Database failover
Audit log
Backups
MonitoringSuppose:
Payment provider latency:
normal = 100 ms
degraded = 10 secondsYour API timeout is:
2 secondsQuestions:
A strong solution should identify the distinction between:
FAILURE
vs
UNKNOWN OUTCOMEBefore production, verify:
Remember fault tolerance as:
FAILURE
│
▼
DETECT
│
┌──────────┴──────────┐
│ │
TRANSIENT PERMANENT
│ │
RETRY FAIL FAST
BACKOFF FALLBACK
│ │
└──────────┬──────────┘
▼
ISOLATE
│
┌──────────┼──────────┐
│ │ │
Circuit Bulkhead Load Shed
Breaker
│
▼
RECOVER
│
┌──────────┼──────────┐
│ │ │
Failover Replay Reconcile
│
▼
RESTORE
REDUNDANCYThe core principle:
A fault-tolerant system does not merely detect failure; it prevents failure from propagating, preserves correctness where required, and restores normal capacity safely.
You learned:
The most important takeaway:
Distributed systems must be designed around partial failure. Assume that components will fail independently, responses can be lost, messages can be duplicated, and networks can become unreliable. Then explicitly design detection, isolation, fallback, recovery, and reconciliation.
Explain:
Fault
Error
Failurewith a database example.
Explain why retries can make an outage worse.
Design a circuit breaker for a payment service.
Design bulkheads for:
Payment
Search
RecommendationsExplain:
Replication vs BackupDesign fault tolerance for a message-processing system.
Include:
Explain why a timeout does not necessarily mean an operation failed.
Next we will move from component-level fault tolerance to large-scale disaster recovery.
We will cover:
Mark it as complete to track your mastery.