A distributed system is not useful if it works only when everything is healthy.
Production systems must continue serving users when:
This is the goal of High Availability (HA).
A simple definition is:
High availability means designing a system so that it continues providing its intended service despite failures.
HA is not the same as:
"Nothing ever fails."Instead:
Failures happen
↓
Detect failure
↓
Fail over / degrade
↓
Continue servingThese terms are related but different.
Availability asks:
Is the service usable right now?
Example:
Request → ResponseReliability asks:
Does the system continue working correctly over time without failure?
A system can be:
Highly available
but occasionally incorrector:
Reliable
but unavailable during planned maintenanceA good production system aims for both.
A simple availability calculation is:
Availability =
Uptime / (Uptime + Downtime)For example:
Uptime = 99 hours
Downtime = 1 hour
Availability =
99 / 100
= 99%Another common representation is:
Availability =
MTBF / (MTBF + MTTR)Where:
Mean Time Between Failures measures the average time between failures.
Higher MTBF generally means:
Failures happen less frequentlyExample:
System operates for 1,000 hours
between failures on average.MTBF:
1000 hoursMean Time To Recovery measures how quickly the system is restored after failure.
Example:
Failure detected
↓
Engineer/system responds
↓
Service restoredIf recovery takes:
5 minutesthen reducing MTTR from:
30 min → 5 mincan significantly improve availability.
Suppose two systems have similar failure frequency.
Failure every 30 days
Recovery = 1 hourFailure every 30 days
Recovery = 5 minutesSystem B has much less downtime.
Therefore HA is not only about preventing failures.
It is also about:
Detecting and recovering from failures quickly.
Availability is often described using "nines".
Examples:
99%
99.9%
99.99%
99.999%Higher availability means less allowed downtime.
Approximate annual downtime:
| Availability | Approx. downtime/year |
|---|---|
| 99% | 3.65 days |
| 99.9% | 8.76 hours |
| 99.99% | 52.56 minutes |
| 99.999% | 5.26 minutes |
| 99.9999% | 31.5 seconds |
These are approximate theoretical values for a 365-day year.
Suppose your target is:
99.999%That allows only about:
5.26 minutes/yearThat budget must potentially include:
Therefore extremely high availability requires significant engineering and operational discipline.
These three terms are frequently used together.
Service Level Indicator
A measurement.
Examples:
Availability
Latency
Error rateExample:
99.97% successful requestsService Level Objective
The target.
Example:
99.95% availabilityService Level Agreement
A formal commitment, often contractual, between a provider and customer.
Suppose:
SLI:
successful request percentage
SLO:
99.9%
SLA:
99.5%Interpretation:
Measurement → SLI
Internal target → SLO
External commitment → SLADo not use these terms interchangeably.
The most fundamental HA technique is:
Don't depend on a single component.
Bad:
User
↓
Server A
↓
Database AIf Server A fails:
Service unavailableBetter:
Load Balancer
/ \
/ \
Server A Server B
\ /
\ /
DatabaseNow one application server can fail without necessarily taking down the service.
A Single Point of Failure (SPOF) is a component whose failure can take down the service.
Example:
Load Balancer
│
▼
Server A
│
▼
Database APotential SPOFs:
Load Balancer
Server A
Database A
Network path
StorageHA design tries to remove or mitigate critical SPOFs.
Common redundancy approaches include:
The right redundancy level depends on the failure domain you need to survive.
A failure domain is a group of resources that can fail together.
Examples:
Server
Rack
Availability Zone
Region
Cloud providerIf all replicas are inside one rack:
Rack failure
↓
All replicas failTherefore:
Replication is only useful if replicas are sufficiently independent.
Cloud providers divide regions into isolated infrastructure locations commonly called:
Availability ZonesConceptually:
Region
├── AZ A
├── AZ B
└── AZ CIf your application runs only in:
AZ Aan AZ-level outage can take it offline.
A multi-AZ design spreads critical components across zones.
Example:
Load Balancer
/ | \
/ | \
AZ-A AZ-B AZ-C
│ │ │
App A App B App C
│ │ │
└──── Database ────┘If one zone fails:
AZ-A ✗traffic can potentially move to:
AZ-B
AZ-CIn active-passive:
Active
│
│ serves traffic
│
Passive
│
│ standbyIf active fails:
Active ✗
↓
Failover
↓
Passive becomes activeAdvantages:
Disadvantages:
Both instances serve traffic:
Load Balancer
/ \
/ \
Active A Active B
│ │
└──────┬──────┘
│
Data LayerIf A fails:
B continues servingAdvantages:
Challenges:
| Property | Active-Active | Active-Passive |
|---|---|---|
| Both serve traffic | Yes | No |
| Resource utilization | Higher | Lower |
| Failover | Usually faster | Usually slower |
| State complexity | Higher | Often lower |
| Operational complexity | Higher | Lower |
| Scalability | Good | More limited |
| Typical use | Stateless apps, global services | Some databases and failover systems |
Stateless services are easier to scale and fail over.
Bad architecture:
Session stored only in Server A memoryIf:
Server A failsthe user's session disappears.
Better:
Application Servers
│
▼
Shared session storeFor example:
Server A ─┐
Server B ─┼──► Redis / Database
Server C ─┘Now any server can handle the request.
A load balancer distributes requests:
Load Balancer
/ | \
▼ ▼ ▼
App A App B App CIf App B fails:
App A ✓
App B ✗
App C ✓health checks allow the load balancer to stop routing traffic to B.
The load balancer itself must also be highly available.
A health check may verify:
HTTP response
Process state
Database connectivity
Dependency healthExample:
GET /healthResponse:
200 OKBut a simple HTTP 200 may not prove the application is actually capable of serving production traffic.
This is why systems often distinguish:
Answers:
Is the process alive?
Answers:
Can this instance safely receive traffic?
Example:
Process alive ✓
Database connection ✗Liveness:
healthyReadiness:
not readyThe load balancer should generally avoid routing production traffic to an instance that is not ready.
HA does not always mean:
Everything works perfectly.Sometimes the best strategy is:
Core functionality continues
non-critical features degradeExample:
Shopping site
Checkout ✓
Payment ✓
Product page ✓
Recommendations ✗
Reviews ✗
Analytics ✗Users can still complete the critical operation.
During failures, categorize features:
Login
Checkout
Payment
Order retrievalSearch
Recommendations
ReviewsPersonalized analytics
Advanced suggestionsThis allows the system to shed non-critical work when dependencies fail.
Suppose:
Checkout Service
│
▼
Recommendation Service
│
XIf checkout synchronously depends on recommendations:
Recommendation failure
↓
Checkout failureThis is bad dependency design.
Instead:
Checkout
│
├── Payment
└── Inventory
Recommendations
│
└── OptionalNon-critical dependencies should not unnecessarily sit on critical request paths.
A cascading failure occurs when one failure causes additional failures.
Example:
Service A slows
↓
Service B waits
↓
Threads accumulate
↓
Service B slows
↓
Service C retries
↓
Traffic increases
↓
More overloadEventually:
Entire system degradesHA design must prevent cascades.
A circuit breaker protects services from repeatedly calling a failing dependency.
States:
CLOSED
↓
Failures increase
↓
OPEN
↓
Wait
↓
HALF-OPEN
↓
Test requestIf dependency recovers:
HALF-OPEN → CLOSEDIf failure continues:
HALF-OPEN → OPENA bulkhead isolates resources.
Suppose one dependency consumes all worker threads:
Dependency A
↓
All threads consumed
↓
Dependency B also failsInstead allocate separate pools:
Pool A → Dependency A
Pool B → Dependency BIf A fails:
Pool A exhausted
Pool B still worksThis is inspired by compartmentalization in ships.
When a system is overloaded, accepting every request can make the system collapse.
Instead:
Traffic
↓
Capacity exceeded
↓
Reject/defer lower-priority work
↓
Protect critical operationsExamples:
HTTP 429
Queue limits
Request prioritization
Admission controlBackpressure means:
A downstream system communicates that it cannot safely accept work at the current rate.
Example:
Producer
↓
Queue
↓
ConsumerIf consumer capacity is:
1,000 msg/secbut producer sends:
10,000 msg/secthe system needs:
Otherwise queues can grow without bound.
Suppose a dependency fails.
100 servers each retry:
1 request
↓
retry
↓
retry
↓
retryInstead of reducing traffic, retries can multiply it.
Example:
Normal:
10,000 req/sec
Failure:
10,000 original
+
20,000 retries
+
40,000 retries
...This can create a retry storm.
Use:
1s
2s
4s
8s
16splus random jitter:
1.2s
2.7s
4.1sThis prevents all clients from retrying at exactly the same moment.
A database is often the most important HA component.
Basic architecture:
Application
│
▼
Leader
/ \
▼ ▼
Replica A Replica BIf leader fails:
Leader ✗
↓
Election / failover
↓
Replica becomes leaderThe exact mechanism depends on the database.
The primary waits for required replicas before confirming the write.
Write
↓
Primary
↓
Replica
↓
ACKBenefits:
Costs:
Primary acknowledges before all replicas have applied the write.
Write
↓
Primary → ACK
Later:
Primary → ReplicaBenefits:
Cost:
Suppose:
Primary A
Replica B
Replica CA fails.
Failover process:
Detect failure
↓
Select candidate
↓
Promote candidate
↓
Redirect traffic
↓
Reconfigure replicationPotential risks:
Recovery Point Objective (RPO) asks:
How much data loss can the business tolerate?
Examples:
RPO = 0means:
No data loss acceptableAnother system might tolerate:
RPO = 5 minutesmeaning losing up to approximately five minutes of recent changes may be acceptable under a disaster scenario.
Recovery Time Objective (RTO) asks:
How quickly must the service be restored?
Example:
RTO = 5 minutesmeans the recovery process should restore the service within the defined target.
RPO and RTO are different:
RPO → data loss tolerance
RTO → downtime toleranceHA and disaster recovery overlap but are not identical.
HA focuses on:
Continuing service through failuresDR focuses more broadly on:
Recovering from major disastersExamples:
Example:
Global DNS / Router
/ \
/ \
Region A Region B
│ │
App + DB App + DBIf Region A becomes unavailable:
Traffic → Region BBut multi-region introduces difficult problems:
Region A
ACTIVE
Region B
PASSIVETraffic normally goes to A.
If A fails:
Failover
↓
Region BSimpler than active-active but may have:
Global Router
/ \
▼ ▼
Region A Region B
Active ActiveBoth regions serve users.
Benefits:
Challenges:
Failover depends on detecting failure.
Typical flow:
Health monitoring
↓
Failure suspected
↓
Confirm according to policy
↓
Remove unhealthy target
↓
Promote replacement
↓
Redirect traffic
↓
Verify recoveryToo aggressive:
false failoversToo conservative:
long downtimeSystem detects and performs failover.
Advantages:
Risks:
Human decides.
Advantages:
Risks:
Critical systems often combine automation with safeguards and controlled human intervention.
Deployments themselves can cause outages.
Bad strategy:
Stop all servers
↓
Deploy
↓
Start serversBetter:
A B C D
Update A
Update B
Update C
Update DMaintain capacity during the rollout.
Two environments:
Blue → current
Green → newDeploy to Green.
Test:
Green ✓Then switch traffic:
Blue ✗
Green ✓If necessary:
Green ✗
Blue ✓This can make rollback fast.
Send a small percentage of traffic to the new version:
Old → 95%
New → 5%Monitor:
If healthy:
20%
50%
100%If unhealthy:
New → 0%This reduces deployment blast radius.
Caching can improve availability by reducing dependency load.
Example:
Application
│
▼
Cache
│
X
Database unavailableIf cached data is still valid:
Serve cached responseThis is a form of graceful degradation.
But cached data may be stale, so the business must define acceptable staleness.
Queues can absorb temporary spikes.
Clients
│
▼
Queue
│
▼
WorkersIf workers temporarily slow down:
Queue buffers workThis can prevent immediate overload.
However:
A queue is not infinite capacity.
Eventually queue depth can become a failure signal.
Suppose:
Normal traffic = 5,000 req/sec
Peak traffic = 8,000 req/secIf infrastructure supports only:
5,500 req/secthe system is not truly highly available under peak conditions.
HA therefore requires enough capacity to survive:
Suppose normal capacity requires:
3 serversAn N+1 design has:
4 serversIf one fails:
3 remainand normal capacity can still be maintained.
If you need to survive two simultaneous failures:
Normal = 3
Extra = 2
Total = 5This increases resilience but also costs more.
The right redundancy level depends on:
Suppose two components are arranged in series:
A → BThe system is available only if both are available.
If:
Availability A = 99.9%
Availability B = 99.9%Then approximately:
System availability
= 0.999 × 0.999
= 0.998001
≈ 99.8001%Dependencies multiply the availability risk.
Suppose two independent servers serve the same function:
┌── Server A ──┐
Request → Response
└── Server B ──┘The system fails only if both fail.
If each has failure probability:
q = 1 - availabilitythen under a simplified independence assumption:
P(both fail) = q²So redundancy can dramatically improve availability.
Real-world failures are often correlated, so this calculation is an approximation.
Two replicas are not truly independent if they share:
Same rack
Same power source
Same zone
Same network path
Same software bug
Same deploymentExample:
Server A ─┐
Server B ─┼── same AZ
Server C ─┘An AZ outage can remove all three.
Therefore physical and logical separation matters.
Blast radius means:
How much of the system is affected by a failure?
Example:
One service instance fails
→ small blast radius
Entire availability zone fails
→ larger blast radius
Entire region fails
→ very large blast radiusGood HA architecture limits blast radius.
An SLO creates an availability budget.
Suppose:
SLO = 99.9%Allowed unavailability is approximately:
0.1%This becomes an error budget.
The team can use it for:
If the error budget is exhausted:
Slow risky changes
Focus on reliabilitySuppose monthly SLO:
99.9%Monthly allowed downtime is approximately:
43.2 minutesIf an incident consumes:
30 minutesremaining budget:
13.2 minutesThe exact accounting depends on the organization's measurement window and SLO definition.
Imagine a video platform.
Critical:
Video playbackOptional:
Recommendations
Comments
Personalized thumbnailsIf recommendation service fails:
Playback continuesFallback:
Trending videosThis is much better than:
Recommendation failure
↓
Entire website failsA practical architecture:
Global DNS
│
┌────────┴────────┐
│ │
Region A Region B
│ │
Load Balancer Load Balancer
/ \ / \
▼ ▼ ▼ ▼
App A App B App C App D
│ │ │ │
└──┬───┘ └──┬───┘
│ │
Cache Cache
│ │
└──────┬───────────┘
│
Replicated DBThis architecture can survive more failures than a single-server design, but the exact guarantees depend on replication and failover configuration.
When designing a highly available system:
Identify critical components.
Identify failure domains.
Remove single points of failure.
Add redundancy.
Add health checks.
Design failover.
Define RTO and RPO.
Add graceful degradation.
Protect against cascading failures.
Test failure scenarios.
Do not only ask:
How does the system work when everything is healthy?Also ask:
What if:
Server fails?
Database fails?
Cache fails?
Queue fails?
Network fails?
Zone fails?
Region fails?
Dependency slows?
Traffic doubles?
Deployment fails?
Configuration is wrong?This mindset is central to production system design.
Chaos engineering deliberately introduces failures to test resilience.
Examples:
Kill server
Introduce latency
Drop network packets
Terminate instances
Fail a dependency
Simulate zone outageThe goal is not to break systems randomly.
The goal is to validate:
Expected failure
↓
Expected response
↓
Service remains within SLOHigh availability means zero downtime.
No.
HA reduces downtime and improves resilience.
No practical system can assume that every failure will be invisible.
More replicas automatically make a system highly available.
Not necessarily.
If all replicas share one failure domain:
Same zone
Same network
Same deploymentone failure can still remove them all.
Active-active is always better than active-passive.
Not necessarily.
Active-active can increase:
The architecture should match requirements.
Automatic failover is always safer.
Automation can fail too.
Poorly designed failover can cause:
Failover must be carefully designed and tested.
Five nines means the system never fails.
No.
It means the defined service has a very small allowed unavailability budget over the measurement period.
The exact SLO definition matters.
High availability is the ability of a system to continue providing its intended service despite component failures. It is achieved using redundancy, failure detection, failover, fault isolation, graceful degradation, and appropriate capacity across failure domains.
Active-active has multiple instances serving traffic simultaneously, while active-passive keeps one or more standby instances that take over after failure. Active-active can improve utilization and failover speed but usually introduces greater state and coordination complexity.
Recovery Time Objective defines how quickly a service should be restored after a failure.
Recovery Point Objective defines how much data loss is acceptable after a failure or disaster.
Answer with a process:
Identify component
↓
Add redundancy
↓
Separate failure domains
↓
Add health checks
↓
Implement failover
↓
Test failureExample:
Single DB
↓
Primary + replicas
↓
Multi-AZ
↓
Automatic failoverUse:
Requirements:
Design:
Global Router
/ \
▼ ▼
Region A Region B
│ │
Load Balancer Load Balancer
/ \ / \
▼ ▼ ▼ ▼
App A App B App C App D
│ │ │ │
└──┬────┘ └──┬────┘
│ │
Cache Cache
│ │
└───────┬─────────┘
│
Database
│
Queue
│
WorkersFor every major component define:
Failure
↓
Detection
↓
Failover
↓
Fallback
↓
RecoverySuppose:
Normal traffic = 100,000 req/secYou deploy:
4 application serversEach supports:
30,000 req/secTotal capacity:
120,000 req/secNow one server fails.
Remaining capacity:
90,000 req/secNormal traffic:
100,000 req/secTherefore the system cannot maintain normal capacity after one failure.
A better design may require:
N+1 capacityrather than merely N instances.
Before calling a system highly available, ask:
High availability can be remembered as:
HIGH AVAILABILITY
│
┌─────────────┼─────────────┐
│ │ │
Redundancy Detection Recovery
│ │ │
▼ ▼ ▼
Multiple Health checks Failover
instances Heartbeats Retry
AZs Monitoring Recovery
Regions Degradation
│
▼
Failure isolation
│
▼
Cascading failure protection
│
▼
Capacity + TestingThe core principle is:
Assume components will fail; design the system so that one failure does not unnecessarily become a service outage.
You learned:
The most important takeaway:
High availability is not a single technology. It is an architectural property created by redundancy, failure detection, failover, isolation, capacity planning, graceful degradation, and continuous failure testing.
Calculate approximate annual downtime for:
99.9%
99.99%
99.999%Explain:
MTBF vs MTTRDesign an active-active application across three availability zones.
Identify all SPOFs in:
Client
↓
Load Balancer
↓
Server
↓
DatabaseExplain how a circuit breaker prevents cascading failures.
Design an N+1 architecture for:
100,000 req/secwhen one server can process:
25,000 req/secExplain:
RPO vs RTOusing a banking example.
Next we will go deeper into fault tolerance.
We will study:
Mark it as complete to track your mastery.