Disaster Recovery (DR) is the discipline of preparing a system to recover from a major disruptive event and restore critical business operations within an acceptable time.
A production system can be highly available and fault tolerant and still need disaster recovery.
For example:
Those are normal failure scenarios.
But what happens when:
That is where Disaster Recovery becomes critical.
Disaster Recovery is the set of:
required to restore a system after a major disaster.
A simple model is:
Normal Operation
|
v
+----------------+
| Production |
| Environment |
+----------------+
|
| disaster
v
+----------------+
| Recovery |
| Environment |
+----------------+
|
v
Restore Service
|
v
Resume BusinessDR is therefore not simply:
"Take backups."
It is a complete operational strategy.
These concepts are related but different.
| Concept | Main Goal |
|---|---|
| Fault tolerance | Continue operating despite failures |
| High availability | Minimize service downtime |
| Disaster recovery | Recover after a major disaster |
| Backup | Preserve recoverable copies of data |
| Business continuity | Keep critical business operations running |
Example:
A server crashes.
Server A
X
|
v
Server BThe load balancer sends traffic to Server B.
That is fault tolerance / high availability.
Now imagine:
Entire Region
X
|
v
Another RegionMoving the workload to another region is a disaster recovery concern.
A system can have:
and still lose data or become unavailable.
Why?
Because redundancy does not protect against every failure.
Consider:
Application
|
v
Database
|
+---- Replica
|
+---- ReplicaIf an administrator accidentally executes:
DROP TABLE customers;replication may faithfully replicate the destructive operation.
So:
Replication != BackupThis is one of the most important DR concepts.
Disasters can have different causes.
Examples:
Usually handled partly through HA and redundancy.
An entire data center may become unavailable.
Examples:
Multi-AZ architectures can reduce this risk.
An entire cloud region may become unavailable.
Example:
Region A
---------
Application
Database
Cache
Storage
X
Region B
---------
Recovery EnvironmentThis requires cross-region DR.
Data may become corrupted because of:
This is especially dangerous because corrupted data can be replicated.
Examples:
DELETE FROM users;or:
rm -rf important-dataor an administrator deletes a production storage bucket.
Backups and retention policies are critical here.
Examples:
DR architecture should consider security isolation for backups.
A software deployment can cause widespread failure.
Example:
Version 1
|
v
Version 2
|
v
Database migration
|
X
Millions of records corruptedRollback alone may not recover corrupted data.
Point-in-time recovery may be required.
Before designing DR, understand what the business actually needs.
Not every system requires the same recovery objective.
Consider two systems:
Downtime may immediately stop transactions.
Several hours of downtime may be acceptable.
Therefore DR requirements should be driven by business impact.
Important questions:
RPO answers:
How much data loss can the business tolerate?
Suppose:
10:00 ---- 11:00 ---- 12:00 ---- DisasterIf RPO is 1 hour, losing approximately the latest hour of data may be acceptable.
If RPO is 5 minutes, the recovery architecture must preserve data much more frequently.
A payment system:
RPO = near zeroA reporting system:
RPO = 24 hoursRPO is fundamentally about data loss tolerance.
RTO answers:
How long can the system remain unavailable?
Example:
Disaster occurs
|
| 15 minutes
v
Service restoredThen:
RTO = 15 minutesRTO is fundamentally about downtime tolerance.
| Property | RPO | RTO |
|---|---|---|
| Meaning | Maximum acceptable data loss | Maximum acceptable downtime |
| Focus | Data | Time |
| Question | How much data can we lose? | How quickly must we recover? |
Example:
RPO = 5 minutes
RTO = 30 minutesMeans:
A backup is a separate recoverable copy of data.
Basic model:
Production
|
v
Backup System
|
v
Backup StorageGood backup systems consider:
A full backup copies the entire dataset.
Example:
Monday
Full Backup
Tuesday
Full Backup
Wednesday
Full BackupAdvantages:
Disadvantages:
An incremental backup stores changes since the previous backup.
Example:
Sunday
Full Backup
Monday
Changes since Sunday
Tuesday
Changes since Monday
Wednesday
Changes since TuesdayAdvantages:
Disadvantages:
A differential backup stores changes since the last full backup.
Example:
Sunday
Full
Monday
Changes since Sunday
Tuesday
Changes since Sunday
Wednesday
Changes since SundayRestore generally requires:
Full backup
+
Latest differentialA snapshot captures the state of storage at a point in time.
Conceptually:
Database Volume
|
+---- Snapshot T1
|
+---- Snapshot T2
|
+---- Snapshot T3Snapshots can be useful for:
But snapshots should not automatically be treated as a complete DR strategy.
If snapshots exist only inside the same failure domain, a larger infrastructure failure may affect them too.
Point-in-time recovery allows restoring data to a selected point in time.
A common architecture is:
Base Backup
+
Transaction Logs / WAL
|
v
Target Recovery TimeFor example:
12:00 backup
12:05 transaction log
12:10 transaction log
12:15 transaction log
12:20 bad deploymentYou may restore to:
12:19instead of restoring only to 12
.This is extremely useful for:
This distinction is critical.
Replication:
Primary
|
v
ReplicaBackup:
Primary
|
v
Independent Recovery CopyReplication improves availability.
Backups improve recoverability.
A replica can contain the same corrupted data as the primary.
A backup may preserve a historical clean state.
Therefore:
HA + Replication + Backupis stronger than any one of them alone.
Backups should not be kept indefinitely without a policy.
A retention policy might be:
Hourly -> 48 hours
Daily -> 30 days
Weekly -> 12 weeks
Monthly -> 7 yearsThe exact policy depends on:
A commonly used backup strategy is:
3 copies of data
2 different storage/media types
1 copy off-siteConceptually:
Production
|
+---- Local backup
|
+---- Secondary storage
|
+---- Off-site backupThe principle reduces dependence on a single failure domain.
For modern cloud systems, the exact implementation can vary, but the underlying idea remains useful:
Do not let one disaster destroy every copy.
A backup can be protected from modification or deletion for a defined period.
Conceptually:
Backup
|
v
Immutable Storage
|
X
Cannot be modified during retention periodThis can protect against:
The important principle is:
A backup that an attacker can immediately delete may not be a reliable recovery boundary.
Backups may contain sensitive information.
Therefore protect:
A strong design separates backup access from normal production access where practical.
One of the biggest DR mistakes is:
"We have backups, therefore we can recover."
A backup is useful only if it can actually be restored.
A DR program should test:
Backup
|
v
Restore
|
v
Validate
|
v
Application starts
|
v
Data verifiedA recovery drill simulates a disaster.
Example:
Scenario:
Primary region unavailableTeam performs:
The goal is to discover gaps before a real disaster.
DR environments can be classified by how ready they are.
Infrastructure is mostly absent or minimally prepared.
Primary Region
|
X
|
v
Recovery Region
|
Provision infrastructure
|
Restore data
|
Start servicesAdvantages:
Disadvantages:
A recovery environment is partially running.
Primary Region
|
| replication
v
Warm Region
|
small running capacityDuring disaster:
Scale up
+
Promote database
+
Route trafficThis provides a balance between cost and recovery speed.
The recovery environment is already running.
Region A
========
Application
Database
Cache
|
| replication
v
Region B
========
Application
Database
CacheFailover can be relatively fast.
But it is expensive and operationally complex.
In active-passive architecture:
Traffic
|
v
+---------------+
| Region A |
| ACTIVE |
+---------------+
|
replication
|
v
+---------------+
| Region B |
| PASSIVE |
+---------------+During failure:
Region A
X
|
v
Promote Region B
|
v
Route trafficBoth regions serve traffic.
Users
|
Global Traffic
Distribution
/ \
v v
Region A Region B
ACTIVE ACTIVEAdvantages:
Challenges:
A DR architecture often needs data in another region.
Example:
Region A
Primary DB
|
| async replication
v
Region B
Standby DBAsynchronous replication introduces a possible RPO gap.
If:
Primary writes:
A
B
C
D
Replica receives:
A
B
Cand the primary region fails, D may be lost.
Therefore replication strategy must be matched to the RPO requirement.
A write may require acknowledgement from multiple locations.
Advantages:
Disadvantages:
The primary confirms the write before the remote copy necessarily receives it.
Advantages:
Disadvantages:
There is a trade-off:
Consistency / RPO
vs
Latency / AvailabilityWhen the primary region fails, traffic needs to reach the recovery environment.
Common mechanisms include:
Conceptually:
Users
|
v
Global Traffic Manager
|
+------ Region A
|
+------ Region BDuring failure:
Region A = unhealthy
Traffic
|
v
Region BDNS can direct clients toward a healthy endpoint.
Example:
app.example.com
|
v
DNS
/ \
Region A Region BIf Region A becomes unhealthy:
DNS
|
v
Region BHowever, DNS failover is not instantaneous.
Caching and TTL behavior can influence how quickly clients observe the change.
A runbook is a documented sequence of operational actions.
A DR runbook might contain:
1. Declare disaster
2. Assign incident commander
3. Confirm affected region
4. Freeze dangerous changes
5. Validate recovery environment
6. Promote database
7. Scale application
8. Update traffic routing
9. Validate critical workflows
10. Monitor error rates
11. Communicate status
12. Begin recovery of primary regionThe exact commands depend on the infrastructure.
During a major incident:
A documented runbook reduces dependency on individual knowledge.
Good runbooks include:
A common mistake is recovering only the application.
Imagine:
Application
|
+---- Database
|
+---- Authentication
|
+---- Payment provider
|
+---- Object storage
|
+---- DNS
|
+---- Secrets
|
+---- Message queueIf the application is restored but authentication is unavailable, the system may still be unusable.
Therefore DR planning must identify dependencies.
Create a dependency graph:
User
|
v
DNS
|
v
Load Balancer
|
v
Application
|
+---- Database
|
+---- Cache
|
+---- Queue
|
+---- Object Storage
|
+---- Authentication
|
+---- External APIsThen ask:
Can every critical dependency recover?
Recovery should follow dependency order.
For example:
1. Networking
2. DNS / traffic management
3. Identity / secrets
4. Database
5. Object storage
6. Message infrastructure
7. Application services
8. Background workers
9. Analytics / non-critical systemsThe exact order varies by architecture.
Disaster recovery is part of a broader concept called Business Continuity.
Business continuity asks:
How does the organization continue critical operations during disruption?
This includes:
DR focuses heavily on technology recovery, while business continuity has a broader organizational scope.
Consider an e-commerce system.
Users
|
v
Global Traffic Manager
/ \
v v
Region A Region B
ACTIVE STANDBY
| |
+-------+-------+ |
| | | |
API Queue Cache |
| replication
v |
Primary DB ------------------+
|
v
Object Storage
|
v
Cross-region backupDuring normal operation:
Region A = primary
Region B = recoveryDuring disaster:
Region A
X
Global Traffic
|
v
Region B
|
v
Promote database
|
v
Scale services
|
v
Validate applicationPayments require special care because duplicate processing can be dangerous.
Suppose:
User pays ₹1,000The request succeeds at the payment provider, but the application crashes before recording the result.
A retry could accidentally create another payment.
Therefore DR architecture should include:
Recovery should not simply replay every request blindly.
These are different.
Application becomes availableCorrect data state is restoredA system can be:
UP but WRONGThat is still a serious disaster.
For example:
Application = HTTP 200
Database = corruptedTherefore DR validation must verify correctness, not merely process availability.
After recovery, validate:
Business-level validation is essential.
DR testing can range from simple to highly realistic.
Teams discuss:
"What would we do if Region A disappeared?"
Low risk.
Teams execute portions of the recovery procedure.
Traffic is actually moved.
A realistic disaster scenario is executed with production-like conditions.
Higher realism means higher risk, so testing must be carefully controlled.
A recovery test may reveal:
This is exactly why testing matters.
Useful metrics include:
Actual data loss during testCompare against target RPO.
Actual recovery timeCompare against target RTO.
successful backups / total backupssuccessful restores / restore attemptsPrimary time - Replica timePercentage of critical workflows that work after recovery.
It is not.
Untested backups are assumptions.
A regional disaster can affect both production and backups.
Availability does not guarantee correctness.
Application recovery is not enough.
People may not know what to do during the incident.
Without measurable objectives, DR architecture cannot be evaluated properly.
A system should match DR investment to business requirements.
Automatic failover itself needs testing.
After the immediate disaster, the organization still needs a controlled path back to normal operations.
When designing DR, ask:
1. What can fail?
2. What is the business impact?
3. What RPO is required?
4. What RTO is required?
5. Which data needs backup?
6. Where should backups live?
7. How frequently should backups occur?
8. How long should backups be retained?
9. How is data restored?
10. How is traffic redirected?
11. What dependencies must recover?
12. Who owns each recovery step?
13. How is recovery tested?
14. How is recovery validated?
15. What happens after recovery?A simplified decision model:
| Requirement | Possible Strategy |
|---|---|
| Low cost, long recovery | Cold standby |
| Moderate recovery requirement | Warm standby |
| Fast recovery | Hot standby |
| Very low RPO | Frequent/synchronous replication where appropriate |
| Historical recovery | Backups + point-in-time recovery |
| Regional resilience | Cross-region architecture |
| High traffic continuity | Active-active |
The architecture must be derived from the required RPO, RTO, business impact, and budget.
Think of DR as multiple layers.
+------------------------------+
| Business Continuity |
+------------------------------+
| DR Runbooks & Operations |
+------------------------------+
| Traffic Failover |
+------------------------------+
| Application Recovery |
+------------------------------+
| Data Recovery |
+------------------------------+
| Backup / Replication |
+------------------------------+
| Infrastructure Recovery |
+------------------------------+A mature DR strategy addresses all layers.
Requirements:
RPO = 24 hours
RTO = 4 hoursA reasonable approach could be:
Primary
|
+---- Daily backups
|
+---- Off-site backup
|
+---- Infrastructure as Code
|
+---- Documented restore procedureThe startup does not necessarily need two fully active regions.
Requirements:
RPO = minutes or lower
RTO = minutesPotential architecture:
Global Traffic
|
+-----+------+
| |
Region A Region B
Active Active/Standby
| |
DB ---------- DB
| |
Backups BackupsAdditional mechanisms:
Infrastructure as Code can make recovery easier.
Instead of manually rebuilding:
Server
Database
Load Balancer
Network
IAM
Queuesconfiguration can be represented as code.
Conceptually:
Infrastructure Code
|
v
Recovery Region
|
v
Rebuild InfrastructureBut IaC alone is not DR.
It does not automatically solve:
Recovery environments need access to:
A common disaster is:
Application restored
|
v
Secret unavailable
|
v
Application cannot startSecrets therefore need their own recovery strategy.
Queues require special consideration.
Suppose:
Producer
|
v
Queue
|
v
WorkerDuring disaster:
DR design must preserve the semantics of the messaging system.
Object storage often contains:
Consider:
Primary Bucket
|
v
Replication
|
v
Recovery BucketAlso consider:
Kubernetes can recreate application infrastructure, but Kubernetes itself does not magically recover application state.
A recovery strategy may involve:
Git / IaC
+
Container Registry
+
Kubernetes Manifests
+
Cluster Recovery
+
Database Recovery
+
Object Storage Recovery
+
Secrets RecoveryThe cluster is only one component of the system.
During recovery, observability is essential.
Monitor:
A recovery system without visibility is difficult to operate safely.
INCIDENT: Primary Region Failure
Phase 1 — Detection
1. Confirm outage.
2. Check monitoring.
3. Verify regional scope.
Phase 2 — Decision
4. Declare DR event.
5. Assign incident commander.
6. Freeze non-essential changes.
Phase 3 — Data
7. Check replication status.
8. Determine latest recoverable point.
9. Promote recovery database.
Phase 4 — Infrastructure
10. Scale recovery environment.
11. Validate dependencies.
12. Load secrets/configuration.
Phase 5 — Traffic
13. Shift traffic.
14. Validate health checks.
Phase 6 — Validation
15. Test login.
16. Test read operations.
17. Test write operations.
18. Test critical business workflows.
Phase 7 — Stabilization
19. Monitor metrics.
20. Communicate status.
21. Start primary-region recovery.
Phase 8 — Post-Incident
22. Reconcile data.
23. Review RPO/RTO.
24. Document lessons.
25. Improve runbook.DR is the strategy and set of processes used to recover systems and business operations after major disruption.
Maximum acceptable data loss measured in time.
Maximum acceptable downtime before service recovery.
No. Replication primarily supports availability and continuity; backups provide historical recovery points.
A recovery environment that requires significant preparation before becoming operational.
Active-active has multiple environments serving traffic, while active-passive keeps one environment primarily as a standby.
Because successful backup creation does not guarantee successful restoration.
To protect against failures affecting the primary geographic failure domain.
Recovery of data to a selected historical point using backups and transaction/log information.
Because the corruption can be replicated to the secondary copy.
Discuss:
Based on required RTO, RPO, cost, operational complexity, and business impact.
Consider:
Possible strategy:
Detect corruption
|
Identify clean recovery point
|
Restore backup
|
Replay logs to target point
|
Validate data
|
Recover application
|
Reconcile missing/new transactionsIt must be:
Design DR for:
E-Commerce PlatformRequirements:
Users: 10 million
Daily orders: 2 million
RPO: 5 minutes
RTO: 30 minutes
Primary region: Region A
Recovery region: Region BRegion A
|
Application ClusterPrimary DB
|
v
Cross-region ReplicaContinuous / frequent logs
+
Daily full backups
+
Immutable retentionGlobal Traffic Manager
|
+---+---+
| |
Region A Region BFailure
|
v
Detect
|
v
Promote Region B
|
v
Route traffic
|
v
ValidateRemember these principles.
Replication -> availability
Backup -> recoveryHow much can we lose?How quickly must we recover?Application alone != recovered systemBackup without restore testing = assumptionSystem UP + wrong data = disasterArchitecture
+
Data
+
Automation
+
Runbooks
+
People
+
TestingIn this chapter we learned:
The central idea is:
Disaster Recovery is the ability to restore a correct and usable system after a major failure, within the business's required RPO and RTO.
Explain the difference between:
HA
Fault Tolerance
Backup
DR
Business ContinuityA company says:
RPO = 1 hour
RTO = 4 hoursExplain exactly what those requirements mean.
Design a DR architecture for a URL shortener.
Consider:
A database replica contains corrupted data because an application bug replicated bad writes.
How would you recover?
Design a DR strategy for a payment service with:
RPO = 0 or near zero
RTO = 5 minutesExplain the trade-offs.
A mature production system does not ask only:
"What happens when a server crashes?"
It also asks:
"What happens when the failure is bigger than our redundancy?"
That leads to the complete recovery model:
Failure
|
v
Detection
|
v
Decision
|
v
Failover / Restore
|
v
Data Recovery
|
v
Service Recovery
|
v
Validation
|
v
Stabilization
|
v
Reconciliation
|
v
Post-Incident ImprovementThat is the core of Disaster Recovery.
The next chapter moves from resilience to security architecture.
We will cover:
Mark it as complete to track your mastery.