So far, we have built the foundation:
Chapter 01 → How a Web Application Works
Chapter 02 → Networking Fundamentals
Chapter 03 → DNS Deep Dive
Chapter 04 → HTTP & API Design
Chapter 05 → Servers & Application ArchitectureNow we reach one of the central questions in System Design:
How do we make an application handle more traffic?
Imagine an application currently handles:
1,000 requests/secondThen users grow and traffic becomes:
5,000 requests/secondLater:
50,000 requests/secondEventually:
1,000,000 requests/secondA system that works perfectly at 1,000 requests/sec may fail completely at 50,000.
Scaling is the process of increasing a system's capacity while keeping required properties such as:
Scaling means increasing the system's ability to handle workload.
Workload can increase because of:
A useful model is:
Traffic / Workload
↓
System Capacity
↓
Can capacity satisfy workload?If:
Workload ≤ Capacitythe system can potentially keep up.
If:
Workload > Capacityqueues grow, latency increases, and eventually requests may fail.
Capacity is the amount of work a component can handle under defined conditions.
For an API:
Capacity = 2,000 requests/secFor a database:
Capacity = 20,000 queries/secFor a worker pool:
Capacity = 500 jobs/secCapacity is not a universal fixed number.
It depends on:
Therefore:
"This server supports 10,000 requests/sec" is incomplete without specifying the workload and performance target.
A system can need scaling in different dimensions.
More CPU is needed.
More RAM is needed.
More data must be stored.
More bandwidth is needed.
More queries or data must be handled.
More concurrent users/connections exist.
More jobs need to be processed.
A common mistake is to scale the wrong resource.
Example:
CPU = 20%
Database = overloadedAdding more application CPU will not solve the database bottleneck.
Vertical scaling means making a machine more powerful.
Example:
Before:
4 CPU
16 GB RAM
↓
After:
16 CPU
64 GB RAMThis is also called:
Scale up
It is often simple.
Architecture:
Client
↓
Server
↓
Databasecan remain largely unchanged.
Advantages:
A machine has practical limits.
Eventually:
CPU cannot be increased indefinitely
RAM cannot be increased indefinitelyThere are also:
If everything depends on one machine:
Machine failure
↓
Entire service unavailableVertical scaling alone therefore does not solve every problem.
Horizontal scaling means adding more machines or instances.
Also called:
Scale out
Example:
Before:
Client
↓
Server 1
After:
┌── Server 1
Client → LB ──┼── Server 2
├── Server 3
└── Server 4Instead of one powerful machine handling all requests, multiple instances share the workload.
| Vertical Scaling | Horizontal Scaling |
|---|---|
| Bigger machine | More machines |
| Simpler initially | More architectural complexity |
| Hardware limits | Can scale across many instances |
| Often easier state management | Works best with stateless services |
| Single-machine failure can matter | Can provide redundancy |
| Scale-up | Scale-out |
A production system may use both.
Suppose one application server handles:
1,000 requests/secTraffic:
1,000 req/secOne server may be sufficient.
Traffic increases:
3,000 req/secPossible design:
Load Balancer
/ | \
↓ ↓ ↓
App 1 App 2 App 3If each instance handles approximately 1,000 req/sec, three instances provide roughly the required compute capacity, subject to real-world overhead and bottlenecks.
Horizontal scaling becomes much easier when application instances are stateless.
Bad design:
User
↓
App 1
↓
Session stored in App 1 memoryNext request:
User
↓
App 2
↓
Session missingBetter:
App 1 ─┐
App 2 ─┼──→ Shared Redis / Database
App 3 ─┘or use an appropriate stateless authentication mechanism.
Then:
Request 1 → App 1
Request 2 → App 3
Request 3 → App 2All instances can serve the user.
A load balancer distributes traffic across multiple servers.
Load Balancer
/ | \
↓ ↓ ↓
App 1 App 2 App 3It can:
We will study load balancers deeply in Chapter 07.
For now, remember:
Horizontal scaling usually needs a mechanism for distributing traffic.
Suppose:
Load Balancer
↓
App 1 ─┐
App 2 ─┼──→ Database
App 3 ─┤
App 4 ─┘We add more application servers:
App 1
App 2
App 3
...
App 100But the database remains:
One databaseNow the database may become the bottleneck.
This teaches an important principle:
Scaling one component can move the bottleneck to another component.
Consider:
Internet
↓
Load Balancer
↓
Application
↓
Cache
↓
DatabaseThe bottleneck might be:
Networkor:
Application CPUor:
Cacheor:
Databaseor:
External APISystem design is partly the process of identifying the current bottleneck and scaling the correct layer.
A useful relationship for capacity reasoning is:
L = λ × WWhere:
L = average number of items in the systemλ = arrival rateW = average time in the systemFor example:
Arrival rate = 100 requests/sec
Average response time = 0.5 secThen approximately:
L = 100 × 0.5
= 50So there are approximately 50 requests in the system on average.
This becomes useful when reasoning about concurrency.
These are different.
How much work is completed per unit time.
Example:
5,000 requests/secHow long one request takes.
Example:
120 msA system can have:
High throughput
+
High latencyor:
Low throughput
+
Low latencyYou must optimize for the actual requirement.
Before scaling, estimate the workload.
Suppose:
10 million usersBut only:
10% active dailyThen:
1 million DAUSuppose each active user generates:
20 API requests/dayThen:
1,000,000 × 20
= 20,000,000 requests/dayAverage requests/sec:
20,000,000 / 86,400
≈ 231 requests/secBut average traffic is not enough.
Peak traffic may be much higher.
Suppose average traffic is:
231 req/secbut peak traffic is 10× average:
≈ 2,310 req/secThe system must be designed around an appropriate peak requirement rather than only the daily average.
Peak factors can come from:
Do not design capacity exactly at expected traffic.
Suppose expected peak:
2,000 req/secand measured safe capacity:
2,500 req/secHeadroom:
500 req/secor:
25%Headroom helps absorb:
The appropriate amount depends on the system.
A typical scalable application architecture:
Internet
↓
Load Balancer
↓
┌────────────┼────────────┐
↓ ↓ ↓
App 1 App 2 App 3
│ │ │
└────────────┼────────────┘
↓
Cache
↓
DatabaseThe application tier is horizontally scalable.
Instead of manually adding servers:
Traffic increases
↓
More instancesan autoscaling system can adjust capacity automatically.
Conceptually:
CPU / RPS / Queue Depth
↓
Autoscaler
↓
Add / Remove instancesExample:
Target CPU = 60%If average CPU becomes:
85%the system can add instances.
If traffic decreases:
CPU = 20%it may remove instances.
CPU is useful, but not always the best scaling metric.
For APIs, requests per second can be a better signal.
Example:
Target = 500 req/sec/instanceIf:
Total traffic = 2,000 req/secwe may need roughly:
4 instancesassuming each instance can reliably handle the target workload.
For asynchronous systems:
Queue depth
↓
Autoscaler
↓
WorkersExample:
Queue = 10,000 jobs
↓
Add workersWhen:
Queue = 50 jobsworkers can scale down.
This is often more meaningful than CPU for background workloads.
Autoscaling needs policies.
Examples:
If CPU > 70% for 5 minutes
add instancesor:
If queue depth > 5,000
add workersThere are trade-offs:
Suppose an autoscaler behaves like this:
CPU high
↓
Add 10 servers
↓
CPU becomes low
↓
Remove 10 servers
↓
CPU high again
↓
Add 10 serversThis repeated scaling behavior is undesirable.
It can be caused by:
Autoscaling systems need stabilization.
Suppose every request queries the database.
100,000 requests/sec
↓
100,000 DB queries/secThis may overwhelm the database.
Add caching:
100,000 requests/sec
↓
Cache
├── 90,000 hits
└── 10,000 misses
↓
DatabaseThe database load can drop substantially.
Caching is therefore not only a latency optimization.
It can also be a scalability mechanism.
We will study caching deeply later.
Suppose:
90% reads
10% writesScaling strategies can include:
The correct strategy depends on access patterns.
Suppose:
20% reads
80% writesCaching alone may not solve the bottleneck.
Potential strategies include:
These will become important later in the course.
Suppose an API receives:
POST /generate-reportGenerating the report takes:
30 secondsInstead of holding the HTTP request open:
Client
↓
API
↓
Generate report
↓
30 seconds
↓
Responsewe can use:
Client
↓
API
↓
Queue
↓
202 AcceptedThen:
Queue
↓
Worker
↓
Generate report
↓
StorageThis allows the API tier to handle more concurrent requests.
A queue can absorb temporary bursts.
Suppose:
Normal workload = 1,000 jobs/sec
Burst = 5,000 jobs/sec
Worker capacity = 1,500 jobs/secWithout a queue:
4,000 jobs/sec rejected/failedWith a queue:
5,000 jobs/sec
↓
Queue
↓
1,500 jobs/sec processed
↓
Backlog gradually decreasesA queue does not create infinite capacity.
It changes immediate rejection into delayed processing.
Batching and queueing can improve throughput.
But users may experience additional latency.
For example:
Synchronous:
request → process → response
latency = 100 msAsynchronous:
request → queue → worker → result
latency = secondsThe correct choice depends on whether the operation must be synchronous.
A database can become the main bottleneck.
Possible strategies include:
Vertical scaling
Caching
Read replicas
Partitioning
Sharding
Database optimization
Connection poolingWe will cover these systematically in later chapters.
For now, understand:
Adding more application servers does not automatically scale the database.
For read-heavy workloads:
Primary DB
/ \
↓ ↓
Replica 1 Replica 2Writes go to the primary.
Reads can potentially go to replicas.
Example:
Write
↓
Primary
Read
↓
ReplicaThis can increase read capacity.
But replication introduces consistency considerations.
We will study replication in Chapter 13.
Suppose users repeatedly request:
images
videos
CSS
JavaScriptServing everything from application servers is inefficient.
A CDN can cache content near users.
User
↓
CDN Edge
├── Cache Hit → Response
└── Cache Miss
↓
OriginThis reduces:
CDNs will be covered in Chapter 08.
Suppose:
20 application serversEach has:
50 DB connectionsPotential total:
20 × 50 = 1,000 DB connectionsIf the database safely supports only 300 connections, simply adding application servers can make the database problem worse.
This is a crucial scaling lesson:
Every horizontally scaled layer can multiply pressure on its dependencies.
Suppose one API request calls:
Service A
Service B
Service C
Service DOne incoming request creates four downstream requests.
If traffic is:
10,000 API requests/secthen downstream traffic can become approximately:
40,000 requests/secbefore considering retries or additional calls.
This is called fan-out.
High fan-out can create scaling problems.
API
│
┌───────────┼───────────┐
↓ ↓ ↓
Service A Service B Service C
↓ ↓ ↓
Database Cache External APIIf Service B becomes slow, the whole API may become slow.
Scaling the API servers may not solve the problem.
Consider:
Service A
↓
Service B
↓
DatabaseDatabase becomes slow.
Then:
Service B waits
↓
connections occupied
↓
Service A waits
↓
API requests accumulate
↓
Application workers exhausted
↓
Entire service becomes unavailableThis is a cascading failure.
Scaling requires thinking about dependencies, not only the main server.
Suppose traffic suddenly increases from:
10,000 req/secto:
100,000 req/secAutoscaling takes time.
Rate limiting can protect the system during the scaling delay.
Traffic
↓
Rate Limiter
↓
Allowed traffic
↓
ApplicationExcess traffic may receive:
429 Too Many RequestsThis protects system capacity.
Suppose:
Capacity = 10,000 req/sec
Traffic = 20,000 req/secThe system may intentionally reject some requests.
Possible strategy:
Critical requests → allow
Low-priority requests → reject/deferThis is especially useful when not all operations are equally important.
Consider an e-commerce system.
High priority:
Payment
Order creation
CheckoutLower priority:
Recommendations
Analytics
Personalized suggestionsDuring overload:
Keep checkout functioning
Reduce recommendation trafficThis is a form of graceful degradation and load prioritization.
A common misconception:
"Just keep adding servers."
In reality, bottlenecks can move to:
Database
Cache
Network
Load Balancer
Storage
External APIs
Queues
Coordination systemsEventually the architecture itself may need to change.
Even the load balancer can become a bottleneck.
Large systems may use:
DNS / Anycast
↓
Multiple load balancers
↓
Application clustersor managed load-balancing infrastructure.
The principle is:
A supposedly central component must also have sufficient capacity and redundancy.
Storage requirements can grow from:
100 GB
↓
1 TB
↓
100 TB
↓
PetabytesAt large scale, systems may use:
Storage architecture becomes its own system-design problem.
Suppose one dataset is too large for one machine.
Partition it:
Data
├── Partition 1
├── Partition 2
├── Partition 3
└── Partition 4Each partition stores a subset of the data.
Example:
Users A–F → Partition 1
Users G–L → Partition 2
Users M–R → Partition 3
Users S–Z → Partition 4This concept leads toward sharding.
We will study database partitioning and sharding later.
Replication means maintaining multiple copies.
Example:
Primary
↓
Replica 1
Replica 2
Replica 3Replication can improve:
But it creates:
Replication stores additional copies in persistent or semi-persistent systems.
Caching stores frequently needed data closer to consumers.
Client
↓
Cache
↓ miss
DatabaseThe key trade-off is:
freshness
↕
performanceWe will study this deeply in caching chapters.
Some work does not need to happen during the user request.
Move it out:
Synchronous path
↓
Queue
↓
WorkersThis reduces the work performed directly on the request path.
Example:
User registration
↓
Create account
↓
Return response
Background:
↓
Send welcome email
↓
Generate analytics event
↓
Update recommendation modelThe critical path is the work required before a response can be returned.
Example:
Request
↓
Authentication
↓
Database
↓
ResponseIf email sending is not required before the response:
Request
↓
Authentication
↓
Database
↓
Response
Background:
↓
EmailRemoving unnecessary work from the critical path improves latency and often scalability.
Suppose:
API
↓
Service A
↓
Service B
↓
Service Cbut Service C only performs analytics.
If analytics does not affect the user response:
API
↓
Service A
↓
Service B
↓
Response
Background:
Service CThis can significantly reduce request-path work.
Suppose:
Peak traffic = 10,000 req/secOne application instance safely handles:
500 req/secRequired instances:
10,000 / 500
= 20Suppose we want 25% headroom:
20 × 1.25
= 25 instancesIf we also want to survive the loss of 5 instances:
25 + 5
= 30 instancesThis is a simplified capacity model.
Real capacity planning should account for workload variation, dependency limits, autoscaling delay, failure domains, and measured performance.
Suppose:
10 million redirects/dayAverage:
10,000,000 / 86,400
≈ 116 requests/secSuppose peak is:
10× average
≈ 1,160 requests/secArchitecture:
Users
↓
CDN / Load Balancer
↓
Stateless API Servers
↓
Redis Cache
↓
DatabaseMost popular URLs can be served from cache.
Suppose users upload:
10,000 images/minuteDo not make application servers perform every operation synchronously.
Better:
Client
↓
Upload API
↓
Object Storage
↓
Queue
↓
Image Workers
├── Resize
├── Compress
└── Generate thumbnailsThis separates upload traffic from processing capacity.
Video processing is CPU/GPU intensive.
A better architecture:
Client
↓
Upload Service
↓
Object Storage
↓
Job Queue
↓
Worker Fleet
├── Worker 1
├── Worker 2
├── Worker 3
└── Worker N
↓
Processed Video
↓
Object StorageWorkers can scale according to:
queue depth
processing time
CPU/GPU utilizationSuppose:
1 million notificationsInstead of sending synchronously:
API
↓
Send 1 million notifications
↓
Responseuse:
API
↓
Queue
↓
Notification Workers
↓
Email/SMS/Push providersThe queue absorbs bursts and workers control processing rate.
Queue
│
Queue depth = 50,000
│
▼
Autoscaler
┌───────┼───────┐
▼ ▼ ▼
Worker Worker Worker
│ │ │
└───────┼───────┘
▼
StorageAs backlog increases:
queue depth ↑
↓
workers ↑As backlog decreases:
queue depth ↓
↓
workers ↓Scaling is not only about performance.
Suppose:
10 servers → ₹X/month
100 servers → ₹10X/monthIf traffic grows only 2× but infrastructure cost grows 10×, architecture may need optimization.
System design balances:
Performance
Availability
Reliability
Complexity
CostRunning too much capacity can waste resources.
Example:
Traffic = 100 req/sec
Capacity = 100,000 req/secThe system may be extremely reliable, but unnecessarily expensive.
The correct capacity depends on:
Under-provisioning causes:
High CPU
↓
High latency
↓
Timeouts
↓
Retries
↓
More traffic
↓
Higher CPU
↓
FailureThis can create a feedback loop.
A particularly dangerous pattern:
Service overloaded
↓
Requests become slow
↓
Clients timeout
↓
Clients retry
↓
Traffic increases
↓
Service becomes even slowerTherefore:
Scaling and retry policies must be designed together.
Retries without capacity planning can amplify failures.
Suppose one instance normally processes:
500 req/secAt overload:
700 req/seclatency rises.
Clients timeout.
Retries create:
900 req/secNow the system is even more overloaded.
Therefore production systems need:
A practical approach:
Find:
RPS
latency
CPU
memory
DB load
cache hit rate
queue depth
error rateDo not guess.
Remove unnecessary work.
If workload is cache-friendly.
Add instances where appropriate.
Database, queue, storage, etc.
Move non-critical work off the request path.
Automate capacity changes.
Rate limiting, backpressure, load shedding.
The database may be the real bottleneck.
Local sessions can prevent effective horizontal scaling.
Peak traffic matters.
The system can fail during small traffic spikes.
Retries can amplify outages.
More servers can create too many database connections.
Expensive background work should often be asynchronous.
Queues delay work; they do not eliminate capacity limits.
CPU may be low while queue depth or latency is extremely high.
Always identify the bottleneck first.
Vertical scaling makes one machine more powerful.
Horizontal scaling adds more machines or instances.
Because requests can be routed to any healthy instance without depending on local state.
You increase application compute capacity, but may increase pressure on:
Possible mechanisms:
CDN
Caching
Load balancing
Autoscaling
Rate limiting
Queueing
Load shedding
Graceful degradationThe correct combination depends on workload.
A simplified estimate:
Required instances
≈ Peak traffic / Safe capacity per instanceThen add appropriate headroom and failure capacity.
Because the system must remain within acceptable performance limits during expected high-load periods.
It reduces repeated work and decreases load on downstream systems such as databases.
When the work does not need to complete before the user response and can tolerate asynchronous processing.
No.
The bottleneck may move to:
Database
Cache
Network
Storage
Queue
External dependency
Coordination layerA mechanism that prevents incoming work from overwhelming downstream capacity.
Design a scalable URL shortener.
Requirements:
100 million URLs
10 million redirects/day
10× peak traffic
99.9% availability
Low redirect latencyStart with:
Client
↓
Application
↓
DatabaseThen evolve it.
Client
↓
App
↓
DatabaseClient
↓
Load Balancer
↓
App 1 / App 2 / App 3
↓
DatabaseClient
↓
Load Balancer
↓
Stateless App Servers
↓
Redis
↓
DatabaseNow answer:
Design:
Internet
│
▼
CDN
│
▼
Load Balancer
│
┌──────────┼──────────┐
▼ ▼ ▼
App 1 App 2 App 3
│ │ │
└──────────┼──────────┘
│
┌───────┴────────┐
▼ ▼
Redis PostgreSQL
│
▼
ReplicaAdd:
Queue
↓
WorkersThen define:
When someone asks:
"How would you scale this system?"
Do not immediately say:
"Add more servers."
Instead think:
1. What is the workload?
↓
2. What is the peak?
↓
3. What is the current capacity?
↓
4. What is the bottleneck?
↓
5. Can we optimize it?
↓
6. Can we cache?
↓
7. Can we scale horizontally?
↓
8. Are instances stateless?
↓
9. Can dependencies handle the added load?
↓
10. Can expensive work become asynchronous?
↓
11. Do we need autoscaling?
↓
12. How do we protect the system during overload?This is much stronger than simply proposing more machines.
You should now understand:
Before moving forward, make sure you can explain:
We now have multiple application servers.
The next question is:
How do we distribute traffic intelligently across them?
We will deeply study:
Client
↓
Load Balancer
├── Server 1
├── Server 2
├── Server 3
└── Server 4Topics will include:
Mark it as complete to track your mastery.