File upload looks simple:
User → Server → DiskAt small scale, this can work.
At production scale, however, files introduce very different requirements from normal API requests:
This chapter connects the storage concepts from Chapter 17 with a complete production-grade media system.
A simple architecture is:
Browser
|
| POST /upload
v
Application Server
|
v
Object StorageFor a small application this may be acceptable.
The application receives the entire file and then sends it to object storage.
The problem is that the application server becomes part of the file-data path.
For a 1 GB file:
User
|
| 1 GB
v
App Server
|
| 1 GB
v
Object StorageThe application server consumes:
At large scale, this becomes inefficient.
A common production architecture is:
┌─────────────────┐
│ Application API │
└────────┬────────┘
|
create upload
|
v
Upload URL
|
v
User ─────────────────────────────→ Object StorageThe application server does not transfer the actual file bytes.
Instead:
This architecture dramatically reduces application-server bandwidth usage.
A pre-signed URL is a temporary URL that grants permission to perform a specific storage operation.
Conceptually:
POST /uploads
|
v
API creates temporary permission
|
v
Pre-signed URL
|
v
Client → Object StorageThe URL should normally have:
The client should not receive permanent cloud credentials.
Giving a browser unrestricted storage credentials would be dangerous.
A compromised client could potentially:
Pre-signed URLs provide limited authorization.
For reliable large uploads, create an upload session.
Example:
POST /uploadsRequest:
{
"filename": "vacation.mp4",
"size": 524288000,
"contentType": "video/mp4"
}Response:
{
"uploadId": "upl_123",
"objectKey": "users/42/uploads/upl_123/original.mp4",
"status": "created"
}The upload session can have states:
CREATED
|
UPLOADING
|
UPLOADED
|
PROCESSING
|
READYFailure states might include:
FAILED
CANCELLED
EXPIREDThis state machine becomes important for reliable systems.
Without a session, the system may not know:
An upload record can contain:
upload_id
user_id
filename
object_key
content_type
expected_size
received_size
status
created_at
expires_at
checksumThe database stores metadata.
Object storage stores the actual bytes.
Uploading a 5 KB image is very different from uploading a 20 GB video.
For large files, sending one HTTP request is fragile.
Suppose:
10 GB fileand the network fails after:
9.5 GBIf the system requires restarting the entire upload, the user wastes:
9.5 GBof transfer.
Instead, divide the file into parts.
Example:
10 GB file
+--------+--------+--------+--------+
| Part 1 | Part 2 | Part 3 | Part 4 |
+--------+--------+--------+--------+Each part can be uploaded independently.
For example:
Part size = 100 MB
10 GB / 100 MB ≈ 100 partsThe client can upload several parts concurrently.
┌─ Part 1 ─→ Storage
├─ Part 2 ─→ Storage
File ─────────┼─ Part 3 ─→ Storage
├─ Part 4 ─→ Storage
└─ Part 5 ─→ StorageAfter all parts are uploaded, the client or server asks storage to complete the multipart upload.
Typical lifecycle:
1. Initiate multipart upload
|
v
2. Receive upload ID
|
v
3. Upload parts
|
v
4. Track part numbers + checksums
|
v
5. Retry failed parts
|
v
6. Complete upload
|
v
7. Object becomes availableThe storage provider generally handles the final assembly.
The application should track the upload session and completion state.
Resumable uploads allow the client to continue after interruption.
Example:
Part 1 ✓
Part 2 ✓
Part 3 ✓
Part 4 ✗
Part 5 ✓Only Part 4 needs to be retried.
This is especially useful for:
Users expect progress information.
A simple percentage is:
progress =
uploadedBytes / totalBytes * 100For example:
uploaded = 400 MB
total = 1000 MB
progress = 40%The browser can track client-side upload progress.
For multipart uploads, progress can be calculated from completed parts plus the current part.
Example:
Completed parts: 600 MB
Current part: 50 MB of 100 MB
Total uploaded = 650 MBFor more advanced systems, maintain progress in the database or a fast store.
Example:
upload_id
uploaded_bytes
total_bytes
status
updated_atHowever, do not write to the database on every tiny progress event.
That can create excessive write load.
Instead:
Never trust the filename or client-provided MIME type.
A client can claim:
Content-Type: image/pngwhile sending something completely different.
Validation should consider:
Extension
+
Declared MIME type
+
File signature / magic bytes
+
Actual size
+
Application-specific rulesExample:
avatar.pngshould actually contain a valid image format.
Before accepting an upload, define limits.
Example:
Profile image: 10 MB
Document: 50 MB
Video: 5 GBEnforce limits as early as possible.
Why?
Because unlimited uploads create:
Do not use the original filename directly as the storage path.
Bad:
/uploads/user-input-filenamePotential problems include:
Instead generate an internal object key.
Example:
users/42/uploads/8b9f2c/originalStore the original filename separately as metadata.
A file can be represented by a cryptographic hash.
Example:
SHA-256(file)Conceptually:
same bytes
|
v
same hashThis enables duplicate detection.
Example:
File A → SHA-256 → abc123
File B → SHA-256 → abc123The system can recognize that both files have identical content.
Suppose 10,000 users upload the exact same 100 MB file.
Without deduplication:
10,000 × 100 MB
= 1,000,000 MB
≈ 1 TBWith content-addressed deduplication, the underlying object may be stored once.
Metadata can represent ownership separately:
User A → object abc123
User B → object abc123
User C → object abc123Important consideration:
Do not delete the shared object until all references are gone.
User-uploaded files are untrusted input.
A production system may run malware scanning before making a file available.
Architecture:
Client
|
v
Object Storage
|
v
Queue
|
v
Malware Scanner
|
+---- clean ----→ Processing
|
+---- infected → QuarantineA newly uploaded file should not automatically become downloadable to other users.
A useful design is to separate uploaded files into states or locations:
quarantine/
processed/
public/
private/Example:
Upload
|
v
Quarantine
|
v
Security scan
|
+---- infected → isolated
|
+---- clean → processingThis reduces the chance that unsafe content is immediately served.
A single uploaded image may need multiple variants.
Example:
Original
|
+── thumbnail
+── small
+── medium
+── large
+── Web-optimizedArchitecture:
Object Storage
|
v
Message Queue
|
v
Image Worker
|
+---- thumbnail
+---- small
+---- medium
+---- optimizedThe original should normally be preserved if business requirements require it.
Suppose a user uploads:
photo.jpgThe system can create:
photo-original.jpg
photo-200.jpg
photo-800.jpg
photo-1600.jpgThe application can choose the appropriate version based on:
This reduces bandwidth.
A realistic pipeline might be:
Upload
|
v
Validate
|
v
Malware Scan
|
v
Decode Image
|
v
Resize
|
v
Compress
|
v
Generate Variants
|
v
Store Results
|
v
Mark READYEach stage should ideally be independently observable and retryable.
Video is significantly more expensive.
A user may upload:
1080p.mp4but the application may need:
240p
360p
480p
720p
1080pIt may also need different codecs or streaming formats.
Architecture:
Upload
|
v
Object Storage
|
v
Queue
|
v
Video Processor
|
+---- Transcode
+---- Extract metadata
+---- Generate thumbnails
+---- Generate preview
+---- Generate streaming segments
|
v
Processed MediaSuppose transcoding takes:
3 minutesDo not keep the original upload HTTP request open for three minutes.
Instead:
POST /videos
|
v
Create processing job
|
v
Return 202 Accepted
|
v
Worker processes asynchronouslyThe client can later query:
GET /videos/{id}/statusExample:
{
"status": "processing",
"progress": 63
}A queue decouples upload from processing.
┌──────────────┐
│ Object Store │
└──────┬───────┘
|
v
┌──────────────┐
│ Message Queue│
└──────┬───────┘
|
┌───────────┼───────────┐
v v v
Worker 1 Worker 2 Worker 3Benefits:
Suppose:
Uploads = 10,000/hour
Processing capacity = 5,000/hourThe queue grows.
That is not necessarily a failure.
The queue provides a buffer.
But the system must monitor:
queue depth
oldest message age
processing latency
failure rate
worker utilizationIf the queue grows continuously, add workers or increase processing capacity.
Media processing can fail because of:
Transient errors can be retried.
Permanent errors should eventually fail.
A typical strategy:
Attempt 1 → fail
|
v
wait
|
Attempt 2 → fail
|
v
wait longer
|
Attempt 3 → fail
|
v
Dead-letter queueUse exponential backoff.
Example:
1s
2s
4s
8s
16sAdd jitter to avoid synchronized retries.
Workers may receive the same job more than once.
Therefore processing should be idempotent where possible.
Suppose:
job = generate-thumbnail(upload_123)If executed twice, the system should not produce inconsistent results or corrupt state.
A worker can use a deterministic output key:
uploads/123/thumbnail-400.webpBefore writing:
Does output already exist?or maintain a processing record:
job_id
upload_id
operation
status
attempt_countDistributed systems commonly provide at-least-once delivery.
That means:
same message
may be processed more than onceTherefore:
at-least-once delivery
+
idempotent consumeris a common design.
Do not assume:
message received once = operation executed onceA media object can have:
UPLOADING
UPLOADED
SCANNING
PROCESSING
READY
FAILED
DELETEDFor video:
UPLOADED
|
SCANNING
|
TRANSCODING
|
GENERATING_THUMBNAILS
|
PUBLISHING
|
READYThe state should have valid transitions.
For example:
READY → PROCESSINGmight not be allowed unless a reprocessing operation is explicitly requested.
Do not put all media metadata inside the object store.
A database can store:
media_id
owner_id
object_key
original_filename
content_type
size
checksum
status
visibility
created_at
updated_at
duration
width
heightObject storage stores:
actual bytesDatabase:
"What is this file?"Object storage:
"Here are the bytes."A system usually has different visibility models.
Anyone with the URL may access the object.
Useful for:
Only authorized users may access it.
Useful for:
A private file should not simply expose a permanent public URL.
Instead:
Client
|
| GET /files/123
v
Application
|
| authorize
v
Generate signed download URL
|
v
Client
|
v
Object Storage / CDNThe URL should expire.
Example:
valid for 5 minutesAuthorization should happen before generating access.
Example:
User A requests File X
|
v
Is User A allowed?
|
+--+--+
| |
YES NO
| |
v v
Signed 403
URLDo not rely solely on an unguessable object key for authorization.
Security should be enforced by an explicit access-control policy.
A signed URL can contain authorization information that the storage/CDN validates.
Typical properties:
resource
expiration
signatureThis allows:
temporary accesswithout exposing permanent credentials.
Large media should generally be delivered through a CDN when global or high-volume delivery is required.
Architecture:
User
|
v
CDN
|
+---- cache hit → response
|
+---- cache miss
|
v
Object StorageBenefits:
Large files often need partial downloads.
HTTP supports byte ranges.
Conceptually:
Range: bytes=0-999999The server can return only that portion.
This is useful for:
Example:
10 GB video
User jumps to minute 45
Client does not necessarily need
to download the first 45 minutes.A media delivery system can optimize downloads through:
For large files, avoid routing every byte through the application server.
For video platforms, a single large MP4 may not be sufficient.
The system can create multiple representations:
240p
360p
480p
720p
1080pThe client can select an appropriate representation based on network conditions.
Common streaming approaches include segmented adaptive streaming such as HLS or MPEG-DASH.
Conceptually:
Master Manifest
|
+---- 240p
+---- 360p
+---- 720p
+---- 1080pEach representation is divided into segments.
A production architecture may look like:
┌───────────────┐
│ Application API│
└───────┬───────┘
|
Upload Session
|
v
Client ───────────────────→ Object Storage
|
v
Queue
|
┌────────────────┼────────────────┐
v v v
Virus Scanner Image Worker Video Worker
| | |
└────────────────┼────────────────┘
v
Processed Storage
|
v
CDN
|
v
UsersThe API manages metadata and authorization.
Object storage manages bytes.
Workers manage expensive processing.
Queues provide asynchronous coordination.
CDN handles delivery.
Abandoned uploads are common.
Example:
User starts 5 GB upload
↓
User closes browser
↓
Multipart upload remains incompleteIf never cleaned up, storage resources accumulate.
Use lifecycle cleanup.
Example:
Incomplete upload older than 24 hours
|
v
AbortAlso clean:
Suppose:
Database row deleted
Object remainsor:
Object deleted
Database row remainsNow the system is inconsistent.
Possible solutions include:
A reconciliation job can find:
objects without metadata
metadata without objectsImportant controls include:
Know who is uploading.
Verify what the user is allowed to upload.
Prevent abuse.
Do not trust only the filename.
Scan untrusted files.
Do not directly use user-controlled paths.
Prevent upload abuse.
Limit per-user or per-tenant storage.
Protect data at rest and in transit.
Do not accidentally expose sensitive objects.
Suppose a malicious user sends:
10,000 upload requests/minuteEven before files are transferred, this can overload the upload-session API.
Rate-limit:
create upload
complete upload
generate signed URL
download authorizationYou can apply limits by:
For multi-tenant systems:
Tenant A → 100 GB
Tenant B → 1 TB
Tenant C → 10 GBTrack:
used_bytes
quota_bytesBefore creating a new upload:
used + expected_size <= quotaBut remember that concurrent uploads create race conditions.
Quota enforcement may require reservation.
Suppose a user has:
1 GB remainingTwo simultaneous uploads each request:
700 MBIf both check independently:
700 MB <= 1 GB
700 MB <= 1 GBBoth may be accepted.
Actual usage:
1.4 GBTo avoid this, reserve capacity:
available = quota - used - reservedThen:
reservation + expected_sizemust fit.
Example:
Internet
|
v
API Gateway
|
v
Upload Service
|
+------------+------------+
| |
v v
Metadata DB Signed URL
|
v
Object Storage
|
v
Queue
|
+-------------------+-------------------+
| | |
v v v
Scanner Image Worker Video Worker
| | |
+-------------------+-------------------+
|
v
Processed Storage
|
v
CDNThis separates responsibilities cleanly.
A database can technically store binary data.
But for large media workloads, object storage is usually more appropriate because it is designed for:
The database should usually store metadata and relationships.
Example APIs:
POST /uploads
GET /uploads/{id}
POST /uploads/{id}/complete
POST /uploads/{id}/cancel
GET /uploads/{id}/status
DELETE /uploads/{id}For a multipart implementation:
POST /uploads/{id}/parts
POST /uploads/{id}/completeThe exact API depends on the storage provider and architecture.
A useful model:
+-----------+
| CREATED |
+-----+-----+
|
v
+-----------+
| UPLOADING |
+-----+-----+
|
complete
|
v
+-----------+
| UPLOADED |
+-----+-----+
|
v
+-----------+
| SCANNING |
+-----+-----+
|
v
+-----------+
| PROCESSING|
+-----+-----+
|
v
+-----------+
| READY |
+-----------+Failure can transition to:
FAILEDCancellation can transition to:
CANCELLEDSolution:
resume failed partSolution:
queue retrySolution:
dead-letter queueSolution:
idempotent workerSolution:
resumable uploadSolution:
reconciliationSolution:
verification/reconciliationMonitor the entire pipeline.
Important metrics:
upload success rate
upload failure rate
average upload duration
bytes uploaded
active uploadsqueue depth
oldest message age
retry count
dead-letter countprocessing duration
success rate
failure rate
worker utilization
CPU
memorytotal bytes
object count
storage growth
orphaned objectscache hit ratio
origin traffic
bandwidth
latency
error rateMedia systems can become expensive.
Costs can come from:
Storage
+
PUT/GET operations
+
Bandwidth
+
CDN
+
Transcoding
+
Scanning
+
Worker compute
+
DatabaseFor example, a video platform may spend significant resources on transcoding rather than raw storage.
Optimization strategies include:
Requirements:
Architecture:
Mobile/Web Client
|
v
Upload API
|
+---- Metadata DB
|
+---- Signed Upload URL
|
v
Object Storage
|
v
Queue
|
Malware Scanner
|
v
Image Worker
|
+---------+---------+
| | |
v v v
Thumb Medium Large
| | |
+---------+---------+
|
v
Processed Store
|
v
CDN
|
v
UsersRequirements:
Architecture:
Client
|
v
Upload API
|
v
Multipart Upload
|
v
Object Storage
|
v
Queue
|
+---- Scan
|
+---- Extract Metadata
|
+---- Transcode
|
+---- Generate Thumbnails
|
+---- Generate Segments
|
v
Processed Storage
|
v
CDN
|
v
ViewerApplication server:
simpler
but expensive at scaleDirect object storage:
more architecture
but scales betterSynchronous:
simple
poor for expensive workAsynchronous:
more components
better scalability and reliabilityNo deduplication:
simpler
higher storage costDeduplication:
more complexity
lower storage usagePublic:
simple
not appropriate for private contentSigned:
secure temporary access
more authorization complexitySending every large file through the application server.
Using permanent cloud credentials in the browser.
Trusting the filename extension.
Processing videos synchronously.
Not supporting resumable uploads.
Not cleaning abandoned multipart uploads.
Assuming queue messages execute exactly once.
Deleting an object without checking shared references.
Exposing private objects publicly.
No quota or rate limit.
No reconciliation process.
No observability around the media pipeline.
When asked:
Design a file upload system.
Start with requirements.
Ask:
What file types?
Maximum file size?
How many uploads/day?
Public or private?
Need resumability?
Need image processing?
Need video transcoding?
Need CDN?
Need virus scanning?
Retention requirements?
Multi-region?Then define:
API
Metadata DB
Object storage
Upload mechanism
Queue
Workers
CDN
Security
ObservabilityThen discuss scale and failures.
Answer:
Pre-signed URLs allow clients to upload directly to object storage using temporary, scoped authorization. This avoids routing large file payloads through application servers and improves scalability.
Answer:
Multipart upload divides a large file into independently transferable parts. Failed parts can be retried without restarting the entire upload, and multiple parts can be uploaded concurrently.
Answer:
Media processing such as image transformation and video transcoding can be CPU- and time-intensive. Asynchronous queues and workers prevent long-running processing from blocking user-facing HTTP requests and allow processing capacity to scale independently.
Answer:
Assume at-least-once delivery and make workers idempotent. Use deterministic output keys, processing records, idempotency keys, or conditional writes so repeated execution does not create inconsistent results.
Answer:
Authenticate
↓
Authorize
↓
Generate short-lived signed URL
↓
Client downloads through storage/CDNDo not expose permanent public URLs for private content.
Build a production-style media service.
Implement:
POST /uploads
GET /uploads/:id
GET /uploads/:id/status
POST /uploads/:id/complete
DELETE /uploads/:idAdd:
For image files:
original
thumbnail
medium
largeFor private files:
signed download URLMedia
----------------
id
owner_id
object_key
original_name
content_type
size
checksum
status
visibility
created_at
updated_atProcessing job:
MediaJob
----------------
id
media_id
operation
status
attempt_count
last_error
created_at
updated_atBefore considering the system production-ready, verify:
[ ] Direct upload supported
[ ] Pre-signed URLs
[ ] Large-file support
[ ] Multipart uploads
[ ] Resumable uploads
[ ] Upload expiration
[ ] File validation
[ ] Malware scanning
[ ] Safe object keys
[ ] Content hashing
[ ] Optional deduplication
[ ] Metadata DB
[ ] Queue
[ ] Async workers
[ ] Retries
[ ] Idempotency
[ ] Dead-letter handling
[ ] Private/public access control
[ ] Signed downloads
[ ] CDN
[ ] Range requests
[ ] Cleanup
[ ] Reconciliation
[ ] Quotas
[ ] Rate limiting
[ ] Encryption
[ ] Monitoring
[ ] Alerting
[ ] Cost controlsRemember this architecture:
CONTROL PLANE
|
v
Application
|
metadata / auth / policy
|
v
DATA PLANE
|
v
Object Storage
|
v
Queue
|
v
Media Workers
|
v
Processed Storage
|
v
CDN
|
v
UsersThe application controls the system.
Object storage holds the bytes.
Queues coordinate asynchronous work.
Workers perform expensive transformations.
CDNs deliver content efficiently.
In this chapter we learned how a production-grade file and media system works.
The key concepts are:
The most important architectural principle is:
Do not make the application server carry
large media traffic or perform long-running
media processing synchronously.
Separate:
API
Storage
Processing
DeliveryThat separation allows each part of the system to scale independently.
Before moving forward, you should be able to explain:
In the next chapter we will move from media pipelines to the broader concept of message queues.
We will learn:
Mark it as complete to track your mastery.