Nextgen Blog
HomeLearnAIQuantumRoboticsSecuritySpaceAbout
Nextgen Blog

By Anubhav โ€” full stack engineer, IIT Mandi CSE. Deep dives into AI, software engineering, and the technologies shaping the next generation of tech.

Navigate

  • Home
  • Learn
  • Blog
  • Archive
  • About

Topics

  • Artificial Intelligence
  • Quantum Computing
  • Robotics
  • Cybersecurity
  • Space Tech

Connect

Built with โค๏ธ & By - Anubhav

ยฉ 2026 Nextgen Blog by Anubhav. All rights reserved.

Privacy PolicyTerms of Service
Phase 04 ยท Operations & SecurityChapter 24 of 33
23.
Phase 04Read time ~22 minDifficulty intermediate

Kubernetes Troubleshooting ๐Ÿ”ง

Warrior, this is one of the most important chapters in the entire Kubernetes course.

In this chapter
  • 01The Kubernetes Troubleshooting Mindset
  • 02The Golden Troubleshooting Workflow
  • 03First Command: kubectl get
  • 04Check Everything
  • 05Check Pod Details
  • 06Kubernetes Events
  • 07Problem #1 โ€” CrashLoopBackOff ๐Ÿ’ฅ
  • 08Debug CrashLoopBackOff
  • 09Example: Node.js Crash
  • 10Another CrashLoopBackOff Example
  • 11Problem #2 โ€” OOMKilled ๐Ÿ’พ
  • 12Debug OOMKilled
  • 13Node.js + OOMKilled
  • 14Problem #3 โ€” ImagePullBackOff
  • 15Common Image Problems
  • 16Wrong image name
  • 17Wrong tag
  • 18Private registry
  • 19Registry authentication
  • 20Debug ImagePullBackOff
  • 21ErrImagePull vs ImagePullBackOff
  • 22Problem #4 โ€” Pending Pod
  • 23Common Causes of Pending
  • 24Insufficient CPU
  • 25Insufficient memory
  • 26Node affinity
  • 27Taint
  • 28PVC issue
  • 29Topology constraints
  • 30Pending Pod Debugging
  • 31Pending + Cluster Autoscaler
  • 32Problem #5 โ€” ContainerCreating
  • 33Problem #6 โ€” CreateContainerConfigError
  • 34ConfigMap Problems
  • 35Problem #7 โ€” Readiness Probe Failed
  • 36Why Readiness Matters
  • 37Common Readiness Probe Problems
  • 38Wrong port
  • 39Wrong path
  • 40Application binds only to localhost
  • 41Application starts slowly
  • 42Problem #8 โ€” Service Not Working
  • 43The Classic Service Problem
  • 44Check EndpointSlices
  • 45Service Port vs TargetPort
  • 46Problem #9 โ€” DNS Failure
  • 47Fully Qualified Service Name
  • 48Problem #10 โ€” Ingress Not Working
  • 49Ingress Debugging
  • 50vs 502
  • 51404
  • 52502/503
  • 53Check Ingress Controller
  • 54Problem #11 โ€” PVC Pending
  • 55Check StorageClass
  • 56Problem #12 โ€” Node NotReady
  • 57Node Conditions
  • 58Node Resource Investigation
  • 59Problem #13 โ€” Deployment Not Updating
  • 60Check Deployment Image
  • 61Image Tag Problem
  • 62Problem #14 โ€” Rollout Stuck
  • 63Rollout History
  • 64Debugging Network Connectivity
  • 65DNS + Connectivity Test
  • 66NetworkPolicy Troubleshooting
  • 67Debugging DNS Itself
  • 68Problem #15 โ€” Secret Is Wrong
  • 69Debugging Environment Variables
  • 70kubectl exec
  • 71Important: Minimal Images
  • 72Ephemeral Debugging Containers
  • 73A Complete Troubleshooting Decision Tree
  • 74Troubleshooting by Status
  • 75A Real Production Example
  • 76Step 1
  • 77Step 2
  • 78Step 3
  • 79Step 4
  • 80Step 5
  • 81Step 6
  • 82Step 7
  • 83The Most Important Rule
  • 84Verification After Fix
  • 85Interview Questions ๐ŸŽฏ
  • 86Q1. What do you do when a Pod is in CrashLoopBackOff?
  • 87Q2. How do you troubleshoot Pending Pods?
  • 88Q3. How do you troubleshoot ImagePullBackOff?
  • 89Q4. What does OOMKilled mean?
  • 90Q5. How do you troubleshoot a Service?
  • 91Q6. Why can a Pod be Running but not receive traffic?
  • 92Q7. How do you troubleshoot Ingress?
  • 93Q8. How do you troubleshoot PVC Pending?
  • 94Q9. How do you troubleshoot a Node NotReady?
  • 95Q10. What are the first three Kubernetes debugging commands?
  • 96Your Kubernetes Troubleshooting Cheat Sheet ๐Ÿง 
  • 97Pod problem
  • 98Events
  • 99Deployment
  • 100Service
  • 101Resources
  • 102Nodes
  • 103Storage
  • 104Network
  • 105Next โ†’ Chapter 24: Rolling Updates & Rollbacks ๐Ÿš€
In this chapter (105 sections)
01The Kubernetes Troubleshooting Mindset02The Golden Troubleshooting Workflow03First Command: kubectl get04Check Everything05Check Pod Details06Kubernetes Events07Problem #1 โ€” CrashLoopBackOff ๐Ÿ’ฅ08Debug CrashLoopBackOff09Example: Node.js Crash10Another CrashLoopBackOff Example11Problem #2 โ€” OOMKilled ๐Ÿ’พ12Debug OOMKilled13Node.js + OOMKilled14Problem #3 โ€” ImagePullBackOff15Common Image Problems16Wrong image name17Wrong tag18Private registry19Registry authentication20Debug ImagePullBackOff21ErrImagePull vs ImagePullBackOff22Problem #4 โ€” Pending Pod23Common Causes of Pending24Insufficient CPU25Insufficient memory26Node affinity27Taint28PVC issue29Topology constraints30Pending Pod Debugging31Pending + Cluster Autoscaler32Problem #5 โ€” ContainerCreating33Problem #6 โ€” CreateContainerConfigError34ConfigMap Problems35Problem #7 โ€” Readiness Probe Failed36Why Readiness Matters37Common Readiness Probe Problems38Wrong port39Wrong path40Application binds only to localhost41Application starts slowly42Problem #8 โ€” Service Not Working43The Classic Service Problem44Check EndpointSlices45Service Port vs TargetPort46Problem #9 โ€” DNS Failure47Fully Qualified Service Name48Problem #10 โ€” Ingress Not Working49Ingress Debugging50vs 5025140452502/50353Check Ingress Controller54Problem #11 โ€” PVC Pending55Check StorageClass56Problem #12 โ€” Node NotReady57Node Conditions58Node Resource Investigation59Problem #13 โ€” Deployment Not Updating60Check Deployment Image61Image Tag Problem62Problem #14 โ€” Rollout Stuck63Rollout History64Debugging Network Connectivity65DNS + Connectivity Test66NetworkPolicy Troubleshooting67Debugging DNS Itself68Problem #15 โ€” Secret Is Wrong69Debugging Environment Variables70kubectl exec71Important: Minimal Images72Ephemeral Debugging Containers73A Complete Troubleshooting Decision Tree74Troubleshooting by Status75A Real Production Example76Step 177Step 278Step 379Step 480Step 581Step 682Step 783The Most Important Rule84Verification After Fix85Interview Questions ๐ŸŽฏ86Q1. What do you do when a Pod is in CrashLoopBackOff?87Q2. How do you troubleshoot Pending Pods?88Q3. How do you troubleshoot ImagePullBackOff?89Q4. What does OOMKilled mean?90Q5. How do you troubleshoot a Service?91Q6. Why can a Pod be Running but not receive traffic?92Q7. How do you troubleshoot Ingress?93Q8. How do you troubleshoot PVC Pending?94Q9. How do you troubleshoot a Node NotReady?95Q10. What are the first three Kubernetes debugging commands?96Your Kubernetes Troubleshooting Cheat Sheet ๐Ÿง 97Pod problem98Events99Deployment100Service101Resources102Nodes103Storage104Network105Next โ†’ Chapter 24: Rolling Updates & Rollbacks ๐Ÿš€

In real projects, knowing Kubernetes isn't just about creating Deployments and Services.

The real skill is:

When something breaks, can you quickly identify where the problem is and fix it?

We'll build a systematic troubleshooting method instead of randomly running commands.


1. The Kubernetes Troubleshooting Mindset#

When something doesn't work, don't immediately change YAML.

First ask:

What exactly is broken?
        โ†“
Which layer is broken?
        โ†“
Can I reproduce it?
        โ†“
What evidence do I have?
        โ†“
What is the smallest fix?

Think in layers:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚        Application            โ”‚
โ”‚   Node.js / Next.js / etc.    โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚       Container / Pod         โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚       Deployment / Job        โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚          Service              โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚        Ingress / DNS          โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚       Node / Cluster          โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚      Cloud / Infrastructure   โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Find the broken layer first.


2. The Golden Troubleshooting Workflow#

Memorize this:

1. kubectl get
       โ†“
2. kubectl describe
       โ†“
3. kubectl logs
       โ†“
4. kubectl logs --previous
       โ†“
5. Check Events
       โ†“
6. Check Service
       โ†“
7. Check Endpoints
       โ†“
8. Check DNS / Network
       โ†“
9. Check Node / Resources

The three commands you should instinctively remember are:

kubectl get
kubectl describe
kubectl logs

3. First Command: kubectl get#

Suppose your application isn't working.

Start with:

kubectl get pods

Example:

NAME                         READY   STATUS             RESTARTS
node-api-7d9f8c6b-x2k4m      1/1     Running            0
node-api-7d9f8c6b-q8m2p      0/1     CrashLoopBackOff   7

Immediately you know:

One Pod is healthy and another is repeatedly crashing.

Don't modify anything yet.

Investigate.


4. Check Everything#

Useful command:

kubectl get all

You'll see resources such as:

NAME                            READY
pod/node-api-xxx                1/1
 
NAME                    TYPE        CLUSTER-IP
service/node-api        ClusterIP   10.96.20.10
 
NAME                         READY
deployment.apps/node-api     2/2

For production, namespace matters:

kubectl get all -n production

5. Check Pod Details#

Run:

kubectl describe pod <pod-name>

Example:

kubectl describe pod node-api-xxx

The most important section is usually:

Events:

Example:

Warning  FailedScheduling
0/3 nodes are available:
2 Insufficient memory
1 node has untolerated taint

Now you have evidence.


6. Kubernetes Events#

Events are extremely useful.

Run:

kubectl get events

For a namespace:

kubectl get events -n production

Sort by time:

kubectl get events \
  --sort-by=.lastTimestamp

Events can reveal:

FailedScheduling
FailedMount
Failed
BackOff
Unhealthy
Pulling
Pulled
Created
Started
Killing

For beginners:

When something unexpected happens in Kubernetes, check Events.


7. Problem #1 โ€” CrashLoopBackOff ๐Ÿ’ฅ#

One of the most common Kubernetes errors.

You see:

NAME       READY   STATUS
node-api   0/1     CrashLoopBackOff

What does it mean?

It means roughly:

The container starts, crashes, Kubernetes restarts it, it crashes again, and Kubernetes increasingly backs off between restart attempts.

It does not itself tell you the root cause.


8. Debug CrashLoopBackOff#

First:

kubectl logs <pod>

Then:

kubectl logs <pod> --previous

Then:

kubectl describe pod <pod>

Look for:

Last State:
  Terminated
  Reason:
  Exit Code:

9. Example: Node.js Crash#

Logs:

Error: Cannot connect to database
    at connectDatabase (/app/db.js:42:11)

Now you know:

Kubernetes isn't necessarily the root problem.
 
Your application is crashing because DB connection failed.

Potential causes:

Wrong DATABASE_URL
Wrong Secret
Database unavailable
DNS failure
NetworkPolicy
Credentials invalid

10. Another CrashLoopBackOff Example#

Suppose:

Error: listen EADDRINUSE: address already in use

This is an application/container problem.

Don't immediately blame Kubernetes.

Check:

Application
โ†“
Port configuration
โ†“
Container configuration

11. Problem #2 โ€” OOMKilled ๐Ÿ’พ#

You see:

Last State:
  Terminated
  Reason: OOMKilled

OOM means:

Out Of Memory.

Suppose your Deployment has:

resources:
  limits:
    memory: "512Mi"

But the application needs more memory.

The container can be terminated when it exceeds its memory limit.


12. Debug OOMKilled#

Run:

kubectl describe pod <pod>

Look for:

Reason: OOMKilled

Then:

kubectl top pod <pod>

if metrics are available.

Check:

resources:
  requests:
    memory: "256Mi"
 
  limits:
    memory: "512Mi"

Questions to ask:

Is the memory limit too low?
Is the application leaking memory?
Did traffic increase?
Is Node.js heap growing?
Are large files/models being loaded?

13. Node.js + OOMKilled#

This is especially relevant for Node.js applications.

Suppose:

Node.js API
โ†“
512Mi memory limit
โ†“
Large JSON processing
โ†“
Memory usage increases
โ†“
Container exceeds limit
โ†“
OOMKilled

You may need to investigate:

  • memory leak
  • large request payloads
  • large database result sets
  • caching
  • inefficient data processing
  • Node.js heap configuration
  • concurrency
  • application architecture

Don't blindly increase memory.

First understand why memory increased.


14. Problem #3 โ€” ImagePullBackOff#

Suppose:

kubectl get pods

shows:

node-api   0/1   ImagePullBackOff

This usually means Kubernetes cannot successfully pull the container image.

Check:

kubectl describe pod <pod>

Look at Events.

You might see:

Failed to pull image

15. Common Image Problems#

Wrong image name#

image: my-node-appp:1.0

instead of:

image: my-node-app:1.0

Wrong tag#

image: my-node-api:999

when tag doesn't exist.

Private registry#

The cluster doesn't have permission to pull it.

Registry authentication#

You may need:

imagePullSecrets

or cloud workload/node identity depending on your environment.


16. Debug ImagePullBackOff#

Run:

kubectl describe pod <pod>

Find:

Events:

Example:

Failed to pull image "myrepo/node-api:5.0"
manifest unknown

Now the issue is clear:

Image/tag doesn't exist.


17. ErrImagePull vs ImagePullBackOff#

You may see:

ErrImagePull

This means image pulling failed.

Then Kubernetes retries.

Eventually you may see:

ImagePullBackOff

Think:

Image pull fails
      โ†“
ErrImagePull
      โ†“
Retry
      โ†“
Backoff
      โ†“
ImagePullBackOff

18. Problem #4 โ€” Pending Pod#

Example:

NAME       READY   STATUS
node-api   0/1     Pending

A Pending Pod hasn't been successfully scheduled and started.

Run:

kubectl describe pod <pod>

Look at Events.

You might see:

0/3 nodes are available:
3 Insufficient cpu

19. Common Causes of Pending#

Insufficient CPU#

Insufficient cpu

Insufficient memory#

Insufficient memory

Node affinity#

Pod requires:

workload=gpu

but no node has that label.

Taint#

Node has:

workload=gpu:NoSchedule

and Pod doesn't have the appropriate toleration.

PVC issue#

Pod requires storage that isn't available.

Topology constraints#

Pod cannot be placed according to topology rules.


20. Pending Pod Debugging#

Use:

kubectl describe pod <pod>

Then ask:

Is there enough CPU?
Is there enough memory?
Are nodeSelector/affinity rules valid?
Are there taints?
Does the Pod tolerate them?
Is PVC available?
Are topology constraints too strict?

21. Pending + Cluster Autoscaler#

Remember Chapter 16.

Suppose:

HPA
 โ†“
Pods increase
 โ†“
No node has enough capacity
 โ†“
Pods Pending
 โ†“
Cluster Autoscaler
 โ†“
New node
 โ†“
Scheduler
 โ†“
Pod Running

If your cloud cluster has a Cluster Autoscaler configured, a Pending Pod caused by insufficient capacity may trigger node scaling.

But:

Not every Pending Pod is a Cluster Autoscaler problem.

If the reason is:

untolerated taint

adding another generic node may not solve it.


22. Problem #5 โ€” ContainerCreating#

You may see:

STATUS
ContainerCreating

for a long time.

This means Kubernetes is still trying to prepare/start the container.

Potential causes:

Image pulling
Volume mounting
Secret/ConfigMap issues
CNI/network setup
Container runtime issues

Run:

kubectl describe pod <pod>

Again:

Events are your friend.


23. Problem #6 โ€” CreateContainerConfigError#

Example:

CreateContainerConfigError

Often indicates a configuration dependency problem.

For example, your Deployment says:

envFrom:
  - secretRef:
      name: database-secret

But:

database-secret

doesn't exist.

Check:

kubectl get secrets

and:

kubectl describe pod <pod>

You may find:

secret "database-secret" not found

24. ConfigMap Problems#

Suppose:

envFrom:
  - configMapRef:
      name: node-api-config

but ConfigMap doesn't exist.

Check:

kubectl get configmaps

or:

kubectl get cm

Then:

kubectl describe pod <pod>

25. Problem #7 โ€” Readiness Probe Failed#

Suppose:

NAME       READY   STATUS
node-api   0/1     Running

This is an important situation.

The container is running.

But:

READY = 0

means the Pod isn't currently ready to receive normal Service traffic.

Check:

kubectl describe pod <pod>

You may see:

Readiness probe failed:
HTTP probe failed with statuscode: 503

26. Why Readiness Matters#

Remember Chapter 14.

Container running
       โ†“
Readiness probe
       โ†“
Success?
   โ”Œโ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”
  Yes      No
   โ†“        โ†“
Traffic   No normal
allowed   Service traffic

So:

Running โ‰  Ready

This is one of the most important Kubernetes concepts.


27. Common Readiness Probe Problems#

Wrong port#

Application:

3000

Probe:

port: 8080

Wrong path#

Application:

/health

Probe:

path: /healthz

Application binds only to localhost#

For container networking, your server generally needs to listen on:

0.0.0.0

rather than only:

127.0.0.1

Application starts slowly#

Use an appropriate:

startupProbe

28. Problem #8 โ€” Service Not Working#

Suppose:

Frontend
   โ†“
backend-service
   โ†“
No response

First:

kubectl get svc

Then:

kubectl describe svc backend-service

Check:

Selector
Port
TargetPort
Endpoints

29. The Classic Service Problem#

Service:

selector:
  app: backend

Pod:

labels:
  app: node-api

They don't match.

Therefore:

Service
  โ†“
selector app=backend
  โ†“
No matching Pods
  โ†“
No endpoints

Check:

kubectl get pods --show-labels

and:

kubectl describe svc backend-service

30. Check EndpointSlices#

Modern Kubernetes uses EndpointSlices.

Run:

kubectl get endpointslices

Or:

kubectl get endpointslices \
  -l kubernetes.io/service-name=backend-service

If your Service has no ready backend endpoints, investigate:

Selector
Pod labels
Pod readiness
Port configuration

31. Service Port vs TargetPort#

Suppose your Node.js app listens on:

3000

Service:

ports:
  - port: 80
    targetPort: 3000

Correct.

Traffic:

Client
  โ†“
Service :80
  โ†“
Pod :3000

But if you accidentally write:

targetPort: 8080

your Service may route traffic to the wrong port.

Result:

Connection refused

32. Problem #9 โ€” DNS Failure#

Suppose backend should be reachable at:

http://backend-service

but your frontend gets:

getaddrinfo ENOTFOUND backend-service

First verify:

kubectl get svc

Then test DNS from inside the cluster.

For example:

kubectl run dns-test \
  --image=busybox:1.36 \
  -it --rm \
  --restart=Never \
  -- nslookup backend-service

Depending on the image, nslookup may not be available; you can use an image with DNS tools such as busybox or dnsutils.


33. Fully Qualified Service Name#

Suppose Service:

backend-service

exists in:

production

You can use:

backend-service.production

or:

backend-service.production.svc.cluster.local

The standard pattern is:

<service>.<namespace>.svc.cluster.local

34. Problem #10 โ€” Ingress Not Working#

Suppose:

https://example.com

returns:

404

or:

502 Bad Gateway

Don't immediately blame the application.

Debug layer by layer:

DNS
 โ†“
Load Balancer
 โ†“
Ingress Controller
 โ†“
Ingress
 โ†“
Service
 โ†“
EndpointSlices
 โ†“
Pods

35. Ingress Debugging#

First:

kubectl get ingress

Then:

kubectl describe ingress <ingress-name>

Check:

Host
Path
Backend Service
Backend Port
Events

Then:

kubectl get svc

Then:

kubectl get endpointslices

Then:

kubectl get pods

36. 404 vs 502#

These can give clues, although exact behavior depends on your Ingress Controller.

404#

Potentially:

Wrong host
Wrong path
Ingress routing rule
Controller configuration

502/503#

Potentially:

Service has no healthy endpoints
Backend unreachable
Wrong targetPort
Application unavailable

Use actual controller logs and events to confirm.


37. Check Ingress Controller#

For example, if using an NGINX-based controller:

kubectl get pods -A

Find the controller Pods.

Then:

kubectl logs <ingress-controller-pod> -n <namespace>

The exact namespace and controller depend on your installation.


38. Problem #11 โ€” PVC Pending#

Suppose:

kubectl get pvc

shows:

NAME           STATUS
postgres-pvc   Pending

Your Pod may also remain Pending.

Run:

kubectl describe pvc postgres-pvc

Look for Events.

Possible causes:

No StorageClass
No suitable provisioner
Invalid access mode
Insufficient storage
Cloud storage problem
Zone/topology constraints

39. Check StorageClass#

Run:

kubectl get storageclass

Then:

kubectl describe storageclass <name>

Remember:

PVC
 โ†“
StorageClass
 โ†“
Provisioner
 โ†“
PV
 โ†“
Actual storage

40. Problem #12 โ€” Node NotReady#

Check:

kubectl get nodes

Example:

NAME     STATUS
node-1   Ready
node-2   NotReady

Investigate:

kubectl describe node node-2

Look for:

Conditions
Events
Taints
Allocatable resources
Capacity

41. Node Conditions#

You may see conditions such as:

Ready
MemoryPressure
DiskPressure
PIDPressure
NetworkUnavailable

For example:

DiskPressure=True

could indicate a disk-space issue.

This can affect Pods and scheduling.


42. Node Resource Investigation#

Use:

kubectl top nodes

if metrics are available.

Look at:

CPU
Memory

Then:

kubectl describe node <node>

to understand:

Capacity
Allocatable
Allocated resources
Conditions
Taints
Pods

43. Problem #13 โ€” Deployment Not Updating#

Suppose you changed:

image: node-api:2.0

but Pods still appear to run the old version.

Check:

kubectl rollout status deployment/node-api

Then:

kubectl get deployment node-api

and:

kubectl get rs

Then:

kubectl get pods

44. Check Deployment Image#

Run:

kubectl describe deployment node-api

Look for:

Image:

You can also inspect:

kubectl get deployment node-api \
  -o=jsonpath='{.spec.template.spec.containers[*].image}'

45. Image Tag Problem#

Suppose you use:

image: node-api:latest

You build a new image with the same tag.

Kubernetes may not necessarily pull the new image in the way you expect, depending on image pull policy and node cache.

For production deployments, immutable versioned tags are much safer:

node-api:1.0.0
node-api:1.0.1
node-api:1.0.2

Even better, deployments can use immutable image digests.


46. Problem #14 โ€” Rollout Stuck#

Check:

kubectl rollout status deployment/node-api

Maybe:

Waiting for deployment "node-api" rollout to finish...

Check:

kubectl get pods

Maybe new Pods are:

0/1 Running

Then:

kubectl describe pod <new-pod>

Maybe readiness probe is failing.

Or:

ImagePullBackOff

Or:

CrashLoopBackOff

This demonstrates an important point:

A Deployment rollout problem often has a lower-level Pod problem underneath it.


47. Rollout History#

Check:

kubectl rollout history deployment/node-api

If a new version is broken:

kubectl rollout undo deployment/node-api

Then:

kubectl rollout status deployment/node-api

We'll study production rollouts and rollback strategies in Chapter 24.


48. Debugging Network Connectivity#

Suppose:

Frontend Pod
      โ†“
Backend Service
      โ†“
Timeout

Don't immediately modify NetworkPolicy.

Test from inside the cluster.

Run a temporary debugging Pod:

kubectl run network-debug \
  --image=curlimages/curl \
  -it --rm \
  --restart=Never \
  -- sh

Then:

curl http://backend-service

Test a specific port:

curl http://backend-service:80

49. DNS + Connectivity Test#

Inside the debugging container:

nslookup backend-service

Then:

curl http://backend-service

This separates:

DNS problem

from:

Application/network connectivity problem

50. NetworkPolicy Troubleshooting#

Remember Chapter 20.

Suppose:

Frontend โ†’ Backend

used to work.

Then you add:

default-deny

and suddenly it fails.

Check:

kubectl get networkpolicy -A

Then:

kubectl describe networkpolicy <policy>

Ask:

Is frontend allowed to talk to backend?
Is backend port allowed?
Is DNS allowed?
Is egress allowed?
Are namespace selectors correct?

NetworkPolicy debugging requires understanding the actual source and destination Pods.


51. Debugging DNS Itself#

If service DNS doesn't work:

kubectl get pods -n kube-system

Look for your cluster DNS components, commonly CoreDNS.

Then inspect:

kubectl logs -n kube-system <coredns-pod>

You can also inspect:

kubectl get svc -n kube-system

Look for the cluster DNS Service.


52. Problem #15 โ€” Secret Is Wrong#

Suppose your app says:

Authentication failed

But the Pod is Running.

Potentially:

Secret value incorrect

Check that the Secret exists:

kubectl get secret

Don't casually print sensitive values into terminal history or logs.

You can inspect metadata:

kubectl describe secret database-secret

Remember:

Kubernetes Secrets are not automatically equivalent to a full external secret-management system.


53. Debugging Environment Variables#

You can inspect environment variables inside a container when appropriate:

kubectl exec -it <pod> -- env

But be careful:

This can expose secrets in your terminal output.

For production secrets, avoid indiscriminate dumping.

Instead, verify configuration through safer targeted checks.


54. kubectl exec#

Sometimes logs aren't enough.

You can enter a running container:

kubectl exec -it <pod> -- /bin/sh

Then:

ls

or:

env

or:

cat /etc/resolv.conf

or:

wget ...

depending on available tools.


55. Important: Minimal Images#

Your production image may be:

node:alpine

or an even smaller hardened image.

It might not contain:

curl
ping
nslookup
bash
netstat

So don't assume every debugging command exists inside your application container.

That's why temporary debug Pods are useful.


56. Ephemeral Debugging Containers#

Modern Kubernetes environments may support ephemeral containers for troubleshooting running Pods.

Conceptually:

Application Pod
 โ”œโ”€โ”€ Node.js container
 โ””โ”€โ”€ Temporary debug container

This lets you troubleshoot without rebuilding the application image just to add debugging tools.

Availability and permissions depend on your cluster/version/setup.


57. A Complete Troubleshooting Decision Tree#

This is worth saving.

                 Application broken
                        โ”‚
                        โ†“
                 kubectl get pods
                        โ”‚
             โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
             โ†“          โ†“          โ†“
           Pending    Running    Failed/Crash
             โ”‚          โ”‚          โ”‚
             โ†“          โ†“          โ†“
         describe    logs       logs
             โ”‚          โ”‚          โ”‚
             โ†“          โ†“       --previous
       Scheduling?    Ready?        โ”‚
       Resources?      โ”‚           โ†“
       PVC?            โ”‚       Exit code?
       Taints?         โ”‚       OOMKilled?
                       โ”‚       App error?
                       โ†“
                  Service check
                       โ”‚
                       โ†“
                EndpointSlices
                       โ”‚
                       โ†“
                  DNS / curl
                       โ”‚
                       โ†“
                 NetworkPolicy
                       โ”‚
                       โ†“
                Ingress / DNS
                       โ”‚
                       โ†“
                  Node / Cloud

58. Troubleshooting by Status#

Memorize this table:

Status First suspicion First command
CrashLoopBackOff App/container crash kubectl logs --previous
OOMKilled Memory exceeded kubectl describe pod
ImagePullBackOff Image/registry issue kubectl describe pod
ErrImagePull Image pull failed kubectl describe pod
Pending Scheduling/resource/PVC kubectl describe pod
ContainerCreating Volume/image/network setup kubectl describe pod
CreateContainerConfigError Config/Secret dependency kubectl describe pod
Running 0/1 Readiness failure kubectl describe pod
Service no response Selector/port/endpoints kubectl describe svc
PVC Pending Storage provisioning kubectl describe pvc
Node NotReady Node health/resources kubectl describe node

59. A Real Production Example#

Imagine your company deploys:

Next.js
   โ†“
Ingress
   โ†“
Node.js API
   โ†“
PostgreSQL

Users report:

"The website is loading, but checkout fails."

You investigate.

Step 1#

kubectl get pods -n production

Everything is:

Running

So don't conclude everything is healthy.


Step 2#

Check API logs:

kubectl logs -n production <api-pod>

You find:

Database connection timeout

Step 3#

Check PostgreSQL Pod:

kubectl get pods -n production

Postgres is:

Running

Step 4#

Check PostgreSQL Service:

kubectl get svc -n production

Exists.


Step 5#

Check endpoints:

kubectl get endpointslices -n production

Postgres has no ready endpoints.


Step 6#

Inspect Postgres:

kubectl describe pod postgres-xxx -n production

You discover:

Readiness probe failed

Step 7#

Inspect logs:

kubectl logs postgres-xxx -n production

You discover:

Database storage is full

Now you have a chain:

Disk full
   โ†“
PostgreSQL unhealthy
   โ†“
Readiness fails
   โ†“
No ready endpoints
   โ†“
Node.js cannot connect
   โ†“
Checkout fails

This is real troubleshooting.

You didn't randomly change the API.

You followed evidence.


60. The Most Important Rule#

When troubleshooting Kubernetes:

Don't jump directly to the solution. Follow the evidence.

Bad troubleshooting:

Pod broken
โ†“
Increase CPU
โ†“
Restart
โ†“
Delete Pod
โ†“
Hope it works

Good troubleshooting:

Symptom
 โ†“
Status
 โ†“
Events
 โ†“
Logs
 โ†“
Dependencies
 โ†“
Metrics
 โ†“
Root cause
 โ†“
Fix
 โ†“
Verify

61. Verification After Fix#

Suppose you fix the problem.

Don't stop at:

"It seems fixed."

Verify.

kubectl get pods

Then:

kubectl rollout status deployment/node-api

Then:

kubectl get svc

Then test:

curl ...

Then inspect logs:

kubectl logs <pod>

And if monitoring exists:

Error rate โ†“
Latency โ†“
Restarts stable
Pods Ready

A fix isn't complete until you verify the system recovered.


62. Interview Questions ๐ŸŽฏ#

Q1. What do you do when a Pod is in CrashLoopBackOff?#

kubectl logs <pod>
kubectl logs <pod> --previous
kubectl describe pod <pod>

Then inspect application errors, exit code, probes, resource limits, and Events.


Q2. How do you troubleshoot Pending Pods?#

kubectl describe pod <pod>

Look for:

  • insufficient CPU/memory
  • taints
  • affinity
  • nodeSelector
  • topology constraints
  • PVC issues

Q3. How do you troubleshoot ImagePullBackOff?#

kubectl describe pod <pod>

Check:

  • image name
  • tag
  • registry
  • authentication
  • imagePullSecrets
  • registry/network access

Q4. What does OOMKilled mean?#

The container was terminated because it exceeded its memory limit or otherwise encountered an out-of-memory condition.

Investigate actual usage, memory limits, application behavior, and node conditions.


Q5. How do you troubleshoot a Service?#

Check:

kubectl get svc
kubectl describe svc <service>
kubectl get endpointslices
kubectl get pods --show-labels

Verify:

selector
Pod labels
port
targetPort
readiness

Q6. Why can a Pod be Running but not receive traffic?#

Because:

Running โ‰  Ready

The readiness probe may be failing.


Q7. How do you troubleshoot Ingress?#

Check:

DNS
Ingress
Ingress Controller
Service
EndpointSlices
Pods

Commands:

kubectl get ingress
kubectl describe ingress <ingress>
kubectl get svc
kubectl get endpointslices
kubectl logs <ingress-controller>

Q8. How do you troubleshoot PVC Pending?#

kubectl describe pvc <pvc>
kubectl get storageclass

Check provisioning, StorageClass, access modes, topology and storage backend.


Q9. How do you troubleshoot a Node NotReady?#

kubectl describe node <node>

Check:

Conditions
Events
DiskPressure
MemoryPressure
PIDPressure
kubelet
networking

Q10. What are the first three Kubernetes debugging commands?#

A very good answer:

kubectl get
kubectl describe
kubectl logs

Then use Events, metrics, Services, EndpointSlices and network debugging depending on the symptom.


63. Your Kubernetes Troubleshooting Cheat Sheet ๐Ÿง #

Pod problem#

kubectl get pods
kubectl describe pod <pod>
kubectl logs <pod>
kubectl logs <pod> --previous

Events#

kubectl get events --sort-by=.lastTimestamp

Deployment#

kubectl get deployment
kubectl describe deployment <deployment>
kubectl rollout status deployment/<deployment>
kubectl rollout history deployment/<deployment>

Service#

kubectl get svc
kubectl describe svc <service>
kubectl get endpointslices

Resources#

kubectl top pods
kubectl top nodes

Nodes#

kubectl get nodes
kubectl describe node <node>

Storage#

kubectl get pvc
kubectl describe pvc <pvc>
kubectl get storageclass

Network#

kubectl run network-debug \
  --image=curlimages/curl \
  -it --rm \
  --restart=Never \
  -- sh

Then:

curl http://service-name

๐Ÿ”ฅ Final Mental Model#

When your Kubernetes application breaks:

                     SOMETHING BROKE
                           โ”‚
                           โ†“
                    kubectl get
                           โ”‚
                           โ†“
                    What's wrong?
                           โ”‚
          โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
          โ†“                โ†“                โ†“
       Pod issue       Service issue      Node issue
          โ”‚                โ”‚                โ”‚
          โ†“                โ†“                โ†“
      describe           describe         describe
          โ”‚                โ”‚                โ”‚
          โ†“                โ†“                โ†“
        logs           endpoints        resources
          โ”‚                โ”‚                โ”‚
          โ†“                โ†“                โ†“
      --previous        DNS/curl        kubelet/events
          โ”‚                โ”‚                โ”‚
          โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                           โ†“
                       Metrics
                           โ†“
                        Root Cause
                           โ†“
                          Fix
                           โ†“
                        Verify

If you become comfortable with this flow, you'll be able to troubleshoot a surprisingly large percentage of everyday Kubernetes incidents.


๐ŸŽฏ Chapter 23 Key Takeaways#

Remember these 10 rules:

  1. kubectl get โ†’ understand the current state.
  2. kubectl describe โ†’ inspect details and Events.
  3. kubectl logs โ†’ inspect application output.
  4. --previous โ†’ inspect the previous crashed container.
  5. CrashLoopBackOff โ†’ find why the application is crashing.
  6. Pending โ†’ investigate scheduling/resources/storage/constraints.
  7. ImagePullBackOff โ†’ investigate image/registry/authentication.
  8. Running โ‰  Ready โ†’ check readiness probes.
  9. Service problems โ†’ check selectors + EndpointSlices + ports.
  10. Always verify the fix instead of assuming it worked.

Next โ†’ Chapter 24: Rolling Updates & Rollbacks ๐Ÿš€#

We'll now move from troubleshooting into production deployment strategies.

You'll learn:

  • Rolling Updates
  • ReplicaSets during deployments
  • maxSurge
  • maxUnavailable
  • Zero-downtime deployments
  • Rollout history
  • Rollbacks
  • Blue-Green deployments
  • Canary deployments
  • Deployment strategies
  • Database migration problems during deployments
  • Version compatibility
  • Production deployment workflow
  • How Kubernetes safely moves from v1 โ†’ v2 without taking your application offline.

Finished this chapter?

Mark it as complete to track your mastery.

Previous ChapterChapter 22 โ€” Kubernetes Monitoring ๐Ÿ“ŠNext ChapterChapter 24 โ€” Rolling Updates & Rollbacks ๐Ÿš€
PrevNext