When an application fails in Kubernetes, seeing a Pod in CrashLoopBackOff, ImagePullBackOff, or Pending does not necessarily tell us what the actual problem is.

It tells us where to start investigating.

One of the most common mistakes during troubleshooting is trying to immediately fix the status shown by kubectl.

For example:

1CrashLoopBackOff

is not the root cause.

It means that a container is repeatedly failing and Kubernetes is applying a progressive delay before trying to start it again.

The actual cause could be related to the application, configuration, resources, health checks, or an external dependency.

In this article, we are going to build a systematic process to investigate four of the most common Pod states:

• CrashLoopBackOff

• ImagePullBackOff

• Pods stuck in Pending

• Pods stuck in Terminating

Before you start: do not restart the Pod immediately

When an incident occurs, it can be tempting to run:

1kubectl delete pod <pod>

and wait for Kubernetes to create a new one.

In some cases, this can temporarily restore the service.

But it can also remove useful evidence that could help us understand the root cause.

Before changing anything, review:

Pod State
   |
   +--- Events
   |
   +--- Logs
   |
   +--- Previous container state
   |
   +--- Resources
   |
   +--- Configuration

A good investigation starts by preserving evidence.

A basic troubleshooting flow

Before investigating a specific error, we can get a general view of the cluster.

1kubectl get pods -A -o wide

This allows us to quickly identify:

• namespace;

• status;

• restart count;

• IP address;

• node where the Pod is running;

• Pod age.

Then we can review recent Events:

1kubectl get events -A \
2  --sort-by='.lastTimestamp'

Events are particularly useful when Kubernetes is having problems with:

1Scheduling
2Image pulls
3Volumes
4Health checks
5Nodes
6Evictions

If we already know which Pod is affected:

1kubectl describe pod <pod> -n <namespace>

kubectl describe combines much of the information we need for an initial investigation:

1Containers
2State
3Last State
4Restart Count
5Conditions
6Volumes
7Node
8Events

Then we can inspect the logs:

1kubectl logs <pod> -n <namespace>

If the container has already restarted, there is another very important command:

1kubectl logs <pod> \
2  -n <namespace> \
3  --previous

This retrieves logs from the previous instance of the container.

1. CrashLoopBackOff

Suppose we find:

1NAME       READY   STATUS             RESTARTS
2api-xyz    0/1     CrashLoopBackOff   7

CrashLoopBackOff means that the container is entering a cycle similar to this:

Container starts
      |
      v
Container fails
      |
      v
Kubernetes restarts it
      |
      v
Container fails again
      |
      v
Backoff
      |
      +----------------+
                       |
                       v
                 Retry again

Kubernetes progressively increases the time between restart attempts to avoid continuously restarting a container that keeps failing.

The important question is:

Why is the process inside the container terminating?

Step 1: inspect the Pod

1kubectl describe pod api-xyz -n production

Pay particular attention to:

1State:
2Last State:
3Reason:
4Exit Code:
5Restart Count:
6Events:

We might find something like:

1Last State:     Terminated
2Reason:         Error
3Exit Code:      1

This tells us that the process exited with an error.

But we could also find:

1Reason: OOMKilled
2Exit Code: 137

In that case, the investigation changes completely and should focus on memory usage.

Step 2: inspect previous logs

Start with:

1kubectl logs api-xyz -n production

But if the container restarts quickly:

1kubectl logs api-xyz \
2  -n production \
3  --previous

This command often contains the most useful clue.

For example:

1ERROR: DATABASE_URL environment variable is missing

Now we have a much more specific hypothesis.

Step 3: inspect configuration

We can inspect the complete Pod specification:

1kubectl get pod api-xyz \
2  -n production \
3  -o yaml

And if it belongs to a Deployment:

1kubectl get deployment api \
2  -n production \
3  -o yaml

Check:

1environment variables
2Secrets
3ConfigMaps
4command
5args
6resources
7volume mounts
8probes
9image

An incorrect configuration may allow the container to be created successfully but cause the application to fail immediately after startup.

Step 4: inspect health checks

We may also find Events such as:

1Liveness probe failed

or:

1Startup probe failed

In this case, review:

1livenessProbe:
2startupProbe:
3readinessProbe:

An application that needs 40 seconds to start can end up in a restart loop if a health check starts too early.

Common causes of CrashLoopBackOff

CrashLoopBackOff
       |
       +--- Application crash
       |
       +--- Missing environment variable
       |
       +--- Invalid Secret / ConfigMap
       |
       +--- Dependency unavailable
       |
       +--- OOMKilled
       |
       +--- Liveness probe
       |
       +--- Startup probe
       |
       +--- Invalid command / arguments

This is why CrashLoopBackOff should be treated as a symptom, not the root cause.

2. ImagePullBackOff

Now suppose we find:

1NAME       READY   STATUS             RESTARTS
2api-abc    0/1     ImagePullBackOff   0

In this case, the container has not started running our application yet.

The failure happens earlier:

Pod
 |
 v
kubelet
 |
 v
Container runtime
 |
 v
Registry
 |
 +--- Image
 +--- Authentication
 +--- DNS
 +--- Network
 +--- TLS

Again, start with:

1kubectl describe pod api-abc \
2  -n production

The Events may show errors such as:

1manifest unknown

1pull access denied

1unauthorized

1x509: certificate signed by unknown authority

or:

1no such host

Each error points us in a different direction.

Confirm the image

1kubectl get pod api-abc \
2  -n production \
3  -o jsonpath='{.spec.containers[*].image}'

For example:

1registry.example.com/platform/api:2.4.1

Confirm that the following are correct:

• registry;

• repository;

• image name;

• tag.

A simple typo can be enough to cause ImagePullBackOff.

Private registries

If we use a private registry, inspect available Secrets:

1kubectl get secrets -n production

And check which imagePullSecrets the Pod is using:

1kubectl get pod api-abc \
2  -n production \
3  -o jsonpath='{.spec.imagePullSecrets}'

There is also an important point to remember:

Just because your laptop can run docker pull does not mean that the Kubernetes node can do the same.

The kubelet and container runtime on the node need access to the registry.

That means the issue could also involve:

1DNS
2Routing
3Firewall
4Proxy
5TLS
6Registry availability

3. Pods stuck in Pending

Another common scenario:

1NAME       READY   STATUS    RESTARTS
2api-123    0/1     Pending   0

With Pending, the first question is:

Was the scheduler able to assign the Pod to a node?

The fastest way to find out is:

1kubectl describe pod api-123 \
2  -n production

We might find:

1Warning  FailedScheduling

followed by something like:

10/3 nodes are available:
22 Insufficient cpu,
31 node(s) had untolerated taint

This information is much more useful than only seeing STATUS=Pending.

Insufficient resources

Check the Pod resource requests:

1kubectl get pod api-123 \
2  -n production \
3  -o yaml

For example:

1resources:
2  requests:
3    cpu: "2"
4    memory: "4Gi"

Then inspect the nodes:

1kubectl describe nodes

and check current metrics:

1kubectl top nodes

But there is an important difference to understand.

The scheduler primarily makes placement decisions using resource requests, not simply the real-time consumption shown by kubectl top.

We could have:

1Actual Node CPU:      30%
2Requested CPU:        95%

and new Pods may still be unable to schedule because there is not enough unallocated requested capacity.

Other FailedScheduling causes

CPU and memory are not the only possible causes.

We should also investigate:

1nodeSelector
2nodeAffinity
3podAffinity
4podAntiAffinity
5taints
6tolerations
7topologySpreadConstraints
8PVC
9ResourceQuota

For example:

1kubectl get nodes --show-labels

and:

1kubectl describe node <node>

can help identify problems related to labels and taints.

4. Pods stuck in Terminating

Now suppose we find:

1NAME       READY   STATUS        AGE
2api-456    1/1     Terminating   2d

Pods do not disappear instantly when we delete them.

Kubernetes allows them to perform a graceful shutdown before removing them.

Start with:

1kubectl describe pod api-456 \
2  -n production

and:

1kubectl get pod api-456 \
2  -n production \
3  -o yaml

Look specifically for:

1deletionTimestamp:
2finalizers:
3terminationGracePeriodSeconds:

Common causes

A Pod can remain in Terminating because of:

Application not handling SIGTERM
        |
preStop hook blocked
        |
Volume cannot unmount
        |
CSI problem
        |
Finalizer
        |
Node unavailable

We can identify the node with:

1kubectl get pod api-456 \
2  -n production \
3  -o wide

Then check it:

1kubectl get node <node>

If the Pod uses volumes:

1kubectl describe pod api-456 \
2  -n production

and:

1kubectl get pvc -n production

can help us identify storage-related issues.

What about force deletion?

We can run:

1kubectl delete pod api-456 \
2  -n production \
3  --grace-period=0 \
4  --force

But this should be a deliberate action, not the first troubleshooting step.

Force deleting a Pod removes its Kubernetes API representation without necessarily guaranteeing that every underlying process or resource has already been terminated cleanly.

This requires even more caution with stateful workloads.

The first objective should be understanding why the Pod is not terminating normally.

When kubectl exec is not enough

Many modern container images are intentionally minimal or even distroless.

You may try:

1kubectl exec -it api-xyz -- sh

and discover that the container does not include:

1sh
2curl
3dig
4tcpdump
5ps

That does not mean we should permanently install debugging tools inside our production images.

Instead, we can use an ephemeral debugging container:

1kubectl debug api-xyz \
2  -n production \
3  -it \
4  --image=ubuntu

kubectl debug allows us to add troubleshooting tools without modifying the original application image.

It can also be used to investigate nodes:

1kubectl debug node/worker-01 \
2  -it \
3  --image=ubuntu

This creates a debugging environment that can help us inspect the node.

Metrics, Logs, and Events answer different questions

One of the most important parts of Kubernetes troubleshooting is understanding that a single source of information is usually not enough.

                 Problem
                    |
        +-----------+-----------+
        |           |           |
      Events      Metrics      Logs
        |           |           |
        v           v           v
   What did K8s?  How much?   What happened?

Events

Useful for:

1FailedScheduling
2FailedMount
3FailedAttachVolume
4FailedPull
5BackOff
6Unhealthy
7Evicted

Command:

1kubectl get events -A \
2  --sort-by='.lastTimestamp'

Metrics

Useful for:

1CPU
2Memory
3Node utilization
4Container utilization

For example:

1kubectl top pods -A
2kubectl top nodes

In environments with Prometheus, we can also inspect historical behavior and correlate incidents with changes in traffic or resource utilization.

Logs

Useful for understanding what happened inside the application:

1kubectl logs <pod>

or:

1kubectl logs <pod> --previous

In production, these logs should usually also be centralized using platforms such as Loki, OpenSearch, Elasticsearch, Splunk, Datadog, or similar solutions.

A troubleshooting process we can reuse

For these and many other Kubernetes problems, we can use the same general flow:

 1                  Symptom
 2                     |
 3                     v
 4             Define blast radius
 5                     |
 6                     v
 7              Check Pod state
 8                     |
 9                     v
10                 Events
11                     |
12                     v
13             Logs / Metrics
14                     |
15                     v
16               Configuration
17                     |
18                     v
19         Dependencies / Infrastructure
20                     |
21                     v
22                 Root cause
23                     |
24                     v
25                   Fix
26                     |
27                     v
28                 Validate
29                     |
30                     v
31                 Prevent

Before changing anything, try to answer:

1What is failing?
2When did it start?
3What changed?
4Does it affect one Pod or several?
5Does it affect one specific node?
6What do the Events say?
7What do the logs show?
8What do the metrics show?

This helps reduce trial-and-error troubleshooting.

Command reference

 1# Pods
 2kubectl get pods -A -o wide
 3
 4# Details and Events
 5kubectl describe pod <pod> -n <namespace>
 6
 7# Logs
 8kubectl logs <pod> -n <namespace>
 9kubectl logs <pod> -n <namespace> --previous
10
11# Events
12kubectl get events -A --sort-by='.lastTimestamp'
13
14# Resources
15kubectl top pods -A
16kubectl top nodes
17
18# Configuration
19kubectl get pod <pod> -n <namespace> -o yaml
20kubectl get deployment <deployment> -n <namespace> -o yaml
21
22# Nodes
23kubectl get nodes -o wide
24kubectl describe node <node>
25
26# Debug container
27kubectl debug <pod> -n <namespace> -it --image=ubuntu
28
29# Node debugging
30kubectl debug node/<node> -it --image=ubuntu

Conclusion

States such as CrashLoopBackOff, ImagePullBackOff, Pending, and Terminating are starting points for troubleshooting, not complete diagnoses.

The key is combining different signals:

 1Kubernetes State
 2       +
 3     Events
 4       +
 5      Logs
 6       +
 7    Metrics
 8       |
 9       v
10   Root Cause

In the next article, we will continue with another common group of Kubernetes problems:

CPU, memory, OOMKilled, throttling, and Pod evictions.

We will look at why a Pod can experience performance problems even when the Kubernetes node appears to have available resources.