A Pod can be Running and still have serious performance problems.

An application can become slow, continuously consume memory, restart because of OOMKilled, or experience CPU throttling even when the Kubernetes node appears to have resources available.

In this second article of our Kubernetes troubleshooting series, we will investigate:

• high CPU usage;

• high memory usage;

• OOMKilled;

• CPU throttling;

• Pod evictions;

• node-level resource pressure.

The objective is not simply to find "which Pod is using the most resources."

We want to answer:

Is the problem in the application, its requests and limits, or the capacity of the node?

Requests, limits, and actual usage

Before investigating CPU or memory, we need to distinguish three concepts:

1                Resources
2                    |
3       +------------+------------+
4       |            |            |
5    Request        Limit       Usage
6       |            |            |
7       v            v            v
8 Scheduling     Enforcement    Actual

Requests

A request tells Kubernetes how much of a resource a workload asks for.

For example:

1resources:
2  requests:
3    cpu: "250m"
4    memory: "256Mi"

The scheduler uses these values when deciding which node can run the Pod.

A container can use more than its request when resources are available and its limits allow it.

Limits

A limit places a constraint on resource consumption.

1resources:
2  requests:
3    cpu: "250m"
4    memory: "256Mi"
5  limits:
6    cpu: "500m"
7    memory: "512Mi"

CPU and memory limits behave differently.

 1CPU limit
 2    |
 3    v
 4CPU throttling
 5
 6Memory limit
 7    |
 8    v
 9Memory pressure / allocation
10    |
11    v
12Possible OOM kill

A container is not terminated simply because it consumes too much CPU. The kernel constrains its available CPU time through cgroup mechanisms.

Memory works differently. When memory cannot be allocated within the applicable cgroup constraint, the kernel's OOM mechanisms can terminate a process.

1. High CPU usage

Suppose an alert reports increased latency and we find a Pod consuming a large amount of CPU.

A good first check is:

1kubectl top pods -A

To see individual containers:

1kubectl top pod <pod> \
2  -n <namespace> \
3  --containers

We can also sort Pods by CPU:

1kubectl top pods \
2  -n <namespace> \
3  --sort-by=cpu

This gives us a recent snapshot of resource consumption.

However:

kubectl top is not a replacement for a complete monitoring system.

Its values normally come from Metrics Server and are intended to provide lightweight resource signals suitable for Kubernetes components such as autoscalers.

To understand when a problem started or how it evolved over hours or days, we need historical monitoring.

What should we ask when CPU increases?

Do not immediately assume that the workload needs more CPU.

Start here:

 1High CPU
 2   |
 3   +--- Did traffic increase?
 4   |
 5   +--- Was the application recently changed?
 6   |
 7   +--- Is one endpoint CPU intensive?
 8   |
 9   +--- Is a dependency responding slowly?
10   |
11   +--- Is there a loop or runaway process?
12   |
13   +--- Did background workload increase?
14   |
15   +--- Is the container being throttled?

Check recent revisions:

1kubectl rollout history \
2  deployment/<deployment> \
3  -n <namespace>

Then inspect resource configuration:

1kubectl get deployment <deployment> \
2  -n <namespace> \
3  -o yaml

Look for:

1resources:
2  requests:
3    cpu:
4  limits:
5    cpu:

CPU usage vs CPU limit

Suppose we have:

1CPU request:  250m
2CPU limit:    500m
3CPU usage:    490m

The container is operating close to its CPU limit.

Now suppose the node reports:

1Node CPU usage: 35%

That may initially look contradictory.

It is not.

The container has its own CPU constraint.

 1Node
 216 CPUs available
 3      |
 4      |        Plenty of CPU available
 5      |
 6      +------------------------------+
 7                                     |
 8Container                            |
 9CPU limit = 500m                     |
10      |                              |
11      v                              |
12reaches limit                        |
13      |                              |
14      v                              |
15CPU throttling <---------------------+

A node can therefore have available CPU while an individual workload is constrained by its CPU limit.

2. CPU throttling

CPU throttling deserves its own investigation.

A workload can show:

1Pods: Running
2Node CPU: Normal
3Memory: Normal
4Errors: Low
5
6Latency: HIGH

and still be constrained by CPU.

CPU limits are enforced through kernel and cgroup mechanisms. Once a workload exhausts its permitted CPU quota for a scheduling period, its execution can be delayed until more CPU time becomes available.

For latency-sensitive workloads, this can matter significantly.

Prometheus metrics

Depending on your container runtime and metrics collection stack, you may have metrics such as:

1container_cpu_usage_seconds_total
2
3container_cpu_cfs_throttled_periods_total
4
5container_cpu_cfs_periods_total
6
7container_cpu_cfs_throttled_seconds_total

The exact metrics and labels available can differ between environments, so verify what your monitoring stack actually exposes before relying on a query.

For CPU consumption, a query might look like:

1rate(
2  container_cpu_usage_seconds_total{
3    namespace="production",
4    pod=~"api-.*",
5    container!="POD",
6    container!=""
7  }[5m]
8)

When the CFS period metrics are available, a throttled-period ratio can be calculated with a query such as:

1sum by (namespace, pod, container) (
2  rate(container_cpu_cfs_throttled_periods_total[5m])
3)
4/
5sum by (namespace, pod, container) (
6  rate(container_cpu_cfs_periods_total[5m])
7)

Do not interpret throttling in isolation.

Correlate:

 1CPU throttling
 2      +
 3CPU usage
 4      +
 5request rate
 6      +
 7latency
 8      +
 9application behavior
10      |
11      v
12Better diagnosis

How should CPU throttling be resolved?

There is no universal answer.

Depending on the cause, we might:

• optimize CPU-intensive code;

• increase CPU requests;

• modify CPU limits;

• increase replicas;

• adjust autoscaling;

• separate background workloads;

• investigate noisy neighbors;

• increase node capacity.

Removing every CPU limit is not automatically the correct answer either.

Limits can help protect shared environments from workloads that would otherwise consume excessive resources.

The appropriate configuration depends on the workload's behavior and the required balance between performance and isolation.

3. High memory usage

Start again with:

1kubectl top pods -A

and:

1kubectl top pod <pod> \
2  -n <namespace> \
3  --containers

However, a single measurement cannot tell us whether memory looks like this:

1Normal high usage
2
3Memory
4  ^
5  |        ____________
6  |       /
7  |______/
8                 time

or this:

 1Possible leak
 2
 3Memory
 4  ^
 5  |                 /
 6  |              /
 7  |           /
 8  |        /
 9  |_____/
10                 time

Historical metrics are particularly important for memory investigations.

Questions to ask about memory

 1High Memory
 2    |
 3    +--- Is usage continuously growing?
 4    |
 5    +--- Does it stabilize?
 6    |
 7    +--- Does it correlate with traffic?
 8    |
 9    +--- Was the application changed?
10    |
11    +--- Is caching involved?
12    |
13    +--- Could there be a memory leak?
14    |
15    +--- Is a queue growing?
16    |
17    +--- Does the runtime have a configured heap?
18    |
19    +--- Are we approaching the memory limit?

Inspect configured resources:

1kubectl get pod <pod> \
2  -n <namespace> \
3  -o yaml

For example:

1resources:
2  requests:
3    memory: "512Mi"
4  limits:
5    memory: "1Gi"

Then compare:

1Memory usage
2       vs
3Memory request
4       vs
5Memory limit

4. OOMKilled

One of the clearest symptoms is:

1Reason: OOMKilled

Start with:

1kubectl describe pod <pod> \
2  -n <namespace>

We can also inspect the previous container state:

1kubectl get pod <pod> \
2  -n <namespace> \
3  -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

And retrieve the previous container's logs:

1kubectl logs <pod> \
2  -n <namespace> \
3  --previous

OOMKilled vs Exit Code 137

This distinction is important.

You may encounter:

1Exit Code: 137

137 conventionally represents:

1128 + 9 = 137
2
3signal 9 = SIGKILL

However:

Exit code 137 by itself does not prove that the process was killed because of an out-of-memory condition.

A process can receive SIGKILL for other reasons.

Look for Kubernetes reporting:

1Reason: OOMKilled

and correlate it with:

• container state;

• Events;

• memory metrics;

• logs;

• node conditions.

Container OOM vs node memory pressure

Another important distinction is:

 1          Memory problem
 2                |
 3        +-------+-------+
 4        |               |
 5        v               v
 6 Container OOM      Node Pressure
 7        |               |
 8 Memory/cgroup      Node running
 9 constraint         low on memory
10        |               |
11        v               v
12 OOMKilled          Possible eviction

These are not the same incident.

Container OOM

For example:

1limits:
2  memory: "512Mi"

If the workload cannot allocate the memory it needs within its applicable constraint, the kernel's OOM mechanism can kill a process.

Node memory pressure

Multiple workloads may instead consume enough memory that the node itself becomes resource constrained.

Check:

1kubectl describe node <node>

Look for:

1Conditions:
2  MemoryPressure:

And inspect current usage:

1kubectl top node <node>

How should OOMKilled be resolved?

Increasing the memory limit may be correct.

But first answer:

Why does the application need more memory?

Possible explanations include:

 1OOMKilled
 2   |
 3   +--- Memory leak
 4   |
 5   +--- Limit too low
 6   |
 7   +--- Higher traffic
 8   |
 9   +--- Unbounded cache
10   |
11   +--- Incorrect heap configuration
12   |
13   +--- Large query / processing job
14   |
15   +--- Excessive worker concurrency
16   |
17   +--- Memory-backed volume usage

Changing:

1resources:
2  requests:
3    memory: "512Mi"
4  limits:
5    memory: "1Gi"

to:

1resources:
2  requests:
3    memory: "1Gi"
4  limits:
5    memory: "2Gi"

may be appropriate when measurements demonstrate that the additional memory is legitimate.

If the workload has a memory leak, increasing the limit may simply delay the next failure.

5. Pod evictions

Now suppose we find:

1STATUS: Evicted

Start with:

1kubectl describe pod <pod> \
2  -n <namespace>

Then investigate its node:

1kubectl describe node <node>

Kubernetes supports node-pressure eviction, where the kubelet can terminate Pods to reclaim constrained node resources.

The kubelet monitors signals associated with resources such as:

1Memory
2Disk
3Filesystem inodes

When configured thresholds are crossed, Pods can be selected for eviction so the node can reclaim resources.

Node conditions

A good starting point is:

1kubectl get nodes

followed by:

1kubectl describe node <node>

Look for conditions such as:

1MemoryPressure
2DiskPressure
3PIDPressure

Then inspect current usage:

1kubectl top node <node>

List workloads assigned to that node:

1kubectl get pods -A \
2  --field-selector spec.nodeName=<node> \
3  -o wide

This helps identify which workloads were colocated on the affected node.

Evicted is not OOMKilled

These symptoms are easy to confuse.

 1OOMKilled
 2   |
 3A process/container is killed
 4as part of an OOM condition
 5
 6             !=
 7
 8Evicted
 9   |
10kubelet terminates a Pod
11to reclaim constrained
12node resources

The investigation therefore differs.

For OOMKilled, the container and its memory behavior are primary investigation targets.

For Evicted, the node and its resource pressure become especially important.

6. QoS Classes

Kubernetes assigns a Quality of Service class to Pods based on their CPU and memory resource configuration.

Check it with:

1kubectl get pod <pod> \
2  -n <namespace> \
3  -o jsonpath='{.status.qosClass}'

The classes are:

1Guaranteed
2Burstable
3BestEffort

Guaranteed

With traditional per-container resource configuration, a Pod qualifies as Guaranteed when every container has CPU and memory requests and limits, with the request equal to the limit for each resource.

For example:

1resources:
2  requests:
3    cpu: "500m"
4    memory: "512Mi"
5  limits:
6    cpu: "500m"
7    memory: "512Mi"

Burstable

A Pod with at least some CPU or memory resource configuration that does not meet the Guaranteed criteria is generally classified as Burstable.

BestEffort

A Pod without CPU or memory requests or limits is normally:

1BestEffort

For example:

1resources: {}

QoS and eviction

QoS matters when Kubernetes selects workloads during node resource pressure, but it is not the only consideration.

Pod priority and whether resource consumption exceeds requests can also affect eviction decisions.

Therefore, avoid reducing every eviction decision to:

1BestEffort -> Burstable -> Guaranteed

as if QoS class alone completely determined the result.

The actual node-pressure eviction process considers additional factors.

7. Requests that are too low

Consider:

1resources:
2  requests:
3    cpu: "50m"
4    memory: "64Mi"
5  limits:
6    cpu: "2"
7    memory: "2Gi"

while the workload normally consumes:

1CPU:     800m
2Memory:  900Mi

The Pod may run successfully.

However, its requests tell the scheduler that the workload needs considerably fewer resources than it typically consumes.

Across many workloads, this can contribute to:

1Node appears to have capacity
2        |
3Scheduler places more Pods
4        |
5Actual usage grows
6        |
7Node becomes saturated
8        |
9Resource pressure

Requests should therefore not be selected solely to make Pods easier to schedule.

They should reasonably reflect the workload's resource requirements.

8. What if requests are too high?

The opposite problem also exists.

Suppose we configure:

1requests:
2  cpu: "4"
3  memory: "8Gi"

for a workload that typically consumes:

1CPU:     200m
2Memory:  300Mi

This can waste schedulable capacity.

Even when:

1kubectl top nodes

shows low actual utilization, the scheduler may refuse additional Pods because Pod placement is based on requested resources.

Inspect:

1kubectl describe node <node>

and find:

1Allocated resources:

This helps us compare:

1Actual utilization
2        vs
3Requested resources

9. Useful Prometheus metrics

kubectl top is useful for a quick snapshot.

During a real incident, we usually need to understand how resource behavior changed over time.

Metrics commonly available from kubelet/cAdvisor pipelines can include:

1container_cpu_usage_seconds_total
2
3container_memory_working_set_bytes
4
5container_cpu_cfs_throttled_periods_total
6
7container_cpu_cfs_periods_total
8
9container_cpu_cfs_throttled_seconds_total

kube-state-metrics can separately expose Kubernetes object state such as configured requests and limits.

Exact metric availability and labels depend on the monitoring architecture.

CPU usage

For example:

1sum by (namespace, pod) (
2  rate(
3    container_cpu_usage_seconds_total{
4      container!="",
5      container!="POD"
6    }[5m]
7  )
8)

Memory working set

1sum by (namespace, pod) (
2  container_memory_working_set_bytes{
3    container!="",
4    container!="POD"
5  }
6)

Instead of only asking for the current value, inspect:

130 minutes before
2       |
3incident begins
4       |
5resource growth
6       |
7failure
8       |
9recovery

The trend is often more useful than the snapshot.

10. Correlating resources with application behavior

Consider the following timeline:

114:00 CPU normal
214:05 Deployment
314:10 CPU increases
414:12 Latency increases
514:15 CPU throttling increases
614:20 5xx increases

We now have a much stronger story than:

"CPU is high."

Our hypothesis becomes:

 1Deployment
 2    |
 3    v
 4CPU growth
 5    |
 6    v
 7Throttling
 8    |
 9    v
10Latency
11    |
12    v
13Errors

The next investigation might use application logs and distributed traces to discover which new execution path is consuming additional CPU.

11. Investigating the node

When the evidence suggests that the problem affects an entire node:

1kubectl describe node <node>

and:

1kubectl top node <node>

are useful starting points.

If deeper investigation is required and permissions allow it:

1kubectl debug node/<node> \
2  -it \
3  --image=ubuntu

From the debugging environment, host resources can be inspected according to the permissions and environment available.

For example:

1free -h

1df -h

1df -i

And in environments where administrators have direct host access, kubelet logs can also be investigated:

1journalctl -u kubelet

Troubleshooting flow

We can summarize the investigation as:

 1               Performance issue
 2                       |
 3                       v
 4                Define impact
 5                       |
 6                       v
 7              kubectl top / metrics
 8                       |
 9             +---------+---------+
10             |                   |
11             v                   v
12            CPU                Memory
13             |                   |
14      +------+------+      +-----+------+
15      |             |      |            |
16 High usage    Throttling  Growth     OOMKilled
17      |             |      |            |
18      +------+------+      +-----+------+
19             |                   |
20             v                   v
21        Requests/limits      Requests/limits
22             |                   |
23             +---------+---------+
24                       |
25                       v
26                  Check node
27                       |
28          +------------+-------------+
29          |            |             |
30      CPU usage   MemoryPressure DiskPressure
31          |            |             |
32          +------------+-------------+
33                       |
34                       v
35              Correlate application
36                       |
37                       v
38             Logs / Metrics / Traces
39                       |
40                       v
41                   Root cause

Command reference

 1# Current usage
 2kubectl top pods -A
 3kubectl top nodes
 4
 5# Container-level usage
 6kubectl top pod <pod> \
 7  -n <namespace> \
 8  --containers
 9
10# Sort by usage
11kubectl top pods \
12  -n <namespace> \
13  --sort-by=cpu
14
15kubectl top pods \
16  -n <namespace> \
17  --sort-by=memory
18
19# Pod state
20kubectl describe pod <pod> \
21  -n <namespace>
22
23# Previous container state
24kubectl get pod <pod> \
25  -n <namespace> \
26  -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
27
28# Previous logs
29kubectl logs <pod> \
30  -n <namespace> \
31  --previous
32
33# Resource configuration
34kubectl get deployment <deployment> \
35  -n <namespace> \
36  -o yaml
37
38# Node investigation
39kubectl describe node <node>
40kubectl top node <node>
41
42# Pods running on a node
43kubectl get pods -A \
44  --field-selector spec.nodeName=<node> \
45  -o wide
46
47# QoS class
48kubectl get pod <pod> \
49  -n <namespace> \
50  -o jsonpath='{.status.qosClass}'
51
52# Node debugging
53kubectl debug node/<node> \
54  -it \
55  --image=ubuntu

What not to do

During an incident, avoid automatic responses such as:

1OOMKilled
2    -> increase memory
3
4High CPU
5    -> increase CPU
6
7Evicted
8    -> add another node

Those actions might temporarily mitigate the problem without resolving its cause.

Instead:

 1Symptom
 2   |
 3   v
 4Measure
 5   |
 6   v
 7Correlate
 8   |
 9   v
10Find constraint
11   |
12   v
13Understand workload
14   |
15   v
16Change
17   |
18   v
19Validate

After making a change, verify:

• CPU;

• memory;

• throttling;

• restart count;

• latency;

• error rate;

• node pressure;

• application behavior.

Conclusion

Resource problems in Kubernetes can occur at multiple levels:

1Application
2     |
3Container
4     |
5Pod
6     |
7Node
8     |
9Cluster

An OOMKilled container does not automatically mean the node ran out of memory.

High CPU does not automatically mean we need more CPUs.

And an evicted Pod is not the same as a process being terminated by the OOM killer.

To investigate these incidents correctly, connect:

 1Requests + Limits
 2       +
 3Actual Usage
 4       +
 5Container State
 6       +
 7Node Conditions
 8       +
 9Historical Metrics
10       +
11Application behavior
12       |
13       v
14    Root Cause

In the next article in this series, we will troubleshoot Kubernetes networking:

• Services;

• EndpointSlices;

• DNS;

• Ingress and Gateway API;

• NetworkPolicy;

• CNI;

• Pod-to-Pod connectivity;

• outbound connectivity.

We will follow the traffic path layer by layer to determine exactly where communication is failing.