Kubernetes Troubleshooting: Health Checks, Readiness, Liveness y Startup Probes

A Kubernetes Pod can be Running while still being unavailable for traffic.

It can also be running a perfectly functional application and start restarting because a health check is incorrectly configured.

Kubernetes provides three primary types of probes:

1Startup
2Liveness
3Readiness

Although their configuration looks similar, they answer very different questions.

 1Startup
 2   |
 3   v
 4Has the application finished starting?
 5
 6Liveness
 7   |
 8   v
 9Is the application still functioning,
10or should it be restarted?
11
12Readiness
13   |
14   v
15Is the application ready
16to receive traffic?

Confusing these responsibilities can turn a small failure into a much larger incident.

In this fourth article of our Kubernetes Troubleshooting series, we will investigate:

• readiness probes;

• liveness probes;

• startup probes;

• HTTP, TCP, exec, and gRPC probes;

• timeouts and thresholds;

• Pods that are Running but NotReady;

• probe-induced restart loops;

• slow-starting applications;

• external dependencies in health checks;

• cascading failures;

• how to design probes that help instead of making incidents worse.

First: the three probes do not do the same thing

This is the most important distinction in this article.

 1                        Container
 2                           |
 3          +----------------+----------------+
 4          |                |                |
 5          v                v                v
 6       Startup          Liveness         Readiness
 7          |                |                |
 8       Started?          Alive?        Can receive
 9                                          traffic?
10          |                |                |
11        FAIL             FAIL             FAIL
12          |                |                |
13          v                v                v
14       Restart          Restart       Remove from
15                                      normal Service
16                                         traffic

A readiness failure does not normally restart the container.

A liveness failure can cause the kubelet to terminate the container and apply its restart policy.

A startup probe protects applications that require additional startup time.

1. Readiness probes

A readiness probe answers:

Is this container able to receive traffic right now?

Suppose we have three Pods:

1                Service
2                   |
3          +--------+--------+
4          |        |        |
5          v        v        v
6        Pod A    Pod B    Pod C
7        Ready    Ready   NotReady

Normal Service traffic should go to endpoints considered ready.

Pod C can remain:

1Running

while not being:

1Ready

That is a valid and important Kubernetes state.

Example readiness probe

1readinessProbe:
2  httpGet:
3    path: /ready
4    port: 8080
5  initialDelaySeconds: 5
6  periodSeconds: 5
7  timeoutSeconds: 2
8  failureThreshold: 3

Conceptually:

 1GET /ready
 2    |
 3    +--- Success
 4    |       |
 5    |       v
 6    |     Ready
 7    |
 8    +--- Failure
 9            |
10            v
11         NotReady
12            |
13            v
14    No normal Service traffic

The application process continues running.

Pod is Running but 0/1 Ready

A common situation is:

1kubectl get pods -n production

returning:

1NAME                    READY   STATUS    RESTARTS
2api-7bd95fcb85-rx7mw    0/1     Running   0

This tells us:

1Container
2   |
3   +--- Running
4   |
5   +--- Not Ready

Start with:

1kubectl describe pod \
2  api-7bd95fcb85-rx7mw \
3  -n production

Look for Events such as:

1Readiness probe failed:
2HTTP probe failed with statuscode: 503

or:

1dial tcp 10.244.2.15:8080:
2connect: connection refused

What is the readiness probe actually checking?

Inspect the workload:

1kubectl get deployment api \
2  -n production \
3  -o yaml

Find:

1readinessProbe:

Then ask:

1Correct path?
2Correct port?
3Correct protocol?
4Is the application listening?
5Is the timeout realistic?
6Is the HTTP response valid?
7Does the endpoint depend on other systems?

We can also reproduce the check manually:

1kubectl exec \
2  -n production \
3  <pod> -- \
4  curl -v http://127.0.0.1:8080/ready

If the production image does not contain curl, use a debugging container or temporary debug Pod instead.

Readiness and EndpointSlices

When a Pod becomes not ready, Kubernetes reflects this in the backend information used by Services.

Inspect it:

1kubectl get endpointslices \
2  -n production \
3  -l kubernetes.io/service-name=api-service \
4  -o yaml

Look for:

1conditions:
2  ready: false

This connects health checks with the networking path we investigated in the previous article:

 1Readiness Probe
 2      |
 3      v
 4Pod NotReady
 5      |
 6      v
 7EndpointSlice condition changes
 8      |
 9      v
10Normal Service traffic stops

2. Liveness probes

A liveness probe asks a different question:

Is the application still healthy enough to keep running?

For example:

1livenessProbe:
2  httpGet:
3    path: /live
4    port: 8080
5  periodSeconds: 10
6  timeoutSeconds: 2
7  failureThreshold: 3

If the probe fails enough consecutive times:

 1Liveness fails
 2      |
 3      v
 4Failure threshold reached
 5      |
 6      v
 7kubelet stops container
 8      |
 9      v
10Restart policy applies
11      |
12      v
13Container starts again

This can recover applications from conditions such as certain deadlocks.

But it can also be dangerous.

When liveness becomes the problem

Imagine an application under heavy load.

Normally:

1/live -> 20 ms

During a traffic spike:

1/live -> 1.5 seconds

But the probe has:

1timeoutSeconds: 1
2failureThreshold: 3

The result can become:

 1Traffic increases
 2      |
 3      v
 4Application slows
 5      |
 6      v
 7Liveness times out
 8      |
 9      v
10Pod restarts
11      |
12      v
13Less capacity
14      |
15      v
16Remaining Pods receive more traffic
17      |
18      v
19They become slower
20      |
21      v
22More liveness failures
23      |
24      v
25More restartstxt

We have created a cascading failure.

The application started with a performance degradation.

The health-check configuration progressively removed capacity.

3. Startup probes

Startup probes are especially useful for applications that need significant initialization time.

For example:

 1Container starts
 2      |
 3      v
 4Load configuration
 5      |
 6      v
 7Initialize runtime
 8      |
 9      v
10Warm caches
11      |
12      v
13Application ready

Suppose this takes:

190 seconds

If liveness begins too early, Kubernetes can terminate the process while it is still legitimately starting.

Without a startup probe

 1Container starts
 2      |
 3      v
 4Application starting...
 5      |
 6      v
 7Liveness begins
 8      |
 9      v
10FAIL
11      |
12      v
13Restart
14      |
15      v
16Application starting...
17      |
18      v
19FAIL

Eventually we may see:

1CrashLoopBackOff

even though the application might have successfully started if given enough time.

With a startup probe

1startupProbe:
2  httpGet:
3    path: /startup
4    port: 8080
5  periodSeconds: 10
6  failureThreshold: 30

This provides an approximate maximum startup window of:

110 seconds × 30 failures = 300 seconds

While startup has not succeeded:

1Startup probe
2      |
3      +--- checking
4      |
5Liveness disabled
6Readiness disabled

Once startup succeeds:

1Startup succeeds
2      |
3      +------------+
4      |            |
5      v            v
6  Liveness     Readiness
7   begins        begins

initialDelaySeconds vs startupProbe

We could configure:

1livenessProbe:
2  initialDelaySeconds: 120

but a startup probe usually communicates the intent more precisely for variable startup times.

A fixed delay means:

1Wait 120 seconds

A startup probe means:

 1Check startup
 2     |
 3     +--- ready after 20 seconds
 4     |         |
 5     |         v
 6     |     continue
 7     |
 8     +--- still starting
 9               |
10               v
11            keep checking

This protects slow startups without necessarily waiting for the maximum possible delay.

4. Troubleshooting probe-induced restart loops

Suppose:

1kubectl get pods -n production

shows:

1NAME                    READY   STATUS             RESTARTS
2api-6b8cff746c-r27ft    0/1     CrashLoopBackOff   8

Do not immediately assume that the application process is crashing on its own.

Run:

1kubectl describe pod \
2  api-6b8cff746c-r27ft \
3  -n production

and suppose we find:

1Warning  Unhealthy
2Liveness probe failed

Our hypothesis changes immediately.

Check:

1kubectl logs \
2  api-6b8cff746c-r27ft \
3  -n production \
4  --previous

and inspect the probe configuration:

1kubectl get deployment api \
2  -n production \
3  -o yaml

Investigate the timeline

We want to determine:

 1Container starts
 2       |
 3       v
 4When does the probe start?
 5       |
 6       v
 7How long does startup take?
 8       |
 9       v
10How frequently is it checked?
11       |
12       v
13How long until failure?
14       |
15       v
16Why does the endpoint fail?

For example:

1livenessProbe:
2  httpGet:
3    path: /health
4    port: 8080
5  initialDelaySeconds: 3
6  periodSeconds: 2
7  timeoutSeconds: 1
8  failureThreshold: 2

may be far too aggressive for some applications.

5. Probe parameters we need to understand

initialDelaySeconds

Controls when probing begins after the container starts.

1initialDelaySeconds: 10

For workloads with variable or long initialization, it should not automatically replace a proper startup probe.

periodSeconds

Controls how frequently the probe runs.

1periodSeconds: 10

Running probes unnecessarily often can create additional overhead.

timeoutSeconds

Controls how long Kubernetes waits for the probe result.

1timeoutSeconds: 2

An unrealistically short timeout can produce false failures under load.

failureThreshold

Controls the number of consecutive failures before Kubernetes considers the probe failed.

1failureThreshold: 3

Together with periodSeconds, this determines much of the failure tolerance.

successThreshold

Controls how many consecutive successes are required after failure.

Readiness can use values greater than one.

For liveness and startup probes:

1successThreshold = 1

is required.

6. HTTP probes

An HTTP probe can look like:

1readinessProbe:
2  httpGet:
3    path: /ready
4    port: 8080
5    scheme: HTTP

We can reproduce it:

1curl -v http://<pod-ip>:8080/ready

Investigate:

1HTTP status
2response time
3connection errors
4TLS
5application logs

7. TCP probes

A TCP probe checks whether a TCP connection can be established.

1livenessProbe:
2  tcpSocket:
3    port: 8080

Conceptually:

1Can kubelet establish TCP connection?
2             |
3        +----+----+
4        |         |
5       YES        NO

But a successful connection does not prove that the application is logically healthy.

A service can accept:

1TCP -> OK

while returning:

1HTTP 500

for every request.

Choose the probe mechanism based on what you actually need to validate.

8. Exec probes

A probe can also execute a command inside the container:

1livenessProbe:
2  exec:
3    command:
4      - /bin/sh
5      - -c
6      - test -f /tmp/healthy

This allows application-specific checks.

However, exec probes involve executing processes within the container.

Very frequent exec probes across dense clusters can create unnecessary overhead, so they should be used deliberately.

9. gRPC probes

Kubernetes also supports native probes for gRPC workloads.

For example:

1livenessProbe:
2  grpc:
3    port: 50051
4  initialDelaySeconds: 10

This can avoid exposing an additional HTTP endpoint solely for Kubernetes health checks.

There are configuration differences compared with HTTP and TCP probes, however.

For example, gRPC probes do not support named ports in the same way HTTP and TCP probes do.

Always validate probe configuration against the protocol being used.

10. Should liveness check the database?

This is one of the most important health-check design questions.

Suppose:

1/live
2  |
3  +--- application
4  +--- PostgreSQL
5  +--- Redis
6  +--- payment API

Now PostgreSQL has an outage.

 1PostgreSQL unavailable
 2       |
 3       v
 4/live fails
 5       |
 6       v
 7Every application Pod
 8fails liveness
 9       |
10       v
11Every Pod restarts
12       |
13       v
14Application capacity collapses

Restarting application Pods does not repair PostgreSQL.

We are using liveness to react to a failure that restarting the local process cannot fix.

A better separation

A useful model is:

1/live
2  |
3  +--- Is my process internally healthy?
4
5/ready
6  |
7  +--- Can I currently serve useful traffic?

For example:

 1Database unavailable
 2       |
 3       v
 4Readiness fails
 5       |
 6       v
 7Pod removed from normal traffic
 8       |
 9       v
10Container remains running

But readiness also requires careful design.

11. The danger of checking every dependency in readiness

Suppose the API uses:

1API
2 |
3 +-- PostgreSQL
4 +-- Redis
5 +-- External Payment API

and readiness requires all three to be healthy.

The payment provider has a small outage:

 1Payment API unavailable
 2      |
 3      v
 4Every API Pod becomes NotReady
 5      |
 6      v
 7Service has zero ready backends
 8      |
 9      v
10Entire API unavailable

But perhaps:

1GET /products
2GET /profile
3GET /orders

could still work.

When designing readiness, ask:

Does failure of this dependency actually make the entire Pod unable to serve useful traffic?

The answer depends on the application architecture.

12. Probes should be cheap

A health endpoint should not execute an expensive database query every few seconds.

Avoid designs such as:

 1/readiness
 2    |
 3    v
 4Complex DB query
 5    |
 6    v
 7Multiple downstream calls
 8    |
 9    v
10Storage operation

multiplied by:

1500 Pods
2×
3probe every 5 seconds

The health-check infrastructure itself can become significant load.

A good probe should normally be:

1fast
2cheap
3predictable
4purpose-specific

13. Observe probe failures

Events are an excellent first signal:

1kubectl get events \
2  -n production \
3  --sort-by='.lastTimestamp'

Look for:

1Unhealthy
2Readiness probe failed
3Liveness probe failed
4Startup probe failed

But Events are not historical monitoring.

Correlate:

 1Probe failures
 2      +
 3Restarts
 4      +
 5CPU
 6      +
 7Memory
 8      +
 9Latency
10      +
11Traffic
12      +
13Deployments

For example:

110:00 Deployment
210:03 Traffic increases
310:05 CPU saturation
410:06 Liveness failures
510:07 Restarts increase
610:08 Available replicas decrease
710:09 5xx spike

That is much more informative than simply seeing Unhealthy.

14. Logs

When a probe fails:

1kubectl logs <pod> \
2  -n production

If the container restarted:

1kubectl logs <pod> \
2  -n production \
3  --previous

Correlate timestamps between Events and application logs.

You might discover:

1probe timeout
2      |
3      v
4same timestamp
5      |
6      v
7long GC pause

or:

1readiness returns 503
2      |
3      v
4connection pool exhausted

or:

1startup failure
2      |
3      v
4database migration still running

15. Metrics and traces

Probes indicate whether an instance is available.

They do not necessarily explain why the instance is degraded.

Suppose:

1Readiness failures increasing

while:

1CPU normal
2Memory normal
3Latency high

A distributed trace could reveal:

1API                    20 ms
2 |
3 +-- Inventory         15 ms
4 |
5 +-- Database        4.2 sec

Now we know which dependency deserves further investigation.

Health checks are one signal.

Logs, metrics, and traces provide the context.

16. Rolling updates and readiness

Readiness becomes especially important during deployments.

 1Old Pods
 2   |
 3   +--- Ready
 4
 5New Pod starts
 6   |
 7   v
 8NotReady
 9   |
10   v
11Initialize
12   |
13   v
14Ready
15   |
16   v
17Receives traffic

Without an appropriate readiness check, a new application instance can begin receiving traffic before it is genuinely prepared to serve requests correctly.

This can produce failures that appear only during rollouts.

17. Readiness Gates

For more advanced scenarios, Kubernetes allows additional conditions to participate in Pod readiness through:

1readinessGates:

Conceptually:

1Containers Ready
2       +
3Custom condition Ready
4       |
5       v
6Pod Ready

This allows controllers or external conditions to influence readiness.

Most workloads do not need readiness gates, but they are useful when availability depends on something that cannot be accurately represented by a normal container probe.

18. Probe failures isolated to one node

Suppose:

1node-01
2  Pods -> probes OK
3
4node-02
5  Pods -> probes FAIL

Now the application itself may not be the problem.

Check placement:

1kubectl get pods \
2  -n production \
3  -o wide

Then:

1kubectl describe node node-02

Possible causes include:

1node CPU saturation
2network failure
3CNI problem
4kubelet problem
5disk pressure
6packet loss

Blast radius is again an important diagnostic signal.

19. What to inspect with kubectl describe

During troubleshooting:

1kubectl describe pod <pod> \
2  -n <namespace>

inspect:

1State
2Last State
3Ready
4Restart Count
5Conditions
6Liveness
7Readiness
8Startup
9Events

Then:

1kubectl get pod <pod> \
2  -n <namespace> \
3  -o yaml

to inspect the complete probe configuration.

20. A practical configuration

A workload might use:

 1startupProbe:
 2  httpGet:
 3    path: /startup
 4    port: 8080
 5  periodSeconds: 5
 6  failureThreshold: 30
 7
 8livenessProbe:
 9  httpGet:
10    path: /live
11    port: 8080
12  periodSeconds: 10
13  timeoutSeconds: 2
14  failureThreshold: 3
15
16readinessProbe:
17  httpGet:
18    path: /ready
19    port: 8080
20  periodSeconds: 5
21  timeoutSeconds: 2
22  failureThreshold: 3

But these values are not universal recommendations.

They should be derived from actual data about:

1startup time
2normal response time
3load behavior
4recovery time
5dependency behavior
6traffic patterns

Copying another service's probe configuration without understanding those characteristics can create new problems.

Troubleshooting flow

 1                 Pod problem
 2                     |
 3                     v
 4            Is container Running?
 5                     |
 6                     v
 7               Is Pod Ready?
 8                     |
 9          +----------+----------+
10          |                     |
11         NO                    YES
12          |                     |
13          v                     v
14   Readiness Events       Restarting?
15          |                     |
16          v                     v
17 Probe endpoint works?    Liveness / Startup?
18          |                     |
19          v                     v
20 Path / port / timeout    Probe timing
21 dependency / overload    startup duration
22          |                     |
23          +----------+----------+
24                     |
25                     v
26           Metrics + Logs + Events
27                     |
28                     v
29              Check dependencies
30                     |
31                     v
32                Root cause
33                     |
34                     v
35                  Change
36                     |
37                     v
38                 Validate

Command reference

 1# Pod state
 2kubectl get pods \
 3  -n <namespace>
 4
 5kubectl get pods \
 6  -n <namespace> \
 7  -o wide
 8
 9# Pod details and probe Events
10kubectl describe pod <pod> \
11  -n <namespace>
12
13# Complete configuration
14kubectl get pod <pod> \
15  -n <namespace> \
16  -o yaml
17
18kubectl get deployment <deployment> \
19  -n <namespace> \
20  -o yaml
21
22# Logs
23kubectl logs <pod> \
24  -n <namespace>
25
26kubectl logs <pod> \
27  -n <namespace> \
28  --previous
29
30# Events
31kubectl get events \
32  -n <namespace> \
33  --sort-by='.lastTimestamp'
34
35# EndpointSlices
36kubectl get endpointslices \
37  -n <namespace> \
38  -l kubernetes.io/service-name=<service> \
39  -o yaml
40
41# Resource metrics
42kubectl top pods \
43  -n <namespace>
44
45kubectl top pod <pod> \
46  -n <namespace> \
47  --containers
48
49# Deployment history
50kubectl rollout history \
51  deployment/<deployment> \
52  -n <namespace>
53
54# Test endpoints
55curl -v http://<pod-ip>:<port>/ready
56curl -v http://<pod-ip>:<port>/live
57
58# Debug when application image lacks tools
59kubectl debug <pod> \
60  -n <namespace> \
61  -it \
62  --image=ubuntu

What not to do

Avoid automatically responding with:

1Readiness failing
2       |
3       v
4Restart Pod

or:

1Liveness failing
2       |
3       v
4Increase timeout

or:

1Slow startup
2       |
3       v
4Set initialDelaySeconds = 300

Understand the failure first.

 1Failure
 2   |
 3   v
 4Understand probe purpose
 5   |
 6   v
 7Reproduce probe
 8   |
 9   v
10Correlate Events
11   |
12   v
13Check metrics + logs
14   |
15   v
16Identify application/dependency issue
17   |
18   v
19Adjust probe or application
20   |
21   v
22Validate under realistic load

Conclusion

Kubernetes probes are not simply health checks.

They can directly change cluster behavior.

 1Readiness failure
 2       |
 3       v
 4Traffic changes
 5
 6Liveness failure
 7       |
 8       v
 9Container lifecycle changes
10
11Startup failure
12       |
13       v
14Container lifecycle changes

That is why a poorly designed probe can be dangerous.

Each probe should answer one specific question:

1Startup:
2Has the application finished starting?
3
4Liveness:
5Can restarting this process recover it?
6
7Readiness:
8Can this instance serve useful traffic now?

Before adding any dependency to a health check, ask:

What will Kubernetes do when this dependency fails, and will that action actually help?

That question prevents many dangerous probe configurations.

In the next article we will troubleshoot Kubernetes Nodes:

• NotReady;

• MemoryPressure;

• DiskPressure;

• PIDPressure;

• kubelet;

• container runtime;

• node networking;

• failures concentrated on a single node;

• determining when an incident has moved from the workload layer to the host layer.