This post is also available in: Español
Kubernetes networking problems can be difficult to diagnose because the same application symptom can originate from many different layers.
An application might report:
Connection refused
Connection timed out
No route to host
Name or service not known
502 Bad Gateway
503 Service Unavailable
but the actual problem could be somewhere along this path:
Application
|
Pod
|
Pod Network
|
Service
|
EndpointSlice
|
NetworkPolicy
|
Ingress / Gateway
|
Load Balancer
|
DNS
|
External Network
The most effective strategy is not to start changing configurations.
Instead, follow the connection one layer at a time until we can identify exactly where communication stops working.
In this third article of our Kubernetes Troubleshooting series, we will investigate:
Pod-to-Pod connectivity;
Services;
EndpointSlices;
DNS;
ports and
targetPort;NetworkPolicies;
Ingress;
Gateway API;
LoadBalancers;
CNI;
outbound connectivity.
First, understand the Kubernetes network model
Before troubleshooting, we need to understand a few expectations of Kubernetes networking.
Each Pod receives its own IP address inside the cluster.
Under the Kubernetes network model, and unless intentional network segmentation prevents it, Pods should be able to communicate with other Pods even when those Pods are running on different nodes.
Containers within the same Pod share a network namespace and can communicate over:
localhost
Services add another abstraction:
Client Pod
|
v
Service
|
v
EndpointSlice
|
+------ Pod A
|
+------ Pod B
|
+------ Pod C
The Service provides a stable access point while backend Pods can be created, deleted, or receive different IP addresses.
Our troubleshooting approach
For networking, we will use one simple rule:
Test the closest destination first, then move outward one layer at a time.
For example, if:
https://api.example.com
does not work, we should not immediately start changing public DNS or restarting the Ingress Controller.
Instead test:
Application listening?
|
v
Pod IP reachable?
|
v
Service reachable?
|
v
EndpointSlice correct?
|
v
DNS working?
|
v
NetworkPolicy allows traffic?
|
v
Ingress / Gateway route?
|
v
Load Balancer?
|
v
Public DNS?
Every successful test eliminates possible causes.
1. Start with the Pods
Suppose we have:
frontend
|
v
api-service
|
v
api Pods
and the frontend reports:
connection refused
First inspect the backend Pods:
kubectl get pods \
-n production \
-o wide
Look at:
STATUS;READY;IP;
NODE;
restart count.
For example:
NAME READY STATUS IP
api-7cf8f76bcd-jx9pq 1/1 Running 10.244.1.23
api-7cf8f76bcd-qt8fz 1/1 Running 10.244.2.15
Running does not guarantee that the application is accepting connections, but it gives us a starting point.
2. Is the application actually listening?
A surprisingly common Kubernetes networking problem is not a Kubernetes networking problem at all.
The application may not be listening on the expected port.
If the container image includes the necessary tools:
kubectl exec \
-n production \
<pod> -- \
ss -lntp
We can also test the application locally:
kubectl exec \
-n production \
<pod> -- \
curl -v http://127.0.0.1:8080/health
If this fails:
Pod
|
X
Application
there is little value in troubleshooting the Service yet.
First determine why the process is not accepting connections.
A classic problem: listening only on localhost
Suppose the application binds to:
127.0.0.1:8080
Inside the container:
curl http://127.0.0.1:8080
may work.
But other Pods reach the application using its Pod IP:
10.244.1.23:8080
If the process only listens on loopback, that connection can fail.
Applications running in Kubernetes commonly need to listen on:
0.0.0.0
or another appropriate Pod interface.
3. Test the Pod IP directly
From another Pod, test the backend address.
A temporary debugging Pod is useful:
kubectl run network-debug \
-n production \
--rm -it \
--restart=Never \
--image=curlimages/curl \
-- sh
From inside it:
curl -v http://10.244.1.23:8080
If this works:
Debug Pod
|
v
Pod IP
|
v
Application
|
OK
we have verified:
basic Pod networking;
the destination port;
the application.
Now we can move to the Service layer.
If it fails, investigate:
Application
Pod networking
NetworkPolicy
CNI
Node networking
4. Investigate the Service
List Services:
kubectl get svc \
-n production
Then inspect the affected Service:
kubectl describe svc api-service \
-n production
Pay attention to:
Selector
Type
IP Families
ClusterIP
Port
TargetPort
A Service object can exist correctly in the Kubernetes API while still not providing working connectivity.
Service port vs targetPort
Consider:
ports:
- port: 80
targetPort: 8080
The flow is:
Client
|
| :80
v
Service
|
| :8080
v
Pod
port is the Service port exposed to clients.
targetPort identifies the backend port.
If the application listens on:
8080
but the Service uses:
targetPort: 3000
DNS can work and the Service can exist, but traffic will not reach the expected application port.
5. EndpointSlices: does the Service have backends?
This is one of the most important checks.
Modern Kubernetes troubleshooting should primarily inspect EndpointSlices:
kubectl get endpointslices \
-n production \
-l kubernetes.io/service-name=api-service
For more detail:
kubectl describe endpointslice \
-n production \
-l kubernetes.io/service-name=api-service
EndpointSlice is the API Kubernetes uses to efficiently represent Service backends.
Conceptually:
Service
|
v
EndpointSlice
|
+--- 10.244.1.23:8080
|
+--- 10.244.2.15:8080
If endpoints exist, we can test those IP addresses directly.
What if there are no endpoints?
Suppose we have:
Service
|
v
EndpointSlice
|
X
No backend Pods
One of the first things to inspect is the Service selector.
kubectl get svc api-service \
-n production \
-o yaml
For example:
selector:
app: api
Now inspect Pod labels:
kubectl get pods \
-n production \
--show-labels
If the Pods have:
app=backend-api
but the Service selects:
app=api
there is no match.
The Service will not have the expected Pod backends.
Readiness matters too
Suppose the labels match correctly.
But:
kubectl get pods -n production
shows:
NAME READY
api-abc 0/1
api-def 0/1
A failing readiness probe can prevent those Pods from being considered ready endpoints for normal Service traffic.
Check:
kubectl describe pod api-abc \
-n production
Then inspect EndpointSlice conditions:
kubectl get endpointslices \
-n production \
-l kubernetes.io/service-name=api-service \
-o yaml
EndpointSlices can expose conditions such as:
ready
serving
terminating
These help consumers distinguish whether an endpoint is ready to receive traffic and whether it is terminating.
6. Test the Service from inside the cluster
Once backend endpoints are confirmed:
kubectl run network-debug \
-n production \
--rm -it \
--restart=Never \
--image=curlimages/curl \
-- sh
Then:
curl -v http://api-service
or:
curl -v http://api-service:80
We can also test the ClusterIP directly:
curl -v http://10.96.120.15:80
This creates a useful distinction.
Pod IP works, Service IP fails
Client
|
+---- Pod IP --------> OK
|
+---- Service IP ----> FAIL
Now focus on:
Service configuration
EndpointSlice
Service proxy implementation
Node networking
Depending on the cluster networking architecture, Service traffic may be implemented by kube-proxy or another dataplane implementation.
Service IP works, Service DNS fails
Service ClusterIP
|
OK
Service name
|
FAIL
That points strongly toward DNS.
7. DNS troubleshooting
Kubernetes provides DNS-based Service discovery.
From a Pod in the same namespace we can normally use:
api-service
From another namespace:
api-service.production
or a fully qualified service name:
api-service.production.svc.cluster.local
The cluster domain is configurable, so cluster.local is common but should not be treated as universal.
Test name resolution
From a Pod with DNS tools:
nslookup api-service
Then:
nslookup api-service.production
And:
nslookup api-service.production.svc.cluster.local
Also inspect:
cat /etc/resolv.conf
A typical configuration might include:
search production.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
This helps explain why short names can resolve within one namespace but not another.
Check CoreDNS
If DNS is failing across several workloads:
kubectl get pods \
-n kube-system \
-l k8s-app=kube-dns
Inspect the DNS Service:
kubectl get svc \
-n kube-system
And logs:
kubectl logs \
-n kube-system \
-l k8s-app=kube-dns
A production cluster can have several replicas or containers involved, so during an incident it is useful to identify exactly which instances are processing queries.
An important distinction
Consider:
nslookup api-service
|
FAIL
nslookup kubernetes.default
|
FAIL
This suggests a broader DNS problem.
But if:
nslookup kubernetes.default
|
OK
nslookup api-service
|
FAIL
investigate:
Service name
Namespace
Service existence
DNS record creation
before deciding that CoreDNS itself is completely unavailable.
8. NetworkPolicy
Another common scenario is:
Pod A -> Pod B
worked before a networking policy change.
Start with:
kubectl get networkpolicy -A
Then:
kubectl describe networkpolicy \
<policy> \
-n production
Also inspect labels:
kubectl get pods \
-n production \
--show-labels
because NetworkPolicy selects Pods using labels.
NetworkPolicies are additive
This behavior is important.
When multiple NetworkPolicies select the same Pod, they are not processed as ordered firewall rules where the first match wins.
Allowed traffic from applicable policies is combined.
For Pod-to-Pod communication:
Source Pod
|
| Egress must allow
v
Network
|
| Ingress must allow
v
Destination Pod
When both sides are isolated, the source egress policy and destination ingress policy must both permit the connection.
Test a NetworkPolicy hypothesis
From the source Pod:
kubectl exec \
-n frontend \
<frontend-pod> -- \
curl -v --connect-timeout 5 \
http://api-service.production:8080
Or test TCP connectivity:
nc -vz \
api-service.production \
8080
If:
DNS resolves
|
v
Correct destination
|
v
Connection timeout
NetworkPolicy, routing, or firewall filtering become stronger hypotheses.
Remember the CNI implementation
Creating a NetworkPolicy object does not automatically mean a cluster can enforce it.
The cluster networking implementation must support NetworkPolicy enforcement.
Therefore:
NetworkPolicy exists
|
X
does not necessarily mean
traffic is being enforced
Actual behavior depends on the networking plugin.
9. Same-node communication works, cross-node communication fails
This is a valuable signal for CNI or infrastructure problems.
Suppose:
Node A
|
+-- Pod A
|
| works
v
+-- Pod B
but:
Node A Node B
| |
Pod A -----------------> Pod C
FAIL
That significantly changes the investigation.
Check Pod placement:
kubectl get pods \
-A \
-o wide
Then nodes:
kubectl get nodes -o wide
Potential areas include:
CNI agents;
routing;
encapsulation;
firewall;
MTU;
node interfaces;
IP address management.
Identify the networking stack
Different Kubernetes distributions use different networking components.
Start with:
kubectl get pods \
-n kube-system \
-o wide
You might find implementations such as:
Calico
Cilium
Flannel
Antrea
cloud-provider-specific networking
Each implementation has its own commands, metrics, and architecture.
Do not start using Calico- or Cilium-specific troubleshooting commands until you know which network implementation the cluster actually uses.
10. CNI and IP allocation failures
If newly created Pods cannot obtain working networking, Events can provide useful clues.
kubectl describe pod <pod> \
-n production
Look for messages related to:
sandbox creation
network setup
CNI
IP allocation
Also review cluster Events:
kubectl get events -A \
--sort-by='.lastTimestamp'
This can reveal whether several Pods or nodes began failing at approximately the same time.
11. MTU: the problem that can look random
Some networking failures look like:
Small request
|
OK
Large request
|
timeout
or:
TCP connection
|
established
|
TLS handshake / large response
|
FAIL
One possible cause is an MTU mismatch.
This can appear in networks using:
overlays;
VXLAN;
VPNs;
cloud networking;
tunnels.
Inspect interfaces:
ip link
and routing:
ip route
MTU changes should only be made with an understanding of the CNI and underlying network.
Changing MTU values through trial and error can introduce additional failures.
12. Ingress troubleshooting
Suppose:
curl http://api-service.production
works inside the cluster.
But:
https://api.example.com
does not work externally.
That tells us:
Pod OK
Service OK
DNS OK
|
v
Problem is farther upstream
Now investigate Ingress.
kubectl get ingress -A
Then:
kubectl describe ingress <name> \
-n production
Inspect:
IngressClass
Host
Path
Backend Service
Backend Port
TLS
Events
Address
Ingress requires a controller
Creating:
kind: Ingress
does not by itself implement the traffic routing.
An Ingress Controller must watch those resources and configure the corresponding data plane.
Check:
kubectl get ingressclass
Then identify the implementation used by the cluster.
Inspect its Pods and logs according to its namespace:
kubectl get pods \
-n <ingress-controller-namespace>
kubectl logs \
-n <ingress-controller-namespace> \
<controller-pod>
Do not assume that every Kubernetes cluster uses ingress-nginx.
502 vs 503
The exact meaning depends on the proxy or controller implementation, but these status codes can guide the investigation.
For example:
External Client
|
v
Ingress Controller
|
+---- backend unavailable
|
+---- backend connection failure
|
+---- timeout
Always correlate:
controller logs;
EndpointSlices;
application logs;
latency metrics.
Do not diagnose solely from the HTTP status code.
13. Gateway API
For new designs, Gateway API should also be considered.
The Kubernetes project recommends Gateway rather than extending the older Ingress API.
Ingress remains supported, but its API is frozen.
A typical Gateway API traffic path might look like:
Client
|
Gateway
|
HTTPRoute
|
Service
|
EndpointSlice
|
Pods
Depending on the installed API and implementation:
kubectl get gateway -A
kubectl get httproute -A
Then:
kubectl describe gateway <gateway> \
-n <namespace>
kubectl describe httproute <route> \
-n <namespace>
Gateway API also requires an appropriate implementation/controller. Kubernetes does not automatically provide a Gateway dataplane simply because Gateway resources exist.
14. LoadBalancer remains Pending
Another common problem is:
kubectl get svc
showing:
TYPE EXTERNAL-IP
LoadBalancer <pending>
Ask:
Service type LoadBalancer
|
v
Who provides the load balancer?
In a cloud environment, this can involve:
cloud controller;
IAM or permissions;
network configuration;
quota;
subnets;
load balancer controllers.
In bare-metal and local environments, an additional load-balancer implementation may be required.
Kubernetes defines the LoadBalancer abstraction, but the surrounding infrastructure must fulfill it.
15. Public DNS
Suppose the load balancer works:
Load Balancer
|
OK
but:
api.example.com
|
FAIL
Check public DNS:
dig api.example.com
or:
nslookup api.example.com
Compare the result with the expected load balancer address.
A particularly useful test is:
curl -v \
--resolve api.example.com:443:<IP> \
https://api.example.com
This allows us to test:
HTTP
TLS
Host-based routing
against a specific IP without relying on normal DNS resolution.
16. TLS
A networking problem can actually be a TLS problem.
Start with:
curl -vk https://api.example.com
For more detail:
openssl s_client \
-connect api.example.com:443 \
-servername api.example.com
Inspect:
certificate chain
expiration
SAN
issuer
SNI
hostname
If Kubernetes stores the certificate in a Secret:
kubectl get secret <tls-secret> \
-n production
and inspect the relevant Ingress or Gateway resource to confirm which certificate should be presented.
17. Pods cannot reach the Internet
Now investigate the opposite direction:
Pod
|
v
Internet
|
X
Start from inside the Pod:
curl -v https://example.com
Then separate DNS from connectivity:
nslookup example.com
If DNS succeeds but the connection fails:
DNS
|
OK
|
v
TCP connection
|
FAIL
investigate:
NetworkPolicy egress;
routing;
NAT;
firewall;
proxy;
cloud network controls;
CNI.
DNS vs egress
For example:
nslookup example.com
works.
But:
curl https://example.com
times out.
DNS resolution is working, so restarting CoreDNS is unlikely to address the actual problem.
If:
nslookup example.com
|
FAIL
then determine whether the Pod can reach its configured DNS server before moving to other networking layers.
18. Database connection timeout
The same methodology works for database connectivity.
Suppose an application reports:
connection to database timed out
Break the connection into stages:
Application
|
v
DNS resolution
|
v
TCP connection
|
v
TLS
|
v
Authentication
|
v
Database
DNS:
nslookup database.example.internal
TCP:
nc -vz \
database.example.internal \
5432
If the connection succeeds, we have already demonstrated that:
DNS + routing + TCP path
work to some degree.
We can then investigate:
TLS;
credentials;
database availability;
connection limits;
application connection pools.
Good networking troubleshooting is also about proving which layers are not broken.
19. Connection refused vs timeout
These symptoms can provide useful clues.
Connection refused
Client
|
v
Destination reached
|
v
Nothing accepting connection
Possible causes include:
process not listening;
wrong port;
proxy actively rejecting the connection.
Timeout
Client
|
v
Packets sent
|
?
|
No response
Possible causes include:
NetworkPolicy;
firewall;
routing;
unavailable destination;
severe packet loss.
These are not absolute rules, but they help determine the next test.
20. Packet capture
When application-level tests are not enough, packet capture can show us what is happening on the wire.
Depending on permissions and environment:
tcpdump -i any -nn \
host 10.244.2.15 \
and port 8080
Look for:
SYN
SYN-ACK
RST
retransmissions
For example:
Client Server
SYN -------------------->
SYN -------------------->
SYN -------------------->
no response
is very different from:
Client Server
SYN -------------------->
<-------------- RST
Packet capture helps determine how far traffic actually travels.
21. Debugging a specific node
Suppose:
kubectl get pods -A -o wide
reveals:
Pods on node-01 -> OK
Pods on node-02 -> FAIL
Our hypothesis shifts away from a generic application failure toward something associated with:
node-02
CNI
routes
interfaces
firewall
With appropriate permissions:
kubectl debug node/node-02 \
-it \
--image=ubuntu
Then inspect:
ip addr
ip route
ip link
For deeper incidents, inspect the relevant CNI agent logs and metrics as well.
22. Metrics
Not every networking incident causes a total outage.
Other symptoms include:
Packet loss
Latency
Connection resets
Retries
Timeouts
Useful signals can include:
request rate
error rate
request duration
TCP resets
packet drops
retransmissions
DNS latency
DNS failures
connection counts
Exact metrics depend on the CNI, ingress/gateway implementation, service mesh, cloud platform, and observability stack.
23. Distributed tracing
Consider:
Frontend
|
v
API
|
v
Payment Service
|
v
Database
The user experiences a request taking:
5.2 seconds
A distributed trace might reveal:
Frontend 20 ms
|
API 35 ms
|
Payment Service 5.1 sec
|
Database 25 ms
Now we know which dependency deserves further investigation.
Traces do not replace network tests, but they are extremely useful for identifying which hop in a distributed request path is slow or failing.
24. Logs
Networking incidents can require correlating logs from several layers:
Application logs
Ingress/Gateway logs
CNI logs
CoreDNS logs
Cloud load balancer logs
Firewall logs
Where possible, correlate using:
timestamp
request ID
trace ID
source IP
destination
status code
This creates a much more complete view than inspecting each component independently.
Complete troubleshooting flow
The process can be summarized as:
Request fails
|
v
Application alive?
|
v
Listening on port?
|
v
Pod IP works?
|
v
Service works?
|
v
EndpointSlices correct?
|
v
DNS works?
|
v
NetworkPolicy allows?
|
v
Cross-node networking works?
|
v
CNI healthy?
|
v
Ingress / Gateway works?
|
v
Load Balancer works?
|
v
Public DNS works?
|
v
TLS works?
|
v
Root cause
Not every incident requires every step.
Each test should eliminate possible causes and narrow the investigation.
Command reference
# Pods and nodes
kubectl get pods -A -o wide
kubectl get nodes -o wide
# Services
kubectl get svc -A
kubectl describe svc <service> -n <namespace>
# EndpointSlices
kubectl get endpointslices \
-n <namespace> \
-l kubernetes.io/service-name=<service>
kubectl get endpointslices \
-n <namespace> \
-l kubernetes.io/service-name=<service> \
-o yaml
# Labels
kubectl get pods \
-n <namespace> \
--show-labels
# DNS
nslookup <service>
nslookup <service>.<namespace>
cat /etc/resolv.conf
# CoreDNS
kubectl get pods \
-n kube-system \
-l k8s-app=kube-dns
kubectl logs \
-n kube-system \
-l k8s-app=kube-dns
# NetworkPolicy
kubectl get networkpolicy -A
kubectl describe networkpolicy \
<policy> \
-n <namespace>
# Ingress
kubectl get ingress -A
kubectl describe ingress \
<ingress> \
-n <namespace>
kubectl get ingressclass
# Gateway API
kubectl get gateway -A
kubectl get httproute -A
# Temporary debug Pod
kubectl run network-debug \
-n <namespace> \
--rm -it \
--restart=Never \
--image=curlimages/curl \
-- sh
# Node debugging
kubectl debug node/<node> \
-it \
--image=ubuntu
# Network tools
curl -v <url>
nslookup <hostname>
dig <hostname>
nc -vz <hostname> <port>
ip addr
ip route
ip link
tcpdump -i any -nn
What not to do
Avoid starting with actions such as:
Service doesn't work
|
v
Restart CoreDNS
or:
502
|
v
Restart Ingress Controller
or:
Timeout
|
v
Disable NetworkPolicy
These changes modify the system without demonstrating the root cause.
A better workflow is:
Symptom
|
v
Test one boundary
|
v
Record result
|
v
Eliminate hypotheses
|
v
Move one layer
|
v
Find failure point
|
v
Fix
|
v
Validate
Conclusion
Kubernetes networking becomes much easier to understand when we stop treating it as a single component.
A request can pass through:
Client
|
DNS
|
Load Balancer
|
Gateway / Ingress
|
Service
|
EndpointSlice
|
Pod network
|
Application
Each layer can be tested independently.
That is why one of the most useful questions during troubleshooting is:
How far do we know the connection works?
If we can prove:
Pod IP -> works
Service IP -> works
Service DNS -> works
Ingress -> fails
we have already eliminated a large part of the system from the investigation.
The objective is to replace trial-and-error troubleshooting with evidence.
In the next article we will investigate Kubernetes health checks in depth:
readiness probes;
liveness probes;
startup probes;
timeouts;
thresholds;
probe-induced restart loops;
external dependencies in health endpoints;
cascading failures caused by incorrectly designed probes.
Comments