Kubernetes Troubleshooting: Networking, Services, DNS, NetworkPolicy, Ingress y CNI

Learn how to troubleshoot Kubernetes networking from Pods and Services through EndpointSlices, DNS, NetworkPolicy, Ingress, Gateway API, LoadBalancers, and CNI.

This post is also available in: Español

Kubernetes networking problems can be difficult to diagnose because the same application symptom can originate from many different layers.

An application might report:

Connection refused
Connection timed out
No route to host
Name or service not known
502 Bad Gateway
503 Service Unavailable

but the actual problem could be somewhere along this path:

Application
|
Pod
|
Pod Network
|
Service
|
EndpointSlice
|
NetworkPolicy
|
Ingress / Gateway
|
Load Balancer
|
DNS
|
External Network

The most effective strategy is not to start changing configurations.

Instead, follow the connection one layer at a time until we can identify exactly where communication stops working.

In this third article of our Kubernetes Troubleshooting series, we will investigate:

  • Pod-to-Pod connectivity;

  • Services;

  • EndpointSlices;

  • DNS;

  • ports and targetPort;

  • NetworkPolicies;

  • Ingress;

  • Gateway API;

  • LoadBalancers;

  • CNI;

  • outbound connectivity.


First, understand the Kubernetes network model

Before troubleshooting, we need to understand a few expectations of Kubernetes networking.

Each Pod receives its own IP address inside the cluster.

Under the Kubernetes network model, and unless intentional network segmentation prevents it, Pods should be able to communicate with other Pods even when those Pods are running on different nodes.

Containers within the same Pod share a network namespace and can communicate over:

localhost

Services add another abstraction:

Client Pod
|
v
Service
|
v
EndpointSlice
|
+------ Pod A
|
+------ Pod B
|
+------ Pod C

The Service provides a stable access point while backend Pods can be created, deleted, or receive different IP addresses.


Our troubleshooting approach

For networking, we will use one simple rule:

Test the closest destination first, then move outward one layer at a time.

For example, if:

https://api.example.com

does not work, we should not immediately start changing public DNS or restarting the Ingress Controller.

Instead test:

Application listening?
|
v
Pod IP reachable?
|
v
Service reachable?
|
v
EndpointSlice correct?
|
v
DNS working?
|
v
NetworkPolicy allows traffic?
|
v
Ingress / Gateway route?
|
v
Load Balancer?
|
v
Public DNS?

Every successful test eliminates possible causes.


1. Start with the Pods

Suppose we have:

frontend
|
v
api-service
|
v
api Pods

and the frontend reports:

connection refused

First inspect the backend Pods:

kubectl get pods \
-n production \
-o wide

Look at:

  • STATUS;

  • READY;

  • IP;

  • NODE;

  • restart count.

For example:

NAME READY STATUS IP
api-7cf8f76bcd-jx9pq 1/1 Running 10.244.1.23
api-7cf8f76bcd-qt8fz 1/1 Running 10.244.2.15

Running does not guarantee that the application is accepting connections, but it gives us a starting point.


2. Is the application actually listening?

A surprisingly common Kubernetes networking problem is not a Kubernetes networking problem at all.

The application may not be listening on the expected port.

If the container image includes the necessary tools:

kubectl exec \
-n production \
<pod> -- \
ss -lntp

We can also test the application locally:

kubectl exec \
-n production \
<pod> -- \
curl -v http://127.0.0.1:8080/health

If this fails:

Pod
|
X
Application

there is little value in troubleshooting the Service yet.

First determine why the process is not accepting connections.


A classic problem: listening only on localhost

Suppose the application binds to:

127.0.0.1:8080

Inside the container:

curl http://127.0.0.1:8080

may work.

But other Pods reach the application using its Pod IP:

10.244.1.23:8080

If the process only listens on loopback, that connection can fail.

Applications running in Kubernetes commonly need to listen on:

0.0.0.0

or another appropriate Pod interface.


3. Test the Pod IP directly

From another Pod, test the backend address.

A temporary debugging Pod is useful:

kubectl run network-debug \
-n production \
--rm -it \
--restart=Never \
--image=curlimages/curl \
-- sh

From inside it:

curl -v http://10.244.1.23:8080

If this works:

Debug Pod
|
v
Pod IP
|
v
Application
|
OK

we have verified:

  • basic Pod networking;

  • the destination port;

  • the application.

Now we can move to the Service layer.

If it fails, investigate:

Application
Pod networking
NetworkPolicy
CNI
Node networking

4. Investigate the Service

List Services:

kubectl get svc \
-n production

Then inspect the affected Service:

kubectl describe svc api-service \
-n production

Pay attention to:

Selector
Type
IP Families
ClusterIP
Port
TargetPort

A Service object can exist correctly in the Kubernetes API while still not providing working connectivity.


Service port vs targetPort

Consider:

ports:
- port: 80
targetPort: 8080

The flow is:

Client
|
| :80
v
Service
|
| :8080
v
Pod

port is the Service port exposed to clients.

targetPort identifies the backend port.

If the application listens on:

8080

but the Service uses:

targetPort: 3000

DNS can work and the Service can exist, but traffic will not reach the expected application port.


5. EndpointSlices: does the Service have backends?

This is one of the most important checks.

Modern Kubernetes troubleshooting should primarily inspect EndpointSlices:

kubectl get endpointslices \
-n production \
-l kubernetes.io/service-name=api-service

For more detail:

kubectl describe endpointslice \
-n production \
-l kubernetes.io/service-name=api-service

EndpointSlice is the API Kubernetes uses to efficiently represent Service backends.

Conceptually:

Service
|
v
EndpointSlice
|
+--- 10.244.1.23:8080
|
+--- 10.244.2.15:8080

If endpoints exist, we can test those IP addresses directly.


What if there are no endpoints?

Suppose we have:

Service
|
v
EndpointSlice
|
X
No backend Pods

One of the first things to inspect is the Service selector.

kubectl get svc api-service \
-n production \
-o yaml

For example:

selector:
app: api

Now inspect Pod labels:

kubectl get pods \
-n production \
--show-labels

If the Pods have:

app=backend-api

but the Service selects:

app=api

there is no match.

The Service will not have the expected Pod backends.


Readiness matters too

Suppose the labels match correctly.

But:

kubectl get pods -n production

shows:

NAME READY
api-abc 0/1
api-def 0/1

A failing readiness probe can prevent those Pods from being considered ready endpoints for normal Service traffic.

Check:

kubectl describe pod api-abc \
-n production

Then inspect EndpointSlice conditions:

kubectl get endpointslices \
-n production \
-l kubernetes.io/service-name=api-service \
-o yaml

EndpointSlices can expose conditions such as:

ready
serving
terminating

These help consumers distinguish whether an endpoint is ready to receive traffic and whether it is terminating.


6. Test the Service from inside the cluster

Once backend endpoints are confirmed:

kubectl run network-debug \
-n production \
--rm -it \
--restart=Never \
--image=curlimages/curl \
-- sh

Then:

curl -v http://api-service

or:

curl -v http://api-service:80

We can also test the ClusterIP directly:

curl -v http://10.96.120.15:80

This creates a useful distinction.


Pod IP works, Service IP fails

Client
|
+---- Pod IP --------> OK
|
+---- Service IP ----> FAIL

Now focus on:

Service configuration
EndpointSlice
Service proxy implementation
Node networking

Depending on the cluster networking architecture, Service traffic may be implemented by kube-proxy or another dataplane implementation.


Service IP works, Service DNS fails

Service ClusterIP
|
OK
Service name
|
FAIL

That points strongly toward DNS.


7. DNS troubleshooting

Kubernetes provides DNS-based Service discovery.

From a Pod in the same namespace we can normally use:

api-service

From another namespace:

api-service.production

or a fully qualified service name:

api-service.production.svc.cluster.local

The cluster domain is configurable, so cluster.local is common but should not be treated as universal.


Test name resolution

From a Pod with DNS tools:

nslookup api-service

Then:

nslookup api-service.production

And:

nslookup api-service.production.svc.cluster.local

Also inspect:

cat /etc/resolv.conf

A typical configuration might include:

search production.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

This helps explain why short names can resolve within one namespace but not another.


Check CoreDNS

If DNS is failing across several workloads:

kubectl get pods \
-n kube-system \
-l k8s-app=kube-dns

Inspect the DNS Service:

kubectl get svc \
-n kube-system

And logs:

kubectl logs \
-n kube-system \
-l k8s-app=kube-dns

A production cluster can have several replicas or containers involved, so during an incident it is useful to identify exactly which instances are processing queries.


An important distinction

Consider:

nslookup api-service
|
FAIL
nslookup kubernetes.default
|
FAIL

This suggests a broader DNS problem.

But if:

nslookup kubernetes.default
|
OK
nslookup api-service
|
FAIL

investigate:

Service name
Namespace
Service existence
DNS record creation

before deciding that CoreDNS itself is completely unavailable.


8. NetworkPolicy

Another common scenario is:

Pod A -> Pod B

worked before a networking policy change.

Start with:

kubectl get networkpolicy -A

Then:

kubectl describe networkpolicy \
<policy> \
-n production

Also inspect labels:

kubectl get pods \
-n production \
--show-labels

because NetworkPolicy selects Pods using labels.


NetworkPolicies are additive

This behavior is important.

When multiple NetworkPolicies select the same Pod, they are not processed as ordered firewall rules where the first match wins.

Allowed traffic from applicable policies is combined.

For Pod-to-Pod communication:

Source Pod
|
| Egress must allow
v
Network
|
| Ingress must allow
v
Destination Pod

When both sides are isolated, the source egress policy and destination ingress policy must both permit the connection.


Test a NetworkPolicy hypothesis

From the source Pod:

kubectl exec \
-n frontend \
<frontend-pod> -- \
curl -v --connect-timeout 5 \
http://api-service.production:8080

Or test TCP connectivity:

nc -vz \
api-service.production \
8080

If:

DNS resolves
|
v
Correct destination
|
v
Connection timeout

NetworkPolicy, routing, or firewall filtering become stronger hypotheses.


Remember the CNI implementation

Creating a NetworkPolicy object does not automatically mean a cluster can enforce it.

The cluster networking implementation must support NetworkPolicy enforcement.

Therefore:

NetworkPolicy exists
|
X
does not necessarily mean
traffic is being enforced

Actual behavior depends on the networking plugin.


9. Same-node communication works, cross-node communication fails

This is a valuable signal for CNI or infrastructure problems.

Suppose:

Node A
|
+-- Pod A
|
| works
v
+-- Pod B

but:

Node A Node B
| |
Pod A -----------------> Pod C
FAIL

That significantly changes the investigation.

Check Pod placement:

kubectl get pods \
-A \
-o wide

Then nodes:

kubectl get nodes -o wide

Potential areas include:

  • CNI agents;

  • routing;

  • encapsulation;

  • firewall;

  • MTU;

  • node interfaces;

  • IP address management.


Identify the networking stack

Different Kubernetes distributions use different networking components.

Start with:

kubectl get pods \
-n kube-system \
-o wide

You might find implementations such as:

Calico
Cilium
Flannel
Antrea
cloud-provider-specific networking

Each implementation has its own commands, metrics, and architecture.

Do not start using Calico- or Cilium-specific troubleshooting commands until you know which network implementation the cluster actually uses.


10. CNI and IP allocation failures

If newly created Pods cannot obtain working networking, Events can provide useful clues.

kubectl describe pod <pod> \
-n production

Look for messages related to:

sandbox creation
network setup
CNI
IP allocation

Also review cluster Events:

kubectl get events -A \
--sort-by='.lastTimestamp'

This can reveal whether several Pods or nodes began failing at approximately the same time.


11. MTU: the problem that can look random

Some networking failures look like:

Small request
|
OK
Large request
|
timeout

or:

TCP connection
|
established
|
TLS handshake / large response
|
FAIL

One possible cause is an MTU mismatch.

This can appear in networks using:

  • overlays;

  • VXLAN;

  • VPNs;

  • cloud networking;

  • tunnels.

Inspect interfaces:

ip link

and routing:

ip route

MTU changes should only be made with an understanding of the CNI and underlying network.

Changing MTU values through trial and error can introduce additional failures.


12. Ingress troubleshooting

Suppose:

curl http://api-service.production

works inside the cluster.

But:

https://api.example.com

does not work externally.

That tells us:

Pod OK
Service OK
DNS OK
|
v
Problem is farther upstream

Now investigate Ingress.

kubectl get ingress -A

Then:

kubectl describe ingress <name> \
-n production

Inspect:

IngressClass
Host
Path
Backend Service
Backend Port
TLS
Events
Address

Ingress requires a controller

Creating:

kind: Ingress

does not by itself implement the traffic routing.

An Ingress Controller must watch those resources and configure the corresponding data plane.

Check:

kubectl get ingressclass

Then identify the implementation used by the cluster.

Inspect its Pods and logs according to its namespace:

kubectl get pods \
-n <ingress-controller-namespace>
kubectl logs \
-n <ingress-controller-namespace> \
<controller-pod>

Do not assume that every Kubernetes cluster uses ingress-nginx.


502 vs 503

The exact meaning depends on the proxy or controller implementation, but these status codes can guide the investigation.

For example:

External Client
|
v
Ingress Controller
|
+---- backend unavailable
|
+---- backend connection failure
|
+---- timeout

Always correlate:

  • controller logs;

  • EndpointSlices;

  • application logs;

  • latency metrics.

Do not diagnose solely from the HTTP status code.


13. Gateway API

For new designs, Gateway API should also be considered.

The Kubernetes project recommends Gateway rather than extending the older Ingress API.

Ingress remains supported, but its API is frozen.

A typical Gateway API traffic path might look like:

Client
|
Gateway
|
HTTPRoute
|
Service
|
EndpointSlice
|
Pods

Depending on the installed API and implementation:

kubectl get gateway -A
kubectl get httproute -A

Then:

kubectl describe gateway <gateway> \
-n <namespace>
kubectl describe httproute <route> \
-n <namespace>

Gateway API also requires an appropriate implementation/controller. Kubernetes does not automatically provide a Gateway dataplane simply because Gateway resources exist.


14. LoadBalancer remains Pending

Another common problem is:

kubectl get svc

showing:

TYPE EXTERNAL-IP
LoadBalancer <pending>

Ask:

Service type LoadBalancer
|
v
Who provides the load balancer?

In a cloud environment, this can involve:

  • cloud controller;

  • IAM or permissions;

  • network configuration;

  • quota;

  • subnets;

  • load balancer controllers.

In bare-metal and local environments, an additional load-balancer implementation may be required.

Kubernetes defines the LoadBalancer abstraction, but the surrounding infrastructure must fulfill it.


15. Public DNS

Suppose the load balancer works:

Load Balancer
|
OK

but:

api.example.com
|
FAIL

Check public DNS:

dig api.example.com

or:

nslookup api.example.com

Compare the result with the expected load balancer address.

A particularly useful test is:

curl -v \
--resolve api.example.com:443:<IP> \
https://api.example.com

This allows us to test:

HTTP
TLS
Host-based routing

against a specific IP without relying on normal DNS resolution.


16. TLS

A networking problem can actually be a TLS problem.

Start with:

curl -vk https://api.example.com

For more detail:

openssl s_client \
-connect api.example.com:443 \
-servername api.example.com

Inspect:

certificate chain
expiration
SAN
issuer
SNI
hostname

If Kubernetes stores the certificate in a Secret:

kubectl get secret <tls-secret> \
-n production

and inspect the relevant Ingress or Gateway resource to confirm which certificate should be presented.


17. Pods cannot reach the Internet

Now investigate the opposite direction:

Pod
|
v
Internet
|
X

Start from inside the Pod:

curl -v https://example.com

Then separate DNS from connectivity:

nslookup example.com

If DNS succeeds but the connection fails:

DNS
|
OK
|
v
TCP connection
|
FAIL

investigate:

  • NetworkPolicy egress;

  • routing;

  • NAT;

  • firewall;

  • proxy;

  • cloud network controls;

  • CNI.


DNS vs egress

For example:

nslookup example.com

works.

But:

curl https://example.com

times out.

DNS resolution is working, so restarting CoreDNS is unlikely to address the actual problem.

If:

nslookup example.com
|
FAIL

then determine whether the Pod can reach its configured DNS server before moving to other networking layers.


18. Database connection timeout

The same methodology works for database connectivity.

Suppose an application reports:

connection to database timed out

Break the connection into stages:

Application
|
v
DNS resolution
|
v
TCP connection
|
v
TLS
|
v
Authentication
|
v
Database

DNS:

nslookup database.example.internal

TCP:

nc -vz \
database.example.internal \
5432

If the connection succeeds, we have already demonstrated that:

DNS + routing + TCP path

work to some degree.

We can then investigate:

  • TLS;

  • credentials;

  • database availability;

  • connection limits;

  • application connection pools.

Good networking troubleshooting is also about proving which layers are not broken.


19. Connection refused vs timeout

These symptoms can provide useful clues.

Connection refused

Client
|
v
Destination reached
|
v
Nothing accepting connection

Possible causes include:

  • process not listening;

  • wrong port;

  • proxy actively rejecting the connection.

Timeout

Client
|
v
Packets sent
|
?
|
No response

Possible causes include:

  • NetworkPolicy;

  • firewall;

  • routing;

  • unavailable destination;

  • severe packet loss.

These are not absolute rules, but they help determine the next test.


20. Packet capture

When application-level tests are not enough, packet capture can show us what is happening on the wire.

Depending on permissions and environment:

tcpdump -i any -nn \
host 10.244.2.15 \
and port 8080

Look for:

SYN
SYN-ACK
RST
retransmissions

For example:

Client Server
SYN -------------------->
SYN -------------------->
SYN -------------------->
no response

is very different from:

Client Server
SYN -------------------->
<-------------- RST

Packet capture helps determine how far traffic actually travels.


21. Debugging a specific node

Suppose:

kubectl get pods -A -o wide

reveals:

Pods on node-01 -> OK
Pods on node-02 -> FAIL

Our hypothesis shifts away from a generic application failure toward something associated with:

node-02
CNI
routes
interfaces
firewall

With appropriate permissions:

kubectl debug node/node-02 \
-it \
--image=ubuntu

Then inspect:

ip addr
ip route
ip link

For deeper incidents, inspect the relevant CNI agent logs and metrics as well.


22. Metrics

Not every networking incident causes a total outage.

Other symptoms include:

Packet loss
Latency
Connection resets
Retries
Timeouts

Useful signals can include:

request rate
error rate
request duration
TCP resets
packet drops
retransmissions
DNS latency
DNS failures
connection counts

Exact metrics depend on the CNI, ingress/gateway implementation, service mesh, cloud platform, and observability stack.


23. Distributed tracing

Consider:

Frontend
|
v
API
|
v
Payment Service
|
v
Database

The user experiences a request taking:

5.2 seconds

A distributed trace might reveal:

Frontend 20 ms
|
API 35 ms
|
Payment Service 5.1 sec
|
Database 25 ms

Now we know which dependency deserves further investigation.

Traces do not replace network tests, but they are extremely useful for identifying which hop in a distributed request path is slow or failing.


24. Logs

Networking incidents can require correlating logs from several layers:

Application logs
Ingress/Gateway logs
CNI logs
CoreDNS logs
Cloud load balancer logs
Firewall logs

Where possible, correlate using:

timestamp
request ID
trace ID
source IP
destination
status code

This creates a much more complete view than inspecting each component independently.


Complete troubleshooting flow

The process can be summarized as:

Request fails
|
v
Application alive?
|
v
Listening on port?
|
v
Pod IP works?
|
v
Service works?
|
v
EndpointSlices correct?
|
v
DNS works?
|
v
NetworkPolicy allows?
|
v
Cross-node networking works?
|
v
CNI healthy?
|
v
Ingress / Gateway works?
|
v
Load Balancer works?
|
v
Public DNS works?
|
v
TLS works?
|
v
Root cause

Not every incident requires every step.

Each test should eliminate possible causes and narrow the investigation.


Command reference

# Pods and nodes
kubectl get pods -A -o wide
kubectl get nodes -o wide
# Services
kubectl get svc -A
kubectl describe svc <service> -n <namespace>
# EndpointSlices
kubectl get endpointslices \
-n <namespace> \
-l kubernetes.io/service-name=<service>
kubectl get endpointslices \
-n <namespace> \
-l kubernetes.io/service-name=<service> \
-o yaml
# Labels
kubectl get pods \
-n <namespace> \
--show-labels
# DNS
nslookup <service>
nslookup <service>.<namespace>
cat /etc/resolv.conf
# CoreDNS
kubectl get pods \
-n kube-system \
-l k8s-app=kube-dns
kubectl logs \
-n kube-system \
-l k8s-app=kube-dns
# NetworkPolicy
kubectl get networkpolicy -A
kubectl describe networkpolicy \
<policy> \
-n <namespace>
# Ingress
kubectl get ingress -A
kubectl describe ingress \
<ingress> \
-n <namespace>
kubectl get ingressclass
# Gateway API
kubectl get gateway -A
kubectl get httproute -A
# Temporary debug Pod
kubectl run network-debug \
-n <namespace> \
--rm -it \
--restart=Never \
--image=curlimages/curl \
-- sh
# Node debugging
kubectl debug node/<node> \
-it \
--image=ubuntu
# Network tools
curl -v <url>
nslookup <hostname>
dig <hostname>
nc -vz <hostname> <port>
ip addr
ip route
ip link
tcpdump -i any -nn

What not to do

Avoid starting with actions such as:

Service doesn't work
|
v
Restart CoreDNS

or:

502
|
v
Restart Ingress Controller

or:

Timeout
|
v
Disable NetworkPolicy

These changes modify the system without demonstrating the root cause.

A better workflow is:

Symptom
|
v
Test one boundary
|
v
Record result
|
v
Eliminate hypotheses
|
v
Move one layer
|
v
Find failure point
|
v
Fix
|
v
Validate

Conclusion

Kubernetes networking becomes much easier to understand when we stop treating it as a single component.

A request can pass through:

Client
|
DNS
|
Load Balancer
|
Gateway / Ingress
|
Service
|
EndpointSlice
|
Pod network
|
Application

Each layer can be tested independently.

That is why one of the most useful questions during troubleshooting is:

How far do we know the connection works?

If we can prove:

Pod IP -> works
Service IP -> works
Service DNS -> works
Ingress -> fails

we have already eliminated a large part of the system from the investigation.

The objective is to replace trial-and-error troubleshooting with evidence.

In the next article we will investigate Kubernetes health checks in depth:

  • readiness probes;

  • liveness probes;

  • startup probes;

  • timeouts;

  • thresholds;

  • probe-induced restart loops;

  • external dependencies in health endpoints;

  • cascading failures caused by incorrectly designed probes.

Comments