What is the first command to run when a Kubernetes pod is failing?

Run `kubectl describe pod -n `. The Events section at the bottom reveals scheduling failures, image pull errors, liveness probe failures, and OOMKills — covering the majority of pod issues in one command.

How do I see logs from a crashed pod in Kubernetes?

Use `kubectl logs --previous` to fetch logs from the last terminated container. If the pod has multiple containers, add `-c `. For very recent crashes, combine with `--tail=200` to limit output.

How do I debug a Kubernetes node that is NotReady?

SSH onto the node and check `systemctl status kubelet`. Common causes are disk pressure (check with `df -h`), memory pressure (`free -m`), or a crashed container runtime (`systemctl status containerd`). Review node conditions with `kubectl describe node `.

What tools help with Kubernetes debugging beyond kubectl?

Useful tools include `kubectl-debug` (ephemeral debug containers), `stern` (multi-pod log tailing), `k9s` (terminal dashboard), `Prometheus + Grafana` (metrics), and AI-powered tools like KubeIntellect that automate root cause analysis.

How do I debug a Kubernetes service that is not routing traffic?

Verify the service selector matches pod labels with `kubectl get endpoints `. Check that pods are Running and Ready. Use `kubectl exec` to curl the service's ClusterIP directly from inside the cluster. Confirm NetworkPolicy rules aren't blocking traffic.

How do I debug DNS failures in Kubernetes?

Run a debug pod with `kubectl run dnstest --image=busybox:1.28 --rm -it --restart=Never -- nslookup kubernetes.default`. If that fails, check CoreDNS pods are running in kube-system and inspect their logs with `kubectl logs -n kube-system -l k8s-app=kube-dns`.

How do I debug a PersistentVolumeClaim that is stuck in Pending?

Run `kubectl describe pvc ` and look at the Events. Common causes: no PersistentVolume matches the StorageClass and access mode, the storage provisioner is not running, or the requested capacity exceeds what any PV offers. Check `kubectl get storageclass` to verify the provisioner is registered.

What does the kubectl debug command actually do?

`kubectl debug` creates a copy of a pod, or attaches an ephemeral container to a running one, so you can troubleshoot without changing the original workload. It is the supported way to get a shell next to a container that has no shell of its own — distroless and scratch images, for example. Unlike `kubectl exec`, it does not require the target container to ship a debugging toolchain, because the ephemeral container brings its own image.

What is a kubectl debug example for a pod with no shell?

Run `kubectl debug -it --image=busybox:1.28 --target= `. This attaches an ephemeral busybox container that shares the target's process namespace, so you can inspect its filesystem through /proc/1/root and see its processes. Add `--copy-to= ` instead of `--target` when the pod is crash-looping and you need a copy whose command you can override.

How do I use kubectl debug on a node?

Run `kubectl debug node/ -it --image=busybox`. This starts a privileged pod on that node with the host filesystem mounted at /host and the host namespaces joined, which lets you inspect kubelet logs, container-runtime state, disk pressure, and networking from inside the node. Use it when a node is NotReady and SSH access is unavailable.

What is the difference between kubectl describe pod and kubectl logs?

`kubectl describe pod` shows the pod's declared state and — most importantly — the Events at the bottom, which explain scheduling failures, image pull errors, probe failures, and OOMKills. `kubectl logs` shows what the application itself printed. If the container never started, there are no logs to read and describe is the only useful command; if the container started and then misbehaved, logs are where the answer is.

How do I grep kubectl logs for a specific error?

Pipe them: `kubectl logs | grep -i error`. For a crashed previous instance use `kubectl logs --previous | grep -i error`. To search every pod behind a label at once, use `kubectl logs -l app= --all-containers --tail=-1 | grep -i error`. Add `--since=1h` to bound the window so you are not scanning the whole retained buffer.

How do I make kubectl output more verbose for debugging?

Add `-v= ` to any command. `-v=6` shows the HTTP requests kubectl makes to the API server, `-v=8` adds full request and response bodies, and `-v=9` includes curl-equivalent commands. This is how you tell an RBAC denial apart from a networking failure or a malformed request when a command fails for no obvious reason.

Why does kubectl describe pod show no events?

Kubernetes events have a time-to-live — one hour by default in most distributions — and are garbage-collected after it expires. If the Events section reads ` `, the incident is simply older than that window. An empty list is not evidence that nothing went wrong. Use `kubectl get events --sort-by=.lastTimestamp` while the incident is fresh, and fall back to the audit log or your logging backend for anything older.

How do I see logs from an init container?

Name it explicitly: `kubectl logs -c `. Init containers are tracked separately from application containers, so a pod stuck in `Init:0/2` produces nothing from a plain `kubectl logs`. Use `kubectl get pod -o jsonpath='{.status.initContainerStatuses[*].state}'` to see which init container is blocking and why.

How do I debug a Kubernetes service that resolves but refuses connections?

If DNS resolves but connections are refused, the Service has endpoints but nothing is listening on the target port. Confirm with `kubectl get endpointslices -l kubernetes.io/service-name= `, then check that `targetPort` matches the port the container actually binds — a Service pointing at port 8080 while the app listens on 3000 resolves fine and fails on connect.

Kubernetes Debugging Guide: Diagnose Any Cluster Issue

Kubernetes debugging starts with a single command: kubectl describe pod <pod-name> -n <namespace>. The Events section at the bottom covers the majority of pod failures in one read. From there, a proven methodology takes you through logs, exec inspection, resource analysis, and network tracing — in that order.

Kubernetes is resilient by design, but when something breaks, the blast radius can be wide and the signal buried deep in logs, events, and metrics spread across dozens of components. This guide walks through a proven debugging methodology for the most common failure categories, with the exact commands needed at each step.

Start with Cluster-Wide Visibility

Before diving into individual pods, get a bird's-eye view of what's unhealthy:

# See all non-Running pods across every namespace
kubectl get pods -A --field-selector='status.phase!=Running'

# Spot nodes under pressure
kubectl get nodes
kubectl describe nodes | grep -A5 "Conditions:"

# Recent cluster events sorted by time
kubectl get events -A --sort-by='.lastTimestamp' | tail -40

This surfaces the scope of the incident immediately. A single bad node failing its health checks looks very different from a misconfigured deployment rolling out CrashLoopBackOff pods on every node.

Pod-Level Debugging Workflow

Once you've identified a problematic pod, follow this sequence:

1. Read the pod description

kubectl describe pod <pod-name> -n <namespace>

The Events section at the bottom is the most informative part. Look for:

FailedScheduling — insufficient CPU/memory or node affinity conflicts
ErrImagePull / ImagePullBackOff — bad image tag or missing registry credentials
Liveness probe failed — app not responding on the expected port/path
OOMKilled — container exceeded its memory limit
FailedMount — PVC not bound or secret key missing

2. Fetch logs (including from previous crash)

# Current container logs
kubectl logs <pod-name> -n <namespace> --tail=100

# Logs from the last crashed container
kubectl logs <pod-name> -n <namespace> --previous --tail=200

# Follow logs in real time
kubectl logs <pod-name> -n <namespace> -f

3. Exec in for live inspection

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

# Inside the container:
env | grep -i db       # check env vars
curl localhost:8080/health
cat /etc/config/app.yaml

If the container image has no shell, launch an ephemeral debug container:

kubectl debug -it <pod-name> --image=busybox --target=<container-name>

Reading kubectl describe Properly

kubectl describe is the command most engineers run first and read least carefully. It prints the object's resolved state followed by an Events feed, and the two answer different questions: the state tells you what Kubernetes intends, the events tell you what actually happened.

Describing a pod and its events

# Full description — state first, Events last
kubectl describe pod <pod-name> -n <namespace>

# Just the events, sorted chronologically across the namespace
kubectl get events -n <namespace> --sort-by=.lastTimestamp

# Events for one object only
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

Events are stored with a TTL — one hour by default in most distributions. If describe shows <none> under Events, the incident is older than that window and you need the audit log or your logging backend instead. An empty Events list is not evidence that nothing went wrong.

Reading a specific container inside the pod

The describe output lists every container separately, each with its own State, Last State, Ready flag, and Restart Count. Last State is where an exit code from a previous crash appears — the single most useful field when a pod is restarting in a loop.

# Exit code and reason for the previous run of a specific container
kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason} {.status.containerStatuses[*].lastState.terminated.exitCode}'

# Init containers are tracked separately from app containers
kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.status.initContainerStatuses[*].state}'

# Logs from an init container that already finished
kubectl logs <pod-name> -n <namespace> -c <init-container-name>

A pod stuck in Init:0/2 is waiting on an init container, and kubectl logs without -c will not show you why. Name the init container explicitly.

Describing nodes

kubectl describe node <node-name>

# Node-level events only
kubectl get events --field-selector involvedObject.kind=Node

On a node, read the Conditions block before anything else. MemoryPressure, DiskPressure, and PIDPressure being True explain evictions that look inexplicable at the pod level. The Allocated resources block shows requests as a percentage of capacity — this is what the scheduler reasons about, and a node can be 100% requested while sitting near-idle on actual usage, which is the usual cause of a FailedScheduling you cannot otherwise explain.

Capturing everything at once

# Dump cluster-wide diagnostic state to a directory
kubectl cluster-info dump --all-namespaces --output-directory=./cluster-dump

# Node logs via the kubelet (Kubernetes 1.27+)
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/logs/"

cluster-info dump is the right command when you need to hand an incident to someone else, or when events are about to expire and you want them preserved. It writes pod specs, logs, and events for every namespace, so send it to a directory rather than to your terminal.

Debugging with Ephemeral Containers

Distroless and scratch-based images have no shell, making traditional exec impossible. Ephemeral debug containers solve this without modifying the pod spec. According to the official Kubernetes documentation on debugging running pods, ephemeral containers are available as a stable feature since Kubernetes 1.25:

# Attach a debug container to a running pod
kubectl debug -it <pod-name> -n <namespace>   --image=nicolaka/netshoot   --target=<container-name>

# nicolaka/netshoot includes: curl, dig, nslookup, tcpdump,
# netstat, ping, traceroute, ss, iperf3, and more

The debug container shares the target container's network namespace, so you can inspect its open ports, run DNS lookups, and trace network connections without any changes to the application image.

# Copy a running pod to a new pod with debug tools added
kubectl debug <pod-name> -it --copy-to=debug-pod   --image=busybox --share-processes

Diagnosing Resource Exhaustion

Resource issues are among the most common — and most misdiagnosed — Kubernetes failures. A pod showing Pending may simply have requests that no node can satisfy. Pods without explicit resource requests are first to be evicted under node memory pressure, per the Kubernetes node-pressure eviction documentation:

# Visualize allocatable vs. requested per node
kubectl describe nodes | grep -A10 "Allocated resources"

# Top consumers right now
kubectl top pods -A --sort-by=memory
kubectl top nodes

# Check namespace-level resource quotas
kubectl describe resourcequota -n <namespace>

If a node shows MemoryPressure=True, the kubelet will start evicting pods — starting with those that exceed their memory requests. Always set both requests and limits on critical workloads.

# Find pods with no resource requests set (eviction risk)
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.spec.containers[].resources.requests == null) |
  [.metadata.namespace, .metadata.name] | @tsv'

Network Debugging

Services not routing traffic is a classic Kubernetes head-scratcher. The most common cause is a label selector mismatch:

# Check if any pods are selected
kubectl get endpoints <service-name> -n <namespace>
# Empty "Endpoints: <none>" means the selector matches nothing

# Compare service selector vs pod labels
kubectl get svc <service-name> -o jsonpath='{.spec.selector}'
kubectl get pods -n <namespace> --show-labels

# Test connectivity from inside the cluster
kubectl exec -it <any-pod> -n <namespace> --   curl -v http://<service>.<namespace>.svc.cluster.local:<port>/health

If endpoints look correct but traffic still fails, check NetworkPolicy objects — a blanket deny-all policy will silently drop packets with no error message on either side.

# List all NetworkPolicies in a namespace
kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <name> -n <namespace>

# Check if a specific port is blocked
kubectl exec -it <pod> -- nc -zv <target-ip> <port>

DNS Debugging

DNS failures inside a cluster often look like connection timeouts or connection refused errors, making them hard to distinguish from routing failures. Always test DNS explicitly first:

# Run a one-shot DNS test pod
kubectl run dnstest --image=busybox:1.28 --rm -it   --restart=Never -- nslookup kubernetes.default

# Test a specific service DNS name
kubectl run dnstest --image=busybox:1.28 --rm -it   --restart=Never -- nslookup <service>.<namespace>.svc.cluster.local

# Check CoreDNS is running
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Read CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Common DNS failure causes: CoreDNS pods in CrashLoopBackOff, the pod's dnsPolicy set to None without a dnsConfig, or a NetworkPolicy blocking UDP port 53 from pods to CoreDNS.

# Verify DNS config inside a pod
kubectl exec -it <pod-name> -- cat /etc/resolv.conf
# Should show: nameserver 10.96.0.10 (or cluster DNS IP)
# and: search <namespace>.svc.cluster.local svc.cluster.local cluster.local

Storage and PersistentVolume Debugging

Storage issues manifest as pods stuck in Pending (PVC not bound) or as FailedMount events (volume exists but can't be attached):

# Check PVC status
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

# Find PVs and their binding status
kubectl get pv
kubectl describe pv <pv-name>

# Check storage class provisioner
kubectl get storageclass
kubectl describe storageclass <name>

If a PVC is stuck in Pending with no events, the provisioner is likely not running. Check for the provisioner's pod in the relevant namespace. For ReadWriteMany access mode, not all storage drivers support it — check the driver documentation.

# PVC stuck in Terminating? Force remove the finalizer
kubectl patch pvc <name> -n <namespace>   -p '{"metadata":{"finalizers":null}}'

Debugging Multi-Container Pods

Pods with multiple containers (sidecars, init containers) require specifying the container name in most kubectl commands:

# List containers in a pod
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'

# Logs from a specific container
kubectl logs <pod-name> -c <container-name> --previous

# Exec into a specific container
kubectl exec -it <pod-name> -c <sidecar-name> -- /bin/sh

# Check init container status
kubectl describe pod <pod-name> | grep -A20 "Init Containers:"

# Init container logs
kubectl logs <pod-name> -c <init-container-name>

Init container failures: If an init container exits with a non-zero code, the main containers never start. The pod status shows Init:CrashLoopBackOff or Init:Error. Always check init container logs first when the main app never appears to start.

Node-Level Debugging

When a node goes NotReady or pods on a specific node all fail simultaneously, investigate at the node level:

# Check node conditions in detail
kubectl describe node <node-name>

# SSH to the node (or use node-shell plugin)
kubectl node-shell <node-name>

# On the node — check kubelet
systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager

# Check container runtime
systemctl status containerd
crictl ps  # list running containers

# Disk usage
df -h
du -sh /var/log/pods/* | sort -hr | head -20

# Memory
free -m
cat /proc/meminfo

Disk pressure is particularly sneaky — it fills up with old pod logs and evicted image layers. Configure log rotation and set imagePullPolicy: Always only where needed.

Debugging StatefulSets

StatefulSets have ordered, sequential pod creation — a stuck pod at ordinal N blocks all pods from ordinal N+1 onward:

# Check StatefulSet status
kubectl describe statefulset <name> -n <namespace>

# Individual pod status (ordered 0, 1, 2...)
kubectl get pods -n <namespace> -l app=<label> --sort-by='.metadata.name'

# PVC per pod (each pod gets its own)
kubectl get pvc -n <namespace> -l app=<label>

# If a pod is stuck, check its specific PVC
kubectl describe pvc <pvc-name-for-pod-0>

Reading the Audit Trail

For intermittent failures or issues that resolved themselves, cluster events expire after one hour by default (controlled by --event-ttl on the API server, per the kube-apiserver reference). Persistent event storage via kube-state-metrics + Prometheus or a dedicated backend like Loki is essential for post-incident analysis.

# Events sorted by time, including older ones
kubectl get events -n <namespace>   --sort-by='.lastTimestamp'   --field-selector='type=Warning'

# Watch events in real time during a deploy
kubectl get events -n <namespace> -w

Pro tip: Save your debugging session with kubectl describe pod > incident.txt before the pod is rescheduled and its history disappears. Events are gone after 1 hour — export them to persistent storage if you need them for a postmortem.

Debugging Stuck Rollouts

A deployment rollout that never completes blocks future deployments. The most common causes are readiness probe failures on the new version or insufficient cluster capacity:

# Check rollout status
kubectl rollout status deployment/<name> -n <namespace>

# See what the new pods are doing
kubectl get pods -n <namespace> | grep -v Running
kubectl describe pod <new-pod-name>

# Check rollout history
kubectl rollout history deployment/<name>

# Emergency rollback
kubectl rollout undo deployment/<name> -n <namespace>

# Pause a rollout to investigate
kubectl rollout pause deployment/<name>
kubectl rollout resume deployment/<name>

When Manual Debugging Takes Too Long

Running through these steps manually under production pressure is slow and error-prone. KubeIntellect automates the entire sequence — it collects describe output, logs, metrics, and events, correlates them across the timeline, and surfaces the root cause with a one-line fix in under two seconds. Ask it in plain English: "Why is my api pod crashing in production?"

For a structured approach to finding the underlying cause of any incident, see the Kubernetes Root Cause Analysis guide. For a quick-reference of copy-paste commands by failure type, see the K8s Troubleshooting Playbook.

Kubernetes Debugging Guide: Diagnose Any Cluster Issue

Start with Cluster-Wide Visibility

Pod-Level Debugging Workflow

1. Read the pod description

2. Fetch logs (including from previous crash)

3. Exec in for live inspection

Reading kubectl describe Properly

Describing a pod and its events

Reading a specific container inside the pod

Describing nodes

Capturing everything at once

Debugging with Ephemeral Containers

Diagnosing Resource Exhaustion

Network Debugging

DNS Debugging

Storage and PersistentVolume Debugging

Debugging Multi-Container Pods

Node-Level Debugging

Debugging StatefulSets

Reading the Audit Trail

Debugging Stuck Rollouts

When Manual Debugging Takes Too Long

Let KubeIntellect debug this automatically

Frequently Asked Questions

Start with Cluster-Wide Visibility

Pod-Level Debugging Workflow

1. Read the pod description

2. Fetch logs (including from previous crash)

3. Exec in for live inspection

Reading kubectl describe Properly

Describing a pod and its events

Reading a specific container inside the pod

Describing nodes

Capturing everything at once

Debugging with Ephemeral Containers

Diagnosing Resource Exhaustion

Network Debugging

DNS Debugging

Storage and PersistentVolume Debugging

Debugging Multi-Container Pods

Node-Level Debugging

Debugging StatefulSets

Reading the Audit Trail

Debugging Stuck Rollouts

When Manual Debugging Takes Too Long

Let KubeIntellect debug this automatically

Frequently Asked Questions

Related Kubernetes Guides