Fix CrashLoopBackOff in Kubernetes: root causes and solutions

When a Kubernetes pod enters CrashLoopBackOff, the container starts but then immediately exits. Kubernetes waits, sees the exit, and restarts it again—looping forever. This guide shows you how to find the root cause and fix it.

Symptom

The pod status shows CrashLoopBackOff. The restart count keeps incrementing. Events show messages like:

Name:           example-app-5f7c9d4bfb-7x8kq
Namespace:      default
Status:         Running
Restarts:       47 (2 minutes ago)
Last State:     Terminated
  Reason:       Error
  Exit Code:    1

Or in kubectl get events:

pod-name   CrashLoopBackOff   exit status 1
pod-name   BackOff            Back-off restarting failed container

Why it happens

Ranked by likelihood:

1. Application crashes on startup — Your code has a bug, throws an unhandled exception, or segfaults before it reaches the main server loop.
2. Malformed command or args — The command or args in the pod spec are wrong, missing quotes, or point to a non-existent binary.
3. Missing environment variable, config file, or secret — The app expects a config file or env var that is not mounted or set.
4. Liveness or startup probe fails immediately — Kubernetes kills the pod thinking it is unhealthy, even though it hasn’t finished starting.
5. Out of memory (OOM) — The app allocates too much memory too quickly and the kernel terminates it (exit code 137).
6. Dependency is not ready — The app tries to connect to a database or service that is not yet running and fails hard instead of retrying.
7. Permission denied or filesystem issue — The app cannot write to required directories or read config files due to ownership or permission rules.

Fix

1. Application crashes on startup

Get the actual error message:

# See the last output before the crash
kubectl logs <pod-name> --previous

# Repeat the logs command several times to watch multiple restart cycles
kubectl logs <pod-name> --previous
kubectl logs <pod-name> --previous

If --previous returns nothing (no previous container), the pod crashed so fast that no logs were written. Use an ephemeral debug container to inspect the filesystem:

# Start a debug pod with the same image and mount points
kubectl debug <pod-name> -it --image=<same-base-image>

Inside the debug container, check for:
– Syntax errors in startup scripts: cat /app/start.sh | bash -x
– Missing libraries: ldd /app/binary
– Directory structure: ls -la /app

2. Malformed command or args

Inspect the exact command:

kubectl get pod <pod-name> -o yaml

Look for the command and args fields:

containers:
- name: app
  command: ["/bin/sh"]
  args: ["-c", "node /app/server.js --port=8080"]

Common mistakes:
– A flag and its value split by mistake: args: ["--config", "=/etc/app/config.yaml"] should be args: ["--config=/etc/app/config.yaml"] or args: ["--config", "/etc/app/config.yaml"]
– Wrong path: command: ["/bin/bash"] but the image is based on Alpine (use /bin/sh instead)
– Forgotten slash: command: ["node"] instead of command: ["/usr/bin/node"]

Fix it in your Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: example:latest
        command: ["/bin/sh"]
        args: ["-c", "exec node /app/server.js"]

Then redeploy:

kubectl rollout restart deployment/example-app

3. Missing environment variable, config file, or secret

Check what the app expects:

kubectl logs <pod-name> --previous | grep -i "not found\|undefined\|missing"

Verify that all env vars and mounts are present:

kubectl get pod <pod-name> -o yaml | grep -A 20 "env:\|volumeMounts:"

Add missing config as a ConfigMap:

# Create a ConfigMap from a file
kubectl create configmap app-config --from-file=config.yaml

Mount it in your Deployment:

containers:
- name: app
  volumeMounts:
  - name: config
    mountPath: /etc/app
    readOnly: true
volumes:
- name: config
  configMap:
    name: app-config

For secrets (API keys, passwords), use a Secret:

kubectl create secret generic app-secrets --from-literal=DB_PASSWORD=mypass

Then reference it:

containers:
- name: app
  env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: DB_PASSWORD

Redeploy:

kubectl rollout restart deployment/example-app

4. Liveness or startup probe fails immediately

The probe is killing the pod before it has time to start. Check the current probe:

kubectl get pod <pod-name> -o yaml | grep -A 10 "livenessProbe:\|startupProbe:"

The most common issue: initialDelaySeconds is too short. If your app takes 10 seconds to start and the probe runs after 5 seconds, it will fail and the pod will restart.

Fix it in your Deployment:

containers:
- name: app
  livenessProbe:
    httpGet:
      path: /health
      port: 8080
    initialDelaySeconds: 30        # Give app time to start
    periodSeconds: 10              # Check every 10 seconds
    failureThreshold: 3            # Restart after 3 failures
  startupProbe:
    httpGet:
      path: /health
      port: 8080
    failureThreshold: 30           # Allow up to 300 seconds (30 * 10s) to become healthy
    periodSeconds: 10

For a database that takes time to initialize:

startupProbe:
  exec:
    command: ["pg_isready", "-h", "localhost"]
  failureThreshold: 30
  periodSeconds: 10
livenessProbe:
  exec:
    command: ["pg_isready", "-h", "localhost"]
  initialDelaySeconds: 5
  periodSeconds: 10

Redeploy and monitor:

kubectl rollout restart deployment/example-app
kubectl describe pod <pod-name>

5. Out of memory (OOM)

Exit code 137 with reason OOMKilled means the kernel terminated the process for using too much memory. Check memory usage:

# If metrics-server is installed
kubectl top pod <pod-name>

# Or describe to see if OOMKilled is in the state
kubectl describe pod <pod-name> | grep -A 5 "Last State"

The pod spec likely has a memory limit that is too low. Increase it:

containers:
- name: app
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"        # Increase this
      cpu: "500m"

If you have a JVM app, also set heap limits:

JAVA_OPTS="-Xmx512m -Xms256m"

For Node.js, set the max old space size:

node --max-old-space-size=512 app.js

Apply the updated spec:

kubectl apply -f deployment.yaml
kubectl rollout restart deployment/example-app

6. Dependency is not ready

The app fails hard when it cannot connect to a database, cache, or other service at startup. Use an init container to wait for the dependency:

containers:
- name: app
  image: example:latest
initContainers:
- name: wait-for-db
  image: busybox:1.36
  command: ['sh', '-c', 'until nc -z postgres.default.svc.cluster.local 5432; do echo waiting for postgres; sleep 2; done']

Or add retry logic to your application startup:

for i in {1..30}; do
  if psql -h postgres.default.svc.cluster.local -U user -d mydb -c "SELECT 1"; then
    break
  fi
  echo "Database not ready, retry $i/30"
  sleep 2
done
exec node /app/server.js

Ensure the Service for the dependency exists and is running:

kubectl get svc postgres
kubectl get pod -l app=postgres

7. Permission denied or filesystem issue

The app cannot read or write required files. Check the pod’s running user and file permissions:

kubectl debug <pod-name> -it --image=<same-image>

Inside the debug container:

# What user are you?
id

# Can you read the config file?
ls -la /etc/app/config.yaml
cat /etc/app/config.yaml

# Can you write to the data directory?
touch /data/test.txt
rm /data/test.txt

If permission is denied, fix it in your Dockerfile or pod spec:

FROM alpine:3.20
RUN adduser -D appuser
RUN mkdir -p /app /data && chown -R appuser:appuser /app /data
USER appuser

Or in your Deployment:

containers:
- name: app
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumeMounts:
  - name: data
    mountPath: /data
volumes:
- name: data
  emptyDir: {}

Prevent it

1. Test the container locally first:

   docker run --rm example:latest
   

   Let it run for at least 10 seconds to catch immediate crashes.

2. Set correct resource requests and limits to avoid OOM:

   resources:
     requests:
       memory: "256Mi"
       cpu: "100m"
     limits:
       memory: "512Mi"
       cpu: "500m"
   

3. Use startup probes if your app takes time to start:

   startupProbe:
     httpGet:
      path: /health
      port: 8080
    failureThreshold: 30
    periodSeconds: 10
   

4. Run the app as a non-root user in your Dockerfile.

5. Use init containers to wait for dependencies before starting the main app.

6. Log startup errors to stderr so they appear in kubectl logs --previous.

FAQ

How do I see why a pod is in CrashLoopBackOff?

Run kubectl logs <pod> --previous to see the last crash. Then kubectl describe pod <pod> to see restart count and probe failures. Finally, kubectl get events to see recent warnings.

Does CrashLoopBackOff mean the image pull failed?

No, that would be ImagePullBackOff. CrashLoopBackOff means the container started but exited with a non-zero status. Check kubectl logs --previous to see the actual error.

How long does Kubernetes wait before retrying after a crash?

It uses exponential backoff: 10s, 20s, 40s, 80s, 160s, 300s (capped at 5 min). So after 5 minutes of crashes, retries happen every 5 minutes.

Can a liveness probe cause CrashLoopBackOff?

Yes. If the probe fails and restartPolicy is Always, Kubernetes restarts the pod. Increase initialDelaySeconds to give the app time to start. Show probe config with kubectl get pod -o yaml.