Fix OOMKilled Pods in Kubernetes: diagnose and size memory limits

OOMKilled means a pod exceeded its memory limit and the kernel terminated it (exit code 137). This happens because either your memory limit is too low or your app has a memory leak. This guide shows you how to diagnose, size limits correctly, and understand QoS classes.

Symptom

The pod restarts repeatedly and kubectl describe pod shows:

Name:           example-app-5f7c9d4bfb-7x8kq
Namespace:      default
Status:         Running
Restarts:       12
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Signal:       9
  Last Probe:   2m30s ago
  Reason:       None

Alternatively, the pod shows no OOMKilled reason but consistently exits with code 137:

Exit Code:    137
Signal:       9

In kubectl get events:

pod-name   OOMKilled   The container was OOMKilled

Memory usage may be visible in kubectl top pod:

NAME                       CPU(cores)   MEMORY(bytes)
example-app-5f7c9d4bfb    100m         1536Mi    # At or near the limit

Why it happens

Ranked by likelihood:

1. Memory limit is too low for the app’s normal workload — The app needs more memory to store data, cache, or process requests, but the limit does not allow it.
2. Memory leak in the app — The app allocates memory over time without releasing it, eventually hitting the limit even if the initial need was smaller.
3. JVM or runtime heap not configured — Java, Node.js, or Python is using all available system memory instead of respecting an explicit heap size.
4. Sudden spike in data processing — A batch job or report generation requires temporarily more memory than the limit allows.
5. No memory request set — The pod has a limit but no request, so Kubernetes schedules it on a node that looks free but is actually under pressure.
6. QoS class is Burstable, not Guaranteed — The pod can be evicted if the node runs low on memory, even if it hasn’t technically exceeded the limit.

Fix

1. Memory limit is too low

First, measure how much memory your app actually uses:

# Install metrics-server if not present (usually pre-installed on cloud clusters)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Wait 1-2 minutes for metrics to be collected
sleep 120

# Check real-time usage
kubectl top pod <pod-name>

# Monitor over time (watch for peaks)
kubectl top pod <pod-name> --containers

Record the peak memory usage. Then increase the limit by 20–50% as a safety margin:

# If peak is 512 Mi, set limit to 768 Mi
kubectl set resources deployment <deployment-name> \
  --limits=memory=768Mi \
  --requests=memory=512Mi

Or edit the Deployment directly:

kubectl edit deployment <deployment-name>

And update the resources section:

containers:
- name: app
  resources:
    requests:
      memory: "512Mi"
      cpu: "100m"
    limits:
      memory: "768Mi"        # Increased from 512 Mi
      cpu: "500m"

Apply and verify:

kubectl rollout restart deployment/<deployment-name>
kubectl top pod <pod-name>    # Verify new pod has enough memory

2. Memory leak in the app

If the app’s memory grows over time until it hits the limit, there is a leak. Monitor memory over several hours or days:

# Collect memory usage every 30 seconds and save to a file
watch -n 30 "kubectl top pod <pod-name> >> memory-log.txt"

If memory consistently grows, the app has a leak. Fix the leak in the application code:

• Java: Use heap dumps to find retained objects:

  jmap -dump:live,format=b,file=heap.bin <pid>
  

• Node.js: Use clinic.js or heap snapshots:

  node --inspect=0.0.0.0:9229 app.js
  # Then connect DevTools and take heap snapshots
  

• Python: Use memory_profiler or objgraph to trace allocations.

For a quick workaround while fixing the leak, set a lower memory limit and increase the restart policy:

restartPolicy: Always
containers:
- name: app
  resources:
    limits:
      memory: "512Mi"    # Lower limit forces faster restart

This is not a fix but buys time while you debug the actual leak.

3. JVM or runtime heap not configured

If your app runs on a JVM, Node.js, or Python, configure the heap or memory limit explicitly:

Java/JVM:
Set the heap size in your Dockerfile or pod spec:

FROM openjdk:17-slim
ENV JAVA_OPTS="-Xmx512m -Xms256m"
COPY app.jar /app.jar
ENTRYPOINT ["java", "$JAVA_OPTS", "-jar", "/app.jar"]

Or pass it as an environment variable:

containers:
- name: app
  image: myapp:jvm
  env:
  - name: JAVA_OPTS
    value: "-Xmx512m -Xms256m"

Node.js:
Set the max-old-space-size:

FROM node:20-alpine
ENV NODE_OPTIONS="--max-old-space-size=512"
COPY . /app
WORKDIR /app
CMD ["node", "server.js"]

Or:

containers:
- name: app
  env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=512"

Python:
Some frameworks pre-allocate memory. Check if there is a memory configuration:

containers:
- name: app
  env:
  - name: PYTHONUNBUFFERED
    value: "1"    # Helps with logging in containers

Redeploy:

kubectl rollout restart deployment/<deployment-name>

4. Sudden spike in data processing

For batch jobs or report generation, use a separate resource class with higher limits:

apiVersion: batch/v1
kind: Job
metadata:
  name: heavy-report
spec:
  template:
    spec:
      containers:
      - name: report
        image: myapp:latest
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "3Gi"
            cpu: "1500m"
      restartPolicy: Never

For long-running apps that occasionally spike, consider using a Vertical Pod Autoscaler (VPA) to automatically adjust limits based on historical usage:

# Install VPA (if not already installed)
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

# Then annotate your Deployment
kubectl annotate deployment <deployment-name> \
  vpa.k8s.io/updateMode=auto \
  vpa.k8s.io/updatePolicy.updateMode=auto

5. No memory request set

If you set a limit but no request, Kubernetes treats the request as equal to the limit. This causes scheduling issues. Always set both:

resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"
    cpu: "500m"

The rule of thumb:
– Request = minimum memory needed (for scheduling)
– Limit = maximum the app should ever use (for enforcement)
– Ratio: limit should be 1.5–2x the request

Update your Deployment:

kubectl set resources deployment <deployment-name> \
  --requests=memory=256Mi,cpu=100m \
  --limits=memory=512Mi,cpu=500m

6. QoS class is Burstable instead of Guaranteed

Check the QoS class:

kubectl get pod <pod-name> -o yaml | grep qosClass

Three QoS classes exist:
– Guaranteed: requests == limits (highest priority, never evicted for memory)
– Burstable: requests < limits (medium priority, can be evicted)
– BestEffort: no requests or limits (lowest priority, evicted first)

For production apps, use Guaranteed:

resources:
  requests:
    memory: "512Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"    # Same as request
    cpu: "100m"

Apply and restart:

kubectl apply -f deployment.yaml
kubectl rollout restart deployment/<deployment-name>

Prevent it

1. Always measure before setting limits:

   kubectl top pod <pod-name>
   

   Set requests and limits based on actual observed peak usage, not guesses.

2. Set both requests and limits, never just one:

   resources:
     requests:
       memory: "256Mi"
     limits:
       memory: "512Mi"
   

3. Configure JVM/runtime heap explicitly:

   ENV JAVA_OPTS="-Xmx512m -Xms256m"
   ENV NODE_OPTIONS="--max-old-space-size=512"
   

4. Enable metrics-server to monitor memory usage:

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
   

5. Use a Vertical Pod Autoscaler (VPA) for automatic right-sizing:

   apiVersion: autoscaling.k8s.io/v1
   kind: VerticalPodAutoscaler
   metadata:
     name: example-vpa
   spec:
     targetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: example-app
     updatePolicy:
       updateMode: "Auto"
   

6. Profile your app for memory leaks during development:
   – Java: heap dumps
   – Node.js: clinic.js or DevTools
   – Python: memory_profiler

7. Set startup probes to give the app time to allocate initial memory:

   startupProbe:
     httpGet:
       path: /health
       port: 8080
     failureThreshold: 30
     periodSeconds: 10
   

FAQ

How do I know if a pod was OOMKilled?

Run kubectl describe pod <pod> and look for ‘Reason: OOMKilled’ in Last State, or exit code 137. Also check kubectl get events.

Why is my pod’s memory limit different from what I set?

Kubernetes sets limits and requests separately. If you only set a limit, the request defaults to the limit. View both with kubectl get pod -o yaml | grep -A 3 resources:.

How do I measure how much memory my app actually uses?

Use kubectl top pod <pod> to see real-time memory usage. If metrics-server is not installed, install it first (usually pre-installed on cloud clusters).

What is the difference between requests and limits?

Requests are guarantees (for scheduling); limits are caps (for enforcement). If a pod exceeds its limit, it gets OOMKilled. Always set both, with limit 1.5–2x the request.