The Container That Will Not Stay Dead
Your phone buzzes. The service is down, and docker ps says Restarting (1) 4 seconds ago. Four seconds later it says the same thing. The container is not failing once. It is failing, restarting, failing, and restarting, and each cycle buries the one log line you need under fifty copies of its neighbor.
The fix is an order of operations, not a clever trick. Read the five fields that Docker already recorded, stop the loop so the evidence holds still, then reproduce the failure in a shell you control. Most restart storms come from one of seven causes, and each leaves a different fingerprint in those fields.
This post is the “it is looping right now” procedure. For what each exit code means, see the Docker exit codes reference. For why restart policies and healthchecks are separate things, see healthcheck vs restart policy. The Kubernetes version (CrashLoopBackOff) gets its own section near the end.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/docker/container-restart-storms
Step 1: Confirm It Is a Storm
A running container with one restart is a hiccup. A storm shows a restart count that climbs while uptime stays tiny.
docker ps --format '{{.Names}}\t{{.Status}}'storm-lab-missing-env-1 Restarting (2) 39 seconds agostorm-lab-oom-1 Restarting (137) 39 seconds agostorm-lab-race-app-1 Up 39 secondsThe number in parentheses is the last exit code. Restarting means Docker is sitting in its backoff delay between attempts. Now ask the daemon for the five fields that matter:
docker inspect --format \ 'restarts={{.RestartCount}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}' \ storm-lab-oom-1restarts=10 exit=137 oom=true err= started=2026-10-25T13:03:57.96Z finished=2026-10-25T13:03:58.07ZStarted and finished are 0.1 seconds apart, so the process died almost instantly. Exit 137 plus oom=true means the kernel killed it for memory. .State.Error stays empty here, but it fills in when Docker itself could not start the process (a missing binary, a bad mount, a port conflict).
You can also watch the deaths as they happen. The die event carries the exit code as an attribute:
docker events --filter container=storm-lab-oom-1 --filter event=die \ --format '{{.Time}} exitCode={{.Actor.Attributes.exitCode}}'The attributes on a die event are exitCode, execDuration, image, and name, plus any labels. Add --since 10m --until "$(date +%s)" and the command returns instead of streaming forever, which is what you want inside a script.
How Long You Are Waiting Between Restarts
Docker does not hammer a failing container. The restart manager in the moby source (daemon/internal/restartmanager/restartmanager.go) starts the delay at 100 milliseconds, doubles it on every failure, and caps it at 1 minute. Any run of 10 seconds or more resets the delay back to 100 ms.
That gives you this schedule: 100 ms, 200 ms, 400 ms, 800 ms, 1.6 s, 3.2 s, 6.4 s, 12.8 s, 25.6 s, 51.2 s, then 60 s forever. In the lab the gaps between log lines were roughly 2, 3, 7, 13, 26, and 52 seconds, because starting the container takes time on top of the delay.
Two consequences follow. A crash that happens at second 11 is never “backed off”, so a service that dies every 15 seconds restarts at full speed forever. And once the loop reaches the 60 second cap, docker ps can show Restarting for most of a minute at a time, which makes a half-hour-old storm look calm. The count in .RestartCount is the number to trust.
The docs also say a restart policy “only takes effect after a container starts successfully”, meaning up for at least 10 seconds. Do not read that literally: the lab’s oom service lived 0.1 seconds per run and still restarted 10 times. In practice the 10 seconds decides whether the backoff resets. What the rule does catch is a container that never starts at all (bad mount, missing binary). That is a start error, not a restart loop, and it shows up in .State.Error instead.
Step 2: Read the Logs of the Run That Just Died
docker logs --timestamps --tail 50 storm-lab-missing-env-1docker logs --since 2m storm-lab-missing-env-12026-10-25T13:03:07.341Z sh: DB_PASSWORD: DB_PASSWORD is not set2026-10-25T13:03:09.073Z sh: DB_PASSWORD: DB_PASSWORD is not set2026-10-25T13:03:12.455Z sh: DB_PASSWORD: DB_PASSWORD is not setLogs belong to the container, not to the process. With the default json-file and local drivers, every restart appends to the same log, so you see all runs interleaved. The timestamps are how you tell run 9 from run 10. Use --tail so you do not scroll through a thousand identical stack traces. If the log volume itself becomes the problem, the Docker logs disk guide covers rotation.
Step 3: Stop the Loop So the Evidence Holds Still
A restarting container keeps overwriting FinishedAt and ExitCode. Freeze it:
docker update --restart=no storm-lab-oom-1docker stop storm-lab-oom-1 # harmless if it already exiteddocker update --restart=no changes the policy on a live container, no recreate needed. In the lab the container sat at Exited (137) after this and stayed there.
For Compose, know what happens next. Running docker compose up -d against an unchanged file started the stopped container again but left the policy at no. Compose only recreates a container when its config hash changes, and a recreate brings back whatever restart: says in the file. I confirmed this with --force-recreate: the policy went straight back to unless-stopped. So edit the file first if you want the loop gone for good, or run docker compose stop <service> and leave the file alone while you investigate.
Step 4: Get a Shell Where the App Dies Instantly
You cannot docker exec into a container that lives for 80 milliseconds. Skip the dead container and start a new one with the shell as the entrypoint, using the same image, env, and mounts:
docker run --rm -it --entrypoint sh \ --env-file ./app.env -v ./data:/data \ myorg/myapp:1.4With Compose, the service definition supplies all of that for you:
docker compose run --rm --entrypoint sh missing-envNow run the real command by hand and watch it fail where you can see it. If the shell works but the app does not, the problem is in the app’s startup, not the image.
Two forensic tools cover the case where the dead container itself holds the answer:
docker cp storm-lab-oom-1:/var/log/app.log ./app.log # works on stopped containersdocker commit storm-lab-oom-1 forensics:latest # freeze its filesystem as an imagedocker run --rm -it --entrypoint sh forensics:latestdocker cp reads from a stopped container without starting it. docker commit turns the dead filesystem into an image you can poke at with a shell, which beats guessing about what the app wrote before it died. Delete the image afterward.
The Seven Usual Suspects
Each cause leaves a signature. Match the fields from Step 1 to this table.
| Cause | Exit code | Other signal |
|---|---|---|
| Missing env var or secret | 1 or 2 | Log names the variable, run time under a second |
| Dependency not ready | 1 | “connection refused” or “no response” in the log, loop stops by itself |
| OOM kill | 137 | .State.OOMKilled is true, memory limit set |
| Bind mount or UID mismatch | 1, 126, or a start error | “permission denied” in the log or .State.Error |
| Port already bound | 1 | “address already in use” in the log, or a start error from the daemon |
Entrypoint script with set -e | whatever the failing command returned | Last log line is the command before the silence |
| Orchestrator killing an unhealthy container | 137 or 143 | Not plain Docker, see below |
Missing env var. The lab’s missing-env service uses ${DB_PASSWORD:?...} in a shell and dies in under a second with exit code 2. Fix it with environment: or env_file:. When variables come from several places, the Compose env precedence post explains which source wins.
Dependency race. The app starts before the database accepts connections, exits, and the restart policy retries until the database is up. The loop heals itself, which is why this one hides in production for months. In the lab, race-app restarted 9 times while slow-db took 40 seconds to boot, then stayed up. The fix is to make Compose wait for health, not just for the container to exist:
fixed-app: depends_on: slow-db: condition: service_healthy slow-db: healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 3s timeout: 3s retries: 20 start_period: 40sThe fixed-app service in the same lab finished with a restart count of 0.
OOM kill. Exit 137 alone is ambiguous, since a plain docker kill also gives 137. The OOMKilled flag is what confirms memory. The lab’s oom service has a 16 MiB limit and inflates a shell variable until the cgroup kills it. The fix is a higher limit or a smaller footprint. For how the kernel picks its victim, see the Linux OOM killer post.
Mount and permission trouble. A bind mount owned by root while the container runs as UID 1000 gives “permission denied” on the first write. Compare ls -ln on the host path with the user: in your service. With the short -v host:container syntax, a host path that does not exist gets created as a directory by Docker, and the app then fails trying to read a “file” that is a folder.
Port conflict. If the host port is taken, Docker cannot start the container, so the error lands in .State.Error, not the app log. If the app binds a port inside the container, the message comes from the app. Run ss -ltnp | grep :8080 on the host to see who owns it.
set -e in the entrypoint. One failing grep or curl in the script ends the whole container with that command’s status, and the log may show nothing after the last echo. Run the script with sh -x from the shell you opened in Step 4. That prints each command as it runs.
An orchestrator restarting unhealthy containers. Plain Docker marks a container unhealthy and does nothing else. Swarm, Kubernetes (via liveness probes), and add-on tools like autoheal do restart it. If a container restarts with exit 137 or 143 and the app log looks fine, check whether something is killing it on a failed health check. A healthcheck that is too strict on a slow-starting app is a classic cause, and Docker healthcheck patterns covers start_period tuning.
The Kubernetes Version: CrashLoopBackOff
Kubernetes runs the same loop, with different tools. kubectl get pods shows CrashLoopBackOff in the STATUS column, and the kubelet is the thing doing the restarting.
The backoff, per the Pod lifecycle docs: the delay starts at 10 seconds, doubles (10s, 20s, 40s and so on), caps at 300 seconds (5 minutes), and resets after the container has run for 10 minutes without problems. That is far slower than Docker, so a CrashLoopBackOff pod can look dead for five minutes between attempts.
Newer releases made this tunable. As of Kubernetes v1.37.1 (the current release when I checked), the KubeletCrashLoopBackOffMax gate is Beta and on by default since v1.35. It lets you set crashLoopBackOff.maxContainerRestartPeriod between 1s and 300s in the kubelet config, per node. The separate ReduceDefaultCrashLoopBackOffDecay gate (alpha since v1.33, off by default) changes the cluster-wide defaults to a 1 second start and a 60 second cap. Do not count on either being enabled on a managed cluster.
The triage order mirrors Docker:
kubectl get pods -n myappkubectl describe pod myapp-7d9c5 -n myappkubectl logs myapp-7d9c5 -n myapp --previouskubectl get events -n myapp --sort-by=.lastTimestampIn describe, look at the container’s Last State block. It carries Reason (such as Error or OOMKilled), Exit Code, and the start and finish times, which map directly to Docker’s .State fields. kubectl logs --previous is the key flag: plain kubectl logs shows the new, probably empty, run, while --previous shows the one that just died.
To get a shell where the app dies instantly, copy the pod and override the command:
kubectl debug myapp-7d9c5 -n myapp -it --copy-to=myapp-debug \ --container=app -- sh--copy-to makes a new pod from the original spec, and the trailing command replaces the container’s command. That is the cluster equivalent of --entrypoint sh. Delete the copy with kubectl delete pod myapp-debug -n myapp when you are done. If you are moving a Compose stack into a cluster, the Compose to Helm migration post covers where these probes come from.
Write It Down for Next Time
The cheapest alert is a restart-count check. This runs from cron and prints when any container has restarted more than 5 times:
#!/usr/bin/env bash# cron: */5 * * * * /usr/local/bin/restart-watch.shdocker ps -q | xargs -r docker inspect \ --format '{{.RestartCount}} {{.Name}}' | awk '$1 > 5 { print "restart storm: " $2 " (" $1 " restarts)" }'Pipe the output to your mail or push tool. If you already run Prometheus, two real metrics cover the rest. On Kubernetes, kube-state-metrics exposes kube_pod_container_status_restarts_total, a counter you can alert on with increase(...[15m]) > 3. For Docker hosts, cAdvisor exposes container_oom_events_total for OOM kills and container_start_time_seconds for spotting a container whose start time keeps moving. The setup for that stack is in the cAdvisor per-container metrics post.
The restart-triage.sh script in the example repo prints everything from Steps 1 and 2 in one go. Run it first and your 2 AM self will thank you.
Common Questions
Why does my Docker container keep restarting?
The container process exits, and the restart policy (always, unless-stopped, or on-failure) tells Docker to start it again. Check docker inspect for .State.ExitCode and .State.OOMKilled, then read docker logs --tail 50 to find why the process exits. The exit code narrows the cause to a short list.
How do I stop a Docker container from restarting without deleting it?
Run docker update --restart=no <container>, then docker stop <container>. The container keeps its filesystem, logs, and config, so you can still use docker cp, docker logs, and docker commit. With Compose, docker compose stop <service> does the same without editing the file.
What is the maximum delay between Docker restart attempts?
The maximum delay is 1 minute. Docker starts at 100 milliseconds and doubles the delay after each failed run. A container that stays up for 10 seconds or longer resets the delay to 100 ms. Kubernetes uses a different schedule, starting at 10 seconds and capping at 300 seconds by default.
Does Docker restart a container that is marked unhealthy?
No, plain Docker only marks it unhealthy and takes no action. A restart policy reacts to the process exiting, not to a failed healthcheck. Docker Swarm and Kubernetes liveness probes do restart unhealthy containers, and so do add-on tools such as autoheal that watch for the unhealthy status.
