Container Lifecycle

A container moves through a small set of states. Created means the runtime has prepared the root filesystem, namespaces, cgroups and configuration, but no process is running yet. Running means PID 1, the entrypoint process, is executing. Paused freezes all processes through the cgroup freezer and is rarely used in production. Restarting appears when a restart policy is bringing the container back after an exit, with backoff. Exited, or stopped, means the main process ended and its exit code is recorded. Removed means the writable layer and metadata are deleted. The transition that matters most is stopping, because it decides whether deployments, scale-ins and node drains lose in-flight requests.

SIGTERM is a polite request to terminate: the process can catch it, stop accepting new work, finish in-flight requests, close connections and exit cleanly. The JVM turns SIGTERM into an orderly shutdown that runs shutdown hooks, which is how Spring Boot's graceful shutdown drains requests, and the exit code is then 143. SIGKILL cannot be caught, blocked or handled: the kernel ends the process immediately, so in-flight requests fail, buffers are not flushed and locks may be left behind, with exit code 137. docker stop and Kubernetes send SIGTERM first, wait for the grace period, and only then send SIGKILL, so the application must finish within that window.

Exec form, written as a JSON array such as ["java","-jar","app.jar"], starts java directly as PID 1, so signals sent by Docker or the kubelet reach the JVM and trigger its shutdown hooks. Shell form, written as plain text, wraps the command in /bin/sh -c, making the shell PID 1. Many shells do not forward SIGTERM to their child, so the JVM never hears it, the runtime waits for the full grace period and then sends SIGKILL, dropping in-flight requests and slowing every deploy. In the note's example, switching to exec form halved deploy time. If you need a wrapper script, it must end with exec java so the JVM replaces the shell.

Exit code 137 means the process was killed with SIGKILL, because shells and runtimes report 128 plus the signal number, and SIGKILL is 9. In containers there are two common causes. The kernel OOM killer ended the process because the container exceeded its memory limit, which Kubernetes shows as last state reason OOMKilled. Or the grace period expired after SIGTERM and the runtime forced the kill, which points to signals not reaching the app or a drain that takes too long. The distinction matters: the first needs memory tuning, such as heap percentage plus off-heap headroom, while the second needs entrypoint and shutdown fixes. A healthy SIGTERM exit is usually 143.

Make the old pod leave the load balancer before it stops serving, and let it finish what it has. Run the app as PID 1 with an exec-form entrypoint so it receives SIGTERM. Add a preStop delay of about ten seconds, because endpoint removal propagates asynchronously to kube-proxy and the ALB. Enable server.shutdown=graceful with a phase timeout above the longest request, and set terminationGracePeriodSeconds above preStop plus drain, 40 seconds for ShopNow checkout, with ALB deregistration delay aligned. On the new side, readiness probes must pass only when the pod can serve. Use maxUnavailable 0 with surge, and verify with zero 5xx responses across deploys.

PID 1 is the init process of the container's PID namespace, and the kernel treats it specially. Signals for which PID 1 has not installed a handler are ignored rather than applying the default action, so a process that does not handle SIGTERM simply keeps running until SIGKILL. PID 1 also inherits orphaned child processes and must reap them with wait, otherwise they remain as zombies that consume PID table entries. The JVM installs a SIGTERM handler, so it shuts down correctly as PID 1, but it does not reap unrelated children. That is why a container spawning ImageMagick needs tini or docker run --init as PID 1 to forward signals and reap zombies.

A slow-starting JVM must not be killed or sent traffic before it is ready. Add a startup probe whose failureThreshold multiplied by periodSeconds covers the worst observed start, for example 30 times 2 seconds; liveness and readiness probes only begin after it succeeds, so no premature restarts. Make readiness reflect real readiness, including warm-up of critical paths. Then shorten startup itself: AppCDS, the Java 25 AOT cache and lazy initialisation where safe, reducing ShopNow checkout from 9.8 to 3.9 seconds, or CRaC where supported. Give enough CPU during startup, because CPU throttling stretches class loading and JIT time, and keep the image small for faster pulls on new nodes.

When a pod is deleted, the API server marks it Terminating and, in parallel, the endpoints controller removes it from Service endpoints; kube-proxy, ingress controllers and the AWS load balancer controller update asynchronously. The kubelet runs the preStop hook for each container, then sends SIGTERM to PID 1. The application stops accepting work, drains in-flight requests and closes resources. The terminationGracePeriodSeconds clock covers preStop and shutdown together; if containers are still running when it expires, the kubelet sends SIGKILL. Native sidecars stop after the main containers. Because endpoint removal and SIGTERM race each other, a short preStop delay is what prevents requests arriving at a stopping process.

Measure it rather than trusting configuration. The primary signal is zero 5xx responses at the load balancer during rollouts, compared with a baseline, ideally checked automatically by the deployment pipeline or Argo Rollouts analysis. Next, track exit codes: pods ending with 137 should be close to zero, apart from OOM investigations, because each one means a forced kill. Track termination duration against the grace period, plus consumer metrics such as Kafka rebalance time and duplicate message rates after deploys. Then test the hard cases deliberately: node drains, Spot interruptions and deploys during load tests. ShopNow proves it with zero failed requests across 50 deploys a week.

Spot capacity can be reclaimed with only a two-minute warning, so shutdown must be fast and automatic. An interruption handler, either Karpenter's native interruption handling through an SQS queue or the AWS Node Termination Handler, receives the notice, cordons the node and drains it, so pods go through the normal preStop and SIGTERM path. Every pod's preStop plus drain must finish well inside two minutes, which rules out long-running requests or batch steps without checkpoints. Run enough replicas across instance types and AZs, use PodDisruptionBudgets, and make startup fast so replacements become ready quickly. Work that cannot stop in time belongs on on-demand capacity.