What is a Graceful Shutdown?
A graceful shutdown is the process where a system, service, or application is stopped in a managed and controlled manner, allowing it to finish processing active requests or jobs and close network connections properly.
Unlike a forceful shutdown, which can interrupt running tasks and cause data loss, it ensures that all ongoing processes are completed before terminating the service.
Why We Need It
-
Prevents Data Loss and Corruption: If you cut the power or force-close an app while a process is writing to disk or performing some db operation, you can corrupt your file systems or leave incomplete data. A graceful shutdown gives the app time to finish writing and flush buffers.
-
Avoids Dropped User Requests: In web servers, dropping a process instantly severs active connections. A graceful shutdown allows the Go server to stop accepting new requests while it completes in-flight ones.
-
Seamless Deployments & Scaling: In cloud environments, when using Docker or Kubernetes, instances are constantly spun up and down during rolling updates. Graceful shutdowns ensure instances properly terminate, avoiding downtime and user-facing errors.
Prerequisites
What do you need to follow this article?
- Working knowledge of Golang.
- Knowledge of HTTP.
Understanding Signals
Before we proceed, we need to understand how the operating system actually tells your application to stop. It does this through signals.
A signal is a standardized asynchronous notification sent to a running process to inform it that an important event has occurred. These events can originate from the user (like pressing Ctrl+C), from the operating system, or even from other programs. When a signal is delivered, the OS interrupts the normal flow of the process to handle it.
How a Process Responds to Signals
When a process receives a signal, one of three things happens:
- Signal handler: The process can register a custom handler (a function) for a specific signal. When that signal comes in, the handler runs instead of the default behavior.
- Default action: If no custom handler is registered, the process follows the default behavior for that signal. Depending on the signal, this might mean terminating, stopping, continuing, or ignoring the process.
- Unblockable signals: Some signals, like SIGKILL, cannot be caught or ignored. They will always terminate the process, no matter what.
Common Signals
Here are the signals you will encounter most often:
- SIGINT (Signal Interrupt): Sent when the user presses Ctrl+C in the terminal.
- SIGTERM (Signal Terminate): A polite request for a process to terminate. This is what Kubernetes sends to your pod during shutdown.
- SIGKILL: A forceful termination signal that cannot be caught or ignored.
- SIGSEGV (Segmentation Violation): Sent when a program attempts to access memory that doesn't belong to it.
- SIGSTOP and SIGCONT: Used to pause and resume process execution.
- SIGQUIT: Quit signal (such as Ctrl+D).
For our implementation, we mostly care about SIGTERM and SIGINT. SIGTERM is what orchestrators like Kubernetes send when they want your app to stop, and SIGINT is what you get when you press Ctrl+C in the terminal of your running server.
How Go Handles Termination
Now that we understand signals, let's look at what Go actually does when your app receives one.
When your Go app gets a SIGTERM, the runtime first catches it using a built-in handler. It checks if a custom handler has been registered. If not, the runtime disables its own handler temporarily and sends the same signal (SIGTERM) to the application again. This time, the OS handles it using the default behavior, which is to terminate the process and clean up any remaining resources.
The important thing to note here is that Go gives us the ability to intercept these signals before the default behavior kicks in. We can override the signal handling by registering our own handler using the os/signal package. This is what we will use to implement our graceful shutdown.
Option 1: Using Channels
The first approach is to use a channel that receives signals:
package main
import (
"fmt"
"os"
"os/signal"
"syscall"
)
func main() {
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
// normal cooking here (server setup, etc...)
<-signalChan
fmt.Println("Received, shutting down...")
}We use a buffered channel with a capacity of 1 here to prevent blocking. signal.Notify tells Go to relay SIGINT and SIGTERM to our channel instead of using the default behavior. The <-signalChan line blocks until a signal is received.
Option 2: Using Context (the Preferred Approach)
Starting with Go 1.16, you can simplify signal handling by using signal.NotifyContext, which ties signal handling directly to context cancellation:
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()NotifyContext returns a copy of the parent context that automatically cancels itself whenever one of the specified signals is received. This is the preferred approach because contexts are already deeply integrated into Go's standard library, and it lets us propagate the shutdown signal to all parts of our application naturally.
Building the Graceful Shutdown
Now that we have found a way to intercept shutdown signals, let's build the actual implementation. There are a few things we need to handle before we can wire it together.
Don't Drop Traffic During the Transition
This is something a lot of people miss. In a containerized environment (and many other orchestrated environments with external load balancers), you should not stop accepting new requests immediately. Even after a container is marked for shutdown or a pod is marked for termination, it might still receive traffic for a few moments before the load balancer pings again to know that it is inactive.
To avoid connection errors during this short window, the correct approach is to fail the health check first. This tells the load balancer to stop sending you traffic:
var isShuttingDown atomic.Bool
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
if isShuttingDown.Load() {
http.Error(w, "Shutting down", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte("OK"))
})After updating our availability status, we wait a small interval for the next health check ping from the load balancer. This gives it time to notice we are no longer healthy and stop routing new requests to us.
Shutting Down the Server and Notifying Handlers
Once the load balancer knows to leave us alone, we can actually shut down the server. With net/http, we do this by calling http.Server.Shutdown. This method stops the server from accepting new connections and waits for all active requests to complete before shutting down.
It returns in only two situations:
- Good case: All active connections are closed and all handlers have finished processing.
- "Oof"(Bad) case: The context passed to
Shutdown(ctx)expires before the handlers finish. In this case, the server gives up waiting and returns an error.
But there is a problem. Handlers are not automatically aware when the server is shutting down. If a handler is doing a long-running operation (like a big database query or a slow external API call), it has no way of knowing it should wrap things up.
We can solve this by setting the server's BaseContext field. BaseContext lets us provide a custom context that gets shared across all incoming connections. If we make that context cancellable, we can cancel it during shutdown, and every handler will be able to detect that it's time to stop:
outgoingContext, stopOutgoingContext := context.WithCancel(context.Background())
server := &http.Server{
Addr: ":8080",
BaseContext: func(_ net.Listener) context.Context {
return outgoingContext
},
}Here, outgoingContext is our cancellable context, and we wire it into the server through BaseContext. Now every handler can check r.Context().Done() to see if the server is shutting down, and exit early if needed.
The Full Implementation
Here is the full implementation with all the pieces connected:
package main
import (
"context"
"log"
"net"
"net/http"
"os/signal"
"sync/atomic"
"syscall"
"time"
)
var isShuttingDown atomic.Bool
func main() {
signalContext, stopSignalContext := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stopSignalContext()
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
if isShuttingDown.Load() {
http.Error(w, "Shutting down", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte("OK"))
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
select {
case <-time.After(2 * time.Second):
w.WriteHeader(http.StatusOK)
w.Write([]byte("Aktiv!"))
case <-r.Context().Done():
http.Error(w, "Request cancelled.", http.StatusRequestTimeout)
}
})
outgoingContext, stopOutgoingContext := context.WithCancel(context.Background())
server := &http.Server{
Addr: ":8080",
BaseContext: func(_ net.Listener) context.Context {
return outgoingContext
},
}
go func() {
log.Println("Server running on http://localhost:8080")
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
panic(err)
}
}()
<-signalContext.Done()
stopSignalContext()
isShuttingDown.Store(true)
log.Println("shutdown signal received, shutting down...")
time.Sleep(5 * time.Second)
log.Println("Health check propagated, waiting for ongoing requests to finish.")
shutdownContext, cancelShutdownContext := context.WithTimeout(context.Background(), 15*time.Second)
defer cancelShutdownContext()
err := server.Shutdown(shutdownContext)
if err != nil {
log.Println("Shutdown timed out, cancelling in-flight requests.")
stopOutgoingContext()
time.Sleep(3 * time.Second)
server.Close()
}
log.Println("Server shut down gracefully.")
}Let me walk through the shutdown sequence so you can see how all the pieces work together:
- A SIGINT or SIGTERM arrives.
signalContextgets cancelled, and the<-signalContext.Done()line unblocks. - We set
isShuttingDownto true, so the/healthendpoint starts returning503 Service Unavailable. - We wait 5 seconds. This gives the load balancer time to notice the failed health check and stop sending us new traffic.
- We call
server.Shutdown()with a 15-second timeout. This stops the server from accepting new connections and waits for all in-flight requests to finish. - If requests complete within the timeout, we are done. If they don't, we cancel
outgoingContextto signal all handlers to stop, wait a bit, and then force-close the server.
A Note on time.Sleep vs time.After
If you noticed, the root handler uses time.After inside a select instead of a plain time.Sleep. This is intentional. time.Sleep blocks the goroutine unconditionally, so it will not respond to context cancellation. That completely defeats the purpose of what we are trying to achieve with the BaseContext. The handler would just sit there sleeping even though the server is trying to shut down.
Instead, we can use a select that listens to both the timer and the context. You can use this helper function for context-aware sleeping:
func Sleep(ctx context.Context, duration time.Duration) error {
select {
case <-time.After(duration):
return nil
case <-ctx.Done():
return ctx.Err()
}
}This way, if the context gets cancelled while we are waiting, we return immediately instead of blocking until the full duration elapses.
Demo
Here is a video showing the graceful shutdown in action:
Conclusion
From the implementation above, the server shuts down gracefully, handling in-flight requests properly and releasing resources. The key takeaways are:
- Intercept signals using
signal.NotifyContextso you control what happens when your app receives a SIGTERM or SIGINT. - Fail the health check first before stopping the server. This gives your load balancer time to stop routing new traffic to you.
- Use
server.Shutdown()to let in-flight requests complete before closing. - Set a
BaseContextso your handlers can detect the shutdown and exit early if needed. - Use context-aware sleep instead of
time.Sleepin handlers, otherwise your graceful shutdown is not actually graceful.
These patterns apply to any Go HTTP server, whether you are running on bare metal, in Docker, or on Kubernetes. The specifics of the timeouts will depend on your setup (how often your load balancer health checks, how long your longest request takes), but the structure stays the same.
(If you want to see the full codebase, check the repository. Play around with it and lmk what you think)
