feat: add graceful shutdown with drain window and idle-connection grace period

[kaiburjack]

Motivation

Currently, when receiving SIGTERM (i.e. when used in Kubernetes), the Go HTTP server will immediately initiate shutdown, closing its listen socket and also all currently idle connections immediately.
This is problematic when doing a rolling update in Kubernetes, because:

1. kube-proxy takes a while to notice a terminating pod and updating the node's iptables/netfilter rules (roughly 5s). During that time, new connections can still be routed to the terminated gcsproxy pod/process, resulting in TCP connection resets.
2. by default, the Go HTTP Server closes idle connections on shutdown immediately. This is also problematic, because it can result in the race condition of a client (using HTTP persistent/keep-alive connections) to initiate a new request right as the server is closing the idle connection.
   The HTTP/1.1 RFC 9112 also briefly discusses this:

For example, a client might have started to send a new request at the same time that the server has decided to close the “idle” connection. From the server’s point of view, the connection is being closed while it was idle, but from the client’s point of view, a request is in progress.

We have seen all of these problems in our Kubernetes production deployment of gcsproxy (and other services) and have forked it because of this (and because of many other features we would like to have): https://github.com/HBTGmbH/gcsproxy
Now, that the main gcsproxy project is actively being maintained again, we would like to add these features to the upstream project as well.

I've also written some blog posts about all of these problems :)
https://medium.com/hamburger-berater-team/achieving-zero-downtime-in-kubernetes-d9e3e5b5927c

What this PR adds

1. A configurable "drain duration" during which nothing is stopped or shutdown or closed: People in the Kubernetes world usually do this with a preStop sleep lifecycle hook. However, it also does not hurt when the service itself provides this as well. This is to avoid the problem 1. mentioned above. During this drain period, requests will receive a Connection: close response header in the response, indicating to the client to not resuse this connection anymore, because the server is about to shutdown. Eventually, the random connection routing in iptables/netfilter will result in all such active connections being routed somewhere else
2. After this drain duration, the listen socket is closed, not accepting any further new connections. Existing connections and existing requests on such connections are unaffected.
3. A configurable "idle grace" duration during which idle connections are not closed, to avoid problem 2. mentioned above. This duration should ideally be set to the maximum client-side idle/keep-alive timeout. Once, no idle connections exist anymore (because they either have seen a next request, for which we respond with Connection: close, or the client closed it on their own, due to client-side idle/keep-alive timeout), the server is shutdown.
4. A configurable "shutdown timeout", for Go's HTTP Server Shutdown(ctx), effectively waiting for this long for currently active requests to finish on active connections.

[daichirata]

Thank you for adding graceful shutdown support! I left a few comments.

[daichirata]

Since this wait may end before minCloseDelay elapses, I wonder if naming this setting something like idleGracePeriod would better reflect its behavior. This would also apply to the corresponding CLI flag and documentation. What do you think?

[kaiburjack]

Yes, very good idea. That is definitely not a "minimum" delay.

[kaiburjack]

I've incorporated the suggested changes in the latest (squashed) commit.

[daichirata]

With the current default, shutdown may wait indefinitely if an in-flight request gets stuck, for example while streaming to a slow client or waiting on GCS. Would a finite default such as 30 seconds be safer, while still allowing users to explicitly set 0 if they want to wait indefinitely? What do you think?

[kaiburjack]

Yes, setting it to something reasonable, like the proposed 30s is a good idea.
The other flags should keep their 0s default, though, IMO, since it is highly deployment-scenario-dependent what values make sense there, and we should keep the behaviour mostly compliant with the previous behaviour before these changes.

[kaiburjack]

I've incorporated the suggested changes in the latest (squashed) commit.