Repository navigation
Conversation
If the Proxy is killed without closing its listeners (SIGKILL, OOM kill, crash), the Unix socket file stays on disk and the next start fails with "bind: address already in use" until the file is removed by hand. v1 removed the path before listening. When Listen fails with EADDRINUSE on a Unix socket, remove the path and retry once, but only if it is a socket and connecting to it is refused. A socket with a live listener, or a path that is not a socket, is left alone and the original error is returned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
If the Proxy exits without closing its listeners (SIGKILL, OOM kill, crash), the Unix socket file stays on disk. The next start then fails for that instance with
bind: address already in use, and keeps failing until someone deletes the file by hand. A clean shutdown (SIGTERM/SIGINT) is not affected, because closing anet.UnixListenerunlinks the socket.This happens whenever the socket directory outlives the process, for example:
--unix-socket /cloudsqlon anemptyDirshared with the app container. If the proxy container is OOM-killed, or killed after a failed liveness probe (SIGKILL once the grace period is over, e.g. when--max-sigterm-delayis longer thanterminationGracePeriodSeconds), the kubelet restarts it in the same pod. TheemptyDirsurvives container restarts, so the restarted proxy goes into CrashLoopBackOff until the pod is deleted.Restart=on-failureafter the process crashed or was killed whenTimeoutStopSecexpired.The v1 Proxy removed an existing file before listening on a Unix socket (
listenInstancecalledremove(cfg.Address)beforenet.Listen). In v2,newUnixSocketMountalready tolerates the Postgres socket directory left over from a previous run ("Make the directory only if it hasn't already been created"), but not the.s.PGSQL.5432socket inside it, nor the socket file used for MySQL/SQL Server.Rather than removing the path unconditionally like v1 did, this change only acts when
Listenfails withEADDRINUSEon a Unix socket. In that case, if the path is a socket and connecting to it is refused (nothing is listening on it), the file is removed andListenis retried once. A socket that still accepts connections (for example one owned by another running Proxy), or a path that is not a socket, is left alone and the original error is returned as before. I did not test on Windows; theEADDRINUSE/ECONNREFUSEDchecks are not expected to match the WSA error codes there, so the existing error path is kept.unix-socket-pathand the FUSE temp-dir sockets go through the samenewSocketMountpath, so they get the same behavior (FUSE not tested here).Reproduction
No instance or network access is needed:
--sql-dataskips the engine-version lookup for Unix sockets, nothing is dialed until a client connects, and the API endpoints point at a closed local port.Second start on main (socket directory shown as
<dir>):With this change:
Starting a second Proxy while the first one is still running on the same directory still fails with
bind: address already in use, and the first Proxy's socket keeps accepting connections.TestClientInitializationWithUnixSocketInUsecovers this case and passes on main and with this change.Tests
TestClientInitializationWithStaleUnixSocket(MySQL socket file and Postgres.s.PGSQL.5432) leaves a socket with no listener behind and checks thatNewClientstarts and the socket accepts connections. It fails on main withbind: address already in use.TestClientInitializationWithUnixSocketInUsechecks that a socket with a live listener is not removed and that startup still fails.Ran
go test -short ./cmd/... ./internal/...,go vet ./...andgolangci-lint run ./internal/proxy/...on macOS, plusGOOS=windowsandGOOS=linuxgo vetfor./internal/proxy/and./cmd/.This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.