Skip to content

fix: replace stale Unix socket left behind by an unclean shutdown - #2727

Open
0z5a wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
0z5a:fix/replace-stale-unix-socket
Open

0z5a wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
0z5a:fix/replace-stale-unix-socket

Conversation

@0z5a

@0z5a 0z5a commented Nov 15, 2026

Copy link
Copy Markdown

Description of changes

If the Proxy exits without closing its listeners (SIGKILL, OOM kill, crash), the Unix socket file stays on disk. The next start then fails for that instance with bind: address already in use, and keeps failing until someone deletes the file by hand. A clean shutdown (SIGTERM/SIGINT) is not affected, because closing a net.UnixListener unlinks the socket.

This happens whenever the socket directory outlives the process, for example:

  • A Kubernetes sidecar running with --unix-socket /cloudsql on an emptyDir shared with the app container. If the proxy container is OOM-killed, or killed after a failed liveness probe (SIGKILL once the grace period is over, e.g. when --max-sigterm-delay is longer than terminationGracePeriodSeconds), the kubelet restarts it in the same pod. The emptyDir survives container restarts, so the restarted proxy goes into CrashLoopBackOff until the pod is deleted.
  • A systemd unit with Restart=on-failure after the process crashed or was killed when TimeoutStopSec expired.

The v1 Proxy removed an existing file before listening on a Unix socket (listenInstance called remove(cfg.Address) before net.Listen). In v2, newUnixSocketMount already tolerates the Postgres socket directory left over from a previous run ("Make the directory only if it hasn't already been created"), but not the .s.PGSQL.5432 socket inside it, nor the socket file used for MySQL/SQL Server.

Rather than removing the path unconditionally like v1 did, this change only acts when Listen fails with EADDRINUSE on a Unix socket. In that case, if the path is a socket and connecting to it is refused (nothing is listening on it), the file is removed and Listen is retried once. A socket that still accepts connections (for example one owned by another running Proxy), or a path that is not a socket, is left alone and the original error is returned as before. I did not test on Windows; the EADDRINUSE/ECONNREFUSED checks are not expected to match the WSA error codes there, so the existing error path is kept.

unix-socket-path and the FUSE temp-dir sockets go through the same newSocketMount path, so they get the same behavior (FUSE not tested here).

Reproduction

No instance or network access is needed: --sql-data skips the engine-version lookup for Unix sockets, nothing is dialed until a client connects, and the API endpoints point at a closed local port.

mkdir -p /tmp/csql
cloud-sql-proxy --sql-data --token dummy --enable-builtin-metrics=false \
  --sqladmin-api-endpoint http://localhost:1/ --sqldata-api-endpoint localhost:1 \
  --unix-socket /tmp/csql proj:region:inst &
sleep 2; kill -KILL $!
cloud-sql-proxy --sql-data --token dummy --enable-builtin-metrics=false \
  --sqladmin-api-endpoint http://localhost:1/ --sqldata-api-endpoint localhost:1 \
  --unix-socket /tmp/csql proj:region:inst

Second start on main (socket directory shown as <dir>):

[proj:region:inst] could not listen to address <dir>/proj:region:inst/.s.PGSQL.5432: listen unix <dir>/proj:region:inst/.s.PGSQL.5432: bind: address already in use
Error starting proxy: [proj:region:inst] Unable to mount socket: listen unix <dir>/proj:region:inst/.s.PGSQL.5432: bind: address already in use
The proxy has encountered a terminal error: unable to start: [proj:region:inst] Unable to mount socket: listen unix <dir>/proj:region:inst/.s.PGSQL.5432: bind: address already in use

With this change:

[proj:region:inst] Removed stale Unix socket <dir>/proj:region:inst/.s.PGSQL.5432
[proj:region:inst] Listening on <dir>/proj:region:inst/.s.PGSQL.5432
The proxy has started successfully and is ready for new connections!

Starting a second Proxy while the first one is still running on the same directory still fails with bind: address already in use, and the first Proxy's socket keeps accepting connections. TestClientInitializationWithUnixSocketInUse covers this case and passes on main and with this change.

Tests

  • TestClientInitializationWithStaleUnixSocket (MySQL socket file and Postgres .s.PGSQL.5432) leaves a socket with no listener behind and checks that NewClient starts and the socket accepts connections. It fails on main with bind: address already in use.
  • TestClientInitializationWithUnixSocketInUse checks that a socket with a live listener is not removed and that startup still fails.
--- FAIL: TestClientInitializationWithStaleUnixSocket (0.00s)
    --- FAIL: TestClientInitializationWithStaleUnixSocket/MySQL (0.00s)
    --- FAIL: TestClientInitializationWithStaleUnixSocket/Postgres (0.00s)

Ran go test -short ./cmd/... ./internal/..., go vet ./... and golangci-lint run ./internal/proxy/... on macOS, plus GOOS=windows and GOOS=linux go vet for ./internal/proxy/ and ./cmd/.

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

If the Proxy is killed without closing its listeners (SIGKILL, OOM
kill, crash), the Unix socket file stays on disk and the next start
fails with "bind: address already in use" until the file is removed by
hand. v1 removed the path before listening.

When Listen fails with EADDRINUSE on a Unix socket, remove the path and
retry once, but only if it is a socket and connecting to it is refused.
A socket with a live listener, or a path that is not a socket, is left
alone and the original error is returned.
@0z5a
0z5a requested a review from a team as a code owner November 15, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants