Skip to content

stats: cancel old streams when containers restart - #7510

Open
mako042 wants to merge 1 commit into
docker:masterfrom
mako042:fix/stats-cancel-removed-collectors
Open

mako042 wants to merge 1 commit into
docker:masterfrom
mako042:fix/stats-cancel-removed-collectors

Conversation

@mako042

@mako042 mako042 commented Nov 11, 2026 •

Copy link
Copy Markdown

Summary

Running docker stats without container arguments opens another stats stream each time a container restarts. The die handler removes the container from the display, but leaves its request open. The next start creates a new collector for the same ID, so connections and collectors accumulate even after the restart loop ends.

Cancel the collector when removing its display entry, and cancel outstanding requests when the command returns. Let the decoder stop on cancellation too. Only initial collectors use the startup wait group, and container-ID lookups take the stats lock to avoid races during restarts.

Testing

TestRunStatsRestart checks that each exit closes the old HTTP request and the next start opens a new one. It fails on unmodified master and passes with this patch:

./scripts/with-go-mod.sh go test ./cli/command/container -run '^TestRunStatsRestart$' -v -count=1

I also ran 20 container restarts against Engine 29.8.1. With CLI 29.9.3, socket FDs rose from 24 to 44 and stayed there after a pause. With this patch, they stayed at 24. The separate mock-API test with --all also kept one active stream.

The unit suite and 20 runs of the stats tests under -race passed. The container package passed under -race with TestInitTtySizeErrors skipped; that test has a buffer race that I also reproduced on unmodified master. The full daemon-backed e2e suite was not run.

Manual reproduction on Linux

Save this as repro-real-engine.sh. It uses a local Docker Engine, creates one temporary Alpine container, restarts it 20 times, then removes it and stops its stats client. Use a test daemon; the script may pull Alpine 3.22.

#!/bin/sh
# Run against a disposable Linux Docker Engine using its local Unix socket.
# CLI selects the binary under test; docker is used for container lifecycle calls.
set -eu

CLI=${CLI:-docker}
engine_host=${DOCKER_HOST:-unix:///var/run/docker.sock}
name="stats-stream-repro-$$"
stats_pid=
tmpdir=$(mktemp -d /tmp/docker-stats-repro.XXXXXX)

cleanup() {
    if [ -n "$stats_pid" ]; then
        kill "$stats_pid" 2>/dev/null || true
        wait "$stats_pid" 2>/dev/null || true
    fi
    docker --host "$engine_host" rm -f "$name" >/dev/null 2>&1 || true
    rm -f "$tmpdir/stats.log"
    rmdir "$tmpdir"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

socket_count() {
    if ! kill -0 "$stats_pid" 2>/dev/null; then
        cat "$tmpdir/stats.log" >&2
        echo "The stats process exited unexpectedly" >&2
        exit 1
    fi
    find "/proc/$stats_pid/fd" -lname 'socket:*' 2>/dev/null | wc -l
}

"$CLI" --version
docker --host "$engine_host" run -d --name "$name" alpine:3.22 sleep 3600 >/dev/null

# No container argument: this exercises the start/die event handlers.
"$CLI" --host "$engine_host" stats --format '{{json .}}' >"$tmpdir/stats.log" 2>&1 &
stats_pid=$!
sleep 3
before=$(socket_count)

i=0
while [ "$i" -lt 20 ]; do
    docker --host "$engine_host" restart --timeout 0 "$name" >/dev/null
    sleep 1
    i=$((i + 1))
done

sleep 3
after=$(socket_count)
sleep 5
settled=$(socket_count)
printf 'CLI socket FDs: before=%s after_20_restarts=%s settled=%s\n' "$before" "$after" "$settled"

Run against the same Engine with each client:

CLI=/absolute/path/to/docker-29.9.3 sh repro-real-engine.sh
CLI=/absolute/path/to/docker-fixed sh repro-real-engine.sh

docker on PATH handles the container lifecycle; CLI selects the stats binary. Set DOCKER_HOST to a different Unix socket if needed. Keep the stats command without container arguments so it follows start/die events.

Before the fix, socket FDs should grow by roughly one per restart and remain elevated. After the fix, they should settle near the initial count. The count includes the events connection and idle HTTP connections, so exact totals vary.

Release notes

Fix `docker stats` accumulating open streams when containers restart.

Cancel the streaming request for a tracked container when its display entry is removed, so subsequent start events do not accumulate collectors for the same container ID.

Keep event-driven collectors out of the initial-sample wait group, synchronize reads of the tracked ID, and allow decoder notifications to stop on context cancellation.

Signed-off-by: a.p.matveev <a.p.matveev@tmn4.esoft.tech>
@mako042 mako042 changed the title cli/command/container: cancel stats streams on container exit stats: cancel old streams when containers restart Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant