Skip to content

policy: Hold listeners until the first network policy update - #2084

Open
firecow wants to merge 1 commit into
cilium:mainfrom
firecow:first-policy-gates-listener-main
Open

firecow wants to merge 1 commit into
cilium:mainfrom
firecow:first-policy-gates-listener-main

Conversation

@firecow

@firecow firecow commented Oct 9, 2026

Copy link
Copy Markdown

After a cilium-envoy restart the L7 listeners start accepting connections before the first NetworkPolicy update is installed, so a connection accepted in that window gets no policy filter state and every request on it answers 500 for as long as the client keeps it open, which this fixes by holding each listener's initialization until the first policy update is installed, or for at most 5 seconds so a stalled policy stream cannot keep listeners down.

@firecow
firecow requested a review from a team as a code owner October 9, 2026 08:34
@firecow
firecow requested a review from nezdolik October 9, 2026 08:34
@moberghammer

Copy link
Copy Markdown

👍🏻

@jrajahalme

Copy link
Copy Markdown
Member

@firecow Did you verify that the Listener init manager wait on policy update meaningfully changes behavior? The wait delays both listener socket listen() and accept() calls, so clients could get connection refused responses instead during the wait window.

Also, please squash commits as we don't do that automatically on merge :-)

@nezdolik

Copy link
Copy Markdown
Contributor

+1 on @jrajahalme point
@firecow so which response code would client get while listener initialization is postponed waiting on policy update?

@jrajahalme

Copy link
Copy Markdown
Member

Also needs rebasing due to conflict with main due to #2056.

Listeners started accepting connections before the first NetworkPolicy
update was installed, so connections accepted in that window had no
policy for local endpoints and every request on them failed with 500
for the lifetime of the connection. Each listener now waits for the
first policy update, or 5 seconds at most.

Signed-off-by: Mads Jon Nielsen <madsjon@gmail.com>
@firecow
firecow force-pushed the first-policy-gates-listener-main branch from 7a879ff to 7a2358d Compare October 21, 2026 06:35
@firecow

firecow commented Oct 21, 2026 •

Copy link
Copy Markdown
Author

@jrajahalme @nezdolik Yes. On minikube with Cilium main and cilium-envoy from this branch, plus a 3 s sleep before the first NPDS update to widen the window, a keep-alive connection opened in the window got 500 on every request for the whole test without this change, and 200 with it.

Neither build gave connection refused. While no listener is accepting, the client's connect just stalls until the listener starts, as it already does during a cilium-envoy restart. The change extends that by the time until the first policy update, so a connect may wait one TCP retransmit longer instead of succeeding and then failing every request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants