Repository navigation
Conversation
|
This pull request has had no activity for 14 days and is now marked stale. |
github-actions
Bot
added
the
state:stale
9328dc9 to
6af3db2
Compare
github-actions
Bot
removed
the
state:stale
|
/ok to test 6af3db2 |
|
These PR needs changes with the tree topology design and shared supervisor logic we have, please think through it, happy to discuss over a call |
mrunalp
added
area:gateway
|
Label |
|
/ok to test 6af3db2 |
Consistent-hash placement only logged its redirects, so operators had no way to see how often replicas redirect supervisors or whether supervisors end up where the ring places them. The gateway now counts each SessionRedirect it sends, labeled by the step that sent it: connect when it answers a connecting supervisor, or shutdown once a stopping replica has queued a handoff on an open session. It also counts redirected supervisors it admits although its own ring names another owner, once the session's acceptance is durable, so a cancelled connect cannot lose the count. Under supervisor_topology: gateway the shared supervisor's host moves pool sessions itself, so a redirected hello there is the designed state, not ring disagreement. Those gateways export no off-ring series at all and never read the ring on the connect path. Both counters start at zero and carry no sandbox or replica IDs. The gateway metrics page, the HA guide, and the debug skill document what the counters cannot see under the shared topology. Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Each replica builds its placement ring from live membership, but nothing showed how many replicas a given replica had in its ring, so a replica that dropped out of its peers' rings, or a ring cleared after failed refreshes, went unnoticed. openshell_server_gateway_ring_members now reports the replicas in this replica's ring. Every ring write goes through one helper that records the size under the same lock: the periodic refresh, leaving the ring at shutdown, and clearing a stale ring. Only replicas that join the ring export the gauge, so SQLite gateways and replicas without a peer endpoint show no series rather than a misleading 0. The HA guide's Monitor Capacity table gains a Session placement row scoped to gateways that place supervisor sessions by ring: under the multi-tenant topology the shared supervisor's host moves sessions, and that section says the placement metrics do not reflect those moves. Part of NVIDIA#4223 Closes NVIDIA#4688 Signed-off-by: Emilien Macchi <emacchi@redhat.com>
6af3db2 to
5e82f09
Compare
|
Thought it through against #5052 and RFC 0018 rather than waiting for a call. Redirects and ring size are unchanged: they count gateway actions that hold in both regimes. The off-ring counter is the one that wasn't honest under the tree, so it now exists only on gateways that place supervisor sessions by ring. Under supervisor_topology: gateway the series isn't exported at all (the startup init path is gated too), because a redirected hello there is the host following its design, not ring disagreement. The docs now say host-driven moves never appear in redirects_total, and shutdown counts are a lower bound for pool sessions. I left preferred_peer_redirect pool-aware out on purpose: #5052 dropped gateway-side pool placement, so I took that as the boundary you want. The one edge I found is a pool session hostless for more than 5 seconds getting ring-redirected away from where its host reattaches. Happy to file a narrow issue for it if you want to weigh in. |
|
/ok to test 5e82f09 |
Summary
Adds placement metrics for consistent-hash session placement (#3661), scoped to the topologies where each signal is honest: redirects and ring size everywhere, off-ring admissions only where the gateway itself places sessions by ring.
Related Issue
Closes #4688
Part of #4223
#4688 still needs
state:acceptedfor the acceptance check.Changes
openshell_server_supervisor_session_redirects_total{reason}counts eachSessionRedirecta replica sends,connectwhen it answers a connecting supervisor andshutdownwhen a stopping replica queues a handoff. A handoff that can't be queued doesn't count.openshell_server_supervisor_session_off_ring_admissions_totalcounts redirected supervisors admitted although this replica's ring names another owner, recorded once the session's acceptance is durable. It exists only on gateways that place supervisor sessions by ring: undersupervisor_topology: gateway(feat(server): keep pool sandbox sessions on their supervisor's gateway #5052) the shared supervisor's host moves sessions, a redirected hello is the designed state, and the series is neither exported nor counted. Dashboards that alert on it should tolerate its absence on pool gateways.openshell_server_gateway_ring_membersreports the replicas in this replica's ring. Every ring write goes through one helper that records it under the same lock, and only replicas that join the ring export it.Testing
mise run rust:lint,mise run rust:format:check,cargo test -p openshell-server --features test-support(2664 passed),mise run test:rust:postgres(40 passed),mise run markdown:lint,mise run docs:build:strictpass locally. Pool-regime tests install aSupervisorHostControllerand assert the record and init paths stay silent; the PostgreSQL tests drive a real connect redirect and the membership worker.Checklist