Skip to content

fix: scope socket events and queue UI per user in multiuser mode#9358

Open
lstein wants to merge 10 commits into
invoke-ai:mainfrom
lstein:fix/multiuser-event-scoping
Open

fix: scope socket events and queue UI per user in multiuser mode#9358
lstein wants to merge 10 commits into
invoke-ai:mainfrom
lstein:fix/multiuser-event-scoping

Conversation

@lstein

@lstein lstein commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Supersedes #9314 and #9353, which fixed overlapping parts of the same problem: in multiuser mode, socket events for one user's generations leaked into other users' (especially admins') personal UI — hijacking the gallery's auto-switch, the progress bar, the favicon badge, and queue counts. This PR combines both fixes, resolves their conflicting handling of invocation_complete with the invalidate-only "middle ground" agreed with @JPPhoto, and restructures the frontend event handling so the per-user routing logic is explicit and readable (addressing @JPPhoto's review feedback on legibility).

It also fixes a bug found while smoke-testing this branch: bulk cancels (cancel_all_except_current / delete_all_except_current) emitted no events at all, so other clients' queue badges went stale when an admin canceled their items.

How multiuser event handling works after this PR

Backend: room-based routing (invokeai/app/api/sockets.py)

On connect (_handle_connect), each authenticated socket joins its personal room user:<user_id>; admin sockets additionally join admin. In single-user mode the lone client joins user:system and admin. Queue subscribers also join the queue room (default).

_handle_queue_event routes each queue event by type:

Event Owner Admins Other users (queue room)
invocation_progress full
invocation_started / complete / error full full
queue_item_status_changed, batch_enqueued full full sanitized companion (user_id="redacted", identifiers/errors cleared)
queue_items_retried, queue_items_canceled (new) full, scoped to own item ids full sanitized companion (no ids, no owners)
queue_cleared (user-scoped) full full sanitized companion
queue_cleared (unscoped) full broadcast to the whole queue room

Every owner+admin delivery is a single emit to [user room, admin] — python-socketio dedups recipients across a room list, so a socket in both rooms (an admin who owns the item, or the single-user client, which is always in both) receives exactly one copy instead of processing the event twice.

Sanitized companions exist so non-owners' queue lists and badge totals refetch without seeing whose items changed or what they were. _owner_and_admin_sids() computes the skip_sid list so owners and admins never receive a redacted second copy that would clobber their full-fidelity caches.

Progress events are owner-only because no admin UI consumes other users' progress — this is the primary guard against progress-bar hijacking; the frontend filter below is defense in depth.

Frontend: ownership decided at the listener

getEventScope(getState, data) (src/services/events/eventScope.ts) is the single classifier used everywhere. It returns:

  • 'own' — the current user's event (in single-user mode there is no authenticated user, so every event is 'own' and behavior is identical to before);
  • 'foreign' — another user's full event, which only admin clients ever receive (via the admin room);
  • 'sanitized' — a redacted companion (user_id === "redacted").

It replaces the three near-duplicate predicates (isOwnEvent, isOwnItem, inline 'redacted' checks) that were previously scattered mid-handler.

Classification requires auth.user, so in multiuser mode useSocketIO defers the socket connection until it has hydrated from /me — see the review-fixes section below.

setEventListeners.tsx routes each event at the socket.on callback, before any state is touched, so downstream handlers only ever see events they should act on:

  • invocation_started / invocation_progress / invocation_error: dropped unless 'own'. Foreign events must not reach the workflowExecutionCoordinator, node execution states, or $lastProgressEvent.
  • invocation_complete: 'own' → the coordinator → buildOnInvocationComplete; otherwise → buildOnForeignInvocationComplete.
  • queue_item_status_changed: 'own'buildOnQueueItemStatusChanged; otherwise → buildOnNonOwnerQueueItemStatusChanged.
  • queue_items_canceled (new): tag invalidation for queue caches, plus per-item tags when the event names ids.

Key handlers (each now free of per-user conditionals, since scope is decided upstream):

  • buildOnInvocationComplete (onInvocationComplete.tsx) — own completions only: fetches result image DTOs, optimistically updates gallery caches (board totals, image name lists), auto-switches the gallery to the new image, clears the progress indicator.
  • buildOnForeignInvocationComplete (same file) — the middle ground for admins observing other users. No DTO fetches, no optimistic cache edits, no auto-switch, no progress clear: it dispatches a single invalidateTags(FOREIGN_GALLERY_REFRESH_TAGS) (ImageNameList, BoardImagesTotal, board list, VirtualBoards). RTK Query only refetches invalidated queries with active subscribers, so an admin actively viewing another user's board stays fresh within one round-trip while idle admin clients do no work at all.
  • buildOnQueueItemStatusChanged / buildOnNonOwnerQueueItemStatusChanged (onQueueItemStatusChanged.tsx, extracted from a ~120-line inline handler) — the own path does the optimistic queue cache updates, failure toast, and progress clear; the non-owner path (sanitized companion or admin-room copy) only invalidates queue tags.

Per-user queue counts. SessionQueueStatus gains user_pending / user_in_progress, populated for the requesting user. ProgressBar, the favicon badge (useSyncFaviconQueueStatus), and the invoke buttons use them via useUserHasActiveQueueItems, falling back to the global counts in single-user mode. The invoke buttons now spin while the user has queued sessions, so enqueueing gives feedback even when another user's generation occupies the processor. A full queue_item_status_changed event embeds the owner's fresh per-user counts, so the owner's personal UI updates from the event without waiting for a refetch; the sanitized companion nulls them, and withRetainedUserCounts (queueStatusEvents.ts) retains the last known values for statuses that carry none until the accompanying invalidation refetches them.

Bulk cancel notification (new). cancel_all_except_current and delete_all_except_current previously updated rows in one SQL statement and emitted nothing — only the initiating client (via its own mutation) refreshed. They now emit queue_items_canceled with the affected item ids grouped by owner, routed like queue_items_retried (see table above), so every client's badge and queue list update immediately.

Resolution of the #9314 / #9353 conflict

The two PRs disagreed on foreign invocation_complete: #9353 kept admins' gallery caches live via optimistic updates; #9314 (per review) ignored foreign events entirely, which left an admin's queue tab showing an item completed while the gallery beside it never showed the image. The middle ground adopted here — invalidate-only — gets the freshness of the former at nearly the simplicity of the latter, with no per-image fetches and no partial gating inside handlers.

Code-review fixes (ce65777, 274374c)

A review pass over the finished branch surfaced eight issues, fixed in ce65777:

  • Socket connection now waits for auth hydration. The socket used to connect with the localStorage token as soon as the app mounted, before /me populated auth.user — so a reload during the user's own running generation misclassified their events as 'foreign' in that window, silently dropping one-shot side effects (progress, node execution states, gallery auto-switch, the failure toast) that never replay. useSocketIO now defers connect() until auth.user has hydrated whenever a token is present; the 'foreign' fallback in getEventScope remains as defense in depth.
  • Single-emit delivery everywhere. queue_item_status_changed, batch_enqueued, and the generic queue-item fallback still used two separate emits (user room, then admin room), so a socket in both rooms received and processed every event twice — in single-user mode, that was every client on every status change. They now use the same single emit(room=[user_room, "admin"]) as the invocation events.
  • useIsGenerationInProgress is now user-scoped, so the Dockview progress/canvas tabs no longer react to other users' generations.
  • delete_by_destination no longer double-signals the current item: it is canceled with its own per-item status event, so it is excluded from the bulk queue_items_canceled event (still deleted and counted).
  • The invoke-button spinner respects the full disabled state (queue.isDisabled || !canWriteImages), so it can no longer animate on a disabled button.
  • Queue invalidation tags share one core (QUEUE_CHANGED_TAGS, spread by all five queue event handlers), preventing the hand-copied lists from drifting apart.
  • Narrow invalidation for admin-room full events: the non-owner status-changed handler now uses id-scoped BatchStatus / QueueCountsByDestination tags when the event carries real ids, reserving type-level invalidation for the sanitized companion — so an admin client on a busy queue no longer refetches every batch/destination query on every foreign status change.

A second adversarial review pass over the updated branch (including the fixes above) confirmed the fixes held up and surfaced two more items, fixed in 274374c:

  • In-tab logout/session expiry now tears the socket down. This gap is pre-existing (the socket was never disconnected on an SPA-navigation logout, keeping the old user's room membership — and private events — until the next login's full-page reload), but it belongs to the code this PR reworks. socketOptions is now derived from the redux auth token instead of a one-time localStorage read, making the token a dependency of the connect effect: logout re-runs it, disconnecting the old socket. This also removes the dual token source (redux vs. mount-time localStorage).
  • Corrected a misleading comment on the own-path SessionQueueStatus invalidation: the per-user counts now arrive in the event's embedded queue_status; the refetch is kept because it refreshes .processor (which the optimistic write never touches) and reconciles event drift — the comment previously suggested it existed for the counts, inviting removal as "redundant".

Testing

  • tests/app/test_invocation_event_socketio.py — backend routing for invocation events.
  • tests/app/routers/test_multiuser_authorization.py — sanitized-companion routing for status changes, retries, cancels, and clears (full events stay private; companions carry no identifying data; owners/admins skipped; full events delivered in a single emit).
  • tests/app/services/session_queue/test_session_queue_bulk_cancel_events.py — bulk cancel/delete emit queue_items_canceled grouped by owner (regression test for the stale-badge bug), and delete_by_destination excludes the separately-canceled current item.
  • tests/app/services/session_queue/test_session_queue_status_user_scoping.py — per-user counts.
  • Frontend: eventScope.test.ts, onInvocationComplete.test.ts (own path auto-switches; foreign path invalidates only, never fetches DTOs, never touches selection/progress), setEventListeners.test.ts (cross-user isolation: foreign events don't reach the coordinator/node states/progress; sanitized and foreign status changes are invalidate-only with id-scoped tags for admin full events; single-user mode unaffected), queueStatusEvents.test.ts.
  • Manual two-user smoke test: badge counts per user, no gallery hijack, progress bar/favicon scoped, invoke-button spinner, admin bulk cancel updates other users' badges.

Follow-ups (out of scope)

  • Centralize the hand-maintained sanitization field lists (the router's sanitize_queue_item_for_user plus the model_copy(update=...) companion blocks in sockets.py) into .sanitized() methods on the event models, so a newly added sensitive field cannot silently leak from a missed site.
  • workflowExecutionCoordinator's per-item owner map is now mostly a guard (foreign events are filtered upstream); it could be simplified in a later PR — note the queue_cleared listener is deliberately not scope-filtered, which is what keeps the map load-bearing today.
  • If admin clients watching very busy queues show refetch churn from the invalidate-only gallery refresh, a debounce can be layered on.

🤖 Generated with Claude Code

lstein and others added 6 commits July 16, 2026 07:45
In multiuser mode, invocation events were broadcast to the whole queue
room, leaking every user's activity into everyone's client. Now:

- invocation_progress goes to the owner's user room only
- invocation_started/complete/error emit once to [owner room, admin room]
- model load events carry the owner's user_id and follow the same routing
- queue_items_retried emits the full event to each affected owner and to
  admins, plus a sanitized companion (empty id lists, user_id lists) to
  the rest of the queue room so other users' badge totals refresh;
  _owner_and_admin_sids accepts a collection because a retry batch may
  span multiple users' items

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SessionQueueStatus gains user_pending and user_in_progress, populated
for the requesting user (admins included). The global pending/in_progress
counts include other users' generations in multiuser mode, so user-facing
indicators need counts scoped to the current user. Null in contexts with
no requesting user (socket event payloads); frontend consumers fall back
to the global counts, which is correct in single-user mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two modules to be wired into the socket listeners next:

- eventScope.ts: one canonical classifier for socket event ownership
  ('own' | 'foreign' | 'sanitized'), replacing the near-duplicate
  isOwnEvent/isOwnItem/redacted predicates scattered across handlers
- onQueueItemStatusChanged.tsx: the queue_item_status_changed handler
  extracted from setEventListeners.tsx, split into an own-event path
  (optimistic cache updates, failure toast, progress clear) and a
  non-owner path (queue tag invalidation only), mirroring the
  buildOnInvocationComplete pattern

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Admin clients receive every user's invocation and queue item events via
the "admin" socket room; these must not drive personal UI state. Route
events by ownership at the socket listener, so each handler only ever
sees the events it owns:

- foreign invocation_started/progress/error are dropped (backend already
  routes progress owner-only; this is defense in depth)
- foreign invocation_complete downgrades to tag invalidation only
  (buildOnForeignInvocationComplete): no DTO fetches, no optimistic cache
  edits, no auto-switch, no progress clear. RTK Query only refetches
  invalidated queries with active subscribers, so an admin actively
  viewing another user's board stays fresh while idle clients do no work
- non-owner queue_item_status_changed (sanitized companion or admin-room
  copy) only invalidates queue tags

With ownership decided upstream, buildOnInvocationComplete and the
own-event queue status handler lose their per-user gating and the
workflow execution coordinator never records foreign items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rent user

The progress bar, favicon badge, and invoke buttons used the global
queue counts, so another user's generations made them light up. They now
prefer the per-user counts (falling back to the global counts in
single-user mode), and the invoke buttons spin whenever the current user
has queued sessions - feedback that their session is enqueued even while
another user's generation occupies the processor. Queue status events
carry null per-user counts, so cached statuses retain the last fetched
per-user counts until the accompanying tag invalidation refetches them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cancel_all_except_current and delete_all_except_current update rows in a
single SQL statement and emit no per-item queue_item_status_changed, so
only the initiating client (via its own mutation's tag invalidation)
learned that pending items left the queue. When an admin canceled other
users' items, those users' badge counts stayed stale until an unrelated
event refetched them.

Add a queue_items_canceled event, emitted by both bulk operations with
the affected item ids grouped by owner. Routing mirrors
queue_items_retried: each owner receives the full event scoped to their
own item ids, admins receive all of them, and a sanitized companion (no
ids, no owners) reaches the rest of the queue room so every subscriber's
queue list and badge counts refetch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generated OpenAPI schema lists component schemas alphabetically;
QueueItemsCanceledEvent was hand-spliced after QueueItemsRetriedEvent,
so the openapi-checks and typegen-checks CI jobs failed on the ordering.
Regenerated both files so they byte-match CI's generator output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lstein lstein added the 6.14.x label Jul 16, 2026
@lstein lstein moved this to 6.14.x Theme: USER EXPERIENCE in Invoke - Community Roadmap Jul 16, 2026
lstein and others added 3 commits July 16, 2026 11:20
Correctness:
- Guard the ImageViewer's direct queue_item_status_changed and
  invocation_progress listeners by ownership, so sanitized companions and
  foreign full events no longer blank another user's live progress preview.
- Classify events as foreign while multiuser auth is hydrating (token
  present, user not yet loaded); previously other users' admin-room events
  ran full own-event handling during the /me window after a page reload.
- Restore CurrentSessionQueueItem/NextSessionQueueItem/InvocationCacheStatus/
  BatchStatus/QueueCountsByDestination invalidations for non-owner status
  changes, so admin UI (cancel-current button, canvas queue counts) stays
  fresh on other users' activity.
- Emit queue_items_canceled from all bulk mutations (cancel_by_batch_ids,
  cancel_by_destination, delete_by_destination, cancel_by_queue_id), not
  just cancel/delete_all_except_current.
- Embed the owner's per-user counts in queue_item_status_changed events so
  personal UI (progress bar, spinner, favicon) updates from the event
  instead of stalling on stale counts until a refetch lands.
- Skip admin sids on bulk-event owner-room emits so an admin owner (and
  every single-user socket) receives exactly one copy.
- Skip intermediate images in the foreign invocation-complete handler; they
  never appear in the gallery, so the invalidation was pure waste.
- Fail closed in _handle_queue_event's default branch: unrouted events that
  carry user identity go to owner+admin rooms, never the whole queue room.

Cleanup:
- Share one bulk-event routing helper in sockets.py, one collect-and-emit
  helper in the session queue, and one tag-invalidation helper in the
  socket listeners (retried/canceled were verbatim copies).
- Extract getUserScopedQueueCounts as the single source of the per-user
  count fallback (progress bar, favicon, spinner), and one InvokeButtonIcon
  shared by both invoke buttons.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- defer the socket connection until auth.user hydrates from /me so the
  user's own events cannot be misclassified as foreign during that window,
  silently dropping one-shot side effects (progress, node execution states,
  gallery auto-switch, failure toast) that never replay
- emit queue_item_status_changed, batch_enqueued, and generic queue item
  events to the owner+admin rooms in a single call so a socket in both
  rooms (admin owners; every client in single-user mode) receives one copy
- scope useIsGenerationInProgress to the current user's own counts so the
  progress workspace tabs stop reacting to other users' generations
- exclude the separately-canceled current item from the bulk
  queue_items_canceled event in delete_by_destination; it already emits its
  own per-item status event
- include board write access in the invoke button spinner's disabled state
  so the spinner never renders on a disabled button
- share the queue invalidation tag core across the five queue event
  handlers, and use id-scoped BatchStatus/QueueCountsByDestination tags for
  admin-room full events instead of type-level over-invalidation

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mment

- derive socketOptions from the redux auth token instead of a one-time
  localStorage read, making the token a dependency of the connect effect:
  an in-tab logout or session expiry now disconnects the socket instead of
  leaving it connected with the old user's room membership (and private
  events) until the next full page reload; also reset $socket to null on
  teardown
- correct the SessionQueueStatus invalidation comment: the per-user counts
  now arrive in the event's embedded queue_status, and the refetch is kept
  for the processor status and event-drift reconciliation, not the counts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.x api frontend PRs that change frontend files python PRs that change python files python-tests PRs that change python tests services PRs that change app services

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

2 participants