feat: migrate Global Async Queries onto the Global Task Framework

[villebro]

SUMMARY

Epic feature branch. This PR is the integration target for the multi-step migration of Global Async Queries (GAQ) onto the Global Task Framework (GTF). Individual step PRs are reviewed and merged into gaq-to-gtf; this PR accumulates them and merges into master in one go once the epic is complete. It is kept as a draft tracker until then.

Superset currently has two overlapping background-execution systems:

• GAQ (GLOBAL_ASYNC_QUERIES) — runs chart-data queries in a dedicated Celery task (load_chart_data_into_cache) and notifies the browser through a bespoke Redis Streams transport (AsyncQueryManager + the /api/v1/async_event/ polling endpoint, or the external superset-websocket server). Results are handed back via a cached qc-<hash> query-context descriptor and a result_url.
• GTF (GLOBAL_TASK_FRAMEWORK) — a newer unified background-task abstraction: a @task/.schedule() API, a tasks table, dedup by task_key, progress/timeouts/cancellation, abort handlers, wait_for_completion, a Task List UI, and a REST API.

This epic deprecates GAQ's internal plumbing and re-implements async chart data on top of GTF, while keeping GLOBAL_ASYNC_QUERIES as the operator-facing switch for whether chart queries run asynchronously. Two secondary goals: consolidate coordination primitives (locks, pub/sub, streams) behind one service + one config (DISTRIBUTED_COORDINATION_CONFIG), and harmonize QueryObject serialization on one canonical JSON-safe representation.

Key architectural insight. Per-query results are already the atomic cached unit: QueryContextProcessor.get_df_payload_result(query_obj) executes and caches exactly one QueryObject under its own query_cache_key (which folds in datasource, extra_cache_keys, RLS, and impersonation). The qc-<hash> entry holds no results — it is only a descriptor. So the atomic async unit is the QueryObject keyed by query_cache_key, and per-query dedup by that key is safe across users.

Progress tracker

Every step PR targets gaq-to-gtf.

Step	Scope	Status	PR
0	Coordination Service (locks/pub-sub/streams/KV/await consolidation; non-breaking GLOBAL_ASYNC_QUERIES_CACHE_BACKEND deprecation)	✅ Merged into branch	#43316
1	Coordination cleanup: Pub/Sub → Redis Streams for wait/notify (event-driven, removes the 1s poll; metastore-poll fallback)	✅ Merged into branch	#43409
2	GTF task dependencies (DAG) via task_dependencies junction table (+ chain-icon dependency display in the Task List)	✅ Merged into branch	#43408
3	Canonical QueryObject serialization	✅ Merged into branch	#43410
4	GTF chart-data cutover — per-QueryObject tasks (superset.query_object_v1) + GET /api/v1/task/status_changes cursor poll + client re-request; removes the qc-<hash> wrapper / result_url / /data/<cache_key> replay AND rips out AsyncQueryManager / /api/v1/async_event/ / GLOBAL_ASYNC_QUERIES_CACHE_BACKEND; embedded-guest task visibility. Async is polling-only (WS transport deferred to step 6).	✅ Merged into branch	#43424
5	async_mode per-request opt-in + GLOBAL_ASYNC_QUERIES → GLOBAL_TASK_FRAMEWORK auto-enable + GLOBAL_ASYNC_QUERIES_DEFAULT / per-dashboard override	✅ Merged into branch	#43429
6	Generalize superset-websocket into a general-purpose push transport	🔵 In progress	branch villebro/gtf-websocket
7	Realtime list views via entity-change pub/sub — Task List updates displayed rows live (lossy, per-entity-type Pub/Sub); no new-row insertion. First surface of a general realtime-list pattern (later Dashboards/Charts/Datasets).	⚪ Not started (rides 0–1, 6)	—
8	Official websocket image — build superset-websocket in the main multi-stage Dockerfile and ship it on the official image (launchable from the main image via an alternate entrypoint), so the realtime transport needs no self-built image	⚪ Not started (rides 6)	—

Shipped so far (merged into gaq-to-gtf)

• Step 0 — Coordination Service (refactor(coordination): centralize distributed coordination in a single service #43316). New superset/coordination/ service consolidating distributed locks, pub/sub, streams, and key/value over one DISTRIBUTED_COORDINATION_CONFIG connection, with wait_for_signal/listen_for_signal await-notify helpers. Coordinator resolves DISTRIBUTED_COORDINATION_CONFIG only; GLOBAL_ASYNC_QUERIES_CACHE_BACKEND is deprecated (GAQ-only fallback). GTF TaskManager, the distributed lock, and AsyncQueryManager route through it. Non-breaking.
• Step 1 — Coordination cleanup: Streams-based await/notify (refactor(coordination): reliable Redis Streams await/notify (replace at-most-once pub/sub) #43409). Moves the coordination service's await/notify off at-most-once pub/sub onto Redis Streams: CoordinationService.notify() (XADD, MAXLEN 1 + TTL) plus stream-blocking wait_for_signal/listen_for_signal (event-driven, no polling) when DISTRIBUTED_COORDINATION_CONFIG is set, metastore polling otherwise. GTF task completion/abort emit via notify(). Pub/sub (publish) retained only as an explicit best-effort nudge. New backend primitives xread/stream_last_id/expire; new config DISTRIBUTED_COORDINATION_SIGNAL_TTL (default 24h) bounds signal-stream retention. Removes the ~1s DB-poll-with-backend that the lossy pub/sub design required.
• Step 2 — GTF task dependencies / DAG (feat(gtf): add task dependencies (DAG) with chain-icon Task List display #43408). task_dependencies junction table + migration (FK ON DELETE CASCADE); Task.dependencies self-referential M2M exposing the prerequisite Task entities in one selectin. TaskOptions.depends_on accepts Task entities / UUIDs / strings; block-and-wait scheduler gate with all_success semantics (fails fast, cascades transitively); chain-icon dependency column + "waiting on N prerequisites" indicator in the Task List; depends_on in the REST API; superset-core abstract TaskDependency model. Submit persists edges in O(1) round-trips; the gate does zero extra reads when prerequisites are already terminal.
• Step 3 — Canonical single-query serialization (feat(common): canonical single-query serialization for async chart data #43410). superset/common/query_serialization.py: serialize_query(query_context, index) → JSON-safe payload of the raw query dict + datasource/form_data/result_type/result_format/force/custom_cache_timeout; load_serialized_query(payload) rebuilds a single-query QueryContext via QueryContextFactory (the same path that produced it), so the reconstructed query hashes to an identical query_cache_key. force/custom_cache_timeout (context-level) carried explicitly so they survive per query. The atomic unit Step 4 runs as an async task.
• Step 4 — GTF chart-data cutover (feat(gaq): chart-data cutover — per-QueryObject GTF tasks + task-status polling #43424). Async /chart/data now fans out into one SHARED GTF task per QueryObject (superset.query_object_v1, keyed by query_cache_key); the 202 returns {task_ids} and the client polls GET /api/v1/task/status_changes (cursor-based {uuid: {status, progress}}, TaskFilter-scoped, task_type filter) then re-issues the request from the warm per-query cache. Contribution queries depends_on the totals task. No coordinator task, no qc-<hash> wrapper. Embedded guests get task visibility + cancellation via a token-derived guest_key (task_subscribers.guest_key, folded into the task-dependencies migration). Rips out AsyncQueryManager, the /api/v1/async_event/ REST API, the legacy load_chart_data_into_cache job, and the GAQ JWT/transport config + dedicated GLOBAL_ASYNC_QUERIES_CACHE_BACKEND — coordination is DISTRIBUTED_COORDINATION_CONFIG-only. Frontend transport rewritten (asyncEvent.ts) to the cursor poll (multi-waiter-safe for deduplicated tasks). Kept GLOBAL_ASYNC_QUERIES flag + GLOBAL_ASYNC_QUERIES_POLLING_DELAY. Real-time WS push retired (polling-only) until step 6.
• Step 5 — async opt-in per request + GTF auto-enable (feat(gaq): async chart data opt-in per request (async_mode) + auto-enable GTF #43429). /chart/data runs async only when the request sets async_mode (absent = synchronous 200, so programmatic API clients are unaffected); gated additionally on the flag + full-JSON + caching (ChartDataRestApi._should_run_async). GLOBAL_ASYNC_QUERIES force-enables GLOBAL_TASK_FRAMEWORK at startup. New frontend-only config GLOBAL_ASYNC_QUERIES_DEFAULT (default true). Frontend resolveAsyncMode() policy chain (feature flag → per-dashboard override → deployment default) injects async_mode on full-JSON renders (via an explicit enableAsyncMode opt-in so direct response.json.result readers stay sync); StatefulChart opts in via an injected hook; native-filter requests carry the override too. Per-dashboard override UI in the dashboard Properties modal (json_metadata.async_mode).

Dependency graph

Step 0  Coordination Service
  └── Step 1  Coordination cleanup: Streams await/notify
Step 2  GTF task dependencies (DAG)            ┐
Step 3  Canonical QueryObject serialization    ┘→ Step 4  GTF chart-data cutover
                                                     └── Step 5  AsyncQueryManager notification + flag auto-enable + async_mode
Step 6  Generalize superset-websocket (rides Steps 0–1; can proceed in parallel)
  └── Step 7  Realtime list views via entity-change pub/sub (Task List first)

Design decisions (locked with product owner)

1. Completion transport (fully GTF-native polling). No coordinator task and no firehose: the 202 body is {task_ids}, and the client polls GET /api/v1/task/status_changes (a cursor-based {uuid: {status, progress}} batch, TaskFilter-scoped, filtered to superset.query_object_v1), aggregating the query tasks' own statuses itself — all SUCCESS → re-request (served from the warm per-query cache); any terminal non-success → error. Cancel via POST /api/v1/task/<uuid>/cancel. This lets us remove the bespoke /api/v1/async_event/ REST API and the entire GAQ firehose/AsyncQueryManager. GTF owns its own completion emission (per-task, via the coordination service); the WebSocket transport (Step 6) will subscribe to GTF, not to any GAQ stream. See Realtime transport & channel topology below.
2. Feature-flag interaction: GLOBAL_ASYNC_QUERIES=on force-enables GLOBAL_TASK_FRAMEWORK at startup (with a log line), mirroring the DASHBOARD_RBAC auto-migration precedent.
3. Atomic unit / dedup: one GTF task per QueryObject, task_key = query_cache_key, TaskScope.SHARED (safe cross-user dedup — the key encodes RLS/impersonation).
4. Result reassembly: drop the qc-<hash> descriptor. On completion the client re-issues the same chart-data POST, which now hits the per-query DATA cache and returns synchronously.
5. Cross-query coupling: add a general depends_on capability to GTF; the chart-data orchestrator sets edges only where real coupling exists (contribution dependents → the totals query). Independent queries still run in parallel.
6. Serialization: standardize on a canonical, JSON-safe, self-contained QueryObject representation (to_dict() + json_int_dttm_ser + datasource ref + result_type/result_format/force).
7. Coordination service (prerequisite, shipped in PR 0): unify locks, pub/sub, and streams behind one service + config. GLOBAL_ASYNC_QUERIES_CACHE_BACKEND is deprecated, not removed — non-breaking.
8. Dependency display (PR 1 UI): show DAG edges in the Task List via a compact chain-icon (🔗) popover column mirroring the existing Details popover pattern — the popover lists each prerequisite with its TaskStatusIcon; a "waiting on N prerequisites" indicator surfaces blocked (block-and-wait) tasks. The list stays flat (no tree/expandable rows — server-side pagination + arbitrary sort make subtree grouping fragile); a richer status-colored DAG graph is deferred to a detail drawer once multi-node DAGs exist.
9. Async is opt-in per request (backward-compatible): GLOBAL_ASYNC_QUERIES only makes async available; a request-level async_mode flag (endpoint default false) decides per request. The server treats an absent async_mode as sync, so programmatic /chart/data consumers keep the synchronous 200 flow. The frontend resolves the async_mode it sends via a policy chain — per-dashboard override (Default / Force enabled / Force disabled, in the dashboard properties editor) → deployment default GLOBAL_ASYNC_QUERIES_DEFAULT (frontend-only policy input, default true) → feature-flag gate. Default true keeps the UI's existing async behavior (non-breaking for the UI) while API clients are sync-by-default; operators dial it down globally or per dashboard for gradual rollout.

Follow-on phase — generalize the WebSocket server (PR 7+, distributed into the stepping stones)

Beyond the core migration, generalize superset-websocket from a GAQ-specific event tail into a general server→browser push transport any feature can use (real-time GTF task/progress updates in the Task List, report/alert notifications, extension events), all riding the coordination-service streams. This depends only on PR 0 (already merged), so the work is threaded into the stepping-stone PRs where each already touches the relevant code: 7a + 7c → PR 3 (emit the generic type-tagged envelope via a push_to_channel helper, keeping the GAQ shape as one type), 7b → PR 4 (extract a shared, feature-agnostic channel-token service), 7e groundwork → PR 5 (topic-routable asyncEvent.ts), 7f → PR 6 (Node consumes the generic envelope; drop the legacy type, last). The remaining dedicated step is 7d — real-time GTF task updates in the Task List (push instead of poll), which pairs with PR 1's dependency display. Related: expose coordination to extensions via a superset_core.coordination abstract surface (injected like superset_core.tasks).

Follow-up — Streams-based coordination (cleanup, after PR 1)

Redis Pub/Sub is documented at-most-once / fire-and-forget (a message is "forever lost" on subscriber disconnect), so CoordinationService.wait_for_signal today wakes on a pub/sub nudge but re-checks the DB predicate every _PUBSUB_TICK_SECONDS (=1.0s) as a delivery backstop — i.e. it degrades to ~1s polling even when Redis is up, and each poll currently loads the full Task ORM (two selectin relations) when only status is needed. A dedicated follow-up will move publish_completion/publish_abort onto Redis Streams (persisted, at-least-once — Redis's own recommendation for stronger delivery) with a blocking XREAD from a captured id: genuinely event-driven, race-free, and no busy-poll when Redis is available; the no-backend fallback becomes a status-only batched read (get_status, always-fresh scalar select, also detecting self-cancellation mid-wait); stream growth bounded with MAXLEN/EXPIRE + prune. It touches the shared PR 0 primitive (GTF completion/abort + sync-join), so it gets its own PR + CI. PR 1 deliberately does not block on it (the 1s poll is correct, just not optimal).

Realtime transport & channel topology (strategic direction)

This epic's coordination work sets up a general realtime transport for the whole app, not just GAQ. Two distinct needs, two mechanisms — deliberately kept separate:

• Guaranteed signalling (task waiters, the coordinator joining its query tasks, distributed-lock handoff) → per-task / per-key Redis Streams, already implemented in the coordination service (wait_for_completion blocks on a per-task signal; Step 1). Point-addressed, replayable, at-least-once, no firehose scan. Correctness depends on delivery, so it must be a stream.
• Lossy realtime UI (list views reflecting entity state without a manual refresh) → one Redis Pub/Sub channel per entity type (entity-changes:task, later :dashboard, :chart, :dataset). A mounted list view holds exactly one subscription and filters client-side to the uuids it currently renders. Best-effort is fine: a dropped event just leaves a row briefly stale until the next update or refresh.

Why per-entity-type is the right granularity (rejecting both extremes):

• Per-entity channel (one per row) → the view churns subscribe/unsubscribe on every page change and scroll; N subscriptions constantly opening/closing. Rejected.
• Global firehose (all entity types on one channel) → every list view receives events for entity types it doesn't display and must discard them; wasteful IO that grows with total app activity. This is exactly the legacy GAQ async-events-full pattern — we move away from it, not extend it.
• Per-entity-type → one stable subscription per mounted list view; the only wasted delivery is same-type rows not currently on screen (bounded, trivially filtered). General, reusable, minimal network IO.

Firehose disposition. The GAQ async-events-full firehose and AsyncQueryManager were removed in Step 4; async chart-data is polling-only until Step 6 adds the GTF-native WebSocket transport. New realtime uses the per-entity-type Pub/Sub above. Step 7 is the first concrete consumer of this pattern (the Task List), paving the way to make every list view in Superset realtime.

Step 7 — Realtime list views via entity-change pub/sub (capstone)

Make list views reflect entity state in real time instead of requiring a manual refresh, starting with the Task List (live status / progress / completion rate on currently-displayed rows). Semantics (locked): lossy (per-entity-type Pub/Sub, best-effort); update-only — only rows currently displayed are patched in place; no new-row insertion (too noisy — new entities appear on the next refresh/navigation). Backend emits a change event at the existing Task status-transition / publish_completion chokepoints onto entity-changes:task; the frontend Task List subscribes (via the Step 6 generalized transport), filtering to the uuids on the current page. Establishes the reusable pattern later extended to Dashboards, Charts, Datasets, etc.

Security

No change to the role/capability matrix. Per-query dedup by query_cache_key is safe across users because the key encodes RLS + impersonation + datasource; workers run under override_user. Guest/embedded channels remain HMAC-derived. Each step PR is reviewed against SECURITY.md.

Backward compatibility

The operator switch stays GLOBAL_ASYNC_QUERIES. Async now runs on the Global Task Framework over DISTRIBUTED_COORDINATION_CONFIG. Breaking changes (the whole branch merges to master as a unit, so these land together): Step 4 removes the qc-<hash> result descriptor + /api/v1/chart/data/<cache_key> replay, the bespoke /api/v1/async_event/ REST API, AsyncQueryManager + the GAQ firehose, the GAQ JWT/cookie/transport config, and the dedicated GLOBAL_ASYNC_QUERIES_CACHE_BACKEND (coordination is DISTRIBUTED_COORDINATION_CONFIG-only). Clients poll/cancel via the GTF task API; polling auth is the normal session, embedded-guest visibility is SECRET_KEY-derived. UPDATING.md will carry the operator migration notes. async_mode per-request opt-in (with GLOBAL_ASYNC_QUERIES_DEFAULT / per-dashboard override) lands in Step 5.

TESTING INSTRUCTIONS

Per step PR (see each PR for specifics). End-to-end for the epic: with GLOBAL_ASYNC_QUERIES=on (GTF auto-enabled) and DISTRIBUTED_COORDINATION_CONFIG set, load a dashboard with mixed multi-query + contribution charts; confirm tasks appear in the Task List UI as superset.query_object_v1, the charts resolve via the status_changes poll, cancellation works, and a cached second load short-circuits to 200.

ADDITIONAL INFORMATION

• Has associated issue
• Required feature flags: GLOBAL_ASYNC_QUERIES (auto-enables GLOBAL_TASK_FRAMEWORK); GLOBAL_ASYNC_QUERIES_DEFAULT (frontend async default, true)
• Changes UI (Task List dependency column; dashboard async-mode dropdown)
• Includes DB Migration (PR 1: task_dependencies)
• Introduces new feature or API (async_mode request flag; task depends_on)
• Removes existing feature or API (Step 4: qc-<hash> path + /api/v1/async_event/ REST API)

[netlify]

✅ Deploy Preview for superset-docs-preview ready!

Name	Link
🔨 Latest commit	c7fcebf
🔍 Latest deploy log	https://app.netlify.com/projects/superset-docs-preview/deploys/6a8917697accc00008653eb0
😎 Deploy Preview	https://deploy-preview-43407--superset-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codecov]

Codecov Report

❌ Patch coverage is 79.20792% with 105 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.83%. Comparing base (5812c0e) to head (077c6d3).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/daos/tasks.py	43.75%	25 Missing and 2 partials ⚠️
superset/tasks/async_queries.py	62.26%	20 Missing ⚠️
superset/coordination/utils.py	0.00%	12 Missing ⚠️
superset/tasks/api.py	47.05%	9 Missing ⚠️
superset/coordination/base.py	95.08%	2 Missing and 4 partials ⚠️
superset/commands/tasks/cancel.py	58.33%	3 Missing and 2 partials ⚠️
superset/commands/tasks/submit.py	84.84%	4 Missing and 1 partial ⚠️
superset/tasks/guest.py	64.28%	4 Missing and 1 partial ⚠️
superset/tasks/scheduler.py	73.68%	4 Missing and 1 partial ⚠️
superset/tasks/filters.py	25.00%	2 Missing and 1 partial ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #43407      +/-   ##
==========================================
+ Coverage   78.82%   78.83%   +0.01%     
==========================================
  Files        2876     2879       +3     
  Lines      164459   164342     -117     
  Branches    37956    37985      +29     
==========================================
- Hits       129634   129561      -73     
+ Misses      32378    32341      -37     
+ Partials     2447     2440       -7     

Flag	Coverage Δ
hive	38.17% <35.00%> (+0.09%)	⬆️
mysql	57.75% <52.08%> (-0.02%)	⬇️
postgres	57.78% <52.08%> (-0.03%)	⬇️
presto	40.11% <35.62%> (+0.10%)	⬆️
python	83.51% <78.12%> (-0.03%)	⬇️
sqlite	57.47% <52.08%> (-0.02%)	⬇️
unit	73.59% <75.41%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.