[cuebot] Fix issue on getWhatDependsOn(Frame) (#2278) by aghiles · Pull Request #1 · aghiles/OpenCue

aghiles · 2026-07-23T20:51:01Z

Related Issues

Summarize your change.

86664de Fixes the indentation to make queries legible
ea5abd0 Fixes a bug on the GET_WHAT_DEPENDS_ON_FRAME query. Ready
[cuebot] Missing parentheses in getWhatDependsOn(Frame) SQL changes WHERE clause scope AcademySoftwareFoundation/OpenCue#2277 for more details

Summary by CodeRabbit

Chores
Reformatted internal database access code for improved consistency and
readability. No functional changes or impact to user-facing features.

…ion#2278) ## Related Issues Fixes AcademySoftwareFoundation#2277 ## Summarize your change. - 86664de Fixes the indentation to make queries legible - ea5abd0 Fixes a bug on the `GET_WHAT_DEPENDS_ON_FRAME` query. Ready AcademySoftwareFoundation#2277 for more details  ## Summary by CodeRabbit * **Chores** * Reformatted internal database access code for improved consistency and readability. No functional changes or impact to user-facing features. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2278)

## Summarize your change. Queries were wrapped with `// spotless: off/on` to allow formatting them for legibility. The content of queries has been checked to ensure they contain exactly the same string as before. ## LLM usage disclosure Claude Opus was used to apply the changes and to write a scrip that confirmed the string content is exactly the same.  ## Summary by CodeRabbit * **Style** * Reformatted SQL statement constants across the database access layer for improved code readability and formatting consistency. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2297)

…olumn in two (AcademySoftwareFoundation#2313) ## Related Issues - AcademySoftwareFoundation#2314 ## Summarize your change. Bug: Down host or misreporting hosts can publish free > total (seen on DOWN hosts where /mcp metrics were never refreshed), which makes used negative and the free-fill rect extend past the cell - bleeding the bar's green into the columns to the left. New changes: 1) Bug fix: Temp bar overflow - `_paintDifferenceBar` computed `used = total - free`, which became negative when stale hosts published `free > total`. - The free-fill rect was then built with `rect.adjusted(<negative>, 0, 0, 0)`, shifting its left edge beyond the cell boundary. - Since the Temp delegate paints last in the row, its green free portion bled leftward across Idle Memory, Total Memory, GPU Memory, Physical, and Swap, causing the "green bar spanning many columns". - Clamp `used` to `[0, total]` so misreporting hosts render as fully free (all green) without overflowing the cell. - Add `painter.setClipRect(rect)` as a safeguard so future arithmetic issues cannot paint outside the intended cell. - Tighten the early-return guard to also bail when `total <= 0`. 2) New improvement: "Temp Free" column split - The combined "Temp Free" column value (e.g.: 23.5G (50%)) display made it difficult to sort by absolute free space. Sorting by percentage could rank a `1.0G / 100%` host above a `900G / 45%` host, which is counterproductive when scanning for actual available headroom. - Replace the single Temp Free column with two: a) "Temp Free" (e.g. `"23.5G"`), sorted by "free_mcp" and b) "Temp Free %" (e.g. `"50%"`), sorted by usage ratio and left empty when "total_mcp" is unknown - The adjacent Temp bar continues to sort by ratio for visual continuity. - `_formatTempCell` is replaced by `_formatTempFreeAmount` and `_formatTempFreePercent`. 3) Column width tuning: Increase default widths for columns that were truncating content under production fonts: - GPU Memory - Total Memory - Idle Memory - Temp Free - Temp Free % - Idle Cores - Idle GPUs - GPU Mem - GPU Mem Idle - Ping - Hardware - Locked - ThreadMode - Header labels and common values now fit without clipping or hover-only visibility, making Monitor Hosts easier to scan. 4) Update tests: - Replace `_formatTempCell` tests with dedicated helper coverage: - Amount renders even when total is unknown - Percentage rounds to the nearest integer - Percentage remains empty when total is unavailable - Add a delegate-wiring assertion for the new `Temp Free %` column at index `10`. Docs: - Update `cuecommander-administration-guide.md` to document all three Temp-related columns (`Temp`, `Temp Free`, `Temp Free %`) and their sorting behavior.

…ior (AcademySoftwareFoundation#2316) ## Related Issues - AcademySoftwareFoundation#2314 ## Summarize your change. Revise the 'Temp Free' and 'Temp Free %' columns tooltip to reflect percentage-based /mcp/ free space reporting, ratio-based sorting, and empty values when total /mcp/ size is unavailable.

…wareFoundation#2321) The windows-tests job was timing out because every PR rebuilt all of rqd's transitive deps from scratch on a slow Windows runner, with no job timeout (default 6h). .github/workflows/rust-pipeline.yml: - Add Swatinem/rust-cache@v2 to all four jobs so target/ and the registry are reused across runs (typically 5-10x faster on Windows after the first build). - Add timeout-minutes per job (30-60 min) so stalled runs fail fast. - Set CARGO_INCREMENTAL=0 (no upside on ephemeral CI, hurts clean builds) plus CARGO_NET_RETRY / RUSTUP_MAX_RETRIES = 10 for Windows registry flakiness. - Split windows-tests into `cargo test -p rqd --no-run` then `cargo test -p rqd` so compile-vs-run timing is obvious and the cache lands even when a test fails. - Drop `--verbose` from the Windows jobs . Log volume noticeably slows Windows runners.  ## Summary by CodeRabbit * **Chores** * Updated CI/CD pipeline configuration to improve build reliability and performance through enhanced caching mechanisms and retry configurations.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2321)

Fixes AcademySoftwareFoundation#2311  ## Summary by CodeRabbit * **Bug Fixes** * Corrected exit signal computation for Docker containers when exit codes exceed standard thresholds. * Fixed exit-status interpretation on Unix systems for proper signal detection. * **Tests** * Added validation tests for exit-status handling across container and Unix environments. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2312)

The application os not properly closing the grpc channel at shutdown, leaving this SEVERE warning: ``` Channel ManagedChannelImpl{logId=2787, target=elk0815:8444} was not shutdown properly ``` Explanation: `RqdClientGrpc` (cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java:71-89) holds a Guava `LoadingCache<String, ManagedChannel>`. The `removalListener` calls `conn.shutdown()` **on cache eviction** — but never on application shutdown. The bean has no `destroy-method` (applicationContext-service.xml:39, no destroy hook), and `RqdClientGrpc` has no `shutdown()` method at all. So when cuebot stops: - The cache is GC'd - Each `ManagedChannel` is finalized **without** `shutdown()` having been called - gRPC logs `SEVERE: Channel ... was not shutdown properly!!!` per leaked channel, with the stack capturing where the channel was first allocated (the `RuntimeException: ManagedChannel allocation site` — that's a diagnostic stack, not a real exception) ## LLM usage disclosure Claude Opus was used for investigating the origin of the log and proposing a fix  ## Summary by CodeRabbit * **Bug Fixes** * Improved application shutdown procedure to properly clean up network resources and prevent potential resource leaks during shutdown. * Enhanced shutdown sequencing to ensure proper initialization and cleanup order of internal services. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2274)

…d Frame (AcademySoftwareFoundation#2325) ## Related Issues - AcademySoftwareFoundation#2319 ## Summarize your change. Exposes when an object became eligible to run (left DEPEND for WAITING, or the job's submission time when never blocked) so callers can measure how long a frame waited to be picked up by a render proc: wait_for_pickup = frame.startTime() - frame.eligibleTime() Proto files - Adds `eligible_time` field to Frame, Job, NestedJob, and Layer. Cuebot - V41 migration adds `ts_eligible` column to the frame, layer, and job tables. - `layer.ts_eligible` and `job.ts_eligible` default to `current_timestamp`; existing rows are backfilled from `job.ts_started`. - Extends the existing DEPEND -> WAITING trigger so it stamps `frame.ts_eligible` whenever a frame unblocks. - V42 migration adds a SETUP -> WAITING trigger to stamp `ts_eligible` on the path frames actually take through Cuebot (frames are inserted as SETUP and bulk-transitioned to WAITING by `JobManagerService.activateJob`, so V41's BEFORE INSERT trigger never fires in practice). - V42 also backfills `ts_eligible` for frames that already made the SETUP -> WAITING jump before this migration ran. - `WhiteboardDaoJdbc` and `NestedWhiteboardDaoJdbc` map the column into the proto's `eligible_time` via a `getEligibleTimeInEpoch` helper, falling back to the job's submission time when `ts_eligible` is NULL (frames still in DEPEND). PyCue - Adds `Frame.eligibleTime()`. - Adds `Job.eligibleTime(format=None)` and `Layer.eligibleTime(format=None)`, mirroring the format-string behavior of `Job.startTime()`. - Propagates `eligible_time` through `NestedJob.asJob()`. - New unit tests cover all four wrappers. CueGUI - Adds an "Eligible Time" column to `FrameMonitorTree`. - Adds an "Eligible" column to `JobMonitorTree` and `LayerMonitorTree`. - "Eligible" was chosen over "Available" because the latter can imply the object will run next, when other gates (resources, paused state, etc.) may still block dispatch. Docs - Adds `eligibleTime` to the Job and Frame REST API reference schemas and example payloads. - Documents the new monitor-tree columns in the Cuetopia monitoring guide.

…hinx_docs.sh (AcademySoftwareFoundation#2334) ## Summarize your change. - The `aswf/ci-opencue:2023` container has no `pip` binary on PATH, so the job failed with `pip: command not found`. - Switch both `pip install` calls to `python -m pip install`, matching `ci/run_python_tests.sh`.

…SRF) (AcademySoftwareFoundation#2332) ## Related Issues - AcademySoftwareFoundation#2333 ## Summarize your change. - Upgrade `next` from ^14.2.35 to ^15.5.18 (resolves to 15.5.18) to fix GHSA-c4j6-fc7j-m34r: self-hosted Next.js Node server can proxy crafted WebSocket upgrade requests to arbitrary internal/external destinations, enabling SSRF against cloud metadata endpoints and internal services. - Upgrade `eslint-config-next` from 14.0.4 to ^15.5.18 to align with the Next.js major version. - Regenerate `package-lock.json` so `npm ci` in Docker resolves the patched dependency tree correctly. - Address Next.js 15 breaking change: `next/dynamic` no longer supports `ssr: false` inside Server Components. Moved client-only `DataTable` dynamic import into `app/jobs/data-table-client.tsx`, marked with `"use client"`, and imported it from `app/page.tsx` to preserve behavior. - Verified `npm run build` and clean `docker compose build --no-cache cueweb` both succeed; container starts and serves correctly on :3000. Not affected: CueWeb does not use `next/headers`, `next/cache`, middleware, `unstable_*` APIs, or async `params`/`searchParams`, so no additional Next 15 migration changes were required. See: https://cybersecuritynews.com/next-js-vulnerability-exposes-credentials/ Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com> Co-authored-by: Filipe Lacerda <tifilipebr@gmail.com>

…demySoftwareFoundation#2335) ## Related Issues Fixes AcademySoftwareFoundation#2284 ## Summary Adds a bell icon per job row in the cueweb jobs table. Clicking subscribes the browser to a system notification when the job reaches FINISHED. State is stored in localStorage. An app-wide poller checks subscribed jobs every 15 seconds and fires browser notifications via the Web Notifications API. The bell has three visual states: - **Outline bell**: not subscribed → click to subscribe - **Filled bell**: subscribed, waiting → click to cancel - **Filled bell + green dot**: notification fired → click to clear The bell is disabled (faded, with tooltip) on jobs that are already FINISHED when first viewed. ## Screenshots ### 1. First-click permission prompt <img width="800" alt="browser permission prompt on first subscribe" src="https://github.com/user-attachments/assets/69544b4a-b219-4fd0-8de9-4448f7760ffc" /> ### 2. Subscribed (bell turns filled) <img width="800" alt="bell shows filled BellRing icon after subscribe" src="https://github.com/user-attachments/assets/22083793-6a8a-4f66-831c-d115e73bcdf1" /> ### 3. Notified (OS notification fires, bell shows green dot) <img width="800" alt="bell shows filled + green dot after job finished and notification fired" src="https://github.com/user-attachments/assets/094a409e-6f48-45b4-9681-21abcc497ce3" /> ### 4. Disabled bell on already-finished jobs <img width="800" alt="bell faded on FINISHED jobs, tooltip explains why" src="https://github.com/user-attachments/assets/aed24fbc-4e76-4297-9eef-d1edc4461951" /> ### 5. Permission denied (toast refusal) <img width="800" alt="toast warning when browser notifications are blocked" src="https://github.com/user-attachments/assets/33bea528-4ff1-4ea0-ab21-7837f97bbddf" /> ## LLM usage disclosure Assisted-by: Claude / Opus 4.7 Used for implementation planning, initial code drafting, and writing unit tests.  ## Summary by CodeRabbit * **New Features** * Bell icon in the jobs table to subscribe/unsubscribe to job completion notifications. * Background poller that checks subscribed jobs and triggers browser notifications when jobs finish. * Subscriptions persist across sessions and can be managed from the UI. * Permission request for browser notifications with user-facing warning if denied. * **Tests** * Added comprehensive tests covering subscription CRUD, defensive parsing, and notification selection logic.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2335?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)   --------- Signed-off-by: Michael Vallido <vallido.michael@gmail.com>

… another user (AcademySoftwareFoundation#2340) ## Related Issues Fixes AcademySoftwareFoundation#2324 ## Summarize your change. This PR adds a **Take Ownership** action to the CueGUI Hosts context menu so users can reclaim NIMBY-locked hosts that are currently deeded to someone else, without requiring manual pycue scripting or admin intervention. ## Main changes: 1. New Hosts action in CueGUI (`cuegui/MenuActions.py`, `cuegui/HostMonitorTree.py`) - **Take Ownership** entry in the Hosts right-click menu. - Enabled only when exactly one host is selected and that host is `NIMBY_LOCKED` (matches the backend's `OwnerManagerService.takeOwnership` invariant). - Re-checks `lock_state` inside the action itself as defense-in-depth against a stale selection. 2. Explicit ownership-transfer UX - Prompts for the username that should own the host (defaults to `getpass.getuser()`). - Resolves the owner via `opencue.api.getOwner(name)`; if the username doesn't exist yet, lazily creates the owner record via `findShow(SHOW or "pipe").createOwner(name)`, same pattern as `LocalBookingWidget.deedLocalhost`. - Looks up the host's current deed and: - If the host has **no current deed** -> skip the confirmation entirely (nothing to transfer from). - If the host is owned by **another user** -> confirm with `Host <name> is currently owned by <user>. Take ownership?`. - If the host is **already owned by the requested user** -> no-op confirmation. - If the deed lookup **fails** for an unexpected reason -> log the exception and prompt with `Host <name> ownership could not be determined. Take ownership?`. - On confirm, calls `owner.takeOwnership(host_name)` and refreshes the host row. - Owner creation happens **after** confirmation (verified by a dedicated test) so a cancelled prompt never leaves a stray owner record behind. - Errors at any stage (getOwner, createOwner, takeOwnership) surface through `cuegui.Utils.showErrorMessageBox` instead of leaking gRPC tracebacks. 3. Reused existing backend; completed missing client-side plumbing - Uses existing `OwnerInterface.TakeOwnership` flow, no backend API change required. - The backend already replaces any existing deed atomically (`deedDao.deleteDeed(host)` -> `deedDao.insertDeed(owner, host)` in `OwnerManagerService.takeOwnership`). - Added a new pycue wrapper `Host.getDeed()` (`pycue/opencue/wrappers/host.py`) so CueGUI can read the current owner via the deed before confirmation. The underlying gRPC `HostInterface.GetDeed` was already implemented in `ManageHost.java`; only the Python-side wrapper was missing. 4. Tests (`cuegui/tests/test_menu_actions.py`) Added `HostActionsTests` coverage for: - `test_canTakeOwnership`: NIMBY-only enablement gate. - `test_takeOwnership`: Happy path with cross-user confirmation. - `test_takeOwnership_missingOwnerCreatesAfterConfirm`: Verifies `createOwner` is **only** called after the user confirms. - `test_takeOwnership_deedLookupFailureStillPrompts`: Generic deed-lookup failure still prompts so the user can decide. - `test_takeOwnership_unownedHostSkipsConfirmation`: `EntityNotFoundException` (host has no deed) is distinguished from a generic lookup failure, and the confirmation dialog is skipped. - `test_takeOwnership_ownerLookupFailure`: Owner lookup failure surfaces an error dialog without proceeding. - `test_takeOwnership_ignored_for_non_nimby`: Non-NIMBY host is a no-op even if the action is somehow invoked. Focused test slice runs cleanly: QT_QPA_PLATFORM=offscreen python -m pytest cuegui/tests/test_menu_actions.py -k HostActionsTests 5. Documentation (`docs/`) - `docs/_docs/user-guides/cuecommander-administration-guide.md`: Monitor Hosts -> Manage Host States: added the new action with its NIMBY-only gate and confirmation behavior. - `docs/_docs/tutorials/using-cuegui.md`: Added **Take Ownership (NIMBY-locked only)** to the right-click host menu tree. - `docs/_docs/reference/CueGUI-app.md`: Added the action to the Managing hosts "common actions include" list. - pycue Sphinx docs auto-pick up `Host.getDeed()` from its docstring via `automodule :members:`: no manual `.rst` edit needed. ## Why? The backend already supports atomic ownership replacement, but CueGUI had no UI surface for reclaiming a host owned by another user, the only workaround was a pycue shell session or an admin request. This closes that usability gap directly in the Hosts workflow. ## LLM usage disclosure: Olaiwon Ismail Model used: GPT-5.3-Codex (GitHub Copilot) Usage: - Helped me understand the codebase - Analyzed existing CueGUI host action architecture and pycue ownership wrappers - Assisted with syntax and boilerplate generation for the new host action and UI gating logic - Generated unit test scaffolding to execute the focused test slice Co-authored-by: Olaiwon Ismail <olaiwonismail@gmail.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com>

…SoftwareFoundation#2336) ## Related Issues - AcademySoftwareFoundation#2287 ## Summarize your change. The existing `Age` column in the jobs table displays time in HHH:MM format (e.g., `048:32`), which requires mental conversion to understand at a glance. I added a new `Readable Age` column that shows the same datum in a more natural format: - Jobs under a day: `2h 14m` - Jobs over a day: `3d 4h` The column is hidden by default and can be enabled through the column chooser dropdown, so existing workflows aren't disrupted. The header pairs with the existing `Age` column to make the relationship (same value, different format) obvious. Implementation notes: - New formatter `secondsToHumanAge` in `cueweb/app/utils/layers_frames_utils.ts` - New column `readable age` in `cueweb/app/jobs/columns.tsx` with a numeric `sortingFn` so rows sort by actual elapsed seconds (not the formatted string) - `getJobAgeInSeconds` clamps to non-negative and floors to whole seconds, so the sort key always matches what the formatter displays (addresses CodeRabbit feedback on clock-skew / fractional-second edge cases) - Docs updated: new row in the Job Information Columns table in `docs/_docs/user-guides/cueweb-user-guide.md` Testing: - Formatter edge cases verified: negative values, zero, minutes-only, hours-only, multi-day jobs - Confirmed the column appears in the chooser, is hidden by default, and sorts correctly by actual age in seconds - Verified end-to-end in the sandbox stack (`docker compose --profile all up`) Co-authored-by: Vishal Kumar Singh <vishal.kr.singh2021@gmail.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com>

## Summary - add a hover tooltip to CueWeb job progress bars with per-state frame counts and percentages - centralize progress segment and tooltip calculations in a utility - cover progress percentages, tooltip rows, and zero-frame jobs with unit tests Fixes AcademySoftwareFoundation#2285 ## Testing - npm test -- --runTestsByPath app/__tests__/api/utils/job_progress_utils.test.ts - npx tsc --noEmit - git diff --check

## Related Issues - AcademySoftwareFoundation#2290 ## Summarize your change. [cueweb] Add frame state filter chips - Add frame-state filter chips above CueWeb frame tables with per-state counts - Support OR-based filtering for selected frame states - Persist selected frame states in the `frameStates` URL query parameter - Add unit tests covering frame state counts and filtering behavior [cueweb] Improve frame state filter parsing and pagination behavior - Trim whitespace and deduplicate values when parsing the `frameStates` URL parameter, ensuring URLs like `?frameStates=WAITING, RUNNING` correctly preserve valid states - Reset pagination to page 1 whenever frame state filters change, preventing empty result pages after narrowing filters - Preserve the current page during polling-based refreshes via `autoResetPageIndex: false` [cueweb/docs] Document job progress tooltip and frame state filter chips - Update the user guide, reference, tutorial, quick-start, additional guides, and CueWeb README - Document the job progress bar tooltip (AcademySoftwareFoundation#2331), including per-state frame counts and percentages - Document frame state filter chips (AcademySoftwareFoundation#2330), including: - Per-state counts - OR-combined filtering behavior - URL persistence through the `frameStates` query parameter - Whitespace-tolerant and deduplicated parsing - Pagination reset behavior when filters change ## Testing - npm test -- --runTestsByPath app/__tests__/api/utils/frame_columns.test.ts - npx tsc --noEmit - git diff --check Co-authored-by: Mukunda Katta <mukunda.vjcs6@gmail.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com>

…ademySoftwareFoundation#2327) When there's only one active show, the logic to randomize show order inadvertently removes it from the list, leading to a frozen queue.  ## Summary by CodeRabbit * **Bug Fixes** * Improved shuffle behavior so host dispatching now randomizes show order correctly when shuffle is enabled. * **Tests** * Updated unit test expectations to reflect the corrected dispatch behavior.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2327)

…SoftwareFoundation#2343) ## Related Issues - AcademySoftwareFoundation#2344 ## Summarize your change. - Replace QStyle.CE_ProgressBar in ProgressDelegate with manual QPainter rendering (dark background, green chunk, centered text) - macOS aqua style ignores opts.rect inside item-view delegate paints and renders its thin animated indicator at the row's leftmost cell, painting a stray blue line over the Name column - Manual painting matches the cross-platform approach already used by JobProgressBarDelegate, so the Progress column now renders consistently on macOS, Linux, and Windows - Clamp progress to [0, 100] to guard against backend over/underreports

…reFoundation#2346) ## Related Issues - AcademySoftwareFoundation#2347 ## Summarize your change. - Bump @sentry/nextjs from ^8.52.0 to ^10.53.1 to resolve high-severity vulnerabilities in rollup (path traversal) and the Sentry dependency chain. - Run npm audit fix to update transitive dependencies, resolving: - form-data (critical: unsafe random boundary) - @babel/plugin-transform-modules-systemjs (arbitrary code generation) - flatted (DoS / prototype pollution) - lodash (code injection, prototype pollution) - minimatch 3.x/5.x/8.x/9.x (ReDoS) - picomatch 2.x/4.x (ReDoS, method injection) - serialize-javascript (RCE, DoS) - terser-webpack-plugin (via serialize-javascript) Build, type-check, and full test suite (36/36) pass.

…areFoundation#2348) ## Related Issues - AcademySoftwareFoundation#2284 ## Summarize your change. PR AcademySoftwareFoundation#2335 added the per-job subscribe bell to the CueWeb Jobs table but shipped without doc updates. Add coverage across all cueweb doc surfaces that enumerate UI features, columns, or developer components so the feature is discoverable from every entry point. - user-guides/cueweb-user-guide.md: Add Notify column to Job Information Columns and a Job-finished notifications subsection under Real-time Updates and Monitoring covering the three bell states, the permission prompt, the 15s poll cadence, localStorage persistence, and auto-cleanup of deleted jobs. - developer-guide/cueweb-development.md: Register JobSubscriptionPoller under Core Components and SubscribeBell under UI Components; add a Subscription store note covering the cueweb:job-subscriptions key, the cueweb:subscriptions-changed event bus, and the defensive parser. - reference/cueweb.md: Add Notify column to the Jobs Table and a behavior table covering trigger, polling, notification, persistence, auto-cleanup, and cross-component sync. - quick-starts/quick-start-cueweb.md: Add Job-finished Notifications bullet to Expected Interface and step 7 to Frame Operations. - tutorials/cueweb-tutorial.md: Add a Subscribe to Job Completion step to Viewing Your Jobs and a bullet to Auto-refresh Settings.

…cademySoftwareFoundation#2349) ## Related Issues - AcademySoftwareFoundation#2279 ## Summarize your change. Replicates the CueGUI Comments dialog (cuegui/cuegui/Comments.py) in CueWeb: list, add, edit, and delete per-job comments, plus per-browser predefined-comment macros. - New page at app/jobs/[job-name]/comments with comment list, sanitized markdown preview, editor, and New / Save / Delete actions. Edit and delete are gated by author check. - Predefined macro CRUD stored in localStorage (cueweb-comment-macros), matching CueGUI's Add / Edit / Delete predefined comment workflow. - Proxy routes: POST /api/job/getcomments -> JobInterface/GetComments POST /api/job/action/addcomment -> JobInterface/AddComment POST /api/comment/action/save -> CommentInterface/Save POST /api/comment/action/delete -> CommentInterface/Delete - Helpers getJobComments / addJobComment / saveJobComment / deleteJobComment in app/utils. - Sticky-note indicator next to job names when Job.hasComment is true, with username threaded through TanStack Table meta so the indicator click opens the page with the right author context. - "Comments" entry added to the job-row context menu. - Markdown rendered via react-markdown + rehype-sanitize. Docs updated across user guide, reference, REST API reference, developer guide, tutorial, other-guides, quick start, and concepts.

…ySoftwareFoundation#2350) ## Related Issues - AcademySoftwareFoundation#2351 ## Summarize your change. Adds the standard OpenCue Apache 2.0 license header to all CueWeb source files (TS, TSX, JS, JSX, CSS) that were missing it, using the JS/TS-equivalent /* ... */ block comment form of the canonical Python header. - 107 files updated across app/, components/, lib/, public/workers/, jest/, root config (next.config.js, tailwind.config.{js,ts}, postcss.config.js, jest.config.js), and the Sentry config entries. - For files that begin with a "use client" / "use server" / "use strict" directive, the directive remains on line 1 (Next.js / V8 requirement) and the header is inserted immediately below it. - app/__tests__/utils/subscription_utils.test.ts keeps its `@jest-environment jsdom` docblock as the first comment so Jest still picks up the environment override; the license header sits below it. - next-env.d.ts is intentionally skipped, it is auto-generated and carries an explicit "should not be edited" note. No behavior changes. Type check and the full Jest suite (42 tests) pass; the cueweb Docker image builds clean.

…mySoftwareFoundation#2337) ## Related Issues - AcademySoftwareFoundation#2326 ## Summarize your change. Adds `submissionTime()` to `Frame`, exposing the parent job's submission timestamp directly on the frame object. This avoids requiring callers to fetch the parent job or overload `eligibleTime()` to infer submission time. `Frame.startTime()` represents when the frame began executing on a render host, not when the job was submitted. Since `Job.startTime()` and `Layer.startTime()` already serve as submission timestamps for those objects, only `Frame` needed this additional accessor. With `submissionTime()`, callers can now compute frame lifecycle timing directly from a single `Frame` object: - `blocked_by_depends = frame.eligibleTime() - frame.submissionTime()` - `blocked_by_pickup = frame.startTime() - frame.eligibleTime()` - `total_turnaround = frame.stopTime() - frame.submissionTime()` Proto files - Adds `submission_time` field to `Frame`. Cuebot - Updates `WhiteboardDaoJdbc.FRAME_MAPPER` to populate `submission_time` from the existing `job.ts_started` join (already aliased as `job_ts_started` for the `eligibleTime()` fallback). - No database migration is required, since the source column already exists. PyCue - Adds `Frame.submissionTime()`. - Includes a new unit test covering the Python wrapper. CueGUI - Adds a new "Submission Time" column to `FrameMonitorTree`, positioned next to the existing "Eligible Time" column. - Re-anchors the `*_COLUMN` visual-index constants in `FrameMonitorTree`, which had drifted from their intended columns over years of insertions (last touched in 2018). The new "Submission Time" column made the staleness visible by shifting `LASTLINE_COLUMN` further out of place: - `PROC_COLUMN`: 5 -> 6 (was pointing at GPUs; now correctly points to Host). - `CHECKPOINT_COLUMN`: 7 -> 8 (was pointing at Retries; now correctly points to the icon-only `_CheckpointEnabled` column where the checkmark decoration belongs). - `RUNTIME_COLUMN`: 9 -> 10 (was pointing at the hidden `_CheckpointEnabled`; now correctly points to Runtime). - `MEMORY_COLUMN`: 11 -> 12 (was pointing at LLU; now correctly points to Memory (RSS)). - `LASTLINE_COLUMN`: 15 -> 20 (was pointing at Remain; now correctly points to Last Line). - As a side effect, `redrawRunning()` now emits `dataChanged` over the correct Runtime -> Last Line range, restoring smooth repaints for Runtime/Memory/Last Line cells on running frames. The `PROC_COLUMN` foreground-color and alignment, plus the `CHECKPOINT_COLUMN` icon decoration, also land on the right cells now. - Adds a header comment noting these are visual indices that must be updated in lockstep when columns are inserted, removed, or reordered. Docs - Adds `submissionTime` to the Frame REST API reference schema and example payloads. - Updates the Cuetopia monitoring guide to document the new column. VERSION.in - Bumped up to 1.22

…Foundation#2352) The satisfy logic that runs to clean up stale depends would catch both EATEN and SUCCESS frames as a sign its dependents should be cleaned, but this behavior is only acceptable when `depend.satisfy_only_on_frame_success` is false.  ## Summary by CodeRabbit ## Release Notes * **New Features** * Added `depend.satisfy_only_on_frame_success` configuration flag to control how EATEN frames are treated during dependency recovery. When enabled (default), only SUCCEEDED frames satisfy dependencies; when disabled, EATEN frames also count as completion.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2352?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

…cademySoftwareFoundation#2341) ## Related Issues - AcademySoftwareFoundation#2335 ## Summarize your change. Builds on top of the per-job subscribe bell (AcademySoftwareFoundation#2335): - Replace the OS browser Notification API with the existing react-toastify channel (toastSuccess). Removes the first-click permission prompt; the bell now subscribes/unsubscribes immediately. Drops requestNotificationPermission() and the "permission denied" toast warning from the bell click handler. - SSR-guard getSubscriptions(): return {} when window is undefined, matching subscribeToChanges and the (now removed) requestNotificationPermission helper. - Cross-tab sync in subscribeToChanges(): also listen for the browser 'storage' event so a mutation in one tab updates bells in other open tabs. - Harden the poller tick: * Wrap each getJob() call in try/catch so one failed fetch does not reject Promise.all and silently lose the tick. * Wrap the whole tick body in try/catch so failures show up as a console.error instead of an unhandled rejection. * Re-read each entry from localStorage right before firing and skip if notifiedAt is no longer null. Narrows the cross-tab race window where two tabs both pick the same FINISHED entry and toast twice.

…reFoundation#2328) The scheduler was inadvertently booking non-threadable jobs on threadable machines. Going against cuebot's behavior.  ## Summary by CodeRabbit * **Refactor** * Streamlined thread-mode validation logic in the scheduler's core reservation and host matching systems. * Removed redundant code paths and consolidated validation logic. * **Tests** * Enhanced test coverage for thread-mode compatibility validation across different thread-mode configurations.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2328)

…yer (AcademySoftwareFoundation#2338) ## Related Issues - AcademySoftwareFoundation#2339 ## Summarize your change. [cuebot/proto/pycue/cuegui/docs] Add Layer startTime() and stopTime() Layer previously had no start/stop time accessors. Callers that needed the execution window for a layer had to fetch every frame and compute MIN/MAX client-side, which was wasteful in CueGUI (N extra RPCs per refresh for a job with N layers) and unavailable to tools that only queried layers. This change denormalizes layer timing onto `layer_stat` and exposes the values directly on the Layer API: - `Layer.startTime()` = `layer_stat.ts_started` (stamped on first entry to RUNNING) - `Layer.stopTime()` = `layer_stat.ts_stopped` (stamped on each exit from RUNNING, but surfaced as 0 until every frame on the layer has stopped) `stopTime()` remains `0` while any frame is still pending, running, or in DEPEND, mirroring `Job.stopTime()` so callers can use the same "is it done?" idiom consistently. Proto - Add `start_time` (field 24) and `stop_time` (field 25) to the `Layer` message. Cuebot - Add `ts_started` and `ts_stopped` (`TIMESTAMP WITH TIME ZONE`) to `layer_stat`. - Maintain both values through the existing `trigger__update_frame_status_counts` trigger (`AFTER UPDATE ON frame`): * entry to RUNNING stamps `ts_started` from `NEW.ts_started` via `COALESCE(...)` (first-writer-wins; retries do not update it) * exit from RUNNING stamps `ts_stopped` from `NEW.ts_stopped` (latest-writer-wins) - Copy timestamps from the updated frame row instead of sampling `current_timestamp`, ensuring `layer_stat.ts_stopped` exactly matches `MAX(frame.ts_stopped)`. - Preserve `COALESCE(..., current_timestamp)` as a fallback for callers that change state without explicitly updating timestamps. - Update `GET_LAYER` and `GET_LAYER_WITH_LIMITS` to read `layer_stat.ts_started` and `layer_stat.ts_stopped` directly, replacing two correlated frame-table aggregates per layer query. - Move "stopTime stays 0 until all frames are done" logic into `WhiteboardDaoJdbc.LAYER_MAPPER` using existing counters: `int_waiting_count + int_running_count + int_depend_count == 0` - Align layer timing behavior with existing denormalized `job.ts_started` / `job.ts_stopped`. - Add V43 migration to create and backfill both columns from existing frame aggregates, with no manual follow-up required. PyCue - Add `Layer.startTime(format=None)` and `Layer.stopTime(format=None)`, matching `Job.startTime()` formatting behavior. - Add unit tests for both epoch and formatted outputs. CueGUI - Add "Start Time" and "Stop Time" columns to `LayerMonitorTree`, alongside the existing "Eligible" column. Docs - Add `startTime` and `stopTime` to the Layer example payload in the REST API reference. - Document the new monitor-tree columns in the Cuetopia monitoring guide. VERSION.in - Bump version to 1.23.

CueJobMonitorTree was fetching the same job data twice every 22s: once via cached getJobWhiteboard, then once per group via an uncached getJobs(id=...). On a show with N populated groups, one tick cost 1 + N gRPC round-trips and SQL executions. Changes: - Add `repeated Job inline_jobs = 18` to NestedGroup (strictly additive; `repeated string jobs` retained). Cuebot populates it from the existing GET_NESTED_GROUPS row data — the only column added to the SELECT is `str_loki_url`. Reuses WhiteboardDaoJdbc.JOB_MAPPER. - Bump whiteboard cache TTL 5s→10s and add per-show single-flight (synchronized-on-show), so concurrent clients share one SQL execution per TTL window. - cuegui: drop UPDATE_INTERVAL 22s→5s; override _update with a skip-if-running guard; rewrite _processUpdate as an incremental diff (takeChild only stale IDs, no clear()), so selection, scroll, and expansion survive add/remove/reparent. - cuegui: consume inline_jobs; fall back to opencue.api.getJobs against older Cuebot. ## LLM usage disclosure Claude Opus was used to implement this optimization  ## Summary by CodeRabbit * **New Features** * Display inline job data within nested job groups. * **Performance** * Faster job tree refresh (22s → 5s). * Incremental tree updates to avoid full rebuilds. * Improved whiteboard caching with per-show refresh control and longer timeout. * **Bug Fixes** * Ensure work/task completion callbacks fire even on failure. * **Tests** * Added a test verifying inline jobs populate in nested groups. * **Chores** * Project version updated to 1.24.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2370?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)   --------- Signed-off-by: Diego Tavares <dtavares@imageworks.com>

…2371) The following warning has been poluting cuegui's log for a while. This QT warning can be triggered by different locations, this PR treats one of them. ``` WARNING Main Qt thread-affinity violation: File "cuegui/ThreadPool.py", line 218, in run result = work[0]() File "cuegui/HostMonitorTree.py", line 291, in _getUpdate parent.updateOSFilterList(os_values) File "cuegui/HostMonitor.py", line 483, in updateOSFilterList action = QtWidgets.QAction(menu) File "cuegui/Main.py", line 140, in warning_handler logger.warning("Qt thread-affinity violation:\n%s", "".join(traceback.format_stack())) ``` The previous solution called a parent function from a different thread directly. This fix uses a signal to allow triggering the same function on its own thread. ## LLM usage disclosure Claude Code was used to propose a fix once the issue was identified.  ## Summary by CodeRabbit * **Bug Fixes** * Resolved thread-safety issues in host monitoring by implementing proper synchronization for OS filter list updates. This prevents race conditions and potential crashes when the background worker discovers and updates operating system values during host monitoring operations.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2371?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

…oftwareFoundation#2378) For some unknown reason, this change was resulting on a Segmentation Fault when a combination of unrelated factors were involved. For now this PR simple reverts the previous version in an effort to avoid crashes.  ## Summary by CodeRabbit * **Bug Fixes** * Stopped emitting completion signals for tasks that fail, preventing failed background jobs from appearing as completed and avoiding premature cleanup of job state. * Improved background fetch error handling with clearer logging, distinct handling for transient RPC errors, and ensured failed fetches return safely so they are retried on the next update tick.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2378?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)   --------- Signed-off-by: Diego Tavares <dtavares@imageworks.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

…uite now 11) One MIXED run on the FULL farm where the two fragmentations INTERSECT: 8 random capability tags scatter across the hosts (--tags 8) AND 25% of hosts/ layers are GPU (--gpu 0.25), so a GPU layer tagged capN fits only hosts that are BOTH GPU-capable and tagged capN, while plain tagged CPU work competes on the same machines. tag_gpu_watch.py samples continuously and the verdict asserts: - zero tag-placement violations (the same host.str_tags ~* layer.str_tags regex predicate cuebot's dispatch query uses); - zero procs of GPU layers on GPU-less hosts; - no host oversubscribed on GPU units or GPU memory (SUM(proc.int_gpus_reserved) <= host.int_gpus, same for gpu_mem), and no negative int_gpus_idle / int_gpu_mem_idle; - coverage floors so the verdict cannot pass vacuously: peak GPU procs >= SIM_TAGGPU_MIN_GPU (default 50) and ALL N tag pools ran work; - GPU utilization floor: peak GPU-unit utilization >= SIM_TAGGPU_MIN_UTIL (default 40%), computed over GPU-CAPABLE HOSTS ONLY -- most of the farm has no GPUs and a farm-wide percentage would be meaningless. GPU utilization is also GRAPHED: the watcher writes run_taggpu.csv and analysis/plot_run.py renders a dedicated <tag>_gpu.png panel (GPU-unit util% + GPU-memory util% over GPU hosts, GPU procs on the twin axis). Tag demand is forced uniform for this scenario (SIM_TAG_SKEW=1.0): farm_spec's steep default skew (0.3) is a deliberate starvation profile for stranding studies, under which cold pools get ~zero jobs and the every-pool coverage floor can never be met. Fragmentation still binds -- each job stays confined to its ~1/N slice. Measured on this box (full farm, 180s): PASS with zero violations, peak 1361 GPU procs = 72.5% of GPU units (30.5% of GPU memory -- the mix is unit-bound, not mem-bound, and the new panel shows exactly that), all 8 pools active.

The planner re-acquired the Postgres advisory lock every tick and released it at tick end, so leadership could bounce between cuebots and each new leader re-planned the whole farm from a cold snapshot. That is redundant work with no payoff. The scheduler is single-writer by design: one cuebot plans the whole farm each tick. Running more cuebots cannot make planning faster, because only the lock holder plans and the others cannot help with the same tick. Additional cuebots are therefore backups, not extra capacity. Spreading the work across them would only duplicate the farm-wide planning and waste resources. So hold the lock once, on a dedicated (non-pooled) connection, for the leader's lifetime. Standbys stay idle and only take over when the holder's session dies (Postgres releases the session-scoped lock on connection loss), which is real failover, not load sharing. The lock connection is a raw DriverManager connection so HikariCP idle-reaping and leak-detection cannot silently release it.

Layers declare licenses in their environment (CUE_LICENSES=hengine,katana); the planner books them only while the license server reports free seats. Seats are per frame (floating) or per machine (host_based, shared by all frames on the machine, packed onto as few machines as possible). LicenseSource polls an http endpoint or a script wrapping the vendor CLI: scheduler.license.provider=http://lic-reporter:9101/licenses scheduler.license.provider=script:/site/bin/cue_licenses.sh Either returns the same JSON (hosts is optional; available is server truth, net of every consumer): {"queried_at": <epoch s>, "licenses": [{"name": "hengine", "total": 800, "available": 794, "host_based": false, "hosts": [{"host": "wolf1018", "count": 1}]}]} Tuning, all under scheduler.license.*: poll_seconds, timeout_seconds, stale_seconds, inflight_pad_seconds, headroom.<name> (seats withheld per license, e.g. headroom.hengine=5), env_key (default CUE_LICENSES), and denied_exit_statuses (vendor exit codes meaning "no license"). The planner corrects the sample with DB-derived in-flight bookings (consistent across failover) and per-license headroom for interactive users. host_based caps are bounded via `available`, so a provider that reports no host list (sesictrl only reports per user) still cannot be oversubscribed. Stale sample, unknown license or no provider: the work is held, not run blind. A configured license-denied exit status requeues the frame without spending a retry. Licensing CONSTRAINS, it never prioritizes. Having few licenses is not a claim to the farm: licensed layers go through the same priority lottery as everything else, and a full license pool simply holds the layer, costing nobody anything. The only placement bias a license adds is where that layer's OWN frames land (packing onto already-seated machines, to burn fewer seats); other jobs never pay for it beyond the marginal stranding delta on those hosts. That packing bonus is deliberately stronger than the ordinary cache-warmth locality bonus (16 per license pool vs 8, dominant over the E-PVM score spread): a seat is scarcer than a warm cache. No schema change, no legacy dispatcher contact, no enable flag: the layer's declaration is the switch. Sim: fake license server (counts real farm usage plus artist holds), scenarios LICENSE and LICENSE_NO_HOSTS, licensed load in FAILOVER, and every scenario is now gated on completed frames. Standby cuebots poll through the script: flavour of the provider, so FAILOVER proves both transports. Also removes the static per-host limit (b_host_limit): unreachable by users (no proto/API/GUI) and blind to seats held outside the cue. Restores the upstream plan-read and frame-start statements. Frame-count Limits unchanged; the seat bonus keeps its property name (scheduler.host_limit_seat_bonus).

A proc row left behind by a failed completion (or any crash) wedges the planner permanently: every batch commit hits c_proc_uk, rolls back whole, and the next tick replans the same frame. Reproduced deterministically: plant terminal orphan procs on never-dispatched WAITING frames at the head of the dispatch order; on current code booking stops (13 failed ticks in a row, orphans untouched). PASS requires eviction + uninterrupted booking, so this is the gate for the coming fix. SIM_POISON_MODE=stall is the organic variant (Aghiles): SIGSTOP cuebot so completion acks time out and RQD re-sends, SIGCONT into the duplicate flood, the mechanism that manufactured the orphans in the real incident.

Frame completions were processed one at a time on the gRPC threads that received them: every completed frame was its own transaction, dozens of threads contending over the same frame, proc, host and stat rows. At farm scale that contention is the completion path's ceiling, and the suite shows it: batching the completions into the scheduler tick lifts every high-churn scenario (OOM 62 to 88 frames/s, DEPENDS 58 to 91, LOCALITY 60 to 100, TAGS_GPU 56 to 90, FAILOVER 53 to 209, and the priority flood runs at 165 frames/s while keeping the low-priority stream fed). How it works now: - The report thread only acks, resolves (pure reads) and queues the completion. Every cuebot drains its own queue at the start of its tick, leader and standby alike; the database stays the shared truth. - The drain applies completions in chunks, each chunk one transaction: frames stopped (state+version guarded, stat counters pre-locked in sorted order), procs deleted with their live reserved values, host and accounting resources refunded. The planner then plans against a snapshot where every freed core is already visible. - Lock order is procs, then hosts, then stats. Procs first because single-proc writers (the OOM memory bump trigger) take proc then host; hosts before stats matches the booking commit. - Follow-up work per completed frame (depends, layer/job completion, usage counters) runs on one dedicated non-droppable thread, so the tick stays a few batched statements regardless of completion rate. Batching also kills a whole failure class. Concurrent inline completions could race each other, a job shutdown and the batch commit into leaving an ORPHANED proc behind (frame back to WAITING, proc row alive); that one corpse collides with c_proc_uk on replan, rolls back the entire batch commit, and poisons every tick after it. The whole farm stops booking over one row. With completions applied by one writer in tick order that interleaving no longer exists, and two layers of armor clean up corpses from any other source: the batch commit evicts stale procs sitting on frames it just won, and a janitor sweep deletes procs whose frame has not been RUNNING for 10s. The POISON scenario committed previously reproduces the wedge deterministically and now passes in both modes (planted corpses and a SIGSTOP'd cuebot): orphans evicted, zero failed ticks, booking uninterrupted. Drain-at-tick-start is also how Plow, the schema author's scheduler at WETA, applies completions: the scheduler consumes the completion queue first on every pass, rolling the accounting into one transaction, and runs with near zero contention. That design is confirmed here. Crash semantics are deliberate: a queued completion lost with the process leaves its frame RUNNING, and host-report reconciliation requeues it. A few seconds of redone work beats at-least-once machinery. All 13 verify scenarios pass.

The locality bonus only sees LIVE procs: the moment a host loses its last proc of a layer, the layer's pull on that host vanishes, even though its locally cached data (texture caches, NFS client caches) is still on the machine. Measured over 88 minutes and 555k frame starts: 77.5% of starts landed live-warm, but once a host went dark nothing brought the layer back, natural revisits died within a minute, and 40.7% of starts ran cold. Placement now keeps a decayed pull on vacated (host, layer) pairs: bonus = locality_bonus * (1 - foreignFrames / window) Age is displacement, not wall clock. What invalidates a local cache is other layers' frames writing their data over yours, so age is counted in frames of other work booked onto the host since the layer left it. The scheduler keeps a per-host booking odometer; each warmth entry stores the reading at the layer's last completion there. A busy host cools one step per foreign frame; an idle host's odometer never moves, so its entries stay fully warm no matter how long it idles, because nothing displaced them. The map is fed by the completion drain and read only on the planner thread (no locking), expires by the same odometer comparison (self cleaning, bounded by hosts x co-resident layers), and adds per tick only work linear in completions, bookings and map size plus one hash lookup per scored (host, candidate) pair. Live bonus always outranks warm, and fit, reservations, tags and licenses are filtered before any bonus, so warmth only breaks ties among hosts that could all take the work. scheduler.locality_window_frames (default 64, 0 disables) is the cache size over a typical frame's cache footprint, a site property. Same-feed A/B in the sim: cold starts fell from 40.7% to 19.4%, the live-warm rate rose from 80% to 90%, refill affinity reached 90% over 40k refills, and POISON plus the throughput gates were unaffected. Scheduler.md 3.7 documents the model with a worked example; the sim's locality watcher gains a frames mode with a warmth-age histogram, the measurement behind these numbers.

…t-only)

…ss facilities Two eligibility gates existed in the legacy dispatcher but not in the scheduler's candidate query, so jobs the old path renders were silently starved (or misplaced) by the new one. This is the same "old books it, new doesn't" class as the host-name tag gap, found by auditing every legacy clause against the planner's: * OS: a host may advertise SEVERAL OSes, comma-separated in host_stat.str_os ("rhel7,rhel9" on mid-migration boxes). The legacy dispatcher expands that into str_os IN ('rhel7','rhel9'); the scheduler compared j.str_os to the raw string, exactly, so every os-pinned job was invisible on such hosts, forever. Match any advertised value with str_os = ANY(string_to_array(?, ',')). * Facility: every legacy job-finding query binds job.pk_facility to the host's facility (jobs render next to their assets; a job must never cross sites). The candidate query had no facility clause at all, and the frame-level plan read does not re-check it, so the planner booked jobs onto other facilities' hosts. Carry the alloc's facility through the host snapshot into the group key and bind it in the candidate query. Both are proven by a new PARITY verify scenario, the drift detector for this whole class: inject_parity.py submits one job per eligibility archetype (plain, os-pinned on multi-OS hosts, dual-tag alternation, other-facility) and records which of them ever book; run_verify runs the battery once under "--mode old" (PARITY_OLD gates the legacy baseline: everything books except the other-facility job) and once under "--mode new" (PARITY_NEW fails on ANY difference between the booked sets, in either direction). On the unfixed scheduler the diff was old-only=[parity_os], new-only=[parity_facother]; with this change the sets are identical. Hosts advertise the multi-OS string via a new SIM_HOST_OS knob (farm_spec.os_attrs feeding the RenderHost SP_OS attribute, sent by the registrar, both pingers and the report loop). The verify suite grows to 16 scenarios. SchedulerTests cover the new facility field in the host-spec key.

Reporting only, no scheduling changes. At DEBUG: every group logs candidates=N every tick (zero-candidate groups were invisible before); a zero-candidate group prints per-layer explain lines with one boolean per eligibility gate (tagOk osOk facOk hasSub underBurst underJobCap fitsCores hasWaiting limitOk folderOk managedOk); every candidate that placed nothing logs its binding constraint. At WARN, always on: a layer planned scheduler.plan_zero_warn_ticks consecutive ticks (default 40) whose planHost read returns zero frames is reported with the layer and last host: the signature of a commit-side gate the planner does not model. Enable DEBUG by adding to cuebot's JAVA_OPTS: -Dlogging.level.com.imageworks.spcue.dispatcher.Scheduler=DEBUG (The LOGGING_LEVEL_* env form misses this logger: relaxed binding lowercases the class name. Env-only setups should target the package: LOGGING_LEVEL_COM_IMAGEWORKS_SPCUE_DISPATCHER=DEBUG.)

Layer tags are regex fragments typed by users, and the candidate query compiles every pending layer's tags in one statement: one malformed tag ("comp(") aborted the whole tick, every tick, farm-wide. The legacy dispatcher runs the same regex per show, so a bad tag only poisons that show. Match that blast radius: catch the failure, skip that group for the tick with a throttled WARN carrying the database error, and keep planning the other groups.

Same fixture, host and values as DispatcherDaoTests; the only intended delta is scheduler.enabled=facility. Asserts the scheduler's candidate query finds exactly what the legacy job query finds: the fixture job on the fixture host, an os-pinned job on a multi-OS host, and refusal of a cross-facility job. CI-resident sibling of the simulator's PARITY scenario; it would have caught the multi-OS and facility drifts at commit time. Two Scheduler read methods widen to package-private as test seams.

ThreadMode.ALL hosts (NIMBY workstations by default) run only threadable layers. The planner was blind to the attribute: it parked non-threadable layers on ALL hosts (idle workstations score best), planHost's legacy re-check found zero frames, and the layer burned its one commit per tick forever while legacy booked it elsewhere. Carry int_thread_mode through the host snapshot into the group key (one bit: legacy normalizes every mode but ALL to AUTO) and bind the legacy threadability clause in the candidate query and the DEBUG explain. Parity tests cover refusal and booking on ALL hosts; the refusal test was written first and failed against the unfixed scheduler.

Entire-Checkpoint: f68a943135a3

Fix the batch-start locking javadoc, note the transaction requirement on the stat pre-locks, add a legend to the score formula, explain the advisory lock choice, and clean comment punctuation. Comments and docs only.

Expose the scheduler's behaviour as Prometheus metrics and read them from a Grafana dashboard. Metrics cover tick duration, frames dispatched per show, group pass reasons and totals, per-show farm cores, fragmentation by reason, and active vs inactive host-spec groups, under a human-readable label vocabulary. The dashboard draws per-show throughput and farm share as stacked bars, so each show's contribution and the farm-wide total read at a glance. On the simulator side, add the fragmentation scenario tooling, a multi-show feeder (sim1 to sim5) with per-show priority, and multi-tag hosts, and start each cuebot with the Prometheus collector on so the stack can be scraped.

A permissive layer (loose tags, or plain 'general') is a candidate in every host-spec group it fits. The within-group break already caps it to one host per group per tick, but nothing stopped it being planned again in the next group: each group re-planned it onto its own host, the parallel per-host plan reads then pulled the same waiting frames, and every copy but one lost the frame.int_version race at commit. The waste was real: candidate scans, reads and VirtualProc construction, plus idle cores stolen from siblings that booked nothing. Add a tick-wide placedLayerIds set (cleared with plannedByHost). submitCommit records the placed layer, and the candidate loop skips a layer already placed in an earlier group this tick. The skip sits after seenLayerIds.add, so reservation sweeping still sees the layer, and before any host/cap mutation, so a duplicate consumes no simulated resources. It keys on placement, not candidacy, so a layer that could not fit an earlier group is still tried in later ones. Measured on the simulator under a 120-tag, 30% run-anywhere farm (about 120 host-spec groups, 1553 hosts): raceLost fell from about 97% of planned to 0 (planned now equals committed). Utilisation is unchanged: at this tag count the farm is fragmentation-limited, not planning-limited. sim: guard the fix with a TAGMAX scenario in the --verify battery. tagmax_watch reads the Scheduler's per-window stat line and fails if raceLost exceeds a small fraction of planned (default 0.10) across the fragmented farm, with planned and host-spec-group floors so it cannot pass on an idle run. Add the --tagmax-test flag, its wiring and a README row, plus SIM_GENERAL_FRAC in farm_spec: the fraction of layers that carry no capability tag (run-anywhere 'general' work, a candidate in every group). 0 by default; the scenario uses 120 tags and 0.3.

Break the 527-line doTick into small, single-purpose methods so the tick reads as its phases: the completion drain, the leadership gate, then 1. snapshot, 2. group, 3. plan, 4. commit. The extracted methods (planGroup, planBookings, recordCommitted, stampWarmthAndLaunch, snapshotFarmFill, grantReservations, trimOverFolderCeiling, trimOverLicensePools, clearTickScratch, drainResolvedCompletions, expireDisplacedWarmth, and friends) each carry a plain prose header. runTick stays a thin Quartz harness that times the pass and rolls the leader counters into the window summary; doTick returns the procs dispatched, or -1 for a standby that did not plan.

Documentation, comments, and one rename. No logic changes. Scheduler.java: * Fixed 7 factually wrong or stale comments: the all-hosts query filter, the launch-pool backpressure policy (a drop policy, not caller-runs), the licenses source, the farm-snapshot timing, a nonexistent license property, waitingFrameCount, and dispatchSupport. * Field comments now say what each field is used for and point to the main caller, rather than restating the declared type. * Trimmed duplicated design essays to a gist plus a Scheduler.md pointer (priority lottery, batched accounting, cross-group dedup, E-PVM placement, group fragmentation). * Removed an orphaned tick-algorithm block, lowercased emphasis capitals, dropped leftover HTML and stray dashes, and re-scoped a few comments. * Renamed stampWarmthAndLaunch to launchCommitted, named for its primary job (launch); the cache-warmth stamping it also does is now in the method header. Scheduler.md (full consistency sweep against Scheduler.java): * snapshot: readBookableHosts / SELECT_BOOKABLE_HOSTS renamed to readAllHosts / SELECT_ALL_HOSTS; the query is UP + OPEN (busy or idle), and the minimum idle core cut happens later, per group, in planGroup. * tick loop: document stage 0 (completion drain plus cache warmth expiry on every Cuebot) and the leadership gate that precedes the placement pipeline. * host spec key: full six part tuple (alloc, facility, tags, os, gpu, thread mode), not four. * stat line: add drained, reservedCores, backfilledCores and the optional lic segment to both the sample and the prose. * drop the stale "no schema changes" claims; fix field name glosses (layerCoresMin / layerMemMin) and soften the new file count.

A blocked wide layer could reserve a host and drain it, but two gaps let it starve anyway: - The reservation was not firm. A running frame or a higher-priority reserver could seize the host mid-drain, so the wide job never assembled its block and stranded forever. - Grants were ordered strictly by priority, so a low-priority wide job was starved of the scarce reservation budget by any steady higher-priority stream. Fix both: - Make reservations firm. reservationAllows is owner-only and pickReservationTarget only ever claims a free host, so once a layer holds a host nothing takes it away, not a running frame and not another reserver. Higher-priority work still borrows the draining host's spare cores through EASY backfill (never owning), so the owner is never delayed; the host drains and the wide job runs. - Grant by a priority-weighted lottery, the same one the dispatcher uses (key = random()^(1/priority), Efraimidis-Spirakis), so a low-priority wide job keeps a proportional share of the budget instead of being starved. The RESERVATIONS --verify scenario asserts the stranded wide jobs reserve, drain, and actually run, with a farm-wide throughput floor so a dead farm cannot pass.

Stats only, no scheduling change. Each tick puts every waiting frame from the candidate layers into one of six buckets: flowing, capacity, no fit, limit, no license, held. The tally goes to the Prometheus gauge cue_scheduler_waiting_frames. A live booking ledger feeds cue_scheduler_running_frames, the denominator, so the board shows each cause as a percent of all frames. No SQL is used for stats. The fragmentation metric is removed. The board gains waitlist and utilisation panels. The verify battery asserts each bucket fires.

A legacy trigger rejects any plus that lands over a cap. When a user lowers a job's max cores, or an admin shrinks a subscription burst, below live usage, the batched flush aborts. The pluses wedge in the retry buffer, completions keep subtracting, and the mirror goes negative (seen in production as a job at -14 cores). The flush now writes each guarded table as a cap-neutral pair of updates that the trigger skips; the cap is net unchanged. The planner still enforces caps at plan time. The CAPDROP scenario drops both caps under load and fails on divergence, a negative mirror, or a rejected flush.

PRODENV now runs first in the verify battery: 300 seconds on the full farm while a seeded chaos driver mutates the live environment the way admins do all day. Limit caps churn, capped folders appear and eat running jobs, host tags come and go, job max cores and subscription bursts random-walk, hosts lock and unlock. The watcher asserts decay-style cap invariants (over-cap usage may only drain, never grow), mirror-vs-proc truth everywhere, zero rejected flushes, zero negative counters, and a throughput floor, then reconciles every ledger in a quiet tail. The five feeder shows are renamed to showA..showE in the simulator seed and helpers. SIM_VERIFY_ONLY=NAME[,NAME] now runs a subset of the battery.

Production showed 128 one-core frames of one layer filling a single 128-core host while the rest of the farm sat free: the locality bonus (8.0) dwarfs the E-PVM spread (hundredths), so same-layer frames pile onto one machine until it is full. That concentrates blast radius and phase-locked IO on one chassis. New knob scheduler.layer_host_max_frac (default 0.25; 0 disables): one layer may hold at most this fraction of a host's cores, never below 8 frames so small hosts still anchor a cache-warm batch. Hosts at their share are skipped at selection and the commit is clamped, so the flood spills to the next host. The LAYERCAP scenario reproduces the pile-up with the cap off (128 frames on one host, 2 hosts total) and asserts the cap holds at 0.25 (same flood, 15 hosts, zero violations); LOCALITY still passes with the cap on (refill affinity 27.7% against the 15% floor).

Each RQD host report now feeds an in-memory health ledger: swap use and kernel system time (a new sysTime report attribute; real RQD still needs a small patch to send it). After each tick the scheduler publishes the cue_farm_health_* gauges per host spec group and per hardware shape, with no database read. The dashboard gets a farm health row, the sim plants sick hosts, and the new HEALTH verify scenario asserts that the sickness shows up on the metrics endpoint.

DiegoTavares and others added 30 commits May 11, 2026 19:08

akheffache and others added 14 commits July 26, 2026 22:49

cuebot: move long inline comment blocks into function headers (commen…

f912cfe

…t-only)

Fix query issues on DispatchQuery

103872b

Entire-Checkpoint: f68a943135a3

cuebot: comment and doc cleanup from review feedback

097a7be

Fix the batch-start locking javadoc, note the transaction requirement on the stat pre-locks, add a legend to the score formula, explain the advisory lock choice, and clean comment punctuation. Comments and docs only.

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from c7f1c00 to aec256d Compare August 13, 2026 02:41

akheffache added 3 commits August 14, 2026 00:05

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from 0bb32a2 to efb0513 Compare August 14, 2026 18:54

akheffache added 2 commits August 15, 2026 02:26

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch 3 times, most recently from c23c771 to abfaafd Compare August 19, 2026 04:22

akheffache added 2 commits August 19, 2026 05:31

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from abfaafd to a6647aa Compare August 19, 2026 05:36

akheffache added 3 commits August 19, 2026 15:31

Navigation Menu

[cuebot] Fix issue on getWhatDependsOn(Frame) (#2278) - #1

[cuebot] Fix issue on getWhatDependsOn(Frame) (#2278)#1
aghiles wants to merge 126 commits into
masterfrom
claude/gifted-clarke-0vluzy-v2

aghiles commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

Conversation

aghiles commented Jul 23, 2026

Related Issues

Summarize your change.

Summary by CodeRabbit

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants