diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 920f76cb3..ab1063ff5 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -32,7 +32,7 @@ one to open. | LoRAs | `dw/pipeline_processors/adapters.py`: `active_loras`, `load_loras` | A `loras` entry whose `model_name` is null is switched off. | `tests/test_lora_disable.py::TestLoadLoras::test_an_all_null_entry_loads_nothing` | | Step progress | `dw/pipeline_processors/progress.py`: `reported_progress_bars` | A pipeline with no step callback reports its denoise steps through its progress bars. | `tests/test_modular_progress.py::test_every_advance_of_the_bar_is_reported` | | Step timing | `dw/pipeline_processors/timing.py`: `StepTimer`, `step_timer`, `timed_step`, `mark` | Only a pipeline step that ran is timed, on the worker's monotonic clock: `step_timer` opens a `StepTimer` for a step with a `pipeline` or `pipeline_reference` and None for any other, which also hides an outer step's timer from a composed child. Load, call, denoise start, each pass and save reach the open timer through `mark`, and nowhere else. Denoise start is never taken inside a compiled region: a whole-compiled denoiser (`_compiled_call_impl` set) has that callable wrapped for the call instead of a forward pre-hook, which `fullgraph: true` cannot trace. A span with a missing mark is null, never guessed. `record_step` puts the reading on the step's manifest entry and `step_end`, and a reused step gets no `timing`. A chain sums its pipeline calls into one `timing` with `segments: N`. `ManifestEntry.timing` declares the shape (`docs/WORKSPACES.md` *Step timing*). | `tests/test_step_timing.py::test_a_missing_denoise_start_nulls_encode_and_first_pass`, `tests/test_step_timing.py::test_a_reused_step_carries_no_timing_even_when_one_is_passed`, `tests/test_step_timing.py::test_a_chain_sums_its_segments_and_takes_one_median_over_both`, `tests/test_step_timing.py::test_a_task_step_run_through_the_step_loop_has_no_timing`, `tests/test_step_timing.py::test_the_denoise_hook_marks_the_first_forward_once_and_removes_itself`, `tests/test_step_timing.py::test_a_fullgraph_compiled_denoiser_runs_and_still_gets_the_start_mark` | -| Step-cache skip counts | `dw/pipeline_processors/timing.py`: `CacheCounter`, `counting_forwards`, `report_cache_stats`, `StepTimer.cache_counted`; `dw/pipeline_processors/components.py`: `stateful_cache_context`, `get_cache_transformer` | For `first_block` and `mag`, a forward hook on the cache transformer counts computed or skipped once per forward, reading the leader hook's `should_compute` inside the cond/uncond `cache_context` the pipeline opened, so classifier-free guidance counts both forwards. `stateful_cache_context` registers the hook for one pipeline call; its `finally` emits `cache_stats` before `_reset_stateful_cache()` and removes the hook after the event. Private diffusers state that has moved gives null counts and one `cache_stats_unavailable` warning, and never fails the run; `taylorseer`, `faster` and `text_kv` report type and config only, with no warning. The counts reach the step's `timing.cache` only through `StepTimer.cache_counted`; a chain sums its calls' counts, and a null in any call nulls the sum. A step with no `cache` has no `cache` key and no event (`docs/ACCELERATION.md` *Reading the skip counts*). | `tests/test_cache_stats.py::test_cfg_counts_two_forwards_per_step_and_they_add_up`, `tests/test_cache_stats.py::test_the_event_names_the_step_and_segment_and_goes_out_before_the_reset`, `tests/test_cache_stats.py::test_moved_hook_state_nulls_the_counts_warns_once_and_the_run_finishes`, `tests/test_cache_stats.py::test_the_private_diffusers_names_are_where_the_counter_reads_them`, `tests/test_cache_stats.py::test_the_timer_sums_counts_and_nulls_on_any_null`, `tests/test_cache_stats.py::test_types_without_a_decision_report_type_only_and_no_warning` | +| Step-cache skip counts | `dw/pipeline_processors/timing.py`: `CacheCounter`, `counting_forwards`, `report_cache_stats`, `StepTimer.cache_counted`; `dw/pipeline_processors/components.py`: `stateful_cache_context`; `dw/pipeline_processors/mag_calibration.py`: `get_cache_transformer`, `captured_mag_calibration` | For `first_block` and `mag`, a forward hook on the cache transformer counts computed or skipped once per forward, reading the leader hook's `should_compute` inside the cond/uncond `cache_context` the pipeline opened, so classifier-free guidance counts both forwards. `stateful_cache_context` registers the hook for one pipeline call; its `finally` emits `cache_stats` before `_reset_stateful_cache()` and removes the hook after the event. Private diffusers state that has moved gives null counts and one `cache_stats_unavailable` warning, and never fails the run; `taylorseer`, `faster` and `text_kv` report type and config only, with no warning. The denoiser a cache attaches to (`get_cache_transformer`) and the MagCache calibration capture (`captured_mag_calibration`, the end-of-run log record turned into `cache_stats`-side job events, #891/#927) live in `mag_calibration.py`, which `components.py` imports and re-exports for `pipeline.py`, never the reverse. The counts reach the step's `timing.cache` only through `StepTimer.cache_counted`; a chain sums its calls' counts, and a null in any call nulls the sum. A step with no `cache` has no `cache` key and no event (`docs/ACCELERATION.md` *Reading the skip counts*). | `tests/test_cache_stats.py::test_cfg_counts_two_forwards_per_step_and_they_add_up`, `tests/test_cache_stats.py::test_the_event_names_the_step_and_segment_and_goes_out_before_the_reset`, `tests/test_cache_stats.py::test_moved_hook_state_nulls_the_counts_warns_once_and_the_run_finishes`, `tests/test_cache_stats.py::test_the_private_diffusers_names_are_where_the_counter_reads_them`, `tests/test_cache_stats.py::test_the_timer_sums_counts_and_nulls_on_any_null`, `tests/test_cache_stats.py::test_types_without_a_decision_report_type_only_and_no_warning` | | Community pipelines | `dw/community_pipelines/` (e.g. `pipeline_ltx2_refine.py`: `LTX2RefinePipeline`), `dw/introspection.py`: `community_pipelines`, `load_allowed_class` | A pipeline dw ships is named by its dotted path (`dw.community_pipelines..`), loads untrusted because it is under `dw`, and is listed by `list_pipelines` and resolved by `get_pipeline_signature` - only the modules in that directory, never a caller-chosen import. `LTX2RefinePipeline` VAE-encodes `video` at `width` x `height` and passes it unnormalized as `latents`, because `LTX2Pipeline.prepare_latents` normalizes 5-D latents itself. | `tests/test_ltx2_refine_pipeline.py::TestIntrospection::test_only_the_shipped_modules_resolve`, `tests/test_ltx2_refine_pipeline.py::TestEncode::test_the_latents_are_left_unnormalized_for_the_parent` | | Adding a task | `dw/tasks/registry.py`: `register_command`, `RegistryTable` | A task is a function registered with `@register_command`, and the signature of its implementation is its argument schema. Its validate-time rules are declared on the same decorator - `domains`, `choices`, `static_check`, `whole_numbers` and `media_arguments` - and `TASK_ARGUMENT_DOMAINS`, `TASK_ARGUMENT_CHOICES`, `TASK_STATIC_CHECKS`, `TASK_WHOLE_NUMBER_ARGUMENTS` and `TASK_MEDIA_ARGUMENTS` are read-only views derived from them, so there is no second table to forget (#692). The registry lives in `dw/tasks/registry.py`; `dw/tasks/task.py` re-exports it, and a task module may register its own handler, as `dw/tasks/beats.py` does, provided `dw/tasks/task.py` imports it (`docs/TASKS.md` *Adding a task*). The built-in handlers are grouped by family the same way - `dw/tasks/audio_handlers.py`, `dw/tasks/video_handlers.py`, `dw/tasks/finish_handlers.py` and `dw/tasks/model_handlers.py` (#790) - and `dw/tasks/task.py` keeps the general-purpose commands, the processor dispatch and `Task`. | `tests/test_task_discovery.py::TestDescribeTask::test_signature_becomes_the_schema`, `tests/test_task_registry_rules.py::test_every_derived_table_is_the_registrys_view`, `tests/test_task_registry_rules.py::test_every_static_check_is_registered_to_exactly_one_command`, `tests/test_task_registry_rules.py::test_every_media_argument_is_a_parameter_of_its_command` | | Task argument domains | `dw/task_domains.py`: `check_arguments`, `task_argument_errors`; `dw/task_problems.py`: `face_track_errors`, `beats_errors`, `cuts_errors`, `lut_errors` | A numeric domain that a signature cannot express is declared on the command's `@register_command(domains=...)`, and it is checked at validation and again at run time. A numeric argument is read by `whole_number` or `real_number`, the only numeric coercion in `dw/tasks/`, and validation applies the same rule (`number_problem`), so a value is refused at both or at neither: `"3.0"` is a whole number, `"3.5"` for one of `whole_numbers` is not, and `true` is no number (#774). `Task.run` coerces every declared argument through `coerce_arguments` before dispatch, so a handler sees a number, and an argument that has no range besides being a number declares `finite`. A command's cross-argument rules are its registered `static_check`, which `task_argument_errors` runs at validation (`attribute_voices`' `voices_argument_errors`, kept in `dw/tasks/voice_attribution.py`, among them). A literal enum choice (`choice_errors`) is refused at validation by the same module. The per-family rules past a single domain live in `dw/task_problems.py`, which reads `task_domains`' helpers and is never imported by it (`task_argument_errors` reaches them only through the registered `static_check`, so `import_cycles` stays 0; #790): the `ingredients_grid` image count against `max_images` (`ingredients_grid_errors`), and `crop_face_track`'s cross-argument rules (`face_track_problems`, `face_track_errors`: crop size a multiple of the step's `multiple` (32 by default), `remainder` below `modulus`, `gate_zero` above `gate_full`, padding and confidence limits), which the task checks again at run time, as `analyze_beats`' are too (`beats_problems`, `beats_errors`: `min_bpm` below `max_bpm`, and anchors that are one kind per list, ascending, with whole beat indexes; `tests/test_analyze_beats.py`), and `plan_cuts`' (`cuts_problems`, `cuts_errors`: a bare-string transcript (`transcript_problem`, which `plan_cuts` also calls at run time), `min_scene_s` above `max_scene_s`, `segment_by: "beat"` without beats), and `apply_lut`'s (`lut_source_problem`, `palette_problem`, `lut_errors`: exactly one of `lut` or `palette`, and a palette of 2..16 `#rrggbb` colours, re-checked at run time by `check_lut_source`), and `check_face_detector_source` vets its Hub repo and `.onnx` file before any download. `select`'s index stays in `task_domains` with the other media-timing rules (`select_index_problem`, `select_rule_problems`: a whole number, at least 0, and below the candidate count when the count is known, i.e. at run time or when `candidates` is a written list). | `tests/test_task_domains.py::TestTheStaticPass::test_a_negative_frame_count_is_refused_at_its_path`, `tests/test_numeric_coercion.py::TestGradeExposure::test_a_string_number_grades_like_the_number`, `tests/test_task_domains.py::TestAtRunTime::test_slice_audio_refuses_a_negative_count`, `tests/test_ingredients_grid.py::TestStaticValidation::test_bad_layout_and_fit`, `tests/test_face_track.py::TestStaticValidation::test_bad_literal_is_an_error_at_its_argument`, `tests/test_face_track.py::TestCropFaceTrack::test_bad_arguments_are_refused_at_run_time`, `tests/test_face_track.py::TestStaticValidation::test_a_declared_grid_is_accepted`, `tests/test_face_track.py::TestDetectorSource::test_traversing_repo_is_refused`, `tests/test_select_validation.py::TestSelectErrors::test_an_out_of_range_literal_index_names_both_ends`, `tests/test_lut.py::TestPaletteAtValidate::test_both_refuse_naming_the_two`, `tests/test_lut.py::TestPalette::test_a_bad_palette_refuses_naming_the_entry` | @@ -67,12 +67,14 @@ one to open. | Concept | Owner | Rule | Enforced by | | --- | --- | --- | --- | | Run directories | `dw/runs.py`: `open_run`, `workflow_identity`, `new_run_id` | Each execution writes into `///` with a `manifest.json`, unless the flat layout is chosen (the test pins the flat-layout exception). | `tests/test_runs.py::TestRunDirectories::test_the_flat_layout_writes_where_it_always_did` | -| A file's role (final / intermediate / input) | `dw/roles.py` | Decided per step: result.role, then a final/intermediate subfolder, then data flow (a step a media-writing or unsaved consumer reads is intermediate). Engine records it at run end, with `role_declared` where result.role stated it; the server reports only recorded roles, so an older run has none. At read time the gallery (`GET /api/gallery`, its `role=` filter, the metadata route) also demotes a file a later run of the same workspace read - an `output:` input (an unpinned `latest` is the newest run started before the reader) or an `asset:` keep_output kept from it - unless its role was declared (`consumed_outputs`, `listed_role`, once per listing; no manifest is rewritten). | `tests/test_roles.py`, `tests/test_roles.py::test_a_file_a_later_run_read_is_consumed`, `tests/test_roles.py::test_listed_role`, `tests/test_gallery_roles.py::test_a_file_a_later_run_read_lists_as_intermediate`, `tests/test_gallery_roles.py::test_a_declared_final_stays_final` | +| A file's role (final / intermediate / input) | `dw/roles.py` | Decided per step: result.role, then a final/intermediate subfolder, then data flow (a step a media-writing or unsaved consumer reads is intermediate). Engine records it at run end, with `role_declared` where result.role stated it; a run that predates roles recorded none, so the listing falls back on the file's own `final`/`intermediate` subfolder, then the demotion below, and a file neither places has none. At read time the gallery (`GET /api/gallery`, its `role=` filter, the metadata route) also demotes a file a later run of the same workspace read - an `output:` input (an unpinned `latest` is the newest run started before the reader) or an `asset:` keep_output kept from it - unless its role was declared (`consumed_outputs`, `listed_role`, once per listing; no manifest is rewritten). | `tests/test_roles.py`, `tests/test_roles.py::test_a_file_a_later_run_read_is_consumed`, `tests/test_roles.py::test_listed_role`, `tests/test_roles.py::test_listed_role_of_a_run_that_recorded_none`, `tests/test_gallery_roles.py::test_older_run_reads_its_intermediate_subfolder_as_intermediate`, `tests/test_gallery_roles.py::test_a_pre_roles_file_a_later_run_read_lists_as_intermediate`, `tests/test_gallery_roles.py::test_metadata_agrees_with_the_listing_for_an_older_run`, `tests/test_gallery_roles.py::test_a_file_a_later_run_read_lists_as_intermediate`, `tests/test_gallery_roles.py::test_a_declared_final_stays_final` | | Notes on runs, files, assets, workspaces | dw/notes.py | notes.json beside what it describes (run dir or asset dir; a file's own note else the run's), workspace.json at a workspace root; only the server writes them, never the engine; ≤4000 chars, plain text. | tests/test_notes.py | | Run versions | `dw/runs.py`: `open_run`, `run_versions`, `record_run_versions`, `copy_runs_in` | A run takes its number once, when it opens (or, copied in from another workspace, when `copy_runs_in` places it), so deleting a middle run leaves a gap rather than renumbering. | `tests/test_runs.py::TestRunVersions::test_a_deleted_middle_run_leaves_a_gap_rather_than_renumbering`, `tests/test_runs.py::TestRunVersions::test_runs_started_in_the_same_second_still_number_upward` | | Run-version surfaces | `dw/runs.py`: `run_versions`, `dw/server/routes/gallery.py`, `dw/server/outputs.py`, `dw_mcp/tools_catalog.py`: `list_gallery`, `ui/src/lib/pages/GalleryPage.svelte` | Only `dw/runs.py` computes the number, and every other module reads it back. | `tests/test_runs.py::TestRunVersions::test_an_output_reference_can_name_a_run_by_its_version` | +| What the server says about an output file | `dw/server/outputs.py`: `output_annotations` | The gallery listing (`dw/server/routes/gallery.py`) and the metadata route (`dw/server/routes/media.py`) describe an output file's `role`, `note`, `note_source`, `run_note`, `starred` and `version` through `output_annotations`, and neither builds those fields on its own. | `tests/test_gallery_roles.py::test_metadata_agrees_with_the_listing_on_a_starred_noted_demoted_file` | | Result subfolders | `dw/subfolders.py`: `subfolder_errors`, `dw/security.py`: `SUBFOLDER_PATTERN` | A step's `result.subfolder` is checked for shape at validation. | `tests/test_subfolders.py::TestValidationErrorsIntegration::test_validation_errors_reports_a_bad_subfolder` | | Template subfolder roles | `workflows/templates/`, `dw/workflows/` | Every saving step of a template is marked `final` or `intermediate`, with at least one `final`, and packaged builtins stay unmarked. | `tests/test_template_subfolders.py::test_every_saving_step_of_a_template_names_its_role`, `tests/test_template_subfolders.py::test_the_packaged_builtins_stay_unmarked` | +| Frames carrying their rate | `dw/media_types.py`: `FrameList` (re-exported by `dw/tasks/video_utils.py`), `rated_videos` | The one list of frames (or of videos) that carries `.fps` and `.shots`: a video loaded from a file (`dw/argument_media.py`) and a step that knows its rate but would return bare frame lists both answer it - `h3_tiles`' tiled refine `videos`, and every pipeline step whose pipeline reports its own `fps` (MiniMax-H3's property; `dw/output_extraction.py`: `attach_video_frame_rate`, called by `pipeline.py`'s `_run_once` beside `attach_audio_sample_rate`, #1056) - through `rated_videos`, the batch and each video in it, so `pair_audio` and the result writer read `getattr(video, "fps", None)` instead of fitting and playing at the default 8 fps. A pipeline that reports no rate leaves its lists bare. No second class of that name. | `tests/test_h3_tiles.py::TestTileLoopReporting::test_the_videos_are_dws_one_frame_list`, `tests/test_pipeline_components.py::TestVideoFrameRate::test_a_dict_outputs_videos_carry_the_pipelines_rate`, `tests/test_pipeline_components.py::TestVideoFrameRate::test_a_pipeline_with_no_rate_leaves_the_lists_bare`, `tests/test_modular_output_properties.py::TestTheBasesOwnTakeSavesBesideIt::test_the_base_take_pairs_at_its_own_rate_with_no_padding`, `tests/test_video_utils.py` (the `FrameList` cases) | | JSON records beside media | `dw/media_types.py`: `JsonRecord`, `dw/result.py`: `Result.save_artifact` | A dict marked `JsonRecord` is saved whole as one `.json`, whatever content type the step declares, and is never split into one file per key. | `tests/test_face_track.py::test_json_record_is_saved_whole_under_a_video_content_type`, `tests/test_face_track.py::test_the_task_result_saves_crops_as_video_and_track_as_one_json` | | Reading a JSON record back | `dw/locations.py`: `load_json_record` | A task reads a saved JSON record (`fit_to_model`'s, `crop_face_track`'s) only through `load_json_record`: the path is confined like any media path, then the extension must be `.json`, then the size is checked against `MAX_JSON_SIZE`, and only then is it parsed. `fit._read_fit` and `face_track._read_track` call it, never `open`. | `tests/test_locations.py::TestLoadJsonRecord::test_traversal_out_of_the_roots_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_a_non_json_extension_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_an_oversized_file_is_refused_before_it_is_read`, `tests/test_fit_to_model.py::TestRestore::test_fit_path_with_traversal_is_refused` | | Shots in a joined video | `dw/shots.py` | Every `AudioVideo` constructor decides what to do with `shots`, and the sample spans are measured from the joined waveform. | `tests/test_shots.py::test_every_audio_video_constructor_site_is_decided`, `tests/test_shots.py::TestConcatVideosShots::test_an_overrun_track_is_measured_not_derived` | @@ -95,20 +97,22 @@ one to open. | Admission | `dw/server/admission.py`: `admit` | Validate, submit, rerun and enhance load and check a request once, and `JobManager.submit` does not check it again. | `tests/test_admission.py::test_a_submit_expands_once` | | Variable constraints | `dw/variable_constraints.py`: `apply_constraints`; the grid arithmetic `aligned` / `aligned_down` | A model's rule about a value is declared in the workflow's `variable_constraints`, and it is checked at validation and again before anything loads. `plan_cuts` (and its static check) rounds render lengths through `aligned` / `aligned_down`, never its own copy, so a planned `num_frames` is one the template's constraint accepts; `crop_face_track` pads its crops to the step's `modulus`/`remainder` grid through `aligned` too (`padding_to_grid`). | `tests/test_variable_constraints.py::TestTheGrid::test_without_snap_an_off_grid_value_is_refused_not_rounded`, `tests/test_variable_constraints.py::TestTheGrid::test_plan_cuts_rounds_through_the_constraint_owner`, `tests/test_face_track.py::TestPadding::test_padding_to_grid_rounds_through_the_constraint_owner`, `tests/test_variable_constraints.py::TestTheRunTimePass::test_an_illegal_value_is_refused_before_anything_loads` | | Bound acknowledgement | `dw/plan.py`: `fingerprint`, `dw/server/admission.py`: `check_bound_acknowledgement` | A bound `acknowledged_cost` is refused with 409 when the plan's fingerprint or its required downloads changed. The fingerprint is keyed only by what can change the output: `DOCUMENTATION_KEYS` (`cost`, `cost_drivers`, `vram_estimate`, descriptions, catalog shape/traits), a constraint's `reason` and a result's role, subfolder and file name are left out, of the workflow and of each composed child; every schema key is sorted into left out or kept. A number digests by value, not spelling (`12.0` as `12`), and the seed stays out (#85). | `tests/test_plan.py::TestFingerprintKeys::test_the_exclusion_lists_are_pinned`, `tests/test_plan.py::TestTheCatalogH3Template::test_a_client_round_trip_keeps_it`, `tests/test_plan.py::TestFingerprintKeys::test_every_top_level_schema_key_is_sorted`, `tests/test_server.py::TestBoundRerun::test_a_rerun_bound_to_a_stale_plan_is_refused`, `tests/test_server.py::TestBoundAcknowledgement::test_a_matching_fingerprint_queues` | -| Per-repo gating | `dw/plan.py`: `build_plan` | Each gated repo, including a `loras` entry's repo, is probed and reported on its own. | `tests/test_plan.py::TestDownloadsRequired::test_a_gated_repo_this_token_lacks_access_to_is_blocked` | +| Per-repo gating | `dw/plan_downloads.py`: `_model_info`, `_probe_gate_blocked`, assembled by `dw/plan.py`: `build_plan` | Each gated repo, including a `loras` entry's repo, is probed and reported on its own. | `tests/test_plan.py::TestDownloadsRequired::test_a_gated_repo_this_token_lacks_access_to_is_blocked` | +| Download sources | `dw/plan_downloads.py`: `downloads_required`, `_collect_sources`, `_collect_component_overrides`, `_sub_workflow_paths` (which `plan.estimate` walks children by too; the module never imports `dw/plan.py` back, and `plan.py` re-exports `downloads_required` and `gate_warnings`) | `downloads_required` lists every `model_name`, `loras` repo, single-file URL and `load_components.pretrained_model_name_or_path` override (string, per-component dict, `default`), once each. The base repo stays listed whole: an override's replaced subfolder is not subtracted, so the list is a floor. | `tests/test_plan.py::TestDownloadsRequired::test_a_load_components_repo_override_is_listed` | +| Task weights | each fetching task's `*_source(arguments)` (e.g. `dw/tasks/remove_object.py`: `remover_source`); `dw/tasks/weights.py`: `argument_or_default`, `file_entries`, `package_dir`; `dw/tasks/registry.py`: `required_weights_hook`, `register_weights_fallback`; `dw/tasks/image_utils.py`: `processor_weights_hook`; `dw/security.py`: `validate_weight_name`; `dw/plan_downloads.py`: `_collect_task_weights`, `_file_info` | A model-backed task declares `required_weights(arguments)` on its `register_command`, and `downloads_required` lists those repos beside the pipeline `model_name`s (a shared repo once; a reference or local-path argument none). A single-file task lists `{repo, filename, revision}` entries, checked per file against the cache; `restore_faces` also lists facexlib's weights as URL entries. A task's declaration and its loader read the same resolution of which argument wins (a single-file task's `*_source(arguments)`, an overridable whole-repo task's `argument_or_default` call, a fixed repo's module constant), so they cannot differ; `argument_or_default` is the shared helper only where `None` means the default (`remover_source` and `upscaler_source` treat any falsy value as absent; `spandrel_source` has no default). `file_entries` builds a file entry (its name checked by `validate_weight_name`), `package_dir` finds facexlib's weights dir without importing it, and `_file_info` sizes one file. The lookup path is acyclic: `dw/locations.py` reads the registry's tables and the task modules import the location policy and `weights.py`, so the registry imports no task module (an image processor's hook reaches it through `register_weights_fallback`, which `image_utils` calls at import) and `weights.py` imports only `dw/security.py`, where `validate_weight_name` lives for that reason. A task module the hook lookup imports keeps `transformers` out of its module top (`hf_pipeline` is a lazy wrapper, patched by name in the tests), so a validate never pays for it. | `tests/test_task_weights.py`, `tests/test_task_weights.py::TestTheHookPathStaysAcyclic::test_weights_imports_only_the_security_leaf_from_dw`, `tests/test_task_weights.py::TestTheHookPathStaysAcyclic::test_the_registry_imports_nothing_from_dw` | | Observed cost | `dw/server/observed_cost.py`: `declared_drivers`, `ObservedCosts._on_this_card`, `ObservedCosts.card` | `observed` comes only from this box's finished jobs on the named card (the one `route` names, for `plan.estimate`), else the server's own card; `ObservedCosts.card` owns a card's (kind, name, default). Rows are matched by card name whatever index they ran on (a job from before jobs carried a `device` counts only while the server is on the default card), and never writes `cost`. Cold and warm runs are reported separately. | `tests/test_observed_cost.py::TestColdIsNotWarm::test_the_two_are_reported_separately_each_with_its_runs`, `tests/test_observed_cost.py::TestByCard::test_only_rows_from_this_card_count_whichever_index_they_had`, `tests/test_observed_cost.py::TestByCard::test_rows_from_before_jobs_carried_a_device_do_not_count_off_the_default_card` | | The estimate | `dw/plan.py`: `estimate`, `_tempered`; the driver scaling is `dw/cost_scaling.py`: `scale_curated`, `_driver_scaling` | `plan.estimate` quotes the observed figure ahead of the curated one, and blends it toward the curated figure below three runs. A `cost` entry is measured at the workflow's variable defaults; a run moving `num_inference_steps`, `width`/`height` or `num_frames` (per list entry, summed) scales it in proportion (`scale_curated`, which `dw/server/deps.py` reuses for an inherited figure), basis `derived` with `scaled` naming each axis, times the list-length re-price, and that scaled figure is what a thin observed one blends toward (#866). | `tests/test_plan.py::TestObservedEstimate::test_history_beats_a_curated_figure`, `tests/test_plan.py::TestLowConfidenceObservedEstimate::test_a_single_run_blends_toward_the_curated_figure`, `tests/test_plan.py::TestDriverScaling::test_steps_canvas_and_frames_multiply`, `tests/test_plan.py::TestScalingAccuracy::test_within_thirty_percent` | | Cost drivers | `dw/server/observed_cost.py`: `declared_drivers` | A cost driver that names no declared variable is dropped. | `tests/test_observed_cost.py::TestComparability::test_a_driver_naming_no_variable_is_dropped`, `tests/test_observed_cost.py::TestTheCatalogsDriversAreReal::test_every_declared_driver_is_a_variable_of_its_workflow` | | Raw workflow GET | `dw/server/routes/library.py`: `get_workflow` | The raw workflow GET is served verbatim, with no `observed`, because the editor saves what it reads. | — | | VRAM projection | `dw/vram_estimate.py`, `dw/guides.py`: `guide_lengths` | A template's `vram_estimate` is projected per step after `for_each` expansion, and only the largest step over the ceiling is reported. With `bytes_per_guide_voxel` declared, each guide the step lays in (its own non-null `guides` plus a guide chain's) adds its snapped frames times the canvas; an unprobeable clip is charged at `num_frames`, and validate, admission and run project the same number. | `tests/test_vram_estimate.py::test_only_the_largest_over_ceiling_step_is_reported_once_per_cost_entry`, `tests/test_vram_guides.py::test_validate_apply_and_required_agree_on_a_guided_definition` | | H3 Ref2VA VRAM numbers | `workflows/templates/minimax/` | Every Ref2VA template declares `base_gb` 16.0, `bytes_per_voxel` 28.71 and `gb_per_reference` 1.0. | `tests/test_h3_vram_ceiling.py::test_every_ref2va_minimax_template_declares_gb_per_reference` | -| Inherited VRAM ceiling | `dw/vram_inheritance.py`, `dw/server/deps.py`: `ceiling_index` (the catalog parse is `_catalog_listing`, shared with the cost lookup below) | A workflow with no `vram_estimate` is matched against the catalog by pipeline identity, and is warned, never refused. | `tests/test_vram_inheritance.py::test_every_template_declaring_one_identity_declares_the_same_numbers` | +| Inherited VRAM ceiling | `dw/vram_inheritance.py`, `dw/server/deps.py`: `ceiling_index` (the catalog parse is `_catalog_listing`, shared with the cost lookup below) | A workflow with no `vram_estimate` is matched against the catalog by pipeline identity, and is warned, never refused. Under the ceiling it still gets a `vram_config_differs` warning when its offload or quantization differs from the template's, which the estimate was calibrated on (#954). | `tests/test_vram_inheritance.py::test_every_template_declaring_one_identity_declares_the_same_numbers`, `tests/test_vram_inheritance.py::TestConfigDiffers::test_missing_offload_under_the_ceiling_notes_it` | | Inherited cost | `dw/server/deps.py`: `inherited_observed`, `dw/vram_inheritance.py`: `inherited_differences`, `dw/plan.py`: `_observed` (basis `inherited`), `dw/server/routes/jobs.py`: `_inherited_cost_warnings` | An inline workflow (no catalog name) is priced from the observed runs of the catalog template with the same `component_type` + `model_name` + `workflow` key as `vram_inheritance`; the template with the most cold runs wins, then name. `differs` lists offload, quantization, frame count, attention backend, cache, LoRA and reference count (the `references` list lengths, per-entry `item:` ones read off the `for_each` list; named, not scaled, #915) set differently from the template, and sets `low_confidence` when it lists any (#866). More pipeline steps than the template run (a `for_each` counted per member, against the list the bucket ran, so a fallback to the default bucket re-prices a longer list) re-price its minutes linearly, and the mean frames, inference steps, width and height per pipeline step are scaled by `scale_curated` itself - the one rule the catalog figure uses, so an inline copy and its template quote one request alike - with `scaled` naming why and `low_confidence` set (#851, #866). It is warned, never refused. | `tests/test_inherited_cost.py::TestPricedByPipelineIdentity::test_an_inline_copy_inherits_the_templates_runs`, `tests/test_inherited_cost.py::TestDifferences::test_offload_quantization_and_frames_are_named`, `tests/test_inherited_cost.py::TestDifferences::test_a_reference_count_other_than_the_templates_is_named`, `tests/test_inherited_cost.py::TestThePlanReportsIt::test_the_estimate_is_inherited_and_the_validate_answer_warns`, `tests/test_inherited_cost.py::TestTheTemplateAndAnInlineCopyAgree::test_the_same_minutes_within_rounding`, `tests/test_inherited_cost.py::TestAListLongerThanTheFallbackBucket::test_an_inline_six_shot_copy_scales_by_members` | | Projected host-RAM peak | `dw/server/host_ram_need.py`: `memory_key`, `projected_peak`, `host_memory_block`; the fit is `dw/host_memory_projection.py`: `projected_mb`; history by key is `dw/server/job_history.py`: `peaks_for_key` | A run's memory key is the sorted pipeline identity of its pipeline steps plus every `quantization_config` / `offload` / `group_offload` value, with arguments substituted through `replace_variables`; never the workflow's name, so a renamed or inline copy shares the template's history. `observed` is the max peak over this machine's successful same-key runs and comes ahead of the list-run fit, unless the shape holds its pipeline across the list and the fit is the larger (the key leaves list length out, #910); with neither there is no figure. It never refuses; dispatch may delay a start on it (`dw/server/dispatch.py`: `Dispatcher._next_dispatch`, #869) (`Admission.host_ram_need`, validate's `plan.host_memory`). | `tests/test_host_ram_need.py::TestMemoryKey::test_an_inline_copy_without_its_metadata_has_the_same_key`, `tests/test_host_ram_need.py::TestMemoryKey::test_a_different_weights_dtype_is_a_different_key`, `tests/test_host_ram_need.py::TestProjectedPeak::test_an_observed_peak_beats_the_fit`, `tests/test_host_ram_need.py::TestALongerListThanObserved::test_an_accumulating_list_takes_the_larger_fit`, `tests/test_host_ram_need.py::TestProjectedPeak::test_a_failed_run_is_not_observed`, `tests/test_host_ram_need.py::TestValidateRoute::test_an_observed_run_is_the_plans_host_memory` | | Acceleration warnings | `dw/acceleration_warnings.py`: `acceleration_warnings`, `CACHE_MIN_USEFUL_STEPS`, `CUDA_ONLY_ATTENTION_BACKENDS` (the last entry of `WARNING_CHECKS`) | An acceleration setting that can't pay off (a `first_block`/`mag` cache under `CACHE_MIN_USEFUL_STEPS`, any `taylorseer` cache, `compile.fullgraph` with a cache, a cache on a pipeline class with no transformer or whose transformer lacks `enable_cache()`, a CUDA-only `attention_backend` on a non-CUDA device) is a warning and never a refusal, and an unresolvable value (steps, backend, device) leaves it silent. | `tests/test_validation.py::TestAccelerationWarnings::test_warnings_never_refuse`, `tests/test_validation.py::TestAccelerationWarnings::test_steps_that_resolve_to_nothing_are_silent`, `tests/test_validation.py::TestAccelerationWarnings::test_an_unknown_device_is_silent`, `tests/test_validation.py::TestAccelerationWarnings::test_an_unresolvable_backend_is_silent_on_mps` | | H3 sparse attention | `dw/sparse_attention.py`: `sparse_call`, `install_sparse`, `install_sparse_for_call`, `apply_sparse_settings`, `SparseH3AttnProcessor`; wired from `dw/pipeline_processors/components.py`: `configure_components` and `dw/pipeline_processors/pipeline.py`: `apply_runtime_settings` | Whether a pipeline call runs sparse is decided once per call, all or nothing (the call's first kernel invocation included); it never changes diffusers' process-wide backend and never reaches `AttentionBackendName`; it installs by a processor swap plus forward hooks, never a `forward` wrapper; and `sparse_tau` is a runtime setting, not part of the weights identity (#931). | `tests/test_sparse_attention_processor.py`, `tests/test_sparse_attention_wiring.py`, `tests/test_sparse_attention_tiles.py`, `tests/test_sparse_attention_backends.py` | | H3 adapter partition | `dw/adapter_compatibility.py`; the server's index is `dw/server/deps.py` `lora_index` | The LoRA catalog decides first, on the server only: a `proven` entry's `workflow` refuses a step in the other partition, a `trial`/`rejected` entry's only warns. A file no entry places falls to its name: an FL2VA LoRA on a reference step is refused, and one whose name says neither `ref2v` nor `fl2v` is warned. | `tests/test_h3_adapters.py::TestWhatIsRefused::test_a_keyframe_adapter_on_the_reference_path`, `tests/test_h3_adapters.py::TestWhatIsRefused::test_an_unrecognised_name_is_a_warning_not_a_refusal`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_proven_entry_refuses_the_other_partition`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_weaker_entry_only_warns`, `tests/test_server_loras.py::TestTheCatalogPlacesAnAdapter::test_a_trial_entry_warns_and_promoting_it_refuses` | -| H3 audio hold, refine and guide layout | Five modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules asked before validation and before the call; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants), `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`) and `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`), which import diffusers when they load, so the two owners import them only inside `blocks()` and `guide_blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline.py` adds its blocks for a `guides` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/hold_audio.py`: `hold_audio_errors`, `refine_strength_errors` | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and they are no-ops without `hold_audio` / `refine_strength`. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers` | +| H3 audio hold, refine, tiled refine and guide layout | Six modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules, `refine_tiles` among them, asked before validation and before the call; `tile_plan`, the tile layout on the 32 px grid, shared by validate and the run; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants), `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`; `HELD_LATENTS_OUTPUT` and `encoded_hold_audio`, the hold block's encode handed back to it), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`) and `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`), which import diffusers when they load, so the two owners import them only inside `blocks()` and `guide_blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline.py` adds its blocks for a `guides` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/pipeline_processors/h3_tiles.py`: the tiled refine (the tile loop, the latent-space feather blend, the per-tile crop of one whole-canvas noise draw, the decode-free graph variant, and the one spatially tiled decode), which imports `h3_rules`, `h3_hold` and `media_types` (for `FrameList`; and `events`, `progress`) and is imported by `pipeline.py` and, lazily, `components.py` (which attaches the variant at load, behind `h3_hold.core_denoise_sequences`), never by `h3_rules` or `h3_hold`, `dw/hold_audio.py`: `hold_audio_errors`, `refine_strength_errors` | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and they are no-ops without `hold_audio` / `refine_strength`. The tiled refine adds a second graph built at load beside them (`h3_tiles.decode_free_blocks`: the decode and text-encoder blocks dropped, every kept block the graph's own instance, so hold and refine stay in) and swaps it into `pipeline._blocks` only for the duration of the one text-encode call and the tile calls (`swapped_blocks`, restored on failure). Every tile runs the hold block, but the track is encoded once per refine: the first tile is asked for the block's encode (`HELD_LATENTS_OUTPUT`) and every later tile's `hold_audio` carries it (`encoded_hold_audio`, an `EncodedAudioReference`), which the block writes in place of running the audio VAE. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_tiles.py::TestDecodeFree::test_variant_has_no_decode_and_shares_blocks`, `tests/test_h3_tiles.py::TestTileLoop::test_blocks_are_restored_when_a_tile_fails`, `tests/test_h3_tiles.py::TestTileLoop::test_the_track_is_encoded_by_one_tile_and_handed_to_the_rest`, `tests/test_h3_hold_audio.py::TestHoldBlock::test_a_carried_encode_is_written_without_encoding_again`, `tests/test_h3_tiles.py`, `tests/test_h3_rules.py` (`tile_plan` and the `refine_tiles` rules), `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers` | | IC-LoRA reference scale | `workflows/templates/ltx2/` | The three conditioning templates run at `reference_downscale_factor: 1`, and the two upscalers at 2. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_reference_is_encoded_at_the_output_resolution`, `tests/test_ltx2_ic_loras.py::TestTheGenerativeUpscaleMatchesTheSameCard::test_it_loads_the_upscaler_at_factor_two` | | IC-LoRA numbers | `workflows/templates/ltx2/` | Every number in the IC-LoRA templates comes from the vendor's model card. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_strength_is_the_cards_default`, `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_defaults_are_the_trained_bucket` | | IC-LoRA prompt genre | `prompts/ltx2/` | A stored prompt tagged `ic-lora` is checked against its trained caption form, not against the T2V paragraph rule. | `tests/test_ltx_prompt_library.py::test_an_ic_lora_prompt_is_in_its_trained_form` | @@ -122,7 +126,7 @@ one to open. | A different result | `dw/server/routes/jobs.py`, `dw_mcp/tools_jobs.py`: `rerun_job` | `rerun` with `new_seed` draws a fresh seed into the workflow's seed variable. | `tests/test_rerun_new_seed.py::test_a_rerun_with_a_new_seed_draws_one_into_that_variable` | | Worker protocol | `dw/worker_protocol.py`: `parse_reply` | Every command and reply is a frozen dataclass that travels as a wire dict. A run event's keys sit beside the wire's own `type`, so an event's `type` key (`cache_stats`' cache type) crosses as `event_type` and comes back as `type`. | `tests/test_worker_messages.py::test_from_wire_inverts_to_wire`, `tests/test_cache_stats.py::test_the_event_crosses_the_worker_wire_with_its_cache_type`, `tests/test_worker_messages.py::test_an_unknown_reply_type_is_kept_whole_rather_than_raised` | | Persistent worker | `dw/worker.py`, `dw/worker_manager.py`, `dw/serve.py` | Jobs run in spawned workers (one per card) that keep models loaded, so a change to engine code needs a server restart. | — | -| Worker card pinning | `dw/devices.py`: `worker_environment`, `pinned_environment`, `resolve_serve_devices`, `check_device_present`, `label_of` | A worker named with a CUDA index is spawned with `CUDA_VISIBLE_DEVICES` set to that card and `DW_DEVICE=cuda` (a bare `cuda` pins nothing), so index-0 reads inside it measure its own card; `--devices`/`devices` takes one entry per worker (a card named twice, or a bare `cuda` in a list of several, is refused), a card the box lacks is refused at startup naming the cards present, and a job stores the card it ran on as `device_ordinal` and `device_card` (#693), job and history alike: rerun affinity reads the ordinal, observed cost buckets by the card, and the `device` label `"cuda:N "` is only `label_of` joining the two - nothing splits a label back apart. | `tests/test_worker_manager.py::TestWorkerPinning::test_a_cuda_index_pins_the_worker_through_the_environment`, `tests/test_worker_manager.py::TestWorkerPinning::test_a_bare_cuda_sets_no_pin`, `tests/test_serve_main.py::TestConfigureDevices::test_a_refusal_exits_with_code_2`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_the_machine_lacks_is_refused_naming_the_cards_present`, `tests/test_serve_main.py::TestResolveServeDevices::test_two_entries_are_both_kept`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_named_twice_is_refused` | +| Worker card pinning | `dw/devices.py`: `worker_environment`, `pinned_environment`, `resolve_serve_devices`, `check_device_present`, `label_of` | A worker named with a CUDA index is spawned with `CUDA_VISIBLE_DEVICES` set to that card and `DW_DEVICE=cuda` (a bare `cuda` pins nothing), so index-0 reads inside it measure its own card; `--devices`/`devices` takes one entry per worker (a card named twice, or a bare `cuda` in a list of several, is refused), a card the box lacks is refused at startup naming the cards present (in a list of several it is dropped with a warning and the server starts on the rest; only a list with no card present refuses), and a job stores the card it ran on as `device_ordinal` and `device_card` (#693), job and history alike: rerun affinity reads the ordinal, observed cost buckets by the card, and the `device` label `"cuda:N "` is only `label_of` joining the two - nothing splits a label back apart. | `tests/test_worker_manager.py::TestWorkerPinning::test_a_cuda_index_pins_the_worker_through_the_environment`, `tests/test_worker_manager.py::TestWorkerPinning::test_a_bare_cuda_sets_no_pin`, `tests/test_serve_main.py::TestConfigureDevices::test_a_refusal_exits_with_code_2`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_the_machine_lacks_is_refused_naming_the_cards_present`, `tests/test_serve_main.py::TestResolveServeDevices::test_two_entries_are_both_kept`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_gone_from_a_list_is_dropped_and_reported`, `tests/test_serve_main.py::TestConfigureDevices::test_a_pulled_card_in_the_setting_starts_on_the_rest_and_warns`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_named_twice_is_refused` | | Failed-run reporting | `dw/worker.py`, `dw/worker_protocol.py`: `Failed`, `Cancelled` | A failed or cancelled run's reply still carries the manifest of the steps that ran. | `tests/test_worker_execute.py::test_failure_carries_the_manifest_of_the_steps_that_ran`, `tests/test_worker_execute.py::test_cancellation_carries_the_manifest_too` | | Failure path redaction | `dw/path_redaction.py`: `redact_paths`, called by `dw/worker.py` | A failed run's message and traceback name a file under the job's asset roots or output directory by its `asset:`/`output:` reference, never its absolute server path. | `tests/test_worker_execute.py::test_a_failure_names_an_asset_by_reference_not_by_server_path`, `tests/test_path_redaction.py` | | Run-time warnings | `dw/events.py`: `emit_warning` | A warning found at run time is emitted as an event, so it reaches the caller and not just the log. | `tests/test_concat_videos.py::TestWarningsReachTheCaller::test_the_level_spread_warning_is_emitted_as_an_event` | @@ -136,7 +140,7 @@ one to open. | Output comparison | `dw/media_compare.py`: `compare`, `psnr`, `ssim`, `mean_abs_diff`, `check_request`, `crop_box` | Metrics always compare the same pixels: one shared crop is the measured region, and per-output crops are display only, leaving metrics on the full frame. An output whose size differs from the baseline's, or a moment past its end, gets null metrics with a reason; nothing is resized to force a number. SSIM is on Rec. 601 luma (SSIM's convention, not `image_ops.luma`). It runs in the server process and decodes through `dw/media.py` and `open_bounded`. | `tests/test_compare_route.py::test_a_shared_crop_is_the_region_measured`, `tests/test_compare_route.py::test_per_output_crops_are_display_only`, `tests/test_compare_route.py::test_a_size_mismatch_gets_no_numbers_and_says_why`, `tests/test_compare_route.py::test_a_moment_past_the_shorter_clip_is_a_blank_cell` | | Signal processing | `dw/dsp.py` | It measures and transforms waveforms, decides nothing, and imports nothing from `dw`. | `tests/test_media_layering.py::test_dsp_imports_nothing_from_dw` | | Assessment probes | `dw/tasks/assess.py`, `dw/assessment_rules.py` | Exactly three registered commands are assessment probes. Each measures a finished file, and every rule names a real probe field. | `tests/test_assessment_rules.py::TestRuleProbesAreRealCommands::test_assessment_is_exactly_the_three_probes`, `tests/test_assessment_rules.py::TestRuleProbesAreRealCommands::test_every_rule_names_a_registered_json_command` | -| Script check | `dw/tasks/script_check.py`: `check`; `dw/script_lines.py`: `parse_lines`, `parse_shots`, `shot_names_error` | `check_script` aligns a take's heard words to its expected lines and guards out words heard over silence and repetition loops (Whisper's hallucination over music). It builds its findings with `assessment_rules.finding()` and reads the guard's floor and window (`DEAD_AIR_FLOOR_DBFS`, `DEAD_AIR_WINDOW`) rather than copying them; the loudest-window measure is `dsp.loudest_window_dbfs` (`script_check.word_level` wraps it), which `transcribe_audio`'s `suspect: "silent"` mark reads with the same floor and window; the repetition-loop detector (`script_lines.looped_texts`, `REPEAT_RUN_MIN`, `REPEAT_MAX_PERIOD`) lives in `dw/script_lines.py` and backs both `check_script`'s `repetition` discard and `transcribe_audio`'s `suspect: "repetition"`, and `script_lines.suspect_runs`/`SUSPECT_LEFT_OUT` are what `plan_cuts` groups and leaves out suspect chunks by - so neither task imports the other. It is not an assessment probe and adds nothing to `RULES`. Its literal `lines` is refused at validation through `task_problems` (`script_lines_errors`), which calls the same `parse_lines` the task runs, and a literal `shots` through the same `parse_shots` and `shot_names_error`; the parsers live in `dw/script_lines.py`, not the task, so that `task_problems` reaches them without importing `script_check` (the `import_cycles` ratchet in `scripts/arch_metrics.py`). A line's `shot` may be the short `@name` of the one recorded shot that carries it (`script_lines.matching_shot_names`, `with_full_shot_names`, shared by the task and the literal-`shots` check); which member key a recorded name carries comes from `references.member_keys`, beside `MEMBER_SEPARATOR` and `for_each.member_name`, never a re-derived pattern: a key ends at the first character outside `references.VARIABLE_NAME_CHARACTERS`, the set `security.validate_variable_name` (which checks a for_each entry's name) builds its pattern from. With `audio` an `output:`/`asset:`/path whose run manifest records shots, `dw/shot_span_preflight.py: script_shot_name_errors` (the `script_shot_names` error check) runs `shot_names_error` at validation, reading the map through `runs.shots_beside`. Its shot map comes from `assess.resolve_shots`, the one the probes use, never a copy; it places each shot on the soundtrack with `assess.sample_span`, decides repeated shot names with `shots.duplicate_shot_names`, and asks `locations.is_http_url` whether the take is a URL. | `tests/test_script_check.py::TestRegistration::test_registered_as_json_command_without_assessment_flag`, `tests/test_script_check.py::TestEmptyLines::test_words_only_over_silence_are_discarded_not_found`, `tests/test_script_check.py::TestLinesErrors::test_literal_with_a_colon_is_not_a_reference`, `tests/test_script_check.py::TestShotValidation::test_a_short_member_name_matches_the_one_shot_carrying_it`, `tests/test_script_check.py::TestShotValidation::test_a_valid_variable_name_comes_back_whole_as_a_member_key`, `tests/test_shot_span_preflight.py::TestScriptShotNames::test_a_wrong_shot_name_is_an_error_at_validate_time` | +| Script check | `dw/tasks/script_check.py`: `check`; `dw/tasks/script_take.py`: `guard_words`, `word_level`, `shot_spans`; `dw/script_lines.py`: `parse_lines`, `parse_shots`, `shot_names_error` | `check_script` aligns a take's heard words to its expected lines and guards out words heard over silence and repetition loops (Whisper's hallucination over music). It builds its findings with `assessment_rules.finding()` and reads the guard's floor and window (`DEAD_AIR_FLOOR_DBFS`, `DEAD_AIR_WINDOW`) rather than copying them; the loudest-window measure is `dsp.loudest_window_dbfs` (`script_take.word_level` wraps it; the guard, the transcription levelling and the shot placement are `dw/tasks/script_take.py`, which `script_check` reads one way - nothing there reads the rule tables), which `transcribe_audio`'s `suspect: "silent"` mark reads with the same floor and window; the repetition-loop detector (`script_lines.looped_texts`, `REPEAT_RUN_MIN`, `REPEAT_MAX_PERIOD`) lives in `dw/script_lines.py` and backs both `check_script`'s `repetition` discard and `transcribe_audio`'s `suspect: "repetition"`, and `script_lines.suspect_runs`/`SUSPECT_LEFT_OUT` are what `plan_cuts` groups and leaves out suspect chunks by - so neither task imports the other. It is not an assessment probe and adds nothing to `RULES`. Its literal `lines` is refused at validation through `task_problems` (`script_lines_errors`), which calls the same `parse_lines` the task runs, and a literal `shots` through the same `parse_shots` and `shot_names_error`; the parsers live in `dw/script_lines.py`, not the task, so that `task_problems` reaches them without importing `script_check` (the `import_cycles` ratchet in `scripts/arch_metrics.py`). A line's `shot` may be the short `@name` of the one recorded shot that carries it (`script_lines.matching_shot_names`, `with_full_shot_names`, shared by the task and the literal-`shots` check); which member key a recorded name carries comes from `references.member_keys`, beside `MEMBER_SEPARATOR` and `for_each.member_name`, never a re-derived pattern: a key ends at the first character outside `references.VARIABLE_NAME_CHARACTERS`, the set `security.validate_variable_name` (which checks a for_each entry's name) builds its pattern from. With `audio` an `output:`/`asset:`/path whose run manifest records shots, `dw/shot_span_preflight.py: script_shot_name_errors` (the `script_shot_names` error check) runs `shot_names_error` at validation, reading the map through `runs.shots_beside`. Its shot map comes from `assess.resolve_shots`, the one the probes use, never a copy; it places each shot on the soundtrack with `assess.sample_span`, decides repeated shot names with `shots.duplicate_shot_names`, and asks `locations.is_http_url` whether the take is a URL. | `tests/test_script_check.py::TestRegistration::test_registered_as_json_command_without_assessment_flag`, `tests/test_script_check.py::TestEmptyLines::test_words_only_over_silence_are_discarded_not_found`, `tests/test_script_check.py::TestLinesErrors::test_literal_with_a_colon_is_not_a_reference`, `tests/test_script_check.py::TestShotValidation::test_a_short_member_name_matches_the_one_shot_carrying_it`, `tests/test_script_check.py::TestShotValidation::test_a_valid_variable_name_comes_back_whole_as_a_member_key`, `tests/test_shot_span_preflight.py::TestScriptShotNames::test_a_wrong_shot_name_is_an_error_at_validate_time` | | Audio sync against a source slice | `dw/tasks/measure_sync.py`: `measure_sync`; `dw/dsp.py`: `envelope_lag` | `measure_sync` cross-correlates the onset envelopes of a shot's audio and its source slice and answers the lag and a 0-1 confidence. A positive lag means the audio is LATE (its events fall after the reference's), negative early; `lag_seconds` is None for a silent or flat track. It builds its findings (`sync_unmeasurable`, `sync_low_confidence`, `audio_out_of_sync`) with `assessment_rules.finding()` over module-local rule dicts and adds nothing to `RULES`; it reads the silence floor from `beats.SILENT_DBFS`. `envelope_lag` imports nothing from `dw`. | `tests/test_measure_sync.py::TestLag::test_late_audio_is_a_positive_lag`, `tests/test_measure_sync.py::TestLag::test_early_audio_is_a_negative_lag`, `tests/test_measure_sync.py::TestUnmeasurable::test_a_silent_reference_has_no_lag` | ## Security and architecture guardrails @@ -168,6 +172,7 @@ one to open. | dw disk caches | `dw/disk_caches.py`; `dw/server/jobs.py`: `JobManager.hold_dispatch`; `dw/server/routes/system.py`: `clear_cache`, `delete_cache_entry` | Members `inductor`, `triton`, `quantized`. Clear only a root carrying the `.dw-cache` marker, never while a job is running or queued (`quantized` also not during a model download); an entry delete takes only a 64-hex id, checked before any path is built; clear renames into `.clearing-*` then deletes. The busy check and the dispatch hold are one step in `JobManager.hold_dispatch`, kept for the whole clear or entry delete: a job submitted meanwhile is accepted but stays queued until it ends. | `tests/test_disk_caches.py`, `tests/test_disk_caches.py::test_job_submitted_during_a_clear_waits_for_it` | | Job-to-run link | `dw/server/job_record.py`, `dw/server/jobs.py`: `JobManager.realized` | A job records `run_id`, `run_dir` and `run_version`, and `JobManager.realized` reads that run's `workflow.json`, confined to the output root. | `tests/test_server_jobs.py::test_realized_reads_the_file_the_run_wrote`, `tests/test_server_jobs.py::test_realized_refuses_a_run_dir_that_escapes_the_output_root` | | Job history | `dw/server/job_history.py`: `JobHistory`, `_ADDED_COLUMNS` | Finished jobs persist in `jobs.sqlite`, in WAL mode, so the Jobs view survives a restart. A database from an older release gains each column it lacks from the `_ADDED_COLUMNS` table when it is opened. | `tests/test_server.py::test_job_history_survives_restart_and_reruns`, `tests/test_job_history_wal.py::test_the_jobs_database_uses_wal_mode`, `tests/test_server_jobs.py::test_a_v0_13_0_database_gains_the_columns_added_since` | +| Job fields, live and persisted | `dw/server/job_record.py`: `Job.summary`, `Job.detail`; `dw/server/job_history.py`: `_RECORDED`, `_ADDED_COLUMNS`, `recent_summaries`, `_to_detail` | A job field is declared once per side: on `Job` (and its `summary()`/`detail()`), and in history as an `_ADDED_COLUMNS` entry plus one `(column, getter)` in `_RECORDED`. History readers take columns by name, never by position, and a historical job serves the same summary and detail keys a live one does, apart from the live-only poll state and the history-only markers the parity test declares. | `tests/test_job_history_parity.py::test_history_serves_the_keys_a_live_job_does` | | HTTP security | `dw/server/http_security.py`: `install_middleware`, `query_token_ok` | Every API route needs the token, and only a route marked `query_token_ok` takes it as `?token=`. | `tests/test_security_auth.py::TestTheTokenGate::test_every_api_spelling_needs_the_token`, `tests/test_security_auth.py::TestTheTokenGate::test_the_query_token_is_refused_where_it_is_not_allowed` | | Prompt enhancement | `dw/server/enhancers.py`: `PRESETS`, `_T2I_SYSTEM_PROMPT` | `PRESETS` is the Enhance panel's menu (label, curated LLMs, preselect hints); a preset's family-specific knowledge, its spec, lives in a builtin under `dw/workflows/`, never in Python. `t2i`'s generic, family-agnostic system prompt stays in Python. A third preset moves the menu into builtin metadata (#696). | `tests/test_prompt_library.py::test_every_enhancer_preset_targets_a_known_family`, `tests/test_server.py::TestEnhance::test_presets_are_listed`, `tests/test_server.py::TestEnhance::test_enhance_workflows_are_schema_valid`, `tests/test_server.py::TestEnhance::test_an_unknown_preset_is_a_client_error` | @@ -178,6 +183,7 @@ one to open. | `dw_mcp` stays torch-free | `dw_mcp/` | `dw_mcp` reaches `dw.serve` over HTTP and imports no `dw` module, because `dw/__init__.py` pulls in torch. | `tests/test_mcp_server.py::TestStartupWeight::test_the_server_starts_without_importing_the_engine` | | Tool surface | `dw_mcp/server.py`, `dw_mcp/tools_*.py` | Only these modules import the MCP SDK, and each tool body is a one-line call into a handler. | `tests/test_mcp_server.py::test_the_wiring_table_covers_every_registered_tool`, `tests/test_mcp_server.py::test_the_stated_tool_count_is_the_registered_one` | | Surface text budget | the tool docstrings in `dw_mcp/tools_*.py`, the instructions in `dw_mcp/server.py` | The instructions and each tool description stay at or under 2,048 characters, and the whole surface stays within its token budget. | `tests/test_mcp_server.py`: `SURFACE_BUDGET`, `CLIENT_TEXT_LIMIT`; `tests/test_mcp_server.py::test_no_text_the_agent_reads_is_cut_off_by_the_client`, `tests/test_mcp_server.py::test_the_tool_surface_fits_the_budget` | +| Guide pointers in resident text | the tool docstrings in `dw_mcp/tools_*.py`, the instructions in `dw_mcp/server.py`, each plugin `SKILL.md` and its `references/` | Every `get_guide(...)` / guide-section pointer opens a real section through `dw/server/guides.py`'s `get_guide`, and the tools that point at a guide keep their pointer when text is trimmed to fit the budget. | `tests/test_guide_pointers.py` (`TOOLS_THAT_POINT`) | | Spending needs consent | each handler behind a tool that takes `acknowledged_cost` (`dw_mcp/diagnose.py`, `models.py`, `prompts.py`, `workspaces.py`) | Every tool that takes `acknowledged_cost` refuses until it is set; `delete_workspace`'s refusal is the server's 409. | `tests/test_mcp_twins.py::test_every_tool_that_takes_acknowledged_cost_refuses_without_it` | | Copies of engine rules | the constants and word lists in `dw_mcp/` and `dw/run.py` | `dw_mcp` cannot import its owners, so each copy equals its owner; a copy that needs no twin is deleted, not pinned. | `tests/test_mcp_twins.py` | | The CLI is a client of the server | `dw/run.py` | `dw.run` queues on `dw.serve` through `dw_mcp.client`, so `dw` imports `dw_mcp` and never the reverse. | `tests/test_mcp_server.py::TestStartupWeight::test_the_server_starts_without_importing_the_engine`, `tests/test_mcp_twins.py` | diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index d015a611e..088443bc4 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -47,7 +47,7 @@ pyproject; install.ps1 installs it explicitly on Windows. ## Platform-Specific Dependencies -**All platforms (core ML):** peft, transformers, accelerate, safetensors, controlnet_aux, sentencepiece, torchsde, torchao, optimum-quanto, gguf, kornia, ftfy, sdnq, spandrel, facexlib (spandrel + facexlib back the `upscale` and `restore_faces` tasks). On first use, `restore_faces` has facexlib download its face detection and parsing weights (about 110 MB) from facexlib's own GitHub releases into the facexlib package directory. It is the one model download that bypasses the Hugging Face cache, dw's download manager and the `plan.downloads_required` estimate. +**All platforms (core ML):** peft, transformers, accelerate, safetensors, controlnet_aux, sentencepiece, torchsde, torchao, optimum-quanto, gguf, kornia, ftfy, sdnq, spandrel, facexlib (spandrel + facexlib back the `upscale` and `restore_faces` tasks). On first use, `restore_faces` has facexlib download its face detection and parsing weights (about 110 MB) from facexlib's own GitHub releases into the facexlib package directory. It is the one model download that bypasses the Hugging Face cache and dw's download manager; a validate's `plan.downloads_required` lists the two weights as URL entries while they are missing from that directory. **All platforms (sparse attention):** comfy-kitchen (Sol-Attn sparse attention for MiniMax H3's generated-video rows, #931). Its compiled CUDA/HIP kernels ship for Linux and Windows; elsewhere it installs an eager-only build. diff --git a/docs/MCP.md b/docs/MCP.md index 94f72e32e..cbda32031 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -233,7 +233,7 @@ when no single workflow covers it. | `get_health()` | — | Check that the server is alive, and which machine answered: `version`, `device`, whether any model process is resident (`worker_alive`), and the queue depth. The engine runs one job per GPU: `workers` has one entry per card - `device`, `name`, `vram_gb`, `alive`, its own `current_job`, and the worker's `host_memory_rss_mb` while it runs - and the top-level `current_job` is the longest-running job. `worker_alive: false` is the normal idle state on a server that has not run a job since startup or the last memory clear - not a fault - the on-demand worker starts with the next job (#206) | | `get_server_info()` | — | What this installation can do and where it keeps things: `device` (the accelerator a run will use), `version`, the `workspace` this session is working in and the workflow/asset/output/prompt `directories` of *that* workspace, the bind address and port, whether a token is required, and whether MCP is mounted. Check the device before authoring - a CUDA-only choice (bitsandbytes, `torch.compile`, flash attention) is not available on an `mps` or `cpu` server. `runtime` (#222) reports the Python version, torch version and the CUDA version torch was built against, the NVIDIA driver version (when `nvidia-smi` is reachable), and the installed versions of diffusers, transformers, accelerate, bitsandbytes, peft, safetensors and sentencepiece (`null` for one not installed) - for diagnosing an environment mismatch between boxes without shelling in. `caches` lists dw's disk caches (`inductor`, `triton`, `quantized`), each with `name`, `bytes`, `files`, `enabled`, `clearable` and `disk_free`, plus `entry_count` for a cache with entries, and `cache_entries=true` lists them (`id`, `model`, `subfolder`, `component`, `bytes`, `files`, `unreadable`, `created`, `last_hit`; the quantizer and dtype are not in `/api/caches`); it is omitted when the server has no `/api/caches`. There is no MCP tool to clear one - that is `DELETE /api/caches/{name}` or `.../entries/{entry}` ([SERVER.md](SERVER.md)) | | `list_jobs(limit=20, status=None, workspace=None)` | optional `limit` (newest N), `status` (one state or a comma-separated set of `queued`, `running`, `succeeded`, `failed`, `cancelled`), `workspace` | List queued, running and recent jobs, **newest first** (a queued job held for host RAM carries `waiting_for`). Bounded by default: the unbounded listing was over a client's tool-result limit on a server with a few months of history, which made it a tool that could not be called at all. `total` says how many matched and `truncated`/`next` say so when the answer was cut - raise `limit` or narrow with `status`. Without `workspace`, a named workspace lists its own jobs and the default one lists every job the server holds | -| `list_gallery(limit=50, subfolder=None, only_orphans=False, workspace=None, folder=None, version=None, media=False, role=None)` | `limit`, `subfolder`, `only_orphans`, `workspace`, `folder`, `version`, `media`, `role` | List generated output files, newest first. A name is `//`, where `` may sit in the subfolder the step chose (`final/episode.mp4`); each entry carries `folder` (the workflow) and `subfolder` (by convention `final` or `intermediate`, `''` when the step chose none, any path the workflow wrote otherwise), and `subfolder="final"` lists only deliverables. Each entry also carries `role`, `final` or `intermediate` (null for a flat-layout file with no run; `intermediate` once a later step or a later run read the file, unless `result.role` declared it); `role=["final"]` lists only those, and the `role` filter defaults to every file. See `dw/roles.py` for how a role is decided. A file starred in the web UI's gallery (its keep mark) carries `starred: true`; an unstarred one carries no `starred`. Each entry carries `run_id` and `version` - that run's ordinal among the workflow's runs, which is how one of several runs that wrote the same basename is named to a person: the web UI labels the same file `v5`. The number is assigned when the run opens and never renumbered, so deleting a run leaves a gap rather than sliding the rest down (a failed run, or a rerun that reused every step, leaves one too - it took a number and may have nothing to list), and it is `null` under the flat output layout, which has no runs. `folder` with `version` lists that one run's files, and `output:/v5/` names one in a workflow; every other tool takes `name`. Each entry also carries a ready-made `url`, already scoped to the workspace that made it - a hand-built `/outputs/` URL 404s for anything but the default workspace. `only_orphans=True` inverts the call: instead of files, it returns run directories with no media anywhere under them (`runs`, each `{name, mtime}`) - a run whose output was deleted before `delete_output` could remove it by name, or one that failed before writing anything; `subfolder` does not apply in this mode, and `name` is exactly what `delete_output` accepts (#170). `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | +| `list_gallery(limit=50, subfolder=None, only_orphans=False, workspace=None, folder=None, version=None, media=False, role=None)` | `limit`, `subfolder`, `only_orphans`, `workspace`, `folder`, `version`, `media`, `role` | List generated output files, newest first. A name is `//`, where `` may sit in the subfolder the step chose (`final/episode.mp4`); each entry carries `folder` (the workflow) and `subfolder` (by convention `final` or `intermediate`, `''` when the step chose none, any path the workflow wrote otherwise), and `subfolder="final"` lists only deliverables. Each entry also carries `role`, `final` or `intermediate` (null for a flat-layout file with no run, or an older run's file that neither its subfolder nor a later run's reads place; `intermediate` once a later step or a later run read the file, unless `result.role` declared it); `role=["final"]` lists only those, and the `role` filter defaults to every file. See `dw/roles.py` for how a role is decided. A file starred in the web UI's gallery (its keep mark) carries `starred: true`; an unstarred one carries no `starred`. Each entry carries `run_id` and `version` - that run's ordinal among the workflow's runs, which is how one of several runs that wrote the same basename is named to a person: the web UI labels the same file `v5`. The number is assigned when the run opens and never renumbered, so deleting a run leaves a gap rather than sliding the rest down (a failed run, or a rerun that reused every step, leaves one too - it took a number and may have nothing to list), and it is `null` under the flat output layout, which has no runs. `folder` with `version` lists that one run's files, and `output:/v5/` names one in a workflow; every other tool takes `name`. Each entry also carries a ready-made `url`, already scoped to the workspace that made it - a hand-built `/outputs/` URL 404s for anything but the default workspace. `only_orphans=True` inverts the call: instead of files, it returns run directories with no media anywhere under them (`runs`, each `{name, mtime}`) - a run whose output was deleted before `delete_output` could remove it by name, or one that failed before writing anything; `subfolder` does not apply in this mode, and `name` is exactly what `delete_output` accepts (#170). `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | | `get_gallery_metadata(name, envelope=False, workspace=None, interval_seconds=None)` | `name`, `workspace` | Get the metadata embedded in a generated file — or, when `name` is an `asset:` reference, what an *input* asset holds (`source` says which; `job` is null for an asset). Reading an input's duration, frame count, fps and sample rate before a run is how a caller learns the `total_frames`, `fps` and `sample_rate` a workflow expects it to supply: the exact workflow and arguments that produced it, and, for audio/video, a `media` block (duration, rate, channels, fps, size, peak/mean dBFS) and `findings` - each level problem the server measured against `dw/audio_qc.py`'s thresholds (`full_scale`, `near_silent`), with the fix. Only an image (PNG/JPEG/WebP) embeds `metadata` this way — it is always null for audio and video, and `next` then points at `get_job_workflow(job_id)` when `job` is known, or says a kept asset carries no provenance at all when it isn't. `envelope=true` adds `media.envelope` — `rms_dbfs` and `peak_dbfs` one entry per second, or per `interval_seconds` (default 1.0, 0.05 to 3600; 0.1 places a vocal onset; a finer-than-default width that would need over 2400 bins is refused with the interval that fits) — which is what locates something in a track rather than measuring the whole of it. `media.shots` is set on an output joined from shots: one `{name, start_frame, num_frames, start_sample, num_samples}` per shot, as the join measured them (see the run manifest in docs/WORKFLOW_GUIDE.md), else null. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | ### Media @@ -277,7 +277,7 @@ tools) rather than relying on the session pin. | Tool | Arguments | Purpose | | --- | --- | --- | -| `validate_workflow(workflow=None, name=None, workspace=None, arguments=None)` | exactly one of `workflow` (inline definition) or `name` (a stored workflow, as `list_workflows` reports it), optional `workspace`, optional `arguments` | Check a workflow against the schema and against real pipeline signatures. Free and instant. Validating by name uses the workflow file's own directory as the base directory, so it sees what a run would. Returns every schema violation in `errors`, each with the JSON path it sits at, so a draft is fixed in one pass, and a `previous_result:` that names no earlier step is one of them. `warnings` covers what still runs but is probably wrong - a signature mismatch, and, for a list-driven variable, an entry key no step reads, at the entry's path. `workspace` names the workspace for this one call without switching the session to it - use it to pin a job whose `output:` or `asset:` references live in a workspace other than the session's. Pass the same `arguments` you will pass to `run_workflow` and they are checked too - an undeclared or renamed variable name, a value that will not coerce to the declared type, and an `asset:`, `prompt:` or `output:` reference that names nothing this workspace can reach, each reported at `arguments.`. `checked_arguments` lists what was covered, so a `valid: true` about the stored defaults cannot be mistaken for one about your values. A reference set the model would refuse - too many images, videos or audio clips, or, for MiniMax-H3, audio as the only reference - is an error here too, rather than a failure minutes into a run you acknowledged. `run_workflow` makes the same check and refuses a bad argument rather than queuing a job that fails on its first step. A valid answer carries `plan` - the fingerprint, step count, list lengths, `downloads_required`, `estimate` (with `basis`), `host_memory` (`{projected_mb, basis, runs}`, the projected host-RAM peak; `basis` is `observed` from same-key successful runs on this machine or `fit`; absent when there is no figure) and `elided_steps` for the arguments given; quote from it. `steps` counts what will run: a step nothing reads and which saves no file does not run, and is named in `elided_steps` instead | +| `validate_workflow(workflow=None, name=None, workspace=None, arguments=None)` | exactly one of `workflow` (inline definition) or `name` (a stored workflow, as `list_workflows` reports it), optional `workspace`, optional `arguments` | Check a workflow against the schema and against real pipeline signatures. Free and instant. Validating by name uses the workflow file's own directory as the base directory, so it sees what a run would. Returns every schema violation in `errors`, each with the JSON path it sits at, so a draft is fixed in one pass, and a `previous_result:` that names no earlier step is one of them. `warnings` covers what still runs but is probably wrong - a signature mismatch, and, for a list-driven variable, an entry key no step reads, at the entry's path. `workspace` names the workspace for this one call without switching the session to it - use it to pin a job whose `output:` or `asset:` references live in a workspace other than the session's. Pass the same `arguments` you will pass to `run_workflow` and they are checked too - an undeclared or renamed variable name, a value that will not coerce to the declared type, and an `asset:`, `prompt:` or `output:` reference that names nothing this workspace can reach, each reported at `arguments.`. `checked_arguments` lists what was covered, so a `valid: true` about the stored defaults cannot be mistaken for one about your values. A reference set the model would refuse - too many images, videos or audio clips, or, for MiniMax-H3, audio as the only reference - is an error here too, rather than a failure minutes into a run you acknowledged. `run_workflow` makes the same check and refuses a bad argument rather than queuing a job that fails on its first step. A valid answer carries `plan` - the fingerprint, step count, list lengths, `downloads_required` (pipeline `model_name`s and the repos and single files task steps fetch), `estimate` (with `basis`), `host_memory` (`{projected_mb, basis, runs}`, the projected host-RAM peak; `basis` is `observed` from same-key successful runs on this machine or `fit`; absent when there is no figure) and `elided_steps` for the arguments given; quote from it. `steps` counts what will run: a step nothing reads and which saves no file does not run, and is named in `elided_steps` instead | | `list_workspaces()` | — | The server's workspaces and which one this session is using. Each has its own workflows, assets and outputs; the prompt library is shared by all of them. Compact entries are `name`/`default`/`usage` plus any `note` (clipped, `note_truncated` when cut); `detail=true` adds each entry's folder paths | | `use_workspace(name)` | `name` | Work in that workspace for the rest of the session - every later call reads and writes there. This is how to keep your work out of another agent's namespace rather than sharing the default one. Checked against the server, so a typo fails here rather than scoping every later call to nothing. On a `dw.serve --mcp` endpoint the pin is shared by every connected client (#298, see the note above the table) - a `warning` field appears when this call overwrote a pin already pointed away from `default` | | `create_workspace(name, use=False)` | `name`, `use` | Create a workspace. Pass use=true to switch this session to it as well; otherwise the session stays where it was and the result says so. The result names the workspace (`name`, `default`, `current`, `next`), not its folders - `list_workspaces(detail=true)` is the opt-in for those. `use=true` carries the same mounted-server caveat and `warning` field as `use_workspace` | @@ -331,7 +331,7 @@ references written in the same session. | Tool | Arguments | Purpose | | --- | --- | --- | | `list_models()` | — | (Catalog) List what the Hugging Face cache holds, largest first | -| `download_model(repo_id, acknowledged_cost=False)` | `repo_id`, `acknowledged_cost` | Fetch a model repo into the cache. Costs disk and bandwidth, so it passes the gate. Returns as soon as the download starts | +| `download_model(repo_id, acknowledged_cost=False)` | `repo_id`, `acknowledged_cost` | Fetch a model repo into the cache. Costs disk and bandwidth, so it passes the gate. Returns as soon as the download starts. It fetches the whole repo at `main`, so it does not pre-fetch a single pinned file a task entry names; the task fetches that on its first run | | `list_downloads()` | — | List downloads the server is running or recently ran | | `cancel_download(download_id)` | `download_id` | Stop a running download; the record shows `cancel_requested: true` until it is `cancelled`. Partial files stay cached and resume on a retry | | `delete_model(repo, acknowledged_cost=False)` | `repo`, `acknowledged_cost` | Delete every cached revision of one repo. Not recoverable locally | @@ -372,7 +372,9 @@ answers with a `plan`; pass `acknowledged_cost={"fingerprint": plan.fingerprint, "minutes": plan.estimate.minutes, "downloads": [...]}` and the server refuses with 409 if the run's shape changed between the quote and the call - a longer list, a stored prompt edited meanwhile, weights that now -have to be downloaded - naming the new plan so the agent re-quotes. Bare +have to be downloaded - naming the new plan so the agent re-quotes. The +`plan` itself, passed unedited, binds the same way: with no `downloads` key, +its `downloads_required` repos (a file entry by its repo) and `estimate.minutes` stand in. Bare `true` still works and is for a `plan` that came back null; the job records which form it got (`acknowledged: none | boolean | bound`). @@ -388,7 +390,7 @@ The intended loop: verdict covers the stored definition and its stock defaults, not the values you wrote, and its `plan` is the number to say out loud: `estimate.minutes` with its `basis`, plus each `downloads_required` - entry as a line item of its own + entry as a line item of its own (task steps' repos and single files are included) 2. `run_workflow` with `acknowledged_cost` bound to the plan (`{fingerprint, minutes, downloads}`), or `true` when there was no plan — pass a name straight from `list_workflows` as `workflow_path`; queues the job and returns diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 98e3cf5fb..41d0efbf4 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -7,6 +7,143 @@ notes from commits at tag time (see below). This section is a scratch pad for items a branch's author wants the next release note to name; clear it when a release ships. +### 0.16.0 + + + +A release about reaching past a 24 GB card and about removing things from +a picture. MiniMax H3 refines to 1920x1088 in tiles, the plan prices every +weight a task step will fetch before a run is accepted, and the cost quotes +and VRAM ceilings for LTX-2.5 and Z-Image are calibrated against real runs. +A prompt-free object-removal task joins the catalog with a small inpaint +model beside it, and check-script is sturdier on looped, clipped and +repeated takes. + +**H3 tiled refine to 1920x1088** (#1043, #1048, #1056) + +- An H3 refine step takes `refine_tiles: {width, height, overlap}` and + denoises an upscaled plate in feathered tiles on the 32 px grid, so a + 1920x1088 refine fits a 24 GB card. The tiles blend in latent space and + the canvas decodes once; the soundtrack held from the base take is + encoded once per refine, not once per tile. The guide's *Refining above + 768p in tiles* section is the recipe: 544p base, `upscale_h3_latents`, + tiled refine on the 768p turbo LoRA, one decode (about 12 min on an RTX + 3090 for 124 frames). +- `upscale_h3_latents` caps at 1920x1088 (or 1088x1920) and validate + refuses past it; references, guides, keyframes and `` are + refused in tile mode; validate prices tile mode by the tile and the + decode, not as a full-canvas denoise. +- Every pipeline step whose pipeline reports its own frame rate now hands + its `videos` on carrying it, so `pair_audio` on an H3 base take fits the + track at 24 fps instead of the 8 fps fallback. The guide's keep-the-544p + recipe no longer asks for `result.fps`, which runs after the task. +- `analyze_seams` reports look measures (lightness, contrast, saturation, + detail) at each seam beside the motion fields, so a re-roll can say + which side stepped (#1031, #1042). The `script-to-video` skill says when to + re-roll a seam (#1034). +- H3 dialogue shots drop the turbo LoRA, the enhancer writes spoken and + sung lines as `` blocks, and locked-frame lip sync is FL2VA with + `hold_audio` (#985, #989, #992). `slice_audio` with `start_seconds` and + `num_frames` keeps the frame length (#990). A one-segment H3 chain is + bit-identical to a native run (#1046). + +**Object removal** (#956, #973, #974, #946) + +- `remove_object` erases a masked region with LaMa, prompt-free, and + `templates/remove-object` and `templates/segment-and-remove` (SAM2 mask + from a box or a label, then erase) run it. `segment` takes each box's + best-rated SAM2 mask rather than its first. +- `models/z-image-inpaint` configures `templates/inpaint` with Z-Image + Turbo for masked add-or-replace, a fraction of FLUX.1 Fill's 54 GB. The + guide's *Erasing versus repainting a masked region* section routes + between the two. + +**Cost, downloads and admission** (#975, #977, #978, #981, #1037, #1038, #948, #954) + +- A validate's `downloads_required` lists the Hugging Face repos and + single files task steps fetch (`segment`, `remove_background`, + `diffusion_upscale`, `transcribe_audio`, `check_script`, + `generate_speech`, `attribute_voices`, `text_generation`, + `image_to_text`, `judge`, the image processors, `upscale`, + `restore_faces`, `remove_object`, `upscale_h3_latents`, + `crop_face_track`, `interpolate_frames`, and facexlib's two weights), + plus `load_components` repo overrides. Entries gain optional `filename` + and `revision`; the OpenAPI document and the UI's types carry them. +- The validate plan passed back verbatim is a bound acknowledgement. A + bound `acknowledged_cost` taken from a validate made before this deploy + is answered 409 `downloads` when the workflow has one of these tasks and + its repo is not cached: re-validate and resend the new `acknowledge`. + The 409's rebind plan carries the observed estimate validate quotes. +- The observed estimate is the median of the newest runs, not the whole + history, and an inherited quote pays the model load once across scaled + steps. A task-only workflow's unknown estimate names why it is unpriced + (`unpriced_reason`), and validate notes `vram_config_differs` when a + step's offload or quantization differs from the calibrated template. +- Validate reports one VRAM error per projection. LTX-2.5's + `upscale-clip`, `image-to-video`, `keyframes`, `two-stage`, + `refine-clip` and `reference-sheet` declare calibrated `vram_estimate`s, + so a 1080p or 481-frame request past a 24 GB card is refused at + validate instead of running out of memory late in the denoise; the + image-to-video default and the `ltx-2.5` skill move to 401 frames + (#1050, #1052, #1053, #1054). `models/z-image` declares one over batch, + width and height (#945). + +**check-script and audio** (#953, #955, #959-#971, #1039, #1041) + +- A looped cut is re-transcribed shot by shot when shots are known; a + scripted loop keeps its first take and flags `transcript_loop`; a chunk + repeating the previous span is discarded as `repeated_span`; a second + copy of a line inside its matched words is `unscripted_speech`; audible + untranscribed sound between two lines, and after the last one, is + reported and placed in the shot holding most of it; a clipped tail is + judged against the line's loudest window, so a score bed is not a clip. +- `separate_stems` warns `stacked_vocals` over layered-stereo spans of the + vocal stem (#947). `normalize_audio(limit)` warns when the limiter works + mostly on one shot of a cut, and reads a saved cut's manifest shots when + given its path. + +**Server, worker and validation** (#928, #944, #949, #950, #951, #952, #957, #972, #976, #993, #995, #996) + +- A worker whose CUDA context a device-side assert poisoned is recycled + instead of answering every later job with the same error. A pulled card + in a `--devices` list no longer stops the server: it starts on the cards + present and warns. `scripts/dw-serve.service` sets + `RestartPreventExitStatus=2` so a refused configuration stops the unit + rather than looping; reinstall the unit to pick it up. +- A file of a run with no recorded role lists as intermediate when in + `intermediate/` or read by a later run; manifest roles are stamped on + the job at finish so history serves what a live job did; a deleted + workspace's job rows stay resolvable by id. +- Validate refuses a bare relative media name that resolves nowhere, + naming `output:` for an own output; an unknown argument is + reported as itself; a document error no longer hides an unknown argument + in the same call; no undertrained-LoRA warning on a null LoRA whose steps + were raised. +- A redirect hop the outbound guard refuses now names the target and the + URL asked for, so a refused hop reads apart from an upstream error + (#1055). + +**Web UI** + +- The flow view routes column-skipping edges in lanes and merges same-pair + labels. The jobs page filters follow the URL hash. Jobs and models list + the day (with the year only when it is not this one) and give the exact + timestamp where it matters. undici 8.12.3. + +**Engine structure** + +- The task-weights hook path is acyclic (`validate_weight_name` in + `dw/security.py`, processor weights registered into the task registry), + and `transformers` no longer loads into the server's validate path. + `dw/plan_downloads.py`, `dw/tasks/script_take.py` and + `dw/pipeline_processors/mag_calibration.py` split out of their parents; + `docs/ARCHITECTURE.md` names each seam. + +**Dependencies and deploy** + +- No dependency floors changed. The engine changes in this release need a + server restart (the persistent worker) after deploy. + ### 0.15.0 diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 3c46143ea..dd576e779 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -199,7 +199,13 @@ One policy now answers all of it, untrusted: (`MAX_MEDIA_REDIRECTS`), and each `Location` passes the same scheme and host policy before it is dialed. A public URL answering `302` to `http://127.0.0.1:8765/api/server` is refused with the target named, and - nothing is fetched from it. Images are decoded from the fetched bytes and + nothing is fetched from it. A refused hop's error always begins + `Refusing a redirect to '' (from ''): ` and + then gives the rule that refused it (an internal address, a scheme other + than http(s), an unresolvable host), whereas an error status from a URL + itself stays the HTTP client's own `403 Client Error: Forbidden for url: + ...` - so a refused redirect and an upstream failure read differently + (#1055). Images are decoded from the fetched bytes and still go through diffusers' `load_image`, so EXIF orientation and RGB conversion are unchanged. Each hop's host is resolved once, by the policy, and dialed at the address it checked - so a name that answers differently diff --git a/docs/SECURITY_QUICKREF.md b/docs/SECURITY_QUICKREF.md index 79bff9a1e..350a1232b 100644 --- a/docs/SECURITY_QUICKREF.md +++ b/docs/SECURITY_QUICKREF.md @@ -109,4 +109,5 @@ Invalid: file:///etc/passwd, ftp://server/file | `Path contains dangerous pattern matching \.\.` | Path traversal | Use absolute or `./` relative paths | | `Invalid variable name: my.var` | Special characters | Use `my_var` instead | | `Invalid URL: URL scheme not allowed: file` | Non-http scheme | Use https or local file path | +| `Refusing a redirect to '' (from ''): ...` | A media URL redirected to an internal host or a non-http scheme | Host the media where it is served directly; a refused hop is never fetched | | `Argument contains dangerous characters: ...` | Shell metacharacters | Remove `` ` `` `$` `\|` `&` `;` `>` `<` | diff --git a/docs/SERVER.md b/docs/SERVER.md index 5dd42d321..9c636d98f 100644 --- a/docs/SERVER.md +++ b/docs/SERVER.md @@ -47,7 +47,10 @@ the same choice, and the flag wins over it. A list (`--devices cuda:0,cuda:1`) runs one worker per card, each running one job at a time. A card named twice, or a bare `cuda` in a list of several, is refused at startup, as is a card the machine lacks, with the message naming the cards that are present -(`cuda:0 (NVIDIA GeForce RTX 4090)`, ...). +(`cuda:0 (NVIDIA GeForce RTX 4090)`, ...). A list survives a card being +pulled: an entry the machine lacks is dropped with a warning naming it, and +the server starts on the rest. Only a list none of whose cards is present +(or a single absent card) is refused. Each job's record carries `device` - the card it ran on, as the server addresses it (`"cuda:1 NVIDIA GeForce RTX 3090"`), set when the job starts @@ -200,7 +203,7 @@ running: | Route | What it does | | --- | --- | -| `POST /api/jobs` | Queue a run: `{"workflow_path": ...}` or an inline `{"workflow": {...}, "base_dir": ...}`, plus `arguments` for variable overrides. `workflow_path` accepts a stored workflow name as listed by `/api/workflows` (with or without `.json`, nested names included), or a relative/absolute path that still resolves under `--workflow-dir` - confined the same way the `/api/workflows` CRUD routes are; a path that names a real file outside that directory is rejected with 400, not opened. Answers with argument warnings from signature checking. Takes an optional `acknowledged_cost`: `true` is recorded as `acknowledged: boolean`; the object `{fingerprint, minutes, downloads}` from a validate answer's `plan` is `bound` - the server re-plans the run for the arguments given and answers **409** when the fingerprint differs or a repo in `downloads_required` is not in `downloads` (a download that has since vanished is not a refusal); the body is `{"detail": {message, reason: "fingerprint" \| "downloads" \| "unplannable", acknowledged, plan, acknowledge}}` with the current plan, so the caller re-quotes from it, and `acknowledge` - the `{fingerprint, minutes, downloads}` to resend - whenever there is a plan. A `null` in `downloads` (a URL with no repo) is ignored. `minutes` is recorded, never compared. Nothing is required: the web UI and every caller that sends nothing are `acknowledged: none`, and every job answer and history row carries `acknowledged` (and `acknowledged_cost` when bound). `POST /api/jobs/{id}/rerun` takes the same field and checks against the stored spec; a fresh seed does not change a fingerprint. | +| `POST /api/jobs` | Queue a run: `{"workflow_path": ...}` or an inline `{"workflow": {...}, "base_dir": ...}`, plus `arguments` for variable overrides. `workflow_path` accepts a stored workflow name as listed by `/api/workflows` (with or without `.json`, nested names included), or a relative/absolute path that still resolves under `--workflow-dir` - confined the same way the `/api/workflows` CRUD routes are; a path that names a real file outside that directory is rejected with 400, not opened. Answers with argument warnings from signature checking. Takes an optional `acknowledged_cost`: `true` is recorded as `acknowledged: boolean`; the object `{fingerprint, minutes, downloads}` from a validate answer's `plan` is `bound` - the server re-plans the run for the arguments given and answers **409** when the fingerprint differs or a repo in `downloads_required` (pipeline `model_name`s and the repos task steps fetch) is not in `downloads` (a download that has since vanished is not a refusal); the body is `{"detail": {message, reason: "fingerprint" \| "downloads" \| "unplannable", acknowledged, plan, acknowledge}}` with the current plan, so the caller re-quotes from it, and `acknowledge` - the `{fingerprint, minutes, downloads}` to resend - whenever there is a plan. A `null` in `downloads` (a URL with no repo) is ignored. The validate `plan` sent unedited is bound too: when `downloads` is absent, its `downloads_required` repos (and `estimate.minutes`) are what was acknowledged; an explicit `downloads`, even `[]`, wins. `minutes` is recorded, never compared. Nothing is required: the web UI and every caller that sends nothing are `acknowledged: none`, and every job answer and history row carries `acknowledged` (and `acknowledged_cost` when bound). `POST /api/jobs/{id}/rerun` takes the same field and checks against the stored spec; a fresh seed does not change a fingerprint. | | `GET /api/jobs?workspace=&status=&limit=` | Queue + history summaries, oldest first, with `total` beside them. `status` narrows to one state or a comma-separated set (`queued`, `running`, `succeeded`, `failed`, `cancelled`; anything else is a 400); `limit` keeps the newest N, and `total` still reports how many matched, so a bounded answer cannot be mistaken for a complete one. Each summary carries `device`, the card the job ran on (`"cuda:1 NVIDIA GeForce RTX 3090"`; null while queued and for older jobs), and `rerun_of`, present on a job started by rerun: the id of the job it reran; absent for a fresh submit. No parameters means every job, which is what the web UI polls | | `GET /api/jobs/{id}` | Full detail: spec, events, manifest, error. A manifest entry for a step served from the step cache carries `reused: true`. Every entry carries `subfolder` - the in-run subfolder the step's `result.subfolder` chose, `''` for none. A `for_each` step appears in the manifest as its members (`shot@wide_open`, `shot@closeup`), because the manifest records what ran; the run's `workflow.json` keeps the `for_each` form, because it records what was asked Carries `output_kinds`: each manifest file mapped to its media kind (`image`, `video`, `audio`, `text`, or null), and each file-carrying run event (`step_end`) carries the same map for its own files. The entry of a pipeline step that ran carries `timing` (see [Step timing](WORKSPACES.md#step-timing)). Carries `rerun_of` - present on a job started by rerun: the id of the job it reran; absent for a fresh submit. | | `DELETE /api/jobs/{id}/run` | Delete the run directory a finished job wrote, whole, from the output root the job ran against (no workspace selector applies): `{job_id, run_dir, deleted, run_swept}`. 404 for an unknown job or one with no run directory, 409 for one still queued or running | @@ -426,7 +429,7 @@ The editor's forms come from these; they are just as usable from scripts: when the worker is busy or did not answer - a workflow with no `seed` also gets a warning saying so, since `0` alone does not distinguish a disabled cache from an empty one); - `downloads_required`, each `model_name` the hub cache does not hold as + `downloads_required`, each `model_name` the hub cache does not hold, and each repo a task step fetches (`dw/tasks/weights.py`; a repo shared with a pipeline step is listed once, a reference or local-path argument gives none) as `{repo, gb, gated, access_blocked}` (`gb` from the hub, `null` when it could not be asked - `?sizes=false` skips the hub, and then `gated` and `access_blocked` are `null` too); `gated` is the hub's own field for the @@ -445,7 +448,21 @@ The editor's forms come from these; they are just as usable from scripts: offline) - "unknown" is not "not blocked". `validate_workflow`'s `warnings` carries one line per entry with `access_blocked: true`. Each `from_single_file` URL is `{repo: null, url, gb: null, gated: null, - access_blocked: null}`, since a direct file URL is never gated; and + access_blocked: null}`, since a direct file URL is never gated. + A task that fetches one file rather than a whole repo is listed as a file + entry, `{repo, filename, revision, gb, gated, access_blocked}`: `revision` + is a 40-character commit pin or `null` for `main`; `gb` is that one file's + size, from a HEAD of the file (`null` offline or with `?sizes=false`); + `gated` is `null`, since the HEAD that sizes it does not say; and + `access_blocked` is `true` when that HEAD is refused by the gate, `false` + when it succeeds, `null` when unknown. It is listed when that file at that + revision is not in the hub cache (checked per file, so another cached file + of the same repo does not count), and a whole-repo entry for the same repo + subsumes it. Acknowledgements still compare by `repo`. `restore_faces` + also lists facexlib's weights as URL entries (`detection_Resnet50_Final.pth`, + and `parsing_parsenet.pth` unless `use_parse` is false) while they are + missing from facexlib's weights directory; they download outside the + Hugging Face cache. And `estimate`, `{minutes, basis, device, measured_on, partial, runs}` from this box's own history when it has one and otherwise from the workflow's `cost` block - @@ -467,7 +484,8 @@ The editor's forms come from these; they are just as usable from scripts: naming the drivers that moved, because it was measured at another setting of them; `other_device` (no entry for the serving backend; the first entry's figure, which is a warning rather - than a quote) or `unknown` (no cost block, or more than one list changed + than a quote) or `unknown` (no cost block - a workflow of task steps alone says so in + `unpriced_reason`, since it has no pipeline to inherit from - or more than one list changed so there is nothing honest to extrapolate along) or `inherited` (an inline workflow has no catalog name, so its price is this server's `observed` runs of the catalog template that loads the same pipeline - @@ -595,8 +613,9 @@ The editor's forms come from these; they are just as usable from scripts: `subfolders` list every distinct value over the whole tree, `''` always a member of each so root-level files stay selectable. Each entry also carries `role`, `final` or `intermediate` (null for a flat-layout file with - no run; `intermediate` once a later run read the file, unless the step's - `result.role` declared it), and `GET /api/gallery?role=final` lists only + no run, or a file of a run that recorded no role whose subfolder says + nothing and that no later run read; `intermediate` once a later run read + the file, unless the step's `result.role` declared it), and `GET /api/gallery?role=final` lists only those (repeatable; a null role counts as final). See `dw/roles.py` for how a role is decided; the metadata route reports the file's `role` and the run's `inputs` too diff --git a/docs/TASKS.md b/docs/TASKS.md index 4127bb184..7372bfbd8 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -107,6 +107,15 @@ A new task is: automatically, so a module it does not import registers nothing; 3. a section in this file. +A task that loads Hub weights also declares `required_weights(arguments)`, +named on the decorator as +`register_command(required_weights="dw.tasks..required_weights")`. It +returns `[{"repo": ...}]` for the step's arguments (an empty list when the +repo is not known yet, e.g. a `previous_result:` `model_name`), and resolves +each repo with `argument_or_default` from `dw/tasks/weights.py`, the same call +the loader makes, so the declaration and the load cannot name different +repos. A validate's `downloads_required` lists those repos; `tests/test_task_weights.py` pins it. + ## Image Processing ### ControlNet Preprocessors @@ -139,6 +148,8 @@ Generate control images for ControlNet pipelines: | `depth_estimator` | Depth hint generation | | `depth_estimator_tensor` | Depth hint as tensor | +A validate's `downloads_required` lists the repo each detector fetches when it is not cached: `depth` `Intel/dpt-hybrid-midas`; `depth_estimator` and `depth_estimator_tensor` `Intel/dpt-large`; the `controlnet_aux` detectors `lllyasviel/Annotators` (`teed` `fal-ai/teed`, `anyline` `TheMistoAI/MistoLine`, `sam` `ybelkada/segment-anything`); `segmentation` `openmmlab/upernet-convnext-small`. `zoe_depth` and `dw_pose` fetch outside the Hugging Face cache and are not listed. + All accept an `image` argument with processing parameters: ```json @@ -175,6 +186,8 @@ All accept an `image` argument with processing parameters: | `add_watermark` | Add visible text watermark | `text`, `position`, `opacity`, `font_size`, `color`, `margin` | | `get_image_size` | Return `{width, height}` dict | | +A validate's `downloads_required` lists `briaai/RMBG-2.0` for `remove_background` when it is not cached. + ### EXIF Stripping Remove all EXIF metadata, GPS coordinates, camera info, and timestamps from images for privacy-safe preprocessing: @@ -1099,6 +1112,8 @@ face-detail pass that needs the face large and still: | `detector_file` | No | The detector file in that repo, a bare `.onnx` name. Default `face_detection_yunet_2023mar.onnx` | | `device` | No | Where detection runs | +A validate's `downloads_required` lists the detector file as a file entry when it is not cached, pinned at its revision for the default repo and file. + `modulus`, `remainder` and `multiple` follow `plan_cuts`: the model's grid is the workflow's to declare. The defaults (8, 1, 32) are LTX's 8n+1 frames and 32-pixel steps, so a workflow that names none gets the output it always did; @@ -1251,10 +1266,10 @@ returns frames without it, and this puts it back: | Argument | Required | Description | | -------- | -------- | ----------- | -| `video` | Yes | The frames - a frame list, a frame array or tensor, or an audio+video pair whose own soundtrack is replaced; their own rate is carried through to the output, so `result.fps` is only needed to override it (frames that carry none are written at 8 fps) | +| `video` | Yes | The frames - a frame list, a frame array or tensor, or an audio+video pair whose own soundtrack is replaced; their own rate is carried through to the output, so `result.fps` is only needed to override it. A file brings its rate along, and so does a pipeline step whose pipeline reports one (MiniMax-H3's `videos` carry 24 fps, #1056); frames that carry none are fitted and written at 8 fps unless `fps` says otherwise | | `audio` | Yes | The soundtrack - a waveform, the earlier step whose video carried one, or the path or URL of an audio or video file; the last two bring their sample rate along. A mono track is fine: an mp4 audio stream takes stereo and nothing else, so saving duplicates the one channel into two and warns that it did | | `sample_rate` | No | Sample rate of the waveform. Required unless `audio` carries one; given here it wins | -| `fps` | No | The rate the frames play at, only needed when they carry none of their own. Used solely to work out how long the video is - what `fit` and the length-mismatch check measure the track against - and is never written to the file; that's `result.fps`, which sets the rate the output plays at and defaults to 8 fps when the frames carry none | +| `fps` | No | The rate the frames play at, only needed when they carry none of their own. What `fit` and the length-mismatch check measure the track against, and the rate the paired video then carries to the writer. `result.fps` is not a substitute: the writer applies it after the fit has run, so declaring it alone on rate-less frames writes a 24 fps picture under a track padded to the 8 fps length (`video_fps_defaulted` says so, #1056) | | `fit` | No | `"video"` cuts or pads the track with silence to the length of the frames, warning either way (`audio_padded_to_video` / `audio_trimmed_to_video`). Left unset (the default) the track is used as it is, and a length that disagrees with the frames' is warned about rather than corrected (`audio_video_length_mismatch`). Any other value is refused at run time, not by `validate_workflow` | `fit`'s guarantee is exact for the waveform handed to the encoder, not for @@ -1315,7 +1330,7 @@ a length still passes the whole track along: | Argument | Required | Description | | -------- | -------- | ----------- | | `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | -| `start_seconds` / `duration_seconds` | One pair | The slice in seconds; either may be omitted | +| `start_seconds` / `duration_seconds` | One pair | The slice in seconds; either may be omitted. `start_seconds` with `num_frames` + `fps` and no duration takes its length from the frames | | `start_frame` / `num_frames` / `fps` | One pair | The slice in video frames; `fps` is required, start and count may be omitted | | `lead_frames` | No | Frame form only, default 0: extra audio before the cut. The slice starts at `start_frame - lead_frames` and still runs `num_frames`, so `start_frame` 48 with `lead_frames` 12 at 24 fps is audio from frame 36 for `num_frames` frames. A whole number, 0 or above; refused with the seconds form, and refused when `start_frame - lead_frames` is below zero (by `validate_workflow` for literals, at run time otherwise) | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | @@ -2161,6 +2176,18 @@ Every seam between shots, audio and picture: | `seams[].frame_delta` | The largest single-frame picture change across the seam | | `seams[].typical_delta` | The larger shot's own typical frame-to-frame change, floored | | `seams[].jump_ratio` | `frame_delta` divided by `typical_delta` | +| `seams[].saturation_step` | Mean saturation (per-pixel max(RGB) - min(RGB)) of the 4 frames after the seam minus the 4 before it (before a dissolve's fade), clipped to the shots. Signed: positive means the incoming shot is more saturated | +| `seams[].lightness_step` | The same for mean Rec. 709 luma. Signed: negative means the incoming shot is darker | +| `seams[].contrast_step` | The same for the luma standard deviation. Signed | +| `seams[].look_ratio` | The largest of the three steps' sizes divided by that measure's own local rate: the 90th percentile of its change from one 4-frame mean to the next inside either shot, floored at 1.0. A step inside a shot that is itself changing that way (a sunrise) scores low | +| `seams[].look_ratio_measure` | Which measure gave `look_ratio`: `saturation`, `lightness` or `contrast` | +| `seams[].detail_ratio` | Mean high-pass detail (luma against its 3x3 blur) after the seam divided by before it: below 1 means the incoming shot is softer. Null when the before window has almost no detail (below 0.5) | + +The six look fields are measures, not rules: none raises a finding, and they +appear in neither `rules_applied` nor `rules_skipped`. They are on the 0-255 +scale of a 128x72 colour copy of each frame, and the numbers are not +perceptual - a `look_ratio` of 10 is not a calibrated "visible". They are null +when the file has no picture. ### analyze_sync_drift @@ -2233,7 +2260,7 @@ score bed. "command": "separate_stems", "arguments": { "audio": "asset:song.mp3" } }, - "result": { "content_type": "audio/wav", "file_name": "stems" } + "result": { "content_type": "audio/wav", "file_base_name": "stems" } }, { "name": "lyrics", @@ -2250,15 +2277,40 @@ score bed. | `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | -The result saves one file per stem, `-vocals`, `-drums`, `-bass` +The result saves one file per stem, `-vocals`, `-drums`, `-bass` and `-other`, and a later step reads one as `previous_result:.vocals`. Stems are 44.1 kHz stereo, whatever the source was, and sum back to the mix. The model is fixed (`htdemucs`, run in fp32), its weights download on first use, and it needs the `demucs` package. On MPS a failed separation falls back to the CPU and warns (`separation_cpu_fallback`). +**Stacked vocals.** A chorus is often recorded as layered stereo doubles - every +Music 3 chorus is - and htdemucs returns all of them in the one `vocals` stem. +When the stem's left and right channels sing different lines over a sung span, +the step warns `stacked_vocals` and names the spans in m:ss (up to 8, then "and +N more"): + +``` +separate_stems: the vocals stem holds stacked (layered stereo) voices over +1:19–1:34, 1:59–2:09; treat those spans as more than one voice before +processing them as one +``` + +Read it before working on a span as one voice: pitch or formant work on a +stacked span distorts it, and an `attribute_voices` window or a lip-sync shot +over one has two singers in it. Pick a span outside the flagged ones, or plan +the shot as a two-shot. The stems themselves are unchanged. The check is a 1 s +window test on the stereo stem: the window must be sung (half its frames voiced, +and within 14 dB of the stem's level), with an L/R correlation under 0.45 and a +side/mid energy ratio over 0.4. Windows merge across a one-window gap, and +spans under 2 s are dropped. A mono source, an instrumental and a single doubled +word never flag it. The thresholds are the `STACK_*` constants in +`dw/tasks/voice_attribution.py`. + ## Voice attribution +A validate's `downloads_required` lists `speechbrain/spkrec-ecapa-voxceleb` only. The demucs separator (and `separate_stems`) downloads outside the Hugging Face cache and is not listed. + Which reference voice sings each line of a song, by timbre - staging lip-sync shots for a generated song needs its section-to-singer map, and a generation model (MiniMax Music3 included) does not hand one back. Pitch @@ -2506,8 +2558,8 @@ Pass through arguments directly. Useful for organizing data flow. ## Videos in image tasks -Every image command - the upscalers, face restoration, segmentation and the -image processors - takes a video where it takes an image: an `AudioVideo` from +Every image command - the upscalers, face restoration, segmentation, object +removal and the image processors - takes a video where it takes an image: an `AudioVideo` from a generation, `concat_videos` or `dissolve_videos` step, or a frame array from `video_frames`. The command runs over the frames one at a time and returns one video artifact, its soundtrack carried through untouched, so a generated clip @@ -2559,6 +2611,8 @@ Upscale images using spandrel-compatible super-resolution models (ESRGAN, SwinIR | `tile_size` | No | Tile size for large images (default: 512) | | `tile_overlap` | No | Overlap between tiles in pixels (default: 32) | +A validate's `downloads_required` lists the `model_name` / `filename` file when it is not cached, or the whole repo when no `filename` is given (such a repo must hold exactly one weight file); a `previous_result:` or local-path model gives none. + Large images are automatically tiled to avoid GPU memory issues. Models can be loaded from HuggingFace Hub repos or local `.pth`/`.safetensors` files. **Examples:** @@ -2573,6 +2627,8 @@ Two modes are available: - **x4** (default): `StableDiffusionUpscalePipeline` — 4x upscale via `stabilityai/stable-diffusion-x4-upscaler` - **x2**: `StableDiffusionLatentUpscalePipeline` — 2x upscale via `stabilityai/sd-x2-latent-upscaler` +A validate's `downloads_required` lists the repo for the `mode` given; `model_name` overrides it. + ```json { "task": { @@ -2632,6 +2688,8 @@ Restore and enhance faces in images using spandrel-compatible face restoration m | `eye_dist_threshold` | No | Skip faces with eye distance below this (default: 5) | | `upsample_img` | No | Pre-upscaled background image (e.g., from a prior upscale step) | +A validate's `downloads_required` lists the `model_name` / `filename` file when it is not cached (the whole repo when no `filename` is given), and facexlib's two weights as URL entries while they are missing from facexlib's weights directory: `detection_Resnet50_Final.pth` always and `parsing_parsenet.pth` unless `use_parse` is false. Those still download outside the Hugging Face cache. + Models are loaded via spandrel, so any `.pth`/`.safetensors` face restoration weights work. CodeFormer requires `pip install spandrel-extra-arches` (non-commercial license). The detector and parser are loaded once per settings/device and reused across frames; `release_models` drops them with the restorer. **Example:** [restore-faces.json](../workflows/templates/restore-faces.json) — Generate a portrait, then restore faces with GFPGAN v1.4. @@ -2684,6 +2742,8 @@ This gives the best results: the super-resolution model handles background detai Detect and segment objects using text prompts via GroundingDINO + SAM2. Returns a binary mask image suitable for inpainting workflows. +A validate's `downloads_required` lists `IDEA-Research/grounding-dino-base` and `facebook/sam2-hiera-large`; `model_name` and `sam_model_name` override them. + ```json { "task": { @@ -2706,12 +2766,58 @@ Detect and segment objects using text prompts via GroundingDINO + SAM2. Returns | `threshold` | No | Detection confidence threshold (default: 0.3) | | `invert` | No | Invert the output mask (default: false) | -Returns a grayscale PIL Image (mode "L") — white (255) for detected objects, black (0) for background. Use with inpainting pipelines like FluxFillPipeline. +Returns a grayscale PIL Image (mode "L") — white (255) for detected objects, black (0) for background. SAM2 proposes three masks for each detected box, and segment keeps the one SAM2 rates best. That is usually the whole object: on the docs cup photo the first proposal was the body alone, without the lid and with a hole at the logo. To erase what it found, pass the mask to [`remove_object`](#object-removal), which chains with no `invert`. To paint something new in its place, pass it to a prompted inpainter such as FluxFillPipeline. **Examples:** - [segment.json](../workflows/templates/segment.json) — Segment an object from an image -- [segment-and-inpaint.json](../workflows/templates/segment-and-inpaint.json) — Segment, then inpaint the masked region +- [segment-and-remove.json](../workflows/templates/segment-and-remove.json) — Segment, then erase the object with `remove_object` (no prompt) +- [segment-and-inpaint.json](../workflows/templates/segment-and-inpaint.json) — Segment, then repaint the masked region with a prompt + +## Object Removal + +Erase what a mask covers and fill the hole from the surrounding texture, with no prompt (#956). `remove_object` runs [LaMa](https://github.com/advimman/lama) (big-lama) through spandrel: no diffusion, so a warm run takes well under a second on a GPU, and it runs on CUDA, MPS and CPU. + +```json +{ + "task": { + "command": "remove_object", + "arguments": { + "image": "asset:photo.png", + "mask_image": "previous_result:mask", + "dilate": 8 + } + }, + "result": { "content_type": "image/png" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `image` | Yes | PIL Image or reference - a video runs frame by frame, see [Videos in image tasks](#videos-in-image-tasks) | +| `mask_image` | Yes | White where to erase, black where to keep | +| `dilate` | No | Pixels to grow the mask by before the fill, a whole number 0 or above (default: 8) | +| `model_name` | No | Hugging Face repo id of the weights, never a local path (default: `michaelgold/big-lama`, at a pinned revision) | +| `weight_name` | No | The `.safetensors` file in that repo (default: `big-lama.safetensors`) | + +**The mask.** White means fill, the polarity [`segment`](#object-segmentation) returns, so `segment` → `remove_object` chains with no `invert`. The mask is converted to grayscale and binarized at 128 (at or above is fill). A mask of a different size from the image is resized to it (nearest). An all-black mask, as `segment` returns when it finds nothing, gives the image back unchanged with a warning in the job's events, and loads no model. + +**`dilate`.** LaMa fills cleanly only when the mask overshoots the object's edge, and a `segment` mask is tight to it. The default 8 px suits a segmented object. Raise it (16 to 24) for a soft edge, a shadow or a glow around the object. Use 0 only for a hand-drawn mask that already has a margin. + +**The output** is the input's size. Only the region inside the dilated mask comes from the model: every pixel outside it is the input's own, unchanged. + +**Video.** Frames (e.g. from `video_frames`) are cleaned one at a time with the same still mask, and come back at the same count and fps, with the audio kept. There is no temporal model, so a fill can flicker from frame to frame, and the mask cannot follow a moving object. + +**Limits.** LaMa suits small to medium objects: a cup, a sign, a watermark, a passer-by in the distance. A large region, such as a person filling a third of the frame, comes back smeared. For that, a prompted inpainter whose prompt names the background is the remaining route, and it may paint a patch that does not match (#946). + +**Weights.** `michaelgold/big-lama`, file `big-lama.safetensors` (about 200 MB), read at revision `98ab7cface89a594b381f32fc7a9f2cae3f91ccc`. `biali/big-lama` holds a byte-identical copy and is the fallback if that repo disappears; it is not tried automatically. LaMa is by advimman (Suvorov et al., *Resolution-robust Large Mask Inpainting with Fourier Convolutions*) and is licensed **Apache-2.0**. The Hub repo's card says MIT, but the upstream licence is the one that applies. The task reads safetensors only: `validate_workflow` refuses a `weight_name` that is not `.safetensors` (the common `big-lama.pt` is a TorchScript file) and a `model_name` that is a local path, and the loader refuses both again at run time. Weights that do not load as an inpainting model are refused too. The network runs in fp32 on every device. + +**First run.** The weights download on the first run, about 200 MB. A validate's `downloads_required` lists `michaelgold/big-lama` / `big-lama.safetensors` as a file entry (pinned at that revision for the default pair only; another `model_name` or `weight_name` is read at `main`) while it is not cached, so the download is quoted beforehand. + +**Examples:** + +- [remove-object.json](../workflows/templates/remove-object.json) — Erase what a supplied mask covers +- [segment-and-remove.json](../workflows/templates/segment-and-remove.json) — Find an object by name, then erase it ## Image Captioning @@ -2719,6 +2825,8 @@ Generate text captions from images using a vision-language model. Transformers 5 removed the dedicated `image-to-text` pipeline this task used to build, along with the BLIP/ViT-GPT2/GIT captioning models that ran on it. Captioning now goes through the same `image-text-to-text` pipeline as any other VLM, so `model_name` needs a vision-language model (SmolVLM, Qwen2.5-VL, LLaVA, etc.) and `prompt` is a question put to the model rather than a text fragment to continue. +A validate's `downloads_required` lists `HuggingFaceTB/SmolVLM-256M-Instruct` for `image_to_text` and `judge`; `model_name` overrides it. + ```json { "task": { @@ -2828,6 +2936,8 @@ The built-in `h3_context_ir` workflow applies this to its own output, so a workf Generate or expand text using a local language model. Useful for expanding short prompts into detailed image generation prompts, rewriting text, or other text-to-text tasks. +A validate's `downloads_required` lists `Qwen/Qwen2.5-1.5B-Instruct`, or `HuggingFaceTB/SmolVLM-256M-Instruct` when an `image` is given; `model_name` overrides it. A `system_prompt.from_hub` file is not listed. + ```json { "task": { @@ -2912,7 +3022,15 @@ It is merged after everything else, so it can override `repetition_penalty` and ## Speech Generation -Speak a line of text with a local text-to-speech model. The result is a waveform carrying the rate its model generated at, so it composes with [`slice_audio`](#slice_audio), [`fade_audio`](#fade_audio) and [`pair_audio`](#pair_audio) directly ([`concat_videos`](#concat_videos) and `dissolve_videos` join videos — pair the track onto a video first). +Speak a line of text with a local text-to-speech model. A validate's `downloads_required` lists `suno/bark-small` (`model_name` overrides it), plus `speechbrain/spkrec-xvect-voxceleb` when `speaker_embedding` is given. Any transformers text-to-speech checkpoint can be named as `model_name`; three families are wired for their voice controls: + +| Family | Checkpoint | Voice | +|--------|-----------|-------| +| Bark | `suno/bark-small` (the command's default) | Distinct speakers by `voice_preset` | +| MMS / VITS | `facebook/mms-tts-eng`, and `mms-tts-` per language (the template's default) | One voice, a quarter Bark's size; a multi-speaker VITS takes `speaker_id` in `forward_params` | +| SpeechT5 | `microsoft/speecht5_tts` | The voice of a reference clip, given as `speaker_embedding` | + +The result is a waveform carrying the rate its model generated at, so it composes with [`slice_audio`](#slice_audio), [`fade_audio`](#fade_audio) and [`pair_audio`](#pair_audio) directly ([`concat_videos`](#concat_videos) and `dissolve_videos` join videos — pair the track onto a video first). ```json { @@ -2967,6 +3085,8 @@ A speech model is worth releasing before a video model loads — set `release_mo ## Speech Transcription +A validate's `downloads_required` lists `openai/whisper-base`; `model_name` overrides it. `check_script` uses the same model. + Transcribe spoken audio to text with a local Whisper-class model. The word-correctness of a TTS deliverable — a dropped line, a mid-sentence truncation — can only be inferred from duration and timing arithmetic without this; `transcribe_audio` checks it directly against the text the deliverable was supposed to speak. ```json @@ -3000,6 +3120,8 @@ To compare a take against the lines it was meant to speak, use [`check_script`]( ## Script check +A validate's `downloads_required` lists `openai/whisper-base` for its transcription; `model_name` overrides it. + Whether a dialogue take speaks its script. Confirming it used to mean transcribing the take and comparing the transcript to the script by eye, which missed a word Whisper dropped, an H3 tag read aloud, and a last word cut off by the end of the file. `check_script` does the comparison and reports where to look. It transcribes the take with [`transcribe_audio`](#speech-transcription) word timestamps - the same code and model cache, no second ASR path. Every heard word is measured, and one at or below the dead-air floor is discarded as unheard, because Whisper invents words over silence. So is every word of a repetition loop - the same words over and over (`Pre-pre-pre-...`), which Whisper invents over music or room tone loud enough to pass the floor. H3 markup is stripped from each expected line, and the heard words are aligned to the expected words in order (`difflib`). Each line scores the similarity of its own expected and heard words. It decides nothing: findings are places to look, not verdicts. It is not an assessment probe (`assessment=True`); like [`attribute_voices`](#voice-attribution) it answers a JSON document. @@ -3035,27 +3157,29 @@ Markup stripped from an expected line: `[Language] ...`, ``, | ---- | ---------- | --------------------- | ---- | | `line_mismatch` | A line's similarity is below `similarity`; a dropped line comes back with `heard: ""` | The line's similarity / `similarity` | `{line, seconds, word}` - the line's first heard word, or where it should have been | | `tag_spoken` | A word from the line's stripped markup was heard, and is not also a word of the line's dialogue | The word heard / none | `{line, seconds, word}` | -| `line_clipped_at_end` | A line's last heard word ends inside the final `CLIP_TAIL_SECONDS` of its shot (when the line has one) or of the file, and that tail is still above `GUARD_FLOOR_DBFS` | The tail's level in dBFS / `GUARD_FLOOR_DBFS` | `{line, seconds, word, shot}` - `shot` when the line has one | +| `transcript_loop` | A repetition loop (`REPEAT_RUN_MIN` periods of up to `REPEAT_MAX_PERIOD` words) made only of words the script holds - Whisper stuck on a line it heard, so speech after it can be missing from `heard` and lines read 0.0 without being wrong (#961). The loop's first occurrence is kept as heard when it has real duration and speech level; the rest is under `discarded`, one entry per loop (`count`, `start`..`end`). A line heard nowhere, at or after the loop, reads `heard: null`, `similarity: null`, `not_transcribed: "transcript_loop"` and raises no `line_mismatch`. When the shots are known the cut is then transcribed again one shot at a time and the findings and `lines` come from that pass (`transcription_pass: "per_shot"`, each line `pass: "per_shot"`, #963) | Chunks in the loop / `REPEAT_RUN_MIN` | `{line: null, seconds, end}` | +| `line_clipped_at_end` | A line's last heard word ends inside the final `CLIP_TAIL_SECONDS` of its shot (when the line has one) or of the file, and that tail is still above `GUARD_FLOOR_DBFS` and within `CLIP_TAIL_BELOW_WORD_DB` of the last word's own loudest window (a quieter tail is a score bed or room tone, not the word - #953) | The tail's level in dBFS / `GUARD_FLOOR_DBFS` | `{line, seconds, word, shot}` - `shot` when the line has one | | `speech_where_silent` | `lines` is `[]` and words were heard above the floor and outside a repetition loop | Number of words heard / 0 | `{line: null, seconds, word}` - the first; `shot` when shots are known | | `speech_in_silent_shot` | Shots are known, at least one line names a shot, and words were heard above the floor and outside a repetition loop in a shot no line names. A word is in the shot its midpoint falls in; one a line in another shot matched (a word straddling the cut) is that line's, not counted | Number of words heard in the shot / 0 | `{line: null, seconds, word, shot}` - the shot's first counted word, `seconds` no earlier than the shot's start | | `unscripted_speech` | Shots are known and a line names the shot; words were heard inside it that align to no line (a line spoken twice, an ad-lib) | Number of unmatched words in the shot / 0 | `{line: null, seconds, end, word, shot, repeats_line}` - the first such word; `repeats_line` is true when those words repeat or contain one of the shot's lines | -| `speech_not_transcribed` | Between two words of one matched line there is a gap of `UNHEARD_MIN_SECONDS` or more, or one of its words lasts `UNHEARD_MIN_SECONDS` + 2 `UNHEARD_MARGIN_SECONDS` or more, and the interior (trimmed by `UNHEARD_MARGIN_SECONDS` each side) holds a window above `UNHEARD_FLOOR_DBFS`: audible speech the ASR left out of its text, which the line's timing absorbed (#923) | The gap's length in seconds / `UNHEARD_FLOOR_DBFS` | `{line, seconds, end, shot}` - `shot` when the line has one | +| `speech_not_transcribed` | Between two words of one matched line, or between one line's last word and the next line's first (#960; the finding sits on the next line, and a gap holding heard-but-unmatched words is `unscripted_speech`'s), there is a gap of `UNHEARD_MIN_SECONDS` or more, or one of its words lasts `UNHEARD_MIN_SECONDS` + 2 `UNHEARD_MARGIN_SECONDS` or more, and the interior (trimmed by `UNHEARD_MARGIN_SECONDS` each side) holds a window above `UNHEARD_FLOOR_DBFS`: audible speech the ASR left out of its text, which the line's timing absorbed (#923) | The gap's length in seconds / `UNHEARD_FLOOR_DBFS` | `{line, seconds, end, shot}` - `shot` when the line has one | -Every finding is severity `warn`; `at.shot` is set whenever the shot is known. A rule that does not apply to the call is listed in `rules_skipped` with its reason: `speech_in_silent_shot` and `unscripted_speech` are skipped when no shots are known (`line_clipped_at_end` then looks at the file's end only), when the shots carry neither a sample span nor a frame span the take's frame rate can place, or when no line names a shot (no shot is known to be meant silent). +Every finding is severity `warn`; `at.shot` is set whenever the shot is known. A rule that does not apply to the call is listed in `rules_skipped` with its reason: `speech_in_silent_shot` and `unscripted_speech` are skipped when no shots are known (`line_clipped_at_end` then looks at the file's end only), when the shots carry neither a sample span nor a frame span the take's frame rate can place, or when no line names a shot (no shot is known to be meant silent). A take checked alone, with no shot map (an `asset:` file with no manifest beside it), therefore never raises `unscripted_speech`, however many words are `unmatched`: read `unmatched` there, or pass `shots`. With shots known there is no minimum word count - one unmatched word inside a named shot fires it. -The thresholds are module constants in `dw/tasks/script_check.py`, deliberately not in `RULES` (`dw/assessment_rules.py`): every rule there must fire on a synthetic file that holds no script. The answer echoes them under `thresholds`. +The thresholds are module constants in `dw/tasks/script_check.py` (the guard's `GUARD_FLOOR_DBFS` and `GUARD_WINDOW_SECONDS` in its sibling `dw/tasks/script_take.py`, beside `guard_words`), deliberately not in `RULES` (`dw/assessment_rules.py`): every rule there must fire on a synthetic file that holds no script. The answer echoes them under `thresholds`. | Constant | Value | What crossing it does | | -------- | ----- | --------------------- | | `DEFAULT_SIMILARITY` | 0.85 | A line below this similarity is a `line_mismatch` (the `similarity` argument overrides it) | | `GUARD_FLOOR_DBFS` | -65 dBFS (= `DEAD_AIR_FLOOR_DBFS`, `shot_dead_air`'s floor) | A heard word whose loudest window is at or below it is discarded as unheard; also the level a clipped tail must exceed | -| `UNHEARD_MIN_SECONDS` / `UNHEARD_MARGIN_SECONDS` / `UNHEARD_FLOOR_DBFS` | 1.0 s / 0.25 s / -40 dBFS | A gap in a line's words (or a stretched word) this long, with a window above the floor in its margin-trimmed interior, is a `speech_not_transcribed` | +| `UNHEARD_MIN_SECONDS` / `UNHEARD_MARGIN_SECONDS` / `UNHEARD_FLOOR_DBFS` | 1.0 s / 0.25 s / -40 dBFS | A gap in a line's words or between two lines (or a stretched word) this long, with a window above the floor in its margin-trimmed interior, is a `speech_not_transcribed` | | `GUARD_WINDOW_SECONDS` | 0.05 s (= `DEAD_AIR_WINDOW`, `shot_dead_air`'s window) | The window a heard word is measured in - its loudest one, so a span overhanging a pause does not average a real word away | | `CLIP_TAIL_SECONDS` | 0.25 s | The final stretch of the file a line's last word must end in to be `line_clipped_at_end` | +| `CLIP_TAIL_BELOW_WORD_DB` | 15 dB | A tail more than this far under the last word's loudest window is not the word still sounding (a bed under a finished line), so not `line_clipped_at_end` | | `REPEAT_RUN_MIN` | 6 | Heard words repeating the same unit this many times over, back to back, are a repetition loop: every chunk holding one is discarded (`reason: "repetition"`), however loud | | `REPEAT_MAX_PERIOD` | 4 words | The longest repeating unit a loop is looked for in - `pre pre pre` has period 1, `thank you thank you` period 2 | -The result: `findings`, `lines[]` (`expected`, `heard`, `similarity`, `start`, `end`, `shot`), `discarded[]` (`text`, `start`, `end`, `level_dbfs`, `reason` - `below_floor` or `repetition`), `unmatched[]` (heard words aligned to no line), `transcript`, `model_name`, `rules_applied`, `rules_skipped[]` (`rule`, `reason`) and `thresholds`. A line's `shot` is the shot it names, else the shot its heard words overlap most (null when none is known). `shots` is the map as placed - `[{name, start, end}]` in seconds on the take, or null when none is known - and `shots_source` says where it came from: `argument`, `artifact`, `manifest` or `none`. +The result: `findings`, `lines[]` (`expected`, `heard`, `similarity`, `start`, `end`, `shot`), `discarded[]` (`text`, `start`, `end`, `level_dbfs`, `reason` - `below_floor`, `repeated_span` (a chunk repeating the previous one's exact span) or `repetition`, `count` - a repetition loop is one entry spanning its chunks), `unmatched[]` (heard words aligned to no line), `transcript`, `model_name`, `rules_applied`, `rules_skipped[]` (`rule`, `reason`) and `thresholds`. A line's `shot` is the shot it names, else the shot its heard words overlap most (null when none is known). `shots` is the map as placed - `[{name, start, end}]` in seconds on the take, or null when none is known - and `shots_source` says where it came from: `argument`, `artifact`, `manifest` or `none`. How the alignment reads a take: @@ -3093,7 +3217,7 @@ Increase video frame rate using RIFE (Real-Time Intermediate Flow Estimation). T | `model_name` | No | HuggingFace repo with RIFE v4.13 weights (default: `imaginairy/rife-interpolation`) | | `filename` | No | Weights filename within the repo (default: `rife-flownet-4.13.2.safetensors`) | -Uses vendored IFNet v4.13 architecture. Weights are downloaded from HuggingFace Hub on first use. +Uses vendored IFNet v4.13 architecture. Weights are downloaded from HuggingFace Hub on first use. A validate's `downloads_required` lists the `model_name` / `filename` file when it is not cached. **Example:** [interpolate-frames.json](../workflows/templates/interpolate-frames.json) — Generate video with Mochi, then 2x interpolate from 30fps to 60fps. diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index e6695ce2a..7cb3b0e04 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -494,14 +494,21 @@ step (for a `for_each`, the heaviest member). It warns rather than refuses: your offload and quantization config may differ from the template's. Cut frames, size or references until it goes quiet, or declare your own `vram_estimate`, which always wins and is judged exactly as a -template's. A pipeline the catalog declares no ceiling for gets no warning - +template's. Under the ceiling it is not always silent: the template's +estimate was calibrated on its own offload and quantization, so a workflow +that sets either differently (no `offload` where the template sets +`model`, say) gets one warning of kind `vram_config_differs` naming both +settings and the projection - real usage may be higher. Over the ceiling the +one `vram_projection_inherited` warning names that difference instead. A pipeline the catalog declares no ceiling for gets no warning - silence there is not a clearance - and H3's `t2va` and `ref2va` are different identities with different ceilings. The projection is `base_gb`, plus `bytes_per_voxel` times the product of the `voxel_variables`, plus `gb_per_reference` per non-null reference, plus `bytes_per_guide_voxel` times each guide's frames times the canvas (every -voxel variable but `num_frames`). The guides counted are the step's own +voxel variable but `num_frames`); an absent key adds nothing. A declared +estimate refuses on the guide term as on the rest; an inherited one only +warns. The guides counted are the step's own `guides` with a non-null `video`, at their snapped length, and the one a `continuity: "guide"` chain appends (`guide_frames`, 22 or 39). A guide clip is probed at validate, at admission and at run; one that cannot be @@ -822,6 +829,25 @@ workflow. `reference-to-video` and `dialogue-short` templates). The `identity-referenced` trait in the listing marks the workflows that take one. +### Erasing versus repainting a masked region + +A request to *erase or remove* an object and a request to *put something +new* in its place are different operations, and only one of them takes a +prompt. + +- **Erase or remove an object** → the `remove_object` task: prompt-free, it + fills the hole from the surrounding scene, and suits small to medium + objects (a cup, a sign, a watermark). `templates/remove-object.json` takes + a mask you already have; `templates/segment-and-remove.json` masks the + object by name first. A large region, such as a person filling a third of + the frame, comes back smeared. +- **Put something new inside a mask** → a prompted inpainter, + `models/z-image-inpaint` or `templates/inpaint.json`. The prompt names what + to paint. Do not use one to erase: given a prompt for "empty background" it + paints a new object or a patch that does not match (#946). + +`docs/TASKS.md` *Object Removal* has the task's arguments and its mask rules. + ### A spoken scene breaking into a song A musical number: spoken shots, each carrying its own dialogue audio, then @@ -1046,7 +1072,11 @@ Pass `probe="analyze_seams"` (or either of the other two probe names) to get that one probe's full body: every seam's or shot's measurements, not just the ones that crossed a rule. `detail=true` adds every applicable probe's full body under `probes`. An unknown probe is refused before -anything is read, and the error names the three probes. +anything is read, and the error names the three probes. Each seam's full +record also carries look measures - saturation, lightness and contrast +steps, `look_ratio` and `detail_ratio` - that catch a colour or brightness +shift `jump_ratio` misses; no rule reads them (the tasks guide's +`analyze_seams` field table). **The answer.** @@ -1762,6 +1792,10 @@ open at about 3 of 6 sung-word onsets, against 6 of 6 with the reference alone, speech to camera rather than singing. #808 measured the audio's offset, not mouth onsets, so how closely the mouth follows a held-plus-referenced song is still a visual check. Hold is also for picture that must fit audio exactly, such as motion cut to music. +On `fl2va`, hold is the locked-frame lip-sync route (#985: `image-to-video`'s fl2va, a first +frame plus a held 5.17 s song, int8 turbo, two seeds): frame 0 kept the picture's pixels and +the mouth followed the song as in an audio-referenced Ref2VA shot, at about a fifth of the +denoise time of Ref2VA `[keyframe completion + audio reuse]`, which keeps only the framing. `refine_strength` also uses hold, to keep the base pass's own audio, which was generated jointly with that video - a different case from lip sync to supplied audio (see the refine section below). @@ -1804,7 +1838,7 @@ declared variable, so reference a `guides` variable or edit the argument. | `frame` | a multiple of 17 (a VAE chunk boundary): 0, 17, 34, ... | | clip length | 1, 5 or 17m + 5 frames (22, 39, 56, ... 124); any other length is cut down to the longest such length (n < 5 gives 1, 5 <= n < 22 gives 5, else 17 * ((n - 5) // 17) + 5) with a warning | | extent | `frame` + length must not run past `num_frames` (H3 rounds it up to 17n + 5; default 124); ending exactly at the end is fine | -| count | at most 4 guides per step (a VRAM limit: each guide frame adds attention rows, and the template's `vram_estimate` charges each guide's frames on the canvas); `guides: []` is the same as none | +| count | at most 4 guides per step. Each guide is also checked against the card: each guide frame adds attention rows, and the template's `vram_estimate` charges each guide's frames on the canvas; `guides: []` is the same as none | | where | `t2va` or `fl2va` on an H3 pipeline; not with `references` (`ref2va`), not on a non-H3 pipeline | Validation refuses all of the above before the run (`dw/guides.py`); a `previous_result:` @@ -1884,7 +1918,10 @@ So this is a measurement path, not a catalog template: the catalog's native 768p is `templates/minimax/video-with-audio-768p`, and it is the one to use when faces matter. Target `width`/`height` must be multiples of 16, each between 1x and 4x the base -latents' own size, and within H3's 1344x768 (or portrait 768x1344) canvas. The latents +latents' own size, and within the upscale cap of 1920x1088 (or portrait 1088x1920); +validation refuses a literal size off the grid or over the cap before the run. Above +1344x768 the plate is for a tiled refine ([Refining above 768p in +tiles](#refining-above-768p-in-tiles)), and upscale-only it is soft. The latents pass from `base` to `up` to `decode` entirely in memory. A `previous_result:` property on a modular step names a key of the dict its `output` list returns, spelled as the pipeline spells it: `base.latents`, `base.audio` and `base.sampling_rate` - not @@ -1897,10 +1934,10 @@ upscaling only makes sense for a take chosen from something already reviewed, so The upscaler weights (~691 MB, `LBH-123-AI/Minimax_h3_latent_Upscaler`, Apache-2.0, read at a pinned revision) and the H3 VAE download on first use, the same as any other -model. Neither is counted in `plan.downloads_required`, since that walk collects -`from_pretrained_arguments` sources on pipeline steps and does not see a task -argument naming a Hugging Face repo - a box that has run the base pass before but never -this task can still stall mid-run pulling the upscaler. +model. A validate's `plan.downloads_required` lists the upscaler file as a file entry +(`{repo, filename, revision, ...}`) while it is not cached, and the VAE is covered by +the pipeline step's whole `MiniMaxAI/MiniMax-H3` entry, so a box that has run the base +pass before but never this task is told of the upscaler download beforehand. ```json { @@ -2104,12 +2141,16 @@ this task can still stall mid-run pulling the upscaler. ``` To keep the 544p take beside the promotion for comparison, add one more `pair_audio` -step with `"video": "previous_result:base.videos"` and the same `audio`/`sample_rate`, -saved with `"result": {"content_type": "video/mp4", "fps": 24, "subfolder": +step with `"video": "previous_result:base.videos"` and the same `audio`/`sample_rate` +(and `fit`), saved with `"result": {"content_type": "video/mp4", "subfolder": "intermediate"}`. `base.videos` is the pipeline's batch - a list holding the one video - and `pair_audio` unwraps a batch of one (a batch of several is refused, since one track -goes under one video). The dict's frames carry no frame rate, so the result's `fps` -says it; without it the file is written at the 8 fps fallback. +goes under one video). The H3 step hands its `videos` on carrying the model's 24 fps, +the way the tiled refine's do, so `pair_audio` fits the track against 5.17 s and the +file is written at 24 fps without being told. Frames that carry no rate - a plain list +from a task - need `pair_audio`'s own `fps` argument, not the result's: the result's +`fps` is applied by the writer after `fit` has already measured the frames at the 8 fps +fallback, so it would re-time the picture under a track padded to three times its length. ### Refining the upscaled latents: refine_strength @@ -2451,6 +2492,379 @@ Validation refuses a `refine_strength` that is not a number between 0 and 1 (exc refine with no `latents` or no `hold_audio`, a step that is not H3, and a `num_inference_steps` below 2. +To refine a canvas above 1344x768, such as a 1920x1088 plate, see [Refining above 768p in +tiles](#refining-above-768p-in-tiles). + +### Refining above 768p in tiles + +H3's refine runs at 1344x768 at most, and `upscale_h3_latents` now reaches 1920x1088 (or +1088x1920). An upscale-only plate above 768p is soft: `upscale_h3_latents` to 1920x1088 +alone, decoded, is not a 1080p render, since the interpolation adds no detail. To put the +detail in, give the refine step `refine_tiles`, which refines a `latents` canvas bigger than +H3's own in overlapping tiles. It is a second argument to `refine_strength`, not a +replacement: the strength, the held audio and the step count mean what they did above. + +```json +{"refine_tiles": {"width": 1344, "height": 768, "overlap": 256}} +``` + +The step plans the tiles on the 32 px grid: the fewest per axis whose overlap is at least +the requested one, spread evenly, the last flush to the edge. A 1920x1088 canvas at +1344x768 with overlap 256 is 2x2, tiles at x 0 and 576, y 0 and 320. The noise is drawn +once for the whole canvas from the step's generator and cropped per tile, so overlapping +tiles start from the same noise. Each tile is denoised through H3 without its decode block, +and the prompt is encoded once. The tiles are blended in latent space with a linear feather +across each overlap, the weights normalized, and the blended canvas is decoded once with VAE +spatial tiling. The step returns the full-canvas `latents`, the video and stage 1's held +audio, once. Its `videos` carry H3's 24 fps, so the `pair_audio` below writes them at +that rate without being told. The job's events show one log line per tile and one decoding phase. + +It is slow. A 1920x1088 refine is roughly 4 tiles, each a 768p refine, plus a 1080p decode: +about 25-30 minutes, against about 9.6 minutes for a native 768p take (an estimate, not +measured). The validate projection prices the denoise at the tile's size and the decode at the +canvas's, so a tiled refine does not warn as if it denoised 1080p at once. + +```json +{ + "id": "H3LatentUpscaleRefineTiles", + "description": "Promote a MiniMax-H3 take from 960x544 to a 1920x1088 canvas in latent space, refine it in overlapping 1344x768 tiles, and pair the held audio.", + "variables": { + "prompt": "prompt:minimax/fox_dawn_context_ir", + "num_frames": 124, + "num_inference_steps": 5, + "video_shift": 6.0, + "audio_shift": 3.0, + "weights_dtype": "{int4}", + "lora_scale": 1.0, + "lora_alpha": null, + "lora_model_name": "lightx2v/Minimax-h3-Turbo", + "lora_weight_name": "minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors", + "lora_adapter_name": "turbo", + "seed": 42 + }, + "variable_constraints": { + "num_frames": { + "modulus": 17, + "remainder": 5, + "reason": "the video VAE encodes 17 * n + 5 frames" + } + }, + "seed": "variable:seed", + "steps": [ + { + "name": "base", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "t2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "variable:lora_adapter_name", + "scale": "variable:lora_scale", + "alpha": "variable:lora_alpha" + } + ], + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "num_frames": "variable:num_frames", + "width": 960, + "height": 544, + "num_inference_steps": "variable:num_inference_steps", + "output": [ + "videos", + "audio", + "sampling_rate", + "latents" + ] + } + } + }, + { + "name": "up", + "task": { + "command": "upscale_h3_latents", + "arguments": { + "latents": "previous_result:base.latents", + "width": 1920, + "height": 1088 + } + } + }, + { + "name": "refine", + "pipeline": { + "configuration": { + "component_type": "ModularPipeline", + "pre_load_modules": [ + "sdnq" + ], + "load_components": { + "dtype": "torch.bfloat16", + "quantization_config": { + "transformer": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "use_quantized_matmul": true, + "dequantize_fp32": false, + "modules_to_not_convert": [ + "proj_in", + "audio_proj_in", + "context_embedder", + "time_embedder", + "time_proj", + "token_refiner", + "norm_out", + "proj_out", + "audio_proj_out" + ] + } + }, + "text_encoder": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "variable:weights_dtype", + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false, + "modules_to_not_convert": [ + ".model.visual", + "lm_head" + ] + } + }, + "vae": { + "configuration": { + "config_type": "sdnq.SDNQConfig" + }, + "arguments": { + "weights_dtype": "{int8}", + "quant_conv": true, + "use_quantized_matmul_conv": true, + "quantization_device": "cuda", + "return_device": "cpu", + "dequantize_fp32": false + } + } + } + }, + "components": { + "transformer": { + "group_offload": { + "offload_type": "block_level", + "num_blocks_per_group": 2, + "use_stream": true, + "record_stream": true, + "low_cpu_mem_usage": true + } + }, + "text_encoder": { + "remove_modules": [ + "lm_head" + ] + }, + "text_encoder.model": { + "truncate_layers": { + "language_model.layers": 51 + }, + "group_offload": { + "offload_type": "leaf_level" + } + }, + "vae": { + "device": "cuda", + "residency": "on_demand" + }, + "audio_vae": { + "device": "cuda", + "residency": "on_demand" + } + } + }, + "from_pretrained_arguments": { + "model_name": "MiniMaxAI/MiniMax-H3", + "workflow": "t2va" + }, + "loras": [ + { + "model_name": "variable:lora_model_name", + "weight_name": "variable:lora_weight_name", + "adapter_name": "variable:lora_adapter_name", + "scale": "variable:lora_scale", + "alpha": "variable:lora_alpha" + } + ], + "scheduler": { + "shift": "variable:video_shift" + }, + "audio_scheduler": { + "shift": "variable:audio_shift" + }, + "arguments": { + "prompt": "variable:prompt", + "num_frames": "variable:num_frames", + "num_inference_steps": "variable:num_inference_steps", + "output": [ + "videos", + "audio", + "sampling_rate" + ], + "latents": "previous_result:up", + "refine_strength": 0.2, + "refine_tiles": { + "width": 1344, + "height": 768, + "overlap": 256 + }, + "hold_audio": "previous_result:base.audio" + } + } + }, + { + "name": "mux", + "task": { + "command": "pair_audio", + "arguments": { + "video": "previous_result:refine.videos", + "audio": "previous_result:base.audio", + "sample_rate": "previous_result:base.sampling_rate", + "fit": "video" + } + }, + "result": { + "content_type": "video/mp4", + "fps": 24, + "subfolder": "final" + } + } + ] +} +``` + +The refine step uses the 768p turbo LoRA with its own schedule (`video_shift` 6, `audio_shift` +3, alpha left to the file), as `workflows/templates/minimax/video-with-audio-768p.json` does. It +carries no `width` or `height`: the canvas is the `latents`. Naming them is allowed, and must +equal the canvas, which is the latent height x width x 16. + +Validation and the run refuse, naming `refine_tiles`: + +- `refine_tiles` without `refine_strength`, or without what refine needs: `latents`, + `hold_audio`, an H3 step and `num_inference_steps` of at least 2; +- a `width`, `height` or `overlap` that is not an integer multiple of 32; +- a tile larger than 1344x768 (or 768x1344), or larger than the canvas; +- an `overlap` below 192, or at or above half the tile's shorter side; +- a reference image or video, a ``, a keyframe or `guides` on the step; +- an explicit `width` or `height` that does not match the `latents` canvas. + ### Chained Video Generation Video pipelines generate short clips - a `chain` block on a pipeline step runs the diff --git a/docs/img/cup-object-mask.png b/docs/img/cup-object-mask.png new file mode 100644 index 000000000..2d419eaa0 Binary files /dev/null and b/docs/img/cup-object-mask.png differ diff --git a/docs/proposals/audits/2026-10-h3-wangp-probes.md b/docs/proposals/audits/2026-10-h3-wangp-probes.md new file mode 100644 index 000000000..e78b0a510 --- /dev/null +++ b/docs/proposals/audits/2026-10-h3-wangp-probes.md @@ -0,0 +1,205 @@ +# MiniMax-H3 claims from WanGP's field notes: record-only probes (#985 stage A, #988) + +Probe date: 2026-10-25. Server `lem`, develop @ b80dec77. The sung +`measure_sync` figures were re-measured on develop @ f9b96b20 (the #990 fix). +Agent: claude-opus-5-5 (anthropic). Workspace `qa-h3-wangp-probes`. +No engine or skill file was changed. This document only records results. + +Pinned configuration (Don, at plan approval), the same for every cited job: +- `weights_dtype` **int8**, with the turbo LoRA **on**. Each job's `note` + states both. +- Ref2VA (P1, P4 arm b): `transformer_ref`, LoRA + `minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors`, 9 steps, video + shift 12, audio shift 3, `sparse` attention. +- FL2VA (P4 arm a): `transformer`, LoRA + `minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors`, 5 steps, shift 6. +- 960x544, 124 frames (5.17 s at 24 fps). Every shot holds its source slice + with `hold_audio`. + +Each arm ran on seeds 1001 and 2002. A claim is **confirmed** only when both +seeds of the claim-broken arm show the effect and neither seed of the +claim-followed arm does. It is **refuted** when the broken arm doesn't show +the effect, and **inconclusive** when the seeds disagree. Claim 4 has no +broken arm (it asks whether a route works, not whether a rule matters). It is +confirmed when both FL2VA seeds keep frame 0 and their mouths follow the +track, and refuted when either fails. + +With `hold_audio`, the output soundtrack *is* the source slice. So the +transcripts and `measure_sync` can't tell the arms apart: they measure audio +against audio, not lips. The mouth frames decide every verdict here. + +## Verdicts + +| # | Claim (as stated on #985) | Verdict | Jobs | Observation | +| --- | --- | --- | --- | --- | +| 1 | "Lip-sync without ``": for lip-sync to supplied audio, leave the words out of `` and ask for natural lip movements synchronized to the track. | **refuted** | sung: 3d9a4e7377fa, 6d7f6faf8e60 (a); d2384deebb50, f27b7a7af223 (b). spoken: c029d63a15e8, c0763cc81d6b (a); 8451e9fd9289, e0790f00b4c2 (b) | With the track held, the words in `` did no harm. On both tracks and both seeds, arm a's mouth opens and shuts at the same instants as arm b's, and the identity holds. See *Probe 1*. | +| 4 | "FL2VA with a held soundtrack" lip-syncs. Our skill says "FL2VA takes no audio" (SKILL.md:31). | **confirmed** | 52195bad071b, f4fe60dcca4d (FL2VA); 422f22a7df71, 8e42ec21cf9a (Ref2VA keyframe) | FL2VA plus `hold_audio` keeps the input picture pixel for pixel at frame 0. On both seeds its mouth follows the held song with the same open and shut pattern as the audio-referenced Ref2VA shots. #852's Ref2VA keyframe route also lip-syncs, but frame 0 keeps only the framing: the arms hang instead of being folded, and the shot is wider. See *Probe 4*. | + +Not run: none. Claims 2 and 3 were dropped and deferred by the plan, not +probed. + +### `measure_sync` figures + +| Track | Jobs | `lag_seconds` | `confidence` | `reference_seconds` | Findings | Measured by | +| --- | --- | --- | --- | --- | --- | --- | +| sung (la-vela) | all 4 P1 sung, all 4 P4 | 0.0 | 0.979 | 5.1667 | none | re-measure job 85fb684df2ef | +| spoken (hal-voice) | all 4 P1 spoken | 0.0 | 0.756 | 2.499 | none | each job's own `sync` step | + +Lag 0.0 is expected, because the output audio is the held slice. The +confidence is the same for every sung job for the same reason: each one +plays the same 5.17 s of the song. + +**Sung figures re-measured after #990.** The first in-job `sync` steps of +the sung jobs read 0.0 / 0.175 (`sync_low_confidence`) with +`reference_seconds` 118.751. The cause was #990: `slice_audio` ignored +`num_frames` when given `start_seconds` and no duration, so each sung +`slice` step wrote 118.751 s of song from 42.5 s onward. `measure_sync` then +compared a 5.17 s shot against that whole length. + +**The shots themselves were not affected.** Both inputs that read the slice +crop it to the shot's `num_frames / fps` from its start: +- `hold_audio` encodes with `max_duration` and plays a fitted copy + (`dw/pipeline_processors/h3_hold_steps.py:219,236`); +- the Ref2VA `