diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md new file mode 100644 index 0000000000..7d4a91b0b9 --- /dev/null +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -0,0 +1,618 @@ +# DL3: an output dataset says what produced it + +Branch `refactor/DL3-output-provenance`. Gap 6 of `D1-cellpose-n2v2-custom-functions.md` (PR #1230). + +**Lands after DL1 and after four library releases.** Two pins cannot be installed from PyPI yet: + +- `python-introspect>=0.3.0,<0.4` (set in `pyproject.toml`; 0.3.0 is on PyPI). ObjectState, + formbridge, pyqt-reactive and zmqruntime all declare `python-introspect<0.3`, so the resolver + refuses this branch until each of them releases with the bound widened to `<0.4`. Their + submodules run against 0.3.0 here (formbridge and ObjectState suites: no failure; zmqruntime: two + IPC transport tests fail with `EOFError` in `multiprocessing`, not in the codec; pyqt-reactive + not run). +- PolyStore: the storage part is PolyStore PR OpenHCSDev/PolyStore#41, to be released as **0.6.4** + (0.6.3 is the arraybridge 0.5.0 migration and requires `arraybridge>=0.5.0,<0.6`). OpenHCS main + is on arraybridge 0.4.1 until `refactor/DL1-device-and-memory-types` merges. Until then the + submodule `external/PolyStore` points at `5301d35` (branch `feat/file-content-identity`, still + based on PolyStore 0.6.1 and still titled 0.6.3) and `pyproject.toml` still says + `polystore>=0.6.3,<0.7`, which is wrong. At that time set: `polystore>=0.6.4,<0.7`, + `arraybridge>=0.5.0,<0.6`, submodule at the 0.6.4 tag. + +**Current state: see "Third hand-off" at the end.** Earlier sections are the first two agents'; +where they describe a union decoded by its fields, one metadata transaction, or +`OpenHCSMetadataTarget.saved_path_producers`, the third hand-off replaces them. + +## What is wrong (measured on `origin/main` at `1b5fe99f5`) + +- `openhcs_metadata.json` of an output dataset has no pipeline, function, parameter, source-hash + or model-hash key (keys of a real output: `subdirectories..{workspace_mapping, + source_metadata, source_projection, image_files, ...}`; `OpenHCSMetadataFields`, + `openhcs/core/virtual_workspace_metadata.py:325`). +- The pipeline code that reaches the server is rendered with `clean_mode=True` + (`openhcs/core/pipeline_document.py:96`); function kwargs equal to the signature default are + dropped (`openhcs/serialization/pycodify_formatters.py:150`). No non-GUI caller passes `False`. +- A declared input file is resolved to an absolute path and checked to exist + (`CompilationPathResolver.resolve`, `openhcs/core/pipeline/compilation_session.py:102`); no + content identity is taken. PolyStore hashes only to verify a caller-supplied hash + (`DataSource.read_verified_source_bytes`, `external/PolyStore/src/polystore/base.py:366`). +- Config resolution already returns, per field, the scope and class that supplied the value + (`ObjectState.resolve_saved_object`, used at `compiler.py:923`), and + `ResolvedPipelineDefinition.step_provenance` keeps it for steps. The compiler reads one field of + it (`compiler.py:1727`) and the orchestrator discards it for the pipeline config + (`orchestrator.py:1057`). Nothing writes it out. +- The producer of an output is stated in exactly one typed place: + `CompiledStepPlan.producer_identity(...)` (`compiled_step_plan.py:386`) builds a + `StreamProducerIdentity` (step name, pipeline position, step scope id, output kind and key). It is + sent with every viewer item (`viewer_transport.py:343`). On disk only its `output_key` survives + (`function_outputs.py:869`), plus a `_step` token in artifact file names + (`path_planner.py:2289`). Measurement tables, ROI archives and label images are not listed in the + metadata document at all. +- Function identity: `FunctionMetadata.composite_key` (`registry:name`); the library version is + known per registry only (`LibraryRegistryBase.get_library_version`); a custom function's source + hash is stamped on the callable (`declaration_revision`) but its text is not kept with it. + +## Design + +Standard used: P10 Definition 2.1(c) and Proposition 6.1 (the output carries the declaration that +produced it), P9 Theorem 5.3 (a log or private attribute does not make the source recoverable; it +has to be in the declared output), P9 "Several fact versions" (evidence that crosses versions +carries the version), P9 "Only executed, contract-guaranteed channels count" (always written, no +opt-in), P9 Theorem 5.5 (no parallel side table once the record determines the source). + +**One record, one writer, one time.** The compiler builds a `RunRecord` for each compilation, from +the facts only it holds, and puts it on the `CompiledExecutionBundle`. The parent process fills in +the content of declared input files when execution starts, and the existing single writer of the +output dataset's metadata, `OpenHCSMetadataTarget.finalize_completed_dataset` (parent process, one +locked transaction per `openhcs_metadata.json`, also on failure), stores it in that document: + +``` +openhcs_metadata.json + subdirectories: {...} (unchanged) + run_records: {: RunRecord} record id = sha256 of the record's canonical JSON + produced_files: {: {"run_record": , "producer": {...}}} +``` + +A record no file refers to any more is dropped in the same transaction, so a dataset written twice +keeps exactly the records its files need. + +**What the record holds (sufficient to re-run).** +- `pipeline_code`: the pipeline document rendered complete (`clean_mode=False`: every function + parameter and every stored config field is written). `global_config_code`: the saved global + config rendered the same way. These are the *stored documents*; rebuilding executes them. +- `configuration`: the *resolved* configuration, one entry per leaf field of the effective config + and of each step: the value (as its Python expression, from pycodify) and its source: the scope + id and class ObjectState reported, or "set here" when the object itself holds the value. This is + a different read kind from the document and is kept beside it, never merged into it. +- `software`: OpenHCS version, Python version, platform. +- `steps[]`: name, pipeline position, step scope id (the two fields a producer identity carries), + declared input and output memory type, the assigned device per framework, and `functions[]`. +- `functions[]`: identity (below), `parameters` and `declared_files`. +- `custom_sources`: `{sha256: source text}`, each text once. + +**Function identity is content.** The function reference family answers it +(`FunctionReference.identity_record`): the registry key or import path, the distribution and version +of the module that defines the function, and the distributions and versions of the modules that +module imports (this is where Cellpose's version comes from for a custom function: its source's +imports). A custom function adds the sha256 of its source; the text is stored once under +`custom_sources`. The text is kept on the source's own namespace object +(`CustomFunctionSourceNamespace`), which already owns the hash. + +**Defaults are recorded.** Each parameter of the function's signature appears in `parameters` with +its effective value and a source: `pipeline` (written in the pipeline), `function_default` (taken +from the signature of this function at the recorded version), or `run_time` (supplied by the +platform: an artifact input, a bound config value). The source is a small family of classes; its +JSON `kind` is derived from them. The editable pipeline the user saves is not changed: it stays +concise (`clean_mode=True`). Only the record's copy is rendered complete. + +**Declared files are identified by the storage layer.** PolyStore gains +`DataSource.file_content(path) -> FileContent`, a family: `HashedFileContent(sha256, size_bytes)` +or `UnhashedFileContent(reason)`; `FileManager.file_content(path, backend)` delegates to the +backend. The path declaration family says what its kind needs +(`DatasetPathDeclaration.content_at_run_start`): an input file is hashed; an input directory, an +output file and an output directory are recorded as not hashed, each with its reason. Hashing +happens once per file per run, in the parent process when execution starts (not at compile: a +compiled plan can be executed later, and the bytes that matter are the ones the run reads). At +the end of the run the writer asks again for size and modification time; a file that changed +during the run is recorded as `changed_during_run`. Functions hash nothing. + +Cost, measured here on a 1 GiB file (`.claude/scratch/hashcost.py`, NVMe, CPU with SHA +extensions): sha256 0.58 s with the file in the page cache, 2.15 s after dropping it; blake2b +1.03 s, md5 1.22 s. That is cheap enough to be always on, so there is no switch. A backend with no +physical file answers `UnhashedFileContent` with its reason; the record never omits the entry. + +**Per-output linkage is the existing producer identity, written down.** No second linkage: the +`StreamProducerIdentity` a viewer item already carries is the value stored under +`produced_files[path].producer`. Workers already report what they saved +(`StepExecutionObservation`); that report gains the producer of each saved path, taken from the +same two calls the streaming path uses (`producer_identity_for_main_flow`, +`producer_identity_for_artifact`). The reader matches a producer to its step by step scope id and +pipeline position. A measurement row is linked through its table file; a viewer item through the +identity it arrived with. The `_step` file-name token is not read by anything here. + +**One reader.** `RunRecord.for_dataset(root)`, `RunRecord.for_output_file(path)` (walks up to the +dataset's metadata document; returns the record and the step) and `RunRecord.rebuild()` (registers +the recorded custom sources after checking their hashes, executes the two documents, returns the +pipeline document and global config). The agent capability `openhcs_read_run_record` (MCP tool and +generated CLI command) wraps the same calls. + +**Dependencies on sibling branches.** +- DL1 (device and memory types): the device is read from `CompiledStepPlan.device_assignment`, the + place it is stated at this branch point. If DL1 moves or extends that statement (CPU devices, + conversions), the record follows by reading the new one. +- DL2 (declared files between steps): an input file an earlier step of the same run writes does + not exist when the run starts. At this branch point compile rejects that pipeline; when DL2 + lands, such a file must be hashed after its producer finishes. `content_at_run_start` is the + single place that decision is made. + +## Guards and tests + +Unit (`tests/unit/test_run_record.py`): default parameters appear with `function_default`; a custom +function's record changes when its body changes and its name does not; a declared input file is +hashed by the file manager and a declared output file is recorded as not hashed with a reason; the +resolved configuration names the scope and class of an inherited value; the record id is stable +for equal input; a file with no entry is reported as unrecorded, not guessed from its name. +Integration (`tests/integration/test_run_record_round_trip.py`): run a small pipeline, read the +record from one output file, rebuild, run again, compare outputs byte for byte. + +## Done when + +The Cellpose example leaves a record with the custom source hash, every parameter, Cellpose's +version and the device; the N2V2 denoise example records the model file's hash; gates pass. + +## Progress notes + +**Wired (commit `c2d689e61`).** `CompiledExecutionBundle.run_declaration` (required; built in +`compile_pipelines` before `_resolve_pipeline_once`, step values added after it; pickles with the +bundle). `StepExecutionObservation.producer_by_saved_path`, filled by +`OpenHCSMetadataTarget.saved_path_producers` at the two places a step's observation is built. +`execute_bundle_request` calls `RunRecord.at_run_start` before workers exist and passes the record +to `finalize_completed_dataset`; `reconcile_completed_dataset` writes it inside its one `update`. +A dataset root that received only non-image files gets a document; a document the run saved +nothing under (the input plate) is left untouched. The real ZMQ journey writes a record (59 kB for +one step, of which 246 installed distributions are most). + +**What the design had wrong, and what was done instead.** +- `clean_mode=False` did not write a parameter the pipeline leaves out (it only stopped dropping + written ones). Complete rendering now adds the signature default of every parameter of every + pattern function (`with_every_parameter_written`, `pycodify_formatters.py`), except parameters + the platform supplies when it calls the function (artifact inputs, `slice_by_slice`, + `dtype_config`): writing their signature default would replace the supplied value. +- Two default kinds had no formatter and would have stopped compilation (`range` in + `skimage:filters.frangi`, `numpy.sum` in `skimage:measure.block_reduce`). Both now have one; + `test_every_registered_function_default_can_be_written_in_the_record` guards the registry + installed here (283 functions; CuPy and pyclesperanto are not installed on this machine). +- The WIP stored the *effective* config as the global document. Rebuilding against it gave equal + values but different sources (everything inherited from the global scope). The record now stores + the saved global config (`global_config_code`, as this design says); with none set it stores the + default-constructed one, which the round trip shows resolves identically. +- A step's values are read from `AbstractStep.configuration_parameters()` (the function pattern is + not configuration). An inherited path is checked before descending: a nested config the step + leaves unset is inherited whole and ObjectState lists it as one path. +- The producer of a main-flow image is the step that produced the pixels. A step that returns its + image unchanged saves a file whose producer is the earlier step: the same identity the viewer + item carries. +- A rebuilt run writes every parameter, so its record says `pipeline` where the first said + `function_default`. Nothing else differs apart from the plate path and the submission scope + (`submission_` in step scope ids). Output files (TIFF and CSV) are byte-identical. + +**Not done.** `changed_during_run` (asking again for size and modification time at the end of the +run): the design text above still describes it; it is dropped, a file modified during a run is +not detected. + +## Hand-off (2026-10-12): state, results, what remains + +A fresh agent continues from here. PolyStore part: PR OpenHCSDev/PolyStore#41 (branch +`feat/file-content-identity`, commit `8ee8ce4`, version 0.6.3, not merged, not on PyPI). This +branch pins `polystore>=0.6.3,<0.7` and points the submodule at that commit. + +### Implemented and committed + +| Commit | What | +|---|---| +| `7ea5440e1` | this receipt's design | +| `1786bbc13` | `openhcs/core/function_identity.py`; `identity()` on the function reference and registry entry families; custom source text kept on `CustomFunctionSourceNamespace`; `content_at_run_start` on the declared path family; `openhcs/core/run_record.py` | +| `c2d689e61` | bundle carries `RunDeclaration`; workers report `producer_by_saved_path`; `RunRecord.at_run_start` in `execute_bundle_request`; one-transaction write in `reconcile_completed_dataset`; complete rendering writes every parameter; `tests/unit/test_run_record.py` (17), `tests/integration/test_run_record_round_trip.py` | +| `09a712ad5` | `openhcs_read_run_record` capability (`openhcs/agent/services/run_record_service.py`), `tests/unit/test_run_record_capability.py` (7) | +| `d33b72496` | merge of `origin/main` | + +### Acceptance run (CPU venv, torch 2.14.1+cpu, Cellpose 3.1.1.3, 3 BBBC039 fields) + +Scripts and outputs: `.claude/scratch/acceptance/` (untracked). NumPy-declared Cellpose source +from commit `47fbed348` (the torch-declared one cannot compile without CUDA; DL1's gap). + +- Cellpose then `measure_labelled_objects`: a label TIFF resolves to step `segment`, a + measurement CSV to step `measure`. `custom_source_sha256` equals sha256 of the source file. + All 7 Cellpose parameters are recorded as `function_default`; `min_area_px=20` as `pipeline`. + `imports.libraries`: cellpose 3.1.1.3, numpy 2.0.2. Record 78,764 bytes, metadata document + 97,633 bytes. Compile 2.53 s, execute 19.97 s. +- Body changed, name kept: source hash `a2824fff…` -> `bca58da6…`, record id `7ce7240d…` -> + `37e46efc…`; each dataset stores its own text. +- Re-run from the record in a fresh process with nothing registered: 3 main images, the label + TIFF and both step CSVs byte-identical. ROI zips differ only in an embedded absolute + `source_path` and zip timestamps (all `.roi` members identical); `segmentation_summary.txt` + lists absolute paths; `metaxpress_style_summary.csv` has a different `Plate ID`. +- N2V2 denoise (NumPy-declared copy; model file left at its default): `declared_files` entry + `declared_as: "input file"`, `hashed: true`, sha256 `0340245e…`, 124,144,631 bytes, equal to + hashlib; after overwriting the model the recorded hash is `5a2582ea…`. +- Hash cost: standalone 1 GiB, sha256, 0.58 s warm and 2.15 s after dropping the page cache + (blake2b 1.03 s, md5 1.22 s). Through `filemanager.file_content` under load (full suite running, + 2 GB free RAM): 12.56 s first read, then 1.29 s and 1.46 s. +- Capability default response: 52,808 bytes for a dataset root, 68,175 for one file. + +Example (Cellpose step, trimmed): + +```json +{"name": "segment", "pipeline_position": 0, + "step_scope_id": "…/runA/source::pipeline::submission_7fd30cd3c3c0::step_0", + "input_memory_type": "numpy", "output_memory_type": "numpy", "assigned_gpu_devices": [], + "functions": [{ + "identity": {"key": "openhcs:cellpose_segment", "module": "openhcs.processing.custom_functions", + "name": "cellpose_segment", + "imports": {"libraries": [{"distribution": "cellpose", "version": "3.1.1.3"}, + {"distribution": "numpy", "version": "2.0.2"}], + "modules_without_distribution": ["openhcs"]}, + "custom_source_sha256": "a2824fffc06b7c906d3af193fa4ff21ba24a7a5ddf0a16e54419aa7eb077d9af"}, + "parameters": [ + {"name": "image", "value": "", "source": {"kind": "run_time"}}, + {"name": "model", "value": "cellpose_segment.__openhcs_source_a2824f….__openhcs_member_….NUCLEI", + "source": {"kind": "function_default", "function": "openhcs:cellpose_segment"}}, + {"name": "diameter", "value": "None", "source": {"kind": "function_default", "function": "openhcs:cellpose_segment"}}, + {"name": "flow_threshold", "value": "0.4", "source": {"kind": "function_default", "function": "openhcs:cellpose_segment"}}], + "declared_files": []}]} +``` + +### Gates + +- `guardrails_local.sh` on `d33b72496`: **rc=1**. Lines: `ratchet openhcs exit 1`, + `ratchet scripts exit 0`, `ratchet benchmark exit 0`, `38 passed, 22 warnings in 68.54s`, + `nra-r1 exit 1`, `guardrails rc=1`. Ratchet delta: `GodClassExcess PipelineCompiler +7`, + `GodClassExcess OpenHCSMetadataTarget +24`, `LongBooleanChain +1`, + `BooleanChainTerms openhcs/core/steps/abstract.py +1`, + `BooleanChainTerms openhcs/core/virtual_workspace_metadata.py +4`, + `ForeignAbsenceProbe` +1 in `virtual_workspace_metadata.py` and +1 in + `serialization/pycodify_formatters.py`, `StringSubscript +21`, + `AnyAnnotation openhcs/core/run_record.py +6`. NRA increased: `unmodeled_record_shape` in + `openhcs/core/run_record.py` and `openhcs/agent/services/run_record_service.py`. +- `verify_branch.sh DL3`: see "Full suite" below. +- Not run: `smoke_local.sh`, official30 parity (required: compilation, execution and + materialization changed). + +### What remains, in order + +1. **Model the record as nominal types** (clears `StringSubscript`, `AnyAnnotation`, + `unmodeled_record_shape`): `RunRecord` holds `content: Mapping[str, Any]` and the reader and + the capability service index it by string. Replace with frozen dataclasses (software, step, + function, parameter, declared file) that encode to and decode from JSON in one place; + `RunRecordService` reads attributes. +2. **Move the added code out of the two oversized classes**: `saved_path_producers` out of + `OpenHCSMetadataTarget` (it belongs with `ProducedFiles`), the declaration calls out of + `PipelineCompiler`. Remove the two new absence probes and the lengthened boolean chains + (`StepExecutionObservation.is_empty`, `reconcile_completed_dataset`). +3. **Defect: a declared path left at its default is recorded as written in the pipeline**, with + the resolved absolute path as its value. `CallableContract.resolve_declared_paths` + (`callable_contract.py:~1296`) puts the resolved default into the kwargs and `_parameters` + (`run_record.py`) treats every kwarg as written. Record the written or default value with its + true source and keep the resolved path only in `declared_files`. Failing test first. +4. **`enabled`** is written in `pipeline_code` function kwargs but has no `parameters` entry. +5. **Device on CPU**: `assigned_gpu_devices: []` does not say the step ran on host memory. State + it explicitly from the place DL1 leaves the assignment. +6. **Enum defaults of custom functions** render as + `name.__openhcs_source_.__openhcs_member_.MEMBER` (same mangling as D1 gap 10). +7. **Transitive libraries**: `imports.libraries` lists direct imports only (torch is not listed + for Cellpose; it is only in `software.installed_distributions`). Decide: record the + requirement closure of the direct imports, or state the rule in the record. +8. **Plate-level outputs are unrecorded**: `results/metaxpress_style_summary.csv` (analysis + consolidation) has no `produced_files` entry; `for_output_file` raises `UnrecordedOutputFile` + for it. It needs a producer identity of its own kind (it is not a step output). +9. **Record size**: 59 to 79 kB per record, mostly installed distributions and per-step + configuration repeating absolute scope ids. The capability's shortened response is still 53 kB. +10. **Fallback to remove or justify**: with no saved global config, `RunDeclaration` stores a + default-constructed one. +11. **Reader API**: no public "is this a dataset root"; `ProducedFile` has neither dataset root + nor relative path, so the capability's read of the metadata document above a file bypasses + the agent path policy; a directory without metadata reports as an unrecorded file. +12. `changed_during_run` is not implemented; update the design text or implement. +13. Reproducibility leaks found by the round trip, outside this branch's scope: ROI zip metadata + and `segmentation_summary.txt` embed absolute source paths; zip timestamps. +14. Gates: guardrails to `rc=0`, `verify_branch.sh`, `smoke_local.sh`, parity under + `heavy_slot.sh` and `official30.lock`. Then mark the PR ready. PolyStore #41 must merge and + release 0.6.3 before CI here can install the pin. +15. Delete `.claude/venv-cpu` (2.3 GB) and `.claude/scratch/` in this worktree. +16. For the #1230 branch: nothing in the example functions has to change for this gap; the count + table's own `model` and `diameter_px` columns are no longer the only place those are stated. + +### Full suite (`verify_branch.sh DL3`, on `d33b72496`, lines verbatim) + +``` +suite: = 7 failed, 11754 passed, 25 skipped, 358 warnings, 6 subtests passed in 1948.52s (0:32:28) = +collected 11785, reported 11761, crash lines 0 +new vs known: 5 +only on DL3: +tests/unit/agent/test_mcp_server.py::test_mcp_stdio_server_roundtrip_returns_errors_as_payloads +tests/unit/test_mcp_dev_client_terminal_input.py::test_long_terminal_command_reuses_original_parser_and_client[...] +tests/unit/test_pycodify_formatters.py::test_function_step_round_trip_omits_injected_values_but_keeps_runtime_settings[False-expected_kwargs1] +fails on both with a different error: +``` + +Failing ids outside `known.fail` (5), with the believed cause (none investigated beyond this): + +| Id | Rerun alone | Believed cause | +|---|---|---| +| `tests/unit/test_pycodify_formatters.py::test_function_step_round_trip_omits_injected_values_but_keeps_runtime_settings[False-expected_kwargs1]` | fails on DL3 only | **This branch.** `clean_mode=False` now writes every parameter; the test expects only the written kwargs in the full rendering. Same change of meaning as the one already made in `tests/unit/agent/test_authoring_roundtrip_boundaries.py`; restate the expectation for the full rendering, keep the clean one. | +| `tests/unit/test_mcp_dev_client_terminal_input.py::test_long_terminal_command_reuses_original_parser_and_client[...]` | fails on DL3 only | **Probably this branch**: the generated CLI gained `read-run-record`, or the complete rendering changed a long command's text. Not looked at. | +| `tests/unit/agent/test_mcp_server.py::test_mcp_stdio_server_roundtrip_returns_errors_as_payloads` | fails on DL3 only in the gate's rerun; passed alone earlier (`1 passed` with `-k stdio_server_roundtrip`) | Seen before as a 5 s stdio-initialize timeout under load (acceptance run and suite ran together). Rerun on a quiet machine before treating it as real; if it persists, the new tool lengthens server start. | +| `tests/unit/agent/test_mcp_affine_inspection_progress.py::test_resident_continuous_connections_keep_affinity_and_wire_progress[original-continuous]` | passed on rerun (both trees) | load-dependent | +| `tests/unit/test_ack_return_route_journey.py::test_two_native_stream_producers_keep_deferred_ack_ownership[TransportMode.IPC]` | passed on rerun (both trees) | load-dependent | + +The remaining 2 of the 7 failures are the ids in `known.fail`. Fix the first two rows as step 0 +of "What remains". + +## Second hand-off (2026-10-12): what changed, what remains + +Commit `3a55d1fe4` (pushed). Stopped at the context limit after items 1 and 2 of the continuation +brief; nothing else is started in the tree. + +### Done + +**Brief item 1: the record is nominal types** (`openhcs/core/run_record.py`, +`openhcs/core/function_identity.py`). + +| Type | Holds | +|---|---| +| `RunRecord` | `software`, `pipeline_code`, `global_config_code`, `configuration`, `steps`, `custom_sources` (text by sha256, each once) | +| `SoftwareEnvironment` | OpenHCS, Python, platform, `installed_distributions` (version by name) | +| `StepRecord` | `calls: StepCalls`, `configuration: tuple[ConfigValuesFromOwner, ...]` | +| `StepCalls` | name, pipeline position, step scope id, memory types, `assigned_gpu_devices: tuple[GpuDevice, ...]`, `functions` | +| `FunctionRecord` | `identity`, group, position, memory types, `parameters`, `declared_files` | +| `FunctionIdentity` / `CustomFunctionIdentity` | key, module, name, `defined_in` and `imports` (`ModuleLibraries`); the custom one adds `source_sha256`. The text is not copied into the identity: `source_texts()` reads it from the registered `CustomFunctionSourceNamespace` that owns the hash, and the record stores it once under `custom_sources` | +| `RecordedParameter` | name, value (Python expression, `None` when no declaration states it), `source: type[ParameterSource]` | +| `ParameterSource` family | `WrittenInPipeline` (`pipeline`), `FunctionDefault` (`function_default`; the function and version are the identity of the enclosing `FunctionRecord`), `SuppliedAtRunTime` (`run_time`). Each says how its value is written (`value_expression`) | +| `ConfigValuesFromOwner` | `owner: ConfigValueOwner(source, scope_id, type_name)`, `values` by field path. `source` is `type[ConfigValueSource]`: `SetOnObject` (`set_here`) or `InheritedFrom` (`inherited`) | +| `DeclaredFile` | parameter, `declared_as`, path, `content: HashedFileContent \| UnhashedFileContent` (PolyStore's family) | +| `DatasetRunRecords` | the two keys of the metadata document: `run_records` by id, `produced_files: Mapping[str, StoredProducedFile(run_record, producer: StreamProducerIdentity)]` | +| `ProducedFile` | what the reader returns for one file: `record`, `producer`, `.step -> StepCalls` | + +**One codec.** Encode is `python_introspect.to_jsonable`, decode is +`python_introspect.dataclass_from_mapping` (`external/python-introspect/src/python_introspect/ +jsonable.py`, `dataclass_projection.py`); `RunRecord.to_json` / `from_json` and +`DatasetRunRecords.in_document` only call them. No `to_record` method is left in OpenHCS. A class +of the two source families is stored as its registry key (`AutoRegisterMeta`, the codec's existing +rule for `type[Family]`). The identity family and the file-content family carry data, and the codec +has no nominal tag for a family *instance*: they are declared as unions of dataclasses and the +decoder picks the member whose declared fields match exactly (it rejects undeclared fields and +raises on an ambiguous match). `test_stored_json_reads_back_as_the_same_record` covers every type. +**Open point for the owner:** that union decode is by field set, not by name. If that is not +acceptable, the fix belongs in python-introspect (a registered family instance stored as key plus +fields), not in a second codec here. PolyStore's `FileContent.to_record` / `from_record` +(`hashed: bool` key) is now unused by OpenHCS and can be dropped from #41 when it is rebased. + +**Reader and capability return the types.** `RunRecord.for_dataset`, `RunRecord.for_output_file`, +`ProducedFile.step`. `openhcs_read_run_record`: `RunRecordEntry(record_id, steps: tuple[StepCalls, +...], record: RunRecord | None)`; the request flag is `include_full_record` (was +`include_sources`); the digest-and-`shortened_fields` form is gone. Contract fixture regenerated +(only that tool changed). + +**Size (part of brief item 5, measured on the two-step unit pipeline in the shared venv, 246 +distributions):** default response `steps` 2,277 bytes; full record 47,633 bytes (software 5,252, +pipeline_code 12,000, global_config_code 7,453, configuration 9,846, steps 12,984). Grouping +resolved values by owner took the pipeline configuration from 22.9 kB to 9.8 kB and each step's +from 12.3 kB to 5.3 kB. Not re-measured on the Cellpose run. + +**Brief item 2: the two suite failures.** +- `test_pycodify_formatters.py::test_function_step_round_trip_omits_injected_values_but_keeps_runtime_settings[False-...]`: + the first agent had changed what `clean_mode=False` means (the hook was in + `FunctionStepFormatter.format`). The test states the stored-document promise (every stored + field, no parameter the pipeline does not state) and it holds again: the hook is removed, and the + two tests that had been weakened for it (`test_authoring_roundtrip_boundaries.py`, + `test_source_round_trip.py`) are restored to main's text. The record's copy is its own + rendering, `PipelineDocumentCodec.render_with_every_parameter` (completes each step's pattern + with `with_every_parameter_written`, then renders unabridged). + `test_saved_pipeline_text_stays_short_and_the_record_copy_is_complete` asserts all three. +- `test_mcp_dev_client_terminal_input.py::test_long_terminal_command_reuses_original_parser_and_client`: + not this branch. It pumps a 4 kB+ command through a PTY child with a 15 s deadline. Alone on the + branch at load average 13: 13.4 s and 8.5 s (passed), then 19.4 s and 17.3 s (one failed). With + the child importing `origin/main`'s package under the same load: 15.07 s and 15.01 s, both + failed, twice. Left as is. +- `test_mcp_server.py` stdio tests: 5 s `session.initialize()` timeout under `-n 4`; `2 passed` + alone. + +Tests run here: `tests/unit/test_run_record.py`, `test_run_record_capability.py`, +`tests/integration/test_run_record_round_trip.py`, `test_mcp_tool_contract.py`, +`test_pycodify_formatters.py`, `test_source_round_trip.py`, +`test_authoring_roundtrip_boundaries.py`: all pass. Full suite, guardrails, smoke and parity were +not run after these changes. + +### What remains, in order (numbers are the continuation brief's) + +3. **Shrink the two oversized classes; ratchet to no increase.** Not started. Expected to be + cleared by item 1 but unverified: `StringSubscript`, `AnyAnnotation`, `unmodeled_record_shape`. + Still present: `GodClassExcess PipelineCompiler +7` (the seven lines are the + `RunDeclaration.before_step_resolution` call, `with_resolved_steps`, and `run_declaration=` at + three bundle constructions), `OpenHCSMetadataTarget +24` (`saved_path_producers` and the + additions in `finalize_completed_dataset`), the boolean chains in + `StepExecutionObservation.is_empty` and `AtomicMetadataWriter.reconcile_completed_dataset`, + and two absence probes (`ModuleExportedCallableFormatter.can_format` in + `pycodify_formatters.py`; one in `virtual_workspace_metadata.py`). Notes from reading the + compiler: the declaration needs the submitted documents before `_resolve_pipeline_once` + replaces the steps in place, and the per-step provenance that only + `ResolvedPipelineDefinition` holds; nothing on the bundle or the plans carries that + provenance, so the orchestrator (the single caller, `orchestrator.py:739`) cannot build it + after the fact without the compiler handing something over. The two early returns in + `compile_pipelines` build the same empty bundle twice. `saved_path_producers` belongs with + `ProducedFiles` in `run_record.py`. Then merge `origin/main` and run `guardrails_local.sh`. +4. **Defects in what is recorded**, failing test first, none started: + (3) declared path left at its default: `CompiledFunctionInvocation.from_normalized_item` + (`function_patterns.py:~926`) replaces the kwargs with `resolve_declared_paths`' result, so the + invocation no longer knows which paths the pipeline wrote; it has to keep that (the stated + value and its source per declared path) so `_parameters` can say `function_default` and + `declared_files` holds the resolved path. (4) `enabled` has no parameters entry. (5) device on + a host run: `StepCalls.assigned_gpu_devices` is still an empty tuple; state it positively from + `FrameworkDeviceAssignment` and note the DL1 dependency. (6) enum defaults of custom functions + render with the mangled path. (7) libraries: direct imports only; add the requirements + closure, each list named. (8) plate-level outputs have no `produced_files` entry. (10) no + saved global config: `RunDeclaration.before_step_resolution` still falls back to + `GlobalPipelineConfig()`. (12) `changed_during_run`: still in the design text, not + implemented. +5. **Reader**: a public "is this a dataset root / which dataset contains this file" answer + through the dataset access owner; `DatasetRunRecords.of_dataset` and `_dataset_root_of` still + read the metadata file directly, and `RunRecordTarget.at` probes for the file, so the + capability's read of the document above a file is not checked by the agent path policy. Size: + re-measure on the Cellpose record. +6. **PolyStore 0.6.4**: not started. Rebase #41 onto PolyStore main, version 0.6.4, drop the + unused `to_record` / `from_record` if the owner agrees, suite in a fresh venv, merge, annotated + tag, publish. Then this branch's pin (see the top of this document). +7. **Acceptance re-run** (Cellpose on CPU, three BBBC039 fields), then delete `.claude/venv-cpu` + (2.4 GB) and `.claude/scratch/` (1.2 GB): both still present in the worktree. The acceptance + scripts under `.claude/scratch/acceptance/` read the old dict form and need the typed reader. +8. Gates (the landing agent's): guardrails `rc=0`, `verify_branch.sh`, smoke, parity. The PR is + still a draft; its body does not yet say "lands after DL1". +9. Outside this branch, unchanged: ROI zip metadata and `segmentation_summary.txt` embed absolute + source paths; zip timestamps. + +## Third hand-off (2026-10-12): steps A and B done, C to E not started + +Commits `18c4d56ae` (A) and `1a1c8c1eb` (B), on a merge of `origin/main` at `6845c3a36`. + +### A. Stored family members name their class + +**Where the tag is declared.** python-introspect 0.3.0 (PR OpenHCSDev/python-introspect#23, +merged, annotated tag `v0.3.0`, on PyPI through the manual script after the publish workflow sat +queued for 34 minutes). `python_introspect.jsonable`: + +- `MemberFamily.declared_by(annotation)` says whether a declared field type admits several + classes: a union with two or more dataclass members (`UnionMembers`), an abstract dataclass + (`ConcreteSubclasses`: its concrete subclasses, as loaded), or a class whose metaclass is + `AutoRegisterMeta` (`RegisteredMembers`: the registry). The three are subclasses; each answers + `members()`; the registered one reads a tag through `registered_class_from_key`. +- `class_tag(member)` is the one place a tag is derived: the registry key when the class is + registered (stable when the module or class is renamed), otherwise `__name__`. +- Stored form: `{tag: fields}` (one key; no reserved field name). `to_jsonable` now follows each + dataclass's declared field types through optional, `Annotated`, tuple/list/sequence/set and + mapping declarations; `dataclass_from_mapping` reads the class from the tag. +- Errors: unknown tag is `ValueError` naming the known members; a value outside the declared + family, a non-string registry key, or two members with one tag is `TypeError`; an untagged + object in a family field is refused, never matched by its fields. +- A type with its own `to_jsonable.register` projection keeps its whole stored form (P1 + Proposition 3.3 is about the guess; those types already write a nominal key of their own). +- Cost of reading declarations when encoding: a dataclass whose field types do not resolve at run + time is a `TypeError` naming it. One encoded class was affected and is fixed + (`UiLiveMeasurementEntryState.preview`, import moved out of `TYPE_CHECKING`). A live import of + about 2,545 first-party dataclasses found 92 whose annotations do not resolve; none of the others + was found at a `to_jsonable` call, but call sites that encode arbitrary values were not all + traced (`formbridge/bridge/object_state.py:257`, `core/debug_view_models.py:199`, + `core/materialization/core.py:3015`, `mcp/dev_client_rendering.py:375,397`, + `agent/services/artifact_plan_inspection_service.py:443,586`). The full suite is the check. + +**The record's two families** are registered and the fields are declared by the family: +`FunctionRecord.identity: FunctionIdentity` (`defined_by`: `installed_module`, `source_text`) and +`DeclaredFile.content: FileContent` (PolyStore, `content_kind`: `hashed`, `not_hashed`). +PolyStore's `FileContent.to_record` / `from_record`, the `hashed` flag and its hand-kept +`_kind_by_hashed` table are deleted on the #41 branch (certificate: **derived**, the registry and +the codec own both facts; nothing in OpenHCS called them). + +**Every user of the codec, checked** (live import plus `rg`; about 470 dataclasses reachable from +about 300 decode targets): + +| Kind | Where | Effect | +|---|---|---| +| Union of dataclasses decoded by fields | only the two record fields above | now registered families | +| Registered or abstract family with its own projection | `PositionedAxis.spec: AxisSpec` (`PayloadAxes.to_mapping`), `source_spatial_domain: SourceSpatialDomain` (`to_mapping`, key `spatial_dimensions`), `MeasurementTable.rows: ColumnarRows`, `SourceImageProvenance*` | unchanged: they write their own nominal key | +| Field typed with one registered class, decoded | `DebugSnapshotReadResponse.snapshot: DebugSnapshot` (zmq debug response), `ViewerWindowValidationSummaryResult.state: ViewerWindowStateResult \| None` (MCP result) | now `{registry key: fields}` | +| Registered family fields, encoded only | `SourceBindingsConfig.bindings` / `match_plan` (and the step and lazy variants), `GlobalPipelineConfig` / `PipelineConfig` streaming configs, `ObjectRelationship.declaration`, `Output.content`, CPA workspace panels and axes, `DebugProgressRecord.context` / `.snapshot`, `benchmark/openhcs_measured_run.py:101` | tagged in what they encode; `SourceBindingsConfig.declaration_identity()` hashes that JSON, so its value changes (whether it is persisted was not checked) | +| Concrete dataclass with dataclass subclasses as the field type | formbridge `UiActionIdentity`, `UiCodeDocumentIdentity`, `UiStateSurfaceIdentity`, `UiWindowIdentity`, `UiSemanticAddress`, `EndpointConnectionSpec`; `NapariDisplayConfig`; `ViewerPayloadControlOptions`; `GallerySourceEvidence` | **not a family, unchanged.** A subclass instance in such a field is written with its own fields and the decoder then refuses the undeclared ones. Open point: either the codec refuses a subclass instance when encoding, or these bases declare a family | + +**Stored formats that changed:** the `run_records` section of `openhcs_metadata.json` (new on this +branch, nothing released), MCP results and zmq debug responses that carry the fields in the third +and fourth rows (both ends are the same version), and the source-bindings declaration identity. +**Nothing on disk has to be regenerated:** no tracked JSON holds a run record (searched all +12,378 `*.json` / `*.jsonl` files), the only fixture decoded through the codec +(`tests/fixtures/mcp/start-owned-runtime-400.json`) has no family field, and MCP tools publish +`dict[str, Any]` as their result schema. Files under `paper/figures/slas/` hold UI summaries with +`identity` / `semantic_address` objects (the last row: unchanged form) and no code decodes them. +`paper/**`, `validation/**`, `benchmark/results/**` were not touched. + +**Dependants whose upper bound excludes python-introspect 0.3.0:** ObjectState 4.0.0 +(`>=0.2.0,<0.3`), formbridge 0.1.4 (`>=0.2.7,<0.3`), pyqt-reactive 0.8.0 (`>=0.2.6,<0.3`), +zmqruntime 0.6.4 (`>=0.2.5,<0.3`). pycodify and PolyStore do not depend on it; arraybridge has no +upper bound. None was released from here. + +Tests: python-introspect 191 pass on Python 3.10, 3.12 and 3.13 (round trip per member of each +kind of family, members with identical fields, containers and optional fields, unknown tag, +untagged data refused, own projection kept, unresolved field type). OpenHCS: the three run-record +test files and `test_mcp_tool_contract.py` pass; of the 2,951 tests in files that touch the codec, +the failures were a test-order leak (`tests/unit/test_axis_family_witness.py` leaves the +RemoteSensing axis family active for the files that follow it in the same xdist worker; with that +file left out the 40 affected files give 1,410 passed and 1 failed, +`test_installed_mcp_demo.py::test_portable_source_normalization_defers_catalog_and_execution_runtimes`, +which fails inside `RegistryService.metadata_for_canonical_key` in a subprocess and was not +compared with main). + +### B. The two oversized classes and the ratchet + +- `OpenHCSMetadataTarget` and `AtomicMetadataWriter` are byte-identical to main again + (`function_outputs.py` differs from main only in `finalize_function_step_outputs`, which asks + `ProducedFiles.stated_by_step`). Deleted: `OpenHCSMetadataTarget.saved_path_producers` (moved, + same body), the `run_record` / `produced_files` parameters, `ProducedFiles.write_into` + (certificate for all: **derived**, the same statements now run in `run_record.py`). +- **The record is stored by its owner in a second locked update.** `RunRecord.store_with_saved_files` + (called in the same `finally` as `finalize_completed_dataset`, so also after a failure) asks + `RunRecordWriter(AtomicMetadataWriter).store` for each dataset root the run saved a file under. + This replaces the design text's "one locked transaction": the dataset's `subdirectories` and the + record are now two consecutive transactions on the same document, each under its lock. A crash + between them leaves a dataset without its record. That is the price of not putting the record's + logic into the writer's transaction; if one transaction is required, the writer needs an + extension point, which adds lines to it. +- **The compiler still makes the two declaration calls.** The previous note holds: the documents + must be rendered before `_resolve_pipeline_once` replaces the steps, and only its result carries + the per-step provenance. What changed is that both empty-compilation returns go through + `CompiledExecutionBundle.with_nothing_to_run`, so `PipelineCompiler` is 3 lines smaller than on + main instead of 7 larger. +- `StepExecutionObservation.is_empty` is `not any(field values)`, derived from the dataclass + fields (all six were in the hand-written chain). The absence probe in + `virtual_workspace_metadata.py` went with the revert; the `ModuleExportedCallableFormatter` + probe is not counted by the ratchet and is unchanged. +- Two `StringSubscript` hits: `del parameters["func"]` (now a filter on the value being the + step's pattern) and a quoted annotation in `run_record.py`. + `CustomFunctionSourceNamespace.attribute_of_revision` names the namespace attribute once. + +Gate (the one this job runs), after merging `origin/main`: + +``` +ratchet openhcs exit 0 +ratchet scripts exit 0 +ratchet benchmark exit 0 +38 passed, 22 warnings in 39.28s +nra-r1 exit 0: "increased": [] +base 6845c3a363b1d24a6eeaf006f2efce3658e7804f head 1a1c8c1ebe8ae8021fe17f463b7471388add4e44 guardrails rc=0 +``` + +Tests run for B: the run-record files, `test_function_outputs.py`, +`test_orchestrator_lane_planning.py`, the four publication test files restored to main's text, +and 73 unit files that touch custom functions, observations, bundles or compilation: with the +witness file left out, 913 passed, 1 failed +(`test_mcp_server.py::test_mcp_stdio_server_roundtrip_returns_errors_as_payloads`, the 5 s stdio +timeout under xdist the second hand-off describes). Full suite, smoke and parity were not run. + +### What remains, in order + +C. **Defects in what is recorded**, failing test first, none started. The second hand-off's + item 4 is still the exact list: (3) declared path left at its default, (4) `enabled`, + (5) device on a host run (depends on DL1), (6) enum defaults of custom functions, (7) direct + imports and the requirements closure, (8) plate-level outputs, (10) no saved global config + (`RunDeclaration.before_step_resolution` still falls back to `GlobalPipelineConfig()`), + (12) `changed_during_run`. +D. **PolyStore 0.6.4.** Clone at `.claude/ps-clone` (branch `feat/file-content-identity`, pushed + at `5301d35`: #41 plus the registered family). Still to do: rebase onto PolyStore main + (`5fdb617`), version 0.6.4 with `uv.lock`, retitle the PR, suite in a fresh venv, merge, + annotated tag, publish. The pre-rebase commit `5301d35` is what the submodule needs until DL1, + so keep it reachable (push it under another branch name before the rebase is force-pushed). +E. **Reader** (second hand-off item 5: dataset-root answer through the dataset access owner; the + capability's metadata read through `openhcs/agent/path_policy.py`), then the **acceptance + re-run** with the scripts in `.claude/scratch/acceptance/` moved to the typed reader, the + short-response size on the Cellpose record, then delete `.claude/venv-cpu` and + `.claude/scratch/` (both still present, with `.claude/pi-clone` and `.claude/ps-clone`). +F. Decide the two open points above: one transaction or two for the record; concrete bases with + subclasses in codec fields. +G. Gates (the landing agent's): `verify_branch.sh`, smoke, parity. The PR stays a draft. +H. Outside this branch, unchanged: ROI zip metadata and `segmentation_summary.txt` embed absolute + source paths; zip timestamps; `test_axis_family_witness.py` leaks the active axis family. diff --git a/external/PolyStore b/external/PolyStore index d9aed37873..5301d35b4f 160000 --- a/external/PolyStore +++ b/external/PolyStore @@ -1 +1 @@ -Subproject commit d9aed378736bee67cbcba75cb66c0a25074b876a +Subproject commit 5301d35b4fe6037a46f4186f7d33537e746294bb diff --git a/external/python-introspect b/external/python-introspect index a88fe56f78..a9428c3b5a 160000 --- a/external/python-introspect +++ b/external/python-introspect @@ -1 +1 @@ -Subproject commit a88fe56f78b523ad5c791e3f868bb35d7d6b3f12 +Subproject commit a9428c3b5a5f7c4091049ac977dff8fd48f76b2e diff --git a/openhcs/agent/capabilities.py b/openhcs/agent/capabilities.py index ebdfe00409..84ccf1099d 100644 --- a/openhcs/agent/capabilities.py +++ b/openhcs/agent/capabilities.py @@ -143,6 +143,8 @@ from openhcs.agent.dto.plate import ( PlateFileQueryRequest, PlateFileQueryResult, + RunRecordReadRequest, + RunRecordReadResult, PlateFileStreamRequest, PlateFileStreamResult, PlateImageSampleRequest, @@ -2352,6 +2354,46 @@ class QueryPlateFilesCapability(MainThreadProgressCapability, PlatePathCapabilit ) +class ReadRunRecordCapability(PlatePathCapability): + name = "openhcs_read_run_record" + cli_command = "read-run-record" + title = "Read run record" + description = ( + "Read-only: what produced an output dataset. Pass an output dataset root " + "(the directory holding openhcs_metadata.json) to get every run record " + "by record id, or one output file inside a dataset to get the producer " + "identity stored for that file, the producing step and the record of " + "that run. A file the dataset does not list is an error, never a guess " + "from its name. Each record lists what every step ran: each function " + "with its identity (registry key, defining and imported library " + "versions, sha256 of a custom function's source), every parameter with " + "its value and source (pipeline, function_default, run_time), declared " + "files with content hashes, and devices. Pass include_full_record=true " + "to add the complete stored record: pipeline and global config " + "documents, resolved configuration with the owner of every value, " + "custom source texts and all installed distributions." + ) + exposition = PlatePathCapability.exposition.refine( + workflow_stage=CapabilityWorkflowStage.INSPECTION, + visibility=CapabilityVisibility.STANDARD, + ) + service = "run_record" + data_exposure = ( + "local_output_path", + "pipeline_source", + "resolved_configuration", + "custom_function_source", + "installed_distribution_versions", + ) + security_requirements = ("AgentPathPolicy readable root",) + input_contract = RunRecordReadRequest + output_contract = RunRecordReadResult + invocation = AgentDataclassRequestServiceInvocation( + service=lambda context: context.run_record_service, + method=lambda service, request: service.read(request), + ) + + class SamplePlateImageCapability(MainThreadProgressCapability, PlatePathCapability): name = "openhcs_sample_plate_image" cli_command = "sample-plate-image" diff --git a/openhcs/agent/dto/__init__.py b/openhcs/agent/dto/__init__.py index d0033d795e..c7b45a3354 100644 --- a/openhcs/agent/dto/__init__.py +++ b/openhcs/agent/dto/__init__.py @@ -130,6 +130,10 @@ "PlatePathInspectionRequest", "PlatePathInspectionResult", "PlateWorkspacePreparationOperation", + "ProducedFileEntry", + "RunRecordEntry", + "RunRecordReadRequest", + "RunRecordReadResult", "SelectedPlateFileQueryTarget", "SelectedPlateFileQueryRequest", "SelectedPlateFileQueryResult", diff --git a/openhcs/agent/dto/plate.py b/openhcs/agent/dto/plate.py index 9079b42e4b..8b5aa67f98 100644 --- a/openhcs/agent/dto/plate.py +++ b/openhcs/agent/dto/plate.py @@ -9,10 +9,13 @@ from typing import ClassVar from metaclass_registry import AutoRegisterMeta +from polystore.streaming.identity import StreamProducerIdentity from zmqruntime.config import TransportMode from formbridge.contracts import ( AGENT_PARAMETER_DESCRIPTION_METADATA_KEY, + AgentCliArgumentSpec, + AgentDataclassCliRequest, AgentError, AgentResultEnvelope, AgentResourceRef, @@ -23,6 +26,7 @@ from openhcs.core.component_filters import ComponentFilters from openhcs.core.config import NapariDisplayConfig from openhcs.core.dataset_sources.choice import AutoDetectedSource, DatasetSourceChoice +from openhcs.core.run_record import RunRecord, StepCalls from openhcs.core.plate_file_inventory import ( PlateFileInventoryQuery, PlateFileKind, @@ -1339,3 +1343,71 @@ class SelectedPlateFileStreamResult(AgentResultEnvelope): selected_plate: JsonObject = field(default_factory=dict) target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot stream: PlateFileStreamResult | None = None + + +@dataclass(frozen=True, slots=True) +class RunRecordReadRequest(AgentDataclassCliRequest): + """Read what produced an output dataset, or one output file inside one. + + ``path`` is an output dataset root (the directory that holds the OpenHCS + metadata document) or one output file below such a root. + ``include_full_record`` adds the complete stored record to each entry: the + pipeline and global config documents, the resolved configuration with the + owner of every value, custom function source texts and every installed + distribution. + """ + + path: str + include_full_record: bool = False + + @classmethod + def agent_cli_argument_specs(cls) -> tuple[AgentCliArgumentSpec, ...]: + return ( + AgentCliArgumentSpec( + field_name="path", + positional=True, + help="Output dataset root, or one output file inside one.", + ), + *super(RunRecordReadRequest, cls).agent_cli_argument_specs(), + ) + + +@dataclass(frozen=True, slots=True) +class RunRecordEntry: + """One stored run record, addressed by the sha256 of its stored JSON. + + ``steps`` says what each step ran: every function with its identity (a + custom function by the sha256 of its source), every parameter with its + value and source, and every declared file with its content hash. + ``record`` is the complete record, present when the request asked for it. + """ + + record_id: str + steps: tuple[StepCalls, ...] + record: RunRecord | None = None + + +@dataclass(frozen=True, slots=True) +class ProducedFileEntry: + """One output file with the producer identity stored for it and its producing step.""" + + path: str + record_id: str + producer: StreamProducerIdentity + step: StepCalls + + +@dataclass(frozen=True, kw_only=True, slots=True) +class RunRecordReadResult(AgentResultEnvelope): + """Run records read from one output dataset. + + For a dataset root ``records`` holds every record and ``produced_files`` is + empty; for an output file ``produced_files`` holds that file and + ``records`` holds the one record of the run that wrote it. + """ + + path: str + records: tuple[RunRecordEntry, ...] = () + produced_files: tuple[ProducedFileEntry, ...] = () + errors: tuple[AgentError, ...] = () + warnings: tuple[AgentWarning, ...] = () diff --git a/openhcs/agent/dto/ui_bridge.py b/openhcs/agent/dto/ui_bridge.py index 75dcb9eddb..2badb184c6 100644 --- a/openhcs/agent/dto/ui_bridge.py +++ b/openhcs/agent/dto/ui_bridge.py @@ -4,7 +4,6 @@ from dataclasses import dataclass, field from enum import Enum -from typing import TYPE_CHECKING from python_introspect import ( JsonObject, ) @@ -35,10 +34,7 @@ UiMutationRequestTokenCarrier, UiStateSurfaceEnvelope, ) -if TYPE_CHECKING: - from openhcs.core.progress.live_measurements import ( - LiveMeasurementTablePreview, - ) +from openhcs.core.progress.live_measurements import LiveMeasurementTablePreview def _dataset_workflow_enum() -> type[Enum]: diff --git a/openhcs/agent/services/run_record_service.py b/openhcs/agent/services/run_record_service.py new file mode 100644 index 0000000000..c170d0a7ad --- /dev/null +++ b/openhcs/agent/services/run_record_service.py @@ -0,0 +1,100 @@ +"""Read the record of what produced an output dataset, for agents.""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from dataclasses import dataclass +from pathlib import Path + +from openhcs.agent.dto.common import SCHEMA_VERSION +from openhcs.agent.dto.plate import ( + ProducedFileEntry, + RunRecordEntry, + RunRecordReadRequest, + RunRecordReadResult, +) +from openhcs.agent.exceptions import AgentFacingErrorMixin +from openhcs.agent.path_policy import AgentPathPolicy +from openhcs.core.run_record import RunRecord, UnrecordedOutputFile +from openhcs.core.virtual_workspace_metadata import get_metadata_path + + +class OutputFileNotRecorded(AgentFacingErrorMixin, LookupError): + """The dataset's metadata does not say which step produced the requested file.""" + + agent_error_code = "run_record_output_file_unrecorded" + agent_error_hint = ( + "Pass the output dataset root to list its run records, or a file the " + "dataset's metadata lists under produced_files." + ) + + +def _entry(record: RunRecord, request: RunRecordReadRequest) -> RunRecordEntry: + """What each step ran, with the complete stored record when the request asks for it.""" + return RunRecordEntry( + record_id=record.record_id, + steps=tuple(step.calls for step in record.steps), + record=record if request.include_full_record else None, + ) + + +@dataclass(frozen=True, slots=True) +class RunRecordTarget(ABC): + """A readable path a run record is asked for: a dataset root or a file inside one.""" + + path: Path + + @classmethod + def at(cls, path: Path) -> "RunRecordTarget": + """A directory holding the metadata document is a dataset root; anything else is a file.""" + if Path(get_metadata_path(path)).is_file(): + return DatasetRootTarget(path) + return OutputFileTarget(path) + + @abstractmethod + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: + """Read this target with the detail the request asks for.""" + + +class DatasetRootTarget(RunRecordTarget): + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: + return RunRecordReadResult( + schema_version=SCHEMA_VERSION, + path=str(self.path), + records=tuple( + _entry(record, request) + for record in RunRecord.for_dataset(self.path).values() + ), + ) + + +class OutputFileTarget(RunRecordTarget): + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: + try: + produced = RunRecord.for_output_file(self.path) + except UnrecordedOutputFile as error: + raise OutputFileNotRecorded(str(error), path=str(self.path)) from error + return RunRecordReadResult( + schema_version=SCHEMA_VERSION, + path=str(self.path), + records=(_entry(produced.record, request),), + produced_files=( + ProducedFileEntry( + path=str(self.path), + record_id=produced.record.record_id, + producer=produced.producer, + step=produced.step, + ), + ), + ) + + +class RunRecordService: + """Wrap ``RunRecord``'s readers behind the agent path policy.""" + + def __init__(self, path_policy: AgentPathPolicy) -> None: + self._path_policy = path_policy + + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: + target = RunRecordTarget.at(self._path_policy.assert_readable(request.path)) + return target.read(request) diff --git a/openhcs/core/compiled_execution.py b/openhcs/core/compiled_execution.py index 940c40a4bc..c69d560a00 100644 --- a/openhcs/core/compiled_execution.py +++ b/openhcs/core/compiled_execution.py @@ -10,6 +10,7 @@ ) from openhcs.core.config import GlobalPipelineConfig, MultiprocessingStartMethod from openhcs.core.context.processing_context import CompiledContext +from openhcs.core.run_record import RunDeclaration from openhcs.core.steps.abstract import AbstractStep from openhcs.core.worker_start_policy import ( WorkerStartDecision, @@ -132,6 +133,8 @@ class CompiledExecutionBundle: transport_contexts: Mapping[str, CompiledContext] worker_assignments: Mapping[str, list[str]] runtime_environment: CompiledRuntimeEnvironmentPlan + run_declaration: RunDeclaration + """The documents and resolved configuration this bundle was compiled from.""" @property def axis_ids(self) -> tuple[str, ...]: @@ -156,6 +159,29 @@ def requires_parent_runtime_observation(self) -> bool: for plan in context.step_plans.values() ) + @classmethod + def with_nothing_to_run( + cls, + pipeline_definition: Sequence[AbstractStep], + effective_config: GlobalPipelineConfig, + *, + server_mode: bool, + run_declaration: RunDeclaration, + ) -> "CompiledExecutionBundle": + """The bundle of a compilation that selected no axis value or no enabled step.""" + + return cls.from_runtime_contexts( + pipeline_definition=pipeline_definition, + runtime_contexts={}, + worker_assignments={}, + runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( + effective_config, + compiled_contexts={}, + server_mode=server_mode, + ), + run_declaration=run_declaration, + ) + @classmethod def from_runtime_contexts( cls, @@ -164,6 +190,7 @@ def from_runtime_contexts( runtime_contexts: Mapping[str, CompiledContext], worker_assignments: Mapping[str, list[str]], runtime_environment: CompiledRuntimeEnvironmentPlan, + run_declaration: RunDeclaration, ) -> "CompiledExecutionBundle": from openhcs.core.function_step_transport import FunctionStepTransportAuthority @@ -178,6 +205,7 @@ def from_runtime_contexts( key: list(value) for key, value in worker_assignments.items() }, runtime_environment=runtime_environment, + run_declaration=run_declaration, ) @classmethod @@ -187,6 +215,7 @@ def from_unassigned_runtime_contexts( pipeline_definition: Sequence[AbstractStep], runtime_contexts: Mapping[str, CompiledContext], runtime_environment: CompiledRuntimeEnvironmentPlan, + run_declaration: RunDeclaration, ) -> "CompiledExecutionBundle": """Create a bundle before worker ownership has been assigned.""" @@ -195,6 +224,7 @@ def from_unassigned_runtime_contexts( runtime_contexts=runtime_contexts, worker_assignments={}, runtime_environment=runtime_environment, + run_declaration=run_declaration, ) def for_transport_serialization( diff --git a/openhcs/core/function_identity.py b/openhcs/core/function_identity.py new file mode 100644 index 0000000000..abbb5d2ab6 --- /dev/null +++ b/openhcs/core/function_identity.py @@ -0,0 +1,178 @@ +"""What a processing function is, stated as content: where it comes from and at which version. + +A function's name can stay while its body, its library or its defaults change. +The identity here is what a record of a run stores so that a later reader can +tell which code answered. +""" + +from __future__ import annotations + +import ast +import importlib.metadata +import importlib.util +import sys +from collections.abc import Mapping +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path +from types import MappingProxyType +from typing import ClassVar + +from metaclass_registry import AutoRegisterMeta + +from openhcs.runtime.environment_provenance import InstalledDistributionVersion + + +@lru_cache(maxsize=1) +def _distributions_by_top_level_module() -> dict[str, tuple[str, ...]]: + return { + module: tuple(sorted(set(distributions))) + for module, distributions in importlib.metadata.packages_distributions().items() + } + + +@dataclass(frozen=True, slots=True) +class ModuleLibraries: + """The distributions behind a set of imported modules. + + A module that belongs to no installed distribution and is not part of the + standard library is listed by name, never dropped: the reader must be able + to see that its version is unknown. + """ + + libraries: tuple[InstalledDistributionVersion, ...] + modules_without_distribution: tuple[str, ...] + + @classmethod + def of_modules(cls, module_names: frozenset[str]) -> "ModuleLibraries": + by_module = _distributions_by_top_level_module() + libraries: set[InstalledDistributionVersion] = set() + unknown: set[str] = set() + for top_level in {name.partition(".")[0] for name in module_names}: + if top_level in sys.stdlib_module_names: + continue + distributions = by_module.get(top_level, ()) + if not distributions: + unknown.add(top_level) + libraries.update( + InstalledDistributionVersion( + distribution, importlib.metadata.version(distribution) + ) + for distribution in distributions + ) + return cls( + tuple(sorted(libraries, key=lambda item: (item.name.casefold(), item.version))), + tuple(sorted(unknown)), + ) + + @classmethod + def imported_by_source(cls, source_text: str) -> "ModuleLibraries": + """Every absolute import in the source, wherever it is written.""" + imported: set[str] = set() + for node in ast.walk(ast.parse(source_text)): + if isinstance(node, ast.Import): + imported.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + imported.add(node.module) + return cls.of_modules(frozenset(imported)) + + @classmethod + def imported_by_module(cls, module_name: str) -> "ModuleLibraries": + """The imports of an importable module, read from its source file.""" + spec = importlib.util.find_spec(module_name) + origin = None if spec is None else spec.origin + if origin is None or not origin.endswith(".py"): + return cls.of_modules(frozenset()) + return cls.imported_by_source(Path(origin).read_text(encoding="utf-8")) + + +@dataclass(frozen=True, slots=True) +class FunctionIdentity(metaclass=AutoRegisterMeta): + """A function exported by an installed module. + + The identities are registered under ``defined_by``; a stored identity names + its class by that key. + + ``key`` is the registry key (``registry:name``) or the import path + (``python:module:name``); ``defined_in`` is the distribution and version of + the module that defines it; ``imports`` are the distributions that module + imports. + """ + + __registry_key__ = "defined_by" + defined_by: ClassVar[str] = "installed_module" + + key: str + module: str + name: str + defined_in: ModuleLibraries + imports: ModuleLibraries + + @classmethod + def of_module_export(cls, *, key: str, module: str, name: str) -> "FunctionIdentity": + return cls( + key=key, + module=module, + name=name, + defined_in=ModuleLibraries.of_modules(frozenset({module})), + imports=ModuleLibraries.imported_by_module(module), + ) + + def recorded_source_hashes(self) -> tuple[str, ...]: + """The sha256 of each source text this identity refers to; none for an installed module.""" + return () + + def source_texts(self) -> Mapping[str, str]: + """Source texts this identity refers to, by sha256; none for an installed module.""" + return MappingProxyType({}) + + +@dataclass(frozen=True, slots=True) +class CustomFunctionIdentity(FunctionIdentity): + """A function defined by user-written source text, identified by that text's hash.""" + + defined_by: ClassVar[str] = "source_text" + + source_sha256: str + + @classmethod + def of_source( + cls, *, key: str, module: str, name: str, source_sha256: str, source_text: str + ) -> "CustomFunctionIdentity": + return cls( + key=key, + module=module, + name=name, + defined_in=ModuleLibraries.of_modules(frozenset()), + imports=ModuleLibraries.imported_by_source(source_text), + source_sha256=source_sha256, + ) + + def recorded_source_hashes(self) -> tuple[str, ...]: + return (self.source_sha256,) + + def source_texts(self) -> Mapping[str, str]: + """The text, from the registered source namespace that owns this hash.""" + from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, + ) + from openhcs.processing.custom_functions.source_namespace import ( + CustomFunctionSource, + CustomFunctionSourceNamespace, + ) + + declaration = CustomFunctionRuntimeRegistry.declaration_for_source( + CustomFunctionSource(self.name, self.source_sha256) + ) + if declaration is None: + raise LookupError( + f"Custom function {self.name!r} is no longer registered with source " + f"{self.source_sha256}; its text cannot be recorded." + ) + return MappingProxyType( + { + self.source_sha256: CustomFunctionSourceNamespace.bound_to( + declaration.metadata.func + ).code + } + ) diff --git a/openhcs/core/function_reference.py b/openhcs/core/function_reference.py index 3f08f3a0d2..0bad03ba9f 100644 --- a/openhcs/core/function_reference.py +++ b/openhcs/core/function_reference.py @@ -21,6 +21,7 @@ FunctionStepExecutionScope, ) from openhcs.core.function_contract_metadata import FunctionContractAttribute +from openhcs.core.function_identity import FunctionIdentity if TYPE_CHECKING: from openhcs.core.steps.abstract import AbstractStep @@ -86,6 +87,10 @@ def source_fragment(self, context: FormatContext) -> SourceFragment: def resolve(self) -> Callable: """Resolve this reference through its nominal transport authority.""" + @abstractmethod + def identity(self) -> FunctionIdentity: + """Where the referenced function comes from and at which version.""" + def require_current_declaration(self, resolved: Callable) -> Callable: """Return ``resolved`` only when its source proof matches compilation.""" @@ -122,6 +127,13 @@ def resolve(self) -> Callable: "is not importable in this process." ) + def identity(self) -> FunctionIdentity: + return FunctionIdentity.of_module_export( + key=self.composite_key, + module=self.original_module, + name=self.function_name, + ) + @dataclass(frozen=True, kw_only=True) class RegistryFunctionReference(FunctionReference): @@ -161,6 +173,14 @@ def resolve(self) -> Callable: return RegistryService.resolve_function_reference(self) + def identity(self) -> FunctionIdentity: + """The registry entry answers: a custom entry is identified by its source.""" + from openhcs.processing.backends.lib_registry.registry_service import ( + RegistryService, + ) + + return RegistryService.metadata_for_canonical_key(self.composite_key).identity() + @dataclass(frozen=True, kw_only=True) class ModuleExportRegistryFunctionReference( diff --git a/openhcs/core/orchestrator/bundle_execution.py b/openhcs/core/orchestrator/bundle_execution.py index 03341f3dc5..e95ae5c712 100644 --- a/openhcs/core/orchestrator/bundle_execution.py +++ b/openhcs/core/orchestrator/bundle_execution.py @@ -84,6 +84,7 @@ from openhcs.core.steps.function_artifact_materialization import ( ArtifactMaterializationTargetPlan, ) +from openhcs.core.run_record import ProducedFiles, RunRecord from openhcs.core.steps.function_outputs import ( OpenHCSMetadataTarget, ) @@ -251,6 +252,13 @@ def execute_bundle_request( f"with max_workers={validated.actual_max_workers}." ) + # Before any worker exists: each declared input file is hashed once, + # and the hash is of the bytes this run reads. + run_record = RunRecord.at_run_start( + request.execution_bundle, + filemanager=orchestrator.filemanager, + backend=Backend.DISK.value, + ) executor_resources = WorkerExecutorFactory( log_file_base=request.log_file_base, progress_queue=validated.progress_queue, @@ -341,6 +349,7 @@ def execute_bundle_request( ), ), ) + run_record.store_with_saved_files(executing_contexts) if execution_results.is_success(): viewer_states_by_port = settle_viewer_state( visualizers, @@ -644,6 +653,13 @@ def execute_dataset_scoped_steps( StepExecutionObservation( materialized_locations_by_address=MappingProxyType({}), source_projection_entries_by_target=projection_entries, + producer_by_saved_path=( + ProducedFiles.stated_by_step( + owner_context, + owner_plan.require_function_execution_ready(), + artifact_materializations=materializations, + ) + ), ), ) ) diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index 30bc6f421c..ed36ed6619 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -56,6 +56,7 @@ StepAxisFilterSet, ) from openhcs.core.pipeline.funcstep_contract_validator import FuncStepContractValidator +from openhcs.core.run_record import RunDeclaration from openhcs.core.pipeline.compilation_session import ( CompilationPathResolver, CompilationDatasetScope, @@ -1534,17 +1535,16 @@ def compile_pipelines( axis_filter, effective_config, ) + run_declaration = RunDeclaration.before_step_resolution( + orchestrator, pipeline_definition + ) if not axis_values_to_process: logger.warning("No axis values found to process based on filter.") - return CompiledExecutionBundle.from_runtime_contexts( - pipeline_definition=pipeline_definition, - runtime_contexts={}, - worker_assignments={}, - runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( - effective_config, - compiled_contexts={}, - server_mode=is_zmq_execution, - ), + return CompiledExecutionBundle.with_nothing_to_run( + pipeline_definition, + effective_config, + server_mode=is_zmq_execution, + run_declaration=run_declaration, ) logger.info( @@ -1554,19 +1554,16 @@ def compile_pipelines( pipeline_inputs = PipelineCompiler._resolve_pipeline_once( orchestrator, pipeline_definition ) + run_declaration = run_declaration.with_resolved_steps(pipeline_inputs) if not pipeline_definition: logger.warning( "All steps were disabled. Pipeline is empty after filtering." ) - return CompiledExecutionBundle.from_runtime_contexts( - pipeline_definition=pipeline_definition, - runtime_contexts={}, - worker_assignments={}, - runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( - effective_config, - compiled_contexts={}, - server_mode=is_zmq_execution, - ), + return CompiledExecutionBundle.with_nothing_to_run( + pipeline_definition, + effective_config, + server_mode=is_zmq_execution, + run_declaration=run_declaration, ) path_resolver = CompilationPathResolver( dataset_scope=CompilationDatasetScope.from_path(orchestrator.dataset_root), @@ -1654,6 +1651,7 @@ def compile_pipelines( runtime_contexts=compiled_contexts, worker_assignments=worker_assignments, runtime_environment=runtime_environment, + run_declaration=run_declaration, ) PipelineCompiler._write_compilation_debug_bundle_if_configured( effective_config, diff --git a/openhcs/core/pipeline_document.py b/openhcs/core/pipeline_document.py index 8abde39d74..5b1999741b 100644 --- a/openhcs/core/pipeline_document.py +++ b/openhcs/core/pipeline_document.py @@ -57,6 +57,31 @@ def from_values( ), ) + @classmethod + def render_with_every_parameter(cls, document: PipelineDocument) -> str: + """Render the copy a run's record keeps: nothing is left to a default. + + The saved pipeline (``render``) leaves out what equals a default, so it + means whatever the installed defaults are when it is loaded. This text + writes every stored field and, for every function, every parameter it + takes a value for, so it means the same after a default changes. + """ + + from openhcs.serialization.pycodify_formatters import ( + with_every_parameter_written, + ) + + return cls.render( + PipelineDocument( + pipeline_config=document.pipeline_config, + pipeline_steps=[ + step.with_function_spec(with_every_parameter_written(step.func)) + for step in document.pipeline_steps + ], + ), + clean_mode=False, + ) + @classmethod def from_namespace( cls, diff --git a/openhcs/core/run_record.py b/openhcs/core/run_record.py new file mode 100644 index 0000000000..9dc21b29e9 --- /dev/null +++ b/openhcs/core/run_record.py @@ -0,0 +1,853 @@ +"""The record of what produced an output dataset. + +Three stages, each with one owner: + +* ``RunDeclaration``: what only the compiler holds (the stored documents and + the resolved configuration with the source of every value). It travels on + the ``CompiledExecutionBundle``. +* ``RunRecord.at_run_start``: the parent process combines the declaration with + the compiled step plans and asks the storage layer what each declared input + file holds. +* ``ProducedFiles`` and ``RunRecord`` are written into the output dataset's + ``openhcs_metadata.json`` by its one writer and read back from there. + +Two read kinds are kept apart on purpose. ``pipeline_code`` and +``global_config_code`` are stored documents: executing them rebuilds the +run. ``configuration``, ``steps[].configuration`` and ``parameters`` are the +resolved values, each with the declaration that supplied it. + +The stored form is the JSON ``python_introspect.to_jsonable`` writes for these +dataclasses and ``python_introspect.dataclass_from_mapping`` reads back; there +is no other encoder or decoder. A class of a registered family (a parameter's +or a configuration value's source) is stored as its registry key. A member of +a registered family that carries data (a function identity, a file's content) +is stored as ``{registry key: fields}``, so the stored form names its class. +""" + +from __future__ import annotations + +import hashlib +import inspect +import json +from abc import ABC, abstractmethod +from collections.abc import Iterable, Iterator, Mapping, Sequence +from dataclasses import dataclass, field, fields, is_dataclass, replace +from pathlib import Path +from types import MappingProxyType +from typing import TYPE_CHECKING, ClassVar + +import pycodify +from metaclass_registry import AutoRegisterMeta +from polystore.file_content import FileContent +from polystore.streaming.identity import StreamProducerIdentity +from python_introspect import JsonObject, JsonValue, dataclass_from_mapping, to_jsonable + +from openhcs.core.function_identity import FunctionIdentity + +from openhcs.core.steps.abstract import StepExecutionObservation +from openhcs.core.vfs_protocol import FileManagerLike +from openhcs.core.virtual_workspace_metadata import METADATA_CONFIG, AtomicMetadataWriter + +if TYPE_CHECKING: + from openhcs.core.context.processing_context import ExecutingContext + from openhcs.core.steps.function_artifact_materialization import ( + MaterializedRuntimeArtifact, + ) + from openhcs.core.compiled_execution import CompiledExecutionBundle + from openhcs.core.compiled_step_plan import CompiledStepPlan + from openhcs.core.config import GlobalPipelineConfig + from openhcs.core.function_patterns import CompiledFunctionInvocation + from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator + from openhcs.core.steps.abstract import AbstractStep + from openhcs.core.pipeline.compilation_session import ResolvedPipelineDefinition + from pycodify.documents import PipelineDocument + + + +def python_expression(value: object) -> str: + """Write a value the way generated pipeline code writes it.""" + import openhcs.serialization.pycodify_formatters # noqa: F401 (registers formatters) + + return pycodify.to_source(value, pycodify.FormatContext(clean_mode=False)).code + + +class ParameterSource(ABC, metaclass=AutoRegisterMeta): + """Which declaration supplied the value a function parameter ran with. + + Each source is a subclass; ``name`` is its spelling in the stored record. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + + @classmethod + @abstractmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + """The value as a Python expression, or None when no declaration states it.""" + + +class WrittenInPipeline(ParameterSource): + """The pipeline document states the value.""" + + name = "pipeline" + + @classmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + return python_expression(written[parameter.name]) + + +class FunctionDefault(ParameterSource): + """The pipeline leaves the parameter out; the signature of the recorded function supplies it. + + The function and its version are the identity of the function record the + parameter belongs to. + """ + + name = "function_default" + + @classmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + return python_expression(parameter.default) + + +class SuppliedAtRunTime(ParameterSource): + """The platform passes the value when it calls the function; no declaration states it.""" + + name = "run_time" + + @classmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + return None + + +@dataclass(frozen=True, slots=True) +class RecordedParameter: + """One parameter of a function, the value it ran with and where the value came from.""" + + name: str + value: str | None + source: type[ParameterSource] + + +class ConfigValueSource(ABC, metaclass=AutoRegisterMeta): + """How a configured object came by a value: it holds it, or ObjectState resolved it.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + + +class SetOnObject(ConfigValueSource): + """The configured object itself holds the value.""" + + name = "set_here" + + +class InheritedFrom(ConfigValueSource): + """ObjectState resolved the value from another scope or class.""" + + name = "inherited" + + +def _type_name(owner: type | None) -> str | None: + return None if owner is None else f"{owner.__module__}.{owner.__qualname__}" + + +@dataclass(frozen=True, slots=True) +class ConfigValueOwner: + """The scope and class that supplied configuration values, and how.""" + + source: type[ConfigValueSource] + scope_id: str | None + type_name: str | None + + +@dataclass(frozen=True, slots=True) +class ConfigValuesFromOwner: + """The resolved values one owner supplied, by field path, as Python expressions.""" + + owner: ConfigValueOwner + values: Mapping[str, str] + + +ObjectStateProvenance = Mapping[str, tuple[str | None, type | None]] + + +def _field_values(configured: object) -> dict[str, object]: + return {item.name: getattr(configured, item.name) for item in fields(configured)} + + +def _values_with_owner( + values: Mapping[str, object], + provenance: ObjectStateProvenance, + *, + holder: ConfigValueOwner, + prefix: str = "", +) -> Iterator[tuple[ConfigValueOwner, str, str]]: + for name, value in values.items(): + path = f"{prefix}{name}" + if path in provenance: + scope_id, source_type = provenance[path] + yield ( + ConfigValueOwner(InheritedFrom, scope_id, _type_name(source_type)), + path, + python_expression(value), + ) + elif is_dataclass(value) and not isinstance(value, type): + yield from _values_with_owner( + _field_values(value), + provenance, + holder=holder, + prefix=f"{path}.", + ) + else: + yield holder, path, python_expression(value) + + +def resolved_values( + values: Mapping[str, object], + provenance: ObjectStateProvenance, + *, + scope_id: str | None, + holder_type: type, +) -> tuple[ConfigValuesFromOwner, ...]: + """Every leaf value of a resolved object, grouped by the owner that supplied it. + + ObjectState lists the paths it inherited; a path it does not list is held + by the object itself. A nested config the object leaves unset is inherited + whole and listed as one path, so the inherited check comes before descent. + Each owner is written once, with all the values it supplied. + """ + by_owner: dict[ConfigValueOwner, dict[str, str]] = {} + for owner, path, expression in _values_with_owner( + values, + provenance, + holder=ConfigValueOwner(SetOnObject, scope_id, _type_name(holder_type)), + ): + by_owner.setdefault(owner, {})[path] = expression + return tuple( + ConfigValuesFromOwner(owner, MappingProxyType(owned)) + for owner, owned in by_owner.items() + ) + + +@dataclass(frozen=True, slots=True) +class RunDeclaration: + """What the compiler alone knows about a run: documents and resolved configuration.""" + + pipeline_code: str + global_config_code: str + configuration: tuple[ConfigValuesFromOwner, ...] + step_configurations: tuple[tuple[ConfigValuesFromOwner, ...], ...] = () + """Resolved values of each enabled step, by compiled step index.""" + + @classmethod + def before_step_resolution( + cls, + orchestrator: "PipelineOrchestrator", + pipeline_definition: Sequence[AbstractStep], + ) -> "RunDeclaration": + """Render the documents as submitted, before the compiler resolves the steps in place.""" + from objectstate import ObjectState, ObjectStateRegistry + from objectstate.global_config import get_saved_global_config + + from openhcs.core.config import GlobalPipelineConfig + from openhcs.core.config_document import ConfigDocumentCodec + from openhcs.core.pipeline_document import PipelineDocumentCodec + + # With no saved global config set, ObjectState resolves against the + # declared defaults: the document a default-constructed config states. + saved_global_config = ( + get_saved_global_config(GlobalPipelineConfig) or GlobalPipelineConfig() + ) + effective_config, provenance = ObjectState.resolve_saved_object( + orchestrator.pipeline_config, + ancestor_objects_with_scopes=( + ObjectStateRegistry.get_ancestor_objects_with_scopes(None, use_saved=True) + ), + ) + return cls( + pipeline_code=PipelineDocumentCodec.render_with_every_parameter( + PipelineDocumentCodec.from_values( + pipeline_config=orchestrator.pipeline_config, + pipeline_steps=pipeline_definition, + ) + ), + global_config_code=ConfigDocumentCodec.render( + saved_global_config, + expected_config_type=GlobalPipelineConfig, + clean_mode=False, + ), + configuration=resolved_values( + _field_values(effective_config), + provenance, + scope_id=str(orchestrator.dataset_root), + holder_type=type(orchestrator.pipeline_config), + ), + ) + + def with_resolved_steps( + self, pipeline: "ResolvedPipelineDefinition" + ) -> "RunDeclaration": + """Add each enabled step's resolved values from the compiler's one resolution.""" + return replace( + self, + step_configurations=tuple( + resolved_values( + step.configuration_parameters(), + pipeline.step_provenance[index], + scope_id=pipeline.step_scope_ids[index], + holder_type=type(step), + ) + for index, step in enumerate(pipeline.steps) + ), + ) + + +@dataclass(frozen=True, slots=True) +class DeclaredFile: + """One declared path parameter and what the storage layer says the file holds.""" + + parameter: str + declared_as: str + path: str + content: FileContent + + +@dataclass(frozen=True, slots=True) +class FunctionRecord: + """One function call of a step: which function, with which values and files.""" + + identity: FunctionIdentity + group: str + position: int + input_memory_type: str | None + output_memory_type: str | None + parameters: tuple[RecordedParameter, ...] + declared_files: tuple[DeclaredFile, ...] + + @classmethod + def of_invocation( + cls, + invocation: "CompiledFunctionInvocation", + *, + filemanager: FileManagerLike, + backend: str, + ) -> "FunctionRecord": + contract = invocation.contract + return cls( + identity=contract.func.identity(), + group=python_expression(invocation.key.group_key), + position=invocation.key.position, + input_memory_type=invocation.input_memory_type, + output_memory_type=invocation.output_memory_type, + parameters=_parameters(invocation), + declared_files=tuple( + DeclaredFile( + parameter=parameter_name, + declared_as=declaration.plain_name, + path=str(path), + content=declaration.content_at_run_start( + Path(path), filemanager=filemanager, backend=backend + ), + ) + for parameter_name, (declaration, path) in ( + contract.declared_path_values(invocation.kwargs_dict).items() + ) + ), + ) + + +def _parameters(invocation: "CompiledFunctionInvocation") -> tuple[RecordedParameter, ...]: + """Every parameter of the function's signature with its effective value and source.""" + contract = invocation.contract + written = invocation.kwargs_dict + signature = contract.canonical_signature.parameters + supplied_at_run_time = { + *contract.runtime_owned_parameter_names, + invocation.primary_input_parameter_name, + } + sources: dict[str, type[ParameterSource]] = {} + for name, parameter in signature.items(): + if name in written: + sources[name] = WrittenInPipeline + elif name in supplied_at_run_time or parameter.default is inspect.Parameter.empty: + sources[name] = SuppliedAtRunTime + else: + sources[name] = FunctionDefault + keyword_only = inspect.Parameter.KEYWORD_ONLY + return tuple( + RecordedParameter( + name, + source.value_expression( + signature.get(name, inspect.Parameter(name, keyword_only)), written + ), + source, + ) + for name, source in { + **sources, + **{name: WrittenInPipeline for name in written if name not in signature}, + }.items() + ) + + +@dataclass(frozen=True, slots=True) +class GpuDevice: + """The GPU device assigned to one framework's arrays.""" + + memory_type: str + device_id: int + + +@dataclass(frozen=True, slots=True) +class StepCalls: + """What one step ran: its identity in the pipeline, its devices and its function calls.""" + + name: str + pipeline_position: int | None + step_scope_id: str | None + input_memory_type: str | None + output_memory_type: str | None + assigned_gpu_devices: tuple[GpuDevice, ...] + functions: tuple[FunctionRecord, ...] + + @classmethod + def of_plan( + cls, + plan: "CompiledStepPlan", + *, + filemanager: FileManagerLike, + backend: str, + ) -> "StepCalls": + return cls( + name=plan.step_name, + pipeline_position=plan.pipeline_position, + step_scope_id=plan.step_scope_id, + input_memory_type=plan.input_memory_type, + output_memory_type=plan.output_memory_type, + assigned_gpu_devices=tuple( + GpuDevice(binding.memory_type.value, binding.device_id) + for binding in plan.device_assignment.bindings + ), + functions=tuple( + FunctionRecord.of_invocation( + invocation, filemanager=filemanager, backend=backend + ) + for invocation in plan.compiled_function_pattern.iter_invocations() + ), + ) + + def is_named_by(self, producer: StreamProducerIdentity) -> bool: + """A producer identity names a step by the two fields it carries.""" + return ( + self.step_scope_id == producer.step_scope_id + and self.pipeline_position == producer.pipeline_position + ) + + +@dataclass(frozen=True, slots=True) +class StepRecord: + """One step of the run: what it ran, and its resolved configuration by owner.""" + + calls: StepCalls + configuration: tuple[ConfigValuesFromOwner, ...] + + +@dataclass(frozen=True, slots=True) +class SoftwareEnvironment: + """The interpreter and the installed distributions the run executed in.""" + + openhcs: str + python: str + python_implementation: str + platform: str + machine: str + installed_distributions: Mapping[str, str] + """Version by distribution name.""" + + @classmethod + def current(cls) -> "SoftwareEnvironment": + import openhcs + from openhcs.runtime.environment_provenance import RuntimeEnvironmentSnapshot + + environment = RuntimeEnvironmentSnapshot.current() + return cls( + openhcs=openhcs.__version__, + python=environment.python_version, + python_implementation=environment.python_implementation, + platform=environment.sys_platform, + machine=environment.machine, + installed_distributions=MappingProxyType( + {item.name: item.version for item in environment.installed_distributions} + ), + ) + + +class UnrecordedOutputFile(LookupError): + """The dataset's metadata does not say which step produced this file.""" + + +@dataclass(frozen=True, slots=True) +class RunRecord: + """What produced an output dataset. Its id is the sha256 of its stored JSON.""" + + software: SoftwareEnvironment + pipeline_code: str + global_config_code: str + configuration: tuple[ConfigValuesFromOwner, ...] + steps: tuple[StepRecord, ...] + custom_sources: Mapping[str, str] + """Source text of each custom function, by the sha256 its identity states.""" + + def to_json(self) -> JsonObject: + """The stored form: the one encoder, ``python_introspect.to_jsonable``.""" + return to_jsonable(self) + + @classmethod + def from_json(cls, stored: JsonObject) -> "RunRecord": + """Read the stored form: the one decoder, ``python_introspect.dataclass_from_mapping``.""" + return dataclass_from_mapping(cls, stored) + + @property + def record_id(self) -> str: + return hashlib.sha256( + json.dumps(self.to_json(), sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + + def store_with_saved_files(self, contexts: Mapping[str, "ExecutingContext"]) -> None: + """Write this record into the document of each output dataset the run saved files under. + + Runs in the parent process once the run has ended, also after a failure, + right after the dataset's own metadata is published. Each document + receives the producer of every file saved under its dataset root. + """ + from openhcs.core.steps.function_outputs import OpenHCSMetadataTarget + + saved = ProducedFiles( + StepExecutionObservation.combine( + context.completed_step_outputs for context in contexts.values() + ).producer_by_saved_path + ) + declared_dataset_roots = { + Path(target.dataset_root) + for context in contexts.values() + for plan in context.step_plans.values() + for target in OpenHCSMetadataTarget.for_plan(plan) + } + for dataset_root, files in saved.by_dataset_root(declared_dataset_roots).items(): + RunRecordWriter().store(dataset_root, files, self) + + @classmethod + def at_run_start( + cls, + bundle: "CompiledExecutionBundle", + *, + filemanager: FileManagerLike, + backend: str, + ) -> "RunRecord": + """Combine the compiler's declaration with the step plans and the declared files' content. + + Called once per run in the parent process before any worker starts, so + a declared input file is hashed once and the hash is of the bytes the + run reads. + """ + declaration = bundle.run_declaration + steps: dict[int, StepRecord] = {} + for axis_id in bundle.axis_ids: + context = next( + context + for context in bundle.runtime_contexts.values() + if context.require_axis_id() == axis_id + ) + for index, plan in context.step_plans.items(): + if index not in steps: + steps[index] = StepRecord( + StepCalls.of_plan(plan, filemanager=filemanager, backend=backend), + declaration.step_configurations[index], + ) + custom_sources: dict[str, str] = {} + for step in steps.values(): + for function in step.calls.functions: + custom_sources.update(function.identity.source_texts()) + return cls( + software=SoftwareEnvironment.current(), + pipeline_code=declaration.pipeline_code, + global_config_code=declaration.global_config_code, + configuration=declaration.configuration, + steps=tuple(steps[index] for index in sorted(steps)), + custom_sources=MappingProxyType(custom_sources), + ) + + def step_for(self, producer: StreamProducerIdentity) -> StepCalls: + """The step a producer identity names.""" + matches = [ + step.calls for step in self.steps if step.calls.is_named_by(producer) + ] + if len(matches) != 1: + raise LookupError( + f"Run record {self.record_id} has {len(matches)} steps for producer " + f"{producer.step_name!r} at position {producer.pipeline_position}." + ) + return matches[0] + + @classmethod + def for_dataset(cls, dataset_root: str | Path) -> Mapping[str, "RunRecord"]: + """Every record the dataset's metadata document holds, by record id.""" + return DatasetRunRecords.of_dataset(Path(dataset_root)).run_records + + @classmethod + def for_output_file(cls, output_file: str | Path) -> "ProducedFile": + """The record and producing step of one output file. + + The answer comes only from what the writer stored for this path; a file + the dataset does not list raises instead of being guessed from its name. + """ + path = Path(output_file).resolve() + dataset_root = _dataset_root_of(path) + stored = DatasetRunRecords.of_dataset(dataset_root) + entry = stored.produced_files.get(path.relative_to(dataset_root).as_posix()) + if entry is None: + raise UnrecordedOutputFile( + f"{path} is not recorded as a step output in " + f"{_metadata_path(dataset_root)}." + ) + return ProducedFile( + record=stored.run_records[entry.run_record], producer=entry.producer + ) + + def rebuild(self) -> tuple["PipelineDocument", "GlobalPipelineConfig"]: + """Rebuild the pipeline document and global config this record was run with. + + Custom functions are registered from the recorded source text (not + persisted), after checking that each text still has its recorded hash. + """ + from openhcs.core.config import GlobalPipelineConfig + from openhcs.core.config_document import ConfigDocumentCodec + from openhcs.core.pipeline_document import PipelineDocumentCodec + from openhcs.processing.custom_functions import CustomFunctionManager + from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, + ) + from openhcs.processing.custom_functions.source_namespace import ( + CustomFunctionSource, + ) + + function_name_by_source = { + source_sha256: function.identity.name + for step in self.steps + for function in step.calls.functions + for source_sha256 in function.identity.recorded_source_hashes() + } + for recorded_sha256, source_text in self.custom_sources.items(): + if CustomFunctionSource.content_digest(source_text.encode("utf-8")) != ( + recorded_sha256 + ): + raise ValueError( + f"Run record {self.record_id}: custom source {recorded_sha256} does " + "not match its recorded hash." + ) + source = CustomFunctionSource( + function_name_by_source[recorded_sha256], recorded_sha256 + ) + if CustomFunctionRuntimeRegistry.declaration_for_source(source) is None: + CustomFunctionManager().register_from_code( + source_text, persist=False, emit_signal=False + ) + return ( + PipelineDocumentCodec.from_source(self.pipeline_code), + ConfigDocumentCodec.from_source( + self.global_config_code, + expected_config_type=GlobalPipelineConfig, + ), + ) + + +@dataclass(frozen=True, slots=True) +class StoredProducedFile: + """What the metadata document stores for one saved file.""" + + run_record: str + producer: StreamProducerIdentity + + +@dataclass(frozen=True, slots=True) +class ProducedFile: + """One output file, the step that wrote it and the record of the run.""" + + record: RunRecord + producer: StreamProducerIdentity + + @property + def step(self) -> StepCalls: + return self.record.step_for(self.producer) + + +@dataclass(frozen=True, slots=True) +class DatasetRunRecords: + """The part of a dataset's metadata document that says what produced its files. + + The document stores each field of this class under the field's name. + """ + + run_records: Mapping[str, RunRecord] = field(default_factory=dict) + produced_files: Mapping[str, StoredProducedFile] = field(default_factory=dict) + """By path relative to the dataset root.""" + + @classmethod + def in_document(cls, document: JsonObject) -> "DatasetRunRecords": + return dataclass_from_mapping( + cls, + { + item.name: document[item.name] + for item in fields(cls) + if item.name in document + }, + ) + + @classmethod + def of_dataset(cls, dataset_root: Path) -> "DatasetRunRecords": + return cls.in_document( + json.loads(_metadata_path(dataset_root).read_text(encoding="utf-8")) + ) + + def with_run( + self, record: RunRecord, produced: Mapping[str, StreamProducerIdentity] + ) -> "DatasetRunRecords": + """Add one run's files; keep only the records some file still refers to.""" + record_id = record.record_id + produced_files = { + **self.produced_files, + **{ + path: StoredProducedFile(record_id, producer) + for path, producer in produced.items() + }, + } + referenced = {entry.run_record for entry in produced_files.values()} + return DatasetRunRecords( + run_records={ + stored_id: stored + for stored_id, stored in {**self.run_records, record_id: record}.items() + if stored_id in referenced + }, + produced_files=produced_files, + ) + + +def _metadata_path(dataset_root: Path) -> Path: + from openhcs.core.virtual_workspace_metadata import get_metadata_path + + return Path(get_metadata_path(dataset_root)) + + +def _dataset_root_of(output_file: Path) -> Path: + for directory in output_file.parents: + if _metadata_path(directory).is_file(): + return directory + raise UnrecordedOutputFile( + f"No directory above {output_file} holds an OpenHCS metadata document." + ) + + +@dataclass(frozen=True, slots=True) +class ProducedFiles: + """The saved paths of one run, each with the producer identity its writer stated.""" + + producer_by_path: Mapping[Path, StreamProducerIdentity] + + @staticmethod + def stated_by_step( + context: ExecutingContext, + plan: CompiledStepPlan, + *, + artifact_materializations: tuple[MaterializedRuntimeArtifact, ...], + ) -> Mapping[Path, StreamProducerIdentity]: + """The producer identity of each file this step saved under a metadata document. + + Both statements are the ones the viewer stream uses: a main-flow record + carries its own identity; a saved artifact's comes from its output plan. + """ + from openhcs.core.steps.function_outputs import OpenHCSMetadataTarget + + producers: dict[Path, StreamProducerIdentity] = {} + for target in OpenHCSMetadataTarget.for_execution( + context, plan, artifact_materializations=artifact_materializations + ): + if not OpenHCSMetadataTarget.hosts_metadata_document(target.backend): + continue + for record in target.produced_records(context, plan): + producers[Path(record.path_under(target.output_dir))] = ( + record.producer_identity + ) + artifact_target = plan.runtime_artifact_materialization + if artifact_target.has_persistent_target: + backend = artifact_target.require_persistent_backend() + if OpenHCSMetadataTarget.hosts_metadata_document(backend): + for saved in artifact_materializations: + producer = plan.producer_identity_for_artifact( + saved.materialization.output_plan + ) + for output in saved.outputs_for_backend(backend): + producers[Path(output.path)] = producer + return MappingProxyType(producers) + + def by_dataset_root( + self, dataset_roots: Iterable[Path] + ) -> Mapping[Path, "ProducedFiles"]: + """Split the files by the declared output dataset root each one lies under. + + The roots are the ones the compiled plans declare. Where one declared + root contains another, a file belongs to the nearest. + """ + nearest_first = sorted(set(dataset_roots), key=lambda root: -len(root.parts)) + by_root: dict[Path, dict[Path, StreamProducerIdentity]] = {} + for path, producer in self.producer_by_path.items(): + root = next( + (root for root in nearest_first if path.is_relative_to(root)), None + ) + if root is None: + raise ValueError( + f"Saved file {path} lies under no declared output dataset root " + f"({', '.join(map(str, nearest_first))})." + ) + by_root.setdefault(root, {})[path] = producer + return MappingProxyType( + {root: ProducedFiles(MappingProxyType(files)) for root, files in by_root.items()} + ) + + +class RunRecordWriter(AtomicMetadataWriter): + """Stores a run's record in a dataset's metadata document, under that document's lock.""" + + def store(self, dataset_root: Path, files: ProducedFiles, record: RunRecord) -> None: + """Add the record with the files the run saved under ``dataset_root``. + + A record no file refers to any more is dropped in the same update. + """ + + def update(document: dict[str, JsonValue] | None) -> dict[str, JsonValue]: + document = self._ensure_subdirectories_structure(document) + document.update( + to_jsonable( + DatasetRunRecords.in_document(document).with_run( + record, + { + path.relative_to(dataset_root).as_posix(): producer + for path, producer in files.producer_by_path.items() + }, + ) + ) + ) + return document + + self._execute_update(METADATA_CONFIG.metadata_path(dataset_root), update) diff --git a/openhcs/core/steps/abstract.py b/openhcs/core/steps/abstract.py index 2cb064dea0..3c88585d79 100644 --- a/openhcs/core/steps/abstract.py +++ b/openhcs/core/steps/abstract.py @@ -10,11 +10,12 @@ import inspect from pathlib import Path from collections.abc import Mapping -from dataclasses import dataclass, field, is_dataclass +from dataclasses import dataclass, field, fields, is_dataclass from types import MappingProxyType from typing import TYPE_CHECKING, Iterable, get_type_hints from objectstate import get_base_type_for_lazy, semantic_values_equal +from polystore.streaming.identity import StreamProducerIdentity # Import LazyStepMaterializationConfig for type hints @@ -65,6 +66,11 @@ class StepExecutionObservation: "OpenHCSMetadataTarget", "VirtualWorkspaceSourceProjectionEntries" ] = field(default_factory=dict) + producer_by_saved_path: Mapping[Path, StreamProducerIdentity] = field( + default_factory=dict + ) + """The producer identity its writer stated for each saved file.""" + def __post_init__(self) -> None: projections = dict(self.source_projection_entries_by_target) if projections: @@ -93,13 +99,7 @@ def __post_init__(self) -> None: @property def is_empty(self) -> bool: """Whether this completed output carrier has no facts to transport.""" - return not ( - self.materialized_locations_by_address - or self.runtime_export_paths - or self.hook_observations - or self.sample_numbers_by_export_path - or self.source_projection_entries_by_target - ) + return not any(getattr(self, carried.name) for carried in fields(self)) @classmethod def empty(cls) -> "StepExecutionObservation": @@ -124,7 +124,14 @@ def combine( hook_observations = [] sample_numbers = {} source_projections = {} + producers = {} for observation in observations: + for path, producer in observation.producer_by_saved_path.items(): + if producers.setdefault(path, producer) != producer: + raise ValueError( + f"Saved file {path} has two producers: " + f"{producers[path]} and {producer}." + ) for ( address, values, @@ -163,6 +170,7 @@ def combine( for target, entries in source_projections.items() } ), + MappingProxyType(producers), ) @@ -267,6 +275,11 @@ def declaration_parameters(self) -> dict[str, object]: if not name.startswith("_") } + def configuration_parameters(self) -> dict[str, object]: + """The values that configure this step, apart from what it computes.""" + + return self.declaration_parameters() + def same_declaration(self, other: object) -> bool: """Return whether another step is the same complete declaration.""" diff --git a/openhcs/core/steps/function_outputs.py b/openhcs/core/steps/function_outputs.py index bd9ebf5655..a2af0c6090 100644 --- a/openhcs/core/steps/function_outputs.py +++ b/openhcs/core/steps/function_outputs.py @@ -56,6 +56,7 @@ SourceProjection, SourceProjectionMetadataSerializer, ) +from openhcs.core.run_record import ProducedFiles from openhcs.core.steps.abstract import StepExecutionObservation from openhcs.core.steps.function_artifact_materialization import ( ArtifactMaterializationTargetPlan, @@ -122,6 +123,9 @@ def finalize_function_step_outputs( projection_entries = OpenHCSMetadataTarget.observe_for_step( context, plan, artifact_materializations=materializations ) + producers = ProducedFiles.stated_by_step( + context, plan, artifact_materializations=materializations + ) else: _profile_finalization_phase( "finalize_memory_outputs", @@ -150,11 +154,19 @@ def finalize_function_step_outputs( ), plan, ) + producers = _profile_finalization_phase( + "finalize_saved_path_producers", + lambda: ProducedFiles.stated_by_step( + context, plan, artifact_materializations=materializations + ), + plan, + ) return replace( StepExecutionObservation.combine( item.observation(plan, context) for item in materializations ), source_projection_entries_by_target=projection_entries, + producer_by_saved_path=producers, ) diff --git a/openhcs/core/steps/function_step.py b/openhcs/core/steps/function_step.py index bfc8ccbd27..056ec68e5f 100644 --- a/openhcs/core/steps/function_step.py +++ b/openhcs/core/steps/function_step.py @@ -57,6 +57,15 @@ def with_function_spec(self, func: FunctionSpec) -> "FunctionStep": step.func = func return step + def configuration_parameters(self) -> dict[str, object]: + """The function pattern is what the step computes, not its configuration.""" + + return { + name: value + for name, value in self.declaration_parameters().items() + if value is not self.func + } + def occurrence_authorities(self) -> tuple[object, ...]: """Prefer callable-pattern authority before nominal step authority.""" diff --git a/openhcs/core/vfs_protocol.py b/openhcs/core/vfs_protocol.py index cc4eb9022f..7d2fe3053c 100644 --- a/openhcs/core/vfs_protocol.py +++ b/openhcs/core/vfs_protocol.py @@ -7,7 +7,9 @@ from dataclasses import dataclass from pathlib import Path from types import MappingProxyType -from typing import Annotated, Any, get_args, get_origin +from typing import Annotated, Any, ClassVar, get_args, get_origin + +from polystore.file_content import FileContent, UnhashedFileContent class FileManagerLike(ABC): @@ -35,6 +37,10 @@ def exists(self, path: str | Path, backend: str) -> bool: def is_dir(self, path: str | Path, backend: str) -> bool: """Return whether a path is a directory in the selected backend.""" + @abstractmethod + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + """Say what bytes a file holds, or why the backend cannot hash it.""" + @abstractmethod def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: """Load an object from the selected backend.""" @@ -103,6 +109,9 @@ def ensure_directory(self, directory: str | Path, backend: str) -> str: class DatasetPathDeclaration(ABC): """Nominal declaration for a path resolved against one compilation plate.""" + plain_name: ClassVar[str] + """What this kind of path is, for a reader of a run's record.""" + @abstractmethod def validate_target( self, @@ -113,6 +122,16 @@ def validate_target( ) -> None: """Validate the resolved target without creating it.""" + @abstractmethod + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + """Identify the bytes a run will read here, or say why there are none to hash.""" + @classmethod def from_annotation(cls, annotation: Any) -> "DatasetPathDeclaration | None": """Return the sole declaration carried by one structured annotation.""" @@ -146,6 +165,17 @@ def _from_annotation(cls, annotation: Any) -> tuple["DatasetPathDeclaration", .. class PlateInputFileDeclaration(DatasetPathDeclaration): """Existing plate-relative input file.""" + plain_name: ClassVar[str] = "input file" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + return filemanager.file_content(target, backend) + def validate_target( self, target: Path, @@ -163,6 +193,20 @@ def validate_target( class PlateInputDirectoryDeclaration(DatasetPathDeclaration): """Existing plate-relative input directory.""" + plain_name: ClassVar[str] = "input directory" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + del target, filemanager, backend + return UnhashedFileContent( + "A declared input directory is recorded by path; its files are not hashed." + ) + def validate_target( self, target: Path, @@ -184,6 +228,20 @@ def validate_target( class PlateOutputFileDeclaration(DatasetPathDeclaration): """Plate-relative output file whose writer owns creation.""" + plain_name: ClassVar[str] = "output file" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + del target, filemanager, backend + return UnhashedFileContent( + "This run writes the file; it has no content when the run starts." + ) + def validate_target( self, target: Path, @@ -198,6 +256,20 @@ def validate_target( class PlateOutputDirectoryDeclaration(DatasetPathDeclaration): """Plate-relative output directory whose writer owns creation.""" + plain_name: ClassVar[str] = "output directory" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + del target, filemanager, backend + return UnhashedFileContent( + "This run writes the directory; it has no content when the run starts." + ) + def validate_target( self, target: Path, diff --git a/openhcs/mcp/context.py b/openhcs/mcp/context.py index 233399e5cf..97344a7c64 100644 --- a/openhcs/mcp/context.py +++ b/openhcs/mcp/context.py @@ -31,6 +31,7 @@ PipelineAuthoringService, ) from openhcs.agent.services.plate_inspection_service import PlateInspectionService + from openhcs.agent.services.run_record_service import RunRecordService from openhcs.agent.services.plate_streaming_service import PlateStreamingService from openhcs.agent.services.runtime_server_service import RuntimeServerService from openhcs.agent.services.selected_plate_service import SelectedPlateService @@ -62,6 +63,7 @@ class OpenHCSAgentContext: "_object_state_field_help_service", "_pipeline_service", "_plate_inspection_service", + "_run_record_service", "_plate_streaming_service", "_runtime_server_service", "_viewer_endpoint_discovery_service", @@ -109,6 +111,7 @@ def __init__( self._object_state_field_help_service = object_state_field_help_service self._pipeline_service = pipeline_service self._plate_inspection_service = plate_inspection_service + self._run_record_service: "RunRecordService | None" = None self._plate_streaming_service = plate_streaming_service self._runtime_server_service = runtime_server_service self._viewer_endpoint_discovery_service = viewer_endpoint_discovery_service @@ -212,6 +215,14 @@ def plate_inspection_service(self) -> "PlateInspectionService": self._plate_inspection_service = PlateInspectionService(self.path_policy) return self._plate_inspection_service + @property + def run_record_service(self) -> "RunRecordService": + if self._run_record_service is None: + from openhcs.agent.services.run_record_service import RunRecordService + + self._run_record_service = RunRecordService(self.path_policy) + return self._run_record_service + @property def plate_streaming_service(self) -> "PlateStreamingService": if self._plate_streaming_service is None: diff --git a/openhcs/processing/backends/lib_registry/unified_registry.py b/openhcs/processing/backends/lib_registry/unified_registry.py index 5a49fd60e5..b6606421c1 100644 --- a/openhcs/processing/backends/lib_registry/unified_registry.py +++ b/openhcs/processing/backends/lib_registry/unified_registry.py @@ -137,6 +137,17 @@ def composite_key(self) -> str: def require_current_declaration(self) -> None: """Validate declaration lifetimes specialized by metadata owners.""" + def identity(self) -> "FunctionIdentity": + """Where this function comes from and at which version.""" + from openhcs.core.function_identity import FunctionIdentity + + import_identity = self.import_identity + return FunctionIdentity.of_module_export( + key=self.composite_key, + module=import_identity.module_name, + name=import_identity.function_name, + ) + @property def display_name(self) -> str: """Human-readable function name for catalogs and selectors.""" diff --git a/openhcs/processing/custom_functions/manager.py b/openhcs/processing/custom_functions/manager.py index 3497985b91..2589026d6d 100644 --- a/openhcs/processing/custom_functions/manager.py +++ b/openhcs/processing/custom_functions/manager.py @@ -633,7 +633,7 @@ def _prepare_source( function_name=declaration.__name__, content_sha256=CustomFunctionSource.content_digest(code.encode("utf-8")), ) - CustomFunctionSourceNamespace(source, namespace).bind(declaration) + CustomFunctionSourceNamespace(source, namespace, code).bind(declaration) try: return project_custom_function( declaration, diff --git a/openhcs/processing/custom_functions/runtime_registry.py b/openhcs/processing/custom_functions/runtime_registry.py index 1d94d4eb7e..3392bccc3f 100644 --- a/openhcs/processing/custom_functions/runtime_registry.py +++ b/openhcs/processing/custom_functions/runtime_registry.py @@ -14,6 +14,7 @@ from typing import TYPE_CHECKING from openhcs.core.function_contract_metadata import FunctionContractAttribute +from openhcs.core.function_identity import CustomFunctionIdentity from openhcs.processing.backends.lib_registry.unified_registry import FunctionMetadata from openhcs.processing.custom_functions.source_namespace import ( CustomFunctionSource, @@ -60,6 +61,18 @@ def require_current_declaration(self) -> None: super().require_current_declaration() CustomFunctionRuntimeRegistry.require_current_metadata(self) + def identity(self) -> CustomFunctionIdentity: + """The source text that defines this function, by hash, and what it imports.""" + namespace = CustomFunctionSourceNamespace.bound_to(self.func) + import_identity = self.import_identity + return CustomFunctionIdentity.of_source( + key=self.composite_key, + module=import_identity.module_name, + name=import_identity.function_name, + source_sha256=namespace.source.content_sha256, + source_text=namespace.code, + ) + class CustomFunctionCanonicalLookup: """Independent source-lookup capability composed with the registry template.""" diff --git a/openhcs/processing/custom_functions/source_namespace.py b/openhcs/processing/custom_functions/source_namespace.py index 7b3d90f807..d5a46c821a 100644 --- a/openhcs/processing/custom_functions/source_namespace.py +++ b/openhcs/processing/custom_functions/source_namespace.py @@ -81,12 +81,34 @@ class CustomFunctionSourceNamespace: source: CustomFunctionSource bindings: dict[str, object] = field(repr=False) + code: str = field(repr=False) + """The exact text ``source.content_sha256`` is the hash of.""" member_prefix: ClassVar[str] = "__openhcs_member_" + def __post_init__(self) -> None: + if CustomFunctionSource.content_digest(self.code.encode("utf-8")) != ( + self.source.content_sha256 + ): + raise ValueError( + f"Custom function source {self.source.function_name!r}: the text " + "does not have the declared content hash." + ) + + @classmethod + def bound_to(cls, declaration: Callable) -> "CustomFunctionSourceNamespace": + """Return the namespace ``bind`` attached to this declaration.""" + revision = vars(declaration)[FunctionContractAttribute.declaration_revision] + return vars(declaration)[cls.attribute_of_revision(revision)] + + @staticmethod + def attribute_of_revision(content_sha256: str) -> str: + """The attribute of a callable that holds the namespace of one source revision.""" + return f"__openhcs_source_{content_sha256}" + @property def export_attribute(self) -> str: """Derive the revision-specific namespace address on its callable.""" - return f"__openhcs_source_{self.source.content_sha256}" + return self.attribute_of_revision(self.source.content_sha256) def member_attribute(self, relative_qualname: str) -> str: """Encode one lexical name, without colliding with namespace members.""" diff --git a/openhcs/serialization/pycodify_formatters.py b/openhcs/serialization/pycodify_formatters.py index 5231509a78..dfa04e26dd 100644 --- a/openhcs/serialization/pycodify_formatters.py +++ b/openhcs/serialization/pycodify_formatters.py @@ -1,6 +1,7 @@ from __future__ import annotations import inspect +import sys from collections.abc import Callable from pathlib import Path @@ -38,6 +39,42 @@ def format(self, value, context: FormatContext) -> SourceFragment: return to_source(self.reference_for(value), context) +class RangeFormatter(SourceFormatter): + """A ``range`` (for example a signature default) is written as its own call.""" + + priority = 50 + + def can_format(self, value) -> bool: + return isinstance(value, range) + + def format(self, value: range, context: FormatContext) -> SourceFragment: + return SourceFragment(repr(value), frozenset()) + + +class ModuleExportedCallableFormatter(SourceFormatter): + """A callable object its module exports under its own name, by import. + + NumPy's dispatchers (``numpy.sum`` as a signature default) are callable + objects, not functions; they load back as themselves from their module. + """ + + priority = 20 + + def can_format(self, value) -> bool: + if not callable(value): + return False + module = sys.modules.get(getattr(value, "__module__", None) or "") + name = getattr(value, "__qualname__", None) + return ( + module is not None + and isinstance(name, str) + and getattr(module, name, None) is value + ) + + def format(self, value, context: FormatContext) -> SourceFragment: + return imported_name(value.__module__, value.__qualname__, context) + + class MaterializationSpecFormatter(SourceFormatter): priority = 110 @@ -120,6 +157,48 @@ def _exported_callable_parameter_exclusions(func) -> set[str]: return excluded.difference(contract.overridable_runtime_parameter_names) +def _signature_defaults(func) -> dict[str, object]: + """The parameters this function takes from its signature when a pattern leaves them out. + + A parameter the platform supplies when it calls the function (an artifact + input, a value bound from the step's configuration) is not one of them: + writing its signature default would replace the supplied value. + """ + + public_func = _public_pattern_callable(func) + supplied_when_called = { + *_exported_callable_parameter_exclusions(public_func), + *CallableContract.from_callable(public_func).runtime_owned_parameter_names, + } + return { + name: parameter.default + for name, parameter in inspect.signature(public_func).parameters.items() + if parameter.default is not inspect.Parameter.empty + and name not in supplied_when_called + } + + +def with_every_parameter_written(pattern): + """Write each function of a pattern with every parameter it has a value for. + + A pattern that leaves a parameter out runs with the signature default of + the installed function; the complete document states that value. + """ + + if _is_pattern_tuple(pattern): + func, args = pattern + return (func, {**_signature_defaults(func), **args}) + if isinstance(pattern, list): + return [with_every_parameter_written(item) for item in pattern] + if isinstance(pattern, dict): + return { + key: with_every_parameter_written(item) for key, item in pattern.items() + } + if _public_pattern_callable(pattern) is not None: + return (pattern, _signature_defaults(pattern)) + return pattern + + class FunctionPatternTupleFormatter(SourceFormatter): priority = 85 diff --git a/pyproject.toml b/pyproject.toml index 2b7408d96a..00c35acf1a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -105,10 +105,10 @@ dependencies = [ "zmqruntime>=0.6.4,<0.7", "pycodify>=0.2.1,<0.3", "objectstate>=3.0.4,<4", - "python-introspect>=0.2.8,<0.3", + "python-introspect>=0.3.0,<0.4", "metaclass-registry>=0.4.0,<0.5", "arraybridge>=0.4.1,<0.5", - "polystore>=0.6.2,<0.7", + "polystore>=0.6.3,<0.7", "formbridge>=0.1.3,<0.2", "pyqt-reactive>=0.7.2,<0.8", ] diff --git a/tests/fixtures/mcp/mcp_tool_contract.json b/tests/fixtures/mcp/mcp_tool_contract.json index 8de654d693..46df4c84a0 100644 --- a/tests/fixtures/mcp/mcp_tool_contract.json +++ b/tests/fixtures/mcp/mcp_tool_contract.json @@ -397,6 +397,7 @@ "openhcs_load_pipeline_file", "openhcs_observe_owned_runtime", "openhcs_query_plate_files", + "openhcs_read_run_record", "openhcs_register_custom_function", "openhcs_render_config_source", "openhcs_render_pipeline_source", @@ -455,6 +456,7 @@ "openhcs_observe_owned_runtime", "openhcs_probe_viewer_window", "openhcs_query_plate_files", + "openhcs_read_run_record", "openhcs_register_custom_function", "openhcs_render_config_source", "openhcs_render_pipeline_source", @@ -560,6 +562,7 @@ "openhcs_observe_owned_runtime", "openhcs_probe_viewer_window", "openhcs_query_plate_files", + "openhcs_read_run_record", "openhcs_register_custom_function", "openhcs_render_config_source", "openhcs_render_pipeline_source", @@ -4448,6 +4451,47 @@ }, "title": "Query plate files" }, + "openhcs_read_run_record": { + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": true, + "readOnlyHint": true, + "title": "Read run record" + }, + "description": "Read-only: what produced an output dataset. Pass an output dataset root (the directory holding openhcs_metadata.json) to get every run record by record id, or one output file inside a dataset to get the producer identity stored for that file, the producing step and the record of that run. A file the dataset does not list is an error, never a guess from its name. Each record lists what every step ran: each function with its identity (registry key, defining and imported library versions, sha256 of a custom function's source), every parameter with its value and source (pipeline, function_default, run_time), declared files with content hashes, and devices. Pass include_full_record=true to add the complete stored record: pipeline and global config documents, resolved configuration with the owner of every value, custom source texts and all installed distributions.", + "execution": null, + "icons": null, + "inputSchema": { + "additionalProperties": false, + "properties": { + "include_full_record": { + "default": false, + "title": "Include Full Record", + "type": "boolean" + }, + "path": { + "title": "Path", + "type": "string" + } + }, + "required": [ + "path" + ], + "title": "openhcs_read_run_recordArguments", + "type": "object" + }, + "meta": { + "openhcs/outputContract": "RunRecordReadResult" + }, + "name": "openhcs_read_run_record", + "outputSchema": { + "additionalProperties": true, + "title": "openhcs_read_run_recordDictOutput", + "type": "object" + }, + "title": "Read run record" + }, "openhcs_register_custom_function": { "annotations": { "destructiveHint": true, diff --git a/tests/integration/test_compiled_artifact_inspection.py b/tests/integration/test_compiled_artifact_inspection.py index 0067b3a6db..fedff805eb 100644 --- a/tests/integration/test_compiled_artifact_inspection.py +++ b/tests/integration/test_compiled_artifact_inspection.py @@ -69,6 +69,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration @contextmanager @@ -191,6 +192,7 @@ def produce(image): transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=object(), + run_declaration=minimal_run_declaration(), ) return ( ZMQCompileArtifactRecord( diff --git a/tests/integration/test_custom_source_transport.py b/tests/integration/test_custom_source_transport.py index 624827fdcb..0f774eff9a 100644 --- a/tests/integration/test_custom_source_transport.py +++ b/tests/integration/test_custom_source_transport.py @@ -25,6 +25,7 @@ from tests.context_support import ( compiled_context, ) +from tests.run_records import minimal_run_declaration WORKTREE = Path(__file__).resolve().parents[2] PYTHON = Path(sys.executable) @@ -498,6 +499,7 @@ def _compiled_custom_contexts(payload, runtime): pipeline_definition=(), runtime_contexts=contexts, worker_assignments={f"worker_{i}": [axis] for i, axis in enumerate(contexts)}, runtime_environment=runtime, + run_declaration=minimal_run_declaration(), ).transport_contexts diff --git a/tests/integration/test_generated_source_relative_paths.py b/tests/integration/test_generated_source_relative_paths.py index da08cb3f81..6ccc49588d 100644 --- a/tests/integration/test_generated_source_relative_paths.py +++ b/tests/integration/test_generated_source_relative_paths.py @@ -6,6 +6,8 @@ from collections.abc import Sequence from pathlib import Path +from polystore.file_content import FileContent, HashedFileContent + from tests.artifact_graphs import admitted_graph from openhcs.core.function_patterns import compile_function_pattern from openhcs.core.function_step_transport import FunctionStepTransportAuthority @@ -36,6 +38,10 @@ def is_dir(self, path: str | Path, backend: str) -> bool: del backend return Path(path).is_dir() + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + del backend + return HashedFileContent.of_physical_file(Path(file_path)) + def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: raise NotImplementedError diff --git a/tests/integration/test_run_record_round_trip.py b/tests/integration/test_run_record_round_trip.py new file mode 100644 index 0000000000..5c187d34bb --- /dev/null +++ b/tests/integration/test_run_record_round_trip.py @@ -0,0 +1,135 @@ +"""DL3: the record read from one output file is enough to run the pipeline again.""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from queue import Queue + +import pytest +from objectstate.lazy_factory import ensure_global_config_context + +from openhcs.core.config import GlobalPipelineConfig, LazyPathPlanningConfig, PipelineConfig +from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator +from openhcs.core.progress import ProgressExecutionContext, set_progress_queue +from openhcs.core.run_record import DatasetRunRecords, RunRecord +from openhcs.core.steps.function_step import FunctionStep +from openhcs.processing.backends.processors.numpy_processor import ( + stack_percentile_normalize, +) +from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, +) +from tests.unit.test_artifact_publication_journey import _plate +from tests.unit.test_run_record import count_bright_source, register # noqa: F401 + + +@pytest.fixture(autouse=True) +def progress_events(): + set_progress_queue(Queue()) + yield + set_progress_queue(None) + + +def run(plate: Path, pipeline_config: PipelineConfig, steps: list) -> Path: + """Run the steps on the plate and return its output dataset root.""" + set_progress_queue(Queue()) + orchestrator = PipelineOrchestrator(plate, pipeline_config=pipeline_config).initialize() + results = orchestrator.execute_compiled_bundle( + orchestrator.compile_pipelines(steps), + progress_queue=Queue(), + progress_context=ProgressExecutionContext( + execution_id="run-record-round-trip", subject_id=str(plate) + ), + ) + assert all(result.is_success() for result in results.values()), { + axis: result.error_message for axis, result in results.items() + } + return plate.with_name(f"{plate.name}_out") + + +def produced_file_bytes(output_root: Path) -> dict[str, str]: + """sha256 of every file the dataset records as a step output, by relative path.""" + produced = DatasetRunRecords.of_dataset(output_root).produced_files + return { + relative: hashlib.sha256((output_root / relative).read_bytes()).hexdigest() + for relative in sorted(produced) + } + + +SUBMISSION_SCOPE = re.compile(r"::pipeline::submission_[0-9a-f]+") + + +def without_run_location(record: RunRecord, plate: Path) -> str: + """The record with this run's plate path and submission scope written as placeholders.""" + text = json.dumps(record.to_json(), sort_keys=True).replace(str(plate), "") + return SUBMISSION_SCOPE.sub("::pipeline::", text) + + +def test_record_read_from_one_output_file_reruns_to_identical_outputs(tmp_path, register): + (tmp_path / "first").mkdir() + (tmp_path / "second").mkdir() + first_plate = _plate(tmp_path / "first" / "source", site_count=2) + first_root = run( + first_plate, + PipelineConfig( + path_planning_config=LazyPathPlanningConfig(output_dir_suffix="_out"), + num_workers=1, + use_threading=True, + ), + [ + FunctionStep( + func=(stack_percentile_normalize, {"low_percentile": 2.0}), name="norm" + ), + FunctionStep(func=register(count_bright_source(0.5)), name="bright"), + ], + ) + first_files = produced_file_bytes(first_root) + tables = [relative for relative in first_files if relative.endswith(".csv")] + images = [relative for relative in first_files if relative.endswith(".tif")] + assert tables and images + + produced = RunRecord.for_output_file(first_root / tables[0]) + assert produced.step.name == "bright" + CustomFunctionRuntimeRegistry.clear() + document, global_config = produced.record.rebuild() + + ensure_global_config_context(GlobalPipelineConfig, global_config) + second_plate = _plate(tmp_path / "second" / "source", site_count=2) + second_root = run(second_plate, document.pipeline_config, list(document.pipeline_steps)) + + assert produced_file_bytes(second_root) == first_files + second_record = RunRecord.for_output_file(second_root / tables[0]).record + differences = differing_values( + json.loads(without_run_location(produced.record, first_plate)), + json.loads(without_run_location(second_record, second_plate)), + ) + # The rebuilt document writes every parameter, so a value the first run took + # from a signature default is, in the second run, written in the pipeline. + # Nothing else differs once the plate path and the submission scope are set aside. + assert set(differences.values()) == {("function_default", "pipeline")} + assert all( + re.fullmatch(r"/steps/\d+/calls/functions/\d+/parameters/\d+/source", where) + for where in differences + ) + + +def differing_values(first, second, path: str = "") -> dict[str, tuple]: + """Every leaf at which two JSON values differ, by its path.""" + if isinstance(first, dict) and isinstance(second, dict): + return { + where: values + for key in sorted({*first, *second}) + for where, values in differing_values( + first.get(key), second.get(key), f"{path}/{key}" + ).items() + } + if isinstance(first, list) and isinstance(second, list) and len(first) == len(second): + return { + where: values + for index, (left, right) in enumerate(zip(first, second, strict=True)) + for where, values in differing_values(left, right, f"{path}/{index}").items() + } + return {} if first == second else {path: (first, second)} diff --git a/tests/run_records.py b/tests/run_records.py new file mode 100644 index 0000000000..e309cd768f --- /dev/null +++ b/tests/run_records.py @@ -0,0 +1,37 @@ +"""The smallest valid run declaration and run record, for tests of other behaviour.""" + +from types import MappingProxyType + +from openhcs.core.run_record import ( + ProducedFiles, + RunDeclaration, + RunRecord, + SoftwareEnvironment, +) + + +def minimal_run_declaration() -> RunDeclaration: + """A declaration of an empty pipeline with no resolved configuration.""" + return RunDeclaration( + pipeline_code="pipeline_steps = []\n", + global_config_code="", + configuration=(), + ) + + +def minimal_run_record() -> RunRecord: + """A record of a run with no steps.""" + declaration = minimal_run_declaration() + return RunRecord( + software=SoftwareEnvironment.current(), + pipeline_code=declaration.pipeline_code, + global_config_code=declaration.global_config_code, + configuration=declaration.configuration, + steps=(), + custom_sources=MappingProxyType({}), + ) + + +def no_produced_files() -> ProducedFiles: + """A run that reports no saved file.""" + return ProducedFiles(MappingProxyType({})) diff --git a/tests/unit/agent/test_agent_services.py b/tests/unit/agent/test_agent_services.py index 5179542cc8..5f170e3da8 100644 --- a/tests/unit/agent/test_agent_services.py +++ b/tests/unit/agent/test_agent_services.py @@ -130,6 +130,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration def sample_processing_function(image, sigma: float = 1.0): @@ -302,6 +303,7 @@ def _compile_inspection_result( transport_contexts=runtime_contexts, worker_assignments=worker_assignments, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ), source_workspace_projection=source_workspace_projection, ) diff --git a/tests/unit/test_artifact_plan_view.py b/tests/unit/test_artifact_plan_view.py index f034d64481..16843dcc82 100644 --- a/tests/unit/test_artifact_plan_view.py +++ b/tests/unit/test_artifact_plan_view.py @@ -65,6 +65,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration class QtApplication: @@ -147,6 +148,7 @@ def transform(image, labels): transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=object(), + run_declaration=minimal_run_declaration(), ) inspection = CompiledArtifactInspection.from_execution_bundle( compile_artifact_id="compile-1", diff --git a/tests/unit/test_catalog_startup_gc_lifetime.py b/tests/unit/test_catalog_startup_gc_lifetime.py index 2eb7113910..0b20d02a01 100644 --- a/tests/unit/test_catalog_startup_gc_lifetime.py +++ b/tests/unit/test_catalog_startup_gc_lifetime.py @@ -23,10 +23,15 @@ def test_startup_freeze_preserves_job_gc_and_releases_source_cycles(): CustomFunctionSource, CustomFunctionSourceNamespace, ) bindings = {"__name__": "startup_lifetime_control"} - exec("def captured_source(): pass", bindings) + code = "def captured_source(): pass" + exec(code, bindings) function = bindings["captured_source"] namespace = CustomFunctionSourceNamespace( - CustomFunctionSource("captured_source", "lifetime-control"), bindings, + CustomFunctionSource( + "captured_source", CustomFunctionSource.content_digest(code.encode()) + ), + bindings, + code, ) namespace.bind(function) captured = weakref.ref(function) diff --git a/tests/unit/test_cellprofiler_pipeline_import.py b/tests/unit/test_cellprofiler_pipeline_import.py index 5ca762110f..4febdbeb8f 100644 --- a/tests/unit/test_cellprofiler_pipeline_import.py +++ b/tests/unit/test_cellprofiler_pipeline_import.py @@ -3,12 +3,14 @@ from __future__ import annotations import ast +import hashlib from collections.abc import Callable, Sequence from dataclasses import replace from pathlib import Path import pytest from objectstate import config_context +from polystore.file_content import FileContent, HashedFileContent from openhcs.constants import Backend from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep @@ -118,6 +120,11 @@ def is_dir(self, path: str | Path, backend: str) -> bool: del backend return Path(path) == Path("pipelines") + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + del backend + stored = self.files[Path(file_path)].encode("utf-8") + return HashedFileContent(hashlib.sha256(stored).hexdigest(), len(stored)) + def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: del backend, kwargs return self.files[Path(file_path)] diff --git a/tests/unit/test_compilation_paths.py b/tests/unit/test_compilation_paths.py index 89a04b208b..9993c31810 100644 --- a/tests/unit/test_compilation_paths.py +++ b/tests/unit/test_compilation_paths.py @@ -11,6 +11,7 @@ import pytest from objectstate import ObjectState, ObjectStateRegistry +from polystore.file_content import FileContent, UnhashedFileContent from python_introspect import Enableable from tests.artifact_graphs import admitted_graph @@ -70,6 +71,12 @@ def is_dir(self, path: str | Path, backend: str) -> bool: self.validation_calls.append(("is_dir", target, backend)) return target in self.directories + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + del backend + return UnhashedFileContent( + f"{type(self).__name__} names {Path(file_path)} without holding its bytes." + ) + def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: raise NotImplementedError diff --git a/tests/unit/test_compiled_execution.py b/tests/unit/test_compiled_execution.py index c64467c2ba..2c235d4ff2 100644 --- a/tests/unit/test_compiled_execution.py +++ b/tests/unit/test_compiled_execution.py @@ -56,6 +56,7 @@ compiled_context, executing_context, ) +from tests.run_records import minimal_run_declaration def _runtime_environment() -> CompiledRuntimeEnvironmentPlan: @@ -91,6 +92,7 @@ def resolve_for_transport(contexts): runtime_contexts={"A01": runtime_context}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert resolver_calls == [{"A01": runtime_context}] @@ -113,6 +115,7 @@ def test_compiled_execution_bundle_derives_axis_ids_from_contexts() -> None: transport_contexts={}, worker_assignments={}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert bundle.axis_ids == ("A01", "B01") @@ -152,6 +155,7 @@ def test_compiled_execution_bundle_derives_runtime_observation_mode( transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert RuntimeObservationMode.for_compiled_bundle(bundle) is expected_mode @@ -201,6 +205,7 @@ def test_consolidation_does_not_require_parent_payload_records( transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert bundle.requires_parent_runtime_observation is False diff --git a/tests/unit/test_function_outputs.py b/tests/unit/test_function_outputs.py index e1020afe26..274cb20461 100644 --- a/tests/unit/test_function_outputs.py +++ b/tests/unit/test_function_outputs.py @@ -74,6 +74,7 @@ RuntimeArtifactMaterialization, ) from openhcs.core.steps.function_artifact_materialization import ArtifactMaterializationTargetPlan +from openhcs.core.run_record import ProducedFiles from openhcs.core.steps.function_outputs import ( MaterializedImageOutputWriter, MemoryOutputWriter, @@ -169,12 +170,14 @@ def test_function_step_metadata_follows_runtime_artifact_persistence( "runtime_artifacts", ), (OpenHCSMetadataTarget, "observe_for_step", "metadata"), + (ProducedFiles, "stated_by_step", "producers"), ): monkeypatch.setattr( authority, method_name, lambda _context, _plan, event=event, **_kwargs: ( - events.append(event), {} if event == "metadata" else () + events.append(event), + {} if event in ("metadata", "producers") else (), )[1], ) @@ -186,6 +189,7 @@ def test_function_step_metadata_follows_runtime_artifact_persistence( "stream", "runtime_artifacts", "metadata", + "producers", ] diff --git a/tests/unit/test_function_step_transport.py b/tests/unit/test_function_step_transport.py index 58ba0af7d7..04c8239aca 100644 --- a/tests/unit/test_function_step_transport.py +++ b/tests/unit/test_function_step_transport.py @@ -49,6 +49,7 @@ from tests.context_support import ( compiled_context, ) +from tests.run_records import minimal_run_declaration def _execute_spawned_custom_invocation( @@ -458,6 +459,7 @@ def test_bundle_transport_preserves_prepared_runtime_contract_identity() -> None runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( GlobalPipelineConfig(), compiled_contexts=contexts, server_mode=False, ), + run_declaration=minimal_run_declaration(), ) transport_context = bundle.transport_contexts["A01"] transport_plan = transport_context.step_plans[0] diff --git a/tests/unit/test_orchestrator_lane_planning.py b/tests/unit/test_orchestrator_lane_planning.py index b464db5442..083a08f49f 100644 --- a/tests/unit/test_orchestrator_lane_planning.py +++ b/tests/unit/test_orchestrator_lane_planning.py @@ -84,6 +84,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration PROGRESS_CONTEXT = ProgressExecutionContext( execution_id="exec", @@ -248,6 +249,7 @@ def _execute_with_visualizer(monkeypatch, visualizer, *, progress_queue=None): transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ) request = BundleExecutionRequest( execution_id="exec", @@ -315,6 +317,7 @@ def release_parent_runtime_resources(self, _bundle): _state=None, is_initialized=lambda: True, microscope_handler=object(), + filemanager=object(), ) return execute_bundle_request(orchestrator, request) @@ -385,6 +388,7 @@ def test_compiled_plate_execution_request_uses_bundle_as_runtime_authority(): transport_contexts={"A01": context}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ) request = BundleExecutionRequest( execution_id="exec", @@ -422,6 +426,7 @@ def test_plate_scope_rejects_omitted_runtime_observations(monkeypatch): transport_contexts={"A01": context}, worker_assignments=None, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ) request = BundleExecutionRequest( execution_id="exec", @@ -539,6 +544,7 @@ def test_resource_modes_select_prepared_runtime_or_serial_transport_contexts(): use_threading=False, start_method=MultiprocessingStartMethod.SPAWN, ), + run_declaration=minimal_run_declaration(), ) cancellation = ExecutionCancellationSignal() resources = ( @@ -1026,6 +1032,7 @@ def fake_execute_worker_lane( use_threading=False, start_method=MultiprocessingStartMethod.FORK, ), + run_declaration=minimal_run_declaration(), ) ) try: diff --git a/tests/unit/test_run_record.py b/tests/unit/test_run_record.py new file mode 100644 index 0000000000..b1549ef6b1 --- /dev/null +++ b/tests/unit/test_run_record.py @@ -0,0 +1,560 @@ +"""DL3: an output dataset says what produced it. One test per rule.""" + +from __future__ import annotations + +import hashlib +import importlib.metadata +import json +import pickle +from pathlib import Path +from queue import Queue + +import inspect + +import pytest +from polystore.streaming.identity import StreamProducerIdentity + +import openhcs.processing.custom_functions.manager as manager_module +from openhcs.constants import Backend +from openhcs.core.callable_contract import CallableContract +from openhcs.core.config import LazyPathPlanningConfig, PipelineConfig +from openhcs.core.function_identity import ModuleLibraries +from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator +from openhcs.core.pipeline_document import PipelineDocumentCodec +from openhcs.core.progress import ProgressExecutionContext, set_progress_queue +from polystore.file_content import HashedFileContent, UnhashedFileContent + +from openhcs.core.run_record import ( + ConfigValueOwner, + DatasetRunRecords, + DeclaredFile, + FunctionDefault, + FunctionRecord, + InheritedFrom, + ProducedFiles, + RecordedParameter, + RunDeclaration, + RunRecord, + RunRecordWriter, + SetOnObject, + SuppliedAtRunTime, + UnrecordedOutputFile, + WrittenInPipeline, + python_expression, +) +from openhcs.runtime.environment_provenance import InstalledDistributionVersion +from openhcs.core.steps.function_step import FunctionStep +from openhcs.core.virtual_workspace_metadata import AtomicMetadataWriter +from openhcs.processing.backends.lib_registry.registry_service import RegistryService +from openhcs.processing.backends.processors.numpy_processor import ( + stack_percentile_normalize, +) +from openhcs.processing.custom_functions import CustomFunctionManager +from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, +) +from tests.run_records import minimal_run_record +from tests.unit.test_artifact_publication_journey import _plate, tiny_counts + +NORMALIZE_KEY = "openhcs:processors_numpy_processor_stack_percentile_normalize" + +COUNT_BRIGHT = '''import numpy as np + +from openhcs.core.artifacts.kinds import MeasurementsArtifactType +from openhcs.core.artifacts.relations import ArtifactMeasurementSubjectRelation +from openhcs.core.artifacts.specs import ArtifactSpec +from openhcs.core.measurement_row_materialization import MeasurementSparseColumnarRows +from openhcs.core.memory import numpy +from openhcs.core.pipeline.function_contracts import artifact_outputs +from openhcs.core.runtime_tabular_values import FieldSpec +from openhcs.core.vfs_protocol import PlateInputFile, PlateOutputFile + + +@numpy +@artifact_outputs( + ArtifactSpec.output( + "bright", + MeasurementsArtifactType, + relations=(ArtifactMeasurementSubjectRelation(),), + ) +) +def count_bright( + image, + threshold: float = THRESHOLD, + weights: PlateInputFile = None, + notes: PlateOutputFile = None, +): + """Count pixels above a threshold.""" + return image, MeasurementSparseColumnarRows.from_rows( + ({"bright": int(np.count_nonzero(image > threshold))},), + fields=(FieldSpec("bright", int),), + ) +''' + + +def count_bright_source(threshold: float) -> str: + return COUNT_BRIGHT.replace("THRESHOLD", repr(threshold)) + + +@pytest.fixture(autouse=True) +def progress_events(): + set_progress_queue(Queue()) + yield + set_progress_queue(None) + + +@pytest.fixture +def register(monkeypatch, tmp_path): + """Register custom source text the way the editor does, without persisting it.""" + storage_dir = tmp_path / "custom_functions" + storage_dir.mkdir() + monkeypatch.setattr( + manager_module, "get_data_file_path", lambda _name, *, create=True: storage_dir + ) + for attribute in ( + "_declarations_by_name", + "_published_exports", + "_preparation_outcomes", + "_preparation_threads", + ): + monkeypatch.setattr(CustomFunctionRuntimeRegistry, attribute, {}) + monkeypatch.setattr(CustomFunctionRuntimeRegistry, "_source_revision", None) + + def register_source(code: str): + CustomFunctionRuntimeRegistry.clear() + [function] = CustomFunctionManager().register_from_code( + code, persist=False, emit_signal=False + ) + return function + + yield register_source + CustomFunctionRuntimeRegistry.clear() + + +def orchestrator_for(plate: Path, **config_fields) -> PipelineOrchestrator: + return PipelineOrchestrator( + plate, + pipeline_config=PipelineConfig( + path_planning_config=LazyPathPlanningConfig(output_dir_suffix="_out"), + num_workers=1, + use_threading=True, + **config_fields, + ), + ).initialize() + + +def record_of(orchestrator: PipelineOrchestrator, steps: list) -> RunRecord: + return RunRecord.at_run_start( + orchestrator.compile_pipelines(steps), + filemanager=orchestrator.filemanager, + backend=Backend.DISK.value, + ) + + +def run(orchestrator: PipelineOrchestrator, steps: list, tmp_path: Path): + set_progress_queue(Queue()) + bundle = orchestrator.compile_pipelines(steps) + results = orchestrator.execute_compiled_bundle( + bundle, + progress_queue=Queue(), + progress_context=ProgressExecutionContext( + execution_id="run-record", subject_id=str(tmp_path) + ), + ) + assert all(result.is_success() for result in results.values()), { + axis: result.error_message for axis, result in results.items() + } + return bundle + + +def only_function(record: RunRecord, step_name: str) -> FunctionRecord: + (step,) = (step for step in record.steps if step.calls.name == step_name) + (function,) = step.calls.functions + return function + + +def parameters_by_name(function: FunctionRecord) -> dict[str, RecordedParameter]: + return {parameter.name: parameter for parameter in function.parameters} + + +def stored_records(dataset_root: Path) -> DatasetRunRecords: + return DatasetRunRecords.in_document( + json.loads((dataset_root / "openhcs_metadata.json").read_text()) + ) + + +def normalize_and_count() -> list: + return [ + FunctionStep(func=(stack_percentile_normalize, {"low_percentile": 2.0}), name="norm"), + FunctionStep(func=tiny_counts, name="count"), + ] + + +def test_parameter_left_out_is_recorded_with_the_function_default(tmp_path): + record = record_of(orchestrator_for(_plate(tmp_path / "source")), normalize_and_count()) + + function = only_function(record, "norm") + parameters = parameters_by_name(function) + assert function.identity.key == NORMALIZE_KEY + assert parameters["low_percentile"] == RecordedParameter( + "low_percentile", "2.0", WrittenInPipeline + ) + assert parameters["high_percentile"] == RecordedParameter( + "high_percentile", "99.0", FunctionDefault + ) + assert parameters["stack"] == RecordedParameter("stack", None, SuppliedAtRunTime) + + +def test_stored_json_reads_back_as_the_same_record(tmp_path, register): + """One encoder and one decoder: every type of the record survives the round trip.""" + plate = _plate(tmp_path / "source") + (plate / "weights.bin").write_bytes(b"weights") + record = record_of( + orchestrator_for(plate), + [ + FunctionStep(func=(stack_percentile_normalize, {"low_percentile": 2.0}), name="norm"), + FunctionStep( + func=( + register(count_bright_source(0.5)), + {"weights": "weights.bin", "notes": "notes/log.txt"}, + ), + name="bright", + ), + ], + ) + + stored = json.loads(json.dumps(record.to_json())) + + assert RunRecord.from_json(stored) == record + assert RunRecord.from_json(stored).record_id == record.record_id + (norm_parameter, *_rest) = stored["steps"][0]["calls"]["functions"][0]["parameters"] + assert norm_parameter == {"name": "stack", "value": None, "source": "run_time"} + # Each family member names its class by its registry key, not by its fields. + library, custom = (step["calls"]["functions"][0] for step in stored["steps"]) + assert list(library["identity"]) == ["installed_module"] + assert list(custom["identity"]) == ["source_text"] + assert [list(declared["content"]) for declared in custom["declared_files"]] == [ + ["hashed"], + ["not_hashed"], + ] + + +def test_saved_pipeline_text_stays_short_and_the_record_copy_is_complete(tmp_path): + steps = normalize_and_count() + record = record_of(orchestrator_for(_plate(tmp_path / "source")), steps) + + document = PipelineDocumentCodec.from_source(record.pipeline_code) + saved_text = PipelineDocumentCodec.render(document) + assert "low_percentile" in saved_text + assert "high_percentile" not in saved_text + assert "description=None" not in saved_text + submitted = PipelineDocumentCodec.from_values( + pipeline_config=PipelineConfig(), pipeline_steps=steps + ) + # Three renderings of one source, three promises: the saved text leaves out + # defaults; the unabridged text writes every stored field but no parameter + # the pipeline does not state; the record's copy leaves nothing to a default. + assert "high_percentile" not in PipelineDocumentCodec.render( + submitted, clean_mode=False + ) + assert "description=None" in PipelineDocumentCodec.render(submitted, clean_mode=False) + assert "high_percentile" in record.pipeline_code + assert "description=None" in record.pipeline_code + assert ( + PipelineDocumentCodec.render(document, clean_mode=False) + == record.pipeline_code + ) + + +def test_compiled_bundle_with_its_declaration_pickles(tmp_path): + bundle = orchestrator_for(_plate(tmp_path / "source")).compile_pipelines( + normalize_and_count() + ) + + restored = pickle.loads(pickle.dumps(bundle.for_transport_serialization())) + + assert restored.run_declaration == bundle.run_declaration + + +def test_custom_function_is_identified_by_its_source_text(tmp_path, register): + orchestrator = orchestrator_for(_plate(tmp_path / "source")) + records = {} + for threshold in (0.5, 0.25): + source = count_bright_source(threshold) + record = record_of( + orchestrator, [FunctionStep(func=register(source), name="bright")] + ) + identity = only_function(record, "bright").identity + sha256 = hashlib.sha256(source.encode("utf-8")).hexdigest() + assert identity.name == "count_bright" + assert identity.source_sha256 == sha256 + assert record.custom_sources == {sha256: source} + records[threshold] = record + + first, second = ( + only_function(record, "bright").identity for record in records.values() + ) + assert first.key == second.key + assert first.source_sha256 != second.source_sha256 + assert records[0.5].record_id != records[0.25].record_id + + +def test_custom_function_imports_name_installed_versions(tmp_path, register): + record = record_of( + orchestrator_for(_plate(tmp_path / "source")), + [FunctionStep(func=register(count_bright_source(0.5)), name="bright")], + ) + + libraries = only_function(record, "bright").identity.imports.libraries + assert ( + InstalledDistributionVersion("numpy", importlib.metadata.version("numpy")) + in libraries + ) + + +def test_module_without_a_distribution_is_listed_not_dropped(): + libraries = ModuleLibraries.imported_by_source( + "import json\nimport numpy\nimport module_no_distribution_provides.part\n" + ) + + assert libraries.modules_without_distribution == ("module_no_distribution_provides",) + assert [library.name for library in libraries.libraries] == ["numpy"] + + +def test_declared_input_file_is_hashed_once_per_run_by_the_file_manager( + tmp_path, register, monkeypatch +): + plate = _plate(tmp_path / "source") + weights = plate / "weights.bin" + weights.write_bytes(b"first weights") + orchestrator = orchestrator_for(plate) + hashed_paths: list[Path] = [] + file_content = orchestrator.filemanager.file_content + + def spy(file_path, backend): + hashed_paths.append(Path(file_path)) + return file_content(file_path, backend) + + monkeypatch.setattr(orchestrator.filemanager, "file_content", spy) + count_bright = register(count_bright_source(0.5)) + + def compiled(): + return orchestrator.compile_pipelines( + [ + FunctionStep( + func=(count_bright, {"weights": "weights.bin", "notes": "notes/log.txt"}), + name="bright", + ) + ] + ) + + def declared_files(bundle) -> dict[str, DeclaredFile]: + record = RunRecord.at_run_start( + bundle, filemanager=orchestrator.filemanager, backend=Backend.DISK.value + ) + return { + declared.parameter: declared + for declared in only_function(record, "bright").declared_files + } + + bundle = compiled() + assert hashed_paths == [] + + declared = declared_files(bundle) + assert hashed_paths == [weights] + assert declared["weights"] == DeclaredFile( + parameter="weights", + declared_as="input file", + path=str(weights), + content=HashedFileContent( + hashlib.sha256(b"first weights").hexdigest(), len(b"first weights") + ), + ) + assert declared["notes"].declared_as == "output file" + assert isinstance(declared["notes"].content, UnhashedFileContent) + assert declared["notes"].content.reason.strip() + + weights.write_bytes(b"second weights!") + assert declared_files(bundle)["weights"].content == HashedFileContent( + hashlib.sha256(b"second weights!").hexdigest(), len(b"second weights!") + ) + + +def test_resolved_configuration_names_the_source_of_each_value(tmp_path, monkeypatch): + resolved_pipelines = [] + with_resolved_steps = RunDeclaration.with_resolved_steps + + def keep_resolved_pipeline(declaration, pipeline): + resolved_pipelines.append(pipeline) + return with_resolved_steps(declaration, pipeline) + + monkeypatch.setattr(RunDeclaration, "with_resolved_steps", keep_resolved_pipeline) + plate = _plate(tmp_path / "source") + record = record_of(orchestrator_for(plate), normalize_and_count()) + + owner_by_path = { + path: (group.owner, value) + for group in record.configuration + for path, value in group.values.items() + } + assert owner_by_path["num_workers"] == ( + ConfigValueOwner(SetOnObject, str(plate), "openhcs.core.config.PipelineConfig"), + "1", + ) + inherited, _value = owner_by_path["materialization_results_path"] + assert inherited.source is InheritedFrom + assert inherited.type_name == "openhcs.core.config.GlobalPipelineConfig" + owners = [group.owner for group in record.configuration] + assert len(owners) == len(set(owners)), "each owner is written once" + + (pipeline,) = resolved_pipelines + assert len(record.steps) == len(pipeline.steps) == 2 + for index, step in enumerate(record.steps): + recorded_names = {path for group in step.configuration for path in group.values} + assert pipeline.step_provenance[index] + assert set(pipeline.step_provenance[index]) <= recorded_names + + +def test_each_output_file_names_the_step_that_produced_it(tmp_path): + plate = _plate(tmp_path / "source") + orchestrator = orchestrator_for(plate) + bundle = run(orchestrator, normalize_and_count(), tmp_path) + output_root = tmp_path / "source_out" + produced = stored_records(output_root).produced_files + (image,) = (path for path in produced if path.endswith(".tif")) + (table,) = (path for path in produced if path.endswith(".csv")) + + image_file = RunRecord.for_output_file(output_root / image) + table_file = RunRecord.for_output_file(output_root / table) + + # "count" returns its image unchanged, so the saved pixels are still the + # ones "norm" produced: the same producer a viewer item for them carries. + assert image_file.step.name == "norm" + assert table_file.step.name == "count" + assert table_file.record.record_id == image_file.record.record_id + (context,) = bundle.runtime_contexts.values() + count_plan = context.step_plans[1] + (output_plan,) = count_plan.artifact_outputs.values() + assert produced[table].producer == count_plan.producer_identity_for_artifact( + output_plan + ) + unlisted = output_root / "A01_s001_w1_z001_t001_unlisted.csv" + unlisted.write_text("not written by a step") + with pytest.raises(UnrecordedOutputFile): + RunRecord.for_output_file(unlisted) + + +def test_input_plate_metadata_receives_no_record(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + + document = json.loads((plate / "openhcs_metadata.json").read_text()) + assert not set(document) & {"run_records", "produced_files"} + assert stored_records(plate) == DatasetRunRecords() + + +def test_second_pipeline_into_the_same_dataset_keeps_only_referenced_records(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + run( + orchestrator_for(plate), + [FunctionStep(func=(stack_percentile_normalize, {"low_percentile": 5.0}), name="norm")], + tmp_path, + ) + + stored = stored_records(tmp_path / "source_out") + referenced = {entry.run_record for entry in stored.produced_files.values()} + assert set(stored.run_records) == referenced + assert len(referenced) == 2 + + +def _table_producer() -> StreamProducerIdentity: + return StreamProducerIdentity.pipeline_output( + output_kind="artifact", + output_key="counts", + projection_key="counts", + step_name="count", + pipeline_position=0, + step_scope_id="plate::pipeline::step_0", + artifact_kind="measurements", + ) + + +def test_dataset_root_that_received_only_a_table_gets_a_document_with_the_record(tmp_path): + results_root = tmp_path / "results_only" + table = results_root / "results" / "A01_counts.csv" + table.parent.mkdir(parents=True) + table.write_text("nonzero\n9\n") + record = minimal_run_record() + + RunRecordWriter().store(results_root, ProducedFiles({table: _table_producer()}), record) + + produced = RunRecord.for_output_file(table) + assert produced.record.record_id == record.record_id + assert produced.producer == _table_producer() + + +def test_saved_file_is_recorded_under_the_nearest_declared_dataset_root(tmp_path): + outer = tmp_path / "plate_out" + inner = outer / "nested_out" + image = outer / "images" / "a.tif" + table = inner / "results" / "a.csv" + produced = ProducedFiles({image: _table_producer(), table: _table_producer()}) + + by_root = produced.by_dataset_root([outer, inner]) + + assert {root: set(files.producer_by_path) for root, files in by_root.items()} == { + outer: {image}, + inner: {table}, + } + with pytest.raises(ValueError, match="no declared output dataset root"): + produced.by_dataset_root([inner]) + + +def test_subdirectory_metadata_updates_keep_the_records(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + metadata_path = tmp_path / "source_out" / "openhcs_metadata.json" + before = stored_records(metadata_path.parent) + assert before.run_records and before.produced_files + + writer = AtomicMetadataWriter() + writer.merge_subdirectory_metadata(metadata_path, {"added": {"note": "kept"}}) + writer.replace_subdirectory_metadata(metadata_path, "replaced", {"note": "kept"}) + + assert stored_records(metadata_path.parent) == before + + +def test_output_dataset_with_records_opens_as_an_input_plate(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + + set_progress_queue(Queue()) + reopened = orchestrator_for(tmp_path / "source_out") + + assert reopened.is_initialized() + record_of(reopened, normalize_and_count()) + + +def test_every_registered_function_default_can_be_written_in_the_record(): + """Compilation writes each default; one it cannot write would stop that pipeline.""" + unwritable = [] + for key, metadata in RegistryService.get_all_functions_with_metadata().items(): + contract = CallableContract.from_callable(metadata.func) + step = FunctionStep(func=metadata.func, name=key) + try: + PipelineDocumentCodec.render_with_every_parameter( + PipelineDocumentCodec.from_values( + pipeline_config=PipelineConfig(), pipeline_steps=[step] + ) + ) + for name, parameter in contract.canonical_signature.parameters.items(): + if ( + parameter.default is not inspect.Parameter.empty + and name not in contract.runtime_owned_parameter_names + ): + python_expression(parameter.default) + except TypeError as error: + unwritable.append((key, str(error))) + + assert unwritable == [] diff --git a/tests/unit/test_run_record_capability.py b/tests/unit/test_run_record_capability.py new file mode 100644 index 0000000000..4848dc7d34 --- /dev/null +++ b/tests/unit/test_run_record_capability.py @@ -0,0 +1,236 @@ +"""openhcs_read_run_record reads only what the dataset's metadata document stores.""" + +import hashlib +import json +from types import SimpleNamespace + +import pytest +from polystore.file_content import HashedFileContent +from polystore.streaming.identity import StreamProducerIdentity +from python_introspect import to_jsonable + +from openhcs.agent.capabilities import ReadRunRecordCapability +from openhcs.agent.path_policy import AgentPathPolicy +from openhcs.agent.services.run_record_service import RunRecordService +from openhcs.core.function_identity import CustomFunctionIdentity, ModuleLibraries +from openhcs.core.run_record import ( + ConfigValueOwner, + ConfigValuesFromOwner, + DatasetRunRecords, + DeclaredFile, + FunctionRecord, + RecordedParameter, + RunRecord, + SetOnObject, + SoftwareEnvironment, + StepCalls, + StepRecord, + SuppliedAtRunTime, + WrittenInPipeline, +) +from openhcs.mcp.execution import McpTransportExecutor +from openhcs.mcp.server import build_server + +CUSTOM_SOURCE = "def count(image):\n return image\n" +CUSTOM_SOURCE_SHA256 = hashlib.sha256(CUSTOM_SOURCE.encode("utf-8")).hexdigest() +NO_LIBRARIES = ModuleLibraries((), ()) +STEP = StepCalls( + name="count", + pipeline_position=0, + step_scope_id="s::step_0", + input_memory_type="numpy", + output_memory_type="numpy", + assigned_gpu_devices=(), + functions=( + FunctionRecord( + identity=CustomFunctionIdentity( + key="openhcs:count", + module="openhcs.processing.custom_functions", + name="count", + defined_in=NO_LIBRARIES, + imports=NO_LIBRARIES, + source_sha256=CUSTOM_SOURCE_SHA256, + ), + group="None", + position=0, + input_memory_type="numpy", + output_memory_type="numpy", + parameters=( + RecordedParameter("image", None, SuppliedAtRunTime), + RecordedParameter("weights", "'weights.bin'", WrittenInPipeline), + ), + declared_files=( + DeclaredFile( + parameter="weights", + declared_as="input file", + path="/data/weights.bin", + content=HashedFileContent("0" * 64, 7), + ), + ), + ), + ), +) +RECORD = RunRecord( + software=SoftwareEnvironment( + openhcs="0.0.0", + python="3.12.0", + python_implementation="CPython", + platform="linux", + machine="x86_64", + installed_distributions={"numpy": "2.0.0", "polystore": "1.0.0"}, + ), + pipeline_code="pipeline = []\n" * 40, + global_config_code="config = None\n", + configuration=( + ConfigValuesFromOwner( + ConfigValueOwner(SetOnObject, "s", "openhcs.core.config.PipelineConfig"), + {"num_workers": "1"}, + ), + ), + steps=(StepRecord(STEP, ()),), + custom_sources={CUSTOM_SOURCE_SHA256: CUSTOM_SOURCE}, +) +RECORD_ID = RECORD.record_id +PRODUCER = StreamProducerIdentity.pipeline_output( + output_kind="artifact", + output_key="counts", + projection_key="counts", + step_name="count", + pipeline_position=0, + step_scope_id="s::step_0", +) + + +def as_json(value): + return json.loads(json.dumps(to_jsonable(value))) + + +@pytest.fixture +def dataset(tmp_path): + root = tmp_path / "dataset" + (root / "results").mkdir(parents=True) + (root / "results" / "a.csv").write_text("n\n1\n", encoding="utf-8") + (root / "results" / "unlisted.csv").write_text("n\n2\n", encoding="utf-8") + (root / "openhcs_metadata.json").write_text( + json.dumps( + { + "subdirectories": {}, + **to_jsonable( + DatasetRunRecords().with_run(RECORD, {"results/a.csv": PRODUCER}) + ), + } + ), + encoding="utf-8", + ) + return root + + +@pytest.fixture +def read(tmp_path): + service = RunRecordService( + AgentPathPolicy.with_roots(readable_roots=(tmp_path,), writable_roots=()) + ) + executor = McpTransportExecutor() + built = build_server( + SimpleNamespace(run_record_service=service), + main_thread_dispatcher=executor.dispatcher, + ) + tool = built._tool_manager.get_tool(ReadRunRecordCapability.name) + + def call(**arguments): + async def exercise(): + return await tool.fn(**arguments) + + return executor.run(exercise) + + yield call + executor.close() + + +def test_dataset_root_returns_what_each_step_ran_by_default(dataset, read): + result = read(path=str(dataset)) + + assert result["errors"] == [] + assert result["produced_files"] == [] + assert result["records"] == [ + {"record_id": RECORD_ID, "steps": [as_json(STEP)], "record": None} + ] + (function,) = result["records"][0]["steps"][0]["functions"] + # A family member is stored under the key its class is registered with. + assert function["identity"]["source_text"]["source_sha256"] == CUSTOM_SOURCE_SHA256 + assert function["parameters"][1] == { + "name": "weights", + "value": "'weights.bin'", + "source": "pipeline", + } + assert function["declared_files"][0]["content"] == { + "hashed": {"sha256": "0" * 64, "size_bytes": 7} + } + + +def test_full_record_is_returned_on_request_and_reads_back(dataset, read): + result = read(path=str(dataset), include_full_record=True) + + (entry,) = result["records"] + assert entry["steps"] == [as_json(STEP)] + assert RunRecord.from_json(entry["record"]) == RECORD + + +@pytest.mark.parametrize("include_full_record", (False, True)) +def test_output_file_returns_its_producer_step_and_record( + dataset, read, include_full_record +): + output_file = dataset / "results" / "a.csv" + + result = read(path=str(output_file), include_full_record=include_full_record) + + assert result["errors"] == [] + assert result["produced_files"] == [ + { + "path": str(output_file), + "record_id": RECORD_ID, + "producer": as_json(PRODUCER), + "step": as_json(STEP), + } + ] + (entry,) = result["records"] + assert entry["record_id"] == RECORD_ID + assert (entry["record"] == as_json(RECORD)) is include_full_record + + +def test_unlisted_file_is_an_error_not_a_guess(dataset, read): + unlisted = dataset / "results" / "unlisted.csv" + + result = read(path=str(unlisted)) + + assert result["ok"] is False + assert result["tool"] == "openhcs_read_run_record" + (error,) = result["errors"] + assert error["code"] == "run_record_output_file_unrecorded" + assert error["message"] == ( + f"{unlisted} is not recorded as a step output in " + f"{dataset / 'openhcs_metadata.json'}." + ) + assert "records" not in result + + +def test_path_outside_readable_roots_is_rejected(tmp_path, read): + result = read(path=str(tmp_path.parent)) + + assert result["errors"][0]["code"] == "agent_path_policy_rejected" + + +def test_cli_command_is_generated_from_the_declaration(): + import openhcs.mcp.dev_client as dev_client + + parser = dev_client._build_parser() + call = dev_client._calls_from_args( + parser.parse_args(("read-run-record", "/data/out", "--include-full-record")) + )[0] + + assert call.name == "openhcs_read_run_record" + assert call.arguments == {"path": "/data/out", "include_full_record": True} + default_call = dev_client._calls_from_args( + parser.parse_args(("read-run-record", "/data/out")) + )[0] + assert default_call.arguments == {"path": "/data/out", "include_full_record": False} diff --git a/tests/unit/test_zmq_compilation.py b/tests/unit/test_zmq_compilation.py index 6fcc024292..86b8db1340 100644 --- a/tests/unit/test_zmq_compilation.py +++ b/tests/unit/test_zmq_compilation.py @@ -23,6 +23,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration class _StrippedStepShell: @@ -164,6 +165,7 @@ def test_reused_compile_artifact_reads_step_names_from_compiled_plans() -> None: transport_contexts={}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) compilation = ZMQCompilationResult(execution_bundle=bundle) artifacts = { @@ -217,6 +219,7 @@ def test_compile_fresh_emits_heartbeat_during_long_compilation() -> None: transport_contexts={}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) orchestrator = _FreshCompileOrchestrator(bundle) @@ -277,6 +280,7 @@ def test_rejected_reuse_preserves_artifact_for_a_valid_request(retain, invalid): transport_contexts={}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) artifact = ZMQCompileArtifactRecord( configs=OpenHCSExecutionConfigBundle(GlobalPipelineConfig(), PipelineConfig()), diff --git a/tests/unit/test_zmq_execution_server.py b/tests/unit/test_zmq_execution_server.py index a45b6bcedc..e55c02c07b 100644 --- a/tests/unit/test_zmq_execution_server.py +++ b/tests/unit/test_zmq_execution_server.py @@ -61,6 +61,7 @@ executing_context, with_facts, ) +from tests.run_records import minimal_run_declaration def test_zmq_execution_context_seeds_saved_global_config_for_compilation() -> None: @@ -416,6 +417,7 @@ def test_zmq_server_admits_compiled_declaration_without_reevaluating_source( compiled_contexts={}, server_mode=True, ), + run_declaration=minimal_run_declaration(), ) server._compiled_artifacts["compile-1"] = ZMQCompileArtifactRecord( execution_id="compile-1", @@ -700,6 +702,7 @@ def test_compiled_source_adoption_owns_fresh_runtime_services_and_live_source_ga compiled_contexts={"A01": context}, server_mode=True, ), + run_declaration=minimal_run_declaration(), ) compiled_zarr = context.filemanager.registry[Backend.ZARR.value] interleaved = PipelineOrchestrator(