From 7ea5440e18a9daf437d9c2db5933d90a21b2cdbc Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 14:32:22 -0400 Subject: [PATCH 01/12] DL3 receipt: design for the record of what produced an output dataset Co-Authored-By: Claude Opus 5.5 --- .../DL3-output-provenance.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 docs/refactor/consolidation-20261009/DL3-output-provenance.md diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md new file mode 100644 index 000000000..214d5e64b --- /dev/null +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -0,0 +1,144 @@ +# DL3: an output dataset says what produced it + +Branch `refactor/DL3-output-provenance`. Gap 6 of `D1-cellpose-n2v2-custom-functions.md` (PR #1230). + +## What is wrong (measured on `origin/main` at `1b5fe99f5`) + +- `openhcs_metadata.json` of an output dataset has no pipeline, function, parameter, source-hash + or model-hash key (keys of a real output: `subdirectories..{workspace_mapping, + source_metadata, source_projection, image_files, ...}`; `OpenHCSMetadataFields`, + `openhcs/core/virtual_workspace_metadata.py:325`). +- The pipeline code that reaches the server is rendered with `clean_mode=True` + (`openhcs/core/pipeline_document.py:96`); function kwargs equal to the signature default are + dropped (`openhcs/serialization/pycodify_formatters.py:150`). No non-GUI caller passes `False`. +- A declared input file is resolved to an absolute path and checked to exist + (`CompilationPathResolver.resolve`, `openhcs/core/pipeline/compilation_session.py:102`); no + content identity is taken. PolyStore hashes only to verify a caller-supplied hash + (`DataSource.read_verified_source_bytes`, `external/PolyStore/src/polystore/base.py:366`). +- Config resolution already returns, per field, the scope and class that supplied the value + (`ObjectState.resolve_saved_object`, used at `compiler.py:923`), and + `ResolvedPipelineDefinition.step_provenance` keeps it for steps. The compiler reads one field of + it (`compiler.py:1727`) and the orchestrator discards it for the pipeline config + (`orchestrator.py:1057`). Nothing writes it out. +- The producer of an output is stated in exactly one typed place: + `CompiledStepPlan.producer_identity(...)` (`compiled_step_plan.py:386`) builds a + `StreamProducerIdentity` (step name, pipeline position, step scope id, output kind and key). It is + sent with every viewer item (`viewer_transport.py:343`). On disk only its `output_key` survives + (`function_outputs.py:869`), plus a `_step` token in artifact file names + (`path_planner.py:2289`). Measurement tables, ROI archives and label images are not listed in the + metadata document at all. +- Function identity: `FunctionMetadata.composite_key` (`registry:name`); the library version is + known per registry only (`LibraryRegistryBase.get_library_version`); a custom function's source + hash is stamped on the callable (`declaration_revision`) but its text is not kept with it. + +## Design + +Standard used: P10 Definition 2.1(c) and Proposition 6.1 (the output carries the declaration that +produced it), P9 Theorem 5.3 (a log or private attribute does not make the source recoverable; it +has to be in the declared output), P9 "Several fact versions" (evidence that crosses versions +carries the version), P9 "Only executed, contract-guaranteed channels count" (always written, no +opt-in), P9 Theorem 5.5 (no parallel side table once the record determines the source). + +**One record, one writer, one time.** The compiler builds a `RunRecord` for each compilation, from +the facts only it holds, and puts it on the `CompiledExecutionBundle`. The parent process fills in +the content of declared input files when execution starts, and the existing single writer of the +output dataset's metadata, `OpenHCSMetadataTarget.finalize_completed_dataset` (parent process, one +locked transaction per `openhcs_metadata.json`, also on failure), stores it in that document: + +``` +openhcs_metadata.json + subdirectories: {...} (unchanged) + run_records: {: RunRecord} record id = sha256 of the record's canonical JSON + produced_files: {: {"run_record": , "producer": {...}}} +``` + +A record no file refers to any more is dropped in the same transaction, so a dataset written twice +keeps exactly the records its files need. + +**What the record holds (sufficient to re-run).** +- `pipeline_code`: the pipeline document rendered complete (`clean_mode=False`: every function + parameter and every stored config field is written). `global_config_code`: the saved global + config rendered the same way. These are the *stored documents*; rebuilding executes them. +- `configuration`: the *resolved* configuration, one entry per leaf field of the effective config + and of each step: the value (as its Python expression, from pycodify) and its source: the scope + id and class ObjectState reported, or "set here" when the object itself holds the value. This is + a different read kind from the document and is kept beside it, never merged into it. +- `software`: OpenHCS version, Python version, platform. +- `steps[]`: name, pipeline position, step scope id (the two fields a producer identity carries), + declared input and output memory type, the assigned device per framework, and `functions[]`. +- `functions[]`: identity (below), `parameters` and `declared_files`. +- `custom_sources`: `{sha256: source text}`, each text once. + +**Function identity is content.** The function reference family answers it +(`FunctionReference.identity_record`): the registry key or import path, the distribution and version +of the module that defines the function, and the distributions and versions of the modules that +module imports (this is where Cellpose's version comes from for a custom function: its source's +imports). A custom function adds the sha256 of its source; the text is stored once under +`custom_sources`. The text is kept on the source's own namespace object +(`CustomFunctionSourceNamespace`), which already owns the hash. + +**Defaults are recorded.** Each parameter of the function's signature appears in `parameters` with +its effective value and a source: `pipeline` (written in the pipeline), `function_default` (taken +from the signature of this function at the recorded version), or `run_time` (supplied by the +platform: an artifact input, a bound config value). The source is a small family of classes; its +JSON `kind` is derived from them. The editable pipeline the user saves is not changed: it stays +concise (`clean_mode=True`). Only the record's copy is rendered complete. + +**Declared files are identified by the storage layer.** PolyStore gains +`DataSource.file_content(path) -> FileContent`, a family: `HashedFileContent(sha256, size_bytes)` +or `UnhashedFileContent(reason)`; `FileManager.file_content(path, backend)` delegates to the +backend. The path declaration family says what its kind needs +(`DatasetPathDeclaration.content_at_run_start`): an input file is hashed; an input directory, an +output file and an output directory are recorded as not hashed, each with its reason. Hashing +happens once per file per run, in the parent process when execution starts (not at compile: a +compiled plan can be executed later, and the bytes that matter are the ones the run reads). At +the end of the run the writer asks again for size and modification time; a file that changed +during the run is recorded as `changed_during_run`. Functions hash nothing. + +Cost, measured here on a 1 GiB file (`.claude/scratch/hashcost.py`, NVMe, CPU with SHA +extensions): sha256 0.58 s with the file in the page cache, 2.15 s after dropping it; blake2b +1.03 s, md5 1.22 s. That is cheap enough to be always on, so there is no switch. A backend with no +physical file answers `UnhashedFileContent` with its reason; the record never omits the entry. + +**Per-output linkage is the existing producer identity, written down.** No second linkage: the +`StreamProducerIdentity` a viewer item already carries is the value stored under +`produced_files[path].producer`. Workers already report what they saved +(`StepExecutionObservation`); that report gains the producer of each saved path, taken from the +same two calls the streaming path uses (`producer_identity_for_main_flow`, +`producer_identity_for_artifact`). The reader matches a producer to its step by step scope id and +pipeline position. A measurement row is linked through its table file; a viewer item through the +identity it arrived with. The `_step` file-name token is not read by anything here. + +**One reader.** `RunRecord.for_dataset(root)`, `RunRecord.for_output_file(path)` (walks up to the +dataset's metadata document; returns the record and the step) and `RunRecord.rebuild()` (registers +the recorded custom sources after checking their hashes, executes the two documents, returns the +pipeline document and global config). The agent capability `openhcs_read_run_record` (MCP tool and +generated CLI command) wraps the same calls. + +**Dependencies on sibling branches.** +- DL1 (device and memory types): the device is read from `CompiledStepPlan.device_assignment`, the + place it is stated at this branch point. If DL1 moves or extends that statement (CPU devices, + conversions), the record follows by reading the new one. +- DL2 (declared files between steps): an input file an earlier step of the same run writes does + not exist when the run starts. At this branch point compile rejects that pipeline; when DL2 + lands, such a file must be hashed after its producer finishes. `content_at_run_start` is the + single place that decision is made. + +## Guards and tests + +Unit (`tests/unit/test_run_record.py`): default parameters appear with `function_default`; a custom +function's record changes when its body changes and its name does not; a declared input file is +hashed by the file manager and a declared output file is recorded as not hashed with a reason; the +resolved configuration names the scope and class of an inherited value; the record id is stable +for equal input; a file with no entry is reported as unrecorded, not guessed from its name. +Integration (`tests/integration/test_run_record_round_trip.py`): run a small pipeline, read the +record from one output file, rebuild, run again, compare outputs byte for byte. + +## Done when + +The Cellpose example leaves a record with the custom source hash, every parameter, Cellpose's +version and the device; the N2V2 denoise example records the model file's hash; gates pass. + +## Progress notes + +(filled in as work proceeds) From 1786bbc136f19538f45e32f532dc4fab81cf108d Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 14:38:42 -0400 Subject: [PATCH 02/12] WIP DL3: function identity, declared-file content, run record model Function references and registry entries state where a function comes from (distribution, version, imports; source hash for custom functions). Declared path kinds say what content a run reads. RunRecord builds, reads and rebuilds the record. PolyStore 0.6.3 (file_content) pinned. Not wired into the compiler, the workers or the metadata writer yet. Co-Authored-By: Claude Opus 5.5 --- external/PolyStore | 2 +- openhcs/core/function_identity.py | 161 +++++ openhcs/core/function_reference.py | 20 + openhcs/core/run_record.py | 588 ++++++++++++++++++ openhcs/core/vfs_protocol.py | 74 ++- .../backends/lib_registry/unified_registry.py | 11 + .../processing/custom_functions/manager.py | 2 +- .../custom_functions/runtime_registry.py | 13 + .../custom_functions/source_namespace.py | 17 + pyproject.toml | 2 +- .../unit/test_catalog_startup_gc_lifetime.py | 9 +- 11 files changed, 893 insertions(+), 6 deletions(-) create mode 100644 openhcs/core/function_identity.py create mode 100644 openhcs/core/run_record.py diff --git a/external/PolyStore b/external/PolyStore index d9aed3787..8ee8ce44a 160000 --- a/external/PolyStore +++ b/external/PolyStore @@ -1 +1 @@ -Subproject commit d9aed378736bee67cbcba75cb66c0a25074b876a +Subproject commit 8ee8ce44a1dfc3dc59433584d1d19952231f5815 diff --git a/openhcs/core/function_identity.py b/openhcs/core/function_identity.py new file mode 100644 index 000000000..fb4137307 --- /dev/null +++ b/openhcs/core/function_identity.py @@ -0,0 +1,161 @@ +"""What a processing function is, stated as content: where it comes from and at which version. + +A function's name can stay while its body, its library or its defaults change. +The identity here is what a record of a run stores so that a later reader can +tell which code answered. +""" + +from __future__ import annotations + +import ast +import importlib.metadata +import importlib.util +import sys +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path + +from openhcs.runtime.environment_provenance import InstalledDistributionVersion + + +@lru_cache(maxsize=1) +def _distributions_by_top_level_module() -> dict[str, tuple[str, ...]]: + return { + module: tuple(sorted(set(distributions))) + for module, distributions in importlib.metadata.packages_distributions().items() + } + + +@dataclass(frozen=True, slots=True) +class ModuleLibraries: + """The distributions behind a set of imported modules. + + A module that belongs to no installed distribution and is not part of the + standard library is listed by name, never dropped: the reader must be able + to see that its version is unknown. + """ + + libraries: tuple[InstalledDistributionVersion, ...] + modules_without_distribution: tuple[str, ...] + + @classmethod + def of_modules(cls, module_names: frozenset[str]) -> "ModuleLibraries": + by_module = _distributions_by_top_level_module() + libraries: set[InstalledDistributionVersion] = set() + unknown: set[str] = set() + for top_level in {name.partition(".")[0] for name in module_names}: + if top_level in sys.stdlib_module_names: + continue + distributions = by_module.get(top_level, ()) + if not distributions: + unknown.add(top_level) + libraries.update( + InstalledDistributionVersion( + distribution, importlib.metadata.version(distribution) + ) + for distribution in distributions + ) + return cls( + tuple(sorted(libraries, key=lambda item: (item.name.casefold(), item.version))), + tuple(sorted(unknown)), + ) + + @classmethod + def imported_by_source(cls, source_text: str) -> "ModuleLibraries": + """Every absolute import in the source, wherever it is written.""" + imported: set[str] = set() + for node in ast.walk(ast.parse(source_text)): + if isinstance(node, ast.Import): + imported.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + imported.add(node.module) + return cls.of_modules(frozenset(imported)) + + @classmethod + def imported_by_module(cls, module_name: str) -> "ModuleLibraries": + """The imports of an importable module, read from its source file.""" + spec = importlib.util.find_spec(module_name) + origin = None if spec is None else spec.origin + if origin is None or not origin.endswith(".py"): + return cls.of_modules(frozenset()) + return cls.imported_by_source(Path(origin).read_text(encoding="utf-8")) + + def to_record(self) -> dict[str, object]: + return { + "libraries": [ + {"distribution": library.name, "version": library.version} + for library in self.libraries + ], + "modules_without_distribution": list(self.modules_without_distribution), + } + + +@dataclass(frozen=True, slots=True) +class FunctionIdentity: + """A function exported by an installed module. + + ``key`` is the registry key (``registry:name``) or the import path + (``python:module:name``); ``defined_in`` is the distribution and version of + the module that defines it; ``imports`` are the distributions that module + imports. + """ + + key: str + module: str + name: str + defined_in: ModuleLibraries + imports: ModuleLibraries + + @classmethod + def of_module_export(cls, *, key: str, module: str, name: str) -> "FunctionIdentity": + return cls( + key=key, + module=module, + name=name, + defined_in=ModuleLibraries.of_modules(frozenset({module})), + imports=ModuleLibraries.imported_by_module(module), + ) + + def custom_sources(self) -> dict[str, str]: + """Source texts this identity refers to by hash; none for an installed module.""" + return {} + + def to_record(self) -> dict[str, object]: + return { + "key": self.key, + "module": self.module, + "name": self.name, + "defined_in": self.defined_in.to_record(), + "imports": self.imports.to_record(), + } + + +@dataclass(frozen=True, slots=True) +class CustomFunctionIdentity(FunctionIdentity): + """A function defined by user-written source text, identified by that text's hash.""" + + source_sha256: str + source_text: str + + @classmethod + def of_source( + cls, *, key: str, module: str, name: str, source_sha256: str, source_text: str + ) -> "CustomFunctionIdentity": + return cls( + key=key, + module=module, + name=name, + defined_in=ModuleLibraries.of_modules(frozenset()), + imports=ModuleLibraries.imported_by_source(source_text), + source_sha256=source_sha256, + source_text=source_text, + ) + + def custom_sources(self) -> dict[str, str]: + return {self.source_sha256: self.source_text} + + def to_record(self) -> dict[str, object]: + return { + **FunctionIdentity.to_record(self), + "custom_source_sha256": self.source_sha256, + } diff --git a/openhcs/core/function_reference.py b/openhcs/core/function_reference.py index 3f08f3a0d..0bad03ba9 100644 --- a/openhcs/core/function_reference.py +++ b/openhcs/core/function_reference.py @@ -21,6 +21,7 @@ FunctionStepExecutionScope, ) from openhcs.core.function_contract_metadata import FunctionContractAttribute +from openhcs.core.function_identity import FunctionIdentity if TYPE_CHECKING: from openhcs.core.steps.abstract import AbstractStep @@ -86,6 +87,10 @@ def source_fragment(self, context: FormatContext) -> SourceFragment: def resolve(self) -> Callable: """Resolve this reference through its nominal transport authority.""" + @abstractmethod + def identity(self) -> FunctionIdentity: + """Where the referenced function comes from and at which version.""" + def require_current_declaration(self, resolved: Callable) -> Callable: """Return ``resolved`` only when its source proof matches compilation.""" @@ -122,6 +127,13 @@ def resolve(self) -> Callable: "is not importable in this process." ) + def identity(self) -> FunctionIdentity: + return FunctionIdentity.of_module_export( + key=self.composite_key, + module=self.original_module, + name=self.function_name, + ) + @dataclass(frozen=True, kw_only=True) class RegistryFunctionReference(FunctionReference): @@ -161,6 +173,14 @@ def resolve(self) -> Callable: return RegistryService.resolve_function_reference(self) + def identity(self) -> FunctionIdentity: + """The registry entry answers: a custom entry is identified by its source.""" + from openhcs.processing.backends.lib_registry.registry_service import ( + RegistryService, + ) + + return RegistryService.metadata_for_canonical_key(self.composite_key).identity() + @dataclass(frozen=True, kw_only=True) class ModuleExportRegistryFunctionReference( diff --git a/openhcs/core/run_record.py b/openhcs/core/run_record.py new file mode 100644 index 000000000..d4a23d3fa --- /dev/null +++ b/openhcs/core/run_record.py @@ -0,0 +1,588 @@ +"""The record of what produced an output dataset. + +Three stages, each with one owner: + +* ``RunDeclaration``: what only the compiler holds (the stored documents and + the resolved configuration with the source of every value). It travels on + the ``CompiledExecutionBundle``. +* ``RunRecord.at_run_start``: the parent process combines the declaration with + the compiled step plans and asks the storage layer what each declared input + file holds. +* ``ProducedFiles`` and ``RunRecord`` are written into the output dataset's + ``openhcs_metadata.json`` by its one writer and read back from there. + +Two read kinds are kept apart on purpose. ``pipeline_code`` and +``effective_config_code`` are stored documents: executing them rebuilds the +run. ``configuration``, ``steps[].configuration`` and ``parameters`` are the +resolved values, each with the declaration that supplied it. +""" + +from __future__ import annotations + +import hashlib +import inspect +import json +from abc import ABC, abstractmethod +from collections.abc import Iterable, Iterator, Mapping, Sequence +from dataclasses import dataclass, fields, is_dataclass, replace +from pathlib import Path +from types import MappingProxyType +from typing import TYPE_CHECKING, Any, ClassVar + +import pycodify +from polystore.streaming.identity import StreamProducerIdentity + +from openhcs.core.vfs_protocol import DatasetPathDeclaration, FileManagerLike + +if TYPE_CHECKING: + from openhcs.core.compiled_execution import CompiledExecutionBundle + from openhcs.core.compiled_step_plan import CompiledStepPlan + from openhcs.core.config import GlobalPipelineConfig + from openhcs.core.function_patterns import CompiledFunctionInvocation + from openhcs.core.pipeline.compilation_session import ResolvedPipelineDefinition + from pycodify.documents import PipelineDocument + +RUN_RECORDS_KEY = "run_records" +PRODUCED_FILES_KEY = "produced_files" +JsonRecord = dict[str, Any] + + +def python_expression(value: object) -> str: + """Write a value the way generated pipeline code writes it.""" + import openhcs.serialization.pycodify_formatters # noqa: F401 (registers formatters) + + return pycodify.to_source(value, pycodify.FormatContext(clean_mode=False)).code + + +class ValueSource(ABC): + """Which declaration supplied a recorded value.""" + + kind: ClassVar[str] + + @abstractmethod + def details(self) -> JsonRecord: + """The fields that identify the supplying declaration.""" + + def to_record(self) -> JsonRecord: + return {"kind": self.kind, **self.details()} + + +@dataclass(frozen=True, slots=True) +class WrittenInPipeline(ValueSource): + """The pipeline document states the value.""" + + kind: ClassVar[str] = "pipeline" + + def details(self) -> JsonRecord: + return {} + + +@dataclass(frozen=True, slots=True) +class FunctionDefault(ValueSource): + """The pipeline leaves the parameter out; the function's signature supplies it.""" + + kind: ClassVar[str] = "function_default" + function_key: str + + def details(self) -> JsonRecord: + return {"function": self.function_key} + + +@dataclass(frozen=True, slots=True) +class SuppliedAtRunTime(ValueSource): + """The platform passes the value when it calls the function.""" + + kind: ClassVar[str] = "run_time" + + def details(self) -> JsonRecord: + return {} + + +@dataclass(frozen=True, slots=True) +class SetOnObject(ValueSource): + """The configured object itself holds the value.""" + + kind: ClassVar[str] = "set_here" + scope_id: str | None + + def details(self) -> JsonRecord: + return {"scope": self.scope_id} + + +@dataclass(frozen=True, slots=True) +class InheritedFrom(ValueSource): + """ObjectState resolved the value from this scope and class.""" + + kind: ClassVar[str] = "inherited" + scope_id: str | None + source_type: type | None + + def details(self) -> JsonRecord: + source_type = self.source_type + return { + "scope": self.scope_id, + "type": ( + None + if source_type is None + else f"{source_type.__module__}.{source_type.__qualname__}" + ), + } + + +@dataclass(frozen=True, slots=True) +class RecordedValue: + """One resolved value and the declaration that supplied it.""" + + name: str + expression: str + source: ValueSource + + def to_record(self) -> JsonRecord: + return { + "name": self.name, + "value": self.expression, + "source": self.source.to_record(), + } + + +ObjectStateProvenance = Mapping[str, tuple[str | None, type | None]] + + +def _declared_names(configured: object) -> tuple[str, ...]: + """The names a configured object is constructed from.""" + if is_dataclass(configured): + return tuple(item.name for item in fields(configured)) + return tuple( + name + for name, parameter in inspect.signature(type(configured)).parameters.items() + if parameter.kind + not in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD) + ) + + +def resolved_values( + resolved: object, + provenance: ObjectStateProvenance, + *, + scope_id: str | None, + prefix: str = "", +) -> Iterator[RecordedValue]: + """Every leaf value of a resolved object, with the source ObjectState reported. + + ObjectState lists the fields it inherited; a field it does not list is held + by the object itself. A nested config that was inherited whole is one value. + """ + for name in _declared_names(resolved): + path = f"{prefix}{name}" + value = getattr(resolved, name) + inherited = provenance.get(path) + if inherited is not None: + yield RecordedValue(path, python_expression(value), InheritedFrom(*inherited)) + elif is_dataclass(value) and not isinstance(value, type) and fields(value): + yield from resolved_values( + value, provenance, scope_id=scope_id, prefix=f"{path}." + ) + else: + yield RecordedValue(path, python_expression(value), SetOnObject(scope_id)) + + +@dataclass(frozen=True, slots=True) +class RunDeclaration: + """What the compiler alone knows about a run: documents and resolved configuration.""" + + pipeline_code: str + effective_config_code: str + configuration: tuple[RecordedValue, ...] + step_configurations: Mapping[int, tuple[RecordedValue, ...]] = MappingProxyType({}) + + @classmethod + def before_step_resolution( + cls, + orchestrator: Any, + pipeline_definition: Sequence[Any], + ) -> "RunDeclaration": + """Render the documents as submitted, before the compiler resolves the steps in place.""" + from objectstate import ObjectState, ObjectStateRegistry + + from openhcs.core.config import GlobalPipelineConfig + from openhcs.core.config_document import ConfigDocumentCodec + from openhcs.core.pipeline_document import PipelineDocumentCodec + + effective_config, provenance = ObjectState.resolve_saved_object( + orchestrator.pipeline_config, + ancestor_objects_with_scopes=( + ObjectStateRegistry.get_ancestor_objects_with_scopes(None, use_saved=True) + ), + ) + return cls( + pipeline_code=PipelineDocumentCodec.render( + PipelineDocumentCodec.from_values( + pipeline_config=orchestrator.pipeline_config, + pipeline_steps=pipeline_definition, + ), + clean_mode=False, + ), + effective_config_code=ConfigDocumentCodec.render( + effective_config, + expected_config_type=GlobalPipelineConfig, + clean_mode=False, + ), + configuration=tuple( + resolved_values( + effective_config, + provenance, + scope_id=str(orchestrator.dataset_root), + ) + ), + ) + + def with_resolved_steps( + self, pipeline: "ResolvedPipelineDefinition" + ) -> "RunDeclaration": + """Add each enabled step's resolved values from the compiler's one resolution.""" + return replace( + self, + step_configurations=MappingProxyType( + { + index: tuple( + resolved_values( + step, + pipeline.step_provenance[index], + scope_id=pipeline.step_scope_ids[index], + ) + ) + for index, step in enumerate(pipeline.steps) + } + ), + ) + + +def _parameters(invocation: "CompiledFunctionInvocation", function_key: str) -> list[JsonRecord]: + """Every parameter of the function's signature with its effective value and source.""" + contract = invocation.contract + written = invocation.kwargs_dict + supplied_at_run_time = { + *contract.runtime_owned_parameter_names, + invocation.primary_input_parameter_name, + } + recorded: list[RecordedValue] = [] + for name, parameter in contract.canonical_signature.parameters.items(): + if name in written: + recorded.append( + RecordedValue(name, python_expression(written[name]), WrittenInPipeline()) + ) + elif name in supplied_at_run_time or parameter.default is inspect.Parameter.empty: + recorded.append( + RecordedValue(name, "", SuppliedAtRunTime()) + ) + else: + recorded.append( + RecordedValue( + name, + python_expression(parameter.default), + FunctionDefault(function_key), + ) + ) + return [value.to_record() for value in recorded] + + +def _declared_files( + invocation: "CompiledFunctionInvocation", + *, + filemanager: FileManagerLike, + backend: str, +) -> list[JsonRecord]: + """Each declared path parameter, with what the storage layer says the file holds.""" + declared: Mapping[str, tuple[DatasetPathDeclaration, object]] = ( + invocation.contract.declared_path_values(invocation.kwargs_dict) + ) + return [ + { + "parameter": parameter_name, + "declared_as": declaration.plain_name, + "path": str(path), + "content": declaration.content_at_run_start( + Path(path), filemanager=filemanager, backend=backend + ).to_record(), + } + for parameter_name, (declaration, path) in declared.items() + ] + + +def _step_record( + plan: "CompiledStepPlan", + configuration: Iterable[RecordedValue], + *, + filemanager: FileManagerLike, + backend: str, +) -> tuple[JsonRecord, dict[str, str]]: + custom_sources: dict[str, str] = {} + functions = [] + for invocation in plan.compiled_function_pattern.iter_invocations(): + identity = invocation.contract.func.identity() + custom_sources.update(identity.custom_sources()) + functions.append( + { + "identity": identity.to_record(), + "group": python_expression(invocation.key.group_key), + "position": invocation.key.position, + "input_memory_type": invocation.input_memory_type, + "output_memory_type": invocation.output_memory_type, + "parameters": _parameters(invocation, identity.key), + "declared_files": _declared_files( + invocation, filemanager=filemanager, backend=backend + ), + } + ) + return ( + { + "name": plan.step_name, + "pipeline_position": plan.pipeline_position, + "step_scope_id": plan.step_scope_id, + "input_memory_type": plan.input_memory_type, + "output_memory_type": plan.output_memory_type, + "assigned_gpu_devices": [ + {"memory_type": binding.memory_type.value, "device_id": binding.device_id} + for binding in plan.device_assignment.bindings + ], + "functions": functions, + "configuration": [value.to_record() for value in configuration], + }, + custom_sources, + ) + + +@dataclass(frozen=True, slots=True) +class ProducedFile: + """One output file, the step that wrote it and the record of the run.""" + + record: "RunRecord" + producer: StreamProducerIdentity + + @property + def step(self) -> JsonRecord: + return self.record.step_for(self.producer) + + +class UnrecordedOutputFile(LookupError): + """The dataset's metadata does not say which step produced this file.""" + + +@dataclass(frozen=True, slots=True) +class RunRecord: + """What produced an output dataset, as stored: JSON-native, addressed by its content.""" + + content: Mapping[str, Any] + + @property + def record_id(self) -> str: + return hashlib.sha256( + json.dumps(self.content, sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + + @classmethod + def at_run_start( + cls, + bundle: "CompiledExecutionBundle", + *, + filemanager: FileManagerLike, + backend: str, + ) -> "RunRecord": + """Combine the compiler's declaration with the step plans and the declared files' content. + + Called once per run in the parent process before any worker starts, so + a declared input file is hashed once and the hash is of the bytes the + run reads. + """ + import openhcs + from openhcs.runtime.environment_provenance import RuntimeEnvironmentSnapshot + + declaration = bundle.run_declaration + steps: dict[int, JsonRecord] = {} + custom_sources: dict[str, str] = {} + for axis_id in bundle.axis_ids: + context = next( + context + for context in bundle.runtime_contexts.values() + if context.require_axis_id() == axis_id + ) + for index, plan in context.step_plans.items(): + if index in steps: + continue + steps[index], sources = _step_record( + plan, + declaration.step_configurations.get(index, ()), + filemanager=filemanager, + backend=backend, + ) + custom_sources.update(sources) + environment = RuntimeEnvironmentSnapshot.current() + return cls( + { + "software": { + "openhcs": openhcs.__version__, + "python": environment.python_version, + "python_implementation": environment.python_implementation, + "platform": environment.sys_platform, + "machine": environment.machine, + "installed_distributions": { + item.name: item.version + for item in environment.installed_distributions + }, + }, + "pipeline_code": declaration.pipeline_code, + "effective_config_code": declaration.effective_config_code, + "configuration": [value.to_record() for value in declaration.configuration], + "steps": [steps[index] for index in sorted(steps)], + "custom_sources": custom_sources, + } + ) + + @property + def steps(self) -> Sequence[JsonRecord]: + return self.content["steps"] + + def step_for(self, producer: StreamProducerIdentity) -> JsonRecord: + """The step a producer identity names, matched on the two fields it carries.""" + matches = [ + step + for step in self.steps + if step["step_scope_id"] == producer.step_scope_id + and step["pipeline_position"] == producer.pipeline_position + ] + if len(matches) != 1: + raise LookupError( + f"Run record {self.record_id} has {len(matches)} steps for producer " + f"{producer.step_name!r} at position {producer.pipeline_position}." + ) + return matches[0] + + @classmethod + def for_dataset(cls, dataset_root: str | Path) -> Mapping[str, "RunRecord"]: + """Every record the dataset's metadata document holds, by record id.""" + return MappingProxyType( + { + record_id: cls(content) + for record_id, content in _metadata_document(Path(dataset_root)) + .get(RUN_RECORDS_KEY, {}) + .items() + } + ) + + @classmethod + def for_output_file(cls, output_file: str | Path) -> ProducedFile: + """The record and producing step of one output file. + + The answer comes only from what the writer stored for this path; a file + the dataset does not list raises instead of being guessed from its name. + """ + path = Path(output_file).resolve() + dataset_root = _dataset_root_of(path) + document = _metadata_document(dataset_root) + entry = document.get(PRODUCED_FILES_KEY, {}).get( + path.relative_to(dataset_root).as_posix() + ) + if entry is None: + raise UnrecordedOutputFile( + f"{path} is not recorded as a step output in " + f"{_metadata_path(dataset_root)}." + ) + return ProducedFile( + record=cls(document[RUN_RECORDS_KEY][entry["run_record"]]), + producer=StreamProducerIdentity.from_payload(entry["producer"]), + ) + + def rebuild(self) -> tuple["PipelineDocument", "GlobalPipelineConfig"]: + """Rebuild the pipeline document and global config this record was run with. + + Custom functions are registered from the recorded source text (not + persisted), after checking that each text still has its recorded hash. + """ + from openhcs.core.config import GlobalPipelineConfig + from openhcs.core.config_document import ConfigDocumentCodec + from openhcs.core.pipeline_document import PipelineDocumentCodec + from openhcs.processing.custom_functions import CustomFunctionManager + from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, + ) + from openhcs.processing.custom_functions.source_namespace import ( + CustomFunctionSource, + ) + + function_name_by_source = { + function["identity"]["custom_source_sha256"]: function["identity"]["name"] + for step in self.steps + for function in step["functions"] + if "custom_source_sha256" in function["identity"] + } + for recorded_sha256, source_text in self.content["custom_sources"].items(): + if CustomFunctionSource.content_digest(source_text.encode("utf-8")) != ( + recorded_sha256 + ): + raise ValueError( + f"Run record {self.record_id}: custom source {recorded_sha256} does " + "not match its recorded hash." + ) + source = CustomFunctionSource( + function_name_by_source[recorded_sha256], recorded_sha256 + ) + if CustomFunctionRuntimeRegistry.declaration_for_source(source) is None: + CustomFunctionManager().register_from_code( + source_text, persist=False, emit_signal=False + ) + return ( + PipelineDocumentCodec.from_source(self.content["pipeline_code"]), + ConfigDocumentCodec.from_source( + self.content["effective_config_code"], + expected_config_type=GlobalPipelineConfig, + ), + ) + + +def _metadata_path(dataset_root: Path) -> Path: + from openhcs.core.virtual_workspace_metadata import get_metadata_path + + return Path(get_metadata_path(dataset_root)) + + +def _metadata_document(dataset_root: Path) -> Mapping[str, Any]: + return json.loads(_metadata_path(dataset_root).read_text(encoding="utf-8")) + + +def _dataset_root_of(output_file: Path) -> Path: + for directory in output_file.parents: + if _metadata_path(directory).is_file(): + return directory + raise UnrecordedOutputFile( + f"No directory above {output_file} holds an OpenHCS metadata document." + ) + + +@dataclass(frozen=True, slots=True) +class ProducedFiles: + """The saved paths of one run, each with the producer identity its writer stated.""" + + producer_by_path: Mapping[Path, StreamProducerIdentity] + + def write_into( + self, + document: dict[str, Any], + *, + dataset_root: Path, + record: RunRecord, + ) -> None: + """Store the record and this run's files; drop records no file refers to any more.""" + records = document.setdefault(RUN_RECORDS_KEY, {}) + produced = document.setdefault(PRODUCED_FILES_KEY, {}) + record_id = record.record_id + records[record_id] = dict(record.content) + for path, producer in self.producer_by_path.items(): + if not path.is_relative_to(dataset_root): + continue + produced[path.relative_to(dataset_root).as_posix()] = { + "run_record": record_id, + "producer": dict(producer.to_payload()), + } + referenced = {entry["run_record"] for entry in produced.values()} | {record_id} + for stale in set(records) - referenced: + del records[stale] diff --git a/openhcs/core/vfs_protocol.py b/openhcs/core/vfs_protocol.py index cc4eb9022..7d2fe3053 100644 --- a/openhcs/core/vfs_protocol.py +++ b/openhcs/core/vfs_protocol.py @@ -7,7 +7,9 @@ from dataclasses import dataclass from pathlib import Path from types import MappingProxyType -from typing import Annotated, Any, get_args, get_origin +from typing import Annotated, Any, ClassVar, get_args, get_origin + +from polystore.file_content import FileContent, UnhashedFileContent class FileManagerLike(ABC): @@ -35,6 +37,10 @@ def exists(self, path: str | Path, backend: str) -> bool: def is_dir(self, path: str | Path, backend: str) -> bool: """Return whether a path is a directory in the selected backend.""" + @abstractmethod + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + """Say what bytes a file holds, or why the backend cannot hash it.""" + @abstractmethod def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: """Load an object from the selected backend.""" @@ -103,6 +109,9 @@ def ensure_directory(self, directory: str | Path, backend: str) -> str: class DatasetPathDeclaration(ABC): """Nominal declaration for a path resolved against one compilation plate.""" + plain_name: ClassVar[str] + """What this kind of path is, for a reader of a run's record.""" + @abstractmethod def validate_target( self, @@ -113,6 +122,16 @@ def validate_target( ) -> None: """Validate the resolved target without creating it.""" + @abstractmethod + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + """Identify the bytes a run will read here, or say why there are none to hash.""" + @classmethod def from_annotation(cls, annotation: Any) -> "DatasetPathDeclaration | None": """Return the sole declaration carried by one structured annotation.""" @@ -146,6 +165,17 @@ def _from_annotation(cls, annotation: Any) -> tuple["DatasetPathDeclaration", .. class PlateInputFileDeclaration(DatasetPathDeclaration): """Existing plate-relative input file.""" + plain_name: ClassVar[str] = "input file" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + return filemanager.file_content(target, backend) + def validate_target( self, target: Path, @@ -163,6 +193,20 @@ def validate_target( class PlateInputDirectoryDeclaration(DatasetPathDeclaration): """Existing plate-relative input directory.""" + plain_name: ClassVar[str] = "input directory" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + del target, filemanager, backend + return UnhashedFileContent( + "A declared input directory is recorded by path; its files are not hashed." + ) + def validate_target( self, target: Path, @@ -184,6 +228,20 @@ def validate_target( class PlateOutputFileDeclaration(DatasetPathDeclaration): """Plate-relative output file whose writer owns creation.""" + plain_name: ClassVar[str] = "output file" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + del target, filemanager, backend + return UnhashedFileContent( + "This run writes the file; it has no content when the run starts." + ) + def validate_target( self, target: Path, @@ -198,6 +256,20 @@ def validate_target( class PlateOutputDirectoryDeclaration(DatasetPathDeclaration): """Plate-relative output directory whose writer owns creation.""" + plain_name: ClassVar[str] = "output directory" + + def content_at_run_start( + self, + target: Path, + *, + filemanager: FileManagerLike, + backend: str, + ) -> FileContent: + del target, filemanager, backend + return UnhashedFileContent( + "This run writes the directory; it has no content when the run starts." + ) + def validate_target( self, target: Path, diff --git a/openhcs/processing/backends/lib_registry/unified_registry.py b/openhcs/processing/backends/lib_registry/unified_registry.py index 5a49fd60e..b6606421c 100644 --- a/openhcs/processing/backends/lib_registry/unified_registry.py +++ b/openhcs/processing/backends/lib_registry/unified_registry.py @@ -137,6 +137,17 @@ def composite_key(self) -> str: def require_current_declaration(self) -> None: """Validate declaration lifetimes specialized by metadata owners.""" + def identity(self) -> "FunctionIdentity": + """Where this function comes from and at which version.""" + from openhcs.core.function_identity import FunctionIdentity + + import_identity = self.import_identity + return FunctionIdentity.of_module_export( + key=self.composite_key, + module=import_identity.module_name, + name=import_identity.function_name, + ) + @property def display_name(self) -> str: """Human-readable function name for catalogs and selectors.""" diff --git a/openhcs/processing/custom_functions/manager.py b/openhcs/processing/custom_functions/manager.py index 67ff41bda..348caf4b2 100644 --- a/openhcs/processing/custom_functions/manager.py +++ b/openhcs/processing/custom_functions/manager.py @@ -533,7 +533,7 @@ def _prepare_source( function_name=declaration.__name__, content_sha256=CustomFunctionSource.content_digest(code.encode("utf-8")), ) - CustomFunctionSourceNamespace(source, namespace).bind(declaration) + CustomFunctionSourceNamespace(source, namespace, code).bind(declaration) try: return project_custom_function( declaration, diff --git a/openhcs/processing/custom_functions/runtime_registry.py b/openhcs/processing/custom_functions/runtime_registry.py index 825957e5f..25ccecf45 100644 --- a/openhcs/processing/custom_functions/runtime_registry.py +++ b/openhcs/processing/custom_functions/runtime_registry.py @@ -14,6 +14,7 @@ from typing import TYPE_CHECKING from openhcs.core.function_contract_metadata import FunctionContractAttribute +from openhcs.core.function_identity import CustomFunctionIdentity from openhcs.processing.backends.lib_registry.unified_registry import FunctionMetadata from openhcs.processing.custom_functions.source_namespace import ( CustomFunctionSource, @@ -60,6 +61,18 @@ def require_current_declaration(self) -> None: super().require_current_declaration() CustomFunctionRuntimeRegistry.require_current_metadata(self) + def identity(self) -> CustomFunctionIdentity: + """The source text that defines this function, by hash, and what it imports.""" + namespace = CustomFunctionSourceNamespace.bound_to(self.func) + import_identity = self.import_identity + return CustomFunctionIdentity.of_source( + key=self.composite_key, + module=import_identity.module_name, + name=import_identity.function_name, + source_sha256=namespace.source.content_sha256, + source_text=namespace.code, + ) + class CustomFunctionCanonicalLookup: """Independent source-lookup capability composed with the registry template.""" diff --git a/openhcs/processing/custom_functions/source_namespace.py b/openhcs/processing/custom_functions/source_namespace.py index 56bb1c051..7b96866b4 100644 --- a/openhcs/processing/custom_functions/source_namespace.py +++ b/openhcs/processing/custom_functions/source_namespace.py @@ -57,8 +57,25 @@ class CustomFunctionSourceNamespace: source: CustomFunctionSource bindings: dict[str, object] = field(repr=False) + code: str = field(repr=False) + """The exact text ``source.content_sha256`` is the hash of.""" member_prefix: ClassVar[str] = "__openhcs_member_" + def __post_init__(self) -> None: + if CustomFunctionSource.content_digest(self.code.encode("utf-8")) != ( + self.source.content_sha256 + ): + raise ValueError( + f"Custom function source {self.source.function_name!r}: the text " + "does not have the declared content hash." + ) + + @classmethod + def bound_to(cls, declaration: Callable) -> "CustomFunctionSourceNamespace": + """Return the namespace ``bind`` attached to this declaration.""" + revision = vars(declaration)[FunctionContractAttribute.declaration_revision] + return vars(declaration)[f"__openhcs_source_{revision}"] + @property def export_attribute(self) -> str: """Derive the revision-specific namespace address on its callable.""" diff --git a/pyproject.toml b/pyproject.toml index 2b7408d96..4ff06b44a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -108,7 +108,7 @@ dependencies = [ "python-introspect>=0.2.8,<0.3", "metaclass-registry>=0.4.0,<0.5", "arraybridge>=0.4.1,<0.5", - "polystore>=0.6.2,<0.7", + "polystore>=0.6.3,<0.7", "formbridge>=0.1.3,<0.2", "pyqt-reactive>=0.7.2,<0.8", ] diff --git a/tests/unit/test_catalog_startup_gc_lifetime.py b/tests/unit/test_catalog_startup_gc_lifetime.py index 2eb711391..0b20d02a0 100644 --- a/tests/unit/test_catalog_startup_gc_lifetime.py +++ b/tests/unit/test_catalog_startup_gc_lifetime.py @@ -23,10 +23,15 @@ def test_startup_freeze_preserves_job_gc_and_releases_source_cycles(): CustomFunctionSource, CustomFunctionSourceNamespace, ) bindings = {"__name__": "startup_lifetime_control"} - exec("def captured_source(): pass", bindings) + code = "def captured_source(): pass" + exec(code, bindings) function = bindings["captured_source"] namespace = CustomFunctionSourceNamespace( - CustomFunctionSource("captured_source", "lifetime-control"), bindings, + CustomFunctionSource( + "captured_source", CustomFunctionSource.content_digest(code.encode()) + ), + bindings, + code, ) namespace.bind(function) captured = weakref.ref(function) From c2d689e61c3266c5ee9ea5574db51bf5fd713856 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 15:04:27 -0400 Subject: [PATCH 03/12] WIP DL3: compile, run and metadata writer carry the run record The compiled bundle carries a RunDeclaration (complete pipeline document, saved global config document, resolved configuration with sources). Workers report the producer identity of each saved file. The parent hashes declared input files once before workers start and the metadata writer stores the record and the produced files in the dataset document's one transaction. Complete rendering (clean_mode=False) now writes the signature default of every function parameter a pattern leaves out, except those the platform supplies; range and module-exported callable defaults get formatters. The record stores the saved global config (global_config_code), not the effective config: rebuilding then resolves through the same hierarchy. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/compiled_execution.py | 7 + openhcs/core/orchestrator/bundle_execution.py | 16 + openhcs/core/pipeline/compiler.py | 8 + openhcs/core/run_record.py | 143 +++-- openhcs/core/steps/abstract.py | 20 + openhcs/core/steps/function_outputs.py | 82 ++- openhcs/core/steps/function_step.py | 7 + openhcs/core/virtual_workspace_metadata.py | 21 +- openhcs/serialization/pycodify_formatters.py | 81 +++ .../test_compiled_artifact_inspection.py | 2 + .../test_custom_source_transport.py | 2 + .../test_generated_source_relative_paths.py | 6 + ...rimary_segmentation_diagnostics_journey.py | 3 +- .../integration/test_run_record_round_trip.py | 141 +++++ tests/run_records.py | 34 ++ tests/unit/agent/test_agent_services.py | 2 + tests/unit/test_artifact_plan_view.py | 2 + .../unit/test_artifact_publication_journey.py | 4 +- .../unit/test_cellprofiler_pipeline_import.py | 7 + tests/unit/test_compilation_paths.py | 7 + tests/unit/test_compiled_execution.py | 5 + tests/unit/test_function_outputs.py | 30 +- tests/unit/test_function_step_transport.py | 2 + .../unit/test_named_checkpoint_publication.py | 6 +- tests/unit/test_orchestrator_lane_planning.py | 9 +- tests/unit/test_persisted_image_metadata.py | 4 +- .../test_persisted_roi_only_publication.py | 9 +- tests/unit/test_run_record.py | 507 ++++++++++++++++++ tests/unit/test_source_round_trip.py | 17 +- tests/unit/test_zmq_compilation.py | 4 + tests/unit/test_zmq_execution_server.py | 3 + 31 files changed, 1111 insertions(+), 80 deletions(-) create mode 100644 tests/integration/test_run_record_round_trip.py create mode 100644 tests/run_records.py create mode 100644 tests/unit/test_run_record.py diff --git a/openhcs/core/compiled_execution.py b/openhcs/core/compiled_execution.py index 940c40a4b..73970e259 100644 --- a/openhcs/core/compiled_execution.py +++ b/openhcs/core/compiled_execution.py @@ -10,6 +10,7 @@ ) from openhcs.core.config import GlobalPipelineConfig, MultiprocessingStartMethod from openhcs.core.context.processing_context import CompiledContext +from openhcs.core.run_record import RunDeclaration from openhcs.core.steps.abstract import AbstractStep from openhcs.core.worker_start_policy import ( WorkerStartDecision, @@ -132,6 +133,8 @@ class CompiledExecutionBundle: transport_contexts: Mapping[str, CompiledContext] worker_assignments: Mapping[str, list[str]] runtime_environment: CompiledRuntimeEnvironmentPlan + run_declaration: RunDeclaration + """The documents and resolved configuration this bundle was compiled from.""" @property def axis_ids(self) -> tuple[str, ...]: @@ -164,6 +167,7 @@ def from_runtime_contexts( runtime_contexts: Mapping[str, CompiledContext], worker_assignments: Mapping[str, list[str]], runtime_environment: CompiledRuntimeEnvironmentPlan, + run_declaration: RunDeclaration, ) -> "CompiledExecutionBundle": from openhcs.core.function_step_transport import FunctionStepTransportAuthority @@ -178,6 +182,7 @@ def from_runtime_contexts( key: list(value) for key, value in worker_assignments.items() }, runtime_environment=runtime_environment, + run_declaration=run_declaration, ) @classmethod @@ -187,6 +192,7 @@ def from_unassigned_runtime_contexts( pipeline_definition: Sequence[AbstractStep], runtime_contexts: Mapping[str, CompiledContext], runtime_environment: CompiledRuntimeEnvironmentPlan, + run_declaration: RunDeclaration, ) -> "CompiledExecutionBundle": """Create a bundle before worker ownership has been assigned.""" @@ -195,6 +201,7 @@ def from_unassigned_runtime_contexts( runtime_contexts=runtime_contexts, worker_assignments={}, runtime_environment=runtime_environment, + run_declaration=run_declaration, ) def for_transport_serialization( diff --git a/openhcs/core/orchestrator/bundle_execution.py b/openhcs/core/orchestrator/bundle_execution.py index 03341f3dc..12bed2121 100644 --- a/openhcs/core/orchestrator/bundle_execution.py +++ b/openhcs/core/orchestrator/bundle_execution.py @@ -84,6 +84,7 @@ from openhcs.core.steps.function_artifact_materialization import ( ArtifactMaterializationTargetPlan, ) +from openhcs.core.run_record import RunRecord from openhcs.core.steps.function_outputs import ( OpenHCSMetadataTarget, ) @@ -251,6 +252,13 @@ def execute_bundle_request( f"with max_workers={validated.actual_max_workers}." ) + # Before any worker exists: each declared input file is hashed once, + # and the hash is of the bytes this run reads. + run_record = RunRecord.at_run_start( + request.execution_bundle, + filemanager=orchestrator.filemanager, + backend=Backend.DISK.value, + ) executor_resources = WorkerExecutorFactory( log_file_base=request.log_file_base, progress_queue=validated.progress_queue, @@ -340,6 +348,7 @@ def execute_bundle_request( executing_contexts ), ), + run_record=run_record, ) if execution_results.is_success(): viewer_states_by_port = settle_viewer_state( @@ -644,6 +653,13 @@ def execute_dataset_scoped_steps( StepExecutionObservation( materialized_locations_by_address=MappingProxyType({}), source_projection_entries_by_target=projection_entries, + producer_by_saved_path=( + OpenHCSMetadataTarget.saved_path_producers( + owner_context, + owner_plan.require_function_execution_ready(), + artifact_materializations=materializations, + ) + ), ), ) ) diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index 30bc6f421..a15af1493 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -56,6 +56,7 @@ StepAxisFilterSet, ) from openhcs.core.pipeline.funcstep_contract_validator import FuncStepContractValidator +from openhcs.core.run_record import RunDeclaration from openhcs.core.pipeline.compilation_session import ( CompilationPathResolver, CompilationDatasetScope, @@ -1534,6 +1535,9 @@ def compile_pipelines( axis_filter, effective_config, ) + run_declaration = RunDeclaration.before_step_resolution( + orchestrator, pipeline_definition + ) if not axis_values_to_process: logger.warning("No axis values found to process based on filter.") return CompiledExecutionBundle.from_runtime_contexts( @@ -1545,6 +1549,7 @@ def compile_pipelines( compiled_contexts={}, server_mode=is_zmq_execution, ), + run_declaration=run_declaration, ) logger.info( @@ -1554,6 +1559,7 @@ def compile_pipelines( pipeline_inputs = PipelineCompiler._resolve_pipeline_once( orchestrator, pipeline_definition ) + run_declaration = run_declaration.with_resolved_steps(pipeline_inputs) if not pipeline_definition: logger.warning( "All steps were disabled. Pipeline is empty after filtering." @@ -1567,6 +1573,7 @@ def compile_pipelines( compiled_contexts={}, server_mode=is_zmq_execution, ), + run_declaration=run_declaration, ) path_resolver = CompilationPathResolver( dataset_scope=CompilationDatasetScope.from_path(orchestrator.dataset_root), @@ -1654,6 +1661,7 @@ def compile_pipelines( runtime_contexts=compiled_contexts, worker_assignments=worker_assignments, runtime_environment=runtime_environment, + run_declaration=run_declaration, ) PipelineCompiler._write_compilation_debug_bundle_if_configured( effective_config, diff --git a/openhcs/core/run_record.py b/openhcs/core/run_record.py index d4a23d3fa..d5da5fd60 100644 --- a/openhcs/core/run_record.py +++ b/openhcs/core/run_record.py @@ -12,7 +12,7 @@ ``openhcs_metadata.json`` by its one writer and read back from there. Two read kinds are kept apart on purpose. ``pipeline_code`` and -``effective_config_code`` are stored documents: executing them rebuilds the +``global_config_code`` are stored documents: executing them rebuilds the run. ``configuration``, ``steps[].configuration`` and ``parameters`` are the resolved values, each with the declaration that supplied it. """ @@ -115,18 +115,22 @@ class InheritedFrom(ValueSource): kind: ClassVar[str] = "inherited" scope_id: str | None - source_type: type | None + source_type_name: str | None + + @classmethod + def reported_by_object_state( + cls, scope_id: str | None, source_type: type | None + ) -> "InheritedFrom": + """Keep the class by name: the declaration is pickled with the compiled bundle.""" + return cls( + scope_id, + None + if source_type is None + else f"{source_type.__module__}.{source_type.__qualname__}", + ) def details(self) -> JsonRecord: - source_type = self.source_type - return { - "scope": self.scope_id, - "type": ( - None - if source_type is None - else f"{source_type.__module__}.{source_type.__qualname__}" - ), - } + return {"scope": self.scope_id, "type": self.source_type_name} @dataclass(frozen=True, slots=True) @@ -148,20 +152,12 @@ def to_record(self) -> JsonRecord: ObjectStateProvenance = Mapping[str, tuple[str | None, type | None]] -def _declared_names(configured: object) -> tuple[str, ...]: - """The names a configured object is constructed from.""" - if is_dataclass(configured): - return tuple(item.name for item in fields(configured)) - return tuple( - name - for name, parameter in inspect.signature(type(configured)).parameters.items() - if parameter.kind - not in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD) - ) +def _field_values(configured: object) -> dict[str, object]: + return {item.name: getattr(configured, item.name) for item in fields(configured)} def resolved_values( - resolved: object, + values: Mapping[str, object], provenance: ObjectStateProvenance, *, scope_id: str | None, @@ -169,18 +165,21 @@ def resolved_values( ) -> Iterator[RecordedValue]: """Every leaf value of a resolved object, with the source ObjectState reported. - ObjectState lists the fields it inherited; a field it does not list is held - by the object itself. A nested config that was inherited whole is one value. + ObjectState lists the paths it inherited; a path it does not list is held + by the object itself. A nested config the object leaves unset is inherited + whole and listed as one path, so the inherited check comes before descent. """ - for name in _declared_names(resolved): + for name, value in values.items(): path = f"{prefix}{name}" - value = getattr(resolved, name) - inherited = provenance.get(path) - if inherited is not None: - yield RecordedValue(path, python_expression(value), InheritedFrom(*inherited)) - elif is_dataclass(value) and not isinstance(value, type) and fields(value): + if path in provenance: + yield RecordedValue( + path, + python_expression(value), + InheritedFrom.reported_by_object_state(*provenance[path]), + ) + elif is_dataclass(value) and not isinstance(value, type): yield from resolved_values( - value, provenance, scope_id=scope_id, prefix=f"{path}." + _field_values(value), provenance, scope_id=scope_id, prefix=f"{path}." ) else: yield RecordedValue(path, python_expression(value), SetOnObject(scope_id)) @@ -191,9 +190,10 @@ class RunDeclaration: """What the compiler alone knows about a run: documents and resolved configuration.""" pipeline_code: str - effective_config_code: str + global_config_code: str configuration: tuple[RecordedValue, ...] - step_configurations: Mapping[int, tuple[RecordedValue, ...]] = MappingProxyType({}) + step_configurations: tuple[tuple[RecordedValue, ...], ...] = () + """Resolved values of each enabled step, by compiled step index.""" @classmethod def before_step_resolution( @@ -203,11 +203,17 @@ def before_step_resolution( ) -> "RunDeclaration": """Render the documents as submitted, before the compiler resolves the steps in place.""" from objectstate import ObjectState, ObjectStateRegistry + from objectstate.global_config import get_saved_global_config from openhcs.core.config import GlobalPipelineConfig from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.pipeline_document import PipelineDocumentCodec + # With no saved global config set, ObjectState resolves against the + # declared defaults: the document a default-constructed config states. + saved_global_config = ( + get_saved_global_config(GlobalPipelineConfig) or GlobalPipelineConfig() + ) effective_config, provenance = ObjectState.resolve_saved_object( orchestrator.pipeline_config, ancestor_objects_with_scopes=( @@ -222,14 +228,14 @@ def before_step_resolution( ), clean_mode=False, ), - effective_config_code=ConfigDocumentCodec.render( - effective_config, + global_config_code=ConfigDocumentCodec.render( + saved_global_config, expected_config_type=GlobalPipelineConfig, clean_mode=False, ), configuration=tuple( resolved_values( - effective_config, + _field_values(effective_config), provenance, scope_id=str(orchestrator.dataset_root), ) @@ -242,17 +248,15 @@ def with_resolved_steps( """Add each enabled step's resolved values from the compiler's one resolution.""" return replace( self, - step_configurations=MappingProxyType( - { - index: tuple( - resolved_values( - step, - pipeline.step_provenance[index], - scope_id=pipeline.step_scope_ids[index], - ) + step_configurations=tuple( + tuple( + resolved_values( + step.configuration_parameters(), + pipeline.step_provenance[index], + scope_id=pipeline.step_scope_ids[index], ) - for index, step in enumerate(pipeline.steps) - } + ) + for index, step in enumerate(pipeline.steps) ), ) @@ -283,6 +287,11 @@ def _parameters(invocation: "CompiledFunctionInvocation", function_key: str) -> FunctionDefault(function_key), ) ) + recorded.extend( + RecordedValue(name, python_expression(value), WrittenInPipeline()) + for name, value in written.items() + if name not in contract.canonical_signature.parameters + ) return [value.to_record() for value in recorded] @@ -411,7 +420,7 @@ def at_run_start( continue steps[index], sources = _step_record( plan, - declaration.step_configurations.get(index, ()), + declaration.step_configurations[index], filemanager=filemanager, backend=backend, ) @@ -431,7 +440,7 @@ def at_run_start( }, }, "pipeline_code": declaration.pipeline_code, - "effective_config_code": declaration.effective_config_code, + "global_config_code": declaration.global_config_code, "configuration": [value.to_record() for value in declaration.configuration], "steps": [steps[index] for index in sorted(steps)], "custom_sources": custom_sources, @@ -533,7 +542,7 @@ def rebuild(self) -> tuple["PipelineDocument", "GlobalPipelineConfig"]: return ( PipelineDocumentCodec.from_source(self.content["pipeline_code"]), ConfigDocumentCodec.from_source( - self.content["effective_config_code"], + self.content["global_config_code"], expected_config_type=GlobalPipelineConfig, ), ) @@ -564,6 +573,30 @@ class ProducedFiles: producer_by_path: Mapping[Path, StreamProducerIdentity] + def by_dataset_root( + self, dataset_roots: Iterable[Path] + ) -> Mapping[Path, "ProducedFiles"]: + """Split the files by the declared output dataset root each one lies under. + + The roots are the ones the compiled plans declare. Where one declared + root contains another, a file belongs to the nearest. + """ + nearest_first = sorted(set(dataset_roots), key=lambda root: -len(root.parts)) + by_root: dict[Path, dict[Path, StreamProducerIdentity]] = {} + for path, producer in self.producer_by_path.items(): + root = next( + (root for root in nearest_first if path.is_relative_to(root)), None + ) + if root is None: + raise ValueError( + f"Saved file {path} lies under no declared output dataset root " + f"({', '.join(map(str, nearest_first))})." + ) + by_root.setdefault(root, {})[path] = producer + return MappingProxyType( + {root: ProducedFiles(MappingProxyType(files)) for root, files in by_root.items()} + ) + def write_into( self, document: dict[str, Any], @@ -571,18 +604,22 @@ def write_into( dataset_root: Path, record: RunRecord, ) -> None: - """Store the record and this run's files; drop records no file refers to any more.""" + """Store the record with the files this run saved under this dataset root. + + A document the run saved nothing under is left as it is. A record no + file refers to any more is dropped. + """ + if not self.producer_by_path: + return records = document.setdefault(RUN_RECORDS_KEY, {}) produced = document.setdefault(PRODUCED_FILES_KEY, {}) record_id = record.record_id records[record_id] = dict(record.content) for path, producer in self.producer_by_path.items(): - if not path.is_relative_to(dataset_root): - continue produced[path.relative_to(dataset_root).as_posix()] = { "run_record": record_id, "producer": dict(producer.to_payload()), } - referenced = {entry["run_record"] for entry in produced.values()} | {record_id} + referenced = {entry["run_record"] for entry in produced.values()} for stale in set(records) - referenced: del records[stale] diff --git a/openhcs/core/steps/abstract.py b/openhcs/core/steps/abstract.py index 2cb064dea..d1a4104f1 100644 --- a/openhcs/core/steps/abstract.py +++ b/openhcs/core/steps/abstract.py @@ -15,6 +15,7 @@ from typing import TYPE_CHECKING, Iterable, get_type_hints from objectstate import get_base_type_for_lazy, semantic_values_equal +from polystore.streaming.identity import StreamProducerIdentity # Import LazyStepMaterializationConfig for type hints @@ -65,6 +66,11 @@ class StepExecutionObservation: "OpenHCSMetadataTarget", "VirtualWorkspaceSourceProjectionEntries" ] = field(default_factory=dict) + producer_by_saved_path: Mapping[Path, StreamProducerIdentity] = field( + default_factory=dict + ) + """The producer identity its writer stated for each saved file.""" + def __post_init__(self) -> None: projections = dict(self.source_projection_entries_by_target) if projections: @@ -99,6 +105,7 @@ def is_empty(self) -> bool: or self.hook_observations or self.sample_numbers_by_export_path or self.source_projection_entries_by_target + or self.producer_by_saved_path ) @classmethod @@ -124,7 +131,14 @@ def combine( hook_observations = [] sample_numbers = {} source_projections = {} + producers = {} for observation in observations: + for path, producer in observation.producer_by_saved_path.items(): + if producers.setdefault(path, producer) != producer: + raise ValueError( + f"Saved file {path} has two producers: " + f"{producers[path]} and {producer}." + ) for ( address, values, @@ -163,6 +177,7 @@ def combine( for target, entries in source_projections.items() } ), + MappingProxyType(producers), ) @@ -267,6 +282,11 @@ def declaration_parameters(self) -> dict[str, object]: if not name.startswith("_") } + def configuration_parameters(self) -> dict[str, object]: + """The values that configure this step, apart from what it computes.""" + + return self.declaration_parameters() + def same_declaration(self, other: object) -> bool: """Return whether another step is the same complete declaration.""" diff --git a/openhcs/core/steps/function_outputs.py b/openhcs/core/steps/function_outputs.py index bd9ebf565..ad230b184 100644 --- a/openhcs/core/steps/function_outputs.py +++ b/openhcs/core/steps/function_outputs.py @@ -56,6 +56,7 @@ SourceProjection, SourceProjectionMetadataSerializer, ) +from openhcs.core.run_record import ProducedFiles, RunRecord from openhcs.core.steps.abstract import StepExecutionObservation from openhcs.core.steps.function_artifact_materialization import ( ArtifactMaterializationTargetPlan, @@ -122,6 +123,9 @@ def finalize_function_step_outputs( projection_entries = OpenHCSMetadataTarget.observe_for_step( context, plan, artifact_materializations=materializations ) + producers = OpenHCSMetadataTarget.saved_path_producers( + context, plan, artifact_materializations=materializations + ) else: _profile_finalization_phase( "finalize_memory_outputs", @@ -150,11 +154,19 @@ def finalize_function_step_outputs( ), plan, ) + producers = _profile_finalization_phase( + "finalize_saved_path_producers", + lambda: OpenHCSMetadataTarget.saved_path_producers( + context, plan, artifact_materializations=materializations + ), + plan, + ) return replace( StepExecutionObservation.combine( item.observation(plan, context) for item in materializations ), source_projection_entries_by_target=projection_entries, + producer_by_saved_path=producers, ) @@ -1058,22 +1070,80 @@ def observe_for_step( observations[replace(target, artifact_materializations=())] = entries return MappingProxyType(observations) + @classmethod + def saved_path_producers( + cls, + context: ExecutingContext, + plan: CompiledStepPlan, + *, + artifact_materializations: tuple[MaterializedRuntimeArtifact, ...], + ) -> Mapping[Path, StreamProducerIdentity]: + """The producer identity of each file this step saved under a metadata document. + + Both statements are the ones the viewer stream uses: a main-flow record + carries its own identity; a saved artifact's comes from its output plan. + """ + producers: dict[Path, StreamProducerIdentity] = {} + for target in cls.for_execution( + context, plan, artifact_materializations=artifact_materializations + ): + if not cls.hosts_metadata_document(target.backend): + continue + for record in target.produced_records(context, plan): + producers[Path(record.path_under(target.output_dir))] = ( + record.producer_identity + ) + artifact_target = plan.runtime_artifact_materialization + if artifact_target.has_persistent_target: + backend = artifact_target.require_persistent_backend() + if cls.hosts_metadata_document(backend): + for saved in artifact_materializations: + producer = plan.producer_identity_for_artifact( + saved.materialization.output_plan + ) + for output in saved.outputs_for_backend(backend): + producers[Path(output.path)] = producer + return MappingProxyType(producers) + @classmethod def finalize_completed_dataset( cls, compiled_contexts: Mapping[str, ExecutingContext], *, runtime_observations: Iterable[RuntimeExecutionObservation] = (), + run_record: RunRecord, ) -> None: - """Publish completed or partial saved outputs once at the receiving boundary.""" + """Publish completed or partial saved outputs once at the receiving boundary. + Each output dataset's document also receives the run's record and the + producer of every file the run saved under that dataset root. + """ + + runtime_observations = tuple(runtime_observations) target_contexts: dict[OpenHCSMetadataTarget, ExecutingContext] = {} + declared_dataset_roots: set[Path] = set() for context in compiled_contexts.values(): for plan in context.step_plans.values(): + targets = cls.for_plan(plan) + declared_dataset_roots.update( + Path(target.dataset_root) for target in targets + ) if not plan.create_openhcs_metadata: continue - for target in cls.for_plan(plan): + for target in targets: target_contexts.setdefault(target, context) + produced_files_by_document = { + METADATA_CONFIG.metadata_path(dataset_root): produced_files + for dataset_root, produced_files in ProducedFiles( + StepExecutionObservation.combine( + context_observation.outputs + for observation in runtime_observations + for context_observation in observation.contexts + ).producer_by_saved_path + ) + .by_dataset_root(declared_dataset_roots) + .items() + } produced_by_document: dict[Path, dict[OpenHCSMetadataTarget, list[VirtualWorkspaceSourceProjectionEntries]]] = {} for observation in runtime_observations: @@ -1093,7 +1163,9 @@ def finalize_completed_dataset( targets_by_document.setdefault( METADATA_CONFIG.metadata_path(owner.dataset_root), {} )[owner] = context - for metadata_path in dict.fromkeys((*targets_by_document, *produced_by_document)): + for metadata_path in dict.fromkeys( + (*targets_by_document, *produced_by_document, *produced_files_by_document) + ): AtomicMetadataWriter().reconcile_completed_dataset( metadata_path, targets_by_document.get(metadata_path, {}), @@ -1101,6 +1173,10 @@ def finalize_completed_dataset( target: VirtualWorkspaceSourceProjectionEntries.combine(entries) for target, entries in produced_by_document.get(metadata_path, {}).items() }, + produced_files=produced_files_by_document.get( + metadata_path, ProducedFiles(MappingProxyType({})) + ), + run_record=run_record, ) diff --git a/openhcs/core/steps/function_step.py b/openhcs/core/steps/function_step.py index bfc8ccbd2..a417fcb17 100644 --- a/openhcs/core/steps/function_step.py +++ b/openhcs/core/steps/function_step.py @@ -57,6 +57,13 @@ def with_function_spec(self, func: FunctionSpec) -> "FunctionStep": step.func = func return step + def configuration_parameters(self) -> dict[str, object]: + """The function pattern is what the step computes, not its configuration.""" + + parameters = self.declaration_parameters() + del parameters["func"] + return parameters + def occurrence_authorities(self) -> tuple[object, ...]: """Prefer callable-pattern authority before nominal step authority.""" diff --git a/openhcs/core/virtual_workspace_metadata.py b/openhcs/core/virtual_workspace_metadata.py index 036b406f7..8a270e42c 100644 --- a/openhcs/core/virtual_workspace_metadata.py +++ b/openhcs/core/virtual_workspace_metadata.py @@ -38,6 +38,7 @@ from openhcs.core.axes import Axis, AxisFamily, ColourAxis if TYPE_CHECKING: + from openhcs.core.run_record import ProducedFiles, RunRecord from openhcs.core.context.processing_context import ExecutingContext from openhcs.core.steps.function_outputs import OpenHCSMetadataTarget @@ -223,11 +224,22 @@ def reconcile_completed_dataset( produced_entries_by_target: Mapping[ OpenHCSMetadataTarget, VirtualWorkspaceSourceProjectionEntries ] = MappingProxyType({}), + produced_files: ProducedFiles, + run_record: RunRecord, ) -> None: - """Admit and reconcile all targets from one locked plate document.""" - if not produced_entries_by_target and not Path(metadata_path).is_file() and not any( + """Admit and reconcile all targets from one locked plate document. + + The same transaction stores the run's record with the files the run + saved under this document's dataset root. + """ + if ( + not produced_entries_by_target + and not produced_files.producer_by_path + and not Path(metadata_path).is_file() + and not any( target.contains_outputs(context) - for target, context in target_contexts.items() + for target, context in target_contexts.items() + ) ): return @@ -268,6 +280,9 @@ def update(data): VirtualWorkspaceSourceProjectionEntries(MappingProxyType({})), ), ) + produced_files.write_into( + data, dataset_root=Path(metadata_path).parent, record=run_record + ) return data self._execute_update(metadata_path, update) diff --git a/openhcs/serialization/pycodify_formatters.py b/openhcs/serialization/pycodify_formatters.py index 5231509a7..249846beb 100644 --- a/openhcs/serialization/pycodify_formatters.py +++ b/openhcs/serialization/pycodify_formatters.py @@ -1,6 +1,7 @@ from __future__ import annotations import inspect +import sys from collections.abc import Callable from pathlib import Path @@ -38,6 +39,42 @@ def format(self, value, context: FormatContext) -> SourceFragment: return to_source(self.reference_for(value), context) +class RangeFormatter(SourceFormatter): + """A ``range`` (for example a signature default) is written as its own call.""" + + priority = 50 + + def can_format(self, value) -> bool: + return isinstance(value, range) + + def format(self, value: range, context: FormatContext) -> SourceFragment: + return SourceFragment(repr(value), frozenset()) + + +class ModuleExportedCallableFormatter(SourceFormatter): + """A callable object its module exports under its own name, by import. + + NumPy's dispatchers (``numpy.sum`` as a signature default) are callable + objects, not functions; they load back as themselves from their module. + """ + + priority = 20 + + def can_format(self, value) -> bool: + if not callable(value): + return False + module = sys.modules.get(getattr(value, "__module__", None) or "") + name = getattr(value, "__qualname__", None) + return ( + module is not None + and isinstance(name, str) + and getattr(module, name, None) is value + ) + + def format(self, value, context: FormatContext) -> SourceFragment: + return imported_name(value.__module__, value.__qualname__, context) + + class MaterializationSpecFormatter(SourceFormatter): priority = 110 @@ -120,6 +157,48 @@ def _exported_callable_parameter_exclusions(func) -> set[str]: return excluded.difference(contract.overridable_runtime_parameter_names) +def _signature_defaults(func) -> dict[str, object]: + """The parameters this function takes from its signature when a pattern leaves them out. + + A parameter the platform supplies when it calls the function (an artifact + input, a value bound from the step's configuration) is not one of them: + writing its signature default would replace the supplied value. + """ + + public_func = _public_pattern_callable(func) + supplied_when_called = { + *_exported_callable_parameter_exclusions(public_func), + *CallableContract.from_callable(public_func).runtime_owned_parameter_names, + } + return { + name: parameter.default + for name, parameter in inspect.signature(public_func).parameters.items() + if parameter.default is not inspect.Parameter.empty + and name not in supplied_when_called + } + + +def with_every_parameter_written(pattern): + """Write each function of a pattern with every parameter it has a value for. + + A pattern that leaves a parameter out runs with the signature default of + the installed function; the complete document states that value. + """ + + if _is_pattern_tuple(pattern): + func, args = pattern + return (func, {**_signature_defaults(func), **args}) + if isinstance(pattern, list): + return [with_every_parameter_written(item) for item in pattern] + if isinstance(pattern, dict): + return { + key: with_every_parameter_written(item) for key, item in pattern.items() + } + if _public_pattern_callable(pattern) is not None: + return (pattern, _signature_defaults(pattern)) + return pattern + + class FunctionPatternTupleFormatter(SourceFormatter): priority = 85 @@ -231,6 +310,8 @@ def format(self, value: FunctionStep, context: FormatContext) -> SourceFragment: current_value = step_values[name] else: current_value = param.default + if name == "func" and not context.clean_mode: + current_value = with_every_parameter_written(current_value) if name in default_values: default_value = default_values[name] diff --git a/tests/integration/test_compiled_artifact_inspection.py b/tests/integration/test_compiled_artifact_inspection.py index 0067b3a6d..fedff805e 100644 --- a/tests/integration/test_compiled_artifact_inspection.py +++ b/tests/integration/test_compiled_artifact_inspection.py @@ -69,6 +69,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration @contextmanager @@ -191,6 +192,7 @@ def produce(image): transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=object(), + run_declaration=minimal_run_declaration(), ) return ( ZMQCompileArtifactRecord( diff --git a/tests/integration/test_custom_source_transport.py b/tests/integration/test_custom_source_transport.py index 624827fdc..0f774eff9 100644 --- a/tests/integration/test_custom_source_transport.py +++ b/tests/integration/test_custom_source_transport.py @@ -25,6 +25,7 @@ from tests.context_support import ( compiled_context, ) +from tests.run_records import minimal_run_declaration WORKTREE = Path(__file__).resolve().parents[2] PYTHON = Path(sys.executable) @@ -498,6 +499,7 @@ def _compiled_custom_contexts(payload, runtime): pipeline_definition=(), runtime_contexts=contexts, worker_assignments={f"worker_{i}": [axis] for i, axis in enumerate(contexts)}, runtime_environment=runtime, + run_declaration=minimal_run_declaration(), ).transport_contexts diff --git a/tests/integration/test_generated_source_relative_paths.py b/tests/integration/test_generated_source_relative_paths.py index da08cb3f8..6ccc49588 100644 --- a/tests/integration/test_generated_source_relative_paths.py +++ b/tests/integration/test_generated_source_relative_paths.py @@ -6,6 +6,8 @@ from collections.abc import Sequence from pathlib import Path +from polystore.file_content import FileContent, HashedFileContent + from tests.artifact_graphs import admitted_graph from openhcs.core.function_patterns import compile_function_pattern from openhcs.core.function_step_transport import FunctionStepTransportAuthority @@ -36,6 +38,10 @@ def is_dir(self, path: str | Path, backend: str) -> bool: del backend return Path(path).is_dir() + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + del backend + return HashedFileContent.of_physical_file(Path(file_path)) + def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: raise NotImplementedError diff --git a/tests/integration/test_primary_segmentation_diagnostics_journey.py b/tests/integration/test_primary_segmentation_diagnostics_journey.py index c70e45d2e..c5ff60ab5 100644 --- a/tests/integration/test_primary_segmentation_diagnostics_journey.py +++ b/tests/integration/test_primary_segmentation_diagnostics_journey.py @@ -14,6 +14,7 @@ from openhcs.core.axes import Ungrouped from openhcs.domains.microscopy.axes import Microscopy from openhcs.core.dataset_sources.source_bindings_source import SourceBindingsSource +from tests.run_records import minimal_run_record def _field(empty=False): @@ -364,7 +365,7 @@ def step(func, name, kwargs): primary_plan = context.step_plans[0] assert primary_plan.write_backend == "memory" assert primary_plan.materialized_output is None - OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts) + OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts, run_record=minimal_run_record()) metadata_path = METADATA_CONFIG.metadata_path(primary_plan.output_dataset_root) metadata_document = json.loads(metadata_path.read_text()) persisted_source_provenance = SourceImageProvenance.from_mapping( diff --git a/tests/integration/test_run_record_round_trip.py b/tests/integration/test_run_record_round_trip.py new file mode 100644 index 000000000..bfc1a94a0 --- /dev/null +++ b/tests/integration/test_run_record_round_trip.py @@ -0,0 +1,141 @@ +"""DL3: the record read from one output file is enough to run the pipeline again.""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from queue import Queue + +import pytest +from objectstate.lazy_factory import ensure_global_config_context + +from openhcs.core.config import GlobalPipelineConfig, LazyPathPlanningConfig, PipelineConfig +from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator +from openhcs.core.progress import ProgressExecutionContext, set_progress_queue +from openhcs.core.run_record import PRODUCED_FILES_KEY, RunRecord +from openhcs.core.steps.function_step import FunctionStep +from openhcs.processing.backends.processors.numpy_processor import ( + stack_percentile_normalize, +) +from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, +) +from tests.unit.test_artifact_publication_journey import _plate +from tests.unit.test_run_record import count_bright_source, register # noqa: F401 + + +@pytest.fixture(autouse=True) +def progress_events(): + set_progress_queue(Queue()) + yield + set_progress_queue(None) + + +def run(plate: Path, pipeline_config: PipelineConfig, steps: list) -> Path: + """Run the steps on the plate and return its output dataset root.""" + set_progress_queue(Queue()) + orchestrator = PipelineOrchestrator(plate, pipeline_config=pipeline_config).initialize() + results = orchestrator.execute_compiled_bundle( + orchestrator.compile_pipelines(steps), + progress_queue=Queue(), + progress_context=ProgressExecutionContext( + execution_id="run-record-round-trip", subject_id=str(plate) + ), + ) + assert all(result.is_success() for result in results.values()), { + axis: result.error_message for axis, result in results.items() + } + return plate.with_name(f"{plate.name}_out") + + +def produced_file_bytes(output_root: Path) -> dict[str, str]: + """sha256 of every file the dataset records as a step output, by relative path.""" + produced = json.loads((output_root / "openhcs_metadata.json").read_text())[ + PRODUCED_FILES_KEY + ] + return { + relative: hashlib.sha256((output_root / relative).read_bytes()).hexdigest() + for relative in sorted(produced) + } + + +SUBMISSION_SCOPE = re.compile(r"::pipeline::submission_[0-9a-f]+") + + +def without_run_location(record: RunRecord, plate: Path) -> str: + """The record with this run's plate path and submission scope written as placeholders.""" + text = json.dumps(record.content, sort_keys=True).replace(str(plate), "") + return SUBMISSION_SCOPE.sub("::pipeline::", text) + + +def test_record_read_from_one_output_file_reruns_to_identical_outputs(tmp_path, register): + (tmp_path / "first").mkdir() + (tmp_path / "second").mkdir() + first_plate = _plate(tmp_path / "first" / "source", site_count=2) + first_root = run( + first_plate, + PipelineConfig( + path_planning_config=LazyPathPlanningConfig(output_dir_suffix="_out"), + num_workers=1, + use_threading=True, + ), + [ + FunctionStep( + func=(stack_percentile_normalize, {"low_percentile": 2.0}), name="norm" + ), + FunctionStep(func=register(count_bright_source(0.5)), name="bright"), + ], + ) + first_files = produced_file_bytes(first_root) + tables = [relative for relative in first_files if relative.endswith(".csv")] + images = [relative for relative in first_files if relative.endswith(".tif")] + assert tables and images + + produced = RunRecord.for_output_file(first_root / tables[0]) + assert produced.step["name"] == "bright" + CustomFunctionRuntimeRegistry.clear() + document, global_config = produced.record.rebuild() + + ensure_global_config_context(GlobalPipelineConfig, global_config) + second_plate = _plate(tmp_path / "second" / "source", site_count=2) + second_root = run(second_plate, document.pipeline_config, list(document.pipeline_steps)) + + assert produced_file_bytes(second_root) == first_files + second_record = RunRecord.for_output_file(second_root / tables[0]).record + differences = differing_values( + json.loads(without_run_location(produced.record, first_plate)), + json.loads(without_run_location(second_record, second_plate)), + ) + # The rebuilt document writes every parameter, so a value the first run took + # from a signature default is, in the second run, written in the pipeline. + # Nothing else differs once the plate path and the submission scope are set aside. + assert set(differences.values()) == { + ("function_default", "pipeline"), + ("openhcs:processors_numpy_processor_stack_percentile_normalize", None), + ("openhcs:count_bright", None), + } + assert all( + re.fullmatch(r"/steps/\d+/functions/\d+/parameters/\d+/source/(kind|function)", where) + for where in differences + ) + + +def differing_values(first, second, path: str = "") -> dict[str, tuple]: + """Every leaf at which two JSON values differ, by its path.""" + if isinstance(first, dict) and isinstance(second, dict): + return { + where: values + for key in sorted({*first, *second}) + for where, values in differing_values( + first.get(key), second.get(key), f"{path}/{key}" + ).items() + } + if isinstance(first, list) and isinstance(second, list) and len(first) == len(second): + return { + where: values + for index, (left, right) in enumerate(zip(first, second, strict=True)) + for where, values in differing_values(left, right, f"{path}/{index}").items() + } + return {} if first == second else {path: (first, second)} diff --git a/tests/run_records.py b/tests/run_records.py new file mode 100644 index 000000000..58da79ec7 --- /dev/null +++ b/tests/run_records.py @@ -0,0 +1,34 @@ +"""The smallest valid run declaration and run record, for tests of other behaviour.""" + +from types import MappingProxyType + +from openhcs.core.run_record import ProducedFiles, RunDeclaration, RunRecord + + +def minimal_run_declaration() -> RunDeclaration: + """A declaration of an empty pipeline with no resolved configuration.""" + return RunDeclaration( + pipeline_code="pipeline_steps = []\n", + global_config_code="", + configuration=(), + ) + + +def minimal_run_record() -> RunRecord: + """A record with every stored key and no steps.""" + declaration = minimal_run_declaration() + return RunRecord( + { + "software": {}, + "pipeline_code": declaration.pipeline_code, + "global_config_code": declaration.global_config_code, + "configuration": [], + "steps": [], + "custom_sources": {}, + } + ) + + +def no_produced_files() -> ProducedFiles: + """A run that reports no saved file.""" + return ProducedFiles(MappingProxyType({})) diff --git a/tests/unit/agent/test_agent_services.py b/tests/unit/agent/test_agent_services.py index 5179542cc..5f170e3da 100644 --- a/tests/unit/agent/test_agent_services.py +++ b/tests/unit/agent/test_agent_services.py @@ -130,6 +130,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration def sample_processing_function(image, sigma: float = 1.0): @@ -302,6 +303,7 @@ def _compile_inspection_result( transport_contexts=runtime_contexts, worker_assignments=worker_assignments, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ), source_workspace_projection=source_workspace_projection, ) diff --git a/tests/unit/test_artifact_plan_view.py b/tests/unit/test_artifact_plan_view.py index f034d6448..16843dcc8 100644 --- a/tests/unit/test_artifact_plan_view.py +++ b/tests/unit/test_artifact_plan_view.py @@ -65,6 +65,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration class QtApplication: @@ -147,6 +148,7 @@ def transform(image, labels): transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=object(), + run_declaration=minimal_run_declaration(), ) inspection = CompiledArtifactInspection.from_execution_bundle( compile_artifact_id="compile-1", diff --git a/tests/unit/test_artifact_publication_journey.py b/tests/unit/test_artifact_publication_journey.py index d7e0785f3..5771e2fc5 100644 --- a/tests/unit/test_artifact_publication_journey.py +++ b/tests/unit/test_artifact_publication_journey.py @@ -57,6 +57,7 @@ from openhcs.core.pipeline.function_contracts import ( FullStackLabels, ) +from tests.run_records import minimal_run_record @artifact_outputs(ArtifactSpec.output("objects", ObjectLabelsArtifactType)) @@ -177,7 +178,8 @@ def test_converted_checkpoint_has_typed_address_on_final_reconciliation(tmp_path for index, step in enumerate(steps) )) OpenHCSMetadataTarget.finalize_completed_dataset( - contexts, runtime_observations=(observations,) + contexts, runtime_observations=(observations,), + run_record=minimal_run_record(), ) saved = tmp_path / "source_out" / "converted" files = tuple(saved.glob("*.tif")) diff --git a/tests/unit/test_cellprofiler_pipeline_import.py b/tests/unit/test_cellprofiler_pipeline_import.py index 5ca762110..4febdbeb8 100644 --- a/tests/unit/test_cellprofiler_pipeline_import.py +++ b/tests/unit/test_cellprofiler_pipeline_import.py @@ -3,12 +3,14 @@ from __future__ import annotations import ast +import hashlib from collections.abc import Callable, Sequence from dataclasses import replace from pathlib import Path import pytest from objectstate import config_context +from polystore.file_content import FileContent, HashedFileContent from openhcs.constants import Backend from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep @@ -118,6 +120,11 @@ def is_dir(self, path: str | Path, backend: str) -> bool: del backend return Path(path) == Path("pipelines") + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + del backend + stored = self.files[Path(file_path)].encode("utf-8") + return HashedFileContent(hashlib.sha256(stored).hexdigest(), len(stored)) + def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: del backend, kwargs return self.files[Path(file_path)] diff --git a/tests/unit/test_compilation_paths.py b/tests/unit/test_compilation_paths.py index 5ff4b4f58..8a8ee83c7 100644 --- a/tests/unit/test_compilation_paths.py +++ b/tests/unit/test_compilation_paths.py @@ -11,6 +11,7 @@ import pytest from objectstate import ObjectState, ObjectStateRegistry +from polystore.file_content import FileContent, UnhashedFileContent from python_introspect import Enableable from tests.artifact_graphs import admitted_graph @@ -70,6 +71,12 @@ def is_dir(self, path: str | Path, backend: str) -> bool: self.validation_calls.append(("is_dir", target, backend)) return target in self.directories + def file_content(self, file_path: str | Path, backend: str) -> FileContent: + del backend + return UnhashedFileContent( + f"{type(self).__name__} names {Path(file_path)} without holding its bytes." + ) + def load(self, file_path: str | Path, backend: str, **kwargs: object) -> object: raise NotImplementedError diff --git a/tests/unit/test_compiled_execution.py b/tests/unit/test_compiled_execution.py index c64467c2b..2c235d4ff 100644 --- a/tests/unit/test_compiled_execution.py +++ b/tests/unit/test_compiled_execution.py @@ -56,6 +56,7 @@ compiled_context, executing_context, ) +from tests.run_records import minimal_run_declaration def _runtime_environment() -> CompiledRuntimeEnvironmentPlan: @@ -91,6 +92,7 @@ def resolve_for_transport(contexts): runtime_contexts={"A01": runtime_context}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert resolver_calls == [{"A01": runtime_context}] @@ -113,6 +115,7 @@ def test_compiled_execution_bundle_derives_axis_ids_from_contexts() -> None: transport_contexts={}, worker_assignments={}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert bundle.axis_ids == ("A01", "B01") @@ -152,6 +155,7 @@ def test_compiled_execution_bundle_derives_runtime_observation_mode( transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert RuntimeObservationMode.for_compiled_bundle(bundle) is expected_mode @@ -201,6 +205,7 @@ def test_consolidation_does_not_require_parent_payload_records( transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) assert bundle.requires_parent_runtime_observation is False diff --git a/tests/unit/test_function_outputs.py b/tests/unit/test_function_outputs.py index 8a2605856..3d45eb72b 100644 --- a/tests/unit/test_function_outputs.py +++ b/tests/unit/test_function_outputs.py @@ -112,6 +112,7 @@ from openhcs.core.streaming_config_declarations import NapariViewer from openhcs.core.payload_axes import PayloadAxes from openhcs.core.config import IncludePartitions +from tests.run_records import minimal_run_record, no_produced_files def _publish_saved_step(context, plan, *, artifact_materializations=()): @@ -130,6 +131,7 @@ def _publish_saved_step(context, plan, *, artifact_materializations=()): metadata_path, {target: context for target in entries if target.create_openhcs_metadata}, produced_entries_by_target=entries, + produced_files=no_produced_files(), run_record=minimal_run_record(), ) @@ -169,12 +171,14 @@ def test_function_step_metadata_follows_runtime_artifact_persistence( "runtime_artifacts", ), (OpenHCSMetadataTarget, "observe_for_step", "metadata"), + (OpenHCSMetadataTarget, "saved_path_producers", "producers"), ): monkeypatch.setattr( authority, method_name, lambda _context, _plan, event=event, **_kwargs: ( - events.append(event), {} if event == "metadata" else () + events.append(event), + {} if event in ("metadata", "producers") else (), )[1], ) @@ -186,6 +190,7 @@ def test_function_step_metadata_follows_runtime_artifact_persistence( "stream", "runtime_artifacts", "metadata", + "producers", ] @@ -238,7 +243,8 @@ def test_metadata_document_targets_follow_backend_declaration( lambda _self, metadata_path, *_args, **_kwargs: documents.append(metadata_path), ) OpenHCSMetadataTarget.finalize_completed_dataset( - {"A01": SimpleNamespace(step_plans={0: plan})} + {"A01": SimpleNamespace(step_plans={0: plan})}, + run_record=minimal_run_record(), ) assert documents == ( [METADATA_CONFIG.metadata_path(root)] if expected_targets else [] @@ -1710,7 +1716,7 @@ def test_produced_projection_metadata_persists_typed_collapsed_semantics( Backend.MEMORY.value, ) _publish_saved_step(context, plan) - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) subdirectory = json.loads( (dataset_root / "openhcs_metadata.json").read_text(encoding="utf-8") @@ -2185,7 +2191,7 @@ def reject_generated_path_parse(_filename): # must supply the same addresses, calibration and exact saved coverage. for record in records: filemanager.delete(record.output_path, Backend.MEMORY.value) - OpenHCSMetadataTarget.finalize_completed_dataset({well: context}) + OpenHCSMetadataTarget.finalize_completed_dataset({well: context}, run_record=minimal_run_record()) subdirectory = json.loads((dataset_root / "openhcs_metadata.json").read_text())[ FIELDS.SUBDIRECTORIES ]["images"] @@ -2206,14 +2212,14 @@ def reject_generated_path_parse(_filename): unregistered = output_dir / "unowned_s001_w1_z001_t001.tif" tifffile.imwrite(unregistered, pixels) with pytest.raises(MetadataWriteError, match="lack typed produced addresses"): - OpenHCSMetadataTarget.finalize_completed_dataset({well: context}) + OpenHCSMetadataTarget.finalize_completed_dataset({well: context}, run_record=minimal_run_record()) assert metadata_path.read_bytes() == previous_metadata unregistered.unlink() # This fixture owns the synthetic file. if len(paths) > 1: removed = paths[0] (dataset_root / removed).unlink() - OpenHCSMetadataTarget.finalize_completed_dataset({well: context}) + OpenHCSMetadataTarget.finalize_completed_dataset({well: context}, run_record=minimal_run_record()) reconciled = json.loads(metadata_path.read_text())[FIELDS.SUBDIRECTORIES][ "images" ] @@ -2330,10 +2336,12 @@ def test_completed_plate_metadata_includes_outputs_written_after_owner_axis( dataset_root / "openhcs_metadata.json", {target: owner_context for target in combined_facts}, produced_entries_by_target=combined_facts, + produced_files=no_produced_files(), run_record=minimal_run_record(), ) OpenHCSMetadataTarget.finalize_completed_dataset( - {"A01": owner_context, "B03": follower_context} + {"A01": owner_context, "B03": follower_context}, + run_record=minimal_run_record(), ) metadata = json.loads( @@ -2360,7 +2368,7 @@ def test_completed_plate_metadata_skips_unmaterialized_output_target(tmp_path): plan.create_openhcs_metadata = True context.step_plans = {plan.step_index: plan} - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) assert not (dataset_root / "openhcs_metadata.json").exists() @@ -2529,9 +2537,9 @@ def test_declared_image_destinations_publish_and_reconcile_after_value_cleanup( if stray_image: tifffile.imwrite(tmp_path / "images/review/unowned.tif", pixels) with pytest.raises(MetadataWriteError, match="lack typed produced addresses"): - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) return - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) subdirectories = json.loads((tmp_path / "openhcs_metadata.json").read_text())[ FIELDS.SUBDIRECTORIES ] @@ -2646,7 +2654,7 @@ def test_actual_array_exports_only_publish_declared_raster_inventory( context, plan, artifact_materializations=materializations ) context.runtime_value_store.clear() - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) metadata_path = tmp_path / "openhcs_metadata.json" subdirectories = json.loads(metadata_path.read_text())[FIELDS.SUBDIRECTORIES] assert set(subdirectories) == {"images/exports"} diff --git a/tests/unit/test_function_step_transport.py b/tests/unit/test_function_step_transport.py index 58ba0af7d..04c8239ac 100644 --- a/tests/unit/test_function_step_transport.py +++ b/tests/unit/test_function_step_transport.py @@ -49,6 +49,7 @@ from tests.context_support import ( compiled_context, ) +from tests.run_records import minimal_run_declaration def _execute_spawned_custom_invocation( @@ -458,6 +459,7 @@ def test_bundle_transport_preserves_prepared_runtime_contract_identity() -> None runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( GlobalPipelineConfig(), compiled_contexts=contexts, server_mode=False, ), + run_declaration=minimal_run_declaration(), ) transport_context = bundle.transport_contexts["A01"] transport_plan = transport_context.step_plans[0] diff --git a/tests/unit/test_named_checkpoint_publication.py b/tests/unit/test_named_checkpoint_publication.py index 92f04d293..0fa1b84de 100644 --- a/tests/unit/test_named_checkpoint_publication.py +++ b/tests/unit/test_named_checkpoint_publication.py @@ -48,6 +48,7 @@ ) from openhcs.core.axes import TileAxis from openhcs.domains.microscopy.axes import Microscopy +from tests.run_records import minimal_run_record @@ -112,7 +113,8 @@ def test_named_and_ordinary_checkpoint_inventory_owns_every_address(tmp_path, ma for key, context in contexts.items() )) OpenHCSMetadataTarget.finalize_completed_dataset( - contexts, runtime_observations=(observations,) + contexts, runtime_observations=(observations,), + run_record=minimal_run_record(), ) root = tmp_path / "source_out" metadata = json.loads((root / "openhcs_metadata.json").read_text()) @@ -165,4 +167,4 @@ def test_named_and_ordinary_checkpoint_inventory_owns_every_address(tmp_path, ma # error. This is not a missing-address permissive fallback. tifffile.imwrite(root / "checkpoints" / "unaddressed.tif", np.zeros((8, 8), dtype=np.uint16)) with pytest.raises(MetadataWriteError, match="Saved images lack typed produced addresses.*unaddressed"): - OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts) + OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts, run_record=minimal_run_record()) diff --git a/tests/unit/test_orchestrator_lane_planning.py b/tests/unit/test_orchestrator_lane_planning.py index b464db544..8dfca887a 100644 --- a/tests/unit/test_orchestrator_lane_planning.py +++ b/tests/unit/test_orchestrator_lane_planning.py @@ -84,6 +84,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration PROGRESS_CONTEXT = ProgressExecutionContext( execution_id="exec", @@ -248,6 +249,7 @@ def _execute_with_visualizer(monkeypatch, visualizer, *, progress_queue=None): transport_contexts={"A01": context}, worker_assignments={}, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ) request = BundleExecutionRequest( execution_id="exec", @@ -315,6 +317,7 @@ def release_parent_runtime_resources(self, _bundle): _state=None, is_initialized=lambda: True, microscope_handler=object(), + filemanager=object(), ) return execute_bundle_request(orchestrator, request) @@ -385,6 +388,7 @@ def test_compiled_plate_execution_request_uses_bundle_as_runtime_authority(): transport_contexts={"A01": context}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ) request = BundleExecutionRequest( execution_id="exec", @@ -422,6 +426,7 @@ def test_plate_scope_rejects_omitted_runtime_observations(monkeypatch): transport_contexts={"A01": context}, worker_assignments=None, runtime_environment=runtime_environment, + run_declaration=minimal_run_declaration(), ) request = BundleExecutionRequest( execution_id="exec", @@ -539,6 +544,7 @@ def test_resource_modes_select_prepared_runtime_or_serial_transport_contexts(): use_threading=False, start_method=MultiprocessingStartMethod.SPAWN, ), + run_declaration=minimal_run_declaration(), ) cancellation = ExecutionCancellationSignal() resources = ( @@ -1026,6 +1032,7 @@ def fake_execute_worker_lane( use_threading=False, start_method=MultiprocessingStartMethod.FORK, ), + run_declaration=minimal_run_declaration(), ) ) try: @@ -1335,7 +1342,7 @@ def test_compiled_execution_returns_settled_nonpersistent_viewer_state_before_cl monkeypatch.setattr( compiled_plate_execution_module.OpenHCSMetadataTarget, "finalize_completed_dataset", - lambda _contexts, *, runtime_observations: events.append("metadata"), + lambda _contexts, *, runtime_observations, run_record: events.append("metadata"), ) response = ViewerControlResponse( payload={ diff --git a/tests/unit/test_persisted_image_metadata.py b/tests/unit/test_persisted_image_metadata.py index 50e3f9408..1e6834009 100644 --- a/tests/unit/test_persisted_image_metadata.py +++ b/tests/unit/test_persisted_image_metadata.py @@ -46,6 +46,7 @@ from openhcs.domains.microscopy.axes import Microscopy from openhcs.core.payload_axes import PayloadAxes from openhcs.core.axes import ColourAxis +from tests.run_records import minimal_run_record, no_produced_files def metadata_fixture(): @@ -167,7 +168,8 @@ def receive_projection_entries(writer, metadata_path, subdirectory, entries): backend="disk", results_dir=None, create_openhcs_metadata=False, ) writer.reconcile_completed_dataset( - metadata_path, {}, produced_entries_by_target={target: entries} + metadata_path, {}, produced_entries_by_target={target: entries}, + produced_files=no_produced_files(), run_record=minimal_run_record(), ) diff --git a/tests/unit/test_persisted_roi_only_publication.py b/tests/unit/test_persisted_roi_only_publication.py index e32e793c2..0f2afa75e 100644 --- a/tests/unit/test_persisted_roi_only_publication.py +++ b/tests/unit/test_persisted_roi_only_publication.py @@ -49,6 +49,7 @@ executing_context, with_facts, ) +from tests.run_records import minimal_run_record, no_produced_files def _publish_saved_step(context, plan, *, artifact_materializations=()): @@ -67,6 +68,7 @@ def _publish_saved_step(context, plan, *, artifact_materializations=()): metadata_path, {target: context for target in entries if target.create_openhcs_metadata}, produced_entries_by_target=entries, + produced_files=no_produced_files(), run_record=minimal_run_record(), ) @@ -246,7 +248,7 @@ def test_saved_roi_only_batch_publishes_and_reconciles_without_rendering( for target in owner.reconciliation_targets(publication_context) ) assert plate / destination in reconciled - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}, run_record=minimal_run_record()) assert result_paths(plate, publication_context) == (output.path,) @@ -275,7 +277,7 @@ def test_result_only_destination_saved_through_zarr_passthrough_publishes_comple plan = plan_for(plate, Backend.ZARR.value) publication_context = with_facts(publication_context, step_plans={0: plan}) _publish_saved_step(publication_context, plan, artifact_materializations=(outcome,)) - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}, run_record=minimal_run_record()) published = json.loads(METADATA_CONFIG.metadata_path(plate).read_text()) entry = published["subdirectories"]["results"] @@ -345,6 +347,7 @@ def test_measurement_only_step_does_not_reconcile_pending_image_producer( AtomicMetadataWriter().reconcile_completed_dataset( metadata_path, {target: publication_context}, produced_entries_by_target=staged, + produced_files=no_produced_files(), run_record=minimal_run_record(), ) assert not metadata_path.exists() @@ -360,7 +363,7 @@ def test_measurement_only_step_does_not_reconcile_pending_image_producer( SourceProjectionMetadataSerializer.projection_fields(((projection, virtual_path),)), ) publication_context = with_facts(publication_context, step_plans={0: plan}) - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}, run_record=minimal_run_record()) reconciled = json.loads(metadata_path.read_text())[FIELDS.SUBDIRECTORIES][destination] assert reconciled[FIELDS.IMAGE_FILES] == [virtual_path] assert len(reconciled[FIELDS.SOURCE_PROJECTION]) == 1 diff --git a/tests/unit/test_run_record.py b/tests/unit/test_run_record.py new file mode 100644 index 000000000..ea375c6b4 --- /dev/null +++ b/tests/unit/test_run_record.py @@ -0,0 +1,507 @@ +"""DL3: an output dataset says what produced it. One test per rule.""" + +from __future__ import annotations + +import hashlib +import importlib.metadata +import json +import pickle +from pathlib import Path +from queue import Queue + +import inspect + +import pytest +from polystore.streaming.identity import StreamProducerIdentity + +import openhcs.processing.custom_functions.manager as manager_module +from openhcs.constants import Backend +from openhcs.core.callable_contract import CallableContract +from openhcs.core.config import LazyPathPlanningConfig, PipelineConfig +from openhcs.core.function_identity import ModuleLibraries +from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator +from openhcs.core.pipeline_document import PipelineDocumentCodec +from openhcs.core.progress import ProgressExecutionContext, set_progress_queue +from openhcs.core.run_record import ( + PRODUCED_FILES_KEY, + RUN_RECORDS_KEY, + ProducedFiles, + RunDeclaration, + RunRecord, + UnrecordedOutputFile, + python_expression, +) +from openhcs.core.steps.function_step import FunctionStep +from openhcs.core.virtual_workspace_metadata import AtomicMetadataWriter +from openhcs.processing.backends.lib_registry.registry_service import RegistryService +from openhcs.processing.backends.processors.numpy_processor import ( + stack_percentile_normalize, +) +from openhcs.processing.custom_functions import CustomFunctionManager +from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, +) +from tests.run_records import minimal_run_record +from tests.unit.test_artifact_publication_journey import _plate, tiny_counts + +NORMALIZE_KEY = "openhcs:processors_numpy_processor_stack_percentile_normalize" + +COUNT_BRIGHT = '''import numpy as np + +from openhcs.core.artifacts.kinds import MeasurementsArtifactType +from openhcs.core.artifacts.relations import ArtifactMeasurementSubjectRelation +from openhcs.core.artifacts.specs import ArtifactSpec +from openhcs.core.measurement_row_materialization import MeasurementSparseColumnarRows +from openhcs.core.memory import numpy +from openhcs.core.pipeline.function_contracts import artifact_outputs +from openhcs.core.runtime_tabular_values import FieldSpec +from openhcs.core.vfs_protocol import PlateInputFile, PlateOutputFile + + +@numpy +@artifact_outputs( + ArtifactSpec.output( + "bright", + MeasurementsArtifactType, + relations=(ArtifactMeasurementSubjectRelation(),), + ) +) +def count_bright( + image, + threshold: float = THRESHOLD, + weights: PlateInputFile = None, + notes: PlateOutputFile = None, +): + """Count pixels above a threshold.""" + return image, MeasurementSparseColumnarRows.from_rows( + ({"bright": int(np.count_nonzero(image > threshold))},), + fields=(FieldSpec("bright", int),), + ) +''' + + +def count_bright_source(threshold: float) -> str: + return COUNT_BRIGHT.replace("THRESHOLD", repr(threshold)) + + +@pytest.fixture(autouse=True) +def progress_events(): + set_progress_queue(Queue()) + yield + set_progress_queue(None) + + +@pytest.fixture +def register(monkeypatch, tmp_path): + """Register custom source text the way the editor does, without persisting it.""" + storage_dir = tmp_path / "custom_functions" + storage_dir.mkdir() + monkeypatch.setattr( + manager_module, "get_data_file_path", lambda _name, *, create=True: storage_dir + ) + for attribute in ( + "_declarations_by_name", + "_published_exports", + "_preparation_outcomes", + "_preparation_threads", + ): + monkeypatch.setattr(CustomFunctionRuntimeRegistry, attribute, {}) + monkeypatch.setattr(CustomFunctionRuntimeRegistry, "_source_revision", None) + + def register_source(code: str): + CustomFunctionRuntimeRegistry.clear() + [function] = CustomFunctionManager().register_from_code( + code, persist=False, emit_signal=False + ) + return function + + yield register_source + CustomFunctionRuntimeRegistry.clear() + + +def orchestrator_for(plate: Path, **config_fields) -> PipelineOrchestrator: + return PipelineOrchestrator( + plate, + pipeline_config=PipelineConfig( + path_planning_config=LazyPathPlanningConfig(output_dir_suffix="_out"), + num_workers=1, + use_threading=True, + **config_fields, + ), + ).initialize() + + +def record_of(orchestrator: PipelineOrchestrator, steps: list) -> RunRecord: + return RunRecord.at_run_start( + orchestrator.compile_pipelines(steps), + filemanager=orchestrator.filemanager, + backend=Backend.DISK.value, + ) + + +def run(orchestrator: PipelineOrchestrator, steps: list, tmp_path: Path): + set_progress_queue(Queue()) + bundle = orchestrator.compile_pipelines(steps) + results = orchestrator.execute_compiled_bundle( + bundle, + progress_queue=Queue(), + progress_context=ProgressExecutionContext( + execution_id="run-record", subject_id=str(tmp_path) + ), + ) + assert all(result.is_success() for result in results.values()), { + axis: result.error_message for axis, result in results.items() + } + return bundle + + +def by_name(records: list[dict]) -> dict[str, dict]: + return {record["name"]: record for record in records} + + +def only_function(record: RunRecord, step_name: str) -> dict: + (step,) = (step for step in record.steps if step["name"] == step_name) + (function,) = step["functions"] + return function + + +def normalize_and_count() -> list: + return [ + FunctionStep(func=(stack_percentile_normalize, {"low_percentile": 2.0}), name="norm"), + FunctionStep(func=tiny_counts, name="count"), + ] + + +def test_parameter_left_out_is_recorded_with_the_function_default(tmp_path): + record = record_of(orchestrator_for(_plate(tmp_path / "source")), normalize_and_count()) + + function = only_function(record, "norm") + parameters = by_name(function["parameters"]) + assert function["identity"]["key"] == NORMALIZE_KEY + assert parameters["low_percentile"] == { + "name": "low_percentile", + "value": "2.0", + "source": {"kind": "pipeline"}, + } + assert parameters["high_percentile"] == { + "name": "high_percentile", + "value": "99.0", + "source": {"kind": "function_default", "function": NORMALIZE_KEY}, + } + assert parameters["stack"]["source"] == {"kind": "run_time"} + + +def test_saved_pipeline_text_stays_short_and_the_record_copy_is_complete(tmp_path): + steps = normalize_and_count() + record = record_of(orchestrator_for(_plate(tmp_path / "source")), steps) + + document = PipelineDocumentCodec.from_source(record.content["pipeline_code"]) + saved_text = PipelineDocumentCodec.render(document) + assert "low_percentile" in saved_text + assert "high_percentile" not in saved_text + assert "description=None" not in saved_text + assert "high_percentile" in record.content["pipeline_code"] + assert "description=None" in record.content["pipeline_code"] + assert ( + PipelineDocumentCodec.render(document, clean_mode=False) + == record.content["pipeline_code"] + ) + + +def test_compiled_bundle_with_its_declaration_pickles(tmp_path): + bundle = orchestrator_for(_plate(tmp_path / "source")).compile_pipelines( + normalize_and_count() + ) + + restored = pickle.loads(pickle.dumps(bundle.for_transport_serialization())) + + assert restored.run_declaration == bundle.run_declaration + + +def test_custom_function_is_identified_by_its_source_text(tmp_path, register): + orchestrator = orchestrator_for(_plate(tmp_path / "source")) + records = {} + for threshold in (0.5, 0.25): + source = count_bright_source(threshold) + record = record_of( + orchestrator, [FunctionStep(func=register(source), name="bright")] + ) + identity = only_function(record, "bright")["identity"] + sha256 = hashlib.sha256(source.encode("utf-8")).hexdigest() + assert identity["name"] == "count_bright" + assert identity["custom_source_sha256"] == sha256 + assert record.content["custom_sources"] == {sha256: source} + records[threshold] = record + + first, second = ( + only_function(record, "bright")["identity"] for record in records.values() + ) + assert first["key"] == second["key"] + assert first["custom_source_sha256"] != second["custom_source_sha256"] + assert records[0.5].record_id != records[0.25].record_id + + +def test_custom_function_imports_name_installed_versions(tmp_path, register): + record = record_of( + orchestrator_for(_plate(tmp_path / "source")), + [FunctionStep(func=register(count_bright_source(0.5)), name="bright")], + ) + + libraries = only_function(record, "bright")["identity"]["imports"]["libraries"] + assert {"distribution": "numpy", "version": importlib.metadata.version("numpy")} in ( + libraries + ) + + +def test_module_without_a_distribution_is_listed_not_dropped(): + libraries = ModuleLibraries.imported_by_source( + "import json\nimport numpy\nimport module_no_distribution_provides.part\n" + ) + + assert libraries.modules_without_distribution == ("module_no_distribution_provides",) + assert [library.name for library in libraries.libraries] == ["numpy"] + + +def test_declared_input_file_is_hashed_once_per_run_by_the_file_manager( + tmp_path, register, monkeypatch +): + plate = _plate(tmp_path / "source") + weights = plate / "weights.bin" + weights.write_bytes(b"first weights") + orchestrator = orchestrator_for(plate) + hashed_paths: list[Path] = [] + file_content = orchestrator.filemanager.file_content + + def spy(file_path, backend): + hashed_paths.append(Path(file_path)) + return file_content(file_path, backend) + + monkeypatch.setattr(orchestrator.filemanager, "file_content", spy) + count_bright = register(count_bright_source(0.5)) + + def compiled(): + return orchestrator.compile_pipelines( + [ + FunctionStep( + func=(count_bright, {"weights": "weights.bin", "notes": "notes/log.txt"}), + name="bright", + ) + ] + ) + + def declared_files(bundle) -> dict[str, dict]: + record = RunRecord.at_run_start( + bundle, filemanager=orchestrator.filemanager, backend=Backend.DISK.value + ) + return { + declared["parameter"]: declared + for declared in only_function(record, "bright")["declared_files"] + } + + bundle = compiled() + assert hashed_paths == [] + + declared = declared_files(bundle) + assert hashed_paths == [weights] + assert declared["weights"] == { + "parameter": "weights", + "declared_as": "input file", + "path": str(weights), + "content": { + "hashed": True, + "sha256": hashlib.sha256(b"first weights").hexdigest(), + "size_bytes": len(b"first weights"), + }, + } + assert declared["notes"]["declared_as"] == "output file" + assert declared["notes"]["content"]["hashed"] is False + assert declared["notes"]["content"]["reason"].strip() + + weights.write_bytes(b"second weights!") + assert declared_files(bundle)["weights"]["content"] == { + "hashed": True, + "sha256": hashlib.sha256(b"second weights!").hexdigest(), + "size_bytes": len(b"second weights!"), + } + + +def test_resolved_configuration_names_the_source_of_each_value(tmp_path, monkeypatch): + resolved_pipelines = [] + with_resolved_steps = RunDeclaration.with_resolved_steps + + def keep_resolved_pipeline(declaration, pipeline): + resolved_pipelines.append(pipeline) + return with_resolved_steps(declaration, pipeline) + + monkeypatch.setattr(RunDeclaration, "with_resolved_steps", keep_resolved_pipeline) + plate = _plate(tmp_path / "source") + record = record_of(orchestrator_for(plate), normalize_and_count()) + + configuration = by_name(record.content["configuration"]) + assert configuration["num_workers"]["value"] == "1" + assert configuration["num_workers"]["source"] == {"kind": "set_here", "scope": str(plate)} + inherited = configuration["materialization_results_path"]["source"] + assert inherited["kind"] == "inherited" + assert inherited["type"] == "openhcs.core.config.GlobalPipelineConfig" + + (pipeline,) = resolved_pipelines + assert len(record.steps) == len(pipeline.steps) == 2 + for index, step in enumerate(record.steps): + recorded_names = {value["name"] for value in step["configuration"]} + assert pipeline.step_provenance[index] + assert set(pipeline.step_provenance[index]) <= recorded_names + + +def test_each_output_file_names_the_step_that_produced_it(tmp_path): + plate = _plate(tmp_path / "source") + orchestrator = orchestrator_for(plate) + bundle = run(orchestrator, normalize_and_count(), tmp_path) + output_root = tmp_path / "source_out" + produced = json.loads((output_root / "openhcs_metadata.json").read_text())[ + PRODUCED_FILES_KEY + ] + (image,) = (path for path in produced if path.endswith(".tif")) + (table,) = (path for path in produced if path.endswith(".csv")) + + image_file = RunRecord.for_output_file(output_root / image) + table_file = RunRecord.for_output_file(output_root / table) + + # "count" returns its image unchanged, so the saved pixels are still the + # ones "norm" produced: the same producer a viewer item for them carries. + assert image_file.step["name"] == "norm" + assert table_file.step["name"] == "count" + assert table_file.record.record_id == image_file.record.record_id + (context,) = bundle.runtime_contexts.values() + count_plan = context.step_plans[1] + (output_plan,) = count_plan.artifact_outputs.values() + assert produced[table]["producer"] == dict( + count_plan.producer_identity_for_artifact(output_plan).to_payload() + ) + unlisted = output_root / "A01_s001_w1_z001_t001_unlisted.csv" + unlisted.write_text("not written by a step") + with pytest.raises(UnrecordedOutputFile): + RunRecord.for_output_file(unlisted) + + +def test_input_plate_metadata_receives_no_record(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + + document = json.loads((plate / "openhcs_metadata.json").read_text()) + assert RUN_RECORDS_KEY not in document + assert PRODUCED_FILES_KEY not in document + + +def test_second_pipeline_into_the_same_dataset_keeps_only_referenced_records(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + run( + orchestrator_for(plate), + [FunctionStep(func=(stack_percentile_normalize, {"low_percentile": 5.0}), name="norm")], + tmp_path, + ) + + document = json.loads((tmp_path / "source_out" / "openhcs_metadata.json").read_text()) + referenced = {entry["run_record"] for entry in document[PRODUCED_FILES_KEY].values()} + assert set(document[RUN_RECORDS_KEY]) == referenced + assert len(referenced) == 2 + + +def _table_producer() -> StreamProducerIdentity: + return StreamProducerIdentity.pipeline_output( + output_kind="artifact", + output_key="counts", + projection_key="counts", + step_name="count", + pipeline_position=0, + step_scope_id="plate::pipeline::step_0", + artifact_kind="measurements", + ) + + +def test_dataset_root_that_received_only_a_table_gets_a_document_with_the_record(tmp_path): + results_root = tmp_path / "results_only" + table = results_root / "results" / "A01_counts.csv" + table.parent.mkdir(parents=True) + table.write_text("nonzero\n9\n") + record = minimal_run_record() + + AtomicMetadataWriter().reconcile_completed_dataset( + results_root / "openhcs_metadata.json", + {}, + produced_files=ProducedFiles({table: _table_producer()}), + run_record=record, + ) + + produced = RunRecord.for_output_file(table) + assert produced.record.record_id == record.record_id + assert produced.producer == _table_producer() + + +def test_saved_file_is_recorded_under_the_nearest_declared_dataset_root(tmp_path): + outer = tmp_path / "plate_out" + inner = outer / "nested_out" + image = outer / "images" / "a.tif" + table = inner / "results" / "a.csv" + produced = ProducedFiles({image: _table_producer(), table: _table_producer()}) + + by_root = produced.by_dataset_root([outer, inner]) + + assert {root: set(files.producer_by_path) for root, files in by_root.items()} == { + outer: {image}, + inner: {table}, + } + with pytest.raises(ValueError, match="no declared output dataset root"): + produced.by_dataset_root([inner]) + + +def test_subdirectory_metadata_updates_keep_the_records(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + metadata_path = tmp_path / "source_out" / "openhcs_metadata.json" + before = json.loads(metadata_path.read_text()) + assert before[RUN_RECORDS_KEY] and before[PRODUCED_FILES_KEY] + + writer = AtomicMetadataWriter() + writer.merge_subdirectory_metadata(metadata_path, {"added": {"note": "kept"}}) + writer.replace_subdirectory_metadata(metadata_path, "replaced", {"note": "kept"}) + + after = json.loads(metadata_path.read_text()) + assert after[RUN_RECORDS_KEY] == before[RUN_RECORDS_KEY] + assert after[PRODUCED_FILES_KEY] == before[PRODUCED_FILES_KEY] + + +def test_output_dataset_with_records_opens_as_an_input_plate(tmp_path): + plate = _plate(tmp_path / "source") + run(orchestrator_for(plate), normalize_and_count(), tmp_path) + + set_progress_queue(Queue()) + reopened = orchestrator_for(tmp_path / "source_out") + + assert reopened.is_initialized() + record_of(reopened, normalize_and_count()) + + +def test_every_registered_function_default_can_be_written_in_the_record(): + """Compilation writes each default; one it cannot write would stop that pipeline.""" + unwritable = [] + for key, metadata in RegistryService.get_all_functions_with_metadata().items(): + contract = CallableContract.from_callable(metadata.func) + step = FunctionStep(func=metadata.func, name=key) + try: + PipelineDocumentCodec.render( + PipelineDocumentCodec.from_values( + pipeline_config=PipelineConfig(), pipeline_steps=[step] + ), + clean_mode=False, + ) + for name, parameter in contract.canonical_signature.parameters.items(): + if ( + parameter.default is not inspect.Parameter.empty + and name not in contract.runtime_owned_parameter_names + ): + python_expression(parameter.default) + except TypeError as error: + unwritable.append((key, str(error))) + + assert unwritable == [] diff --git a/tests/unit/test_source_round_trip.py b/tests/unit/test_source_round_trip.py index 09417710e..e2e66d260 100644 --- a/tests/unit/test_source_round_trip.py +++ b/tests/unit/test_source_round_trip.py @@ -325,7 +325,22 @@ def test_pipeline_document_loads_back_every_pattern_shape(clean_mode): loaded = PipelineDocumentCodec.from_source(source) assert raw(loaded.pipeline_config) == raw(document.pipeline_config), source - assert raw(loaded.pipeline_steps) == raw(document.pipeline_steps), source + if clean_mode: + assert raw(loaded.pipeline_steps) == raw(document.pipeline_steps), source + else: + # The full text writes every function parameter, so what loads back + # states the signature defaults the original left out: the same + # declaration, the same short text, and the same full text again. + assert all( + loaded_step.same_declaration(step) + for loaded_step, step in zip( + loaded.pipeline_steps, document.pipeline_steps, strict=True + ) + ), source + assert PipelineDocumentCodec.render(loaded) == PipelineDocumentCodec.render( + document + ) + assert PipelineDocumentCodec.render(loaded, clean_mode=False) == source assert samples.unsampled == [] assert "path_root = Path('/data/screen')" in source assert "CropModule.Shape.ELLIPSE" in source diff --git a/tests/unit/test_zmq_compilation.py b/tests/unit/test_zmq_compilation.py index 6fcc02429..86b8db134 100644 --- a/tests/unit/test_zmq_compilation.py +++ b/tests/unit/test_zmq_compilation.py @@ -23,6 +23,7 @@ from tests.context_support import ( executing_context, ) +from tests.run_records import minimal_run_declaration class _StrippedStepShell: @@ -164,6 +165,7 @@ def test_reused_compile_artifact_reads_step_names_from_compiled_plans() -> None: transport_contexts={}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) compilation = ZMQCompilationResult(execution_bundle=bundle) artifacts = { @@ -217,6 +219,7 @@ def test_compile_fresh_emits_heartbeat_during_long_compilation() -> None: transport_contexts={}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) orchestrator = _FreshCompileOrchestrator(bundle) @@ -277,6 +280,7 @@ def test_rejected_reuse_preserves_artifact_for_a_valid_request(retain, invalid): transport_contexts={}, worker_assignments={"worker_0": ["A01"]}, runtime_environment=_runtime_environment(), + run_declaration=minimal_run_declaration(), ) artifact = ZMQCompileArtifactRecord( configs=OpenHCSExecutionConfigBundle(GlobalPipelineConfig(), PipelineConfig()), diff --git a/tests/unit/test_zmq_execution_server.py b/tests/unit/test_zmq_execution_server.py index a45b6bced..e55c02c07 100644 --- a/tests/unit/test_zmq_execution_server.py +++ b/tests/unit/test_zmq_execution_server.py @@ -61,6 +61,7 @@ executing_context, with_facts, ) +from tests.run_records import minimal_run_declaration def test_zmq_execution_context_seeds_saved_global_config_for_compilation() -> None: @@ -416,6 +417,7 @@ def test_zmq_server_admits_compiled_declaration_without_reevaluating_source( compiled_contexts={}, server_mode=True, ), + run_declaration=minimal_run_declaration(), ) server._compiled_artifacts["compile-1"] = ZMQCompileArtifactRecord( execution_id="compile-1", @@ -700,6 +702,7 @@ def test_compiled_source_adoption_owns_fresh_runtime_services_and_live_source_ga compiled_contexts={"A01": context}, server_mode=True, ), + run_declaration=minimal_run_declaration(), ) compiled_zarr = context.filemanager.registry[Backend.ZARR.value] interleaved = PipelineOrchestrator( From 5869373961947a7f2aee87680af83936a466caa4 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 15:06:58 -0400 Subject: [PATCH 04/12] DL3 receipt: progress notes on the run record wiring Co-Authored-By: Claude Opus 5.5 --- .../DL3-output-provenance.md | 38 ++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md index 214d5e64b..a792cbc69 100644 --- a/docs/refactor/consolidation-20261009/DL3-output-provenance.md +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -141,4 +141,40 @@ version and the device; the N2V2 denoise example records the model file's hash; ## Progress notes -(filled in as work proceeds) +**Wired (commit `c2d689e61`).** `CompiledExecutionBundle.run_declaration` (required; built in +`compile_pipelines` before `_resolve_pipeline_once`, step values added after it; pickles with the +bundle). `StepExecutionObservation.producer_by_saved_path`, filled by +`OpenHCSMetadataTarget.saved_path_producers` at the two places a step's observation is built. +`execute_bundle_request` calls `RunRecord.at_run_start` before workers exist and passes the record +to `finalize_completed_dataset`; `reconcile_completed_dataset` writes it inside its one `update`. +A dataset root that received only non-image files gets a document; a document the run saved +nothing under (the input plate) is left untouched. The real ZMQ journey writes a record (59 kB for +one step, of which 246 installed distributions are most). + +**What the design had wrong, and what was done instead.** +- `clean_mode=False` did not write a parameter the pipeline leaves out (it only stopped dropping + written ones). Complete rendering now adds the signature default of every parameter of every + pattern function (`with_every_parameter_written`, `pycodify_formatters.py`), except parameters + the platform supplies when it calls the function (artifact inputs, `slice_by_slice`, + `dtype_config`): writing their signature default would replace the supplied value. +- Two default kinds had no formatter and would have stopped compilation (`range` in + `skimage:filters.frangi`, `numpy.sum` in `skimage:measure.block_reduce`). Both now have one; + `test_every_registered_function_default_can_be_written_in_the_record` guards the registry + installed here (283 functions; CuPy and pyclesperanto are not installed on this machine). +- The WIP stored the *effective* config as the global document. Rebuilding against it gave equal + values but different sources (everything inherited from the global scope). The record now stores + the saved global config (`global_config_code`, as this design says); with none set it stores the + default-constructed one, which the round trip shows resolves identically. +- A step's values are read from `AbstractStep.configuration_parameters()` (the function pattern is + not configuration). An inherited path is checked before descending: a nested config the step + leaves unset is inherited whole and ObjectState lists it as one path. +- The producer of a main-flow image is the step that produced the pixels. A step that returns its + image unchanged saves a file whose producer is the earlier step: the same identity the viewer + item carries. +- A rebuilt run writes every parameter, so its record says `pipeline` where the first said + `function_default`. Nothing else differs apart from the plate path and the submission scope + (`submission_` in step scope ids). Output files (TIFF and CSV) are byte-identical. + +**Not done.** `changed_during_run` (asking again for size and modification time at the end of the +run). The agent capability `openhcs_read_run_record` is being written separately and reads +`effective_config_code`; that key is now `global_config_code`. From 09a712ad5ada6a519cb3f1916a0497115472ad7b Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 15:08:50 -0400 Subject: [PATCH 05/12] DL3: openhcs_read_run_record capability (MCP tool and CLI command) Reads the record of what produced an output dataset, or of one output file, through RunRecord. Long texts are returned as hash and length unless asked for. The complete pipeline text writes every parameter; the authoring round-trip test states that for the full rendering. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/capabilities.py | 40 ++++ openhcs/agent/dto/__init__.py | 4 + openhcs/agent/dto/plate.py | 68 ++++++ openhcs/agent/services/run_record_service.py | 146 +++++++++++++ openhcs/mcp/context.py | 11 + tests/fixtures/mcp/mcp_tool_contract.json | 44 ++++ .../test_authoring_roundtrip_boundaries.py | 10 +- tests/unit/test_run_record_capability.py | 197 ++++++++++++++++++ 8 files changed, 518 insertions(+), 2 deletions(-) create mode 100644 openhcs/agent/services/run_record_service.py create mode 100644 tests/unit/test_run_record_capability.py diff --git a/openhcs/agent/capabilities.py b/openhcs/agent/capabilities.py index ebdfe0040..ceb3013c3 100644 --- a/openhcs/agent/capabilities.py +++ b/openhcs/agent/capabilities.py @@ -143,6 +143,8 @@ from openhcs.agent.dto.plate import ( PlateFileQueryRequest, PlateFileQueryResult, + RunRecordReadRequest, + RunRecordReadResult, PlateFileStreamRequest, PlateFileStreamResult, PlateImageSampleRequest, @@ -2352,6 +2354,44 @@ class QueryPlateFilesCapability(MainThreadProgressCapability, PlatePathCapabilit ) +class ReadRunRecordCapability(PlatePathCapability): + name = "openhcs_read_run_record" + cli_command = "read-run-record" + title = "Read run record" + description = ( + "Read-only: what produced an output dataset. Pass an output dataset root " + "(the directory holding openhcs_metadata.json) to get every run record " + "by record id, or one output file inside a dataset to get the producer " + "identity stored for that file, the producing step's entry (name, " + "pipeline position, functions with identity, parameters with their " + "sources, declared files with content hashes, devices) and the record of " + "that run. A file the dataset does not list is an error, never a guess " + "from its name. By default pipeline_code, global_config_code, " + "custom_sources texts and software.installed_distributions are replaced " + "by their sha256 and length and named in shortened_fields; pass " + "include_sources=true for the full texts." + ) + exposition = PlatePathCapability.exposition.refine( + workflow_stage=CapabilityWorkflowStage.INSPECTION, + visibility=CapabilityVisibility.STANDARD, + ) + service = "run_record" + data_exposure = ( + "local_output_path", + "pipeline_source", + "resolved_configuration", + "custom_function_source", + "installed_distribution_versions", + ) + security_requirements = ("AgentPathPolicy readable root",) + input_contract = RunRecordReadRequest + output_contract = RunRecordReadResult + invocation = AgentDataclassRequestServiceInvocation( + service=lambda context: context.run_record_service, + method=lambda service, request: service.read(request), + ) + + class SamplePlateImageCapability(MainThreadProgressCapability, PlatePathCapability): name = "openhcs_sample_plate_image" cli_command = "sample-plate-image" diff --git a/openhcs/agent/dto/__init__.py b/openhcs/agent/dto/__init__.py index d0033d795..c7b45a335 100644 --- a/openhcs/agent/dto/__init__.py +++ b/openhcs/agent/dto/__init__.py @@ -130,6 +130,10 @@ "PlatePathInspectionRequest", "PlatePathInspectionResult", "PlateWorkspacePreparationOperation", + "ProducedFileEntry", + "RunRecordEntry", + "RunRecordReadRequest", + "RunRecordReadResult", "SelectedPlateFileQueryTarget", "SelectedPlateFileQueryRequest", "SelectedPlateFileQueryResult", diff --git a/openhcs/agent/dto/plate.py b/openhcs/agent/dto/plate.py index 9079b42e4..3ead9b2a0 100644 --- a/openhcs/agent/dto/plate.py +++ b/openhcs/agent/dto/plate.py @@ -13,6 +13,8 @@ from formbridge.contracts import ( AGENT_PARAMETER_DESCRIPTION_METADATA_KEY, + AgentCliArgumentSpec, + AgentDataclassCliRequest, AgentError, AgentResultEnvelope, AgentResourceRef, @@ -1339,3 +1341,69 @@ class SelectedPlateFileStreamResult(AgentResultEnvelope): selected_plate: JsonObject = field(default_factory=dict) target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot stream: PlateFileStreamResult | None = None + + +@dataclass(frozen=True, slots=True) +class RunRecordReadRequest(AgentDataclassCliRequest): + """Read what produced an output dataset, or one output file inside one. + + ``path`` is an output dataset root (the directory that holds the OpenHCS + metadata document) or one output file below such a root. + ``include_sources`` returns the long texts in full; otherwise each is + replaced by its sha256 and length and named in ``shortened_fields``. + """ + + path: str + include_sources: bool = False + + @classmethod + def agent_cli_argument_specs(cls) -> tuple[AgentCliArgumentSpec, ...]: + return ( + AgentCliArgumentSpec( + field_name="path", + positional=True, + help="Output dataset root, or one output file inside one.", + ), + *super(RunRecordReadRequest, cls).agent_cli_argument_specs(), + ) + + +@dataclass(frozen=True, slots=True) +class RunRecordEntry: + """One stored run record, addressed by the sha256 of its full content. + + ``shortened_fields`` names every field of ``record`` whose value was + replaced by ``{"shortened": true, "sha256": ..., "length": ...}``; it is + empty when the record is returned in full. + """ + + record_id: str + record: JsonObject + shortened_fields: tuple[str, ...] = () + + +@dataclass(frozen=True, slots=True) +class ProducedFileEntry: + """One output file with the producer identity stored for it and its producing step.""" + + path: str + record_id: str + producer: JsonObject + step: JsonObject + + +@dataclass(frozen=True, kw_only=True, slots=True) +class RunRecordReadResult(AgentResultEnvelope): + """Run records read from one output dataset. + + For a dataset root ``records`` holds every record and ``produced_files`` is + empty; for an output file ``produced_files`` holds that file and + ``records`` holds the one record of the run that wrote it. + """ + + path: str + include_sources: bool + records: tuple[RunRecordEntry, ...] = () + produced_files: tuple[ProducedFileEntry, ...] = () + errors: tuple[AgentError, ...] = () + warnings: tuple[AgentWarning, ...] = () diff --git a/openhcs/agent/services/run_record_service.py b/openhcs/agent/services/run_record_service.py new file mode 100644 index 000000000..93f42dbec --- /dev/null +++ b/openhcs/agent/services/run_record_service.py @@ -0,0 +1,146 @@ +"""Read the record of what produced an output dataset, for agents.""" + +from __future__ import annotations + +import hashlib +import json +from abc import ABC, abstractmethod +from dataclasses import dataclass +from pathlib import Path + +from python_introspect import JsonObject + +from openhcs.agent.dto.common import SCHEMA_VERSION +from openhcs.agent.dto.plate import ( + ProducedFileEntry, + RunRecordEntry, + RunRecordReadRequest, + RunRecordReadResult, +) +from openhcs.agent.exceptions import AgentFacingErrorMixin +from openhcs.agent.path_policy import AgentPathPolicy +from openhcs.core.run_record import RunRecord, UnrecordedOutputFile +from openhcs.core.virtual_workspace_metadata import get_metadata_path + + +class OutputFileNotRecorded(AgentFacingErrorMixin, LookupError): + """The dataset's metadata does not say which step produced the requested file.""" + + agent_error_code = "run_record_output_file_unrecorded" + agent_error_hint = ( + "Pass the output dataset root to list its run records, or a file the " + "dataset's metadata lists under produced_files." + ) + + +def _digest(text: str) -> JsonObject: + return { + "shortened": True, + "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), + "length": len(text), + } + + +def _shortened(record: RunRecord) -> RunRecordEntry: + """The record with each long text replaced by its sha256 and length. + + ``installed_distributions`` is digested as its sorted compact JSON text. + """ + content = record.content + software = content["software"] + custom_sources = content["custom_sources"] + return RunRecordEntry( + record_id=record.record_id, + record={ + **content, + "software": { + **software, + "installed_distributions": _digest( + json.dumps( + software["installed_distributions"], + sort_keys=True, + separators=(",", ":"), + ) + ), + }, + "pipeline_code": _digest(content["pipeline_code"]), + "global_config_code": _digest(content["global_config_code"]), + "custom_sources": { + source_sha256: _digest(text) + for source_sha256, text in custom_sources.items() + }, + }, + shortened_fields=( + "software.installed_distributions", + "pipeline_code", + "global_config_code", + *(f"custom_sources.{source_sha256}" for source_sha256 in custom_sources), + ), + ) + + +def _in_full(record: RunRecord) -> RunRecordEntry: + return RunRecordEntry(record_id=record.record_id, record=dict(record.content)) + + +@dataclass(frozen=True, slots=True) +class RunRecordTarget(ABC): + """A readable path a run record is asked for: a dataset root or a file inside one.""" + + path: Path + + @classmethod + def at(cls, path: Path) -> "RunRecordTarget": + """A directory holding the metadata document is a dataset root; anything else is a file.""" + if Path(get_metadata_path(path)).is_file(): + return DatasetRootTarget(path) + return OutputFileTarget(path) + + @abstractmethod + def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: + """Read this target; ``entry`` turns a ``RunRecord`` into its returned form.""" + + +class DatasetRootTarget(RunRecordTarget): + def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: + return RunRecordReadResult( + schema_version=SCHEMA_VERSION, + path=str(self.path), + include_sources=request.include_sources, + records=tuple( + entry(record) for record in RunRecord.for_dataset(self.path).values() + ), + ) + + +class OutputFileTarget(RunRecordTarget): + def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: + try: + produced = RunRecord.for_output_file(self.path) + except UnrecordedOutputFile as error: + raise OutputFileNotRecorded(str(error), path=str(self.path)) from error + return RunRecordReadResult( + schema_version=SCHEMA_VERSION, + path=str(self.path), + include_sources=request.include_sources, + records=(entry(produced.record),), + produced_files=( + ProducedFileEntry( + path=str(self.path), + record_id=produced.record.record_id, + producer=dict(produced.producer.to_payload()), + step=produced.step, + ), + ), + ) + + +class RunRecordService: + """Wrap ``RunRecord``'s readers behind the agent path policy.""" + + def __init__(self, path_policy: AgentPathPolicy) -> None: + self._path_policy = path_policy + + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: + target = RunRecordTarget.at(self._path_policy.assert_readable(request.path)) + return target.read(request, _in_full if request.include_sources else _shortened) diff --git a/openhcs/mcp/context.py b/openhcs/mcp/context.py index 233399e5c..97344a7c6 100644 --- a/openhcs/mcp/context.py +++ b/openhcs/mcp/context.py @@ -31,6 +31,7 @@ PipelineAuthoringService, ) from openhcs.agent.services.plate_inspection_service import PlateInspectionService + from openhcs.agent.services.run_record_service import RunRecordService from openhcs.agent.services.plate_streaming_service import PlateStreamingService from openhcs.agent.services.runtime_server_service import RuntimeServerService from openhcs.agent.services.selected_plate_service import SelectedPlateService @@ -62,6 +63,7 @@ class OpenHCSAgentContext: "_object_state_field_help_service", "_pipeline_service", "_plate_inspection_service", + "_run_record_service", "_plate_streaming_service", "_runtime_server_service", "_viewer_endpoint_discovery_service", @@ -109,6 +111,7 @@ def __init__( self._object_state_field_help_service = object_state_field_help_service self._pipeline_service = pipeline_service self._plate_inspection_service = plate_inspection_service + self._run_record_service: "RunRecordService | None" = None self._plate_streaming_service = plate_streaming_service self._runtime_server_service = runtime_server_service self._viewer_endpoint_discovery_service = viewer_endpoint_discovery_service @@ -212,6 +215,14 @@ def plate_inspection_service(self) -> "PlateInspectionService": self._plate_inspection_service = PlateInspectionService(self.path_policy) return self._plate_inspection_service + @property + def run_record_service(self) -> "RunRecordService": + if self._run_record_service is None: + from openhcs.agent.services.run_record_service import RunRecordService + + self._run_record_service = RunRecordService(self.path_policy) + return self._run_record_service + @property def plate_streaming_service(self) -> "PlateStreamingService": if self._plate_streaming_service is None: diff --git a/tests/fixtures/mcp/mcp_tool_contract.json b/tests/fixtures/mcp/mcp_tool_contract.json index 8de654d69..a8864b995 100644 --- a/tests/fixtures/mcp/mcp_tool_contract.json +++ b/tests/fixtures/mcp/mcp_tool_contract.json @@ -397,6 +397,7 @@ "openhcs_load_pipeline_file", "openhcs_observe_owned_runtime", "openhcs_query_plate_files", + "openhcs_read_run_record", "openhcs_register_custom_function", "openhcs_render_config_source", "openhcs_render_pipeline_source", @@ -455,6 +456,7 @@ "openhcs_observe_owned_runtime", "openhcs_probe_viewer_window", "openhcs_query_plate_files", + "openhcs_read_run_record", "openhcs_register_custom_function", "openhcs_render_config_source", "openhcs_render_pipeline_source", @@ -560,6 +562,7 @@ "openhcs_observe_owned_runtime", "openhcs_probe_viewer_window", "openhcs_query_plate_files", + "openhcs_read_run_record", "openhcs_register_custom_function", "openhcs_render_config_source", "openhcs_render_pipeline_source", @@ -4448,6 +4451,47 @@ }, "title": "Query plate files" }, + "openhcs_read_run_record": { + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": true, + "readOnlyHint": true, + "title": "Read run record" + }, + "description": "Read-only: what produced an output dataset. Pass an output dataset root (the directory holding openhcs_metadata.json) to get every run record by record id, or one output file inside a dataset to get the producer identity stored for that file, the producing step's entry (name, pipeline position, functions with identity, parameters with their sources, declared files with content hashes, devices) and the record of that run. A file the dataset does not list is an error, never a guess from its name. By default pipeline_code, global_config_code, custom_sources texts and software.installed_distributions are replaced by their sha256 and length and named in shortened_fields; pass include_sources=true for the full texts.", + "execution": null, + "icons": null, + "inputSchema": { + "additionalProperties": false, + "properties": { + "include_sources": { + "default": false, + "title": "Include Sources", + "type": "boolean" + }, + "path": { + "title": "Path", + "type": "string" + } + }, + "required": [ + "path" + ], + "title": "openhcs_read_run_recordArguments", + "type": "object" + }, + "meta": { + "openhcs/outputContract": "RunRecordReadResult" + }, + "name": "openhcs_read_run_record", + "outputSchema": { + "additionalProperties": true, + "title": "openhcs_read_run_recordDictOutput", + "type": "object" + }, + "title": "Read run record" + }, "openhcs_register_custom_function": { "annotations": { "destructiveHint": true, diff --git a/tests/unit/agent/test_authoring_roundtrip_boundaries.py b/tests/unit/agent/test_authoring_roundtrip_boundaries.py index 30605b323..5e8eaaeff 100644 --- a/tests/unit/agent/test_authoring_roundtrip_boundaries.py +++ b/tests/unit/agent/test_authoring_roundtrip_boundaries.py @@ -162,8 +162,14 @@ def test_real_authoring_render_parse_retains_registry_memory_contract( original_step.func, original_step.name ) ) - if kwargs != {"gamma": 1, "gain": 1} or not clean: - assert step.func[1] == kwargs + if clean: + if kwargs != {"gamma": 1, "gain": 1}: + assert step.func[1] == kwargs + else: + # The complete text writes every parameter, so a later change of a + # default cannot change what the text means. + assert kwargs.items() <= step.func[1].items() + assert set(step.func[1]) > set(kwargs) def test_registry_reference_formatting_uses_alias_without_resolving( diff --git a/tests/unit/test_run_record_capability.py b/tests/unit/test_run_record_capability.py new file mode 100644 index 000000000..627fa553a --- /dev/null +++ b/tests/unit/test_run_record_capability.py @@ -0,0 +1,197 @@ +"""openhcs_read_run_record reads only what the dataset's metadata document stores.""" + +import hashlib +import json +from types import SimpleNamespace + +import pytest +from polystore.streaming.identity import StreamProducerIdentity + +from openhcs.agent.capabilities import ReadRunRecordCapability +from openhcs.agent.path_policy import AgentPathPolicy +from openhcs.agent.services.run_record_service import RunRecordService +from openhcs.core.run_record import RunRecord +from openhcs.mcp.execution import McpTransportExecutor +from openhcs.mcp.server import build_server + +PIPELINE_CODE = "pipeline = []\n" * 40 +CUSTOM_SOURCE = "def count(image):\n return image\n" +CUSTOM_SOURCE_SHA256 = hashlib.sha256(CUSTOM_SOURCE.encode("utf-8")).hexdigest() +STEP = { + "name": "count", + "pipeline_position": 0, + "step_scope_id": "s::step_0", + "input_memory_type": "numpy", + "output_memory_type": "numpy", + "assigned_gpu_devices": [], + "functions": [], + "configuration": [], +} +RECORD = { + "software": { + "openhcs": "0.0.0", + "python": "3.12.0", + "installed_distributions": {"numpy": "2.0.0", "polystore": "1.0.0"}, + }, + "pipeline_code": PIPELINE_CODE, + "global_config_code": "config = None\n", + "configuration": [], + "steps": [STEP], + "custom_sources": {CUSTOM_SOURCE_SHA256: CUSTOM_SOURCE}, +} +RECORD_ID = RunRecord(RECORD).record_id +PRODUCER = dict( + StreamProducerIdentity.pipeline_output( + output_kind="artifact", + output_key="counts", + projection_key="counts", + step_name="count", + pipeline_position=0, + step_scope_id="s::step_0", + ).to_payload() +) + + +def _digest(text): + return { + "shortened": True, + "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), + "length": len(text), + } + + +@pytest.fixture +def dataset(tmp_path): + root = tmp_path / "dataset" + (root / "results").mkdir(parents=True) + (root / "results" / "a.csv").write_text("n\n1\n", encoding="utf-8") + (root / "results" / "unlisted.csv").write_text("n\n2\n", encoding="utf-8") + (root / "openhcs_metadata.json").write_text( + json.dumps( + { + "subdirectories": {}, + "run_records": {RECORD_ID: RECORD}, + "produced_files": { + "results/a.csv": {"run_record": RECORD_ID, "producer": PRODUCER} + }, + } + ), + encoding="utf-8", + ) + return root + + +@pytest.fixture +def read(tmp_path): + service = RunRecordService( + AgentPathPolicy.with_roots(readable_roots=(tmp_path,), writable_roots=()) + ) + executor = McpTransportExecutor() + built = build_server( + SimpleNamespace(run_record_service=service), + main_thread_dispatcher=executor.dispatcher, + ) + tool = built._tool_manager.get_tool(ReadRunRecordCapability.name) + + def call(**arguments): + async def exercise(): + return await tool.fn(**arguments) + + return executor.run(exercise) + + yield call + executor.close() + + +def test_dataset_root_returns_every_record_shortened_by_default(dataset, read): + result = read(path=str(dataset)) + + assert result["errors"] == [] + assert result["include_sources"] is False + assert result["produced_files"] == [] + (entry,) = result["records"] + assert entry["record_id"] == RECORD_ID + assert entry["shortened_fields"] == [ + "software.installed_distributions", + "pipeline_code", + "global_config_code", + f"custom_sources.{CUSTOM_SOURCE_SHA256}", + ] + record = entry["record"] + assert set(record) == set(RECORD) + assert record["pipeline_code"] == _digest(PIPELINE_CODE) + assert record["global_config_code"] == _digest(RECORD["global_config_code"]) + assert record["custom_sources"] == {CUSTOM_SOURCE_SHA256: _digest(CUSTOM_SOURCE)} + assert record["software"]["installed_distributions"] == _digest( + '{"numpy":"2.0.0","polystore":"1.0.0"}' + ) + assert record["software"]["openhcs"] == "0.0.0" + assert record["steps"] == [STEP] + + +def test_dataset_root_with_sources_returns_the_stored_record(dataset, read): + result = read(path=str(dataset), include_sources=True) + + assert result["include_sources"] is True + assert result["records"] == [ + {"record_id": RECORD_ID, "record": RECORD, "shortened_fields": []} + ] + + +@pytest.mark.parametrize("include_sources", (False, True)) +def test_output_file_returns_its_producer_step_and_record(dataset, read, include_sources): + output_file = dataset / "results" / "a.csv" + + result = read(path=str(output_file), include_sources=include_sources) + + assert result["errors"] == [] + assert result["produced_files"] == [ + { + "path": str(output_file), + "record_id": RECORD_ID, + "producer": PRODUCER, + "step": STEP, + } + ] + (entry,) = result["records"] + assert entry["record_id"] == RECORD_ID + assert (entry["record"] == RECORD) is include_sources + assert (entry["shortened_fields"] == []) is include_sources + + +def test_unlisted_file_is_an_error_not_a_guess(dataset, read): + unlisted = dataset / "results" / "unlisted.csv" + + result = read(path=str(unlisted)) + + assert result["ok"] is False + assert result["tool"] == "openhcs_read_run_record" + (error,) = result["errors"] + assert error["code"] == "run_record_output_file_unrecorded" + assert error["message"] == ( + f"{unlisted} is not recorded as a step output in " + f"{dataset / 'openhcs_metadata.json'}." + ) + assert "records" not in result + + +def test_path_outside_readable_roots_is_rejected(tmp_path, read): + result = read(path=str(tmp_path.parent)) + + assert result["errors"][0]["code"] == "agent_path_policy_rejected" + + +def test_cli_command_is_generated_from_the_declaration(): + import openhcs.mcp.dev_client as dev_client + + parser = dev_client._build_parser() + call = dev_client._calls_from_args( + parser.parse_args(("read-run-record", "/data/out", "--include-sources")) + )[0] + + assert call.name == "openhcs_read_run_record" + assert call.arguments == {"path": "/data/out", "include_sources": True} + default_call = dev_client._calls_from_args( + parser.parse_args(("read-run-record", "/data/out")) + )[0] + assert default_call.arguments == {"path": "/data/out", "include_sources": False} From f3ae7a42a84c5e00ed78645cf5a8bb51e9567c5a Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 15:34:33 -0400 Subject: [PATCH 06/12] DL3 receipt: hand-off state, acceptance results, what remains Co-Authored-By: Claude Opus 5.5 --- .../DL3-output-provenance.md | 126 +++++++++++++++++- 1 file changed, 124 insertions(+), 2 deletions(-) diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md index a792cbc69..6ffdd6444 100644 --- a/docs/refactor/consolidation-20261009/DL3-output-provenance.md +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -176,5 +176,127 @@ one step, of which 246 installed distributions are most). (`submission_` in step scope ids). Output files (TIFF and CSV) are byte-identical. **Not done.** `changed_during_run` (asking again for size and modification time at the end of the -run). The agent capability `openhcs_read_run_record` is being written separately and reads -`effective_config_code`; that key is now `global_config_code`. +run): the design text above still describes it; it is dropped, a file modified during a run is +not detected. + +## Hand-off (2026-10-12): state, results, what remains + +A fresh agent continues from here. PolyStore part: PR OpenHCSDev/PolyStore#41 (branch +`feat/file-content-identity`, commit `8ee8ce4`, version 0.6.3, not merged, not on PyPI). This +branch pins `polystore>=0.6.3,<0.7` and points the submodule at that commit. + +### Implemented and committed + +| Commit | What | +|---|---| +| `7ea5440e1` | this receipt's design | +| `1786bbc13` | `openhcs/core/function_identity.py`; `identity()` on the function reference and registry entry families; custom source text kept on `CustomFunctionSourceNamespace`; `content_at_run_start` on the declared path family; `openhcs/core/run_record.py` | +| `c2d689e61` | bundle carries `RunDeclaration`; workers report `producer_by_saved_path`; `RunRecord.at_run_start` in `execute_bundle_request`; one-transaction write in `reconcile_completed_dataset`; complete rendering writes every parameter; `tests/unit/test_run_record.py` (17), `tests/integration/test_run_record_round_trip.py` | +| `09a712ad5` | `openhcs_read_run_record` capability (`openhcs/agent/services/run_record_service.py`), `tests/unit/test_run_record_capability.py` (7) | +| `d33b72496` | merge of `origin/main` | + +### Acceptance run (CPU venv, torch 2.14.1+cpu, Cellpose 3.1.1.3, 3 BBBC039 fields) + +Scripts and outputs: `.claude/scratch/acceptance/` (untracked). NumPy-declared Cellpose source +from commit `47fbed348` (the torch-declared one cannot compile without CUDA; DL1's gap). + +- Cellpose then `measure_labelled_objects`: a label TIFF resolves to step `segment`, a + measurement CSV to step `measure`. `custom_source_sha256` equals sha256 of the source file. + All 7 Cellpose parameters are recorded as `function_default`; `min_area_px=20` as `pipeline`. + `imports.libraries`: cellpose 3.1.1.3, numpy 2.0.2. Record 78,764 bytes, metadata document + 97,633 bytes. Compile 2.53 s, execute 19.97 s. +- Body changed, name kept: source hash `a2824fff…` -> `bca58da6…`, record id `7ce7240d…` -> + `37e46efc…`; each dataset stores its own text. +- Re-run from the record in a fresh process with nothing registered: 3 main images, the label + TIFF and both step CSVs byte-identical. ROI zips differ only in an embedded absolute + `source_path` and zip timestamps (all `.roi` members identical); `segmentation_summary.txt` + lists absolute paths; `metaxpress_style_summary.csv` has a different `Plate ID`. +- N2V2 denoise (NumPy-declared copy; model file left at its default): `declared_files` entry + `declared_as: "input file"`, `hashed: true`, sha256 `0340245e…`, 124,144,631 bytes, equal to + hashlib; after overwriting the model the recorded hash is `5a2582ea…`. +- Hash cost: standalone 1 GiB, sha256, 0.58 s warm and 2.15 s after dropping the page cache + (blake2b 1.03 s, md5 1.22 s). Through `filemanager.file_content` under load (full suite running, + 2 GB free RAM): 12.56 s first read, then 1.29 s and 1.46 s. +- Capability default response: 52,808 bytes for a dataset root, 68,175 for one file. + +Example (Cellpose step, trimmed): + +```json +{"name": "segment", "pipeline_position": 0, + "step_scope_id": "…/runA/source::pipeline::submission_7fd30cd3c3c0::step_0", + "input_memory_type": "numpy", "output_memory_type": "numpy", "assigned_gpu_devices": [], + "functions": [{ + "identity": {"key": "openhcs:cellpose_segment", "module": "openhcs.processing.custom_functions", + "name": "cellpose_segment", + "imports": {"libraries": [{"distribution": "cellpose", "version": "3.1.1.3"}, + {"distribution": "numpy", "version": "2.0.2"}], + "modules_without_distribution": ["openhcs"]}, + "custom_source_sha256": "a2824fffc06b7c906d3af193fa4ff21ba24a7a5ddf0a16e54419aa7eb077d9af"}, + "parameters": [ + {"name": "image", "value": "", "source": {"kind": "run_time"}}, + {"name": "model", "value": "cellpose_segment.__openhcs_source_a2824f….__openhcs_member_….NUCLEI", + "source": {"kind": "function_default", "function": "openhcs:cellpose_segment"}}, + {"name": "diameter", "value": "None", "source": {"kind": "function_default", "function": "openhcs:cellpose_segment"}}, + {"name": "flow_threshold", "value": "0.4", "source": {"kind": "function_default", "function": "openhcs:cellpose_segment"}}], + "declared_files": []}]} +``` + +### Gates + +- `guardrails_local.sh` on `d33b72496`: **rc=1**. Lines: `ratchet openhcs exit 1`, + `ratchet scripts exit 0`, `ratchet benchmark exit 0`, `38 passed, 22 warnings in 68.54s`, + `nra-r1 exit 1`, `guardrails rc=1`. Ratchet delta: `GodClassExcess PipelineCompiler +7`, + `GodClassExcess OpenHCSMetadataTarget +24`, `LongBooleanChain +1`, + `BooleanChainTerms openhcs/core/steps/abstract.py +1`, + `BooleanChainTerms openhcs/core/virtual_workspace_metadata.py +4`, + `ForeignAbsenceProbe` +1 in `virtual_workspace_metadata.py` and +1 in + `serialization/pycodify_formatters.py`, `StringSubscript +21`, + `AnyAnnotation openhcs/core/run_record.py +6`. NRA increased: `unmodeled_record_shape` in + `openhcs/core/run_record.py` and `openhcs/agent/services/run_record_service.py`. +- `verify_branch.sh DL3`: see "Full suite" below. +- Not run: `smoke_local.sh`, official30 parity (required: compilation, execution and + materialization changed). + +### What remains, in order + +1. **Model the record as nominal types** (clears `StringSubscript`, `AnyAnnotation`, + `unmodeled_record_shape`): `RunRecord` holds `content: Mapping[str, Any]` and the reader and + the capability service index it by string. Replace with frozen dataclasses (software, step, + function, parameter, declared file) that encode to and decode from JSON in one place; + `RunRecordService` reads attributes. +2. **Move the added code out of the two oversized classes**: `saved_path_producers` out of + `OpenHCSMetadataTarget` (it belongs with `ProducedFiles`), the declaration calls out of + `PipelineCompiler`. Remove the two new absence probes and the lengthened boolean chains + (`StepExecutionObservation.is_empty`, `reconcile_completed_dataset`). +3. **Defect: a declared path left at its default is recorded as written in the pipeline**, with + the resolved absolute path as its value. `CallableContract.resolve_declared_paths` + (`callable_contract.py:~1296`) puts the resolved default into the kwargs and `_parameters` + (`run_record.py`) treats every kwarg as written. Record the written or default value with its + true source and keep the resolved path only in `declared_files`. Failing test first. +4. **`enabled`** is written in `pipeline_code` function kwargs but has no `parameters` entry. +5. **Device on CPU**: `assigned_gpu_devices: []` does not say the step ran on host memory. State + it explicitly from the place DL1 leaves the assignment. +6. **Enum defaults of custom functions** render as + `name.__openhcs_source_.__openhcs_member_.MEMBER` (same mangling as D1 gap 10). +7. **Transitive libraries**: `imports.libraries` lists direct imports only (torch is not listed + for Cellpose; it is only in `software.installed_distributions`). Decide: record the + requirement closure of the direct imports, or state the rule in the record. +8. **Plate-level outputs are unrecorded**: `results/metaxpress_style_summary.csv` (analysis + consolidation) has no `produced_files` entry; `for_output_file` raises `UnrecordedOutputFile` + for it. It needs a producer identity of its own kind (it is not a step output). +9. **Record size**: 59 to 79 kB per record, mostly installed distributions and per-step + configuration repeating absolute scope ids. The capability's shortened response is still 53 kB. +10. **Fallback to remove or justify**: with no saved global config, `RunDeclaration` stores a + default-constructed one. +11. **Reader API**: no public "is this a dataset root"; `ProducedFile` has neither dataset root + nor relative path, so the capability's read of the metadata document above a file bypasses + the agent path policy; a directory without metadata reports as an unrecorded file. +12. `changed_during_run` is not implemented; update the design text or implement. +13. Reproducibility leaks found by the round trip, outside this branch's scope: ROI zip metadata + and `segmentation_summary.txt` embed absolute source paths; zip timestamps. +14. Gates: guardrails to `rc=0`, `verify_branch.sh`, `smoke_local.sh`, parity under + `heavy_slot.sh` and `official30.lock`. Then mark the PR ready. PolyStore #41 must merge and + release 0.6.3 before CI here can install the pin. +15. Delete `.claude/venv-cpu` (2.3 GB) and `.claude/scratch/` in this worktree. +16. For the #1230 branch: nothing in the example functions has to change for this gap; the count + table's own `model` and `diameter_px` columns are no longer the only place those are stated. From 8eae47bd7ff8f9839a0fbb69b05406c9fe9d5a78 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 15:44:28 -0400 Subject: [PATCH 07/12] DL3 receipt: full-suite result and the failing ids Co-Authored-By: Claude Opus 5.5 --- .../DL3-output-provenance.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md index 6ffdd6444..df10d70cb 100644 --- a/docs/refactor/consolidation-20261009/DL3-output-provenance.md +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -300,3 +300,29 @@ Example (Cellpose step, trimmed): 15. Delete `.claude/venv-cpu` (2.3 GB) and `.claude/scratch/` in this worktree. 16. For the #1230 branch: nothing in the example functions has to change for this gap; the count table's own `model` and `diameter_px` columns are no longer the only place those are stated. + +### Full suite (`verify_branch.sh DL3`, on `d33b72496`, lines verbatim) + +``` +suite: = 7 failed, 11754 passed, 25 skipped, 358 warnings, 6 subtests passed in 1948.52s (0:32:28) = +collected 11785, reported 11761, crash lines 0 +new vs known: 5 +only on DL3: +tests/unit/agent/test_mcp_server.py::test_mcp_stdio_server_roundtrip_returns_errors_as_payloads +tests/unit/test_mcp_dev_client_terminal_input.py::test_long_terminal_command_reuses_original_parser_and_client[...] +tests/unit/test_pycodify_formatters.py::test_function_step_round_trip_omits_injected_values_but_keeps_runtime_settings[False-expected_kwargs1] +fails on both with a different error: +``` + +Failing ids outside `known.fail` (5), with the believed cause (none investigated beyond this): + +| Id | Rerun alone | Believed cause | +|---|---|---| +| `tests/unit/test_pycodify_formatters.py::test_function_step_round_trip_omits_injected_values_but_keeps_runtime_settings[False-expected_kwargs1]` | fails on DL3 only | **This branch.** `clean_mode=False` now writes every parameter; the test expects only the written kwargs in the full rendering. Same change of meaning as the one already made in `tests/unit/agent/test_authoring_roundtrip_boundaries.py`; restate the expectation for the full rendering, keep the clean one. | +| `tests/unit/test_mcp_dev_client_terminal_input.py::test_long_terminal_command_reuses_original_parser_and_client[...]` | fails on DL3 only | **Probably this branch**: the generated CLI gained `read-run-record`, or the complete rendering changed a long command's text. Not looked at. | +| `tests/unit/agent/test_mcp_server.py::test_mcp_stdio_server_roundtrip_returns_errors_as_payloads` | fails on DL3 only in the gate's rerun; passed alone earlier (`1 passed` with `-k stdio_server_roundtrip`) | Seen before as a 5 s stdio-initialize timeout under load (acceptance run and suite ran together). Rerun on a quiet machine before treating it as real; if it persists, the new tool lengthens server start. | +| `tests/unit/agent/test_mcp_affine_inspection_progress.py::test_resident_continuous_connections_keep_affinity_and_wire_progress[original-continuous]` | passed on rerun (both trees) | load-dependent | +| `tests/unit/test_ack_return_route_journey.py::test_two_native_stream_producers_keep_deferred_ack_ownership[TransportMode.IPC]` | passed on rerun (both trees) | load-dependent | + +The remaining 2 of the 7 failures are the ids in `known.fail`. Fix the first two rows as step 0 +of "What remains". From 3a55d1fe456ad37200571acd53f657ced90933e6 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 16:03:18 -0400 Subject: [PATCH 08/12] DL3: the run record is frozen dataclasses with one JSON codec; the record's pipeline text is its own rendering Item 1. RunRecord, StepRecord, StepCalls, FunctionRecord, RecordedParameter, DeclaredFile, SoftwareEnvironment, ConfigValuesFromOwner, DatasetRunRecords replace the nested dicts. A parameter's source and a configuration value's source are registered class families stored as their registry key. The stored JSON is python_introspect.to_jsonable of the record and is read back with python_introspect.dataclass_from_mapping; the identity and file-content families are read as unions of dataclasses. Resolved configuration is grouped by the owner that supplied it, so each scope id is written once. The reader and the openhcs_read_run_record capability return these types; the capability returns what each step ran by default and the whole record on request. Item 2. clean_mode=False keeps its meaning (every stored field, no parameter the pipeline does not state). The record's copy is a separate rendering, PipelineDocumentCodec.render_with_every_parameter. The two tests that were weakened for the changed flag are restored. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/capabilities.py | 16 +- openhcs/agent/dto/plate.py | 28 +- openhcs/agent/services/run_record_service.py | 72 +- openhcs/core/function_identity.py | 64 +- openhcs/core/pipeline_document.py | 25 + openhcs/core/run_record.py | 706 +++++++++++------- openhcs/serialization/pycodify_formatters.py | 2 - tests/fixtures/mcp/mcp_tool_contract.json | 6 +- .../integration/test_run_record_round_trip.py | 18 +- tests/run_records.py | 23 +- .../test_authoring_roundtrip_boundaries.py | 10 +- tests/unit/test_run_record.py | 215 ++++-- tests/unit/test_run_record_capability.py | 197 +++-- tests/unit/test_source_round_trip.py | 17 +- 14 files changed, 806 insertions(+), 593 deletions(-) diff --git a/openhcs/agent/capabilities.py b/openhcs/agent/capabilities.py index ceb3013c3..84ccf1099 100644 --- a/openhcs/agent/capabilities.py +++ b/openhcs/agent/capabilities.py @@ -2362,14 +2362,16 @@ class ReadRunRecordCapability(PlatePathCapability): "Read-only: what produced an output dataset. Pass an output dataset root " "(the directory holding openhcs_metadata.json) to get every run record " "by record id, or one output file inside a dataset to get the producer " - "identity stored for that file, the producing step's entry (name, " - "pipeline position, functions with identity, parameters with their " - "sources, declared files with content hashes, devices) and the record of " + "identity stored for that file, the producing step and the record of " "that run. A file the dataset does not list is an error, never a guess " - "from its name. By default pipeline_code, global_config_code, " - "custom_sources texts and software.installed_distributions are replaced " - "by their sha256 and length and named in shortened_fields; pass " - "include_sources=true for the full texts." + "from its name. Each record lists what every step ran: each function " + "with its identity (registry key, defining and imported library " + "versions, sha256 of a custom function's source), every parameter with " + "its value and source (pipeline, function_default, run_time), declared " + "files with content hashes, and devices. Pass include_full_record=true " + "to add the complete stored record: pipeline and global config " + "documents, resolved configuration with the owner of every value, " + "custom source texts and all installed distributions." ) exposition = PlatePathCapability.exposition.refine( workflow_stage=CapabilityWorkflowStage.INSPECTION, diff --git a/openhcs/agent/dto/plate.py b/openhcs/agent/dto/plate.py index 3ead9b2a0..8b5aa67f9 100644 --- a/openhcs/agent/dto/plate.py +++ b/openhcs/agent/dto/plate.py @@ -9,6 +9,7 @@ from typing import ClassVar from metaclass_registry import AutoRegisterMeta +from polystore.streaming.identity import StreamProducerIdentity from zmqruntime.config import TransportMode from formbridge.contracts import ( @@ -25,6 +26,7 @@ from openhcs.core.component_filters import ComponentFilters from openhcs.core.config import NapariDisplayConfig from openhcs.core.dataset_sources.choice import AutoDetectedSource, DatasetSourceChoice +from openhcs.core.run_record import RunRecord, StepCalls from openhcs.core.plate_file_inventory import ( PlateFileInventoryQuery, PlateFileKind, @@ -1349,12 +1351,14 @@ class RunRecordReadRequest(AgentDataclassCliRequest): ``path`` is an output dataset root (the directory that holds the OpenHCS metadata document) or one output file below such a root. - ``include_sources`` returns the long texts in full; otherwise each is - replaced by its sha256 and length and named in ``shortened_fields``. + ``include_full_record`` adds the complete stored record to each entry: the + pipeline and global config documents, the resolved configuration with the + owner of every value, custom function source texts and every installed + distribution. """ path: str - include_sources: bool = False + include_full_record: bool = False @classmethod def agent_cli_argument_specs(cls) -> tuple[AgentCliArgumentSpec, ...]: @@ -1370,16 +1374,17 @@ def agent_cli_argument_specs(cls) -> tuple[AgentCliArgumentSpec, ...]: @dataclass(frozen=True, slots=True) class RunRecordEntry: - """One stored run record, addressed by the sha256 of its full content. + """One stored run record, addressed by the sha256 of its stored JSON. - ``shortened_fields`` names every field of ``record`` whose value was - replaced by ``{"shortened": true, "sha256": ..., "length": ...}``; it is - empty when the record is returned in full. + ``steps`` says what each step ran: every function with its identity (a + custom function by the sha256 of its source), every parameter with its + value and source, and every declared file with its content hash. + ``record`` is the complete record, present when the request asked for it. """ record_id: str - record: JsonObject - shortened_fields: tuple[str, ...] = () + steps: tuple[StepCalls, ...] + record: RunRecord | None = None @dataclass(frozen=True, slots=True) @@ -1388,8 +1393,8 @@ class ProducedFileEntry: path: str record_id: str - producer: JsonObject - step: JsonObject + producer: StreamProducerIdentity + step: StepCalls @dataclass(frozen=True, kw_only=True, slots=True) @@ -1402,7 +1407,6 @@ class RunRecordReadResult(AgentResultEnvelope): """ path: str - include_sources: bool records: tuple[RunRecordEntry, ...] = () produced_files: tuple[ProducedFileEntry, ...] = () errors: tuple[AgentError, ...] = () diff --git a/openhcs/agent/services/run_record_service.py b/openhcs/agent/services/run_record_service.py index 93f42dbec..c170d0a7a 100644 --- a/openhcs/agent/services/run_record_service.py +++ b/openhcs/agent/services/run_record_service.py @@ -2,14 +2,10 @@ from __future__ import annotations -import hashlib -import json from abc import ABC, abstractmethod from dataclasses import dataclass from pathlib import Path -from python_introspect import JsonObject - from openhcs.agent.dto.common import SCHEMA_VERSION from openhcs.agent.dto.plate import ( ProducedFileEntry, @@ -33,56 +29,15 @@ class OutputFileNotRecorded(AgentFacingErrorMixin, LookupError): ) -def _digest(text: str) -> JsonObject: - return { - "shortened": True, - "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), - "length": len(text), - } - - -def _shortened(record: RunRecord) -> RunRecordEntry: - """The record with each long text replaced by its sha256 and length. - - ``installed_distributions`` is digested as its sorted compact JSON text. - """ - content = record.content - software = content["software"] - custom_sources = content["custom_sources"] +def _entry(record: RunRecord, request: RunRecordReadRequest) -> RunRecordEntry: + """What each step ran, with the complete stored record when the request asks for it.""" return RunRecordEntry( record_id=record.record_id, - record={ - **content, - "software": { - **software, - "installed_distributions": _digest( - json.dumps( - software["installed_distributions"], - sort_keys=True, - separators=(",", ":"), - ) - ), - }, - "pipeline_code": _digest(content["pipeline_code"]), - "global_config_code": _digest(content["global_config_code"]), - "custom_sources": { - source_sha256: _digest(text) - for source_sha256, text in custom_sources.items() - }, - }, - shortened_fields=( - "software.installed_distributions", - "pipeline_code", - "global_config_code", - *(f"custom_sources.{source_sha256}" for source_sha256 in custom_sources), - ), + steps=tuple(step.calls for step in record.steps), + record=record if request.include_full_record else None, ) -def _in_full(record: RunRecord) -> RunRecordEntry: - return RunRecordEntry(record_id=record.record_id, record=dict(record.content)) - - @dataclass(frozen=True, slots=True) class RunRecordTarget(ABC): """A readable path a run record is asked for: a dataset root or a file inside one.""" @@ -97,24 +52,24 @@ def at(cls, path: Path) -> "RunRecordTarget": return OutputFileTarget(path) @abstractmethod - def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: - """Read this target; ``entry`` turns a ``RunRecord`` into its returned form.""" + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: + """Read this target with the detail the request asks for.""" class DatasetRootTarget(RunRecordTarget): - def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: return RunRecordReadResult( schema_version=SCHEMA_VERSION, path=str(self.path), - include_sources=request.include_sources, records=tuple( - entry(record) for record in RunRecord.for_dataset(self.path).values() + _entry(record, request) + for record in RunRecord.for_dataset(self.path).values() ), ) class OutputFileTarget(RunRecordTarget): - def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: + def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: try: produced = RunRecord.for_output_file(self.path) except UnrecordedOutputFile as error: @@ -122,13 +77,12 @@ def read(self, request: RunRecordReadRequest, entry) -> RunRecordReadResult: return RunRecordReadResult( schema_version=SCHEMA_VERSION, path=str(self.path), - include_sources=request.include_sources, - records=(entry(produced.record),), + records=(_entry(produced.record, request),), produced_files=( ProducedFileEntry( path=str(self.path), record_id=produced.record.record_id, - producer=dict(produced.producer.to_payload()), + producer=produced.producer, step=produced.step, ), ), @@ -143,4 +97,4 @@ def __init__(self, path_policy: AgentPathPolicy) -> None: def read(self, request: RunRecordReadRequest) -> RunRecordReadResult: target = RunRecordTarget.at(self._path_policy.assert_readable(request.path)) - return target.read(request, _in_full if request.include_sources else _shortened) + return target.read(request) diff --git a/openhcs/core/function_identity.py b/openhcs/core/function_identity.py index fb4137307..89d4f949a 100644 --- a/openhcs/core/function_identity.py +++ b/openhcs/core/function_identity.py @@ -11,9 +11,11 @@ import importlib.metadata import importlib.util import sys +from collections.abc import Mapping from dataclasses import dataclass from functools import lru_cache from pathlib import Path +from types import MappingProxyType from openhcs.runtime.environment_provenance import InstalledDistributionVersion @@ -80,15 +82,6 @@ def imported_by_module(cls, module_name: str) -> "ModuleLibraries": return cls.of_modules(frozenset()) return cls.imported_by_source(Path(origin).read_text(encoding="utf-8")) - def to_record(self) -> dict[str, object]: - return { - "libraries": [ - {"distribution": library.name, "version": library.version} - for library in self.libraries - ], - "modules_without_distribution": list(self.modules_without_distribution), - } - @dataclass(frozen=True, slots=True) class FunctionIdentity: @@ -116,18 +109,13 @@ def of_module_export(cls, *, key: str, module: str, name: str) -> "FunctionIdent imports=ModuleLibraries.imported_by_module(module), ) - def custom_sources(self) -> dict[str, str]: - """Source texts this identity refers to by hash; none for an installed module.""" - return {} + def recorded_source_hashes(self) -> tuple[str, ...]: + """The sha256 of each source text this identity refers to; none for an installed module.""" + return () - def to_record(self) -> dict[str, object]: - return { - "key": self.key, - "module": self.module, - "name": self.name, - "defined_in": self.defined_in.to_record(), - "imports": self.imports.to_record(), - } + def source_texts(self) -> Mapping[str, str]: + """Source texts this identity refers to, by sha256; none for an installed module.""" + return MappingProxyType({}) @dataclass(frozen=True, slots=True) @@ -135,7 +123,6 @@ class CustomFunctionIdentity(FunctionIdentity): """A function defined by user-written source text, identified by that text's hash.""" source_sha256: str - source_text: str @classmethod def of_source( @@ -148,14 +135,33 @@ def of_source( defined_in=ModuleLibraries.of_modules(frozenset()), imports=ModuleLibraries.imported_by_source(source_text), source_sha256=source_sha256, - source_text=source_text, ) - def custom_sources(self) -> dict[str, str]: - return {self.source_sha256: self.source_text} + def recorded_source_hashes(self) -> tuple[str, ...]: + return (self.source_sha256,) + + def source_texts(self) -> Mapping[str, str]: + """The text, from the registered source namespace that owns this hash.""" + from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, + ) + from openhcs.processing.custom_functions.source_namespace import ( + CustomFunctionSource, + CustomFunctionSourceNamespace, + ) - def to_record(self) -> dict[str, object]: - return { - **FunctionIdentity.to_record(self), - "custom_source_sha256": self.source_sha256, - } + declaration = CustomFunctionRuntimeRegistry.declaration_for_source( + CustomFunctionSource(self.name, self.source_sha256) + ) + if declaration is None: + raise LookupError( + f"Custom function {self.name!r} is no longer registered with source " + f"{self.source_sha256}; its text cannot be recorded." + ) + return MappingProxyType( + { + self.source_sha256: CustomFunctionSourceNamespace.bound_to( + declaration.metadata.func + ).code + } + ) diff --git a/openhcs/core/pipeline_document.py b/openhcs/core/pipeline_document.py index 8abde39d7..5b1999741 100644 --- a/openhcs/core/pipeline_document.py +++ b/openhcs/core/pipeline_document.py @@ -57,6 +57,31 @@ def from_values( ), ) + @classmethod + def render_with_every_parameter(cls, document: PipelineDocument) -> str: + """Render the copy a run's record keeps: nothing is left to a default. + + The saved pipeline (``render``) leaves out what equals a default, so it + means whatever the installed defaults are when it is loaded. This text + writes every stored field and, for every function, every parameter it + takes a value for, so it means the same after a default changes. + """ + + from openhcs.serialization.pycodify_formatters import ( + with_every_parameter_written, + ) + + return cls.render( + PipelineDocument( + pipeline_config=document.pipeline_config, + pipeline_steps=[ + step.with_function_spec(with_every_parameter_written(step.func)) + for step in document.pipeline_steps + ], + ), + clean_mode=False, + ) + @classmethod def from_namespace( cls, diff --git a/openhcs/core/run_record.py b/openhcs/core/run_record.py index d5da5fd60..04b40ccd2 100644 --- a/openhcs/core/run_record.py +++ b/openhcs/core/run_record.py @@ -15,6 +15,11 @@ ``global_config_code`` are stored documents: executing them rebuilds the run. ``configuration``, ``steps[].configuration`` and ``parameters`` are the resolved values, each with the declaration that supplied it. + +The stored form is the JSON ``python_introspect.to_jsonable`` writes for these +dataclasses and ``python_introspect.dataclass_from_mapping`` reads back; there +is no other encoder or decoder. A class of a registered family (a parameter's +or a configuration value's source) is stored as its registry key. """ from __future__ import annotations @@ -24,27 +29,31 @@ import json from abc import ABC, abstractmethod from collections.abc import Iterable, Iterator, Mapping, Sequence -from dataclasses import dataclass, fields, is_dataclass, replace +from dataclasses import dataclass, field, fields, is_dataclass, replace from pathlib import Path from types import MappingProxyType -from typing import TYPE_CHECKING, Any, ClassVar +from typing import TYPE_CHECKING, ClassVar import pycodify +from metaclass_registry import AutoRegisterMeta +from polystore.file_content import HashedFileContent, UnhashedFileContent from polystore.streaming.identity import StreamProducerIdentity +from python_introspect import JsonObject, JsonValue, dataclass_from_mapping, to_jsonable + +from openhcs.core.function_identity import CustomFunctionIdentity, FunctionIdentity -from openhcs.core.vfs_protocol import DatasetPathDeclaration, FileManagerLike +from openhcs.core.vfs_protocol import FileManagerLike if TYPE_CHECKING: from openhcs.core.compiled_execution import CompiledExecutionBundle from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import GlobalPipelineConfig from openhcs.core.function_patterns import CompiledFunctionInvocation + from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator + from openhcs.core.steps.abstract import AbstractStep from openhcs.core.pipeline.compilation_session import ResolvedPipelineDefinition from pycodify.documents import PipelineDocument -RUN_RECORDS_KEY = "run_records" -PRODUCED_FILES_KEY = "produced_files" -JsonRecord = dict[str, Any] def python_expression(value: object) -> str: @@ -54,99 +63,114 @@ def python_expression(value: object) -> str: return pycodify.to_source(value, pycodify.FormatContext(clean_mode=False)).code -class ValueSource(ABC): - """Which declaration supplied a recorded value.""" +class ParameterSource(ABC, metaclass=AutoRegisterMeta): + """Which declaration supplied the value a function parameter ran with. + + Each source is a subclass; ``name`` is its spelling in the stored record. + """ - kind: ClassVar[str] + __registry_key__ = "name" + __skip_if_no_key__ = True - @abstractmethod - def details(self) -> JsonRecord: - """The fields that identify the supplying declaration.""" + name: ClassVar[str | None] = None - def to_record(self) -> JsonRecord: - return {"kind": self.kind, **self.details()} + @classmethod + @abstractmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + """The value as a Python expression, or None when no declaration states it.""" -@dataclass(frozen=True, slots=True) -class WrittenInPipeline(ValueSource): +class WrittenInPipeline(ParameterSource): """The pipeline document states the value.""" - kind: ClassVar[str] = "pipeline" + name = "pipeline" + + @classmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + return python_expression(written[parameter.name]) - def details(self) -> JsonRecord: - return {} +class FunctionDefault(ParameterSource): + """The pipeline leaves the parameter out; the signature of the recorded function supplies it. -@dataclass(frozen=True, slots=True) -class FunctionDefault(ValueSource): - """The pipeline leaves the parameter out; the function's signature supplies it.""" + The function and its version are the identity of the function record the + parameter belongs to. + """ - kind: ClassVar[str] = "function_default" - function_key: str + name = "function_default" - def details(self) -> JsonRecord: - return {"function": self.function_key} + @classmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + return python_expression(parameter.default) -@dataclass(frozen=True, slots=True) -class SuppliedAtRunTime(ValueSource): - """The platform passes the value when it calls the function.""" +class SuppliedAtRunTime(ParameterSource): + """The platform passes the value when it calls the function; no declaration states it.""" - kind: ClassVar[str] = "run_time" + name = "run_time" - def details(self) -> JsonRecord: - return {} + @classmethod + def value_expression( + cls, parameter: inspect.Parameter, written: Mapping[str, object] + ) -> str | None: + return None @dataclass(frozen=True, slots=True) -class SetOnObject(ValueSource): +class RecordedParameter: + """One parameter of a function, the value it ran with and where the value came from.""" + + name: str + value: str | None + source: type[ParameterSource] + + +class ConfigValueSource(ABC, metaclass=AutoRegisterMeta): + """How a configured object came by a value: it holds it, or ObjectState resolved it.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + + +class SetOnObject(ConfigValueSource): """The configured object itself holds the value.""" - kind: ClassVar[str] = "set_here" - scope_id: str | None + name = "set_here" - def details(self) -> JsonRecord: - return {"scope": self.scope_id} +class InheritedFrom(ConfigValueSource): + """ObjectState resolved the value from another scope or class.""" -@dataclass(frozen=True, slots=True) -class InheritedFrom(ValueSource): - """ObjectState resolved the value from this scope and class.""" + name = "inherited" - kind: ClassVar[str] = "inherited" - scope_id: str | None - source_type_name: str | None - @classmethod - def reported_by_object_state( - cls, scope_id: str | None, source_type: type | None - ) -> "InheritedFrom": - """Keep the class by name: the declaration is pickled with the compiled bundle.""" - return cls( - scope_id, - None - if source_type is None - else f"{source_type.__module__}.{source_type.__qualname__}", - ) +def _type_name(owner: type | None) -> str | None: + return None if owner is None else f"{owner.__module__}.{owner.__qualname__}" + - def details(self) -> JsonRecord: - return {"scope": self.scope_id, "type": self.source_type_name} +@dataclass(frozen=True, slots=True) +class ConfigValueOwner: + """The scope and class that supplied configuration values, and how.""" + + source: type[ConfigValueSource] + scope_id: str | None + type_name: str | None @dataclass(frozen=True, slots=True) -class RecordedValue: - """One resolved value and the declaration that supplied it.""" +class ConfigValuesFromOwner: + """The resolved values one owner supplied, by field path, as Python expressions.""" - name: str - expression: str - source: ValueSource - - def to_record(self) -> JsonRecord: - return { - "name": self.name, - "value": self.expression, - "source": self.source.to_record(), - } + owner: ConfigValueOwner + values: Mapping[str, str] ObjectStateProvenance = Mapping[str, tuple[str | None, type | None]] @@ -156,33 +180,58 @@ def _field_values(configured: object) -> dict[str, object]: return {item.name: getattr(configured, item.name) for item in fields(configured)} -def resolved_values( +def _values_with_owner( values: Mapping[str, object], provenance: ObjectStateProvenance, *, - scope_id: str | None, + holder: ConfigValueOwner, prefix: str = "", -) -> Iterator[RecordedValue]: - """Every leaf value of a resolved object, with the source ObjectState reported. - - ObjectState lists the paths it inherited; a path it does not list is held - by the object itself. A nested config the object leaves unset is inherited - whole and listed as one path, so the inherited check comes before descent. - """ +) -> Iterator[tuple[ConfigValueOwner, str, str]]: for name, value in values.items(): path = f"{prefix}{name}" if path in provenance: - yield RecordedValue( + scope_id, source_type = provenance[path] + yield ( + ConfigValueOwner(InheritedFrom, scope_id, _type_name(source_type)), path, python_expression(value), - InheritedFrom.reported_by_object_state(*provenance[path]), ) elif is_dataclass(value) and not isinstance(value, type): - yield from resolved_values( - _field_values(value), provenance, scope_id=scope_id, prefix=f"{path}." + yield from _values_with_owner( + _field_values(value), + provenance, + holder=holder, + prefix=f"{path}.", ) else: - yield RecordedValue(path, python_expression(value), SetOnObject(scope_id)) + yield holder, path, python_expression(value) + + +def resolved_values( + values: Mapping[str, object], + provenance: ObjectStateProvenance, + *, + scope_id: str | None, + holder_type: type, +) -> tuple[ConfigValuesFromOwner, ...]: + """Every leaf value of a resolved object, grouped by the owner that supplied it. + + ObjectState lists the paths it inherited; a path it does not list is held + by the object itself. A nested config the object leaves unset is inherited + whole and listed as one path, so the inherited check comes before descent. + Each owner is written once, with all the values it supplied. + """ + by_owner: dict[ConfigValueOwner, dict[str, str]] = {} + for owner, path, expression in _values_with_owner( + values, + provenance, + holder=ConfigValueOwner(SetOnObject, scope_id, _type_name(holder_type)), + ): + by_owner.setdefault(owner, {})[path] = expression + return tuple( + ConfigValuesFromOwner(owner, MappingProxyType(owned)) + for owner, owned in by_owner.items() + ) @dataclass(frozen=True, slots=True) @@ -191,15 +240,15 @@ class RunDeclaration: pipeline_code: str global_config_code: str - configuration: tuple[RecordedValue, ...] - step_configurations: tuple[tuple[RecordedValue, ...], ...] = () + configuration: tuple[ConfigValuesFromOwner, ...] + step_configurations: tuple[tuple[ConfigValuesFromOwner, ...], ...] = () """Resolved values of each enabled step, by compiled step index.""" @classmethod def before_step_resolution( cls, - orchestrator: Any, - pipeline_definition: Sequence[Any], + orchestrator: "PipelineOrchestrator", + pipeline_definition: Sequence["AbstractStep"], ) -> "RunDeclaration": """Render the documents as submitted, before the compiler resolves the steps in place.""" from objectstate import ObjectState, ObjectStateRegistry @@ -221,24 +270,22 @@ def before_step_resolution( ), ) return cls( - pipeline_code=PipelineDocumentCodec.render( + pipeline_code=PipelineDocumentCodec.render_with_every_parameter( PipelineDocumentCodec.from_values( pipeline_config=orchestrator.pipeline_config, pipeline_steps=pipeline_definition, - ), - clean_mode=False, + ) ), global_config_code=ConfigDocumentCodec.render( saved_global_config, expected_config_type=GlobalPipelineConfig, clean_mode=False, ), - configuration=tuple( - resolved_values( - _field_values(effective_config), - provenance, - scope_id=str(orchestrator.dataset_root), - ) + configuration=resolved_values( + _field_values(effective_config), + provenance, + scope_id=str(orchestrator.dataset_root), + holder_type=type(orchestrator.pipeline_config), ), ) @@ -249,128 +296,194 @@ def with_resolved_steps( return replace( self, step_configurations=tuple( - tuple( - resolved_values( - step.configuration_parameters(), - pipeline.step_provenance[index], - scope_id=pipeline.step_scope_ids[index], - ) + resolved_values( + step.configuration_parameters(), + pipeline.step_provenance[index], + scope_id=pipeline.step_scope_ids[index], + holder_type=type(step), ) for index, step in enumerate(pipeline.steps) ), ) -def _parameters(invocation: "CompiledFunctionInvocation", function_key: str) -> list[JsonRecord]: +@dataclass(frozen=True, slots=True) +class DeclaredFile: + """One declared path parameter and what the storage layer says the file holds.""" + + parameter: str + declared_as: str + path: str + content: HashedFileContent | UnhashedFileContent + + +@dataclass(frozen=True, slots=True) +class FunctionRecord: + """One function call of a step: which function, with which values and files.""" + + identity: FunctionIdentity | CustomFunctionIdentity + group: str + position: int + input_memory_type: str | None + output_memory_type: str | None + parameters: tuple[RecordedParameter, ...] + declared_files: tuple[DeclaredFile, ...] + + @classmethod + def of_invocation( + cls, + invocation: "CompiledFunctionInvocation", + *, + filemanager: FileManagerLike, + backend: str, + ) -> "FunctionRecord": + contract = invocation.contract + return cls( + identity=contract.func.identity(), + group=python_expression(invocation.key.group_key), + position=invocation.key.position, + input_memory_type=invocation.input_memory_type, + output_memory_type=invocation.output_memory_type, + parameters=_parameters(invocation), + declared_files=tuple( + DeclaredFile( + parameter=parameter_name, + declared_as=declaration.plain_name, + path=str(path), + content=declaration.content_at_run_start( + Path(path), filemanager=filemanager, backend=backend + ), + ) + for parameter_name, (declaration, path) in ( + contract.declared_path_values(invocation.kwargs_dict).items() + ) + ), + ) + + +def _parameters(invocation: "CompiledFunctionInvocation") -> tuple[RecordedParameter, ...]: """Every parameter of the function's signature with its effective value and source.""" contract = invocation.contract written = invocation.kwargs_dict + signature = contract.canonical_signature.parameters supplied_at_run_time = { *contract.runtime_owned_parameter_names, invocation.primary_input_parameter_name, } - recorded: list[RecordedValue] = [] - for name, parameter in contract.canonical_signature.parameters.items(): + sources: dict[str, type[ParameterSource]] = {} + for name, parameter in signature.items(): if name in written: - recorded.append( - RecordedValue(name, python_expression(written[name]), WrittenInPipeline()) - ) + sources[name] = WrittenInPipeline elif name in supplied_at_run_time or parameter.default is inspect.Parameter.empty: - recorded.append( - RecordedValue(name, "", SuppliedAtRunTime()) - ) + sources[name] = SuppliedAtRunTime else: - recorded.append( - RecordedValue( - name, - python_expression(parameter.default), - FunctionDefault(function_key), - ) - ) - recorded.extend( - RecordedValue(name, python_expression(value), WrittenInPipeline()) - for name, value in written.items() - if name not in contract.canonical_signature.parameters + sources[name] = FunctionDefault + keyword_only = inspect.Parameter.KEYWORD_ONLY + return tuple( + RecordedParameter( + name, + source.value_expression( + signature.get(name, inspect.Parameter(name, keyword_only)), written + ), + source, + ) + for name, source in { + **sources, + **{name: WrittenInPipeline for name in written if name not in signature}, + }.items() ) - return [value.to_record() for value in recorded] -def _declared_files( - invocation: "CompiledFunctionInvocation", - *, - filemanager: FileManagerLike, - backend: str, -) -> list[JsonRecord]: - """Each declared path parameter, with what the storage layer says the file holds.""" - declared: Mapping[str, tuple[DatasetPathDeclaration, object]] = ( - invocation.contract.declared_path_values(invocation.kwargs_dict) - ) - return [ - { - "parameter": parameter_name, - "declared_as": declaration.plain_name, - "path": str(path), - "content": declaration.content_at_run_start( - Path(path), filemanager=filemanager, backend=backend - ).to_record(), - } - for parameter_name, (declaration, path) in declared.items() - ] +@dataclass(frozen=True, slots=True) +class GpuDevice: + """The GPU device assigned to one framework's arrays.""" + memory_type: str + device_id: int -def _step_record( - plan: "CompiledStepPlan", - configuration: Iterable[RecordedValue], - *, - filemanager: FileManagerLike, - backend: str, -) -> tuple[JsonRecord, dict[str, str]]: - custom_sources: dict[str, str] = {} - functions = [] - for invocation in plan.compiled_function_pattern.iter_invocations(): - identity = invocation.contract.func.identity() - custom_sources.update(identity.custom_sources()) - functions.append( - { - "identity": identity.to_record(), - "group": python_expression(invocation.key.group_key), - "position": invocation.key.position, - "input_memory_type": invocation.input_memory_type, - "output_memory_type": invocation.output_memory_type, - "parameters": _parameters(invocation, identity.key), - "declared_files": _declared_files( + +@dataclass(frozen=True, slots=True) +class StepCalls: + """What one step ran: its identity in the pipeline, its devices and its function calls.""" + + name: str + pipeline_position: int | None + step_scope_id: str | None + input_memory_type: str | None + output_memory_type: str | None + assigned_gpu_devices: tuple[GpuDevice, ...] + functions: tuple[FunctionRecord, ...] + + @classmethod + def of_plan( + cls, + plan: "CompiledStepPlan", + *, + filemanager: FileManagerLike, + backend: str, + ) -> "StepCalls": + return cls( + name=plan.step_name, + pipeline_position=plan.pipeline_position, + step_scope_id=plan.step_scope_id, + input_memory_type=plan.input_memory_type, + output_memory_type=plan.output_memory_type, + assigned_gpu_devices=tuple( + GpuDevice(binding.memory_type.value, binding.device_id) + for binding in plan.device_assignment.bindings + ), + functions=tuple( + FunctionRecord.of_invocation( invocation, filemanager=filemanager, backend=backend - ), - } + ) + for invocation in plan.compiled_function_pattern.iter_invocations() + ), + ) + + def is_named_by(self, producer: StreamProducerIdentity) -> bool: + """A producer identity names a step by the two fields it carries.""" + return ( + self.step_scope_id == producer.step_scope_id + and self.pipeline_position == producer.pipeline_position ) - return ( - { - "name": plan.step_name, - "pipeline_position": plan.pipeline_position, - "step_scope_id": plan.step_scope_id, - "input_memory_type": plan.input_memory_type, - "output_memory_type": plan.output_memory_type, - "assigned_gpu_devices": [ - {"memory_type": binding.memory_type.value, "device_id": binding.device_id} - for binding in plan.device_assignment.bindings - ], - "functions": functions, - "configuration": [value.to_record() for value in configuration], - }, - custom_sources, - ) @dataclass(frozen=True, slots=True) -class ProducedFile: - """One output file, the step that wrote it and the record of the run.""" +class StepRecord: + """One step of the run: what it ran, and its resolved configuration by owner.""" - record: "RunRecord" - producer: StreamProducerIdentity + calls: StepCalls + configuration: tuple[ConfigValuesFromOwner, ...] - @property - def step(self) -> JsonRecord: - return self.record.step_for(self.producer) + +@dataclass(frozen=True, slots=True) +class SoftwareEnvironment: + """The interpreter and the installed distributions the run executed in.""" + + openhcs: str + python: str + python_implementation: str + platform: str + machine: str + installed_distributions: Mapping[str, str] + """Version by distribution name.""" + + @classmethod + def current(cls) -> "SoftwareEnvironment": + import openhcs + from openhcs.runtime.environment_provenance import RuntimeEnvironmentSnapshot + + environment = RuntimeEnvironmentSnapshot.current() + return cls( + openhcs=openhcs.__version__, + python=environment.python_version, + python_implementation=environment.python_implementation, + platform=environment.sys_platform, + machine=environment.machine, + installed_distributions=MappingProxyType( + {item.name: item.version for item in environment.installed_distributions} + ), + ) class UnrecordedOutputFile(LookupError): @@ -379,14 +492,29 @@ class UnrecordedOutputFile(LookupError): @dataclass(frozen=True, slots=True) class RunRecord: - """What produced an output dataset, as stored: JSON-native, addressed by its content.""" + """What produced an output dataset. Its id is the sha256 of its stored JSON.""" + + software: SoftwareEnvironment + pipeline_code: str + global_config_code: str + configuration: tuple[ConfigValuesFromOwner, ...] + steps: tuple[StepRecord, ...] + custom_sources: Mapping[str, str] + """Source text of each custom function, by the sha256 its identity states.""" - content: Mapping[str, Any] + def to_json(self) -> JsonObject: + """The stored form: the one encoder, ``python_introspect.to_jsonable``.""" + return to_jsonable(self) + + @classmethod + def from_json(cls, stored: JsonObject) -> "RunRecord": + """Read the stored form: the one decoder, ``python_introspect.dataclass_from_mapping``.""" + return dataclass_from_mapping(cls, stored) @property def record_id(self) -> str: return hashlib.sha256( - json.dumps(self.content, sort_keys=True, separators=(",", ":")).encode("utf-8") + json.dumps(self.to_json(), sort_keys=True, separators=(",", ":")).encode("utf-8") ).hexdigest() @classmethod @@ -403,12 +531,8 @@ def at_run_start( a declared input file is hashed once and the hash is of the bytes the run reads. """ - import openhcs - from openhcs.runtime.environment_provenance import RuntimeEnvironmentSnapshot - declaration = bundle.run_declaration - steps: dict[int, JsonRecord] = {} - custom_sources: dict[str, str] = {} + steps: dict[int, StepRecord] = {} for axis_id in bundle.axis_ids: context = next( context @@ -416,48 +540,28 @@ def at_run_start( if context.require_axis_id() == axis_id ) for index, plan in context.step_plans.items(): - if index in steps: - continue - steps[index], sources = _step_record( - plan, - declaration.step_configurations[index], - filemanager=filemanager, - backend=backend, - ) - custom_sources.update(sources) - environment = RuntimeEnvironmentSnapshot.current() + if index not in steps: + steps[index] = StepRecord( + StepCalls.of_plan(plan, filemanager=filemanager, backend=backend), + declaration.step_configurations[index], + ) + custom_sources: dict[str, str] = {} + for step in steps.values(): + for function in step.calls.functions: + custom_sources.update(function.identity.source_texts()) return cls( - { - "software": { - "openhcs": openhcs.__version__, - "python": environment.python_version, - "python_implementation": environment.python_implementation, - "platform": environment.sys_platform, - "machine": environment.machine, - "installed_distributions": { - item.name: item.version - for item in environment.installed_distributions - }, - }, - "pipeline_code": declaration.pipeline_code, - "global_config_code": declaration.global_config_code, - "configuration": [value.to_record() for value in declaration.configuration], - "steps": [steps[index] for index in sorted(steps)], - "custom_sources": custom_sources, - } + software=SoftwareEnvironment.current(), + pipeline_code=declaration.pipeline_code, + global_config_code=declaration.global_config_code, + configuration=declaration.configuration, + steps=tuple(steps[index] for index in sorted(steps)), + custom_sources=MappingProxyType(custom_sources), ) - @property - def steps(self) -> Sequence[JsonRecord]: - return self.content["steps"] - - def step_for(self, producer: StreamProducerIdentity) -> JsonRecord: - """The step a producer identity names, matched on the two fields it carries.""" + def step_for(self, producer: StreamProducerIdentity) -> StepCalls: + """The step a producer identity names.""" matches = [ - step - for step in self.steps - if step["step_scope_id"] == producer.step_scope_id - and step["pipeline_position"] == producer.pipeline_position + step.calls for step in self.steps if step.calls.is_named_by(producer) ] if len(matches) != 1: raise LookupError( @@ -469,17 +573,10 @@ def step_for(self, producer: StreamProducerIdentity) -> JsonRecord: @classmethod def for_dataset(cls, dataset_root: str | Path) -> Mapping[str, "RunRecord"]: """Every record the dataset's metadata document holds, by record id.""" - return MappingProxyType( - { - record_id: cls(content) - for record_id, content in _metadata_document(Path(dataset_root)) - .get(RUN_RECORDS_KEY, {}) - .items() - } - ) + return DatasetRunRecords.of_dataset(Path(dataset_root)).run_records @classmethod - def for_output_file(cls, output_file: str | Path) -> ProducedFile: + def for_output_file(cls, output_file: str | Path) -> "ProducedFile": """The record and producing step of one output file. The answer comes only from what the writer stored for this path; a file @@ -487,18 +584,15 @@ def for_output_file(cls, output_file: str | Path) -> ProducedFile: """ path = Path(output_file).resolve() dataset_root = _dataset_root_of(path) - document = _metadata_document(dataset_root) - entry = document.get(PRODUCED_FILES_KEY, {}).get( - path.relative_to(dataset_root).as_posix() - ) + stored = DatasetRunRecords.of_dataset(dataset_root) + entry = stored.produced_files.get(path.relative_to(dataset_root).as_posix()) if entry is None: raise UnrecordedOutputFile( f"{path} is not recorded as a step output in " f"{_metadata_path(dataset_root)}." ) return ProducedFile( - record=cls(document[RUN_RECORDS_KEY][entry["run_record"]]), - producer=StreamProducerIdentity.from_payload(entry["producer"]), + record=stored.run_records[entry.run_record], producer=entry.producer ) def rebuild(self) -> tuple["PipelineDocument", "GlobalPipelineConfig"]: @@ -519,12 +613,12 @@ def rebuild(self) -> tuple["PipelineDocument", "GlobalPipelineConfig"]: ) function_name_by_source = { - function["identity"]["custom_source_sha256"]: function["identity"]["name"] + source_sha256: function.identity.name for step in self.steps - for function in step["functions"] - if "custom_source_sha256" in function["identity"] + for function in step.calls.functions + for source_sha256 in function.identity.recorded_source_hashes() } - for recorded_sha256, source_text in self.content["custom_sources"].items(): + for recorded_sha256, source_text in self.custom_sources.items(): if CustomFunctionSource.content_digest(source_text.encode("utf-8")) != ( recorded_sha256 ): @@ -540,24 +634,91 @@ def rebuild(self) -> tuple["PipelineDocument", "GlobalPipelineConfig"]: source_text, persist=False, emit_signal=False ) return ( - PipelineDocumentCodec.from_source(self.content["pipeline_code"]), + PipelineDocumentCodec.from_source(self.pipeline_code), ConfigDocumentCodec.from_source( - self.content["global_config_code"], + self.global_config_code, expected_config_type=GlobalPipelineConfig, ), ) +@dataclass(frozen=True, slots=True) +class StoredProducedFile: + """What the metadata document stores for one saved file.""" + + run_record: str + producer: StreamProducerIdentity + + +@dataclass(frozen=True, slots=True) +class ProducedFile: + """One output file, the step that wrote it and the record of the run.""" + + record: RunRecord + producer: StreamProducerIdentity + + @property + def step(self) -> StepCalls: + return self.record.step_for(self.producer) + + +@dataclass(frozen=True, slots=True) +class DatasetRunRecords: + """The part of a dataset's metadata document that says what produced its files. + + The document stores each field of this class under the field's name. + """ + + run_records: Mapping[str, RunRecord] = field(default_factory=dict) + produced_files: Mapping[str, StoredProducedFile] = field(default_factory=dict) + """By path relative to the dataset root.""" + + @classmethod + def in_document(cls, document: JsonObject) -> "DatasetRunRecords": + return dataclass_from_mapping( + cls, + { + item.name: document[item.name] + for item in fields(cls) + if item.name in document + }, + ) + + @classmethod + def of_dataset(cls, dataset_root: Path) -> "DatasetRunRecords": + return cls.in_document( + json.loads(_metadata_path(dataset_root).read_text(encoding="utf-8")) + ) + + def with_run( + self, record: RunRecord, produced: Mapping[str, StreamProducerIdentity] + ) -> "DatasetRunRecords": + """Add one run's files; keep only the records some file still refers to.""" + record_id = record.record_id + produced_files = { + **self.produced_files, + **{ + path: StoredProducedFile(record_id, producer) + for path, producer in produced.items() + }, + } + referenced = {entry.run_record for entry in produced_files.values()} + return DatasetRunRecords( + run_records={ + stored_id: stored + for stored_id, stored in {**self.run_records, record_id: record}.items() + if stored_id in referenced + }, + produced_files=produced_files, + ) + + def _metadata_path(dataset_root: Path) -> Path: from openhcs.core.virtual_workspace_metadata import get_metadata_path return Path(get_metadata_path(dataset_root)) -def _metadata_document(dataset_root: Path) -> Mapping[str, Any]: - return json.loads(_metadata_path(dataset_root).read_text(encoding="utf-8")) - - def _dataset_root_of(output_file: Path) -> Path: for directory in output_file.parents: if _metadata_path(directory).is_file(): @@ -599,7 +760,7 @@ def by_dataset_root( def write_into( self, - document: dict[str, Any], + document: dict[str, JsonValue], *, dataset_root: Path, record: RunRecord, @@ -611,15 +772,14 @@ def write_into( """ if not self.producer_by_path: return - records = document.setdefault(RUN_RECORDS_KEY, {}) - produced = document.setdefault(PRODUCED_FILES_KEY, {}) - record_id = record.record_id - records[record_id] = dict(record.content) - for path, producer in self.producer_by_path.items(): - produced[path.relative_to(dataset_root).as_posix()] = { - "run_record": record_id, - "producer": dict(producer.to_payload()), - } - referenced = {entry["run_record"] for entry in produced.values()} - for stale in set(records) - referenced: - del records[stale] + document.update( + to_jsonable( + DatasetRunRecords.in_document(document).with_run( + record, + { + path.relative_to(dataset_root).as_posix(): producer + for path, producer in self.producer_by_path.items() + }, + ) + ) + ) diff --git a/openhcs/serialization/pycodify_formatters.py b/openhcs/serialization/pycodify_formatters.py index 249846beb..dfa04e26d 100644 --- a/openhcs/serialization/pycodify_formatters.py +++ b/openhcs/serialization/pycodify_formatters.py @@ -310,8 +310,6 @@ def format(self, value: FunctionStep, context: FormatContext) -> SourceFragment: current_value = step_values[name] else: current_value = param.default - if name == "func" and not context.clean_mode: - current_value = with_every_parameter_written(current_value) if name in default_values: default_value = default_values[name] diff --git a/tests/fixtures/mcp/mcp_tool_contract.json b/tests/fixtures/mcp/mcp_tool_contract.json index a8864b995..46df4c84a 100644 --- a/tests/fixtures/mcp/mcp_tool_contract.json +++ b/tests/fixtures/mcp/mcp_tool_contract.json @@ -4459,15 +4459,15 @@ "readOnlyHint": true, "title": "Read run record" }, - "description": "Read-only: what produced an output dataset. Pass an output dataset root (the directory holding openhcs_metadata.json) to get every run record by record id, or one output file inside a dataset to get the producer identity stored for that file, the producing step's entry (name, pipeline position, functions with identity, parameters with their sources, declared files with content hashes, devices) and the record of that run. A file the dataset does not list is an error, never a guess from its name. By default pipeline_code, global_config_code, custom_sources texts and software.installed_distributions are replaced by their sha256 and length and named in shortened_fields; pass include_sources=true for the full texts.", + "description": "Read-only: what produced an output dataset. Pass an output dataset root (the directory holding openhcs_metadata.json) to get every run record by record id, or one output file inside a dataset to get the producer identity stored for that file, the producing step and the record of that run. A file the dataset does not list is an error, never a guess from its name. Each record lists what every step ran: each function with its identity (registry key, defining and imported library versions, sha256 of a custom function's source), every parameter with its value and source (pipeline, function_default, run_time), declared files with content hashes, and devices. Pass include_full_record=true to add the complete stored record: pipeline and global config documents, resolved configuration with the owner of every value, custom source texts and all installed distributions.", "execution": null, "icons": null, "inputSchema": { "additionalProperties": false, "properties": { - "include_sources": { + "include_full_record": { "default": false, - "title": "Include Sources", + "title": "Include Full Record", "type": "boolean" }, "path": { diff --git a/tests/integration/test_run_record_round_trip.py b/tests/integration/test_run_record_round_trip.py index bfc1a94a0..5c187d34b 100644 --- a/tests/integration/test_run_record_round_trip.py +++ b/tests/integration/test_run_record_round_trip.py @@ -14,7 +14,7 @@ from openhcs.core.config import GlobalPipelineConfig, LazyPathPlanningConfig, PipelineConfig from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.progress import ProgressExecutionContext, set_progress_queue -from openhcs.core.run_record import PRODUCED_FILES_KEY, RunRecord +from openhcs.core.run_record import DatasetRunRecords, RunRecord from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.processors.numpy_processor import ( stack_percentile_normalize, @@ -52,9 +52,7 @@ def run(plate: Path, pipeline_config: PipelineConfig, steps: list) -> Path: def produced_file_bytes(output_root: Path) -> dict[str, str]: """sha256 of every file the dataset records as a step output, by relative path.""" - produced = json.loads((output_root / "openhcs_metadata.json").read_text())[ - PRODUCED_FILES_KEY - ] + produced = DatasetRunRecords.of_dataset(output_root).produced_files return { relative: hashlib.sha256((output_root / relative).read_bytes()).hexdigest() for relative in sorted(produced) @@ -66,7 +64,7 @@ def produced_file_bytes(output_root: Path) -> dict[str, str]: def without_run_location(record: RunRecord, plate: Path) -> str: """The record with this run's plate path and submission scope written as placeholders.""" - text = json.dumps(record.content, sort_keys=True).replace(str(plate), "") + text = json.dumps(record.to_json(), sort_keys=True).replace(str(plate), "") return SUBMISSION_SCOPE.sub("::pipeline::", text) @@ -94,7 +92,7 @@ def test_record_read_from_one_output_file_reruns_to_identical_outputs(tmp_path, assert tables and images produced = RunRecord.for_output_file(first_root / tables[0]) - assert produced.step["name"] == "bright" + assert produced.step.name == "bright" CustomFunctionRuntimeRegistry.clear() document, global_config = produced.record.rebuild() @@ -111,13 +109,9 @@ def test_record_read_from_one_output_file_reruns_to_identical_outputs(tmp_path, # The rebuilt document writes every parameter, so a value the first run took # from a signature default is, in the second run, written in the pipeline. # Nothing else differs once the plate path and the submission scope are set aside. - assert set(differences.values()) == { - ("function_default", "pipeline"), - ("openhcs:processors_numpy_processor_stack_percentile_normalize", None), - ("openhcs:count_bright", None), - } + assert set(differences.values()) == {("function_default", "pipeline")} assert all( - re.fullmatch(r"/steps/\d+/functions/\d+/parameters/\d+/source/(kind|function)", where) + re.fullmatch(r"/steps/\d+/calls/functions/\d+/parameters/\d+/source", where) for where in differences ) diff --git a/tests/run_records.py b/tests/run_records.py index 58da79ec7..e309cd768 100644 --- a/tests/run_records.py +++ b/tests/run_records.py @@ -2,7 +2,12 @@ from types import MappingProxyType -from openhcs.core.run_record import ProducedFiles, RunDeclaration, RunRecord +from openhcs.core.run_record import ( + ProducedFiles, + RunDeclaration, + RunRecord, + SoftwareEnvironment, +) def minimal_run_declaration() -> RunDeclaration: @@ -15,17 +20,15 @@ def minimal_run_declaration() -> RunDeclaration: def minimal_run_record() -> RunRecord: - """A record with every stored key and no steps.""" + """A record of a run with no steps.""" declaration = minimal_run_declaration() return RunRecord( - { - "software": {}, - "pipeline_code": declaration.pipeline_code, - "global_config_code": declaration.global_config_code, - "configuration": [], - "steps": [], - "custom_sources": {}, - } + software=SoftwareEnvironment.current(), + pipeline_code=declaration.pipeline_code, + global_config_code=declaration.global_config_code, + configuration=declaration.configuration, + steps=(), + custom_sources=MappingProxyType({}), ) diff --git a/tests/unit/agent/test_authoring_roundtrip_boundaries.py b/tests/unit/agent/test_authoring_roundtrip_boundaries.py index 5e8eaaeff..30605b323 100644 --- a/tests/unit/agent/test_authoring_roundtrip_boundaries.py +++ b/tests/unit/agent/test_authoring_roundtrip_boundaries.py @@ -162,14 +162,8 @@ def test_real_authoring_render_parse_retains_registry_memory_contract( original_step.func, original_step.name ) ) - if clean: - if kwargs != {"gamma": 1, "gain": 1}: - assert step.func[1] == kwargs - else: - # The complete text writes every parameter, so a later change of a - # default cannot change what the text means. - assert kwargs.items() <= step.func[1].items() - assert set(step.func[1]) > set(kwargs) + if kwargs != {"gamma": 1, "gain": 1} or not clean: + assert step.func[1] == kwargs def test_registry_reference_formatting_uses_alias_without_resolving( diff --git a/tests/unit/test_run_record.py b/tests/unit/test_run_record.py index ea375c6b4..9e0a8ba62 100644 --- a/tests/unit/test_run_record.py +++ b/tests/unit/test_run_record.py @@ -22,15 +22,26 @@ from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.progress import ProgressExecutionContext, set_progress_queue +from polystore.file_content import HashedFileContent, UnhashedFileContent + from openhcs.core.run_record import ( - PRODUCED_FILES_KEY, - RUN_RECORDS_KEY, + ConfigValueOwner, + DatasetRunRecords, + DeclaredFile, + FunctionDefault, + FunctionRecord, + InheritedFrom, ProducedFiles, + RecordedParameter, RunDeclaration, RunRecord, + SetOnObject, + SuppliedAtRunTime, UnrecordedOutputFile, + WrittenInPipeline, python_expression, ) +from openhcs.runtime.environment_provenance import InstalledDistributionVersion from openhcs.core.steps.function_step import FunctionStep from openhcs.core.virtual_workspace_metadata import AtomicMetadataWriter from openhcs.processing.backends.lib_registry.registry_service import RegistryService @@ -155,14 +166,20 @@ def run(orchestrator: PipelineOrchestrator, steps: list, tmp_path: Path): return bundle -def by_name(records: list[dict]) -> dict[str, dict]: - return {record["name"]: record for record in records} +def only_function(record: RunRecord, step_name: str) -> FunctionRecord: + (step,) = (step for step in record.steps if step.calls.name == step_name) + (function,) = step.calls.functions + return function + +def parameters_by_name(function: FunctionRecord) -> dict[str, RecordedParameter]: + return {parameter.name: parameter for parameter in function.parameters} -def only_function(record: RunRecord, step_name: str) -> dict: - (step,) = (step for step in record.steps if step["name"] == step_name) - (function,) = step["functions"] - return function + +def stored_records(dataset_root: Path) -> DatasetRunRecords: + return DatasetRunRecords.in_document( + json.loads((dataset_root / "openhcs_metadata.json").read_text()) + ) def normalize_and_count() -> list: @@ -176,35 +193,67 @@ def test_parameter_left_out_is_recorded_with_the_function_default(tmp_path): record = record_of(orchestrator_for(_plate(tmp_path / "source")), normalize_and_count()) function = only_function(record, "norm") - parameters = by_name(function["parameters"]) - assert function["identity"]["key"] == NORMALIZE_KEY - assert parameters["low_percentile"] == { - "name": "low_percentile", - "value": "2.0", - "source": {"kind": "pipeline"}, - } - assert parameters["high_percentile"] == { - "name": "high_percentile", - "value": "99.0", - "source": {"kind": "function_default", "function": NORMALIZE_KEY}, - } - assert parameters["stack"]["source"] == {"kind": "run_time"} + parameters = parameters_by_name(function) + assert function.identity.key == NORMALIZE_KEY + assert parameters["low_percentile"] == RecordedParameter( + "low_percentile", "2.0", WrittenInPipeline + ) + assert parameters["high_percentile"] == RecordedParameter( + "high_percentile", "99.0", FunctionDefault + ) + assert parameters["stack"] == RecordedParameter("stack", None, SuppliedAtRunTime) + + +def test_stored_json_reads_back_as_the_same_record(tmp_path, register): + """One encoder and one decoder: every type of the record survives the round trip.""" + plate = _plate(tmp_path / "source") + (plate / "weights.bin").write_bytes(b"weights") + record = record_of( + orchestrator_for(plate), + [ + FunctionStep(func=(stack_percentile_normalize, {"low_percentile": 2.0}), name="norm"), + FunctionStep( + func=( + register(count_bright_source(0.5)), + {"weights": "weights.bin", "notes": "notes/log.txt"}, + ), + name="bright", + ), + ], + ) + + stored = json.loads(json.dumps(record.to_json())) + + assert RunRecord.from_json(stored) == record + assert RunRecord.from_json(stored).record_id == record.record_id + (norm_parameter, *_rest) = stored["steps"][0]["calls"]["functions"][0]["parameters"] + assert norm_parameter == {"name": "stack", "value": None, "source": "run_time"} def test_saved_pipeline_text_stays_short_and_the_record_copy_is_complete(tmp_path): steps = normalize_and_count() record = record_of(orchestrator_for(_plate(tmp_path / "source")), steps) - document = PipelineDocumentCodec.from_source(record.content["pipeline_code"]) + document = PipelineDocumentCodec.from_source(record.pipeline_code) saved_text = PipelineDocumentCodec.render(document) assert "low_percentile" in saved_text assert "high_percentile" not in saved_text assert "description=None" not in saved_text - assert "high_percentile" in record.content["pipeline_code"] - assert "description=None" in record.content["pipeline_code"] + submitted = PipelineDocumentCodec.from_values( + pipeline_config=PipelineConfig(), pipeline_steps=steps + ) + # Three renderings of one source, three promises: the saved text leaves out + # defaults; the unabridged text writes every stored field but no parameter + # the pipeline does not state; the record's copy leaves nothing to a default. + assert "high_percentile" not in PipelineDocumentCodec.render( + submitted, clean_mode=False + ) + assert "description=None" in PipelineDocumentCodec.render(submitted, clean_mode=False) + assert "high_percentile" in record.pipeline_code + assert "description=None" in record.pipeline_code assert ( PipelineDocumentCodec.render(document, clean_mode=False) - == record.content["pipeline_code"] + == record.pipeline_code ) @@ -226,18 +275,18 @@ def test_custom_function_is_identified_by_its_source_text(tmp_path, register): record = record_of( orchestrator, [FunctionStep(func=register(source), name="bright")] ) - identity = only_function(record, "bright")["identity"] + identity = only_function(record, "bright").identity sha256 = hashlib.sha256(source.encode("utf-8")).hexdigest() - assert identity["name"] == "count_bright" - assert identity["custom_source_sha256"] == sha256 - assert record.content["custom_sources"] == {sha256: source} + assert identity.name == "count_bright" + assert identity.source_sha256 == sha256 + assert record.custom_sources == {sha256: source} records[threshold] = record first, second = ( - only_function(record, "bright")["identity"] for record in records.values() + only_function(record, "bright").identity for record in records.values() ) - assert first["key"] == second["key"] - assert first["custom_source_sha256"] != second["custom_source_sha256"] + assert first.key == second.key + assert first.source_sha256 != second.source_sha256 assert records[0.5].record_id != records[0.25].record_id @@ -247,9 +296,10 @@ def test_custom_function_imports_name_installed_versions(tmp_path, register): [FunctionStep(func=register(count_bright_source(0.5)), name="bright")], ) - libraries = only_function(record, "bright")["identity"]["imports"]["libraries"] - assert {"distribution": "numpy", "version": importlib.metadata.version("numpy")} in ( - libraries + libraries = only_function(record, "bright").identity.imports.libraries + assert ( + InstalledDistributionVersion("numpy", importlib.metadata.version("numpy")) + in libraries ) @@ -289,13 +339,13 @@ def compiled(): ] ) - def declared_files(bundle) -> dict[str, dict]: + def declared_files(bundle) -> dict[str, DeclaredFile]: record = RunRecord.at_run_start( bundle, filemanager=orchestrator.filemanager, backend=Backend.DISK.value ) return { - declared["parameter"]: declared - for declared in only_function(record, "bright")["declared_files"] + declared.parameter: declared + for declared in only_function(record, "bright").declared_files } bundle = compiled() @@ -303,26 +353,22 @@ def declared_files(bundle) -> dict[str, dict]: declared = declared_files(bundle) assert hashed_paths == [weights] - assert declared["weights"] == { - "parameter": "weights", - "declared_as": "input file", - "path": str(weights), - "content": { - "hashed": True, - "sha256": hashlib.sha256(b"first weights").hexdigest(), - "size_bytes": len(b"first weights"), - }, - } - assert declared["notes"]["declared_as"] == "output file" - assert declared["notes"]["content"]["hashed"] is False - assert declared["notes"]["content"]["reason"].strip() + assert declared["weights"] == DeclaredFile( + parameter="weights", + declared_as="input file", + path=str(weights), + content=HashedFileContent( + hashlib.sha256(b"first weights").hexdigest(), len(b"first weights") + ), + ) + assert declared["notes"].declared_as == "output file" + assert isinstance(declared["notes"].content, UnhashedFileContent) + assert declared["notes"].content.reason.strip() weights.write_bytes(b"second weights!") - assert declared_files(bundle)["weights"]["content"] == { - "hashed": True, - "sha256": hashlib.sha256(b"second weights!").hexdigest(), - "size_bytes": len(b"second weights!"), - } + assert declared_files(bundle)["weights"].content == HashedFileContent( + hashlib.sha256(b"second weights!").hexdigest(), len(b"second weights!") + ) def test_resolved_configuration_names_the_source_of_each_value(tmp_path, monkeypatch): @@ -337,17 +383,25 @@ def keep_resolved_pipeline(declaration, pipeline): plate = _plate(tmp_path / "source") record = record_of(orchestrator_for(plate), normalize_and_count()) - configuration = by_name(record.content["configuration"]) - assert configuration["num_workers"]["value"] == "1" - assert configuration["num_workers"]["source"] == {"kind": "set_here", "scope": str(plate)} - inherited = configuration["materialization_results_path"]["source"] - assert inherited["kind"] == "inherited" - assert inherited["type"] == "openhcs.core.config.GlobalPipelineConfig" + owner_by_path = { + path: (group.owner, value) + for group in record.configuration + for path, value in group.values.items() + } + assert owner_by_path["num_workers"] == ( + ConfigValueOwner(SetOnObject, str(plate), "openhcs.core.config.PipelineConfig"), + "1", + ) + inherited, _value = owner_by_path["materialization_results_path"] + assert inherited.source is InheritedFrom + assert inherited.type_name == "openhcs.core.config.GlobalPipelineConfig" + owners = [group.owner for group in record.configuration] + assert len(owners) == len(set(owners)), "each owner is written once" (pipeline,) = resolved_pipelines assert len(record.steps) == len(pipeline.steps) == 2 for index, step in enumerate(record.steps): - recorded_names = {value["name"] for value in step["configuration"]} + recorded_names = {path for group in step.configuration for path in group.values} assert pipeline.step_provenance[index] assert set(pipeline.step_provenance[index]) <= recorded_names @@ -357,9 +411,7 @@ def test_each_output_file_names_the_step_that_produced_it(tmp_path): orchestrator = orchestrator_for(plate) bundle = run(orchestrator, normalize_and_count(), tmp_path) output_root = tmp_path / "source_out" - produced = json.loads((output_root / "openhcs_metadata.json").read_text())[ - PRODUCED_FILES_KEY - ] + produced = stored_records(output_root).produced_files (image,) = (path for path in produced if path.endswith(".tif")) (table,) = (path for path in produced if path.endswith(".csv")) @@ -368,14 +420,14 @@ def test_each_output_file_names_the_step_that_produced_it(tmp_path): # "count" returns its image unchanged, so the saved pixels are still the # ones "norm" produced: the same producer a viewer item for them carries. - assert image_file.step["name"] == "norm" - assert table_file.step["name"] == "count" + assert image_file.step.name == "norm" + assert table_file.step.name == "count" assert table_file.record.record_id == image_file.record.record_id (context,) = bundle.runtime_contexts.values() count_plan = context.step_plans[1] (output_plan,) = count_plan.artifact_outputs.values() - assert produced[table]["producer"] == dict( - count_plan.producer_identity_for_artifact(output_plan).to_payload() + assert produced[table].producer == count_plan.producer_identity_for_artifact( + output_plan ) unlisted = output_root / "A01_s001_w1_z001_t001_unlisted.csv" unlisted.write_text("not written by a step") @@ -388,8 +440,8 @@ def test_input_plate_metadata_receives_no_record(tmp_path): run(orchestrator_for(plate), normalize_and_count(), tmp_path) document = json.loads((plate / "openhcs_metadata.json").read_text()) - assert RUN_RECORDS_KEY not in document - assert PRODUCED_FILES_KEY not in document + assert not set(document) & {"run_records", "produced_files"} + assert stored_records(plate) == DatasetRunRecords() def test_second_pipeline_into_the_same_dataset_keeps_only_referenced_records(tmp_path): @@ -401,9 +453,9 @@ def test_second_pipeline_into_the_same_dataset_keeps_only_referenced_records(tmp tmp_path, ) - document = json.loads((tmp_path / "source_out" / "openhcs_metadata.json").read_text()) - referenced = {entry["run_record"] for entry in document[PRODUCED_FILES_KEY].values()} - assert set(document[RUN_RECORDS_KEY]) == referenced + stored = stored_records(tmp_path / "source_out") + referenced = {entry.run_record for entry in stored.produced_files.values()} + assert set(stored.run_records) == referenced assert len(referenced) == 2 @@ -459,16 +511,14 @@ def test_subdirectory_metadata_updates_keep_the_records(tmp_path): plate = _plate(tmp_path / "source") run(orchestrator_for(plate), normalize_and_count(), tmp_path) metadata_path = tmp_path / "source_out" / "openhcs_metadata.json" - before = json.loads(metadata_path.read_text()) - assert before[RUN_RECORDS_KEY] and before[PRODUCED_FILES_KEY] + before = stored_records(metadata_path.parent) + assert before.run_records and before.produced_files writer = AtomicMetadataWriter() writer.merge_subdirectory_metadata(metadata_path, {"added": {"note": "kept"}}) writer.replace_subdirectory_metadata(metadata_path, "replaced", {"note": "kept"}) - after = json.loads(metadata_path.read_text()) - assert after[RUN_RECORDS_KEY] == before[RUN_RECORDS_KEY] - assert after[PRODUCED_FILES_KEY] == before[PRODUCED_FILES_KEY] + assert stored_records(metadata_path.parent) == before def test_output_dataset_with_records_opens_as_an_input_plate(tmp_path): @@ -489,11 +539,10 @@ def test_every_registered_function_default_can_be_written_in_the_record(): contract = CallableContract.from_callable(metadata.func) step = FunctionStep(func=metadata.func, name=key) try: - PipelineDocumentCodec.render( + PipelineDocumentCodec.render_with_every_parameter( PipelineDocumentCodec.from_values( pipeline_config=PipelineConfig(), pipeline_steps=[step] - ), - clean_mode=False, + ) ) for name, parameter in contract.canonical_signature.parameters.items(): if ( diff --git a/tests/unit/test_run_record_capability.py b/tests/unit/test_run_record_capability.py index 627fa553a..ad0459c44 100644 --- a/tests/unit/test_run_record_capability.py +++ b/tests/unit/test_run_record_capability.py @@ -5,59 +5,104 @@ from types import SimpleNamespace import pytest +from polystore.file_content import HashedFileContent from polystore.streaming.identity import StreamProducerIdentity +from python_introspect import to_jsonable from openhcs.agent.capabilities import ReadRunRecordCapability from openhcs.agent.path_policy import AgentPathPolicy from openhcs.agent.services.run_record_service import RunRecordService -from openhcs.core.run_record import RunRecord +from openhcs.core.function_identity import CustomFunctionIdentity, ModuleLibraries +from openhcs.core.run_record import ( + ConfigValueOwner, + ConfigValuesFromOwner, + DatasetRunRecords, + DeclaredFile, + FunctionRecord, + RecordedParameter, + RunRecord, + SetOnObject, + SoftwareEnvironment, + StepCalls, + StepRecord, + SuppliedAtRunTime, + WrittenInPipeline, +) from openhcs.mcp.execution import McpTransportExecutor from openhcs.mcp.server import build_server -PIPELINE_CODE = "pipeline = []\n" * 40 CUSTOM_SOURCE = "def count(image):\n return image\n" CUSTOM_SOURCE_SHA256 = hashlib.sha256(CUSTOM_SOURCE.encode("utf-8")).hexdigest() -STEP = { - "name": "count", - "pipeline_position": 0, - "step_scope_id": "s::step_0", - "input_memory_type": "numpy", - "output_memory_type": "numpy", - "assigned_gpu_devices": [], - "functions": [], - "configuration": [], -} -RECORD = { - "software": { - "openhcs": "0.0.0", - "python": "3.12.0", - "installed_distributions": {"numpy": "2.0.0", "polystore": "1.0.0"}, - }, - "pipeline_code": PIPELINE_CODE, - "global_config_code": "config = None\n", - "configuration": [], - "steps": [STEP], - "custom_sources": {CUSTOM_SOURCE_SHA256: CUSTOM_SOURCE}, -} -RECORD_ID = RunRecord(RECORD).record_id -PRODUCER = dict( - StreamProducerIdentity.pipeline_output( - output_kind="artifact", - output_key="counts", - projection_key="counts", - step_name="count", - pipeline_position=0, - step_scope_id="s::step_0", - ).to_payload() +NO_LIBRARIES = ModuleLibraries((), ()) +STEP = StepCalls( + name="count", + pipeline_position=0, + step_scope_id="s::step_0", + input_memory_type="numpy", + output_memory_type="numpy", + assigned_gpu_devices=(), + functions=( + FunctionRecord( + identity=CustomFunctionIdentity( + key="openhcs:count", + module="openhcs.processing.custom_functions", + name="count", + defined_in=NO_LIBRARIES, + imports=NO_LIBRARIES, + source_sha256=CUSTOM_SOURCE_SHA256, + ), + group="None", + position=0, + input_memory_type="numpy", + output_memory_type="numpy", + parameters=( + RecordedParameter("image", None, SuppliedAtRunTime), + RecordedParameter("weights", "'weights.bin'", WrittenInPipeline), + ), + declared_files=( + DeclaredFile( + parameter="weights", + declared_as="input file", + path="/data/weights.bin", + content=HashedFileContent("0" * 64, 7), + ), + ), + ), + ), +) +RECORD = RunRecord( + software=SoftwareEnvironment( + openhcs="0.0.0", + python="3.12.0", + python_implementation="CPython", + platform="linux", + machine="x86_64", + installed_distributions={"numpy": "2.0.0", "polystore": "1.0.0"}, + ), + pipeline_code="pipeline = []\n" * 40, + global_config_code="config = None\n", + configuration=( + ConfigValuesFromOwner( + ConfigValueOwner(SetOnObject, "s", "openhcs.core.config.PipelineConfig"), + {"num_workers": "1"}, + ), + ), + steps=(StepRecord(STEP, ()),), + custom_sources={CUSTOM_SOURCE_SHA256: CUSTOM_SOURCE}, +) +RECORD_ID = RECORD.record_id +PRODUCER = StreamProducerIdentity.pipeline_output( + output_kind="artifact", + output_key="counts", + projection_key="counts", + step_name="count", + pipeline_position=0, + step_scope_id="s::step_0", ) -def _digest(text): - return { - "shortened": True, - "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), - "length": len(text), - } +def as_json(value): + return json.loads(json.dumps(to_jsonable(value))) @pytest.fixture @@ -70,10 +115,9 @@ def dataset(tmp_path): json.dumps( { "subdirectories": {}, - "run_records": {RECORD_ID: RECORD}, - "produced_files": { - "results/a.csv": {"run_record": RECORD_ID, "producer": PRODUCER} - }, + **to_jsonable( + DatasetRunRecords().with_run(RECORD, {"results/a.csv": PRODUCER}) + ), } ), encoding="utf-8", @@ -103,60 +147,55 @@ async def exercise(): executor.close() -def test_dataset_root_returns_every_record_shortened_by_default(dataset, read): +def test_dataset_root_returns_what_each_step_ran_by_default(dataset, read): result = read(path=str(dataset)) assert result["errors"] == [] - assert result["include_sources"] is False assert result["produced_files"] == [] - (entry,) = result["records"] - assert entry["record_id"] == RECORD_ID - assert entry["shortened_fields"] == [ - "software.installed_distributions", - "pipeline_code", - "global_config_code", - f"custom_sources.{CUSTOM_SOURCE_SHA256}", + assert result["records"] == [ + {"record_id": RECORD_ID, "steps": [as_json(STEP)], "record": None} ] - record = entry["record"] - assert set(record) == set(RECORD) - assert record["pipeline_code"] == _digest(PIPELINE_CODE) - assert record["global_config_code"] == _digest(RECORD["global_config_code"]) - assert record["custom_sources"] == {CUSTOM_SOURCE_SHA256: _digest(CUSTOM_SOURCE)} - assert record["software"]["installed_distributions"] == _digest( - '{"numpy":"2.0.0","polystore":"1.0.0"}' - ) - assert record["software"]["openhcs"] == "0.0.0" - assert record["steps"] == [STEP] + (function,) = result["records"][0]["steps"][0]["functions"] + assert function["identity"]["source_sha256"] == CUSTOM_SOURCE_SHA256 + assert function["parameters"][1] == { + "name": "weights", + "value": "'weights.bin'", + "source": "pipeline", + } + assert function["declared_files"][0]["content"] == { + "sha256": "0" * 64, + "size_bytes": 7, + } -def test_dataset_root_with_sources_returns_the_stored_record(dataset, read): - result = read(path=str(dataset), include_sources=True) +def test_full_record_is_returned_on_request_and_reads_back(dataset, read): + result = read(path=str(dataset), include_full_record=True) - assert result["include_sources"] is True - assert result["records"] == [ - {"record_id": RECORD_ID, "record": RECORD, "shortened_fields": []} - ] + (entry,) = result["records"] + assert entry["steps"] == [as_json(STEP)] + assert RunRecord.from_json(entry["record"]) == RECORD -@pytest.mark.parametrize("include_sources", (False, True)) -def test_output_file_returns_its_producer_step_and_record(dataset, read, include_sources): +@pytest.mark.parametrize("include_full_record", (False, True)) +def test_output_file_returns_its_producer_step_and_record( + dataset, read, include_full_record +): output_file = dataset / "results" / "a.csv" - result = read(path=str(output_file), include_sources=include_sources) + result = read(path=str(output_file), include_full_record=include_full_record) assert result["errors"] == [] assert result["produced_files"] == [ { "path": str(output_file), "record_id": RECORD_ID, - "producer": PRODUCER, - "step": STEP, + "producer": as_json(PRODUCER), + "step": as_json(STEP), } ] (entry,) = result["records"] assert entry["record_id"] == RECORD_ID - assert (entry["record"] == RECORD) is include_sources - assert (entry["shortened_fields"] == []) is include_sources + assert (entry["record"] == as_json(RECORD)) is include_full_record def test_unlisted_file_is_an_error_not_a_guess(dataset, read): @@ -186,12 +225,12 @@ def test_cli_command_is_generated_from_the_declaration(): parser = dev_client._build_parser() call = dev_client._calls_from_args( - parser.parse_args(("read-run-record", "/data/out", "--include-sources")) + parser.parse_args(("read-run-record", "/data/out", "--include-full-record")) )[0] assert call.name == "openhcs_read_run_record" - assert call.arguments == {"path": "/data/out", "include_sources": True} + assert call.arguments == {"path": "/data/out", "include_full_record": True} default_call = dev_client._calls_from_args( parser.parse_args(("read-run-record", "/data/out")) )[0] - assert default_call.arguments == {"path": "/data/out", "include_sources": False} + assert default_call.arguments == {"path": "/data/out", "include_full_record": False} diff --git a/tests/unit/test_source_round_trip.py b/tests/unit/test_source_round_trip.py index e2e66d260..09417710e 100644 --- a/tests/unit/test_source_round_trip.py +++ b/tests/unit/test_source_round_trip.py @@ -325,22 +325,7 @@ def test_pipeline_document_loads_back_every_pattern_shape(clean_mode): loaded = PipelineDocumentCodec.from_source(source) assert raw(loaded.pipeline_config) == raw(document.pipeline_config), source - if clean_mode: - assert raw(loaded.pipeline_steps) == raw(document.pipeline_steps), source - else: - # The full text writes every function parameter, so what loads back - # states the signature defaults the original left out: the same - # declaration, the same short text, and the same full text again. - assert all( - loaded_step.same_declaration(step) - for loaded_step, step in zip( - loaded.pipeline_steps, document.pipeline_steps, strict=True - ) - ), source - assert PipelineDocumentCodec.render(loaded) == PipelineDocumentCodec.render( - document - ) - assert PipelineDocumentCodec.render(loaded, clean_mode=False) == source + assert raw(loaded.pipeline_steps) == raw(document.pipeline_steps), source assert samples.unsampled == [] assert "path_root = Path('/data/screen')" in source assert "CropModule.Shape.ELLIPSE" in source From 57376d9bd07d3c2bafbb271590b5e040fe99ab6e Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 16:05:42 -0400 Subject: [PATCH 09/12] DL3 receipt: second hand-off, what is done and what remains Co-Authored-By: Claude Opus 5.5 --- .../DL3-output-provenance.md | 134 ++++++++++++++++++ 1 file changed, 134 insertions(+) diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md index df10d70cb..45efd5a90 100644 --- a/docs/refactor/consolidation-20261009/DL3-output-provenance.md +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -2,6 +2,17 @@ Branch `refactor/DL3-output-provenance`. Gap 6 of `D1-cellpose-n2v2-custom-functions.md` (PR #1230). +**Lands after DL1.** The storage part is PolyStore PR OpenHCSDev/PolyStore#41. PolyStore 0.6.3 is +taken by the arraybridge 0.5.0 migration, so #41 has to be released as **0.6.4**, which requires +`arraybridge>=0.5.0,<0.6`. OpenHCS main is on arraybridge 0.4.1 until +`refactor/DL1-device-and-memory-types` merges, so this branch cannot pin 0.6.4 before that. Until +then the submodule `external/PolyStore` points at the #41 commit (`8ee8ce4`) for local tests and +`pyproject.toml` still says `polystore>=0.6.3,<0.7`, which is wrong and must become `>=0.6.4`. + +**Current state: see "Second hand-off" at the end.** The sections between here and there are the +first agent's; where they describe the stored JSON as nested dicts with `kind` keys, the second +hand-off replaces them. + ## What is wrong (measured on `origin/main` at `1b5fe99f5`) - `openhcs_metadata.json` of an output dataset has no pipeline, function, parameter, source-hash @@ -326,3 +337,126 @@ Failing ids outside `known.fail` (5), with the believed cause (none investigated The remaining 2 of the 7 failures are the ids in `known.fail`. Fix the first two rows as step 0 of "What remains". + +## Second hand-off (2026-10-12): what changed, what remains + +Commit `3a55d1fe4` (pushed). Stopped at the context limit after items 1 and 2 of the continuation +brief; nothing else is started in the tree. + +### Done + +**Brief item 1: the record is nominal types** (`openhcs/core/run_record.py`, +`openhcs/core/function_identity.py`). + +| Type | Holds | +|---|---| +| `RunRecord` | `software`, `pipeline_code`, `global_config_code`, `configuration`, `steps`, `custom_sources` (text by sha256, each once) | +| `SoftwareEnvironment` | OpenHCS, Python, platform, `installed_distributions` (version by name) | +| `StepRecord` | `calls: StepCalls`, `configuration: tuple[ConfigValuesFromOwner, ...]` | +| `StepCalls` | name, pipeline position, step scope id, memory types, `assigned_gpu_devices: tuple[GpuDevice, ...]`, `functions` | +| `FunctionRecord` | `identity`, group, position, memory types, `parameters`, `declared_files` | +| `FunctionIdentity` / `CustomFunctionIdentity` | key, module, name, `defined_in` and `imports` (`ModuleLibraries`); the custom one adds `source_sha256`. The text is not copied into the identity: `source_texts()` reads it from the registered `CustomFunctionSourceNamespace` that owns the hash, and the record stores it once under `custom_sources` | +| `RecordedParameter` | name, value (Python expression, `None` when no declaration states it), `source: type[ParameterSource]` | +| `ParameterSource` family | `WrittenInPipeline` (`pipeline`), `FunctionDefault` (`function_default`; the function and version are the identity of the enclosing `FunctionRecord`), `SuppliedAtRunTime` (`run_time`). Each says how its value is written (`value_expression`) | +| `ConfigValuesFromOwner` | `owner: ConfigValueOwner(source, scope_id, type_name)`, `values` by field path. `source` is `type[ConfigValueSource]`: `SetOnObject` (`set_here`) or `InheritedFrom` (`inherited`) | +| `DeclaredFile` | parameter, `declared_as`, path, `content: HashedFileContent \| UnhashedFileContent` (PolyStore's family) | +| `DatasetRunRecords` | the two keys of the metadata document: `run_records` by id, `produced_files: Mapping[str, StoredProducedFile(run_record, producer: StreamProducerIdentity)]` | +| `ProducedFile` | what the reader returns for one file: `record`, `producer`, `.step -> StepCalls` | + +**One codec.** Encode is `python_introspect.to_jsonable`, decode is +`python_introspect.dataclass_from_mapping` (`external/python-introspect/src/python_introspect/ +jsonable.py`, `dataclass_projection.py`); `RunRecord.to_json` / `from_json` and +`DatasetRunRecords.in_document` only call them. No `to_record` method is left in OpenHCS. A class +of the two source families is stored as its registry key (`AutoRegisterMeta`, the codec's existing +rule for `type[Family]`). The identity family and the file-content family carry data, and the codec +has no nominal tag for a family *instance*: they are declared as unions of dataclasses and the +decoder picks the member whose declared fields match exactly (it rejects undeclared fields and +raises on an ambiguous match). `test_stored_json_reads_back_as_the_same_record` covers every type. +**Open point for the owner:** that union decode is by field set, not by name. If that is not +acceptable, the fix belongs in python-introspect (a registered family instance stored as key plus +fields), not in a second codec here. PolyStore's `FileContent.to_record` / `from_record` +(`hashed: bool` key) is now unused by OpenHCS and can be dropped from #41 when it is rebased. + +**Reader and capability return the types.** `RunRecord.for_dataset`, `RunRecord.for_output_file`, +`ProducedFile.step`. `openhcs_read_run_record`: `RunRecordEntry(record_id, steps: tuple[StepCalls, +...], record: RunRecord | None)`; the request flag is `include_full_record` (was +`include_sources`); the digest-and-`shortened_fields` form is gone. Contract fixture regenerated +(only that tool changed). + +**Size (part of brief item 5, measured on the two-step unit pipeline in the shared venv, 246 +distributions):** default response `steps` 2,277 bytes; full record 47,633 bytes (software 5,252, +pipeline_code 12,000, global_config_code 7,453, configuration 9,846, steps 12,984). Grouping +resolved values by owner took the pipeline configuration from 22.9 kB to 9.8 kB and each step's +from 12.3 kB to 5.3 kB. Not re-measured on the Cellpose run. + +**Brief item 2: the two suite failures.** +- `test_pycodify_formatters.py::test_function_step_round_trip_omits_injected_values_but_keeps_runtime_settings[False-...]`: + the first agent had changed what `clean_mode=False` means (the hook was in + `FunctionStepFormatter.format`). The test states the stored-document promise (every stored + field, no parameter the pipeline does not state) and it holds again: the hook is removed, and the + two tests that had been weakened for it (`test_authoring_roundtrip_boundaries.py`, + `test_source_round_trip.py`) are restored to main's text. The record's copy is its own + rendering, `PipelineDocumentCodec.render_with_every_parameter` (completes each step's pattern + with `with_every_parameter_written`, then renders unabridged). + `test_saved_pipeline_text_stays_short_and_the_record_copy_is_complete` asserts all three. +- `test_mcp_dev_client_terminal_input.py::test_long_terminal_command_reuses_original_parser_and_client`: + not this branch. It pumps a 4 kB+ command through a PTY child with a 15 s deadline. Alone on the + branch at load average 13: 13.4 s and 8.5 s (passed), then 19.4 s and 17.3 s (one failed). With + the child importing `origin/main`'s package under the same load: 15.07 s and 15.01 s, both + failed, twice. Left as is. +- `test_mcp_server.py` stdio tests: 5 s `session.initialize()` timeout under `-n 4`; `2 passed` + alone. + +Tests run here: `tests/unit/test_run_record.py`, `test_run_record_capability.py`, +`tests/integration/test_run_record_round_trip.py`, `test_mcp_tool_contract.py`, +`test_pycodify_formatters.py`, `test_source_round_trip.py`, +`test_authoring_roundtrip_boundaries.py`: all pass. Full suite, guardrails, smoke and parity were +not run after these changes. + +### What remains, in order (numbers are the continuation brief's) + +3. **Shrink the two oversized classes; ratchet to no increase.** Not started. Expected to be + cleared by item 1 but unverified: `StringSubscript`, `AnyAnnotation`, `unmodeled_record_shape`. + Still present: `GodClassExcess PipelineCompiler +7` (the seven lines are the + `RunDeclaration.before_step_resolution` call, `with_resolved_steps`, and `run_declaration=` at + three bundle constructions), `OpenHCSMetadataTarget +24` (`saved_path_producers` and the + additions in `finalize_completed_dataset`), the boolean chains in + `StepExecutionObservation.is_empty` and `AtomicMetadataWriter.reconcile_completed_dataset`, + and two absence probes (`ModuleExportedCallableFormatter.can_format` in + `pycodify_formatters.py`; one in `virtual_workspace_metadata.py`). Notes from reading the + compiler: the declaration needs the submitted documents before `_resolve_pipeline_once` + replaces the steps in place, and the per-step provenance that only + `ResolvedPipelineDefinition` holds; nothing on the bundle or the plans carries that + provenance, so the orchestrator (the single caller, `orchestrator.py:739`) cannot build it + after the fact without the compiler handing something over. The two early returns in + `compile_pipelines` build the same empty bundle twice. `saved_path_producers` belongs with + `ProducedFiles` in `run_record.py`. Then merge `origin/main` and run `guardrails_local.sh`. +4. **Defects in what is recorded**, failing test first, none started: + (3) declared path left at its default: `CompiledFunctionInvocation.from_normalized_item` + (`function_patterns.py:~926`) replaces the kwargs with `resolve_declared_paths`' result, so the + invocation no longer knows which paths the pipeline wrote; it has to keep that (the stated + value and its source per declared path) so `_parameters` can say `function_default` and + `declared_files` holds the resolved path. (4) `enabled` has no parameters entry. (5) device on + a host run: `StepCalls.assigned_gpu_devices` is still an empty tuple; state it positively from + `FrameworkDeviceAssignment` and note the DL1 dependency. (6) enum defaults of custom functions + render with the mangled path. (7) libraries: direct imports only; add the requirements + closure, each list named. (8) plate-level outputs have no `produced_files` entry. (10) no + saved global config: `RunDeclaration.before_step_resolution` still falls back to + `GlobalPipelineConfig()`. (12) `changed_during_run`: still in the design text, not + implemented. +5. **Reader**: a public "is this a dataset root / which dataset contains this file" answer + through the dataset access owner; `DatasetRunRecords.of_dataset` and `_dataset_root_of` still + read the metadata file directly, and `RunRecordTarget.at` probes for the file, so the + capability's read of the document above a file is not checked by the agent path policy. Size: + re-measure on the Cellpose record. +6. **PolyStore 0.6.4**: not started. Rebase #41 onto PolyStore main, version 0.6.4, drop the + unused `to_record` / `from_record` if the owner agrees, suite in a fresh venv, merge, annotated + tag, publish. Then this branch's pin (see the top of this document). +7. **Acceptance re-run** (Cellpose on CPU, three BBBC039 fields), then delete `.claude/venv-cpu` + (2.4 GB) and `.claude/scratch/` (1.2 GB): both still present in the worktree. The acceptance + scripts under `.claude/scratch/acceptance/` read the old dict form and need the typed reader. +8. Gates (the landing agent's): guardrails `rc=0`, `verify_branch.sh`, smoke, parity. The PR is + still a draft; its body does not yet say "lands after DL1". +9. Outside this branch, unchanged: ROI zip metadata and `segmentation_summary.txt` embed absolute + source paths; zip timestamps. + From 18c4d56ae869cb5a4039a98da3366a6f0ad068b5 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 16:28:11 -0400 Subject: [PATCH 10/12] DL3: stored family members name their class (python-introspect 0.3.0) The function identity and the file content of a record are registered families (defined_by: installed_module / source_text; content_kind: hashed / not_hashed) and the record's fields are declared by the family, not by a union. The codec writes {registry key: fields} and reads the class from the key; it no longer picks a member by the fields present. UiLiveMeasurementEntryState imports its preview type at run time: the codec now reads declared field types when encoding. Co-Authored-By: Claude Opus 5.5 --- external/PolyStore | 2 +- external/python-introspect | 2 +- openhcs/agent/dto/ui_bridge.py | 6 +----- openhcs/core/function_identity.py | 13 ++++++++++++- openhcs/core/run_record.py | 12 +++++++----- pyproject.toml | 2 +- tests/unit/test_run_record.py | 8 ++++++++ tests/unit/test_run_record_capability.py | 6 +++--- 8 files changed, 34 insertions(+), 17 deletions(-) diff --git a/external/PolyStore b/external/PolyStore index 8ee8ce44a..5301d35b4 160000 --- a/external/PolyStore +++ b/external/PolyStore @@ -1 +1 @@ -Subproject commit 8ee8ce44a1dfc3dc59433584d1d19952231f5815 +Subproject commit 5301d35b4fe6037a46f4186f7d33537e746294bb diff --git a/external/python-introspect b/external/python-introspect index a88fe56f7..a9428c3b5 160000 --- a/external/python-introspect +++ b/external/python-introspect @@ -1 +1 @@ -Subproject commit a88fe56f78b523ad5c791e3f868bb35d7d6b3f12 +Subproject commit a9428c3b5a5f7c4091049ac977dff8fd48f76b2e diff --git a/openhcs/agent/dto/ui_bridge.py b/openhcs/agent/dto/ui_bridge.py index 75dcb9edd..2badb184c 100644 --- a/openhcs/agent/dto/ui_bridge.py +++ b/openhcs/agent/dto/ui_bridge.py @@ -4,7 +4,6 @@ from dataclasses import dataclass, field from enum import Enum -from typing import TYPE_CHECKING from python_introspect import ( JsonObject, ) @@ -35,10 +34,7 @@ UiMutationRequestTokenCarrier, UiStateSurfaceEnvelope, ) -if TYPE_CHECKING: - from openhcs.core.progress.live_measurements import ( - LiveMeasurementTablePreview, - ) +from openhcs.core.progress.live_measurements import LiveMeasurementTablePreview def _dataset_workflow_enum() -> type[Enum]: diff --git a/openhcs/core/function_identity.py b/openhcs/core/function_identity.py index 89d4f949a..abbb5d2ab 100644 --- a/openhcs/core/function_identity.py +++ b/openhcs/core/function_identity.py @@ -16,6 +16,9 @@ from functools import lru_cache from pathlib import Path from types import MappingProxyType +from typing import ClassVar + +from metaclass_registry import AutoRegisterMeta from openhcs.runtime.environment_provenance import InstalledDistributionVersion @@ -84,15 +87,21 @@ def imported_by_module(cls, module_name: str) -> "ModuleLibraries": @dataclass(frozen=True, slots=True) -class FunctionIdentity: +class FunctionIdentity(metaclass=AutoRegisterMeta): """A function exported by an installed module. + The identities are registered under ``defined_by``; a stored identity names + its class by that key. + ``key`` is the registry key (``registry:name``) or the import path (``python:module:name``); ``defined_in`` is the distribution and version of the module that defines it; ``imports`` are the distributions that module imports. """ + __registry_key__ = "defined_by" + defined_by: ClassVar[str] = "installed_module" + key: str module: str name: str @@ -122,6 +131,8 @@ def source_texts(self) -> Mapping[str, str]: class CustomFunctionIdentity(FunctionIdentity): """A function defined by user-written source text, identified by that text's hash.""" + defined_by: ClassVar[str] = "source_text" + source_sha256: str @classmethod diff --git a/openhcs/core/run_record.py b/openhcs/core/run_record.py index 04b40ccd2..f59ec0e93 100644 --- a/openhcs/core/run_record.py +++ b/openhcs/core/run_record.py @@ -19,7 +19,9 @@ The stored form is the JSON ``python_introspect.to_jsonable`` writes for these dataclasses and ``python_introspect.dataclass_from_mapping`` reads back; there is no other encoder or decoder. A class of a registered family (a parameter's -or a configuration value's source) is stored as its registry key. +or a configuration value's source) is stored as its registry key. A member of +a registered family that carries data (a function identity, a file's content) +is stored as ``{registry key: fields}``, so the stored form names its class. """ from __future__ import annotations @@ -36,11 +38,11 @@ import pycodify from metaclass_registry import AutoRegisterMeta -from polystore.file_content import HashedFileContent, UnhashedFileContent +from polystore.file_content import FileContent from polystore.streaming.identity import StreamProducerIdentity from python_introspect import JsonObject, JsonValue, dataclass_from_mapping, to_jsonable -from openhcs.core.function_identity import CustomFunctionIdentity, FunctionIdentity +from openhcs.core.function_identity import FunctionIdentity from openhcs.core.vfs_protocol import FileManagerLike @@ -314,14 +316,14 @@ class DeclaredFile: parameter: str declared_as: str path: str - content: HashedFileContent | UnhashedFileContent + content: FileContent @dataclass(frozen=True, slots=True) class FunctionRecord: """One function call of a step: which function, with which values and files.""" - identity: FunctionIdentity | CustomFunctionIdentity + identity: FunctionIdentity group: str position: int input_memory_type: str | None diff --git a/pyproject.toml b/pyproject.toml index 4ff06b44a..00c35acf1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -105,7 +105,7 @@ dependencies = [ "zmqruntime>=0.6.4,<0.7", "pycodify>=0.2.1,<0.3", "objectstate>=3.0.4,<4", - "python-introspect>=0.2.8,<0.3", + "python-introspect>=0.3.0,<0.4", "metaclass-registry>=0.4.0,<0.5", "arraybridge>=0.4.1,<0.5", "polystore>=0.6.3,<0.7", diff --git a/tests/unit/test_run_record.py b/tests/unit/test_run_record.py index 9e0a8ba62..ced23f453 100644 --- a/tests/unit/test_run_record.py +++ b/tests/unit/test_run_record.py @@ -228,6 +228,14 @@ def test_stored_json_reads_back_as_the_same_record(tmp_path, register): assert RunRecord.from_json(stored).record_id == record.record_id (norm_parameter, *_rest) = stored["steps"][0]["calls"]["functions"][0]["parameters"] assert norm_parameter == {"name": "stack", "value": None, "source": "run_time"} + # Each family member names its class by its registry key, not by its fields. + library, custom = (step["calls"]["functions"][0] for step in stored["steps"]) + assert list(library["identity"]) == ["installed_module"] + assert list(custom["identity"]) == ["source_text"] + assert [list(declared["content"]) for declared in custom["declared_files"]] == [ + ["hashed"], + ["not_hashed"], + ] def test_saved_pipeline_text_stays_short_and_the_record_copy_is_complete(tmp_path): diff --git a/tests/unit/test_run_record_capability.py b/tests/unit/test_run_record_capability.py index ad0459c44..4848dc7d3 100644 --- a/tests/unit/test_run_record_capability.py +++ b/tests/unit/test_run_record_capability.py @@ -156,15 +156,15 @@ def test_dataset_root_returns_what_each_step_ran_by_default(dataset, read): {"record_id": RECORD_ID, "steps": [as_json(STEP)], "record": None} ] (function,) = result["records"][0]["steps"][0]["functions"] - assert function["identity"]["source_sha256"] == CUSTOM_SOURCE_SHA256 + # A family member is stored under the key its class is registered with. + assert function["identity"]["source_text"]["source_sha256"] == CUSTOM_SOURCE_SHA256 assert function["parameters"][1] == { "name": "weights", "value": "'weights.bin'", "source": "pipeline", } assert function["declared_files"][0]["content"] == { - "sha256": "0" * 64, - "size_bytes": 7, + "hashed": {"sha256": "0" * 64, "size_bytes": 7} } From 1a1c8c1ebe8ae8021fe17f463b7471388add4e44 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 17:01:14 -0400 Subject: [PATCH 11/12] DL3: the run record's owner stores it and states the producers; the two oversized classes are back to main's size - ProducedFiles.stated_by_step replaces OpenHCSMetadataTarget.saved_path_producers. - RunRecord.store_with_saved_files and RunRecordWriter (an AtomicMetadataWriter) store the record in its own locked update right after the dataset's metadata is published. OpenHCSMetadataTarget.finalize_completed_dataset and AtomicMetadataWriter.reconcile_completed_dataset are main's again. - The compiler builds the bundle of an empty compilation through one factory, CompiledExecutionBundle.with_nothing_to_run. - StepExecutionObservation.is_empty is derived from the dataclass fields. - The attribute that holds a custom source namespace is named in one place. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/compiled_execution.py | 23 ++++ openhcs/core/orchestrator/bundle_execution.py | 6 +- openhcs/core/pipeline/compiler.py | 26 ++--- openhcs/core/run_record.py | 110 ++++++++++++++---- openhcs/core/steps/abstract.py | 11 +- openhcs/core/steps/function_outputs.py | 76 +----------- openhcs/core/steps/function_step.py | 8 +- openhcs/core/virtual_workspace_metadata.py | 21 +--- .../custom_functions/source_namespace.py | 9 +- ...rimary_segmentation_diagnostics_journey.py | 3 +- .../unit/test_artifact_publication_journey.py | 4 +- tests/unit/test_function_outputs.py | 28 ++--- .../unit/test_named_checkpoint_publication.py | 6 +- tests/unit/test_orchestrator_lane_planning.py | 2 +- tests/unit/test_persisted_image_metadata.py | 4 +- .../test_persisted_roi_only_publication.py | 9 +- tests/unit/test_run_record.py | 8 +- 17 files changed, 168 insertions(+), 186 deletions(-) diff --git a/openhcs/core/compiled_execution.py b/openhcs/core/compiled_execution.py index 73970e259..c69d560a0 100644 --- a/openhcs/core/compiled_execution.py +++ b/openhcs/core/compiled_execution.py @@ -159,6 +159,29 @@ def requires_parent_runtime_observation(self) -> bool: for plan in context.step_plans.values() ) + @classmethod + def with_nothing_to_run( + cls, + pipeline_definition: Sequence[AbstractStep], + effective_config: GlobalPipelineConfig, + *, + server_mode: bool, + run_declaration: RunDeclaration, + ) -> "CompiledExecutionBundle": + """The bundle of a compilation that selected no axis value or no enabled step.""" + + return cls.from_runtime_contexts( + pipeline_definition=pipeline_definition, + runtime_contexts={}, + worker_assignments={}, + runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( + effective_config, + compiled_contexts={}, + server_mode=server_mode, + ), + run_declaration=run_declaration, + ) + @classmethod def from_runtime_contexts( cls, diff --git a/openhcs/core/orchestrator/bundle_execution.py b/openhcs/core/orchestrator/bundle_execution.py index 12bed2121..e95ae5c71 100644 --- a/openhcs/core/orchestrator/bundle_execution.py +++ b/openhcs/core/orchestrator/bundle_execution.py @@ -84,7 +84,7 @@ from openhcs.core.steps.function_artifact_materialization import ( ArtifactMaterializationTargetPlan, ) -from openhcs.core.run_record import RunRecord +from openhcs.core.run_record import ProducedFiles, RunRecord from openhcs.core.steps.function_outputs import ( OpenHCSMetadataTarget, ) @@ -348,8 +348,8 @@ def execute_bundle_request( executing_contexts ), ), - run_record=run_record, ) + run_record.store_with_saved_files(executing_contexts) if execution_results.is_success(): viewer_states_by_port = settle_viewer_state( visualizers, @@ -654,7 +654,7 @@ def execute_dataset_scoped_steps( materialized_locations_by_address=MappingProxyType({}), source_projection_entries_by_target=projection_entries, producer_by_saved_path=( - OpenHCSMetadataTarget.saved_path_producers( + ProducedFiles.stated_by_step( owner_context, owner_plan.require_function_execution_ready(), artifact_materializations=materializations, diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index a15af1493..ed36ed661 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -1540,15 +1540,10 @@ def compile_pipelines( ) if not axis_values_to_process: logger.warning("No axis values found to process based on filter.") - return CompiledExecutionBundle.from_runtime_contexts( - pipeline_definition=pipeline_definition, - runtime_contexts={}, - worker_assignments={}, - runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( - effective_config, - compiled_contexts={}, - server_mode=is_zmq_execution, - ), + return CompiledExecutionBundle.with_nothing_to_run( + pipeline_definition, + effective_config, + server_mode=is_zmq_execution, run_declaration=run_declaration, ) @@ -1564,15 +1559,10 @@ def compile_pipelines( logger.warning( "All steps were disabled. Pipeline is empty after filtering." ) - return CompiledExecutionBundle.from_runtime_contexts( - pipeline_definition=pipeline_definition, - runtime_contexts={}, - worker_assignments={}, - runtime_environment=CompiledRuntimeEnvironmentPlan.from_global_config( - effective_config, - compiled_contexts={}, - server_mode=is_zmq_execution, - ), + return CompiledExecutionBundle.with_nothing_to_run( + pipeline_definition, + effective_config, + server_mode=is_zmq_execution, run_declaration=run_declaration, ) path_resolver = CompilationPathResolver( diff --git a/openhcs/core/run_record.py b/openhcs/core/run_record.py index f59ec0e93..9dc21b29e 100644 --- a/openhcs/core/run_record.py +++ b/openhcs/core/run_record.py @@ -44,9 +44,15 @@ from openhcs.core.function_identity import FunctionIdentity +from openhcs.core.steps.abstract import StepExecutionObservation from openhcs.core.vfs_protocol import FileManagerLike +from openhcs.core.virtual_workspace_metadata import METADATA_CONFIG, AtomicMetadataWriter if TYPE_CHECKING: + from openhcs.core.context.processing_context import ExecutingContext + from openhcs.core.steps.function_artifact_materialization import ( + MaterializedRuntimeArtifact, + ) from openhcs.core.compiled_execution import CompiledExecutionBundle from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import GlobalPipelineConfig @@ -250,7 +256,7 @@ class RunDeclaration: def before_step_resolution( cls, orchestrator: "PipelineOrchestrator", - pipeline_definition: Sequence["AbstractStep"], + pipeline_definition: Sequence[AbstractStep], ) -> "RunDeclaration": """Render the documents as submitted, before the compiler resolves the steps in place.""" from objectstate import ObjectState, ObjectStateRegistry @@ -519,6 +525,29 @@ def record_id(self) -> str: json.dumps(self.to_json(), sort_keys=True, separators=(",", ":")).encode("utf-8") ).hexdigest() + def store_with_saved_files(self, contexts: Mapping[str, "ExecutingContext"]) -> None: + """Write this record into the document of each output dataset the run saved files under. + + Runs in the parent process once the run has ended, also after a failure, + right after the dataset's own metadata is published. Each document + receives the producer of every file saved under its dataset root. + """ + from openhcs.core.steps.function_outputs import OpenHCSMetadataTarget + + saved = ProducedFiles( + StepExecutionObservation.combine( + context.completed_step_outputs for context in contexts.values() + ).producer_by_saved_path + ) + declared_dataset_roots = { + Path(target.dataset_root) + for context in contexts.values() + for plan in context.step_plans.values() + for target in OpenHCSMetadataTarget.for_plan(plan) + } + for dataset_root, files in saved.by_dataset_root(declared_dataset_roots).items(): + RunRecordWriter().store(dataset_root, files, self) + @classmethod def at_run_start( cls, @@ -736,6 +765,42 @@ class ProducedFiles: producer_by_path: Mapping[Path, StreamProducerIdentity] + @staticmethod + def stated_by_step( + context: ExecutingContext, + plan: CompiledStepPlan, + *, + artifact_materializations: tuple[MaterializedRuntimeArtifact, ...], + ) -> Mapping[Path, StreamProducerIdentity]: + """The producer identity of each file this step saved under a metadata document. + + Both statements are the ones the viewer stream uses: a main-flow record + carries its own identity; a saved artifact's comes from its output plan. + """ + from openhcs.core.steps.function_outputs import OpenHCSMetadataTarget + + producers: dict[Path, StreamProducerIdentity] = {} + for target in OpenHCSMetadataTarget.for_execution( + context, plan, artifact_materializations=artifact_materializations + ): + if not OpenHCSMetadataTarget.hosts_metadata_document(target.backend): + continue + for record in target.produced_records(context, plan): + producers[Path(record.path_under(target.output_dir))] = ( + record.producer_identity + ) + artifact_target = plan.runtime_artifact_materialization + if artifact_target.has_persistent_target: + backend = artifact_target.require_persistent_backend() + if OpenHCSMetadataTarget.hosts_metadata_document(backend): + for saved in artifact_materializations: + producer = plan.producer_identity_for_artifact( + saved.materialization.output_plan + ) + for output in saved.outputs_for_backend(backend): + producers[Path(output.path)] = producer + return MappingProxyType(producers) + def by_dataset_root( self, dataset_roots: Iterable[Path] ) -> Mapping[Path, "ProducedFiles"]: @@ -760,28 +825,29 @@ def by_dataset_root( {root: ProducedFiles(MappingProxyType(files)) for root, files in by_root.items()} ) - def write_into( - self, - document: dict[str, JsonValue], - *, - dataset_root: Path, - record: RunRecord, - ) -> None: - """Store the record with the files this run saved under this dataset root. - A document the run saved nothing under is left as it is. A record no - file refers to any more is dropped. +class RunRecordWriter(AtomicMetadataWriter): + """Stores a run's record in a dataset's metadata document, under that document's lock.""" + + def store(self, dataset_root: Path, files: ProducedFiles, record: RunRecord) -> None: + """Add the record with the files the run saved under ``dataset_root``. + + A record no file refers to any more is dropped in the same update. """ - if not self.producer_by_path: - return - document.update( - to_jsonable( - DatasetRunRecords.in_document(document).with_run( - record, - { - path.relative_to(dataset_root).as_posix(): producer - for path, producer in self.producer_by_path.items() - }, + + def update(document: dict[str, JsonValue] | None) -> dict[str, JsonValue]: + document = self._ensure_subdirectories_structure(document) + document.update( + to_jsonable( + DatasetRunRecords.in_document(document).with_run( + record, + { + path.relative_to(dataset_root).as_posix(): producer + for path, producer in files.producer_by_path.items() + }, + ) ) ) - ) + return document + + self._execute_update(METADATA_CONFIG.metadata_path(dataset_root), update) diff --git a/openhcs/core/steps/abstract.py b/openhcs/core/steps/abstract.py index d1a4104f1..3c88585d7 100644 --- a/openhcs/core/steps/abstract.py +++ b/openhcs/core/steps/abstract.py @@ -10,7 +10,7 @@ import inspect from pathlib import Path from collections.abc import Mapping -from dataclasses import dataclass, field, is_dataclass +from dataclasses import dataclass, field, fields, is_dataclass from types import MappingProxyType from typing import TYPE_CHECKING, Iterable, get_type_hints @@ -99,14 +99,7 @@ def __post_init__(self) -> None: @property def is_empty(self) -> bool: """Whether this completed output carrier has no facts to transport.""" - return not ( - self.materialized_locations_by_address - or self.runtime_export_paths - or self.hook_observations - or self.sample_numbers_by_export_path - or self.source_projection_entries_by_target - or self.producer_by_saved_path - ) + return not any(getattr(self, carried.name) for carried in fields(self)) @classmethod def empty(cls) -> "StepExecutionObservation": diff --git a/openhcs/core/steps/function_outputs.py b/openhcs/core/steps/function_outputs.py index ad230b184..a2af0c609 100644 --- a/openhcs/core/steps/function_outputs.py +++ b/openhcs/core/steps/function_outputs.py @@ -56,7 +56,7 @@ SourceProjection, SourceProjectionMetadataSerializer, ) -from openhcs.core.run_record import ProducedFiles, RunRecord +from openhcs.core.run_record import ProducedFiles from openhcs.core.steps.abstract import StepExecutionObservation from openhcs.core.steps.function_artifact_materialization import ( ArtifactMaterializationTargetPlan, @@ -123,7 +123,7 @@ def finalize_function_step_outputs( projection_entries = OpenHCSMetadataTarget.observe_for_step( context, plan, artifact_materializations=materializations ) - producers = OpenHCSMetadataTarget.saved_path_producers( + producers = ProducedFiles.stated_by_step( context, plan, artifact_materializations=materializations ) else: @@ -156,7 +156,7 @@ def finalize_function_step_outputs( ) producers = _profile_finalization_phase( "finalize_saved_path_producers", - lambda: OpenHCSMetadataTarget.saved_path_producers( + lambda: ProducedFiles.stated_by_step( context, plan, artifact_materializations=materializations ), plan, @@ -1070,80 +1070,22 @@ def observe_for_step( observations[replace(target, artifact_materializations=())] = entries return MappingProxyType(observations) - @classmethod - def saved_path_producers( - cls, - context: ExecutingContext, - plan: CompiledStepPlan, - *, - artifact_materializations: tuple[MaterializedRuntimeArtifact, ...], - ) -> Mapping[Path, StreamProducerIdentity]: - """The producer identity of each file this step saved under a metadata document. - - Both statements are the ones the viewer stream uses: a main-flow record - carries its own identity; a saved artifact's comes from its output plan. - """ - producers: dict[Path, StreamProducerIdentity] = {} - for target in cls.for_execution( - context, plan, artifact_materializations=artifact_materializations - ): - if not cls.hosts_metadata_document(target.backend): - continue - for record in target.produced_records(context, plan): - producers[Path(record.path_under(target.output_dir))] = ( - record.producer_identity - ) - artifact_target = plan.runtime_artifact_materialization - if artifact_target.has_persistent_target: - backend = artifact_target.require_persistent_backend() - if cls.hosts_metadata_document(backend): - for saved in artifact_materializations: - producer = plan.producer_identity_for_artifact( - saved.materialization.output_plan - ) - for output in saved.outputs_for_backend(backend): - producers[Path(output.path)] = producer - return MappingProxyType(producers) - @classmethod def finalize_completed_dataset( cls, compiled_contexts: Mapping[str, ExecutingContext], *, runtime_observations: Iterable[RuntimeExecutionObservation] = (), - run_record: RunRecord, ) -> None: - """Publish completed or partial saved outputs once at the receiving boundary. - - Each output dataset's document also receives the run's record and the - producer of every file the run saved under that dataset root. - """ + """Publish completed or partial saved outputs once at the receiving boundary.""" - runtime_observations = tuple(runtime_observations) target_contexts: dict[OpenHCSMetadataTarget, ExecutingContext] = {} - declared_dataset_roots: set[Path] = set() for context in compiled_contexts.values(): for plan in context.step_plans.values(): - targets = cls.for_plan(plan) - declared_dataset_roots.update( - Path(target.dataset_root) for target in targets - ) if not plan.create_openhcs_metadata: continue - for target in targets: + for target in cls.for_plan(plan): target_contexts.setdefault(target, context) - produced_files_by_document = { - METADATA_CONFIG.metadata_path(dataset_root): produced_files - for dataset_root, produced_files in ProducedFiles( - StepExecutionObservation.combine( - context_observation.outputs - for observation in runtime_observations - for context_observation in observation.contexts - ).producer_by_saved_path - ) - .by_dataset_root(declared_dataset_roots) - .items() - } produced_by_document: dict[Path, dict[OpenHCSMetadataTarget, list[VirtualWorkspaceSourceProjectionEntries]]] = {} for observation in runtime_observations: @@ -1163,9 +1105,7 @@ def finalize_completed_dataset( targets_by_document.setdefault( METADATA_CONFIG.metadata_path(owner.dataset_root), {} )[owner] = context - for metadata_path in dict.fromkeys( - (*targets_by_document, *produced_by_document, *produced_files_by_document) - ): + for metadata_path in dict.fromkeys((*targets_by_document, *produced_by_document)): AtomicMetadataWriter().reconcile_completed_dataset( metadata_path, targets_by_document.get(metadata_path, {}), @@ -1173,10 +1113,6 @@ def finalize_completed_dataset( target: VirtualWorkspaceSourceProjectionEntries.combine(entries) for target, entries in produced_by_document.get(metadata_path, {}).items() }, - produced_files=produced_files_by_document.get( - metadata_path, ProducedFiles(MappingProxyType({})) - ), - run_record=run_record, ) diff --git a/openhcs/core/steps/function_step.py b/openhcs/core/steps/function_step.py index a417fcb17..056ec68e5 100644 --- a/openhcs/core/steps/function_step.py +++ b/openhcs/core/steps/function_step.py @@ -60,9 +60,11 @@ def with_function_spec(self, func: FunctionSpec) -> "FunctionStep": def configuration_parameters(self) -> dict[str, object]: """The function pattern is what the step computes, not its configuration.""" - parameters = self.declaration_parameters() - del parameters["func"] - return parameters + return { + name: value + for name, value in self.declaration_parameters().items() + if value is not self.func + } def occurrence_authorities(self) -> tuple[object, ...]: """Prefer callable-pattern authority before nominal step authority.""" diff --git a/openhcs/core/virtual_workspace_metadata.py b/openhcs/core/virtual_workspace_metadata.py index 8a270e42c..036b406f7 100644 --- a/openhcs/core/virtual_workspace_metadata.py +++ b/openhcs/core/virtual_workspace_metadata.py @@ -38,7 +38,6 @@ from openhcs.core.axes import Axis, AxisFamily, ColourAxis if TYPE_CHECKING: - from openhcs.core.run_record import ProducedFiles, RunRecord from openhcs.core.context.processing_context import ExecutingContext from openhcs.core.steps.function_outputs import OpenHCSMetadataTarget @@ -224,22 +223,11 @@ def reconcile_completed_dataset( produced_entries_by_target: Mapping[ OpenHCSMetadataTarget, VirtualWorkspaceSourceProjectionEntries ] = MappingProxyType({}), - produced_files: ProducedFiles, - run_record: RunRecord, ) -> None: - """Admit and reconcile all targets from one locked plate document. - - The same transaction stores the run's record with the files the run - saved under this document's dataset root. - """ - if ( - not produced_entries_by_target - and not produced_files.producer_by_path - and not Path(metadata_path).is_file() - and not any( + """Admit and reconcile all targets from one locked plate document.""" + if not produced_entries_by_target and not Path(metadata_path).is_file() and not any( target.contains_outputs(context) - for target, context in target_contexts.items() - ) + for target, context in target_contexts.items() ): return @@ -280,9 +268,6 @@ def update(data): VirtualWorkspaceSourceProjectionEntries(MappingProxyType({})), ), ) - produced_files.write_into( - data, dataset_root=Path(metadata_path).parent, record=run_record - ) return data self._execute_update(metadata_path, update) diff --git a/openhcs/processing/custom_functions/source_namespace.py b/openhcs/processing/custom_functions/source_namespace.py index d3b72980c..d5a46c821 100644 --- a/openhcs/processing/custom_functions/source_namespace.py +++ b/openhcs/processing/custom_functions/source_namespace.py @@ -98,12 +98,17 @@ def __post_init__(self) -> None: def bound_to(cls, declaration: Callable) -> "CustomFunctionSourceNamespace": """Return the namespace ``bind`` attached to this declaration.""" revision = vars(declaration)[FunctionContractAttribute.declaration_revision] - return vars(declaration)[f"__openhcs_source_{revision}"] + return vars(declaration)[cls.attribute_of_revision(revision)] + + @staticmethod + def attribute_of_revision(content_sha256: str) -> str: + """The attribute of a callable that holds the namespace of one source revision.""" + return f"__openhcs_source_{content_sha256}" @property def export_attribute(self) -> str: """Derive the revision-specific namespace address on its callable.""" - return f"__openhcs_source_{self.source.content_sha256}" + return self.attribute_of_revision(self.source.content_sha256) def member_attribute(self, relative_qualname: str) -> str: """Encode one lexical name, without colliding with namespace members.""" diff --git a/tests/integration/test_primary_segmentation_diagnostics_journey.py b/tests/integration/test_primary_segmentation_diagnostics_journey.py index c5ff60ab5..c70e45d2e 100644 --- a/tests/integration/test_primary_segmentation_diagnostics_journey.py +++ b/tests/integration/test_primary_segmentation_diagnostics_journey.py @@ -14,7 +14,6 @@ from openhcs.core.axes import Ungrouped from openhcs.domains.microscopy.axes import Microscopy from openhcs.core.dataset_sources.source_bindings_source import SourceBindingsSource -from tests.run_records import minimal_run_record def _field(empty=False): @@ -365,7 +364,7 @@ def step(func, name, kwargs): primary_plan = context.step_plans[0] assert primary_plan.write_backend == "memory" assert primary_plan.materialized_output is None - OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts) metadata_path = METADATA_CONFIG.metadata_path(primary_plan.output_dataset_root) metadata_document = json.loads(metadata_path.read_text()) persisted_source_provenance = SourceImageProvenance.from_mapping( diff --git a/tests/unit/test_artifact_publication_journey.py b/tests/unit/test_artifact_publication_journey.py index 5771e2fc5..d7e0785f3 100644 --- a/tests/unit/test_artifact_publication_journey.py +++ b/tests/unit/test_artifact_publication_journey.py @@ -57,7 +57,6 @@ from openhcs.core.pipeline.function_contracts import ( FullStackLabels, ) -from tests.run_records import minimal_run_record @artifact_outputs(ArtifactSpec.output("objects", ObjectLabelsArtifactType)) @@ -178,8 +177,7 @@ def test_converted_checkpoint_has_typed_address_on_final_reconciliation(tmp_path for index, step in enumerate(steps) )) OpenHCSMetadataTarget.finalize_completed_dataset( - contexts, runtime_observations=(observations,), - run_record=minimal_run_record(), + contexts, runtime_observations=(observations,) ) saved = tmp_path / "source_out" / "converted" files = tuple(saved.glob("*.tif")) diff --git a/tests/unit/test_function_outputs.py b/tests/unit/test_function_outputs.py index 5f560f958..274cb2046 100644 --- a/tests/unit/test_function_outputs.py +++ b/tests/unit/test_function_outputs.py @@ -74,6 +74,7 @@ RuntimeArtifactMaterialization, ) from openhcs.core.steps.function_artifact_materialization import ArtifactMaterializationTargetPlan +from openhcs.core.run_record import ProducedFiles from openhcs.core.steps.function_outputs import ( MaterializedImageOutputWriter, MemoryOutputWriter, @@ -112,7 +113,6 @@ from openhcs.core.streaming_config_declarations import NapariViewer from openhcs.core.payload_axes import PayloadAxes from openhcs.core.config import IncludePartitions -from tests.run_records import minimal_run_record, no_produced_files def _publish_saved_step(context, plan, *, artifact_materializations=()): @@ -131,7 +131,6 @@ def _publish_saved_step(context, plan, *, artifact_materializations=()): metadata_path, {target: context for target in entries if target.create_openhcs_metadata}, produced_entries_by_target=entries, - produced_files=no_produced_files(), run_record=minimal_run_record(), ) @@ -171,7 +170,7 @@ def test_function_step_metadata_follows_runtime_artifact_persistence( "runtime_artifacts", ), (OpenHCSMetadataTarget, "observe_for_step", "metadata"), - (OpenHCSMetadataTarget, "saved_path_producers", "producers"), + (ProducedFiles, "stated_by_step", "producers"), ): monkeypatch.setattr( authority, @@ -243,8 +242,7 @@ def test_metadata_document_targets_follow_backend_declaration( lambda _self, metadata_path, *_args, **_kwargs: documents.append(metadata_path), ) OpenHCSMetadataTarget.finalize_completed_dataset( - {"A01": SimpleNamespace(step_plans={0: plan})}, - run_record=minimal_run_record(), + {"A01": SimpleNamespace(step_plans={0: plan})} ) assert documents == ( [METADATA_CONFIG.metadata_path(root)] if expected_targets else [] @@ -1709,7 +1707,7 @@ def test_produced_projection_metadata_persists_typed_collapsed_semantics( Backend.MEMORY.value, ) _publish_saved_step(context, plan) - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) subdirectory = json.loads( (dataset_root / "openhcs_metadata.json").read_text(encoding="utf-8") @@ -2184,7 +2182,7 @@ def reject_generated_path_parse(_filename): # must supply the same addresses, calibration and exact saved coverage. for record in records: filemanager.delete(record.output_path, Backend.MEMORY.value) - OpenHCSMetadataTarget.finalize_completed_dataset({well: context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({well: context}) subdirectory = json.loads((dataset_root / "openhcs_metadata.json").read_text())[ FIELDS.SUBDIRECTORIES ]["images"] @@ -2205,14 +2203,14 @@ def reject_generated_path_parse(_filename): unregistered = output_dir / "unowned_s001_w1_z001_t001.tif" tifffile.imwrite(unregistered, pixels) with pytest.raises(MetadataWriteError, match="lack typed produced addresses"): - OpenHCSMetadataTarget.finalize_completed_dataset({well: context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({well: context}) assert metadata_path.read_bytes() == previous_metadata unregistered.unlink() # This fixture owns the synthetic file. if len(paths) > 1: removed = paths[0] (dataset_root / removed).unlink() - OpenHCSMetadataTarget.finalize_completed_dataset({well: context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({well: context}) reconciled = json.loads(metadata_path.read_text())[FIELDS.SUBDIRECTORIES][ "images" ] @@ -2329,12 +2327,10 @@ def test_completed_plate_metadata_includes_outputs_written_after_owner_axis( dataset_root / "openhcs_metadata.json", {target: owner_context for target in combined_facts}, produced_entries_by_target=combined_facts, - produced_files=no_produced_files(), run_record=minimal_run_record(), ) OpenHCSMetadataTarget.finalize_completed_dataset( - {"A01": owner_context, "B03": follower_context}, - run_record=minimal_run_record(), + {"A01": owner_context, "B03": follower_context} ) metadata = json.loads( @@ -2361,7 +2357,7 @@ def test_completed_plate_metadata_skips_unmaterialized_output_target(tmp_path): plan.create_openhcs_metadata = True context.step_plans = {plan.step_index: plan} - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) assert not (dataset_root / "openhcs_metadata.json").exists() @@ -2530,9 +2526,9 @@ def test_declared_image_destinations_publish_and_reconcile_after_value_cleanup( if stray_image: tifffile.imwrite(tmp_path / "images/review/unowned.tif", pixels) with pytest.raises(MetadataWriteError, match="lack typed produced addresses"): - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) return - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) subdirectories = json.loads((tmp_path / "openhcs_metadata.json").read_text())[ FIELDS.SUBDIRECTORIES ] @@ -2647,7 +2643,7 @@ def test_actual_array_exports_only_publish_declared_raster_inventory( context, plan, artifact_materializations=materializations ) context.runtime_value_store.clear() - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": context}) metadata_path = tmp_path / "openhcs_metadata.json" subdirectories = json.loads(metadata_path.read_text())[FIELDS.SUBDIRECTORIES] assert set(subdirectories) == {"images/exports"} diff --git a/tests/unit/test_named_checkpoint_publication.py b/tests/unit/test_named_checkpoint_publication.py index 0fa1b84de..92f04d293 100644 --- a/tests/unit/test_named_checkpoint_publication.py +++ b/tests/unit/test_named_checkpoint_publication.py @@ -48,7 +48,6 @@ ) from openhcs.core.axes import TileAxis from openhcs.domains.microscopy.axes import Microscopy -from tests.run_records import minimal_run_record @@ -113,8 +112,7 @@ def test_named_and_ordinary_checkpoint_inventory_owns_every_address(tmp_path, ma for key, context in contexts.items() )) OpenHCSMetadataTarget.finalize_completed_dataset( - contexts, runtime_observations=(observations,), - run_record=minimal_run_record(), + contexts, runtime_observations=(observations,) ) root = tmp_path / "source_out" metadata = json.loads((root / "openhcs_metadata.json").read_text()) @@ -167,4 +165,4 @@ def test_named_and_ordinary_checkpoint_inventory_owns_every_address(tmp_path, ma # error. This is not a missing-address permissive fallback. tifffile.imwrite(root / "checkpoints" / "unaddressed.tif", np.zeros((8, 8), dtype=np.uint16)) with pytest.raises(MetadataWriteError, match="Saved images lack typed produced addresses.*unaddressed"): - OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset(bundle.runtime_contexts) diff --git a/tests/unit/test_orchestrator_lane_planning.py b/tests/unit/test_orchestrator_lane_planning.py index 8dfca887a..083a08f49 100644 --- a/tests/unit/test_orchestrator_lane_planning.py +++ b/tests/unit/test_orchestrator_lane_planning.py @@ -1342,7 +1342,7 @@ def test_compiled_execution_returns_settled_nonpersistent_viewer_state_before_cl monkeypatch.setattr( compiled_plate_execution_module.OpenHCSMetadataTarget, "finalize_completed_dataset", - lambda _contexts, *, runtime_observations, run_record: events.append("metadata"), + lambda _contexts, *, runtime_observations: events.append("metadata"), ) response = ViewerControlResponse( payload={ diff --git a/tests/unit/test_persisted_image_metadata.py b/tests/unit/test_persisted_image_metadata.py index 1e6834009..50e3f9408 100644 --- a/tests/unit/test_persisted_image_metadata.py +++ b/tests/unit/test_persisted_image_metadata.py @@ -46,7 +46,6 @@ from openhcs.domains.microscopy.axes import Microscopy from openhcs.core.payload_axes import PayloadAxes from openhcs.core.axes import ColourAxis -from tests.run_records import minimal_run_record, no_produced_files def metadata_fixture(): @@ -168,8 +167,7 @@ def receive_projection_entries(writer, metadata_path, subdirectory, entries): backend="disk", results_dir=None, create_openhcs_metadata=False, ) writer.reconcile_completed_dataset( - metadata_path, {}, produced_entries_by_target={target: entries}, - produced_files=no_produced_files(), run_record=minimal_run_record(), + metadata_path, {}, produced_entries_by_target={target: entries} ) diff --git a/tests/unit/test_persisted_roi_only_publication.py b/tests/unit/test_persisted_roi_only_publication.py index ff66cde25..6ba9383d3 100644 --- a/tests/unit/test_persisted_roi_only_publication.py +++ b/tests/unit/test_persisted_roi_only_publication.py @@ -49,7 +49,6 @@ executing_context, with_facts, ) -from tests.run_records import minimal_run_record, no_produced_files def _publish_saved_step(context, plan, *, artifact_materializations=()): @@ -68,7 +67,6 @@ def _publish_saved_step(context, plan, *, artifact_materializations=()): metadata_path, {target: context for target in entries if target.create_openhcs_metadata}, produced_entries_by_target=entries, - produced_files=no_produced_files(), run_record=minimal_run_record(), ) @@ -248,7 +246,7 @@ def test_saved_roi_only_batch_publishes_and_reconciles_without_rendering( for target in owner.reconciliation_targets(publication_context) ) assert plate / destination in reconciled - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}) assert result_paths(plate, publication_context) == (output.path,) @@ -277,7 +275,7 @@ def test_result_only_destination_saved_through_zarr_passthrough_publishes_comple plan = plan_for(plate, Backend.ZARR.value) publication_context = with_facts(publication_context, step_plans={0: plan}) _publish_saved_step(publication_context, plan, artifact_materializations=(outcome,)) - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}) published = json.loads(METADATA_CONFIG.metadata_path(plate).read_text()) entry = published["subdirectories"]["results"] @@ -347,7 +345,6 @@ def test_measurement_only_step_does_not_reconcile_pending_image_producer( AtomicMetadataWriter().reconcile_completed_dataset( metadata_path, {target: publication_context}, produced_entries_by_target=staged, - produced_files=no_produced_files(), run_record=minimal_run_record(), ) assert not metadata_path.exists() @@ -363,7 +360,7 @@ def test_measurement_only_step_does_not_reconcile_pending_image_producer( SourceProjectionMetadataSerializer.projection_fields(((projection, virtual_path),)), ) publication_context = with_facts(publication_context, step_plans={0: plan}) - OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}, run_record=minimal_run_record()) + OpenHCSMetadataTarget.finalize_completed_dataset({"A01": publication_context}) reconciled = json.loads(metadata_path.read_text())[FIELDS.SUBDIRECTORIES][destination] assert reconciled[FIELDS.IMAGE_FILES] == [virtual_path] assert len(reconciled[FIELDS.SOURCE_PROJECTION]) == 1 diff --git a/tests/unit/test_run_record.py b/tests/unit/test_run_record.py index ced23f453..b1549ef6b 100644 --- a/tests/unit/test_run_record.py +++ b/tests/unit/test_run_record.py @@ -35,6 +35,7 @@ RecordedParameter, RunDeclaration, RunRecord, + RunRecordWriter, SetOnObject, SuppliedAtRunTime, UnrecordedOutputFile, @@ -486,12 +487,7 @@ def test_dataset_root_that_received_only_a_table_gets_a_document_with_the_record table.write_text("nonzero\n9\n") record = minimal_run_record() - AtomicMetadataWriter().reconcile_completed_dataset( - results_root / "openhcs_metadata.json", - {}, - produced_files=ProducedFiles({table: _table_producer()}), - run_record=record, - ) + RunRecordWriter().store(results_root, ProducedFiles({table: _table_producer()}), record) produced = RunRecord.for_output_file(table) assert produced.record.record_id == record.record_id From 5f8e8ed5c70275b370d5658d5525a5d67860a357 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 17:04:36 -0400 Subject: [PATCH 12/12] DL3 receipt: third hand-off (family tags, record storage, guardrails line, what remains) Co-Authored-By: Claude Opus 5.5 --- .../DL3-output-provenance.md | 176 +++++++++++++++++- 1 file changed, 166 insertions(+), 10 deletions(-) diff --git a/docs/refactor/consolidation-20261009/DL3-output-provenance.md b/docs/refactor/consolidation-20261009/DL3-output-provenance.md index 45efd5a90..7d4a91b0b 100644 --- a/docs/refactor/consolidation-20261009/DL3-output-provenance.md +++ b/docs/refactor/consolidation-20261009/DL3-output-provenance.md @@ -2,16 +2,25 @@ Branch `refactor/DL3-output-provenance`. Gap 6 of `D1-cellpose-n2v2-custom-functions.md` (PR #1230). -**Lands after DL1.** The storage part is PolyStore PR OpenHCSDev/PolyStore#41. PolyStore 0.6.3 is -taken by the arraybridge 0.5.0 migration, so #41 has to be released as **0.6.4**, which requires -`arraybridge>=0.5.0,<0.6`. OpenHCS main is on arraybridge 0.4.1 until -`refactor/DL1-device-and-memory-types` merges, so this branch cannot pin 0.6.4 before that. Until -then the submodule `external/PolyStore` points at the #41 commit (`8ee8ce4`) for local tests and -`pyproject.toml` still says `polystore>=0.6.3,<0.7`, which is wrong and must become `>=0.6.4`. - -**Current state: see "Second hand-off" at the end.** The sections between here and there are the -first agent's; where they describe the stored JSON as nested dicts with `kind` keys, the second -hand-off replaces them. +**Lands after DL1 and after four library releases.** Two pins cannot be installed from PyPI yet: + +- `python-introspect>=0.3.0,<0.4` (set in `pyproject.toml`; 0.3.0 is on PyPI). ObjectState, + formbridge, pyqt-reactive and zmqruntime all declare `python-introspect<0.3`, so the resolver + refuses this branch until each of them releases with the bound widened to `<0.4`. Their + submodules run against 0.3.0 here (formbridge and ObjectState suites: no failure; zmqruntime: two + IPC transport tests fail with `EOFError` in `multiprocessing`, not in the codec; pyqt-reactive + not run). +- PolyStore: the storage part is PolyStore PR OpenHCSDev/PolyStore#41, to be released as **0.6.4** + (0.6.3 is the arraybridge 0.5.0 migration and requires `arraybridge>=0.5.0,<0.6`). OpenHCS main + is on arraybridge 0.4.1 until `refactor/DL1-device-and-memory-types` merges. Until then the + submodule `external/PolyStore` points at `5301d35` (branch `feat/file-content-identity`, still + based on PolyStore 0.6.1 and still titled 0.6.3) and `pyproject.toml` still says + `polystore>=0.6.3,<0.7`, which is wrong. At that time set: `polystore>=0.6.4,<0.7`, + `arraybridge>=0.5.0,<0.6`, submodule at the 0.6.4 tag. + +**Current state: see "Third hand-off" at the end.** Earlier sections are the first two agents'; +where they describe a union decoded by its fields, one metadata transaction, or +`OpenHCSMetadataTarget.saved_path_producers`, the third hand-off replaces them. ## What is wrong (measured on `origin/main` at `1b5fe99f5`) @@ -460,3 +469,150 @@ not run after these changes. 9. Outside this branch, unchanged: ROI zip metadata and `segmentation_summary.txt` embed absolute source paths; zip timestamps. +## Third hand-off (2026-10-12): steps A and B done, C to E not started + +Commits `18c4d56ae` (A) and `1a1c8c1eb` (B), on a merge of `origin/main` at `6845c3a36`. + +### A. Stored family members name their class + +**Where the tag is declared.** python-introspect 0.3.0 (PR OpenHCSDev/python-introspect#23, +merged, annotated tag `v0.3.0`, on PyPI through the manual script after the publish workflow sat +queued for 34 minutes). `python_introspect.jsonable`: + +- `MemberFamily.declared_by(annotation)` says whether a declared field type admits several + classes: a union with two or more dataclass members (`UnionMembers`), an abstract dataclass + (`ConcreteSubclasses`: its concrete subclasses, as loaded), or a class whose metaclass is + `AutoRegisterMeta` (`RegisteredMembers`: the registry). The three are subclasses; each answers + `members()`; the registered one reads a tag through `registered_class_from_key`. +- `class_tag(member)` is the one place a tag is derived: the registry key when the class is + registered (stable when the module or class is renamed), otherwise `__name__`. +- Stored form: `{tag: fields}` (one key; no reserved field name). `to_jsonable` now follows each + dataclass's declared field types through optional, `Annotated`, tuple/list/sequence/set and + mapping declarations; `dataclass_from_mapping` reads the class from the tag. +- Errors: unknown tag is `ValueError` naming the known members; a value outside the declared + family, a non-string registry key, or two members with one tag is `TypeError`; an untagged + object in a family field is refused, never matched by its fields. +- A type with its own `to_jsonable.register` projection keeps its whole stored form (P1 + Proposition 3.3 is about the guess; those types already write a nominal key of their own). +- Cost of reading declarations when encoding: a dataclass whose field types do not resolve at run + time is a `TypeError` naming it. One encoded class was affected and is fixed + (`UiLiveMeasurementEntryState.preview`, import moved out of `TYPE_CHECKING`). A live import of + about 2,545 first-party dataclasses found 92 whose annotations do not resolve; none of the others + was found at a `to_jsonable` call, but call sites that encode arbitrary values were not all + traced (`formbridge/bridge/object_state.py:257`, `core/debug_view_models.py:199`, + `core/materialization/core.py:3015`, `mcp/dev_client_rendering.py:375,397`, + `agent/services/artifact_plan_inspection_service.py:443,586`). The full suite is the check. + +**The record's two families** are registered and the fields are declared by the family: +`FunctionRecord.identity: FunctionIdentity` (`defined_by`: `installed_module`, `source_text`) and +`DeclaredFile.content: FileContent` (PolyStore, `content_kind`: `hashed`, `not_hashed`). +PolyStore's `FileContent.to_record` / `from_record`, the `hashed` flag and its hand-kept +`_kind_by_hashed` table are deleted on the #41 branch (certificate: **derived**, the registry and +the codec own both facts; nothing in OpenHCS called them). + +**Every user of the codec, checked** (live import plus `rg`; about 470 dataclasses reachable from +about 300 decode targets): + +| Kind | Where | Effect | +|---|---|---| +| Union of dataclasses decoded by fields | only the two record fields above | now registered families | +| Registered or abstract family with its own projection | `PositionedAxis.spec: AxisSpec` (`PayloadAxes.to_mapping`), `source_spatial_domain: SourceSpatialDomain` (`to_mapping`, key `spatial_dimensions`), `MeasurementTable.rows: ColumnarRows`, `SourceImageProvenance*` | unchanged: they write their own nominal key | +| Field typed with one registered class, decoded | `DebugSnapshotReadResponse.snapshot: DebugSnapshot` (zmq debug response), `ViewerWindowValidationSummaryResult.state: ViewerWindowStateResult \| None` (MCP result) | now `{registry key: fields}` | +| Registered family fields, encoded only | `SourceBindingsConfig.bindings` / `match_plan` (and the step and lazy variants), `GlobalPipelineConfig` / `PipelineConfig` streaming configs, `ObjectRelationship.declaration`, `Output.content`, CPA workspace panels and axes, `DebugProgressRecord.context` / `.snapshot`, `benchmark/openhcs_measured_run.py:101` | tagged in what they encode; `SourceBindingsConfig.declaration_identity()` hashes that JSON, so its value changes (whether it is persisted was not checked) | +| Concrete dataclass with dataclass subclasses as the field type | formbridge `UiActionIdentity`, `UiCodeDocumentIdentity`, `UiStateSurfaceIdentity`, `UiWindowIdentity`, `UiSemanticAddress`, `EndpointConnectionSpec`; `NapariDisplayConfig`; `ViewerPayloadControlOptions`; `GallerySourceEvidence` | **not a family, unchanged.** A subclass instance in such a field is written with its own fields and the decoder then refuses the undeclared ones. Open point: either the codec refuses a subclass instance when encoding, or these bases declare a family | + +**Stored formats that changed:** the `run_records` section of `openhcs_metadata.json` (new on this +branch, nothing released), MCP results and zmq debug responses that carry the fields in the third +and fourth rows (both ends are the same version), and the source-bindings declaration identity. +**Nothing on disk has to be regenerated:** no tracked JSON holds a run record (searched all +12,378 `*.json` / `*.jsonl` files), the only fixture decoded through the codec +(`tests/fixtures/mcp/start-owned-runtime-400.json`) has no family field, and MCP tools publish +`dict[str, Any]` as their result schema. Files under `paper/figures/slas/` hold UI summaries with +`identity` / `semantic_address` objects (the last row: unchanged form) and no code decodes them. +`paper/**`, `validation/**`, `benchmark/results/**` were not touched. + +**Dependants whose upper bound excludes python-introspect 0.3.0:** ObjectState 4.0.0 +(`>=0.2.0,<0.3`), formbridge 0.1.4 (`>=0.2.7,<0.3`), pyqt-reactive 0.8.0 (`>=0.2.6,<0.3`), +zmqruntime 0.6.4 (`>=0.2.5,<0.3`). pycodify and PolyStore do not depend on it; arraybridge has no +upper bound. None was released from here. + +Tests: python-introspect 191 pass on Python 3.10, 3.12 and 3.13 (round trip per member of each +kind of family, members with identical fields, containers and optional fields, unknown tag, +untagged data refused, own projection kept, unresolved field type). OpenHCS: the three run-record +test files and `test_mcp_tool_contract.py` pass; of the 2,951 tests in files that touch the codec, +the failures were a test-order leak (`tests/unit/test_axis_family_witness.py` leaves the +RemoteSensing axis family active for the files that follow it in the same xdist worker; with that +file left out the 40 affected files give 1,410 passed and 1 failed, +`test_installed_mcp_demo.py::test_portable_source_normalization_defers_catalog_and_execution_runtimes`, +which fails inside `RegistryService.metadata_for_canonical_key` in a subprocess and was not +compared with main). + +### B. The two oversized classes and the ratchet + +- `OpenHCSMetadataTarget` and `AtomicMetadataWriter` are byte-identical to main again + (`function_outputs.py` differs from main only in `finalize_function_step_outputs`, which asks + `ProducedFiles.stated_by_step`). Deleted: `OpenHCSMetadataTarget.saved_path_producers` (moved, + same body), the `run_record` / `produced_files` parameters, `ProducedFiles.write_into` + (certificate for all: **derived**, the same statements now run in `run_record.py`). +- **The record is stored by its owner in a second locked update.** `RunRecord.store_with_saved_files` + (called in the same `finally` as `finalize_completed_dataset`, so also after a failure) asks + `RunRecordWriter(AtomicMetadataWriter).store` for each dataset root the run saved a file under. + This replaces the design text's "one locked transaction": the dataset's `subdirectories` and the + record are now two consecutive transactions on the same document, each under its lock. A crash + between them leaves a dataset without its record. That is the price of not putting the record's + logic into the writer's transaction; if one transaction is required, the writer needs an + extension point, which adds lines to it. +- **The compiler still makes the two declaration calls.** The previous note holds: the documents + must be rendered before `_resolve_pipeline_once` replaces the steps, and only its result carries + the per-step provenance. What changed is that both empty-compilation returns go through + `CompiledExecutionBundle.with_nothing_to_run`, so `PipelineCompiler` is 3 lines smaller than on + main instead of 7 larger. +- `StepExecutionObservation.is_empty` is `not any(field values)`, derived from the dataclass + fields (all six were in the hand-written chain). The absence probe in + `virtual_workspace_metadata.py` went with the revert; the `ModuleExportedCallableFormatter` + probe is not counted by the ratchet and is unchanged. +- Two `StringSubscript` hits: `del parameters["func"]` (now a filter on the value being the + step's pattern) and a quoted annotation in `run_record.py`. + `CustomFunctionSourceNamespace.attribute_of_revision` names the namespace attribute once. + +Gate (the one this job runs), after merging `origin/main`: + +``` +ratchet openhcs exit 0 +ratchet scripts exit 0 +ratchet benchmark exit 0 +38 passed, 22 warnings in 39.28s +nra-r1 exit 0: "increased": [] +base 6845c3a363b1d24a6eeaf006f2efce3658e7804f head 1a1c8c1ebe8ae8021fe17f463b7471388add4e44 guardrails rc=0 +``` + +Tests run for B: the run-record files, `test_function_outputs.py`, +`test_orchestrator_lane_planning.py`, the four publication test files restored to main's text, +and 73 unit files that touch custom functions, observations, bundles or compilation: with the +witness file left out, 913 passed, 1 failed +(`test_mcp_server.py::test_mcp_stdio_server_roundtrip_returns_errors_as_payloads`, the 5 s stdio +timeout under xdist the second hand-off describes). Full suite, smoke and parity were not run. + +### What remains, in order + +C. **Defects in what is recorded**, failing test first, none started. The second hand-off's + item 4 is still the exact list: (3) declared path left at its default, (4) `enabled`, + (5) device on a host run (depends on DL1), (6) enum defaults of custom functions, (7) direct + imports and the requirements closure, (8) plate-level outputs, (10) no saved global config + (`RunDeclaration.before_step_resolution` still falls back to `GlobalPipelineConfig()`), + (12) `changed_during_run`. +D. **PolyStore 0.6.4.** Clone at `.claude/ps-clone` (branch `feat/file-content-identity`, pushed + at `5301d35`: #41 plus the registered family). Still to do: rebase onto PolyStore main + (`5fdb617`), version 0.6.4 with `uv.lock`, retitle the PR, suite in a fresh venv, merge, + annotated tag, publish. The pre-rebase commit `5301d35` is what the submodule needs until DL1, + so keep it reachable (push it under another branch name before the rebase is force-pushed). +E. **Reader** (second hand-off item 5: dataset-root answer through the dataset access owner; the + capability's metadata read through `openhcs/agent/path_policy.py`), then the **acceptance + re-run** with the scripts in `.claude/scratch/acceptance/` moved to the typed reader, the + short-response size on the Cellpose record, then delete `.claude/venv-cpu` and + `.claude/scratch/` (both still present, with `.claude/pi-clone` and `.claude/ps-clone`). +F. Decide the two open points above: one transaction or two for the record; concrete bases with + subclasses in codec fields. +G. Gates (the landing agent's): `verify_branch.sh`, smoke, parity. The PR stays a draft. +H. Outside this branch, unchanged: ROI zip metadata and `segmentation_summary.txt` embed absolute + source paths; zip timestamps; `test_axis_family_witness.py` leaks the active axis family.