feat(readers): read returned tool factories with constructor ownership checks

CHANGELOG.md

@@ -4,6 +4,8 @@
 
 ### Changes
 
+- `diff --application` reads bounded local factories returning OpenAI Agents SDK and Google ADK tool lists. It keeps the returned members and their construction locations, and reports unread constructor identity, import effects, escaping values, or unresolved callers as coverage gaps rather than treating them as established wiring. This is static source analysis; it does not execute a factory or establish runtime enforcement. (#874, #964)
+- Constructor ownership still withholds canonical SDK Runner execution consumers and anonymous callable or generator carriers. These are static coverage limits; readable candidate tools and handoffs remain visible, and incomplete scans do not report them as removed. The direct `from . import agent` ADK package layout is supported. When a binding graph is incomplete, a capability comparison now withholds only what depends on absence (removed and narrowed capabilities, resolved findings) and still reports added, changed and unknown-direction changes, saying so in its notes; a remote Google ADK `McpToolset` whose tools cannot be listed is not itself incomplete binding evidence, so a changed endpoint, `os.environ[...]` credential reference or tool filter on it, nested stdio parameters included, is reported again. An ADK sub-agent behind a constructor gap stays a candidate with an incomplete edge instead of being reported as unmatched. A module that re-exports `eval` or `exec` from `builtins` is no longer treated as an unused import while another module imports it. The proof that no repository file provides a standard-library module reads the directory listing, so it is the same on case-sensitive and case-insensitive filesystems. (#964)
 - **An MCP row names the `env_vars` entries that changed.** Replaying openai/codex-security#1281, a `.mcp.json` that added `CODEX_SECURITY_PLUGIN_ROOT` to a server's `env_vars` array and changed nothing else was a `changed` row reading `no difference in the command name launch_codex_security_mcp, launch arguments, env key names or header key names; the change is in a detail this output does not show, such as the command's path or another setting`, which pointed at the command's path for a change to a variable name. An MCP server grant now publishes `env_var_names` (host-grants `0.9`, extended in place): the plain names its `env_vars` list declares, in declared order, never a `NAME=value` entry, an object, a token-shaped string or an entry the digest's redaction rewrites. The row reads `codex-security: env_vars names +CODEX_SECURITY_PLUGIN_ROOT` on `diff` (text and `--json`), `verify`, the PR comment and `check`, beside `env keys` and `header keys` (`env keys +API_BASE -DEBUG; env_vars names +X -Y`); the same names in another order read `env_vars names in a different order`, and an added or removed server's cell lists them. The label is the key the list is declared under, so no claim is made that a variable is set or read. No value is published or printed, and rotating an `env` value is still no row, because `config_sha256`'s input redacts every `env` value. The field is display-only, left out of grant equality, the inventory digests and saved baselines, so no row, direction, `expands`, severity, expansion signal or `check` decision moves; a change confined to an unpublished entry still says it is not shown. See [STABILITY.md](STABILITY.md#mcp-env-var-names-795). (#795)
 - **An OpenShell policy row names the declarations that changed.** Replaying openshift-kni/ai-sandbox#29, a selected policy that removed `/sandbox/.uv/python` from `filesystem_policy.read_only`, changed binary selectors and added GitHub release destinations printed `authored, OpenShell 0.1.2, 6 endpoint(s), facts <digest> → …7 endpoint(s), facts <digest>`, which names none of them. `diff` (text and `--json`), `verify`, `check` and the PR comment now print the difference: `filesystem read_only -/sandbox/.uv/python`, `destination aiplatform.googleapis.com:443 (l4) binaries +/tmp/**/node_modules/…/claude -/usr/local/bin/claude`, `removed destination oauth2.googleapis.com:443 (l4) via /usr/local/bin/claude` and `added destinations objects.githubusercontent.com:443, release-assets.githubusercontent.com:443 (l4) via …`, with changed request declarations, `include_workdir`, Landlock compatibility and process identity named the same way. Each list names its first entries and counts the rest. Only values the OpenShell inventory already publishes are named, and a value that would change under redaction prints as `<redacted>`. Query and parameter constraints and endpoint options appear by field name, not value. A policy selected or removed whole lists what it declares and which sections and inspected endpoints took an upstream default, as a description and not a proof that authority grew. Display only: the published `rows[]`, direction, `expands`, severity, expansion signals and `check` decisions are unchanged, the existing unknown-direction and runtime-limit wording stays in each row's reason, and no schema changes. (#968)
 - **`diff` classifies a large base tree about eight times faster.** Translating each boundary glob into a regular expression was repeated for every path in the tree and was 74–86% of first-run `diff` time on two large repositories (#698's measurement). The compiled pattern is now cached per pattern; which paths match is unchanged, and a test checks the cached matcher against the previous implementation on over 10,000 random pattern and path pairs. (#698)

README.md

@@ -246,7 +246,12 @@ Filename detection never establishes verified permissions.
 ## What your PR sees
 
 The same run writes `pr-comment.md`, the comment the GitHub Action posts.
-Abridged from the artifact:
+This abridged excerpt is from the published `v1.2.0` release. The current
+source tree reports what a change added or changed but withholds what it
+removed or narrowed when agent binding coverage is incomplete, and names that
+limit; that coverage gap does not change the four blocking findings.
+
+Abridged from the released artifact:
 
 ```text
 ## Agents Shipgate

ci_timing.py

@@ -0,0 +1,164 @@
+"""Best-effort hosted timings for whole files, including canceled CI runs.
+
+This observer never selects tests or changes their result. A completed-file
+record requires a real teardown for every selected case. Missing records are
+unmeasured; an invalidation discards any earlier record for that file/session.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+from collections import Counter
+from dataclasses import dataclass, field
+from pathlib import PurePosixPath
+
+import pytest
+
+PREFIX = "SHIPGATE_TIMING "
+SCHEMA = "shipgate.ci_file_timing/v1"
+_PHASES = {"setup": 1, "call": 2, "teardown": 4}
+
+
+@dataclass
+class _File:
+    selected: int
+    completed: int = 0
+    phases: dict[str, float] = field(default_factory=lambda: dict.fromkeys(_PHASES, 0.0))
+    outcomes: Counter = field(default_factory=Counter)
+    invalid: bool = False
+
+
+class FileTimings:
+    """One immutable selected census and distinct worker phase reports."""
+
+    def __init__(self) -> None:
+        self.ids: tuple[str, ...] | None = None
+        self.cases: dict[str, tuple[str, int, str | None]] = {}
+        self.files: dict[str, _File] = {}
+        self.invalid = False
+
+    def collection(self, ids: list[str]) -> list[dict]:
+        if self.invalid:
+            return []
+        current = tuple(ids)
+        if self.ids is not None:
+            if current != self.ids:
+                self.invalid = True
+                return [{"kind": "invalidate_session", "reason": "worker_collection_mismatch"}]
+            return []
+        if not current or len(current) > 50000 or len(set(current)) != len(current):
+            self.invalid = True
+            return [{"kind": "invalidate_session", "reason": "unsupported_collection"}]
+        paths = [nodeid.split("::", 1)[0] for nodeid in current]
+        if any(not path.startswith("tests/") or not path.endswith(".py")
+               or len(path) > 1024 or ".." in PurePosixPath(path).parts
+               or "\\" in path for path in paths):
+            self.invalid = True
+            return [{"kind": "invalidate_session", "reason": "unsupported_collector"}]
+        counts = Counter(paths)
+        if len(counts) > 1000:
+            self.invalid = True
+            return [{"kind": "invalidate_session", "reason": "collection_limit"}]
+        census = {"kind": "census", "selected": len(current),
+                  "files": dict(sorted(counts.items()))}
+        if len(json.dumps(census, ensure_ascii=True)) > 65536:
+            self.invalid = True
+            return [{"kind": "invalidate_session", "reason": "census_output_limit"}]
+        self.ids = current
+        self.cases = {nodeid: (path, 0, None) for nodeid, path in zip(current, paths, strict=True)}
+        self.files = {path: _File(count) for path, count in counts.items()}
+        digest = hashlib.sha256()
+        for nodeid in current:
+            value = nodeid.encode("utf-8", errors="surrogatepass")
+            digest.update(len(value).to_bytes(8, "big"))
+            digest.update(value)
+        return [{**census, "node_ids_sha256": digest.hexdigest(),
+                 "node_digest_encoding": "length-prefixed-utf8-surrogatepass"}]
+
+    def report(self, nodeid: str, phase: str, duration: float, outcome: str) -> list[dict]:
+        if self.invalid or self.ids is None:
+            return []
+        case = self.cases.get(nodeid)
+        if case is None:
+            self.invalid = True
+            return [{"kind": "invalidate_session", "reason": "unknown_case"}]
+        path, seen, setup_outcome = case
+        timing = self.files[path]
+        if timing.invalid:
+            return []
+        bit = _PHASES.get(phase)
+        if (bit is None or seen & bit or seen & _PHASES["teardown"]
+                or not isinstance(duration, int | float) or isinstance(duration, bool)
+                or not math.isfinite(duration) or duration < 0
+                or outcome not in {"passed", "failed", "skipped"}
+                or (phase != "setup" and not seen & _PHASES["setup"])
+                or (phase == "call" and setup_outcome != "passed")
+                or (phase == "teardown" and (setup_outcome == "passed") != bool(seen & _PHASES["call"]))):
+            timing.invalid = True
+            return [{"kind": "invalidate_file", "file": path,
+                     "reason": "unsupported_or_duplicate_report"}]
+        self.cases[nodeid] = (path, seen | bit, outcome if phase == "setup" else setup_outcome)
+        timing.phases[phase] += duration
+        if not math.isfinite(timing.phases[phase]):
+            timing.invalid = True
+            return [{"kind": "invalidate_file", "file": path, "reason": "nonfinite_total"}]
+        timing.outcomes[outcome] += 1
+        if phase != "teardown":
+            return []
+        timing.completed += 1
+        if timing.completed != timing.selected:
+            return []
+        return [{"kind": "complete_file", "file": path, "selected": timing.selected,
+                 "completed": timing.completed, "phase_seconds": timing.phases.copy(),
+                 "seconds": math.fsum(timing.phases.values()),
+                 "phase_outcomes": dict(sorted(timing.outcomes.items()))}]
+
+
+class HostedTimings:
+    """Controller-only observer registered for the existing load-once CI."""
+
+    def __init__(self, config) -> None:  # noqa: ANN001
+        self.config = config
+        self.timings = FileTimings()
+        self.workers: set[str] = set()
+        self.disabled = False
+
+    def _emit(self, events: list[dict]) -> None:
+        if self.disabled:
+            return
+        terminal = self.config.pluginmanager.getplugin("terminalreporter")
+        if terminal is None:
+            self.disabled = True
+            return
+        try:
+            if events:
+                terminal.ensure_newline()
+                if terminal._tw.width_of_current_line != 0:
+                    terminal.write_line("")
+            for event in events:
+                payload = {"schema": SCHEMA, "workers_observed": len(self.workers), **event}
+                terminal.write_line(PREFIX + json.dumps(payload, ensure_ascii=True, allow_nan=False))
+                terminal._tw.flush()
+        except Exception:
+            # Diagnostic output must not turn passing tests into a failing job.
+            self.disabled = True
+
+    @pytest.hookimpl(optionalhook=True)
+    def pytest_xdist_node_collection_finished(self, node, ids) -> None:  # noqa: ANN001
+        if self.disabled:
+            return
+        try:
+            self.workers.add(node.gateway.id)
+            self._emit(self.timings.collection(ids))
+        except Exception:
+            self.disabled = True
+
+    def pytest_runtest_logreport(self, report) -> None:  # noqa: ANN001
+        if self.disabled:
+            return
+        try:
+            self._emit(self.timings.report(report.nodeid, report.when, report.duration, report.outcome))
+        except Exception:
+            self.disabled = True

conftest.py

@@ -30,6 +30,20 @@
 from ci_sharding import load_seconds, shard_assignment  # noqa: E402
 
 
+def pytest_configure(config) -> None:  # noqa: ANN001
+    """Stream hosted file timings without affecting test selection or results."""
+    if (os.environ.get("GITHUB_ACTIONS") != "true" or hasattr(config, "workerinput")
+            or config.getoption("collectonly")
+            or config.getoption("setuponly")
+            or not config.getoption("numprocesses", default=None)
+            or config.getoption("dist", default=None) != "load"
+            or config.pluginmanager.hasplugin("rerunfailures")):
+        return
+    from ci_timing import HostedTimings
+
+    config.pluginmanager.register(HostedTimings(config), "shipgate_hosted_timings")
+
+
 @pytest.fixture(autouse=True)
 def _scrub_agent_environment(monkeypatch: pytest.MonkeyPatch) -> None:
     """Keep the suite hermetic when it runs inside a coding agent.
@@ -78,9 +92,16 @@ def _shard_selection() -> tuple[int, int] | None:
     return count, index
 
 
+@pytest.hookimpl(tryfirst=True)
 def pytest_collection_modifyitems(config, items) -> None:  # noqa: ANN001
-    """Keep only this shard's files when a shard is requested."""
+    """Mark the nightly-only framework cases, then keep this shard's files.
+
+    The marks go on before pytest applies ``-m``, which is why this runs first.
+    """
+
+    from tests.framework_matrix import mark_nightly_framework_cases
 
+    mark_nightly_framework_cases(items)
     selection = _shard_selection()
     if selection is None:
         return

docs/distribution-surfaces.md

@@ -177,6 +177,13 @@ provenance and makes no claim about a runtime verdict or qualification.
 the `emitted_ci_workflow` row above covers how the installed engine uses its
 record. The top-level classifier records this distinction explicitly.
 
+`ci_sharding.py`, `ci_timing.py` and `conftest.py` are repository test-run
+mechanics. The timing observer records selected-file phase durations and
+outcomes for CI diagnostics; it publishes no engine answer, release verdict
+or adopter-facing command. These modules are classified outside distribution
+surfaces, with observer behavior covered by `tests/test_ci_timing.py` and
+selection covered by `tests/test_shard_partition.py`.
+
 The main-tree `.well-known/agents-shipgate.json` integration enumeration is
 checked against `contract --json` from the same source build, including missing
 and unsupported entries. The site's discovery copy stays pinned to its released