Skip to content

Promote develop to main: runtime metrics and hub convergence#884

Open
ptr727 wants to merge 16 commits into
mainfrom
develop
Open

Promote develop to main: runtime metrics and hub convergence#884
ptr727 wants to merge 16 commits into
mainfrom
develop

Conversation

@ptr727

@ptr727 ptr727 commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Promotion of develop to main. Twelve commits since the last promotion: the runtime-metrics feature, the #880 convergence work, a docs change, and five Dependabot bumps.

Referencing #880 rather than closing it - the hub still has open dispositions there and it is the live coordination channel for them. It gets closed by hand once those land.

What is in it

Runtime metrics (the functional change, and why the floor moved to 3.22)

Hub convergence (#880) - 086cb8b, ba3b01a, f91af07

  • Governance: Repository Boundaries and Write Safety and Verification Discipline vendored, Git and Commit Rules refreshed, and the AGENTS.md structure re-derived against the canonical.
  • Release pipeline: the validate-release entry gate now asserts branch/version consistency in both directions before anything builds, the canonical github-release job brings the release-asset-* cleanup, and the Docker job can no longer push on a partial run.
  • Config: .markdownlint-cli2.jsonc and repo-config/configure.sh re-vendored (the latter gaining the read-only check mode), Docker/README.md on reference-style links, BUILD_CONFIGURATION defaulting to Release, set -Eeuo pipefail standardized, and the publisher pinning ref: github.sha.

Other - #866 (self-contained agent docs), plus Dependabot: #870, #872, #874, #876, #878.

Release impact

version.json is already at 3.22 (raised in #865 for the metrics feature), and README + HISTORY both carry the 3.22 notes, so no version edit is needed here. main currently releases 3.21.x; the next dispatch or scheduled run after this merges publishes 3.22.x as stable and refreshes latest.

Note this promotion is the first to run under the re-vendored release task, so the publish that follows it exercises validate-release and the new artifact cleanup against main for the first time. The main arm of the gate (a public release must carry no prerelease suffix) has not run outside smoke yet, where it self-skips by design.

Verification

Every commit here came through CI on develop with a Copilot review on its head SHA. The full lint gate (markdownlint-cli2, actionlint, editorconfig-checker, cspell) and the husky pre-commit gate are clean on the tip. The validate-release smoke self-skip was confirmed from its job log rather than inferred from a green check.

Merge-commit only, and never with --delete-branch - the head of a promotion PR is develop itself.

ptr727 and others added 12 commits July 18, 2026 17:22
Two agent-facing docs pointed at an external repo a reader of this repo cannot open. Route cross-cutting fixes to the maintainer and phrase defaults without an external attribution. Docs only, no behavior change.
…tics.Metrics (#865)

Add an always-on System.Diagnostics.Metrics meter (PlexCleaner.Process) so a long-running process or monitor pass can be inspected live with dotnet-counters. Overall progress is weighted by input bytes, not file count. Includes per-outcome counters, in-flight and thread gauges, byte-weighted progress and ETA, and file and per-tool duration histograms, all with bounded tags. Ships a Docker counters wrapper. Zero new packages.
* Add per-file partial-credit progress to the runtime metrics

Fold in-flight files' partial progress into the byte-weighted
progress.ratio and eta.seconds gauges so a long scan or re-encode
advances the aggregate continuously instead of only at completion.

- Metrics: per-file FileSink carrying byte weight and an atomic
  permyriad fraction, a lock-free in-flight registry, and a
  ThreadLocal current-file sink captured at each tool call site.
- MediaTool.ExecuteStreamStdOut streams stdout for progress while
  capturing stderr for failure logging.
- Instrument the long ops only: ffprobe full-file scan (pts over
  duration), ffmpeg re-encode (-progress pipe:1), HandBrake
  re-encode (--json Working.Progress). Fast remux/copy/setts stay
  at zero until completion.
- Tests for the fold math and the three tool progress parsers.
- Convert all README and HISTORY inline URIs to reference-style
  links, grouped shields/internal/external and alphabetized.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Track max pts so the scan fraction only moves forward

Packets arrive in demux order with interleaved audio/video and
reordered B-frame timestamps, so raw per-packet PtsTime is
non-monotonic. Report the running max pts over duration so the
in-flight scan fraction advances without backward jitter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Model one-file-per-thread in the observable metrics test

Complete one file and start a second that stays in flight, instead
of starting two files on one thread, so the test matches the
ThreadLocal current-file semantics, and clear the in-flight sink at
the end.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Read the sink duration via Volatile across threads

DurationUs is written on the worker thread and read per progress
line on the tool pipe thread, so back it with a Volatile.Read
getter and a SetDurationUs writer, matching the permyriad fraction
pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Switch runtime metrics to operation-weighted progress

Replace the in-tool partial-credit progress with a simpler
operation-counting model: each heavy full-file operation adds the
file's size to the work total when it starts and to the completed total
when it ends, so progress reflects the actual, non-deterministic work
with no per-file fraction weighting and no parsing of tool progress
output.

- Metrics: work.total/work.completed byte counters incremented at
  operation boundaries via an ambient thread-local file size; drop the
  FileSink partial-credit machinery. bytes.total stays the input-size
  reference.
- Bracket the heavy full-file operations (closed-caption scan, idet,
  ffmpeg re-encode, HandBrake deinterlace, packet analysis, verify)
  with OpStarted/OpCompleted.
- Keep the tool progress-parsing capability (ParseProgressFraction,
  ExecuteStreamStdOut, the Progress()/Json() builders) and its unit
  tests unused by this feature, for the media-tools library.
- NativeAOT: enable EventSourceSupport when PublishAot is set so
  dotnet-counters and dotnet-monitor can read the meter from an AOT
  build; default builds are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Call the work instruments gauges not counters in the docs

work.total and work.completed are run-scoped observable gauges, not
Counter instruments, so name them gauges to avoid confusing
dotnet-counters and OpenTelemetry consumers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Count operation completion only when the tool actually ran

Move OpCompleted after the executed guard so a cancelled or
failed-to-start tool is not credited as completed work, which would
skew progress and ETA on interrupted runs. A non-zero exit still
counts, the tool did the full-file work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Count the packet scan when it ran but exited non-zero

GetPackets returns false on a non-zero exit where ffprobe still ran
the scan, so gate OpCompleted on cancellation instead of the return
value, otherwise a ran-but-non-zero scan strands its work in the
total and skews progress for the rest of the run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Credit the packet scan only when ffprobe actually ran

Gate OpCompleted on the scan having run, evidenced by exit 0 or
captured stderr, instead of a cancellation check, so a cancellation
or a failure to start (empty error) is excluded while a ran-but-non-zero
scan is still counted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Roll back discovered work when an operation never runs

OpStarted adds the file size to the work total up front, so an
operation that is cancelled or fails to start would leave the total
permanently inflated and stop progress converging. Add OpAborted to
roll that size back out, and call it on every abort path so each
started operation resolves to exactly one of completed or aborted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
)

Bumps the actions-deps group with 1 update: [DavidAnson/markdownlint-cli2-action](https://github.com/davidanson/markdownlint-cli2-action).


Updates `DavidAnson/markdownlint-cli2-action` from 24.0.0 to 24.1.0
- [Release notes](https://github.com/davidanson/markdownlint-cli2-action/releases)
- [Commits](DavidAnson/markdownlint-cli2-action@8de2aa0...6bf21b0)

---
updated-dependencies:
- dependency-name: DavidAnson/markdownlint-cli2-action
  dependency-version: 24.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps AwesomeAssertions from 9.4.0 to 9.5.0
Bumps ptr727.LanguageTags from 1.5.48 to 1.5.53

---
updated-dependencies:
- dependency-name: AwesomeAssertions
  dependency-version: 9.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: nuget-deps
- dependency-name: ptr727.LanguageTags
  dependency-version: 1.5.53
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: nuget-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps ptr727.LanguageTags from 1.5.53 to 1.5.56
Bumps ptr727.Utilities from 4.0.18 to 4.0.28

---
updated-dependencies:
- dependency-name: ptr727.LanguageTags
  dependency-version: 1.5.56
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: nuget-deps
- dependency-name: ptr727.Utilities
  dependency-version: 4.0.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: nuget-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…874)

Bumps the actions-deps group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@9c091bb...3d3c42e)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#878)

Bumps the actions-deps group with 1 update: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv).


Updates `astral-sh/setup-uv` from 8.3.2 to 9.0.0
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@11f9893...c771a70)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 9.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Vendor the Repository Boundaries and Write Safety and Verification Discipline
sections, refresh the canonical Git and Commit Rules bullets, and re-vendor
.markdownlint-cli2.jsonc and repo-config/configure.sh.

Convert Docker/README.md to reference-style links, default the Dockerfile
BUILD_CONFIGURATION to Release, standardize set -Eeuo pipefail in every
non github-release run: block, and pin the publisher ref to github.sha.

The github-release job is untouched pending the release-pipeline
reconciliation that carries the prerelease backstop.
Add the canonical validate-release job: it asserts branch<->version
classification once, before the builds, covering both arms (main must be
public, every other branch prerelease) where the old in-job backstop
checked only main and only after building.

Take the canonical github-release job, which adds the release-asset-*
cleanup at its point of consumption, gated to the release-create step
and best-effort. publish-release grants the actions: write it needs.

Adopt the Docker atomicity gate so the image is never pushed on a
partial run, and add expect_release_assets, which the canonical
github-release job reads.

Sweep WORKFLOW.md for the moved gate: D2.2, D5.4, the publish diagram,
the 5A assertions, and the scenario table. Correct two stale style
claims found in the sweep - the shell prologue is set -Eeuo pipefail,
and workflow YAML is pinned LF, not CRLF.
Re-vendor the two verbatim sections the hub batch changed: Git and
Commit Rules (the history-rewrite bullet's grammar) and Verification
Discipline (the lint-gate bullet is now self-contained, so it no longer
points at a section this repo does not carry). Repository Boundaries and
Write Safety is unchanged upstream and verified byte-identical.

Add the structural sections this file was missing: Foundational
Principles, Repository Onboarding and Conformance, Supported Development
Platforms, Editor and Tasks, and Repository Details. Rename Project
Structure to Repository Layout, its canonical name, keeping this repo's
own content.

Sections whose canonical body is not about this repo are adapted rather
than copied: Foundational Principles drops the operational-repo
cross-reference and names this repo's two pull channels, and Repository
Onboarding states the audit obligation without the machinery links that
do not resolve here. Devcontainer is left out - this repo ships none.
Copilot AI review requested due to automatic review settings July 24, 2026 02:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Promotes develop to main, bringing the 3.22 runtime-metrics feature, the #880 hub convergence workflow/governance updates, documentation refreshes, and Dependabot bumps into the stable branch and next scheduled/dispatch release train.

Changes:

  • Add always-on runtime metrics (System.Diagnostics.Metrics) with operation-weighted progress and per-tool/per-outcome instrumentation.
  • Converge CI/CD release orchestration on the updated entry gate + artifact-cleanup model (pin publish to github.sha, validate branch/version classification, avoid partial Docker pushes).
  • Refresh governance/config/docs (AGENTS/WORKFLOW/CODESTYLE/markdownlint) and bump dependency versions.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
WORKFLOW.md Updates workflow spec (shell strictness, LF note, publish ref pinning, validate-release gate, artifact cleanup).
version.json Bumps version floor from 3.21 to 3.22.
repo-config/README.md Documents new configure.sh apply / check modes and added security feature management.
repo-config/configure.sh Adds read-only check mode, safer ruleset lookup, and applies Dependabot security settings.
README.md Updates release summary to 3.22 and adds a Runtime Metrics section + ref-style link normalization.
HISTORY.md Adds 3.22 release notes covering runtime metrics behavior and instruments.
ARCHITECTURE.md Documents runtime metrics architecture, hooks, and progress semantics.
AGENTS.md Adds foundational principles, write-safety + verification discipline sections, and refreshes governance structure.
CODESTYLE.md Clarifies markdownlint expectations (MD033 enabled) and cspell CI scope wording.
.markdownlint-cli2.jsonc Enables MD033 again (keeps MD013 disabled) and updates commentary.
Directory.Packages.props Bumps AwesomeAssertions, ptr727.LanguageTags, and ptr727.Utilities versions.
PlexCleaner/Metrics.cs Introduces the process-wide meter and run-scoped gauges/counters for runtime metrics.
PlexCleaner/ProcessDriver.cs Adds per-run file/byte aggregation and per-file metrics accounting around task execution.
PlexCleaner/Process.cs Records per-outcome state/modified/verifyfailed metrics at the point state is known.
PlexCleaner/MediaTool.cs Adds tool-duration recording and a new stdout-streaming execution path.
PlexCleaner/FfProbeTool.cs Adds tool-duration recording and brackets heavy ops with OpStarted/OpCompleted/OpAborted.
PlexCleaner/FfMpegTool.cs Adds ffmpeg progress parsing helper and brackets heavy ops for operation-weighted progress.
PlexCleaner/FfMpegBuilder.cs Adds -progress pipe:1 builder helper for future progress parsing use.
PlexCleaner/HandBrakeTool.cs Adds HandBrake JSON progress parsing helper and brackets heavy ops for progress weighting.
PlexCleaner/HandBrakeBuilder.cs Adds --json builder helper for future progress parsing use.
PlexCleaner/PlexCleaner.csproj Enables EventSourceSupport for NativeAOT builds to allow metrics consumption.
PlexCleaner/Program.cs Disposes Metrics on program exit.
PlexCleaner/ProcessFile.cs Minor formatting-only change (blank line).
PlexCleanerTests/MetricsTests.cs Adds sequential tests for progress/ETA math, flag enumeration, and meter observability.
PlexCleanerTests/ToolProgressParsingTests.cs Adds tests for ffmpeg/HandBrake progress-line parsing helpers.
Docker/Dockerfile Switches default BUILD_CONFIGURATION to Release and installs counters wrapper.
Docker/counters.sh Adds counters wrapper to read runtime metrics inside the container.
Docker/README.md Converts to reference-style links + adds organized link definitions.
.github/workflows/publish-release.yml Pins publish ref to github.sha and grants actions: write for artifact cleanup.
.github/workflows/build-release-task.yml Adds validate-release gate, improves docker push gating, and adds release-asset cleanup.
.github/workflows/get-version-task.yml Updates comments and bumps checkout pin.
.github/workflows/validate-task.yml Bumps action pins, standardizes set -Eeuo pipefail, and updates uv setup action.
.github/workflows/test-pull-request.yml Standardizes set -Eeuo pipefail.
.github/workflows/merge-bot-pull-request.yml Standardizes set -Eeuo pipefail.
.github/workflows/build-executable-task.yml Bumps checkout pin and adds set -Eeuo pipefail for publish step.
.github/workflows/build-docker-task.yml Bumps checkout pin.
.github/copilot-instructions.md Removes external-repo pointer; routes shared-convention discrepancies to maintainer.

Comment on lines +220 to +223
for id in $ids; do
gh api --method DELETE "repos/$GITHUB_REPOSITORY/actions/artifacts/$id" \
|| echo "::warning::Failed to delete artifact $id; retention-days backstop will reap it."
done
#885)

Bumps the actions-deps group with 1 update: [docker/login-action](https://github.com/docker/login-action).


Updates `docker/login-action` from 4.4.0 to 4.5.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](docker/login-action@af1e73f...06fb636)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 24, 2026 05:45
Bumps ptr727.LanguageTags from 1.5.56 to 1.5.58

---
updated-dependencies:
- dependency-name: ptr727.LanguageTags
  dependency-version: 1.5.58
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: nuget-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings July 24, 2026 05:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Comment thread ARCHITECTURE.md
Comment on lines +82 to +83
- Hooks: `ProcessDriver.ProcessFiles` (the choke point every command and monitor cycle funnels through) drives the file/byte/in-flight instruments and the byte-weighted `progress.ratio`. `Process.ProcessFiles` records the per-`SidecarFile.StatesType` outcomes. `MediaTool` execution paths record `tool.duration`.
- Progress is weighted by input bytes (summed once up front and credited at completion from the same map), not file count.
Re-vendor all 15 applicable verbatim AGENTS.md sections byte-for-byte from
hub main per the per-section fidelity model (spec/section-model.md), add a
Devcontainer intent section (this repo ships none, so it documents the native
toolchain), and keep Repository Layout as this repo's own. Operational
Repositories is operational-only, so N/A here.

Remove two sections the model does not permit, without losing content:
Shared Configuration and Tooling was a near-duplicate of the now-verbatim
canonical sections plus CODESTYLE, so its unique C# format-gate note moves to
CODESTYLE and its release-notes rule to ARCHITECTURE; Logging Conventions is
PlexCleaner-specific app behavior, so it moves whole to ARCHITECTURE. Drop the
Git repo-specific subsection, which broke the section byte-match and is already
covered by the verbatim Merge Gate and Branching Model.
Copilot AI review requested due to automatic review settings July 24, 2026 15:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Comment thread Docker/counters.sh
# docker exec <container> counters # live monitor of the PlexCleaner.Process meter
# docker exec <container> counters collect ... # any other dotnet-counters verb/args, passed through

set -euo pipefail
The hub genericized the verbatim sections (ProjectTemplate#422): the
hub-only 404 links inside them are de-linked to prose, and Operational
Repositories is reclassified appliesTo:* so its cross-references resolve
everywhere. Re-vendor all 16 verbatim sections byte-for-byte from the
current hub main.

Carry Operational Repositories as fleet context, and drop Repository
Onboarding and Conformance, which the hub reclassified as hub-only audit
machinery. The verbatim text now carries zero relative links to hub-only
paths. Devcontainer and Repository Layout stay as this repo's own intent
sections. Description consistency (ProjectTemplate#423) already holds.
Copilot AI review requested due to automatic review settings July 24, 2026 19:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

Docker/counters.sh:9

  • This script starts with set -euo pipefail, but the repo's bash-safety convention is set -Eeuo pipefail (include -E so an ERR trap would inherit into functions/subshells). Update this to match the documented standard used across workflows/scripts.
set -euo pipefail

ARCHITECTURE.md:83

  • The new Runtime Metrics section describes progress.ratio as byte-weighted and credited at completion, but the implementation in Metrics.cs is operation-weighted (work.total/work.completed grows as operations are discovered). This documentation should match the actual metrics model to avoid misleading readers.
- Hooks: `ProcessDriver.ProcessFiles` (the choke point every command and monitor cycle funnels through) drives the file/byte/in-flight instruments and the byte-weighted `progress.ratio`. `Process.ProcessFiles` records the per-`SidecarFile.StatesType` outcomes. `MediaTool` execution paths record `tool.duration`.
- Progress is weighted by input bytes (summed once up front and credited at completion from the same map), not file count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants