Skip to content

Testing guidance prioritizes real behavior and APIs - #235

Merged
Marius Storhaug (MariusStorhaug) merged 2 commits into
mainfrom
behavior-focused-testing
Nov 2, 2026
Merged

Marius Storhaug (MariusStorhaug) merged 2 commits into
mainfrom
behavior-focused-testing

Conversation

@MariusStorhaug

@MariusStorhaug Marius Storhaug (MariusStorhaug) commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Testing guidance prioritizes observable supported behavior, real implementations, and safe actual APIs instead of assertions that merely detect implementation edits. A fast credential-free local loop remains available; protected live checks carry explicit prerequisites, authorization, isolation, cleanup, and honest coverage evidence.

The shared plugin adds a thin testing-skill pointer. This is a standards/plugin-source delivery, not a test-suite migration or consumer rollout: existing installations are unchanged, and the focused skill requires updating the shared msx plugin after this source is merged and available.

New: A focused skill routes test work to the canonical policy

msx-coding-testing routes test design, authoring, and review to the single Testing standard. The skill copies no policy. Consumers can continue using their existing general coding route; to gain the focused trigger, update their shared msx plugin after the source is available. This PR does not install it for them.

Technical details

The actual plugin and marketplace entry advance together from 0.1.1 to 0.1.2. The independently managed marketplace metadata stays 0.1.0. Existing plugin tests pass (4 tests, 49 skill directories), both changed plugin Markdown files lint cleanly, and the new pointer resolves when explicitly passed to the existing link tool (1 target in 1 file).

Changed surface Standards checked Framework docs checked Result
.github/plugin/msx/skills/msx-coding-testing/SKILL.md Documentation Model, Natural Language, Markdown Agent Plugin Marketplaces design; sibling thin-skill pattern Aligned
.github/plugin/msx/plugin.json and .github/plugin/marketplace.json Manifest identity and synchronized versions Agent Plugin Marketplaces design; plugin README Aligned
.github/plugin/README.md Documentation Model, Natural Language, Markdown Agent Plugin Marketplaces design Aligned

Changed: Testing policy distinguishes functional evidence from edit detection

Contributors and agents choose tests around supported functions, commands, and APIs without an arbitrary mocked-unit-test ratio or unnecessary remote calls for pure logic. Mocks and fakes are justified exceptions with explicit limits; a fake or an unexecuted live check does not prove that an external API works. A behavior-preserving refactor must not fail tests solely because implementation text changed.

The engineering principle links to the owning policy instead of requiring a cloud-free full suite. Independent expected results, meaningful examined-unit counts, regression intent, linting, formatting, static analysis, link checks, builds, and temporary-fixture tooling checks remain valid. Production writes and unsafe, destructive, or paid calls gain no authorization from this standard. No runtime, CI, or existing-suite migration is included.

Technical details

Scoped comparison: MSXOrg/docs base aa9b5cecc69f4cbe6aa04a5dd70ff9cdadeeac3d to immutable candidate 607cca0010fa08b9fee43c6f5b07c17c5a051253, in two focused ordinary commits. The complete diff contains seven files; no tests, scripts, workflows, runtime dependencies, or settings-delivery changes are imported. The worktree is clean.

Changed surface Standards checked Framework docs checked Result
src/docs/Coding-Standards/Testing.md Existing standard shape, Documentation Model, Natural Language, Markdown None (no framework-specific docs) Fixed in this PR
src/docs/Ways-of-Working/Principles/Engineering-Practices.md Documentation Model, Natural Language, Markdown Owning Testing policy Fixed in this PR
src/docs/Coding-Standards/index.md Generated index and recursive-navigation conventions Repository README and contribution instructions Aligned

Validation on the committed tree:

  • Existing Pester runner: 80 passed in 7 suites; zero failed, skipped, inconclusive, or unexecuted.
  • Markdown lint: all 5 changed Markdown files, zero issues.
  • Internal links: 127 documentation files scanned, all resolve.
  • Cross-repository links: 7 targets in 131 files, all resolve; the new skill pointer was checked separately as recorded above.
  • Generated index verification and diff whitespace checks pass.
  • Clean site build: repository-pinned Zensical 0.0.59, restored in an isolated environment, no issues. The global 0.0.50 mismatch was detected and is not the pinned build evidence; no repository dependency or shared tool was changed.
  • Exact-head Docs CI: Test, Lint, Build, and Links passed; Publish skipped for the PR. Exact-head cross-repository CI passed.
  • Exact-head Copilot review: approved at 2026-11-02T18:38:24Z, covering all seven changed files. The explicit review loop returned clean with zero new comments; complete review-thread pagination returned no threads.

The scoped Task records baseline/scenario reasoning, both commits, local checks, CI/review evidence, and completed standards/convergence review. Every implementation-plan item is complete; the Task remains open for its normal reviewed-merge completion path. Targeted open issue/PR searches for testing, mocks, and functional tests found no additional fully convergent issue; the settings and release deliveries remain separate. No policy-wording or structural-content tests were added, and no existing suites were changed or deleted.

Native Agent merge was verified before enabling and read back afterward: enabled=true, address_reviews=true, fix_ci=true, resolve_conflicts=true, merge_pr=false. This is maintenance-only authorization. The PR remains draft by request with GitHub auto-merge unset; advancing readiness and any later merge require the appropriate user/owner decision. No merge, enqueue, service publication, permission/rule/label change, or consumer rollout was performed or authorized. No protected live-API experiment campaign was run or claimed. The inspected Docs route only publishes Pages on non-PR events and does not invoke Release Management; no release-decision block or labels are applied.

Related references

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

No active review issues are recorded for the testing-guidance and plugin-skill changes.

No open issues 🎉

What changed in this PR

This PR makes the Testing standard the single owner of behavior-focused testing guidance and adds a plugin skill that points agents to it. It does not migrate existing suites or install the skill into consumers.

Changes:

  • Prioritizes supported behavior, real implementations, independent expected results, and justified test doubles.
  • Preserves a credential-free local loop while defining authorization, isolation, cleanup, and evidence requirements for protected live-API checks.
  • Adds the testing skill pointer and synchronizes plugin versions to 0.1.2.
File Description
src/​docs/​Ways-of-Working/​Principles/​Engineering-Practices.md Delegates local-testing requirements to the Testing standard.
src/​docs/​Coding-Standards/​Testing.md Defines behavior-focused testing policy and protected live-check requirements.
src/​docs/​Coding-Standards/​index.md Updates the Testing standard’s summary.
.github/​plugin/​README.md Includes testing in the coding-skill scope.
.github/​plugin/​msx/​skills/​msx-coding-testing/​SKILL.md Adds a pointer to the canonical Testing standard.
.github/​plugin/​msx/​plugin.json Bumps the plugin version to 0.1.2.
.github/​plugin/​marketplace.json Aligns the marketplace version with the plugin.

Review effort: Balanced

@MariusStorhaug
Marius Storhaug (MariusStorhaug) marked this pull request as ready for review November 2, 2026 18:43
@MariusStorhaug
Marius Storhaug (MariusStorhaug) merged commit 180f78a into main Nov 2, 2026
22 checks passed
@MariusStorhaug
Marius Storhaug (MariusStorhaug) deleted the behavior-focused-testing branch November 2, 2026 18:49
Marius Storhaug (MariusStorhaug) added a commit that referenced this pull request Nov 3, 2026
…rds (#208)

Release guidance defines durable intent, phase-aware recovery, complete
source-bound consumer records, and settings-driven decisions. It
preserves the accepted keyed release routing,
`.github/release.settings.yml` convention, solution-specific path
exceptions, and release-range-first PR records while defining the
lifecycle contract.

This delivery changes documentation, not a producer runtime. It creates
no settings file, legacy filename fallback, versioned release, live
caller change, consumer installation, or governance change. Existing
immutable-version consumers need no action. Implementation owners verify
the invoked producer's actual interface and conformance before adopting
any documented setting or lifecycle control.

<!-- ANNOUNCEMENT STOP -->

## New: Owned moving-tag recovery keeps one explicit trust boundary

Use the procedure only for an intentionally moved MSX-owned major or
minor alias that the consumer's artifact-specific pinning standard
permits. Ownership alone does not authorize mutable references: the
owned Action/reusable-workflow major-tag exception does not permit minor
aliases. Exact version tags are immutable; their movement is an
integrity incident, not routine recovery. SHA- or exact-version-pinned
consumers require no action for this documentation delivery.

If a trusted, permitted alias is stale, verify the intended remote and
back up needed local-only tags outside `refs/tags/` before refreshing.
Renaming a tag within that namespace does not preserve it from pruning.
A verified `git bundle create owned-tags-backup.bundle --tags` followed
by `git bundle verify owned-tags-backup.bundle` can retain original tag
objects and reachable history; keep the backup private and outside
checked-in paths.

The whole-namespace refresh is `git fetch --tags --force --prune
--prune-tags origin`. Force replaces local tag values with that
explicitly selected remote's values, and tag pruning removes local tags
absent there. Accept that complete scope deliberately, review the
output, and do not normalize an altered exact version tag. Naming
`origin` prevents another upstream from choosing the source of this
operation.

For ongoing controlled movement, inspect `remote.origin.fetch` and add
`+refs/tags/*:refs/tags/*` once only when absent. Set clone-local
`remote.origin.prune` and `remote.origin.pruneTags` to true, not
clone-wide pruning defaults that can affect other remotes. Fetch
explicitly from origin. Compare the local alias's tag-object hash with
`git ls-remote --refs origin refs/tags/<alias>`, then resolve its exact
release and immutable source before use. A stale or unverifiable result
blocks use.

This is a general conditional operating procedure, not version-specific
migration history. No actual consumer tag, configuration, or history is
changed by this PR.

<details>
<summary>Technical details</summary>

| Changed surface | Standards checked | Framework docs checked | Result
|
| --- | --- | --- | --- |
| Accept Moved Release Tags guide | Guide structure, Natural Language,
Markdown | Owned-alias trust, tag pruning, immutable-source verification
| Aligned |
| GitHub Actions owned-tag link | Owning coding standard, Markdown |
Allowed major-tag exception and consumer recovery | Aligned |
| Capability indexes and navigation | Front matter, generated indexes |
Documentation Model | Aligned |

The earlier isolated anonymous Git experiment reproduced implicit
upstream selection and pruning of a renamed local-only tag. Complete
40-character identities confirmed that an independent bundle retains the
pruned ref, explicit origin restores the intended alias, and
origin-scoped pruning does not remove origin-owned tags when another
remote is fetched. The verification repositories contained only empty
commits and test tags; no repository tags or releases were changed. The
accepted-main reconciliation leaves that operating procedure intact.

</details>

## Changed: Durable lifecycle and native targets require verified
conformance

The numbered specification retains fixed approved source and notes,
permanent reservations after a successful build or external visibility,
one retained artifact through verification and publication, destination
progress, phase-aware retries, retirement and withdrawal,
stable/candidate authority, current discovery, optional aliases, and
independent post-completion announcements. Version computation uses
permanent reservations and the reviewed aggregate minimum bump. Consumer
record coverage separately starts from the actual latest successful
artifact publication for each product, target, and channel.

Authors resolve publication coverage before writing a release-bound
record: verify the actual artifact and publication receipt, its
immutable source provenance, and the exact candidate; compare those
source trees and retain the scoped PR-base comparison separately.
Include earlier merged, skipped, direct, and otherwise unshipped changes
in the applicable consumer range. Successful Gallery publication
advances module note coverage even if later GitHub completion fails.
Completing that older GitHub record preserves its original snapshot and
does not make its already-published module changes new again. These
authoring baselines do not change aggregate approval, version
reservations, or completed-state discovery.

Consumers compose every applicable source-bound record after their
actual consumed baseline through the target in release order. Notes
retain prerequisites, ordering, required intermediate operations,
verification, or an explicit no-action result with its compatibility
basis. Reading a record does not itself require installing that version.
PRs and diffs provide optional depth; resource docs describe supported
target-state usage, not another migration history. Missing publication
provenance or required instructions blocks the affected publication
rather than becoming a no-action result.

Implementation owners verify the invoked immutable producer and actual
controls before relying on them. A requirement or configuration example
does not prove that an older runtime implements recovery, manual release
creation, candidate routing, alias reconciliation, or announcement
journaling. Exact producer adoption actions belong in its own PR/release
record. Existing operators retain their released interface; an
incomplete release is not repaired by rebuilding under an exposed
version or re-resolving changed metadata. This documentation delivers no
broad lifecycle engine, history backfill, downstream propagation, or
Discord sender.

A duplicate or phase-aware retry reuses its original logical intent,
approved source, frozen version, and retained artifact. A separately
approved corrective release has a distinct immutable approved request
identity, even when reviewed source is unchanged; it receives a new
intent/version and preserves its relationship to the earlier attempt. Do
not reuse a retry key to suppress that newly approved request, supersede
an exposed intent, or reuse a reserved version. An affected product
integration template requires its repository, compatible immutable
commit, producer-source compatibility evidence, and delivered work or a
justified no-change result. This docs-only delivery changes no producer
or integration template, so concrete runtime/template adoption values do
not apply here.

Native publishing identities remain explicit. A PowerShell prerelease
uses numeric `ModuleVersion` plus `PrivateData.PSData.Prerelease`:
canonical `2.5.0-alpha` is core `2.5.0` and label `alpha`, not
prerelease text in `ModuleVersion`. Validate the selected client's
suffix syntax and reject or explicitly map unsupported forms without
collisions or lost ordering.

Native bounded ranges, where supported, can express major/minor update
policies without aliases. A field without a native range resolver can
retain immutable patch updates when permitted bounded selection exists
outside that field. A SHA-pinned Action or reusable workflow does not
need or acquire permission to retain a minor alias for that policy.
Git/container reference fields do not evaluate range-like strings.
Require a permitted matching alias only when no other permitted resolver
supplies the bound; if none can express it, fail without widening to
latest or another boundary. Retain the exact version, full SHA, or
digest required by the artifact's pinning standard and trust policy.

The owning VS Code framework declares stable/prerelease native mappings
together before enabling Marketplace publication. Numeric versions and a
native channel do not imply direct conversion from every canonical
SemVer. An odd-minor convention requires reserved stable space and
proven series/counter allocation. Unsupported mappings or collisions
fail before publication; an enabled required destination is not silently
skipped. This PR specifies the conformance boundary, not a new allocator
or Marketplace publisher.

<details>
<summary>Technical details</summary>

| Changed surface | Standards checked | Framework docs checked | Result
|
| --- | --- | --- | --- |
| Release Management spec/design and index | Specification/design
separation, Natural Language, Markdown | Durable intent, aggregate
minima, distinct publication/version baselines, complete records,
conformance | Aligned |
| Publishing-target contract | Owning target requirements, Markdown |
Two-field module mapping, native coordinates, immutable bounded
selection | Aligned |
| Parent/capability indexes and navigation | Generated-index conventions
| Documentation Model | Aligned |

The earlier native PowerShell manifest verification established numeric
`ModuleVersion` and separate prerelease data; prerelease text does not
parse as `System.Version`. Official Git, PowerShell, and Marketplace
documentation grounded the mapping and trust corrections. No runtime
publication or invented allocator is claimed.

Written consumer-policy cases cover omitted policy retaining the
strongest immutable reference, native minor bounds without aliases,
permitted SHA-pinned patch selection without changing major/minor or
retaining a minor alias, the owned-major exception not authorizing minor
aliases, unsupported resolution failing without widening, immutable
external latest selection, opaque range strings, and patch-only pre-1.0
selection staying inside its chosen minor. These are written conformance
cases, not evidence of a delivered resolver runtime.

**Current immutable candidate:**
`a37884419c6c5ec4da56222be27238ee192b4cd1`. **Scoped implementation
base:** accepted main `bdb581f68fddd84bc129394a94388671d1d14356`,
containing delivered #209, #235, #233,
and #196. The complete base-to-head comparison contains eight
existing lifecycle, target, guide, and navigation files. Current
canonical format, naming, testing, solution-specific exceptions,
repository policy, CI configuration, and VS Code framework guidance
match accepted main without an additional PR delta. This repository's
Docs route is documentation validation and Pages deployment, not a
versioned Release Management artifact route; no release-publication
baseline or `Release decisions` block is fabricated for this PR.

Local validation of the composed candidate passes all **64
functional/tooling Pester tests in six suites**, normal index
regeneration, relative links across 128 pages, seven cross-repository
links, exact-file Markdown lint across eight relevant pages, patch
checks, and a clean build with manifest-pinned Zensical **0.0.59**. The
unavailable pinned build version was restored only in an isolated
temporary environment, which was removed afterward. No dependency
manifest was changed. The accepted testing policy is preserved; no
prose-drift tests were added or restored. The inherited-routing scenario
documents the requirement but does not claim an implemented resolver
test.

**New-head required CI was running at handoff:**
[Lint](https://github.com/MSXOrg/docs/actions/runs/41675815774/job/126055996866),
[Links](https://github.com/MSXOrg/docs/actions/runs/41675815774/job/126055996479),
[Test](https://github.com/MSXOrg/docs/actions/runs/41675815774/job/126055996962),
[Build](https://github.com/MSXOrg/docs/actions/runs/41675815774/job/126055996918),
and [Cross-repository
links](https://github.com/MSXOrg/docs/actions/runs/41675815839/job/126055998868).
All five checks identify `a37884419c6c5ec4da56222be27238ee192b4cd1`.
Earlier green checks are not represented as a result for this new head;
current status is shown by the PR checks.

The previous full automated review
[5728246486](#208 (review))
approved `f68927e94a31520c4db482aa8f2359cc999b80aa` with no active
concerns. It is an earlier checkpoint, not a review of this candidate.
This tick began with all review threads resolved and no
changes-requested review, but with approval blocked. No reviewer was
requested or approval manufactured during conflict maintenance. The
exact resolution review preserves lifecycle authority, reservations,
immutable artifact recovery, source-bound records, and consumer policy
while integrating the accepted keyed-routing rules.

The closing Task's original implementation-plan evidence remains in the
prior delivery history. This maintenance resolves the accepted-main
composition gate and leaves new-head required CI and GitHub's remaining
approval/protection requirements as acceptance gates. Producer
implementation, consumer execution, date strategies, historical
correction, and live governance remain outside this Task. No PR merge,
merge enqueue, retarget, release, or artifact promotion was performed;
automatic landing remains off.

</details>

<details>
<summary>Related references</summary>

- Resolves [#227](#227)
- References
[#166](#166)
- References [#209](#209)

</details>

## Changed: Keyed routing and settings retain their accepted boundaries

The lifecycle retains `ReleaseType.Stable`, `ReleaseType.Prerelease`,
and `ReleaseType.ReleaseCandidate`, each with at most one branch. The
stable route inherits the default branch when the routing object or
stable key is omitted. Validate the effective mapping after inheritance:
unknown keys and duplicate branch assignments fail, including a
candidate or prerelease route assigned to the inherited stable branch.

Prerelease and release-candidate routes publish only their assigned
prerelease series, not stable releases or stable discovery/aliases.
Configured branches retain their push-driven series, and open PRs retain
`release:prerelease` using one explicit owned bump label or the valid
effective `DefaultBump`. Candidate versions retain `rc` and increasing
counters; promotion to stable uses the stable version resolver without
carrying either prerelease counter over. An unreviewed direct push to
the stable line still validates without publishing.

The release solution's optional path remains
`.github/release.settings.yml` under the accepted Naming standard. The
convention does not implement a reader, rename a public input, require a
shared directory, or restrict otherwise supported explicit custom paths.
Verify the immutable producer revision's declared settings input and
default before adoption. `ConfigurationFile`, where exposed, remains
that producer's settings-path input.

PSModule retains `.github/PSModule.yml` through `SettingsPath`,
including module release settings; module repositories do not need a
separate release-solution settings file. The VS Code template retains
`.github/vscode-extension.yml` and its settings keys. Both owning
exceptions are preserved from accepted main. This documentation does not
authorize a reader, template, or live-caller migration, and creates no
legacy fallback.

`DefaultBump` still accepts `patch`, `minor`, or `major`, with shared
`patch` as the documented default. Repository settings and explicitly
supplied supported caller overrides resolve before one owned bump-label
override. Omitted caller inputs preserve repository policy; invalid
supplied settings and conflicting signals fail even when another input
supplies a valid level. A compatible effective default needs no per-PR
bump label. Skip and out-of-scope work validate without publication.

The owned set remains exactly `release:patch`, `release:minor`,
`release:major`, `release:prerelease`, and `release:skip`. Prerelease
uses the resolved bump; skip combines with no other signal. Candidate
and announcement behavior uses supported settings rather than extra
labels. A route without versioned release behavior carries neither
release metadata nor any release label.

Authors retain the finalized PR record, release-range-first procedure,
complete source-bound notes, and scoped supporting evidence already
delivered on main. The no-PR-template-file policy remains in Repository
Standard and its owning governance guidance, not PR Format. Product
integration template evidence and genuine repository-specific
contribution instructions remain separate and intact.

Existing runtime consumers do not rename live files or install software
because this guidance changed. Producer support and any required
adoption actions must be verified and delivered by the relevant
implementation owner.

<details>
<summary>Technical details</summary>

| Changed surface | Standards checked | Framework docs checked | Result
|
| --- | --- | --- | --- |
| Lifecycle routing requirements and design | Specification/design
separation, Natural Language, Markdown | Accepted fixed routing keys,
inherited defaults, branch collisions, prerelease sources and
stable-only authority | Aligned |
| Lifecycle settings and configuration sections | Natural Language,
Markdown, Naming | Optional release-solution file, verified reader/input
boundary, unchanged defaults and typed overrides | Aligned |
| Accepted-base composition | Current PR Format, Testing, Repository
Standard | Release-range-first records, no-template ownership, preserved
PSModule and VS Code exceptions | Aligned |

Conflict maintenance merges only accepted main
`bdb581f68fddd84bc129394a94388671d1d14356` into the existing branch. It
resolves conflicts in release spec/design/index without importing an
unmerged delivery or restoring the older `release-branches` schema. The
lifecycle's artifact, aggregate-approval, recovery, discovery,
announcement, version-reservation, settings-precedence, and
optional-file contracts remain intact. Accepted keyed-routing
requirements are expressed within the existing numbered specification
and design; the inherited-default collision scenario makes the
effective-mapping rule explicit. Shared canonical pages and the VS Code
framework match accepted main exactly.

The #233 and #196 sequential-composition gates are
resolved against their delivered merges, not their superseded
development candidates. No other branch, historical record, live label,
consumer pin, governance setting, or producer runtime was changed.
New-head CI and remaining GitHub acceptance gates are still required;
Agent merge maintenance does not authorize automatic landing.

</details>

<details>
<summary>Related references</summary>

- References [#196](#196)
- References [#233](#233)
- References [#235](#235)

</details>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define behavior-focused testing and a thin testing skill

2 participants