Skip to content

Bootstrap score_coverage from tooling/coverage with tests, lint and docs - #1

Merged
dcalavrezo-qorix merged 17 commits into
mainfrom
dcalavrezo_coverage
Sep 8, 2026
Merged

Bootstrap score_coverage from tooling/coverage with tests, lint and docs#1
dcalavrezo-qorix merged 17 commits into
mainfrom
dcalavrezo_coverage

Conversation

@dcalavrezo-qorix

@dcalavrezo-qorix dcalavrezo-qorix commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

What

Extracts the LLVM source-based coverage pipeline from eclipse-score/tooling
(coverage/ at commit 9a61f42) into this repository as Bazel module
score_coverage, and brings it to the state needed to start the tool
qualification (S-CORE tool-management process, ISO 26262-8 clause 11).

Nine commits, in order:

  1. Bootstrap — module layout with the consumer API at the root
    (@score_coverage//:defs.bzl, //:merger, //:generate_coverage_html,
    //:enable_llvm_coverage_for_death_tests), implementation in
    score_coverage/, the consumer workspace in integration_tests/, hygiene
    targets in tools/, CI workflows. The root package loads only runtime
    dependencies so it stays loadable for consumers.
  2. Gate ported from shell to Python — same CLI; exit code 2 for "no
    verdict" (broken report, bad threshold, tool failure) distinct from a
    failed gate; unrounded comparison (the shell passed 99.995 % at threshold
    100); corrupt LCOV rejected.
  3. Unit tests for justify and effective_coverage — two defects fixed:
    justification path matching by plain suffix (bar.cpp applied to
    foobar.cpp), silent 0/0 totals on a missing index page.
  4. Analysis tests, ground truth, fault injection, self-coverage gate —
    rules_testing tests for coverage_scope; the integration LCOV is compared
    record by record against a hand-derived expected_lcov.dat; three fault
    injections; coverage.py measures the tool's own Python with a CI ratchet.
  5. merger and reporter covered end to end with fake llvm tools.
  6. gcovr (QNX) backend tested against real gcovr 8.6 markup — fixed the
    "Exec / Excl / Total" totals parsing and the missing summary.txt.
  7. Static analysis (ruff, pylint, ty as aspects, findings fail the build)
    and Rust fixtures for the analysis tests; four functions split to stay
    under complexity 15.
  8. docs-as-code tree — user manual with constraints of use, use cases,
    potential-error analysis (HazOp), 32 tool requirements, architecture,
    verification report, release notes.
  9. Generated test-to-requirement links — tests run under
    score_py_pytest with @verifies(...) annotations; the verification
    report renders the generated matrix.

Verification (local)

Check Result
bazel build //... pass
Unit tests (6 suites, 207 cases) + 8 Starlark analysis tests pass
bazel build --config=lint //score_coverage/... //tools/... 0 findings
//tools:format.check, //tools:copyright.check pass
Self coverage (coverage.py) 95.6 % lines / 87.2 % branches, gate 95/87
integration_tests/run_integration_test.sh 15/15 checks pass
bazel run //docs:docs_check 0 warnings, 42 needs, 207 testcase needs

Notes for reviewers

  • Consumer labels change from @score_tooling//coverage:* to
    @score_coverage//:*. Consumers are not migrated by this PR (planned
    last, after qualification).
  • combined_report and llvm_profile_wrapper were tooling-repo-bound and are
    not carried over.
  • The license-check workflow assumes the org secret ECLIPSE_GITLAB_API_TOKEN
    as in module_template.
  • QNX on-target coverage stays out of scope (tooling#427); the orchestrator
    accepts only the LLVM zip report.

Follow-ups (not in this PR)

  • Tool Verification Report in eclipse-score/score (score_tools/).
  • bazel_registry/modules/score_coverage/metadata.json and deprecation
    aliases in score_tooling.
  • Integration test emitting JUnit results with properties so the ground-truth
    requirement gets a generated link.

@dcalavrezo-qorix
Bootstrap score_coverage module from tooling/coverage
71e845b 
Move the LLVM source-based coverage pipeline out of eclipse-score/tooling
(coverage/ at commit 9a61f42) into its own Bazel module, score_coverage.

Layout:
- defs.bzl, BUILD: consumer API at the module root (score_coverage_scope,
  score_coverage_reporter, //:merger, //:generate_coverage_html,
  //:enable_llvm_coverage_for_death_tests). The root package loads only
  runtime dependencies, so it stays loadable for consumers that do not see
  this module's dev dependencies.
- score_coverage/: implementation and unit tests. The Python package is
  named score_coverage (not coverage) so it cannot shadow coverage.py when
  the module's own tests run under bazel coverage.
- integration_tests/: consumer workspace, now overriding score_coverage
  instead of score_tooling; excluded from //... via .bazelignore.
- tools/: copyright and format targets (dev-only loads).
- .github/workflows: tests, integration test, format, copyright, gitlint,
  license check.

Adaptations from the original: @score_tooling//coverage: labels become
@score_coverage//:, the pip hub is renamed pip_score_coverage to avoid a
collision with score_tooling < 3, and the repo-bound combined_report and
llvm_profile_wrapper helpers are not carried over.

Verified locally: bazel build //..., 3 unit tests, format and copyright
checks, and integration_tests/run_integration_test.sh (all 9 checks pass;
raw 58.82% / effective 61.76% line coverage on the fixture workspace).

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Port generate_coverage_html from shell to Python
0322ad0 
The orchestrator holds the COVERAGE_THRESHOLD gate, the one decision
downstream verification reports rely on, and as a shell script it could
neither be unit tested nor covered. It is now score_coverage/
generate_coverage_html.py with the same CLI (--yaml, --archive,
--archive-dir, --platform, --testlogs-subdir, --summary-md, output-dir)
and the same observable behaviour, verified by the unchanged integration
test (all 9 checks pass).

Changes in behaviour, all in the "fail loud, never fail green" direction:
- justify / effective_coverage / coverage_summary are called in-process
  instead of via nested `bazel run`; the three modules take an argv
  parameter for that.
- Exit codes: 0 gate passed, 1 gate failed, 2 no verdict possible (missing
  or invalid report, bad threshold, justification errors). Previously a
  tool failure and a failed gate were both exit 1.
- The gate compares the unrounded percentage; the shell compared the value
  after printf "%.2f", so 99.995% passed a threshold of 100.
- Non-numeric or out-of-range COVERAGE_THRESHOLD is an error instead of a
  shell arithmetic failure.
- Effective coverage is read from report.json instead of grepping
  summary.txt; corrupt LCOV data (LH > LF, malformed records) is an error.

Unit tests (score_coverage/tests/generate_coverage_html_test.py, 40 cases)
cover the gate primitives, argument parsing and the end-to-end flow on a
synthetic workspace, including the fail-loud paths.

rules_shell is no longer a dependency.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Add unit tests for justify and effective_coverage
f0d359f 
The two largest modules of the justification layer had no tests. This adds
72 cases:

- justify_test (41): YAML validation (every rule of validate_yaml, all
  errors reported together), location resolution, platform filtering,
  marker scanning (single-line, START/STOP regions, nesting, unknown ids,
  unbalanced markers, non-UTF-8 input), source collection (bazel-* skipped)
  and the end-to-end manifest for linux/qnx/unfiltered runs including the
  fail paths (missing location file, invalid YAML).
- effective_coverage_test (31): flooring arithmetic, index totals parsing,
  path matching, llvm-cov row/branch post-processing (justified, stale,
  covered-in-any-instantiation, branch-only justifications), index page
  and CSS updates, and the end-to-end report/summary on a synthetic
  llvm-cov report.

Two defects found while writing them, both fixed:
- find_matching_justifications matched by plain string suffix, so a
  justification for bar.cpp also applied to foobar.cpp and could inflate
  the effective coverage. Matching now requires a path-component boundary.
- parse_index_page_totals returned 0/0 silently when index.html was
  missing (the unparseable case warned); it now warns too.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Add analysis tests, ground truth, fault injection, self-coverage gate
8c44935 
Starlark: rules_testing analysis tests for the coverage_scope rule/aspect
(score_coverage/tests/starlark): transitive deps and implementation_deps
are collected, shared deps are listed once, header-only libraries yield no
baseline archive, generated sources are excluded, output groups are as
documented. The fixtures are tagged manual.

Integration test: the LCOV produced for the fixture workspace is now
compared record by record against integration_tests/expected_lcov.dat, a
hand-derived ground truth (line and branch counts explained in the file
header). Three fault-injection checks were added: a corrupt report and a
non-numeric threshold must exit 2, and a misspelt COV_JUSTIFIED id must be
reported and must not raise the effective coverage.

Self coverage: tools/self_coverage_gate.py reads the coverage.py LCOV that
`bazel coverage --combined_report=lcov` produces for this repository's own
Python, prints the per-file C0/C1 table and fails below the thresholds.
CI runs it after the unit tests; the current floor is 69% lines / 63%
branches (reporter.py 31%, merger.py 48%, effective_coverage.py 68%,
the rest above 96%) and is meant to be ratcheted up.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Cover merger and reporter end to end with fake llvm tools
1bc1ec8 
merger_test (+10 cases): main() against a fake Bazel coverage directory and
a fake llvm-profdata (profraw merge order, meta.json content, zip layout,
dangling gcov symlink cleanup, RUST_LLVM_PROFDATA fallback) plus the
fail paths (no objects / no profraw exit 0 without output, missing or
failing llvm-profdata exit 1).

reporter_test (+19 cases): reports file, per-test zip extraction with every
invalid-input variant, tool resolution order, cxxfilt lookup, allowlist and
baseline manifests (missing baseline object is a hard error), run_command
stderr separation, the exact llvm-cov report/show/export command lines via
a logging fake, and main() end to end: merged profdata, allowlist-driven
exclusion regexes, workspace-relative SF paths, HTML title rewrite, empty
zip for no/invalid reports, missing tools and empty allowlist as errors.

merger.main and reporter.main take an argv parameter like the other tools.

Note: the first version of these additions sat after the unittest.main()
guard and silently never ran; the guard now closes both files. Self
coverage moves from 69.6/63.4 to 86.9% lines / 79.4% branches (merger
98.4%, reporter 91.6%); the CI ratchet is raised to 86/79. The remaining
gap is the gcovr (QNX) backend of effective_coverage.py.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Test the gcovr (QNX) backend of effective_coverage against real markup
cf107bb 
The gcovr backend post-processes the HTML that communication's
lcov_to_html.py produces through gcovr 8.6 for QNX C++ coverage. It had no
tests. The fixtures in effective_coverage_test reproduce the exact markup
of a real gcovr 8.6 --html-details report generated from the integration
workspace's LCOV (index summary rows, Directory cell, Box-header, source
rows with linebranch details). 21 new cases cover format detection, page
discovery, path extraction, totals parsing, line/branch/stale processing,
in-place restyling, CSS and banner injection, and main() end to end with
and without --lcov.

Three defects found and fixed:
- _parse_gcovr_index_totals read gcovr's "Exec / Excl / Total" triple
  "20 / 0 / 34" as 20 of 0, so without --lcov every percentage collapsed
  to 0. The summary rows are now parsed by label; the old two-number
  heuristics remain as a fallback for other gcovr versions.
- _main_gcovr never wrote summary.txt (the llvm-cov path did), which
  generate_coverage_html requires, and crashed when the output directory
  did not exist. Both backends now share _write_outputs.
- The unused "print JSON to stdout when --output is missing" branch is
  gone; --output is mandatory.

Self coverage: 95.6% lines / 87.1% branches (effective_coverage.py from
68.5% to 95.2%). CI ratchet raised to 95/87.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Add static analysis (ruff, pylint, ty) and Rust analysis-test fixtures
1b6fd71 
Static analysis follows score_tooling 2.x: tools/linters.bzl instantiates
the aspect_rules_lint aspects from @score_tooling//third_party/lint (ruff
with the python_basics rule set, pylint via score_tooling's binary, ty),
wired as `bazel build --config=lint` with fail_on_violation. CI runs it
after the tests. docs-as-code's pre-commit script targets score_tooling 1.x
tool binaries that no longer exist in 2.x, so the aspect route is used
instead, and unlike docs-as-code's continue-on-error job it is blocking.

Policy, documented in pyproject.toml: McCabe complexity ceiling 15 (the
process limit is LoC-based; ruff's default 10 is a style preference),
pylint's design counters disabled in favour of that single complexity
measure, module size limit 2000 lines per the process, test modules exempt
from docstring/protected-access/TemporaryDirectory rules.

Bringing the code to zero findings:
- 252 auto-fixable modernisations (typing builtins, X | None, isort).
- Four functions split to stay under complexity 15: validate_yaml (40 ->
  per-field helpers), process_html_file (30 -> status, branch, classify
  and restyle helpers), coverage_summary.parse_lcov (16 -> _LcovRecord),
  justify.main (16 -> resolve/scan/write helpers). Behaviour unchanged;
  all 214 unit cases and the 15 integration checks pass.
- Type fixes found by ty: Runfiles.Create() may return None (now an
  error), RunfilesLike Protocol so tests can pass fakes, Path | None
  defaults, Optional narrowing in tests, llvm-cov calls with explicit
  arguments instead of **dict kwargs. The unused workspace_root parameter
  of load_baseline_objects is gone. The NaN check uses math.isnan.

Rust fixtures: score_toolchains_rust 0.10.0 (dev) with the standard
Ferrocene toolchain registered in .bazelrc as in persistency; the analysis
tests now cover rust_library (CcInfo archive) and rust_binary (CrateInfo
sources plus the executable as baseline object).

Self coverage after the refactors: 95.6% lines / 87.2% branches.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Add the docs-as-code tree (WP4)
06c9678 
Documentation built with score_docs_as_code 8.1.1 against the S-CORE process
needs (score_process_description 2.1.2), from docs/BUILD so the root package
keeps loading only runtime dependencies. `bazel run //docs:docs` and
`//docs:docs_check` pass with zero metamodel warnings; 42 needs.

- manual/: adoption guide (six steps, from the former README), command
  reference, constraints of use CSTR-01..10 each mapped to the potential
  error it mitigates, known problems (with upstream llvm-cov references and
  the QNX limitation, tooling#427).
- requirements/: six use cases as stakeholder requirements
  (stkh_req__coverage__uc_*), the potential-error analysis ERR-01..10 with
  HazOp guide words, impact, detection and mitigations, and 32 tool
  requirements (tool_req__coverage_*) with `satisfies` links to the use cases
  and to gd_req__verification_reporting / _report_archiving, tagged with the
  errors they mitigate. The analysis concludes tool impact yes, detection no
  for ERR-02/03/07, hence expected TCL LOW.
- architecture/: two-phase pipeline with a PlantUML diagram, module/consumer
  split, design decisions (report-time filtering, fail loud, gate on LCOV,
  in-process tool calls). Replaces COVERAGE_GUIDE.md.
- verification/: verification report with test inventory (7 suites, 230
  cases), hand-maintained requirement coverage, structural coverage table
  (95.6 % / 87.2 %), static analysis, end-to-end validation, deviations.
- release/: release notes for 0.1.0 relative to score_tooling 2.2.x.

README.md is now a short pointer to the docs; docs.yml publishes them through
the cicd-workflows docs job. The generated ubproject.toml and docs/_build are
ignored.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Generate the test-to-requirement links (WP3/WP4 leftover)
10b5476 
Unit tests now run through score_tooling's score_py_pytest (pytest with the
attribute plugin, --junitxml=$XML_OUTPUT_FILE). A small class decorator,
score_coverage/tests/traceability.py::verifies, applies add_test_properties
to every test_* method of a TestCase class, so each JUnit testcase carries
PartiallyVerifies (tool_req ids), TestType and DerivationTechnique; missing
descriptions are derived from the test name, explicit docstrings win. Outside
Bazel the decorator is a no-op and the tests remain plain unittest.

All 54 test classes (207 cases) are annotated against the 32 tool requirements.
docs() gets test_sources = ["score_coverage/tests"], and the verification
report's hand-maintained mapping is replaced by a generated needtable with the
testlink column (test name plus execution result) and a result pie. 28 of 32
requirements carry generated links; the three scope requirements (Starlark
analysis tests, no properties) and the ground-truth validation (shell
integration test) are listed as verified outside pytest.

CI: the unit-test job copies the score_coverage test.xml files into
tests-report/ and uploads them as artifact "tests-report"; the docs build
moves into tests.yml as a job depending on it and passes the artifact to the
reusable docs workflow, so the published docs show the links. The separate
docs.yml is removed.

Coverage under pytest is unchanged (95.6 % / 87.2 %); lint stays at zero
findings.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Add gitlint configuration aligned with eclipse-score/score
87fc38a 
Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Refresh MODULE.bazel.lock for the score_toolchains_rust extensions
64581de 
bazel mod deps --lockfile_mode=update records the ferrocene_rules_rust_miri
extension of score_toolchains_rust, which the CI lockfile check flagged.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Fix collection of JUnit results for the docs build
c227e72 
The cd into the bazel-testlogs symlink made ../tests-report resolve outside
the workspace, so no file was copied and the empty artifact upload passed
silently (if-no-files-found: warn). Paths are now rewritten without changing
directory and an empty upload is an error.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
AlexanderLanin
AlexanderLanin previously approved these changes Sep 7, 2026
View reviewed changes

@AlexanderLanin AlexanderLanin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for bootstrapping. Suggest you add CODEOWNERS ASAP.

@dcalavrezo-qorix
Move the docs output to _build for the reusable docs workflow
39516c8 
docs() runs from //docs, so the CLI writes docs/_build while the
cicd-workflows docs job archives _build at the workspace root.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
AlexanderLanin
AlexanderLanin previously approved these changes Sep 7, 2026
View reviewed changes
Comment thread .github/workflows/tests.yml Outdated
@dcalavrezo-qorix
Add CODEOWNERS (infrastructure maintainers, as in eclipse-score/tools)
f06cc55 
Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix

dcalavrezo-qorix commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

LGTM for bootstrapping. Suggest you add CODEOWNERS ASAP.

done

dcalavrezo-qorix added a commit that referenced this pull request Sep 8, 2026
@dcalavrezo-qorix
Make score_docs_as_code and score_process_description regular depende…
b6de68a 
…ncies

Consumers (reference_integration, the Tool Verification Report in
eclipse-score/score) consume //docs:needs_json and the docs bundle of this
module, which only resolves when these modules are not dev dependencies.
Same pattern as persistency and time. Review feedback on PR #1.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
Make the docs modules regular (non-dev) dependencies
021fb6d 
Consumers (reference_integration, the Tool Verification Report in
eclipse-score/score) consume //docs:needs_json and the docs bundle of this
module, which only resolves when score_docs_as_code and
score_process_description are not dev dependencies. Same pattern as
persistency and time. Review feedback on PR #1.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
AlexanderLanin
AlexanderLanin previously approved these changes Sep 8, 2026
View reviewed changes
MaximilianSoerenPollak
MaximilianSoerenPollak previously approved these changes Sep 8, 2026
View reviewed changes

@MaximilianSoerenPollak MaximilianSoerenPollak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments and questions but nothing that would hinder this bootstrap PR from coming in, in my opinion.

Comment thread .github/workflows/gitlint.yml
Comment thread .github/workflows/tests.yml Outdated
Comment thread .github/workflows/tests.yml Outdated
Comment thread integration_tests/.bazelversion Outdated
Comment thread REUSE.toml Outdated
Comment thread pyproject.toml
@dcalavrezo-qorix
Address review: Python 3.12, Bazel 8.6.0, drop REUSE, rsync collection
b13e350 
- Python: every S-CORE consumer defaults to 3.12, so the 3.11 toolchain, pip
  hub and lock are dropped; ruff and ty target 3.12. One interpreter version
  also keeps the qualified environment unambiguous.
- Bazel 8.6.0 in both workspaces, matching score, persistency, time, kyron,
  orchestrator, baselibs and config_management.
- REUSE.toml removed: the copyright checker does not need it and no other
  S-CORE module besides devcontainer uses REUSE.
- tests.yml: JUnit results collected with rsync as suggested; the no-op
  --github_user/--github_repo arguments of the docs CLI are gone.
- Verification report and constraints state the new environment.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
MaximilianSoerenPollak
MaximilianSoerenPollak previously approved these changes Sep 8, 2026
View reviewed changes

@MaximilianSoerenPollak MaximilianSoerenPollak left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine for me, with all questions answered and adressed.

@dcalavrezo-qorix
Refresh MODULE.bazel.lock after the switch to Bazel 8.6.0
7e36b68 
The bzlTransitiveDigest of the rules_distroless apt extension changes with
the Bazel version; CI's lockfile check flagged the stale value.

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
@dcalavrezo-qorix
dcalavrezo-qorix merged commit a4fca8f into main Sep 8, 2026
8 checks passed
@dcalavrezo-qorix
dcalavrezo-qorix deleted the dcalavrezo_coverage branch September 8, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MaximilianSoerenPollak MaximilianSoerenPollak MaximilianSoerenPollak approved these changes
@AlexanderLanin AlexanderLanin AlexanderLanin left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dcalavrezo-qorix @AlexanderLanin @MaximilianSoerenPollak