Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ services, and does not make dependency safety claims. Provenance-aware,
Scorecard-aware, and enriched SARIF sample artifacts remain covered by their
focused golden tests because those examples include mocked evidence.

For reviewer-facing interpretation of each checked-in example artifact, see
[reviewer-path.md#artifact-evidence-map](reviewer-path.md#artifact-evidence-map).

For `examples/sample-sarif.sarif`, the script normalizes the generated
`originalUriBaseIds.%SRCROOT%.uri` value to `file:///__PROJECT_ROOT__/` so
checked-in output does not depend on a maintainer's local checkout path.
10 changes: 10 additions & 0 deletions tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ This page is a reproducible evidence checklist for reviewing `sbom-diff-and-risk

For the shortest ordered route through these materials, start with
[reviewer-path.md](reviewer-path.md).

For interpreting checked-in examples, use the
[artifact evidence map](reviewer-path.md#artifact-evidence-map). It separates
deterministic no-network examples, mocked enrichment snapshots, and consumer
workflow templates.

## Project Identity

Expand Down Expand Up @@ -69,6 +74,11 @@ python scripts/regenerate-example-artifacts.py --check
For the exact regeneration scope, see
[example-artifact-regeneration.md](example-artifact-regeneration.md).

This check covers deterministic no-network examples only. Provenance-aware and
Scorecard-aware checked-in examples are focused rendering snapshots built by
golden tests with constructed evidence; they are not live PyPI or Scorecard
lookups.

Generate the strict-policy JSON sidecar:

```powershell
Expand Down
27 changes: 26 additions & 1 deletion tools/sbom-diff-and-risk/docs/reviewer-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Inspect the checked-in examples in this order:

| Step | Artifact | What it proves |
| --- | --- | --- |
| 1 | [sample-report.json](../examples/sample-report.json) | Full machine-readable diff, risk, policy, and metadata shape. |
| 1 | [sample-report.json](../examples/sample-report.json) | Default machine-readable diff, risk, and metadata shape. |
| 2 | [sample-summary.json](../examples/sample-summary.json) | Compact CI-facing `summary` contract. |
| 3 | [sample-policy.json](../examples/sample-policy.json) | Policy-only sidecar for CI consumers. |
| 4 | [sample-report.md](../examples/sample-report.md) | Human-readable reviewer report. |
Expand All @@ -73,6 +73,31 @@ Look for these reviewer anchors:

Stop here if you need to understand the review outputs without running code.

## Artifact evidence map

Use this map before interpreting checked-in examples. The example directory
contains deterministic no-network artifacts, mocked enrichment snapshots, and
consumer workflow templates. They answer different review questions.

| Evidence type | Artifacts | How they are produced | Network status | Reviewer use |
| --- | --- | --- | --- | --- |
| Default diff output | [sample-report.json](../examples/sample-report.json), [sample-summary.json](../examples/sample-summary.json), [sample-report.md](../examples/sample-report.md) | `scripts/regenerate-example-artifacts.py` with the CycloneDX example inputs | No network | Verify the default report, compact summary, and Markdown shapes. |
| Policy output | [sample-policy-warn-report.json](../examples/sample-policy-warn-report.json), [sample-policy-fail-report.json](../examples/sample-policy-fail-report.json), [sample-policy.json](../examples/sample-policy.json) | `scripts/regenerate-example-artifacts.py` with local policy files | No network | Review warning, blocking, and policy-only sidecar semantics. |
| Requirements output | [sample-requirements-report.json](../examples/sample-requirements-report.json), [sample-requirements-report.md](../examples/sample-requirements-report.md) | `scripts/regenerate-example-artifacts.py --only requirements` | No network | Verify the `requirements.txt` parser path and report shape. |
| Strict-policy SARIF | [sample-sarif.sarif](../examples/sample-sarif.sarif) | `scripts/regenerate-example-artifacts.py --only sarif` with normalized source root | No network | Inspect conservative code-scanning output for selected findings. |
| PyPI provenance snapshot | [sample-provenance-report.json](../examples/sample-provenance-report.json), [sample-provenance-report.md](../examples/sample-provenance-report.md), [sample-provenance-report.sarif](../examples/sample-provenance-report.sarif) | Focused golden tests with constructed provenance evidence | No live lookup while testing | Review opt-in provenance rendering and policy semantics, not current PyPI package truth. |
| OpenSSF Scorecard snapshot | [sample-scorecard-report.json](../examples/sample-scorecard-report.json), [sample-scorecard-report.md](../examples/sample-scorecard-report.md), [sample-scorecard-report.sarif](../examples/sample-scorecard-report.sarif) | Focused golden tests with constructed Scorecard evidence | No live lookup while testing | Review opt-in Scorecard rendering and explicit policy gating, not current repository reputation. |
| Consumer workflows | [github-actions-consumer.yml](../examples/github-actions-consumer.yml), [github-actions-policy-consumer.yml](../examples/github-actions-policy-consumer.yml) | Checked-in examples | Not executed by this repository | Copy or inspect CI consumption patterns. |

Reviewer rule:

- no-network generated examples prove deterministic local output still matches
the code
- mocked enrichment snapshots prove report rendering and policy interpretation
for opt-in evidence shapes
- release verification docs prove distribution evidence for the tool itself
- none of these artifacts prove that a third-party dependency is safe

## 15-minute reproduction check

From `tools/sbom-diff-and-risk`, run the deterministic example checks:
Expand Down