Skip to content

feat: Phase 3 label validation enforcement #1613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ashleyshaw merged 8 commits into from
Aug 7, 2026
Merged

feat: Phase 3 label validation enforcement #1613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4db7b67
docs: add CHANGELOG entries for PR #1533 (agent consolidation and fil…
ashleyshaw Aug 5, 2026
344dd40
feat: Add pre-creation label validation script
ashleyshaw Aug 5, 2026
b4113ab
Merge branch 'develop' into feat/phase-3-label-validation-enforcement
ashleyshaw Aug 6, 2026
4278246
fix: Remove unused variables and refactor validation logic
ashleyshaw Aug 6, 2026
78adc80
test: Strengthen whitespace handling test assertion
ashleyshaw Aug 6, 2026
d2debf2
docs: Phase 4 label validation documentation
ashleyshaw Aug 6, 2026
38fbe2b
fix: Remove defective labeling-agent code (Phase 1.3)
ashleyshaw Aug 6, 2026
4a3fd7d
docs: Fix CodeRabbit feedback on label validation documentation
ashleyshaw Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 137 additions & 10 deletions docs/LABELING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
title: "Labeling Strategy & Governance"
description: "Label taxonomy, automation rules, and governance for LightSpeed repositories."
file_type: "documentation"
version: 'v1.0.2'
last_updated: '2026-06-18'
version: 'v1.1.0'
last_updated: '2026-08-06'
author: "LightSpeed Team"
maintainer: "LightSpeed Team"
owners: ["lightspeedwp"]
tags: ["labels", "automation", "governance", "colours", "accessibility"]
tags: ["labels", "automation", "governance", "colours", "accessibility", "validation"]
---

# GitHub Labelling & Automation
Expand All @@ -24,8 +24,9 @@ This document describes how LightSpeed uses GitHub labels to power automation, s
4. [Pull Request Labelling](#pull-request-labelling)
5. [Discussion Labelling](#discussion-labelling)
6. [Automation & Agent Integration](#automation--agent-integration)
7. [Best Practices](#best-practices)
8. [Troubleshooting](#troubleshooting)
7. [Pre-Creation Label Validation (Phase 3)](#pre-creation-label-validation-phase-3)
8. [Best Practices](#best-practices)
9. [Troubleshooting](#troubleshooting)

---

Expand Down Expand Up @@ -327,6 +328,96 @@ All automation reads from these files; there is no hardcoded label logic in agen

---

## Pre-Creation Label Validation (Phase 3)

To prevent bare labels and invalid label combinations, all issues and PRs are validated by an automated validation workflow after creation.

### How It Works

1. **Trigger:** Validation runs on issue/PR `opened`, `edited`, `labeled`, `unlabeled`, and PR `synchronize` events (after creation)
2. **Script:** `scripts/validation/validate-labels-before-creation.cjs` enforces 5 rules
3. **Workflow:** `.github/workflows/validate-issue-labels.yml` posts guidance on failure
Comment on lines +338 to +339

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

fd -t f 'validate-labels-before-creation\.cjs$' .
rg -n 'validate-labels-before-creation\.cjs' docs .github scripts 2>/dev/null || true

Repository: lightspeedwp/.github

Length of output: 2556

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo '--- workflow files with validation label references ---'
fd -t f 'validate-issue-labels\.ya?ml$|validate-labels-before-creation\.cjs$' . | sed 's#^\./##' | sort

echo '--- validate-issue-labels workflow excerpts ---'
for f in $(fd -t f 'validate-issue-labels\.ya?ml$' .); do
  echo "### $f"
  rg -n -C 4 'validate-labels-before-creation|validation|scripts/validation|node ' "$f" || true
done

echo '--- repository root candidates for expected paths ---'
for p in scripts/validation/validate-labels-before-creation.cjs .github/scripts/validation/validate-labels-before-creation.cjs .github/workflows/validate-issue-labels.yml; do
  if [ -f "$p" ]; then
    echo "EXISTS $p"
  else
    echo "MISSING $p"
  fi
done

Repository: lightspeedwp/.github

Length of output: 2248

Align the validation script path with the checked-in location.

validate-labels-before-creation.cjs is checked in at scripts/validation/validate-labels-before-creation.cjs, but workflows/validate-issue-labels.yml still calls .github/scripts/validation/validate-labels-before-creation.cjs. Move the script to the workflow path or update the workflow documentation to reference the current path.

🧰 Tools
🪛 LanguageTool

[uncategorized] ~339-~339: The official name of this software platform is spelled with a capital “H”.
Context: ....cjsenforces 5 rules 3. **Workflow:**.github/workflows/validate-issue-labels.yml` po...

(GITHUB)

📍 Affects 3 files
  • docs/LABELING.md#L338-L339 (this comment)
  • docs/LABELING.md#L469-L473
  • docs/LABELING_EXAMPLES.md#L464-L472
  • docs/LABELING_FAQ.md#L299-L305
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/LABELING.md` around lines 338 - 339, Update the workflow documentation
references to use the checked-in script path
scripts/validation/validate-labels-before-creation.cjs instead of
.github/scripts/validation/validate-labels-before-creation.cjs. Apply this to
docs/LABELING.md lines 338-339 and 469-473, docs/LABELING_EXAMPLES.md lines
464-472, and docs/LABELING_FAQ.md lines 299-305; no script move is needed.

4. **Outcome:** Valid labels pass silently; invalid labels receive a helpful error comment with corrected examples

### Validation Rules

| Rule | Requirement | Example ✅ | Example ❌ |
|------|-------------|-----------|-----------|
| **Rule 1: Existence** | Label must exist in canonical set (.github/labels.yml) | `type:bug` | `type:bugfix` |
| **Rule 2: Family Prefix** | Label must have family prefix (type:, status:, priority:, etc.) | `status:needs-triage` | `needs-triage` |
| **Rule 3: One-hot per Family** | Only one label per family (except meta:, comp:, lang:) | `type:bug`, `status:ready` | `type:bug`, `type:feature` |
| **Rule 4: Required Families** | All issues/PRs must have a `type:*` label | `type:documentation` | (no type:) |
| **Rule 5: Warnings** | Common mistakes flagged with suggestions | `type:bug` (after correction) | `bug` (bare, triggers warning) |

### Validation in Practice

**✅ Valid labels** (all rules pass):

```
type:bug
status:needs-triage
priority:critical
area:ci
meta:needs-changelog
```

**❌ Invalid labels** (caught by validation):

```
bug # ❌ Rule 2 (missing type: prefix)
feature # ❌ Rule 2 (missing type: prefix)
type:bug, type:feature # ❌ Rule 3 (multiple type: labels)
status:ready # ❌ Rule 4 (missing type:*)
urgent # ❌ Rule 2 (bare label, not in canonical set)
```

### Error Messages & Fixes

When validation fails, the workflow posts a comment with:

1. **Issue description** — What's wrong and why
2. **Valid examples** — Copy-paste ready label combinations
3. **Documentation link** — This page and other resources
4. **Canonical label reference** — Link to `.github/labels.yml` (158 total labels)

**Example error comment:**

> **⚠️ Label Validation Failed**
>
> **Labels on this issue:** `bug, feature, ci`
>
> **Issues:**
>
> - Label 'bug' missing required family prefix. Use one of: type:, status:, priority:, area:, meta:, release:, lang:, env:, compat:, comp:
> - Label 'feature' missing required family prefix. Use one of: type:, status:, priority:, area:, meta:, release:, lang:, env:, compat:, comp:
> - Label 'ci' missing required family prefix. Use one of: type:, status:, priority:, area:, meta:, release:, lang:, env:, compat:, comp:
>
> **How to fix:**
>
> 1. Use only canonical labels with family prefixes: `type:`, `status:`, `priority:`, `area:`, `meta:`, etc.
> 2. Check the [canonical labels](https://github.com/lightspeedwp/.github/blob/develop/.github/labels.yml) (158 total)
> 3. Each family allows ONE label (except `meta:` and `comp:` which allow multiple)
> 4. All issues/PRs must have a `type:*` label
Comment on lines +397 to +400

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Locate validator and docs files"
git ls-files | rg '(^|/)(validate|validator|labelling|labelling-?agent|labels|LABELING|LABELING_EXAMPLES|LABELING_FAQ)\.(md|mdx|py|js|ts|yml)$' || true

echo
echo "Search validator-related terminology"
rg -n "REQUIRED_FAMILIES|allowed-multiple|multiple.*area|compat|lang|status|priority|changelog|enforce" -S --glob '!node_modules' --glob '!build' --glob '!dist' .

Repository: lightspeedwp/.github

Length of output: 50377

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Candidate repo root files"
git ls-files | sed -n '1,200p'

echo
echo "Exact markdown docs and likely validator files"
git ls-files '.github/**' 'scripts/**' 'docs/LABELING*.md' | sort | sed -n '1,200p'

echo
echo "Target doc snippets"
for f in docs/LABELING.md docs/LABELING_EXAMPLES.md docs/LABELING_FAQ.md; do
  if [ -f "$f" ]; then
    echo "--- $f ($(wc -l < "$f") lines) ---"
    sed -n '380,435p' "$f" 2>/dev/null || true
    sed -n '410,460p' "$f" 2>/dev/null || true
    sed -n '80,165p' "$f" 2>/dev/null || true
  fi
done

echo
echo "Direct search for label validator symbols"
rg -n '\bREQUIRED_FAMILIES\b|\bALLOWED_MULTI_FAMILY\b|\blanguage|compat|area:|status:|priority:|changelog\b|\blabel'.*validator|validator.*label -S \
  .github scripts docs labels.yml 2>/dev/null || true

Repository: lightspeedwp/.github

Length of output: 32517

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Validate repo contains docs/LABELING*.md:"
git ls-files 'docs/LABELING*.md'

echo
echo "Exact lines around current target snippets:"
sed -n '394,402p' docs/LABELING.md
echo "---"
sed -n '421,431p' docs/LABELING.md
echo "---"
sed -n '421,454p' docs/LABELING_EXAMPLES.md
echo "---"
sed -n '89,99p' docs/LABELING_FAQ.md
echo "---"
sed -n '114,128p' docs/LABELING_FAQ.md
echo "---"
sed -n '142,159p' docs/LABELING_FAQ.md

echo
echo "Locate validator implementation files in scripts/.github:"
git ls-files scripts .github | rg 'label|validate|validator|labelling|labels' || true

echo
echo "Search validator definitions without broad context:"
rg -n "REQUIRED_FAMILIES|REQUIRED.*FAMILY|ALLOWED_MULTI|MULTI|allow.*multiple|one-hot|lang:|area:|compat:|status|priority|changelog|type:" \
  --glob '!node_modules' \
  --glob '!build' \
  --glob '!dist' \
  --glob '!docs/**' \
  --glob '!agents/**' \
  --glob '!saved*' \
  --glob '!SAVED_REPLIES/**' \
  .github scripts 2>/dev/null \
  || true

Repository: lightspeedwp/.github

Length of output: 50377

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Tracked target docs:"
git ls-files 'docs/LABELING.md' 'docs/LABELING_EXAMPLES.md' 'docs/LABELING_FAQ.md' || true

echo
echo "Extract only target doc ranges:"
awk '
  FNR >= 394 && FNR <= 400 { print FILENAME ":" FNR ":" $0 }
  FNR >= 421 && FNR <= 430 { print FILENAME ":" FNR ":" $0 }
' docs/LABELING.md || true
awk '
  FNR >= 421 && FNR <= 454 { print FILENAME ":" FNR ":" $0 }
' docs/LABELING_EXAMPLES.md || true
awk '
  FNR >= 89 && FNR <= 99 { print FILENAME ":" FNR ":" $0 }
  FNR >= 114 && FNR <= 128 { print FILENAME ":" FNR ":" $0 }
  FNR >= 142 && FNR <= 159 { print FILENAME ":" FNR ":" $0 }
' docs/LABELING_FAQ.md || true

echo
echo "Tracked plausible validator implementation files:"
git ls-files | rg '(^scripts/.*label|scripts/.*validator|scripts/.*validation.*label|labeler|labels|labeling|labelling|validation|validator)\.(js|ts|cjs|mjs)$|^\.github/.*label|scripts/.*/label' || true

echo
echo "Search non-document/non-agent code for exact constants or multi-family logic:"
rg -n "\bREQUIRED_FAMILIES\b|\bALLOWED_MULTI_FAMILY\b|\bMULTI_FAMILY\b|\blang:\b|\bcompat:\b|\barea:\b|\bevery.*family|only one|multiple.*famil|family count|one-hot" \
  --glob '!*.md' \
  --glob '!*.mdx' \
  --glob '!node_modules/**' \
  --glob '!build/**' \
  --glob '!dist/**' \
  --glob '!docs/**' \
  --glob '!agents/**' \
  . .github scripts 2>/dev/null | sed -n '1,240p' || true

echo
echo "Extract labeler and enforcement config snippets:"
for f in .github/workflows/labeling.yml .github/label-governance-policy.yml .github/labeler.yml; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    rg -n "meta:needs-changelog|meta:no-changelog|priority:|status:|type:|required|enforce|metadata-governance|changelog" "$f" || true
  fi
done

Repository: lightspeedwp/.github

Length of output: 35480

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Tracked validator source:"
find scripts tests .github/scripts -maxdepth 4 -type f \
  \( -name '*label*validate*' -o -name '*validate*label*' -o -name 'label-validation*' \) | sort

echo
echo "validator source relevant sections:"
if [ -f scripts/validation/validate-labels-before-creation.cjs ]; then
  sed -n '1,190p' scripts/validation/validate-labels-before-creation.cjs
  echo
  echo "tests around required families/multi exceptions"
  sed -n '1,290p' scripts/validation/__tests__/validate-labels-before-creation.test.cjs
fi

echo
echo "All one-hot/multi exception references in source (not broad docs):**"
rg -n "one-hot|multi|multiple|meta:|comp:|lang:|area:|compat:|env:|release:|status:|priority:|type:" \
  scripts/validation scripts/agents/includes scripts/agents tests .github/workflows/validate-issue-labels.yml 2>/dev/null | sed -n '1,260p' || true

Repository: lightspeedwp/.github

Length of output: 38177

Align the label guidance with the enforced validator policy.

The only enforced requirement is type:*; status, priority, area/comp, changelog, and release labels are good labelling policy or automation defaults. Keep those as guidance, and update every exception list to allow only meta:, comp:, and lang: for multiple labels. Remove compat:, area:, and env: from multiple-label examples, and make the priority/normal wording consistent so users are not told the required priority:* label can be omitted.

📍 Affects 3 files
  • docs/LABELING.md#L397-L400 (this comment)
  • docs/LABELING.md#L421-L430
  • docs/LABELING_EXAMPLES.md#L421-L454
  • docs/LABELING_FAQ.md#L89-L99
  • docs/LABELING_FAQ.md#L114-L128
  • docs/LABELING_FAQ.md#L142-L159
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/LABELING.md` around lines 397 - 400, Align the labeling documentation
with the validator policy: require only a type:* label, while presenting status,
priority, area/comp, changelog, and release labels as guidance or automation
defaults. In docs/LABELING.md lines 397-400 and 421-430,
docs/LABELING_EXAMPLES.md lines 421-454, and docs/LABELING_FAQ.md lines 89-99,
114-128, and 142-159, update every multiple-label exception to allow only meta:,
comp:, and lang:, remove compat:, area:, and env:, and make priority/normal
wording consistently indicate whether priority:* is required.

>
> **Valid Examples:**
>
> ```
> type:bug, status:needs-triage, priority:critical, area:ci
> type:feature, priority:normal, area:documentation
> type:task, status:ready, area:automation
> ```

### Re-running Validation

If you receive a validation error:

1. **Remove bare labels** (bug, feature, urgent, ci, docs, etc.)
2. **Add family prefix** (type:bug, area:ci, priority:urgent → priority:critical, etc.)
3. **Edit the issue/PR** to apply corrected labels
4. **Validation re-runs automatically** when labels change

---

## Best Practices

1. **Keep exactly one `status:*` and `priority:*`** on every issue/PR.
Expand All @@ -342,30 +433,66 @@ All automation reads from these files; there is no hardcoded label logic in agen

## Troubleshooting

**Missing or incorrect labels?**
### Label Validation Errors

**"Label 'X' missing required family prefix"**

- The label isn't prefixed (e.g., `bug` instead of `type:bug`)
- **Fix:** Edit the issue/PR and apply canonical labels with family prefix (type:, status:, priority:, area:, etc.)
- **Reference:** See [Validation Rules](#validation-rules) above and `.github/labels.yml` for all 158 canonical labels

**"Label 'X' not found in canonical set"**

- The label doesn't exist in `.github/labels.yml` (typo or custom label)
- **Fix:** Use a canonical label from the 158-label set; custom labels are not allowed
- **Reference:** [Canonical labels](https://github.com/lightspeedwp/.github/blob/develop/.github/labels.yml)

**"Multiple labels from family 'Y' found: [a, b]"**

- You applied more than one label from the same family (e.g., `type:bug` AND `type:feature`)
- **Fix:** Keep only one label per family (except meta:, comp:, lang: which allow multiples)
- **Reference:** [Validation Rules](#validation-rules) — one-hot per family

**"Missing required 'type:\*' label for classification"**

- The issue/PR has no `type:*` label (e.g., missing `type:bug`, `type:feature`, `type:task`)
- **Fix:** Add a `type:*` label that matches the work type
- **Reference:** [Type Labels](#type-labels-type) — choose the correct type for your issue/PR

**Validation failed but I don't see a comment**

- The issue/PR might not have raised a validation event yet
- **Fix:** Edit the issue/PR and save (even without label changes) to trigger validation re-run
- **Alternative:** Remove and re-apply labels to trigger the workflow

### Missing or Incorrect Labels?

- Check `.github/labels.yml` for missing/typo entries
- Verify branch prefix or file pattern matches in `.github/labeler.yml`
- Run `node scripts/agents/includes/check-template-labels.js` to validate issue/PR templates
- Use `scripts/validation/validate-labels-before-creation.cjs` to test labels locally

**Label not applied as expected?**
### Label Not Applied as Expected?

- Review labeler workflow logs in the PR/issue activity
- Check if the labelling workflow is enabled and up-to-date
- Verify the labelling agent has access to read/write labels
- Run the pre-creation validation script to check if labels are canonical

**Want to add a new label or modify rules?**
### Want to Add a New Label or Modify Rules?

1. Update `.github/labels.yml` with the new canonical definition
2. Update `.github/labeler.yml` if you need automatic application rules
3. Update this documentation to describe the new label
4. Create a PR and reference this issue #636
4. Create a PR and reference issue #636
5. **Note:** New labels must follow the family-prefix naming convention (e.g., `area:newarea`, not `newarea`)

**Non-canonical labels appearing?**
### Non-Canonical Labels Appearing?

- The labelling agent automatically migrates old labels to canonical equivalents
- If a label persists, check `.github/label-governance-policy.yml` for exceptions
- Open an issue if a label should be migrated or removed
- Run `scripts/validation/validate-labels-before-creation.cjs` to test label canonicality

---

Expand Down
Loading
Loading