diff --git a/.github/projects/active/label-prefix-enforcement-2026-08-05/IMPLEMENTATION_GUIDE.md b/.github/projects/active/label-prefix-enforcement-2026-08-05/IMPLEMENTATION_GUIDE.md new file mode 100644 index 000000000..99723ae6a --- /dev/null +++ b/.github/projects/active/label-prefix-enforcement-2026-08-05/IMPLEMENTATION_GUIDE.md @@ -0,0 +1,497 @@ +--- +title: "Label Prefix Enforcement Implementation Guide" +description: "Step-by-step implementation procedures for all 5 phases" +file_type: "documentation" +version: "1.0.0" +created_date: "2026-08-07" +last_updated: "2026-08-07" +author: "Claude Code" +maintainer: "LightSpeed Team" +domain: "governance" +status: "active" +tags: + - implementation + - procedures + - phase-execution +--- + +# Implementation Guide: Label Prefix Enforcement Project + +**Document Purpose**: Detailed step-by-step procedures for executing all 5 phases +**Target Audience**: Project leads, DevOps, Governance team +**Success Measure**: All procedures documented and executable + +--- + +## Phase 1: Stop New Violations (TODAY) — Detailed Procedures + +### Phase 1.1: Update CLAUDE.md with Label Rules + +**Timeline**: 30 minutes +**Owner**: Governance Lead +**Reviewers**: Engineering Lead, DevOps + +#### Step 1: Prepare the Content + +**File to modify**: `CLAUDE.md` (root) +**Location**: Add new section "Label Creation Rules (CRITICAL)" after "Key Conventions" section + +**Content Block**: + +```markdown +## Label Creation Rules (CRITICAL) + +When creating issues or PRs programmatically (via CLI, API, or workflow), **ALL labels MUST be from the canonical set in `.github/labels.yml` with their required family prefix**. + +### Valid Label Examples (Prefixed) + +- `type:bug`, `type:feature`, `type:task`, `type:documentation` +- `status:needs-triage`, `status:in-progress`, `status:done` +- `priority:critical`, `priority:important`, `priority:normal` +- `area:ci`, `area:docs`, `area:security`, `area:labels` +- `meta:needs-changelog`, `meta:has-pr` + +### INVALID Label Examples (Bare — DO NOT USE) + +- ❌ `bug` — use `type:bug` +- ❌ `feature` — use `type:feature` +- ❌ `urgent` — use `priority:critical` +- ❌ `ci` — use `area:ci` + +### Reference + +- Source of truth: `.github/labels.yml` (158 canonical labels) +- Labeling guide: `docs/LABELING.md` +- Label taxonomy: `docs/LABEL_STRATEGY.md` +- Root cause analysis: `.github/projects/active/label-prefix-audit-2026-08-05/` +``` + +#### Step 2: Make the Edit + +1. Open `CLAUDE.md` in editor +2. Find section: "Key Conventions" +3. Add new section immediately after +4. Paste the content block above +5. Verify formatting (markdown, bullet points, code blocks) + +#### Step 3: Validate + +```bash +# Run markdown linter +npm run lint:md CLAUDE.md + +# Verify YAML frontmatter (if present) +npm run validate:frontmatter CLAUDE.md +``` + +#### Step 4: Commit & Push + +```bash +git add CLAUDE.md +git commit -m "docs: Add label creation rules to CLAUDE.md (Phase 1 enforcement)" +git push origin $(git rev-parse --abbrev-ref HEAD) +``` + +#### Step 5: Verification Checklist + +- [ ] Section added to CLAUDE.md +- [ ] Markdown linting passes +- [ ] Frontmatter validates (if applicable) +- [ ] Commit message follows convention +- [ ] Push successful +- [ ] PR created/updated with this change + +--- + +### Phase 1.2: Update AGENTS.md with Governance Section + +**Timeline**: 30 minutes +**Owner**: Governance Lead +**Reviewers**: Engineering Lead, Agent Owners + +#### Step 1: Locate and Prepare Content + +**File to modify**: `AGENTS.md` (root) +**Location**: Add new subsection under "Governance & Standards" section (if exists), or create new section + +**Content Block**: + +```markdown +### Label Creation for Programmatic Issue Creation + +When your code creates issues via `gh issue create` or GitHub API: + +1. **Always validate labels against canonical set** (`.github/labels.yml`) +2. **Use required family prefixes** (e.g., `type:bug`, not `bug`) +3. **Verify one-hot per family** (except `meta:` and `comp:` which allow multiples) +4. **Check reference** in `docs/LABELING.md` if unsure + +#### Labeling Agent Implementation + +The labeling agent in `.github/scripts/agents/labeling.agent.js` is the authoritative implementation. Use this as template for any custom label creation. + +**Correct pattern**: +```yaml +labels: + - type:bug # Problem classification + - status:needs-triage # Workflow state + - priority:normal # Urgency + - area:ci # Domain/component +``` + +**Incorrect pattern** (DO NOT USE): + +```yaml +labels: + - bug # Bare label (missing type: prefix) + - needs-triage # Bare label (missing status: prefix) + - urgent # Bare label (missing priority: prefix) +``` + +#### Enforcement & Validation + +- **Pre-creation**: Validate labels before calling `gh issue create` +- **Workflow-level**: GitHub Actions workflows validate labels on issue events +- **Governance**: CLAUDE.md documents rules; AGENTS.md documents implementation + +**Reference**: `.github/scripts/validation/validate-labels-before-creation.cjs` + +``` + +#### Step 2: Make the Edit + +1. Open `AGENTS.md` in editor +2. Find appropriate section (Governance, Standards, or AI Governance) +3. Add subsection with content block above +4. Verify indentation and code block formatting + +#### Step 3: Validate + +```bash +# Markdown linting +npm run lint:md AGENTS.md + +# Verify all inline code blocks have proper syntax highlighting +``` + +#### Step 4: Commit & Push + +```bash +git add AGENTS.md +git commit -m "docs: Add label creation governance to AGENTS.md (Phase 1 enforcement)" +git push origin $(git rev-parse --abbrev-ref HEAD) +``` + +#### Step 5: Verification Checklist + +- [ ] Subsection added to AGENTS.md +- [ ] Code blocks properly formatted +- [ ] Markdown linting passes +- [ ] Commit message follows convention +- [ ] Push successful + +--- + +### Phase 1.3: Delete Defective Code + +**Timeline**: 15 minutes +**Owner**: DevOps/Engineering +**Reviewers**: Security, Code Quality + +#### Step 1: Identify the Defective File + +**Location**: `scripts/agents/includes/labeling-agent.js` +**Issue**: Creates bare labels without family prefixes +**Status**: Superseded by `.github/scripts/agents/labeling.agent.js` + +#### Step 2: Verify Supersession + +Before deletion, confirm the correct implementation exists: + +```bash +# Check both files exist +test -f scripts/agents/includes/labeling-agent.js && echo "Defective file exists" +test -f .github/scripts/agents/labeling.agent.js && echo "Correct implementation exists" + +# Check for any imports/references +grep -r "labeling-agent.js" --include="*.js" --include="*.yml" --include="*.yaml" .github/ || echo "No references found" +grep -r "scripts/agents/includes/labeling-agent" --include="*.js" --include="*.yml" --include="*.yaml" . || echo "No references found" +``` + +#### Step 3: Create Removal Commit + +```bash +git rm scripts/agents/includes/labeling-agent.js +git commit -m "fix: Remove defective labeling-agent.js creating bare labels (Phase 1 enforcement)" +git push origin $(git rev-parse --abbrev-ref HEAD) +``` + +#### Step 4: Verification + +```bash +# Verify file is removed +! test -f scripts/agents/includes/labeling-agent.js && echo "✓ File removed" + +# Check git history +git log --oneline -5 | grep "Remove defective" +``` + +#### Step 5: Verification Checklist + +- [ ] Correct implementation verified (`.github/scripts/agents/labeling.agent.js` exists) +- [ ] No references to defective file found +- [ ] File removed via `git rm` +- [ ] Commit created and pushed +- [ ] File no longer exists in working directory + +--- + +### Phase 1.4: Verification & Sign-Off + +**Timeline**: 15 minutes +**Owner**: Governance Lead +**Sign-off**: Engineering Lead + DevOps + +#### Verification Checklist + +- [ ] CLAUDE.md updated with label rules section +- [ ] AGENTS.md updated with implementation guidance +- [ ] Defective code file deleted +- [ ] All commits pushed +- [ ] No new violations created since Phase 1 start + +#### Success Criteria for Phase 1 + +- **Zero new non-canonical labels created** (audit to confirm) +- **Documentation updated** (CLAUDE.md + AGENTS.md complete) +- **Code cleaned** (defective file removed) +- **Governance enforced** (rules documented in AI instructions) + +--- + +## Phase 2: Fix Existing Issues (24–48 hours) + +### Phase 2.1: Bulk Remediation Strategy + +**Timeline**: 3–5 hours +**Owner**: DevOps/Automation +**Impact**: ~100 issues in #1500–#1600 range + +#### Step 1: Create Remediation Script + +**Script purpose**: Bulk update issue labels from bare to canonical form + +**Input**: List of issues with bare labels +**Output**: Updated issues with canonical labels +**Rollback**: Full git history preserved; can revert via git if needed + +#### Step 2: Run Validation First + +```bash +# Pre-remediation audit +npm run audit:labels -- --mode dry-run + +# Expected output: List of issues + proposed changes +# Review all proposed changes before proceeding +``` + +#### Step 3: Execute Remediation + +```bash +# Run with confirmation prompts +npm run remediate:labels -- --mode interactive + +# Or run directly (use only after manual review) +npm run remediate:labels -- --mode direct --issues-range 1500-1600 +``` + +#### Step 4: Post-Remediation Audit + +```bash +# Run full audit +npm run audit:labels + +# Expected: 0 violations +# Document results in Phase 2 completion report +``` + +### Phase 2.2: Manual Review for Edge Cases + +**Timeline**: 1–2 hours +**Owner**: Engineering Team Lead + +#### Step 1: Identify Edge Cases + +Issues that automated script cannot handle: + +- Custom labels not in canonical set +- Issues needing reclassification +- Complex multi-label scenarios + +#### Step 2: Manual Updates + +For each edge case issue: + +1. Open issue in GitHub +2. Review current labels +3. Determine correct canonical labels +4. Update labels manually +5. Document change in spreadsheet + +#### Step 3: Verification + +```bash +# Re-run audit after manual fixes +npm run audit:labels + +# Should report 0 violations +``` + +### Phase 2.3: Verification & Reporting + +- [ ] All ~100 issues remediated +- [ ] Re-audit confirms 0 violations +- [ ] No automation failures +- [ ] Edge cases manually reviewed +- [ ] Completion report generated + +--- + +## Phase 3: Enforce Validation in Workflows (3–5 days) + +### Phase 3.1: Pre-Creation Validation Integration + +**Timeline**: 2 hours +**Owner**: DevOps + +#### Implementation Steps + +1. **Review validation script**: `.github/scripts/validation/validate-labels-before-creation.cjs` +2. **Integrate into issue creation workflows**: `.github/workflows/issue-*.yml` +3. **Test with dry-run**: Create test issue and verify validation +4. **Document**: Update workflow documentation + +#### Validation Rules Checklist + +- [ ] Label must exist in `.github/labels.yml` +- [ ] Label must include family prefix (no bare labels) +- [ ] One-hot per family (except meta, comp) +- [ ] Type label always required + +### Phase 3.2: Workflow-Level Validation + +**Timeline**: 2–3 hours +**Owner**: DevOps + +1. Add validation job to issue event workflows +2. Add validation checks to PR labeling workflows +3. Add documentation for developers +4. Test edge cases + +### Phase 3.3: Error Handling & Rollback + +- Invalid labels rejected +- Helpful error messages provided +- Workflow logs document violations +- Can re-run after label fixes + +--- + +## Phase 4: Documentation Updates (5–7 days) + +### Phase 4.1: Expand LABELING.md + +Add: + +- Troubleshooting guide +- Common mistakes (bare labels) +- Examples per family +- Scripts reference + +### Phase 4.2: Create FAQ Document + +Questions to address: + +- What are prefixed labels? +- How do I add a new label? +- What if I see an error? +- Where do I find the canonical list? + +### Phase 4.3: Update README References + +Ensure all READMEs reference new governance rules. + +--- + +## Phase 5: Team Training & Communication (Ongoing) + +### Phase 5.1: Slack Announcement + +Post in #engineering and #governance: + +- What changed and why +- Where to find documentation +- How to report issues + +### Phase 5.2: Team Meeting + +- Demonstrate the system +- Answer questions +- Gather feedback + +### Phase 5.3: Monitoring + +- Weekly audits for 1 month +- Track new violations +- Adjust documentation as needed + +--- + +## Rollback Procedures + +### If Phase 1 needs to rollback + +```bash +git revert +``` + +### If Phase 2 issues need fixing + +```bash +# Revert remediation +git revert + +# Fix manually +# Then re-apply with corrections +``` + +### If Phase 3 validation is too strict + +1. Review error cases +2. Update validation rules +3. Re-deploy workflows +4. Test edge cases + +--- + +## Success Metrics by Phase + +| Phase | Metric | Target | +|-------|--------|--------| +| 1 | New violations created | 0 | +| 2 | Issues fixed | 100 | +| 3 | Validation failures handled | 100% | +| 4 | Documentation coverage | 100% | +| 5 | Team understanding | 90%+ | + +--- + +## Contact & Escalation + +- **Phase Lead**: [Name] +- **Technical Contact**: [Name] +- **Escalation**: [Process] + +--- + +*Built with ☕ and 🚀 by LightSpeedWP Governance Team* diff --git a/.github/projects/active/label-prefix-enforcement-2026-08-05/README.md b/.github/projects/active/label-prefix-enforcement-2026-08-05/README.md index 19113c794..ae11ffe58 100644 --- a/.github/projects/active/label-prefix-enforcement-2026-08-05/README.md +++ b/.github/projects/active/label-prefix-enforcement-2026-08-05/README.md @@ -1,10 +1,10 @@ --- title: "Label Prefix Enforcement Project" description: "Remediation and permanent governance for label prefix violations" -file_type: "project-index" -version: "1.0.0" +file_type: "readme" +version: "1.1.0" created_date: "2026-08-05" -updated_date: "2026-08-05" +last_updated: "2026-08-07" author: "Claude Code Audit" maintainer: "LightSpeed Team" domain: "governance" @@ -38,8 +38,10 @@ Comprehensive remediation of ~100 issues (#1500–#1600) with non-canonical labe | Document | Purpose | |----------|---------| | [ACTION_PLAN.md](./ACTION_PLAN.md) | Complete 5-phase remediation roadmap | +| [IMPLEMENTATION_GUIDE.md](./IMPLEMENTATION_GUIDE.md) | **NEW** — Step-by-step procedures for all phases | +| [RISK_MITIGATION.md](./RISK_MITIGATION.md) | **NEW** — Risk assessment & contingency procedures | +| [TESTING_VALIDATION.md](./TESTING_VALIDATION.md) | **NEW** — Testing & validation procedures | | [OPENSPEC_RFC_REFINED.md](./OPENSPEC_RFC_REFINED.md) | Refined OpenSpec RFC incorporating audit results (2.0) | -| [AUDIT_SUMMARY.md](./AUDIT_SUMMARY.md) | Executive summary of audit findings | --- diff --git a/.github/projects/active/label-prefix-enforcement-2026-08-05/RISK_MITIGATION.md b/.github/projects/active/label-prefix-enforcement-2026-08-05/RISK_MITIGATION.md new file mode 100644 index 000000000..79903e17b --- /dev/null +++ b/.github/projects/active/label-prefix-enforcement-2026-08-05/RISK_MITIGATION.md @@ -0,0 +1,517 @@ +--- +title: "Label Prefix Enforcement Risk Assessment & Mitigation" +description: "Comprehensive risk analysis and mitigation strategies" +file_type: "documentation" +version: "1.0.0" +created_date: "2026-08-07" +last_updated: "2026-08-07" +author: "Claude Code" +maintainer: "LightSpeed Team" +domain: "governance" +status: "active" +tags: + - risk-management + - mitigation + - contingency +--- + +# Risk Assessment & Mitigation Plan + +**Document Purpose**: Identify and mitigate risks across all 5 phases +**Target Audience**: Project leads, Risk owners, Stakeholders +**Review Frequency**: Weekly during implementation + +--- + +## Executive Risk Summary + +| Risk Level | Count | Trend | Mitigation Status | +|-----------|-------|-------|------------------| +| 🔴 Critical | 2 | Decreasing | Mitigated | +| 🟠 High | 4 | Stable | Planned | +| 🟡 Medium | 6 | N/A | Preventive | +| 🟢 Low | 8 | N/A | Documented | + +**Overall Project Risk**: 🟢 **LOW** (non-breaking, reversible, well-planned) + +--- + +## Critical Risks (🔴) + +### Risk 1: Automation Creates More Bare Labels During Fix + +**Description**: While fixing existing issues, new code creates more bare labels +**Probability**: Medium (code still vulnerable) +**Impact**: High (defeats the purpose of Phase 2) +**Detection**: Audit would show increase instead of decrease + +#### Mitigation Strategies + +**Primary**: Complete Phase 1 BEFORE Phase 2 + +- Stop defective code (Phase 1.3) must complete first +- Prevents new violations while fixing old ones + +**Secondary**: Run validation immediately after Phase 1 + +```bash +# After Phase 1 completion, audit ASAP +npm run audit:labels + +# Should show 0 NEW violations created since Phase 1 +``` + +**Tertiary**: Automated prevention + +- Phase 3 validation prevents new bare labels from reaching GitHub +- Even if code tries to create bare labels, workflow rejects them + +#### Responsibility + +- **Prevention**: DevOps (Phase 1.3 execution) +- **Detection**: Governance Lead (audit Phase 1→2) +- **Response**: Immediately halt Phase 2 and investigate + +#### Success Criteria + +- ✅ Zero new violations between Phase 1 completion and Phase 2 completion +- ✅ Audit shows declining trend (100 → 0) +- ✅ Re-audit confirms 0 violations exist + +--- + +### Risk 2: Remediation Script Breaks in Complex Label Scenarios + +**Description**: Automated bulk remediation fails or creates invalid label combinations +**Probability**: Medium (100 issues, various states) +**Impact**: High (corrupt data, manual fix needed) +**Detection**: Remediation script errors, validation failures + +#### Mitigation Strategies + +**Primary**: Dry-run before real execution + +```bash +# Phase 2.1 Step 2: Always run dry-run first +npm run audit:labels -- --mode dry-run + +# Review ALL proposed changes +# Get approval before proceeding to direct mode +``` + +**Secondary**: Manual review of edge cases + +- Phase 2.2 dedicates 1–2 hours to edge case review +- Automated script handles 90%+, humans handle remainder + +**Tertiary**: Rollback capability + +```bash +# If remediation causes problems: +git revert + +# Investigate + fix + re-attempt +``` + +**Quaternary**: Test with subset first + +- Run remediation on first 10 issues +- Validate results manually +- Then run on full batch + +#### Responsibility + +- **Prevention**: DevOps (dry-run, test subset) +- **Detection**: QA (manual review of results) +- **Response**: Rollback + investigate + re-plan + +#### Success Criteria + +- ✅ Dry-run passes with no errors +- ✅ Proposed changes reviewed and approved +- ✅ Subset test (10 issues) succeeds +- ✅ Full batch remediation succeeds +- ✅ Post-remediation audit: 0 violations + +--- + +## High Risks (🟠) + +### Risk 3: Validation Too Strict, Blocks Legitimate Issues + +**Description**: Phase 3 validation rules are overly restrictive, reject valid label combinations +**Probability**: Medium (rules need tuning) +**Impact**: High (workflows blocked, team frustrated) + +#### Mitigation Strategies + +**1. Comprehensive Rule Review** (Phase 3 Planning) + +- Review all validation rules with team +- Document legitimate label combinations +- Test edge cases before deployment + +**2. Gradual Rollout** + +- Deploy validation in "warn" mode first +- Log violations without blocking +- Monitor for 24 hours +- Then enable "block" mode + +**3. Clear Error Messages** + +- Validation errors must be understandable +- Include link to `docs/LABELING.md` +- Show corrected label example + +**4. Rapid Response Team** + +- Designated owner for validation issues +- 1-hour response time for reports +- Quick rule adjustments/rollback + +#### Responsibility + +- **Prevention**: DevOps + Engineering Lead (rule design) +- **Detection**: Team feedback + monitoring +- **Response**: Quick adjustment or rollback + +#### Success Criteria + +- ✅ Validation rules documented + approved +- ✅ Edge cases tested and pass +- ✅ Warn mode shows 0 false positives after 24h +- ✅ Team reports no blocking issues + +--- + +### Risk 4: Documentation Doesn't Match Implementation + +**Description**: CLAUDE.md, AGENTS.md, validation rules, and actual code are out of sync +**Probability**: High (common in multi-phase projects) +**Impact**: High (team confusion, violations continue) + +#### Mitigation Strategies + +**1. Single Source of Truth** + +- `.github/labels.yml` is the ONLY source of truth +- All documentation references this file +- Validation code reads from this file +- No hardcoded label lists anywhere + +**2. Regular Sync Checks** + +```bash +# Phase 4.1: Automated verification +npm run validate:label-references + +# Checks: +# - CLAUDE.md examples exist in .github/labels.yml +# - AGENTS.md examples exist in .github/labels.yml +# - Validation script uses .github/labels.yml +# - docs/LABELING.md examples match +``` + +**3. Documentation Review Process** + +- All documentation PRs reviewed for label accuracy +- Link to `.github/labels.yml` in every doc +- Use examples pulled from actual canonical labels + +**4. Automated Sync Tools** + +- Generate documentation from `.github/labels.yml` where possible +- Maintain manual sections separate +- Version control everything + +#### Responsibility + +- **Prevention**: Documentation owner (good practices) +- **Detection**: Automated checks (Phase 4) +- **Response**: Update documentation + +#### Success Criteria + +- ✅ Single source of truth established (`.github/labels.yml`) +- ✅ All documentation references this file +- ✅ Automated sync checks pass +- ✅ No documentation drift detected + +--- + +### Risk 5: Team Adoption & Compliance + +**Description**: Team ignores new rules, creates bare labels anyway +**Probability**: Low→Medium (depends on enforcement) +**Impact**: High (governance failure) + +#### Mitigation Strategies + +**1. Phase 5 Training** (Ongoing) + +- Explain the problem + solution +- Show how to use correct labels +- Q&A session +- Reference documentation + +**2. Automated Enforcement** + +- Phase 3 validation prevents bare labels at source +- No manual effort needed to enforce rules +- Even if team forgets, system rejects invalid labels + +**3. Clear Communication** + +- Slack announcement with links to docs +- Highlight in team meeting +- Include in onboarding docs + +**4. Monitoring & Feedback** + +- Track compliance weekly +- Report metrics to team +- Celebrate reaching 0 violations milestone + +**5. Make It Easy** + +- Provide copy-paste label templates +- Update PR/issue templates with correct labels +- Make correct way the path of least resistance + +#### Responsibility + +- **Prevention**: Governance Lead (clear communication) +- **Detection**: Weekly audits +- **Response**: Training + support + +#### Success Criteria + +- ✅ 90%+ team awareness within 1 week +- ✅ 0 violations in issues created after Phase 5 +- ✅ Team reports clear understanding in survey + +--- + +### Risk 6: Workflow Changes Break CI/CD + +**Description**: Phase 3 validation workflows introduce syntax errors or timeout +**Probability**: Low (well-tested changes) +**Impact**: High (CI broken, all PRs blocked) + +#### Mitigation Strategies + +**1. Local Testing** + +- Test all workflow changes locally first +- Validate YAML syntax +- Test with sample issues + +**2. Staged Rollout** + +- Deploy to non-critical workflow first +- Test with real issues for 24h +- Then deploy to all workflows + +**3. Quick Rollback** + +```bash +# If workflows break: +git revert +git push + +# CI returns to normal within 5 minutes +``` + +**4. Monitoring** + +- Watch workflow runs immediately after deploy +- Set up alerts for workflow failures +- Quick response team on standby + +#### Responsibility + +- **Prevention**: DevOps (testing, staged rollout) +- **Detection**: Automated CI monitoring +- **Response**: Immediate rollback + +#### Success Criteria + +- ✅ Workflow syntax validates locally +- ✅ Staged rollout: 0 errors in 24h +- ✅ Full rollout: 0 errors in 72h +- ✅ Performance: <1s validation overhead + +--- + +## Medium Risks (🟡) + +### Risk 7: Label Audit Tools Become Out-of-Date + +**Description**: Audit scripts don't detect new violation patterns +**Probability**: Low (simple rule set) +**Impact**: Medium (hidden violations reappear) + +**Mitigation**: Quarterly audit script review, add new patterns as discovered + +--- + +### Risk 8: New Team Members Unaware of Label Rules + +**Description**: New hires create bare labels because they don't know the rules +**Probability**: High (onboarding issue) +**Impact**: Medium (minor violations, fixable) + +**Mitigation**: Add label rules to onboarding checklist, point to `docs/LABELING.md` + +--- + +### Risk 9: Third-Party Tools Create Bare Labels + +**Description**: GitHub Actions, bots, or integrations create bare labels +**Probability**: Medium (many tools touch labels) +**Impact**: Medium (defeats automation) + +**Mitigation**: Audit all tools, update as needed, add rules to tool config + +--- + +### Risk 10: Documentation Becomes Stale Over Time + +**Description**: Rules change but documentation doesn't stay updated +**Probability**: High (common issue) +**Impact**: Medium (confusion, non-compliance) + +**Mitigation**: Assign documentation owner, quarterly review, link to source of truth + +--- + +### Risk 11: Performance Impact of Validation + +**Description**: Label validation adds latency to issue creation +**Probability**: Very Low (simple validation) +**Impact**: Medium (if >5s overhead) + +**Mitigation**: Optimize validation script, test performance, cache label list + +--- + +### Risk 12: Stakeholder Resistance to Change + +**Description**: Some teams object to new label rules +**Probability**: Low (rules are reasonable) +**Impact**: Medium (resistance delays adoption) + +**Mitigation**: Clear communication of benefits, show audit results, gather feedback + +--- + +## Contingency Procedures + +### If Phase 1 Fails + +``` +Condition: Defective code not deleted, still creating bare labels +Action: + 1. Investigate why deletion failed + 2. Manually verify file doesn't exist + 3. Check git history + 4. Retry deletion with explicit path + 5. Verify with audit + +Timeline: < 1 hour +Owner: DevOps +``` + +### If Phase 2 Remediation Breaks + +``` +Condition: Remediation script errors, labels corrupted +Action: + 1. Stop remediation immediately (CTRL+C) + 2. Roll back: git revert + 3. Investigate error logs + 4. Fix script + 5. Retry with dry-run + 6. Proceed carefully to next batch + +Timeline: < 2 hours +Owner: DevOps + Engineering +``` + +### If Phase 3 Validation Too Strict + +``` +Condition: Workflow rejects legitimate labels +Action: + 1. Collect examples of blocked valid labels + 2. Review rules with team + 3. Adjust validation rules + 4. Test adjustment with dry-run + 5. Deploy update + 6. Communicate change to team + +Timeline: < 4 hours +Owner: DevOps + Governance +``` + +### If Audit Shows New Violations Appeared + +``` +Condition: Post-remediation audit shows violations didn't decrease +Action: + 1. Investigate source of violations + 2. Check if Phase 1 was complete + 3. Run audit again with verbose output + 4. Identify specific issues causing violations + 5. Manually fix or re-remediate + 6. Re-audit to verify + +Timeline: < 2 hours +Owner: Governance + DevOps +``` + +--- + +## Approval & Sign-Off + +**Risk Assessment Completed By**: Claude Code +**Reviewed By**: [Engineering Lead] +**Approved By**: [Project Sponsor] + +**Date Approved**: [TBD] +**Review Date**: [TBD - suggest 1 week post-Phase 1] + +--- + +## Monitoring & Escalation + +### Weekly Status Report + +- [ ] Phase 1–5 progress +- [ ] Any risks materialized? +- [ ] Mitigation actions taken +- [ ] Audit results +- [ ] Team feedback +- [ ] Next week priorities + +### Escalation Path + +1. **Immediate Issues** (blocking work) + - Contact: Project Lead + - Response: < 1 hour + - Action: Fix or rollback + +2. **Non-Blocking Issues** + - Contact: Governance Lead + - Response: < 4 hours + - Action: Log and track + +3. **Stakeholder Concerns** + - Contact: Project Sponsor + - Response: < 24 hours + - Action: Communicate plan + +--- + +*Built with ☕ and 🚀 by LightSpeedWP Governance Team* diff --git a/.github/projects/active/label-prefix-enforcement-2026-08-05/TESTING_VALIDATION.md b/.github/projects/active/label-prefix-enforcement-2026-08-05/TESTING_VALIDATION.md new file mode 100644 index 000000000..e4801814d --- /dev/null +++ b/.github/projects/active/label-prefix-enforcement-2026-08-05/TESTING_VALIDATION.md @@ -0,0 +1,745 @@ +--- +title: "Label Prefix Enforcement Testing & Validation" +description: "Comprehensive testing procedures and validation checklist" +file_type: "documentation" +version: "1.0.0" +created_date: "2026-08-07" +last_updated: "2026-08-07" +author: "Claude Code" +maintainer: "LightSpeed Team" +domain: "governance" +status: "active" +tags: + - testing + - validation + - qa + - procedures +--- + +# Testing & Validation Procedures + +**Document Purpose**: Define testing and validation procedures for each phase +**Target Audience**: QA, DevOps, Engineering, Governance teams +**Success Measure**: All tests pass, 0 violations confirmed + +--- + +## Testing Strategy Overview + +### Testing Pyramid + +``` + ▲ + ╱│╲ + ╱ │ ╲ E2E Tests (Manual + Automated) + ╱ │ ╲ - Full workflow scenarios + ╱───┼───╲ + ╱ │ ╲ Integration Tests + ╱ │ ╲ - Validation + label creation + ╱──────┼──────╲ +╱ │ ╲ Unit Tests + │ - Individual functions + Base Tests +``` + +### Test Coverage Targets + +| Category | Target | Measurement | +|----------|--------|-------------| +| **Unit Tests** | 90%+ | Function coverage | +| **Integration** | 85%+ | Workflow coverage | +| **E2E Tests** | 100% | Critical paths | +| **Regression** | 0 | New issues created | +| **Edge Cases** | 100% | Known scenarios | + +--- + +## Phase 1 Testing: Stop New Violations + +### Test 1.1: Documentation Accuracy + +**Purpose**: Verify CLAUDE.md and AGENTS.md contain accurate, helpful information +**Owner**: QA +**Timeline**: 30 minutes + +#### Test Steps + +1. **Review CLAUDE.md Section** + - [ ] Section exists and is findable + - [ ] Title is clear: "Label Creation Rules (CRITICAL)" + - [ ] Examples are accurate + - [ ] Invalid examples are actually invalid + - [ ] References point to correct files + +2. **Review AGENTS.md Section** + - [ ] Section exists under governance section + - [ ] Implementation examples are correct + - [ ] Code blocks format properly + - [ ] References match canonical system + +3. **Validate References** + + ```bash + # Verify .github/labels.yml exists + test -f .github/labels.yml && echo "✓ Labels file exists" + + # Verify docs/LABELING.md exists + test -f docs/LABELING.md && echo "✓ LABELING.md exists" + + # Verify docs/LABEL_STRATEGY.md exists + test -f docs/LABEL_STRATEGY.md && echo "✓ LABEL_STRATEGY.md exists" + ``` + +4. **Example Validation** + - All `type:bug`, `type:feature`, etc. examples must exist in `.github/labels.yml` + - Invalid examples (bare labels) must NOT exist in `.github/labels.yml` + +#### Pass Criteria + +- [ ] All sections readable and well-formatted +- [ ] All examples verified against canonical labels +- [ ] All references valid +- [ ] Linting passes (markdown, YAML) + +--- + +### Test 1.2: Code Deletion Verification + +**Purpose**: Confirm defective code is removed and not referenced elsewhere +**Owner**: DevOps +**Timeline**: 15 minutes + +#### Test Steps + +1. **Verify File Removal** + + ```bash + # Check file doesn't exist + if [ -f "scripts/agents/includes/labeling-agent.js" ]; then + echo "❌ FAIL: File still exists" + exit 1 + else + echo "✓ File removed" + fi + ``` + +2. **Check Git History** + + ```bash + # Verify removal was committed + git log --oneline | grep -i "Remove defective" && echo "✓ Commit found" + + # Check for file in previous commit + git show HEAD~1:scripts/agents/includes/labeling-agent.js &>/dev/null && echo "✓ File existed before" + ``` + +3. **Search for References** + + ```bash + # Comprehensive reference search + grep -r "labeling-agent.js" . --include="*.js" --include="*.yml" --include="*.yaml" --include="*.md" && echo "❌ FAIL: References found" || echo "✓ No references" + + grep -r "scripts/agents/includes/labeling" . --include="*.js" --include="*.yml" && echo "❌ FAIL: References found" || echo "✓ No references" + ``` + +4. **Verify Correct Implementation Exists** + + ```bash + test -f .github/scripts/agents/labeling.agent.js && echo "✓ Correct implementation exists" + ``` + +#### Pass Criteria + +- [ ] Defective file completely removed +- [ ] No references to file exist +- [ ] Correct implementation file exists +- [ ] Git history shows proper removal + +--- + +### Test 1.3: Governance Rule Enforcement + +**Purpose**: Verify AI instructions will prevent bare label creation going forward +**Owner**: Governance + QA +**Timeline**: 30 minutes + +#### Test Steps + +1. **CLAUDE.md Governance** + + ```bash + # Check section exists + grep -A 20 "Label Creation Rules" CLAUDE.md | head -25 + + # Verify it mentions: + # - "ALL labels MUST" + # - ".github/labels.yml" + # - "family prefix" + # - Examples of valid AND invalid labels + ``` + +2. **AGENTS.md Implementation Guide** + + ```bash + # Check implementation section + grep -A 30 "Label Creation for Programmatic" AGENTS.md + + # Verify it includes: + # - Reference to canonical system + # - Validation steps + # - Correct vs incorrect patterns + ``` + +3. **Validation Checklist** + - [ ] CLAUDE.md: "Label Creation Rules (CRITICAL)" section exists + - [ ] CLAUDE.md: Valid examples (with prefixes) documented + - [ ] CLAUDE.md: Invalid examples (bare) documented + - [ ] CLAUDE.md: Reference to `.github/labels.yml` included + - [ ] AGENTS.md: Implementation guidance documented + - [ ] AGENTS.md: Code pattern examples included + - [ ] Both files: Markdown linting passes + +#### Pass Criteria + +- [ ] Both documents clearly explain rules +- [ ] Rules are unambiguous and actionable +- [ ] Team can understand without further questions +- [ ] Governance is documented in AI instructions + +--- + +## Phase 2 Testing: Fix Existing Issues + +### Test 2.1: Remediation Script Dry-Run + +**Purpose**: Verify remediation script works correctly without making changes +**Owner**: QA + DevOps +**Timeline**: 1 hour + +#### Test Steps + +1. **Run Dry-Run Audit** + + ```bash + npm run audit:labels -- --mode dry-run + ``` + +2. **Review Proposed Changes** + - [ ] Script identifies all ~100 issues with bare labels + - [ ] Proposed corrections are accurate + - [ ] No false positives (valid labels not flagged) + - [ ] All corrections follow canonical system + +3. **Example Validations** + - Bare `bug` → `type:bug` ✓ + - Bare `feature` → `type:feature` ✓ + - Bare `urgent` → `priority:critical` ✓ + - Bare `ci` → `area:ci` ✓ + +4. **Document Review** + + ```bash + # Save dry-run output for review + npm run audit:labels -- --mode dry-run > /tmp/remediation-plan.txt + + # Manual review by 2+ team members + # Sign off: acceptance needed to proceed + ``` + +#### Pass Criteria + +- [ ] Dry-run completes without errors +- [ ] All proposed changes reviewed and approved +- [ ] No false positives found +- [ ] All changes are valid label corrections +- [ ] 2+ team members have signed off + +--- + +### Test 2.2: Remediation Script Execution (Subset) + +**Purpose**: Test remediation on first 10 issues before full batch +**Owner**: QA + DevOps +**Timeline**: 30 minutes + +#### Test Steps + +1. **Run Remediation on Subset** + + ```bash + # Remediate first 10 issues (dry-run first) + npm run remediate:labels -- --mode dry-run --limit 10 + + # If approved, run real remediation + npm run remediate:labels -- --mode direct --limit 10 + ``` + +2. **Verify Changes** + + ```bash + # For each of the 10 issues, verify in GitHub UI: + # - Old label (bare) is removed + # - New label (prefixed) is added + # - No other labels were changed + ``` + +3. **Post-Remediation Audit** + + ```bash + # Audit after subset remediation + npm run audit:labels + + # Expected: 90 violations remaining (not 100) + ``` + +4. **Check for Side Effects** + - [ ] No other labels changed + - [ ] Workflows still function + - [ ] No automation broke + - [ ] Issues still properly categorized + +#### Pass Criteria + +- [ ] All 10 issues successfully remediated +- [ ] Labels correct in GitHub UI +- [ ] Post-remediation audit confirms reduction +- [ ] No unintended side effects +- [ ] Approved to proceed with full batch + +--- + +### Test 2.3: Full Remediation Execution + +**Purpose**: Execute full remediation on all ~100 issues +**Owner**: DevOps +**Timeline**: 2 hours (execution + audit) + +#### Test Steps + +1. **Execute Full Remediation** + + ```bash + # Full remediation with final approval + npm run remediate:labels -- --mode direct + ``` + +2. **Monitor Execution** + - [ ] Script runs to completion + - [ ] No errors reported + - [ ] Progress indicator shows completion + - [ ] Total issues processed = ~100 + +3. **Post-Remediation Audit** + + ```bash + # Full audit after remediation + npm run audit:labels + + # Expected: 0 violations found + ``` + +4. **Spot Check Issues** + - [ ] Randomly select 20 issues from range #1500–#1600 + - [ ] Verify in GitHub UI that labels are correct + - [ ] No bare labels remain + - [ ] All labels are from canonical set + +5. **Workflow Verification** + - [ ] Automation that depends on prefixed labels works + - [ ] Reports/metrics still accurate + - [ ] No workflow failures + +#### Pass Criteria + +- [ ] Full remediation completes successfully +- [ ] Post-remediation audit: 0 violations +- [ ] Spot check: all sampled issues correct +- [ ] All workflows function normally +- [ ] Phase 2 complete and verified + +--- + +### Test 2.4: Regression Testing + +**Purpose**: Verify fix doesn't break existing functionality +**Owner**: QA +**Timeline**: 1 hour + +#### Test Scenarios + +1. **Issue Search** + + ```bash + # Test filtering by label family + gh issue list --label "type:bug" | wc -l + gh issue list --label "status:needs-triage" | wc -l + gh issue list --label "priority:critical" | wc -l + + # Results should be consistent + ``` + +2. **Workflow Execution** + - [ ] Issue triage workflow works with new labels + - [ ] Release workflow recognizes labels correctly + - [ ] CI/CD workflows function normally + - [ ] Slack notifications still fire + +3. **Reports & Metrics** + - [ ] Label-based metrics are accurate + - [ ] Issue dashboards show correct counts + - [ ] Team reports unaffected + +#### Pass Criteria + +- [ ] All issue searches work +- [ ] All workflows function +- [ ] All reports/metrics accurate +- [ ] Zero regressions detected + +--- + +## Phase 3 Testing: Enforce Validation in Workflows + +### Test 3.1: Unit Tests for Validation Function + +**Purpose**: Test validation logic at function level +**Owner**: QA + DevOps +**Timeline**: 1 hour + +#### Test Cases + +```javascript +// Test: Valid label passes +validateLabel('type:bug') → true ✓ + +// Test: Bare label fails +validateLabel('bug') → false ✓ + +// Test: Invalid label fails +validateLabel('type:invalid-type') → false ✓ + +// Test: One-hot per family passes +validateLabels(['type:bug', 'status:open', 'priority:normal']) → true ✓ + +// Test: Duplicate family fails +validateLabels(['type:bug', 'type:feature']) → false ✓ + +// Test: Meta labels can be multiple +validateLabels(['meta:needs-changelog', 'meta:has-pr']) → true ✓ +``` + +#### Pass Criteria + +- [ ] All unit tests pass +- [ ] Code coverage ≥ 90% +- [ ] Edge cases handled +- [ ] Error messages clear + +--- + +### Test 3.2: Integration Tests with Workflows + +**Purpose**: Test validation integrated into GitHub workflow +**Owner**: QA + DevOps +**Timeline**: 2 hours + +#### Test Scenarios + +1. **Create Issue with Valid Labels** + + ```bash + gh issue create --title "Test" --label "type:bug" --label "status:needs-triage" + + # Expected: Issue created successfully + # Validation passes, issue has correct labels + ``` + +2. **Create Issue with Bare Labels (Should Reject)** + + ```bash + gh issue create --title "Test" --label "bug" --label "urgent" + + # Expected: Issue creation blocked + # Workflow validation rejects bare labels + # Clear error message provided + ``` + +3. **Create PR with Valid Labels** + + ```bash + gh pr create --title "Test" --label "type:feature" --label "status:review" + + # Expected: PR created successfully + ``` + +4. **Create PR with Invalid Labels (Should Reject)** + + ```bash + gh pr create --title "Test" --label "bug" --label "feature" + + # Expected: PR creation blocked + ``` + +5. **Edge Case: One-Hot Violation (Should Reject)** + + ```bash + gh issue create --title "Test" --label "type:bug" --label "type:feature" + + # Expected: Issue creation blocked + # Error: "Multiple type labels not allowed" + ``` + +6. **Edge Case: Meta Multiple Labels (Should Pass)** + + ```bash + gh issue create --title "Test" --label "meta:needs-changelog" --label "meta:has-pr" + + # Expected: Issue created successfully + # Meta labels can have multiples + ``` + +#### Pass Criteria + +- [ ] Valid labels create issues/PRs successfully +- [ ] Bare labels are rejected with clear error +- [ ] Invalid combinations are rejected +- [ ] One-hot rules enforced (except meta) +- [ ] Error messages are helpful +- [ ] Workflows complete in <5 seconds + +--- + +### Test 3.3: Error Handling & Recovery + +**Purpose**: Test error scenarios and recovery procedures +**Owner**: QA +**Timeline**: 1 hour + +#### Test Scenarios + +1. **Invalid Label + Recovery** + + ```bash + # Attempt 1: Create with bare label (fails) + gh issue create --title "Test" --label "bug" + # → Workflow blocks, error message shown + + # Attempt 2: Fix label and retry + gh issue create --title "Test" --label "type:bug" + # → Success + ``` + +2. **Workflow Timeout** + - [ ] Validation completes in <5s + - [ ] No timeout errors + - [ ] Performance acceptable + +3. **API Failures** + - [ ] If label lookup fails, safe fallback behavior + - [ ] Clear error messages + - [ ] No silent failures + +#### Pass Criteria + +- [ ] All error scenarios handled gracefully +- [ ] Clear, helpful error messages +- [ ] Recovery procedures work +- [ ] Performance acceptable + +--- + +## Phase 4 Testing: Documentation Updates + +### Test 4.1: Documentation Accuracy + +**Purpose**: Verify all documentation is accurate and consistent +**Owner**: QA + Documentation owner +**Timeline**: 2 hours + +#### Tests + +1. **Example Consistency** + - All examples in LABELING.md exist in `.github/labels.yml` + - All invalid examples don't exist in canonical system + - No contradictions between documents + +2. **Reference Accuracy** + - All links point to correct files + - All file paths are accurate + - No broken references + +3. **Completeness** + - All label families documented + - All major labels have examples + - Edge cases explained + +4. **Clarity** + - Documentation is understandable + - Examples are clear + - Screenshots/diagrams helpful + +#### Pass Criteria + +- [ ] 100% example accuracy +- [ ] 100% reference accuracy +- [ ] Complete documentation coverage +- [ ] Clear, readable docs + +--- + +### Test 4.2: FAQ Accuracy + +**Purpose**: Verify FAQ answers are correct and helpful +**Owner**: QA + Team Lead +**Timeline**: 1 hour + +#### Test Scenarios + +1. **Answer Verification** + - Each FAQ answer is accurate + - Examples are correct + - Solutions work + +2. **Question Coverage** + - Common questions covered + - Edge cases explained + - Troubleshooting included + +#### Pass Criteria + +- [ ] All FAQ answers verified +- [ ] Common questions covered +- [ ] Examples tested and working + +--- + +## Phase 5 Testing: Team Training + +### Test 5.1: Team Understanding Assessment + +**Purpose**: Verify team understands new label system +**Owner**: Training Lead +**Timeline**: 1 hour per session + +#### Assessment Methods + +1. **Knowledge Quiz** + - 10 questions about label rules + - Target: 90%+ pass rate + - Questions cover: + - What is a family prefix? + - What are valid label families? + - How do I find the canonical list? + - What if I don't know which label? + +2. **Hands-On Practice** + - Create test issues with correct labels + - Fix test issues with bare labels + - Identify valid vs invalid labels + +3. **Feedback Survey** + - Is documentation clear? + - Are rules understood? + - Any questions remaining? + +#### Pass Criteria + +- [ ] 90%+ team members pass knowledge quiz +- [ ] All team members can create valid labels +- [ ] Team members can explain the system +- [ ] Feedback indicates understanding + +--- + +## Automated Testing Suite + +### Continuous Validation + +```bash +# Run before every commit +npm run test + +# Includes: +- Unit tests for validation functions +- Integration tests with label system +- Regression tests for workflows +- Documentation accuracy checks +``` + +### Pre-Deployment Checklist + +```bash +#!/bin/bash +set -e + +# Phase 1 +npm run lint:md CLAUDE.md AGENTS.md +npm run validate:frontmatter CLAUDE.md + +# Phase 2 +npm run audit:labels -- --mode dry-run > /tmp/audit.txt +# Manual review of /tmp/audit.txt required + +# Phase 3 +npm test -- validation.test.js +npm test -- integration.test.js + +# Phase 4 +npm run validate:label-references + +# Phase 5 +npm run audit:labels +# Expected: 0 violations + +echo "✓ All pre-deployment checks passed" +``` + +--- + +## Success Criteria Summary + +| Phase | Test | Passing Status | +|-------|------|----------------| +| 1 | Documentation accuracy | ✓ Required | +| 1 | Code deletion | ✓ Required | +| 1 | Governance enforcement | ✓ Required | +| 2 | Dry-run audit | ✓ Required | +| 2 | Subset remediation | ✓ Required | +| 2 | Full remediation | ✓ Required | +| 2 | Regression testing | ✓ Required | +| 3 | Unit tests | ✓ Required | +| 3 | Integration tests | ✓ Required | +| 3 | Error handling | ✓ Required | +| 4 | Documentation | ✓ Required | +| 4 | FAQ accuracy | ✓ Required | +| 5 | Team assessment | ✓ Required | + +--- + +## Test Execution Timeline + +``` +Week 1: + Phase 1 Tests (Tue–Wed) [6 hours] + Phase 2 Tests (Thu–Fri) [8 hours] + +Week 2: + Phase 3 Tests (Mon–Tue) [6 hours] + Phase 4 Tests (Wed–Thu) [4 hours] + Phase 5 Tests (Fri) [4 hours] + +Total Testing Effort: ~28 hours +Parallel with Phase Execution: Reduce timeline impact +``` + +--- + +## Contact & Escalation + +- **QA Lead**: [Name] +- **Testing Issues**: Raise in #governance-enforcement channel +- **Critical Failures**: Escalate immediately to Project Sponsor + +--- + +*Built with ☕ and 🚀 by LightSpeedWP QA & Governance Teams*