fix(sonar): remediate SonarCloud security and code-quality findings - #191
Conversation
Safe, behavior-preserving remediation of SonarCloud findings across the engine, chassis, tools, agents, tests, workflows, Dockerfiles, shell scripts, and SQL. Local gate green: ruff + ruff-format + mypy(engine/) + contract scanner + 721 unit tests. No suppressions, no rule or quality-gate threshold weakening; issues that could not be fixed without behavior/CI risk are left untouched (see PR body). Security (workflows / docker): - Pin GitHub Actions to immutable commit SHAs (S7637) - Scope supply-chain write permissions to job level (S8233) - Pass Slack webhook secret via env instead of inline expansion (S7636) - Interpolate PR title through env vars to block script injection (S7630) - Add non-root user to the production image; merge RUN layers (S6471, S7031) Code quality (engine / chassis / tools / agents): - logging.exception() inside except blocks (S8572) - concise regex character classes (S6353) - module constants for duplicated literals (S1192) - math.isclose for float equality (S1244) - numpy.random.Generator API; seeded RNG (S6711, S6709) - merge-if, set.update, total_ordering, dup-branch, lambda capture, etc. Tests: - Scope pytest.raises to the single throwing call (S5778) - Narrow over-broad exception assertions (S5958) - monkeypatch for global state; explicit skip reasons (S8997, S1607, S5918, S5779) Shell / SQL: - [[ ]] conditionals, stderr redirect, drop unused var (S7688, S7677, S1481) - explicit column lists instead of SELECT * (SelectStarCheck) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01USEi3CurG9BEbx5bTHHGMx
|
❌ Too Many Reviewable Files Changed
📋 Best Practices for Large Changes
🚫 This PR is blocked until reviewable size limits are met. |
L9 Audit Harness Report
Step Results
Architecture Audit Findings
See Spec Coverage
See Next StepsAll checks passed. Safe to merge. |
There was a problem hiding this comment.
Pull request overview
This PR performs broad, evidence-driven remediation of SonarCloud security and code-quality findings across the CEG repository, spanning engine runtime code, tooling, tests, scripts, containers, and GitHub Actions workflows.
Changes:
- Hardens CI/CD supply chain by pinning GitHub Actions to immutable SHAs and tightening workflow-level permissions.
- Refactors multiple code paths for improved robustness/clarity (e.g., exception logging via
logger.exception, reduced duplicated literals, safer float comparisons, small control-flow simplifications). - Updates tests to narrow exception assertions and reduce over-broad
pytest.raises(...)scopes.
Reviewed changes
Copilot reviewed 78 out of 78 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/validate_domain.py | Simplifies exception handling in compliance checks. |
| tools/spec_extract.py | Centralizes glob/filename constants for spec feature extraction outputs. |
| tools/packet_envelope_gate.py | Simplifies exception tuple in contract-gate parsing. |
| tools/infra/precommit_smoke.sh | Removes unused variable in container health check helper. |
| tools/infra/docker_validate.sh | Sends failure summary to stderr. |
| tools/contract_scanner.py | Extracts repeated literals into module constants for contract rules. |
| tools/check_deprecated_imports.py | Tightens regex for deprecated import detection. |
| tools/audit_harness.py | Extracts step-name constant; simplifies severity icon selection. |
| tools/audit_engine.py | Collapses nested conditionals for required-any token enforcement. |
| tests/unit/test_sync_projection.py | Narrows pytest.raises scope by isolating the throwing call. |
| tests/unit/test_projection_outcome_schemas.py | Narrows pytest.raises scope by isolating the throwing call. |
| tests/unit/test_plasticos_spec_authority.py | Uses explicit pytest.skip(...) for non-authoritative entries. |
| tests/unit/test_packet_envelope.py | Narrows exception assertions to ValidationError with match=. |
| tests/unit/test_hgkr_schema.py | Narrows pytest.raises scope by isolating the throwing call. |
| tests/unit/test_gates_all_types.py | Avoids inlined mocks; clarifies instantiation failure case. |
| tests/unit/test_domain_loader.py | Asserts specific domain-spec exception type + message. |
| tests/unit/test_config_schema.py | Reduces duplicated literals by pre-building shared node lists. |
| tests/test_scoring_extended.py | Narrows pytest.raises scope; avoids repeated factory calls. |
| tests/test_config_loader.py | Avoids repeated loader construction to narrow exception scope. |
| tests/test_compliance_engine.py | Avoids repeated engine construction to narrow exception scope. |
| tests/test_algorithmic_upgrades.py | Narrows exception types; uses monkeypatch for env state. |
| tests/invariants/test_configuration.py | Narrows to ValidationError with message match. |
| tests/integration/test_sync_handler.py | Narrows assertion to the actual raised exception type. |
| tests/integration/test_handlers.py | Narrows pytest.raises scope around asyncio.run(...). |
| tests/integration/test_admin_handler.py | Allows either raising or error-dict return shape (explicit branching). |
| tests/gap_fixes/test_gap5_audit.py | Uses monkeypatch instead of mutating module globals. |
| tests/contracts/test_templates.py | Adds explicit reasons to pytest.skip(...). |
| scripts/scripts-setup.sh | Uses [[ ... ]] tests for safer shell conditionals. |
| scripts/scripts-seed.sh | Uses [[ ... ]] tests; minor conditional cleanup. |
| scripts/scripts-migrate.sh | Uses [[ ... ]] tests; minor conditional cleanup. |
| scripts/scripts-health.sh | Uses [[ ... ]] tests for status-code check. |
| scripts/scripts-gds-trigger.sh | Uses [[ ... ]] tests for JOB dispatch. |
| scripts/scripts-build.sh | Uses [[ ... ]] tests for account-id guard. |
| engine/traversal/resolver.py | Uses logger.exception to preserve traceback on resolution failures. |
| engine/traversal/assembler.py | Regex simplification for label reference extraction. |
| engine/security/P2_9_llm_schemas.py | Tightens schema-hint sanitization regex (removes redundancy). |
| engine/security/_5_llm_security.py | Uses logger.exception when JSON/schema validation fails. |
| engine/resolution/resolver.py | Uses set.update(...) instead of a loop. |
| engine/personas/synthesis.py | Uses math.isclose for float-zero comparison. |
| engine/personas/composer.py | Uses math.isclose for float-zero comparison. |
| engine/packet/packet_store.sql | Replaces SELECT * with explicit column lists in lineage functions. |
| engine/kge/ensemble.py | Extracts repeated “no scores” error string to a constant. |
| engine/kge/compound_e3d.py | Switches to numpy.random.Generator; minor loop simplification. |
| engine/kge/beam_search.py | Adds ordering support via @total_ordering; seeds RNG in scoring helper. |
| engine/intake/intake_compiler.py | Uses math.isclose for float-zero comparison. |
| engine/intake/impact_reporter.py | Simplifies gate passability conditional. |
| engine/intake/crm_field_scanner.py | Extracts repeated separator regex into a constant. |
| engine/health/health_report.py | Collapses nested tier conditional. |
| engine/handlers.py | Regex simplification for property validation; uses logger.exception on match failures. |
| engine/gds/scheduler.py | Uses logger.exception for job failures and drop/pre-drop errors. |
| engine/gates/compiler.py | Extracts AND joiner constant; uses math.isclose for relaxed-penalty checks. |
| engine/feedback/signal_weights.py | Uses math.isclose for float-zero comparison. |
| engine/diagnostics/fingerprint.py | Avoids repeated dict lookup in max(...) key function. |
| engine/contract_enforcement.py | Collapses nested conditionals for envelope_hash requirement. |
| engine/config/units.py | Uses logger.exception to preserve traceback on formula evaluation failure. |
| engine/config/loader.py | Extracts spec.yaml constant used for domain discovery/loading. |
| engine/compliance/pii.py | Simplifies numeric regex classes; avoids variable shadowing; uses logger.exception. |
| engine/compliance/audit.py | Uses logger.exception for persistence failures. |
| engine/auth/capabilities.py | Extracts repeated permission strings to constants; uses math.isclose for expiry. |
| Dockerfile.prod | Adds non-root runtime user for prod image. |
| Dockerfile | Merges RUN layers for entrypoint chmod + user creation/chown. |
| chassis/pii.py | Simplifies numeric regex classes in PII patterns. |
| chassis/chassis_app.py | Uses logger.exception on health route failure. |
| chassis/auth/auth.py | Collapses nested bypass-key checks into a single condition. |
| chassis/actions.py | Uses logger.exception on engine handler import failure. |
| agents/cursor/ingest_lessons.py | Extracts duplicated timestamp literal into a constant; removes unused capture. |
| agents/cursor/gmp_meta_learning.py | Avoids unused locals; adjusts lambda capture; awaits calls directly. |
| agents/cursor/cursor_session_hooks.py | Simplifies tool-id checks for file-touching actions. |
| agents/cursor/cursor_memory_client.py | Extracts JSON content-type literal into a constant. |
| .github/workflows/supply-chain.yml | Least-privilege workflow-level permissions; job-level scopes expected. |
| .github/workflows/release-drafter.yml | Pins action to immutable commit SHA. |
| .github/workflows/lint-autofix.yml | Pins action to immutable commit SHA. |
| .github/workflows/k8s-deploy.yml | Moves Slack webhook secret into env var usage to reduce injection risk. |
| .github/workflows/docker-build.yml | Pins Trivy action to immutable commit SHA. |
| .github/workflows/dev-layer-gmp.yml | Pins Codecov action to immutable commit SHA. |
| .github/workflows/coderabbit-notify.yml | Avoids direct expression interpolation in shell heredoc content. |
| .github/workflows/ci.yml | Pins pre-commit and Codecov actions to immutable commit SHAs. |
| .github/workflows/ci-quality.yml | Pins Semgrep and Codecov actions to immutable commit SHAs. |
Suppressed comments (1)
agents/cursor/gmp_meta_learning.py:791
- Same issue as above: this log line is not formatted and
activeis no longer assigned. Assign the awaited result and log the actual count (or remove the count wording).
# Get active heuristics for next GMP
await engine.get_active_heuristics()
logger.info("active heuristics: {len(active)}")
| data = json.loads(llm_response) | ||
| except json.JSONDecodeError as e: | ||
| logger.error(f"LLM returned invalid JSON: {e}") | ||
| logger.exception("LLM returned invalid JSON") | ||
| raise ValidationError(f"LLM output is not valid JSON: {e}") |
- engine/kge/compound_e3d.py: index entity_list for negative sampling so the sampled tail stays typed as str (fixes mypy arg-type error on the numpy.random.Generator refactor; caught on the py3.12 CI runner) - tools/spec_extract.py: drop duplicate RESEARCH_DIR / RESEARCH_PATTERNS_FILE definitions introduced with the constants block - agents/cursor/gmp_meta_learning.py: log real heuristic counts via structured fields instead of non-interpolated literal braces - tools/packet_envelope_gate.py: replace empty `except ValueError: pass` with an explanatory stderr fallback (control flow unchanged) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01USEi3CurG9BEbx5bTHHGMx
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 78 out of 78 changed files in this pull request and generated no new comments.
Suppressed comments (1)
engine/security/_5_llm_security.py:157
pydantic.ValidationErrorcan’t be instantiated with a plain string in Pydantic v2, soraise ValidationError(f"...")will itself raise aTypeError, masking the real JSON parse failure. Raise a standard exception (e.g.,ValueError) or construct a proper PydanticValidationErrorviaValidationError.from_exception_data(...).
Two pre-existing bugs surfaced during the SonarCloud remediation (#191): 1. engine/security/{P2_9_llm_schemas,_5_llm_security}.py raised pydantic ValidationError with a plain string on the JSON-parse path. In pydantic v2 ValidationError is not string-constructible, so the invalid-JSON branch raised TypeError instead of a meaningful error. Raise ValueError (which pydantic.ValidationError subclasses, so broad-catch callers are unaffected); schema-shape failures still raise pydantic ValidationError via model_validate. Drop the now-unused import. 2. tests/integration/test_sync_handler.py asserted a non-existent SyncGenerator.resolve_endpoint and only passed because pytest.raises swallowed the AttributeError. Rewrite it to exercise the real RULE 3 path: handle_sync rejects an unknown entity_type with ValidationError ("No sync endpoint for entity type") during endpoint resolution, before any Neo4j access (mock driver + real domain loader). test_invalid_json updated to assert the corrected ValueError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01USEi3CurG9BEbx5bTHHGMx



Summary
Evidence-driven remediation of the SonarCloud backlog for
Quantum-L9_Cognitive.Engine.Graphs(488 issues; 433 open at analyzed revision7827d98, which matches this branch's base). Fixes target root causes with the smallest safe change, preserve behavior, and add no suppressions and no rule/quality-gate weakening. ~145 findings fixed across 78 files; the remainder are deferred with explicit rationale below (behavior/CI risk or rule-vs-architecture mismatch), per the remediation contract.Local validation (all green, CI-pinned toolchain)
ruff check .(0.15.12)ruff format --check .mypy engine/(1.14.0)tools/contract_scanner.py-m unit)verify_contracts/packet_envelope_gate/check_deprecated_imports/audit_harnessFixed
Security — workflows / Docker
codecov,pre-commit,trivy,create-pull-request,release-drafter,semgrep(S7637)supply-chain.yml: drop redundant workflow-level write perms → least-privilegecontents: read; every job already scopes its own (S8233)k8s-deploy.yml: passSLACK_WEBHOOK_URLviaenv:instead of inline${{ secrets… }}expansion (S7636)coderabbit-notify.yml: interpolate PR title through shell env vars, not the Actions expression, closing a script-injection vector (S7630)Dockerfile.prod: add non-rootl9user (parity with the other two images) (S6471);Dockerfile: merge RUN layers (S7031)Code quality — engine / chassis / tools / agents
logging.exception()insideexceptblocks + drop redundantexc_info/message object (S8572)\d/\wwhere exactly equivalent (S6353)math.isclose(…, abs_tol=1e-9)for float equality (S1244)numpy.random.GeneratorAPI + seeded RNG (S6711, S6709)set.update,@total_ordering, dup-branch, lambda-capture, unused locals, regex simplification, redundant-except (S1066, S8502, S8500, S1871, S1515, S1481, S8786, S5713, S8510, S5869)Tests
pytest.raisesto the single throwing call (S5778 ×31)match=(S5958 ×13)monkeypatchfor global/env state; explicit skip reasons; assert-out-of-except (S8997, S1607, S5918, S5779)Shell / SQL
[[ ]]conditionals, stderr redirect, drop unused var (S7688 ×11, S7677, S1481)SELECT *in lineage CTEs (SelectStarCheck)Deferred (with rationale — not fixed to protect correctness / mergeability)
pip/poetryversion-lock +--only-binary(S8541/S8544, ~99 across workflows & Dockerfiles) —requirements.txtpinsconstellation-node-sdkas a git URL;--only-binary=:all:would break its source install, and the security rating is all-or-nothing, so a partial pass adds churn to governed CI for zero gate benefit. Versions are already centralized inrequirements-ci.txt.python:S3776cognitive-complexity (60) — advisory (maintainability already rated A); 60 control-flow refactors carry real regression risk. Deferred to a dedicated follow-up rather than risk this green PR.python:S1172unused params (26) — mostly handler-DI (tenant,graph_driver,domain_loader), packet-builder API, and kernel signatures; removal breaks the uniform positional dispatch / callers (Contract C-2). Interface-stability retained.pythonsecurity:S8707CLI path-traversal (14) — developer CLI tools that legitimately read/write operator-supplied paths; confinement would break valid cross-directory use and cannot be locally verified as cleared. Needs an explicit path-policy decision.shell:S5332clear-text (20) — all${API_URL}/ container-localhttp://localhosthealth checks; HTTPS would break the smoke tests.cursor_memory_client.pyTLS/HTTP/IP (S5527/S4830/S4423/S5332/S1313) — intentional, documented direct-IP HTTP to a self-signed internal endpoint that bypasses Cloudflare; "fixing" breaks connectivity.l9-analysis.ymlS8233 — file isDO NOT EDIT — managed by l9-ci-core preset.engine/utils/security.pyS6353 (insanitize_label) —\wis Unicode-aware; substituting it would widen the Cypher-label injection guard (Contract C-9). Correctly left as the exact ASCII class.engine/boot.pyS7497/S7504, plsql:S1192 (RLScurrent_setting), S6019 ×2, S1135 ×2, S7503 async (interface-required), S125 (not dead code) — each behavior-sensitive or a rule/architecture mismatch; see commit for specifics.Two pre-existing engine bugs surfaced (not fixed here — flagging)
SyncGenerator.resolve_endpoint(...)referenced bytests/integration/test_sync_handler.pydoes not exist; the test only passed becausepytest.raises(Exception)swallowed theAttributeError.engine/security/P2_9_llm_schemas.py:105constructsValidationError(f"…")invalidly for pydantic v2 → raisesTypeErroron the invalid-JSON path.Remote verification
SonarCloud re-analysis is PENDING (runs when this revision is analyzed). The baseline quality gate is ERROR solely on
new_security_rating; this PR reduces the vulnerability surface but does not claim remote closure from local reasoning.Generated by Claude Code