chore: add comprehensive pre-commit hooks for security and quality - #197
Open
miguellsfilho wants to merge 3 commits into
Open
chore: add comprehensive pre-commit hooks for security and quality#197miguellsfilho wants to merge 3 commits into
miguellsfilho wants to merge 3 commits into
Conversation
Implements critical pre-commit hooks for security, SQL validation, and code quality. **Security (CRITICAL)** - detect-secrets: Prevents credential leaks (DATABASE_URL, GCP keys, tokens) - Baseline file (.secrets.baseline) manages false positives - Catches secrets BEFORE they reach remote **SQL Linting (CRITICAL)** - sqlfluff validates 48 SQL files (42 PostgreSQL migrations + 6 BigQuery) - Catches syntax errors, prevents production database breakage - Focuses on errors, excludes style rules (layout/capitalisation) **File Hygiene** - Trailing whitespace, EOF newlines, LF line endings - YAML/JSON/TOML syntax validation - Merge conflict markers detection - Large files protection (>1MB) **Python Quality** - Ruff linter + formatter (replaces Black/isort/flake8) - Auto-fixes 300+ issues (imports, formatting) - Legacy code exceptions documented in pyproject.toml **Custom Validators** - validate_feature_registry.py: Validates feature_registry.yaml schema - Prevents typos in feature types/compute fields **Files Created** - .sqlfluff: SQL linting config - .secrets.baseline: Secret detection baseline - scripts/validate_feature_registry.py: Custom validator **Files Modified** - .pre-commit-config.yaml: Added 4 phases (hygiene, security, SQL, Python) - pyproject.toml: Ruff lint section, test/legacy exceptions - 84 Python files auto-formatted by Ruff - SQL files: lowercase types, BigQuery keyword escaping **Note**: mypy disabled temporarily (149 type errors in src/). Will enable gradually after adding type hints. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
**Documentation (ALTA PRIORIDADE)** - Add comprehensive "Development Setup" section to README.md - Document all pre-commit hooks with descriptions and performance - Replace outdated "Padrões de Código" section with current info **Type Checking (ALTA PRIORIDADE)** - Enable mypy for core modules: config.py, models/news.py, models/__init__.py - Add pydantic-settings to mypy additional_dependencies - Preserves type checking on critical modules without blocking commits - TODO: Expand to other modules after fixing type errors **Line Endings (MÉDIA PRIORIDADE)** - Create .gitattributes to enforce LF line endings - Prevents mixed-line-ending issues before they happen - Covers all text files (py, sql, yaml, json, md) **Impact:** - Developers now have clear instructions on using pre-commit - Type checking maintained on 3 core modules (vs 0 in previous commit) - Line ending issues prevented at Git level All 15 hooks passing including mypy on core modules.
Migration integration tests validate that migrations run successfully, not code coverage. They only exercise migration scripts and fixtures, resulting in ~5% coverage of the full codebase. Unit tests (tests.yaml workflow) already enforce 70% coverage requirement. Changes: - Add --no-cov flag to pytest command in ci-migrations.yaml - Prevents false negatives from coverage checks on integration tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Objetivo
Implementa pre-commit hooks abrangentes focando em segurança, validação de SQL e qualidade de código para prevenir incidentes em produção.
📋 O Que Foi Implementado
🔒 Segurança (CRÍTICO)
.secrets.baselinegerencia false positives🛢️ SQL Linting (CRÍTICO)
📝 File Hygiene
🐍 Python Quality
⚙️ Custom Validators
validate_feature_registry.py: Valida schema de feature_registry.yaml📚 Documentação
⚡ Type Checking
🔧 Line Endings
.gitattributesforça LF em todos text files📦 Arquivos Criados
.sqlfluff- Configuração SQL linting.secrets.baseline- Baseline de secrets (16KB, 561 entradas).gitattributes- Normalização de line endingsscripts/validate_feature_registry.py- Validador custom📝 Arquivos Modificados
.pre-commit-config.yaml- 5 fases de hooks (hygiene, security, SQL, Python, custom)pyproject.toml- Ruff lint section, exceções para legacy code e testsREADME.md- Documentação completa sobre pre-commit✅ Testes
Unitários: ✅ 678 passed (coverage 71.73%)
Integração: ✅ 2 passed, 65 skipped (erros esperados - falta PostgreSQL local)
Veredicto: Nenhum teste quebrado pelas mudanças.
🚀 Performance
📊 Hooks Ativados (15 total)
✓ Trim trailing whitespace
✓ Ensure files end with newline
✓ Normalize line endings to LF
✓ Validate YAML syntax
✓ Validate TOML syntax
✓ Check for merge conflict markers
✓ Check for case-insensitive filename conflicts
✓ Check for files > 1MB
✓ Detect secrets and credentials 🔒
✓ Lint SQL files (PostgreSQL) 🛢️
✓ Lint SQL files (BigQuery) 🛢️
✓ Lint Python with Ruff 🐍
✓ Format Python with Ruff 🐍
✓ Type check with mypy (core modules) 🐍
✓ Validate feature_registry.yaml schema ⚙️
🎓 Como Usar
Após merge, desenvolvedores devem rodar:
Documentação completa no README.md seção "Development Setup".
🔄 Próximos Passos (Opcional)
📈 Impacto
Commits: