fix(ci): the invisible-character gate never matched anything - #20
fix(ci): the invisible-character gate never matched anything#20hyperpolymath wants to merge 1 commit into
Conversation
MEASURED 2026-08-27: this gate's pattern caught 0 OF 6 invisible-character test
cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi
override or word joiner.
ROOT CAUSE: the pattern used UTF-8 BYTE sequences (\xc2\xa0) while grep -P
matches CHARACTERS. Bytes c2 a0 are ONE character U+00A0; \xc2\xa0 asks for TWO
characters, U+00C2 then U+00A0, which is never present.
grep -P '\xc2\xa0' -> miss
grep -P '\x{a0}' -> MATCH
Only \x00 worked, being single-byte in both readings.
FIXED: codepoint escapes; C0 control characters \x01-\x08,\x0B,\x0C,\x0E-\x1F
added (TAB/LF/CR excluded); and grep -a, without which grep skips any NUL-bearing
file as binary.
The C0 range matters: a stray BACKSPACE byte made a workflow unparseable in
developer-ecosystem, so it never ran, and this linter called it clean.
Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
VERIFIED: YAML re-parsed, and the corrected pattern was confirmed to catch a real
NBSP before the change was kept.
|
Up to standards ✅🟢 Issues
|
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull Request Overview
This PR improves the invisible-character detection gate by transitioning to PCRE codepoint escapes and adding detection for C0 control characters and BiDi overrides. Although Codacy rates this change as up to standards, a significant logic risk was identified: using grep -P with Unicode escapes (\x{...}) causes the engine to stop processing files that contain invalid UTF-8 sequences.
Given that the gate uses the -a flag to include binary-like files (such as those with NUL bytes), this could lead to silent failures where problematic files are skipped entirely without warning. Additionally, there are no automated regression tests provided to ensure these specific invisible characters are actually caught in a real-world scenario. Addressing the robustness of the regex and adding test cases is recommended before merging.
About this PR
- There are no automated regression tests included in this PR to verify that the updated regex patterns successfully catch the targeted characters (NBSP, ZWSP, BOM, etc.). Without these, it is difficult to guarantee the gate remains functional in the future.
Test suggestions
- Missing: Detection of a Non-Breaking Space (U+00A0) in a source file
- Missing: Detection of C0 control characters (e.g., backspace \x08) in a source file
- Missing: Processing a file containing a NUL byte to ensure it is not skipped
- Missing: Detection of a Byte Order Mark (BOM) at the start of a file
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Missing: Detection of a Non-Breaking Space (U+00A0) in a source file
2. Missing: Detection of C0 control characters (e.g., backspace \x08) in a source file
3. Missing: Processing a file containing a NUL byte to ensure it is not skipped
4. Missing: Detection of a Byte Order Mark (BOM) at the start of a file
Low confidence findings
- The patterns and logic are hardcoded directly within the GitHub Action YAML. For long-term maintainability across different workflows, consider moving the regex and scanning logic into a dedicated script.
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| # non-breaking spaces, null bytes, and other invisible Unicode in source files. | ||
| set +e | ||
| PATTERNS='\xc2\xa0|\xe2\x80\x8b|\xe2\x80\x8c|\xe2\x80\x8d|\xef\xbb\xbf|\xc2\xad|\xe2\x80\x8e|\xe2\x80\x8f|\xe2\x80\xaa|\xe2\x80\xab|\xe2\x80\xac|\xe2\x80\xad|\xe2\x80\xae|\x00' | ||
| PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2060}|\x{feff}' |
There was a problem hiding this comment.
🟡 MEDIUM RISK
The use of Unicode code point escapes (\x{...}) with grep -P can lead to silent failures on files containing invalid UTF-8 sequences. When the PCRE engine encounters a validation error, it stops processing the current file. Since stderr is suppressed, these files are skipped without notice. To ensure the scanner is robust against all file types, consider using raw byte sequences for patterns or prepending (*UTF) only where input is guaranteed to be valid UTF-8.\n\nTry running the following prompt in your coding agent:\n> Replace the Unicode code point escapes in the PATTERNS variable with their equivalent UTF-8 byte sequences (e.g., use '\\xc2\\xa0' instead of '\\x{a0}') to ensure the linter can scan files regardless of encoding validity.
| -o -name '*.idr' -o -name '*.zig' -o -name '*.v' -o -name '*.jl' \ | ||
| -o -name '*.gleam' -o -name '*.hs' -o -name '*.ml' -o -name '*.sh' \) \ | ||
| -exec grep -Prl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | ||
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null |
There was a problem hiding this comment.
⚪ LOW RISK
Suggestion: The -r flag is unnecessary because find already handles the directory recursion. Additionally, using {} + instead of {} ; is more efficient as it bundles multiple filenames into fewer grep invocations, reducing process overhead.\n\nsuggestion\n -exec grep -aPl "$PATTERNS" {} + > /tmp/empty-lint-results.txt 2>/dev/null\n
1 participant
Measured 2026-08-27: this gate caught 0 of 6 invisible-character test cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi override or word joiner.
Root cause
The pattern used UTF-8 byte sequences (
\xc2\xa0) whilegrep -Pmatches characters. Bytesc2 a0are one character U+00A0;\xc2\xa0asks for two, U+00C2 then U+00A0 — never present.Only
\x00worked, being single-byte in both readings. The gate ran, passed, and could not see what it exists to see.Fixed
\x01-\x08,\x0B,\x0C,\x0E-\x1Fadded (TAB/LF/CR excluded)grep -a— without it grep skips any NUL-bearing file as binaryThe C0 range matters: a stray backspace byte made a workflow unparseable in
developer-ecosystem, so it never ran — and this linter called it clean.Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
Verified: YAML re-parsed, and the corrected pattern was confirmed to catch a real NBSP before the change was kept.