Fix surrogate skips, Action injection, and hashing crashes - #118
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd152c8549
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return raw.decode("utf-16") | ||
| except UnicodeDecodeError as error: | ||
| raise ClipboardUnavailableError("clipboard bytes were not valid UTF-8 or UTF-16") from error | ||
| if b"\x00" in raw and len(raw) % 2 == 0: |
There was a problem hiding this comment.
On Linux and macOS, the clipboard commands return UTF-8, but this branch globally interprets every even-length payload containing a NUL as UTF-16LE. For example, valid UTF-8 bytes b"A\x00" become just "A", so clipboard scan, watch, and clean miss the NUL even though it belongs to the default control category. Restrict the BOM-less UTF-16LE heuristic to the Windows PowerShell reader, or otherwise pass the reader's expected encoding into this function.
Useful? React with 👍 / 👎.
|
|
||
| def _json_response(self, status: int, payload: dict[str, object]) -> None: | ||
| body = json.dumps(payload, ensure_ascii=False).encode("utf-8") | ||
| body = json.dumps(payload, ensure_ascii=False).encode("utf-8", "surrogatepass") |
There was a problem hiding this comment.
When an endpoint returns an input lone surrogate—such as /api/guard with on_findings: "report" or a /api/remove-marks miss—surrogatepass emits its WTF-8 byte sequence even though the response declares UTF-8. Standard clients then fail UTF-8 decoding before they can parse the JSON, so this replaces the previous server-side exception with an unusable response. Serialize surrogates as JSON escapes, for example by retaining ensure_ascii=True, rather than writing invalid UTF-8 bytes.
Useful? React with 👍 / 👎.
Lint and the scan/clean/guard/normalize paths now decode CESU-8 lone surrogates instead of skipping or raising. The GitHub Action passes inputs through env vars, clipboard UTF-16LE ASCII is not read as NULs, and sha256_text no longer crashes on U+D800-U+DFFF. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
JSON receipts and the local web API now emit \uXXXX escapes instead of WTF-8 so clients can decode the declared UTF-8. The UTF-16LE-without-BOM clipboard heuristic is limited to PowerShell so Linux/macOS NULs stay in the control category. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
25dc86f to
672de39
Compare
2 participants
Repository audit of the defensive surfaces (lint, scan/clean, guard, normalize, clipboard, GitHub Action, local web API). Full
pytestis green.Bugs
fuckmark lintnever firedsurrogate. Strict UTF-8 cannot contain U+D800–U+DFFF, and CESU-8 / modified-UTF-8 encodings of those codepoints were skipped as “invalid UTF-8”, so the default security set advertised a category that could not fail a CI gate. Lint now decodes those bytes withutf-8+surrogatepassand reports them.--fixwrites with the same encoding so a--select bidi_controlrewrite can keep leftover surrogates instead of crashing.--scan/--clean/guard/normalizeuse the same decoder; the mix path still rejects those files.action.ymlinterpolatedinputs.select,inputs.args, andinputs.pathsinto bash. Those values now enter the step throughenv:and are parsed withshlex.split. The existing workflow hygiene test is extended toaction.yml.A\\x00B\\x00is valid UTF-8 (A\\0B\\0), so UTF-8-first decoding insertedU+0000controls between letters. BOM-less UTF-16LE is used only for PowerShellGet-Clipboard.pbpaste/wl-paste/xclip/xselstay UTF-8, so a real NUL on Linux or macOS remains in thecontrolcategory.protect()/normalize_text()crashed on lone surrogates while hashing the receipt.sha256_textnow usessurrogatepass(identical for every valid scalar value; frozen confirmation hashes are unchanged).surrogatepassemitted WTF-8 (ED A0 80). Clients that decodeapplication/json; charset=utf-8strictly would fail. Responses and receipts now escape lone surrogates as\\uXXXX.fuckmark webPOST withoutContent-Lengthscanned{}. Missing header is now a 400 instead of a silent empty-body success.build_real_measurement_calibration_corpusreturn annotation.MeasurementCalibrationCorpusis imported sotyping.get_type_hintsresolves.Package version stays 0.4.1. Frozen confirmation files and hashes are untouched. Independent of #117.