[rahul05ranjan]

• Closes from_ical(bytes) decodes before unfolding, so folds inside a UTF-8 multi-byte sequence silently corrupt the value (U+FFFD) #2077
• Contributes to Legacy CHARSET=ISO-8859-1 property parameter is ignored: non-UTF-8 .ics files parse to U+FFFD even though the file declares its encoding #2078

Description

Calendar.from_ical(bytes) decodes the whole input to text before unfolding folded content lines, so when a producer folds a line in the middle of a multi-byte UTF-8 sequence, the split bytes are destroyed by the errors="replace" fallback before they can be rejoined — the value silently comes back as U+FFFD soup:

summary = "A" * 70 + "🎉" + "B" * 20
# fold the SUMMARY line inside the 4-byte emoji, RFC 5545-style
event = Calendar.from_ical(ics).walk("VEVENT")[0]
str(event["SUMMARY"])   # 'AAAA…AA\N{REPLACEMENT CHARACTER}\N{REPLACEMENT CHARACTER}\N{REPLACEMENT CHARACTER}\N{REPLACEMENT CHARACTER}BBBB…'

RFC 5545 §3.1 explicitly anticipates that producers may split multi-octet characters when folding, and real .ics exports (older Outlook/Google tooling folds at 75 octets regardless of codepoint boundaries) hit exactly this path for every non-ASCII property value.

The fix unfolds on the raw bytes first (a byte-level twin UFOLD_BYTES of the existing UFOLD regex) in Contentline.from_ical() and Contentlines.from_ical(), then decodes the assembled line — the recovery the issue demonstrates by hand. String inputs take the previous path unchanged.

This is the reading side of the folding story; the writing side was fixed in #73.

Checklist

• I added a change log entry, following the instructions in all subsections under Change log requirements.
• I followed icalendar's Artificial intelligence policy and disclosed my Responsible AI use in my commit messages, if applicable.
• I added or updated tests, if applicable.
• I ran and ensured all tests pass locally by following Run tests.
• I added or edited documentation as necessary, both as docstrings to be rendered in the API documentation and narrative documentation, following the Style guide.

Additional information

Verification: the new test_fold_inside_multibyte_utf8_sequence_issue_2077 in src/icalendar/tests/test_encoding.py fails on current main (emoji → four U+FFFD) and passes with this change. Full suite: full suite 21,542 passed / 584 failed, of which 558 are timezone-identification and doctest host artifacts and 26 are timezone-data-related tzid/exdate tests — all 584 reproduce identically on unmodified main on this machine (Python 3.13.5, run from a checkout without installing). Four pre-existing failures in test_parses_event_with_non_ascii_tzid_issue_237 (pytz/zoneinfo variants) reproduce identically on unmodified main on this machine (Python 3.13.5, no repo install) and are unrelated to this change. Documentation: this is an internal parser fix with no API change, so no docs were touched.

[github-actions]

Thanks for the pull request, @rahul05ranjan! A maintainer will review it shortly.

[github-actions]

Automation signals

Activity patterns show signs of automation.

View full analysis →

Report this account →

Evidence

• Fork spike detected: Burst of 10 fork events in a single 24-hour window
• Issue comment and PR within minutes: Commented on an issue and opened a PR to the same repo within 5 minutes, in 2 repositories (the fastest took 101s)

Last 2 PRs:

• https://redirect.github.com/trustmybot/plugin/pull/2574
• https://redirect.github.com/laaielichter/Ai-edit-studio/pull/66

_{This is an automated analysis by AgentScan}

[github-actions]

One or more PR-quality checks did not pass. Review the PR Quality Checks summary in the workflow run for the failed checks and their details, then make any necessary updates.

Checklist items may remain unchecked while work is in progress. Please tick each item as you complete it.

If you used AI assistance, follow the project's Responsible AI use policy and provide the required disclosures.

[angatha]

There is potential for unfolding twice. This seems to be redundant. Have you checked, if this is correct the way it is now?

[rahul05ranjan]

Good catch @angatha — you're right, it was redundant for bytes input.

I verified it before changing anything: after UFOLD_BYTES has unfolded the raw bytes, the second UFOLD pass on the decoded string can never match (folds cannot reappear during decoding — confirmed with a 20,000-case fuzz harness over folded multi-byte content: zero differences). So the double unfold was a harmless no-op in the bytes path, but still a wasted pass, and the string unfold remains necessary only when the input is a str.

Restructured in a8eb8ab: bytes input now unfolds once on bytes and skips the string pass entirely; str input keeps the single UFOLD unfold. The issue #2077 regression test passes with the restructure, and the four remaining test_encoding.py failures reproduce identically on the branch without this change (environment-related, unrelated to this PR).