[rahul05ranjan]

• Closes Legacy CHARSET=ISO-8859-1 property parameter is ignored: non-UTF-8 .ics files parse to U+FFFD even though the file declares its encoding #2078

Description

from_ical(bytes) runs the input through to_unicode(), whose errors="replace" fallback fires before property parameters are ever looked at. So when an .ics file is not valid UTF-8 but a property explicitly declares its charset with the legacy CHARSET parameter — the marker pre-RFC 5545 producers left behind — the information needed to decode correctly is available and still ignored:

ics = b"...\r\nSUMMARY;CHARSET=ISO-8859-1:Caf\xe9 Meeting\r\n..."
event = Calendar.from_ical(ics).walk("VEVENT")[0]
str(event["SUMMARY"])   # 'Caf\N{REPLACEMENT CHARACTER} Meeting' — the é is gone

This PR decodes each content line with the charset its CHARSET parameter declares:

• the search for CHARSET= is restricted to the parameter section (before the first :), so a quoted parameter value mentioning CHARSET=... is not mistaken for a directive;
• lines without a CHARSET parameter keep the previous best-effort UTF-8 decoding, byte for byte;
• an unknown charset name falls back to the previous behavior instead of raising.

Old exports from Outlook/Lotus tooling that spell non-UTF-8 text this way still circulate (migration dumps, mailing-list archives); for every non-ASCII value in them the summary/location/description currently comes back as U+FFFD soup.

Checklist

• I added a change log entry, following the instructions in all subsections under Change log requirements.
• I followed icalendar's Artificial intelligence policy and disclosed my Responsible AI use in my commit messages, if applicable.
• I added or updated tests, if applicable.
• I ran and ensured all tests pass locally by following Run tests.
• I added or edited documentation as necessary, both as docstrings to be rendered in the API documentation and narrative documentation, following the Style guide.

Additional information

Verification: three new tests in src/icalendar/tests/test_encoding.py — the latin-1 repro (fails on current main with U+FFFD, passes here), the unknown-charset graceful fallback, and the quoted-CHARSET=-text guard. Targeted run (test_parsing.py, test_components_break_on_bad_ics.py, test_encoding.py, test_bom_calendar.py, tests/prop): 1,909 passed / 12 failed, where all 12 are the timezone-data-related tzid host artifacts that reproduce identically on unmodified main on this machine (Python 3.13.5, run from a checkout without installing; the same 12 are listed in the full-suite evidence on my related #2079). No API change, so no docs were touched.

[github-actions]

Thanks for the pull request, @rahul05ranjan! A maintainer will review it shortly.

[github-actions]

Automation signals

Activity patterns show signs of automation.

View full analysis →

Report this account →

Evidence

• Fork spike detected: Burst of 10 fork events in a single 24-hour window
• Issue comment and PR within minutes: Commented on an issue and opened a PR to the same repo within 5 minutes, in 2 repositories (the fastest took 101s)

Last 2 PRs:

• https://redirect.github.com/trustmybot/plugin/pull/2574
• https://redirect.github.com/laaielichter/Ai-edit-studio/pull/66

_{This is an automated analysis by AgentScan}

[github-actions]

One or more PR-quality checks did not pass. Review the PR Quality Checks summary in the workflow run for the failed checks and their details, then make any necessary updates.

Checklist items may remain unchecked while work is in progress. Please tick each item as you complete it.

If you used AI assistance, follow the project's Responsible AI use policy and provide the required disclosures.

[angatha]

Can you please use the possessive quantifier.

[angatha]

Where does this come from?

[rahul05ranjan]

Thanks for the review @angatha — on both points:

Possessive quantifier: happy to switch the capture group to [^;:\r\n]++, with one compatibility note: possessive quantifiers require Python 3.11+, while pyproject.toml still declares requires-python = ">=3.10" and the CI matrix includes 3.10 (where the pattern would raise re.error at import). If a 3.10 drop is planned I'll apply it right away; otherwise the greedy form is safe in this position — the class already excludes the delimiters and nothing follows the group, so there is no backtracking blow-up to guard against. Which way would you like it?

"Where does this come from?" (test_charset_text_inside_quoted_param_value...): that case is one I constructed defensively — it is not from a real-world file or from issue #2078 itself. The scenario behind the PR (legacy CHARSET= declarations in pre-RFC-5545 Outlook/Lotus exports) comes from the issue; the quoted-value test pins a boundary the CHARSET scan could easily get wrong: a CHARSET= that occurs inside a quoted parameter value (here ALTREP="cid:x;CHARSET=ISO-8859-1") is parameter data, not a directive, so the property must still decode as UTF-8. That follows from the parameter grammar in RFC 5545 §3.1, where quoted strings may contain ";" and "=". If you'd rather see it dropped or replaced with a more realistic sample, happy to adjust.