chore(deps): Bump openjd-* Rust crates to 0.7.0 - #359
Conversation
openjd-expr 0.6.0 -> 0.7.0, openjd-model 0.6.1 -> 0.7.0, and openjd-sessions 0.5.6 -> 0.5.7. Three upstream changes land, reconciled against a source diff of the published crates rather than the changelog alone: * openjd-rs#374 escapes control characters, quotes and backslashes in repr_py and repr_python. Reachable here through ExprValue.__repr__, which previously emitted literals CPython cannot parse (a raw newline terminates the string, and a NUL cannot appear in source at all) or that parse as a different value -- 'a\b' is a backspace. * openjd-rs#373 changes FormatString::validate_expressions to take a target_type and return StaticResolution. This is the breaking change the minor bump carries, and the changelog did not mention the new parameter. The binding passes None, matching the resolve and resolve_string calls above, and discards the returned value: the Python method stays pass/fail and still returns None on success. * openjd-rs#369 bounds child output reading in the Windows session helper runner. Not reachable from this repo's tests; openjd-sessions forwards to it from subprocess.rs and its coverage lives upstream. TestExprValueReprEscaping pins the #374 behaviour over 28 inputs -- quotes, backslashes, the C0 controls, DEL, and the non-ASCII characters CPython escapes by category -- using CPython's own repr() as the oracle, which is the contract Expression Language 2.2.6 states. It adds an eval(repr(v)) == v round-trip and negative controls for numeric and keyword text that must pass through unaltered. Reverting the version pins fails 60 of these cases. test_repr_list_path_with_format and test_repr_list_list_path_with_format pinned the old bug: under WINDOWS the normalised '\a' was emitted unescaped, which Python parses as BEL. Both now assert the escaped spelling and round-trip, so nothing regressed -- the bump closed a silent-corruption path those tests had frozen. THIRD-PARTY-LICENSES.txt regenerated with scripts/check_third_party_licenses.sh --update; only the three crate versions changed, with no new transitive dependencies. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
| let lib = profile_for_call(profile); | ||
| self.inner | ||
| .validate_expressions(&st, &lib) | ||
| .validate_expressions(&st, &lib, None) |
There was a problem hiding this comment.
The escaping bug this PR is regression-testing for ExprValue is still live a few lines below, in this same file: PyFormatString::__repr__ (line 136-138) does
format!("FormatString(\"{}\")", self.inner.raw())which interpolates the raw source with no escaping at all — not even for the " delimiter or the backslash. So any format string whose raw text contains a double quote, a backslash, or a control character produces a repr that either does not parse or parses as a different value:
FormatString(hello "world")→FormatString("hello "world"")— unparseable.FormatString(C:\a)→FormatString("C:\a")— parses as BEL, exactly theafailure mode called out in the newtest_repr_list_path_with_formatcomment. Windows-style raw paths in templates hit this routinely.- A raw containing a newline terminates the string literal mid-
repr.
This matters more than usual here because raw() is attacker-influenced template text, and repr() is what ends up in log lines and exception messages.
Given openjd-rs#374 added the escaping writer for ExprValue, would it make sense to route FormatString.__repr__ through the same escaper (and pick the delimiter per CPython, as the new test_repr_list_selects_delimiter_per_element asserts)? Otherwise the two repr implementations in the bindings now disagree on whether their output round-trips.
There was a problem hiding this comment.
Correct, and out of scope for this PR. Reproduced through the binding — all three of your cases:
| raw | repr(FormatString(raw)) |
parses as Python? |
|---|---|---|
hello "world" |
FormatString("hello "world"") |
SyntaxError: invalid syntax |
C:\a |
FormatString("C:\a") |
parses — as BEL |
a\nb |
literal newline mid-repr | SyntaxError: unterminated string literal |
And ExprValue.__repr__ handles all three correctly on this branch, so the two implementations do now disagree on whether their output round-trips. Your reading is right.
Not fixing it here, for two reasons.
First, __repr__ is not in this diff. This PR's hunks in this file are the validate_expressions doc comment and its call site; the format! you quote is pre-existing on mainline and fails identically there. A dependency bump should not carry an unrelated behaviour change to a second method.
Second, the fix you suggest is not currently implementable. The escaping writer #374 added is crate-private:
openjd-expr-0.7.0/src/lib.rs:27: pub(crate) mod py_escape;
openjd-expr-0.7.0/src/py_escape.rs:33: pub(crate) fn write_py_string_literal(...)
So the bindings crate cannot route through it. Closing this properly needs openjd-rs to export the writer (or expose a repr_python-style helper for a bare string) first, then a follow-up here. Routing through ExprValue::String(raw).repr_python() instead would work but embeds the ExprValue(...) wrapper in a FormatString repr, which is worse.
Recorded in the PR description as a known gap with the upstream dependency named, so it does not live only in this thread.
There was a problem hiding this comment.
Correcting my earlier reply on this thread. I said closing this needs openjd-rs to export the escaping writer first. That was wrong about the remedy, though right that py_escape is pub(crate). PyO3 exposes the interpreter's own repr() directly, so no upstream change is needed — and it is a better escaper than the one I was waiting on, since it is correct by construction and is the same oracle these tests assert against. SymbolTable.__repr__ in this crate already delegates to Python's repr, so the pattern was in front of me.
The sessions/ slice is now up as #361, with a py_repr helper the remaining sites can reuse.
Investigating it also showed the problem is wider than this one call site, in a way worth recording here. Two classes:
- No escaping at all, the shape you flagged. 4 sites; 2 are genuinely broken (
FormatString.__repr__,ParsedExpression.__repr__).ExprTypeandRangeExprare safe because their constructors reject anything hostile. - Rust
Debug{:?}, ~23 sites.Debugagrees with Python on the quote, backslash and C0 controls, but renders anything else non-printable as\u{a0}, which CPython rejects outright. 14 of these are reachable from Python. This class is invisible to a reader checking only for a missing escape, which is why the flagged site was the tip of it.
The worst one was not in either of our lists: ActionResult(stdout={:?}) carries captured process output, so a non-ASCII byte in a job's stdout breaks the repr as a matter of course rather than under attack.
One correction to your framing, for the record: this is not reachable via raw() being attacker-influenced in the way FormatString is. Template validation rejects only Cc control characters, so U+00A0 and U+3000 — both Zs — pass into step and job names untouched, which is how they reach these reprs.
Still leaving this thread open: #361 covers sessions/ only, so FormatString.__repr__ itself is unfixed. It is recorded in #359's description as a known gap, and #361's description names the remaining sites.
The doc comment on the binding moved to the 3-argument crate signature but its two derived copies did not, leaving three descriptions of one method disagreeing. _openjd_rs.pyi carried the old wording verbatim, including the paragraphs above it that this branch never touched, which confirms the stub is generated from the doc comment. Regenerated by hand: the local scripts/generate_stubs.sh cannot run on macOS -- its GNU `sed -i` patch step fails on BSD sed, and once applied manually the patch is inert because building stub_gen against the local Python 3.13 defines Py_3_10, so the `#[cfg(Py_3_10)]` guard admits a PyEncodingWarning that this pyo3 does not export. No workflow verifies stub freshness, so CI could not have caught the staleness either way. Also records why "a b" sits in ESCAPING_CASES. It is not a non-ASCII space flattened in editing; it is copied from the upstream py_escape tests, where it is the negative half of the Zs discrimination -- U+0020, U+00A0 and U+3000 are all Zs and CPython escapes only the latter two. The comment now says so, since the enumeration above it listed only the characters that do get escaped. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
| openjd-expr = "0.6.0" | ||
| openjd-model = "0.6.1" | ||
| openjd-sessions = "0.5.6" | ||
| openjd-expr = "0.7.0" |
There was a problem hiding this comment.
The bump silently breaks make_list_err_to_py's message rewriting, because openjd-rs#373 changed the wording of the unresolved-element error in addition to the validate_expressions signature the PR description covers.
rust-bindings/src/expr/expr_value.rs:40-48 parses the upstream headline:
if let Some(rest) = headline.strip_prefix("make_list expected ") {
if let Some((expected, got_part)) = rest.split_once(" element, got ") {
if got_part == "unresolved" { /* reference-parity message */ }In 0.6.0 there was no dedicated unresolved check in make_list, so an unresolved element fell through to the per-type element match and produced make_list expected int element, got unresolved — which matches " element, got ", sets got_part == "unresolved", and yields the documented reference message.
0.7.0 adds an up-front check that short-circuits before the per-type match (crates/openjd-expr/src/value.rs):
if elements.iter().any(Self::is_unresolved) {
return Err(ExpressionError::type_error(
"make_list expected concrete elements, got unresolved",
));
}strip_prefix still succeeds, leaving rest == "concrete elements, got unresolved". That does not contain " element, got " — the text is elements,, so the s breaks the literal — so split_once returns None and the code falls through to headline.to_string(). ExprValue([42, ExprValue.unresolved(ExprType("int"))]) therefore now raises
TypeError: make_list expected concrete elements, got unresolved
instead of the parity message the doc comment on make_list_err_to_py and TestExprValueListConstructionErrors (test/openjd/expr/test_lists.py:822-847) both state is the contract: Cannot construct a list containing unresolved values. Use ExprValue.unresolved() ....
The two regression tests there only assert match="unresolved", and the raw upstream headline happens to contain that word, so both still pass — which is presumably why this went unnoticed. Worth adding the concrete elements spelling to the prefix handling (or matching on got unresolved rather than the " element, got " infix, so a future upstream rewording degrades to the parity message rather than leaking crate-internal make_list phrasing to Python callers), and tightening the tests to assert the full expected message so the next bump catches this.
What changed
Bumps the three
openjd-*Rust crate pins inrust-bindings/Cargo.toml, refreshesCargo.lock, and regeneratesTHIRD-PARTY-LICENSES.txt:openjd-expropenjd-modelopenjd-sessionscargo updatemoved only these three packages, so the license file changes by exactly three lines and no new transitive dependency appears.Upstream changes and how each was verified
The claimed change list came from the release commits, then was reconciled against a source diff of the published crates. That diff surfaced one thing the changelog did not mention: #373 also added a third parameter to
validate_expressions.openjd-rs#374 —
repr_py/repr_pythonescape control characters. Reachable here throughExprValue.__repr__. Before the bump, a value carrying a quote, a backslash or a control character produced a literal CPython cannot parse (a raw newline terminates the string; a NUL cannot appear in source at all) or, worse, one that parses as a different value —'a\b'is a backspace.Verified through the Python binding on 28 inputs using CPython's own
repr()as the oracle, which is the contract Expression Language 2.2.6 states forrepr_py. All pass, including the scalar,list[string],list[path]and nested-list paths.openjd-rs#373 —
FormatString::validate_expressionsreturnsStaticResolution. This is the breaking change the minor bump carries, and it broke the build:The binding now passes
target_type: None— matching theresolveandresolve_stringmethods above, which resolve without a target type, so validation observes exactly the values resolution will produce — and discards the returnedStaticResolution. The Python method is unchanged: pass/fail, returningNoneon success. Exposing the resolved-length bound to Python is a separate feature and is not part of this bump.openjd-rs#369 — bound child output reading in the Windows session helper runner. Not reachable from this repo's tests: it lives in
openjd-sessions' helper binary, needs a live Windows session, andsubprocess.rsis the forwarding call site. Its coverage lives upstream in openjd-rs. Called out as unverified here rather than implied.Tests
TestExprValueReprEscapingpins the #374 behaviour: quotes and backslashes (delimiter selection and doubling), the C0 controls, DEL, and the non-ASCII characters CPython escapes by category — U+0085, U+00A0, U+00AD, U+2028, U+2029, U+3000, U+200B, a private-use character and an astral non-printable — alongside printable non-ASCII that must survive verbatim. Each case is asserted three ways: against CPython'srepr(), as aneval(repr(v)) == vround-trip, and as alist[string]element. Negative controls confirm numeric and keyword text passes through unaltered.Mutation-checked by reverting the version pins (and the
validate_expressionsadaptation they force), rebuilding the extension, and re-running: 60 failed, 33 passed. The tests pin real behaviour rather than restating it. The mutated tree was confirmed to build and import first, so the failure is the behaviour change and not a broken build.Two pre-existing tests changed, and why that is not a regression
test_repr_list_path_with_formatandtest_repr_list_list_path_with_formatfailed after the bump. They had pinned the old bug. UnderPathFormat.WINDOWSthe leading/normalises to\, and the old repr emitted'\a'— which Python parses as BEL, not backslash-a. Their regex'(/|\\)a'accepted that spelling.Both now assert the escaped spelling via CPython's
repr()of the normalised items, and additionally asserteval(repr(v)) == v, which only holds after the fix. Measured before changing them:list[path]WINDOWS['\a', '\b']['\\a', '\\b']TrueNothing regressed; the bump closed a silent-corruption path those two tests had frozen.
Verification
cargo build,cargo fmt --check,cargo clippy --all-targets— cleanxfail_stricttest flipped, so nothing in the suite had recorded these fixes as known gaps.black --check,ruff check,mypy— cleanscripts/check_third_party_licenses.shin verify mode:THIRD-PARTY-LICENSES.txt is up to dateUnverified: the Windows helper-runner change (#369), for the reason given above. Windows CI on this PR exercises the bindings on Windows but not that helper's reader path.
Known gap, not addressed here
Review flagged that
PyFormatString::__repr__(rust-bindings/src/expr/format_string.rs) interpolatesself.inner.raw()into a double-quoted literal with no escaping at all. Reproduced:repr(FormatString(raw))hello "world"FormatString("hello "world"")C:\aFormatString("C:\a")a\nbExprValue.__repr__handles all three correctly on this branch, so the tworeprimplementations in the bindings now disagree on whether their output round-trips. It matters becauseraw()is template-derived text that reaches log lines and exception messages.Deliberately not fixed in this PR. It is pre-existing on mainline (this diff's only hunks in that file are the
validate_expressionsdoc comment and call site), and the natural fix is not currently available: the escaping writer #374 added is crate-private,pub(crate) mod py_escape/pub(crate) fn write_py_string_literal, so the bindings crate cannot route through it. Closing it needs openjd-rs to export the writer or a bare-stringrepr_pythonhelper first, then a follow-up here.