Skip to content

chore(deps): Bump openjd-* Rust crates to 0.7.0 - #359

Merged
leongdl merged 2 commits into
OpenJobDescription:mainlinefrom
leongdl:chore/bump-openjd-crates-0.7.0
Sep 10, 2026
Merged

chore(deps): Bump openjd-* Rust crates to 0.7.0#359
leongdl merged 2 commits into
OpenJobDescription:mainlinefrom
leongdl:chore/bump-openjd-crates-0.7.0

Conversation

@leongdl

@leongdl leongdl commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What changed

Bumps the three openjd-* Rust crate pins in rust-bindings/Cargo.toml, refreshes Cargo.lock, and regenerates THIRD-PARTY-LICENSES.txt:

Crate From To
openjd-expr 0.6.0 0.7.0
openjd-model 0.6.1 0.7.0
openjd-sessions 0.5.6 0.5.7

cargo update moved only these three packages, so the license file changes by exactly three lines and no new transitive dependency appears.

Upstream changes and how each was verified

The claimed change list came from the release commits, then was reconciled against a source diff of the published crates. That diff surfaced one thing the changelog did not mention: #373 also added a third parameter to validate_expressions.

openjd-rs#374 — repr_py / repr_python escape control characters. Reachable here through ExprValue.__repr__. Before the bump, a value carrying a quote, a backslash or a control character produced a literal CPython cannot parse (a raw newline terminates the string; a NUL cannot appear in source at all) or, worse, one that parses as a different value — 'a\b' is a backspace.

Verified through the Python binding on 28 inputs using CPython's own repr() as the oracle, which is the contract Expression Language 2.2.6 states for repr_py. All pass, including the scalar, list[string], list[path] and nested-list paths.

openjd-rs#373 — FormatString::validate_expressions returns StaticResolution. This is the breaking change the minor bump carries, and it broke the build:

error[E0061]: this method takes 3 arguments but 2 arguments were supplied
error[E0308]: expected `Result<(), PyErr>`, found `Result<StaticResolution, PyErr>`

The binding now passes target_type: None — matching the resolve and resolve_string methods above, which resolve without a target type, so validation observes exactly the values resolution will produce — and discards the returned StaticResolution. The Python method is unchanged: pass/fail, returning None on success. Exposing the resolved-length bound to Python is a separate feature and is not part of this bump.

openjd-rs#369 — bound child output reading in the Windows session helper runner. Not reachable from this repo's tests: it lives in openjd-sessions' helper binary, needs a live Windows session, and subprocess.rs is the forwarding call site. Its coverage lives upstream in openjd-rs. Called out as unverified here rather than implied.

Tests

TestExprValueReprEscaping pins the #374 behaviour: quotes and backslashes (delimiter selection and doubling), the C0 controls, DEL, and the non-ASCII characters CPython escapes by category — U+0085, U+00A0, U+00AD, U+2028, U+2029, U+3000, U+200B, a private-use character and an astral non-printable — alongside printable non-ASCII that must survive verbatim. Each case is asserted three ways: against CPython's repr(), as an eval(repr(v)) == v round-trip, and as a list[string] element. Negative controls confirm numeric and keyword text passes through unaltered.

Mutation-checked by reverting the version pins (and the validate_expressions adaptation they force), rebuilding the extension, and re-running: 60 failed, 33 passed. The tests pin real behaviour rather than restating it. The mutated tree was confirmed to build and import first, so the failure is the behaviour change and not a broken build.

Two pre-existing tests changed, and why that is not a regression

test_repr_list_path_with_format and test_repr_list_list_path_with_format failed after the bump. They had pinned the old bug. Under PathFormat.WINDOWS the leading / normalises to \, and the old repr emitted '\a' — which Python parses as BEL, not backslash-a. Their regex '(/|\\)a' accepted that spelling.

Both now assert the escaped spelling via CPython's repr() of the normalised items, and additionally assert eval(repr(v)) == v, which only holds after the fix. Measured before changing them:

before bump after bump round-trips
list[path] WINDOWS ['\a', '\b'] ['\\a', '\\b'] now True

Nothing regressed; the bump closed a silent-corruption path those two tests had frozen.

Verification

  • cargo build, cargo fmt --check, cargo clippy --all-targets — clean
  • Full suite: 5795 passed, 24 skipped, 3 xfailed, coverage 94.25% against the 94% gate. The 3 xfails are pre-existing and unrelated; no xfail_strict test flipped, so nothing in the suite had recorded these fixes as known gaps.
  • black --check, ruff check, mypy — clean
  • scripts/check_third_party_licenses.sh in verify mode: THIRD-PARTY-LICENSES.txt is up to date

Unverified: the Windows helper-runner change (#369), for the reason given above. Windows CI on this PR exercises the bindings on Windows but not that helper's reader path.

Known gap, not addressed here

Review flagged that PyFormatString::__repr__ (rust-bindings/src/expr/format_string.rs) interpolates self.inner.raw() into a double-quoted literal with no escaping at all. Reproduced:

raw repr(FormatString(raw)) parses as Python?
hello "world" FormatString("hello "world"") SyntaxError: invalid syntax
C:\a FormatString("C:\a") parses — as BEL
a\nb literal newline mid-repr SyntaxError: unterminated string literal

ExprValue.__repr__ handles all three correctly on this branch, so the two repr implementations in the bindings now disagree on whether their output round-trips. It matters because raw() is template-derived text that reaches log lines and exception messages.

Deliberately not fixed in this PR. It is pre-existing on mainline (this diff's only hunks in that file are the validate_expressions doc comment and call site), and the natural fix is not currently available: the escaping writer #374 added is crate-private, pub(crate) mod py_escape / pub(crate) fn write_py_string_literal, so the bindings crate cannot route through it. Closing it needs openjd-rs to export the writer or a bare-string repr_python helper first, then a follow-up here.

openjd-expr 0.6.0 -> 0.7.0, openjd-model 0.6.1 -> 0.7.0, and
openjd-sessions 0.5.6 -> 0.5.7.

Three upstream changes land, reconciled against a source diff of the
published crates rather than the changelog alone:

* openjd-rs#374 escapes control characters, quotes and backslashes in
  repr_py and repr_python. Reachable here through ExprValue.__repr__,
  which previously emitted literals CPython cannot parse (a raw newline
  terminates the string, and a NUL cannot appear in source at all) or
  that parse as a different value -- 'a\b' is a backspace.
* openjd-rs#373 changes FormatString::validate_expressions to take a
  target_type and return StaticResolution. This is the breaking change
  the minor bump carries, and the changelog did not mention the new
  parameter. The binding passes None, matching the resolve and
  resolve_string calls above, and discards the returned value: the
  Python method stays pass/fail and still returns None on success.
* openjd-rs#369 bounds child output reading in the Windows session
  helper runner. Not reachable from this repo's tests; openjd-sessions
  forwards to it from subprocess.rs and its coverage lives upstream.

TestExprValueReprEscaping pins the #374 behaviour over 28 inputs --
quotes, backslashes, the C0 controls, DEL, and the non-ASCII characters
CPython escapes by category -- using CPython's own repr() as the oracle,
which is the contract Expression Language 2.2.6 states. It adds an
eval(repr(v)) == v round-trip and negative controls for numeric and
keyword text that must pass through unaltered. Reverting the version
pins fails 60 of these cases.

test_repr_list_path_with_format and test_repr_list_list_path_with_format
pinned the old bug: under WINDOWS the normalised '\a' was emitted
unescaped, which Python parses as BEL. Both now assert the escaped
spelling and round-trip, so nothing regressed -- the bump closed a
silent-corruption path those tests had frozen.

THIRD-PARTY-LICENSES.txt regenerated with
scripts/check_third_party_licenses.sh --update; only the three crate
versions changed, with no new transitive dependencies.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl requested a review from a team as a code owner September 10, 2026 18:57
let lib = profile_for_call(profile);
self.inner
.validate_expressions(&st, &lib)
.validate_expressions(&st, &lib, None)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The escaping bug this PR is regression-testing for ExprValue is still live a few lines below, in this same file: PyFormatString::__repr__ (line 136-138) does

format!("FormatString(\"{}\")", self.inner.raw())

which interpolates the raw source with no escaping at all — not even for the " delimiter or the backslash. So any format string whose raw text contains a double quote, a backslash, or a control character produces a repr that either does not parse or parses as a different value:

  • FormatString(hello "world")FormatString("hello "world"") — unparseable.
  • FormatString(C:\a)FormatString("C:\a") — parses as BEL, exactly the a failure mode called out in the new test_repr_list_path_with_format comment. Windows-style raw paths in templates hit this routinely.
  • A raw containing a newline terminates the string literal mid-repr.

This matters more than usual here because raw() is attacker-influenced template text, and repr() is what ends up in log lines and exception messages.

Given openjd-rs#374 added the escaping writer for ExprValue, would it make sense to route FormatString.__repr__ through the same escaper (and pick the delimiter per CPython, as the new test_repr_list_selects_delimiter_per_element asserts)? Otherwise the two repr implementations in the bindings now disagree on whether their output round-trips.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and out of scope for this PR. Reproduced through the binding — all three of your cases:

raw repr(FormatString(raw)) parses as Python?
hello "world" FormatString("hello "world"") SyntaxError: invalid syntax
C:\a FormatString("C:\a") parses — as BEL
a\nb literal newline mid-repr SyntaxError: unterminated string literal

And ExprValue.__repr__ handles all three correctly on this branch, so the two implementations do now disagree on whether their output round-trips. Your reading is right.

Not fixing it here, for two reasons.

First, __repr__ is not in this diff. This PR's hunks in this file are the validate_expressions doc comment and its call site; the format! you quote is pre-existing on mainline and fails identically there. A dependency bump should not carry an unrelated behaviour change to a second method.

Second, the fix you suggest is not currently implementable. The escaping writer #374 added is crate-private:

openjd-expr-0.7.0/src/lib.rs:27:      pub(crate) mod py_escape;
openjd-expr-0.7.0/src/py_escape.rs:33: pub(crate) fn write_py_string_literal(...)

So the bindings crate cannot route through it. Closing this properly needs openjd-rs to export the writer (or expose a repr_python-style helper for a bare string) first, then a follow-up here. Routing through ExprValue::String(raw).repr_python() instead would work but embeds the ExprValue(...) wrapper in a FormatString repr, which is worse.

Recorded in the PR description as a known gap with the upstream dependency named, so it does not live only in this thread.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting my earlier reply on this thread. I said closing this needs openjd-rs to export the escaping writer first. That was wrong about the remedy, though right that py_escape is pub(crate). PyO3 exposes the interpreter's own repr() directly, so no upstream change is needed — and it is a better escaper than the one I was waiting on, since it is correct by construction and is the same oracle these tests assert against. SymbolTable.__repr__ in this crate already delegates to Python's repr, so the pattern was in front of me.

The sessions/ slice is now up as #361, with a py_repr helper the remaining sites can reuse.

Investigating it also showed the problem is wider than this one call site, in a way worth recording here. Two classes:

  • No escaping at all, the shape you flagged. 4 sites; 2 are genuinely broken (FormatString.__repr__, ParsedExpression.__repr__). ExprType and RangeExpr are safe because their constructors reject anything hostile.
  • Rust Debug {:?}, ~23 sites. Debug agrees with Python on the quote, backslash and C0 controls, but renders anything else non-printable as \u{a0}, which CPython rejects outright. 14 of these are reachable from Python. This class is invisible to a reader checking only for a missing escape, which is why the flagged site was the tip of it.

The worst one was not in either of our lists: ActionResult(stdout={:?}) carries captured process output, so a non-ASCII byte in a job's stdout breaks the repr as a matter of course rather than under attack.

One correction to your framing, for the record: this is not reachable via raw() being attacker-influenced in the way FormatString is. Template validation rejects only Cc control characters, so U+00A0 and U+3000 — both Zs — pass into step and job names untouched, which is how they reach these reprs.

Still leaving this thread open: #361 covers sessions/ only, so FormatString.__repr__ itself is unfixed. It is recorded in #359's description as a known gap, and #361's description names the remaining sites.

Comment thread rust-bindings/src/expr/format_string.rs
Comment thread test/openjd/expr/test_expression_value.py
The doc comment on the binding moved to the 3-argument crate signature
but its two derived copies did not, leaving three descriptions of one
method disagreeing.

_openjd_rs.pyi carried the old wording verbatim, including the
paragraphs above it that this branch never touched, which confirms the
stub is generated from the doc comment. Regenerated by hand: the local
scripts/generate_stubs.sh cannot run on macOS -- its GNU `sed -i` patch
step fails on BSD sed, and once applied manually the patch is inert
because building stub_gen against the local Python 3.13 defines Py_3_10,
so the `#[cfg(Py_3_10)]` guard admits a PyEncodingWarning that this pyo3
does not export. No workflow verifies stub freshness, so CI could not
have caught the staleness either way.

Also records why "a b" sits in ESCAPING_CASES. It is not a non-ASCII
space flattened in editing; it is copied from the upstream py_escape
tests, where it is the negative half of the Zs discrimination -- U+0020,
U+00A0 and U+3000 are all Zs and CPython escapes only the latter two.
The comment now says so, since the enumeration above it listed only the
characters that do get escaped.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl merged commit ffd3f22 into OpenJobDescription:mainline Sep 10, 2026
31 checks passed
Comment thread rust-bindings/Cargo.toml
openjd-expr = "0.6.0"
openjd-model = "0.6.1"
openjd-sessions = "0.5.6"
openjd-expr = "0.7.0"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bump silently breaks make_list_err_to_py's message rewriting, because openjd-rs#373 changed the wording of the unresolved-element error in addition to the validate_expressions signature the PR description covers.

rust-bindings/src/expr/expr_value.rs:40-48 parses the upstream headline:

if let Some(rest) = headline.strip_prefix("make_list expected ") {
    if let Some((expected, got_part)) = rest.split_once(" element, got ") {
        if got_part == "unresolved" { /* reference-parity message */ }

In 0.6.0 there was no dedicated unresolved check in make_list, so an unresolved element fell through to the per-type element match and produced make_list expected int element, got unresolved — which matches " element, got ", sets got_part == "unresolved", and yields the documented reference message.

0.7.0 adds an up-front check that short-circuits before the per-type match (crates/openjd-expr/src/value.rs):

if elements.iter().any(Self::is_unresolved) {
    return Err(ExpressionError::type_error(
        "make_list expected concrete elements, got unresolved",
    ));
}

strip_prefix still succeeds, leaving rest == "concrete elements, got unresolved". That does not contain " element, got " — the text is elements,, so the s breaks the literal — so split_once returns None and the code falls through to headline.to_string(). ExprValue([42, ExprValue.unresolved(ExprType("int"))]) therefore now raises

TypeError: make_list expected concrete elements, got unresolved

instead of the parity message the doc comment on make_list_err_to_py and TestExprValueListConstructionErrors (test/openjd/expr/test_lists.py:822-847) both state is the contract: Cannot construct a list containing unresolved values. Use ExprValue.unresolved() ....

The two regression tests there only assert match="unresolved", and the raw upstream headline happens to contain that word, so both still pass — which is presumably why this went unnoticed. Worth adding the concrete elements spelling to the prefix handling (or matching on got unresolved rather than the " element, got " infix, so a future upstream rewording degrades to the parity message rather than leaking crate-internal make_list phrasing to Python callers), and tightening the tests to assert the full expected message so the next bump catches this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants