Skip to content

🐛 fix(py_info): discover Python 3.6 and 3.7 interpreters - #117

Merged
gaborbernat merged 1 commit into
mainfrom
fix/discover-old-pythons
Aug 12, 2026
Merged

🐛 fix(py_info): discover Python 3.6 and 3.7 interpreters#117
gaborbernat merged 1 commit into
mainfrom
fix/discover-old-pythons

Conversation

@gaborbernat

@gaborbernat gaborbernat commented Aug 12, 2026

Copy link
Copy Markdown
Member

CI jobs that still target Python 3.7 stopped finding their interpreters once nox adopted python-discovery; closes #116. The script interrogating each candidate runs inside that interpreter, so its syntax floor decides which Pythons are discoverable, independent of requires-python. Because it shared _py_info.py with the library, style modernizations raised that floor to 3.8 without anyone noticing: the standalone rewrite pulled in typing.Final, and a pre-commit autoupdate later added walrus operators. 🐛 Candidates on 3.6 or 3.7 died with a SyntaxError buried at debug level, after two identical query attempts.

The target-side collection now lives in its own module, _py_info_collect.py. Collection supports Python 3.6+, reaching back to RHEL 8's system Python, while the rest of the library keeps its modern style. The file parses down to Python 2.7, and a version gate above every other import reports anything older through a dedicated exit code plus a stderr marker, within the single interrogation call; the host classifies a candidate as too old only when both signals agree, so neither an errno collision nor a shim echoing the phrase can misfire. A static vermin check plus CI jobs that exercise discovery against real python:2.7 through python:3.7 containers pin the floor, closing the door on the lint-autoupdate regression path. The disk cache keys entries on the script hash, so caches written by earlier releases re-query instead of misreading the new payload, in both upgrade directions.

A candidate below the floor logs one warning naming the executable and the version found, skips the retry, and the verdict lands in the disk cache so the interpreter is never spawned again; an absolute-path spec pointing at such an interpreter raises RuntimeError with the same message. ⚡ PythonInfo.from_dict builds instances via __new__, so loading a cache entry no longer performs a throwaway collection of the host interpreter. The docs state both floors: the library runs on Python 3.8+ and discovers interpreters down to 3.6, with the floor moving only in a major release.

@gaborbernat gaborbernat added bug Something isn't working documentation Improvements or additions to documentation labels Aug 12, 2026
@gaborbernat
gaborbernat marked this pull request as draft August 12, 2026 00:47
@gaborbernat
gaborbernat force-pushed the fix/discover-old-pythons branch from 2941653 to 5fb1dc2 Compare August 12, 2026 00:58
@henryiii

Copy link
Copy Markdown

I was playing with this (adding 3.7 only is a 10 line change), then saw you had a patch, so ran a Fable review, pasted below, in case it's helpful.

🤖 AI text below 🤖

The review is done: #117 is fundamentally sound — my Docker testing confirmed every headline claim — but the review surfaced 8 verified findings, four of them correctness issues worth fixing before it leaves draft. All 8 are in the findings panel; the short version:

Correctness (the ones that matter):

  1. Fragile stderr classification (_cached_py_info.py:249) — "too old" is detected by substring-scanning the child's entire stderr for unsupported Python version . Anything else printing that phrase (a sitecustomize, a shim) misclassifies a healthy interpreter and suppresses the recovery retry. A dedicated exit code would be structural instead of prose-coupled; today even rewording the marker in one file silently breaks the other, with no cross-file test.
  2. Version gate runs too late (_py_info_collect.py:9) — module-level import sysconfig (3.2+) and 2.7-floor syntax execute before the gate, so 2.6/3.0/3.1 candidates die without the marker and get the old generic failure path, contradicting the documented "one warning naming the version" promise. Move the gate above the imports and/or scope the docs to 2.7 and 3.2+.
  3. "Too old" is never disk-cached and re-warns (_cached_py_info.py:115) — failures return before the disk-cache write, so every new process re-spawns the old interpreter and warns again, and even in-process cache hits re-emit the warning. The "one warning" and "cached to disk" promises both break for an outcome the code knows is permanent.
  4. Dropped .get() fallback (_py_info_collect.py:142) — the new collector indexes sysconfig_paths[key] directly where the old code fell back to "", so interpreters whose scheme lacks stdlib/platstdlib now crash the probe instead of being discovered.

Hardening/cleanup: the vermin test only asserts py3 is not None rather than <= (3, 6), so the exact lint-autoupdate regression class this PR fixes could recur undetected by the fast suite; the script name/path is duplicated at four call sites and re-hashed per lookup; PythonInfo.__init__ round-trips through to_dict() duplicating from_dict; and the old-target CI matrix builds the same wheel four times.

The agent also explicitly refuted one candidate (a suspected 3.6 UnicodeEncodeErrorjson.dumps is ASCII-safe) and dropped low-value nits. Combined with the green functional matrix from earlier (2.7/3.5 rejected cleanly, 3.6/3.7 discovered, cache compatible both directions, 554 tests passing), my overall take: this is a better design than the minimal patch I'd sketched — findings 1–4 are all small, targeted fixes.

The interrogation script runs inside the candidate interpreter, so its
syntax floor decides which Pythons are discoverable, independent of
requires-python. Sharing one file between the library and that script
tied the floor to the library's style: the standalone rewrite pulled in
typing.Final and a pre-commit autoupdate added walrus operators, both
3.8-only, so 3.6 and 3.7 candidates died with a SyntaxError logged at
debug level after two identical attempts (#116).

Split the target-side collection into _py_info_collect.py. Collection
supports 3.6+, reaching RHEL 8's system Python, while the rest of the
library keeps modern style. The file parses down to Python 2.7 and a
version gate above every other import reports older interpreters with
a dedicated exit code plus a stderr marker, all within the single
interrogation call; the host classifies only when both agree, so an
errno collision or a shim echoing the phrase cannot misfire. The
verdict is permanent, so it is written to the disk cache and warned
about once; the retry that cannot succeed is skipped, and an
absolute-path spec raises the same message. A vermin check, a gate
placement test, and CI jobs against real 2.7 through 3.7 containers
keep lint modernizations from raising the floor again. The cache keys
entries on the script hash, so entries written by earlier releases
re-query instead of misreading the new payload.

PythonInfo.from_dict builds via __new__ instead of running a throwaway
collection, cutting the cost of every cache hit.
@gaborbernat
gaborbernat force-pushed the fix/discover-old-pythons branch from 5fb1dc2 to 7097231 Compare August 12, 2026 04:41
@gaborbernat
gaborbernat marked this pull request as ready for review August 12, 2026 14:03
@gaborbernat
gaborbernat merged commit 0986627 into main Aug 12, 2026
19 checks passed
@gaborbernat
gaborbernat deleted the fix/discover-old-pythons branch August 12, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

python-discovery can't locate the Python 3.7 interpreter (should be either fixed, or more clearly indicated as not supported)

2 participants