Skip to content

fix(python): keep builtin-named members of absolutely imported project modules - #2632

Closed
BGamboa13 wants to merge 1 commit into
colbymchenry:mainfrom
BGamboa13:fix/python-src-layout-module-members
Closed

BGamboa13 wants to merge 1 commit into
colbymchenry:mainfrom
BGamboa13:fix/python-src-layout-module-members

Conversation

@BGamboa13

Copy link
Copy Markdown

fix(python): keep builtin-named members of absolutely imported project modules

Problem

A Python project module's function named like a built-in method — get, write, update, items, find, read… — was dropped as a dict/list/file method whenever the module was imported by its absolute package path, which is how src-layout projects import their own code. Only relative imports were recognised as project modules.

# src/app/db.py
def get(key): ...
# tests/test_x.py
from app import db
db.get("k")            # before: no calls edge (treated as dict.get)   after: -> app/db.py::get
import app.store as store
store.write(rec)       # same

What changes

  • isPythonProjectModule falls back to the dotted-module file lookup (findPythonModuleFile), as resolvePythonModuleMember already did.
  • To keep that from linking .get() / .write() on an unrelated object, a receiver that the call's own function rebinds is no longer treated as the module: parameter, with … as / for / except … as target, plain/augmented/annotated assignment, walrus, del, a nested def/class of that name, a comprehension or lambda variable on the call's own statement, a function-local import of a different module under the same name, a match capture. Keyword arguments, comments, strings and sibling or nested function bodies never count as rebinding. An enclosing function's locals apply to a nested def.
  • Function scopes are built once per file (memoized per context, cleared with the other import memos), so indexing time is unchanged.

Resolution only: no extraction change, EXTRACTION_VERSION untouched (main is already at the unreleased 28).

Precision evidence

  • A 1,544-file src-layout Python project: 2,261 new edges (2,247 calls, 14 references). Population check: 2,261/2,261 have a receiver that is the imported project module (checked with the AST, independent of the resolver); 0 receivers rebound in scope. 54/54 hand-checked edges true (100%, 93.4–100% at 95%).
  • store.write incoming edges 41 → 1,675 with 0 false positives; file/socket/buffer .write() calls stay unresolved. Other targets (store.get, doctrine.get, registry.find, …) show the same pattern; edges counted there that were not added by this change already existed on main (e.g. 129 dict.get-style edges to doctrine.get).
  • Coverage of the class: of 2,557 builtin-named alias.name uses where alias is a project module, 2,556 now resolve.
  • Deliberate unresolved-over-wrong (14 edges). Against the first version of this change the shadow guard removes 14 edges on that project, all cases where the receiver is a function-local re-import of a different module under the same name or a match capture; 0 of the removed edges were false. A guard that over-matched on keyword args / comprehensions / nested-def parameters / comments (it dropped 185 correct edges) was tightened with regression tests before this PR.
  • Known limits: a function-local import other as db of a different module is only caught when the file binds that name to more than one module; case db: captures and exotic rebindings are treated conservatively (unresolved). Remaining unresolved uses on that project are class/annotation/value references (pytest.raises(store.Err), except store.Err, annotations), which are unchanged pre-existing behaviour.

Benchmarks

Index of the 1,544-file project: 45.1 s [42.6–65.4] → 41.4 s [37.6–45.4], peak RSS 3.4 → 3.0 GB, DB 137.7 → 137.9 MB (noise; no regression). An earlier version of the scope check re-blanked the whole file per function and doubled index time (≈51 s → ≈115 s); the memoized per-file scope tree fixed that (final: 41.3 s vs 40.5–41.5 s baseline).

Tests

__tests__/python-module-member-builtin-name.test.ts (8 tests): absolute from pkg import mod; import pkg.mod as m; shadowing by param / with as / assignment / for / except as / lambda; receiver text that only looks rebound (kwarg, comprehension, nested-def param, comment, local import of the same module, lambda elsewhere); rebindings that must stay unresolved (comprehension target equal to receiver on the call's statement, closure over an outer param, annotated and augmented assignment, local import of a different module, match capture). Against plain main, 6 of 8 fail. Python-named suites, frameworks-integration, pr19-improvements: 15 files pass. Full suite: 6,898 passed, 2 failed (bundle-launcher, host lacks zip; same on main).

Relation to existing issues/PRs

Not included

Package-root discovery for src layouts (#1921), aliased from-imports (#1453), class/annotation/value references through a module (mod.Class), exceptions in except.

…t modules

`from app import db; db.get(...)` and `import app.db as store;
store.write(...)` were dropped as dict/list/file method calls whenever
the module was imported by its absolute package path (how src-layout
projects import their own code), because only relative imports were
recognised as project modules. Fall back to the dotted-module file
lookup, as the module-member resolver already does.

To keep that from linking `.get()`/`.write()` on an unrelated object, a
receiver the call's own function rebinds is no longer treated as the
module: a parameter, `with ... as` / `for` / `except ... as` target,
assignment (plain, augmented, annotated), walrus, `del`, a nested
def/class of that name, a comprehension or lambda variable on the
call's statement, a local re-import of another module under the same
name, or a `match` capture. Kwargs, comments, strings and sibling or
nested function bodies never count. Function scopes are built once per
file (memoized per context) so indexing time is unchanged.
colbymchenry added a commit that referenced this pull request Oct 30, 2026
…t that module (#3046)

A Python call through a name the file imports as a module, like
other.update(...) after `from pkg import other`, linked to the module's
function whatever the calling function bound `other` to: a parameter, a
local, a with/except/loop target, a walrus, a case capture, a lambda or
comprehension variable. resolveViaImport read the file's import mappings
alone (its only scope check, pythonFunctionModuleAlias from PR #2600,
covers a function-local `import a.b as X`), and isPythonProjectModule,
which lets a builtin-named call like update/get/write past the builtin
filter for a project module, asked the same file-wide question. The
mappings also list from-imports at any depth ahead of plain imports, so a
function-local `from pkg import third as other` became every function's
`other`.

pythonReceiverBinding reads how the code around a reference binds the
name: the reference's own statement (lambda parameters, comprehension
targets), then each enclosing function, innermost first, through its
parameters and own statements (assignment, annotation, for, as, walrus,
del, nested def/class, case captures, imports; global keeps the module's
binding), memoized per function. pythonImportsInScope gives
resolveViaImport's Python branch and isPythonProjectModule only the
receiver's mappings that apply there.

Validation:
- __tests__/python-receiver-rebinding.test.ts, 80 tests (default and
  wasm, LF and CRLF): 48 fail on the base main (ten rebinding shapes,
  the local from-import, the cross-function leak), 32 pass (look-alikes:
  keyword argument, string, comment, comprehension over the module, lambda
  beside the call, global, keyword pattern, same-module local import);
  all pass with the fix. The 26 Python-related test files pass.
- A/B, main vs fix: django's own package (930 files, kernel path) changes
  2 edges' confidence (0.85 to 0.8, same targets: a closure over a
  parameter `apps` and a local `apps = model._meta.apps`); flask drops
  one cli.group() call and lowers 4 to 0.7 with the same targets (a
  nested `def cli` and a pytest fixture parameter `app` named like
  imports); requests, click, mealie and fastapi (package plus tests), and
  the express and gin controls are byte-identical.
- django package index time, median of 3 interleaved: 11.3 s main,
  10.7 s fix.

Reimplements PR #2632 by BGamboa13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry

Copy link
Copy Markdown
Owner

Thanks for this careful work, @BGamboa13. The absolute-import half was covered by #1921 in the meantime, and the rebinding guard you designed is reimplemented in #3046 on top of the existing scope readers, with you credited in the changelog: a parameter or local named like an imported module is no longer taken for that module. It also stops a function-local from … import … as from leaking into the rest of the file, which your fixtures helped find. Closing this in favor of #3046; it ships in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants