Add getLibraryApi and getStdlibApi: extract Python library & stdlib public-API types

[knutwannheden]

Motivation

ty-types currently exposes per-AST-node type attribution for a single file (getTypes). To build type tables for Python libraries — the Python analogue of how Moderne builds Java type tables by scanning a JAR's class files with ASM into JavaType objects — we need a different output shape: the public API surface (module-level declarations, including class members), not the inferred type of every expression in every method body.

This PR adds two methods:

• getLibraryApi — extracts one installed distribution from site-packages.
• getStdlibApi — extracts the standard library (from ty's vendored typeshed) for the project's configured Python version.

Classes defined outside the extraction unit (stdlib/typeshed, other distributions, or other stdlib modules) are emitted as a lightweight classRef rather than fully expanded — mirroring the self-contained type-table model where a referenced-but-not-defined class is a TAG_CLASS_REF. The emitted JSON is consumed Java-side, where the existing TypeTableWriter serializes it to types.bin; reproducing that format (minimal-perfect-hash index, ZSTD framing, the JavaType graph) from Rust was deliberately avoided.

A distribution installs several roots, not one: mypy ships mypy/ and mypyc/, pytest ships _pytest/, pytest/ and a bare py.py. getLibraryApi therefore takes a set of roots and spans the boundary across their union, so a class defined in one root and referenced from another stays a full classLiteral. Extracting one root at a time instead would force the consumer to merge results with disjoint type-id spaces and reunite each cross-root class with its body — and a merge that misses one, or keys on a descriptor lacking a qualified name, lets the V3 writer's first-wins FQN dedup alias the real class body onto the body-less ref, producing a member-less class that still reports success. In mypy, 143 distinct mypy.* classes are referenced across that boundary from mypyc.

Examples

One distribution, several roots (after initialize with a project/venv root from which the packages resolve):

{"jsonrpc":"2.0","method":"getLibraryApi","params":{"roots":["/path/to/site-packages/mypy","/path/to/site-packages/mypyc"]},"id":2}

A root may be a package directory or a bare module file, so pytest's three roots extract in one call:

{"jsonrpc":"2.0","method":"getLibraryApi","params":{"roots":["/site-packages/_pytest","/site-packages/pytest","/site-packages/py.py"]},"id":2}

{
  "modules": [
    { "name": "mypkg.core", "file": "mypkg/core.py",
      "symbols": [ {"name": "Widget", "typeId": 12}, {"name": "make", "typeId": 30} ] }
  ],
  "types": {
    "12": { "kind": "classLiteral", "className": "Widget", "moduleName": "mypkg.core",
            "members": [ {"name": "size", "typeId": 7} ] },
    "7":  { "kind": "instance", "className": "int", "classId": 9 },
    "9":  { "kind": "classRef", "className": "int", "moduleName": "builtins" }
  }
}

Two visibility filters are on by default, each with an opt-out, since a dependency table can be consumed by first-party code that imports library internals:

{"jsonrpc":"2.0","method":"getLibraryApi","params":{"roots":["/site-packages/_pytest"],"includePrivateModules":true,"includeNonExportedSymbols":true},"id":2}

Standard library — modules selects the local unit; everything else (other stdlib modules, builtins when not requested) becomes a classRef. Omit modules for a whole-stdlib dump.

{"jsonrpc":"2.0","method":"getStdlibApi","params":{"modules":["os","collections"]},"id":3}

In both cases, in-unit classes are full classLiterals with members; out-of-unit classes collapse to the new classRef descriptor (identity only).

Summary

• getLibraryApi (src/library.rs): walks each root for .py/.pyi modules (preferring .pyi), enumerates each module's public top-level symbols, and registers their types. roots accepts package directories and bare module files alike; a root named explicitly is extracted whatever its name. root names a single root and is unioned with roots. Modules reached through more than one root are emitted once.
• Visibility opt-outs: includePrivateModules keeps modules with an underscore-prefixed path component; includeNonExportedSymbols keeps module-level symbols that __all__ omits (or, with no __all__, underscore-prefixed ones). Both default to off, so the default output is unchanged.
• LibraryModuleInfo.file is relative to its root's parent (mypkg/core.py), which is the shape the design doc specifies and keeps same-named modules from sibling roots (mypy/main.py vs mypyc/main.py) distinct.
• getStdlibApi (src/library.rs): discovers stdlib modules via ty_module_resolver::all_modules filtered to the standard-library search path (no filesystem walk — typeshed is vendored), excludes _typeshed. Version comes from the initialize project config.
• Public-symbol filtering: respects __all__ when defined; otherwise drops underscore-prefixed names (what ty applies for from x import *). Shared between both methods.
• One entry per symbol name: ty's all_end_of_scope_members chains every end-of-scope declaration with every end-of-scope binding, so a name that is both declared and bound (import, annotated assignment, def, class) arrives twice. Keeping the first gives the declared type precedence, matching ty's own resolution. On six.py this is 111 entries for 72 names; three of them (advance_iterator, callable, print_) differ between the two entries — the def's declared signature versus a union across the if PY3: branches — so deduping is a real precedence choice, not just deflation.
• Boundary-aware registry (src/registry.rs): a Boundary is either UnderRoots(paths) (distribution extraction) or Modules(set) (stdlib extraction). Classes outside the boundary emit the new classRef descriptor. getTypes/getTypeRegistry are unchanged (no boundary).
• New classRef TypeDescriptor variant (src/protocol.rs): className + moduleName + qualifiedName, maps 1:1 to the type-table TAG_CLASS_REF. The qualified name is what joins a ref to the classLiteral carrying the same class's body: a nested class is mod.Outer.Inner on both sides, where moduleName + className alone would rebuild only mod.Inner.
• ruff submodule bump: the ty-types-2 fork now widens dunder_all module visibility (via the existing widen_ty_visibility.sh fix-up list) so __all__ is reachable; adds ty_python_core as a dependency for global_scope.

Known gap, pre-existing and left for its own change: specialForm descriptors carry a bare name and no qualified name, so consumers can still mint colliding names for them.

Test plan

• cargo test — 61 integration tests pass, including:
  • Multi-root: a class defined in root A and referenced from root B is a full classLiteral with members when both roots are given, and a classRef when only B is — the two tests are each other's control.
  • Symbol uniqueness: a module whose statements both declare and bind (import, x: int = 1, def, class) emits each name once, and the surviving entry carries the declared type (int, not Literal[1]) — the second test fails if the dedupe keeps the binding instead.
  • Cross-boundary FQN join: a nested class aliased across roots comes back as a classRef whose qualifiedName is the full dotted path when its defining root is excluded, and as a classLiteral with that same qualifiedName when both roots are given.
  • File roots: a bare module file extracts as one module, including an underscore-named one, since naming a root explicitly is the request.
  • Visibility flags: a private submodule and a public-named symbol absent from __all__ appear only with their respective flag set; each flag leaves the other filter alone.
  • getLibraryApi: lists modules + class symbols; excludes underscore-private modules/packages; prefers .pyi; __all__/underscore symbol filtering; in-package class → full classLiteral, typeshed int → classRef; cross-module in-package class (sibling import) stays a full classLiteral.
  • getStdlibApi: single requested module → its classes full, referenced builtins.str → classRef; multi-module local set (["string","builtins"]) → str full; whole-stdlib dump (no modules) includes os/sys/collections/builtins.
• Each new test confirmed to fail against the pre-change source, so none passes vacuously.
• Verified against real distributions in a venv (release build): extracting mypy + mypyc together drops mypy.* classRefs from 143 to 5, and all 5 remaining are mypy_extensions — a separate distribution installed as its own bare module, correctly outside both roots. mypy/main.py and mypyc/main.py come back as distinct file values.
• Verified against real distributions: pytest's three roots (_pytest, pytest, py.py) extract in one call, py.py included; on _pytest the flags are independent and additive (59 → 78 modules with includePrivateModules, 3794 → 4286 symbols with includeNonExportedSymbols).
• Back-compat: the legacy single root key still extracts (867 modules for mypy).
• Existing getTypes/getTypeRegistry tests remain green (boundary unused there; the boundary generalization is a pure refactor).
• Verified on real six.py: 111 symbols → 72, no name repeated, and advance_iterator/callable/print_ resolve to their declared def signatures.
• cargo clippy --all-targets clean for this crate
• Merged current main (Emit qualifiedName for enum, TypedDict, NewType and nested classes #21, qualifiedName for enums/TypedDicts/NewTypes/nested classes); one conflict in the GenericAlias arm of src/registry.rs, resolved by keeping both sides.