Unofficial tracker: misc. `line_profiler.autoprofile` import-profiling bugs

[TTsangSC]

(Related: #434; TTsangSC#6 which is based on the former)

This is a summary of several issues concerning import-time profiling when using line_profiler.autoprofile. I'm working on fixes (see the linked PRs), but this issue is mainly for documenting purposes.

Only the last target profiled in multi-target imports

Click to expand

(See #433.)

In line_profiler/autoprofile/profmod_extractor.py::ProfmodExtractor, ProfmodExtractor.run() is only set up for single-target imports, so in an import statement like

from foo import bar, baz

bar would not be profiled unless a line_profiler/autoprofile/ast_profile_transformer.py::AstProfileTransformer is also used (kernprof -l --prof-imports --prof-mod=<path/to/the/profiled/code>).

Remedies/Mitigations

• FIX: handling multi-target (from-)import statements #434: deprecates ProfmodExtractor.run(), replacing it with .extract_all() which assumes multi-target imports.

Star-imports (from ... import *) result in SyntaxError

Click to expand

This was first reported by @Erotemic and Fable as a part of TTsangSC#5 (see dev/planning/fable-review-fullrepo-2026-07-05.md). Essentially, since ProfmodExtractor and AstProfileTransformer both do these simple substitutions:

• import <foo.bar>
  -> import <foo.bar>; prof.add_imported_function_or_module(<foo.bar>)
• import <foo.bar> as <baz>
  -> import <foo.bar>; prof.add_imported_function_or_module(<baz>)
• from <foo.bar> import <baz>
  -> from <foo.bar> import <baz>; prof.add_imported_function_or_module(<baz>)
• from <foo.bar> import <baz> as <foobar>
  -> from <foo.bar> import <baz> as <foobar>; prof.add_imported_function_or_module(<foobar>)

A star-import of the form from <foo.bar> import * would result in the syntactically wrong from <foo.bar> import *; prof.add_imported_function_or_module(*). As such, if

• The module from which the star-import pulls names is among the --prof-mod targets (kernprof -l --prof-mod=<foo.bar>), or
• Full-module profiling with import rewriting is requested (kernprof -l --prof-imports --prof-mod=<path/to/the/profiled/code>)

line_profiler/autoprofile/autoprofile.py::run() will error out with SyntaxError.

Remedies/Mitigations

• FIX: handling multi-target (from-)import statements #434: includes the stop-gap measure of catching star-imports, filtering them out, and warning the user against them.
• ENH: handle nested imports and star-imports TTsangSC/line_profiler#6:
  • Introduces a new helper function-slash-pseudo-method line_profiler/autoprofile/line_profiler_utils.py::add_star_import() which actually handles (dynamically) recovering and profiling the imported names.
  • Adds a new kernprof flag --prof-star-imports and the config key-value pairs [tool.line_profiler.kernprof]::prof-star-imports and [tool.line_profiler.autoprofile]::prof_star_imports to control whether to interpolate add_star_import() nodes into the profiled-code AST as with add_imported_function_or_module().

Nested imports and top-level imports not handled on equal footing

Click to expand

Consider the following code (say, ./script.py):

from foo import bar

try:
    from spam import ham
except ImportError:
    from other_spam import ham

Suppose that we do kernprof -l --prof-mod=foo,spam ./script.py.

• Since ./script.py isn't in --prof-mod, we don't do a full module rewrite, and just insert prof.add_imported_function_or_module() calls with ProfmodExtractor.
• However, ProfmodExtractor only handles top-level imports, i.e. those directly in the module body without being nested inside compound statements, like the try-except bloc we have.
• As such, only foo.bar will be passed to the profiler, and spam.ham is missed.

Now suppose that we do instead kernprof -l --prof-mod=foo,spam --prof-mod=./script.py --prof-imports ./script.py.

• Since ./script.py is in --prof-mod and --prof-imports is set, we do a full module AST rewrite with AstProfileTransformer.
• While AstProfileTransformer walks the entire AST and thus is able to catch all the import statements, we lose granularity because all three of foo.bar, spam.ham, and other_spam.ham will be profiled (if their resp. import statements are executed), and there is no option to just select the former two and exclude the last.

Remedies/Mitigations

• ENH: handle nested imports and star-imports TTsangSC/line_profiler#6:

  • Adds kernprof flag --prof-nested-imports and config options [tool.line_profiler.kernprof]::prof-nested-imports and [tool.line_profiler.autoprofile.import_discovery]::* to control the profiling of imports (e.g. exclude imports inside function bodies and loops by default).
  • Extends ProfmodExtractor.extract_all() (introduced in FIX: handling multi-target (from-)import statements #434) to also recover import targets nested inside eligible compound statements (as dictated by the above config options), so that --prof-mod options are respected.
  • Updates AstProfileTransformer (used for --prof-imports) so that it also respects the aforementioned config and doesn’t just rewrite arbitrary imports.

  Note though that this changes the semantics of --prof-imports, in that it no longer unconditionally causes imports to be profiled, being now also subjected to scoping restrictions by --prof-nested-imports.

On-import profiling deduplication results in inconsistent profiling

Click to expand

Consider the following code (say, ./script.py):

def foo() -> None:
    from textwrap import indent

    print(indent('foo\n''bar\n''baz', '* '))


def spam() -> None:
    from textwrap import indent

    print(indent('spam\n''ham\n''eggs', '- '))


if __name__ == '__main__':
    spam()

Suppose that we do kernprof -l --prof-mod=./script.py --prof-imports ./script.py. This rewrites the entire module with AstProfileTransformer and supposedly profiles all (1) locally-defined functions and (2) imports nested arbitrarily deep in the AST. Now AstProfileTransformer keeps track of the prof.add_imported_function_or_module() nodes it (or any prior preprocessing, via the profiled_imports: list[str] parameter) interpolated, and avoids profiling the same import target twice (because add_imported_function_or_module() can descend deep into an object and incur substantial overhead). This is well and all, but in the above example, it results in no profiling data being collected for textwrap.indent(), because prof.add_imported_function_or_module(indent) is only inserted in foo() and not bar().

Remedies/Mitigations

• ENH: handle nested imports and star-imports TTsangSC/line_profiler#6:
  • Updates AstProfileTransformer so that:
    • It accepts profiled_imports: list[str] as before, but since this leaves us with insufficient info for deduplication, a DeprecationWarning is issued.
    • It also takes profiled_imports: Mapping[...] which is the same format as returned by ProfmodExtractor.extract_all() (introduced in FIX: handling multi-target (from-)import statements #434), and uses that to do context-aware on-import profiling deduplication.
  • Updates AstTreeProfiler.profile() and how it uses AstProfileTransformer to make use of the fixed feature.