POC: [Python] try out stubgen-pyx for auto-generated typing stubs

[jorisvandenbossche]

I recently stumbled upon https://github.com/jon-edward/stubgen-pyx, and this might be an interesting option to explore for the pyarrow typing situation.

It is essentially an improved stubgen specifically for cython code (in the past, when we explored options here, the existing stubgens didn't work sufficiently well for cython to be considered). And also https://github.com/NVIDIA/cuda-python is (experimenting with) using this.

This PR, just for illustration purposes, includes the full generated output for the entire pyarrow package (with .pyi files located next to the cython file, but this could of course be put elsewhere as well): stubgen-pyx python/pyarrow/*.pyx.

The interesting part that I learned from looking at the cuda-python code, is that you can actually use python type annotations inside the cython code, and that way we could have inline annotations. Whatever information the tool cannot extract from the cython information, we can augment with adding python type annotations. So essentially that would give an option to have all the custom type annotations inline (which the current PRs for #32609, such as #48622, are putting in separate stub files), and have the stub files be fully generated (i.e. allow us to have only one source of truth / one place to edit).

To illustrate that part, I am going to open a separate PR with just the generated stub file for types.pxi (so it can be compared to #48622), and with a few customizations to improve the generated stubs -> #51681

[rok]

It would be really great to automatically extract stubs from cython!

Whatever information the tool cannot extract from the cython information, we can augment with adding python type annotations.

By that you mean we add more precise type information or do you mean we additionally annotate symbols that are dynamically created at import time (e.g. all kernels)?
If we want to annotate dynamically generated symbols directly we could do that in a separately versioned .pyi files and dynamically inject them with docstrings at wheel build time as we do now. But then, maintenance wise, did we gain enough?

[jorisvandenbossche]

Specifically for the example of dynamically create compute kernels: those are actually pure-python functions (generated in compute.py, not in _compute.pyx), and so I assume it should be possible to update that generation code to directly include type annotations as well (e.g., we now also generate a Signature to add to the generated function.

But in general, I was talking about the cython files. As an example, see the second commit in #51681, which adds some additional type annotations in the cython file (for things that the stubgen cannot infer from the cython type defs), and so are then included in the generated stub file.