Repository navigation
Test and document the on-premise packages on free-threaded Python - #243
Draft
jwrosewell wants to merge 2 commits into
Draft
jwrosewell wants to merge 2 commits into
jwrosewell wants to merge 2 commits into
Conversation
A new workflow builds the Device Detection and IP Intelligence on-premise packages on free-threaded Python 3.13 and 3.14 on Ubuntu, and 3.14 on Windows, and runs their tests with the free Lite data files. The SWIG wrappers declare that the native modules run without the GIL only when SWIGPYTHON_NOGIL is defined, and it is not, so Python turns the GIL back on when it loads them. tests/test_free_threaded.py in each package checks, in a new interpreter, that the GIL is off before the native module loads, on after, and that Python warns about it. The tests skip on every other build of Python.
Each on-premise readme and SwigData docstring says that free-threaded Python is supported, that loading the native module turns the GIL back on with a RuntimeWarning, and that the package therefore gains nothing from free-threading. The readmes give the warning in full, and both say how to start with the GIL on and so without the warning, which is PYTHON_GIL=1 or -X gil=1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
James Rosewell asked for free-threaded Python to be documented and tested. Neither on-premise package says what happens on a free-threaded build (Python 3.13t or 3.14t), and nothing tests one.
Both SWIG wrappers carry the declaration that a module runs without the GIL, but only when
SWIGPYTHON_NOGILis defined, whichswig -nogildoes and these wrappers were not generated with. It is athash_python_wrap.cxxlines 32974 to 32980 andipi_python_wrap.cxxlines 49165 to 49171, with the matchingPyUnstable_Module_SetGILcall at lines 33067 to 33071 and 49258 to 49262. Neither wrapper definesSWIG_PYTHON_THREADS, and neithersetup.pydefines either name. So on a free-threaded build Python turns the GIL back on when it loads each native module.What was confirmed
On Windows, with free-threaded Python 3.14.10 and both native modules built for it from this checkout's
cxxat 178371ac with MSVC, importing each module in a new interpreter gave the following.sys._is_gil_enabled()beforefiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModulefiftyone_ipintelligence_onpremise.IpIntelligenceEngineModuleWith
PYTHON_GIL=1, or with-X gil=1, the GIL was on before and after the import of each module and there was no warning. WithPYTHON_GIL=0it stayed off after the import, with no warning, which is the state the new tests refuse.The rule
The thread safety page of the specification, in specifications#190, says at 1477eaf7, lines 13 to 15 that
1477eaf7 is that draft's head, and the same lines are at its parent 9c9afb6.
What changed
This draft is on top of #241, because the docstring paragraph it adds follows the paragraph #241 adds to the same
SwigDatadocstrings, and a paragraph added onrcthere would conflict with #241. It is opened against #241's branch and retargets torconce #241 merges.Two commits.
The changes, file by file:
.github/workflows/free-threaded-python.yml, runs on a pull request or a push torcormainthat changes either on-premise package, the pipeline packages they use,cxxor the workflow, and on demand. On free-threaded Python 3.13 and 3.14 on Ubuntu and 3.14 on Windows, it checks the build is free-threaded with the GIL off at start, fetches the free Lite data files with the data repository's own script, builds both native modules, and runs both packages' own tests with-rs, so a skipped test shows.tests/test_free_threaded.pyin each on-premise package imports the native module in a new interpreter, withPYTHON_GILremoved from its environment, and asserts thatsys._is_gil_enabled()was False before the import and is True after it, and that the import gave a RuntimeWarning naming the native module. So a change that lets the module load with the GIL still off fails. The tests skip on any other build of Python, so the existing builds are unchanged apart from two skipped tests in each package.PYTHON_GIL=1or run with-X gil=1to start with the GIL on and so without the warning, that the package has not been tested with the GIL kept off byPYTHON_GIL=0, and that the results of one request are still not safe to use from several threads at once.SwigDataclass docstring has a paragraph saying the same, after the paragraph from Results instances are not thread-safe, so reads take no lock #241.How it was verified
Only the two packages this changes were tested locally, on Windows with free-threaded Python 3.14.10, both native modules built for it, and the free Lite data files (
51Degrees-LiteV4.1.hash,51Degrees-LiteV41.ipiand the free ASN file). No licence key, resource key, data file variable orPYTHON_GILwas set.pytest -rs tests, 105 passed and 14 subtests passed, none skipped.pytest -rs tests, 80 passed and 13 subtests passed, none skipped.test_free_threaded.pyskips both of its tests in each package.The new workflow passed on this head in run 44051575693, on free-threaded Python 3.13.18 and 3.14.10 on Ubuntu and 3.14.10 on Windows. Each of the three jobs gave 105 passed for Device Detection and 80 passed for IP Intelligence, none skipped. The quick checks passed too, being UTM link lint (run 44051575485), environment variable names (run 44051575608) and only rc into main (run 44051576242).
This test guards behaviour the packages already have, so it has no red on
rc. To show it would fail, a throwaway branch at 41d3324 changed the GIL assertion in both packages to expect the GIL still off after the import, and ran this workflow on a push to that branch, in run 44051719311. All three jobs failed, each package withAssertionError: 'False False' != 'False True'attests/test_free_threaded.pyline 71, giving 1 failed and 104 passed for Device Detection and 1 failed and 79 passed for IP Intelligence. The throwaway branch ran this workflow itself, as the common-ci build runs on standard Python, where these tests skip. The branch is deleted.No full build was started for this draft.
What remains
rc.swig -nogiland showing the native code and the wrappers are safe without the GIL, which this draft does not attempt.