Skip to content

Specification conformance, an IP Intelligence performance test on the free file - #238

Draft
jwrosewell wants to merge 4 commits into
rcfrom
remote/python-perf-ipi
Draft

jwrosewell wants to merge 4 commits into
rcfrom
remote/python-perf-ipi

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The problem

On rc, ip-intelligence/ci/run-performance-tests.ps1 printed "Skipping performance tests - not yet implemented for IP Intelligence" and measured nothing, so the performance step had no IP Intelligence figures, with a licensed file or without one. Device Detection already had a web benchmark and a console benchmark in that step.

The specification

All at 51Degrees/specifications 6570727d.

ip-intelligence-specification/required-examples.md lines 80 to 87:

Examples follow the device detection rules for on-premise examples. Given no arguments or settings, an example that shows location or registration Properties uses the Lite data file, and so does one whose purpose needs the Enterprise data file, such as a compare, metrics or country overlap example. A performance, reload or update example, whose Properties do not matter to what it shows, uses the Asn data file, which is far smaller to download.

device-detection-specification/required-examples.md line 91, which the first sentence above points to:

Examples MUST run out of the box with the free data files.

pipeline-specification/automated-testing.md lines 17 to 22:

Some level of performance regression testing is carried out in certain areas. This is, of necessity, a gross test, since unit tests are carried out in a wide range of environments during CI and automated build and distribution.

Performance benchmarking is not carried out as part of automated testing. Hence, external test tools, such as JMeter, are not used as part of automated testing.

The change measures IP Intelligence in the build's performance step on a free data file, as that step already measures Device Detection. The step gives its files explicitly, so the Asn rule for a performance example run with no arguments does not decide its file. The step uses the Lite file because both benchmarks read RegisteredCountry, which the Lite file has and the Asn file does not, so each lookup returns a value as it would with a paid file. The repository's AGENTS.md says the performance step runs only when rc itself or the release pull request is built.

The regression testing is done by common-ci, which compares the figures of each configuration with the figures of earlier runs. It reads one summary for each configuration, results_<Name>.json, where <Name> is the configuration's name (common-ci DESIGN.md lines 439 to 442, steps/compare-performance.ps1 line 268), so IP Intelligence's figures have to be in that file to be compared.

The change

  • 9c94b2b, the tests.
    • ci/ip-intelligence-performance.Tests.ps1 is new, a Pester 5 test of ip-intelligence/ci/run-performance-tests.ps1 in the same form as the Device Detection one in #237. Each case lays out a folder as common-ci does, with empty stand-ins for 51Degrees-LiteIpiV41.ipi and ip-intelligence-evidence.yml, and mocks python. The six cases check that the web benchmark exists, that it runs on the Lite file and the evidence with --output, that performance_console.py runs with --data_file the Lite file and --evidence_file the evidence and the output names the Lite file, that results_test.json holds DetectionsPerSecond and AvgMillisecsPerDetection (changed in 4e4bb0f), that the step throws rather than skipping when the Lite file is missing, and that nothing runs when performance is off.
    • ip-intelligence/fiftyone_ipintelligence_examples/tests/test_web_performance_benchmark.py is new. It runs the web benchmark as the step runs it, on the free Asn file and the first 50 evidence records, and checks the JSON it writes with --output and the line it writes to stderr without it.
  • d751511, the fix.
    • ip-intelligence/fiftyone_ipintelligence_examples/performance-tests/test_performance.py is new, in the same form as Device Detection's web benchmark. Each IP address in the evidence becomes REMOTE_ADDR on a mock Flask request, which webevidence gives as server.client-ip. The Pipeline is built with IpIntelligenceOnPremisePipelineBuilder on MaxPerformance, with no automatic update, no temporary copy of the data file, no usage sharing and no JavaScript builder. The lookups are timed with timeit, each reads registeredcountry, and --output prints the same JSON fields as Device Detection's.
    • ip-intelligence/ci/run-performance-tests.ps1 is rewritten. It runs the web benchmark and then performance_console.py, both on assets/51Degrees-LiteIpiV41.ipi and assets/ip-intelligence-evidence.yml, names the file it measures on, and writes the console benchmark's figures to a summary in test-results/performance-summary/, whose name and figures the next two commits correct. It throws when either file is missing, because ip-intelligence/ci/fetch-assets.ps1 always fetches both. It uses the Lite file even when the Enterprise file is fetched, so every run measures the same file.
    • The root ci/run-performance-tests.ps1 already runs each product's own script, so it is unchanged.
  • 4e4bb0f, the tests of the summary.
    • In ci/ip-intelligence-performance.Tests.ps1, "writes the summary the performance step reports" checks that results_test.json is the only file in the summary folder for the configuration named test, and that its figures are IpIntelligence_DetectionsPerSecond and IpIntelligence_AvgMillisecsPerDetection.
    • ci/dispatch.Tests.ps1 has two new cases for the root step. When two products write the summary of the same configuration, the one file keeps the figures of both. When both report the same figure, the step fails for the later product and the file keeps the earlier figure.
  • 4d47dcd, the fix of the summary.
    • At d751511, ip-intelligence/ci/run-performance-tests.ps1 named its loop over the two asset names $name. PowerShell variable names are not case sensitive, so $name and the $Name parameter were one variable, and after the loop the configuration name was ip-intelligence-evidence.yml. The summary was written as results_ip-intelligence-evidence.yml.json, and the artifact of the Ubuntu, Python 3.12 job in the full build of d751511 holds that file beside Device Detection's results_Ubuntu_Python_3.12.json. common-ci reads only results_<Name>.json, so the IP Intelligence figures were uploaded but never compared. The loop now uses $file.
    • Correcting the name alone would have replaced Device Detection's figures. The root step moves each product's results into one folder with Move-Item -Force, Device Detection runs before IP Intelligence because ci/packages.json lists it first, and Device Detection writes results_<Name>.json with the figures DetectionsPerSecond and AvgMillisecsPerDetection, the same file and the same names as the IP Intelligence step at d751511 once its name was right. The IP Intelligence figures are now IpIntelligence_DetectionsPerSecond and IpIntelligence_AvgMillisecsPerDetection, the names 51degrees-java gives them in its one summary for each configuration (ci/run-performance-tests.ps1 lines 16 to 19 and lines 140 and 141).
    • Move-ProductTestResults in ci/dispatch.ps1 adds a product's figures to a summary that another product already wrote for the configuration, rather than replacing that file. A figure both products report fails the step for the later product, and Invoke-ProductStep records that failure as it records a failing script, so every product still runs.
    • Device Detection's figure names are unchanged, so its graphs keep their history.
    • The merge was also run on the results_Ubuntu_Python_3.12.json from the artifacts of the full build of d751511, with IP Intelligence figures added, and common-ci's steps/publish-performance-results.ps1 accepted the result with all four figures.

The runs

  • Red, the test commit 9c94b2b: missing. No CI run ran 9c94b2b, and the tests were not run anywhere before the fix.
  • CI script tests at d751511, run 43936594750: failure on both ubuntu-latest and windows-latest, "Tests Passed: 149, Failed: 1". "writes the summary the performance step reports" failed at ci/ip-intelligence-performance.Tests.ps1 line 97 with "Expected path '.../ip-intelligence/test-results/performance-summary/results_test.json' to exist, but it did not exist.", because the summary was written as results_ip-intelligence-evidence.yml.json.
  • Full build at d751511, run 43938023716, first attempt, success. All eleven Setting 'PullRequestSha' lines name d751511, and Configure printed "Left out 5 macOS entries". It passed in all ten jobs, Ubuntu and Windows with Python 3.10, 3.11, 3.12, 3.13 and 3.14. test_web_performance_benchmark.py passed both its tests in every job, and every job printed "Running Web performance tests on 51Degrees-LiteIpiV41.ipi" and "Running Console performance tests on 51Degrees-LiteIpiV41.ipi".
  • Red, the test commit 4e4bb0f alone on top of d751511, pushed as tmp/red-python-238 and run with the CI script tests workflow, run 44009747094: failure on both ubuntu-latest and windows-latest, "Tests Passed: 149, Failed: 3". "writes the summary the performance step reports" failed with "Expected: 'results_test.json' But was: 'results_ip-intelligence-evidence.yml.json'", "keeps every product's figures in the one summary of a configuration" with "Expected 1, but got $null." for the first product's figure, and "fails the step when two products report the same figure" with "no exception was thrown". The branch was deleted after the run.
  • Green, CI script tests at 4d47dcd, run 44010897487: success on both ubuntu-latest and windows-latest, "Tests Passed: 152, Failed: 0", with the three cases above among the passes.

The figures the full build of d751511 printed for IP Intelligence on the Lite file. The web benchmark timed 20,000 IP addresses in one process. The console benchmark processed 5,000 IP addresses over four worker processes. The last two commits change neither benchmark.

Job Web, lookups per second Web, ms per lookup Console, lookups per second Console, ms per lookup
Ubuntu, Python 3.10 4122 0.243 216 4.640
Ubuntu, Python 3.11 4355 0.230 368 2.715
Ubuntu, Python 3.12 7852 0.127 605 1.653
Ubuntu, Python 3.13 6956 0.144 443 2.257
Ubuntu, Python 3.14 4286 0.233 349 2.864
Windows, Python 3.10 4906 0.204 288 3.474
Windows, Python 3.11 4808 0.208 287 3.483
Windows, Python 3.12 6560 0.152 395 2.534
Windows, Python 3.13 4227 0.237 260 3.847
Windows, Python 3.14 9068 0.110 496 2.016

What remains

  1. The full build of 4d47dcd.
  2. The red run of 9c94b2b on CI.
  3. Whether the step should measure the Lite file or the Asn file. Changing it is one line in run-performance-tests.ps1.
  4. Device Detection's step in Specification conformance, the Device Detection performance test on the free Lite file #237 prefers the TAC file when it is there, where this step always uses the Lite file. The two choices are not yet lined up.
  5. Device Detection's figures keep the names DetectionsPerSecond and AvgMillisecsPerDetection, where 51degrees-java names them DeviceDetection_DetectionsPerSecond and DeviceDetection_AvgMillisecsPerDetection. Renaming them would start Device Detection's graphs again with no history.
  6. The console benchmark starts one worker for each CPU by default, and its header says that on MaxPerformance "each worker loads the whole data file into its own memory". It ran on four workers on these runners. The step does not pass -w, so on a runner with more CPUs it holds more copies of the Lite file.

Before you upgrade

Nothing changes for callers. The changed files are CI scripts, CI tests, and a benchmark and test in the examples package, which ci/packages.json marks as not published, so no package is released for this change. For CI only, the performance step now measures IP Intelligence on the free Lite file and fails, where before it skipped, when the Lite file or the evidence is missing. Its figures go into each configuration's summary beside Device Detection's, and two products that report a figure of the same name fail the step.

…file

The IP Intelligence specification says its examples follow the device detection rules for on-premise examples (ip-intelligence-specification/required-examples.md lines 80 and 81 at 6570727d2af6967390d3c4fbec77b9cf0540d3f3), which say examples MUST run out of the box with the free data files (device-detection-specification/required-examples.md line 91 at the same commit), and every example and test runs on the free files. ip-intelligence/ci/run-performance-tests.ps1 measures nothing: it prints that performance tests are not yet implemented, so the performance step has no IP Intelligence figures at all.

ci/ip-intelligence-performance.Tests.ps1 runs the step with only the free Lite file and the evidence in the assets. It expects a web benchmark at fiftyone_ipintelligence_examples/performance-tests/test_performance.py, in the same place and form as Device Detection's, to be run on the Lite file and the evidence with --output, the console example performance_console.py to be run on the same files, the output to name the Lite file, and a results file to be written with DetectionsPerSecond and AvgMillisecsPerDetection. Without the Lite file the step fails rather than skipping, and with performance turned off it runs nothing.

test_web_performance_benchmark.py runs the web benchmark on the free Asn file and 50 evidence records, and checks the JSON it prints. PF-2.
… file

ip-intelligence/ci/run-performance-tests.ps1 now runs two benchmarks on 51Degrees-LiteIpiV41.ipi and the evidence, which fetch-assets.ps1 always fetches, as the rule that every example and test runs on the free files and ip-intelligence-specification/required-examples.md lines 80 and 81 with device-detection-specification/required-examples.md line 91 at 6570727d2af6967390d3c4fbec77b9cf0540d3f3 require. The root ci/run-performance-tests.ps1 runs it for IP Intelligence as it does for Device Detection.

The web benchmark, fiftyone_ipintelligence_examples/performance-tests/test_performance.py, is in the same form as Device Detection's: it makes each IP address of the evidence the client address of a mock Flask request, processes the request's evidence with the on-premise pipeline on MaxPerformance, and with --output prints the same JSON fields. The console benchmark is the existing performance_console.py example, whose results become results_<Name>.json with DetectionsPerSecond and AvgMillisecsPerDetection, as Device Detection's do.

The step names the file it measured on. When the Lite file or the evidence is missing it fails rather than skipping, and with performance turned off it runs nothing. The Lite file is used rather than the smaller Asn file because it has RegisteredCountry, the property both benchmarks read. PF-2.
@jwrosewell jwrosewell added the remote-conformance Brings the code into conformance with the specification label Nov 16, 2026
…ummary

ci/ip-intelligence-performance.Tests.ps1 now checks that the only file in
the performance summary folder is results_test.json, for the configuration
named test, and that its figures are IpIntelligence_DetectionsPerSecond and
IpIntelligence_AvgMillisecsPerDetection. At d751511 the step writes
results_ip-intelligence-evidence.yml.json instead. The loop over the asset
names in ip-intelligence/ci/run-performance-tests.ps1 uses $name, which
PowerShell treats as the same variable as the $Name parameter, because
variable names are not case sensitive, so the last asset name replaces the
configuration name. common-ci's steps/compare-performance.ps1 reads only
results_<Name>.json, so the IP Intelligence figures were uploaded but never
compared.

ci/dispatch.Tests.ps1 gets two cases for the root step. When two products
write the summary of the same configuration, the one file keeps the figures
of both, and when both report the same figure the step fails rather than
losing one. Move-ProductTestResults moves each product's files with
Move-Item -Force, so the product that runs later replaces the summary of
the one before. Device Detection runs before IP Intelligence and also
writes results_<Name>.json, so correcting the name alone would replace
Device Detection's figures with IP Intelligence's.
…ce summary

ip-intelligence/ci/run-performance-tests.ps1 names its loop over the asset
names $file rather than $name, so the $Name parameter keeps the
configuration name and the summary is results_<Name>.json, the file
common-ci's steps/compare-performance.ps1 reads for each configuration. Its
figures are now IpIntelligence_DetectionsPerSecond and
IpIntelligence_AvgMillisecsPerDetection, because Device Detection reports
DetectionsPerSecond and AvgMillisecsPerDetection in the same file.
51degrees-java's ci/run-performance-tests.ps1 prefixes the Device Detection
and IP Intelligence figures in its one file in the same way.

Move-ProductTestResults in ci/dispatch.ps1 adds a product's figures to a
summary another product already wrote for the configuration, rather than
replacing that file. A figure both products report fails the step for the
later product, and the summary keeps the earlier figure. Invoke-ProductStep
records that failure as it records a failing script, so every product
still runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Brings the code into conformance with the specification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant