Skip to content

Specification conformance, integration tests that skip only what needs a key - #239

Draft
jwrosewell wants to merge 2 commits into
rcfrom
remote/python-integration-skip
Draft

jwrosewell wants to merge 2 commits into
rcfrom
remote/python-integration-skip

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The problem

On rc, the ci/run-integration-tests.ps1 of each product, being Device Detection, IP Intelligence, location and pipeline, threw "No TestResourceKey was passed, so the cloud tests cannot call the cloud service." before running anything when no resource key was passed. So a build without the key ran none of the integration tests, including the many that need no key at all.

The specification

No sentence in 51Degrees/specifications at 6570727d says what an integration step does without a key. The nearest are these.

pipeline-specification/automated-testing.md lines 8 to 13:

The tests provided are almost all unit tests, though some unit tests also serve as integration tests - for example, requiring access to live cloud server instances to pass - however they are not distinguished as such in continuous integration.

device-detection-specification/required-examples.md lines 20 to 23:

Those tests MUST include a run of each example with no arguments or settings, as someone would run it from a fresh clone, so that the run uses the free data files.

This change marks the tests that need the live cloud service with a resource key as live, so they are told apart, and runs every other test when no key is passed. It keeps the rule of #58, that a cloud test fails when it finds no key, so a key that is passed but never reaches a test is still caught.

The change

  • c6d02d4, the test. ci/run-integration-tests.Tests.ps1 is new. It runs each product's script as common-ci runs it, through Invoke-ProductStep in ci/dispatch.ps1 and common-ci's own steps/run-script.ps1, from a folder laid out like a common-ci checkout. The only stand-in is python/run-integration-tests.ps1, which runs no tests and records the packages and the PYTEST_ADDOPTS each call is given. With no key it checks, for all four products, that tests run, that only the live tests are left out, that the step says so for each product, that the Device Detection and IP Intelligence examples and the cloud request engine and its examples run, that the 51Did live tests are skipped with a message, and that the Selenium contract against the cloud example is skipped with a message and nothing is cloned for it. With a key it checks for Device Detection and location that nothing is left out.
  • 892c433, the fix.
    • The four product scripts no longer throw. With no key each prints a ::warning that says the tests needing a resource key are skipped, sets PYTEST_ADDOPTS to -m "not live" for its run, and puts PYTEST_ADDOPTS back afterwards. live is the marker that device-detection/ci/run-unit-tests.ps1 and pipeline/ci/run-unit-tests.ps1 already leave out with the same -m "not live".
    • The tests that need a resource key are marked live, in the Device Detection and IP Intelligence examples, location, and the cloud request engine and its examples, and the marker is registered in each test folder that uses it. The licence-key tests already skip themselves with a reason when no licence key is set, and are unchanged.
    • Pipeline runs the cloud request engine and its examples first and then, only with a key, 51Did on its own with -m "live and not tier", because the two selections cannot share one run. The first run's .coverage is renamed .coverage.cloudrequestengine so the second run's combine reads it. With no key, 51Did has nothing to run, so it and its OWID copy are skipped with a message.
    • IP Intelligence reports the key's products only when there is a key.
    • Device Detection runs the Selenium contract against the cloud example only with a key, and against the on-premise example only with the TAC data file, and clones nothing when it can run neither.
    • test_cloudgettingstartedweb.py in the Device Detection examples and location/tests/test_properties51D.py built a cloud Pipeline when the module was imported, which raises with no key and so failed collection even for deselected tests. Both now build it in setUpClass and fail with the no-key message when they run without a key. The Device Detection test moves from flask_unittest.ClientTestCase to unittest.TestCase with app.test_client().
    • location/tests/test_examples.py passed without a key and did nothing. It is now live and fails with the no-key message, as Fail the cloud tests when no resource key is set, and run the Cloud 51Did workflow #58 made the other live tests do.
    • test_HttpDataSetInException in pipeline/fiftyone_pipeline_cloudrequestengine/tests/test_cloudengine.py sends a key that does not exist, so it moves out of the live class into CloudErrorTests and runs with no key.
    • Each changed package's tox.ini passes PYTEST_ADDOPTS through, and two readme files say how to run the tests that need no key.

The runs

  • Red, CI script tests run 44009105422, failure as expected. The run is the CI script tests workflow started by hand on a temporary branch whose commit, 871188a, is rc 2373d14 with only the test file of c6d02d4 added. On ubuntu-latest and on windows-latest all six cases of "Context with no TestResourceKey" failed, each with "RuntimeException: run-integration-tests failed for: device-detection, ip-intelligence, location, pipeline" from Invoke-ProductStep in ci/dispatch.ps1 line 47, and each job ended "Tests Passed: 145, Failed: 6". On rc each of the four product scripts throws "No TestResourceKey was passed" when there is no key (for example device-detection/ci/run-integration-tests.ps1 lines 19 to 20 at 2373d141), so the step failed for every product before calling the stand-in. The case "with a TestResourceKey" passed on both.
  • Green, the head 892c433: CI script tests run 43969034217, success on ubuntu-latest and windows-latest. On both, all seven cases of "Describing run-integration-tests.ps1" passed, six with no TestResourceKey and one with a key, and each job ended "Tests Passed: 151, Failed: 0".
  • Full build, run 43970313327, first attempt, success. All eleven Setting 'PullRequestSha' lines name 892c433, and Configure printed "Left out 5 macOS entries". It passed in all ten jobs, Ubuntu and Windows with Python 3.10, 3.11, 3.12, 3.13 and 3.14. The build had a resource key, so it ran the step with the key, the live tests included, and printed no warning about a missing key.

What remains

  1. No CI run has yet run the real integration step with no key. The no-key path is shown only by the Pester test, with common-ci's test runner replaced by a stand-in, and the full build had a key and left out macOS.
  2. Tests left out with -m "not live" appear only as pytest's "deselected" count and in the warning, and not one by one in the published results.
  3. With no key and the TAC file present, the Selenium contract still runs against the on-premise example with PAID_RESOURCE_KEY empty. Whether that suite needs the key for the on-premise example is not established.
  4. That no other test module builds a cloud Pipeline when it is imported was checked by reading the tests, not by a run with no key.

Before you upgrade

Nothing changes for callers of the packages, because no package code changes. location/ci/run-integration-tests.ps1 is inside the location path of fiftyone_location in ci/packages.json, and that package's ignore list does not leave out ci/, so the release plan will release fiftyone_location with no change to its code. Every other changed file is a test, a readme, a tox.ini, or outside every package's path. For CI, a build with no resource key now runs every integration test that needs no key and warns that the others are skipped, where before it failed at once.

…eed none

ci/run-integration-tests.ps1 stops before running anything when no
TestResourceKey is passed, for every product, because each product's own
script throws "No TestResourceKey was passed". Two macOS runs stopped there,
so none of the integration tests that need no key ran on them.

ci/run-integration-tests.Tests.ps1 runs the step as common-ci does, through
ci/dispatch.ps1 and common-ci's steps/run-script.ps1, for Device Detection,
IP Intelligence, location and pipeline, with a stand-in for common-ci's
python/run-integration-tests.ps1 that records what each product asked it to
test. With no TestResourceKey it expects:

- every product to run its tests, the step not failing;
- each run to leave out only the tests marked live, which need a resource
  key, through PYTEST_ADDOPTS;
- each product to say that the tests needing a resource key are skipped;
- the Device Detection cloud and examples packages and the IP Intelligence
  examples to be run;
- pipeline to run the cloud request engine and its examples, and to skip
  the 51Did package, whose integration run has only live tests, saying so;
- the Selenium contract against the cloud example to be skipped, saying so,
  with nothing cloned for it.

With a TestResourceKey, Device Detection and location run every test, the
live ones included. CI-1.
…ed none

Each product's ci/run-integration-tests.ps1 now runs when no
TestResourceKey is passed. It prints a warning that the tests needing a
resource key are skipped, sets PYTEST_ADDOPTS to -m "not live" so they are
left out, and runs every other integration test. With a key nothing is left
out, and a live test that finds no key still fails, as before.

- Device Detection, IP Intelligence and location set the selection around
  their one run. IP Intelligence reports what the key carries only when
  there is a key.
- Pipeline runs the cloud request engine and its examples first, then the
  51Did package on its own with its "live and not tier" selection, because
  both runs now see PYTEST_ADDOPTS. The first run's coverage is kept under
  a name the second run's combine reads. With no key the 51Did package has
  no test to run, so it is skipped, saying so, and the OWID source is not
  copied.
- Device Detection skips the Selenium contract against the cloud example
  when there is no key, saying so, and clones and sets up nothing when
  neither example can be run.
- The tox.ini of the Device Detection and IP Intelligence examples, location,
  the cloud request engine and its examples pass PYTEST_ADDOPTS.
- The tests that need a resource key are marked live: the Device Detection
  cloud example tests, the IP Intelligence cloud console and web tests,
  location's example, property and web example tests, the cloud request
  engine's CloudEngineTests and the pipeline examples' resource key cases.
  Each folder registers the marker in its conftest.py, and the four copies
  of the on-premise conftest.py stay the same.
- Two test modules built a cloud pipeline when they were collected, which
  raises with no key, so collecting them failed even when they were left
  out. The Device Detection cloud getting started web test now builds it in
  setUpClass and fails with the no key message, as the other web tests do,
  and location's property tests do the same.
- location's cloud example test passed without a key. It now fails, naming
  the variables, like the other live tests.
- The cloud request engine's test of the error a wrong key gets needs no
  key, so it moves out of CloudEngineTests into a class of its own.
- The licence key tests already skip themselves without a licence key, and
  are unchanged.

The readmes of the Device Detection and pipeline examples say how to run the
tests that need no key. CI-1.
@jwrosewell jwrosewell added the remote-conformance Brings the code into conformance with the specification label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Brings the code into conformance with the specification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant