Repository navigation
Lock every read of IP Intelligence native results, type Latitude and Longitude as Float, and process cloud IP Intelligence with the shared engine logic - #177
Merged
Conversation
The Pipeline specification's Value types page, at 51Degrees/specifications 706de1f0, says in pipeline-specification/features/value-types.md line 99: "A value converted from Azimuth or Declination MUST be given to the caller rounded to 3 decimal places, whether it is given as a number, as a string or as a point in a WKT shape." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/features/value-types.md#L99 The IP Intelligence on-premise engine gives Latitude and Longitude as weighted values from the native getValuesAsWeightedDoubleList, which converts each stored angle with no rounding at the native code this repository builds against. The test processes four addresses and reads both properties as weighted values and by their typed accessors, and checks each is the value of its type nearest a decimal with at most 3 places. Only the Enterprise data file has these properties, so with the free Lite file the test is skipped. Specification conformance item CO-58.
Moves the cxx submodule from 7992af70 to 178371ac, the 51degrees-cxx release candidate, which holds a1c0504d, "Give a number stored as an angle rounded to the decimal places of its text (#177)". The native getValuesAsWeightedDoubleList then gives a value stored as an Azimuth or a Declination rounded to 3 decimal places, as its text already was, so Latitude and Longitude read as weighted values are rounded as the typed accessors were. Answers pipeline-specification/features/value-types.md line 99 at 51Degrees/specifications 706de1f0, "A value converted from Azimuth or Declination MUST be given to the caller rounded to 3 decimal places, whether it is given as a number, as a string or as a point in a WKT shape." Specification conformance item CO-58.
The IP Intelligence data model page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/data-model.md line 10: "All Engines MUST populate data objects that implement Aspect Data as defined in the Pipeline specification." and at line 34: "Element data returned by an IP Intelligence Engine MUST extend `IAspectData`." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/data-model.md#L10 https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/data-model.md#L34 The compiler holds both today, as IPIntelligenceData extends AspectData, and nothing tested them. The tests check that the Element Data type extends AspectData, and that the on-premise engine, with the free Lite data file, and the cloud engine, with a fixed response, each populate Element Data that is AspectData and IPIntelligenceData. The code is unchanged. Specification conformance items IP-1 and IP-2.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says of the accepted Evidence keys in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 84: "This MUST be done at start-up and any time the data is refreshed." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L84 The engine already reads the keys in setEngineMetaData, at start-up and after each refresh, and EvidenceKeysTests checked them after building only. Both free data files accept the same keys, so a refresh to the other file cannot show a change. The new test puts a key no data file has in place of the engine's keys, refreshes, and checks that the keys are again those the native engine gives, and that at start-up they are those too. The code is unchanged. Specification conformance item IP-8.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 128: "- The Element Data MUST read a Property value from `ResultsIpi` only when that Property is requested, and MUST NOT copy Property values out of `ResultsIpi` while the Engine processes the Flow Data." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L128 processEngine already only keeps the native results, and nothing tested it. The test puts a stand-in for the native engine in the engine built from the free Lite file, processes a Flow Data, checks that the results it gave were asked nothing, then reads RegisteredName and checks that one value was read, by position. The code is unchanged. Specification conformance item IP-9.
…e are set The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 395: "An implementation configured with no names MUST create the Engine with no names, and MUST NOT pass the data set's names itself, because an Engine created with names walks graphs during `process`, as reading graphs describes, where one created with no names walks a graph only when a Property of it is first read." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L395 The engine builder already passes only the names it is given, and PropertyIndexTests checked the indexes the engine then has, which would be the same if the data set's names were passed. The tests read the names in the configuration the native engine is created with, for an engine built with no names by the engine builder and by the on-premise pipeline builder, and check that there are none. The code is unchanged. Specification conformance item IP-22.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says of the table of property indexes in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 419: "That table is built once, when the Engine is created, from the names the native code gives, for example through the Engine's `getRequiredProperties`, and MUST NOT be built again from a later data set." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L419 The engine already makes the table once. PropertyIndexTests reloads the same file, and the native engine keeps the names it was created with through a reload, so that test passes even when the table is made again. The new test builds an engine from the free Lite file with no names, then has it create its native engine again from the free ASN data file, whose native engine gives other names, and checks the table is unchanged and has no index for a name only the later data set gives. The code is unchanged. Specification conformance item IP-24.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 520: "Finished Element Data MUST be safe to read from several threads at once, as thread safety describes, so Aspect Data that reads its values from a `ResultsIpi` instance MUST let only one thread at a time read from its results, for example by holding a lock for each read, unless the Aspect Data is a type the language does not let several threads share." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L520 Four threads read one finished Element Data at once, two through the RegisteredName getter and two through the public getValues, from a stand-in for the native engine whose results count the threads inside them and hold each call for a moment. The test checks that no two threads were ever inside the results together. Specification conformance item IP-27.
Each typed read held a lock only around the value read, so the check that the property is available, which asks the native results with containsProperty, ran outside it, and the public getValues held no lock at all. Several threads could then read the native results at once. The on-premise IP Intelligence Element Data now holds one lock, its own, for the whole of each read through tryGetValue, which covers the availability check and every typed read, and for getValues when a caller uses it directly. The lock is taken before the base class's lock on every path, so the two cannot deadlock. Answers ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 520 at 51Degrees/specifications 706de1f0, "Aspect Data that reads its values from a `ResultsIpi` instance MUST let only one thread at a time read from its results, for example by holding a lock for each read". Specification conformance item IP-27.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 152: "When the Evidence holds more than one client IP key, the native code uses one value, chosen in this order: the query keys in the data file's order, then, only when no query key supplied an address, the server keys in the data file's order. So query.client-ip is used ahead of server.true-client-ip-51d, and the Property values describe that one address." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L152 The engine passes every accepted Evidence entry to the native code, which makes the choice, and no test put two client IP keys in one Evidence. With the free Lite data file, the tests check that query.client-ip and server.true-client-ip-51d holding different addresses give the values of the query address, whichever was added first, and that the server key on its own gives the values of its address. The code is unchanged. Specification conformance item IP-36.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 487: "- An Engine created with only the Properties an implementation reads lets the Balanced configuration hold what they read in memory, so an implementation SHOULD create the Engine with only those Properties." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L487 The engine builder already passes only the names it is given, and nothing tested it. The tests set one name through the engine builder and through the on-premise pipeline builder, and check the native engine is created with that name and no other, and that the native engine then holds only that name. The code is unchanged. Specification conformance item IP-26.
The IP Intelligence on-premise page, at 51Degrees/specifications 706de1f0, says under Performance Guidance in ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md line 532, of marshalling data values to and from native representations: "As such, steps SHOULD be taken to minimize this as much as possible:" and at line 539: "- Whenever making a call to native code, be aware of the data that is being passed and check if there is anything that can be done to reduce it." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L532 The engine already passes only the Evidence it accepts and takes the accepted keys from the native engine once for each data set, and nothing tested either. With a stand-in for the native engine, the tests check that a Flow Data holding the client IP and forty other entries passes only the client IP, and that five processes never ask the native engine for its keys and leave the same key filter. The code is unchanged. Specification conformance item IP-28.
The IP Intelligence cloud page, at 51Degrees/specifications 706de1f0, says in ip-intelligence-specification/pipeline-elements/ip-intelligence-cloud.md line 48: "The majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic." https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-cloud.md#L48 The engine extends the shared CloudAspectEngineBase but replaces its processEngine with one of its own, which skips the shared checks: it does not look at whether the Cloud Request Engine ran for the Flow Data, and it fails reading a response that is not there. The tests drive the engine through its public process method with a fixed response, and check that the engine declares no processEngine of its own but the processCloudEngine the shared logic calls, that a response is read, that no response gives the shared warning and no values, and that a request that did not run throws PipelineConfigurationException. Specification conformance item IP-5.
…igence The cloud IP Intelligence engine no longer replaces processEngine. The shared CloudAspectEngineBase logic now finds the Cloud Request Engine's data, checks that the request ran, logs its warning and stops when there is no response, and otherwise calls the engine's processCloudEngine, which reads the engine's part of the response as before. DataModelTests now drives the engine through its public process method, as processEngine is no longer declared in the engine's package. The engine keeps its own property metadata loading, because the shared loading only knows the scalar cloud types and IP Intelligence has weighted, address, WKT and Single types. Answers ip-intelligence-specification/pipeline-elements/ip-intelligence-cloud.md line 48 at 51Degrees/specifications 706de1f0, "The majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic." Specification conformance item IP-5.
This was referenced Nov 16, 2026
…pes them With the Enterprise data file, Angles_RoundedTo3Places stopped at checkWeighted with "java.lang.String cannot be cast to class java.lang.Number", both at 56601cd and at 38fd7ff. The test read each angle with get(name), which asks for weighted values of the type the property meta data gives. The native meta data names Latitude and Longitude "single", which PropertyMetaDataIPI.getType does not know and so gives as String, and the engine then reads the native weighted string list. That text was already rounded, so the test could not show the fault in the weighted double list either way. The test now asks the element data for the weighted values as Float, the type the IP Intelligence data model gives Latitude and Longitude (ip-intelligence-specification/data-model.md at 51Degrees/specifications 706de1f0) and the type getLatitude and getLongitude return. For Float the engine reads the native getValuesAsWeightedDoubleList, so the test fails on the unrounded values the cxx at 56601cd gives and passes with the rounding 38fd7ff brings in. Only the Enterprise data file has these properties, so the test is skipped with the free Lite file. Specification conformance item CO-58.
The IP Intelligence data model gives Latitude and Longitude the type float, and says each property MUST also be exposed as a weighted value (ip-intelligence-specification/data-model.md at 51Degrees/specifications 706de1f0). Their typed accessors, getLatitude and getLongitude, return Float. Read by name with get(name), which names no type, the on-premise engine gives them as weighted values of String instead, because the native property meta data names their type "single", and PropertyMetaDataIPI.getType gives String for any name it does not know. PropertyMetaDataIPITypeTests checks the Java type the meta data gives for each native type name, with the native meta data mocked, so it runs with any data file or none. "single" and "weightedsingle" are to be Float, and the names already mapped keep their types. AngleRoundingTests.Angles_ByName_AreWeightedFloats processes the same four addresses as Angles_RoundedTo3Places and checks every weighted value get(name) gives for Latitude and Longitude is a Float. Only the Enterprise data file has these properties, so with the free Lite file it is skipped. Specification conformance item CO-58.
PropertyMetaDataIPI.getType gave String for the native type name "single", and for "weightedsingle", because the switch did not know them. Latitude and Longitude are "single", so get(name) asked the engine for weighted String values and gave the text of each angle, while getLatitude and getLongitude gave Float, the type the IP Intelligence data model gives them (ip-intelligence-specification/data-model.md at 51Degrees/specifications 706de1f0). Both names now give Float. The element data then reads the weighted values from the native weighted double list as Float, the same number the typed accessors give, rounded to 3 decimal places since 38fd7ff. Specification conformance item CO-58.
The shared Cloud Aspect Engine logic that this work moved the cloud IP Intelligence engine onto already records a failed cloud request and warns about an empty response, which rc added to this engine's own override, so the override goes and the shared path keeps both.
… the engine leaves processing to the shared logic
jwrosewell
marked this pull request as ready for review
November 16, 2026 21:14
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The IP Intelligence packages differed from the specification in three places, and eight more rules had no test.
singleProperty, which is how the data file holds Latitude and Longitude, as aString, where the data model gives both asfloat.getValuesheld no lock at all, so two threads could read the native results at once although each read writes into them.processEnginewith its own, which did not check whether the Cloud Request Engine ran for the Flow Data and failed reading a response that was not there.What the specification says
All at 51Degrees/specifications dbaf099c.
data-model.md lines 48 and 49
ip-intelligence-on-premise.md lines 520 to 525
ip-intelligence-cloud.md lines 48 to 49
The rules that gain a test here, where the code already met them, are in data-model.md lines 10 and 34 (Element Data is Aspect Data) and in ip-intelligence-on-premise.md at line 84 (the accepted Evidence keys read at start-up and on every refresh), lines 128 to 130 (a value read only when it is asked for), lines 152 to 161 (the query client IP used ahead of the server one), lines 395 to 399 (an engine with no names set is created with none), line 419 (the property index table made once), lines 487 to 489 (an engine created with only the names set) and lines 532 to 540 (native code passed only what it needs).
What changed
Links are to the head, 07bb0e9.
singleis aFloat.PropertyMetaDataIPImaps the native typessingleandweightedsingletoFloat, where they fell through toString, so the metadata of Latitude and Longitude gives the type the data model gives.IPIntelligenceDataHashDefaultholds one private lock for the whole of each read throughtryGetValue, covering the availability check and every typed read (lines 422 to 429), and for the publicgetValues(lines 195 to 198). The lock is always taken before the base class's lock, so the two cannot deadlock.IPIntelligenceCloudEngineno longer declares its ownprocessEngine. Its response reading becomesprocessCloudEngine, which the sharedCloudAspectEngineBase.processEnginecalls, as the Device Detection cloud engine does. So a request that did not run now throwsPipelineConfigurationException, and no response gives the shared warning and no values. The engine keeps its own Property metadata loading, because the shared loading knows only the scalar cloud types. Four existing tests in the module that calledprocessEnginenow call the publicprocessorprocessCloudEngine.ip-intelligence.engine.on-premise,ip-intelligence.cloudandip-intelligence. They areDataModelTests(on-premise and cloud),AcceptedEvidenceRefreshTests,ProcessReadsNoValueTests,EngineNamesTestsandOnPremiseEngineNamesTests,PropertyIndexLaterDataSetTests,ClientIpChoiceTestsandProcessMarshallingTests.No public name is removed or renamed.
Before you upgrade
Floatvalues where they wereStringvalues, wherever the type is taken from the metadata. Code that readsgetType()from the metadata of a nativesingleProperty seesFloat.class. The on-premise data also takes its read type from that metadata whenever a caller names none, inIPIntelligenceDataBaseOnPremise.tryGetValueandasKeyMap, soget("latitude"),get("longitude")andasKeyMap()now give weightedFloatvalues where they gave weightedStringvalues. The JSON Builder readsasKeyMap(), so the JSON it writes for those two Properties, which a web page's JavaScript reads, now carries numbers where it carried text. Code that casts either value toString, or a page that compares the JSON value as text, has to read a number instead.PipelineConfigurationException. A subclass that overrodeprocessEngineshould overrideprocessCloudEngineinstead.Tests
Each fix has a test commit before it.
Red, two runs, each job checking out one test commit before its fix and running only that commit's new tests on Java 17.
SharedCloudProcessingTestsPipelineConfigurationException, and no response failed with aNullPointerException. Reading a response passedResultsReadByOneThreadTestsPropertyMetaDataIPITypeTestsSingle_IsFloatandWeightedSingle_IsFloat, each "expected: but was:".OtherNames_AreUnchangedpasses, as it shouldThe two red jobs of run 43901352500 check out commits from before this draft stopped moving the C library, so their native build is of 178371ac. Neither test depends on the C library's version, the first counting how many threads are inside a read at once and the second mocking the native metadata.
The nine test classes in item 4 pass on
rc, as the code met their rules. The compiler already enforces the twoDataModelTestsrules, so no change that still compiles could make them fail. The other seven need the native engine and a free data file, and none has been shown failing on a runner with its rule broken.Green, the full build below, in which these tests ran on the head.
Build
Run 43899947358 built 07bb0e9 merged with
rc81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21. Every job passed, andResultsReadByOneThreadTests(1 test),PropertyMetaDataIPITypeTests(3) andSharedCloudProcessingTests(4) passed in all nine. Each of its tenSetting 'PullRequestSha'lines names 07bb0e9, and its Configure job printed "Left out 6 macOS entries".macOS was left out of this draft's full build. The rc head is built with macOS before it merges into main.
The C library
This draft builds on the C library at 7992af70, the 4.5.106 release, which is what
rchas. It used to move thecxxsubmodule to 178371ac for Latitude and Longitude rounded to 3 decimal places, withAngleRoundingTests(commit 56601cd) to hold it, and #171 carries the same move. Whether Java moves to that C library is a decision held with #171, so commit 07bb0e9 puts the submodule back torc's and takesAngleRoundingTestsout, and the rest of this draft no longer waits on that decision. The rounding test comes back with the move.Base branch
rc81e3d4a is merged into the head with a merge commit. An earlier merge ofrc20f9089 took #175, which changedIPIntelligenceCloudEngine.processEnginein the same place this draft removes it. The override goes, because the shared Cloud Aspect Engine logic this draft moves the engine onto already records a failed cloud request and warns on an empty response, andCloudRequestFailedTestsnow processes through the engine's publicprocess(commit b8dda8b).#171 also locks the reads in
IPIntelligenceDataHashDefault, so whichever of the two merges second must mergercafter the other and keep one lock.What remains
ResultsReadByOneThreadTests,PropertyMetaDataIPITypeTestsand the shared cloud logic have red runs above. Seven of the test classes in item 4 have not been shown failing on a runner, as each needs the native engine and a data file and the code already met its rule.