Skip to content

Lock every read of IP Intelligence native results, type Latitude and Longitude as Float, and process cloud IP Intelligence with the shared engine logic - #177

Merged
jwrosewell merged 21 commits into
rcfrom
remote/java-09
Nov 16, 2026
Merged

jwrosewell merged 21 commits into
rcfrom
remote/java-09

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The IP Intelligence packages differed from the specification in three places, and eight more rules had no test.

  1. The property metadata typed a native single Property, which is how the data file holds Latitude and Longitude, as a String, where the data model gives both as float.
  2. IP Intelligence Element Data held its lock only around the value read itself. The availability check before it called the native results outside the lock, and the public getValues held no lock at all, so two threads could read the native results at once although each read writes into them.
  3. The IP Intelligence cloud engine extends the shared Cloud Aspect Engine but replaced its processEngine with its own, which did not check whether the Cloud Request Engine ran for the Flow Data and failed reading a response that was not there.

What the specification says

All at 51Degrees/specifications dbaf099c.

  1. data-model.md lines 48 and 49

    | Latitude | float |
    | Longitude | float |

  2. ip-intelligence-on-premise.md lines 520 to 525

    Finished Element Data MUST be safe to read from several threads at once, as thread safety describes, so Aspect Data that reads its values from a ResultsIpi instance MUST let only one thread at a time read from its results, for example by holding a lock for each read, unless the Aspect Data is a type the language does not let several threads share.

  3. ip-intelligence-cloud.md lines 48 to 49

    The majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic.

The rules that gain a test here, where the code already met them, are in data-model.md lines 10 and 34 (Element Data is Aspect Data) and in ip-intelligence-on-premise.md at line 84 (the accepted Evidence keys read at start-up and on every refresh), lines 128 to 130 (a value read only when it is asked for), lines 152 to 161 (the query client IP used ahead of the server one), lines 395 to 399 (an engine with no names set is created with none), line 419 (the property index table made once), lines 487 to 489 (an engine created with only the names set) and lines 532 to 540 (native code passed only what it needs).

What changed

Links are to the head, 07bb0e9.

  1. A native single is a Float. PropertyMetaDataIPI maps the native types single and weightedsingle to Float, where they fell through to String, so the metadata of Latitude and Longitude gives the type the data model gives.
  2. One lock for every read of the native results. IPIntelligenceDataHashDefault holds one private lock for the whole of each read through tryGetValue, covering the availability check and every typed read (lines 422 to 429), and for the public getValues (lines 195 to 198). The lock is always taken before the base class's lock, so the two cannot deadlock.
  3. The shared logic processes cloud IP Intelligence. IPIntelligenceCloudEngine no longer declares its own processEngine. Its response reading becomes processCloudEngine, which the shared CloudAspectEngineBase.processEngine calls, as the Device Detection cloud engine does. So a request that did not run now throws PipelineConfigurationException, and no response gives the shared warning and no values. The engine keeps its own Property metadata loading, because the shared loading knows only the scalar cloud types. Four existing tests in the module that called processEngine now call the public process or processCloudEngine.
  4. Nine test classes, for the rules the code already met, in ip-intelligence.engine.on-premise, ip-intelligence.cloud and ip-intelligence. They are DataModelTests (on-premise and cloud), AcceptedEvidenceRefreshTests, ProcessReadsNoValueTests, EngineNamesTests and OnPremiseEngineNamesTests, PropertyIndexLaterDataSetTests, ClientIpChoiceTests and ProcessMarshallingTests.

No public name is removed or renamed.

Before you upgrade

  1. Latitude and Longitude are Float values where they were String values, wherever the type is taken from the metadata. Code that reads getType() from the metadata of a native single Property sees Float.class. The on-premise data also takes its read type from that metadata whenever a caller names none, in IPIntelligenceDataBaseOnPremise.tryGetValue and asKeyMap, so get("latitude"), get("longitude") and asKeyMap() now give weighted Float values where they gave weighted String values. The JSON Builder reads asKeyMap(), so the JSON it writes for those two Properties, which a web page's JavaScript reads, now carries numbers where it carried text. Code that casts either value to String, or a page that compares the JSON value as text, has to read a number instead.
  2. The IP Intelligence cloud engine behaves as the Device Detection one does when the cloud request fails. When the Cloud Request Engine left no response, it now logs the shared warning and gives no values, where it failed reading the response. When the Cloud Request Engine did not run for the Flow Data, it throws PipelineConfigurationException. A subclass that overrode processEngine should override processCloudEngine instead.

Tests

Each fix has a test commit before it.

Red, two runs, each job checking out one test commit before its fix and running only that commit's new tests on Java 17.

Run Test commit Tests Result
43775266418 12b7650 SharedCloudProcessingTests 3 of 4 fail, "The engine should leave processEngine to the shared Cloud Aspect Engine logic.", a request that did not run threw no PipelineConfigurationException, and no response failed with a NullPointerException. Reading a response passed
43901352500 4223be9 ResultsReadByOneThreadTests 1 of 1 fails, "More than one thread read the native results at once. expected:<1> but was:<4>". The job builds the IP Intelligence engine with its native library and the free data file
43901352500 595e3cc PropertyMetaDataIPITypeTests 2 of 3 fail, Single_IsFloat and WeightedSingle_IsFloat, each "expected: but was:". OtherNames_AreUnchanged passes, as it should

The two red jobs of run 43901352500 check out commits from before this draft stopped moving the C library, so their native build is of 178371ac. Neither test depends on the C library's version, the first counting how many threads are inside a read at once and the second mocking the native metadata.

The nine test classes in item 4 pass on rc, as the code met their rules. The compiler already enforces the two DataModelTests rules, so no change that still compiles could make them fail. The other seven need the native engine and a free data file, and none has been shown failing on a runner with its rule broken.

Green, the full build below, in which these tests ran on the head.

Build

Run 43899947358 built 07bb0e9 merged with rc 81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21. Every job passed, and ResultsReadByOneThreadTests (1 test), PropertyMetaDataIPITypeTests (3) and SharedCloudProcessingTests (4) passed in all nine. Each of its ten Setting 'PullRequestSha' lines names 07bb0e9, and its Configure job printed "Left out 6 macOS entries".

macOS was left out of this draft's full build. The rc head is built with macOS before it merges into main.

The C library

This draft builds on the C library at 7992af70, the 4.5.106 release, which is what rc has. It used to move the cxx submodule to 178371ac for Latitude and Longitude rounded to 3 decimal places, with AngleRoundingTests (commit 56601cd) to hold it, and #171 carries the same move. Whether Java moves to that C library is a decision held with #171, so commit 07bb0e9 puts the submodule back to rc's and takes AngleRoundingTests out, and the rest of this draft no longer waits on that decision. The rounding test comes back with the move.

Base branch

rc 81e3d4a is merged into the head with a merge commit. An earlier merge of rc 20f9089 took #175, which changed IPIntelligenceCloudEngine.processEngine in the same place this draft removes it. The override goes, because the shared Cloud Aspect Engine logic this draft moves the engine onto already records a failed cloud request and warns on an empty response, and CloudRequestFailedTests now processes through the engine's public process (commit b8dda8b).

#171 also locks the reads in IPIntelligenceDataHashDefault, so whichever of the two merges second must merge rc after the other and keep one lock.

What remains

  1. Latitude and Longitude rounded to 3 decimal places, with the C library move held with Specification conformance, 51degrees-java, section 13 #171.
  2. ResultsReadByOneThreadTests, PropertyMetaDataIPITypeTests and the shared cloud logic have red runs above. Seven of the test classes in item 4 have not been shown failing on a runner, as each needs the native engine and a data file and the code already met its rule.

The Pipeline specification's Value types page, at 51Degrees/specifications
706de1f0, says in pipeline-specification/features/value-types.md line 99:

  "A value converted from Azimuth or Declination MUST be given to the
  caller rounded to 3 decimal places, whether it is given as a number, as
  a string or as a point in a WKT shape."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/features/value-types.md#L99

The IP Intelligence on-premise engine gives Latitude and Longitude as
weighted values from the native getValuesAsWeightedDoubleList, which
converts each stored angle with no rounding at the native code this
repository builds against. The test processes four addresses and reads
both properties as weighted values and by their typed accessors, and
checks each is the value of its type nearest a decimal with at most 3
places. Only the Enterprise data file has these properties, so with the
free Lite file the test is skipped.

Specification conformance item CO-58.
Moves the cxx submodule from 7992af70 to 178371ac, the 51degrees-cxx
release candidate, which holds a1c0504d, "Give a number stored as an
angle rounded to the decimal places of its text (#177)". The native
getValuesAsWeightedDoubleList then gives a value stored as an Azimuth or
a Declination rounded to 3 decimal places, as its text already was, so
Latitude and Longitude read as weighted values are rounded as the typed
accessors were.

Answers pipeline-specification/features/value-types.md line 99 at
51Degrees/specifications 706de1f0, "A value converted from Azimuth or
Declination MUST be given to the caller rounded to 3 decimal places,
whether it is given as a number, as a string or as a point in a WKT
shape."

Specification conformance item CO-58.
The IP Intelligence data model page, at 51Degrees/specifications
706de1f0, says in ip-intelligence-specification/data-model.md line 10:

  "All Engines MUST populate data objects that implement Aspect Data as
  defined in the Pipeline specification."

and at line 34:

  "Element data returned by an IP Intelligence Engine MUST extend
  `IAspectData`."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/data-model.md#L10
https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/data-model.md#L34

The compiler holds both today, as IPIntelligenceData extends AspectData,
and nothing tested them. The tests check that the Element Data type
extends AspectData, and that the on-premise engine, with the free Lite
data file, and the cloud engine, with a fixed response, each populate
Element Data that is AspectData and IPIntelligenceData. The code is
unchanged.

Specification conformance items IP-1 and IP-2.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says of the accepted Evidence keys in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 84:

  "This MUST be done at start-up and any time the data is refreshed."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L84

The engine already reads the keys in setEngineMetaData, at start-up and
after each refresh, and EvidenceKeysTests checked them after building
only. Both free data files accept the same keys, so a refresh to the other
file cannot show a change. The new test puts a key no data file has in
place of the engine's keys, refreshes, and checks that the keys are again
those the native engine gives, and that at start-up they are those too.
The code is unchanged.

Specification conformance item IP-8.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 128:

  "- The Element Data MUST read a Property value from `ResultsIpi` only
  when that Property is requested, and MUST NOT copy Property values out
  of `ResultsIpi` while the Engine processes the Flow Data."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L128

processEngine already only keeps the native results, and nothing tested
it. The test puts a stand-in for the native engine in the engine built
from the free Lite file, processes a Flow Data, checks that the results
it gave were asked nothing, then reads RegisteredName and checks that one
value was read, by position. The code is unchanged.

Specification conformance item IP-9.
…e are set

The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 395:

  "An implementation configured with no names MUST create the Engine with
  no names, and MUST NOT pass the data set's names itself, because an
  Engine created with names walks graphs during `process`, as reading
  graphs describes, where one created with no names walks a graph only
  when a Property of it is first read."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L395

The engine builder already passes only the names it is given, and
PropertyIndexTests checked the indexes the engine then has, which would be
the same if the data set's names were passed. The tests read the names in
the configuration the native engine is created with, for an engine built
with no names by the engine builder and by the on-premise pipeline
builder, and check that there are none. The code is unchanged.

Specification conformance item IP-22.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says of the table of property indexes in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 419:

  "That table is built once, when the Engine is created, from the names
  the native code gives, for example through the Engine's
  `getRequiredProperties`, and MUST NOT be built again from a later data
  set."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L419

The engine already makes the table once. PropertyIndexTests reloads the
same file, and the native engine keeps the names it was created with
through a reload, so that test passes even when the table is made again.
The new test builds an engine from the free Lite file with no names, then
has it create its native engine again from the free ASN data file, whose
native engine gives other names, and checks the table is unchanged and has
no index for a name only the later data set gives. The code is unchanged.

Specification conformance item IP-24.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 520:

  "Finished Element Data MUST be safe to read from several threads at
  once, as thread safety describes, so Aspect Data that reads its values
  from a `ResultsIpi` instance MUST let only one thread at a time read
  from its results, for example by holding a lock for each read, unless
  the Aspect Data is a type the language does not let several threads
  share."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L520

Four threads read one finished Element Data at once, two through the
RegisteredName getter and two through the public getValues, from a
stand-in for the native engine whose results count the threads inside
them and hold each call for a moment. The test checks that no two threads
were ever inside the results together.

Specification conformance item IP-27.
Each typed read held a lock only around the value read, so the check that
the property is available, which asks the native results with
containsProperty, ran outside it, and the public getValues held no lock at
all. Several threads could then read the native results at once.

The on-premise IP Intelligence Element Data now holds one lock, its own,
for the whole of each read through tryGetValue, which covers the
availability check and every typed read, and for getValues when a caller
uses it directly. The lock is taken before the base class's lock on every
path, so the two cannot deadlock.

Answers ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 520 at 51Degrees/specifications 706de1f0, "Aspect Data that reads its
values from a `ResultsIpi` instance MUST let only one thread at a time
read from its results, for example by holding a lock for each read".

Specification conformance item IP-27.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 152:

  "When the Evidence holds more than one client IP key, the native code
  uses one value, chosen in this order: the query keys in the data file's
  order, then, only when no query key supplied an address, the server keys
  in the data file's order. So query.client-ip is used ahead of
  server.true-client-ip-51d, and the Property values describe that one
  address."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L152

The engine passes every accepted Evidence entry to the native code, which
makes the choice, and no test put two client IP keys in one Evidence. With
the free Lite data file, the tests check that query.client-ip and
server.true-client-ip-51d holding different addresses give the values of
the query address, whichever was added first, and that the server key on
its own gives the values of its address. The code is unchanged.

Specification conformance item IP-36.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 487:

  "- An Engine created with only the Properties an implementation reads
  lets the Balanced configuration hold what they read in memory, so an
  implementation SHOULD create the Engine with only those Properties."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L487

The engine builder already passes only the names it is given, and nothing
tested it. The tests set one name through the engine builder and through
the on-premise pipeline builder, and check the native engine is created
with that name and no other, and that the native engine then holds only
that name. The code is unchanged.

Specification conformance item IP-26.
The IP Intelligence on-premise page, at 51Degrees/specifications
706de1f0, says under Performance Guidance in
ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md
line 532, of marshalling data values to and from native representations:

  "As such, steps SHOULD be taken to minimize this as much as possible:"

and at line 539:

  "- Whenever making a call to native code, be aware of the data that is
  being passed and check if there is anything that can be done to reduce
  it."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-on-premise.md#L532

The engine already passes only the Evidence it accepts and takes the
accepted keys from the native engine once for each data set, and nothing
tested either. With a stand-in for the native engine, the tests check that
a Flow Data holding the client IP and forty other entries passes only the
client IP, and that five processes never ask the native engine for its
keys and leave the same key filter. The code is unchanged.

Specification conformance item IP-28.
The IP Intelligence cloud page, at 51Degrees/specifications 706de1f0, says
in ip-intelligence-specification/pipeline-elements/ip-intelligence-cloud.md
line 48:

  "The majority of the processing SHOULD be handled by the shared Cloud
  Aspect Engine logic."

https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/ip-intelligence-specification/pipeline-elements/ip-intelligence-cloud.md#L48

The engine extends the shared CloudAspectEngineBase but replaces its
processEngine with one of its own, which skips the shared checks: it does
not look at whether the Cloud Request Engine ran for the Flow Data, and it
fails reading a response that is not there. The tests drive the engine
through its public process method with a fixed response, and check that
the engine declares no processEngine of its own but the processCloudEngine
the shared logic calls, that a response is read, that no response gives
the shared warning and no values, and that a request that did not run
throws PipelineConfigurationException.

Specification conformance item IP-5.
…igence

The cloud IP Intelligence engine no longer replaces processEngine. The
shared CloudAspectEngineBase logic now finds the Cloud Request Engine's
data, checks that the request ran, logs its warning and stops when there
is no response, and otherwise calls the engine's processCloudEngine, which
reads the engine's part of the response as before. DataModelTests now
drives the engine through its public process method, as processEngine is
no longer declared in the engine's package.

The engine keeps its own property metadata loading, because the shared
loading only knows the scalar cloud types and IP Intelligence has weighted,
address, WKT and Single types.

Answers ip-intelligence-specification/pipeline-elements/ip-intelligence-cloud.md
line 48 at 51Degrees/specifications 706de1f0, "The majority of the
processing SHOULD be handled by the shared Cloud Aspect Engine logic."

Specification conformance item IP-5.
@jwrosewell jwrosewell added the remote-conformance Specification conformance work from the remote packs label Nov 15, 2026
…pes them

With the Enterprise data file, Angles_RoundedTo3Places stopped at
checkWeighted with "java.lang.String cannot be cast to class
java.lang.Number", both at 56601cd and at 38fd7ff. The test read each
angle with get(name), which asks for weighted values of the type the
property meta data gives. The native meta data names Latitude and
Longitude "single", which PropertyMetaDataIPI.getType does not know and
so gives as String, and the engine then reads the native weighted string
list. That text was already rounded, so the test could not show the fault
in the weighted double list either way.

The test now asks the element data for the weighted values as Float, the
type the IP Intelligence data model gives Latitude and Longitude
(ip-intelligence-specification/data-model.md at 51Degrees/specifications
706de1f0) and the type getLatitude and getLongitude return. For Float the
engine reads the native getValuesAsWeightedDoubleList, so the test fails
on the unrounded values the cxx at 56601cd gives and passes with the
rounding 38fd7ff brings in.

Only the Enterprise data file has these properties, so the test is
skipped with the free Lite file.

Specification conformance item CO-58.
The IP Intelligence data model gives Latitude and Longitude the type
float, and says each property MUST also be exposed as a weighted value
(ip-intelligence-specification/data-model.md at 51Degrees/specifications
706de1f0). Their typed accessors, getLatitude and getLongitude, return
Float. Read by name with get(name), which names no type, the on-premise
engine gives them as weighted values of String instead, because the
native property meta data names their type "single", and
PropertyMetaDataIPI.getType gives String for any name it does not know.

PropertyMetaDataIPITypeTests checks the Java type the meta data gives for
each native type name, with the native meta data mocked, so it runs with
any data file or none. "single" and "weightedsingle" are to be Float, and
the names already mapped keep their types.

AngleRoundingTests.Angles_ByName_AreWeightedFloats processes the same
four addresses as Angles_RoundedTo3Places and checks every weighted value
get(name) gives for Latitude and Longitude is a Float. Only the
Enterprise data file has these properties, so with the free Lite file it
is skipped.

Specification conformance item CO-58.
PropertyMetaDataIPI.getType gave String for the native type name
"single", and for "weightedsingle", because the switch did not know
them. Latitude and Longitude are "single", so get(name) asked the engine
for weighted String values and gave the text of each angle, while
getLatitude and getLongitude gave Float, the type the IP Intelligence
data model gives them (ip-intelligence-specification/data-model.md at
51Degrees/specifications 706de1f0).

Both names now give Float. The element data then reads the weighted
values from the native weighted double list as Float, the same number the
typed accessors give, rounded to 3 decimal places since 38fd7ff.

Specification conformance item CO-58.
The shared Cloud Aspect Engine logic that this work moved the cloud IP
Intelligence engine onto already records a failed cloud request and warns
about an empty response, which rc added to this engine's own override, so
the override goes and the shared path keeps both.
… the engine leaves processing to the shared logic
@jwrosewell jwrosewell changed the title Specification conformance, 51degrees-java, section 9 Round Latitude and Longitude to 3 decimal places, and process cloud IP Intelligence with the shared engine logic Nov 16, 2026
@jwrosewell jwrosewell changed the title Round Latitude and Longitude to 3 decimal places, and process cloud IP Intelligence with the shared engine logic Lock every read of IP Intelligence native results, type Latitude and Longitude as Float, and process cloud IP Intelligence with the shared engine logic Nov 16, 2026
@jwrosewell
jwrosewell marked this pull request as ready for review November 16, 2026 21:14
@jwrosewell
jwrosewell merged commit 933e424 into rc Nov 16, 2026
3 checks passed
@jwrosewell
jwrosewell deleted the remote/java-09 branch November 16, 2026 21:14
@jwrosewell jwrosewell mentioned this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Specification conformance work from the remote packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant