Skip to content

Nine fixes the specification asks for, in usage sharing, data updates, Evidence keys and the JavaScript Builder - #170

Draft
jwrosewell wants to merge 18 commits into
rcfrom
remote/java-14
Draft

jwrosewell wants to merge 18 commits into
rcfrom
remote/java-14

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The Java packages differed from the specification in nine places.

  1. Usage sharing shared a query string parameter only when its name started with 51D_ or was named in the included list, where share all query string parameters is on by default.
  2. With share all Evidence on, usage sharing shared every cookie, the site's own session cookie among them, where only the 51D_ cookies may ever be shared.
  3. A failure in a check started by update polling or by the file watcher was thrown out of the scheduled task, so it was lost and the task stopped, and no later check ran.
  4. Four rows of the data update Errors table gave a different level or status. An integrity check failure was a warning, a response with no Content-MD5 header failed the update, and a missing temporary path and an unexpected error both gave AUTO_UPDATE_REFRESH_FAILED. A failed write of data passed in from memory was not logged as a warning.
  5. The Evidence key whitelist compared keys with their case unless its caller passed a comparator, and the JavaScript Builder read query Evidence only when its prefix was in lower case.
  6. The JavaScript Builder did not minify its script, and had no option for it.
  7. setProperties split a property list only at a comma with white space on both sides, so "IsMobile,BrowserName" was read as one property.
  8. Element Data was found by its key only in the case it was stored under, so data stored as device was not found as Device.
  9. The IP Intelligence getting started examples printed each value without its weighting.

What the specification says

All at 51Degrees/specifications dbaf099c.

  1. usage-sharing-element.md lines 80 to 83

    Share all query string parameters is on by default, so every query string and HTTP form parameter is shared unless a user turns it off.

    Anything else MUST return true. (i.e. it will be shared)

  2. usage-sharing-element.md lines 65 to 69 and line 99

    It MUST also return false, whatever the configuration and even when all Evidence is shared, for any key whose prefix is cookie. and whose field does not start with 51D_.

    The session cookie MUST NOT be shared unless its name starts with 51D_, as no other cookie is.

  3. data-updates.md lines 474 to 477

    A failure in a check started by update polling or by the file watcher MUST be caught and logged, rather than being lost or ending the process, and is then reported to onUpdateCallbacks. A failure MUST NOT stop later checks for updates, so a later good data file is still applied.

  4. data-updates.md lines 481 to 483, with the rows of the Errors table at lines 505 to 507, 511 and 515

    These messages MUST also be consistent across languages to make life easier for 51Degrees support.

  5. evidence.md line 10

    Keys MUST be case-insensitive and will usually be split into <prefix>.<field>.

  6. javascript-builder.md lines 105 to 114, and the SetMinify row, default true, at line 350

    Performance tests MUST be in place to verify the performance impact of this feature.

    If it does have a significant impact, and no alternative libraries offer better performance, a cut-down minification process MAY be implemented in order to achieve the major benefits without a significant performance cost.

  7. pipeline-configuration.md line 50

    Configuration files MUST be human-readable and consistent between languages.

  8. access-to-results.md lines 5 and 11

    Where string keys are used, the following rules apply:

    • String comparisons MUST be case insensitive.
  9. ip-intelligence-specification/required-examples.md line 107

    Examples MUST demonstrate accessing both the value and its associated weighting:

What changed

Links are to the head, 0014283.

  1. Every query string parameter is shared by default. shareAllQueryStringParameters starts true in ShareUsageBuilderBase line 69, with its @DefaultValue and Javadoc to match. Three tests in ShareUsageOptionsTests pinned the old default. Two now expect the new one, and the third turns the option off to test the other options.
  2. Only 51D_ cookies are shared. EvidenceKeyFilterShareUsage.include lines 204 to 212 applies the cookie rule before the share all Evidence check. The session cookie is still included for the tracker, which compares it and never sends it. ShareUsageOptionsTests asserted that cookie.plain is shared when all Evidence is shared, and now expects it not to be.
  3. A failed update check is caught, logged and reported, and later checks still run. In DataUpdateServiceDefault each polling and watcher task catches a failure and logs it at Error as "An unhandled error occurred while checking for automatic updates for engine '<type>'", with the error attached. An error of a kind the service does not expect is reported to the callbacks as the new AUTO_UPDATE_UNKNOWN_ERROR, and the next polling check is still scheduled. The watcher task always resets its watch key (line 743), so it keeps running. A programmatic check and the update on start-up still pass a callback's error to their caller.
  4. The four rows of the Errors table. An integrity check failure is logged at Error with the table's message. A response with no Content-MD5 header is logged as a warning and the update goes on (line 1138). A data file with no temporary path gives the new AUTO_UPDATE_TEMP_PATH_NOT_SET (line 1285). A failed write of data passed in from memory is logged as a warning and the engine is still refreshed (line 577). The two new statuses are added at the end of AutoUpdateStatus. DataUpdateServiceTests.DataUpdateService_UpdateFromUrl_MD5Invalid allowed one warning and now allows the one Error the table gives.
  5. Evidence keys are matched without regard to case. The two EvidenceKeyFilterWhitelist constructors that take no comparator now use String.CASE_INSENSITIVE_ORDER. The constructors that take a comparator are unchanged, and EvidenceKeyFilterAggregatorTests now passes its case-sensitive comparator explicitly. The JavaScript Builder reads query Evidence whatever the case of its prefix (JavaScriptBuilderElement line 458).
  6. The JavaScript Builder minifies its script by default. A new setMinify option on JavaScriptBuilderElementBuilder, default true as the SetMinify row gives, and a new JavaScriptMinifier. The minifier makes one pass over the script, removing comments, indentation and empty lines. It keeps line breaks, so the way the browser inserts semicolons is unchanged, and it keeps strings, template literals and regular expressions as they are. It is the cut-down process the specification allows, so the published package gains no new dependency. The JavaScriptBuilderElement constructor gains a minify argument, and the constructor without it is kept and minifies. Tests that read the rendered text (JavaScriptObjectNameTests, SessionIdAndSequenceTests, JavaScriptUrlTests and three in JavaScriptBuilderTests) set setMinify(false).
  7. A property list is read whatever white space is around its commas. AspectEngineBuilderBase.setProperties splits at each comma, trims each name and skips an empty one.
  8. Element Data keys are compared without regard to case. TypedKeyMapBuilder finds a key with equalsIgnoreCase.
  9. The IP Intelligence getting started examples show each weighting. PropertyHelper gains weightedValues, which gives a single value a weighting of 1, and asWeightedString. GettingStartedOnPrem and GettingStartedCloud read each value and its weighting through them and print, for example, "gb (weighting 1.00)".

No public name is removed or renamed.

Before you upgrade

  1. Every query string and form parameter is now shared by default. A site that sets nothing now shares every query string and HTTP form parameter with usage sharing, apart from the keys that are never shared. Before, it shared only those starting 51D_ and those named in the included list. To keep the old behaviour, call setShareAllQueryStringParameters(false) on the usage sharing builder, or set ShareAllQueryStringParameters to false in the configuration file.
  2. Share all Evidence no longer shares a site's own cookies. With share all Evidence on, a cookie whose name does not start 51D_ is no longer shared, and nor is the session cookie. There is no setting to share them.
  3. The Evidence key whitelist ignores case by default. An EvidenceKeyFilterWhitelist built with a list or a map and no comparator now matches Query.UserAgent to query.useragent. To keep matching with case, pass a case-sensitive comparator to the constructor that takes one.
  4. The JavaScript Builder's script is now minified. The script it returns has no comments, indentation or empty lines. Code that reads the script's text should call setMinify(false) on the builder, or set Minify to false in the configuration file.
  5. Data update statuses. A response with no Content-MD5 header no longer fails the update with AUTO_UPDATE_ERR_MD5_VALIDATION_FAILED, and is logged as a warning instead. A data file with no temporary path gives AUTO_UPDATE_TEMP_PATH_NOT_SET, and an unexpected error gives AUTO_UPDATE_UNKNOWN_ERROR, where both gave AUTO_UPDATE_REFRESH_FAILED. Code that switches on AutoUpdateStatus should handle the two new values. An integrity check failure is now logged at Error rather than Warning.

Tests

Each item has a test commit followed by its fix, and each test commit was run on a runner before its fix.

Red, run 43772345068. The run's commit, c2ec1b9, is the head with only a temporary workflow added. Each of its eleven jobs checks out one test commit, builds the module it tests with the modules it needs on Java 17, runs only that commit's new or changed tests, and fails.

Test commit Tests Result
600b974 ShareUsageOptionsTests 1 of 10 fails, defaults_ShareEveryQueryStringParameter expected true but was false
3897134 EvidenceKeyFilterShareUsageTests 2 of 14 fail, both cookie tests expected false but were true
6a0c78b DataUpdateServiceCheckFailureTests 3 of 3 fail, "The scheduled task threw, so its error is lost and a repeating task stops"
fd93b77 DataUpdateServiceLoggingTests 5 of 29 fail, one for each changed row
b9bbebf EvidenceKeyFilterWhitelistCaseTests 3 of 3 fail
b9bbebf JavaScriptUrlTests 1 of 27 fails, Parameters_QueryPrefixIsCaseInsensitive, "Query evidence with prefix 'Query.' was dropped"
eb9fafa JavaScriptMinifyTests, MinifyOverheadTests do not compile, as JavaScriptMinifier and setMinify do not exist
6a76df2 AspectEngineBuilderPropertiesTests 3 of 5 fail
a455269 FlowDataTests 2 of 41 fail, "There is no data for 'Device'"
4fe3b94 PropertyHelperWeightingTests does not compile, as weightedValues and asWeightedString do not exist
4fe3b94 WeightingShownTest 2 of 2 fail, "GettingStartedOnPrem does not read a weighting" and the same for GettingStartedCloud

The WeightingShownTest job removes PropertyHelperWeightingTests before it builds, because that test does not compile at this commit and the console module's build needs the shared module.

Green, the full build below, in which every one of these tests ran on the head.

Build

Run 43771872937 built 0014283 merged with rc (e2472ee) on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. Each of its ten Setting 'PullRequestSha' lines names 0014283, and its Configure job printed "Left out 6 macOS entries". Every test class named in the red table ran and passed in all nine jobs, MinifyOverheadTests among them.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Merging with #186

#186 also brings the data update Errors table rows into line, and a trial merge of the two heads conflicts in DataUpdateServiceDefault.java, DataUpdateServiceLoggingTests.java and DataUpdateServiceTests.java. Both also add AUTO_UPDATE_TEMP_PATH_NOT_SET to AutoUpdateStatus, this draft at the end and #186 in the middle, and git merges DataUpdateService.java without a conflict, which would declare the constant twice and stop the build. Whichever of the two merges second must first merge rc, after the other has merged, and keep one declaration, at the end of the enum.

What remains

  1. #150 keeps the parts of the data update Errors table this draft does not cover. This draft covers its item 1, the integrity check failure, and adds two of the three statuses in its item 5. The rest of Log the rest of the data update Errors table as the specification gives it #150 remains, including its item 3, that a programmatic check and the update on start-up should pass an unexpected error to their caller rather than log it and return AUTO_UPDATE_UNKNOWN_ERROR, as data-updates.md lines 477 to 479 asks.
  2. The minification time limits are 3 ms for one minification and 5 ms of extra time for each Process call. Timing run 43778506201, on the head with only a temporary workflow added, ran MinifyOverheadTests three times on each kind of runner the release builds on, with Java 17, and printed the medians the test records. The slowest runner is macOS Intel (macos-15-intel), at 0.30 to 0.36 ms for one minification and 0.35 to 0.49 ms of extra time for each Process call, so each limit is about ten times the slowest runner's figure. The others gave 0.13 to 0.18 ms and 0.13 to 0.30 ms (Ubuntu x64 and arm64, Windows, macOS on Apple silicon).

Ready for review, head 0014283, build https://github.com/51Degrees/51degrees-java/actions/runs/43771872937

…ry string parameters is on by default (US-5)
… string parameter is shared unless a user turns the option off (US-5)
…D_, the session cookie among them, is still not shared (US-35, US-36)
… cookie among them, even when all Evidence is shared (US-35, US-36)
@jwrosewell jwrosewell added the remote-conformance Specification conformance work from the remote packs label Nov 15, 2026
…eported, and does not stop later checks (DU-13, DU-14)
… unexpected one as AUTO_UPDATE_UNKNOWN_ERROR, and keep later checks running (DU-13, DU-14)
…e for MD5, temporary path, other errors and writes from memory (DU-15)
…on when Content-MD5 is missing, add AUTO_UPDATE_TEMP_PATH_NOT_SET and AUTO_UPDATE_UNKNOWN_ERROR, and warn on a failed write from memory (DU-15)
… whitelist and the JavaScript builder's query parameters (EV-5)
…t, and read query Evidence in the JavaScript builder without regard to case (EV-5)
…ify option, using a cut-down single pass minifier that keeps line breaks (JB-7)
…t is read the same whatever white space is around the commas (CF-8)
…stored under device is found as Device (PR-3)
…th its weighting, read through a new weightedValues helper (IP-32)
@jwrosewell jwrosewell changed the title Specification conformance, 51degrees-java, section 14 Nine fixes the specification asks for, in usage sharing, data updates, Evidence keys and the JavaScript Builder Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Specification conformance work from the remote packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant