Repository navigation
Ignore the case of Evidence keys in the Device Detection engine, the whitelist filter and the JavaScript Builder - #169
Draft
jwrosewell wants to merge 7 commits into
Draft
jwrosewell wants to merge 7 commits into
jwrosewell wants to merge 7 commits into
Conversation
…e Device Detection engine
… the native code its own key spelling
…e case of its prefix
…aScript Builder's parameters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Evidence keys are meant to ignore case, and in three places Java does not.
DeviceDetectionHashEngine.processEnginefinds each Evidence key among the engine's keys ignoring case, then hands the native code the key as the caller spelled it (lines 385 to 400 at rc b33daa0f, the key at line 397). The native code matches the prefix with case, usingstrncmpinfiftyoneDegreesEvidenceMapPrefix(51degrees-cxxsrc/common/evidence.cat 7992af70, the cxx this repository'srcuses). So a User-Agent added asHEADER.User-AgentorHeader.User-Agentpasses the engine's filter and is then dropped, and the result is the one for no Evidence. The IP Intelligence engine already gives the native code its own spelling of the key (IPIntelligenceOnPremiseEnginelines 375 to 390).EvidenceKeyFilterWhitelistconstructors that take no comparator store the keys in aTreeMapin natural order (lines 42 to 46 and 71 to 75), so an element whose filter is made that way acceptscookie.date-of-birthand notcookie.Date-Of-Birth. Every filter with keys that the packages and the developer examples make passesString.CASE_INSENSITIVE_ORDER, so this reaches an application's own elements, and any element that leaves the comparator out because it reads like the default. The .NET filter ignores case unless it is given a comparer (EvidenceKeyFilterWhitelist.cslines 55 and 56 at 51degrees-dotnet rc f0ba9653).JavaScriptBuilderElement.getParametersfinds the query Evidence it gives the client script withstartsWith("query"), and the session id and sequence it leaves out with an exact match (lines 399 and 400 at rc b33daa0f), so query Evidence an application adds asQuery.Extranever reaches the script. The web integrations add query Evidence in lower case, so it reaches only an application that adds query Evidence itself.What the specification says
evidence.md at 706de1f0, line 10:
What changed
DeviceDetectionHashEngine.processEnginegives the native code the engine's own spelling of each key it matched, as the IP Intelligence engine does.EvidenceKeyFilterWhitelistconstructors that take no comparator now useString.CASE_INSENSITIVE_ORDER, through the constructors that take one. A filter given a comparator still uses it. No public name is added, removed or renamed.JavaScriptBuilderElement.getParametersmatches thequeryprefix ignoring case, withregionMatches, and compares the session id and sequence it leaves out with the key in lower case in the root locale, so query Evidence reaches the script whatever its case and those two never do (commit 9634d9d9).EvidenceKeyFilterAggregatoralready merges a whitelist that ignores case into its own whitelist, so a filter made without a comparator now joins it there rather than being checked on its own.EvidenceKeyFilterAggregatorTests.EvidenceKeyFilterAggregator_CaseSensitivitymade its case-sensitive filter without a comparator. It now passesComparator.naturalOrder(), so it still checks a filter that compares with case beside one that does not.Tests
Written first (commit 0623e68).
TestProcessTests.Process_Hash_CaseInsensitivePrefix, through the newProcessTests.caseInsensitiveEvidencePrefix. A Chrome User-Agent added asHEADER.User-Agentgives a result with a value for every property, as the existingheader.USER-AGENTtest asks.EvidenceKeyFilterWhitelistTests.EvidenceKeyFilterWhitelist_Default_IgnoresCase. A whitelist made from a list holdingheader.user-agentincludesHeader.User-AgentandHEADER.USER-AGENT, and one made from a map holdingquery.Session-Idincludesquery.session-idand gives its order forQUERY.SESSION-ID.JavaScriptUrlTests.Parameters_QueryPrefixInAnyCase(commit 74dd66f3), a test that renders the script through the element with no browser.Query.ExtraandQUERY.Otherreach the script's parameters, andQuery.Session-Iddoes not.The runs check out the submodules, download the free Lite Device Detection file and the User-Agents file with
cxx/data/get-data-files.sh, buildpipeline/pipeline.coreand the Device Detection on-premise engine with the modules they need, its native library included, on Java 17, and run theEvidenceKeyFilter*Testsclasses,TestProcessTestsandEvidenceKeysTests.rcb33daa0 (0623e68 with the workflow, 5bf2ff0): run 43620936140.EvidenceKeyFilterWhitelistTestsfailed 1 of 3 (EvidenceKeyFilterWhitelist_Default_IgnoresCase) andTestProcessTests1 of 9 (Process_Hash_CaseInsensitivePrefix, "Property IsMobile should have value but doesn't"), the two new tests, whileEvidenceKeyFilterAggregatorTests3,EvidenceKeyFilterShareUsageTests10 andEvidenceKeysTests4 with 1 skipped passed.EvidenceKeysTests4 with the same 1 skipped.pipeline/pipeline.javascriptbuilderand runsJavaScriptUrlTestsandJavaScriptBuilderElementBuilderTestson Ubuntu and Windows, Java 17): run 43702386067.Parameters_QueryPrefixInAnyCasefailed on both systems with "Query.Extra is not in the parameters", the parameters object being empty, and the other 26JavaScriptUrlTestsand the 14JavaScriptBuilderElementBuilderTestspassed.JavaScriptUrlTestspassed 27 of 27 andJavaScriptBuilderElementBuilderTests14 of 14 on both systems.Build
Run 43944930371 built c218e23 merged with
rc820a544 on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. Each of its tenSetting 'PullRequestSha'lines names c218e23, and its Configure job printed "Left out 6 macOS entries".macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.
Before you upgrade
An
EvidenceKeyFilterWhitelistmade without a comparator now ignores case, as Evidence keys do. A Flow Element whose filter relied on matching case, for example to acceptquery.aand refusequery.A, now accepts both. To keep comparing with case, pass a comparator such asComparator.naturalOrder(). Evidence given to the Device Detection on-premise engine with its prefix in capitals, such asHeader.User-Agent, is now used. Query Evidence an application adds with its prefix in capitals, such asQuery.Extra, now reaches the JavaScript Builder's client script.Base branch
rc820a544 is merged into the head, c218e23, with a merge commit. The merge brought in #178, which minifies the script by default, so commit c218e23 has the new query prefix test read the script as written withsetMinify(false), as #178 did for the other tests inJavaScriptUrlTests.JavaScriptUrlTestspassed 27 of 27 in all nine jobs.What remains
Nothing for this change.