Skip to content

Ignore the case of Evidence keys in the Device Detection engine, the whitelist filter and the JavaScript Builder - #169

Draft
jwrosewell wants to merge 7 commits into
rcfrom
fix/evidence-key-case
Draft

jwrosewell wants to merge 7 commits into
rcfrom
fix/evidence-key-case

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Evidence keys are meant to ignore case, and in three places Java does not.

  1. The Device Detection on-premise engine ignores a key whose prefix is not in lower case. DeviceDetectionHashEngine.processEngine finds each Evidence key among the engine's keys ignoring case, then hands the native code the key as the caller spelled it (lines 385 to 400 at rc b33daa0f, the key at line 397). The native code matches the prefix with case, using strncmp in fiftyoneDegreesEvidenceMapPrefix (51degrees-cxx src/common/evidence.c at 7992af70, the cxx this repository's rc uses). So a User-Agent added as HEADER.User-Agent or Header.User-Agent passes the engine's filter and is then dropped, and the result is the one for no Evidence. The IP Intelligence engine already gives the native code its own spelling of the key (IPIntelligenceOnPremiseEngine lines 375 to 390).
  2. A whitelist filter made without a comparator compares with case. The two EvidenceKeyFilterWhitelist constructors that take no comparator store the keys in a TreeMap in natural order (lines 42 to 46 and 71 to 75), so an element whose filter is made that way accepts cookie.date-of-birth and not cookie.Date-Of-Birth. Every filter with keys that the packages and the developer examples make passes String.CASE_INSENSITIVE_ORDER, so this reaches an application's own elements, and any element that leaves the comparator out because it reads like the default. The .NET filter ignores case unless it is given a comparer (EvidenceKeyFilterWhitelist.cs lines 55 and 56 at 51degrees-dotnet rc f0ba9653).
  3. The JavaScript Builder passes on query Evidence only when its prefix is in lower case. JavaScriptBuilderElement.getParameters finds the query Evidence it gives the client script with startsWith("query"), and the session id and sequence it leaves out with an exact match (lines 399 and 400 at rc b33daa0f), so query Evidence an application adds as Query.Extra never reaches the script. The web integrations add query Evidence in lower case, so it reaches only an application that adds query Evidence itself.

What the specification says

evidence.md at 706de1f0, line 10:

Keys MUST be case-insensitive and will usually be split into <prefix>.<field>.

What changed

  1. DeviceDetectionHashEngine.processEngine gives the native code the engine's own spelling of each key it matched, as the IP Intelligence engine does.
  2. The two EvidenceKeyFilterWhitelist constructors that take no comparator now use String.CASE_INSENSITIVE_ORDER, through the constructors that take one. A filter given a comparator still uses it. No public name is added, removed or renamed.
  3. JavaScriptBuilderElement.getParameters matches the query prefix ignoring case, with regionMatches, and compares the session id and sequence it leaves out with the key in lower case in the root locale, so query Evidence reaches the script whatever its case and those two never do (commit 9634d9d9).

EvidenceKeyFilterAggregator already merges a whitelist that ignores case into its own whitelist, so a filter made without a comparator now joins it there rather than being checked on its own. EvidenceKeyFilterAggregatorTests.EvidenceKeyFilterAggregator_CaseSensitivity made its case-sensitive filter without a comparator. It now passes Comparator.naturalOrder(), so it still checks a filter that compares with case beside one that does not.

Tests

Written first (commit 0623e68).

  • TestProcessTests.Process_Hash_CaseInsensitivePrefix, through the new ProcessTests.caseInsensitiveEvidencePrefix. A Chrome User-Agent added as HEADER.User-Agent gives a result with a value for every property, as the existing header.USER-AGENT test asks.
  • EvidenceKeyFilterWhitelistTests.EvidenceKeyFilterWhitelist_Default_IgnoresCase. A whitelist made from a list holding header.user-agent includes Header.User-Agent and HEADER.USER-AGENT, and one made from a map holding query.Session-Id includes query.session-id and gives its order for QUERY.SESSION-ID.
  • JavaScriptUrlTests.Parameters_QueryPrefixInAnyCase (commit 74dd66f3), a test that renders the script through the element with no browser. Query.Extra and QUERY.Other reach the script's parameters, and Query.Session-Id does not.

The runs check out the submodules, download the free Lite Device Detection file and the User-Agents file with cxx/data/get-data-files.sh, build pipeline/pipeline.core and the Device Detection on-premise engine with the modules they need, its native library included, on Java 17, and run the EvidenceKeyFilter*Tests classes, TestProcessTests and EvidenceKeysTests.

  • Red, the tests on rc b33daa0 (0623e68 with the workflow, 5bf2ff0): run 43620936140. EvidenceKeyFilterWhitelistTests failed 1 of 3 (EvidenceKeyFilterWhitelist_Default_IgnoresCase) and TestProcessTests 1 of 9 (Process_Hash_CaseInsensitivePrefix, "Property IsMobile should have value but doesn't"), the two new tests, while EvidenceKeyFilterAggregatorTests 3, EvidenceKeyFilterShareUsageTests 10 and EvidenceKeysTests 4 with 1 skipped passed.
  • Green, the change (7f74f14 with the workflow, bf5b7a3): run 43620945237. Every test passed, 3, 9, 3 and 10, and EvidenceKeysTests 4 with the same 1 skipped.
  • Red 2, the JavaScript Builder test on 7f74f14 (74dd66f with a throwaway workflow that builds pipeline/pipeline.javascriptbuilder and runs JavaScriptUrlTests and JavaScriptBuilderElementBuilderTests on Ubuntu and Windows, Java 17): run 43702386067. Parameters_QueryPrefixInAnyCase failed on both systems with "Query.Extra is not in the parameters", the parameters object being empty, and the other 26 JavaScriptUrlTests and the 14 JavaScriptBuilderElementBuilderTests passed.
  • Green 2, the head (9634d9d with the same workflow): run 43702395560. JavaScriptUrlTests passed 27 of 27 and JavaScriptBuilderElementBuilderTests 14 of 14 on both systems.

Build

Run 43944930371 built c218e23 merged with rc 820a544 on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. Each of its ten Setting 'PullRequestSha' lines names c218e23, and its Configure job printed "Left out 6 macOS entries".

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Before you upgrade

An EvidenceKeyFilterWhitelist made without a comparator now ignores case, as Evidence keys do. A Flow Element whose filter relied on matching case, for example to accept query.a and refuse query.A, now accepts both. To keep comparing with case, pass a comparator such as Comparator.naturalOrder(). Evidence given to the Device Detection on-premise engine with its prefix in capitals, such as Header.User-Agent, is now used. Query Evidence an application adds with its prefix in capitals, such as Query.Extra, now reaches the JavaScript Builder's client script.

Base branch

rc 820a544 is merged into the head, c218e23, with a merge commit. The merge brought in #178, which minifies the script by default, so commit c218e23 has the new query prefix test read the script as written with setMinify(false), as #178 did for the other tests in JavaScriptUrlTests. JavaScriptUrlTests passed 27 of 27 in all nine jobs.

What remains

Nothing for this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant