Skip to content

Share every query value by default, no cookie but 51D_ ones, and name every package in usage sharing - #193

Draft
jwrosewell wants to merge 11 commits into
rcfrom
fix/usage-sharing-rules
Draft

jwrosewell wants to merge 11 commits into
rcfrom
fix/usage-sharing-rules

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

The problem

Against today's rc (2373d14), the usage sharing element writes the same values to every record it writes, being Version, Product, the flow elements, Language, LanguageVersion, ServerIP and Platform, although none of them changes while the element lives. It carries no process identifier, so a record that left those values out could not be tied back to them. It also names only the version of fiftyone_pipeline_core, so the spread of each package's releases cannot be measured.

This pull request's title also names sharing every query value by default and sharing no cookie but 51D_ ones. Those rules reached rc in #201, and #218 then settled what query_whitelist=None shares. So against rc this pull request now changes only what follows.

The specification

All at 51Degrees/specifications 07e1fced, pipeline-specification/pipeline-elements/usage-sharing-element.md, on the specification's rc.

Lines 115 to 117:

The element MUST create a process identifier when it is built, and MUST write it to every record as the ProcessId element. A record that leaves out the invariant values below is tied to them by this identifier.

Lines 119 to 126 make the identifier a version 4 UUID written as "32 hexadecimal digits in groups of 8, 4, 4, 4 and 12 separated by hyphens", and say "a reader MUST compare identifiers ignoring letter case".

Lines 128 to 131:

The identifier is never stored, so a process that starts again creates a new one. A process that builds more than one Usage Sharing Element, for example one for each of two Pipelines, creates an identifier for each, because the invariant values can differ between them.

Lines 138 to 145 list the invariant values, being Version, Product, Flow Elements, Packages, Language, LanguageVersion, ServerIP and Platform.

Lines 150 to 153:

The invariant values MUST be written to the first record the element writes. They MUST be written again to the first record the element writes once an hour or more has passed since it last wrote them. They MUST NOT be written to any other record.

Lines 157 to 160:

The values are written again each hour because a record can be lost, for example when the message that holds it cannot be sent. The hour is measured from when the values were last written to a record, whether or not that record arrived.

specifications#189, a draft into the specification's rc, adds a sentence for a value that is not yet known when the first record is written. At its head, 44d815f8, lines 152 to 155:

When one of them is not yet known as the first record is written, for example a server address still being looked up, the element MUST write all of them again to the first record it writes once that value is known. They MUST NOT be written to any other record.

And lines 345 to 347:

The values written again are the same values, apart from one that was not yet known the time before, so a reader that meets them more than once for one process identifier keeps the latest it has read.

Lines 207 to 210:

A record that leaves them out has no Version, Product, FlowElement, Package, Language, LanguageVersion, ServerIP or Platform element, and is otherwise the same.

Lines 256 to 264:

Package - In a record that carries the invariant values, an xml element MUST be written for every 51Degrees package in use, being the package that holds the Pipeline, the package that holds each FlowElement in the Pipeline, and every 51Degrees package those depend on. Each package is written once. The Name attribute is the name the package is published under, as its language's package manager knows it (for example a NuGet package id, an npm package name, a Maven group and artifact id, a Go module path or a crate name), and the text is the version of that package in use, as that package manager gives it.

The change

All in pipeline/fiftyone_pipeline_engines_fiftyone, in share_usage.py and three test files.

  • 63b5fd3 and f6bf51a, the earlier test and change, carried the Package element together with the query and cookie rules that Share every query value by default, and no cookie but 51D_ ones #201 has since brought to rc. What they still add against rc is the Package element. fiftyone_packages finds, from the installed packages' metadata, fiftyone_pipeline_core, the package of each flow element and every 51Degrees package those require, each named once under its published name with its installed version and sorted by name. A package counts as a 51Degrees one when its author or author email names 51Degrees, so an element an application defines, and a package such as requests, adds none. The Package elements follow the FlowElement elements. PackageTests in tests/test_share_usage_rules.py holds this.
  • f563c5b, a merge of rc 2373d14. Its conflicts were resolved by keeping rc's record header for each Pipeline (Name the elements of each pipeline in its usage sharing records #202) with the Package elements moved into it, and rc's filter and test_empty_query_whitelist from Share only 51D_ and listed query parameters when no list is given #218.
  • fd37a90, the tests. tests/test_share_usage_invariant_values.py is new. It sets the element's clock by hand rather than waiting an hour, and checks that the first record carries every invariant value, that records within the hour carry none, that the values come back in the first record once the hour is up and the hour then runs from that record, that a record without them is otherwise the same, that a record written before the server's address is known does not count, that every record carries one version 4 UUID in ProcessId, written first, that a second element has a new identifier, and that a second process has a new identifier. test_the_first_record_names_every_51degrees_package_in_use replaces the earlier test that every record names every package, and tests/test_share_usage_pipelines.py gains test_only_the_first_record_from_each_pipeline_names_its_elements.
  • 27b9266, the fix.
    • ShareUsage creates process_id, a uuid.uuid4() string, when it is built, and every record now opens with <ProcessId>.
    • _write_invariants decides whether a record carries the invariant values. They go in the first record from each Pipeline, and again in the first record written once INVARIANT_VALUES_INTERVAL_SECONDS (3600) have passed since they were last written, whether or not that record is sent, so a record dropped because the queue is full still counts. The time is read from clock, which is time.monotonic unless a test sets it, and is not a setting.
    • A record written before the lookup of the server's address has ended carries the values with an empty ServerIP and does not count as having written them, so the next record carries them again with the address. This meant every record written while the lookup ran carried them, which specifications#189 does not allow, and 19e18bc below changes it.
  • b41d660, the test for specifications#189. test_the_values_go_again_only_in_the_first_record_once_known in tests/test_share_usage_invariant_values.py holds up the lookup of the server's address, writes two records while it runs, lets it end, and writes four more with the clock moved by hand. It checks that the first record carries every invariant value with an empty ServerIP, that the second carries none, that the first record after the lookup carries them all with the address, that the next two carry none although the second of them is more than an hour after the first record, and that the record an hour after the one with the address carries them again.
  • 19e18bc, the fix for specifications#189.
    • _write_invariants now notes the time of every record that carries the values, and whether that record went without the server's address. A later record carries them again only when the lookup has since found the address, or when the hour is up. So a second record written while the lookup runs leaves them out, and the hour runs from the record that carried the address. When the lookup ends without an address, the values are not written again before the hour is up, because there is no address to add.
    • The decision and the values are made under one lock from one reading of the address, in the new private methods _invariant_xml and _constant_xml, so a record never carries an address other than the one the decision was made on. A part made while the lookup runs is never kept, where before a lookup that ended between building the part and keeping it could leave ServerIP empty for the life of the element.
    • get_constant_xml keeps its name and arguments, and no public name changes.

The runs

  • Red for the Package element, run 43730154249, being 63b5fd3 with only a temporary workflow added, failure as expected. On Ubuntu and Windows the package's tests gave "21 failed, 101 passed", and test_share_usage_rules.py alone "9 failed, 2 passed", the PackageTests failing because a record named no package.
  • Green for the Package element, run 43731748002, success, being 514975a, the head before the merge below, with only a temporary workflow added.
  • Red for the invariant values and the process identifier on CI, run 44009118922, failure as expected. The run is the pull request build, without macOS, of a temporary branch whose commit, 9593a77, is rc 2373d14 with only the test changes of fd37a90 added, being test_share_usage_invariant_values.py, test_share_usage_pipelines.py and test_share_usage_rules.py as at 27b9266. rc does not name packages yet, so the rules file also brings the PackageTests of 63b5fd3. All ten jobs failed. In the two jobs whose logs were read, Ubuntu and Windows with Python 3.12, fiftyone_pipeline_engines_fiftyone gave "13 failed, 122 passed", every failure one of the new tests, and every other package's tests passed.
    • The four ProcessIdentifierTests and test_a_record_that_leaves_them_out_is_otherwise_the_same found no ProcessId, for example '<ProcessId>' not found in '<Device><Version>4.5.44</Version>.
    • test_the_invariant_values_are_written_again_once_the_hour_is_up found the values in the records within the hour, with "Lists differ: [True, True, False, True, True] != [False, False, False, False, False]", and test_only_the_first_record_from_each_pipeline_names_its_elements found the flow elements in every record.
    • test_the_first_record_carries_the_invariant_values, test_records_within_the_hour_leave_the_invariant_values_out and test_a_record_written_before_the_server_ip_is_known_is_not_counted stopped at their check of the first record, which held every invariant value but Package, and three PackageTests found no Package element, for example Regex didn't match: '</FlowElement>(<Package [^>]*>[^<]*</Package>)+<Language>'. Those six fail on rc because it names no package, so this run does not reach the later checks of those three invariant tests.
  • Red outside CI at fd37a90, on Windows with Python 3.12.8, the invariant values, pipelines and rules tests gave 10 failed and 11 passed at fd37a90. Each failure was one of the new or changed tests, failing on no ProcessId, on the invariant values in every record, or on the flow elements and packages in every record. test_the_first_record_carries_the_invariant_values passed there, as it must, because the values were already in every record.
  • Green outside CI at 27b9266, on Windows with Python 3.12.8: fiftyone_pipeline_engines_fiftyone 135 passed and 6 subtests passed, fiftyone_pipeline_core 560 passed, and fiftyone_pipeline_engines 125 passed. The invariant values, rules, pipelines and record tests ran five times and gave 30 passed each time.
  • Full build, run 43959931267, first attempt, success. All eleven Setting 'PullRequestSha' lines name 27b9266, and Configure printed "Left out 5 macOS entries". It passed in all ten jobs, Ubuntu and Windows with Python 3.10, 3.11, 3.12, 3.13 and 3.14, and in each test_share_usage_invariant_values.py passed 9, test_share_usage_pipelines.py 2 and test_share_usage_rules.py 10.
  • Red for specifications#189 on CI, run 44010111720, failure as expected. The run is the pull request build, without macOS, of a temporary branch whose commit is b41d660 with only a temporary workflow added. All ten jobs, Ubuntu and Windows with Python 3.10, 3.11, 3.12, 3.13 and 3.14, failed in fiftyone_pipeline_engines_fiftyone with "1 failed, 135 passed", the one failure in each being test_the_values_go_again_only_in_the_first_record_once_known, which found every invariant value in the second record, written while the lookup ran ("Lists differ: [] != ['Version', 'Product', 'FlowElement', ..."). In every job each of the other eleven test summaries reported no failure and no error.
  • Outside CI on Windows with Python 3.14.7, at b41d660 test_share_usage_invariant_values.py gave 1 failed and 9 passed, failing the same way. At 19e18bc fiftyone_pipeline_engines_fiftyone gave 136 passed and 6 subtests passed, and the invariant values, record fields, pipelines and record tests ran five times and gave 41 passed each time.

What remains

  1. One element can be in several Pipelines, and each Pipeline's first record carries that Pipeline's own invariant values, so a reader sees two sets of FlowElement values under one ProcessId. Lines 128 to 131 give an identifier to each element and do not cover one element in several Pipelines. That case is a question for the specification.
  2. specifications#189 is still a draft into the specification's rc. This branch follows its text at 44d815f8, so a change to that text before it is merged needs a change here.
  3. The full build of 19e18bc.
  4. The title of this pull request still names the query and cookie rules that Share every query value by default, and no cookie but 51D_ ones #201 brought to rc.

Before you upgrade

  • Every record now opens with <ProcessId>, a version 4 UUID that is new each time a ShareUsage element is built.
  • Only the first record from each Pipeline, and then the first record written once an hour or more has passed, carries Version, Product, FlowElement, Package, Language, LanguageVersion, ServerIP and Platform. When the first record is written while the server's address is still being looked up, it carries them with an empty ServerIP, and the first record written once the address is found carries them all again. Every other record leaves them out. A reader that meets them more than once for one ProcessId keeps the latest it has read, as specifications#189 says at lines 345 to 347. Anything that reads the records and expects those values in every record must match records to them by ProcessId, as lines 329 to 340 say, and a record with no ProcessId came from a package from before this rule (lines 353 and 354).
  • A record that carries those values names every 51Degrees package in use, with its version, as Package elements.
  • ShareUsage has two new public attributes, process_id and clock, and share_usage a new constant, INVARIANT_VALUES_INTERVAL_SECONDS. No setting is added and no name is removed or renamed.
  • The change is in fiftyone_pipeline_engines_fiftyone, so a release takes it and the published packages that depend on it, being fiftyone_devicedetection_shared, fiftyone_devicedetection_cloud, fiftyone_devicedetection_onpremise, fiftyone_devicedetection, fiftyone_ipintelligence_shared, fiftyone_ipintelligence_cloud, fiftyone_ipintelligence_onpremise and fiftyone_ipintelligence.

A configuration file gives a list of names as one comma-separated string.
The usage sharing element and its evidence key filter read a string given
for query_whitelist or header_blacklist one character at a time, so
query_whitelist="first,second" shares neither parameter and shares a
parameter named "f", and header_blacklist="x-one, x-two" leaves both
headers shared.

These tests fail until a string is split into its names, each trimmed
and with empty entries left out, as the on-premise engines already do
for restricted_properties. They cover the element built in code and from
a configuration file, and the evidence key filter on its own. The test
that a list given in code is used as it is passes today and is there to
keep that behavior.
…uration file gives them

The pipeline configuration specification says "Configuration files MUST
be human-readable and consistent between languages." (pipeline-configuration.md
line 50 at 706de1f0,
https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/features/pipeline-configuration.md?plain=1#L50),
and the types it then lists include a "list of strings (represented as a
comma-separated string)" (line 60,
https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/features/pipeline-configuration.md?plain=1#L60).

ShareUsage and ShareUsageEvidenceKeyFilter read a string given for
query_whitelist or header_blacklist one character at a time. A
configuration file giving query_whitelist as "first,second" shared
neither parameter, but did share a parameter named by any single letter
of the string, and one giving header_blacklist as "x-one, x-two" left
both headers shared.

A string given for either list is now split on commas into its names,
each trimmed, with empty entries left out, as the on-premise engines do
for restricted_properties. An empty string is an empty list. A list or
None is used as it is, so code that passes a list sees no change. When a
string is given, the ShareUsage attributes query_whitelist and
header_blacklist hold the names rather than the string.
The usage sharing specification on its rc now says three things the
element does not do (specifications#155 and #156, merged as 33de68a0 and
30b3741f).

- "Share all query string parameters is on by default, so every query
  string and HTTP form parameter is shared unless a user turns it off."
- "It MUST also return false, whatever the configuration and even when
  all Evidence is shared, for any key whose prefix is `cookie.` and whose
  field does not start with `51D_`."
- "An xml element MUST be written for every 51Degrees package in use,
  being the package that holds the Pipeline, the package that holds each
  FlowElement in the Pipeline, and every 51Degrees package those depend
  on."

test_share_usage_rules.py tests each, through the message the element
sends: query and form values shared by default, the new
share_all_query_string_parameters setting turned off in code and in a
configuration file, only 51D_ cookies shared when all Evidence is, the
session cookie compared but never shared whether or not all Evidence is
shared, and a Package element with its version for every 51Degrees
package in use and no other.

The existing tests that held the old rules change with them. With no
settings a query parameter such as password is now shared, sharing all
Evidence no longer shares a site's cookies, the filter test of sharing
all Evidence expects the same, and the tests of a list of parameters to
share, including those for comma-separated lists, turn sharing every
parameter off, as only then does the list decide.
… every package

The usage sharing specification on its rc (specifications#155 and #156)
says every query string and form parameter is shared unless a user turns
that off, no cookie whose name does not start 51D_ is ever shared, even
when all Evidence is, and each record names every 51Degrees package in
use with its version.

- ShareUsage takes share_all_query_string_parameters, True by default,
  and passes it to both of its filters. With it on, every query string
  and form parameter is shared. With it off, only those starting 51D_
  and those in query_whitelist are, as before. None for query_whitelist
  still shares every parameter.
- ShareUsageEvidenceKeyFilter decides a cookie before it looks at
  share_all, so share_all no longer shares a site's own cookies. The
  tracker's filter still keeps the session cookie when sessions are
  tracked, so track_session works the same whether or not all Evidence is
  shared, and the session cookie is never sent. The filter's own default
  for the new setting is False, so a filter made directly keeps to its
  list as before.
- Each record holds a Package element for every 51Degrees package in use,
  after the FlowElement elements. These are fiftyone_pipeline_core, the
  package of each flow element and every 51Degrees package those
  require, found from the installed packages' metadata, each named once
  under its published name with its installed version. A package is a
  51Degrees one when its author or author email names 51Degrees, as every
  package this repository publishes does, so an element an application
  defines, and a package such as requests, adds none.
Base automatically changed from fix/share-usage-lists to rc November 15, 2026 15:31
test_the_web_request_is_shared_by_the_same_rules checks that a query
parameter of the web request that is not on the list is left out, as it
would be from Evidence. Every query parameter is now shared by default,
so the test turns that off to keep checking the list.
# Conflicts:
#	pipeline/fiftyone_pipeline_engines_fiftyone/src/fiftyone_pipeline_engines_fiftyone/share_usage.py
#	pipeline/fiftyone_pipeline_engines_fiftyone/tests/test_share_usage_lists.py
jwrosewell added a commit that referenced this pull request Nov 15, 2026
Usage sharing now shares every query string and form value by default, and never shares a cookie whose name does not start `51D_`, even when all Evidence is shared. This is the part of [#193](#193) that follows [specifications#155](51Degrees/specifications#155), split from it so that it can merge while #193's Package element waits for a decision.
jwrosewell added a commit that referenced this pull request Nov 15, 2026
One usage sharing element in several pipelines writes, in each record, the flow elements of the pipeline the request came from. It replaces [#199](#199), which was stacked on [#193](#193), so that it does not wait for #193's Package element.
rc names each Pipeline's own flow elements in its records, and from #218
shares only 51D_ and listed query parameters when sharing every query
parameter is off and no list is given. Both are kept. The Package
elements this branch adds are written from the flow elements of the
Pipeline a record came from, beside rc's FlowElement elements.
…r on every record

usage-sharing-element.md at specifications 07e1fced, lines 150 to 153:
"The invariant values MUST be written to the first record the element
writes. They MUST be written again to the first record the element writes
once an hour or more has passed since it last wrote them. They MUST NOT
be written to any other record."

Lines 115 and 116: "The element MUST create a process identifier when it
is built, and MUST write it to every record as the ProcessId element."

The tests move the element's clock by hand rather than waiting an hour.
The package test and the two Pipeline test now expect the packages and
flow elements in the first record only.
… on every record

usage-sharing-element.md at specifications 07e1fced, lines 150 to 153:
"The invariant values MUST be written to the first record the element
writes. They MUST be written again to the first record the element writes
once an hour or more has passed since it last wrote them. They MUST NOT
be written to any other record."

Lines 115 and 116: "The element MUST create a process identifier when it
is built, and MUST write it to every record as the ProcessId element."

The element creates a version 4 UUID when it is built, as process_id, and
writes it first in every record. Version, Product, the FlowElement and
Package elements, Language, LanguageVersion, ServerIP and Platform go in
the first record from each Pipeline and again in the first one written
an hour or more after they were last written, measured by the element's
clock attribute. A record written before the lookup of the server's
address has ended carries an empty ServerIP, so it does not start the
hour and the next record carries the values again.
…the server address is known

The server address is looked up on its own thread, so the first record
can be written before it is known. The usage sharing page, as
specifications#189 changes it, says the element then writes all the
invariant values again to the first record it writes once that value is
known, and to no other record.

The new test holds the lookup up, writes two records while it runs, lets
it end, and then checks that only the first record and the first record
after the lookup carry the values, and that the hour runs from the
record that carried the address.
…he server address is known

A record written while the lookup of the server address was running
carried the invariant values with an empty ServerIP and was not counted,
so every record until the lookup ended carried them. The usage sharing
page, as specifications#189 changes it, allows only the first record,
the first record once the address is known, and the first record once
an hour has passed since they were last written.

The element now notes the time of every record that carries them, and
whether that record went without the address. The next record carries
them again only when the lookup has since found the address, or the
hour is up. The decision and the values are made under one lock from one
reading of the address, so a record never carries an address the
decision was not made on, and a part made while the lookup runs is
never kept.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant