Repository navigation
Share every query value by default, no cookie but 51D_ ones, and name every package in usage sharing - #193
Draft
jwrosewell wants to merge 11 commits into
Draft
Share every query value by default, no cookie but 51D_ ones, and name every package in usage sharing#193jwrosewell wants to merge 11 commits into
jwrosewell wants to merge 11 commits into
Conversation
A configuration file gives a list of names as one comma-separated string. The usage sharing element and its evidence key filter read a string given for query_whitelist or header_blacklist one character at a time, so query_whitelist="first,second" shares neither parameter and shares a parameter named "f", and header_blacklist="x-one, x-two" leaves both headers shared. These tests fail until a string is split into its names, each trimmed and with empty entries left out, as the on-premise engines already do for restricted_properties. They cover the element built in code and from a configuration file, and the evidence key filter on its own. The test that a list given in code is used as it is passes today and is there to keep that behavior.
…uration file gives them The pipeline configuration specification says "Configuration files MUST be human-readable and consistent between languages." (pipeline-configuration.md line 50 at 706de1f0, https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/features/pipeline-configuration.md?plain=1#L50), and the types it then lists include a "list of strings (represented as a comma-separated string)" (line 60, https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/features/pipeline-configuration.md?plain=1#L60). ShareUsage and ShareUsageEvidenceKeyFilter read a string given for query_whitelist or header_blacklist one character at a time. A configuration file giving query_whitelist as "first,second" shared neither parameter, but did share a parameter named by any single letter of the string, and one giving header_blacklist as "x-one, x-two" left both headers shared. A string given for either list is now split on commas into its names, each trimmed, with empty entries left out, as the on-premise engines do for restricted_properties. An empty string is an empty list. A list or None is used as it is, so code that passes a list sees no change. When a string is given, the ShareUsage attributes query_whitelist and header_blacklist hold the names rather than the string.
The usage sharing specification on its rc now says three things the element does not do (specifications#155 and #156, merged as 33de68a0 and 30b3741f). - "Share all query string parameters is on by default, so every query string and HTTP form parameter is shared unless a user turns it off." - "It MUST also return false, whatever the configuration and even when all Evidence is shared, for any key whose prefix is `cookie.` and whose field does not start with `51D_`." - "An xml element MUST be written for every 51Degrees package in use, being the package that holds the Pipeline, the package that holds each FlowElement in the Pipeline, and every 51Degrees package those depend on." test_share_usage_rules.py tests each, through the message the element sends: query and form values shared by default, the new share_all_query_string_parameters setting turned off in code and in a configuration file, only 51D_ cookies shared when all Evidence is, the session cookie compared but never shared whether or not all Evidence is shared, and a Package element with its version for every 51Degrees package in use and no other. The existing tests that held the old rules change with them. With no settings a query parameter such as password is now shared, sharing all Evidence no longer shares a site's cookies, the filter test of sharing all Evidence expects the same, and the tests of a list of parameters to share, including those for comma-separated lists, turn sharing every parameter off, as only then does the list decide.
… every package The usage sharing specification on its rc (specifications#155 and #156) says every query string and form parameter is shared unless a user turns that off, no cookie whose name does not start 51D_ is ever shared, even when all Evidence is, and each record names every 51Degrees package in use with its version. - ShareUsage takes share_all_query_string_parameters, True by default, and passes it to both of its filters. With it on, every query string and form parameter is shared. With it off, only those starting 51D_ and those in query_whitelist are, as before. None for query_whitelist still shares every parameter. - ShareUsageEvidenceKeyFilter decides a cookie before it looks at share_all, so share_all no longer shares a site's own cookies. The tracker's filter still keeps the session cookie when sessions are tracked, so track_session works the same whether or not all Evidence is shared, and the session cookie is never sent. The filter's own default for the new setting is False, so a filter made directly keeps to its list as before. - Each record holds a Package element for every 51Degrees package in use, after the FlowElement elements. These are fiftyone_pipeline_core, the package of each flow element and every 51Degrees package those require, found from the installed packages' metadata, each named once under its published name with its installed version. A package is a 51Degrees one when its author or author email names 51Degrees, as every package this repository publishes does, so an element an application defines, and a package such as requests, adds none.
test_the_web_request_is_shared_by_the_same_rules checks that a query parameter of the web request that is not on the list is left out, as it would be from Evidence. Every query parameter is now shared by default, so the test turns that off to keep checking the list.
# Conflicts: # pipeline/fiftyone_pipeline_engines_fiftyone/src/fiftyone_pipeline_engines_fiftyone/share_usage.py # pipeline/fiftyone_pipeline_engines_fiftyone/tests/test_share_usage_lists.py
This was referenced Nov 15, 2026
jwrosewell
added a commit
that referenced
this pull request
Nov 15, 2026
Usage sharing now shares every query string and form value by default, and never shares a cookie whose name does not start `51D_`, even when all Evidence is shared. This is the part of [#193](#193) that follows [specifications#155](51Degrees/specifications#155), split from it so that it can merge while #193's Package element waits for a decision.
rc names each Pipeline's own flow elements in its records, and from #218 shares only 51D_ and listed query parameters when sharing every query parameter is off and no list is given. Both are kept. The Package elements this branch adds are written from the flow elements of the Pipeline a record came from, beside rc's FlowElement elements.
…r on every record usage-sharing-element.md at specifications 07e1fced, lines 150 to 153: "The invariant values MUST be written to the first record the element writes. They MUST be written again to the first record the element writes once an hour or more has passed since it last wrote them. They MUST NOT be written to any other record." Lines 115 and 116: "The element MUST create a process identifier when it is built, and MUST write it to every record as the ProcessId element." The tests move the element's clock by hand rather than waiting an hour. The package test and the two Pipeline test now expect the packages and flow elements in the first record only.
… on every record usage-sharing-element.md at specifications 07e1fced, lines 150 to 153: "The invariant values MUST be written to the first record the element writes. They MUST be written again to the first record the element writes once an hour or more has passed since it last wrote them. They MUST NOT be written to any other record." Lines 115 and 116: "The element MUST create a process identifier when it is built, and MUST write it to every record as the ProcessId element." The element creates a version 4 UUID when it is built, as process_id, and writes it first in every record. Version, Product, the FlowElement and Package elements, Language, LanguageVersion, ServerIP and Platform go in the first record from each Pipeline and again in the first one written an hour or more after they were last written, measured by the element's clock attribute. A record written before the lookup of the server's address has ended carries an empty ServerIP, so it does not start the hour and the next record carries the values again.
…the server address is known The server address is looked up on its own thread, so the first record can be written before it is known. The usage sharing page, as specifications#189 changes it, says the element then writes all the invariant values again to the first record it writes once that value is known, and to no other record. The new test holds the lookup up, writes two records while it runs, lets it end, and then checks that only the first record and the first record after the lookup carry the values, and that the hour runs from the record that carried the address.
…he server address is known A record written while the lookup of the server address was running carried the invariant values with an empty ServerIP and was not counted, so every record until the lookup ended carried them. The usage sharing page, as specifications#189 changes it, allows only the first record, the first record once the address is known, and the first record once an hour has passed since they were last written. The element now notes the time of every record that carries them, and whether that record went without the address. The next record carries them again only when the lookup has since found the address, or the hour is up. The decision and the values are made under one lock from one reading of the address, so a record never carries an address the decision was not made on, and a part made while the lookup runs is never kept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Against today's
rc(2373d14), the usage sharing element writes the same values to every record it writes, being Version, Product, the flow elements, Language, LanguageVersion, ServerIP and Platform, although none of them changes while the element lives. It carries no process identifier, so a record that left those values out could not be tied back to them. It also names only the version offiftyone_pipeline_core, so the spread of each package's releases cannot be measured.This pull request's title also names sharing every query value by default and sharing no cookie but
51D_ones. Those rules reachedrcin #201, and #218 then settled whatquery_whitelist=Noneshares. So againstrcthis pull request now changes only what follows.The specification
All at 51Degrees/specifications 07e1fced,
pipeline-specification/pipeline-elements/usage-sharing-element.md, on the specification'src.Lines 115 to 117:
Lines 119 to 126 make the identifier a version 4 UUID written as "32 hexadecimal digits in groups of 8, 4, 4, 4 and 12 separated by hyphens", and say "a reader MUST compare identifiers ignoring letter case".
Lines 128 to 131:
Lines 138 to 145 list the invariant values, being Version, Product, Flow Elements, Packages, Language, LanguageVersion, ServerIP and Platform.
Lines 150 to 153:
Lines 157 to 160:
specifications#189, a draft into the specification's
rc, adds a sentence for a value that is not yet known when the first record is written. At its head, 44d815f8, lines 152 to 155:And lines 345 to 347:
Lines 207 to 210:
Lines 256 to 264:
The change
All in
pipeline/fiftyone_pipeline_engines_fiftyone, inshare_usage.pyand three test files.rc. What they still add againstrcis the Package element.fiftyone_packagesfinds, from the installed packages' metadata,fiftyone_pipeline_core, the package of each flow element and every 51Degrees package those require, each named once under its published name with its installed version and sorted by name. A package counts as a 51Degrees one when its author or author email names 51Degrees, so an element an application defines, and a package such asrequests, adds none. ThePackageelements follow theFlowElementelements.PackageTestsintests/test_share_usage_rules.pyholds this.rc2373d14. Its conflicts were resolved by keepingrc's record header for each Pipeline (Name the elements of each pipeline in its usage sharing records #202) with the Package elements moved into it, andrc's filter andtest_empty_query_whitelistfrom Share only 51D_ and listed query parameters when no list is given #218.tests/test_share_usage_invariant_values.pyis new. It sets the element's clock by hand rather than waiting an hour, and checks that the first record carries every invariant value, that records within the hour carry none, that the values come back in the first record once the hour is up and the hour then runs from that record, that a record without them is otherwise the same, that a record written before the server's address is known does not count, that every record carries one version 4 UUID inProcessId, written first, that a second element has a new identifier, and that a second process has a new identifier.test_the_first_record_names_every_51degrees_package_in_usereplaces the earlier test that every record names every package, andtests/test_share_usage_pipelines.pygainstest_only_the_first_record_from_each_pipeline_names_its_elements.ShareUsagecreatesprocess_id, auuid.uuid4()string, when it is built, and every record now opens with<ProcessId>._write_invariantsdecides whether a record carries the invariant values. They go in the first record from each Pipeline, and again in the first record written onceINVARIANT_VALUES_INTERVAL_SECONDS(3600) have passed since they were last written, whether or not that record is sent, so a record dropped because the queue is full still counts. The time is read fromclock, which istime.monotonicunless a test sets it, and is not a setting.test_the_values_go_again_only_in_the_first_record_once_knownintests/test_share_usage_invariant_values.pyholds up the lookup of the server's address, writes two records while it runs, lets it end, and writes four more with the clock moved by hand. It checks that the first record carries every invariant value with an empty ServerIP, that the second carries none, that the first record after the lookup carries them all with the address, that the next two carry none although the second of them is more than an hour after the first record, and that the record an hour after the one with the address carries them again._write_invariantsnow notes the time of every record that carries the values, and whether that record went without the server's address. A later record carries them again only when the lookup has since found the address, or when the hour is up. So a second record written while the lookup runs leaves them out, and the hour runs from the record that carried the address. When the lookup ends without an address, the values are not written again before the hour is up, because there is no address to add._invariant_xmland_constant_xml, so a record never carries an address other than the one the decision was made on. A part made while the lookup runs is never kept, where before a lookup that ended between building the part and keeping it could leave ServerIP empty for the life of the element.get_constant_xmlkeeps its name and arguments, and no public name changes.The runs
test_share_usage_rules.pyalone "9 failed, 2 passed", thePackageTestsfailing because a record named no package.rc2373d14 with only the test changes of fd37a90 added, beingtest_share_usage_invariant_values.py,test_share_usage_pipelines.pyandtest_share_usage_rules.pyas at 27b9266.rcdoes not name packages yet, so the rules file also brings thePackageTestsof 63b5fd3. All ten jobs failed. In the two jobs whose logs were read, Ubuntu and Windows with Python 3.12,fiftyone_pipeline_engines_fiftyonegave "13 failed, 122 passed", every failure one of the new tests, and every other package's tests passed.ProcessIdentifierTestsandtest_a_record_that_leaves_them_out_is_otherwise_the_samefound noProcessId, for example'<ProcessId>' not found in '<Device><Version>4.5.44</Version>.test_the_invariant_values_are_written_again_once_the_hour_is_upfound the values in the records within the hour, with "Lists differ: [True, True, False, True, True] != [False, False, False, False, False]", andtest_only_the_first_record_from_each_pipeline_names_its_elementsfound the flow elements in every record.test_the_first_record_carries_the_invariant_values,test_records_within_the_hour_leave_the_invariant_values_outandtest_a_record_written_before_the_server_ip_is_known_is_not_countedstopped at their check of the first record, which held every invariant value butPackage, and threePackageTestsfound noPackageelement, for exampleRegex didn't match: '</FlowElement>(<Package [^>]*>[^<]*</Package>)+<Language>'. Those six fail onrcbecause it names no package, so this run does not reach the later checks of those three invariant tests.ProcessId, on the invariant values in every record, or on the flow elements and packages in every record.test_the_first_record_carries_the_invariant_valuespassed there, as it must, because the values were already in every record.fiftyone_pipeline_engines_fiftyone135 passed and 6 subtests passed,fiftyone_pipeline_core560 passed, andfiftyone_pipeline_engines125 passed. The invariant values, rules, pipelines and record tests ran five times and gave 30 passed each time.Setting 'PullRequestSha'lines name 27b9266, and Configure printed "Left out 5 macOS entries". It passed in all ten jobs, Ubuntu and Windows with Python 3.10, 3.11, 3.12, 3.13 and 3.14, and in eachtest_share_usage_invariant_values.pypassed 9,test_share_usage_pipelines.py2 andtest_share_usage_rules.py10.fiftyone_pipeline_engines_fiftyonewith "1 failed, 135 passed", the one failure in each beingtest_the_values_go_again_only_in_the_first_record_once_known, which found every invariant value in the second record, written while the lookup ran ("Lists differ: [] != ['Version', 'Product', 'FlowElement', ..."). In every job each of the other eleven test summaries reported no failure and no error.test_share_usage_invariant_values.pygave 1 failed and 9 passed, failing the same way. At 19e18bcfiftyone_pipeline_engines_fiftyonegave 136 passed and 6 subtests passed, and the invariant values, record fields, pipelines and record tests ran five times and gave 41 passed each time.What remains
ProcessId. Lines 128 to 131 give an identifier to each element and do not cover one element in several Pipelines. That case is a question for the specification.rc. This branch follows its text at 44d815f8, so a change to that text before it is merged needs a change here.rc.Before you upgrade
<ProcessId>, a version 4 UUID that is new each time aShareUsageelement is built.ProcessIdkeeps the latest it has read, as specifications#189 says at lines 345 to 347. Anything that reads the records and expects those values in every record must match records to them byProcessId, as lines 329 to 340 say, and a record with noProcessIdcame from a package from before this rule (lines 353 and 354).Packageelements.ShareUsagehas two new public attributes,process_idandclock, andshare_usagea new constant,INVARIANT_VALUES_INTERVAL_SECONDS. No setting is added and no name is removed or renamed.fiftyone_pipeline_engines_fiftyone, so a release takes it and the published packages that depend on it, beingfiftyone_devicedetection_shared,fiftyone_devicedetection_cloud,fiftyone_devicedetection_onpremise,fiftyone_devicedetection,fiftyone_ipintelligence_shared,fiftyone_ipintelligence_cloud,fiftyone_ipintelligence_onpremiseandfiftyone_ipintelligence.