Repository navigation
Keep the Set Headers Element's SetHeader properties for each Pipeline - #185
Merged
Merged
Conversation
The multi-pipeline elements specification says an element that keeps state for the Pipeline it is in keeps that state for each Pipeline it is added to (multi-pipeline-elements.md line 17 at 706de1f0, https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/advanced-features/multi-pipeline-elements.md?plain=1#L17). These tests add one SetHeaderElement to two Pipelines, one holding an element that gives SetHeaderBrowserAccept-CH and the other an element that gives SetHeaderPlatformCritical-CH. test_each_pipeline_gives_its_own_headers processes the Pipelines in each order, twice over, and expects each Pipeline's own headers. This fails today, because the element builds its list of SetHeader properties from the first Pipeline it processes for and keeps that list for every Pipeline. The second Pipeline then looks for an element it does not hold and gives the first Pipeline's header with an empty value, for example Accept-CH set to "" where Critical-CH is expected. test_one_pipeline_gives_the_same_headers_on_every_request processes one Pipeline three times. It passes today and guards that a Pipeline keeps giving its own headers. test_headers_are_given_once_the_properties_are_known processes a Pipeline whose element has no properties on its first request, as a cloud engine has none until the cloud service answers, and then gives the element its properties. It expects no headers and then the header. It passes today, because an empty list is built again on the next request, and guards that a list kept for each Pipeline is not an empty one kept for good. The cloud request engine specification says consumers of that metadata "MUST NOT permanently cache an empty result obtained before discovery has completed" (cloud-request-engine.md line 298 at 706de1f0, https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/pipeline-elements/cloud-request-engine.md?plain=1#L298). test_setheader_properties_holds_the_first_pipelines_properties processes the two Pipelines that share one element and checks that the public setheader_properties holds the first Pipeline's dict of data key to property names, the shape it has today. It passes today and guards that shape.
The multi-pipeline elements specification says "If this feature is enabled, then all Flow Elements that maintain any such state information MUST ensure that they have the means to keep track of isolated instances of this state for each Pipeline to which they are added and use the correct state information when processing." (multi-pipeline-elements.md line 17 at 706de1f0, https://github.com/51Degrees/specifications/blob/706de1f0cb3edd560eb7fd7ef3bb19b93a4883dc/pipeline-specification/advanced-features/multi-pipeline-elements.md?plain=1#L17). The Set Headers Element built its list of SetHeader properties from the first Pipeline it processed for and used that list for every Pipeline. One element added to two Pipelines whose other elements give different SetHeader properties gave the second Pipeline the first one's headers, each with an empty value, and printed that the first one's element was not found. The element now keeps a list for each Pipeline in the private _setheader_properties_by_pipeline, keyed by the Pipeline, and builds each list from that Pipeline's own elements on the Pipeline's first request. Processing uses only those lists. As before, a list that is empty is built again on the next request, so the headers of a cloud engine whose properties arrive after the first request are still given. Two requests that build a list at the same time build the same one, and each reads and writes the dictionary in a single step, so no lock is needed. The public setheader_properties keeps the shape it had, a dict of data key to the names of that element's SetHeader properties. It holds a copy of the first list the element builds that is not empty, set once, so code that reads setheader_properties["device"] keeps working. As it is a copy, changing it no longer changes the headers. PipelineBuilder adds a new Set Headers Element to each Pipeline it builds, so only an application that adds one element to several Pipelines sees its headers change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One Set Headers Element added to several Pipelines now gives each Pipeline its own response headers, built from that Pipeline's own elements, rather than the headers of the first Pipeline it processed for.
Problem
The Set Headers Element built its list of SetHeader properties from the first Pipeline it processed for, and used that list for every Pipeline (
setheaderelement.pyat f6998d0d). One element in two Pipelines whose other elements give different SetHeader properties gave the second Pipeline the first one's headers, each with an empty value, and printed that the first one's element was not found.What the specification says
multi-pipeline-elements.md lines 16 to 19 at 706de1f0:
cloud-request-engine.md lines 298 and 299, which the list kept for each Pipeline must also meet, says consumers of the cloud Property metadata "MUST NOT permanently cache an empty result obtained before discovery has completed".
What changed
8298a3c0, the tests,SetHeaderMultiPipelineTestsintest_setheader.py. One element is added to a Pipeline whose element givesSetHeaderBrowserAccept-CHand to one whose element givesSetHeaderPlatformCritical-CH.test_each_pipeline_gives_its_own_headersprocesses them in each order, twice over, and expects each Pipeline's own header. Three tests pass on rc and guard what must not change, being one Pipeline giving the same headers on every request, a Pipeline whose element has no properties on its first request giving the header once they arrive (as a cloud engine does), and the publicsetheader_propertieskeeping its shape.598949a7, the change. The element keeps a list for each Pipeline in the private_setheader_properties_by_pipeline, keyed by the Pipeline, built from that Pipeline's own elements on its first request, and processing reads only those lists. An empty list is built again on the next request, as before. Two requests that build a list at once build the same one, and each reads and writes the dictionary in one step, so no lock is needed. The publicsetheader_propertieskeeps its shape, a dict of data key to the names of that element's SetHeader properties, and holds a copy of the first list built that is not empty.Before you upgrade
setheader_propertiesis now a copy, so changing it no longer changes the headers sent.PipelineBuilderadds a new Set Headers Element to each Pipeline it builds, so only an application that adds one element to several Pipelines itself sees its headers change, each Pipeline now getting its own.How it was verified
rcf6998d0) plus only a throwaway workflow. On Ubuntu and Windows only the two cases oftest_each_pipeline_gives_its_own_headersfailed, "2 failed, 23 passed".tmp/pr-matrix-185fromrcf6998d0, green in all ten Build and Test jobs (Python 3.10 to 3.14 on Ubuntu and Windows), the web examples and the shared browser tests of the response headers included. Its elevenSetting 'PullRequestSha'lines are each 598949a, the head. macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main. Configure printed "Left out 5 macOS entries".What it releases
fiftyone_pipeline_coreand its dependants, which Monday's release plan releases already.What remains
The other elements that kept one Pipeline's state are each changed in a draft of their own, as the Set Headers Element is here, being the Derived Property Element (#197), the cloud aspect engines (#198), the JSON Builder (#200, stacked on #179), the Usage Sharing Element (#202) and the on-premise engines, which loaded their data file again for each Pipeline (#203).