Repository navigation
Add the Derived Property Element and its example - #156
Draft
jwrosewell wants to merge 7 commits into
Draft
jwrosewell wants to merge 7 commits into
jwrosewell wants to merge 7 commits into
Conversation
Adds the tests for a new pipeline.derivedproperty module and a new developer example, before the code they test. The module is the Derived Property Element of the specification, which produces new properties from the properties other elements in the Pipeline produce, following rules written in a YAML or JSON script. The derived-properties repository is added as the submodule pipeline/derived-properties, at cc30638. Its scripts folder holds the built-in scripts the package ships, and its tests folder holds the case files and rejection cases every language runs. SharedCasesTests runs both, and BuiltInScriptTests fails when BuiltInScript and the scripts folder differ. The other tests cover reading YAML by the YAML 1.2 core schema, so yes stays a string, the conversion of source values, the Pipeline check, the no value message, the logging, the metadata and building from a configuration file. ExampleTests and FindOutMoreTests run the example. The two modules are added to pipeline/pom.xml, the developer examples pom and ci/packages.json, and SnakeYAML is named in the dependency management of pipeline/pom.xml. The tests do not compile until the code follows.
pipeline.derivedproperty is the Derived Property Element of the specification. It is a Flow Element rather than an Aspect Engine, holds no data file, makes no network request and accepts no evidence. Every instance writes into the Element Data under the key derived, as every Translation Engine shares translation. DerivedPropertyElementBuilder takes scripts shipped in the package (BuiltInScript), files on disk with wildcards in the file name, and text from code, and a configuration file names it as DerivedPropertyElement with the Scripts and ScriptFiles build parameters. Each script is parsed with the compose API of SnakeYAML, so every node keeps its line, and plain scalars are resolved by the YAML 1.2 core schema. The validator follows tools/validate.mjs in the derived-properties repository, with the same paths and messages, and reports every fault in every script at once. Per request the element reads each source property once, converts it to the type the script infers without coercing loosely, then runs the checks and the rules. Where a source property is not available the output has no value, using the existing AspectPropertyValue mechanism, and the message names every property that could not be read, in the wording of tools/evaluate.mjs. A request with no Element Data under a key a script reads is logged once per script. When the Pipeline adds the element it checks that some element lists each source property, without judging where that element sits, and that no two Derived Property Elements produce the same property. The developer example builds one Pipeline in code, with a script held as text and a script read from a file beside the example, and the same Pipeline from a configuration file, and shows the message a caller sees when a source property is not available. It ends with a Find out more section.
…ccepts
The tests made each pattern with Path.resolve("*.yaml"), and Windows
refuses a path that holds a wildcard with InvalidPathException, so three
DerivedPropertyElementConfigurationTests failed in every Windows job of
full build 43667464820. The element itself splits a pattern with
File.getName and File.getParentFile and was not affected.
jwrosewell
added a commit
that referenced
this pull request
Nov 15, 2026
…its verify (#161) Tests only. `DataUpdateServiceTests.DataUpdateService_Register_AutoUpdateConfiguredInterval` fails about once in 600 runs, and it failed `Ubuntu_Java_17` in full build [43667464820](https://github.com/51Degrees/51degrees-java/actions/runs/43667464820) of [#156](#156), which changes nothing in `pipeline.engines`. This fixes the cause rather than leaving it to a re-run, which is the rule for a test that fails on timing. [#122](#122), which is open and not yet merged, adds that rule to AGENTS.md.
…for a package never released
…nslation already uses
jwrosewell
added a commit
that referenced
this pull request
Nov 16, 2026
…ith no key to the drafts that carry them The Derived Property Element and its example are #156's, Comparer on the free data file is #158's, and every cloud example ending cleanly with no resource key is #163's, so each file this draft changed only for those is put back to rc's version. The cloud cases of the no-settings tests go with them, as #163's tests cover each cloud example run with no key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Derived Property Element to Java, as .NET and Rust have it, with its example. Closes #110.
The element produces new property values from property values other Flow Elements have already produced, following rules written in a script, YAML or JSON, and writes them into the Element Data under the key
derived. It holds no data file, makes no request over the network and reads no Evidence. The scripts and their test cases are shared by every language in the derived-properties repository, which this repository now holds as the submodulepipeline/derived-propertiesat cc306386, so Java gives the same answers and the same faults as the other languages, case for case.What the specification says
The element is specified in derived-property-element.md at 706de1f0, which states its package behaviour without RFC 2119 keywords. The requirement on each language is in required-examples.md lines 30 to 51:
What changed
pipeline.derivedproperty(pipeline/pipeline.derivedproperty).DerivedPropertyElementBuildertakes the built-in scripts by name (addScript(BuiltInScript), orsetScriptswith a comma separated list, as a configuration file gives it), a script file (addScriptFile, orsetScriptFileswith a comma separated list whose file names may hold wildcards), and a script as text (addScript(String, ...)). Each script is parsed and checked when the element is built, so a script that is not valid stops the build withDerivedScriptValidationException, which names the fault. The built-in scripts,HumanConfidenceandIsGdpr, are packaged from the submodule'sscriptsfolder as resources. Per request the element reads each source property once, runs each script, and gives each derived property either its value or no value with a message that names what was missing.pipeline.developer-examples.derivedproperty(Main.java). Its source properties come from a small element written in the example,SessionElement, so it needs no data file, no key and no network. It shows the four things the specification lists, a script held as text, a script read from ReaderEngagement.yaml beside it, the same Pipeline built from derived-property.xml, and a request without a source property, and ends with a Find out more section. It reads none of the shared scripts.pipeline/pom.xmllists the module and putsorg.snakeyaml:snakeyaml-engine2.7 in its dependency management, the YAML 1.2 readerpipeline.translationalready uses, which the package reads its scripts with (commit 6323472, in place of a second YAML library).ci/packages.jsonlists both modules, published as the other developer examples are, andpipeline/README.mdand AGENTS.md describe the package and the submodule.Tests
Tests first, in 9995692, the code in dcbab24.
SharedCasesTestsruns every case file and every rejection case in the submodule'stestsfolder, the cases every language runs.DerivedPropertyElementTests,DerivedScriptParserTests,SourceValuesTests,BuiltInScriptTestsandDerivedPropertyElementConfigurationTeststest the element, the parser, the reading of source values, the packaged scripts and the configuration file route.ExampleTestsruns it and checks the four parts, andFindOutMoreTestschecks its closing section and its tagged pages.Red 43620956485 ran the test commit plus a throwaway workflow on JDK 17 and JDK 8, and the tests did not compile, as the classes do not exist before the code. Green 43620965905 ran the code commit plus the same workflow, and on both JDKs the module's 213 tests passed, 167 of them the shared cases, as did the example's tests.
The first full build, 43667464820, failed in every Windows job, because
DerivedPropertyElementConfigurationTestsbuilt its file patterns withPath.resolve("*.yaml"), which Windows refuses. c9693a95 builds them as text. The element itself splits a pattern withFile.getNameandFile.getParentFile, so it was not affected. ItsUbuntu_Java_17job failed on a race in a data update test ofpipeline.engines, which this draft does not touch and #161 fixes.Full build
Run 43909692444 built a05afaa merged with
rc81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21. Configure and all nine Build and Test jobs passed, andSharedCasesTestspassed in all nine, read from each job's log. Each of its tenSetting 'PullRequestSha'lines names a05afaa, and its Configure job printed "Left out 6 macOS entries". The run's last job, which only gathers the results, was still waiting for a runner when this was written.An earlier full build, run 43673959199, passed on c9693a9, before the two review fixes and the merge of
rc.macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.
What it releases
pipeline.derivedpropertyandpipeline.developerexamples.derivedpropertyare new packages, released at their first version. The change topipeline/pom.xmlreleases thepipelineparent with the packages that depend on it. Both new entries inci/packages.jsongive aFirstVersion, which the release plan needs for a package that has never been released (commit 0cca79f).Base branch
rc81e3d4a is merged into the branch at a05afaa, with no conflict.What remains
HumanConfidencereads values only the cloud service gives, so in an on-premise Pipeline it gives no value with its message. The example does not use the built-in scripts, as the specification asks.