Skip to content

Add the Derived Property Element and its example - #156

Draft
jwrosewell wants to merge 7 commits into
rcfrom
feature/derived-property-element
Draft

jwrosewell wants to merge 7 commits into
rcfrom
feature/derived-property-element

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Adds the Derived Property Element to Java, as .NET and Rust have it, with its example. Closes #110.

The element produces new property values from property values other Flow Elements have already produced, following rules written in a script, YAML or JSON, and writes them into the Element Data under the key derived. It holds no data file, makes no request over the network and reads no Evidence. The scripts and their test cases are shared by every language in the derived-properties repository, which this repository now holds as the submodule pipeline/derived-properties at cc306386, so Java gives the same answers and the same faults as the other languages, case for case.

What the specification says

The element is specified in derived-property-element.md at 706de1f0, which states its package behaviour without RFC 2119 keywords. The requirement on each language is in required-examples.md lines 30 to 51:

Every reference implementation MUST provide one example for the Derived Property Element. It MUST NOT need a resource key, a data file or a network connection, so that it runs anywhere, and it MUST NOT depend on the content of the shared script repository, because a test in a language repository that read a script from that repository would break whenever a script changed.

The example MUST show all four of the following. [...]

  1. A script held in the example itself and passed to the builder as text, producing a value from source Properties an Element in the example supplies.
  2. A script read from a file beside the example.
  3. The same Pipeline built from a configuration file rather than from code, naming the builder and passing the script through build parameters.
  4. What the caller sees when a source Property named by the script is not available on the request, which is a Property with no value and a message naming what was missing. This case MUST be shown rather than only described, [...]

What changed

  • A new package, pipeline.derivedproperty (pipeline/pipeline.derivedproperty). DerivedPropertyElementBuilder takes the built-in scripts by name (addScript(BuiltInScript), or setScripts with a comma separated list, as a configuration file gives it), a script file (addScriptFile, or setScriptFiles with a comma separated list whose file names may hold wildcards), and a script as text (addScript(String, ...)). Each script is parsed and checked when the element is built, so a script that is not valid stops the build with DerivedScriptValidationException, which names the fault. The built-in scripts, HumanConfidence and IsGdpr, are packaged from the submodule's scripts folder as resources. Per request the element reads each source property once, runs each script, and gives each derived property either its value or no value with a message that names what was missing.
  • A developer example, pipeline.developer-examples.derivedproperty (Main.java). Its source properties come from a small element written in the example, SessionElement, so it needs no data file, no key and no network. It shows the four things the specification lists, a script held as text, a script read from ReaderEngagement.yaml beside it, the same Pipeline built from derived-property.xml, and a request without a source property, and ends with a Find out more section. It reads none of the shared scripts.
  • Wiring. pipeline/pom.xml lists the module and puts org.snakeyaml:snakeyaml-engine 2.7 in its dependency management, the YAML 1.2 reader pipeline.translation already uses, which the package reads its scripts with (commit 6323472, in place of a second YAML library). ci/packages.json lists both modules, published as the other developer examples are, and pipeline/README.md and AGENTS.md describe the package and the submodule.

Tests

Tests first, in 9995692, the code in dcbab24.

  • SharedCasesTests runs every case file and every rejection case in the submodule's tests folder, the cases every language runs.
  • DerivedPropertyElementTests, DerivedScriptParserTests, SourceValuesTests, BuiltInScriptTests and DerivedPropertyElementConfigurationTests test the element, the parser, the reading of source values, the packaged scripts and the configuration file route.
  • The example's ExampleTests runs it and checks the four parts, and FindOutMoreTests checks its closing section and its tagged pages.

Red 43620956485 ran the test commit plus a throwaway workflow on JDK 17 and JDK 8, and the tests did not compile, as the classes do not exist before the code. Green 43620965905 ran the code commit plus the same workflow, and on both JDKs the module's 213 tests passed, 167 of them the shared cases, as did the example's tests.

The first full build, 43667464820, failed in every Windows job, because DerivedPropertyElementConfigurationTests built its file patterns with Path.resolve("*.yaml"), which Windows refuses. c9693a95 builds them as text. The element itself splits a pattern with File.getName and File.getParentFile, so it was not affected. Its Ubuntu_Java_17 job failed on a race in a data update test of pipeline.engines, which this draft does not touch and #161 fixes.

Full build

Run 43909692444 built a05afaa merged with rc 81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21. Configure and all nine Build and Test jobs passed, and SharedCasesTests passed in all nine, read from each job's log. Each of its ten Setting 'PullRequestSha' lines names a05afaa, and its Configure job printed "Left out 6 macOS entries". The run's last job, which only gathers the results, was still waiting for a runner when this was written.

An earlier full build, run 43673959199, passed on c9693a9, before the two review fixes and the merge of rc.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What it releases

pipeline.derivedproperty and pipeline.developerexamples.derivedproperty are new packages, released at their first version. The change to pipeline/pom.xml releases the pipeline parent with the packages that depend on it. Both new entries in ci/packages.json give a FirstVersion, which the release plan needs for a package that has never been released (commit 0cca79f).

Base branch

rc 81e3d4a is merged into the branch at a05afaa, with no conflict.

What remains

  • HumanConfidence reads values only the cloud service gives, so in an on-premise Pipeline it gives no value with its message. The example does not use the built-in scripts, as the specification asks.

Adds the tests for a new pipeline.derivedproperty module and a new
developer example, before the code they test. The module is the Derived
Property Element of the specification, which produces new properties from
the properties other elements in the Pipeline produce, following rules
written in a YAML or JSON script.

The derived-properties repository is added as the submodule
pipeline/derived-properties, at cc30638. Its scripts folder holds the
built-in scripts the package ships, and its tests folder holds the case
files and rejection cases every language runs. SharedCasesTests runs both,
and BuiltInScriptTests fails when BuiltInScript and the scripts folder
differ.

The other tests cover reading YAML by the YAML 1.2 core schema, so yes
stays a string, the conversion of source values, the Pipeline check, the
no value message, the logging, the metadata and building from a
configuration file. ExampleTests and FindOutMoreTests run the example.

The two modules are added to pipeline/pom.xml, the developer examples pom
and ci/packages.json, and SnakeYAML is named in the dependency management
of pipeline/pom.xml. The tests do not compile until the code follows.
pipeline.derivedproperty is the Derived Property Element of the
specification. It is a Flow Element rather than an Aspect Engine, holds no
data file, makes no network request and accepts no evidence. Every
instance writes into the Element Data under the key derived, as every
Translation Engine shares translation.

DerivedPropertyElementBuilder takes scripts shipped in the package
(BuiltInScript), files on disk with wildcards in the file name, and text
from code, and a configuration file names it as DerivedPropertyElement
with the Scripts and ScriptFiles build parameters. Each script is parsed
with the compose API of SnakeYAML, so every node keeps its line, and plain
scalars are resolved by the YAML 1.2 core schema. The validator follows
tools/validate.mjs in the derived-properties repository, with the same
paths and messages, and reports every fault in every script at once.

Per request the element reads each source property once, converts it to
the type the script infers without coercing loosely, then runs the checks
and the rules. Where a source property is not available the output has no
value, using the existing AspectPropertyValue mechanism, and the message
names every property that could not be read, in the wording of
tools/evaluate.mjs. A request with no Element Data under a key a script
reads is logged once per script.

When the Pipeline adds the element it checks that some element lists each
source property, without judging where that element sits, and that no two
Derived Property Elements produce the same property.

The developer example builds one Pipeline in code, with a script held as
text and a script read from a file beside the example, and the same
Pipeline from a configuration file, and shows the message a caller sees
when a source property is not available. It ends with a Find out more
section.
…ccepts

The tests made each pattern with Path.resolve("*.yaml"), and Windows
refuses a path that holds a wildcard with InvalidPathException, so three
DerivedPropertyElementConfigurationTests failed in every Windows job of
full build 43667464820. The element itself splits a pattern with
File.getName and File.getParentFile and was not affected.
jwrosewell added a commit that referenced this pull request Nov 15, 2026
…its verify (#161)

Tests only. `DataUpdateServiceTests.DataUpdateService_Register_AutoUpdateConfiguredInterval` fails about once in 600 runs, and it failed `Ubuntu_Java_17` in full build [43667464820](https://github.com/51Degrees/51degrees-java/actions/runs/43667464820) of [#156](#156), which changes nothing in `pipeline.engines`. This fixes the cause rather than leaving it to a re-run, which is the rule for a test that fails on timing. [#122](#122), which is open and not yet merged, adds that rule to AGENTS.md.
jwrosewell added a commit that referenced this pull request Nov 16, 2026
…ith no key to the drafts that carry them

The Derived Property Element and its example are #156's, Comparer on the
free data file is #158's, and every cloud example ending cleanly with no
resource key is #163's, so each file this draft changed only for those
is put back to rc's version. The cloud cases of the no-settings tests go
with them, as #163's tests cover each cloud example run with no key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant