Skip to content

feat(pipeline): add optional flag to omit intermediary chained stage output - #1057

Open
daanjo3 wants to merge 3 commits into
ldelements:mainfrom
daanjo3:feat/pipeline-chain-optional-emit
Open

daanjo3 wants to merge 3 commits into
ldelements:mainfrom
daanjo3:feat/pipeline-chain-optional-emit

Conversation

@daanjo3

@daanjo3 daanjo3 commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Current state

Currently a chained stage will have it's output be part of the whole output of the pipeline. However, this is not always desirable. I've run into a scenario where I'd like to run a pre-processing step (thus a chained stage), but only use that step's output as input for the next stage, not as pipeline output. This is currently not possible.

Changes

This PR adds the emitOutput option to a stage, which will be true by default. When set to false it will be prevent the output scratch file from being merged into the pipeline output.

Validity of the flag usage is checked in two locations: on a stage to check whether it's child stages end with one that is outputting, and on the pipeline to check whether it's one stage will output.

Considerations & open questions

  • The stageComplete event currently reports the quads that were generated regardless of whether they also end up in the total pipeline output. Depending on how this number is interpreted it may be confusing.

@ddeboer ddeboer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Daan. Before going into the code: could you sketch the scenario you ran into, with the two queries and what the first produces for the second? I can’t tell from the description:

  • What does the pre-processing step do? Materialise a closure, normalise URIs, build a working graph that the next query joins against?
  • Why can’t the next stage’s query do it itself, for instance with a subquery? A query too heavy for the source endpoint would be a good reason, but I’d like to know which reason applies.
  • What goes wrong if the intermediate output does end up in the pipeline output? Is it too large, does it end up in the published graph, does it fail validation downstream?

The docs’ extract → aggregate example has the same gap: it shows how, not why.

Your answer matters for the design too. emitOutput only means something inside a chain, but it sits on every Stage. That’s why it needs two checks in two constructors, and the code admits it: “a stage cannot tell whether it is top-level or a chained child.” false is invalid on a top-level stage without stages and on every last child.

If your case is “pre-process, then produce”, a single option on the parent stage would cover it and leave no invalid states:

new Stage({
  name: 'extract',
  readers: …,
  stages: [aggregate],
  chainOutput: 'last', // default: 'all'
});

'last' can’t express “discard the final stage”, so both validations go away. A per-stage flag only pays off if you need to emit something other than the tail of the chain, such as parent and last but not middle. Do you?

On your open question about stageComplete: once the choice lives on the chain, the count written under the parent’s name can be the quads that actually reach the writers.

Smaller things:

  • No test covers a chain whose emitted output is empty because everything was withheld. Worth checking what the writers and provenance do then.
  • docs/reference/pipeline.md uses straight apostrophes in “the parent stage's or the Pipeline's”; the rest of the docs use ’.

@daanjo3

daanjo3 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

What does the pre-processing step do? Materialise a closure, normalise URIs, build a working graph that the next query joins against?

I'm currently working on the loda-pipeline, which will attempt to transform data from an arbitrary source (often close to following schema-ap-nde to the EDM format (Europeana Data Model). Ideally we have one shared schema-ap-nde => EDM mapping, and if needed, we do a pre-transformation step per source. In this case that would mean mapping the source to schema-ap-nde.

Why can’t the next stage’s query do it itself, for instance with a subquery? A query too heavy for the source endpoint would be a good reason, but I’d like to know which reason applies.

With the aforementioned structure we aim to have a single transformation that maps to our desired output shape. When folding the source-specific mapping into it as a subquery we'd need to keep copies of the schema-ap-nde => edm mapping, which is what we'd like to avoid as these can drift. Splitting the queries also helps make them simpler.

What goes wrong if the intermediate output does end up in the pipeline output? Is it too large, does it end up in the published graph, does it fail validation downstream?

In the case of the loda-pipeline we have a follow-up step that filters out any resources it doesn't need. However with the current structure it does need to go over the full graph including the intermediary step output. The intermediary output causes the graph to nearly double in size, of which we know we don't need half of it. If the intermediate stages don't emit, the final stage's query alone determines the data and shape of the pipeline output, and everything downstream just works on that.

I think that last point is in line with your suggestion to use a setting like chainOutput: 'last' instead, I like that better than my current solution. I'll amend it.

@daanjo3
daanjo3 requested a review from ddeboer November 19, 2026 08:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants