Repository navigation
Conversation
ddeboer
left a comment
There was a problem hiding this comment.
Thanks, Daan. Before going into the code: could you sketch the scenario you ran into, with the two queries and what the first produces for the second? I can’t tell from the description:
- What does the pre-processing step do? Materialise a closure, normalise URIs, build a working graph that the next query joins against?
- Why can’t the next stage’s query do it itself, for instance with a subquery? A query too heavy for the source endpoint would be a good reason, but I’d like to know which reason applies.
- What goes wrong if the intermediate output does end up in the pipeline output? Is it too large, does it end up in the published graph, does it fail validation downstream?
The docs’ extract → aggregate example has the same gap: it shows how, not why.
Your answer matters for the design too. emitOutput only means something inside a chain, but it sits on every Stage. That’s why it needs two checks in two constructors, and the code admits it: “a stage cannot tell whether it is top-level or a chained child.” false is invalid on a top-level stage without stages and on every last child.
If your case is “pre-process, then produce”, a single option on the parent stage would cover it and leave no invalid states:
new Stage({
name: 'extract',
readers: …,
stages: [aggregate],
chainOutput: 'last', // default: 'all'
});'last' can’t express “discard the final stage”, so both validations go away. A per-stage flag only pays off if you need to emit something other than the tail of the chain, such as parent and last but not middle. Do you?
On your open question about stageComplete: once the choice lives on the chain, the count written under the parent’s name can be the quads that actually reach the writers.
Smaller things:
- No test covers a chain whose emitted output is empty because everything was withheld. Worth checking what the writers and provenance do then.
docs/reference/pipeline.mduses straight apostrophes in “the parent stage's or the Pipeline's”; the rest of the docs use ’.
I'm currently working on the loda-pipeline, which will attempt to transform data from an arbitrary source (often close to following schema-ap-nde to the EDM format (Europeana Data Model). Ideally we have one shared schema-ap-nde => EDM mapping, and if needed, we do a pre-transformation step per source. In this case that would mean mapping the source to schema-ap-nde.
With the aforementioned structure we aim to have a single transformation that maps to our desired output shape. When folding the source-specific mapping into it as a subquery we'd need to keep copies of the schema-ap-nde => edm mapping, which is what we'd like to avoid as these can drift. Splitting the queries also helps make them simpler.
In the case of the loda-pipeline we have a follow-up step that filters out any resources it doesn't need. However with the current structure it does need to go over the full graph including the intermediary step output. The intermediary output causes the graph to nearly double in size, of which we know we don't need half of it. If the intermediate stages don't emit, the final stage's query alone determines the data and shape of the pipeline output, and everything downstream just works on that. I think that last point is in line with your suggestion to use a setting like |
Current state
Currently a chained stage will have it's output be part of the whole output of the pipeline. However, this is not always desirable. I've run into a scenario where I'd like to run a pre-processing step (thus a chained stage), but only use that step's output as input for the next stage, not as pipeline output. This is currently not possible.
Changes
This PR adds the
emitOutputoption to a stage, which will betrueby default. When set tofalseit will be prevent the output scratch file from being merged into the pipeline output.Validity of the flag usage is checked in two locations: on a stage to check whether it's child stages end with one that is outputting, and on the pipeline to check whether it's one stage will output.
Considerations & open questions
stageCompleteevent currently reports the quads that were generated regardless of whether they also end up in the total pipeline output. Depending on how this number is interpreted it may be confusing.