Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 83 additions & 4 deletions docs/guide/chain-stage-outputs.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Chain stage outputs

This guide shows how to feed one stage’s output to sub-stages that query it – for example an extraction stage whose result a second stage aggregates.
This guide shows how to feed one stage’s output to sub-stages that query it, and how to choose which stages’ output the chain writes.

## Declare sub-stages

Pass child stages in the parent’s `stages` option. Each child queries the previous stage’s output instead of the dataset’s own distribution:
Pass child stages in the parent’s `stages` option. Each child queries the previous stage’s output instead of the dataset’s own distribution. A chain is a line, not a tree: the second child queries the first child’s output, not the parent’s. To run several queries over the same output, give one child several readers:

```typescript
import { SparqlConstructReader, Stage } from '@lde/pipeline';
Expand All @@ -21,6 +21,84 @@ const parent = new Stage({
});
```

## Choose what the chain writes

The parent’s `chainOutput` option decides which stages’ output reaches the pipeline’s [writers](../reference/pipeline#writer). Pick it by what the later stages do with the earlier output.

### Enrich: write every stage

Use the default, `chainOutput: 'all'`, when each stage adds to the result. A typical case is enrichment: the parent produces your data, and a chained stage selects the terms it links to and fetches their labels from another SPARQL endpoint. The chained stage’s item selector queries the parent’s output; its reader queries the other endpoint for the selected terms:

```typescript
import { Distribution } from '@lde/dataset';
import {
SparqlConstructReader,
SparqlItemSelector,
Stage,
} from '@lde/pipeline';
import type { Reader } from '@lde/pipeline';

const terminology = Distribution.sparql(
new URL('https://terminology.example.org/sparql'),
);
const labels = await SparqlConstructReader.fromFile('term-labels.rq');

// Query the terminology endpoint instead of the parent’s output.
const termLabels: Reader = {
read: (dataset, _distribution, options) =>
labels.read(dataset, terminology, options),
};

const parent = new Stage({
name: 'map',
readers: await SparqlConstructReader.fromFile('map.rq'),
stages: [
new Stage({
name: 'enrich',
itemSelector: new SparqlItemSelector({
query:
'SELECT DISTINCT ?this WHERE { ?record <https://schema.org/about> ?this } ORDER BY ?this',
}),
readers: termLabels,
// Chained stages are tuned for the pipeline’s own store; set
// what suits the remote endpoint.
batchSize: 50,
maxConcurrency: 2,
}),
],
});
```

The writers get the mapped data and the labels together. If the mapped data links to no terms, the enrich stage selects nothing and the writers get the mapped data alone.

A chained stage runs with the batch size and concurrency meant for the store that serves its parent’s output. A stage whose reader queries a remote endpoint should set `batchSize` and `maxConcurrency` for that endpoint, as above.

### Normalise, then map: write only the last stage

Set `chainOutput: 'last'` when the earlier stages only produce input for the next one. A typical case is a shared mapping that expects its input in one shape: a first stage normalises each source to that shape, and the mapping reads the normalised output. The normalised data is scaffolding, so only the mapping’s output should reach the writers:

```typescript
const parent = new Stage({
name: 'normalise',
readers: await SparqlConstructReader.fromFile('normalise.rq'),
chainOutput: 'last', // only 'map' reaches the writers
stages: [
new Stage({
name: 'map',
readers: await SparqlConstructReader.fromFile('map.rq'),
}),
],
});
```

Without `'last'`, the writers would get the normalised data as well as the mapped data, and whatever reads the output would have to filter the normalised data back out.

The stages `'last'` leaves out still run their [validation](../reference/pipeline#validation), and its results are reported – useful for checking that the normalised data has the shape the mapping expects.

### Inspect the output between stages

Whichever you choose, every stage in the chain keeps its own scratch file at `<outputDir>/<stage name>/<dataset>.nt` after the run, including the stages `'last'` leaves out of what the chain writes. Compare one stage’s file with the next to see how a query acts on the previous one’s output. The next run of the dataset overwrites the files.

## Implement a stage output resolver

Between stages, the pipeline writes output to a scratch N-Triples file. A `StageOutputResolver` turns that file into the `Distribution` the next stage queries:
Expand Down Expand Up @@ -70,10 +148,11 @@ For each dataset:

1. The parent stage runs against the dataset’s resolved distribution and writes N-Triples to a scratch file under `<outputDir>/<stage name>/`.
2. `resolve()` turns that file into a `Distribution`; the first sub-stage runs against it, writing its own scratch file. Each further sub-stage chains off the previous one’s output the same way.
3. The scratch files of the whole chain are concatenated and written to the pipeline’s [writers](../reference/pipeline#writer) under the parent stage’s name.
3. The scratch files the parent’s `chainOutput` selects – all of them by default, or only the last stage’s – are concatenated and written to the pipeline’s [writers](../reference/pipeline#writer) under the parent stage’s name. The scratch files stay on disk.
4. `cleanup()` runs when the chain finishes, on success and on failure.

Two constraints apply:

- Chains are quad-only: a stage with `stages` cannot `project`, because a chained stage serialises to N-Triples, which a projected item cannot (see [ADR 13](../decisions/0013-project-inside-the-batch-per-root-type)).
- A chained stage that returns `NotSupported` – for example, its item selector selects nothing – fails the whole chain instead of being skipped, because a later stage cannot chain off missing output.
- A chain is one line: a chained stage cannot have `stages` of its own, and the stages in a chain need distinct names, because each writes its scratch file under its name.
- A chained stage whose item selector selects nothing produces empty output, and the chain continues. Any other `NotSupported` – every reader returning it – fails the whole chain instead of being skipped, because the stage produced no output for the next one to read.
15 changes: 13 additions & 2 deletions docs/reference/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always ret

#### Skipped stages

A stage that cannot run against a dataset is skipped, not failed: when every reader returns `NotSupported`, or the item selector yields no items, the pipeline reports `stageSkipped` and moves on. The selector is peeked before the writer’s first write, so a destructive writer – `SparqlUpdateWriter` issues `CLEAR GRAPH` on first write – does not clear anything for a stage that turns out to be empty. Inside a [chain](#chaining-stages), `NotSupported` fails the chain instead.
A stage that cannot run against a dataset is skipped, not failed: when every reader returns `NotSupported`, or the item selector yields no items, the pipeline reports `stageSkipped` and moves on. The selector is peeked before the writer’s first write, so a destructive writer – `SparqlUpdateWriter` issues `CLEAR GRAPH` on first write – does not clear anything for a stage that turns out to be empty. Inside a [chain](#chaining-stages), a stage whose selector yields no items produces empty output and the chain continues; any other `NotSupported` fails the chain.

### Chaining stages

Expand All @@ -201,7 +201,18 @@ new Pipeline({
});
```

Because the scratch files are N-Triples, chains are quad-only: a stage with `stages` cannot `project`. A chained stage that returns `NotSupported` fails the chain rather than being skipped. See the how-to guide [Chain stage outputs](../guide/chain-stage-outputs).
Because the scratch files are N-Triples, chains are quad-only: a stage with `stages` cannot `project`. A chained stage cannot have `stages` of its own, and the stages in a chain need distinct names. A chained stage whose selector yields no items produces empty output; any other `NotSupported` fails the chain rather than being skipped.

The parent stage’s `chainOutput` option decides which stages’ output reaches the writers:

| `chainOutput` | Writes | Use when |
| ----------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `'all'` (default) | every stage’s output | each stage adds to the result, such as a chained stage that enriches the parent’s output with labels from another endpoint |
| `'last'` | only the last chained stage’s | the earlier stages only produce input for the next, such as a normalisation that a shared mapping then reads |

Either way, every stage’s scratch file stays at `<outputDir>/<stage name>/<dataset>.nt` after the run, so you can inspect what each stage produced; the next run of the dataset overwrites it.

See the how-to guide [Chain stage outputs](../guide/chain-stage-outputs).

### Item Selector

Expand Down
55 changes: 40 additions & 15 deletions packages/pipeline/src/pipeline.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import { createReadStream } from 'node:fs';
import { mkdir, rm, writeFile } from 'node:fs/promises';
import { dirname } from 'node:path';
import { randomUUID } from 'node:crypto';
import { Dataset, Distribution } from '@lde/dataset';
import {
Expand Down Expand Up @@ -146,6 +148,13 @@ export interface PipelineOptions<Out = Quad> {
distributionResolver?: DistributionResolver;
chaining?: {
stageOutputResolver: StageOutputResolver;
/**
* Where every chained stage writes its output, as N-Triples at
* `<outputDir>/<stage name>/<dataset>.nt`, for the next stage to query.
* The files stay after the run – including those a chain with
* `chainOutput: 'last'` leaves out of what it writes – so each step can be
* inspected; the next run of the dataset overwrites them.
*/
outputDir: string;
};
/** Reporter(s) handed every {@link PipelineEvent} of a run, in array order. */
Expand Down Expand Up @@ -1065,8 +1074,8 @@ export class Pipeline<Out = Quad> {
}

/**
* Run a stage with reporting and return whether it was supported.
* Returns `true` if the stage produced results, `false` if NotSupported.
* Run a stage with reporting. Returns the {@link NotSupported} reason when
* the stage was skipped, and nothing when it ran.
*/
private async runStage<Item>(
runId: string,
Expand All @@ -1077,7 +1086,7 @@ export class Pipeline<Out = Quad> {
source: SourceKind,
timeout?: TimeoutPolicy,
unread?: UnreadValueTally,
): Promise<boolean> {
): Promise<NotSupported | undefined> {
this.report(runId, { type: 'stageStart', dataset, stage: stage.name });
const stageStart = Date.now();

Expand Down Expand Up @@ -1131,7 +1140,7 @@ export class Pipeline<Out = Quad> {
message: result.message,
},
});
return false;
return result;
}

this.report(runId, {
Expand All @@ -1143,11 +1152,14 @@ export class Pipeline<Out = Quad> {
durationMs: Date.now() - stageStart,
...summary,
});

return true;
return undefined;
}

/** Run a stage in chained mode, throwing if the stage is not supported. */
/**
* Run a stage in chained mode. A stage whose item selector selects nothing
* produced empty output, which the next stage can still read; any other
* NotSupported throws, because the chain cannot continue without the stage.
*/
private async runChainedStage<Item>(
runId: string,
dataset: Dataset,
Expand All @@ -1157,7 +1169,7 @@ export class Pipeline<Out = Quad> {
source: SourceKind,
timeout?: TimeoutPolicy,
): Promise<void> {
const supported = await this.runStage(
const skipped = await this.runStage(
runId,
dataset,
distribution,
Expand All @@ -1166,7 +1178,7 @@ export class Pipeline<Out = Quad> {
source,
timeout,
);
if (!supported) {
if (skipped && !(skipped instanceof NoItemsSelected)) {
throw new Error(
`Stage '${stage.name}' returned NotSupported in chained mode`,
);
Expand All @@ -1183,6 +1195,8 @@ export class Pipeline<Out = Quad> {
timeout?: TimeoutPolicy,
): Promise<void> {
const { stageOutputResolver, outputDir } = this.chaining!;
// Scratch files of every stage in the chain, in order; `chainOutput`
// decides which of them reach the writers.
const outputFiles: string[] = [];

// Chaining is a quad-only concern: every stage in a chain writes N-Triples
Expand All @@ -1200,6 +1214,10 @@ export class Pipeline<Out = Quad> {
outputDir: `${outputDir}/${chainedStage.name}`,
format: 'n-triples',
});
const outputPath = scratchWriter.getOutputPath(dataset);
// The FileWriter writes no file for empty output, so remove the previous
// run's file first: it must never stand in for this run's output.
await rm(outputPath, { force: true });
const scratchRun = await scratchWriter.openRun(context);
try {
await this.runChainedStage(
Expand All @@ -1217,7 +1235,11 @@ export class Pipeline<Out = Quad> {
await scratchRun.abort(error);
throw error;
}
return scratchWriter.getOutputPath(dataset);
// A stage that produced nothing leaves an empty file, so the next stage
// and the writers read an empty graph rather than a missing file.
await mkdir(dirname(outputPath), { recursive: true });
await writeFile(outputPath, '', { flag: 'a' });
return outputPath;
};

try {
Expand Down Expand Up @@ -1246,13 +1268,16 @@ export class Pipeline<Out = Quad> {
}
}

// 3. Concatenate all output files → the open run, applying the plugins'
// beforeStageWrite transforms once for the chain under the parent stage.
// The concatenation is quads (per the quad-only contract above), written
// to the `Out`-typed run writer – sound because chaining implies `Quad`.
// 3. Concatenate the output files `chainOutput` selects → the open run,
// applying the plugins' beforeStageWrite transforms once for the chain
// under the parent stage. The concatenation is quads (per the quad-only
// contract above), written to the `Out`-typed run writer – sound because
// chaining implies `Quad`.
const emitted =
stage.chainOutput === 'last' ? outputFiles.slice(-1) : outputFiles;
await this.stageWriter(runWriter, stage.name).write(
dataset,
this.readFiles(outputFiles) as unknown as AsyncIterable<Out>,
this.readFiles(emitted) as unknown as AsyncIterable<Out>,
);
} finally {
await stageOutputResolver.cleanup();
Expand Down
2 changes: 2 additions & 0 deletions packages/pipeline/src/pipelineEvent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,8 @@ export interface StageProgressEvent
export interface StageCompleteEvent extends DatasetEventEnvelope<'stageComplete'> {
stage: string;
itemsProcessed: number;
/** The quads this stage generated – including those of a chained stage
* that `chainOutput: 'last'` leaves out of what the chain writes. */
quadsGenerated: number;
durationMs: number;
/** Where the stage spent its time. Always set by the pipeline; optional so
Expand Down
51 changes: 51 additions & 0 deletions packages/pipeline/src/stage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,24 @@ export interface StageOptions<Out = Quad> {
sourceFor?: (dataset: Dataset, distribution: Distribution) => Distribution;
/** Child stages that chain off this stage's output. */
stages?: Stage[];
/**
* Which outputs of this stage’s chain reach the pipeline’s writers:
*
* - `'all'` – this stage’s output and that of every chained stage. Use it
* when each stage adds to the result, such as a chained stage that
* enriches this stage’s output with data from another source.
* - `'last'` – only the last chained stage’s output. Use it when the earlier
* stages produce working graphs that only feed the next stage, such as a
* normalisation that a shared mapping then reads: they still write their
* scratch files for the next stage to query, but the chain leaves them out
* of what it writes. Their validation still runs and is reported.
*
* Requires {@link StageOptions.stages}: a stage without them is no chain, and
* its output is always written.
*
* @default 'all'
*/
chainOutput?: 'all' | 'last';
/**
* Treat a supported stage that produces no quads as a hard failure (throws),
* rather than a legitimately empty result.
Expand Down Expand Up @@ -272,6 +290,9 @@ export class Stage<Out = Quad> {
/** Whether this stage reads a source of its own rather than the dataset’s
* resolved distribution. @see {@link StageOptions.sourceFor} */
readonly sourcesOwnData: boolean;
/** Which outputs of this stage’s chain reach the pipeline’s writers.
* @see {@link StageOptions.chainOutput} */
readonly chainOutput: 'all' | 'last';
/** Maximum concurrent in-flight SPARQL queries. @see {@link StageOptions.maxConcurrency} */
readonly maxConcurrency: number;
private readonly readers: NormalizedReader[];
Expand All @@ -295,6 +316,11 @@ export class Stage<Out = Quad> {
`Stage '${options.name}': 'project' cannot combine with chained 'stages' – a chained stage serializes to N-Triples, which a projected item cannot.`,
);
}
if (options.chainOutput && !options.stages?.length) {
throw new Error(
`Stage '${options.name}': 'chainOutput' requires chained 'stages' – set it on the stage that starts the chain.`,
);
}
// A child is handed its parent's output as its distribution; a `sourceFor`
// would substitute that away, silently reading elsewhere instead of
// continuing the chain. Caught here rather than at run time, where it looks
Expand All @@ -305,6 +331,30 @@ export class Stage<Out = Quad> {
`Stage '${options.name}': chained stage '${sourcedChild.name}' declares 'sourceFor' – a chained stage reads its parent's output, so it cannot source its own data.`,
);
}
// A chain is one line: the pipeline runs only the direct children, so a
// child's own `stages` would silently never run.
const nestingChild = options.stages?.find(
(child) => child.stages.length > 0,
);
if (nestingChild) {
throw new Error(
`Stage '${options.name}': chained stage '${nestingChild.name}' declares 'stages' – a chain is one line, so list its stages under '${options.name}' instead.`,
);
}
// Each stage in a chain writes its scratch file under its own name, so two
// stages with one name would overwrite each other's output.
const chainNames = [
options.name,
...(options.stages ?? []).map((child) => child.name),
];
const duplicateName = chainNames.find(
(name, index) => chainNames.indexOf(name) !== index,
);
if (duplicateName) {
throw new Error(
`Stage '${options.name}': two stages in the chain are named '${duplicateName}' – each writes its scratch file under its name, so the names must differ.`,
);
}
this.name = options.name;
this.stages = options.stages ?? [];
this.readers = normalizeReaders(options.readers);
Expand All @@ -319,6 +369,7 @@ export class Stage<Out = Quad> {
this.queueCapacity = options.queueCapacity;
this.sourceFor = options.sourceFor;
this.sourcesOwnData = options.sourceFor !== undefined;
this.chainOutput = options.chainOutput ?? 'all';
}

/** The validator for this stage, if configured. */
Expand Down
Loading