Repository navigation
Fix #3086: Parser actions: add maxDocumentSize, checked before the parse - #3087
Conversation
|
Checked the two mechanisms this rests on, because both are the kind that can look right and silently do nothing. Both work. Details below, plus one documentation suggestion and one caveat about what I did not test. Verified:
|
| cap | body | branch in route? | outcome |
|---|---|---|---|
| 5 | 10 bytes | yes | IllegalArgumentException: too big (10 > 5) |
| 50 | 10 bytes | yes | passes through |
| unset (0) | 10 bytes | no | passes through |
In the third case the branch's log statement never fires at all, so the guard is genuinely absent from the built route rather than evaluated-and-skipped. That is what the description claims and it holds.
One thing worth a YAML comment, because it surprised me. The precondition uses ${properties:maxDocumentSize:0}, and that expression resolves differently depending on when it runs:
ENTER len=10 maxProp=[${properties:maxDocumentSize:0}] -> 0
placeholder=[{{maxDocumentSize}}] -> 50
At route-template instantiation, which is when precondition: true evaluates it, ${properties:...} sees the template parameter and returns 50 — which is why the guard is correctly included. At runtime the same expression returns 0, because the template-local binding is no longer in scope as a property. It does not matter here, since the precondition has already done its job and the inner comparison uses {{maxDocumentSize}}, but the asymmetry is invisible from the YAML. A future editor could reasonably "tidy" the precondition to use the runtime-correct form, or move the check out of the precondition, and silently disable the cap for everyone. Worth a line saying the precondition is build-time-only.
Verified: length() does not consume the payload
The claim that the guard "adds no copy of the payload" would be worth little if ${length()} drained the body before the parse. It does not — file source, cap 50, 10-byte document:
BEFORE type=org.apache.camel.component.file.GenericFile
ENTER len=10
PASSED
AFTER type=org.apache.camel.component.file.GenericFile value=[0123456789]
Body intact afterwards.
Caveat, stated because the test does not cover it: that exercises a GenericFile body, not a raw InputStream. Camel JBang has stream caching on by default, so the realistic file-source path is fine, but a deployment with stream caching disabled and a genuine non-cached InputStream body is the case I have not tested. If that is reachable through these actions it would be worth one more test; if it is not, worth saying so.
Smaller notes
- Not trusting
CamelFileLengthis the right call — it is caller-supplied by the time it reaches an action, and a guard that believes it is not a guard. 0as "no limit" rather than absent-means-no-limit keeps the property always resolvable, which is what makes the${properties:...}precondition work at all. Consistent with themaxDocumentSizealready onlangchain4j-ingest-sink.- Throwing
IllegalArgumentExceptionnaming the document id matches how the sink reports an oversized document, so the two read the same way in a log. - All checks green, including the 28-minute integration run.
Nothing blocking from me. The precondition comment is the only thing I would ask for, and it is a comment rather than a behaviour change.
Claude Code on behalf of Andrea Cosentino
|
Thanks @oscerd. Addressed in 07d0f6b:
Written with Claude Code on behalf of Jiří Ondrušek. |
…Kamelets Each pipeline is one route over the catalog's file-source, parser action and sink Kamelets; the engine stays the camel-langchain4j-ingest component. IngestPipelineDefinition and IngestPipelineRouteBuilder, the internal copy of that topology, are deleted, and camel-kamelets moves to 4.23.0-SNAPSHOT. Beyond the behaviour-neutral scope apache#9152 describes, the per-pipeline filter.* options (CAMEL-24753) are forwarded to the sink Kamelet, and modality=media embeds a document whole (CAMEL-25027). Both declaration paths build one package-private PipelineSpec, which holds the startup checks. The offline KameletContractTest holds every forwarded parameter to the properties the Kamelets on the classpath declare. @ingest is a CDI qualifier, so @ingest methods are producer methods: no reflection and no method shape rules, and user code is unchanged. A switched-off pipeline's method never runs. Configuration owns every pipeline property but the source, for @ingest pipelines too. The unreleased builder setters are removed and the released embeddingStore, embeddingModel and splitter deprecated. Only source.uri, parser and modality stay build-time; size bounds and media conflicts are left to the component. The migration guide covers it. max-document-size goes to the parser action Kamelets, which reject an oversized raw payload before the parse, instead of the extension's own raw-size guard. Review follow-ups: the ingest Kamelets are native resources whatever quarkus.camel.kamelet.identifiers lists, and the docs cover the id filters on parser pipelines, the file register of directory pipelines and the byte units in media mode. Needs a Camel 4.23.0-SNAPSHOT containing apache/camel#27289 (CAMEL-25027, merged 2026-10-05), plus apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink) and apache/camel-kamelets#3087 (maxDocumentSize on the parser actions); the KameletContractTest guards the Kamelet side of the version gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@jiriOndrusek-agent please rebase and fix conflicts |
…the parse tika-extract-text-action and docling-convert-action reject a payload larger than maxDocumentSize bytes before the parse or conversion would materialize it; the exchange fails with an IllegalArgumentException naming the document. The guard is a choice in precondition mode, so it exists only when the cap is set. It measures with length(): free for byte arrays and cached streams, a streamed count otherwise, and it never trusts a CamelFileLength header a caller could forge. Kamelet twin of the raw-size guard in the Camel Quarkus langchain4j-ingest extension (apache/camel-quarkus#9152). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…raining it length() reads a stream body to its end and closes it, so with stream caching switched off a raw stream under the cap reached Tika or Docling empty. Such a stream is now read into bytes before it is measured; files, byte arrays and cached streams are still measured in place. A comment says why the cap check must stay a precondition. New Citrus test parser-max-size-stream-route covers raw streams with stream caching off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
07d0f6b to
0098a14
Compare
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #3086.
tika-extract-text-actionanddocling-convert-actionget amaxDocumentSizeproperty (bytes, 0 = no limit). A larger payload fails the exchange before the parse or conversion, with anIllegalArgumentExceptionnaming the document.choice: the guard is part of the route only when the cap is set.length(): free for byte arrays and cached streams, a streamed count otherwise; aCamelFileLengthheader is never trusted.Kamelet twin of the raw-size guard in the Camel Quarkus langchain4j-ingest extension (apache/camel-quarkus#9152).
Claude Code on behalf of @JiriOndrusek. AI-generated; please verify before merging.
🤖 Generated with Claude Code