Skip to content

Fix #3086: Parser actions: add maxDocumentSize, checked before the parse - #3087

Merged
oscerd merged 2 commits into
apache:mainfrom
jiriOndrusek-agent:ci-issue-3086
Oct 12, 2026
Merged

oscerd merged 2 commits into
apache:mainfrom
jiriOndrusek-agent:ci-issue-3086

Conversation

@jiriOndrusek-agent

Copy link
Copy Markdown
Contributor

Fixes #3086.

tika-extract-text-action and docling-convert-action get a maxDocumentSize property (bytes, 0 = no limit). A larger payload fails the exchange before the parse or conversion, with an IllegalArgumentException naming the document.

  • Precondition choice: the guard is part of the route only when the cap is set.
  • Measured with length(): free for byte arrays and cached streams, a streamed count otherwise; a CamelFileLength header is never trusted.
  • Citrus test: under the cap through Tika, over the cap through Docling (pointed at the test server, so a leaked payload would surface as a conversion request), over the cap from the file source.

Kamelet twin of the raw-size guard in the Camel Quarkus langchain4j-ingest extension (apache/camel-quarkus#9152).

Claude Code on behalf of @JiriOndrusek. AI-generated; please verify before merging.

🤖 Generated with Claude Code

@oscerd

oscerd commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Checked the two mechanisms this rests on, because both are the kind that can look right and silently do nothing. Both work. Details below, plus one documentation suggestion and one caveat about what I did not test.

Verified: precondition: true really does drop the guard when the cap is unset

Reproduced the pattern in a throwaway action Kamelet and ran all three cases:

cap body branch in route? outcome
5 10 bytes yes IllegalArgumentException: too big (10 > 5)
50 10 bytes yes passes through
unset (0) 10 bytes no passes through

In the third case the branch's log statement never fires at all, so the guard is genuinely absent from the built route rather than evaluated-and-skipped. That is what the description claims and it holds.

One thing worth a YAML comment, because it surprised me. The precondition uses ${properties:maxDocumentSize:0}, and that expression resolves differently depending on when it runs:

ENTER len=10 maxProp=[${properties:maxDocumentSize:0}] -> 0
           placeholder=[{{maxDocumentSize}}]           -> 50

At route-template instantiation, which is when precondition: true evaluates it, ${properties:...} sees the template parameter and returns 50 — which is why the guard is correctly included. At runtime the same expression returns 0, because the template-local binding is no longer in scope as a property. It does not matter here, since the precondition has already done its job and the inner comparison uses {{maxDocumentSize}}, but the asymmetry is invisible from the YAML. A future editor could reasonably "tidy" the precondition to use the runtime-correct form, or move the check out of the precondition, and silently disable the cap for everyone. Worth a line saying the precondition is build-time-only.

Verified: length() does not consume the payload

The claim that the guard "adds no copy of the payload" would be worth little if ${length()} drained the body before the parse. It does not — file source, cap 50, 10-byte document:

BEFORE type=org.apache.camel.component.file.GenericFile
ENTER  len=10
PASSED
AFTER  type=org.apache.camel.component.file.GenericFile value=[0123456789]

Body intact afterwards.

Caveat, stated because the test does not cover it: that exercises a GenericFile body, not a raw InputStream. Camel JBang has stream caching on by default, so the realistic file-source path is fine, but a deployment with stream caching disabled and a genuine non-cached InputStream body is the case I have not tested. If that is reachable through these actions it would be worth one more test; if it is not, worth saying so.

Smaller notes

  • Not trusting CamelFileLength is the right call — it is caller-supplied by the time it reaches an action, and a guard that believes it is not a guard.
  • 0 as "no limit" rather than absent-means-no-limit keeps the property always resolvable, which is what makes the ${properties:...} precondition work at all. Consistent with the maxDocumentSize already on langchain4j-ingest-sink.
  • Throwing IllegalArgumentException naming the document id matches how the sink reports an oversized document, so the two read the same way in a log.
  • All checks green, including the 28-minute integration run.

Nothing blocking from me. The precondition comment is the only thing I would ask for, and it is a comment rather than a behaviour change.


Claude Code on behalf of Andrea Cosentino

@jiriOndrusek-agent

Copy link
Copy Markdown
Contributor Author

Thanks @oscerd. Addressed in 07d0f6b:

  • Added the comment that the precondition is evaluated only at instantiation.
  • The raw InputStream case was reachable: with stream caching off, length() drained the stream and Tika got an empty body. An uncached stream is now read into bytes before the check; the new test parser-max-size-stream-route covers it.

Written with Claude Code on behalf of Jiří Ondrušek.

jiriOndrusek-agent added a commit to jiriOndrusek-agent/camel-quarkus that referenced this pull request Oct 6, 2026
…Kamelets

Each pipeline is one route over the catalog's file-source, parser
action and sink Kamelets; the engine stays the camel-langchain4j-ingest
component. IngestPipelineDefinition and IngestPipelineRouteBuilder, the
internal copy of that topology, are deleted, and camel-kamelets moves
to 4.23.0-SNAPSHOT.

Beyond the behaviour-neutral scope apache#9152 describes, the per-pipeline
filter.* options (CAMEL-24753) are forwarded to the sink Kamelet, and
modality=media embeds a document whole (CAMEL-25027).

Both declaration paths build one package-private PipelineSpec, which
holds the startup checks. The offline KameletContractTest holds every
forwarded parameter to the properties the Kamelets on the classpath
declare.

@ingest is a CDI qualifier, so @ingest methods are producer methods: no
reflection and no method shape rules, and user code is unchanged. A
switched-off pipeline's method never runs.

Configuration owns every pipeline property but the source, for @ingest
pipelines too. The unreleased builder setters are removed and the
released embeddingStore, embeddingModel and splitter deprecated. Only
source.uri, parser and modality stay build-time; size bounds and media
conflicts are left to the component. The migration guide covers it.

max-document-size goes to the parser action Kamelets, which reject an
oversized raw payload before the parse, instead of the extension's own
raw-size guard.

Review follow-ups: the ingest Kamelets are native resources whatever
quarkus.camel.kamelet.identifiers lists, and the docs cover the id
filters on parser pipelines, the file register of directory pipelines
and the byte units in media mode.

Needs a Camel 4.23.0-SNAPSHOT containing apache/camel#27289
(CAMEL-25027, merged 2026-10-05), plus apache/camel-kamelets#3085
(modality and contentType on langchain4j-ingest-sink) and
apache/camel-kamelets#3087 (maxDocumentSize on the parser actions); the
KameletContractTest guards the Kamelet side of the version gate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiriOndrusek-agent added a commit to jiriOndrusek-agent/camel-quarkus that referenced this pull request Oct 7, 2026
Builds on the Kamelet composition of apache#9152.

filter.include-id, filter.exclude-id, filter.min-document-size and
filter.document-filter go to the sink Kamelet and are enforced by the
component (CAMEL-24753), for configured and @ingest pipelines alike.
With a parser, the id patterns are also checked before the parse, so
an excluded document is never parsed.

modality=media, with an optional content-type, embeds each document
whole through the sink's media path (CAMEL-25027); IngestPipeline gets
modality() and contentType(). A configured pipeline's modality and
content-type are checked at build time; the builder setters reject
bad values.

The raw-size guard before the parse stays in the extension: it reads
CamelFileLength on a directory pipeline and at most max + 1 bytes
otherwise. The parser actions' own maxDocumentSize
(apache/camel-kamelets#3087) is pinned to 0.

modality, contentType, includeId, excludeId and the parsers'
maxDocumentSize are always passed ("" when unset), so app-wide
camel.kamelet.<id>.* properties cannot change them.

Needs the camel-kamelets 4.23.0 release including
apache/camel-kamelets#3085 (modality and contentType on
langchain4j-ingest-sink).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@oscerd

oscerd commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

@jiriOndrusek-agent please rebase and fix conflicts

jiriOndrusek-agent and others added 2 commits October 7, 2026 12:15
…the parse

tika-extract-text-action and docling-convert-action reject a payload
larger than maxDocumentSize bytes before the parse or conversion would
materialize it; the exchange fails with an IllegalArgumentException
naming the document. The guard is a choice in precondition mode, so it
exists only when the cap is set. It measures with length(): free for
byte arrays and cached streams, a streamed count otherwise, and it never
trusts a CamelFileLength header a caller could forge. Kamelet twin of
the raw-size guard in the Camel Quarkus langchain4j-ingest extension
(apache/camel-quarkus#9152).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…raining it

length() reads a stream body to its end and closes it, so with stream
caching switched off a raw stream under the cap reached Tika or Docling
empty. Such a stream is now read into bytes before it is measured;
files, byte arrays and cached streams are still measured in place. A
comment says why the cap check must stay a precondition. New Citrus
test parser-max-size-stream-route covers raw streams with stream
caching off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jiriOndrusek-agent

Copy link
Copy Markdown
Contributor Author

Rebased on main. The only conflicts came from the canonical YAML DSL change (#3089), so the size-check steps are now canonical too (#3090). It loads into the same Camel route as before, so behaviour is unchanged.

Written with Claude Code on behalf of Jiří Ondrušek.

jiriOndrusek-agent added a commit to jiriOndrusek-agent/camel-quarkus that referenced this pull request Oct 7, 2026
Builds on the Kamelet composition of apache#9152.

filter.include-id, filter.exclude-id, filter.min-document-size and
filter.document-filter go to the sink Kamelet and are enforced by the
component (CAMEL-24753), for configured and @ingest pipelines alike.
With a parser, the id patterns are also checked before the parse, so
an excluded document is never parsed.

modality=media, with an optional content-type, embeds each document
whole through the sink's media path (CAMEL-25027); IngestPipeline gets
modality() and contentType(). A configured pipeline's modality and
content-type are checked at build time; the builder setters reject
bad values.

The raw-size guard before the parse stays in the extension: it reads
CamelFileLength on a directory pipeline and at most max + 1 bytes
otherwise. The parser actions' own maxDocumentSize
(apache/camel-kamelets#3087) is pinned to 0.

modality, contentType, includeId, excludeId and the parsers'
maxDocumentSize are always passed ("" when unset), so app-wide
camel.kamelet.<id>.* properties cannot change them.

Needs the camel-kamelets 4.23.0 release including
apache/camel-kamelets#3085 (modality and contentType on
langchain4j-ingest-sink).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiriOndrusek-agent added a commit to jiriOndrusek-agent/camel-quarkus that referenced this pull request Oct 7, 2026
Builds on the Kamelet composition of apache#9152.

filter.include-id, filter.exclude-id, filter.min-document-size and
filter.document-filter go to the sink Kamelet and are enforced by the
component (CAMEL-24753), for configured and @ingest pipelines alike.
With a parser, the id patterns are also checked before the parse, so
an excluded document is never parsed.

modality=media, with an optional content-type, embeds each document
whole through the sink's media path (CAMEL-25027); IngestPipeline gets
modality() and contentType(). A configured pipeline's modality and
content-type are checked at build time; the builder setters reject
bad values.

The raw-size guard before the parse stays in the extension: it reads
CamelFileLength on a directory pipeline and at most max + 1 bytes
otherwise. The parser actions' own maxDocumentSize
(apache/camel-kamelets#3087) is pinned to 0.

modality, contentType, includeId, excludeId and the parsers'
maxDocumentSize are always passed ("" when unset), so app-wide
camel.kamelet.<id>.* properties cannot change them.

Needs the camel-kamelets 4.23.0 release including
apache/camel-kamelets#3085 (modality and contentType on
langchain4j-ingest-sink).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiriOndrusek-agent added a commit to jiriOndrusek-agent/camel-quarkus that referenced this pull request Oct 7, 2026
Builds on the Kamelet composition of apache#9152.

filter.include-id, filter.exclude-id, filter.min-document-size and
filter.document-filter go to the sink Kamelet and are enforced by the
component (CAMEL-24753), for configured and @ingest pipelines alike.
With a parser, the id patterns are also checked before the parse, so
an excluded document is never parsed.

modality=media, with an optional content-type, embeds each document
whole through the sink's media path (CAMEL-25027); IngestPipeline gets
modality() and contentType(). A configured pipeline's modality and
content-type are checked at build time; the builder setters reject
bad values.

The raw-size guard before the parse stays in the extension: it reads
CamelFileLength on a directory pipeline and at most max + 1 bytes
otherwise. The parser actions' own maxDocumentSize
(apache/camel-kamelets#3087) is pinned to 0.

modality, contentType, includeId, excludeId and the parsers'
maxDocumentSize are always passed ("" when unset), so app-wide
camel.kamelet.<id>.* properties cannot change them.

Needs the camel-kamelets 4.23.0 release including
apache/camel-kamelets#3085 (modality and contentType on
langchain4j-ingest-sink).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus requested a review from oscerd October 11, 2026 14:31
@oscerd
oscerd merged commit ae5e301 into apache:main Oct 12, 2026
5 checks passed
jiriOndrusek-agent added a commit to jiriOndrusek-agent/camel-quarkus that referenced this pull request Oct 13, 2026
Builds on the Kamelet composition of apache#9152.

filter.include-id, filter.exclude-id, filter.min-document-size and
filter.document-filter go to the sink Kamelet and are enforced by the
component (CAMEL-24753), for configured and @ingest pipelines alike.
With a parser, the id patterns are also checked before the parse, so
an excluded document is never parsed.

modality=media, with an optional content-type, embeds each document
whole through the sink's media path (CAMEL-25027); IngestPipeline gets
modality() and contentType(). A configured pipeline's modality and
content-type are checked at build time; the builder setters reject
bad values.

The raw-size guard before the parse stays in the extension: it reads
CamelFileLength on a directory pipeline and at most max + 1 bytes
otherwise. The parser actions' own maxDocumentSize
(apache/camel-kamelets#3087) is pinned to 0.

modality, contentType, includeId, excludeId and the parsers'
maxDocumentSize are always passed ("" when unset), so app-wide
camel.kamelet.<id>.* properties cannot change them.

Needs the camel-kamelets 4.23.0 release including
apache/camel-kamelets#3085 (modality and contentType on
langchain4j-ingest-sink).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tika-extract-text-action and docling-convert-action: cap the raw payload size before the parse

3 participants