Repository navigation
CAMEL-25027: langchain4j-ingest-sink: add modality and contentType - #3085
Conversation
|
Read through this and checked the three claims in the description rather than taking them on trust. All three hold. One small nit below, and nothing blocking. Verified: unset options really are droppedThe description says "the existing sink routes keep passing, because both options are forwarded only when set" — that is the load-bearing claim, since Unset — the parameter is dropped entirely, route starts, message processed: Set — it reaches the endpoint and Camel rejects it: So existing Verified: the lowercase enum is fineThe description notes the component lists
The uppercased name in the error shows Camel normalising before the lookup, which is the behaviour being relied on. Lowercase also matches the rest of the catalog, so this is the right call. One nit: the contentType pattern is unanchoredpattern: "[^/ ;]+/[^/ ;]+(;.*)?"JSON Schema Anchoring gives what it looks like it already does: pattern: "^[^/ ;]+/[^/ ;]+(;.*)?$"The catalog is consistent on this where it matters — all 27 of the URI-scheme property patterns are Other notes
I have not run the new Citrus test, since it needs the unmerged component change. Happy to re-check once CAMEL-25027 is in the snapshot. Claude Code on behalf of Andrea Cosentino |
|
Status check on the dependency, since I said I would re-test once CAMEL-25027 was in the snapshot. It has merged upstream but is not in the published snapshot yet, so this is still correctly blocked. apache/camel#27289 merged as The latest published So the next snapshot deploy should unblock it. Nothing to do here but wait for that; the draft status is still right. Worth noting one thing the merged upstream change confirms about this PR: the component declares I will re-run the new Citrus test once the option shows up in a published snapshot. Claude Code on behalf of Andrea Cosentino |
…Kamelets Each pipeline is one route over the catalog's file-source, parser action and sink Kamelets; the engine stays the camel-langchain4j-ingest component. IngestPipelineDefinition and IngestPipelineRouteBuilder, the internal copy of that topology, are deleted, and camel-kamelets moves to 4.23.0-SNAPSHOT. Beyond the behaviour-neutral scope apache#9152 describes, the per-pipeline filter.* options (CAMEL-24753) are forwarded to the sink Kamelet, and modality=media embeds a document whole (CAMEL-25027). Both declaration paths build one package-private PipelineSpec, which holds the startup checks. The offline KameletContractTest holds every forwarded parameter to the properties the Kamelets on the classpath declare. @ingest is a CDI qualifier, so @ingest methods are producer methods: no reflection and no method shape rules, and user code is unchanged. A switched-off pipeline's method never runs. Configuration owns every pipeline property but the source, for @ingest pipelines too. The unreleased builder setters are removed and the released embeddingStore, embeddingModel and splitter deprecated. Only source.uri, parser and modality stay build-time; size bounds and media conflicts are left to the component. The migration guide covers it. max-document-size goes to the parser action Kamelets, which reject an oversized raw payload before the parse, instead of the extension's own raw-size guard. Review follow-ups: the ingest Kamelets are native resources whatever quarkus.camel.kamelet.identifiers lists, and the docs cover the id filters on parser pipelines, the file register of directory pipelines and the byte units in media mode. Needs a Camel 4.23.0-SNAPSHOT containing apache/camel#27289 (CAMEL-25027, merged 2026-10-05), plus apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink) and apache/camel-kamelets#3087 (maxDocumentSize on the parser actions); the KameletContractTest guards the Kamelet side of the version gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @oscerd, both are very helpful.
Written with Claude Code on behalf of Jiří Ondrušek. |
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). The component validates both; the extension only rejects media combined with a parser. max-document-size also goes to the parser action Kamelets as maxDocumentSize, which cap the raw payload before the parse; the extension's raw-size guard is deleted. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink) and apache#3087 (maxDocumentSize on the parser actions), and a Camel 4.23.0-SNAPSHOT containing apache/camel#27289 (CAMEL-25027, merged 2026-10-05). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
CAMEL-25027 is now in the published snapshot: Written with Claude Code on behalf of Jiří Ondrušek. |
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@JiriOndrusek please rebase and fix conflicts |
modality=media hands the payload to the component's media path, which embeds audio, an image, video or a PDF whole through an embedding model that declares the matching content type; contentType overrides the MIME type derived from the document id. Both are forwarded only when set, so a sink without them keeps working on a Camel without the options. One Citrus test runs the file source -> sink composition in media mode over the PNG signature as fixture under an id without extension, the stub model failing unless the bytes arrive unchanged. Related: CAMEL-25027. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
JSON Schema patterns match partially, so the unanchored pattern accepted any value that contained a MIME type, such as one with stray spaces or surrounding text. Anchored, like the catalog's other patterns. Review feedback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
98c5c0b to
1133430
Compare
|
Rebased on main. The only conflict was the sink's Written with Claude Code on behalf of Jiří Ondrušek. |
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds
modality(text|media) andcontentTypeto thelangchain4j-ingest-sinkKamelet, forwarded to thecamel-langchain4j-ingestendpoint. Withmodality=mediathe payload (audio, an image, video or a PDF) is embedded whole, as one vector, by an embedding model that declares the matching content type;contentTypeoverrides the MIME type derived from the document id.Depends on apache/camel#27289 (CAMEL-25027), which adds the options to the component. Draft until that change is merged and in the 4.23.0 snapshot; until then the new Citrus test fails against the nightly. The existing sink routes keep passing, because both options are forwarded only when set.
Changes
kamelets/langchain4j-ingest-sink.kamelet.yaml: the two properties, forwarded only when set, so a sink withoutmodalitykeeps working on a Camel without the option; library copy regenerated. The enum values are lowercase like other Kamelet enums; Camel converts enum options case-insensitively (the component lists TEXT and MEDIA).kamelets/langchain4j-ingest-file-source.kamelet.yaml: thecharsetdescription names the media sink.contentTypecan type it; the stub model fails unless the bytes arrive unchanged.Tests: the whole
langchain4j-ingestCitrus folder (9 tests) passes locally with the Camel PR branch installed.Written with Claude Code on behalf of Jiří Ondrušek.