feat(quickwit): add search, ingest, index, source and delete-task

[tanav29]

What changes are being made and why?

Driving Quickwit from Kestra today means io.kestra.plugin.core.http.Request plus hand-built JSON, so every flow repeats the same four things: the base URL, the api/v1 prefix, query-string encoding and error handling. This plugin wraps those once and exposes typed tasks, so a flow only declares what it wants to search, ingest or manage.

This PR turns the plugin-quickwit scaffold into a working plugin for the Quickwit REST API.

Components shipped (11 tasks + 1 polling trigger):

Subpackage	Components
search	Search, Trigger
ingest	Ingest (with Commit enum)
index	Create, Get, List, Delete, Clear
source	Create, Toggle, Delete, ResetCheckpoint
deletetask	Create, List

Shared plumbing

• AbstractQuickwitTask holds the connection properties once — url (required) plus optional basicAuth, headers, connectTimeout and readTimeout — so every component takes them flat, with no extra nesting.
• QuickwitService builds the endpoints, creates the requests and turns failures into IllegalStateException messages that carry Quickwit's own message field. A doc-mapping mismatch therefore reads as doc mapping does not contain field 'timestamp' instead of an opaque HTTP 400.
• models/ holds the API response POJOs, so downstream tasks consume typed outputs rather than Pebble expressions over a raw JSON body.

Notable design decisions

• Authentication is optional by design. Quickwit ships no auth layer — its REST server registers CORS, compression and tracing only, and its OpenAPI document declares no security scheme. basicAuth and headers exist for clusters behind a reverse proxy or an API gateway, and both are documented as such.
• configVersion, not version. Kestra reserves version to pin a plugin version, so Quickwit's index-config format version is named configVersion.
• search.Trigger keeps its watermark in the namespace KV Store, advancing it only after the execution is created, so a document is delivered once. The trigger polls ascending on timestampField; documents that arrive late, with an event timestamp already below the watermark, are not picked up. stateKey allows two triggers to share a watermark deliberately.
• fetchType on Search (FETCH / FETCH_ONE / STORE / NONE) so a large result set can go to internal storage as a URI instead of into the task output.
• Ingest handles the two things that bite users: Quickwit only accepts NDJSON, so the task serializes a document, a list, a kestra:// URI, a file:// path or a JSON string; and with the default commit: AUTO documents are queued but not yet searchable, which WAIT_FOR and FORCE fix. detailedResponse: true returns the parseFailures entry per rejected document.

Also in this PR

• docker-compose.yml starts a single-node Quickwit with every service enabled on port 7280 alongside Kestra, so the examples can be run against a real cluster.
• Tests: WireMock (org.wiremock:wiremock-jetty12) with @KestraTest covers every task, the trigger, the connection options and the error handling — no live cluster needed.
• META-004 is disabled in build.gradle with an explanatory comment: a subpackage is literally named index, so its metadata file collides with the root index.yaml that the same rule requires. The invariant is still enforced by MetadataConsistencyTest.
• Docs: the full how-to (src/main/resources/doc/io.kestra.plugin.quickwit.md), the README rewritten around the actual components, icons and metadata per subpackage.

closes #13

QA Screenshots

Unit tests — `./gradlew build` passing

Local Quickwit up — `curl http://localhost:7280/api/v1/cluster`

Flows runs

Trigger firing on new doc

---

Setup Instructions

• docker compose up -d — starts Kestra + single-node Quickwit on http://localhost:7280
• Create the index per src/main/resources/doc/[io.kestra.plugin.quickwit.md](file:///C:/Users/tanav/c/plugin-quickwit/src/main/resources/doc/io.kestra.plugin.quickwit.md) (or index.Create)
• No credentials needed locally. Behind a proxy/gateway only:
  • basicAuth for reverse-proxy basic auth
  • headers: with Authorization: "Bearer {{ secret('QUICKWIT_GATEWAY_TOKEN') }}" for gateway tokens
• Verify before running flows: curl http://localhost:7280/api/v1/cluster

---

Contributor Checklist ✅

• I have read and followed the plugin contribution guidelines

[jymaire]

Thanks for the work here, the structure is clean (shared base task, QuickwitService, WireMock coverage, per-subpackage metadata). I reviewed the head 0bf5138 against the Kestra contribution guidelines and the Quickwit REST docs; I did not run the build. Inline comments carry suggested fixes.

Main concern: search.Trigger watermark. It uses wall-clock now, while Quickwit's start_timestamp filters on the document timestamp field. Late documents are lost, overflow beyond maxHits is skipped (not delivered later as documented), and a boundary-second document can be delivered twice. The tests don't exercise any of these (the "no refire" test just re-stubs an empty response).

Other guideline gaps (not inline):

• Rendered properties should be prefixed with r (renderedIndex -> rIndex, etc.).
• No YAML flow sanity checks in src/test/resources/flows.
• Tests are WireMock-only; Testcontainers is preferred (the PR already ships a Quickwit service in docker-compose.yml), with .github/setup-unit.sh / docker-compose-ci.yml if so.
• No local QA screenshots in the PR description.
• Two commits are not Conventional Commits (Fix Quickwit request formats and examples, duplicate logic removed); the PR title also ends with a trailing space.
• Some outputs echo inputs (Search.Output.fetchType, Ingest.Output.commit, Trigger.Output.index/query); guidelines say not to duplicate properties in outputs.
• search.Trigger reads the watermark with a warn-and-fallback to "whole result set" on any KV error, which would re-fire on every poll during a KV outage; failing the evaluation seems safer.

Nothing here is a blocker for the overall design; the watermark and the secret/key items are the ones I would fix before merging.

[tanav29]

I am now working on other review that you have told above that are not inline.

[tanav29]

You can now check i have fixed the left issues!

[jymaire]

Follow-up review on ec65eca. Thanks for the watermark rework. Using document timestamps, failing the poll on KV errors, the flow sanity checks and outputs that no longer repeat inputs are all good changes.

This time I checked every suggestion before posting. I applied all of them locally on top of ec65eca, and ./gradlew test passes (90 tests). The new TriggerTest cases fail on the current head.

Apologies: three of my earlier suggestions were wrong, and you were right to revert them.

• @Min on a Property<Integer> breaks validation with HV000030: No validator could be found for constraint 'jakarta.validation.constraints.Min' validating type 'Property<Integer>', so every trigger evaluation and task validation fails. The bounds are now checked at render time instead.
• The : in my length-prefixed key is not a valid KV key character ([a-zA-Z0-9][a-zA-Z0-9._-]*). The corrected suggestion uses -.

I reopened those three threads. The bounds and the key collision are still open, so each one now has a working suggestion.

Blocking, all in search.Trigger:

1. Timestamps are parsed as numbers, but Quickwit returns RFC 3339 strings by default. As a result, every poll against an index with the default output_format fails, including the index from the plugin's own how-to.
2. sort_by: +timestamp sorts descending in Quickwit (+ → Desc, - → Asc), so documents beyond maxHits are still skipped.
3. A truncated page drops the rest of its last second.
4. The watermark is still written before the execution is generated.

The suggestions depend on each other (imports, the timestamp(...) helper, QuickwitService.requireAtLeast), so please apply them together in one batch.

Still open from the first review: Testcontainers coverage, local QA screenshots in the description, and the trailing space in the PR title.

[tanav29]

@jymaire Is there other things i am missing in this pr?

[jymaire]

Follow-up review on 16f4113. Thanks for applying the batch, everything from my last review is in: RFC 3339 parsing, ascending sort with -, the truncated-page hold-back, writing the watermark after the execution is generated, the length-prefixed key and the render-time bounds. I ran ./gradlew test on this head and all 90 tests pass.

I found two new issues, both in the how-to page (io.kestra.plugin.quickwit.md). Each inline comment has a suggestion.

Still open from earlier reviews (unchanged, not repeated inline):

• Testcontainers coverage. Tests are still WireMock-only.
• Local QA screenshots in the PR description.
• The Search.sortBy description should explain Quickwit's prefix semantics (- ascending, + descending), and the timestampField description still says "in seconds". It should mention the supported output_format values (rfc3339 or unix_timestamp_secs).

The PR description also still says late documents are picked up later. Please update it in the same pass as the first inline comment.

[tanav29]

@jymaire I have added the Testcontainer coverage, QuickwitContainerTest.java - one live test: create index → ingest 2 docs (FORCE) → poll search until both hit → delete index. Manual container lifecycle: starts its own Quickwit, aborts gracefully without Docker, and QUICKWIT_IT_URL=http://localhost:7280 runs it against an existing node.