feat(quickwit): add search, ingest, index, source and delete-task

AGENTS.md

@@ -2,36 +2,83 @@
 
 ## What
 
-- Provides plugin components under `io.kestra.plugin.quickwit`.
-- Includes classes such as `Example`, `Trigger`.
+- Plugin components under `io.kestra.plugin.quickwit`, wrapping the [Quickwit](https://quickwit.io) REST API.
+- Tasks and one polling trigger, grouped in five subpackages: `search`, `ingest`, `index`, `source`, `deletetask`.
+- Shared connection properties, response models and HTTP plumbing live in the root package.
 
 ## Why
 
-- What user problem does this solve? Teams need a concrete starting point for building and validating new Kestra plugins without recreating the same project scaffolding from scratch.
-- Why would a team adopt this plugin in a workflow? It gives plugin authors a ready-made reference repo they can adapt alongside their own build, test, and publishing workflow.
-- What operational/business outcome does it enable? It shortens plugin delivery time, reduces setup mistakes, and makes internal or partner plugin development more repeatable.
+- **What user problem does this solve?** Driving Quickwit from Kestra otherwise means `io.kestra.plugin.core.http.Request` plus hand-built JSON, so every flow repeats the base URL, the `api/v1` prefix, query encoding and error handling.
+- **Why would a team adopt this plugin in a workflow?** Search, ingest, index, source and delete-task operations become typed tasks with outputs downstream tasks read directly.
+- **What operational/business outcome does it enable?** Log and trace pipelines that load Quickwit and alert on it stay readable, and a doc-mapping mismatch surfaces as Quickwit's own error message rather than an opaque HTTP status.
 
 ## How
 
 ### Architecture
 
 Single-module plugin. Source packages under `io.kestra.plugin`:
 
-- `quickwit`
+- `quickwit` — `AbstractQuickwitTask` (shared connection properties and request-body helpers),
+  `QuickwitService` (endpoint building, request creation, response and error handling),
+  `models` (API response POJOs)
+- `quickwit.search` — `Search`, `Trigger`
+- `quickwit.ingest` — `Ingest`, `Commit`
+- `quickwit.index` — `Create`, `Get`, `List`, `Delete`, `Clear`, `AbstractQuickwitIndex`
+- `quickwit.source` — `Create`, `Toggle`, `Delete`, `ResetCheckpoint`, `AbstractQuickwitSource`
+- `quickwit.deletetask` — `Create`, `List`
 
-Infrastructure dependencies (Docker Compose services):
+A base class only exists when it has more than one subclass: a trigger extends `AbstractTrigger`, not
+`Task`, so `search` and `deletetask` keep their properties on the concrete components instead of
+declaring an abstract class a single task would extend.
 
-- `app`
+No plugin class lives in the root package: `lintPluginDocs` rejects it (PKG-003). Infrastructure
+dependencies (Docker Compose services):
 
-### Key Plugin Classes
+- `app` (Kestra, with the built plugin mounted)
+- `quickwit` (a single-node Quickwit cluster on port 7280, for local verification)
 
-- `io.kestra.plugin.quickwit.Example`
+### HTTP
 
-### Project Structure
+All calls go through `io.kestra.core.http.client.HttpClient`; no other HTTP client is used. Two
+consequences worth knowing before changing `QuickwitService`:
+
+- `allowFailed` is enabled so the status and raw body are inspected here. `HttpException` sanitizes
+  the body it embeds in its message, so a message parsed out of the exception would be corrupted.
+  Failures are raised as `IllegalStateException` naming the operation, the status and Quickwit's
+  `message` field.
+- `pathSegment` percent-encodes. `URLEncoder` alone emits `+` for a space, which is a literal plus
+  inside a URI path.
+
+### Key plugin classes
+
+- `io.kestra.plugin.quickwit.search.Search` — queries an index, returns hits as `FETCH`, `FETCH_ONE`, `STORE` or `NONE`
+- `io.kestra.plugin.quickwit.search.Trigger` — polling trigger, watermark in the namespace KV Store
+- `io.kestra.plugin.quickwit.ingest.Ingest` — sends documents as NDJSON, with `AUTO`/`WAIT_FOR`/`FORCE` commit
+- `io.kestra.plugin.quickwit.index.Create` — creates an index from a doc mapping
+- `io.kestra.plugin.quickwit.index.Clear` — drops splits and resets checkpoints, keeping the definition
+- `io.kestra.plugin.quickwit.source.Toggle` — pauses and resumes indexing without losing the checkpoint
+- `io.kestra.plugin.quickwit.deletetask.Create` — queues a delete-by-query task
+
+### Authentication
+
+Quickwit ships no authentication layer: its REST server registers CORS, compression and tracing only,
+and its OpenAPI document declares no security scheme. `basicAuth` and `headers` on the base task are
+therefore for deployments behind a reverse proxy or an API gateway, and both are optional.
+
+### Project structure
 
 ```
 plugin-quickwit/
 ├── src/main/java/io/kestra/plugin/quickwit/
+│   ├── AbstractQuickwitTask.java     # shared connection properties
+│   ├── QuickwitService.java          # endpoints, requests, errors
+│   ├── models/                       # API response POJOs
+│   ├── search/  ingest/  index/  source/  deletetask/
+│   └── package-info.java
+├── src/main/resources/
+│   ├── doc/io.kestra.plugin.quickwit.md
+│   ├── icons/                        # plugin-icon.svg + one per subpackage
+│   └── metadata/                     # index.yaml + one per subpackage
 ├── src/test/java/io/kestra/plugin/quickwit/
 ├── build.gradle
 └── README.md
@@ -40,8 +87,29 @@ plugin-quickwit/
 ## Local rules
 
 - Base the wording on the implemented packages and classes, not on template README text.
+- Every task and the trigger need a `@Plugin` example with `full = true`, valid YAML containing
+  `id`, `namespace` and `tasks` or `triggers`. Secrets go through `{{ secret('NAME') }}`.
+- Every property and output field needs a `@Schema`; a `@Schema(title = ...)` must not end with a period.
+- `@PluginProperty(group = ...)` accepts only: `main`, `connection`, `source`, `processing`,
+  `execution`, `destination`, `reliability`, `advanced`, `deprecated`.
+- A property may not be named `version`: Kestra reserves it to pin a plugin version. Quickwit's
+  configuration format version is therefore `configVersion`.
+- The `index` subpackage metadata file must keep its dotted name
+  (`io.kestra.plugin.quickwit.index.yaml`). A leaf-named `index.yaml` would collide with the root
+  metadata file, which is why `META-004` is disabled in `build.gradle`; the invariant it would have
+  checked is enforced by `MetadataConsistencyTest`.
+- Tests use WireMock (`org.wiremock:wiremock-jetty12`) with `@KestraTest`; no live Quickwit is needed.
+  `QuickwitContainerTest` is the one exception: a single create-ingest-search round-trip against a real
+  node via Testcontainers, aborting where Docker is unavailable. Point `QUICKWIT_IT_URL` at a running
+  node (e.g. `http://localhost:7280`) to run it without Docker.
+  Do not name a test method after a WireMock DSL method (`get`, `delete`, `list`): it hides the
+  static import and the DSL call silently resolves to the test method.
+- Run `./gradlew build` before pushing. It runs `lintPluginDocs`, the tests and JaCoCo.
 
 ## References
 
+- https://quickwit.io/docs
+- https://quickwit.io/docs/reference/rest-api
+- https://quickwit.io/docs/reference/query-language
 - https://kestra.io/docs/plugin-developer-guide
-- https://kestra.io/docs/plugin-developer-guide/contribution-guidelines
+- https://kestra.io/docs/plugin-developer-guide/contribution-guidelines

README.md

@@ -39,20 +39,41 @@
 
 ## Why
 
-- What user problem does this solve? Teams need a concrete starting point for building and validating new Kestra plugins without recreating the same project scaffolding from scratch.
-- Why would a team adopt this plugin in a workflow? It gives plugin authors a ready-made reference repo they can adapt alongside their own build, test, and publishing workflow.
-- What operational/business outcome does it enable? It shortens plugin delivery time, reduces setup mistakes, and makes internal or partner plugin development more repeatable.
+- **What user problem does this solve?** Teams that run [Quickwit](https://quickwit.io) today drive it from Kestra with `io.kestra.plugin.core.http.Request` and hand-built JSON: every flow repeats the base URL, the `api/v1` prefix, the query-string encoding, and the error handling. This plugin wraps the Quickwit REST API in typed tasks, so a flow only declares *what* it wants to search, ingest or manage.
+- **Why would a team adopt this plugin in a workflow?** Search, ingest, index, source and delete-task operations become first-class tasks with validated properties and outputs that downstream tasks consume directly (`outputs.search.rows`, `outputs.ingest.numRejectedDocs`), instead of nested Pebble expressions over raw HTTP responses.
+- **What operational/business outcome does it enable?** Log and trace pipelines that load data into Quickwit and alert on it become readable and reviewable: a doc-mapping mismatch surfaces as Quickwit's own error message, and `search.Trigger` reacts to new documents without polling from scratch on every run.
 
 ## What
 
-- Provides plugin components under `io.kestra.plugin.quickwit`.
-- Includes classes such as `Example`, `Trigger`.
+Tasks live under `io.kestra.plugin.quickwit`, grouped by subpackage:
+
+| Subpackage | Components |
+|---|---|
+| `search` | `Search`, `Trigger` |
+| `ingest` | `Ingest` |
+| `index` | `Create`, `Get`, `List`, `Delete`, `Clear` |
+| `source` | `Create`, `Toggle`, `Delete`, `ResetCheckpoint` |
+| `deletetask` | `Create`, `List` |
+
+Every task and the trigger share the same connection properties, declared flat on the component: `url` (required), plus optional `basicAuth`, `headers`, `connectTimeout` and `readTimeout`. Quickwit itself has no authentication layer, so credentials only matter when the cluster sits behind a reverse proxy or an API gateway.
+
+```yaml
+tasks:
+  - id: search
+    type: io.kestra.plugin.quickwit.search.Search
+    url: "http://localhost:7280"
+    index: app-logs
+    query: "severity:ERROR"
+    maxHits: 100
+```
+
+See [src/main/resources/doc/io.kestra.plugin.quickwit.md](src/main/resources/doc/io.kestra.plugin.quickwit.md) for the full how-to, and the [Quickwit REST API reference](https://quickwit.io/docs/reference/rest-api) for the underlying endpoints.
 
 ## Running Kestra locally with this plugin
 
 1. Build the shadow JAR: `./gradlew shadowJar`. The output lands in `build/libs/`.
-2. Run `docker compose up`. `docker-compose.yml` builds `kestra/kestra:latest` and mounts `build/libs/` to `/app/plugins/`, so Kestra picks up the jar on startup.
-3. Kestra UI is available at [localhost:8080](http://localhost:8080).
+2. Run `docker compose up`. `docker-compose.yml` builds `kestra/kestra:latest`, mounts `build/libs/` to `/app/plugins/`, and starts a Quickwit node on port 7280, so Kestra picks up the jar and the flows can be exercised against a real cluster.
+3. Kestra UI is available at [localhost:8080](http://localhost:8080); Quickwit's REST API at [localhost:7280](http://localhost:7280).
 
 ### Plugins folder gotcha
 

build.gradle

@@ -13,6 +13,15 @@ plugins {
 
 def isBuildSnapshot = version.toString().endsWith("-SNAPSHOT")
 
+pluginDocLint {
+    // META-004 cross-checks that each metadata file declares the group of the package it documents.
+    // It cannot be applied to this plugin: a subpackage is literally named `index`, so the linter
+    // resolves its metadata to `metadata/index.yaml`, which is also the very file the rule itself
+    // requires for the plugin root. Both candidate names collide and there is no third option.
+    // The invariant is still enforced, by MetadataConsistencyTest.
+    disabledRules = ['META-004']
+}
+
 repositories {
     mavenLocal()
     mavenCentral()
@@ -87,6 +96,14 @@ dependencies {
     testImplementation "org.junit.jupiter:junit-jupiter-engine"
     testImplementation "org.hamcrest:hamcrest"
     testImplementation "org.hamcrest:hamcrest-library"
+
+    // stubs the Quickwit REST API so every task and the trigger are tested without a live cluster
+    testImplementation "org.wiremock:wiremock-jetty12"
+
+    // a single live round-trip (create -> ingest -> search) against a real Quickwit node;
+    // WireMock covers the error paths, this covers API drift the stubs cannot catch
+    testImplementation "org.testcontainers:testcontainers:1.20.4"
+    testImplementation "org.testcontainers:junit-jupiter:1.20.4"
 }
 
 /**********************************************************************************************************************\

docker-compose.yml

@@ -1,6 +1,23 @@
-version: "3.8"
-
 services:
+  # A single Quickwit node with every service enabled, so the plugin can be exercised against a real
+  # cluster. The REST API answers on 7280, which is the port the task examples use.
+  # https://quickwit.io/docs/get-started/installation
+  quickwit:
+    image: quickwit/quickwit:0.8.2
+    # the image entrypoint is a bare `quickwit`, so without this the container prints its usage and exits
+    command: run
+    environment:
+      # a single node runs every service in one process; no discovery to configure
+      QW_CLUSTER_ID: quickwit-local
+      QW_NODE_ID: quickwit-node-1
+      QW_DATA_DIR: /quickwit-data
+      QW_SERVICES: indexer searcher janitor metastore ingester
+      QW_PEER_SEEDS: quickwit-node-1
+    ports:
+      - "7280:7280"
+    volumes:
+      - quickwit-data:/quickwit-data
+
   app:
     build:
       context: .
@@ -14,3 +31,6 @@ services:
     # location (/tmp)" on some hosts, see kestra-io/kestra#17405
     tmpfs:
       - /tmp
+
+volumes:
+  quickwit-data: