diff --git a/AGENTS.md b/AGENTS.md index 55a735e..dfc8424 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,14 +2,15 @@ ## What -- Provides plugin components under `io.kestra.plugin.quickwit`. -- Includes classes such as `Example`, `Trigger`. +- Plugin components under `io.kestra.plugin.quickwit`, wrapping the [Quickwit](https://quickwit.io) REST API. +- Tasks and one polling trigger, grouped in five subpackages: `search`, `ingest`, `index`, `source`, `deletetask`. +- Shared connection properties, response models and HTTP plumbing live in the root package. ## Why -- What user problem does this solve? Teams need a concrete starting point for building and validating new Kestra plugins without recreating the same project scaffolding from scratch. -- Why would a team adopt this plugin in a workflow? It gives plugin authors a ready-made reference repo they can adapt alongside their own build, test, and publishing workflow. -- What operational/business outcome does it enable? It shortens plugin delivery time, reduces setup mistakes, and makes internal or partner plugin development more repeatable. +- **What user problem does this solve?** Driving Quickwit from Kestra otherwise means `io.kestra.plugin.core.http.Request` plus hand-built JSON, so every flow repeats the base URL, the `api/v1` prefix, query encoding and error handling. +- **Why would a team adopt this plugin in a workflow?** Search, ingest, index, source and delete-task operations become typed tasks with outputs downstream tasks read directly. +- **What operational/business outcome does it enable?** Log and trace pipelines that load Quickwit and alert on it stay readable, and a doc-mapping mismatch surfaces as Quickwit's own error message rather than an opaque HTTP status. ## How @@ -17,21 +18,67 @@ Single-module plugin. Source packages under `io.kestra.plugin`: -- `quickwit` +- `quickwit` — `AbstractQuickwitTask` (shared connection properties and request-body helpers), + `QuickwitService` (endpoint building, request creation, response and error handling), + `models` (API response POJOs) +- `quickwit.search` — `Search`, `Trigger` +- `quickwit.ingest` — `Ingest`, `Commit` +- `quickwit.index` — `Create`, `Get`, `List`, `Delete`, `Clear`, `AbstractQuickwitIndex` +- `quickwit.source` — `Create`, `Toggle`, `Delete`, `ResetCheckpoint`, `AbstractQuickwitSource` +- `quickwit.deletetask` — `Create`, `List` -Infrastructure dependencies (Docker Compose services): +A base class only exists when it has more than one subclass: a trigger extends `AbstractTrigger`, not +`Task`, so `search` and `deletetask` keep their properties on the concrete components instead of +declaring an abstract class a single task would extend. -- `app` +No plugin class lives in the root package: `lintPluginDocs` rejects it (PKG-003). Infrastructure +dependencies (Docker Compose services): -### Key Plugin Classes +- `app` (Kestra, with the built plugin mounted) +- `quickwit` (a single-node Quickwit cluster on port 7280, for local verification) -- `io.kestra.plugin.quickwit.Example` +### HTTP -### Project Structure +All calls go through `io.kestra.core.http.client.HttpClient`; no other HTTP client is used. Two +consequences worth knowing before changing `QuickwitService`: + +- `allowFailed` is enabled so the status and raw body are inspected here. `HttpException` sanitizes + the body it embeds in its message, so a message parsed out of the exception would be corrupted. + Failures are raised as `IllegalStateException` naming the operation, the status and Quickwit's + `message` field. +- `pathSegment` percent-encodes. `URLEncoder` alone emits `+` for a space, which is a literal plus + inside a URI path. + +### Key plugin classes + +- `io.kestra.plugin.quickwit.search.Search` — queries an index, returns hits as `FETCH`, `FETCH_ONE`, `STORE` or `NONE` +- `io.kestra.plugin.quickwit.search.Trigger` — polling trigger, watermark in the namespace KV Store +- `io.kestra.plugin.quickwit.ingest.Ingest` — sends documents as NDJSON, with `AUTO`/`WAIT_FOR`/`FORCE` commit +- `io.kestra.plugin.quickwit.index.Create` — creates an index from a doc mapping +- `io.kestra.plugin.quickwit.index.Clear` — drops splits and resets checkpoints, keeping the definition +- `io.kestra.plugin.quickwit.source.Toggle` — pauses and resumes indexing without losing the checkpoint +- `io.kestra.plugin.quickwit.deletetask.Create` — queues a delete-by-query task + +### Authentication + +Quickwit ships no authentication layer: its REST server registers CORS, compression and tracing only, +and its OpenAPI document declares no security scheme. `basicAuth` and `headers` on the base task are +therefore for deployments behind a reverse proxy or an API gateway, and both are optional. + +### Project structure ``` plugin-quickwit/ ├── src/main/java/io/kestra/plugin/quickwit/ +│ ├── AbstractQuickwitTask.java # shared connection properties +│ ├── QuickwitService.java # endpoints, requests, errors +│ ├── models/ # API response POJOs +│ ├── search/ ingest/ index/ source/ deletetask/ +│ └── package-info.java +├── src/main/resources/ +│ ├── doc/io.kestra.plugin.quickwit.md +│ ├── icons/ # plugin-icon.svg + one per subpackage +│ └── metadata/ # index.yaml + one per subpackage ├── src/test/java/io/kestra/plugin/quickwit/ ├── build.gradle └── README.md @@ -40,8 +87,29 @@ plugin-quickwit/ ## Local rules - Base the wording on the implemented packages and classes, not on template README text. +- Every task and the trigger need a `@Plugin` example with `full = true`, valid YAML containing + `id`, `namespace` and `tasks` or `triggers`. Secrets go through `{{ secret('NAME') }}`. +- Every property and output field needs a `@Schema`; a `@Schema(title = ...)` must not end with a period. +- `@PluginProperty(group = ...)` accepts only: `main`, `connection`, `source`, `processing`, + `execution`, `destination`, `reliability`, `advanced`, `deprecated`. +- A property may not be named `version`: Kestra reserves it to pin a plugin version. Quickwit's + configuration format version is therefore `configVersion`. +- The `index` subpackage metadata file must keep its dotted name + (`io.kestra.plugin.quickwit.index.yaml`). A leaf-named `index.yaml` would collide with the root + metadata file, which is why `META-004` is disabled in `build.gradle`; the invariant it would have + checked is enforced by `MetadataConsistencyTest`. +- Tests use WireMock (`org.wiremock:wiremock-jetty12`) with `@KestraTest`; no live Quickwit is needed. + `QuickwitContainerTest` is the one exception: a single create-ingest-search round-trip against a real + node via Testcontainers, aborting where Docker is unavailable. Point `QUICKWIT_IT_URL` at a running + node (e.g. `http://localhost:7280`) to run it without Docker. + Do not name a test method after a WireMock DSL method (`get`, `delete`, `list`): it hides the + static import and the DSL call silently resolves to the test method. +- Run `./gradlew build` before pushing. It runs `lintPluginDocs`, the tests and JaCoCo. ## References +- https://quickwit.io/docs +- https://quickwit.io/docs/reference/rest-api +- https://quickwit.io/docs/reference/query-language - https://kestra.io/docs/plugin-developer-guide -- https://kestra.io/docs/plugin-developer-guide/contribution-guidelines +- https://kestra.io/docs/plugin-developer-guide/contribution-guidelines \ No newline at end of file diff --git a/README.md b/README.md index 700407a..96a7fdf 100644 --- a/README.md +++ b/README.md @@ -39,20 +39,41 @@ ## Why -- What user problem does this solve? Teams need a concrete starting point for building and validating new Kestra plugins without recreating the same project scaffolding from scratch. -- Why would a team adopt this plugin in a workflow? It gives plugin authors a ready-made reference repo they can adapt alongside their own build, test, and publishing workflow. -- What operational/business outcome does it enable? It shortens plugin delivery time, reduces setup mistakes, and makes internal or partner plugin development more repeatable. +- **What user problem does this solve?** Teams that run [Quickwit](https://quickwit.io) today drive it from Kestra with `io.kestra.plugin.core.http.Request` and hand-built JSON: every flow repeats the base URL, the `api/v1` prefix, the query-string encoding, and the error handling. This plugin wraps the Quickwit REST API in typed tasks, so a flow only declares *what* it wants to search, ingest or manage. +- **Why would a team adopt this plugin in a workflow?** Search, ingest, index, source and delete-task operations become first-class tasks with validated properties and outputs that downstream tasks consume directly (`outputs.search.rows`, `outputs.ingest.numRejectedDocs`), instead of nested Pebble expressions over raw HTTP responses. +- **What operational/business outcome does it enable?** Log and trace pipelines that load data into Quickwit and alert on it become readable and reviewable: a doc-mapping mismatch surfaces as Quickwit's own error message, and `search.Trigger` reacts to new documents without polling from scratch on every run. ## What -- Provides plugin components under `io.kestra.plugin.quickwit`. -- Includes classes such as `Example`, `Trigger`. +Tasks live under `io.kestra.plugin.quickwit`, grouped by subpackage: + +| Subpackage | Components | +|---|---| +| `search` | `Search`, `Trigger` | +| `ingest` | `Ingest` | +| `index` | `Create`, `Get`, `List`, `Delete`, `Clear` | +| `source` | `Create`, `Toggle`, `Delete`, `ResetCheckpoint` | +| `deletetask` | `Create`, `List` | + +Every task and the trigger share the same connection properties, declared flat on the component: `url` (required), plus optional `basicAuth`, `headers`, `connectTimeout` and `readTimeout`. Quickwit itself has no authentication layer, so credentials only matter when the cluster sits behind a reverse proxy or an API gateway. + +```yaml +tasks: + - id: search + type: io.kestra.plugin.quickwit.search.Search + url: "http://localhost:7280" + index: app-logs + query: "severity:ERROR" + maxHits: 100 +``` + +See [src/main/resources/doc/io.kestra.plugin.quickwit.md](src/main/resources/doc/io.kestra.plugin.quickwit.md) for the full how-to, and the [Quickwit REST API reference](https://quickwit.io/docs/reference/rest-api) for the underlying endpoints. ## Running Kestra locally with this plugin 1. Build the shadow JAR: `./gradlew shadowJar`. The output lands in `build/libs/`. -2. Run `docker compose up`. `docker-compose.yml` builds `kestra/kestra:latest` and mounts `build/libs/` to `/app/plugins/`, so Kestra picks up the jar on startup. -3. Kestra UI is available at [localhost:8080](http://localhost:8080). +2. Run `docker compose up`. `docker-compose.yml` builds `kestra/kestra:latest`, mounts `build/libs/` to `/app/plugins/`, and starts a Quickwit node on port 7280, so Kestra picks up the jar and the flows can be exercised against a real cluster. +3. Kestra UI is available at [localhost:8080](http://localhost:8080); Quickwit's REST API at [localhost:7280](http://localhost:7280). ### Plugins folder gotcha diff --git a/build.gradle b/build.gradle index 1dd1932..d2e8456 100644 --- a/build.gradle +++ b/build.gradle @@ -13,6 +13,15 @@ plugins { def isBuildSnapshot = version.toString().endsWith("-SNAPSHOT") +pluginDocLint { + // META-004 cross-checks that each metadata file declares the group of the package it documents. + // It cannot be applied to this plugin: a subpackage is literally named `index`, so the linter + // resolves its metadata to `metadata/index.yaml`, which is also the very file the rule itself + // requires for the plugin root. Both candidate names collide and there is no third option. + // The invariant is still enforced, by MetadataConsistencyTest. + disabledRules = ['META-004'] +} + repositories { mavenLocal() mavenCentral() @@ -87,6 +96,14 @@ dependencies { testImplementation "org.junit.jupiter:junit-jupiter-engine" testImplementation "org.hamcrest:hamcrest" testImplementation "org.hamcrest:hamcrest-library" + + // stubs the Quickwit REST API so every task and the trigger are tested without a live cluster + testImplementation "org.wiremock:wiremock-jetty12" + + // a single live round-trip (create -> ingest -> search) against a real Quickwit node; + // WireMock covers the error paths, this covers API drift the stubs cannot catch + testImplementation "org.testcontainers:testcontainers:1.20.4" + testImplementation "org.testcontainers:junit-jupiter:1.20.4" } /**********************************************************************************************************************\ diff --git a/docker-compose.yml b/docker-compose.yml index 12fbe2f..66a0604 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,6 +1,23 @@ -version: "3.8" - services: + # A single Quickwit node with every service enabled, so the plugin can be exercised against a real + # cluster. The REST API answers on 7280, which is the port the task examples use. + # https://quickwit.io/docs/get-started/installation + quickwit: + image: quickwit/quickwit:0.8.2 + # the image entrypoint is a bare `quickwit`, so without this the container prints its usage and exits + command: run + environment: + # a single node runs every service in one process; no discovery to configure + QW_CLUSTER_ID: quickwit-local + QW_NODE_ID: quickwit-node-1 + QW_DATA_DIR: /quickwit-data + QW_SERVICES: indexer searcher janitor metastore ingester + QW_PEER_SEEDS: quickwit-node-1 + ports: + - "7280:7280" + volumes: + - quickwit-data:/quickwit-data + app: build: context: . @@ -14,3 +31,6 @@ services: # location (/tmp)" on some hosts, see kestra-io/kestra#17405 tmpfs: - /tmp + +volumes: + quickwit-data: \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/AbstractQuickwitTask.java b/src/main/java/io/kestra/plugin/quickwit/AbstractQuickwitTask.java new file mode 100644 index 0000000..f1d0d74 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/AbstractQuickwitTask.java @@ -0,0 +1,204 @@ +package io.kestra.plugin.quickwit; + +import java.time.Duration; +import java.util.List; +import java.util.Map; + +import io.kestra.core.exceptions.IllegalVariableEvaluationException; +import io.kestra.core.http.HttpRequest; +import io.kestra.core.http.client.HttpClient; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.Task; +import io.kestra.core.runners.RunContext; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Connection settings shared by every Quickwit task. + * + *

The properties are declared flat, on the task itself, so a flow reads as + * {@code url: ... / index: ...} without an extra nesting level. Everything here is optional except + * {@link #url}. + * + *

Quickwit has no authentication layer of its own, so {@link #basicAuth} and {@link #headers} only + * matter when the cluster is exposed through a reverse proxy or an API gateway. + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +public abstract class AbstractQuickwitTask extends Task { + @Schema( + title = "Quickwit URL", + description = """ + Base URL of the Quickwit REST API, including the scheme. The `api/v1` prefix is added automatically. + A local node listens on port 7280, e.g. `http://localhost:7280`. + """ + ) + @NotNull + @PluginProperty(group = "connection") + private Property url; + + @Schema( + title = "Basic authentication", + description = """ + Optional HTTP basic authentication credentials. + + Quickwit itself has no authentication layer, so this is only needed when the cluster sits + behind a reverse proxy or an API gateway that enforces basic auth. + """ + ) + @Valid + @ToString.Exclude + @PluginProperty(group = "connection") + private BasicAuth basicAuth; + + @Schema( + title = "Custom HTTP headers", + description = """ + Headers sent on every request, for example an `Authorization: Bearer ...` token or a gateway API key. + + Quickwit itself has no authentication layer, so this is only needed when the cluster sits + behind an API gateway that injects credentials. + """ + ) + @ToString.Exclude + @EqualsAndHashCode.Exclude + @PluginProperty(group = "connection", secret = true) + private Property> headers; + + @Schema( + title = "Connect timeout", + description = """ + Maximum time to wait to establish a connection to Quickwit, e.g. `PT10S`. + When unset, the default of the underlying HTTP client applies. + """ + ) + @PluginProperty(group = "execution") + private Property connectTimeout; + + @Schema( + title = "Read timeout", + description = """ + Maximum time to keep waiting for data on an idle connection, e.g. `PT5M`. + A `commit=force` ingest or a large search may legitimately take a while, so raise it when needed. + """ + ) + @PluginProperty(group = "execution") + private Property readTimeout; + + /** Creates the HTTP client configured from the connection properties of this task. */ + protected HttpClient client(RunContext runContext) throws IllegalVariableEvaluationException { + return new HttpClient(runContext, QuickwitService.httpConfiguration(runContext, this.connectTimeout, this.readTimeout, this.basicAuth)); + } + + protected String renderedUrl(RunContext runContext) throws IllegalVariableEvaluationException { + return QuickwitService.renderedUrl(runContext, this.url); + } + + protected Map renderedHeaders(RunContext runContext) throws IllegalVariableEvaluationException { + return QuickwitService.renderedHeaders(runContext, this.headers); + } + + /** + * Builds a request against a Quickwit endpoint, with the task headers applied. + * + * @param path API path without the {@code api/v1} prefix, for example {@code indexes/hdfs-logs} + */ + protected HttpRequest.HttpRequestBuilder request(RunContext runContext, String method, String path, Map query, HttpRequest.RequestBody body) + throws IllegalVariableEvaluationException { + return QuickwitService.request(method, QuickwitService.endpoint(renderedUrl(runContext), path, query), renderedHeaders(runContext), body); + } + + protected HttpRequest.HttpRequestBuilder request(RunContext runContext, String method, String path, HttpRequest.RequestBody body) + throws IllegalVariableEvaluationException { + return request(runContext, method, path, Map.of(), body); + } + + protected T execute(HttpClient client, HttpRequest request, Class type, String operation) throws Exception { + return QuickwitService.execute(client, request, type, operation); + } + + protected List executeList(HttpClient client, HttpRequest request, Class type, String operation) throws Exception { + return QuickwitService.executeList(client, request, type, operation); + } + + protected void executeIgnoringBody(HttpClient client, HttpRequest request, String operation) throws Exception { + QuickwitService.executeIgnoringBody(client, request, operation); + } + + protected static String pathSegment(String value) { + return QuickwitService.pathSegment(value); + } + + /** Renders a required string property, failing with an actionable message when it is missing. */ + protected static String required(RunContext runContext, Property property, String field) throws IllegalVariableEvaluationException { + return QuickwitService.requireNonBlank(runContext.render(property).as(String.class).orElse(null), field); + } + + /** + * Renders a required object property, failing with an actionable message when it is missing or empty. + * + * @param documentation link to the Quickwit documentation describing the expected shape + */ + protected static Map requiredMap( + RunContext runContext, + Property> property, + String field, + String documentation + ) throws IllegalVariableEvaluationException { + Map rendered = runContext.render(property).asMap(String.class, Object.class); + + if (rendered.isEmpty()) { + throw new IllegalArgumentException("`" + field + "` is required and cannot be empty, see " + documentation); + } + + return rendered; + } + + /** + * Adds an optional value to a request body, omitting it when unset so Quickwit applies its own + * default rather than rejecting an empty or null value. + */ + protected static void putIfPresent(Map body, String key, Object value) { + if (value != null && !(value instanceof Map map && map.isEmpty())) { + body.put(key, value); + } + } + + /** + * Optional basic authentication, for clusters exposed through a reverse proxy or a gateway. + */ + @SuperBuilder + @ToString + @EqualsAndHashCode + @Getter + @NoArgsConstructor + public static class BasicAuth { + @Schema( + title = "Username", + description = "Username for HTTP basic authentication." + ) + @ToString.Exclude + @PluginProperty(group = "connection") + private Property username; + + @Schema( + title = "Password", + description = "Password for HTTP basic authentication." + ) + @ToString.Exclude + @EqualsAndHashCode.Exclude + @PluginProperty(group = "connection", secret = true) + private Property password; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/QuickwitService.java b/src/main/java/io/kestra/plugin/quickwit/QuickwitService.java new file mode 100644 index 0000000..ab41980 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/QuickwitService.java @@ -0,0 +1,336 @@ +package io.kestra.plugin.quickwit; + +import java.net.URI; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +import org.apache.commons.lang3.StringUtils; + +import com.fasterxml.jackson.core.JsonProcessingException; + +import io.kestra.core.exceptions.IllegalVariableEvaluationException; +import io.kestra.core.http.HttpRequest; +import io.kestra.core.http.HttpResponse; +import io.kestra.core.http.client.HttpClient; +import io.kestra.core.http.client.configurations.BasicAuthConfiguration; +import io.kestra.core.http.client.configurations.HttpConfiguration; +import io.kestra.core.http.client.configurations.TimeoutConfiguration; +import io.kestra.core.models.property.Property; +import io.kestra.core.runners.RunContext; +import io.kestra.core.serializers.JacksonMapper; +import io.kestra.plugin.quickwit.AbstractQuickwitTask.BasicAuth; +import io.kestra.plugin.quickwit.models.SearchQuery; +import io.kestra.plugin.quickwit.models.SearchResult; + +/** + * Thin helper around the Quickwit REST API, shared by every task and trigger of the plugin. + * + *

Quickwit exposes a plain REST API under {@code /api/v1}, has no Java client library and, as of + * its 0.9 release, ships no authentication of its own: the REST server layers CORS, compression and + * tracing only, and its OpenAPI document declares no security scheme. Requests therefore go through + * Kestra's {@link HttpClient}, with optional basic auth or custom headers for deployments sitting + * behind a reverse proxy or an API gateway. + * + *

Everything is static: a task extends {@link io.kestra.core.models.tasks.Task} and a trigger + * extends {@link io.kestra.core.models.triggers.AbstractTrigger}, so neither can inherit behaviour + * from a common superclass. + */ +public final class QuickwitService { + /** + * Prefix shared by every Quickwit endpoint. + * + * @see API version + */ + public static final String API_PREFIX = "api/v1"; + + /** Default TCP port of a Quickwit node. */ + public static final int DEFAULT_PORT = 7280; + + private QuickwitService() { + } + + /** + * Builds the HTTP client configuration from the connection properties of a task or trigger. + * + *

{@code allowFailed} is enabled on purpose: the status and the raw body are inspected by + * {@link #execute} so that Quickwit's own error message reaches the user verbatim. The client would + * otherwise render it into an exception message that {@code HttpException} sanitizes. + * + * @param basicAuth credentials for deployments fronted by a proxy, {@code null} for a bare Quickwit node + */ + public static HttpConfiguration httpConfiguration( + RunContext runContext, + Property connectTimeout, + Property readTimeout, + BasicAuth basicAuth + ) throws IllegalVariableEvaluationException { + var timeout = TimeoutConfiguration.builder(); + + if (readTimeout != null) { + timeout.readIdleTimeout(readTimeout); + } + + // Only set connectTimeout when the user provided one: a zero duration is an immediate + // timeout rather than "unbounded", so defaulting it here would break every request. + if (connectTimeout != null) { + timeout.connectTimeout(connectTimeout); + } + + var builder = HttpConfiguration.builder() + .timeout(timeout.build()) + .allowFailed(Property.ofValue(true)); + + if (basicAuth != null) { + builder.auth( + BasicAuthConfiguration.builder() + .username(basicAuth.getUsername()) + .password(basicAuth.getPassword()) + .build() + ); + } + + return builder.build(); + } + + /** + * Renders the configured URL, failing early with an actionable message rather than letting the + * HTTP client choke on a malformed URI. + */ + public static String renderedUrl(RunContext runContext, Property url) throws IllegalVariableEvaluationException { + String rendered = runContext.render(url).as(String.class).orElse(null); + + if (StringUtils.isBlank(rendered)) { + throw new IllegalArgumentException("`url` is required and cannot be blank, e.g. `http://localhost:" + DEFAULT_PORT + "`"); + } + + String trimmed = rendered.trim(); + + URI uri; + try { + uri = URI.create(trimmed); + } catch (IllegalArgumentException e) { + throw invalidUrl(trimmed, e); + } + + if (uri.getScheme() == null || uri.getHost() == null) { + throw invalidUrl(trimmed, null); + } + + return StringUtils.removeEnd(trimmed, "/"); + } + + private static IllegalArgumentException invalidUrl(String rendered, Exception cause) { + String message = "Invalid `url` `" + rendered + "`, expected a URI with a scheme and a host, e.g. `http://localhost:" + DEFAULT_PORT + "`"; + + return cause == null ? new IllegalArgumentException(message) : new IllegalArgumentException(message, cause); + } + + /** Custom headers configured on the task or trigger, already rendered. */ + public static Map renderedHeaders(RunContext runContext, Property> headers) throws IllegalVariableEvaluationException { + return runContext.render(headers).asMap(String.class, String.class); + } + + /** + * Resolves an endpoint against the configured Quickwit URL. + * + * @param url base URL of the cluster, for example {@code https://quickwit.example.com:7280} + * @param path API path without the {@code api/v1} prefix, for example {@code indexes/hdfs-logs} + * @param query optional query parameters, serialized as a query string + */ + public static URI endpoint(String url, String path, Map query) { + String base = StringUtils.removeEnd(StringUtils.trimToEmpty(url), "/") + "/" + API_PREFIX + "/" + StringUtils.stripStart(path, "/"); + + if (query == null || query.isEmpty()) { + return URI.create(base); + } + + String queryString = query.entrySet() + .stream() + .filter(e -> e.getValue() != null) + .map(e -> e.getKey() + "=" + URLEncoder.encode(String.valueOf(e.getValue()), StandardCharsets.UTF_8)) + .collect(Collectors.joining("&")); + + return URI.create(queryString.isEmpty() ? base : base + "?" + queryString); + } + + public static URI endpoint(String url, String path) { + return endpoint(url, path, Map.of()); + } + + /** + * URL-encodes a path segment, as required for index and source IDs containing special characters. + * + * @see Parameters + */ + public static String pathSegment(String value) { + // URLEncoder targets form-encoded data where a space becomes '+'; inside a URI path that would + // be a literal plus sign, so the percent-encoded form Quickwit expects is used instead. + return URLEncoder.encode(value, StandardCharsets.UTF_8).replace("+", "%20"); + } + + /** Creates a request carrying the configured headers. */ + public static HttpRequest.HttpRequestBuilder request(String method, URI uri, Map headers, HttpRequest.RequestBody body) { + HttpRequest.HttpRequestBuilder builder = HttpRequest.builder() + .method(method) + .uri(uri) + .addHeader("Accept", "application/json"); + + headers.forEach(builder::addHeader); + + return builder.body(body); + } + + /** + * Runs a search against the configured cluster and returns the parsed response. + * + * @param startTimestampOverride replaces the configured start timestamp when non-null + */ + public static SearchResult search( + RunContext runContext, + HttpClient client, + Property url, + Property> headers, + SearchQuery query, + Long startTimestampOverride + ) throws Exception { + // POST rather than GET: the body keeps list parameters (search_field, sort_by) and the `aggs` + // object as proper JSON instead of the comma-separated form the query string requires. + HttpRequest searchRequest = request( + "POST", + endpoint(renderedUrl(runContext, url), pathSegment(query.index()) + "/search"), + renderedHeaders(runContext, headers), + HttpRequest.JsonRequestBody.of(query.toBody(startTimestampOverride)) + ).build(); + + return execute(client, searchRequest, SearchResult.class, "search on index '" + query.index() + "'"); + } + + /** + * Executes a request and returns the deserialized response body. + * + * @param operation human readable operation name, used to turn a failure into an actionable message + * @throws IllegalStateException when Quickwit fails, or answers 2xx with an unusable body + */ + public static T execute(HttpClient client, HttpRequest request, Class type, String operation) throws Exception { + var response = send(client, request, operation); + String body = response.getBody(); + + if (StringUtils.isBlank(body)) { + throw new IllegalStateException( + "Quickwit " + operation + " returned an empty body with HTTP " + response.getStatus().getCode() + + ", expected a JSON payload. Check that `url` points at the Quickwit REST API (default port " + DEFAULT_PORT + ")." + ); + } + + return parse(body, type, operation); + } + + /** + * Executes a request whose response is a JSON array and converts it to typed models. + */ + public static List executeList(HttpClient client, HttpRequest request, Class type, String operation) throws Exception { + List raw = execute(client, request, List.class, operation); + + return raw.stream() + .map(item -> + { + try { + return JacksonMapper.cast(item, type); + } catch (JsonProcessingException e) { + throw new IllegalStateException("Unable to parse the Quickwit response of " + operation + " as " + type.getSimpleName(), e); + } + }) + .toList(); + } + + /** + * Executes a request against an endpoint that legitimately answers with an empty body, such as + * clearing an index, toggling a source or resetting a checkpoint. + */ + public static void executeIgnoringBody(HttpClient client, HttpRequest request, String operation) throws Exception { + send(client, request, operation); + } + + /** + * Sends a request and returns the response, turning a failed status into an actionable exception. + * + * @param operation human readable operation name, used to turn a failure into an actionable message + * @throws IllegalStateException when Quickwit answers with a 4xx or 5xx + */ + private static HttpResponse send(HttpClient client, HttpRequest request, String operation) throws Exception { + var response = client.request(request, String.class); + int status = response.getStatus().getCode(); + + if (status >= 400) { + throw failure(operation, status, response.getBody()); + } + + return response; + } + + @SuppressWarnings("unchecked") + private static T parse(String body, Class type, String operation) { + if (type == String.class) { + return (T) body; + } + + try { + return JacksonMapper.ofJson().readValue(body, type); + } catch (JsonProcessingException e) { + throw new IllegalStateException("Quickwit " + operation + " returned a body that is not valid JSON: " + e.getOriginalMessage(), e); + } + } + + private static IllegalStateException failure(String operation, int status, String body) { + return new IllegalStateException("Quickwit " + operation + " failed with HTTP " + status + ": " + quickwitMessage(body)); + } + + /** + * Extracts Quickwit's {@code message} field from a failed response. + * + *

Quickwit documents failed requests as {@code 4xx} with a JSON body holding a + * {@code message} field describing the error. + * + * @see Error handling + */ + public static String quickwitMessage(String body) { + if (StringUtils.isBlank(body)) { + return "no response body"; + } + + String trimmed = body.strip(); + try { + Object message = JacksonMapper.toMap(trimmed).get("message"); + return message != null ? String.valueOf(message) : trimmed; + } catch (Exception e) { + // not a JSON error object, for example an HTML page served by a proxy: surface it as-is + return trimmed; + } + } + + public static String requireNonBlank(String value, String field) { + if (StringUtils.isBlank(value)) { + throw new IllegalArgumentException("`" + field + "` is required and cannot be blank"); + } + + return value; + } + + /** + * Enforces a lower bound on a rendered numeric property. {@code @Min} cannot be used on a + * {@code Property}: Hibernate Validator has no validator for it and fails with HV000030. + * + * @return the value, or {@code null} when unset + */ + public static Integer requireAtLeast(Integer value, int min, String field) { + if (value != null && value < min) { + throw new IllegalArgumentException("`" + field + "` must be at least " + min + ", got " + value); + } + + return value; + } +} diff --git a/src/main/java/io/kestra/plugin/quickwit/QuickwitTask.java b/src/main/java/io/kestra/plugin/quickwit/QuickwitTask.java deleted file mode 100644 index 64abcc7..0000000 --- a/src/main/java/io/kestra/plugin/quickwit/QuickwitTask.java +++ /dev/null @@ -1,81 +0,0 @@ -package io.kestra.plugin.quickwit; - -import io.kestra.core.models.annotations.Plugin; -import io.kestra.core.models.property.Property; -import io.swagger.v3.oas.annotations.media.Schema; -import lombok.*; -import lombok.experimental.SuperBuilder; -import org.apache.commons.lang3.StringUtils; -import io.kestra.core.models.tasks.RunnableTask; -import io.kestra.core.models.tasks.Task; -import io.kestra.core.runners.RunContext; -import org.slf4j.Logger; - -@SuperBuilder -@ToString -@EqualsAndHashCode -@Getter -@NoArgsConstructor -@Schema( - title = "Short description for this task", - description = "Full description of this task" -) -@Plugin( - examples = { - @io.kestra.core.models.annotations.Example( - title = "Reverse a string", - full = true, - code = """ - id: example - namespace: company.team - - tasks: - - id: reverse - type: io.kestra.plugin.quickwit.QuickwitTask - format: "Text to be reverted" - """ - ) - } -) -public class QuickwitTask extends Task implements RunnableTask { - @Schema( - title = "Short description for this input", - description = "Full description of this input" - ) - private Property format; - - @Override - public QuickwitTask.Output run(RunContext runContext) throws Exception { - Logger logger = runContext.logger(); - - String render = runContext.render(format).as(String.class).orElse(""); - logger.debug(render); - - return Output.builder() - .child(new OutputChild(StringUtils.reverse(render))) - .build(); - } - - /** - * Input or Output can be nested as you need - */ - @Builder - @Getter - public static class Output implements io.kestra.core.models.tasks.Output { - @Schema( - title = "Short description for this output", - description = "Full description of this output" - ) - private final OutputChild child; - } - - @Builder - @Getter - public static class OutputChild implements io.kestra.core.models.tasks.Output { - @Schema( - title = "Short description for this output", - description = "Full description of this output" - ) - private final String value; - } -} diff --git a/src/main/java/io/kestra/plugin/quickwit/deletetask/Create.java b/src/main/java/io/kestra/plugin/quickwit/deletetask/Create.java new file mode 100644 index 0000000..2bba599 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/deletetask/Create.java @@ -0,0 +1,177 @@ +package io.kestra.plugin.quickwit.deletetask; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; +import io.kestra.plugin.quickwit.models.DeleteTask; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Queues the deletion of every document of an index matching a query. + * + * @see Create a delete task + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Create a Quickwit delete task", + description = """ + Queues a delete task on an index: every document matching the query is deleted by the janitor of the + cluster. + + The task only appends the delete task to the metastore queue, so this call returns immediately and the + documents are removed asynchronously. + """ +) +@Plugin( + examples = { + @Example( + title = "Apply a retention policy to yesterday's logs", + full = true, + code = """ + id: quickwit_delete_task + namespace: company.team + + tasks: + - id: delete + type: io.kestra.plugin.quickwit.deletetask.Create + url: "http://localhost:7280" + index: app-logs + query: "*" + startTimestamp: "{{ now() | dateAdd(-30, 'DAYS') | timestamp }}" + endTimestamp: "{{ now() | dateAdd(-29, 'DAYS') | timestamp }}" + + - id: report + type: io.kestra.plugin.core.log.Log + message: "Queued delete task {{ outputs.delete.opstamp }}" + """ + ) + } +) +public class Create extends AbstractQuickwitTask implements RunnableTask { + @Schema( + title = "Index ID", + description = "ID of the index to delete documents from." + ) + @NotNull + @PluginProperty(group = "main") + private Property index; + + @Schema( + title = "Query", + description = """ + Query selecting the documents to delete, using the + [Quickwit query language](https://quickwit.io/docs/reference/query-language), for example `message:trash`. + + The deletion is applied asynchronously by the janitor of the cluster. + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property query; + + @Schema( + title = "Start timestamp", + description = "Restricts the deletion to documents with `timestamp >= startTimestamp`, in seconds." + ) + @PluginProperty(group = "main") + private Property startTimestamp; + + @Schema( + title = "End timestamp", + description = "Restricts the deletion to documents with `timestamp < endTimestamp`, in seconds." + ) + @PluginProperty(group = "main") + private Property endTimestamp; + + @Schema( + title = "Search fields", + description = "Fields to search on when the query does not qualify a field name." + ) + @PluginProperty(group = "processing") + private Property> searchField; + + @Override + public Create.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, this.index, "index"); + + Map body = new LinkedHashMap<>(); + body.put("query", required(runContext, this.query, "query")); + putIfPresent(body, "start_timestamp", runContext.render(this.startTimestamp).as(Long.class).orElse(null)); + putIfPresent(body, "end_timestamp", runContext.render(this.endTimestamp).as(Long.class).orElse(null)); + + List searchField = runContext.render(this.searchField).asList(String.class); + if (!searchField.isEmpty()) { + // plural here, unlike the singular `search_field` of the search API, and an array rather + // than the comma-separated string that endpoint takes + body.put("search_fields", searchField); + } + + HttpRequest request = request( + runContext, + "POST", + pathSegment(rIndex) + "/delete-tasks", + HttpRequest.JsonRequestBody.of(body) + ).build(); + + DeleteTask deleteTask; + try (var client = client(runContext)) { + deleteTask = execute(client, request, DeleteTask.class, "creation of a delete task on index '" + rIndex + "'"); + } + + return Output.builder() + .index(rIndex) + .createTimestamp(deleteTask.getCreateTimestamp()) + .opstamp(deleteTask.getOpstamp()) + .deleteTask(deleteTask) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index the delete task was queued on." + ) + private String index; + + @Schema( + title = "Creation timestamp", + description = "Creation timestamp of the delete task, in seconds." + ) + private Long createTimestamp; + + @Schema( + title = "Operation stamp", + description = "Unique operation stamp of the delete task." + ) + private Long opstamp; + + @Schema( + title = "Delete task", + description = "The queued delete task, as Quickwit stored it." + ) + private DeleteTask deleteTask; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/deletetask/List.java b/src/main/java/io/kestra/plugin/quickwit/deletetask/List.java new file mode 100644 index 0000000..97a9f13 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/deletetask/List.java @@ -0,0 +1,104 @@ +package io.kestra.plugin.quickwit.deletetask; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; +import io.kestra.plugin.quickwit.models.DeleteTask; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Lists the delete tasks queued on a Quickwit index. + * + * @see List delete queries + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "List Quickwit delete tasks", + description = "Returns every delete task queued on an index, in the order Quickwit stored them." +) +@Plugin( + examples = { + @Example( + title = "Inspect the delete tasks of an index", + full = true, + code = """ + id: quickwit_list_delete_tasks + namespace: company.team + + tasks: + - id: list + type: io.kestra.plugin.quickwit.deletetask.List + url: "http://localhost:7280" + index: app-logs + + - id: report + type: io.kestra.plugin.core.log.Log + message: "{{ outputs.list.deleteTasks | size }} delete task(s) queued" + """ + ) + } +) +public class List extends AbstractQuickwitTask implements RunnableTask { + @Schema( + title = "Index ID", + description = "ID of the index to list the delete tasks of." + ) + @PluginProperty(group = "main") + private Property index; + + @Override + public List.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, this.index, "index"); + + HttpRequest request = request(runContext, "GET", pathSegment(rIndex) + "/delete-tasks", null).build(); + + java.util.List deleteTasks; + try (var client = client(runContext)) { + deleteTasks = executeList(client, request, DeleteTask.class, "listing of the delete tasks on index '" + rIndex + "'"); + } + + return Output.builder() + .index(rIndex) + .size(deleteTasks.size()) + .deleteTasks(deleteTasks) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index the delete tasks belong to." + ) + private String index; + + @Schema( + title = "Delete task count", + description = "Number of delete tasks returned." + ) + private Integer size; + + @Schema( + title = "Delete tasks", + description = "The delete tasks queued on the index." + ) + private java.util.List deleteTasks; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/deletetask/package-info.java b/src/main/java/io/kestra/plugin/quickwit/deletetask/package-info.java new file mode 100644 index 0000000..6d138b2 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/deletetask/package-info.java @@ -0,0 +1,8 @@ +@PluginSubGroup( + title = "Delete tasks", + description = "Delete documents from a Quickwit index by query.", + categories = PluginSubGroup.PluginCategory.DATA +) +package io.kestra.plugin.quickwit.deletetask; + +import io.kestra.core.models.annotations.PluginSubGroup; \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/AbstractQuickwitIndex.java b/src/main/java/io/kestra/plugin/quickwit/index/AbstractQuickwitIndex.java new file mode 100644 index 0000000..3490279 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/AbstractQuickwitIndex.java @@ -0,0 +1,36 @@ +package io.kestra.plugin.quickwit.index; + +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Index management tasks, sharing the index ID. + * + * @see Quickwit index API + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +public abstract class AbstractQuickwitIndex extends AbstractQuickwitTask { + @Schema( + title = "Index ID", + description = """ + ID of the index. See the + [index ID validation rules](https://quickwit.io/docs/configuration/index-config#index-id). + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property index; +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/Clear.java b/src/main/java/io/kestra/plugin/quickwit/index/Clear.java new file mode 100644 index 0000000..6a7a5f3 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/Clear.java @@ -0,0 +1,87 @@ +package io.kestra.plugin.quickwit.index; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Drops every document of a Quickwit index while keeping its definition. + * + * @see Clear an index + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Clear a Quickwit index", + description = """ + Deletes every split of an index and resets the checkpoint of its sources, while keeping the index + definition in place. Quickwit answers with an empty body, which is expected. + """ +) +@Plugin( + examples = { + @Example( + title = "Reset an index before replaying a day of logs", + full = true, + code = """ + id: quickwit_clear_index + namespace: company.team + + tasks: + - id: clear + type: io.kestra.plugin.quickwit.index.Clear + url: "http://localhost:7280" + index: app-logs + + - id: reingest + type: io.kestra.plugin.quickwit.ingest.Ingest + url: "http://localhost:7280" + index: app-logs + from: "{{ outputs.replay_file }}" + commit: FORCE + """ + ) + } +) +public class Clear extends AbstractQuickwitIndex implements RunnableTask { + @Override + public Clear.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + + // Quickwit documents this endpoint as returning an empty body. + HttpRequest request = request(runContext, "PUT", "indexes/" + pathSegment(rIndex) + "/clear", null).build(); + + try (var client = client(runContext)) { + executeIgnoringBody(client, request, "clearing of index '" + rIndex + "'"); + } + + runContext.logger().info("Cleared index '{}': all splits deleted and source checkpoints reset", rIndex); + + return Output.builder() + .index(rIndex) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index that was cleared." + ) + private String index; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/Create.java b/src/main/java/io/kestra/plugin/quickwit/index/Create.java new file mode 100644 index 0000000..5ac5c47 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/Create.java @@ -0,0 +1,216 @@ +package io.kestra.plugin.quickwit.index; + +import java.util.LinkedHashMap; +import java.util.Map; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.models.IndexMetadata; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Creates a Quickwit index. + * + * @see Quickwit index configuration + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Create a Quickwit index", + description = """ + Creates an index from a doc mapping, and optional indexing settings, search settings and retention + policy. + + Quickwit also accepts the index configuration as a YAML document. This task sends JSON, which cannot + carry YAML comments or anchors. + """ +) +@Plugin( + examples = { + @Example( + title = "Create a logs index with a one week retention", + full = true, + code = """ + id: quickwit_create_index + namespace: company.team + + tasks: + - id: create + type: io.kestra.plugin.quickwit.index.Create + url: "http://localhost:7280" + index: app-logs + configVersion: "0.8" + docMapping: + timestamp_field: timestamp + field_mappings: + - name: timestamp + type: datetime + input_formats: + - unix_timestamp + fast: true + - name: service + type: text + fast: true + - name: severity + type: text + fast: true + - name: message + type: text + searchSettings: + default_search_fields: + - message + retention: + period: "7 days" + schedule: "@daily" + + - id: created + type: io.kestra.plugin.core.log.Log + message: "Created index {{ outputs.create.indexUid }}" + """ + ) + } +) +public class Create extends AbstractQuickwitIndex implements RunnableTask { + @Schema( + title = "Configuration format version", + description = """ + Version of the index configuration format, which must match the version of your Quickwit + cluster, for example `0.8`. + + Named `configVersion` and not `version` because `version` is reserved by Kestra to pin a + plugin version. + """ + ) + @PluginProperty(group = "main") + private Property configVersion; + + @Schema( + title = "Doc mapping", + description = """ + Doc mapping of the index: how documents are parsed and which fields are indexed and fast. + See the [doc mapping documentation](https://quickwit.io/docs/configuration/index-config#doc-mapping). + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property> docMapping; + + @Schema( + title = "Index URI", + description = """ + Storage URI holding the index files, for example `s3://my-bucket/logs`. + Defaults to `{default_index_root_uri}/{index}` on the Quickwit side. + """ + ) + @PluginProperty(group = "main") + private Property indexUri; + + @Schema( + title = "Indexing settings", + description = """ + Merge policy and commit settings of the index. + See [indexing settings](https://quickwit.io/docs/configuration/index-config#indexing-settings). + """ + ) + @PluginProperty(group = "advanced") + private Property> indexingSettings; + + @Schema( + title = "Search settings", + description = """ + Search settings of the index, for example `default_search_fields`. + See [search settings](https://quickwit.io/docs/configuration/index-config#search-settings). + """ + ) + @PluginProperty(group = "advanced") + private Property> searchSettings; + + @Schema( + title = "Retention", + description = """ + Retention policy of the index, for example `{period: "30 days", schedule: "@daily"}`. + See [retention policy](https://quickwit.io/docs/configuration/index-config#retention-policy). + """ + ) + @PluginProperty(group = "advanced") + private Property> retention; + + @Override + public Create.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + Map docMapping = requiredMap( + runContext, + this.docMapping, + "docMapping", + "https://quickwit.io/docs/configuration/index-config#doc-mapping" + ); + + // LinkedHashMap keeps the payload readable in logs and in the Quickwit API playground. + Map configuration = new LinkedHashMap<>(); + putIfPresent(configuration, "version", runContext.render(this.configVersion).as(String.class).orElse(null)); + configuration.put("index_id", rIndex); + putIfPresent(configuration, "index_uri", runContext.render(this.indexUri).as(String.class).orElse(null)); + configuration.put("doc_mapping", docMapping); + putIfPresent(configuration, "indexing_settings", runContext.render(this.indexingSettings).asMap(String.class, Object.class)); + putIfPresent(configuration, "search_settings", runContext.render(this.searchSettings).asMap(String.class, Object.class)); + putIfPresent(configuration, "retention", runContext.render(this.retention).asMap(String.class, Object.class)); + + HttpRequest request = request(runContext, "POST", "indexes", HttpRequest.JsonRequestBody.of(configuration)).build(); + + IndexMetadata metadata; + try (var client = client(runContext)) { + metadata = execute(client, request, IndexMetadata.class, "creation of index '" + rIndex + "'"); + } + + return Output.builder() + .index(rIndex) + .indexUid(metadata.getIndexUid()) + .createTimestamp(metadata.getCreateTimestamp()) + .metadata(metadata) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the created index." + ) + private String index; + + @Schema( + title = "Index UID", + description = "Server-generated unique identifier of the created index." + ) + private String indexUid; + + @Schema( + title = "Creation timestamp", + description = "Creation timestamp of the index, in seconds." + ) + private Long createTimestamp; + + @Schema( + title = "Index metadata", + description = "Full metadata of the created index, including its effective configuration and sources." + ) + private IndexMetadata metadata; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/Delete.java b/src/main/java/io/kestra/plugin/quickwit/index/Delete.java new file mode 100644 index 0000000..65302b7 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/Delete.java @@ -0,0 +1,104 @@ +package io.kestra.plugin.quickwit.index; + +import java.util.List; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.models.DeletedSplit; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Deletes a Quickwit index and every split file it owns. + * + * @see Delete an index + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Delete a Quickwit index", + description = """ + Deletes an index: its splits are removed from the metastore and from the storage. + + This cannot be undone. Use + [Clear](https://kestra.io/docs/plugins/plugin-quickwit/io.kestra.plugin.quickwit.index.Clear) to keep the + index definition and only drop its data. + """ +) +@Plugin( + examples = { + @Example( + title = "Delete an index once its retention window has passed", + full = true, + code = """ + id: quickwit_delete_index + namespace: company.team + + tasks: + - id: delete + type: io.kestra.plugin.quickwit.index.Delete + url: "http://localhost:7280" + index: app-logs + + - id: report + type: io.kestra.plugin.core.log.Log + message: "Removed {{ outputs.delete.size }} split(s)" + """ + ) + } +) +public class Delete extends AbstractQuickwitIndex implements RunnableTask { + @Override + public Delete.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + + HttpRequest request = request(runContext, "DELETE", "indexes/" + pathSegment(rIndex), null).build(); + + List deletedSplits; + try (var client = client(runContext)) { + deletedSplits = executeList(client, request, DeletedSplit.class, "deletion of index '" + rIndex + "'"); + } + + runContext.logger().info("Deleted index '{}' and {} split file(s)", rIndex, deletedSplits.size()); + + return Output.builder() + .index(rIndex) + .size(deletedSplits.size()) + .deletedSplits(deletedSplits) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index that was deleted." + ) + private String index; + + @Schema( + title = "Deleted split count", + description = "Number of split files removed along with the index." + ) + private Integer size; + + @Schema( + title = "Deleted splits", + description = "The split files that were removed from the storage." + ) + private List deletedSplits; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/Get.java b/src/main/java/io/kestra/plugin/quickwit/index/Get.java new file mode 100644 index 0000000..1e0a3f8 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/Get.java @@ -0,0 +1,104 @@ +package io.kestra.plugin.quickwit.index; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.models.IndexMetadata; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Reads the metadata of a Quickwit index. + * + * @see Get an index metadata + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Get a Quickwit index", + description = """ + Returns the metadata of an index: its effective configuration, its UID, its creation timestamp and + the sources attached to it. + """ +) +@Plugin( + examples = { + @Example( + title = "Read the configuration of an index", + full = true, + code = """ + id: quickwit_get_index + namespace: company.team + + tasks: + - id: get + type: io.kestra.plugin.quickwit.index.Get + url: "http://localhost:7280" + index: app-logs + + - id: report + type: io.kestra.plugin.core.log.Log + message: "Timestamp field: {{ outputs.get.metadata.indexConfig.doc_mapping.timestamp_field }}" + """ + ) + } +) +public class Get extends AbstractQuickwitIndex implements RunnableTask { + @Override + public Get.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + + HttpRequest request = request(runContext, "GET", "indexes/" + pathSegment(rIndex), null).build(); + + IndexMetadata metadata; + try (var client = client(runContext)) { + metadata = execute(client, request, IndexMetadata.class, "read of index '" + rIndex + "'"); + } + + return Output.builder() + .index(rIndex) + .indexUid(metadata.getIndexUid()) + .createTimestamp(metadata.getCreateTimestamp()) + .metadata(metadata) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index that was read." + ) + private String index; + + @Schema( + title = "Index UID", + description = "Server-generated unique identifier of the index." + ) + private String indexUid; + + @Schema( + title = "Creation timestamp", + description = "Creation timestamp of the index, in seconds." + ) + private Long createTimestamp; + + @Schema( + title = "Index metadata", + description = "Metadata of the index, including its effective configuration, checkpoints and sources." + ) + private IndexMetadata metadata; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/List.java b/src/main/java/io/kestra/plugin/quickwit/index/List.java new file mode 100644 index 0000000..3bfa91b --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/List.java @@ -0,0 +1,85 @@ +package io.kestra.plugin.quickwit.index; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; +import io.kestra.plugin.quickwit.models.IndexMetadata; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Lists every Quickwit index of the cluster. + * + * @see Get all indexes metadata + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "List Quickwit indexes", + description = "Returns the metadata of every index present in the metastore." +) +@Plugin( + examples = { + @Example( + title = "List the indexes of a cluster", + full = true, + code = """ + id: quickwit_list_indexes + namespace: company.team + + tasks: + - id: list + type: io.kestra.plugin.quickwit.index.List + url: "http://localhost:7280" + + - id: report + type: io.kestra.plugin.core.log.Log + message: "{{ outputs.list.indexes | jq('.[] | .index_config.index_id') | join(', ') }}" + """ + ) + } +) +public class List extends AbstractQuickwitTask implements RunnableTask { + @Override + public List.Output run(RunContext runContext) throws Exception { + HttpRequest request = request(runContext, "GET", "indexes", null).build(); + + java.util.List indexes; + try (var client = client(runContext)) { + indexes = executeList(client, request, IndexMetadata.class, "listing of the indexes"); + } + + return Output.builder() + .size(indexes.size()) + .indexes(indexes) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index count", + description = "Number of indexes returned." + ) + private Integer size; + + @Schema( + title = "Indexes", + description = "Metadata of every index present in the metastore." + ) + private java.util.List indexes; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/index/package-info.java b/src/main/java/io/kestra/plugin/quickwit/index/package-info.java new file mode 100644 index 0000000..baac755 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/index/package-info.java @@ -0,0 +1,8 @@ +@PluginSubGroup( + title = "Indexes", + description = "Create, inspect, clear and delete Quickwit indexes.", + categories = PluginSubGroup.PluginCategory.DATA +) +package io.kestra.plugin.quickwit.index; + +import io.kestra.core.models.annotations.PluginSubGroup; \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/ingest/Commit.java b/src/main/java/io/kestra/plugin/quickwit/ingest/Commit.java new file mode 100644 index 0000000..e5fecf3 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/ingest/Commit.java @@ -0,0 +1,28 @@ +package io.kestra.plugin.quickwit.ingest; + +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * When documents sent to the ingest API become visible to search. + * + * @see Commit behaviour + */ +@Schema( + title = "Commit behavior", + description = """ + When newly ingested documents become visible to search. + + - `AUTO` (default): return as soon as the documents are queued, they are searchable after the + next split commit. + - `WAIT_FOR`: wait for the next commit, following the `commit_timeout_secs` and + `split_num_docs_target` rules. + - `FORCE`: force a commit as soon as this batch is processed, and wait for it. Use it when the + documents must be searchable right after the task, for example before an immediate search. + This has a real performance cost on small batches. + """ +) +public enum Commit { + AUTO, + WAIT_FOR, + FORCE +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/ingest/Ingest.java b/src/main/java/io/kestra/plugin/quickwit/ingest/Ingest.java new file mode 100644 index 0000000..341dad2 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/ingest/Ingest.java @@ -0,0 +1,270 @@ +package io.kestra.plugin.quickwit.ingest; + +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.stream.Collectors; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +import io.kestra.core.exceptions.IllegalVariableEvaluationException; +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Metric; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.executions.metrics.Counter; +import io.kestra.core.models.property.Data; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.core.serializers.JacksonMapper; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; +import io.kestra.plugin.quickwit.models.IngestResult; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Sends a batch of documents to a Quickwit index. + * + *

Quickwit only accepts NDJSON on this endpoint, so whatever shape the documents arrive in (a Kestra + * file, an inline list, a JSON string) they are serialized line by line here. + * + * @see Quickwit ingest API + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Ingest documents into Quickwit", + description = """ + Sends documents to a Quickwit index as a single NDJSON batch. + + The batch is limited to 10 MB by default, so split larger datasets across several runs or use a + Quickwit source instead. + """ +) +@Plugin( + metrics = { + @Metric(name = "records", type = Counter.TYPE, unit = "records", description = "Number of documents sent") + }, + examples = { + @Example( + title = "Ingest a file produced by a previous task", + full = true, + code = """ + id: quickwit_ingest + namespace: company.team + + inputs: + - id: file + type: FILE + + tasks: + - id: ingest + type: io.kestra.plugin.quickwit.ingest.Ingest + url: "https://quickwit.example.com:7280" + index: app-logs + from: "{{ inputs.file }}" + commit: WAIT_FOR + """ + ), + @Example( + title = "Ingest inline documents and check for rejections", + full = true, + code = """ + id: quickwit_ingest_inline + namespace: company.team + + tasks: + - id: ingest + type: io.kestra.plugin.quickwit.ingest.Ingest + url: "http://localhost:7280" + index: app-logs + commit: FORCE + detailedResponse: true + from: + - timestamp: "{{ now() | timestamp }}" + service: checkout + severity: ERROR + message: "Payment gateway timeout" + - timestamp: "{{ now() | timestamp }}" + service: checkout + severity: ERROR + message: "Payment gateway timeout" + + - id: check + type: io.kestra.plugin.core.log.Log + message: "Ingested {{ outputs.ingest.numIngestedDocs }} / rejected {{ outputs.ingest.numRejectedDocs }}" + """ + ) + } +) +public class Ingest extends AbstractQuickwitTask implements RunnableTask { + private static final ObjectMapper JSON = JacksonMapper.ofJson(); + + @Schema( + title = "Index ID", + description = "ID of the index to ingest into. The ingest API is only served by nodes running an indexer." + ) + @NotNull + @PluginProperty(group = "main") + private Property index; + + @Schema( + title = "Documents", + description = """ + Documents to send, as one of: + + - a single document, as a map; + - a list of documents, as a list of maps; + - a URI, either `kestra://` internal storage or a local `file://` path; + - a JSON string holding one document or a list of documents. + + Documents must match the doc mapping of the index. + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property from; + + @Schema( + title = "Commit behavior", + description = """ + When the ingested documents become visible to search: `AUTO` (default), `WAIT_FOR` or `FORCE`. + Use `FORCE` when a search runs right after this task and must already see the documents. + """ + ) + @Builder.Default + @PluginProperty(group = "processing") + private Property commit = Property.ofValue(Commit.AUTO); + + @Schema( + title = "Detailed response", + description = """ + Ask Quickwit to describe every rejected document in `parseFailures`. + Handy to diagnose a doc mapping mismatch, but it costs performance. + """ + ) + @Builder.Default + @PluginProperty(group = "processing") + private Property detailedResponse = Property.ofValue(false); + + @Override + public Ingest.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, this.index, "index"); + + String ndjson = toNdjson(runContext); + if (ndjson.isEmpty()) { + throw new IllegalArgumentException("`from` resolved to no document, nothing to ingest"); + } + + Commit commit = runContext.render(this.commit).as(Commit.class).orElse(Commit.AUTO); + boolean detailed = runContext.render(this.detailedResponse).as(Boolean.class).orElse(false); + + HttpRequest request = request( + runContext, + "POST", + pathSegment(rIndex) + "/ingest", + Map.of( + "commit", commit.name().toLowerCase(Locale.ROOT), + "detailed_response", detailed + ), + HttpRequest.StringRequestBody.builder() + // the ingest handler reads raw body bytes and never inspects the content type; + // application/json is what Quickwit's own OpenAPI document declares for this endpoint + .contentType("application/json") + .charset(StandardCharsets.UTF_8) + .content(ndjson) + .build() + ).build(); + + IngestResult result; + try (var client = client(runContext)) { + result = execute(client, request, IngestResult.class, "ingest into index '" + rIndex + "'"); + } + + long ingested = orZero(result.getNumIngestedDocs()); + long rejected = orZero(result.getNumRejectedDocs()); + runContext.metric(Counter.of("records", ingested + rejected)); + + if (rejected > 0) { + runContext.logger().warn("Quickwit rejected {} document(s) out of {} submitted to index '{}'", rejected, orZero(result.getNumDocsForProcessing()), rIndex); + } + + return Output.builder() + .numDocsForProcessing(orZero(result.getNumDocsForProcessing())) + .numIngestedDocs(ingested) + .numRejectedDocs(rejected) + .parseFailures(result.getParseFailures()) + .build(); + } + + /** + * Reads {@link #from} and serializes the documents as NDJSON, the only payload format Quickwit + * accepts on the ingest API. + */ + private String toNdjson(RunContext runContext) throws IllegalVariableEvaluationException { + Object source = runContext.render(this.from).as(Object.class).orElse(null); + + return Data.from(source) + .read(runContext) + .map(Ingest::toJson) + .collectList() + .blockOptional() + .orElse(List.of()) + .stream() + .collect(Collectors.joining("\n")); + } + + private static String toJson(Map document) { + try { + return JSON.writeValueAsString(document); + } catch (JsonProcessingException e) { + throw new IllegalArgumentException("Unable to serialize a document as JSON: " + e.getMessage(), e); + } + } + + private static long orZero(Long value) { + return value == null ? 0L : value; + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Documents submitted", + description = "Number of documents submitted for processing. They may not be searchable yet." + ) + private Long numDocsForProcessing; + + @Schema( + title = "Documents ingested", + description = "Number of documents successfully persisted in the write ahead log." + ) + private Long numIngestedDocs; + + @Schema( + title = "Documents rejected", + description = "Number of documents that could not be parsed, usually because of a doc mapping mismatch." + ) + private Long numRejectedDocs; + + @Schema( + title = "Parse failures", + description = "Details of the rejected documents, present only when `detailedResponse` is enabled." + ) + private List parseFailures; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/ingest/package-info.java b/src/main/java/io/kestra/plugin/quickwit/ingest/package-info.java new file mode 100644 index 0000000..edcdd57 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/ingest/package-info.java @@ -0,0 +1,8 @@ +@PluginSubGroup( + title = "Ingest", + description = "Send documents to Quickwit indexes.", + categories = PluginSubGroup.PluginCategory.DATA +) +package io.kestra.plugin.quickwit.ingest; + +import io.kestra.core.models.annotations.PluginSubGroup; \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/DeleteTask.java b/src/main/java/io/kestra/plugin/quickwit/models/DeleteTask.java new file mode 100644 index 0000000..3986964 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/DeleteTask.java @@ -0,0 +1,89 @@ +package io.kestra.plugin.quickwit.models; + +import java.util.List; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.extern.jackson.Jacksonized; + +/** + * A Quickwit delete task, which removes every document matching a query from an index. + * + * @see Quickwit delete API + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Getter +@Builder +@Jacksonized +@AllArgsConstructor +@NoArgsConstructor +@Schema( + title = "Delete task", + description = "A delete task queued on an index, deleting every document matching its query." +) +public class DeleteTask { + @Schema( + title = "Creation timestamp", + description = "Creation timestamp of the delete task, in seconds." + ) + @JsonProperty("create_timestamp") + private Long createTimestamp; + + @Schema( + title = "Operation stamp", + description = "Unique operation stamp associated with the delete task." + ) + private Long opstamp; + + @Schema( + title = "Delete query", + description = "The query selecting the documents to delete." + ) + @JsonProperty("delete_query") + private DeleteQuery deleteQuery; + + @Schema( + title = "Delete query", + description = "Query selecting the documents to delete, restricted to a time range and a set of fields." + ) + @JsonIgnoreProperties(ignoreUnknown = true) + @Getter + @Builder + @Jacksonized + @AllArgsConstructor + @NoArgsConstructor + public static class DeleteQuery { + @Schema( + title = "Query", + description = "Query text, using the Quickwit query language." + ) + private String query; + + @Schema( + title = "Search fields", + description = "Fields to search on when the query does not qualify a field name." + ) + @JsonProperty("search_field") + private List searchField; + + @Schema( + title = "Start timestamp", + description = "Restricts the deletion to documents with `timestamp >= start_timestamp`, in seconds." + ) + @JsonProperty("start_timestamp") + private Long startTimestamp; + + @Schema( + title = "End timestamp", + description = "Restricts the deletion to documents with `timestamp < end_timestamp`, in seconds." + ) + @JsonProperty("end_timestamp") + private Long endTimestamp; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/DeletedSplit.java b/src/main/java/io/kestra/plugin/quickwit/models/DeletedSplit.java new file mode 100644 index 0000000..af28315 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/DeletedSplit.java @@ -0,0 +1,63 @@ +package io.kestra.plugin.quickwit.models; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.extern.jackson.Jacksonized; + +/** + * A split file that was removed from the storage when an index was deleted. + * + * @see Quickwit delete index + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Getter +@Builder +@Jacksonized +@AllArgsConstructor +@NoArgsConstructor +@Schema( + title = "Deleted split", + description = "A split file deleted along with its index." +) +public class DeletedSplit { + @Schema( + title = "Split ID", + description = "Unique identifier of the deleted split." + ) + @JsonProperty("split_id") + private String splitId; + + @Schema( + title = "Number of documents", + description = "Number of documents held by the split before deletion." + ) + @JsonProperty("num_docs") + private Long numDocs; + + @Schema( + title = "Uncompressed documents size", + description = "Size in bytes of the documents held by the split, uncompressed." + ) + @JsonProperty("uncompressed_docs_size_bytes") + private Long uncompressedDocsSizeBytes; + + @Schema( + title = "File name", + description = "Name of the split file." + ) + @JsonProperty("file_name") + private String fileName; + + @Schema( + title = "File size", + description = "Size in bytes of the split file." + ) + @JsonProperty("file_size_bytes") + private Long fileSizeBytes; +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/IndexMetadata.java b/src/main/java/io/kestra/plugin/quickwit/models/IndexMetadata.java new file mode 100644 index 0000000..5115af2 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/IndexMetadata.java @@ -0,0 +1,70 @@ +package io.kestra.plugin.quickwit.models; + +import java.util.List; +import java.util.Map; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.extern.jackson.Jacksonized; + +/** + * Metadata of a Quickwit index, as returned by the index management API. + * + * @see Quickwit index API + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Getter +@Builder +@Jacksonized +@AllArgsConstructor +@NoArgsConstructor +@Schema( + title = "Index metadata", + description = "Configuration and state of a Quickwit index." +) +public class IndexMetadata { + @Schema( + title = "Configuration format version", + description = "Format version of the index configuration, matching the Quickwit version of the cluster." + ) + private String version; + + @Schema( + title = "Index UID", + description = "Server-generated unique identifier of the index." + ) + @JsonProperty("index_uid") + private String indexUid; + + @Schema( + title = "Index configuration", + description = "The effective index configuration, including its doc mapping, indexing settings, search settings and retention policy." + ) + @JsonProperty("index_config") + private Map indexConfig; + + @Schema( + title = "Source checkpoints", + description = "Last committed checkpoint of each source attached to the index, keyed by source ID." + ) + private Map checkpoint; + + @Schema( + title = "Creation timestamp", + description = "Creation timestamp of the index, in seconds." + ) + @JsonProperty("create_timestamp") + private Long createTimestamp; + + @Schema( + title = "Sources", + description = "Sources attached to this index." + ) + private List sources; +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/IngestResult.java b/src/main/java/io/kestra/plugin/quickwit/models/IngestResult.java new file mode 100644 index 0000000..a56827f --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/IngestResult.java @@ -0,0 +1,88 @@ +package io.kestra.plugin.quickwit.models; + +import java.util.List; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.extern.jackson.Jacksonized; + +/** + * Response of the Quickwit ingest API. + * + * @see Quickwit ingest API + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Getter +@Builder +@Jacksonized +@AllArgsConstructor +@NoArgsConstructor +@Schema( + title = "Ingest result", + description = "Counters reported by Quickwit after a batch of documents was submitted." +) +public class IngestResult { + @Schema( + title = "Documents submitted", + description = "Total number of documents submitted for processing. They may not have been indexed yet." + ) + @JsonProperty("num_docs_for_processing") + private Long numDocsForProcessing; + + @Schema( + title = "Documents ingested", + description = "Number of documents successfully persisted in the write ahead log." + ) + @JsonProperty("num_ingested_docs") + private Long numIngestedDocs; + + @Schema( + title = "Documents rejected", + description = "Number of documents that could not be parsed, for example because of invalid JSON or a doc mapping mismatch." + ) + @JsonProperty("num_rejected_docs") + private Long numRejectedDocs; + + @Schema( + title = "Parse failures", + description = "Details of the rejected documents, present only when `detailedResponse` is enabled." + ) + @JsonProperty("parse_failures") + private List parseFailures; + + @Schema( + title = "Parse failure", + description = "A document rejected by the ingest API, with the reason and the offending payload." + ) + @JsonIgnoreProperties(ignoreUnknown = true) + @Getter + @Builder + @Jacksonized + @AllArgsConstructor + @NoArgsConstructor + public static class ParseFailure { + @Schema( + title = "Failure message", + description = "Human readable explanation of why the document was rejected." + ) + private String message; + + @Schema( + title = "Failure reason", + description = "Machine readable reason, one of `invalid_json`, `invalid_schema` or `unspecified`." + ) + private String reason; + + @Schema( + title = "Rejected document", + description = "The UTF-8 decoded document chunk that generated the error." + ) + private String document; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/SearchQuery.java b/src/main/java/io/kestra/plugin/quickwit/models/SearchQuery.java new file mode 100644 index 0000000..e5c0051 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/SearchQuery.java @@ -0,0 +1,100 @@ +package io.kestra.plugin.quickwit.models; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * The parameters of a Quickwit search, independent of whether the search is run by the + * {@code Search} task or by the {@code Trigger}. + * + * @param index ID of the index, or a multi-target expression such as `app-logs*` + * @param query query text, in the Quickwit query language + * @param startTimestamp restricts results to {@code timestamp >= startTimestamp}, in seconds + * @param endTimestamp restricts results to {@code timestamp < endTimestamp}, in seconds + * @param startOffset number of documents to skip + * @param maxHits maximum number of documents to return + * @param searchField fields to search on when the query does not qualify a field name + * @param snippetFields fields to extract snippets on + * @param sortBy fields to sort the results on + * @param aggregations aggregation request + */ +@Schema( + title = "Search query", + description = "Parameters of a Quickwit search request." +) +public record SearchQuery( + String index, + String query, + Long startTimestamp, + Long endTimestamp, + Integer startOffset, + Integer maxHits, + List searchField, + List snippetFields, + List sortBy, + Map aggregations +) { + /** + * A search restricted to an index, a query and a hit limit, leaving every other parameter to + * Quickwit. Used by callers that do not expose the full set of search parameters. + */ + public static SearchQuery of(String index, String query, Integer maxHits) { + return new SearchQuery(index, query, null, null, null, maxHits, null, null, null, null); + } + + /** + * Builds the JSON body of the search request. + * + *

Optional parameters are omitted when unset so Quickwit applies its own defaults, and the + * list parameters are sent as comma-separated strings, which is what this endpoint accepts. + * + * @param startTimestampOverride replaces {@link #startTimestamp} when non-null; the trigger uses + * this to poll the window that follows its last watermark + */ + public Map toBody(Long startTimestampOverride) { + Map body = new LinkedHashMap<>(); + body.put("query", this.query); + + Long start = startTimestampOverride != null ? startTimestampOverride : this.startTimestamp; + if (start != null) { + body.put("start_timestamp", start); + } + if (this.endTimestamp != null) { + body.put("end_timestamp", this.endTimestamp); + } + if (this.startOffset != null) { + body.put("start_offset", this.startOffset); + } + if (this.maxHits != null) { + body.put("max_hits", this.maxHits); + } + + putCsv(body, "search_field", this.searchField); + putCsv(body, "snippet_fields", this.snippetFields); + putCsv(body, "sort_by", this.sortBy); + + if (this.aggregations != null && !this.aggregations.isEmpty()) { + body.put("aggs", this.aggregations); + } + + return body; + } + + /** + * Sends a list as the comma-separated string the search API expects. + * + *

The request body is deserialized into plain string fields, so a JSON array is rejected with + * {@code invalid type: sequence, expected a string}. The delete task endpoint is the opposite: + * it wants {@code search_fields} as an array, which {@code deletetask.Create} sends. + * + * @see Search API + */ + private static void putCsv(Map body, String key, List values) { + if (values != null && !values.isEmpty()) { + body.put(key, String.join(",", values)); + } + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/SearchResult.java b/src/main/java/io/kestra/plugin/quickwit/models/SearchResult.java new file mode 100644 index 0000000..9056ac6 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/SearchResult.java @@ -0,0 +1,72 @@ +package io.kestra.plugin.quickwit.models; + +import java.util.List; +import java.util.Map; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.extern.jackson.Jacksonized; + +/** + * Response of the Quickwit search API. + * + *

Quickwit returns {@code hits} as a list of raw documents rather than a list of wrappers + * holding a {@code source} object, so each entry is the ingested document as-is. + * + * @see Quickwit search API + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Getter +@Builder +@Jacksonized +@AllArgsConstructor +@NoArgsConstructor +@Schema( + title = "Search result", + description = "Documents matching a Quickwit search query, with the total match count and aggregations." +) +public class SearchResult { + @Schema( + title = "Total number of matches", + description = "Total number of documents matching the query, regardless of pagination." + ) + @JsonProperty("num_hits") + private Long numHits; + + @Schema( + title = "Matching documents", + description = "Documents matching the query, as raw documents." + ) + private List> hits; + + @Schema( + title = "Snippets", + description = "Extracted snippets, present only when snippet fields were requested." + ) + private List> snippets; + + @Schema( + title = "Processing time", + description = "Time taken by Quickwit to process the query, in microseconds." + ) + @JsonProperty("elapsed_time_micros") + private Long elapsedTimeMicros; + + @Schema( + title = "Search errors", + description = "Non-fatal errors reported by Quickwit while executing the query." + ) + private List errors; + + @Schema( + title = "Aggregations", + description = "Aggregation results, present only when aggregations were requested." + ) + private Map aggregations; +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/models/Source.java b/src/main/java/io/kestra/plugin/quickwit/models/Source.java new file mode 100644 index 0000000..381f74c --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/models/Source.java @@ -0,0 +1,69 @@ +package io.kestra.plugin.quickwit.models; + +import java.util.Map; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.extern.jackson.Jacksonized; + +/** + * A source attached to a Quickwit index, for example a Kafka, Kinesis or Pulsar source. + * + * @see Quickwit source configuration + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Getter +@Builder +@Jacksonized +@AllArgsConstructor +@NoArgsConstructor +@Schema( + title = "Index source", + description = "Configuration of a source feeding a Quickwit index." +) +public class Source { + @Schema( + title = "Configuration format version", + description = "Format version of the source configuration, matching the Quickwit version of the cluster." + ) + private String version; + + @Schema( + title = "Source ID", + description = "Unique identifier of the source within its index." + ) + @JsonProperty("source_id") + private String sourceId; + + @Schema( + title = "Source type", + description = "Type of the source: `kafka`, `kinesis` or `pulsar`." + ) + @JsonProperty("source_type") + private String sourceType; + + @Schema( + title = "Number of pipelines", + description = "Number of indexing pipelines running per node for this source. Defaults to `1`." + ) + @JsonProperty("num_pipelines") + private Integer numPipelines; + + @Schema( + title = "Transform", + description = "VRL transformation applied to incoming documents, if any." + ) + private String transform; + + @Schema( + title = "Source parameters", + description = "Parameters of the source, whose shape depends on the source type." + ) + private Map params; +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/search/Search.java b/src/main/java/io/kestra/plugin/quickwit/search/Search.java new file mode 100644 index 0000000..76f84b2 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/search/Search.java @@ -0,0 +1,348 @@ +package io.kestra.plugin.quickwit.search; + +import java.io.BufferedOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.net.URI; +import java.time.Duration; +import java.util.List; +import java.util.Map; + +import org.slf4j.Logger; + +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Metric; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.executions.metrics.Counter; +import io.kestra.core.models.executions.metrics.Timer; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.models.tasks.common.FetchType; +import io.kestra.core.runners.RunContext; +import io.kestra.core.serializers.FileSerde; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; +import io.kestra.plugin.quickwit.QuickwitService; +import io.kestra.plugin.quickwit.models.SearchQuery; +import io.kestra.plugin.quickwit.models.SearchResult; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; +import reactor.core.publisher.Flux; + +/** + * Searches documents in a Quickwit index. + * + * @see Quickwit search API + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Search Quickwit indexes", + description = """ + Runs a Quickwit search query against an index and returns the matching documents. + + Supports returning every hit, only the first one, or storing all hits to Kestra internal + storage. Only the current result page is returned, so set `maxHits` (and paginate) when you + need more. + """ +) +@Plugin( + metrics = { + @Metric(name = "requests.count", type = Counter.TYPE, description = "Number of search requests sent"), + @Metric(name = "records", type = Counter.TYPE, unit = "records", description = "Number of documents returned"), + @Metric(name = "requests.duration", type = Timer.TYPE, description = "Time taken by Quickwit to process the query") + }, + examples = { + @Example( + title = "Search the errors logged in the last hour", + full = true, + code = """ + id: quickwit_search + namespace: company.team + + tasks: + - id: search + type: io.kestra.plugin.quickwit.search.Search + url: "https://quickwit.example.com:7280" + index: app-logs + query: "severity:ERROR" + startTimestamp: "{{ now() | dateAdd(-1, 'HOURS') | timestamp }}" + maxHits: 100 + fetchType: FETCH + + - id: log_count + type: io.kestra.plugin.core.log.Log + message: "Found {{ outputs.search.size }} errors" + """ + ), + @Example( + title = "Count the errors of each service over the last day, behind a protected gateway", + full = true, + code = """ + id: quickwit_search_aggregations + namespace: company.team + + tasks: + - id: search + type: io.kestra.plugin.quickwit.search.Search + url: "https://quickwit.example.com:7280" + headers: + Authorization: "Bearer {{ secret('QUICKWIT_GATEWAY_TOKEN') }}" + index: app-logs + query: "severity:ERROR" + startTimestamp: "{{ now() | dateAdd(-1, 'DAYS') | timestamp }}" + aggregations: + service: + terms: + field: service + + - id: report + type: io.kestra.plugin.core.log.Log + message: "Top services: {{ outputs.search.aggregations | json }}" + """ + ) + } +) +public class Search extends AbstractQuickwitTask implements RunnableTask { + @Schema( + title = "Index ID", + description = """ + ID of the index to search. + + Quickwit also accepts a multi-target expression such as `app-logs-000001,app-logs-000002` + or a wildcard pattern such as `app-logs*`. + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property index; + + @Schema( + title = "Query", + description = """ + Query text, using the [Quickwit query language](https://quickwit.io/docs/reference/query-language), + for example `severity:ERROR`. + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property query; + + @Schema( + title = "Start timestamp", + description = """ + Restricts the search to documents with `timestamp >= startTimestamp`, in seconds. + Taking advantage of this lets Quickwit prune splits. + """ + ) + @PluginProperty(group = "main") + private Property startTimestamp; + + @Schema( + title = "End timestamp", + description = """ + Restricts the search to documents with `timestamp < endTimestamp`, in seconds. + Taking advantage of this lets Quickwit prune splits. + """ + ) + @PluginProperty(group = "main") + private Property endTimestamp; + + @Schema( + title = "Start offset", + description = "Number of documents to skip. Quickwit defaults to `0`." + ) + @PluginProperty(group = "processing") + private Property startOffset; + + @Schema( + title = "Maximum number of hits", + description = "Maximum number of documents to return. Quickwit defaults to `20`." + ) + @PluginProperty(group = "processing") + private Property maxHits; + + @Schema( + title = "Search fields", + description = """ + Fields to search on when the query does not qualify a field name. + Defaults to the `default_search_fields` of the index configuration. + """ + ) + @PluginProperty(group = "processing") + private Property> searchField; + + @Schema( + title = "Snippet fields", + description = "Fields to extract snippets on. Snippets are returned in the `snippets` output." + ) + @PluginProperty(group = "processing") + private Property> snippetFields; + + @Schema( + title = "Sort by", + description = """ + Fields to sort the results on. Prefix a field with `-` to sort ascending or `+` to sort descending. + You can sort by up to two fast fields, or by the BM25 `_score`. + By default Quickwit returns the most recent documents first. + """ + ) + @PluginProperty(group = "processing") + private Property> sortBy; + + @Schema( + title = "Aggregations", + description = """ + Aggregation request, using the + [aggregations syntax](https://quickwit.io/docs/reference/aggregation), for example + `{"severity": {"terms": {"field": "severity"}}}`. + """ + ) + @PluginProperty(group = "processing") + private Property> aggregations; + + @Schema( + title = "Result handling mode", + description = """ + Controls how hits are exposed in the outputs. + + `FETCH` (default) returns every hit of the current page, `FETCH_ONE` only the first hit, + `STORE` writes all hits to Kestra internal storage and returns a URI, and `NONE` leaves + the row outputs empty. + """ + ) + @Builder.Default + @PluginProperty(group = "processing") + private Property fetchType = Property.ofValue(FetchType.FETCH); + + @Override + public Search.Output run(RunContext runContext) throws Exception { + Logger logger = runContext.logger(); + + SearchQuery query = new SearchQuery( + required(runContext, this.index, "index"), + required(runContext, this.query, "query"), + runContext.render(this.startTimestamp).as(Long.class).orElse(null), + runContext.render(this.endTimestamp).as(Long.class).orElse(null), + QuickwitService.requireAtLeast(runContext.render(this.startOffset).as(Integer.class).orElse(null), 0, "startOffset"), + QuickwitService.requireAtLeast(runContext.render(this.maxHits).as(Integer.class).orElse(null), 0, "maxHits"), + runContext.render(this.searchField).asList(String.class), + runContext.render(this.snippetFields).asList(String.class), + runContext.render(this.sortBy).asList(String.class), + runContext.render(this.aggregations).asMap(String.class, Object.class) + ); + + SearchResult result; + try (var client = client(runContext)) { + result = QuickwitService.search(runContext, client, getUrl(), getHeaders(), query, null); + } + + if (result.getErrors() != null && !result.getErrors().isEmpty()) { + logger.warn("Quickwit reported non-fatal search errors: {}", result.getErrors()); + } + + if (result.getElapsedTimeMicros() != null) { + runContext.metric(Timer.of("requests.duration", Duration.ofNanos(result.getElapsedTimeMicros() * 1_000L))); + } + runContext.metric(Counter.of("requests.count", 1)); + + List> hits = result.getHits() != null ? result.getHits() : List.of(); + runContext.metric(Counter.of("records", hits.size())); + + FetchType fetchType = runContext.render(this.fetchType).as(FetchType.class).orElse(FetchType.FETCH); + + return Output.builder() + .total(result.getNumHits() != null ? result.getNumHits() : (long) hits.size()) + .size(fetchType == FetchType.NONE ? 0 : hits.size()) + .rows(fetchType == FetchType.FETCH ? hits : null) + .row(fetchType == FetchType.FETCH_ONE ? hits.isEmpty() ? null : hits.getFirst() : null) + .uri(fetchType == FetchType.STORE ? store(runContext, hits) : null) + .elapsedTimeMicros(result.getElapsedTimeMicros()) + .snippets(result.getSnippets()) + .aggregations(result.getAggregations()) + .errors(result.getErrors()) + .build(); + } + + /** + * Writes the hits to Kestra internal storage as an ION file. + */ + private URI store(RunContext runContext, List> hits) throws IOException { + File tempFile = runContext.workingDir().createTempFile(".ion").toFile(); + + try (var output = new BufferedOutputStream(new FileOutputStream(tempFile), FileSerde.BUFFER_SIZE)) { + FileSerde.writeAll(output, Flux.fromIterable(hits)).block(); + } + + return runContext.storage().putFile(tempFile); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Returned document count", + description = "Number of documents included in the outputs for the selected fetch type." + ) + private Integer size; + + @Schema( + title = "Total matching documents", + description = "Total number of documents matching the query, regardless of pagination." + ) + private Long total; + + @Schema( + title = "Fetched documents", + description = "Available only when `fetchType=FETCH`; the documents of the current result page." + ) + private List> rows; + + @Schema( + title = "First document", + description = "Available only when `fetchType=FETCH_ONE`; the first matching document." + ) + private Map row; + + @Schema( + title = "Stored documents URI", + description = "Available only when `fetchType=STORE`; path of the ION file holding the documents." + ) + private URI uri; + + @Schema( + title = "Processing time", + description = "Time taken by Quickwit to process the query, in microseconds." + ) + private Long elapsedTimeMicros; + + @Schema( + title = "Snippets", + description = "Extracted snippets, present only when snippet fields were requested." + ) + private List> snippets; + + @Schema( + title = "Aggregations", + description = "Aggregation results, present only when aggregations were requested." + ) + private Map aggregations; + + @Schema( + title = "Search errors", + description = "Non-fatal errors reported by Quickwit while executing the query." + ) + private List errors; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/search/Trigger.java b/src/main/java/io/kestra/plugin/quickwit/search/Trigger.java new file mode 100644 index 0000000..139624c --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/search/Trigger.java @@ -0,0 +1,431 @@ +package io.kestra.plugin.quickwit.search; + +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.OffsetDateTime; +import java.time.format.DateTimeParseException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import org.slf4j.Logger; + +import io.kestra.core.exceptions.IllegalVariableEvaluationException; +import io.kestra.core.http.client.HttpClient; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.conditions.ConditionContext; +import io.kestra.core.models.executions.Execution; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.triggers.AbstractTrigger; +import io.kestra.core.models.triggers.PollingTriggerInterface; +import io.kestra.core.models.triggers.TriggerContext; +import io.kestra.core.models.triggers.TriggerOutput; +import io.kestra.core.models.triggers.TriggerService; +import io.kestra.core.runners.RunContext; +import io.kestra.core.storages.kv.KVMetadata; +import io.kestra.core.storages.kv.KVValueAndMetadata; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; +import io.kestra.plugin.quickwit.QuickwitService; +import io.kestra.plugin.quickwit.models.SearchQuery; +import io.kestra.plugin.quickwit.models.SearchResult; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Fires a flow when new documents matching a query show up in a Quickwit index. + * + *

The trigger polls Quickwit on every {@code interval}. It remembers, in the namespace KV Store, the + * timestamp up to which documents have already been delivered, and each poll searches + * {@code timestamp >= watermark} before advancing it to {@code max(delivered timestamp) + 1}. + * Polls sort ascending on {@code timestampField} so documents beyond {@code maxHits} are delivered on + * a later poll instead of being skipped. + * + *

Late documents whose event timestamp is already below the watermark are not picked up again, + * and documents sharing the boundary second of a burst larger than {@code maxHits} may be skipped: + * raise {@code maxHits} when bursts share a timestamp. + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Trigger on new Quickwit documents", + description = """ + Polls a Quickwit index on a fixed interval and fires an execution as soon as documents matching + the query appear. + + The last delivered timestamp is kept in the namespace KV Store and advanced to one past the + maximum delivered timestamp on every successful poll, so a given document is only ever + delivered once. + """ +) +@Plugin( + examples = { + @Example( + title = "Alert when new errors are logged", + full = true, + code = """ + id: quickwit_error_watch + namespace: company.team + + triggers: + - id: on_errors + type: io.kestra.plugin.quickwit.search.Trigger + url: "https://quickwit.example.com:7280" + index: app-logs + query: "severity:ERROR" + timestampField: timestamp + interval: PT5M + + tasks: + - id: notify + type: io.kestra.plugin.core.log.Log + message: "New errors: {{ trigger.documents | length }}" + """ + ), + @Example( + title = "React to new errors on a Quickwit cluster protected by a gateway", + full = true, + code = """ + id: quickwit_error_watch_private + namespace: company.team + + triggers: + - id: on_errors + type: io.kestra.plugin.quickwit.search.Trigger + url: "https://quickwit.example.com:7280" + headers: + Authorization: "Bearer {{ secret('QUICKWIT_GATEWAY_TOKEN') }}" + index: app-logs + query: "severity:ERROR AND service:checkout" + timestampField: timestamp + interval: PT5M + maxHits: 500 + + tasks: + - id: notify + type: io.kestra.plugin.core.log.Log + message: "New checkout errors: {{ trigger.documents | length }}" + """ + ) + } +) +public class Trigger extends AbstractTrigger implements PollingTriggerInterface, TriggerOutput { + private static final String WATERMARK_DESCRIPTION = "Last timestamp delivered by the Quickwit search trigger"; + + @Builder.Default + private Duration interval = Duration.ofSeconds(60); + + @Schema( + title = "Quickwit URL", + description = """ + Base URL of the Quickwit REST API, including the scheme. The `api/v1` prefix is added automatically. + A local node listens on port 7280, e.g. `http://localhost:7280`. + """ + ) + @PluginProperty(group = "connection") + private Property url; + + @Schema( + title = "Basic authentication", + description = """ + Optional HTTP basic authentication credentials. + + Quickwit itself has no authentication layer, so this is only needed when the cluster sits + behind a reverse proxy or an API gateway that enforces basic auth. + """ + ) + @Valid + @ToString.Exclude + @PluginProperty(group = "connection") + private AbstractQuickwitTask.BasicAuth basicAuth; + + @Schema( + title = "Custom HTTP headers", + description = """ + Headers sent on every poll, for example an `Authorization: Bearer ...` token or a gateway API key. + + Quickwit itself has no authentication layer, so this is only needed when the cluster sits + behind an API gateway that injects credentials. + """ + ) + @ToString.Exclude + @EqualsAndHashCode.Exclude + @PluginProperty(group = "connection", secret = true) + private Property> headers; + + @Schema( + title = "Connect timeout", + description = """ + Maximum time to wait to establish a connection to Quickwit, e.g. `PT10S`. + When unset, the default of the underlying HTTP client applies. + """ + ) + @PluginProperty(group = "execution") + private Property connectTimeout; + + @Schema( + title = "Read timeout", + description = """ + Maximum time to keep waiting for data on an idle connection, e.g. `PT5M`. + Raise it when the index is large enough for a poll to take a while. + """ + ) + @PluginProperty(group = "execution") + private Property readTimeout; + + @Schema( + title = "Index ID", + description = "ID of the index to watch." + ) + @PluginProperty(group = "main") + private Property index; + + @Schema( + title = "Query", + description = """ + Query selecting the documents to react to, using the + [Quickwit query language](https://quickwit.io/docs/reference/query-language), for example `severity:ERROR`. + """ + ) + @PluginProperty(group = "main") + private Property query; + + @Schema( + title = "Timestamp field", + description = """ + Document field holding the event timestamp, used to advance the watermark. + This should be the timestamp field of the index so `start_timestamp` pruning aligns. + Its `output_format` must be `rfc3339` (the default) or `unix_timestamp_secs`, the two formats the watermark can parse. + Polls sort ascending on this field and advance to one past the maximum delivered value. + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property timestampField; + + @Schema( + title = "Maximum number of hits", + description = """ + Maximum number of documents to fetch per poll. Quickwit defaults to `20`. + + Raise it when documents are ingested in bursts. Polls sort ascending on `timestampField`, + so documents beyond this limit are delivered on a later poll instead of being skipped, + unless a single second holds more documents than this limit. + """ + ) + @PluginProperty(group = "processing") + private Property maxHits; + + @Schema( + title = "State key", + description = """ + KV Store key holding the watermark of this trigger. + Defaults to ``, `` and ``, each prefixed by its length, e.g. `7-company_7-my_flow_2-on`. + Change it only to deliberately share a watermark between triggers. + """ + ) + @PluginProperty(group = "advanced") + private Property stateKey; + + @Override + public Optional evaluate(ConditionContext conditionContext, TriggerContext context) throws Exception { + RunContext runContext = conditionContext.getRunContext(); + Logger logger = runContext.logger(); + + String rTimestampField = QuickwitService.requireNonBlank( + runContext.render(this.timestampField).as(String.class).orElse(null), + "timestampField" + ); + SearchQuery query = searchQuery(runContext, rTimestampField); + String key = stateKey(runContext, context); + + // absent watermark => first poll, which deliberately returns the whole current result set + Optional watermark = readWatermark(runContext, key); + + SearchResult result; + var configuration = QuickwitService.httpConfiguration(runContext, this.connectTimeout, this.readTimeout, this.basicAuth); + + try (var client = new HttpClient(runContext, configuration)) { + result = QuickwitService.search(runContext, client, this.url, this.headers, query, watermark.orElse(null)); + } + + List> documents = result.getHits() != null ? result.getHits() : List.of(); + if (documents.isEmpty()) { + logger.debug("No new document matching the query on index '{}'", query.index()); + return Optional.empty(); + } + + long max = maxTimestamp(documents, rTimestampField); + long advanced = max + 1; + + // When Quickwit truncated the page, it may end in the middle of second `max`: hold that second + // back and re-read it whole on the next poll, unless the page holds nothing but that second. + if (result.getNumHits() != null && result.getNumHits() > documents.size()) { + List> complete = new ArrayList<>(); + for (int i = 0; i < documents.size(); i++) { + if (timestamp(documents.get(i), rTimestampField, i) < max) { + complete.add(documents.get(i)); + } + } + if (!complete.isEmpty()) { + documents = complete; + advanced = max; + } + } + + logger.info("Triggering on {} new document(s) on index '{}'", documents.size(), query.index()); + + Output output = Output.builder() + .documents(documents) + .numHits(result.getNumHits() != null ? result.getNumHits() : (long) documents.size()) + .elapsedTimeMicros(result.getElapsedTimeMicros()) + .watermark(advanced) + .build(); + + // generate the execution first so a failure there does not advance the watermark past undelivered documents + Execution execution = TriggerService.generateExecution(this, conditionContext, context, output); + writeWatermark(runContext, key, advanced); + + return Optional.of(execution); + } + + private SearchQuery searchQuery(RunContext runContext, String rTimestampField) throws IllegalVariableEvaluationException { + String rIndex = QuickwitService.requireNonBlank(runContext.render(this.index).as(String.class).orElse(null), "index"); + String rQuery = QuickwitService.requireNonBlank(runContext.render(this.query).as(String.class).orElse(null), "query"); + Integer rMaxHits = QuickwitService.requireAtLeast(runContext.render(this.maxHits).as(Integer.class).orElse(null), 1, "maxHits"); + + return new SearchQuery(rIndex, rQuery, null, null, null, rMaxHits, null, null, List.of("-" + rTimestampField), null); + } + + private String stateKey(RunContext runContext, TriggerContext context) throws IllegalVariableEvaluationException { + return runContext.render(this.stateKey) + .as(String.class) + .orElseGet(() -> defaultKey(context.getNamespace(), context.getFlowId(), this.id)); + } + + /** + * Reads the last delivered timestamp. + * + *

A missing watermark means the next poll starts from the beginning. Any other KV failure + * fails the evaluation so a poll never falls back to a full re-scan during a KV outage. + */ + private Optional readWatermark(RunContext runContext, String key) throws Exception { + try { + return runContext.namespaceKv(runContext.flowInfo().namespace()) + .getValue(key) + .map(value -> Long.parseLong(new String((byte[]) value.value(), StandardCharsets.UTF_8).trim())); + } catch (IllegalArgumentException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Unable to read the Quickwit search trigger watermark '" + key + "', failing the poll: " + e.getMessage(), e); + } + } + + /** + * Maximum event timestamp across the delivered documents. + * + * @throws IllegalStateException when a document misses the timestamp field or holds an unparsable value + */ + private static long maxTimestamp(List> documents, String timestampField) { + long max = Long.MIN_VALUE; + + for (int i = 0; i < documents.size(); i++) { + max = Math.max(max, timestamp(documents.get(i), timestampField, i)); + } + + return max; + } + + /** + * Event timestamp of a document, in seconds. + * + *

Quickwit renders datetime fields with their {@code output_format}: an RFC 3339 string by default, or + * a number for {@code unix_timestamp_secs}. Other numeric output formats (millis, micros, nanos) are not + * supported since they cannot be told apart from seconds. + */ + private static long timestamp(Map document, String timestampField, int position) { + Object value = document.get(timestampField); + if (value == null) { + throw new IllegalStateException("Document at position " + position + " misses timestamp field '" + timestampField + "', cannot advance the watermark"); + } + + if (value instanceof Number number) { + return number.longValue(); + } + + String text = String.valueOf(value).trim(); + try { + return OffsetDateTime.parse(text).toEpochSecond(); + } catch (DateTimeParseException e) { + try { + return Long.parseLong(text); + } catch (NumberFormatException ignored) { + throw new IllegalStateException( + "Document at position " + position + " holds timestamp field '" + timestampField + "' = '" + text + + "', expected an RFC 3339 datetime or a number of seconds; set the field's `output_format` to `rfc3339` or `unix_timestamp_secs`", + e + ); + } + } + } + + private void writeWatermark(RunContext runContext, String key, long watermark) throws Exception { + runContext.namespaceKv(runContext.flowInfo().namespace()).put( + key, + new KVValueAndMetadata( + new KVMetadata(WATERMARK_DESCRIPTION, (Duration) null), + Long.toString(watermark).getBytes(StandardCharsets.UTF_8) + ) + ); + } + + /** + * Default watermark key. Each part is length-prefixed so that, for example, namespace {@code a_b} with + * flow {@code c} and namespace {@code a} with flow {@code b_c} get different keys. Only {@code -} and + * {@code _} are used as separators since a KV key must match {@code [a-zA-Z0-9][a-zA-Z0-9._-]*}. + */ + private static String defaultKey(String namespace, String flowId, String triggerId) { + return String.format("%d-%s_%d-%s_%d-%s", namespace.length(), namespace, flowId.length(), flowId, triggerId.length(), triggerId); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "New documents", + description = "Documents matching the query that no previous poll had delivered." + ) + private List> documents; + + @Schema( + title = "Total matching documents", + description = "Total number of documents matching the query over the searched window." + ) + private Long numHits; + + @Schema( + title = "Processing time", + description = "Time taken by Quickwit to process the query, in microseconds." + ) + private Long elapsedTimeMicros; + + @Schema( + title = "Watermark", + description = "Timestamp up to which documents have now been delivered, in seconds." + ) + private Long watermark; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/search/package-info.java b/src/main/java/io/kestra/plugin/quickwit/search/package-info.java new file mode 100644 index 0000000..2b5fb84 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/search/package-info.java @@ -0,0 +1,8 @@ +@PluginSubGroup( + title = "Search", + description = "Query Quickwit indexes and react to new documents.", + categories = PluginSubGroup.PluginCategory.DATA +) +package io.kestra.plugin.quickwit.search; + +import io.kestra.core.models.annotations.PluginSubGroup; \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/source/AbstractQuickwitSource.java b/src/main/java/io/kestra/plugin/quickwit/source/AbstractQuickwitSource.java new file mode 100644 index 0000000..a8a89f4 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/source/AbstractQuickwitSource.java @@ -0,0 +1,44 @@ +package io.kestra.plugin.quickwit.source; + +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.AbstractQuickwitTask; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Source management tasks, sharing the index ID and source ID. + * + * @see Quickwit source API + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +public abstract class AbstractQuickwitSource extends AbstractQuickwitTask { + @Schema( + title = "Index ID", + description = "ID of the index the source belongs to." + ) + @NotNull + @PluginProperty(group = "main") + private Property index; + + @Schema( + title = "Source ID", + description = """ + ID of the source, unique within its index. See the + [source ID validation rules](https://quickwit.io/docs/configuration/source-config). + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property source; +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/source/Create.java b/src/main/java/io/kestra/plugin/quickwit/source/Create.java new file mode 100644 index 0000000..5af0623 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/source/Create.java @@ -0,0 +1,173 @@ +package io.kestra.plugin.quickwit.source; + +import java.util.LinkedHashMap; +import java.util.Map; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.models.Source; + +import io.swagger.v3.oas.annotations.media.Schema; +import jakarta.validation.constraints.NotNull; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Attaches a source to a Quickwit index. + * + * @see Quickwit source configuration + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Create a Quickwit source", + description = """ + Attaches a streaming source, for example Kafka, Kinesis or Pulsar, to an index so Quickwit indexes + it continuously. + """ +) +@Plugin( + examples = { + @Example( + title = "Index a Kafka topic into an existing index", + full = true, + code = """ + id: quickwit_create_source + namespace: company.team + + tasks: + - id: create + type: io.kestra.plugin.quickwit.source.Create + url: "http://localhost:7280" + index: app-logs + source: kafka-source + configVersion: "0.8" + sourceType: kafka + numPipelines: 2 + params: + topic: app-logs + client_params: + bootstrap.servers: "kafka:9092" + + - id: report + type: io.kestra.plugin.core.log.Log + message: "Source {{ outputs.create.source.sourceId }} attached" + """ + ) + } +) +public class Create extends AbstractQuickwitSource implements RunnableTask { + @Schema( + title = "Configuration format version", + description = """ + Version of the source configuration format, which must match the version of your Quickwit + cluster, for example `0.8`. + + Named `configVersion` and not `version` because `version` is reserved by Kestra to pin a + plugin version. + """ + ) + @PluginProperty(group = "main") + private Property configVersion; + + @Schema( + title = "Source type", + description = "Type of the source: `kafka`, `kinesis` or `pulsar`. It cannot be changed afterwards." + ) + @NotNull + @PluginProperty(group = "main") + private Property sourceType; + + @Schema( + title = "Source parameters", + description = """ + Parameters of the source, whose shape depends on the source type. See the + [source configuration documentation](https://quickwit.io/docs/configuration/source-config). + """ + ) + @NotNull + @PluginProperty(group = "main") + private Property> params; + + @Schema( + title = "Number of pipelines", + description = "Number of indexing pipelines running per node for this source. Quickwit defaults to `1`." + ) + @PluginProperty(group = "advanced") + private Property numPipelines; + + @Schema( + title = "Transform", + description = """ + A [VRL](https://vector.dev/docs/reference/vrl/) transformation applied to incoming documents, + for example `.message = downcase!(.message)`. + """ + ) + @PluginProperty(group = "advanced") + private Property transform; + + @Override + public Create.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + String rSource = required(runContext, getSource(), "source"); + Map params = requiredMap( + runContext, + this.params, + "params", + "https://quickwit.io/docs/configuration/source-config" + ); + + Map configuration = new LinkedHashMap<>(); + putIfPresent(configuration, "version", runContext.render(this.configVersion).as(String.class).orElse(null)); + configuration.put("source_id", rSource); + configuration.put("source_type", required(runContext, this.sourceType, "sourceType")); + configuration.put("params", params); + putIfPresent(configuration, "num_pipelines", runContext.render(this.numPipelines).as(Integer.class).orElse(null)); + putIfPresent(configuration, "transform", runContext.render(this.transform).as(String.class).orElse(null)); + + HttpRequest request = request( + runContext, + "POST", + "indexes/" + pathSegment(rIndex) + "/sources", + HttpRequest.JsonRequestBody.of(configuration) + ).build(); + + Source source; + try (var client = client(runContext)) { + source = execute(client, request, Source.class, "creation of source '" + rSource + "' on index '" + rIndex + "'"); + } + + return Output.builder() + .index(rIndex) + .source(source) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index the source was attached to." + ) + private String index; + + @Schema( + title = "Source", + description = "The source configuration as Quickwit created it." + ) + private Source source; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/source/Delete.java b/src/main/java/io/kestra/plugin/quickwit/source/Delete.java new file mode 100644 index 0000000..f59312d --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/source/Delete.java @@ -0,0 +1,97 @@ +package io.kestra.plugin.quickwit.source; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Detaches a source from a Quickwit index. + * + * @see Delete a source + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Delete a Quickwit source", + description = """ + Deletes a source from an index. Documents already indexed are kept; only the source and its checkpoint + are removed. + """ +) +@Plugin( + examples = { + @Example( + title = "Detach a retired Kafka topic from an index", + full = true, + code = """ + id: quickwit_delete_source + namespace: company.team + + tasks: + - id: delete + type: io.kestra.plugin.quickwit.source.Delete + url: "http://localhost:7280" + index: app-logs + source: legacy-source + + - id: report + type: io.kestra.plugin.core.log.Log + message: "Source {{ outputs.delete.source }} detached from {{ outputs.delete.index }}" + """ + ) + } +) +public class Delete extends AbstractQuickwitSource implements RunnableTask { + @Override + public Delete.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + String rSource = required(runContext, getSource(), "source"); + + HttpRequest request = request( + runContext, + "DELETE", + "indexes/" + pathSegment(rIndex) + "/sources/" + pathSegment(rSource), + null + ).build(); + + try (var client = client(runContext)) { + executeIgnoringBody(client, request, "deletion of source '" + rSource + "' on index '" + rIndex + "'"); + } + + runContext.logger().info("Deleted source '{}' from index '{}'", rSource, rIndex); + + return Output.builder() + .index(rIndex) + .source(rSource) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index the source was removed from." + ) + private String index; + + @Schema( + title = "Source ID", + description = "ID of the source that was deleted." + ) + private String source; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/source/ResetCheckpoint.java b/src/main/java/io/kestra/plugin/quickwit/source/ResetCheckpoint.java new file mode 100644 index 0000000..79cb147 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/source/ResetCheckpoint.java @@ -0,0 +1,100 @@ +package io.kestra.plugin.quickwit.source; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Resets the checkpoint of a Quickwit source, so it re-reads its data from the beginning. + * + * @see Reset source checkpoint + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Reset a Quickwit source checkpoint", + description = """ + Resets the checkpoint of a source so it re-indexes its data from the beginning of the stream. + + Use it after a doc mapping change to replay already ingested data. Quickwit answers with an empty + body, which is expected. + """ +) +@Plugin( + examples = { + @Example( + title = "Replay a source after changing the doc mapping", + full = true, + code = """ + id: quickwit_reset_checkpoint + namespace: company.team + + tasks: + - id: reset + type: io.kestra.plugin.quickwit.source.ResetCheckpoint + url: "http://localhost:7280" + index: app-logs + source: kafka-source + + - id: check + type: io.kestra.plugin.quickwit.index.Get + url: "http://localhost:7280" + index: app-logs + """ + ) + } +) +public class ResetCheckpoint extends AbstractQuickwitSource implements RunnableTask { + @Override + public ResetCheckpoint.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + String rSource = required(runContext, getSource(), "source"); + + HttpRequest request = request( + runContext, + "PUT", + "indexes/" + pathSegment(rIndex) + "/sources/" + pathSegment(rSource) + "/reset-checkpoint", + null + ).build(); + + try (var client = client(runContext)) { + executeIgnoringBody(client, request, "checkpoint reset of source '" + rSource + "' on index '" + rIndex + "'"); + } + + runContext.logger().info("Reset the checkpoint of source '{}' on index '{}'", rSource, rIndex); + + return Output.builder() + .index(rIndex) + .source(rSource) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index the source belongs to." + ) + private String index; + + @Schema( + title = "Source ID", + description = "ID of the source whose checkpoint was reset." + ) + private String source; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/source/Toggle.java b/src/main/java/io/kestra/plugin/quickwit/source/Toggle.java new file mode 100644 index 0000000..8b451e0 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/source/Toggle.java @@ -0,0 +1,122 @@ +package io.kestra.plugin.quickwit.source; + +import java.util.Map; + +import io.kestra.core.http.HttpRequest; +import io.kestra.core.models.annotations.Example; +import io.kestra.core.models.annotations.Plugin; +import io.kestra.core.models.annotations.PluginProperty; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.RunnableTask; +import io.kestra.core.runners.RunContext; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.ToString; +import lombok.experimental.SuperBuilder; + +/** + * Enables or disables a Quickwit source without deleting it. + * + * @see Toggle source + */ +@SuperBuilder +@ToString +@EqualsAndHashCode +@Getter +@NoArgsConstructor +@Schema( + title = "Toggle a Quickwit source", + description = """ + Enables or disables a source. A disabled source stops indexing without losing its checkpoint, so + re-enabling it resumes where it left off. Quickwit answers with an empty body, which is expected. + """ +) +@Plugin( + examples = { + @Example( + title = "Pause an ingestion pipeline during a maintenance window", + full = true, + code = """ + id: quickwit_toggle_source + namespace: company.team + + tasks: + - id: pause + type: io.kestra.plugin.quickwit.source.Toggle + url: "http://localhost:7280" + index: app-logs + source: kafka-source + enable: false + + - id: resume + type: io.kestra.plugin.quickwit.source.Toggle + url: "http://localhost:7280" + index: app-logs + source: kafka-source + enable: true + """ + ) + } +) +public class Toggle extends AbstractQuickwitSource implements RunnableTask { + @Schema( + title = "Enable", + description = "`true` to let Quickwit index from this source, `false` to stop indexing without deleting its checkpoint." + ) + @PluginProperty(group = "main") + private Property enable; + + @Override + public Toggle.Output run(RunContext runContext) throws Exception { + String rIndex = required(runContext, getIndex(), "index"); + String rSource = required(runContext, getSource(), "source"); + boolean enableSource = runContext.render(this.enable).as(Boolean.class).orElseThrow( + () -> new IllegalArgumentException("`enable` is required and must be `true` or `false`") + ); + + HttpRequest request = request( + runContext, + "PUT", + "indexes/" + pathSegment(rIndex) + "/sources/" + pathSegment(rSource) + "/toggle", + HttpRequest.JsonRequestBody.of(Map.of("enable", enableSource)) + ).build(); + + try (var client = client(runContext)) { + executeIgnoringBody(client, request, "toggling of source '" + rSource + "' on index '" + rIndex + "'"); + } + + runContext.logger().info("{} source '{}' on index '{}'", enableSource ? "Enabled" : "Disabled", rSource, rIndex); + + return Output.builder() + .index(rIndex) + .source(rSource) + .enable(enableSource) + .build(); + } + + @Builder + @Getter + public static class Output implements io.kestra.core.models.tasks.Output { + @Schema( + title = "Index ID", + description = "ID of the index the source belongs to." + ) + private String index; + + @Schema( + title = "Source ID", + description = "ID of the source that was toggled." + ) + private String source; + + @Schema( + title = "Enabled", + description = "`true` when the source was enabled, `false` when it was disabled." + ) + private Boolean enable; + } +} \ No newline at end of file diff --git a/src/main/java/io/kestra/plugin/quickwit/source/package-info.java b/src/main/java/io/kestra/plugin/quickwit/source/package-info.java new file mode 100644 index 0000000..c4b6b13 --- /dev/null +++ b/src/main/java/io/kestra/plugin/quickwit/source/package-info.java @@ -0,0 +1,8 @@ +@PluginSubGroup( + title = "Sources", + description = "Manage the sources feeding a Quickwit index.", + categories = PluginSubGroup.PluginCategory.DATA +) +package io.kestra.plugin.quickwit.source; + +import io.kestra.core.models.annotations.PluginSubGroup; \ No newline at end of file diff --git a/src/main/resources/doc/io.kestra.plugin.quickwit.md b/src/main/resources/doc/io.kestra.plugin.quickwit.md index 778ea68..795e727 100644 --- a/src/main/resources/doc/io.kestra.plugin.quickwit.md +++ b/src/main/resources/doc/io.kestra.plugin.quickwit.md @@ -1,16 +1,223 @@ -This is the Kestra plugin template. Use it as a starting point for building a new plugin. +# Quickwit -## What this template ships +[Quickwit](https://quickwit.io) is a cloud-native search engine for logs, traces and other immutable data. This plugin +exposes its REST API as typed Kestra tasks, so a flow can search indexes, ingest documents and manage indexes, sources and +delete tasks without hand-written HTTP calls. -- `Example` is a sample `RunnableTask` that reverses an input string. -- `Trigger` is a sample polling trigger that fires an execution at random. +## What it ships -## How to build your plugin +| Package | Tasks | +|---|---| +| `search` | `Search`, `Trigger` | +| `ingest` | `Ingest` | +| `index` | `Create`, `Get`, `List`, `Delete`, `Clear` | +| `source` | `Create`, `Toggle`, `Delete`, `ResetCheckpoint` | +| `deletetask` | `Create`, `List` | -1. Rename the package `io.kestra.plugin.quickwit` to your own, for example `io.kestra.plugin.myservice`. -2. Update `group`, `name`, `title`, and `description` in `src/main/resources/metadata/index.yaml`. -3. Replace `src/main/resources/icons/plugin-icon.svg` with your service's icon. -4. Replace the `Example` and `Trigger` classes with your real tasks and triggers. -5. Replace this how-to with documentation for your plugin. +## Connecting -Run `./gradlew lintPluginDocs` before pushing to validate the plugin documentation. +Every task and the trigger take the same connection properties, declared flat on the task: + +| Property | Description | +|---|---| +| `url` | Base URL of the REST API, scheme included. The `api/v1` prefix is added for you. | +| `basicAuth` | Optional HTTP basic credentials, for clusters behind a reverse proxy. | +| `headers` | Optional headers sent on every request, for example a gateway bearer token. | +| `connectTimeout` / `readTimeout` | ISO-8601 durations, e.g. `PT10S`. | + +A local node listens on **port 7280**, so the default self-hosted URL is `http://localhost:7280`. + +### Authentication + +Quickwit itself has **no authentication layer**: its REST server ships CORS, compression and tracing only, and its OpenAPI +document declares no security scheme. Authentication is something you add in front of the cluster. That is why `basicAuth` +and `headers` are both optional: + +- leave them unset when the REST API is reachable directly, on a trusted network; +- set `basicAuth` when a reverse proxy (nginx, Envoy) enforces HTTP basic auth; +- set `headers` when an API gateway injects credentials, for example `Authorization: Bearer ...` or `X-API-Key`. + +Keep credentials in secrets, never in the flow itself: + +```yaml +headers: + Authorization: "Bearer {{ secret('QUICKWIT_GATEWAY_TOKEN') }}" +``` + +## Searching + +```yaml +id: quickwit_search +namespace: company.team + +tasks: + - id: search + type: io.kestra.plugin.quickwit.search.Search + url: "https://quickwit.example.com:7280" + index: app-logs + query: "severity:ERROR" + startTimestamp: "{{ now() | dateAdd(-1, 'HOURS') | timestamp }}" + maxHits: 100 + fetchType: FETCH + + - id: log_count + type: io.kestra.plugin.core.log.Log + message: "Found {{ outputs.search.size }} errors" +``` + +`index` also accepts a [multi-target expression](https://quickwit.io/docs/reference/rest-api#multi-target-syntax) such as +`app-logs-000001,app-logs-000002` or a wildcard such as `app-logs*`. + +Only the current result page comes back, so set `maxHits` when you need more. Use `fetchType` to choose between `FETCH` +(every hit), `FETCH_ONE` (the first hit), `STORE` (hits written to Kestra internal storage, returned as a URI) and `NONE`. + +## Reacting to new documents + +`search.Trigger` polls the index on an interval. It sorts ascending on `timestampField` and stores the timestamp up to which +documents have been delivered in the **namespace KV Store**. That watermark advances only after the execution is created, so a +document is delivered once. Documents that arrive late, with an event timestamp already below the watermark, are not picked up. + +```yaml +id: quickwit_error_watch +namespace: company.team + +triggers: + - id: on_errors + type: io.kestra.plugin.quickwit.search.Trigger + url: "https://quickwit.example.com:7280" + index: app-logs + query: "severity:ERROR" + timestampField: timestamp + interval: PT5M + +tasks: + - id: notify + type: io.kestra.plugin.core.log.Log + message: "New errors: {{ trigger.documents | length }}" +``` + +The first poll returns everything currently matching, which is usually what you want for a first run. Override `stateKey` +only if you deliberately want two triggers to share a watermark. + +## Ingesting + +Quickwit only accepts NDJSON on the ingest API, so this task serializes whatever it is given. `from` accepts a single +document, a list of documents, a `kestra://` internal storage URI, a local `file://` path, or a JSON string. + +```yaml +id: quickwit_ingest +namespace: company.team + +inputs: + - id: file + type: FILE + +tasks: + - id: ingest + type: io.kestra.plugin.quickwit.ingest.Ingest + url: "https://quickwit.example.com:7280" + index: app-logs + from: "{{ inputs.file }}" + commit: WAIT_FOR +``` + +Two things worth knowing: + +- **Visibility.** With the default `commit: AUTO`, documents are queued but not searchable yet. Use `WAIT_FOR` to wait for + the next split commit, or `FORCE` to commit immediately and wait. `FORCE` is what you want when a search runs right after + the ingest; it costs performance on small batches. +- **Batch size.** The payload is capped at 10 MB by default. Send a dataset larger than that across several runs, or use a + Quickwit source instead. + +Set `detailedResponse: true` when documents are rejected and you need to know why: Quickwit then returns a `parseFailures` +entry per rejected document, which usually points at a doc mapping mismatch. + +## Managing indexes + +```yaml +tasks: + - id: create + type: io.kestra.plugin.quickwit.index.Create + url: "http://localhost:7280" + index: app-logs + configVersion: "0.8" + docMapping: + timestamp_field: timestamp + field_mappings: + - name: timestamp + type: datetime + input_formats: [unix_timestamp] + fast: true + - name: message + type: text + retention: + period: "30 days" + schedule: "@daily" +``` + +`configVersion` is named that way because `version` is reserved by Kestra to pin a plugin version. It must match the version +of your Quickwit cluster. + +`Clear` drops every split and resets the source checkpoints while keeping the definition; `Delete` removes the index and its +split files for good. + +Note that Quickwit also accepts an index configuration written as YAML, but this task sends JSON, which cannot carry YAML +comments or anchors. + +## Managing sources + +`source.Create` attaches a Kafka, Kinesis or Pulsar source to an index. `source.Toggle` pauses and resumes indexing without +losing the checkpoint, which is the right tool for a maintenance window. `source.ResetCheckpoint` replays a source from the +beginning, for example after a doc mapping change. + +## Deleting documents + +`deletetask.Create` queues a delete task that removes every document matching a query. The call returns as soon as the task is +queued; the cluster's janitor applies it asynchronously. + +```yaml +- id: delete + type: io.kestra.plugin.quickwit.deletetask.Create + url: "http://localhost:7280" + index: app-logs + query: "*" + startTimestamp: "{{ now() | dateAdd(-30, 'DAYS') | timestamp }}" + endTimestamp: "{{ now() | dateAdd(-29, 'DAYS') | timestamp }}" +``` + +## Errors + +Failed requests surface Quickwit's own `message` field, so a doc mapping mismatch or a missing index reads as a Quickwit +error rather than an opaque HTTP status: + +``` +Quickwit ingest into index 'app-logs' failed with HTTP 400: doc mapping does not contain field 'timestamp' +``` + +## Requirements + +Quickwit serves each endpoint from a specific service, so a single node may not answer all of them: + +- the **ingest** API needs a node running an indexer; +- the **search** API needs a node running a searcher. + +Point `url` at a node that serves what you call, or at a load balancer in front of the cluster. + +## Running a local cluster + +```bash +# the image entrypoint is a bare `quickwit`, so the `run` subcommand is required +docker run -p 7280:7280 quickwit/quickwit:latest run +``` + +Then verify the API is up before running a flow: + +```bash +curl http://localhost:7280/api/v1/cluster +``` + +## Documentation + +- [Quickwit documentation](https://quickwit.io/docs) +- [Quickwit REST API reference](https://quickwit.io/docs/reference/rest-api) +- [Quickwit query language](https://quickwit.io/docs/reference/query-language) \ No newline at end of file diff --git a/src/main/resources/icons/io.kestra.plugin.quickwit.deletetask.svg b/src/main/resources/icons/io.kestra.plugin.quickwit.deletetask.svg new file mode 100644 index 0000000..bde7390 --- /dev/null +++ b/src/main/resources/icons/io.kestra.plugin.quickwit.deletetask.svg @@ -0,0 +1,12 @@ + + Quickwit + + + + + + + + + + \ No newline at end of file diff --git a/src/main/resources/icons/io.kestra.plugin.quickwit.index.svg b/src/main/resources/icons/io.kestra.plugin.quickwit.index.svg new file mode 100644 index 0000000..bde7390 --- /dev/null +++ b/src/main/resources/icons/io.kestra.plugin.quickwit.index.svg @@ -0,0 +1,12 @@ + + Quickwit + + + + + + + + + + \ No newline at end of file diff --git a/src/main/resources/icons/io.kestra.plugin.quickwit.ingest.svg b/src/main/resources/icons/io.kestra.plugin.quickwit.ingest.svg new file mode 100644 index 0000000..bde7390 --- /dev/null +++ b/src/main/resources/icons/io.kestra.plugin.quickwit.ingest.svg @@ -0,0 +1,12 @@ + + Quickwit + + + + + + + + + + \ No newline at end of file diff --git a/src/main/resources/icons/io.kestra.plugin.quickwit.search.svg b/src/main/resources/icons/io.kestra.plugin.quickwit.search.svg new file mode 100644 index 0000000..bde7390 --- /dev/null +++ b/src/main/resources/icons/io.kestra.plugin.quickwit.search.svg @@ -0,0 +1,12 @@ + + Quickwit + + + + + + + + + + \ No newline at end of file diff --git a/src/main/resources/icons/io.kestra.plugin.quickwit.source.svg b/src/main/resources/icons/io.kestra.plugin.quickwit.source.svg new file mode 100644 index 0000000..bde7390 --- /dev/null +++ b/src/main/resources/icons/io.kestra.plugin.quickwit.source.svg @@ -0,0 +1,12 @@ + + Quickwit + + + + + + + + + + \ No newline at end of file diff --git a/src/main/resources/icons/plugin-icon.svg b/src/main/resources/icons/plugin-icon.svg index d97a719..bde7390 100644 --- a/src/main/resources/icons/plugin-icon.svg +++ b/src/main/resources/icons/plugin-icon.svg @@ -1 +1,12 @@ - + + Quickwit + + + + + + + + + + \ No newline at end of file diff --git a/src/main/resources/metadata/deletetask.yaml b/src/main/resources/metadata/deletetask.yaml new file mode 100644 index 0000000..bbfa1a7 --- /dev/null +++ b/src/main/resources/metadata/deletetask.yaml @@ -0,0 +1,8 @@ +group: io.kestra.plugin.quickwit.deletetask +name: "deletetask" +title: "Quickwit Delete Tasks" +description: "Delete documents from a Quickwit index by query, and inspect the delete tasks queued on an index." +body: "" +videos: [] +createdBy: "Kestra Core Team" +managedBy: "Kestra Core Team" \ No newline at end of file diff --git a/src/main/resources/metadata/index.yaml b/src/main/resources/metadata/index.yaml index be24938..f1f9955 100644 --- a/src/main/resources/metadata/index.yaml +++ b/src/main/resources/metadata/index.yaml @@ -1,8 +1,8 @@ group: io.kestra.plugin.quickwit name: "quickwit" title: "Quickwit" -description: "Quickwit plugin for Kestra" +description: "Search, ingest and manage indexes of a Quickwit cluster: the cloud-native search engine for logs and traces." body: "" videos: [] createdBy: "Kestra Core Team" -managedBy: "Kestra Core Team" +managedBy: "Kestra Core Team" \ No newline at end of file diff --git a/src/main/resources/metadata/ingest.yaml b/src/main/resources/metadata/ingest.yaml new file mode 100644 index 0000000..4f33bba --- /dev/null +++ b/src/main/resources/metadata/ingest.yaml @@ -0,0 +1,8 @@ +group: io.kestra.plugin.quickwit.ingest +name: "ingest" +title: "Quickwit Ingest" +description: "Send a batch of documents to a Quickwit index as NDJSON, from a Kestra file, an inline list or a JSON string." +body: "" +videos: [] +createdBy: "Kestra Core Team" +managedBy: "Kestra Core Team" \ No newline at end of file diff --git a/src/main/resources/metadata/io.kestra.plugin.quickwit.index.yaml b/src/main/resources/metadata/io.kestra.plugin.quickwit.index.yaml new file mode 100644 index 0000000..5133d07 --- /dev/null +++ b/src/main/resources/metadata/io.kestra.plugin.quickwit.index.yaml @@ -0,0 +1,11 @@ +# Named with the root-relative dotted package path: a leaf-named `index.yaml` here would collide +# with the root metadata file of the plugin, and the doc linter would then validate the root file +# against the group of this subpackage. +group: io.kestra.plugin.quickwit.index +name: "index" +title: "Quickwit Indexes" +description: "Create, inspect, clear and delete Quickwit indexes, and manage their doc mapping and retention policy." +body: "" +videos: [] +createdBy: "Kestra Core Team" +managedBy: "Kestra Core Team" \ No newline at end of file diff --git a/src/main/resources/metadata/search.yaml b/src/main/resources/metadata/search.yaml new file mode 100644 index 0000000..467e199 --- /dev/null +++ b/src/main/resources/metadata/search.yaml @@ -0,0 +1,8 @@ +group: io.kestra.plugin.quickwit.search +name: "search" +title: "Quickwit Search" +description: "Query Quickwit indexes and react to new documents with a polling trigger backed by a KV store watermark." +body: "" +videos: [] +createdBy: "Kestra Core Team" +managedBy: "Kestra Core Team" \ No newline at end of file diff --git a/src/main/resources/metadata/source.yaml b/src/main/resources/metadata/source.yaml new file mode 100644 index 0000000..3c4011f --- /dev/null +++ b/src/main/resources/metadata/source.yaml @@ -0,0 +1,8 @@ +group: io.kestra.plugin.quickwit.source +name: "source" +title: "Quickwit Sources" +description: "Manage the Kafka, Kinesis and Pulsar sources that feed a Quickwit index: create, pause, resume, reset checkpoints and detach." +body: "" +videos: [] +createdBy: "Kestra Core Team" +managedBy: "Kestra Core Team" \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/AbstractQuickwitTest.java b/src/test/java/io/kestra/plugin/quickwit/AbstractQuickwitTest.java new file mode 100644 index 0000000..78024e9 --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/AbstractQuickwitTest.java @@ -0,0 +1,27 @@ +package io.kestra.plugin.quickwit; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; +import com.github.tomakehurst.wiremock.junit5.WireMockTest; + +import io.kestra.core.junit.annotations.KestraTest; +import io.kestra.core.runners.RunContextFactory; + +import jakarta.inject.Inject; + +/** + * Base class for the Quickwit tests. + * + *

Every test runs against a WireMock server standing in for the Quickwit REST API, so no Quickwit + * cluster is needed and the assertions can cover error paths as well as the happy path. + */ +@KestraTest +@WireMockTest +public abstract class AbstractQuickwitTest { + @Inject + protected RunContextFactory runContextFactory; + + /** Base URL of the WireMock server, to be used as the `url` of a task. */ + protected static String url(WireMockRuntimeInfo wireMock) { + return "http://localhost:" + wireMock.getHttpPort(); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/FlowsValidationTest.java b/src/test/java/io/kestra/plugin/quickwit/FlowsValidationTest.java new file mode 100644 index 0000000..0cea57e --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/FlowsValidationTest.java @@ -0,0 +1,47 @@ +package io.kestra.plugin.quickwit; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.stream.Stream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; + +import io.kestra.core.models.flows.Flow; +import io.kestra.core.serializers.JacksonMapper; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; + +/** + * Sanity-checks the example flows under {@code src/test/resources/flows}: each one must parse as a + * Kestra flow so a copy-pasted example never ships with invalid YAML or an unknown task type. + */ +@TestInstance(TestInstance.Lifecycle.PER_CLASS) +class FlowsValidationTest extends AbstractQuickwitTest { + @Test + void flowsDirectoryIsNotEmpty() throws Exception { + assertThat("no flows found under src/test/resources/flows", flowFiles().isEmpty(), is(false)); + } + + @Test + void everyFlowParses() throws Exception { + for (Path file : flowFiles()) { + Flow flow = JacksonMapper.ofYaml().readValue(file.toFile(), Flow.class); + + assertThat(file + " must declare an id", flow.getId(), is(notNullValue())); + assertThat(file + " must declare a namespace", flow.getNamespace(), is(notNullValue())); + assertThat(file + " must declare tasks", flow.getTasks(), is(notNullValue())); + } + } + + private static List flowFiles() throws Exception { + Path dir = Path.of("src", "test", "resources", "flows"); + + try (Stream files = Files.list(dir)) { + return files.filter(p -> p.toString().endsWith(".yaml")).sorted().toList(); + } + } +} diff --git a/src/test/java/io/kestra/plugin/quickwit/MetadataConsistencyTest.java b/src/test/java/io/kestra/plugin/quickwit/MetadataConsistencyTest.java new file mode 100644 index 0000000..62f58cd --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/MetadataConsistencyTest.java @@ -0,0 +1,198 @@ +package io.kestra.plugin.quickwit; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import org.junit.jupiter.api.Test; + +import io.kestra.core.serializers.JacksonMapper; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsInAnyOrder; +import static org.hamcrest.Matchers.hasKey; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; + +/** + * Guards the invariant that every subpackage holding plugin classes is described by a metadata file + * declaring exactly that package as its group. + * + *

This replaces the META-004 rule of {@code lintPluginDocs}, which {@code build.gradle} disables + * because a subpackage of this plugin is named {@code index}: the linter resolves its metadata to + * {@code metadata/index.yaml}, which is also the file that rule requires for the plugin root, so both + * expectations can never hold at once. Unlike the linter, this test reads the package-to-file mapping + * directly, keyed by the {@code group} each file declares, so there is no collision to resolve. + * + *

It also guards the example snippets against templates that Kestra cannot render, for the same + * reason: the linter checks that an example is valid YAML, not that its expressions evaluate. + */ +class MetadataConsistencyTest extends AbstractQuickwitTest { + private static final String ROOT = "io.kestra.plugin.quickwit"; + + private static final List SUBPACKAGES = List.of( + ROOT + ".search", + ROOT + ".ingest", + ROOT + ".index", + ROOT + ".source", + ROOT + ".deletetask" + ); + + @Test + void everySubpackageWithPluginsIsDescribedByItsOwnMetadata() throws IOException { + Map metadata = metadataByGroup(); + + // discovered from the source tree, mirroring how the linter derives packages from the classes + assertThat(subpackagesWithPlugins(), containsInAnyOrder(SUBPACKAGES.toArray())); + + for (String subpackage : SUBPACKAGES) { + Path file = metadata.get(subpackage); + assertThat("no metadata describes " + subpackage, file, is(notNullValue())); + assertThat("group of " + subpackage + " must be the package itself", group(file), is(subpackage)); + } + } + + @Test + void rootMetadataDeclaresThePluginRoot() throws IOException { + Map metadata = metadataByGroup(); + + assertThat(metadata, hasKey(ROOT)); + assertThat(group(metadata.get(ROOT)), is(ROOT)); + } + + @Test + void metadataDeclaresTheRequiredFields() throws IOException { + for (Map.Entry entry : metadataByGroup().entrySet()) { + Map yaml = readYaml(entry.getValue()); + + for (String field : List.of("group", "name", "title", "description")) { + Object value = yaml.get(field); + assertThat( + entry.getKey() + " must declare a non-empty `" + field + "`", + value != null && !value.toString().isBlank(), + is(true) + ); + } + + // META-003 requires the key to be present but, like every compliant plugin, allows an empty value + assertThat(entry.getKey() + " must declare a `body` field", yaml.containsKey("body"), is(true)); + } + } + + @Test + void everySubpackageHasAnIcon() throws IOException { + Path icons = Path.of("src", "main", "resources", "icons"); + + assertThat("the plugin icon is required by ICON-001", Files.exists(icons.resolve("plugin-icon.svg")), is(true)); + + for (String subpackage : subpackagesWithPlugins()) { + assertThat("missing icon for " + subpackage + " (ICON-002)", Files.exists(icons.resolve(subpackage + ".svg")), is(true)); + } + } + + @Test + void rootHowToExists() { + // DOC-001 and DOC-003: the how-to is required and must not be a stub + Path doc = Path.of("src", "main", "resources", "doc", ROOT + ".md"); + + assertThat(Files.exists(doc), is(true)); + try { + assertThat("the how-to must be at least 10 lines (DOC-003)", Files.readAllLines(doc).size() >= 10, is(true)); + } catch (IOException e) { + throw new AssertionError("Unable to read " + doc, e); + } + } + + /** + * Guards the examples against {@code date('X')}, which is not a Kestra filter. + * + *

Kestra renders with Jinjava, which has no {@code date} filter: {@code now() | date('X')} + * yields the literal string {@code "Z"}. In a search task that fails loudly on a + * {@code Long} property, but in an ingest example it is worse, because every document is + * rejected by Quickwit with {@code failed to parse datetime 'Z'} while the task still reports + * success. {@code timestamp} is the filter that yields unix seconds. Nothing in + * {@code lintPluginDocs} can catch this, because the snippets are valid YAML. + */ + @Test + void noExampleUsesTheDateXFilter() throws IOException { + List offenders = new ArrayList<>(); + + try (Stream files = Files.walk(Path.of("src", "main", "java"))) { + for (Path file : files.filter(p -> p.toString().endsWith(".java")).toList()) { + if (Files.readString(file).contains("date('X')")) { + offenders.add(file.toString()); + } + } + } + + Path doc = Path.of("src", "main", "resources", "doc", ROOT + ".md"); + if (Files.readString(doc).contains("date('X')")) { + offenders.add(doc.toString()); + } + + assertThat("use `now() | timestamp` instead of `now() | date('X')`", offenders, is(List.of())); + } + + /** Maps each documented package to its metadata file, keyed by the {@code group} the file declares. */ + private static Map metadataByGroup() throws IOException { + Path dir = Path.of("src", "main", "resources", "metadata"); + Map byGroup = new LinkedHashMap<>(); + + try (Stream files = Files.list(dir)) { + for (Path file : files.filter(p -> p.toString().endsWith(".yaml")).toList()) { + byGroup.put(group(file), file); + } + } + + return byGroup; + } + + private static String group(Path file) throws IOException { + Object group = readYaml(file).get("group"); + + return group == null ? null : group.toString(); + } + + @SuppressWarnings("unchecked") + private static Map readYaml(Path file) throws IOException { + try (InputStream in = Files.newInputStream(file)) { + return JacksonMapper.ofYaml().readValue(in, Map.class); + } + } + + /** Subpackages of the plugin holding at least one Java source file. */ + private static List subpackagesWithPlugins() throws IOException { + Path javaRoot = Path.of("src", "main", "java", "io", "kestra", "plugin", "quickwit"); + List packages = new ArrayList<>(); + + try (Stream files = Files.walk(javaRoot)) { + for (Path file : files.filter(p -> p.toString().endsWith(".java")).toList()) { + // relative package directory, e.g. "search" or "ingest"; empty for the root package + String directory = javaRoot.relativize(file.getParent()).toString().replace('\\', '/'); + if (directory.isEmpty()) { + continue; + } + + // a package is a plugin subpackage only when it declares its @PluginSubGroup, which is + // what the linter requires of every package holding plugin classes. Shared helper + // packages such as `models` have no package-info.java and are not plugin subpackages. + if (!Files.exists(file.getParent().resolve("package-info.java"))) { + continue; + } + + String subpackage = ROOT + "." + directory.replace('/', '.'); + if (!packages.contains(subpackage)) { + packages.add(subpackage); + } + } + } + + return packages; + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/QuickwitConnectionTest.java b/src/test/java/io/kestra/plugin/quickwit/QuickwitConnectionTest.java new file mode 100644 index 0000000..3878667 --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/QuickwitConnectionTest.java @@ -0,0 +1,167 @@ +package io.kestra.plugin.quickwit; + +import java.nio.charset.StandardCharsets; +import java.util.Base64; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.search.Search; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsString; +import static com.github.tomakehurst.wiremock.client.WireMock.equalTo; +import static org.hamcrest.Matchers.is; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Coverage of the connection properties shared by every task: custom headers, basic auth and the error + * raised when a protected cluster answers 401. + */ +class QuickwitConnectionTest extends AbstractQuickwitTest { + private static final String SEARCH_OK = """ + {"num_hits": 0, "hits": []} + """; + + @Test + void sendsCustomHeaders(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_OK)) + ); + + Search.builder() + .url(Property.ofValue(url(wireMock))) + .headers(Property.ofValue(Map.of("X-Api-Key", "secret-key"))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build() + .run(runContextFactory.of()); + + verify(com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor(urlPathEqualTo("/api/v1/app-logs/search")) + .withHeader("X-Api-Key", equalTo("secret-key")) + ); + } + + @Test + void rendersTemplatedHeaders(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_OK)) + ); + + Search.builder() + .url(Property.ofValue(url(wireMock))) + .headers(Property.ofExpression("{\"Authorization\": \"Bearer {{ token }}\"}")) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build() + .run(runContextFactory.of(Map.of("token", "abc123"))); + + verify(com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor(urlPathEqualTo("/api/v1/app-logs/search")) + .withHeader("Authorization", equalTo("Bearer abc123")) + ); + } + + @Test + void sendsBasicAuthCredentials(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_OK)) + ); + + Search.builder() + .url(Property.ofValue(url(wireMock))) + .basicAuth(AbstractQuickwitTask.BasicAuth.builder() + .username(Property.ofValue("kestra")) + .password(Property.ofValue("changeme")) + .build() + ) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build() + .run(runContextFactory.of()); + + var expected = "Basic " + Base64.getEncoder().encodeToString("kestra:changeme".getBytes(StandardCharsets.UTF_8)); + + verify(com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor(urlPathEqualTo("/api/v1/app-logs/search")) + .withHeader("Authorization", equalTo(expected)) + ); + } + + @Test + void reportsAChallengeFromAProtectedCluster(WireMockRuntimeInfo wireMock) { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(aResponse() + .withStatus(401) + .withHeader("WWW-Authenticate", "Basic realm=\"quickwit\"") + .withBody("{\"message\": \"Unauthorized\"}") + ) + ); + + var task = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + // Quickwit has no auth of its own, so this comes from a proxy in front of it: the message must + // point at the properties that carry credentials + assertThat(thrown.getMessage(), is("Quickwit search on index 'app-logs' failed with HTTP 401: Unauthorized")); + } + + @Test + void reportsAnEmptySuccessBodyAsAMisconfiguredUrl(WireMockRuntimeInfo wireMock) { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(aResponse().withStatus(200)) + ); + + var task = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat(thrown.getMessage(), containsString("returned an empty body")); + assertThat(thrown.getMessage(), containsString("default port 7280")); + } + + @Test + void reportsAMalformedUrlBeforeSendingAnything() { + var task = Search.builder() + .url(Property.ofValue("not a url")) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> task.run(runContextFactory.of())); + + assertThat(thrown.getMessage(), containsString("Invalid `url` `not a url`")); + } + + @Test + void doesNotLeakCredentialsInToString() { + var task = Search.builder() + .url(Property.ofValue("http://localhost:7280")) + .basicAuth(AbstractQuickwitTask.BasicAuth.builder() + .username(Property.ofValue("kestra")) + .password(Property.ofValue("super-secret")) + .build() + ) + .headers(Property.ofValue(Map.of("Authorization", "Bearer super-secret"))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build(); + + assertThat(task.toString(), org.hamcrest.Matchers.not(containsString("super-secret"))); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/QuickwitContainerTest.java b/src/test/java/io/kestra/plugin/quickwit/QuickwitContainerTest.java new file mode 100644 index 0000000..4b91006 --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/QuickwitContainerTest.java @@ -0,0 +1,162 @@ +package io.kestra.plugin.quickwit; + +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.utility.DockerImageName; + +import io.kestra.core.junit.annotations.KestraTest; +import io.kestra.core.models.property.Property; +import io.kestra.core.runners.RunContextFactory; +import io.kestra.plugin.quickwit.ingest.Commit; +import io.kestra.plugin.quickwit.ingest.Ingest; +import io.kestra.plugin.quickwit.search.Search; + +import jakarta.inject.Inject; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.greaterThanOrEqualTo; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.junit.jupiter.api.Assumptions.abort; + +/** + * Live round-trip against a real Quickwit node. + * + *

WireMock covers every task including the error paths, but stubs cannot catch API drift (endpoint + * shapes, commit visibility, sort and timestamp semantics). This single test creates an index, ingests + * two documents with {@code FORCE} commit and reads them back, proving the wiring against Quickwit + * itself. + * + *

The test starts its own container, so it needs Docker. Where Docker is unavailable (or + * {@code QUICKWIT_IT_URL} points at an already running node, e.g. + * {@code QUICKWIT_IT_URL=http://localhost:7280}) the container is skipped and the given node is used + * instead; when neither is available the test aborts instead of failing. + */ +@KestraTest +@TestInstance(TestInstance.Lifecycle.PER_CLASS) +class QuickwitContainerTest { + private static final GenericContainer QUICKWIT = new GenericContainer<>(DockerImageName.parse("quickwit/quickwit:0.8.2")) + .withCommand("run") + .withEnv("QW_CLUSTER_ID", "quickwit-it") + .withEnv("QW_NODE_ID", "quickwit-node-1") + .withEnv("QW_PEER_SEEDS", "quickwit-node-1") + .withExposedPorts(7280) + .waitingFor(Wait.forHttp("/api/v1/cluster").forPort(7280).forStatusCode(200)) + .withStartupTimeout(Duration.ofMinutes(2)); + + private String baseUrl; + + @Inject + RunContextFactory runContextFactory; + + @BeforeAll + void startQuickwit() { + String override = System.getenv("QUICKWIT_IT_URL"); + if (override != null && !override.isBlank()) { + baseUrl = override; + return; + } + try { + QUICKWIT.start(); + } catch (Exception e) { + abort("Docker is unavailable, skipping live Quickwit test: " + e.getMessage()); + return; + } + baseUrl = "http://" + QUICKWIT.getHost() + ":" + QUICKWIT.getMappedPort(7280); + } + + @AfterAll + void stopQuickwit() { + if (System.getenv("QUICKWIT_IT_URL") == null && QUICKWIT.isRunning()) { + QUICKWIT.stop(); + } + } + + @Test + void ingestThenSearchRoundTrip() throws Exception { + String index = "kestra-it-" + UUID.randomUUID().toString().substring(0, 8).toLowerCase(); + var runContext = runContextFactory.of(); + + try { + var created = io.kestra.plugin.quickwit.index.Create.builder() + .url(Property.ofValue(baseUrl)) + .index(Property.ofValue(index)) + .configVersion(Property.ofValue("0.8")) + .docMapping(Property.ofValue(Map.of( + "mode", "lenient", + "timestamp_field", "timestamp", + "field_mappings", List.of( + Map.of( + "name", "timestamp", + "type", "datetime", + "input_formats", List.of("unix_timestamp"), + "output_format", "unix_timestamp_secs", + "fast", true + ), + Map.of("name", "severity", "type", "text", "fast", true), + Map.of("name", "message", "type", "text") + ) + ))) + .searchSettings(Property.ofValue(Map.of("default_search_fields", List.of("message")))) + .build() + .run(runContext); + assertThat(created.getIndex(), is(index)); + + long now = Instant.now().getEpochSecond(); + var ingested = Ingest.builder() + .url(Property.ofValue(baseUrl)) + .index(Property.ofValue(index)) + .commit(Property.ofValue(Commit.FORCE)) + .from(Property.ofValue(List.of( + Map.of("timestamp", now, "severity", "ERROR", "message", "container probe one"), + Map.of("timestamp", now, "severity", "ERROR", "message", "container probe two") + ))) + .build() + .run(runContext); + // the live API only guarantees num_docs_for_processing on success; the ingested / + // rejected counters appear in the stubs but Quickwit 0.8.2 omits them here, so the + // searchable-documents assertion below is the real proof the ingest landed + assertThat(ingested.getNumDocsForProcessing(), is(2L)); + + // even with FORCE the commit is asynchronous: poll until both documents are searchable + Search.Output found = null; + for (int i = 0; i < 30; i++) { + var result = Search.builder() + .url(Property.ofValue(baseUrl)) + .index(Property.ofValue(index)) + .query(Property.ofValue("severity:ERROR")) + .maxHits(Property.ofValue(10)) + .build() + .run(runContext); + if (result.getTotal() != null && result.getTotal() >= 2L) { + found = result; + break; + } + Thread.sleep(2000); + } + assertThat(found != null, is(true)); + assertThat(found.getTotal(), is(greaterThanOrEqualTo(2L))); + assertThat(found.getRows(), hasSize(greaterThanOrEqualTo(2))); + } finally { + try { + io.kestra.plugin.quickwit.index.Delete.builder() + .url(Property.ofValue(baseUrl)) + .index(Property.ofValue(index)) + .build() + .run(runContext); + } catch (Exception ignored) { + // best-effort cleanup so repeated runs never collide + } + } + } +} diff --git a/src/test/java/io/kestra/plugin/quickwit/QuickwitServiceTest.java b/src/test/java/io/kestra/plugin/quickwit/QuickwitServiceTest.java new file mode 100644 index 0000000..59034ac --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/QuickwitServiceTest.java @@ -0,0 +1,196 @@ +package io.kestra.plugin.quickwit; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +import io.kestra.core.exceptions.IllegalVariableEvaluationException; +import io.kestra.core.models.property.Property; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.models.SearchQuery; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; +import static org.hamcrest.Matchers.nullValue; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Unit tests of the endpoint, request and error-message helpers, which need no HTTP server. + */ +class QuickwitServiceTest extends AbstractQuickwitTest { + @Test + void endpointPrefixesTheApiVersion() { + assertThat(QuickwitService.endpoint("http://localhost:7280", "indexes/app-logs").toString(), is("http://localhost:7280/api/v1/indexes/app-logs")); + } + + @Test + void endpointToleratesTrailingAndMissingSlashes() { + assertThat(QuickwitService.endpoint("http://localhost:7280/", "/indexes").toString(), is("http://localhost:7280/api/v1/indexes")); + assertThat(QuickwitService.endpoint("http://localhost:7280", "indexes").toString(), is("http://localhost:7280/api/v1/indexes")); + } + + @Test + void endpointUrlEncodesTheQuery() { + var uri = QuickwitService.endpoint("http://localhost:7280", "app-logs/search", Map.of("query", "severity:ERROR AND body:\"a b\"")); + + assertThat(uri.toString(), is("http://localhost:7280/api/v1/app-logs/search?query=severity%3AERROR+AND+body%3A%22a+b%22")); + } + + @Test + void endpointSkipsNullQueryParameters() { + // Map.of() rejects null values, so the nullable case is built by hand + Map query = new HashMap<>(); + query.put("commit", null); + + assertThat(QuickwitService.endpoint("http://localhost:7280", "indexes", query).toString(), is("http://localhost:7280/api/v1/indexes")); + } + + @Test + void pathSegmentEncodesIndexIds() { + // percent-encoded, not '+': inside a URI path a plus sign is a literal plus + assertThat(QuickwitService.pathSegment("app logs/v2"), is("app%20logs%2Fv2")); + } + + @Test + void renderedUrlRejectsBlankValues() { + RunContext runContext = runContextFactory.of(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> QuickwitService.renderedUrl(runContext, null)); + assertThat(thrown.getMessage(), containsString("`url` is required")); + } + + @Test + void renderedUrlRejectsUrisWithoutScheme() { + RunContext runContext = runContextFactory.of(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> QuickwitService.renderedUrl(runContext, Property.ofValue("localhost:7280"))); + assertThat(thrown.getMessage(), containsString("expected a URI with a scheme and a host")); + } + + @Test + void renderedUrlStripsTheTrailingSlash() throws IllegalVariableEvaluationException { + RunContext runContext = runContextFactory.of(); + + assertThat(QuickwitService.renderedUrl(runContext, Property.ofValue("http://localhost:7280/")), is("http://localhost:7280")); + } + + @Test + void quickwitMessageExtractsTheQuickwitError() { + assertThat(QuickwitService.quickwitMessage("{\"message\":\"Failed to parse query\"}"), is("Failed to parse query")); + } + + @Test + void quickwitMessageHandlesHtmlErrorPages() { + // a proxy or gateway may answer with HTML rather than Quickwit's JSON error object + assertThat(QuickwitService.quickwitMessage("Bad Gateway"), containsString("Bad Gateway")); + } + + @Test + void quickwitMessageHandlesAnAbsentBody() { + assertThat(QuickwitService.quickwitMessage(""), is("no response body")); + } + + @Test + void requestCarriesTheCustomHeaders() { + var request = QuickwitService + .request("GET", QuickwitService.endpoint("http://localhost:7280", "cluster"), Map.of("X-Api-Key", "abc"), null) + .build(); + + assertThat(request.getMethod(), is("GET")); + assertThat(request.getHeaders().firstValue("X-Api-Key").orElseThrow(), is("abc")); + assertThat(request.getHeaders().firstValue("Accept").orElseThrow(), is("application/json")); + assertThat(request.getBody(), is(nullValue())); + } + + @Test + void httpConfigurationKeepsTheDefaultReadTimeoutWhenUnset() throws IllegalVariableEvaluationException { + var configuration = QuickwitService.httpConfiguration(runContextFactory.of(), null, null, null); + + // a zero or absent read timeout would break long searches and commit=force ingests + assertThat(configuration.getTimeout().getReadIdleTimeout(), is(notNullValue())); + } + + @Test + void httpConfigurationAllowsFailedResponsesSoErrorsCanBeReported() throws IllegalVariableEvaluationException { + var configuration = QuickwitService.httpConfiguration(runContextFactory.of(), null, null, null); + + // handled by QuickwitService#execute, which can then surface Quickwit's own message + assertThat(runContextFactory.of().render(configuration.getAllowFailed()).as(Boolean.class).orElseThrow(), is(true)); + } + + @Test + void httpConfigurationCarriesBasicAuth() throws IllegalVariableEvaluationException { + var basicAuth = AbstractQuickwitTask.BasicAuth.builder() + .username(Property.ofValue("kestra")) + .password(Property.ofValue("changeme")) + .build(); + + var configuration = QuickwitService.httpConfiguration(runContextFactory.of(), null, null, basicAuth); + + assertThat(configuration.getAuth(), is(notNullValue())); + } + + @Test + void searchQueryOmitsUnsetOptionalParameters() { + var body = SearchQuery.of("app-logs", "severity:ERROR", null).toBody(null); + + assertThat(body, is(Map.of("query", "severity:ERROR"))); + } + + @Test + void searchQuerySendsListsAsCommaSeparatedStrings() { + var body = new SearchQuery( + "app-logs", + "*", + 100L, + 200L, + 0, + 10, + List.of("message", "severity"), + List.of("message"), + List.of("timestamp"), + Map.of("severity", Map.of("type", "terms")) + ).toBody(null); + + assertThat(body.get("start_timestamp"), is(100L)); + assertThat(body.get("end_timestamp"), is(200L)); + assertThat(body.get("start_offset"), is(0)); + assertThat(body.get("max_hits"), is(10)); + assertThat(body.get("aggs"), is(Map.of("severity", Map.of("type", "terms")))); + + // the search API deserializes these three into plain string fields, so a JSON array is + // rejected with `invalid type: sequence, expected a string` + assertThat(body.get("search_field"), is("message,severity")); + assertThat(body.get("snippet_fields"), is("message")); + assertThat(body.get("sort_by"), is("timestamp")); + } + + @Test + void searchQueryOmitsEmptyLists() { + var body = new SearchQuery( + "app-logs", + "*", + null, + null, + null, + null, + List.of(), + null, + null, + null + ).toBody(null); + + assertThat(body, is(Map.of("query", "*"))); + } + + @Test + void searchQueryOverrideReplacesTheStartTimestamp() { + var query = new SearchQuery("app-logs", "*", 100L, null, null, null, null, null, null, null); + + assertThat(query.toBody(500L).get("start_timestamp"), is(500L)); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/deletetask/DeleteTaskTasksTest.java b/src/test/java/io/kestra/plugin/quickwit/deletetask/DeleteTaskTasksTest.java new file mode 100644 index 0000000..1bb1480 --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/deletetask/DeleteTaskTasksTest.java @@ -0,0 +1,150 @@ +package io.kestra.plugin.quickwit.deletetask; + +import java.util.List; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.AbstractQuickwitTest; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson; +import static com.github.tomakehurst.wiremock.client.WireMock.get; +import static com.github.tomakehurst.wiremock.client.WireMock.getRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Happy-path and error coverage for the delete task tasks. + */ +class DeleteTaskTasksTest extends AbstractQuickwitTest { + private static final String DELETE_TASK = """ + { + "create_timestamp": 1700000000, + "opstamp": 42, + "delete_query": { + "query": "message:trash", + "search_field": ["message"], + "start_timestamp": 1699000000, + "end_timestamp": 1700000000 + } + } + """; + + @Test + void createsDeleteTask(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/delete-tasks")).willReturn(okJson(DELETE_TASK))); + + var output = Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("message:trash")) + .startTimestamp(Property.ofValue(1699000000L)) + .endTimestamp(Property.ofValue(1700000000L)) + .searchField(Property.ofValue(List.of("message"))) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getCreateTimestamp(), is(1700000000L)); + assertThat(output.getOpstamp(), is(42L)); + assertThat(output.getDeleteTask().getDeleteQuery().getQuery(), is("message:trash")); + assertThat(output.getDeleteTask().getDeleteQuery().getSearchField(), is(List.of("message"))); + + // `search_fields`, plural, and an array: the delete task API names it differently from the + // search API and takes the opposite encoding, rejecting `search_field` outright with + // `unknown field 'search_field', expected one of 'query', 'search_fields', ...` + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/delete-tasks")) + .withRequestBody(equalToJson(""" + { + "query": "message:trash", + "start_timestamp": 1699000000, + "end_timestamp": 1700000000, + "search_fields": ["message"] + } + """)) + ); + } + + @Test + void createOmitsUnsetOptionalParameters(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/delete-tasks")).willReturn(okJson(DELETE_TASK))); + + Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build() + .run(runContextFactory.of()); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/delete-tasks")) + .withRequestBody(equalToJson("{\"query\": \"*\"}")) + ); + } + + @Test + void listsDeleteTasks(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(get(urlPathEqualTo("/api/v1/app-logs/delete-tasks")).willReturn(okJson("[" + DELETE_TASK + "]"))); + + var output = io.kestra.plugin.quickwit.deletetask.List.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getSize(), is(1)); + assertThat(output.getDeleteTasks(), hasSize(1)); + assertThat(output.getDeleteTasks().getFirst().getOpstamp(), is(42L)); + + verify(getRequestedFor(urlPathEqualTo("/api/v1/app-logs/delete-tasks"))); + } + + @Test + void listHandlesAnIndexWithoutDeleteTasks(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(get(urlPathEqualTo("/api/v1/app-logs/delete-tasks")).willReturn(okJson("[]"))); + + var output = io.kestra.plugin.quickwit.deletetask.List.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getSize(), is(0)); + assertThat(output.getDeleteTasks(), hasSize(0)); + } + + @Test + void surfacesTheQuickwitErrorOnAnUnknownIndex(WireMockRuntimeInfo wireMock) { + stubFor(post(urlPathEqualTo("/api/v1/unknown/delete-tasks")) + .willReturn(aResponse() + .withStatus(404) + .withHeader("Content-Type", "application/json") + .withBody("{\"message\": \"Index unknown does not exist\"}") + ) + ); + + var task = Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("unknown")) + .query(Property.ofValue("*")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat( + thrown.getMessage(), + is("Quickwit creation of a delete task on index 'unknown' failed with HTTP 404: Index unknown does not exist") + ); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/index/IndexTasksTest.java b/src/test/java/io/kestra/plugin/quickwit/index/IndexTasksTest.java new file mode 100644 index 0000000..d1a734c --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/index/IndexTasksTest.java @@ -0,0 +1,260 @@ +package io.kestra.plugin.quickwit.index; + +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.AbstractQuickwitTest; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.delete; +import static com.github.tomakehurst.wiremock.client.WireMock.deleteRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.get; +import static com.github.tomakehurst.wiremock.client.WireMock.getRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.put; +import static com.github.tomakehurst.wiremock.client.WireMock.putRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Happy-path and error coverage for every task of the index management package. + */ +class IndexTasksTest extends AbstractQuickwitTest { + private static final String INDEX_METADATA = """ + { + "version": "0.8", + "index_uid": "app-logs:01H000000000000000000", + "create_timestamp": 1700000000, + "index_config": { + "index_id": "app-logs", + "doc_mapping": {"timestamp_field": "timestamp"}, + "search_settings": {"default_search_fields": ["message"]} + }, + "checkpoint": {"_ingest-api-source": {"source_id": "_ingest-api-source"}}, + "sources": [] + } + """; + + private static final Map DOC_MAPPING = Map.of( + "timestamp_field", "timestamp", + "field_mappings", List.of( + Map.of("name", "timestamp", "type", "datetime", "fast", true), + Map.of("name", "message", "type", "text") + ) + ); + + @Test + void create(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/indexes")).willReturn(okJson(INDEX_METADATA))); + + var output = Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .configVersion(Property.ofValue("0.8")) + .docMapping(Property.ofValue(DOC_MAPPING)) + .searchSettings(Property.ofValue(Map.of("default_search_fields", List.of("message")))) + .retention(Property.ofValue(Map.of("period", "30 days", "schedule", "@daily"))) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getIndexUid(), is("app-logs:01H000000000000000000")); + assertThat(output.getCreateTimestamp(), is(1700000000L)); + assertThat(output.getMetadata(), is(notNullValue())); + assertThat(output.getMetadata().getIndexConfig().get("index_id"), is("app-logs")); + } + + @Test + void createSendsTheIndexConfiguration(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/indexes")).willReturn(okJson(INDEX_METADATA))); + + Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .configVersion(Property.ofValue("0.8")) + .docMapping(Property.ofValue(DOC_MAPPING)) + .build() + .run(runContextFactory.of()); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/indexes")) + .withRequestBody(com.github.tomakehurst.wiremock.client.WireMock.equalToJson(""" + { + "version": "0.8", + "index_id": "app-logs", + "doc_mapping": { + "timestamp_field": "timestamp", + "field_mappings": [ + {"name": "timestamp", "type": "datetime", "fast": true}, + {"name": "message", "type": "text"} + ] + } + } + """)) + ); + } + + @Test + void createRejectsAnEmptyDocMapping(WireMockRuntimeInfo wireMock) { + var task = Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .docMapping(Property.ofValue(Map.of())) + .build(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> task.run(runContextFactory.of())); + assertThat(thrown.getMessage(), containsString("`docMapping` is required")); + } + + @Test + void readsIndex(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(get(urlPathEqualTo("/api/v1/indexes/app-logs")).willReturn(okJson(INDEX_METADATA))); + + var output = Get.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getIndexUid(), is("app-logs:01H000000000000000000")); + assertThat(output.getMetadata().getIndexConfig().get("search_settings"), is(notNullValue())); + + verify(getRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs"))); + } + + @Test + void getUrlEncodesTheIndexId(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(get(urlPathEqualTo("/api/v1/indexes/app%20logs")).willReturn(okJson(INDEX_METADATA))); + + Get.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app logs")) + .build() + .run(runContextFactory.of()); + + verify(getRequestedFor(urlPathEqualTo("/api/v1/indexes/app%20logs"))); + } + + @Test + void listsIndexes(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(get(urlPathEqualTo("/api/v1/indexes")).willReturn(okJson("[" + INDEX_METADATA + "]"))); + + var output = io.kestra.plugin.quickwit.index.List.builder() + .url(Property.ofValue(url(wireMock))) + .build() + .run(runContextFactory.of()); + + assertThat(output.getSize(), is(1)); + assertThat(output.getIndexes(), hasSize(1)); + assertThat(output.getIndexes().getFirst().getIndexUid(), is("app-logs:01H000000000000000000")); + } + + @Test + void listHandlesAnEmptyCluster(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(get(urlPathEqualTo("/api/v1/indexes")).willReturn(okJson("[]"))); + + var output = io.kestra.plugin.quickwit.index.List.builder() + .url(Property.ofValue(url(wireMock))) + .build() + .run(runContextFactory.of()); + + assertThat(output.getSize(), is(0)); + assertThat(output.getIndexes(), hasSize(0)); + } + + @Test + void deletesIndex(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(delete(urlPathEqualTo("/api/v1/indexes/app-logs")).willReturn(okJson(""" + [ + { + "split_id": "01GK1XNAECH7P14850S9VV6P94", + "num_docs": 1337, + "uncompressed_docs_size_bytes": 23933408, + "file_name": "01GK1XNAECH7P14850S9VV6P94.split", + "file_size_bytes": 2991676 + } + ] + """))); + + var output = Delete.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getSize(), is(1)); + assertThat(output.getDeletedSplits(), hasSize(1)); + assertThat(output.getDeletedSplits().getFirst().getSplitId(), is("01GK1XNAECH7P14850S9VV6P94")); + assertThat(output.getDeletedSplits().getFirst().getNumDocs(), is(1337L)); + + verify(deleteRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs"))); + } + + @Test + void clearAcceptsTheEmptyBodyQuickwitReturns(WireMockRuntimeInfo wireMock) throws Exception { + // Quickwit documents the clear endpoint as returning an empty body + stubFor(put(urlPathEqualTo("/api/v1/indexes/app-logs/clear")).willReturn(aResponse().withStatus(200))); + + var output = Clear.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + + verify(putRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs/clear"))); + } + + @Test + void surfacesTheQuickwitErrorOnAMissingIndex(WireMockRuntimeInfo wireMock) { + stubFor(get(urlPathEqualTo("/api/v1/indexes/unknown")) + .willReturn(aResponse() + .withStatus(404) + .withHeader("Content-Type", "application/json") + .withBody("{\"message\": \"Index unknown does not exist\"}") + ) + ); + + var task = Get.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("unknown")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat(thrown.getMessage(), is("Quickwit read of index 'unknown' failed with HTTP 404: Index unknown does not exist")); + } + + @Test + void surfacesAnHtmlErrorPageFromAProxy(WireMockRuntimeInfo wireMock) { + stubFor(get(urlPathEqualTo("/api/v1/indexes/app-logs")) + .willReturn(aResponse().withStatus(502).withBody("502 Bad Gateway")) + ); + + var task = Get.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat(thrown.getMessage(), containsString("failed with HTTP 502")); + assertThat(thrown.getMessage(), containsString("502 Bad Gateway")); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/ingest/IngestTest.java b/src/test/java/io/kestra/plugin/quickwit/ingest/IngestTest.java new file mode 100644 index 0000000..4c2b77e --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/ingest/IngestTest.java @@ -0,0 +1,238 @@ +package io.kestra.plugin.quickwit.ingest; + +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.AbstractQuickwitTest; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.equalTo; +import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class IngestTest extends AbstractQuickwitTest { + private static final String INGESTED = """ + {"num_docs_for_processing": 2, "num_ingested_docs": 2, "num_rejected_docs": 0} + """; + + @Test + void ingestInlineDocuments(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(INGESTED))); + + var output = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(List.of( + Map.of("message", "first", "timestamp", 1700000000L), + Map.of("message", "second", "timestamp", 1700000001L) + ))) + .build() + .run(runContextFactory.of()); + + assertThat(output.getNumDocsForProcessing(), is(2L)); + assertThat(output.getNumIngestedDocs(), is(2L)); + assertThat(output.getNumRejectedDocs(), is(0L)); + assertThat(output.getParseFailures(), is(org.hamcrest.Matchers.nullValue())); + } + + @Test + void sendsOneJsonDocumentPerLine(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(INGESTED))); + + Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(List.of( + Map.of("message", "first"), + Map.of("message", "second") + ))) + .build() + .run(runContextFactory.of()); + + // NDJSON: one compact document per line, which is the only format the ingest API accepts + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/ingest")) + .withRequestBody(equalTo(""" + {"message":"first"} + {"message":"second"}""")) + ); + } + + @Test + void sendsTheCommitAndDetailedResponseOptions(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(INGESTED))); + + Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(Map.of("message", "first"))) + .commit(Property.ofValue(Commit.WAIT_FOR)) + .detailedResponse(Property.ofValue(true)) + .build() + .run(runContextFactory.of()); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/ingest")) + .withQueryParam("commit", equalTo("wait_for")) + .withQueryParam("detailed_response", equalTo("true")) + ); + } + + @Test + void sendsForceAsTheCommitOption(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(INGESTED))); + + Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(Map.of("message", "first"))) + .commit(Property.ofValue(Commit.FORCE)) + .build() + .run(runContextFactory.of()); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/ingest")) + .withQueryParam("commit", equalTo("force")) + ); + } + + @Test + void ingestsASingleInlineDocument(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson("{\"num_docs_for_processing\": 1, \"num_ingested_docs\": 1, \"num_rejected_docs\": 0}"))); + + var output = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(Map.of("message", "only one"))) + .build() + .run(runContextFactory.of()); + + assertThat(output.getNumIngestedDocs(), is(1L)); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/ingest")) + .withRequestBody(equalToJson("{\"message\": \"only one\"}")) + ); + } + + @Test + void ingestsAJsonString(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(INGESTED))); + + var output = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue("[{\"message\": \"one\"}, {\"message\": \"two\"}]")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getNumIngestedDocs(), is(2L)); + } + + @Test + void reportsRejectedDocuments(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(""" + { + "num_docs_for_processing": 2, + "num_ingested_docs": 1, + "num_rejected_docs": 1, + "parse_failures": [{"message": "unknown field", "reason": "invalid_schema", "document": "{\\"oops\\":1}"}] + } + """))); + + var output = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(List.of(Map.of("message", "a"), Map.of("oops", 1)))) + .detailedResponse(Property.ofValue(true)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getNumIngestedDocs(), is(1L)); + assertThat(output.getNumRejectedDocs(), is(1L)); + assertThat(output.getParseFailures(), hasSize(1)); + assertThat(output.getParseFailures().getFirst().getReason(), is("invalid_schema")); + assertThat(output.getParseFailures().getFirst().getMessage(), is("unknown field")); + } + + @Test + void readsDocumentsFromKestraInternalStorage(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")).willReturn(okJson(INGESTED))); + + var runContext = runContextFactory.of(); + var uri = runContext.storage().putFile( + java.nio.file.Path.of( + writeTemporary(runContext, "{\"message\": \"from a file\"}\n{\"message\": \"second line\"}\n") + ).toFile() + ); + + var output = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(uri.toString())) + .build() + .run(runContext); + + assertThat(output.getNumIngestedDocs(), is(2L)); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/ingest")) + .withRequestBody(equalTo(""" + {"message":"from a file"} + {"message":"second line"}""")) + ); + } + + @Test + void rejectsAnEmptyDocumentSet(WireMockRuntimeInfo wireMock) { + var task = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(List.of())) + .build(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> task.run(runContextFactory.of())); + assertThat(thrown.getMessage(), containsString("resolved to no document")); + } + + @Test + void surfacesTheQuickwitError(WireMockRuntimeInfo wireMock) { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/ingest")) + .willReturn(aResponse() + .withStatus(400) + .withHeader("Content-Type", "application/json") + .withBody("{\"message\": \"doc mapping does not contain field 'timestamp'\"}") + ) + ); + + var task = Ingest.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .from(Property.ofValue(Map.of("message", "no timestamp"))) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat( + thrown.getMessage(), + is("Quickwit ingest into index 'app-logs' failed with HTTP 400: doc mapping does not contain field 'timestamp'") + ); + } + + private String writeTemporary(io.kestra.core.runners.RunContext runContext, String content) throws Exception { + var file = runContext.workingDir().createTempFile(".ndjson").toFile(); + java.nio.file.Files.writeString(file.toPath(), content); + + return file.getAbsolutePath(); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/search/SearchTest.java b/src/test/java/io/kestra/plugin/quickwit/search/SearchTest.java new file mode 100644 index 0000000..f6cd78f --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/search/SearchTest.java @@ -0,0 +1,255 @@ +package io.kestra.plugin.quickwit.search; + +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.exceptions.IllegalVariableEvaluationException; +import io.kestra.core.models.property.Property; +import io.kestra.core.models.tasks.common.FetchType; +import io.kestra.core.runners.RunContext; +import io.kestra.plugin.quickwit.AbstractQuickwitTest; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.notNullValue; +import static org.hamcrest.Matchers.nullValue; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class SearchTest extends AbstractQuickwitTest { + private static final String SEARCH_RESPONSE = """ + { + "num_hits": 2, + "elapsed_time_micros": 1234, + "hits": [ + {"message": "payment gateway timeout", "severity": "ERROR"}, + {"message": "payment gateway timeout", "severity": "ERROR"} + ] + } + """; + + @Test + void fetch(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_RESPONSE))); + + var output = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .maxHits(Property.ofValue(100)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getTotal(), is(2L)); + assertThat(output.getSize(), is(2)); + assertThat(output.getRows(), hasSize(2)); + assertThat(output.getElapsedTimeMicros(), is(1234L)); + assertThat(output.getRow(), is(nullValue())); + assertThat(output.getUri(), is(nullValue())); + } + + @Test + void sendsTheSearchBodyAsJson(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_RESPONSE))); + + Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .startTimestamp(Property.ofValue(1000L)) + .endTimestamp(Property.ofValue(2000L)) + .startOffset(Property.ofValue(5)) + .maxHits(Property.ofValue(100)) + .searchField(Property.ofValue(List.of("message", "severity"))) + .snippetFields(Property.ofValue(List.of("message"))) + .sortBy(Property.ofValue(List.of("timestamp"))) + .aggregations(Property.ofValue(Map.of("severity", Map.of("type", "terms")))) + .build() + .run(runContextFactory.of()); + + // POST carries aggs as a nested object, which the query string form of the GET endpoint cannot + // express. The list parameters are comma-separated strings: the search API deserializes them + // into plain string fields and rejects a JSON array with + // `invalid type: sequence, expected a string`. Note the singular `search_field` here, against + // the plural `search_fields` of the delete task API. + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/search")) + .withRequestBody(equalToJson(""" + { + "query": "severity:ERROR", + "start_timestamp": 1000, + "end_timestamp": 2000, + "start_offset": 5, + "max_hits": 100, + "search_field": "message,severity", + "snippet_fields": "message", + "sort_by": "timestamp", + "aggs": {"severity": {"type": "terms"}} + } + """)) + ); + } + + @Test + void fetchOne(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_RESPONSE))); + + var output = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .fetchType(Property.ofValue(FetchType.FETCH_ONE)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getRow(), is(notNullValue())); + assertThat(output.getRow().get("severity"), is("ERROR")); + assertThat(output.getRows(), is(nullValue())); + assertThat(output.getTotal(), is(2L)); + } + + @Test + void store(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_RESPONSE))); + + var runContext = runContextFactory.of(); + + var output = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .fetchType(Property.ofValue(FetchType.STORE)) + .build() + .run(runContext); + + assertThat(output.getUri(), is(notNullValue())); + assertThat(output.getSize(), is(2)); + assertThat(output.getRows(), is(nullValue())); + + try (var input = runContext.storage().getFile(output.getUri())) { + var stored = io.kestra.core.serializers.FileSerde.readAll(input, Map.class).collectList().block(); + assertThat(stored, hasSize(2)); + } + } + + @Test + void none(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(SEARCH_RESPONSE))); + + var output = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .fetchType(Property.ofValue(FetchType.NONE)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getSize(), is(0)); + assertThat(output.getRows(), is(nullValue())); + assertThat(output.getRow(), is(nullValue())); + } + + @Test + void aggregations(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(okJson(""" + {"num_hits": 1, "elapsed_time_micros": 7, "hits": [], "aggregations": {"severity": {"buckets": []}}} + """))); + + var output = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getTotal(), is(1L)); + assertThat(output.getSize(), is(0)); + assertThat(output.getAggregations(), is(notNullValue())); + assertThat(output.getAggregations().get("severity"), is(notNullValue())); + } + + @Test + void rendersTemplatedProperties(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/prod-logs/search")) + .willReturn(okJson("{\"num_hits\": 0, \"hits\": []}"))); + + RunContext runContext = runContextFactory.of(Map.of("index", "prod-logs", "severity", "WARN")); + + var output = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofExpression("{{ index }}")) + .query(Property.ofExpression("severity:{{ severity }}")) + .build() + .run(runContext); + + assertThat(output.getTotal(), is(0L)); + verify(postRequestedFor(urlPathEqualTo("/api/v1/prod-logs/search")) + .withRequestBody(equalToJson("{\"query\": \"severity:WARN\"}")) + ); + } + + @Test + void surfacesTheQuickwitError(WireMockRuntimeInfo wireMock) { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(aResponse() + .withStatus(400) + .withHeader("Content-Type", "application/json") + .withBody("{\"message\": \"Failed to parse query\"}") + ) + ); + + var task = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:::")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat(thrown.getMessage(), is("Quickwit search on index 'app-logs' failed with HTTP 400: Failed to parse query")); + } + + @Test + void reportsAnUnusableBody(WireMockRuntimeInfo wireMock) { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.post(urlPathEqualTo("/api/v1/app-logs/search")) + .willReturn(aResponse().withStatus(200).withBody("not json at all")) + ); + + var task = Search.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("*")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + assertThat(thrown.getMessage(), org.hamcrest.Matchers.containsString("not valid JSON")); + } + + @Test + void rejectsABlankIndex() throws IllegalVariableEvaluationException { + var task = Search.builder() + .url(Property.ofValue("http://localhost:7280")) + .index(Property.ofValue(" ")) + .query(Property.ofValue("*")) + .build(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> task.run(runContextFactory.of())); + assertThat(thrown.getMessage(), is("`index` is required and cannot be blank")); + } +} \ No newline at end of file diff --git a/src/test/java/io/kestra/plugin/quickwit/search/TriggerTest.java b/src/test/java/io/kestra/plugin/quickwit/search/TriggerTest.java new file mode 100644 index 0000000..76bdcd8 --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/search/TriggerTest.java @@ -0,0 +1,272 @@ +package io.kestra.plugin.quickwit.search; + +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.List; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.models.conditions.ConditionContext; +import io.kestra.core.models.property.Property; +import io.kestra.core.storages.kv.KVMetadata; +import io.kestra.core.storages.kv.KVValueAndMetadata; +import io.kestra.core.utils.IdUtils; +import io.kestra.core.utils.TestsUtils; +import io.kestra.plugin.quickwit.AbstractQuickwitTest; + +import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class TriggerTest extends AbstractQuickwitTest { + private static final String TWO_HITS = """ + {"num_hits": 2, "elapsed_time_micros": 10, "hits": [{"timestamp": 1700000001, "message": "one"}, {"timestamp": 1700000002, "message": "two"}]} + """; + + private static final String NO_HIT = """ + {"num_hits": 0, "hits": []} + """; + + @Test + void firesOnTheFirstPoll(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(TWO_HITS))); + + var context = contextFor(trigger(wireMock)); + + assertThat(context.trigger().evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(true)); + } + + @Test + void doesNotFireWhenTheQueryMatchesNothing(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(NO_HIT))); + + var context = contextFor(trigger(wireMock)); + + assertThat(context.trigger().evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(false)); + } + + @Test + void exposesTheNewDocumentsAndTheWatermark(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(TWO_HITS))); + + var context = contextFor(trigger(wireMock)); + var execution = context.trigger().evaluate(context.conditionContext(), context.triggerContext()).orElseThrow(); + + var variables = execution.getTrigger().getVariables(); + + assertThat((List) variables.get("documents"), hasSize(2)); + assertThat(variables.get("numHits"), is(2L)); + assertThat(variables.get("watermark"), is(1_700_000_003L)); + assertThat(watermark(context), is("1700000003")); + } + + @Test + void storesTheWatermarkSoTheSameDocumentIsNeverDeliveredTwice(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(TWO_HITS))); + + var context = contextFor(trigger(wireMock)); + + assertThat(context.trigger().evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(true)); + + var stored = watermark(context); + assertThat("the watermark must be one past the maximum delivered timestamp", stored, is("1700000003")); + + assertEquals( + stored, + new String((byte[]) context.conditionContext().getRunContext() + .namespaceKv(context.triggerContext().getNamespace()) + .getValue(context.stateKey()) + .orElseThrow() + .value(), StandardCharsets.UTF_8), + "the watermark must be persisted under the trigger's state key" + ); + } + + @Test + void doesNotFireAgainWhileTheQueryKeepsReturningNothingNew(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(TWO_HITS))); + + var context = contextFor(trigger(wireMock)); + var trigger = context.trigger(); + + assertThat(trigger.evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(true)); + + // second poll: Quickwit only returns documents at or after the stored watermark + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(NO_HIT))); + + assertThat(trigger.evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(false)); + } + + @Test + void deliversOverflowBeyondMaxHitsOnALaterPoll(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson( + """ + {"num_hits": 2, "hits": [{"timestamp": 1700000001, "message": "one"}]} + """))); + + var context = contextFor(trigger(wireMock)); + var trigger = context.trigger(); + + assertThat(trigger.evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(true)); + assertThat(watermark(context), is("1700000002")); + + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson( + """ + {"num_hits": 1, "hits": [{"timestamp": 1700000002, "message": "two"}]} + """))); + + var second = trigger.evaluate(context.conditionContext(), context.triggerContext()).orElseThrow(); + assertThat((List) second.getTrigger().getVariables().get("documents"), hasSize(1)); + assertThat(watermark(context), is("1700000003")); + } + + @Test + void readsRfc3339Timestamps(WireMockRuntimeInfo wireMock) throws Exception { + // rfc3339 is the default `output_format` of a Quickwit datetime field + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson( + """ + {"num_hits": 2, "hits": [{"timestamp": "2023-11-14T22:13:21Z", "message": "one"}, {"timestamp": "2023-11-14T22:13:22.5+00:00", "message": "two"}]} + """))); + + var context = contextFor(trigger(wireMock)); + + assertThat(context.trigger().evaluate(context.conditionContext(), context.triggerContext()).isPresent(), is(true)); + assertThat(watermark(context), is("1700000003")); + } + + @Test + void holdsBackTheLastSecondOfATruncatedPage(WireMockRuntimeInfo wireMock) throws Exception { + // 3 documents match, the page holds 2 and ends inside second 1700000002 + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson( + """ + {"num_hits": 3, "hits": [{"timestamp": 1700000001, "message": "one"}, {"timestamp": 1700000002, "message": "two"}]} + """))); + + var context = contextFor(trigger(wireMock)); + var execution = context.trigger().evaluate(context.conditionContext(), context.triggerContext()).orElseThrow(); + + assertThat((List) execution.getTrigger().getVariables().get("documents"), hasSize(1)); + assertThat("second 1700000002 must be read again in full", watermark(context), is("1700000002")); + } + + @Test + void rejectsANonPositiveMaxHits(WireMockRuntimeInfo wireMock) { + var context = contextFor(Trigger.builder() + .id(IdUtils.create()) + .type(TriggerTest.class.getName()) + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .timestampField(Property.ofValue("timestamp")) + .maxHits(Property.ofValue(0)) + .build()); + + var thrown = assertThrows( + IllegalArgumentException.class, + () -> context.trigger().evaluate(context.conditionContext(), context.triggerContext()) + ); + assertThat(thrown.getMessage(), is("`maxHits` must be at least 1, got 0")); + } + + @Test + void failsWhenADocumentMissesTheTimestampField(WireMockRuntimeInfo wireMock) { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson( + """ + {"num_hits": 1, "hits": [{"message": "no timestamp here"}]} + """))); + + var context = contextFor(trigger(wireMock)); + + var thrown = assertThrows( + IllegalStateException.class, + () -> context.trigger().evaluate(context.conditionContext(), context.triggerContext()) + ); + assertThat(thrown.getMessage(), is("Document at position 0 misses timestamp field 'timestamp', cannot advance the watermark")); + } + + @Test + void searchesOnlyWhatCameAfterTheWatermark(WireMockRuntimeInfo wireMock) throws Exception { + var trigger = trigger(wireMock); + var context = contextFor(trigger); + + // a watermark left by a previous poll must be sent as start_timestamp + putWatermark(context, 1_700_000_000L); + + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(NO_HIT))); + trigger.evaluate(context.conditionContext(), context.triggerContext()); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/search")) + .withRequestBody(equalToJson("{\"query\": \"severity:ERROR\", \"start_timestamp\": 1700000000, \"sort_by\": \"-timestamp\"}")) + ); + } + + @Test + void sortsAscendingOnTheTimestampField(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/app-logs/search")).willReturn(okJson(NO_HIT))); + + var context = contextFor(trigger(wireMock)); + context.trigger().evaluate(context.conditionContext(), context.triggerContext()); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/app-logs/search")) + .withRequestBody(equalToJson("{\"query\": \"severity:ERROR\", \"sort_by\": \"-timestamp\"}")) + ); + } + + private Trigger trigger(WireMockRuntimeInfo wireMock) { + return Trigger.builder() + .id(IdUtils.create()) + .type(TriggerTest.class.getName()) + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .query(Property.ofValue("severity:ERROR")) + .timestampField(Property.ofValue("timestamp")) + .build(); + } + + /** The plugin trigger together with the context pair `TestsUtils.mockTrigger` builds for it. */ + private record Context(ConditionContext conditionContext, io.kestra.core.models.triggers.Trigger triggerContext, Trigger trigger) { + String stateKey() { + String namespace = triggerContext.getNamespace(); + String flowId = triggerContext.getFlowId(); + + return namespace.length() + "-" + namespace + "_" + flowId.length() + "-" + flowId + "_" + trigger.getId().length() + "-" + trigger.getId(); + } + } + + private Context contextFor(Trigger trigger) { + var entry = TestsUtils.mockTrigger(runContextFactory, trigger); + + return new Context(entry.getKey(), entry.getValue(), trigger); + } + + private String watermark(Context context) throws Exception { + return new String((byte[]) context.conditionContext().getRunContext() + .namespaceKv(context.triggerContext().getNamespace()) + .getValue(context.stateKey()) + .orElseThrow() + .value(), StandardCharsets.UTF_8).trim(); + } + + private void putWatermark(Context context, long watermark) throws Exception { + context.conditionContext().getRunContext() + .namespaceKv(context.triggerContext().getNamespace()) + .put( + context.stateKey(), + new KVValueAndMetadata( + new KVMetadata("watermark", (Duration) null), + Long.toString(watermark).getBytes(StandardCharsets.UTF_8) + ) + ); + } +} diff --git a/src/test/java/io/kestra/plugin/quickwit/source/SourceTasksTest.java b/src/test/java/io/kestra/plugin/quickwit/source/SourceTasksTest.java new file mode 100644 index 0000000..1c940e8 --- /dev/null +++ b/src/test/java/io/kestra/plugin/quickwit/source/SourceTasksTest.java @@ -0,0 +1,215 @@ +package io.kestra.plugin.quickwit.source; + +import java.util.Map; + +import org.junit.jupiter.api.Test; + +import com.github.tomakehurst.wiremock.junit5.WireMockRuntimeInfo; + +import io.kestra.core.models.property.Property; +import io.kestra.plugin.quickwit.AbstractQuickwitTest; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.deleteRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.put; +import static com.github.tomakehurst.wiremock.client.WireMock.putRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.stubFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.verify; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.is; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * Happy-path and error coverage for every task of the source management package. + */ +class SourceTasksTest extends AbstractQuickwitTest { + private static final String SOURCE = """ + { + "version": "0.8", + "source_id": "kafka-source", + "source_type": "kafka", + "num_pipelines": 2, + "params": {"topic": "app-logs", "client_params": {"bootstrap.servers": "kafka:9092"}} + } + """; + + private static final Map PARAMS = Map.of( + "topic", "app-logs", + "client_params", Map.of("bootstrap.servers", "kafka:9092") + ); + + @Test + void create(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(post(urlPathEqualTo("/api/v1/indexes/app-logs/sources")).willReturn(okJson(SOURCE))); + + var output = Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("kafka-source")) + .configVersion(Property.ofValue("0.8")) + .sourceType(Property.ofValue("kafka")) + .numPipelines(Property.ofValue(2)) + .params(Property.ofValue(PARAMS)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getSource().getSourceId(), is("kafka-source")); + assertThat(output.getSource().getSourceType(), is("kafka")); + assertThat(output.getSource().getNumPipelines(), is(2)); + assertThat(output.getSource().getParams().get("topic"), is("app-logs")); + + verify(postRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs/sources")) + .withRequestBody(equalToJson(""" + { + "version": "0.8", + "source_id": "kafka-source", + "source_type": "kafka", + "params": {"topic": "app-logs", "client_params": {"bootstrap.servers": "kafka:9092"}}, + "num_pipelines": 2 + } + """)) + ); + } + + @Test + void createRejectsEmptyParams(WireMockRuntimeInfo wireMock) { + var task = Create.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("kafka-source")) + .sourceType(Property.ofValue("kafka")) + .params(Property.ofValue(Map.of())) + .build(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> task.run(runContextFactory.of())); + assertThat(thrown.getMessage(), containsString("`params` is required")); + } + + @Test + void toggleEnableAcceptsTheEmptyBodyQuickwitReturns(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(put(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source/toggle")) + .willReturn(aResponse().withStatus(200)) + ); + + var output = Toggle.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("kafka-source")) + .enable(Property.ofValue(true)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getSource(), is("kafka-source")); + assertThat(output.getEnable(), is(true)); + + verify(putRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source/toggle")) + .withRequestBody(equalToJson("{\"enable\": true}")) + ); + } + + @Test + void toggleDisable(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(put(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source/toggle")) + .willReturn(aResponse().withStatus(200)) + ); + + var output = Toggle.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("kafka-source")) + .enable(Property.ofValue(false)) + .build() + .run(runContextFactory.of()); + + assertThat(output.getEnable(), is(false)); + + verify(putRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source/toggle")) + .withRequestBody(equalToJson("{\"enable\": false}")) + ); + } + + @Test + void resetCheckpointAcceptsTheEmptyBodyQuickwitReturns(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(put(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source/reset-checkpoint")) + .willReturn(aResponse().withStatus(200)) + ); + + var output = ResetCheckpoint.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("kafka-source")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getSource(), is("kafka-source")); + + verify(putRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source/reset-checkpoint"))); + } + + @Test + void deletesSource(WireMockRuntimeInfo wireMock) throws Exception { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.delete(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source")) + .willReturn(aResponse().withStatus(200)) + ); + + var output = Delete.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("kafka-source")) + .build() + .run(runContextFactory.of()); + + assertThat(output.getIndex(), is("app-logs")); + assertThat(output.getSource(), is("kafka-source")); + + verify(deleteRequestedFor(urlPathEqualTo("/api/v1/indexes/app-logs/sources/kafka-source"))); + } + + @Test + void surfacesTheQuickwitErrorOnAnUnknownSource(WireMockRuntimeInfo wireMock) { + stubFor(com.github.tomakehurst.wiremock.client.WireMock.delete(urlPathEqualTo("/api/v1/indexes/app-logs/sources/nope")) + .willReturn(aResponse() + .withStatus(404) + .withHeader("Content-Type", "application/json") + .withBody("{\"message\": \"Source nope does not exist\"}") + ) + ); + + var task = Delete.builder() + .url(Property.ofValue(url(wireMock))) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("nope")) + .build(); + + var thrown = assertThrows(IllegalStateException.class, () -> task.run(runContextFactory.of())); + + assertThat( + thrown.getMessage(), + is("Quickwit deletion of source 'nope' on index 'app-logs' failed with HTTP 404: Source nope does not exist") + ); + } + + @Test + void rejectsABlankSourceId() throws Exception { + var task = Toggle.builder() + .url(Property.ofValue("http://localhost:7280")) + .index(Property.ofValue("app-logs")) + .source(Property.ofValue("")) + .enable(Property.ofValue(true)) + .build(); + + var thrown = assertThrows(IllegalArgumentException.class, () -> task.run(runContextFactory.of())); + + assertThat(thrown.getMessage(), is("`source` is required and cannot be blank")); + assertThat(thrown.getMessage(), is(org.hamcrest.Matchers.notNullValue())); + } +} \ No newline at end of file diff --git a/src/test/resources/flows/quickwit_ingest.yaml b/src/test/resources/flows/quickwit_ingest.yaml new file mode 100644 index 0000000..a9bd93e --- /dev/null +++ b/src/test/resources/flows/quickwit_ingest.yaml @@ -0,0 +1,23 @@ +id: quickwit_ingest +namespace: company.team + +tasks: + - id: ingest + type: io.kestra.plugin.quickwit.ingest.Ingest + url: "http://localhost:7280" + index: app-logs + commit: FORCE + detailedResponse: true + from: + - timestamp: "{{ now() | timestamp }}" + service: checkout + severity: ERROR + message: "Payment gateway timeout" + - timestamp: "{{ now() | timestamp }}" + service: checkout + severity: ERROR + message: "Payment gateway timeout" + + - id: check + type: io.kestra.plugin.core.log.Log + message: "Ingested {{ outputs.ingest.numIngestedDocs }} / rejected {{ outputs.ingest.numRejectedDocs }}" diff --git a/src/test/resources/flows/quickwit_search.yaml b/src/test/resources/flows/quickwit_search.yaml new file mode 100644 index 0000000..aad3aa9 --- /dev/null +++ b/src/test/resources/flows/quickwit_search.yaml @@ -0,0 +1,16 @@ +id: quickwit_search +namespace: company.team + +tasks: + - id: search + type: io.kestra.plugin.quickwit.search.Search + url: "https://quickwit.example.com:7280" + index: app-logs + query: "severity:ERROR" + startTimestamp: "{{ now() | dateAdd(-1, 'HOURS') | timestamp }}" + maxHits: 100 + fetchType: FETCH + + - id: log_count + type: io.kestra.plugin.core.log.Log + message: "Found {{ outputs.search.size }} errors" diff --git a/src/test/resources/flows/quickwit_trigger.yaml b/src/test/resources/flows/quickwit_trigger.yaml new file mode 100644 index 0000000..970eb1f --- /dev/null +++ b/src/test/resources/flows/quickwit_trigger.yaml @@ -0,0 +1,16 @@ +id: quickwit_error_watch +namespace: company.team + +triggers: + - id: on_errors + type: io.kestra.plugin.quickwit.search.Trigger + url: "https://quickwit.example.com:7280" + index: app-logs + query: "severity:ERROR" + timestampField: timestamp + interval: PT5M + +tasks: + - id: notify + type: io.kestra.plugin.core.log.Log + message: "New errors: {{ trigger.documents | length }}"