diff --git a/integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java b/integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java index 5d66e479f..4be3d6be1 100644 --- a/integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java +++ b/integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java @@ -112,7 +112,7 @@ public class ElasticsearchVectorStore extends BaseVectorStore /** Default vector dimensionality used when {@code dims} is not provided. */ public static final int DEFAULT_DIMENSION = 768; - /** The maximum number of documents that can be retrieved in get. */ + /** Default request size used by {@code get} when no limit is provided. */ public static final int MAX_RESULT_WINDOW = 10000; public static final String DEFAULT_METADATA_FIELD = "_metadata"; @@ -336,23 +336,31 @@ public Map getStoreKwargs() { /** * Retrieve documents from the vector store. * - *

If ids is not provided, this method will retrieve documents according to {@code limit}, - * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null, - * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an - * Elasticsearch ceiling). + *

When {@code ids} is non-empty, documents are retrieved directly by ID and the filter, + * limit, offset, and {@code filter_query} arguments are not applied. * - *

The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native - * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via - * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma - * implementation. + *

Otherwise, {@code filters} provides equality-only matching against document metadata. Each + * entry is translated to an Elasticsearch {@code term} query on {@code + * ..keyword}, and multiple entries are combined with AND semantics. Because + * Elasticsearch dynamic mapping creates {@code .keyword} sub-fields only for strings, filters + * on non-string metadata values do not match; use a raw {@code filter_query} for those values. + * A raw Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code + * extraArgs}; when both forms are present, they are combined with AND semantics. * - * @param ids The ids of the documents. - * @param collection The name of the collection to be retrieved. If is null, retrieve the - * default collection. - * @param filters Unified filter DSL. Currently ignored — see method Javadoc. - * @param limit Maximum number of documents to return; falls back to {@link - * ElasticsearchVectorStore#MAX_RESULT_WINDOW} when null. - * @param extraArgs Additional arguments. (offset, filter_query, etc.) + *

The {@code limit} parameter takes precedence over a {@code limit} value in {@code + * extraArgs}. If neither limit is provided, the request size defaults to {@link + * ElasticsearchVectorStore#MAX_RESULT_WINDOW} (10,000). The effective Elasticsearch + * result-window limit is controlled by the target index's {@code index.max_result_window} + * setting, which defaults to 10,000 but is configurable. Elasticsearch rejects requests when + * the combined {@code offset} and request size exceed that setting rather than truncating them. + * {@code extraArgs} may also contain an {@code offset}. + * + * @param ids The IDs of documents to retrieve directly. + * @param collection The collection name, or null to use the default collection. + * @param filters Equality-only metadata filters combined with AND semantics. + * @param limit Maximum number of documents to return for filtered or unfiltered searches. + * @param extraArgs Additional arguments, including {@code offset} and raw JSON {@code + * filter_query}. * @return List of documents retrieved. */ @Override @@ -379,22 +387,24 @@ public List get( } /** - * Delete documents in the vector store. + * Delete documents from the vector store. * - *

If ids is not provided, this method will delete documents matched the {@code filter_query} - * in additional arguments. If neither {@code filter_query} nor {@code filters} is provided, - * this method will delete all the documents. + *

When {@code ids} is non-empty, documents are deleted directly by ID and {@code filters} + * and {@code filter_query} are not applied. * - *

The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native - * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via - * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma - * implementation. + *

Otherwise, {@code filters} provides equality-only matching against document metadata. Each + * entry is translated to an Elasticsearch {@code term} query on {@code + * ..keyword}, and multiple entries are combined with AND semantics. Because + * Elasticsearch dynamic mapping creates {@code .keyword} sub-fields only for strings, filters + * on non-string metadata values do not match; use a raw {@code filter_query} for those values. + * A raw Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code + * extraArgs}; when both forms are present, they are combined with AND semantics. If neither + * form is supplied, all documents in the collection are deleted. * - * @param ids The ids of the documents. - * @param collection The name of the collection the documents belong to. If is null, use the - * default collection. - * @param filters Unified filter DSL. Currently ignored — see method Javadoc. - * @param extraArgs Additional arguments. (filter_query, etc.) + * @param ids The IDs of documents to delete directly. + * @param collection The collection name, or null to use the default collection. + * @param filters Equality-only metadata filters combined with AND semantics. + * @param extraArgs Additional arguments, including raw JSON {@code filter_query}. */ @Override public void delete( @@ -571,19 +581,26 @@ private void deleteDocuments( * Executes a KNN vector search using a pre-computed embedding. * *

The method prepares a KNN search request using the supplied {@code embedding} and merges - * default arguments from the store with the provided {@code args}. Optional filter queries - * (JSON DSL) restrict the documents the KNN search may match, so the nearest neighbours are + * default arguments from the store with the provided {@code args}. {@code filters} provides + * equality-only matching against metadata fields. Each entry targets {@code + * ..keyword}; multiple entries are combined with AND semantics. Because + * Elasticsearch dynamic mapping creates {@code .keyword} sub-fields only for strings, filters + * on non-string metadata values do not match; use a raw {@code filter_query} for those values. + * + *

A raw Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code + * args}. When both filter forms are present, they are combined with AND semantics. The combined + * filter restricts the documents the KNN search may match, so the nearest neighbours are * selected from among the matching documents rather than filtered out afterwards. Up to {@code * k} matching documents are returned even when the closest vectors overall do not match. * - * @param embedding The embedding vector to search with - * @param limit Maximum number of items the caller is interested in; used as a fallback for - * {@code k} if not explicitly provided - * @param collection The index to query search. If is null, search the default index. - * @param args Additional arguments. Supported keys: {@code k}, {@code num_candidates}, {@code - * filter_query} - * @return A list of matching documents, possibly empty - * @throws RuntimeException if the search request fails + * @param embedding The embedding vector to search with. + * @param limit Maximum number of items requested; used as a fallback for {@code k}. + * @param collection The collection name, or null to use the default collection. + * @param filters Equality-only metadata filters combined with AND semantics. + * @param args Additional arguments. Supported keys are {@code k}, {@code num_candidates}, and + * raw JSON {@code filter_query}. + * @return A list of matching documents, possibly empty. + * @throws RuntimeException if the search request fails. */ @SuppressWarnings({"rawtypes", "unchecked"}) @Override