fix: shrink oversized embedding input instead of dropping the document

`createEmbeddingOrEmpty` logs a failed embedding as "transient/non-fatal" and
moves on. For a rate limit that is right. For an over-length input it is not:
the input is deterministic, so the same document fails on every rebuild, and
the index keeps a permanent hole that looks like a passing run.

Observed on a 567-table MySQL connection during a full reindex:

    Skipping embedding for RELATIONSHIP document b7efe9d2-... due to
    transient/non-fatal error: 400: Invalid 'input': maximum context length
    is 8192 tokens.

The cause is the character budget, not a missing one. `truncate` already cut
input to `app.embedding.max-chars` (default 30,000) on the stated assumption of
"roughly 4 chars per token", which would be 7,500 tokens. That ratio holds for
prose. It does not hold for what this service embeds: schema and relationship
documents are dense identifiers, underscores, punctuation and repeated
scaffolding, which tokenize closer to 2-3 chars per token. At that density
30,000 chars is 10,000-15,000 tokens and the provider rejects the call.

No fixed ratio is safe across content, so this stops betting on one.
`LlmErrorCategory.CONTEXT_LENGTH` already documents itself as "never retry; the
caller may trim" — until now nothing trimmed. `embedWithShrink` halves the
budget on each CONTEXT_LENGTH rejection (30,000 -> 15,000 -> ... -> 1,875,
floor 1,000) and lets the provider decide when the call fits. That needs no
tokenizer dependency and stays correct for any content and any model window.

Deliberately narrow:
- Only CONTEXT_LENGTH shrinks. Retries and fail-open are untouched for every
  other category, since a smaller input answers nothing about a rate limit or
  a rejected credential.
- The inner attempt runs with fail-open off so the rejection reaches the shrink
  loop; fail-open would convert it into an empty vector indistinguishable from
  a real one. The operator's fail-open setting is still honoured once shrinking
  is exhausted.
- Shrinking a batch only affects members longer than the budget, so one
  oversized text costs the short ones nothing.
- Bounded at 4 halvings, so a pathological document cannot loop.

Tests cover all three paths: shrink-then-succeed, exhaust-then-fail-open, and
no-shrink for a category shrinking cannot fix.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: geekypunk

backend/src/main/java/com/dbaagent/service/EmbeddingService.java

@@ -31,6 +31,10 @@ public class EmbeddingService {
 
     /** Retry budget: 2 retries after the first try, matching the pre-delegation service. */
     private static final int MAX_RETRY_ATTEMPTS = 2;
+    /** Halvings allowed before giving up: 30,000 chars reaches ~1,875 in four. */
+    private static final int MAX_SHRINK_ATTEMPTS = 4;
+    /** Floor for shrinking. Below this the document is too small to be worth indexing. */
+    private static final int MIN_EMBEDDING_CHARS = 1_000;
     private static final long RETRY_BACKOFF_MS = 750L;
 
     /**
@@ -87,9 +91,23 @@ public EmbeddingService(
         this.maxRetryAfter = maxRetryAfter;
     }
 
-    /** text-embedding-3-large accepts 8192 tokens, roughly 4 chars per token. */
-    private String truncate(String text) {
-        return (text != null && text.length() > maxChars) ? text.substring(0, maxChars) : text;
+    /**
+     * Cut {@code text} to a character budget.
+     *
+     * <p>The budget is a <em>guess</em> at the model's token window, and it is wrong often
+     * enough to matter. The old default assumed "roughly 4 chars per token", which holds
+     * for prose but not for what this service actually embeds: schema and relationship
+     * documents are dense identifiers — {@code ORDERS.customer_id}, underscores,
+     * punctuation, repeated scaffolding — that tokenize closer to 2-3 chars per token. At
+     * that density the 30,000-char default is 10,000-15,000 tokens, well past the 8,192 a
+     * text-embedding-3-large call accepts, and the provider rejects the request outright.
+     *
+     * <p>No fixed ratio is safe across content, so the budget is not trusted to be right.
+     * {@link #embedWithShrink} lets the provider's own CONTEXT_LENGTH rejection drive the
+     * budget down until the call fits.
+     */
+    private String truncate(String text, int budget) {
+        return (text != null && text.length() > budget) ? text.substring(0, budget) : text;
     }
 
     /**
@@ -98,8 +116,52 @@ private String truncate(String text) {
     public List<Double> createEmbedding(String text) {
         LlmCredentials credentials = requireCredentials();
         LlmEmbeddingProvider provider = registry.embeddingProvider(credentials.providerId());
-        return attempt(provider, credentials,
-                () -> provider.embed(truncate(text), credentials), List.of());
+        return embedWithShrink(provider, credentials,
+                budget -> provider.embed(truncate(text, budget), credentials),
+                List.of());
+    }
+
+    /**
+     * Run an embedding call, halving the character budget each time the provider says the
+     * input is too long.
+     *
+     * <p>{@link LlmErrorCategory#CONTEXT_LENGTH} already documents itself as "never retry;
+     * the caller may trim" — but until now no caller trimmed. The rejection fell through to
+     * fail-open, the document was skipped, and because the input is deterministic it was
+     * skipped again on every subsequent rebuild. That is a permanent hole in the index
+     * wearing the costume of a transient blip.
+     *
+     * <p>Shrinking is deliberately driven by the provider rather than by counting tokens
+     * locally: it needs no tokenizer dependency, and it stays correct for any content and
+     * any model window, including ones whose ratio we have never measured.
+     *
+     * <p>Only CONTEXT_LENGTH shrinks. Every other category keeps its existing behaviour —
+     * retries and fail-open are unchanged — because shrinking the input answers nothing
+     * about a rate limit or a bad credential.
+     */
+    private <T> T embedWithShrink(LlmEmbeddingProvider provider, LlmCredentials credentials,
+                                  java.util.function.IntFunction<T> call, T failOpenValue) {
+        int budget = maxChars;
+        for (int shrink = 0; ; shrink++) {
+            final int attemptBudget = budget;
+            try {
+                // mayFailOpen=false: a CONTEXT_LENGTH rejection has to reach us. Fail-open
+                // would turn it into an empty vector indistinguishable from a real one.
+                return attempt(provider, credentials, () -> call.apply(attemptBudget), failOpenValue, false);
+            } catch (RuntimeException e) {
+                LlmErrorCategory category = provider.classify(e);
+                if (category != LlmErrorCategory.CONTEXT_LENGTH
+                        || shrink >= MAX_SHRINK_ATTEMPTS
+                        || budget <= MIN_EMBEDDING_CHARS) {
+                    // Out of room to shrink, or not a length problem: honour the operator's
+                    // fail-open setting exactly as before this method existed.
+                    return handleFailure(category, credentials, e, failOpenValue, failOpen);
+                }
+                int next = Math.max(MIN_EMBEDDING_CHARS, budget / 2);
+                log.warn("Embedding input rejected as too long at {} chars; retrying at {}", budget, next);
+                budget = next;
+            }
+        }
     }
 
     /**
@@ -110,9 +172,11 @@ public List<Double> createEmbedding(String text) {
     public List<List<Double>> createEmbeddings(List<String> texts) {
         LlmCredentials credentials = requireCredentials();
         LlmEmbeddingProvider provider = registry.embeddingProvider(credentials.providerId());
-        List<String> truncated = texts.stream().map(this::truncate).toList();
-        return attempt(provider, credentials,
-                () -> provider.embedBatch(truncated, credentials),
+        // Shrinking the budget only touches texts longer than it, so one oversized member
+        // of a batch cannot cost the short ones any content.
+        return embedWithShrink(provider, credentials,
+                budget -> provider.embedBatch(
+                        texts.stream().map(t -> truncate(t, budget)).toList(), credentials),
                 Collections.nCopies(texts.size(), List.of()));
     }
 

backend/src/test/java/com/dbaagent/service/EmbeddingServiceTest.java

@@ -476,4 +476,73 @@ void cosineSimilarityOfIdenticalVectorsIsOne() {
         assertThat(service.cosineSimilarity(List.of(1.0, 2.0, 3.0), List.of(1.0, 2.0, 3.0)))
                 .isCloseTo(1.0, org.assertj.core.data.Offset.offset(1e-9));
     }
+
+    @Test
+    void shrinksTheInputWhenTheProviderRejectsItAsTooLong() {
+        var resolver = mock(LlmConfigResolver.class);
+        var registry = mock(LlmProviderRegistry.class);
+        var provider = mock(LlmEmbeddingProvider.class);
+
+        when(resolver.resolveEmbedding()).thenReturn(creds());
+        when(registry.embeddingProvider("openai")).thenReturn(provider);
+        // Stands in for a model whose real token window is reached well before the
+        // character budget: anything over 15,000 chars is rejected outright.
+        when(provider.embed(anyString(), any())).thenAnswer(call -> {
+            String sent = call.getArgument(0);
+            if (sent.length() > 15_000) {
+                throw new RuntimeException("maximum context length is 8192 tokens");
+            }
+            return List.of(0.5);
+        });
+        when(provider.classify(any())).thenReturn(LlmErrorCategory.CONTEXT_LENGTH);
+
+        var service = new EmbeddingService(resolver, registry, 30_000, false);
+
+        // Without shrinking this document is dropped forever; the point of the fix is
+        // that it comes back embedded rather than empty.
+        assertThat(service.createEmbedding("x".repeat(50_000))).containsExactly(0.5);
+
+        var captor = ArgumentCaptor.forClass(String.class);
+        verify(provider, times(2)).embed(captor.capture(), any());
+        assertThat(captor.getAllValues().get(0)).hasSize(30_000);
+        assertThat(captor.getAllValues().get(1)).hasSize(15_000);
+    }
+
+    @Test
+    void givesUpAfterTheShrinkFloorAndStillHonoursFailOpen() {
+        var resolver = mock(LlmConfigResolver.class);
+        var registry = mock(LlmProviderRegistry.class);
+        var provider = mock(LlmEmbeddingProvider.class);
+
+        when(resolver.resolveEmbedding()).thenReturn(creds());
+        when(registry.embeddingProvider("openai")).thenReturn(provider);
+        when(provider.embed(anyString(), any()))
+                .thenThrow(new RuntimeException("maximum context length is 8192 tokens"));
+        when(provider.classify(any())).thenReturn(LlmErrorCategory.CONTEXT_LENGTH);
+
+        var service = new EmbeddingService(resolver, registry, 30_000, true);
+
+        // Shrinking is bounded, so a pathological document cannot loop forever.
+        assertThat(service.createEmbedding("x".repeat(50_000))).isEmpty();
+        verify(provider, times(5)).embed(anyString(), any()); // 30k, 15k, 7.5k, 3.75k, 1.875k
+    }
+
+    @Test
+    void doesNotShrinkForFailuresThatShrinkingCannotFix() {
+        var resolver = mock(LlmConfigResolver.class);
+        var registry = mock(LlmProviderRegistry.class);
+        var provider = mock(LlmEmbeddingProvider.class);
+
+        when(resolver.resolveEmbedding()).thenReturn(creds());
+        when(registry.embeddingProvider("openai")).thenReturn(provider);
+        when(provider.embed(anyString(), any())).thenThrow(new RuntimeException("nope"));
+        when(provider.classify(any())).thenReturn(LlmErrorCategory.AUTH);
+
+        var service = new EmbeddingService(resolver, registry, 30_000, true);
+
+        assertThat(service.createEmbedding("x".repeat(50_000))).isEmpty();
+        // A rejected credential says nothing about input size; retrying smaller would
+        // just multiply the failed calls.
+        verify(provider, times(1)).embed(anyString(), any());
+    }
 }