refactor(search): unify incremental synchronous indexing

[hlubek]

MCP backfill embedded one entry at a time, leaving small requests underfilled; CLI indexing prepared every pending chunk before embedding. Both now use one synchronous iterator that fills batches across entries while retaining only active work and publishing complete entry versions as they finish.

CLI progress counts published entries without a chunk-count pre-pass. Remove embed.NewBatcher and its exclusive exports: consumers own concurrent batching through Embedder. Keep embed.Batched for request splitting. Consumers must migrate before adopting this API removal.

Validated with incremental-consumption, vector-ownership and interruption/retry tests; full repository and example-module tests, vet, lint, build and graph smoke check pass.

Greptile Summary

This PR consolidates CLI and application search indexing around an incremental synchronous pipeline.

• Packs chunks across entry boundaries while publishing only complete entry versions.
• Preserves completed work across interruption and skips published versions on retry.
• Changes CLI progress from chunk totals to published-entry totals.
• Retains embed.Batched request splitting while removing the concurrent batcher API.
• Adds coverage for incremental consumption, vector ownership, empty and oversized entries, invalid vectors, and interruption recovery.

Confidence Score: 4/5

The PR appears safe to merge, with only a non-blocking misleading progress note for empty entries.

The indexing and publication paths preserve complete-entry ownership, retries, configured batching, and store compatibility; the only accepted concern is CLI presentation when publication occurs without a batch callback.

Files Needing Attention: cmd/sdd/progress.go

Important Files Changed

Filename	Overview
internal/handlers/index_pipeline.go	Introduces the shared incremental preparation, batching, embedding, and complete-entry publication pipeline.
internal/model/index_work.go	Tracks chunk ownership across batches and identifies entries ready for publication.
internal/handlers/handler_index.go	Migrates CLI build and lazy-fill indexing to the shared stream while preserving per-entry manifest durability and force behavior.
internal/handlers/search_index.go	Migrates synchronous application reconciliation to incremental cross-entry batching with publication-aware retry skipping.
cmd/sdd/progress.go	Reports published entries and cumulative chunks, but batchless empty-entry publication can render an unset or stale activity note.
cmd/sdd/serve.go	Preserves the IndexEmbedder wrapper in runtime composition so synchronous indexing receives its configured batch size.
pkg/application/types/search_entry.go	Centralizes publication vector validation through the internal vector validator.
internal/model/batches.go	Adds a lazy fixed-size batching iterator that flushes pending work before propagating source errors.

Prompt To Fix All With AI

### Issue 1
cmd/sdd/progress.go:50
**Empty entries show stale progress**

When an entry has no chunks, it is published without calling `OnBatchStart`, but this code still formats the note using `batchNote`. The first empty entry therefore shows only ` · 0 chunks published`, while an empty entry in connected-repository indexing can keep showing the previous repository’s batch. Clear or construct the note for publications that have no batch.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "sdd: signal tactical Implemented increme..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.