docpipe

Document ingestion for retrieval systems: get text out of a document, split it into units worth retrieving, and turn those into vectors.

Extracted from a production Thai government document system, where the hard part was never the embedding call — it was everything around it.

Chunking legal documents by their own structure instead of by character count is worth 5–10× in retrieval precision. Each statute uploaded as a single file, scored on whether the best-matching chunk identifies the clause that answers the question:

	Document	char-window	structure-aware
Thai	กฎกระทรวง on state property (54 ข้อ)	8%	83%
Vietnamese	Luật nghĩa vụ quân sự 2015 (62 Điều)	16%	75%
EU English	GDPR — Regulation (EU) 2016/679 (99 Articles)	33%	91%
Japanese	個人情報保護法 / APPI (185 条)	0%	83%

Four languages, four legal domains, four drafting traditions — and the baselines are predictable from clause density, which was tested by predicting the Japanese result before running it. Reproduce any of them in one command — cargo run --release --example eval_legal_chunking [-- vi|eu|ja] — see eval/ for the setup, every miss, and why the window collapses: its chunks retrieve relevant text but cannot be cited, which for legal work makes them worthless.

use docpipe::{ai::{AiClient, AiConfig}, pipeline, profile::ChunkCfg};

let ai = AiClient::new(AiConfig::from_env());
let out = pipeline::prepare(&ai, text, &ChunkCfg::default(), true).await?;

for (chunk, vector) in out.chunks.iter().zip(out.vectors.iter()) {
    // index chunk.content at chunk.char_start..chunk.char_end
}

What's actually here

Structure-aware chunking for legal documents. Statutes and regulations are split by their own units, with the chapter breadcrumb and clause label prepended, so each chunk is a self-contained, citable clause instead of an arbitrary window that cuts a clause in half:

หมวด ๒ การจัดหาประโยชน์ › ข้อ ๙
กำหนดระยะเวลาเช่าไม่เกินสามสิบปี
เว้นแต่มีเหตุจำเป็น

Two drafting traditions ship today and the right one is detected automatically:

Jurisdiction	Citable unit	Headings	Benchmarked
Thai	ข้อ / มาตรา	หมวด / ส่วนที่	yes — 8% → 83%
Vietnamese	Điều	Chương / Mục (Roman numerals)	yes — 16% → 75%
EU English	Article	CHAPTER / Section (Roman numerals)	yes — 33% → 91%
Japanese	第N条	第N章 / 第N節 (kanji numerals)	yes — 0% → 83%

Adding one is a Grammar — a handful of keywords — plus a fixture pair under eval/. The grammar is the cheap part; the measurement is the point, and a jurisdiction isn't really done until it has numbers.

Ordinary prose falls back to a sliding character window, so you can point this at a mixed corpus. The strategy that actually ran is returned, not re-derived — your telemetry can't disagree with reality.

Text extraction, including from scans. extract::text() reads a PDF's text layer and falls back to OCR when there isn't one — which is what a scanned document is, and you can't tell until you try. TextSource reports which answered, because OCR'd text carries recognition error:

cargo run --features pdf --example extract_text -- contract.pdf

PDF text-layer reading is behind the pdf feature (pdf-extract is a heavy dependency; callers who already have text shouldn't pay for it). The scanned path works without it.

OCR with a vision-model upgrade path. Tesseract locally (with pdftoppm for scanned PDFs), and if you point TyphoonConfig::base_url at any OpenAI-compatible vision endpoint, that becomes the preferred engine with Tesseract as the per-page fallback. Both are optional: with neither installed, OCR cleanly returns None rather than failing your pipeline.

Per-document-kind configuration. profile::Stages describes how a class of document should be processed — OCR mode and language, chunk strategy and sizes, summariser window, which stages run at all — as plain serde data you can store wherever you keep configuration.

An OpenAI-compatible client that talks to Ollama, vLLM, LiteLLM, OpenAI or Azure without a code change, and reports token usage on every call so you can meter it.

What's deliberately not here

• No database, no storage, no HTTP. This crate computes; persisting the result is yours.
• No authorization. It never decides who may see a result, only how to produce one.
• No Office formats. Extracting .docx/.xlsx reliably means shelling out to LibreOffice — a 400 MB dependency and a subprocess, which is a deployment decision, not a library one. Convert to PDF first, or hand in the text. PDFs and images are covered.
• No configuration discovery. Everything takes the config it needs. The one exception is an explicit Config::from_env() you can decline to call — so two callers in one process can be configured differently, and tests are hermetic.

Two ways in

pipeline::prepare does chunk-then-embed in one call. If you report per-stage progress, use pipeline::chunk and pipeline::embed_chunks separately so you can write your own telemetry between them — there's no observer/callback abstraction, because a sync callback can't await a host that persists progress, and a channel drained by a spawned task reorders events against the caller's own.

Using it from another language

docpipe-json is a line-oriented JSON front end over pipeline::chunk, for hosts that aren't Rust and don't want an FFI binding. It adds no capability — same call, reachable over a pipe.

$ echo '{"id":"a","text":"หมวด ๑\nข้อ ๑ ก\nข้อ ๒ ข\n"}' | cargo run --bin docpipe-json
{"id":"a","ok":true,"strategy":"legal","chunks":[{"seq":0,"content":"…","char_start":0,"char_end":24}]}

It handles files too — {"file":{"path":"contract.pdf"}} extracts then chunks, and "extract_only":true stops after the text. Files arrive as a path, never as bytes in the request: a 50MB scan would be 67MB of base64 on a single line, which is less a wire format than a denial of service against your own parser. The host already has the bytes on disk; it downloaded them.

The response carries source — embedded or ocr — because OCR'd text carries recognition error and downstream code usually wants to know.

One request per line, one response per line, in order, with id echoed back. Line-oriented rather than one document per process because a host indexing a corpus chunks thousands of documents and would otherwise pay process startup for each; over stdin rather than argv because document text routinely exceeds the ~32k command-line cap on Windows. A malformed line fails that line only — a batch processor that dies on document 400 of 5000 leaves the host unable to tell what succeeded.

cfg and legal_allowed mirror the library arguments, so the caller keeps the kill switch. A born-digital PDF needs the pdf feature; a scan needs Tesseract and pdftoppm on PATH, or a Typhoon endpoint. Neither is required for plain text, and a build without them still answers — with ok:false and a reason, rather than by pretending the document was empty.

Kill switches belong to the caller

Deployment-wide switches arrive as arguments, not as configuration this crate reads: chunk_text_labelled(.., legal_allowed), Stages::citations_on(env_allows). A per-document profile can narrow what the operator permits, never widen it.

Status

Extracted from a running system and used by it, but young as a standalone crate: the API may still move before 1.0.

CI runs tests (--all-features, so the pdf path is compiled too), clippy with warnings denied, the offline chunking example, and a docs build with broken intra-doc links treated as errors. The retrieval benchmark is deliberately not in CI: it needs an embedding endpoint, and a benchmark that quietly depends on whichever model a runner happens to have is worse than one you run on purpose.

Licence

MIT.