Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,26 @@ MISTRAL_OCR_MODEL=mistral-ocr-latest
MISTRAL_OCR_API_URL=https://api.mistral.ai/v1/ocr
OCR_SEGMENT_PAGES=20
OCR_REQUEST_TIMEOUT_SECONDS=300
# Alternatives: OCR_PROVIDER=mock|pypdf|chandra
# Alternatives: OCR_PROVIDER=mock|pypdf|chandra|paddleocr

# --- OCR (Baidu PaddleOCR / PP-OCR, local CPU) ---
# OCR_PROVIDER=paddleocr
# Model tier: medium (34.5M params, default) | small (7.7M) | tiny (1.5M).
# Or an explicit pair: PADDLEOCR_MODEL=PP-OCRv6_medium_det,PP-OCRv6_medium_rec
# PADDLEOCR_MODEL=medium
# PADDLEOCR_OCR_VERSION=PP-OCRv6
# PADDLEOCR_LANG=
# PADDLEOCR_DEVICE=cpu
# PADDLEOCR_DET_MODEL=
# PADDLEOCR_REC_MODEL=
# PADDLEOCR_MIN_SCORE=0
# PADDLEOCR_IMAGE_DPI=192
# PADDLEOCR_MAX_WORKERS=1
# Leave false. PaddlePaddle 3.3 + oneDNN crashes on CPU (PIR -> oneDNN conversion);
# only set true if you are on a build where that crash does not occur.
# PADDLEOCR_ENABLE_MKLDNN=false
# Where the checkpoint is cached (set in docker-compose to the mounted volume).
# PADDLE_PDX_CACHE_HOME=/opt/hf-cache/paddlex

# --- LLM: Gemma 4 (translation + domain tagging) ---
AGRINET_GEMMA_BASE_URL=https://dev-vistaar.da.gov.in/gemma4
Expand Down
15 changes: 10 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,16 @@ FROM python:3.10-slim
WORKDIR /app

# Install system dependencies
# apt-get is skipped — curl is not needed at runtime.
# Uncomment below only if you need libreoffice (office→PDF) or curl:
# RUN apt-get update && apt-get install -y --no-install-recommends \
# curl \
# && rm -rf /var/lib/apt/lists/*
# libgomp1 — OpenMP runtime; paddlepaddle's core library links against it and
# refuses to import without it ("Can not import paddle core ...").
# libgl1 + libglib2.0-0 — the runtime shared libraries opencv-contrib-python
# (a PaddleX dependency) loads at import time.
# curl is not needed at runtime; add it only if you add a healthcheck that uses it.
RUN apt-get update && apt-get install -y --no-install-recommends \
libgomp1 \
libgl1 \
libglib2.0-0 \
&& rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:0.12.5 /uv /uvx /usr/local/bin/

Expand Down
25 changes: 24 additions & 1 deletion ENV.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ OCR is a **separate process** (not in `docker-compose`). The worker calls

| Variable | Default | Purpose |
|----------|---------|---------|
| `OCR_PROVIDER` | `chandra` | OCR provider (`chandra`, `mock`, `pypdf`) |
| `OCR_PROVIDER` | `chandra` | OCR provider (`chandra`, `mistral`, `paddleocr`, `mock`, `pypdf`) |
| `OCR_MODEL` | `chandra` | Model name |
| `CHANDRA_VLLM_BASE_URL` | `''` | Base URL; local HF default `http://localhost:8010/v1` |
| `CHANDRA_OCR_API_URL` | `''` | Alternate full OCR API URL (overrides base path build) |
Expand All @@ -124,6 +124,29 @@ OCR is a **separate process** (not in `docker-compose`). The worker calls
| `OCR_SEGMENT_PAGES` | `20` | Segment size for long docs |
| `CHANDRA_HF_HOME` / `HF_HOME` | — | Hugging Face cache (HF server / scripts) |

**Baidu PaddleOCR (local CPU lane):**

`OCR_PROVIDER=paddleocr` runs PP-OCR in-process — no GPU, no remote inference server.
The checkpoint is downloaded on first use into `PADDLE_PDX_CACHE_HOME` (set by
`docker-compose.yml` to a directory on the mounted `HF_HOME` volume, so it survives a
rebuild). Available on the **linux/amd64** image only: `paddlepaddle` publishes no
aarch64 wheels.

| Variable | Default | Purpose |
|----------|---------|---------|
| `PADDLEOCR_MODEL` | `medium` | PP-OCRv6 tier (`medium` \| `small` \| `tiny`), or an explicit `"<det>,<rec>"` model-name pair |
| `PADDLEOCR_OCR_VERSION` | `PP-OCRv6` | `PP-OCRv6` \| `PP-OCRv5` \| `PP-OCRv4` |
| `PADDLEOCR_LANG` | `''` | Checkpoint language hint (`en`, `ch`, `dev`, …); empty = the checkpoint default |
| `PADDLEOCR_DET_MODEL` | `''` | Explicit detector model name (wins over `PADDLEOCR_MODEL`) |
| `PADDLEOCR_REC_MODEL` | `''` | Explicit recogniser model name (wins over `PADDLEOCR_MODEL`) |
| `PADDLEOCR_DEVICE` | `cpu` | `cpu` or `gpu` / `gpu:0` |
| `PADDLEOCR_MIN_SCORE` | `0` | Drop recognised lines below this confidence |
| `PADDLEOCR_IMAGE_DPI` | `192` | Page render DPI |
| `PADDLEOCR_MAX_WORKERS` | `1` | Parallel OCR workers |
| `PADDLEOCR_ENABLE_MKLDNN` | `false` | Intel oneDNN acceleration. Leave `false`: PaddlePaddle 3.3 crashes on CPU in the PIR → oneDNN conversion (`ConvertPirAttribute2RuntimeAttribute not support`) and the `FLAGS_*` env vars do not work around it |
| `PADDLE_PDX_CACHE_HOME` | — | Where the checkpoint is cached (`/opt/hf-cache/paddlex` in compose) |
| `PADDLE_PDX_MODEL_SOURCE` | `huggingface` | PaddleX model host; `modelscope` is the alternative |

**Local start (real model, GPU + torch + model download):**

```bash
Expand Down
6 changes: 6 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,9 @@ services:
# HF cache path propagated into the container
HF_HOME: ${HF_HOME:-/opt/hf-cache}
HF_HUB_CACHE: ${HF_HOME:-/opt/hf-cache}
# PaddleX (Baidu PaddleOCR) weight cache — kept on the same mounted volume so
# the on-demand checkpoint download survives a container rebuild.
PADDLE_PDX_CACHE_HOME: ${HF_HOME:-/opt/hf-cache}/paddlex
ports:
- "127.0.0.1:${API_HOST_PORT:-8011}:8001"
depends_on:
Expand Down Expand Up @@ -295,6 +298,9 @@ services:
ALLOWED_FILE_PATHS: /app/books,/data/documents
HF_HOME: ${HF_HOME:-/opt/hf-cache}
HF_HUB_CACHE: ${HF_HOME:-/opt/hf-cache}
# PaddleX (Baidu PaddleOCR) weight cache — the worker is the process that
# actually runs OCR, so the checkpoint is downloaded and cached here.
PADDLE_PDX_CACHE_HOME: ${HF_HOME:-/opt/hf-cache}/paddlex
depends_on:
temporal:
condition: service_healthy
Expand Down
2 changes: 2 additions & 0 deletions pipeline/ocr/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from .chandra_vllm import ChandraVllmOcrProvider
from .mistral_ocr import MistralOcrProvider
from .mock import MockOcrProvider
from .paddleocr_local import PaddleOcrProvider
from .service import get_ocr_provider, load_ocr_config, ocr_pdf, ocr_pdf_in_segments
from .text_layer import split_pdf_range

Expand All @@ -13,6 +14,7 @@
"MockOcrProvider",
"OcrConfig",
"OcrProvider",
"PaddleOcrProvider",
"get_ocr_provider",
"load_ocr_config",
"ocr_pdf",
Expand Down
13 changes: 13 additions & 0 deletions pipeline/ocr/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,19 @@ class OcrConfig:
image_dpi: int = 192
request_timeout_seconds: float = 300.0

# Baidu PaddleOCR (local, CPU) lane. These are separate fields rather than
# reusing ``model``/``image_dpi`` because the PaddleOCR knobs are spelled
# differently from the Chandra/Mistral ones, and one deployment may want both
# lanes configured at once (switch with OCR_PROVIDER).
paddleocr_version: str = "PP-OCRv6"
paddleocr_lang: str = ""
paddleocr_device: str = "cpu"
paddleocr_det_model: str = ""
paddleocr_rec_model: str = ""
paddleocr_min_score: float = 0.0
paddleocr_image_dpi: int = 192
paddleocr_enable_mkldnn: bool = False


PageDict = dict

Expand Down
Loading