DocuMind Pro — Multi-PDF RAG Intelligence System

╔══════════════════════════════════════════════════════════════════════════════╗
║                                                                              ║
║  ██████╗  ██████╗  ██████╗██╗   ██╗███╗   ███╗██╗███╗   ██╗██████╗          ║
║  ██╔══██╗██╔═══██╗██╔════╝██║   ██║████╗ ████║██║████╗  ██║██╔══██╗         ║
║  ██║  ██║██║   ██║██║     ██║   ██║██╔████╔██║██║██╔██╗ ██║██║  ██║         ║
║  ██║  ██║██║   ██║██║     ██║   ██║██║╚██╔╝██║██║██║╚██╗██║██║  ██║         ║
║  ██████╔╝╚██████╔╝╚██████╗╚██████╔╝██║ ╚═╝ ██║██║██║ ╚████║██████╔╝         ║
║  ╚═════╝  ╚═════╝  ╚═════╝ ╚═════╝ ╚═╝     ╚═╝╚═╝╚═╝  ╚═══╝╚═════╝          ║
║                                                                              ║
║             Multi-PDF RAG Intelligence System v1.0.0                         ║
║ FastAPI · Streamlit · Gemini 2.5 Flash · ChromaDB · BM25 · RAGAS · Docker   ║
╚══════════════════════════════════════════════════════════════════════════════╝

Architecture

                           ┌────────────────────────────────────────────┐
                           │         Streamlit Frontend (8501)          │
                           │ Premium UI • Chat • Metrics • Upload       │
                           └──────────────────┬─────────────────────────┘
                                              │ HTTP REST
                           ┌──────────────────▼─────────────────────────┐
                           │          FastAPI Backend (8000)            │
                           │ Upload • Query • Metrics • Cache           │
                           └──────────────────┬─────────────────────────┘
                                              │
                         ┌────────────────────▼────────────────────┐
                         │          Document Processing Pipeline    │
                         │                                          │
                         │ PDF Upload (Multiple PDFs)               │
                         │              │                           │
                         │ PDFPlumber Extraction                    │
                         │              │                           │
                         │ Semantic Chunking                        │
                         │              │                           │
                         │ Gemini Embedding-001 (3072 Dimensions)   │
                         │              │                           │
                         └───────┬───────────────┬──────────────────┘
                                 │               │
                                 ▼               ▼
                         ChromaDB Vector DB     BM25 Index
                                 │               │
                                 └──────┬────────┘
                                        ▼
                           Reciprocal Rank Fusion (RRF)
                                        │
                                        ▼
                           CrossEncoder Re-ranking
                                        │
                                        ▼
                           Gemini 2.5 Flash Generator
                                        │
                                        ▼
                      Citation-aware Answer + Confidence Score
                                        │
                           ┌────────────┴────────────┐
                           ▼                         ▼
                    Semantic Cache             RAGAS Evaluation
                           │                         │
                           └────────────┬────────────┘
                                        ▼
                                 SQLite Metrics

Features

• Multi-PDF Upload & Indexing
• PDFPlumber Text Extraction
• Semantic-aware Chunking
• Gemini Embedding-001 (3072-Dimensional Embeddings)
• ChromaDB Vector Database
• BM25 Sparse Retrieval
• Hybrid Retrieval Pipeline
• Reciprocal Rank Fusion (RRF)
• CrossEncoder Re-ranking
• Gemini 2.5 Flash Answer Generation
• Citation-aware Responses
• Confidence Score
• Semantic Cache
• RAGAS Evaluation
• SQLite Metrics Dashboard
• FastAPI REST API
• Streamlit Premium Dashboard
• LangSmith Tracing Support
• Apify Web Context (Optional)
• Docker & Docker Compose Support
• Dark Premium Responsive UI

RAG Pipeline

Document Indexing

1. Upload multiple PDF documents.
2. Extract text page-by-page using PDFPlumber.
3. Perform semantic-aware chunking.
4. Generate Gemini Embedding-001 vectors.
5. Store embeddings in ChromaDB.
6. Build BM25 sparse index.

Query Pipeline

1. Receive user question.
2. Check Semantic Cache.
3. Dense Retrieval (ChromaDB Top-20)
4. Sparse Retrieval (BM25 Top-20)
5. Reciprocal Rank Fusion
6. CrossEncoder Re-ranking
7. Context Assembly
8. Gemini 2.5 Flash Response Generation
9. Citation Generation
10. Confidence Score Calculation
11. Background RAGAS Evaluation
12. Store Metrics in SQLite

Quick Start

Prerequisites

• Python 3.11+
• Docker Desktop
• Google Gemini API Key

Clone Project

git clone <repository-url>
cd documind-pro

Configure Environment

cp .env.example .env

Fill your API keys.

Install

python -m venv .venv

# Windows
.venv\Scripts\activate

pip install -r requirements.txt

Run Backend

uvicorn api.main:app --reload

Backend

http://localhost:8000

Swagger

http://localhost:8000/docs

Run Frontend

streamlit run frontend/app.py

Frontend

http://localhost:8501

Run Docker

docker compose up --build

API Reference

Method	Endpoint	Description
POST	/upload	Upload and index PDFs
POST	/query	Ask questions
GET	/documents	List indexed documents
DELETE	/documents/{doc_id}	Delete document
GET	/metrics	Evaluation metrics
GET	/cache/stats	Cache statistics
DELETE	/cache	Clear semantic cache
GET	/health	Health check

Required API Keys

Environment Variable	Service	Required
GOOGLE_API_KEY	Google Gemini	Yes
LANGCHAIN_API_KEY	LangSmith	Optional
APIFY_API_TOKEN	Apify	Optional

Manual Configuration

1. Copy .env.example → .env
2. Add Google Gemini API Key.
3. Start Backend.
4. Start Frontend.

Project Structure

documind-pro/
├── api/
├── cache/
├── evaluation/
├── frontend/
├── pipeline/
├── rag/
├── retrieval/
├── workflows/
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── .env.example
└── README.md

Tech Stack

Layer	Technology
Backend	FastAPI
Frontend	Streamlit
LLM	Gemini 2.5 Flash
Embeddings	Gemini Embedding-001
Orchestration	LangChain
Dense Retrieval	ChromaDB
Sparse Retrieval	BM25
Re-ranking	HuggingFace CrossEncoder
Evaluation	RAGAS
Database	SQLite
Web Context	Apify (Optional)
Observability	LangSmith
Containerization	Docker
Language	Python 3.11

Troubleshooting

Invalid Gemini API Key

Verify your GOOGLE_API_KEY in .env.

Embedding Model Error

Use:

models/gemini-embedding-001

Gemini Rate Limit

Wait for the free-tier quota to reset or use another API key.

ChromaDB Issues

Delete the local data/chroma directory and re-index your documents.

Gmail Notification Not Working

Reconnect Gmail OAuth inside n8n.

API Not Reachable

Ensure the backend is running on port 8000 before starting Streamlit.

License

MIT License

Author

Aman Nanda

B.Sc. Artificial Intelligence

Generative AI • RAG Engineering • LLM Applications • AI Automation • FastAPI • Streamlit • LangChain • Gemini • ChromaDB • Docker