Request Source Access

System Architecture

Data Pipeline (pipeline/) Raw DOJ PDFs → OCR text extraction → Chunking (512 tokens, 64 overlap)
→ all-MiniLM-L6-v2 embeddings → Pinecone (3,363,251 vectors, 384 dims)
→ Full text → SQLite corpus DB → IDrive e2 S3

↓

RAG Backend (backend/backend.py) FastAPI — deployed on Render.com (always-on, no cold starts)
• Embeds queries with all-MiniLM-L6-v2
• Searches Pinecone for top-8 relevant passages (cosine similarity)
• Streams LLM responses via OpenAI-compatible API
• Serves full document text from SQLite corpus
• Auto-downloads corpus DB from IDrive e2 on first boot

↓

Web Interface Vanilla HTML/CSS/JS — served directly by FastAPI
• Light-themed research chat interface
• Streaming AI responses with source citations
• Expandable source cards with passage text
• "View Full Document" modal with page navigation
• Clickable DOJ PDF links in citations

Tech Stack

Vector search	Pinecone serverless, cosine similarity
Embeddings	sentence-transformers all-MiniLM-L6-v2 (384 dims)
LLM	OpenAI-compatible streaming API
Backend	Python FastAPI + uvicorn
Document storage	SQLite (local) + IDrive e2 S3 (cloud)
Hosting	Render.com (Docker, always-on)
Pipeline	Python — PDF ingestion, chunking, embedding, migration

Repository Structure

epstein-files-rag/
├── README.md
├── pipeline/ ← One-time data pipeline (gated)
│ ├── README.md ← Step-by-step pipeline instructions
│ ├── local_indexer.py ← Build vector index from corpus
│ └── migrate_to_pinecone.py
├── backend/ ← Python FastAPI RAG server
│ ├── README.md
│ ├── backend.py ← FastAPI server + HTML interface
│ ├── Dockerfile
│ └── render.yaml
└── requirements.txt

The pipeline/ directory — including PDF ingestion, chunking, embedding generation, and Pinecone migration tools — is available upon approved request.

Data Sources

Corpus	rhowardstone/Epstein-research-data — pre-extracted OCR text from the DOJ Epstein Files corpus
Original docs	U.S. Department of Justice — Epstein Files Transparency Act releases
Embedding model	sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, Apache 2.0)

Pipeline scripts and full source code available for researchers, journalists, and developers. Requests reviewed within 2–3 business days.

Request received.
Thank you — your access request has been submitted. You will be contacted within 2–3 business days.

Direct Contact

Dr. Tapang Ivo Tanku

PhD in Data Science — Chomsky Tanku Consulting

chomskytanku.com/contact

Live Demo Available

Try the fully functional research assistant — no account required.

Open Research Tool