A retrieval-augmented chatbot over Python's official documentation for asyncio
and typing, with an admin dashboard that tracks every question asked, which
documentation chunks were retrieved, and where retrieval struggles.
Built as a focused demo of production RAG patterns: structured ingestion, citation-grounded answers, observability, and input hardening — deployable end-to-end on free-tier hosting.
Live demo: https://rag-python-doc.vercel.app
Try a question like:
How do I run multiple coroutines concurrently?
You'll get a streamed answer with clickable citations back to
docs.python.org. The admin dashboard at /admin is password-gated and
shows every question, the chunks retrieved, and any inputs flagged by the
moderation + injection layers.
Hosted on Render's free tier — first request after a cold start may take ~30 seconds while the service wakes up.
| Chat with citations | Admin dashboard |
|---|---|
 |
 |
┌──────────────────────────────┐
│ React + TS + Vite (Vercel) │
│ Chat UI Admin Dashboard │
└──────────────┬───────────────┘
│ JSON / SSE
▼
┌─────────────────────────────────────────────────────────┐
│ FastAPI backend (Render) │
│ ┌───────────────┐ ┌────────────────┐ ┌────────────┐ │
│ │ /api/chat │ │ /api/admin/* │ │ session │ │
│ │ (SSE stream) │ │ (cookie auth) │ │ cookie │ │
│ └───────┬───────┘ └────────┬───────┘ └────────────┘ │
│ │ │ │
│ ┌───────▼──────────┐ ┌─────▼───────────────────────┐ │
│ │ RAG pipeline │ │ SQLite (aiosqlite) │ │
│ │ • moderation │ │ • questions │ │
│ │ • injection │ │ • retrieved_sources │ │
│ │ detection │ │ • flagged_inputs │ │
│ │ • retrieval │ └─────────────────────────────┘ │
│ │ • OpenAI stream │ │
│ └───────┬──────────┘ │
└──────────┼──────────────────────────────────────────────┘
▼
ChromaDB (on-disk)
python_docs collection
• api-entry chunks (one per dl.py.function/class/method)
• prose chunks at section boundaries
• metadata: source_url, heading_path, module, anchor
- Backend: Python 3.11, FastAPI, SQLAlchemy 2 (async), aiosqlite, LlamaIndex, OpenAI SDK
- Vector store: ChromaDB, on-disk, persisted at
backend/data/chroma/ - Frontend: React + TypeScript (strict) + Vite + Tailwind
- Docs source: official Python 3.12 HTML archive, pinned to a patch version
# 1. Clone and enter
git clone <repo> && cd python-doc-chat
# 2. Backend (requires Python 3.11+ — install via pyenv/asdf if needed)
cd backend
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp ../.env.example ../.env # then fill in OPENAI_API_KEY and ADMIN_PASSWORD
# 3. Ingest Python docs (one-time, ~2 min; embeds ~400 chunks)
python scripts/ingest.py
# 4. Seed the admin dashboard from the captured fixture
python scripts/seed_db.py
# 5. Run the API
uvicorn app.main:app --reload
# 6. Frontend (in a second terminal)
cd ../frontend
npm install && npm run devInteractive API docs auto-generated by FastAPI at http://localhost:8000/docs.
This project ships with a first-class retrieval eval harness rather than a
scratch script. backend/scripts/eval.py runs a golden set of five queries
against the live Chroma index and asserts that expected documentation anchors
appear within a target top-k rank.
cd backend && python scripts/eval.py # print results to stdout
cd backend && python scripts/eval.py --write-md # regenerate EVAL_RESULTS.mdExit code is non-zero on unexpected regressions. Failures that match a
documented weakness are marked KNOWN FAIL and don't fail the suite — if a
tuning change flips one to IMPROVED, that's the signal to update the file.
Current baseline: EVAL_RESULTS.md — 3/5 pass, 2 known failures documented honestly. The known failures inform the What I'd do differently at scale section.
Three layers, in order of enforcement strength:
- OpenAI Moderation API on every user input. Flagged inputs return a canned refusal and are logged with their flagged categories. This does block.
- Regex pattern detection for known prompt-injection shapes (role-swap
attempts, system-prompt leaks, "ignore previous instructions" variants).
Matches are logged to
flagged_inputsbut do not block — they exist as observability signal for the admin dashboard. - Scoped system prompt + Pydantic length cap. The model is told explicitly to answer only from retrieved context and to refuse off-topic questions. Input is capped at 1000 characters at the schema layer.
What this is, honestly: observability-grade, not security-grade. A determined adversary will get past the regex list. The point of this layer is to give the admin dashboard a meaningful signal about what people try, not to make the system bulletproof.
What a production version would add: a trained classifier for injection scoring (not regex), output moderation on the generated response (not just input), a secondary LLM pass to verify the response stays on-topic, and rate limits tied to session identity rather than IP.
Password-protected via a single ADMIN_PASSWORD env var. Login POSTs the
password, backend issues an HTTP-only signed cookie (itsdangerous), subsequent
requests gated by a FastAPI Depends().
Dashboard shows:
- All questions with timestamp, session ID, retrieved sources, generated answer
- Questions-per-day chart (last 14 days)
- Most-retrieved doc sections
- Mean retrieval similarity per question
- Flagged inputs table (moderation + regex) with filters
scripts/capture_seed.py runs each demo prompt once through the actual chat
pipeline (moderation → retrieval → OpenAI → flag detection) and saves the
result to data/seed_fixture.json. scripts/seed_db.py loads that fixture
on startup or on demand. The dashboard shows actual model output, real
retrieval scores, and a real moderation block — captured once, committed to
the repo, replayed deterministically. Costs ~$0.03 in OpenAI credit to
recapture; the fixture is offline + free to seed thereafter.
- Backend → Render (free web service). ChromaDB index is committed to the
repo because Render's free tier has no persistent disk. Rebuilds happen by
running
scripts/ingest.py --rebuildlocally and committing the result. - Frontend → Vercel (free static hosting).
The eval harness (EVAL_RESULTS.md) surfaces two specific weaknesses in the
baseline cosine-similarity retriever:
- Q1 — prose chunks outrank terse API signatures on natural-language
queries. "How do I run multiple coroutines concurrently?" returns generic
coroutine prose instead of
asyncio.gather/asyncio.TaskGroup, even though those chunks exist and rank first on literal-name queries. Fix: enrich each API chunk's embedding text with a natural-language intro derived from the first paragraph of its<dd>description, so a how-to question matches the API entry's semantic content rather than just its signature. - Q5 — term ambiguity + module imbalance. "What is a Protocol?" returns
only
asyncio.*Protocolchunks because asyncio contributes 8+ Protocol- related entries while typing contributes exactly one. Cosine similarity has no mechanism to enforce module diversity. Fix: apply MMR (Maximal Marginal Relevance) re-ranking to diversify top-k across modules — retrieve top-20, then greedily pick 5 that balance relevance with inter-result dissimilarity. - Broader mitigation for both: hybrid BM25 + vector retrieval with
reciprocal rank fusion. Lexical matching gives literal terms (
Protocol,gather) a floor when the embedding model under-weights them, and the fusion score tends to smooth over the prose-vs-API mismatch from Q1.
- ChromaDB → managed vector DB (Pinecone, pgvector, or Turbopuffer). A single-node embedded store is fine up to ~10⁵ chunks; beyond that it bottlenecks on ingest, replication, and cross-region latency.
- SQLite → managed Postgres with proper migrations (Alembic). SQLite is here to keep the demo free-tier and reviewable; it's not the right pick above a handful of concurrent writers.
- Ingest as a pipeline, not a script. Version each chunk by
(doc_version, source_url, content_hash), support incremental re-embedding on upstream docs changes, and run it on a schedule with dbt-style freshness checks. - Expand the eval. Grow the golden set from 5 to ~50 queries drawn from real admin-dashboard traffic; add Recall@k and NDCG metrics; gate CI on it; track scores over time so tuning changes have a visible regression signal.
- Output-side safety. Moderate the generated response, not just input; add a secondary model to verify factual grounding in the retrieved context.
- Auth. Swap the single-password admin for SSO (Clerk, Auth0, WorkOS) and add per-user rate limits on the chat endpoint.
- Observability. Ship structured logs + traces to Honeycomb/Datadog; track retrieval latency, LLM latency, and token cost as first-class metrics.
backend/
app/
main.py FastAPI entry, router mounts, CORS, lifespan
core/ config (pydantic-settings), shared dependencies
routers/ chat.py, admin.py — one APIRouter per feature
rag/ parser, chunker, indexer, retriever, prompts
db/ SQLAlchemy async models + session
security/ injection patterns, moderation, admin auth dep
schemas/ Pydantic request/response models
data/
raw/ downloaded HTML (gitignored)
chroma/ vector store (committed — see Deployment above)
app.db SQLite runtime DB (gitignored)
scripts/
ingest.py one-shot: download → chunk → embed → index
eval.py golden-query retrieval eval → EVAL_RESULTS.md
capture_seed.py one-time: run prompts through real pipeline, snapshot to JSON
seed_db.py load data/seed_fixture.json into the admin DB
frontend/
src/
pages/ Chat.tsx, AdminLogin.tsx, AdminDashboard.tsx
components/ reusable UI
lib/ typed API client
MIT.