From 77d62eebc6ceb724bcb285a0655b3cc29a0008d0 Mon Sep 17 00:00:00 2001 From: Rajkumar Date: Sat, 12 Sep 2026 15:38:51 +0530 Subject: [PATCH] (feature) ci and cd via github actions implemented rather than jenkins --- .github/CI-CD.md | 164 ++++++++++ .github/workflows/cd.yml | 166 ++++++++++ .github/workflows/ci.yml | 225 +++++++++++++ ARCHITECTURE.md | 539 +++++++++++++++++++++++++++++++ README.md | 120 +++---- apps/genAI/app/agent/Qnagent.py | 7 +- apps/genAI/app/main.py | 26 +- apps/genAI/app/rag/rag_system.py | 7 +- apps/genAI/app/utils/config.py | 5 +- apps/genAI/app/utils/llm.py | 1 + apps/genAI/pyproject.toml | 26 ++ contribution.md | 147 ++++++--- docker-compose.prod.yml | 8 +- issues.md | 70 +++- planning.md | 43 ++- setup.md | 34 +- 16 files changed, 1429 insertions(+), 159 deletions(-) create mode 100644 .github/CI-CD.md create mode 100644 .github/workflows/cd.yml create mode 100644 .github/workflows/ci.yml create mode 100644 ARCHITECTURE.md create mode 100644 apps/genAI/pyproject.toml diff --git a/.github/CI-CD.md b/.github/CI-CD.md new file mode 100644 index 0000000..a936eb9 --- /dev/null +++ b/.github/CI-CD.md @@ -0,0 +1,164 @@ +# CI/CD Guide + +How automated checks and deployment work for this project. + +**Contributing?** You only need the [CI](#ci-what-runs-on-your-pr) section — +it explains what runs on your pull request and how to reproduce any failure +locally. The deployment section is for maintainers with production access. + +--- + +## CI (what runs on your PR) + +Every pull request runs [`ci.yml`](workflows/ci.yml). Six jobs run in parallel, +so you get all the failures at once instead of one at a time. + +| Job | What it checks | +|---|---| +| **Lint & type-check** | ESLint and TypeScript across every workspace, then a full build | +| **Prisma schema** | Migrations apply cleanly to a fresh database, and `schema.prisma` matches them | +| **Python checks** | The GenAI service compiles, and `ruff` passes | +| **Build images** | All four Dockerfiles still build | +| **Validate compose** | Both `docker-compose` files are well-formed | +| **Secret scan** | No credentials committed anywhere in history | + +### Reproducing a failure locally + +Run the same commands CI does, from the repo root: + +```bash +# Lint & type-check +pnpm run lint +pnpm run check-types +pnpm run build + +# Python checks (needs apps/genAI/.venv, see below) +cd apps/genAI +python -m compileall -q app +pip install ruff && ruff check app + +# Docker build (one service) +docker build -f apps/frontend/Dockerfile.prod . + +# Compose validation +docker compose -f docker-compose.yml config --quiet +``` + +For Prisma, CI applies migrations to a throwaway database. To do the same +against your local one: + +```bash +cd packages/db +pnpm exec prisma validate +pnpm exec prisma migrate deploy +``` + +If you changed `schema.prisma`, you must also commit a migration. CI fails +otherwise: + +```bash +cd packages/db +pnpm exec prisma migrate dev --name describe_your_change +``` + +### Working on the GenAI service + +The Python service is not covered by `pnpm install`. Set it up once: + +```bash +cd apps/genAI +python3 -m venv .venv +.venv/bin/pip install -r requirements.txt +cp .env.example .env # then add your own API keys +``` + +It needs free API keys from [Groq](https://console.groq.com/keys), +[Google AI Studio](https://aistudio.google.com/apikey), and +[Tavily](https://tavily.com). Without them the service still starts and +`/health` works — only the AI endpoints return an error telling you which key +is missing. + +### Things to know + +-**ESLint currently reports pre-existing errors** in the frontend. That step + is set to report without blocking, so it will not fail your PR. Please do not + add new ones — and a PR that cleans them up is welcome. +-**Ruff is enforced** for Python. Most issues auto-fix with + `ruff check app --fix`. +-**Docker builds are cached**, but a cold run takes a while. The GenAI image + is the slowest because of its ML dependencies. + +--- + +## Deployment (maintainers) + +>This section needs Docker Hub and production server access. Contributors can +>skip it — deploys happen automatically after a PR is merged. + +Pushing to `main` triggers [`cd.yml`](workflows/cd.yml), which builds and +pushes four images to Docker Hub, scans them with Trivy, then deploys over SSH. + +### Required secrets + +Configured under **Settings → Secrets and variables → Actions**. + +Repository secrets: + +| Secret | Purpose | +|---|---| +| `DOCKERHUB_USERNAME` | Docker Hub account owning the images | +| `DOCKERHUB_TOKEN` | Docker Hub access token, not a password | +| `DATABASE_URL` | Build arg so `prisma generate` can run during the image build | + +Environment secrets, on an environment named `production`: + +| Secret | Purpose | +|---|---| +| `SSH_HOST` | Server hostname or IP | +| `SSH_USER` | User permitted to run `docker` | +| `SSH_PRIVATE_KEY` | Private key, full PEM including BEGIN/END lines | +| `SSH_PORT` | Optional, defaults to `22` | +| `DEPLOY_PATH` | Absolute path to the repo checkout on the server | + +Keeping deploy credentials on the environment rather than the repository limits +them to the `deploy` job, and lets you require a reviewer before anything +reaches production (**Environments → production → Required reviewers**). + +### Server prerequisites + +The pipeline does not bootstrap the server. Before the first deploy: + +1. Docker and the Compose plugin installed. +2. Repository cloned at `DEPLOY_PATH`. +3. `.env.prod` present there with production values: `POSTGRES_USER`, + `POSTGRES_PASSWORD`, `POSTGRES_DB`, `DATABASE_URL`, `REDIS_PASSWORD`, + `JWT_SECRET`, `FRONTEND_URL`, the GenAI keys, and `ALLOWED_ORIGINS`. +4. The public key matching `SSH_PRIVATE_KEY` in `~/.ssh/authorized_keys`. + +`.env.prod` is never committed and never written by the pipeline. It lives only +on the server. + +### Image tags and rollback + +Every build publishes `:latest` and `:`. The deploy pins the SHA, so +the running version is always unambiguous — and rolling back is one command: + +```bash +cd +IMAGE_TAG= docker compose -f docker-compose.prod.yml up -d +``` + +### Database migrations + +Migrations are **not** run by the deploy job. The `backend` service applies them +itself on startup, via its compose command. CI proves on every PR that the +migration history still applies to a clean database, so a broken migration is +caught before it reaches the server. + +### Known gaps + +-Trivy scanning is report-only; findings appear in logs but do not block a + deploy. +-No staging environment — `main` goes straight to production. +-`docker compose up -d` recreates changed containers, so deploys have brief + downtime. There is no rolling update. diff --git a/.github/workflows/cd.yml b/.github/workflows/cd.yml new file mode 100644 index 0000000..386e413 --- /dev/null +++ b/.github/workflows/cd.yml @@ -0,0 +1,166 @@ +name: CD + +on: + push: + branches: [main] + # Allows a manual redeploy from the Actions tab without a new commit. + workflow_dispatch: + +# Never let two deploys touch the server at once, and do not cancel a +# deploy midway: a half-applied rollout is worse than a queued one. +concurrency: + group: production-deploy + cancel-in-progress: false + +permissions: + contents: read + +env: + REGISTRY_NAMESPACE: codeheist + +jobs: + build-and-push: + name: Build & push ${{ matrix.service }} + runs-on: ubuntu-latest + timeout-minutes: 45 + + strategy: + fail-fast: false + matrix: + include: + - service: frontend + image: rexial-frontend + dockerfile: apps/frontend/Dockerfile.prod + - service: http-server + image: rexial-http-server + dockerfile: apps/http-server/Dockerfile.prod + - service: ws-server + image: rexial-ws-server + dockerfile: apps/ws-server/Dockerfile.prod + - service: genai + image: rexial-genai + dockerfile: apps/genAI/Dockerfile.prod + + steps: + - uses: actions/checkout@v4 + + - uses: docker/setup-buildx-action@v3 + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Build and push + uses: docker/build-push-action@v6 + with: + context: . + file: ${{ matrix.dockerfile }} + push: true + # Tag with the commit SHA as well as latest, so a bad deploy can + # be rolled back to an exact known-good image. + tags: | + ${{ env.REGISTRY_NAMESPACE }}/${{ matrix.image }}:latest + ${{ env.REGISTRY_NAMESPACE }}/${{ matrix.image }}:${{ github.sha }} + build-args: | + DATABASE_URL=${{ secrets.DATABASE_URL }} + cache-from: type=gha,scope=${{ matrix.service }} + cache-to: type=gha,mode=max,scope=${{ matrix.service }} + + - name: Scan image for vulnerabilities + uses: aquasecurity/trivy-action@v0.36.0 + with: + image-ref: ${{ env.REGISTRY_NAMESPACE }}/${{ matrix.image }}:${{ github.sha }} + severity: HIGH,CRITICAL + ignore-unfixed: true + format: table + # Reported, not enforced: a CVE in a base image should not block + # a deploy until you have decided how to triage it. + exit-code: "0" + + deploy: + name: Deploy to production + needs: build-and-push + runs-on: ubuntu-latest + timeout-minutes: 20 + + # Attach to a GitHub Environment so you can require manual approval + # and scope the deploy secrets to production only. + environment: + name: production + + steps: + - name: Deploy over SSH + uses: appleboy/ssh-action@v1.2.0 + with: + host: ${{ secrets.SSH_HOST }} + username: ${{ secrets.SSH_USER }} + key: ${{ secrets.SSH_PRIVATE_KEY }} + port: ${{ secrets.SSH_PORT || 22 }} + script_stops: true + script: | + set -euo pipefail + + cd ${{ secrets.DEPLOY_PATH }} + + # Take the exact commit this workflow built. + git fetch --all + git checkout main + git reset --hard ${{ github.sha }} + + # Pin to this commit's images rather than :latest, so the + # deployed version is unambiguous and rollback is exact. + export IMAGE_TAG=${{ github.sha }} + + echo "Pulling images tagged $IMAGE_TAG" + docker compose -f docker-compose.prod.yml pull \ + frontend backend ws-server genai + + # Migrations are not run here: the backend service applies + # them itself on startup via its compose command + # (db:generate:prod && db:migrate && start). + echo "Starting services" + docker compose -f docker-compose.prod.yml up -d + + echo "Waiting for containers to settle" + sleep 20 + + docker compose -f docker-compose.prod.yml ps + + # Fail the deploy if any expected service is not running. + for svc in frontend backend ws-server genai db redis; do + state=$(docker compose -f docker-compose.prod.yml ps \ + --format '{{.State}}' "$svc" || true) + echo " $svc: ${state:-missing}" + if [ "$state" != "running" ]; then + echo "ERROR: $svc is not running" + docker compose -f docker-compose.prod.yml logs --tail 50 "$svc" + exit 1 + fi + done + + docker image prune -f + echo "Deployment complete" + + rollback-hint: + name: Report failure + needs: deploy + if: failure() + runs-on: ubuntu-latest + + steps: + - name: Explain how to roll back + run: | + echo "Deploy of ${{ github.sha }} failed." + echo "" + echo "Every build is tagged with its commit SHA, so rolling back" + echo "is a matter of pinning the last known-good one:" + echo "" + echo " ssh @" + echo " cd " + echo " IMAGE_TAG= \\" + echo " docker compose -f docker-compose.prod.yml up -d" + echo "" + echo "Past tags: https://hub.docker.com/r/codeheist/rexial-http-server/tags" + exit 1 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..c1e26d4 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,225 @@ +name: CI + +on: + pull_request: + branches: [main] + push: + branches: [main] + +# A new push to the same branch cancels the previous run, so PRs do not +# queue up stale builds. +concurrency: + group: ci-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +env: + # Points at the throwaway Postgres service container in the prisma job. + # Other jobs only need it to be a syntactically valid URL for + # `prisma generate`, which does not connect. Not a real credential. + DATABASE_URL: postgresql://postgres:postgres@localhost:5432/rexial_ci + +jobs: + lint-and-types: + name: Lint & type-check + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + + - name: Set up Node + uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + + # The genai workspace package runs its turbo tasks through Python, + # so the interpreter has to exist before turbo runs. + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + # http-server and ws-server import generated Prisma types, so the + # client must exist before anything type-checks. + - name: Generate Prisma client + run: pnpm --filter @repo/db run db:generate + + # Reporting only for now: the frontend has ~20 pre-existing eslint + # errors. Drop continue-on-error once those are cleaned up so lint + # becomes a merge gate. + - name: Lint + run: pnpm run lint + continue-on-error: true + + - name: Type-check + run: pnpm run check-types + + - name: Build all workspaces + run: pnpm run build + + prisma: + name: Prisma schema + runs-on: ubuntu-latest + timeout-minutes: 10 + + # Applying migrations for real is the only way to prove they still + # work on a clean database, which is exactly what a deploy does. + services: + postgres: + image: postgres:15-alpine + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: rexial_ci + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 10s + --health-timeout 5s + --health-retries 10 + + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + - name: Validate schema + working-directory: packages/db + run: pnpm exec prisma validate + + - name: Apply migrations to a clean database + working-directory: packages/db + run: pnpm exec prisma migrate deploy + + - name: Check schema matches the migrations + working-directory: packages/db + # Exit code 2 means schema.prisma describes something the applied + # migrations do not, i.e. a migration is missing. + run: | + pnpm exec prisma migrate diff \ + --from-config-datasource \ + --to-schema ./prisma/schema.prisma \ + --exit-code + + python: + name: Python checks + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: pip + cache-dependency-path: apps/genAI/requirements.txt + + # Compile-checks every module for syntax errors without installing + # the multi-GB ML dependency tree. + - name: Compile-check sources + working-directory: apps/genAI + run: python -m compileall -q app + + - name: Lint with ruff + working-directory: apps/genAI + run: | + pip install ruff + ruff check app --output-format=github + + docker: + name: Build ${{ matrix.service }} image + runs-on: ubuntu-latest + timeout-minutes: 45 + + strategy: + # Let every image finish so one failure does not mask the others. + fail-fast: false + matrix: + include: + - service: frontend + dockerfile: apps/frontend/Dockerfile.prod + - service: http-server + dockerfile: apps/http-server/Dockerfile.prod + - service: ws-server + dockerfile: apps/ws-server/Dockerfile.prod + - service: genai + dockerfile: apps/genAI/Dockerfile.prod + + steps: + - uses: actions/checkout@v4 + + - uses: docker/setup-buildx-action@v3 + + - name: Build image + uses: docker/build-push-action@v6 + with: + context: . + file: ${{ matrix.dockerfile }} + # Validation only: never publish from a pull request. + push: false + load: false + build-args: | + DATABASE_URL=${{ env.DATABASE_URL }} + cache-from: type=gha,scope=${{ matrix.service }} + cache-to: type=gha,mode=max,scope=${{ matrix.service }} + + compose: + name: Validate compose files + runs-on: ubuntu-latest + timeout-minutes: 5 + + steps: + - uses: actions/checkout@v4 + + - name: Validate dev compose + run: | + touch apps/genAI/.env + docker compose -f docker-compose.yml config --quiet + + - name: Validate prod compose + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: rexial + REDIS_PASSWORD: placeholder + run: | + # .env.prod lives on the deploy server and is not in the repo, + # so stand in an empty one just to check the file's structure. + touch .env.prod + docker compose -f docker-compose.prod.yml config --quiet + rm -f .env.prod + + security: + name: Secret scan + runs-on: ubuntu-latest + timeout-minutes: 10 + + steps: + - uses: actions/checkout@v4 + with: + # gitleaks needs history to scan commits, not just the tree. + fetch-depth: 0 + + - name: Scan for committed secrets + uses: gitleaks/gitleaks-action@v2 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..1b0fc60 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,539 @@ +# Rexial Architecture + +An in-depth look at how Rexial is put together: the services, how they talk to +each other, and why the boundaries fall where they do. + +New here? Read [contribution.md](contribution.md) first to get the project +running, then come back for the *why*. + +--- + +## Contents + +-[The short version](#the-short-version) +-[System overview](#system-overview) +-[Why a monorepo](#why-a-monorepo) +-[The services](#the-services) +-[Data model](#data-model) +-[Flow 1: Creating a quiz](#flow-1-creating-a-quiz) +-[Flow 2: AI question generation](#flow-2-ai-question-generation) +-[Flow 3: Running a live quiz](#flow-3-running-a-live-quiz) +-[Caching and scale](#caching-and-scale) +-[Deployment topology](#deployment-topology) +-[Design decisions](#design-decisions) +-[Known limitations](#known-limitations) + +--- + +## The short version + +Rexial is a live quiz platform. A host builds a quiz, generates a join code, +and starts a session; participants join with that code and answer questions in +real time, scored on speed, with a live leaderboard. + +It is split into **four services** in one monorepo: + +| Service | Stack | Port | Responsibility | +|---|---|---|---| +| `frontend` | React + Vite + Tailwind | 5173 | All UI | +| `http-server` | Express + Prisma | 4000 | Auth, CRUD, session setup | +| `ws-server` | `ws` + Redis | 8080 | Live gameplay, timers, scoring | +| `genAI` | FastAPI + LangChain | 8000 | AI question generation from PDFs | + +Backed by **PostgreSQL** (source of truth) and **Redis** (hot cache + pub/sub). + +--- + +## System overview + +``` + ┌──────────────────────────┐ + │ Browser │ + │ React SPA (frontend) │ + └────┬──────────┬──────┬───┘ + │ │ │ + REST (axios) │ WS │ │ multipart + JWT in header │ (live) │ │ (PDF upload) + │ │ │ + ┌───────────────▼──┐ ┌────▼──────▼────┐ ┌──────────────┐ + │ http-server │ │ ws-server │ │ genAI │ + │ Express :4000 │ │ ws :8080 │ │ FastAPI :8000│ + │ │ │ │ │ │ + │ • auth (JWT) │ │ • live session │ │ • PDF -> quiz│ + │ • quiz CRUD │ │ • timers │ │ • RAG Q&A │ + │ • invites/email │ │ • scoring │ │ over PDFs │ + │ • start session │ │ • leaderboard │ │ • chat │ + └────────┬─────────┘ └───┬────────┬───┘ └──────┬───────┘ + │ │ │ │ + │ Prisma │ │ cache │ HTTPS + │ │ │ pub/sub │ + ┌────▼────────────────▼──┐ ┌──▼──────────┐ │ + │ PostgreSQL │ │ Redis │ │ + │ (source of truth) │ │ (hot state) │ │ + └────────────────────────┘ └─────────────┘ │ + │ + ┌──────────▼─────────┐ + │ External LLM APIs │ + │ Groq · Gemini · │ + │ Tavily │ + └────────────────────┘ +``` + +**The key split:** `http-server` owns everything *before* and *after* a quiz +runs. `ws-server` owns the quiz *while* it is running. They share a database +but never call each other — the handoff happens through the `QuizSession` row. + +--- + +## Why a monorepo + +Turborepo with pnpm workspaces: + +``` +Rexial/ +├── apps/ +│ ├── frontend/ React SPA +│ ├── http-server/ Express REST API +│ ├── ws-server/ WebSocket server +│ └── genAI/ Python FastAPI service +├── packages/ +│ ├── db/ Prisma schema + generated client (@repo/db) +│ ├── ui/ Shared components (scaffolded, not yet used) +│ ├── eslint-config/ +│ └── typescript-config/ +└── docker-compose.yml +``` + +The reason is `packages/db`. Both Node servers need identical database types; +publishing that to a registry for two consumers would be pure overhead. As a +workspace package, `@repo/db` is imported directly and a schema change surfaces +as a type error in both servers immediately. + +`genAI` is in the workspace too, but only as a thin shim — its `package.json` +scripts shell out to Python so `pnpm dev` can start all four services at once. +Its real dependencies live in `requirements.txt`. + +--- + +## The services + +### frontend — React SPA + +Vite + React 19 + Tailwind 4, Zustand for auth state, React Router for routing. + +Two API clients, deliberately separate: + +| Client | Base URL | Timeout | Why | +|---|---|---|---| +| `api` | `VITE_API_URL` → :4000 | 10s | Normal CRUD; attaches JWT, logs out on 401 | +| `genaiApi` | `VITE_GENAI_URL` → :8000 | **120s** | LLM calls are slow; 10s would abort every request | + +The WebSocket connection is opened directly by the `LiveQuiz` screen, not +through a client wrapper. + +### http-server — Express REST API + +Everything transactional. Routes under `/api/v1`: + +``` +/auth + POST /register bcrypt hash, returns JWT + POST /login verify, returns JWT + GET /me [auth] current user + (Google OAuth via passport is also wired up) + +/quizzes + POST / [auth] create quiz + GET / [auth] list my quizzes + GET /:quizId quiz + questions + answers + POST /:quizId/generate-access-code create the join code + POST /:quizId/questions add a question + its answers + POST /:quizId/start-session [auth] create QuizSession -> returns sessionId + +/quizzes (invites) + POST /:quizId/invite email a co-organizer + POST /invite/accept/:token [auth] accept + +/sessions + POST /join participant joins by code -> participantId + GET /:sessionId/leaderboard +``` + +Auth is a stateless JWT in the `Authorization` header. Participants are +deliberately **not** required to register — they get a `Participant` row tied to +a session, not a `User`. + +### ws-server — live gameplay + +A raw `ws` server. Every message is `{ type, payload }`. + +**Client → server** + +| Type | Sent by | Effect | +|---|---|---| +| `join` | both | Binds the socket to a session and role | +| `quiz:start` | organizer | Starts the quiz | +| `quiz:next-question` | organizer | Broadcasts the next question, starts its timer | +| `quiz:submit-answer` | participant | Scores the answer, updates the leaderboard | +| `quiz:end` | organizer | Ends the session | + +**Server → client** + +| Type | Meaning | +|---|---| +| `participants:sync` | Full participant list (sent on organizer join) | +| `participant:joined` | Someone new joined the lobby | +| `quiz:start` | Quiz has begun | +| `quiz:question` | The current question — **answers stripped of `isCorrect`** | +| `quiz:timer-tick` | Countdown, once per second | +| `quiz:question-results` | Correct answer revealed after time expires | +| `quiz:leaderboard` | Updated standings | +| `quiz:ended` | Final results | + +Two security-relevant details: + +1. **Questions are sanitized before broadcast.** The `quiz:question` payload + maps answers down to `{ id, text }` only. The client literally cannot know + which option is correct, so you cannot cheat by reading the socket. +2. **Scoring happens server-side.** The client sends `timeMs`; the server + decides correctness and points. + +Scoring rewards speed: + +```js +points = isCorrect ? Math.max(10, 1000 - timeMs) : 0 +``` + +A correct answer in 200ms scores 800; one at 3s floors at 10. Wrong answers +score nothing. + +### genAI — AI question generation + +FastAPI, isolated from the Node services so the ML dependency tree (torch, +sentence-transformers, faiss) never touches them. + +| Endpoint | Purpose | +|---|---| +| `POST /generate-quiz` | **Structured JSON** questions — what the UI uses | +| `POST /generate-questions` | Same, as a readable text blob | +| `POST /ask-pdf` | RAG question-answering over an uploaded PDF | +| `POST /chat` | General chat; routes to web search when needed | +| `GET /health` | Liveness, used by the container healthcheck | + +Providers: **Groq** for question generation, **Gemini** for chat, **Tavily** +for web search, **HuggingFace** sentence-transformers for embeddings. + +All clients are built **lazily on first use**. Constructing them at import time +meant a missing API key crashed the whole service on startup — including +`/health`, which made the container fail its healthcheck and restart forever. + +--- + +## Data model + +``` +User ──┬──< Quiz ──┬──< Question ──< Answer + │ │ + │ └──< QuizSession ──┬──< Participant ──< ParticipantAnswer + │ │ │ + └──< QuizOrganizer └──────────────────────────┘ + (co-hosts, invite flow) +``` + +| Model | Notes | +|---|---| +| `User` | Registered accounts. Hosts and co-organizers. | +| `Quiz` | Owned by a user. Holds `joinCode` and `QuizStatus`. | +| `QuizOrganizer` | Co-host link with `OrganizerRole` + `InviteStatus`. | +| `Question` / `Answer` | `Answer.isCorrect` is **never** sent to participants. | +| `QuizSession` | One live run of a quiz. Tracks `currentQuestionIndex`, `SessionStatus`. | +| `Participant` | A player in one session. **Not** a `User` — no signup needed. | +| `ParticipantAnswer` | One submission: answer, `timeMs`, `isCorrect`, `points`. | + +`Quiz` is the template; `QuizSession` is one run of it. The same quiz can be +hosted repeatedly, each run with its own participants and leaderboard. + +--- + +## Flow 1: Creating a quiz + +``` +Host frontend http-server PostgreSQL + │ │ │ │ + │ create quiz │ │ │ + ├─────────────────────►│ POST /quizzes │ │ + │ ├──────────────────►│ create Quiz │ + │ │ ├──────────────────►│ + │ │◄──────────────────┤ quizId │ + │ │ │ │ + │ add question ×N │ │ │ + ├─────────────────────►│ POST /:id/questions │ + │ ├──────────────────►│ Question + Answers│ + │ │ ├──────────────────►│ + │ │ │ │ + │ generate join code │ │ │ + ├─────────────────────►│ POST /:id/generate-access-code │ + │ ├──────────────────►│ joinCode, ACTIVE │ + │ │◄──────────────────┤ │ + │ "ABC123" │ │ │ + │ │ │ │ + │ host live │ │ │ + ├─────────────────────►│ POST /:id/start-session │ + │ ├──────────────────►│ create QuizSession│ + │ │◄──────────────────┤ sessionId │ + │ │ │ + │ └──► navigate to /quiz/manage/:sessionId + │ (from here, ws-server takes over) +``` + +--- + +## Flow 2: AI question generation + +``` +Host frontend genAI Groq + │ │ │ │ + │ upload PDF │ │ │ + │ + "5 questions │ │ │ + │ on chapter 2" │ │ │ + ├────────────────►│ │ │ + │ │ POST /generate-quiz (multipart) │ + │ ├─────────────────►│ │ + │ │ │ save_upload() │ + │ │ │ ↳ sanitize name │ + │ │ │ │ + │ │ │ PyPDFLoader │ + │ │ │ ↳ chunk 3000/300 │ + │ │ │ ↳ whole PDF as │ + │ │ │ context (no │ + │ │ │ retrieval here) │ + │ │ │ │ + │ │ │ prompt + context │ + │ │ ├──────────────────►│ + │ │ │◄──────────────────┤ JSON + │ │ │ │ + │ │ │ _extract_json() │ + │ │ │ ↳ strip fences │ + │ │ │ validate: │ + │ │ │ • exactly 4 opts │ + │ │ │ • exactly 1 true │ + │ │ │ • drop the rest │ + │ │◄─────────────────┤ questions[] │ + │ │ │ │ + │◄────────────────┤ review panel │ │ + │ │ │ │ + │ edit / deselect │ │ │ + │ then "Add to Quiz" │ │ + ├────────────────►│ │ │ + │ │ POST /quizzes/:id/questions ×N │ + │ ├──────────────────────────────► http-server +``` + +**Generation does not use retrieval.** Worth being precise, because the service +is described as "RAG" overall: `/generate-quiz` chunks the PDF and passes the +**entire** text as context, since questions should cover the whole document. +Only `/ask-pdf` does true RAG — it embeds chunks into a vector store and +retrieves the top 5 matches for the question. So the embedding model is never +loaded during quiz generation. + +Two more things worth noting. + +**The model is not trusted.** Its output is parsed defensively (`_extract_json` +handles markdown fences and chatty preambles) and then validated. Any question +without exactly 4 options and exactly 1 correct answer is **discarded**. A +malformed question is dropped rather than shown to the host or saved. + +**The human is the last gate.** Generated questions land in a review panel. +Nothing is written to the database until the host confirms. + +--- + +## Flow 3: Running a live quiz + +``` +Organizer ws-server Redis PostgreSQL Participants + │ │ │ │ │ + │ join (ORGANIZER) │ │ │ │ + ├─────────────────►│ getCachedSession│ │ │ + │ ├────────────────►│ miss │ │ + │ │ ├─────────────►│ │ + │◄─────────────────┤ participants:sync │ │ + │ │ │ │ │ + │ │◄───────────────────────────────────────────────┤ join + │◄─────────────────┤ participant:joined ────────────────────────────►│ + │ │ │ │ │ + │ quiz:start │ │ │ │ + ├─────────────────►│ │ │ │ + │◄─────────────────┤ quiz:start ────────────────────────────────────►│ + │ │ │ │ │ + │ quiz:next-question │ │ │ + ├─────────────────►│ strip isCorrect │ │ │ + │◄─────────────────┤ quiz:question ─────────────────────────────────►│ + │ │ startQuestionTimer │ │ + │◄─────────────────┤ quiz:timer-tick (1/sec) ──────────────────────►│ + │ │ │ │ │ + │ │◄───────────────────────────────────────────────┤ submit + │ │ score it │ │ │ answer + │ │ points = │ │ │ + │ │ max(10,1000-ms)│ │ │ + │ ├────────────────────────────────►│ ParticipantAnswer + │ ├────────────────►│ leaderboard │ │ + │ │ │ │ │ + │ │ timer expires │ │ │ + │◄─────────────────┤ quiz:question-results ────────────────────────►│ + │◄─────────────────┤ quiz:leaderboard ─────────────────────────────►│ + │ │ │ │ │ + │ quiz:end │ │ │ │ + ├─────────────────►│ clearSessionCache │ │ + │◄─────────────────┤ quiz:ended ───────────────────────────────────►│ +``` + +The organizer drives the pace — questions advance on `quiz:next-question`, not +automatically. Timers live in `timeManager.ts`, keyed per session. + +--- + +## Caching and scale + +### Redis as a cache + +Hot session state is cached so a 50-player session does not hammer Postgres on +every answer: + +| Key | Holds | +|---|---| +| `quiz:session:${sessionId}` | Session + quiz metadata | +| `quiz:questions:${sessionId}` | Questions with answers | +| `quiz:participant:${participantId}` | Participant record | +| `quiz:leaderboard:${sessionId}` | Current standings | + +Writes go to Postgres and invalidate the relevant key +(`invalidateParticipants`, `invalidateLeaderboard`). `clearSessionCache` wipes +everything when a session ends. Postgres stays the source of truth; Redis is +never authoritative. + +### Redis pub/sub for horizontal scale + +This is the part that makes `ws-server` scalable, and it is easy to miss. + +A WebSocket connection is pinned to one process. With two `ws-server` +instances behind a load balancer, an organizer on instance A and a participant +on instance B would never see each other's events. + +`broadcastToSession` solves it by doing two things at once: + +```js +// 1. deliver to sockets on THIS instance +for (const client of clients) { ...client.ws.send(...) } + +// 2. publish to every OTHER instance +pub.publish(sessionChannel(sessionId), envelope) +``` + +Each instance subscribes to `session:*` and relays what it receives to its own +clients. The envelope carries a `_sid` (a per-process UUID), and the subscriber +**drops messages it published itself** — otherwise local clients would receive +every event twice. + +``` + instance A Redis instance B + ┌──────────┐ ┌──────────┐ + │ organizer│──broadcast──┬──► local sockets │ │ + └──────────┘ │ │ │ + └──► publish session:X ──►│ relay ──►│ participant + └──────────┘ + (A ignores its own _sid) +``` + +So the WS tier can scale horizontally today. The compose file runs a single +instance, but nothing in the code assumes that. + +--- + +## Deployment topology + +``` + Internet + │ + ▼ + ┌────────────────────────┐ + │ nginx-proxy-manager │ :80 :443 (TLS, routing) + │ │ :81 (admin UI) + └──┬─────┬─────┬─────┬───┘ + │ │ │ │ + ┌────────▼┐ ┌──▼───┐ ┌▼─────┐ ┌▼──────┐ + │frontend │ │backend│ │ ws │ │ genai │ + │ (nginx) │ │ :4000 │ │:8080 │ │ :8000 │ + └─────────┘ └───┬───┘ └──┬───┘ └───────┘ + │ │ + ┌─────▼────┐ ┌─▼──────┐ + │ postgres │ │ redis │ + │ (volume) │ │(volume)│ + └──────────┘ └────────┘ + + all on the private `rexial-network` bridge +``` + +Only nginx is exposed. Everything else is reachable only inside the Docker +network. + +> **Warning:** Port **81** is the nginx-proxy-manager admin UI. It is published on the +>host, so it should be firewalled to trusted IPs — anyone who reaches it can +>re-route your traffic. + +Images are built by GitHub Actions and tagged `:latest` **and** `:`; +the deploy pins the SHA so rollback is exact. See +[.github/CI-CD.md](.github/CI-CD.md). + +Migrations are applied by the `backend` container on startup, not by the deploy +job: + +``` +db:generate:prod && db:migrate && start +``` + +--- + +## Design decisions + +**Why split HTTP and WebSocket servers?** +Different scaling shapes. REST traffic is bursty and stateless; WS connections +are long-lived and stateful. Splitting lets you run many WS instances during a +big quiz without over-provisioning the REST tier. + +**Why is the AI service Python and separate?** +The libraries that matter (torch, sentence-transformers, faiss) are Python-only. +Isolating them keeps a multi-GB dependency tree out of the Node images, lets the +service scale independently, and means an LLM outage degrades one feature rather +than taking down the platform. + +**Why do participants not need accounts?** +Signup is the biggest drop-off in a classroom or event. A `Participant` row +belongs to a session, not a user. Registering later is optional and only needed +to see past history. + +**Why cache in Redis instead of in process memory?** +In-process cache breaks the moment you run two instances — and the same Redis +is already needed for pub/sub. + +**Why is scoring server-side?** +It is the only place it can be trusted. The client never learns which answer is +correct until results are revealed. + +--- + +## Known limitations + +Honest list of what is not solved yet. + +| Area | Limitation | +|---|---| +| **Join codes** | Generation has a race condition — two concurrent requests can collide (tracked in [issues.md](issues.md) #1) | +| **AI vector store** | `InMemoryVectorStore` is rebuilt per request; no caching, so the same PDF is re-embedded every time (#5) | +| **AI cost** | No rate limiting or per-user quota on generation endpoints | +| **Uploads** | PDFs are written to a volume and never cleaned up | +| **Reconnection** | A participant who drops mid-quiz rejoins the lobby, but in-flight question state is not restored | +| **Deploys** | `docker compose up -d` recreates containers — brief downtime, no rolling update | +| **Environments** | No staging; `main` deploys straight to production | +| **Tests** | No automated test suite. CI covers lint, types, builds and migrations, but not behaviour | +| **Frontend lint** | ~20 pre-existing eslint errors; the CI lint step reports without blocking (#6) | diff --git a/README.md b/README.md index 388d83c..1e1bd6d 100644 --- a/README.md +++ b/README.md @@ -1,110 +1,114 @@ # ***Real-Time Quiz Platform*** -## 📌 Overview +## Overview Rexial is a **real-time quiz hosting platform** where users can create, host, and participate in quizzes seamlessly using a unique join code. It is designed to provide an interactive and engaging experience with live dashboards, avatars, and collaborative hosting. The platform evolves in multiple versions: -- **Version 1:** Core real-time quiz system -- **Version 2:** AI-powered quiz generation -- **Version 3:** Live video-based quiz experience +-**Version 1:** Core real-time quiz system +-**Version 2:** AI-powered quiz generation +-**Version 3:** Live video-based quiz experience --- ## Version - 1 (DEMO-VIDEO) ## [[Watch the Demo video of version-1]](/assets/versio1-video//Screencast%20From%202026-04-18%2000-37-22.mp4) -## 🛠️ Tech Stack +## Tech Stack ### Monorepo & Tooling -- **TurboRepo** – High-performance monorepo setup -- **pnpm Workspaces** – Efficient dependency management +-**TurboRepo** – High-performance monorepo setup +-**pnpm Workspaces** – Efficient dependency management ### Frontend -- **React** – UI development +-**React** – UI development ### Backend -- **Node.js + Express** – API and server logic -- **WebSockets** – Real-time communication +-**Node.js + Express** – API and server logic +-**WebSockets** – Real-time communication ### Database -- **PostgreSQL** – Relational database -- **Prisma ORM** – Type-safe database access +-**PostgreSQL** – Relational database +-**Prisma ORM** – Type-safe database access ### Version 2 (AI Features) -- **Python** – AI/ML services for quiz generation -- **GenAI APIs / LLMs** – Content-based question generation +-**Python** – AI/ML services for quiz generation +-**GenAI APIs / LLMs** – Content-based question generation ### Version 3 (Live Streaming) -- **WebRTC** – Real-time video/audio communication +-**WebRTC** – Real-time video/audio communication --- -## ✨ Features +## Features -### Version 1 – Core Features -- Create and host quizzes -- Unique quiz join code system -- Participants can join using code -- Co-host support -- Real-time quiz flow using WebSockets -- Live dashboard after each question -- Final leaderboard/dashboard at the end -- Random avatar assigned to participants +### Version 1 – Core Features +- Create and host quizzes +- Unique quiz join code system +- Participants can join using code +- Co-host support +- Real-time quiz flow using WebSockets +- Live dashboard after each question +- Final leaderboard/dashboard at the end +- Random avatar assigned to participants --- -### 🤖 Version 2 – AI Integration -- Generate quizzes using prompts -- Subject-based quiz generation -- Upload PDFs/PPTs to auto-generate questions -- Smart content understanding using GenAI -- Dynamic quiz creation pipeline +### Version 2 – AI Integration +- Generate quizzes using prompts +- Subject-based quiz generation +- Upload PDFs/PPTs to auto-generate questions +- Smart content understanding using GenAI +- Dynamic quiz creation pipeline --- -### 🎥 Version 3 – Live Streaming Experience -- Real-time video quiz sessions -- Host-guided quiz interactions -- Interactive learning environment -- Combine live discussion with quiz attempts -- Feedback and explanations during quiz +### Version 3 – Live Streaming Experience +- Real-time video quiz sessions +- Host-guided quiz interactions +- Interactive learning environment +- Combine live discussion with quiz attempts +- Feedback and explanations during quiz --- ### DevOps & Deployment -- **Docker** – Containerization -- ** Jenkins ** – CI/CD pipelines -- **AWS** – Cloud hosting and infrastructure +-**Docker** – Containerization +-**GitHub Actions** – CI/CD pipelines +-**AWS** – Cloud hosting and infrastructure ### Future DevOps Enhancements -- **Kubernetes** – Container orchestration -- Advanced scaling and monitoring tools +-**Kubernetes** – Container orchestration +-Advanced scaling and monitoring tools --- -## 🧩 Architecture Overview +## Architecture Overview -- **Monorepo structure** using TurboRepo -- Separate apps/services: - - `frontend` (React) - - `http-server` (Express REST API) - - `ws-server` (WebSockets, live quiz) - - `genAI` (Python / FastAPI – RAG, question generation, chat) -- Real-time communication via **WebSockets** -- Containerized using **Docker** -- CI/CD pipelines via ** Jenkins ** -- Hosted on **AWS** -- Future scalability with **Kubernetes** +-**Monorepo structure** using TurboRepo +-Separate apps/services: + -`frontend` (React) + -`http-server` (Express REST API) + -`ws-server` (WebSockets, live quiz) + -`genAI` (Python / FastAPI – RAG, question generation, chat) +-Real-time communication via **WebSockets** +-Containerized using **Docker** +-CI/CD pipelines via **GitHub Actions** — see [.github/CI-CD.md](.github/CI-CD.md) +-Hosted on **AWS** +-Future scalability with **Kubernetes** + + **[Read the full architecture guide →](ARCHITECTURE.md)** — service +boundaries, data model, request flows, caching, and the design decisions +behind them. --- -## 🚀 Getting Started +## Getting Started ### Prerequisites -- Node.js -- pnpm -- PostgreSQL +-Node.js +-pnpm +-PostgreSQL ### Installation diff --git a/apps/genAI/app/agent/Qnagent.py b/apps/genAI/app/agent/Qnagent.py index b0a7b04..f495afb 100644 --- a/apps/genAI/app/agent/Qnagent.py +++ b/apps/genAI/app/agent/Qnagent.py @@ -1,7 +1,8 @@ -from app.utils.llm import get_google_llm -from app.utils.config import TAVILY_API_KEY, require -from langchain_tavily import TavilySearch from langchain_core.messages import HumanMessage +from langchain_tavily import TavilySearch + +from app.utils.config import TAVILY_API_KEY, require +from app.utils.llm import get_google_llm _search = None diff --git a/apps/genAI/app/main.py b/apps/genAI/app/main.py index 7744d41..38a90c2 100644 --- a/apps/genAI/app/main.py +++ b/apps/genAI/app/main.py @@ -1,17 +1,17 @@ -from fastapi import FastAPI, UploadFile, File, HTTPException,Form, Request -from fastapi.responses import JSONResponse -from app.agent.Qnagent import chat +import os +import shutil +from pathlib import Path + +from fastapi import FastAPI, File, Form, HTTPException, Request, UploadFile from fastapi.middleware.cors import CORSMiddleware +from fastapi.responses import JSONResponse from pydantic import BaseModel -from pathlib import Path -import os -import shutil -from app.rag.rag_system import generate_questions,ask_pdf,generate_quiz +from app.agent.Qnagent import chat +from app.rag.rag_system import ask_pdf, generate_questions, generate_quiz from app.utils.config import MissingAPIKey from app.utils.errors import provider_http_error - app = FastAPI(title="Rexial GenAI Service") ALLOWED_ORIGINS = [ @@ -90,7 +90,7 @@ def chat_endpoint(request: ChatRequest): except MissingAPIKey: raise except Exception as exc: - raise provider_http_error(exc) + raise provider_http_error(exc) from exc return { "response": response @@ -113,7 +113,7 @@ async def generate_pdf_questions( except MissingAPIKey: raise except Exception as exc: - raise provider_http_error(exc) + raise provider_http_error(exc) from exc return { "message": "Questions generated successfully", @@ -139,7 +139,7 @@ async def ask_pdf_question( except MissingAPIKey: raise except Exception as exc: - raise provider_http_error(exc) + raise provider_http_error(exc) from exc return { "message": "Answer generated successfully", @@ -168,11 +168,11 @@ async def generate_quiz_endpoint( raise HTTPException( status_code=502, detail=f"The model returned an unusable response: {exc}" - ) + ) from exc except MissingAPIKey: raise except Exception as exc: - raise provider_http_error(exc) + raise provider_http_error(exc) from exc if not questions: raise HTTPException( diff --git a/apps/genAI/app/rag/rag_system.py b/apps/genAI/app/rag/rag_system.py index 7728df4..6271037 100644 --- a/apps/genAI/app/rag/rag_system.py +++ b/apps/genAI/app/rag/rag_system.py @@ -1,14 +1,15 @@ import json import re + from langchain_community.document_loaders import PyPDFLoader -from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_community.vectorstores import InMemoryVectorStore from langchain_huggingface import HuggingFaceEmbeddings -from app.prompts.rag_Qna_generate_prompt import QUESTION_GENERATION_PROMPT +from langchain_text_splitters import RecursiveCharacterTextSplitter + from app.prompts.quiz_json_prompt import QUIZ_JSON_PROMPT +from app.prompts.rag_Qna_generate_prompt import QUESTION_GENERATION_PROMPT from app.utils.llm import get_groq_llm - EMBEDDING_MODEL = "sentence-transformers/all-mpnet-base-v2" _embeddings = None diff --git a/apps/genAI/app/utils/config.py b/apps/genAI/app/utils/config.py index a893864..330f567 100644 --- a/apps/genAI/app/utils/config.py +++ b/apps/genAI/app/utils/config.py @@ -1,6 +1,7 @@ -from dotenv import load_dotenv -from pathlib import Path import os +from pathlib import Path + +from dotenv import load_dotenv # Load apps/genAI/.env explicitly. A bare load_dotenv() searches from the # current working directory, which misses the file when the service is diff --git a/apps/genAI/app/utils/llm.py b/apps/genAI/app/utils/llm.py index 587bcf7..d428857 100644 --- a/apps/genAI/app/utils/llm.py +++ b/apps/genAI/app/utils/llm.py @@ -1,5 +1,6 @@ from langchain_google_genai import ChatGoogleGenerativeAI from langchain_groq import ChatGroq + from app.utils.config import GOOGLE_API_KEY, GROQ_API_KEY, require GROQ_MODEL = "openai/gpt-oss-120b" diff --git a/apps/genAI/pyproject.toml b/apps/genAI/pyproject.toml new file mode 100644 index 0000000..f3ea83a --- /dev/null +++ b/apps/genAI/pyproject.toml @@ -0,0 +1,26 @@ +[tool.ruff] +target-version = "py311" +line-length = 100 +src = ["."] + +[tool.ruff.lint] +select = [ + "E", # pycodestyle errors + "F", # pyflakes + "I", # import sorting + "UP", # pyupgrade + "B", # bugbear +] + +ignore = [ + # FastAPI's dependency injection is expressed as call-valued defaults + # (File(...), Form(...)), which bugbear flags by design. + "B008", + # Provider SDKs raise their own exception types, so the AI endpoints + # deliberately catch broadly and translate in provider_http_error. + "BLE001", +] + +[tool.ruff.lint.per-file-ignores] +# Prompt modules are long strings, not code to wrap. +"app/prompts/*.py" = ["E501"] diff --git a/contribution.md b/contribution.md index a460d9a..b17c8a4 100644 --- a/contribution.md +++ b/contribution.md @@ -1,45 +1,45 @@ -# 🤝 First Contribution Guide (For Beginners step-by-step guide) +# First Contribution Guide (For Beginners step-by-step guide) Welcome! This guide will help you make your first contribution to the project. --- -## 🚀 1. Fork the Repository +## 1. Fork the Repository -- Go to the project repository on GitHub -- Click the **Fork** button (top right) -- This creates your own copy of the repo +-Go to the project repository on GitHub +-Click the **Fork** button (top right) +-This creates your own copy of the repo --- -## 📥 2. Clone Your Fork +## 2. Clone Your Fork ```bash git clone https://github.com//.git cd ``` -## 🔗 3. Add Upstream Remote -* This lets you sync with the original repository:- +## 3. Add Upstream Remote +*This lets you sync with the original repository:- ```bash git remote add upstream https://github.com//.git git remote -v ``` -## 📦 4. Install Dependencies +## 4. Install Dependencies ```bash pnpm install ``` -## ⚙️ 5. Setup Environment -* Create a `.env` file in the `/packages/db` & `/apps/http-server`: +## 5. Setup Environment +*Create a `.env` file in the `/packages/db` & `/apps/http-server`: -* In `/packages/db` put it: - +*In `/packages/db` put it: - ```bash DATABASE_URL=postgresql://postgres:postgres@db:5432/mydb ``` -* and in `/apps/http-server` put it :- +*and in `/apps/http-server` put it :- ```bash PORT=4000 @@ -52,32 +52,88 @@ FRONTEND_URL=http://localhost:5173 ``` -## 🐳 6. Start the Project +*The **GenAI service** (`/apps/genAI`) has its own env. Copy the example: + +```bash +cp apps/genAI/.env.example apps/genAI/.env +``` + +*Then add your own free API keys — [Groq](https://console.groq.com/keys), + [Google AI Studio](https://aistudio.google.com/apikey), + [Tavily](https://tavily.com): + +```bash +GROQ_API_KEY= +GOOGLE_API_KEY= +TAVILY_API_KEY= +``` + +> **Note:** Without these keys the app still runs — only the AI features return an +>error saying which key is missing. You can skip them unless you are working +>on the GenAI service. + +## 6. Start the Project ```bash docker-compose up --build ``` -## 🌱 7. Create a New Branch -* Always create a new branch for your work: +## 7. Create a New Branch +*Always create a new branch for your work: ```bash git checkout -b feature/ ``` #### Examples: -* `feature/add-auth` -* `fix/ws-connection-bug` +*`feature/add-auth` +*`fix/ws-connection-bug` + +## 8. Make Your Changes +*Follow project structure: + +*apps/ → services (http-server, ws-server, frontend, genAI) +*apps/genAI → Python / FastAPI service for AI quiz generation (RAG) +*packages/db → Prisma + database + +*Keep changes small and focused +*Follow existing code style + +> Unsure how the pieces fit together? [ARCHITECTURE.md](ARCHITECTURE.md) +>explains each service, the data model, and how a live quiz actually runs. + -## ✍️ 8. Make Your Changes -* Follow project structure: +## 9. Working on the GenAI Service (optional) -* apps/ → services (http-server, ws-server, frontend) -* packages/db → Prisma + database +*The AI service is **Python**, so `pnpm install` does not cover it +*If your change touches `/apps/genAI`, set up a virtual environment: -* Keep changes small and focused -* Follow existing code style +```bash +cd apps/genAI +python3 -m venv .venv +.venv/bin/pip install -r requirements.txt +``` + +*Run it on its own (Docker does this for you otherwise): + +```bash +pnpm --filter genai dev # starts on http://localhost:8000 +``` + +*Interactive API docs are at http://localhost:8000/docs — handy for trying + the endpoints without the frontend +*Python code is linted with **ruff**, and CI enforces it: + +```bash +.venv/bin/ruff check app --fix +``` + +#### What the service does +*`POST /generate-quiz` → reads an uploaded PDF and returns multiple-choice + questions as JSON (this is what the **Generate with AI** button uses) +*`POST /ask-pdf` → answers a question about an uploaded PDF +*`POST /chat` → general chat, searches the web when the question needs it -## 🧪 9. Test Your Changes -* Make sure everything works: +## 10. Test Your Changes +*Make sure everything works: ```bash pnpm run dev ``` @@ -88,8 +144,8 @@ pnpm run dev docker-compose up ``` -## 💾 10. Commit Changes -* Write clear commit message +## 11. Commit Changes +*Write clear commit message ```bash git add . git commit -m "feat: add user authentication" @@ -97,29 +153,36 @@ git commit -m "feat: add user authentication" ##### Common prefixes: -* feat: new feature -* fix: bug fix -* docs: documentation -* refactor: code improvement +*feat: new feature +*fix: bug fix +*docs: documentation +*refactor: code improvement -## ⬆️ 11. Push to Your Fork +## 12. Push to Your Fork ```bash git push origin feature/ ``` -## 🔁 12. Create Pull Request (PR) -* Go to your fork on GitHub -* Click Compare & Pull Request -* Add: - * Clear title - * Description of changes - * Screenshots (if UI changes) +## 13. Automated Checks + +*When you open a PR, GitHub Actions runs lint, type-checks, builds, and tests automatically +*If something fails, click **Details** on the failed check to see why +*See [.github/CI-CD.md](.github/CI-CD.md) for what each check does and how to reproduce it locally + + +## 14. Create Pull Request (PR) +*Go to your fork on GitHub +*Click Compare & Pull Request +*Add: + *Clear title + *Description of changes + *Screenshots (if UI changes) -## 🔄 13. Sync with Upstream (Important) -* Before new work, sync your fork +## 15. Sync with Upstream (Important) +*Before new work, sync your fork ```bash git checkout main git pull upstream main diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 5d911b7..e1f80af 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -26,7 +26,7 @@ services: build: context: . dockerfile: apps/frontend/Dockerfile.prod - image: codeheist/rexial-frontend:latest + image: codeheist/rexial-frontend:${IMAGE_TAG:-latest} container_name: rexial-frontend-prod restart: always networks: @@ -42,7 +42,7 @@ services: dockerfile: apps/http-server/Dockerfile.prod args: DATABASE_URL: ${DATABASE_URL} - image: codeheist/rexial-http-server:latest + image: codeheist/rexial-http-server:${IMAGE_TAG:-latest} container_name: rexial-http-server-prod restart: always env_file: @@ -66,7 +66,7 @@ services: dockerfile: apps/ws-server/Dockerfile.prod args: DATABASE_URL: ${DATABASE_URL} - image: codeheist/rexial-ws-server:latest + image: codeheist/rexial-ws-server:${IMAGE_TAG:-latest} container_name: rexial-ws-server-prod restart: always env_file: @@ -84,7 +84,7 @@ services: build: context: . dockerfile: apps/genAI/Dockerfile.prod - image: codeheist/rexial-genai:latest + image: codeheist/rexial-genai:${IMAGE_TAG:-latest} container_name: rexial-genai-prod restart: always env_file: diff --git a/issues.md b/issues.md index 4cfa2d5..38e8e18 100644 --- a/issues.md +++ b/issues.md @@ -2,21 +2,21 @@ ## (1)First Issue -#### ```Race Condition``` -#### -> Check the route in /apps/http-server/routes +#### ```Race Condition``` +#### -> Check the route in /apps/http-server/routes ```bash /api/v1/quizzes/:quizId/generate-access-code ``` -* In this route, we have to tackle race-condition while generating unique-access code for joining quiz. -* If two user say A & B. If they request at the same time and accidently get the same code... +*In this route, we have to tackle race-condition while generating unique-access code for joining quiz. +*If two user say A & B. If they request at the same time and accidently get the same code... -* Request - A +*Request - A ```bash generateCode → AC123A ``` -* Request - B +*Request - B ```bash generateCode → AC123A ``` @@ -24,7 +24,7 @@ -> SO this may give some concurrency issues. #### ***Final Command*** - -> See the route and analyze the prisma queries and try to optimise the code to stop these concurrency issues + -> See the route and analyze the prisma queries and try to optimise the code to stop these concurrency issues ## (2)Second Issue (Functionality Add on) @@ -32,19 +32,61 @@ -> Backend route `/apps/http-server/routes/quiz.ts` #### -> Check the page in frontend `/apps/frontend/src/screens/QuizBuilder.tsx` - * This is the page where we can create question - * But in this page after creating Questions there is no option of deleting Questions. - * Therefore, you have to implement the functionalty of deleting the created questions + *This is the page where we can create question + *But in this page after creating Questions there is no option of deleting Questions. + *Therefore, you have to implement the functionalty of deleting the created questions #### ***Final Command*** - -> See and use the page and after understanding the codebase, Implementing this Feature in frontend and backend both. + -> See and use the page and after understanding the codebase, Implementing this Feature in frontend and backend both. ## (3) Third Issue (Functionality Add on) -> Backend route `/apps/http-server/routes/quiz.ts` #### -> Check the page in frontend `/apps/frontend/src/screens/QuizBuilder.tsx` - * This is the page where we can create question - * But in this page after creating Questions there is no option of editing or updating Questions. - * Therefore, you have to implement the functionalty of updating the created questions + *This is the page where we can create question + *But in this page after creating Questions there is no option of editing or updating Questions. + *Therefore, you have to implement the functionalty of updating the created questions + + + ## (4) Fourth Issue (GenAI - Functionality Add on) + + -> GenAI service `/apps/genAI/app/main.py` + #### -> Check the component in frontend `/apps/frontend/src/components/AiQuizGenerator.tsx` + *This is the panel where a host uploads a PDF and generates questions with AI + *Right now if the AI writes one bad question, the host can only delete it + *There is no way to say "regenerate just this one question" + *Therefore, implement a re-generate option for a single question + +#### ***Final Command*** + -> Add an endpoint that regenerates one question from the same PDF context, + and wire a small re-roll button into the review list in the frontend + + + ## (5) Fifth Issue (GenAI - Performance) + + -> RAG code `/apps/genAI/app/rag/rag_system.py` + *Every request re-reads the PDF, re-chunks it and rebuilds the vector store + *So asking two questions about the same PDF does all the work twice + *The vector store is `InMemoryVectorStore`, so nothing survives a restart + +#### ***Final Command*** + -> Cache the built vector store per PDF (hash the file contents as the key) + so repeat requests on the same document skip the embedding step + + + ## (6) Sixth Issue (Good First Issue - Frontend) + + -> Frontend `/apps/frontend/src` + *`pnpm run lint` currently reports around 20 eslint errors + *Mostly `@typescript-eslint/no-explicit-any` and unused variables + *Because of this, the lint step in CI is set to report without blocking + *Files affected: `Dashboard.tsx`, `JoinQuiz.tsx`, `LiveQuiz.tsx`, + `Login.tsx`, `Signup.tsx`, `InviteAccept.tsx`, `QuizBuilder.tsx` + +#### ***Final Command*** + -> Replace the `any` types with real types (the Prisma types from `@repo/db` + are a good starting point) and remove unused variables. Once `pnpm run lint` + passes, remove `continue-on-error: true` from the Lint step in + `.github/workflows/ci.yml` so lint becomes a real merge gate diff --git a/planning.md b/planning.md index 39330e7..4ff01c1 100644 --- a/planning.md +++ b/planning.md @@ -23,7 +23,7 @@ _________________ `` Rexial ``________________ ### (3) => Setting questions for users as quiz host -> Quiz organnisers can set their custom questions by own - -> and also set the time-limit for each question by own + -> and also set the time-limit for each question by own @@ -66,8 +66,8 @@ _________________ `` Rexial ``________________ Database: Firebase Firestore for rooms/leagues (atomic updates); Redis for live leaderboards. - (ii) auto genearted quize by uploading the pdf of the content and syllabus - (iii) AI generated quiz in just one prompt with dificulty levels + (ii) auto genearted quize by uploading the pdf of the content and syllabus [SHIPPED v2] + (iii) AI generated quiz in just one prompt with dificulty levels [SHIPPED v2] @@ -78,6 +78,43 @@ _________________ `` Rexial ``________________ + ## ``v2 Shipped: GenAI Service`` + +### (5) => AI quiz generation from a PDF + +Implemented as a separate Python service at ``apps/genAI`` (FastAPI), so the +AI work stays out of the Node servers and can scale on its own. + +**What a host can do now** +-> In the Quiz Builder, click **Generate with AI** +-> Upload a PDF (syllabus, notes, chapter) and describe what they want + e.g. *"5 medium questions about chapter 2"* +-> The service reads the PDF and writes multiple-choice questions with + 4 options each, one correct answer, and a difficulty label +-> Generated questions land in a **review step** first: the host can edit the + wording, fix an option, change which answer is correct, or drop a question +-> Nothing is saved to the database until the host confirms + +**Endpoints** (see ``apps/genAI/app/main.py``) +-> ``POST /generate-quiz`` structured JSON questions, used by the UI +-> ``POST /generate-questions`` the same thing as a readable text blob +-> ``POST /ask-pdf`` question answering over an uploaded PDF (RAG) +-> ``POST /chat`` general chat, with web search when needed + +**How it works** +-> PDF is chunked, embedded with ``sentence-transformers``, and searched + with an in-memory vector store (RAG) +-> Groq runs question generation; Gemini runs chat; Tavily handles web search +-> Model output is validated before it is returned: any question without + exactly 4 options and exactly 1 correct answer is discarded rather than + shown to the host + +### Still open on the AI side +-> No persistence of uploaded PDFs between requests (each call re-reads) +-> Question generation is one-shot; no "regenerate this one question" yet +-> No caching, so generating twice from the same PDF costs twice + + ## Schema planning... #### Here is the basic version of schema of Qtrive diff --git a/setup.md b/setup.md index cedee11..bc6291c 100644 --- a/setup.md +++ b/setup.md @@ -1,6 +1,6 @@ -# 💡 Project Setup Guide (Turborepo + Pnpm + Docker): +# Project Setup Guide (Turborepo + Pnpm + Docker): -### 🗂️ Project Folder Structure +### Project Folder Structure ``` ├── apps/ │ ├── http-server/ @@ -17,15 +17,15 @@ └── package.json ``` -### 🎟️ Prerequisites +### Prerequisites #### Make sure you have the following installes:- - - * Node.js (>= 20) - * pnpm (>= 9) - * Python (>= 3.11) # for the genAI service - * Docker - * Docker Compose + + *Node.js (>= 20) + *pnpm (>= 9) + *Python (>= 3.11) # for the genAI service + *Docker + *Docker Compose #### (1) Now, fork & clone the repository ```bash @@ -33,7 +33,7 @@ git clone https://github.com/TheCodeHeist-Coder/Qtrive.git cd Qtrive ``` -#### (2) Install pnpm globally if not installed +#### (2) Install pnpm globally if not installed ```bash npm install -g pnpm @@ -46,13 +46,13 @@ pnpm install #### (4) Setup Environment Variables -* Create a `.env` in `/packages/db` and put this ennvironment variable +*Create a `.env` in `/packages/db` and put this ennvironment variable ```bash DATABASE_URL=postgresql://postgres:postgres@db:5432/mydb ``` -* Create a `.env` file in `/apps/http-server` and put these variables +*Create a `.env` file in `/apps/http-server` and put these variables ```bash PORT=4000 @@ -64,7 +64,7 @@ FRONTEND_URL=http://localhost:5173 ``` -* Create a `.env` file in `/apps/genAI` (copy from `.env.example`) +*Create a `.env` file in `/apps/genAI` (copy from `.env.example`) ```bash cp apps/genAI/.env.example apps/genAI/.env ``` @@ -79,9 +79,9 @@ TAVILY_API_KEY= ```bash docker-compose up --build ``` -* Build all services (http-server, ws-server, frontend, genAI) -* Start PostgreSQL database -* Start all containers +*Build all services (http-server, ws-server, frontend, genAI) +*Start PostgreSQL database +*Start all containers #### (6) Access the application @@ -95,7 +95,7 @@ PostgreSQL → localhost:5432 --- -### 🤖 Running the GenAI service without Docker +### Running the GenAI service without Docker The Python service is part of the Turborepo workspace, so `pnpm dev` at the repo root starts it alongside the JS apps — but it needs a virtualenv first: