Repository navigation
fix: replace inaccessible PDF in RAG example - #1732
Open
jeeth-kataria wants to merge 1 commit into
Open
jeeth-kataria wants to merge 1 commit into
jeeth-kataria wants to merge 1 commit into
Conversation
Use the dated W3C WCAG 1.0 checklist for document-grounded QA. Document prerequisites and add an explicit structural e2e example check. Fixes generative-computing#373 Assisted-by: Codex Signed-off-by: Jeeth <jeethkataria9798@icloud.com>
rafvasq
requested changes
Oct 16, 2026
Contributor
There was a problem hiding this comment.
Thanks @jeeth-kataria! The script and test change look good to me. For the example's README, I think that the documentation can be kept short and consistent with how other examples are described you can minimize the scope of this PR to simply updating the document used and example using it because the README will need to be re-organized but that's outside of this PR.
Comment on lines
+18
to
+65
| Document-grounded question answering using W3C's | ||
| [WCAG 1.0 checklist PDF](https://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505/full-checklist.pdf). | ||
| The question asks what the three checkpoint priority levels mean; the answer is | ||
| in the document's **Priorities** section on page 2. In brief, developers must | ||
| satisfy Priority 1 checkpoints, should satisfy Priority 2 checkpoints, and may | ||
| address Priority 3 checkpoints. Generated wording can vary. | ||
|
|
||
| This dated, seven-page publication is historical sample material, not current | ||
| accessibility guidance. It replaces a USDA PDF that returned HTTP 403 for users | ||
| outside the US. The PDF remains hosted by W3C under its | ||
| [document license](https://www.w3.org/Consortium/Legal/copyright-documents-19990405); | ||
| no copy is bundled with this example. | ||
|
|
||
| **Prerequisites:** Install the Docling extra, have internet access to download | ||
| the PDF and Docling's conversion models, and run Ollama with `granite4.2:3b` | ||
| (the current `start_session()` default). FAISS and sentence-transformers are | ||
| not needed for this example. OCR is disabled because the PDF contains text; | ||
| Docling may still download layout and table models on the first conversion. | ||
|
|
||
| From the repository root: | ||
|
|
||
| ```bash | ||
| uv sync --frozen --extra docling | ||
| ollama serve # run in a separate terminal if Ollama is not already running | ||
| ollama pull granite4.2:3b | ||
| uv run python docs/examples/rag/mellea_pdf.py | ||
| ``` | ||
|
|
||
| From another working directory, use absolute paths to the checkout and script: | ||
|
|
||
| ```bash | ||
| uv run --project /path/to/mellea python /path/to/mellea/docs/examples/rag/mellea_pdf.py | ||
| ``` | ||
|
|
||
| The document URL is independent of the working directory. PDF conversion happens | ||
| before the session is created, so document-loading failures can be distinguished | ||
| from model-inference failures. The `e2e, ollama, slow` markers opt the example | ||
| into explicit slow-test runs. Its assertions check only that the result is a | ||
| non-empty string, not the factual content of the answer. The `slow` marker | ||
| accounts for cold conversion-model downloads and inference; ordinary test runs | ||
| exclude it. `CICD=1` does not skip this structural test, but current PR CI does | ||
| not collect Python examples. | ||
|
|
||
| To run it through pytest, including the slow marker: | ||
|
|
||
| ```bash | ||
| uv run pytest docs/examples/rag/mellea_pdf.py -m "e2e and ollama" --no-cov -v -s | ||
| ``` |
Contributor
There was a problem hiding this comment.
Extra context not needed at the moment. I think this can be kept to one-line for now as it was before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Issue
Fixes #373
Description
Replace the USDA PDF (HTTP 403 from our Indian network) with W3C's dated WCAG 1.0 checklist. Ask about its three priority definitions, update the grounding label and prerequisites, and disable OCR for this text PDF. The document remains externally hosted; no binary or dependency is added.
Add structural assertions and
e2e, ollama, slowmarkers. Also correct the default-model pull instruction intest/README.md, as suggested by the maintainer. No library, adapter or workflow code changes.The maintainer waived US verification and requested this PR in the issue discussion. No US run was performed; worldwide accessibility is not claimed.
Testing
Actual local validation:
mellea.start_session()'s defaultgranite4.2:3b, without model overrides. Recorded prompt includes the complete document underwcag_checklist; response metadata confirms Granite. Root and absolute-path execution from another directory succeeded. Earlier directory checks substituted inference, and earlier Llama runs used an override; neither is the default-model evidence.CICD=1 HF_HUB_DISABLE_XET=1 uv run --no-sync pytest docs/examples/rag/mellea_pdf.py -m 'e2e and ollama' --no-cov -v -s: 1 passed in 46.51s on the final branch. Latest answer correctly states all three definitions, one sentence each. Semantic review is manual; another run added less precise qualifiers, so structural success does not guarantee answer quality.uv run --no-sync pytest test/test_example_collection.py --no-cov -q: 20 passed in 12.58s.uv run --no-sync pre-commit run --all-files: passed all hooks, including full mypy. Initial attempts lacked optional SDKs/pip; automatic stub installation then caused version drift. Installed locked dev extras and compatible local-only stubs, restoring locked NumPy/Requests. No dependency files changed.git diff --cached --check: passed before commit.Broader suite attempt (
CICD=1 HF_HUB_DISABLE_XET=1 uv run --no-sync pytest test --no-cov --maxfail=1 -q) stopped on the 900s uncertainty-adapter test timeout: 646 passed, 62 skipped, 136 deselected, 1 xpassed, 1 error. No complete-suite pass or remote CI result is claimed.HF_HUB_DISABLE_XET=1was used for verification after an hf-xet conversion-weight download stalled; it is not in the example.Marker rationale
The example checks only string type and non-empty output, not generated facts. Cold conversion-weight/model downloads justify
slow; explicit-m 'e2e and ollama'overrides the default slow exclusion.CICD=1skips qualitative items, not this structural check.Current PR CI installs Docling and pulls Granite but does not collect Python examples. Notebook CI collects notebooks only; nightly defaults exclude this slow example. This enables explicit execution, not new automated PR-CI coverage.
Attribution
Codex assisted with investigation, code and validation. Commit uses the human author's existing Git identity, DCO sign-off and
Assisted-by: Codex.Adding a new component, requirement, sampling strategy, or tool?
If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.
NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.