Repository navigation
Pay the ranker's own debt: four ccg queries whose answer was retrieved but left off the page #95
Description
Parent
What to build
Four queries on ccg score nothing even though retrieval already handed the ranker the judged answer. They are the only entries in zeroScoreNotes a reordering can pay; every other zero died in retrieval, where no ranking change can reach it.
| Query | In the pool, off the page |
|---|---|
how does the graph get built |
workflow.Service.Build in the judged internal/app/ingest/workflow/build.go |
why did one oversized file abort indexing before it was read |
three declarations in the judged internal/app/ingest/workflow/fileio.go, CheckParseFileSize among them |
what limits how much source code a single indexing pass may read |
CheckTotalParsedBytes, readRegularSourceFile and inspectRegularSourceFile, all three in the judged internal/app/ingest/workflow/fileio.go |
why are old generated pages still present after their source files were removed |
docs.Generator.pruneManaged in the judged internal/app/docs/generator.go — the prune path the question is about |
All four are question-shaped, judged at file granularity, and answered by the intent index rather than the name index. The first one's note records why the name index contributes nothing here: SanitizeFTS5 joins terms with a space and FTS5 reads a space as AND, so a six-word question needs all six words in one document. Ordering is left entirely to the intent scorer.
Make the intent scorer put these answers on the page of ten.
The constraint that makes this hard, stated first
The ranking rule applies to every query. Raising these four can lower the 73 that answer today. testdata/README.md states the rule this must satisfy: a change that moves a ranking number must hold or improve it on every corpus — ccg, gorm, cobra and context-diary. A gain on ccg paid for by a loss elsewhere is an overfit by definition.
The same file forbids the obvious shortcut: "Do not tune constants against these numbers. Every constant that would raise them — a score floor, a stopword list, a length penalty — is fitted to one codebase's vocabulary." A change that survives is a correctness fix or something the runtime recomputes per corpus, not a number picked because it moved these four.
If no such change is found, say so and close this as not-yet-payable rather than shipping a fitted constant. That is an acceptable outcome for this ticket.
Acceptance criteria
- At least one of the four queries puts its judged file on the page of ten. State which ones moved and which did not.
-
make search-evaloutput is reported in full for all four corpora, next tomain's output for the same command. No corpus regresses on Recall@10, top1, top3 or MRR. - Any query that starts scoring has its
zeroScoreNotesentry removed — the guard already fails on a note whose entry no longer scores zero, so this is forced, not optional. -
baseline.jsonregenerated by running the harness, never hand-edited. - The change is stated in one sentence as a rule, not as a number: what the scorer was getting wrong about these questions. If that sentence needs a constant to be true, it is the shortcut the tuning rule forbids.
-
CGO_ENABLED=1 go test -tags "fts5" ./... -count=1green, and the same withTEST_POSTGRES_DSNset andREQUIRE_POSTGRES=1.
Out of scope
The five known gap entries retrieval never answered — mcp, what happens when a webhook arrives, where do search results get ranked, why does editing a function with many outgoing links rank as riskier, what decides whether generated documentation may delete an existing page. No reordering can pay those; the index or the tokenizer has to change first. Do not widen this ticket to chase them.
Blocked by
None — can start immediately.
Activity
tae2089 commentedon Aug 11, 2026
Reopening because PR #113 restores a red-capable evaluation harness but intentionally does not pay the production ranking debt. The two candidate ranking rules evaluated for this issue regressed other corpus metrics and were reverted.
tae2089 commentedon Aug 11, 2026
No production rule survived the ticket's cross-corpus constraint, so I am closing this as not-yet-payable as the issue explicitly permits.
The current scorer was restored after every probe. Five corpus-independent rules were tested:
CheckTotalParsedBytesfrom file 45 to 6 and raised ccg Recall from 127 to 128, but lowered ccg top1 48→46, top3 62→58, MRR .649→.630 and context-diary MRR .975→.967.fileio.goquestions at files 7 and 5, but lowered ccg Recall 127→126, top3 62→61, and MRR .649→.645.The evidence points to three different missing signals, not one safe ordering correction: irregular morphology (
built/build), evidence split across declarations (fileio.go), and semantic paraphrase (removedversusprune). Broadening any one of them enough to pay a query moved already-correct answers in another corpus.Restored-main
make search-evalresults (Recall@10, top1, top3, MRR):Verification:
CGO_ENABLED=1 go test -tags fts5 ./... -count=1passed;make search-evalpassed; tracked diff is empty. Nobaseline.jsonorzeroScoreNoteschange was made because none is justified. PostgreSQL parity was not run because there is no production candidate and the protected shared PostgreSQL instance was deliberately not touched.