Skip to content

reqtxt: record entry provenance, and close three pip-parity gaps - #49

Merged
jonyoder merged 3 commits into
mainfrom
feat/46-reqtxt-gaps
Aug 28, 2026
Merged

jonyoder merged 3 commits into
mainfrom
feat/46-reqtxt-gaps

Conversation

@jonyoder

Copy link
Copy Markdown
Collaborator

Closes #46.

Two commits: provenance, then three pip-parity fixes.

Source on every entry (the substantial piece)

Flatten splices every include level into one flat File and consumes each IncludeEntry, and OptionEntry carried only {Name, Value}. So after flattening there was nothing left to attribute an entry to a file with, and no entry carried a line number at all.

That matters most for whole-file options, because Pre() and IndexURL() are any-wins across the flattened result. A single --pre nested three includes deep, even inside a -c subtree, silently changes what a caller collects. "Pre-releases were enabled" is a materially worse diagnostic than "enabled by line 2 of extra.txt", and on a machine with no network the first is undiagnosable.

Source{Path, Line} is now on all four entry types, with SourceOf for a caller that wants provenance without a type switch, and WithPath so a bare Parse can record a path it otherwise cannot know. Flatten supplies it per file.

Stamped in Parse's line loop rather than at the eight construction sites: every entry from a logical line shares that line, including per-requirement options gathered from continuations, so one pass cannot miss a site the way eight edits could.

Flatten appends its WithPath after the caller's opts rather than prepending, so a caller's own WithPath cannot attribute every included file to one path. TestSource_CallerWithPathDoesNotOverrideFlatten pins that, since getting it backwards would be invisible.

Two consumer diagnostics are covered end to end because they are why this exists: --pre leaking out of a constraints subtree, and a -r nested inside a -c resetting constraint-ness so those pins become requirements.

Test churn: 13 existing subtests compared whole entry structs and now carry an expected Source. I set the real line numbers rather than zeroing the field, so those assertions gained coverage instead of an exemption.

Three pip-parity fixes

All three verified against pip 26.1's own source and packaging 26.3, not from reading this package.

--all-releases, --only-final, --use-feature were missing from knownOptions and all three are in pip's SUPPORTED_OPTIONS. The consequence was worse than losing normalization: an option missing from the table is assumed boolean, so its argument was dispatched as its own line and became a fabricated requirement. --use-feature 2020-resolver produced a package named 2020-resolver from a valid pip file, and a consumer resolving the closure of every requirement then hunts for something that does not exist. TestKnownOptions_CoversPipSupportedOptions now asserts every long option in SUPPORTED_OPTIONS is handled, so the gap cannot silently reopen.

Worth flagging for consumers: --all-releases and --only-final are pip's per-package replacements for --pre, which pip refuses to combine with them.

A standalone --hash was a hard error. pip logs "line %s has --hash but no requirement, and will be ignored" (req_file.py:217-219) and carries on, so the file installs fine and rejecting it made this package stricter than pip. Now surfaced as a file-level option rather than dropped, so a caller can reproduce pip's warning — with a test pinning that it does not attach to a preceding requirement, which is why it was an error to begin with.

Flatten(path, nil) panicked on the first dereference. Returns an error now.

What deliberately did not change

requests; with an empty marker stays rejected. #46 claimed pip accepts it; checked against packaging 26.3, Requirement("requests;") raises InvalidRequirement: Expected a marker variable or quoted string. The rejection is faithful, and TestTrailingSemicolonStillRejected records that so it is not re-litigated from memory.

I also considered making unknown options a hard error to match pip exactly, and discarding an unknown option's trailing tokens. Both replace one arity guess with another: assuming boolean is the right guess for --frob foo, where foo really is a requirement, and the wrong one for --timeout 60. The fix is to leave fewer options unknown, not to change the guess. The reasoning is in a comment at that branch so the next reader does not redo it.

Verification

gofmt clean, go vet ./... clean, full module suite green.

🤖 Generated with Claude Code

jonyoder and others added 3 commits August 28, 2026 06:38
Flatten splices every include level into one flat File, consuming and
discarding each IncludeEntry, and OptionEntry carried only {Name, Value}.
So after flattening there was nothing left to attribute an entry to a file
with, and no line number on any entry at all.

That matters most for whole-file options, because File.Pre() and
IndexURL() are any-wins across the flattened result. A single "--pre"
nested three includes deep -- even inside a "-c" subtree -- silently
changes what a caller collects, and "pre-releases were enabled" is a
materially worse diagnostic than "enabled by line 2 of extra.txt". For a
consumer building an air-gapped bundle, the first is undiagnosable on a
machine with no network.

Source{Path, Line} is now on all four entry types, plus SourceOf for a
caller that only wants provenance and not a type switch. WithPath lets a
bare Parse record a path it cannot otherwise know, since Parse takes
content rather than a filename; Flatten supplies it per file
automatically.

Stamped in Parse's line loop rather than at the eight construction sites.
Every entry from a logical line shares that line, including the
per-requirement options gathered from its continuations, so one pass
cannot miss a site the way eight edits could.

Flatten APPENDS its WithPath after the caller's opts rather than
prepending, so a caller's own WithPath cannot attribute every included
file to a single path. TestSource_CallerWithPathDoesNotOverrideFlatten
pins that ordering, since getting it backwards would be invisible.

Test churn: 13 existing subtests compared whole entry structs and now
carry an expected Source. I set the real line numbers rather than zeroing
the field, so those assertions gained coverage instead of an exemption.

Two consumer-facing diagnostics are covered end to end because they are
the reason this exists: --pre leaking out of a constraints subtree, and a
"-r" nested inside a "-c" resetting constraint-ness so those pins become
requirements.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three real divergences from pip, verified against pip 26.1's own source
and packaging 26.3 rather than from reading.

1. --all-releases, --only-final and --use-feature were missing from
   knownOptions, and all three are in pip's SUPPORTED_OPTIONS. The
   consequence was worse than "not normalized": an option missing from the
   table is assumed boolean, so its argument was dispatched as its own
   line and became a FABRICATED requirement. "--use-feature 2020-resolver"
   yielded a package named "2020-resolver" from a valid pip file, and a
   consumer resolving the closure of every requirement then hunts for a
   package that does not exist. All three take a value (pip: type="str").

   TestKnownOptions_CoversPipSupportedOptions now asserts every long option
   in pip's SUPPORTED_OPTIONS is handled, so this gap cannot silently
   reopen.

   Worth noting for consumers: --all-releases and --only-final are pip's
   per-package replacements for --pre, which pip refuses to combine with
   them.

2. A standalone --hash was a hard error. pip logs "line %s has --hash but
   no requirement, and will be ignored" (req_file.py:217-219) and carries
   on, so the file installs fine and rejecting it made this package
   stricter than the thing it models. It is now surfaced as a file-level
   option rather than dropped, so a caller can reproduce pip's warning,
   and a test pins that it does not attach to a preceding requirement --
   which is why it was an error in the first place.

3. Flatten(path, nil) panicked on the first dereference. It returns an
   error now; in a CLI a stack trace is a materially worse failure and a
   caller cannot tell the two apart from outside.

What did NOT change, deliberately: "requests;" with an empty marker stays
rejected. A review claimed pip accepts it. Checked against packaging 26.3,
Requirement("requests;") raises InvalidRequirement, so the rejection is
faithful. TestTrailingSemicolonStillRejected records that so the claim is
not re-litigated from memory.

Also considered and rejected: making unknown options a hard error to match
pip exactly, and discarding an unknown option's trailing tokens. Both
replace one arity guess with another -- assuming boolean is RIGHT for
"--frob foo", where foo really is a requirement, and wrong for
"--timeout 60". The fix is to leave fewer options unknown, not to change
the guess.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jonyoder
jonyoder force-pushed the feat/46-reqtxt-gaps branch from 39cad49 to 3e96ba8 Compare August 28, 2026 10:39
@jonyoder
jonyoder merged commit 7231c32 into main Aug 28, 2026
4 checks passed
@jonyoder
jonyoder deleted the feat/46-reqtxt-gaps branch August 28, 2026 10:39
jonyoder added a commit that referenced this pull request Aug 28, 2026
Dates [Unreleased] as 0.9.0 (2026-08-28) and adds its compare link. CHANGELOG only.

A minor bump under the 0.x policy because two changes alter observable behaviour: reqtxt now recognizes --all-releases/--only-final/--use-feature (an unrecognized option was assumed boolean, so its argument became a fabricated requirement), and a standalone --hash line is accepted rather than rejected, matching pip.

Contents since v0.8.0: tags manylinux floor fix (#43), marker EvaluateUndecidable/Variables (#47), tags IsCompatibleOrNewer/CompileAnyLibc/Archs (#48), reqtxt Source and pip-parity fixes (#49), dependency bumps (#50).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reqtxt: entries carry no provenance; unknown options fabricate requirements; two cases stricter than pip; nil open panics

1 participant