gh-136640: Correct the description of AST location attributes

[v0ropaev]

Two sentences in the ast.AST block of Doc/library/ast.rst say things that are not true.

"Instances of ast.expr and ast.stmt subclasses have lineno, col_offset, end_lineno, and end_col_offset attributes." Eight types carry an attributes (...) clause in Parser/Python.asdl, not two: stmt, expr, excepthandler, arg, keyword, alias, pattern, type_param. So an ExceptHandler, an alias or a keyword has full position information while the docs say it should not:

>>> [c.__name__ for c in vars(ast).values()
...  if isinstance(c, type) and issubclass(c, ast.AST) and c.__base__ is ast.AST
...  and 'end_col_offset' in c._attributes]
['expr', 'stmt', 'excepthandler', 'arg', 'keyword', 'alias', 'pattern', 'type_param']

"the end positions are not required by the compiler and are therefore optional." True for six of those eight. pattern and type_param declare int end_lineno, int end_col_offset rather than int? (Python.asdl lines 146 and 153), and the compiler enforces it:

>>> t = ast.parse("match x:\n case 1: pass")
>>> del next(n for n in ast.walk(t) if isinstance(n, ast.pattern)).end_lineno
>>> compile(t, "<s>", "exec")
TypeError: required field "end_lineno" missing from pattern

The same deletion on a stmt node compiles fine.

The wording

The first paragraph is the wording @AA-Turner approved on #136868, with the two inline suggestion blocks from that review applied — they were never committed before the author stopped working on it and the PR was closed. I took them verbatim from the review rather than paraphrasing:

Most classes in the :mod:!ast module have the :attr:lineno, :attr:col_offset,
:attr:end_lineno, and :attr:end_col_offset attributes, including all subclasses of

I checked "most" rather than trusting it: 77 of the 126 ast.AST subclasses exported by the module carry lineno, so it holds.

The second paragraph is the part #136868 did not address. It is one clause naming the two exceptions.

The :ref:above `` link uses the same target and the same phrasing the file already uses forty lines earlier, so nothing new is introduced.

Scope

Doc/library/ast.rst only — 7 insertions, 3 deletions, one file. No code, no ASDL, no test, no behaviour change, so no news entry; skip news applies the way it did on #136868, which was approved with this same single file and nothing else.

Verified on a build of main (3.16.0a0), not on a released interpreter.

• Issue: Inaccurate description of ast.AST location attributes #136640

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34746020 | 📁 Comparing 701553d against main (6893326)

  🔍 Preview build  

14 files changed · ± 14 modified

± Modified

• c-api/interp-lifecycle.html
• c-api/module.html
• c-api/stable.html
• c-api/threads.html
• deprecations/c-api-pending-removal-in-future.html
• deprecations/index.html
• howto/remote_debugging.html
• library/ast.html
• library/turtle.html
• whatsnew/3.12.html
• and 4 more...

[picnixz]

This does not help more. Even if other classes MAY have attributes, the current wording is still not wrong. I find the "most have the attributes" still useless because it does not tell me which ones really have it and I would still need hasattr checks in general.

[picnixz]

I do not like this much because this makes the implementation very CPython specific. I wonder why those nodes actually need them in the first place.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[v0ropaev]

Both taken, and the second one sent me to the record rather than to the prose.

"which ones really have it"

Fair — "most classes" leaves you exactly where you started. The eight are now named, and the claim is that nothing else has them, so no hasattr is needed. That set is exact rather than approximate:

>>> EIGHT = (ast.stmt, ast.expr, ast.excepthandler, ast.arg,
...          ast.keyword, ast.alias, ast.pattern, ast.type_param)
>>> allsub = [c for c in vars(ast).values()
...           if isinstance(c, type) and issubclass(c, ast.AST)]
>>> has   = {c for c in allsub if 'lineno' in c._attributes}
>>> under = {c for c in allsub if issubclass(c, EIGHT)}
>>> len(has), has == under
(77, True)

77 of the 126 exported subclasses, and the two sets coincide with nothing extra on either side. Those eight are also precisely the types with an attributes clause in the grammar, which the page reproduces forty lines up, so the prose and the grammar block now agree instead of the prose describing a subset of it.

"very CPython specific" — and why those nodes need them

You are right that "the compiler requires them" documents an internal rule, so that clause is gone. The text now points at something the reader already has on the page: the literalinclude of Parser/Python.asdl declares the end positions int? for six of the eight and int for pattern and type_param. That is the published abstract grammar, not compiler behaviour.

On why those two differ — it is historical, not semantic, and it was a deliberate decision. End positions arrived in 3.8 (#11605). The classes that predate them had to keep accepting nodes constructed by code written before 3.8, so the fields stayed optional. pattern had no such callers, and @isidentical said so when it was added (gh-88058, comment 1093911564):

I'm not really sure whether we should continue following this old tradition of int? end_* attributes for new sum types like pattern, considering that it was done for supporting the creation of old nodes 3.8< without any change on 3.8>=. pattern subclasses will only be created on 3.10>= so we should be safe by requiring end_* attributes.

type_param was then brought in line for the same reason in gh-106145 — filed precisely because PEP 695's nodes had inherited the old int? tradition without needing to — where @JelleZijlstra answered "Good catch, I think we should fix this."

So the optionality is a backwards-compatibility artefact of 3.8, and the old sentence presented it as a design rule with no exceptions. One clause naming the split, and the reason, seemed more useful than dropping it and leaving the grammar block to be noticed.

Doc/library/ast.rst only, now +12/−9.

[picnixz]

Please do not post LLM answers. We value human contributions, not ones simply prompted so take the time to edit answers and remove superfluous and annoying AI expressions.

As for the updated text, is there a need to name all subclasses then? and I would rather say that classes added after Python 3.8 are expected to have the end positions. And having regression test for that ruke may be relevant as well.

[v0ropaev]

Sorry about the tone of that — you're right, and I've cut it out.

Rule instead of the two exceptions, as you suggested: the classes that already existed when end positions arrived in 3.8 keep them optional so pre-3.8 code can still build those nodes, and anything added later requires them. The grammar says which is which, so the text just points there.

I added the regression test to test_asdl_parser, which already parses Python.asdl. It walks every type with an attributes clause and requires int rather than int? unless the type is one of the six that predate 3.8. Flipping pattern back to int? fails it with "pattern postdates the 3.8 addition of end positions", which is what would have caught type_param in gh-106145.

On naming the eight classes — I kept that because your first comment was that "most classes" still leaves you reaching for hasattr, and the list makes the answer exact: those eight and their subclasses are 77 of the 126 AST classes the module exports, and nothing outside them has the attributes. Happy to drop it back to a pointer at the grammar if you'd rather; it's one sentence either way.