From 3e35db7df2cfce72f9bf2e0320c1d036a42768e2 Mon Sep 17 00:00:00 2001 From: Dmitry Voropaev Date: Thu, 24 Sep 2026 01:14:42 +0300 Subject: [PATCH 1/3] gh-136640: Correct the description of AST location attributes Eight of the types generated from ``Parser/Python.asdl`` carry the ``lineno``/``col_offset``/``end_lineno``/``end_col_offset`` attributes -- ``stmt``, ``expr``, ``excepthandler``, ``arg``, ``keyword``, ``alias``, ``pattern`` and ``type_param`` -- but the documentation named only ``ast.expr`` and ``ast.stmt``. The same block also stated that the end positions are always optional. That holds for the six types that declare them as ``int?`` in the grammar; ``pattern`` and ``type_param`` declare them as ``int``, and compiling a tree whose ``pattern`` or ``type_param`` node is missing ``end_lineno`` raises ``TypeError``. The first paragraph follows the wording approved in GH-136868 and the two review suggestions on it, except that it keeps the original "Instances of": the classes themselves expose nothing, so ``hasattr(ast.Name, 'lineno')`` is false. The cross-reference uses ``:ref:``, matching the rest of the file. --- Doc/library/ast.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index e30dadc9733d37b..85a4d1921e0e527 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -90,9 +90,11 @@ Node classes end_lineno end_col_offset - Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have + Instances of most classes in the :mod:`!ast` module have the :attr:`lineno`, :attr:`col_offset`, :attr:`end_lineno`, and - :attr:`end_col_offset` attributes. The :attr:`lineno` and :attr:`end_lineno` + :attr:`end_col_offset` attributes, including all subclasses of + :class:`ast.expr`, :class:`ast.stmt` and others (see the abstract grammar + :ref:`above `). The :attr:`lineno` and :attr:`end_lineno` are the first and last line numbers of source text span (1-indexed so the first line is line 1) and the :attr:`col_offset` and :attr:`end_col_offset` are the corresponding UTF-8 byte offsets of the first and last tokens that @@ -100,7 +102,9 @@ Node classes UTF-8 internally. Note that the end positions are not required by the compiler and are - therefore optional. The end offset is *after* the last symbol, for example + therefore optional, except for :class:`ast.pattern` and + :class:`ast.type_param` nodes, for which the compiler requires them. + The end offset is *after* the last symbol, for example one can get the source segment of a one-line expression node using ``source_line[node.col_offset : node.end_col_offset]``. From 380832d66855e99877893b685b1c060a5580936e Mon Sep 17 00:00:00 2001 From: Dmitry Voropaev Date: Thu, 24 Sep 2026 17:44:56 +0300 Subject: [PATCH 2/3] gh-136640: name the classes that carry location attributes Review asked for two things the previous wording did not give. "Most classes have the attributes" does not tell a reader which ones, so a hasattr check is still needed. The eight classes are now named. The set is exact: of the 126 ast.AST subclasses the module exports, the 77 that carry lineno are precisely the subclasses of those eight, with no exceptions in either direction. The end-position clause no longer asserts a compiler rule. The abstract grammar reproduced on the same page already declares them int? for six classes and int for pattern and type_param, so the text points at that instead. The reason those two differ is historical rather than semantic: end positions arrived in 3.8 and the older classes had to keep accepting nodes built without them, while pattern (3.10) and type_param (3.12) had no such callers to support. --- Doc/library/ast.rst | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 85a4d1921e0e527..753bccf3afbd21a 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -90,21 +90,24 @@ Node classes end_lineno end_col_offset - Instances of most classes in the :mod:`!ast` module have the - :attr:`lineno`, :attr:`col_offset`, :attr:`end_lineno`, and - :attr:`end_col_offset` attributes, including all subclasses of - :class:`ast.expr`, :class:`ast.stmt` and others (see the abstract grammar - :ref:`above `). The :attr:`lineno` and :attr:`end_lineno` + Instances of :class:`ast.stmt`, :class:`ast.expr`, + :class:`ast.excepthandler`, :class:`ast.arg`, :class:`ast.keyword`, + :class:`ast.alias`, :class:`ast.pattern` and :class:`ast.type_param` + subclasses have the :attr:`lineno`, :attr:`col_offset`, + :attr:`end_lineno`, and :attr:`end_col_offset` attributes. These are the + classes carrying an ``attributes`` clause in the abstract grammar + :ref:`above `; no other class has them. + The :attr:`lineno` and :attr:`end_lineno` are the first and last line numbers of source text span (1-indexed so the first line is line 1) and the :attr:`col_offset` and :attr:`end_col_offset` are the corresponding UTF-8 byte offsets of the first and last tokens that generated the node. The UTF-8 offset is recorded because the parser uses UTF-8 internally. - Note that the end positions are not required by the compiler and are - therefore optional, except for :class:`ast.pattern` and - :class:`ast.type_param` nodes, for which the compiler requires them. - The end offset is *after* the last symbol, for example + The abstract grammar also says which of the four may be omitted: the end + positions are declared ``int?`` for the classes that predate them, and + ``int`` for :class:`ast.pattern` and :class:`ast.type_param`, which were + added afterwards. The end offset is *after* the last symbol, for example one can get the source segment of a one-line expression node using ``source_line[node.col_offset : node.end_col_offset]``. From 701553d92839da2ba80e3e6951276009be4838eb Mon Sep 17 00:00:00 2001 From: Dmitry Voropaev Date: Thu, 24 Sep 2026 22:50:43 +0300 Subject: [PATCH 3/3] gh-136640: state the end-position rule and test it Review preferred the rule over the two exceptions: the classes that existed when end positions arrived in 3.8 keep them optional so that pre-3.8 code can still build those nodes, and anything added later requires them. test_asdl_parser checks that rule against Python.asdl, so a type added with int? end positions fails instead of being noticed years later, as type_param was in gh-106145. --- Doc/library/ast.rst | 10 ++++++---- Lib/test/test_asdl_parser.py | 28 ++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 753bccf3afbd21a..32cfe25e3a0ea34 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -104,10 +104,12 @@ Node classes generated the node. The UTF-8 offset is recorded because the parser uses UTF-8 internally. - The abstract grammar also says which of the four may be omitted: the end - positions are declared ``int?`` for the classes that predate them, and - ``int`` for :class:`ast.pattern` and :class:`ast.type_param`, which were - added afterwards. The end offset is *after* the last symbol, for example + The end positions are optional on the classes that already existed when + they were added in Python 3.8, so that code written before then can still + build those nodes; a class added afterwards requires them. The abstract + grammar spells this out, declaring the two fields ``int?`` in the first + case and ``int`` in the second. The end offset is *after* the last + symbol, for example one can get the source segment of a one-line expression node using ``source_line[node.col_offset : node.end_col_offset]``. diff --git a/Lib/test/test_asdl_parser.py b/Lib/test/test_asdl_parser.py index b9df6568123ea9e..e04fc0678bab2c7 100644 --- a/Lib/test/test_asdl_parser.py +++ b/Lib/test/test_asdl_parser.py @@ -74,6 +74,34 @@ def test_attributes(self): self.assertEqual(repr(stmt.attributes[2]), 'Field(int, end_lineno, quantifiers=[OPTIONAL])') self.assertEqual(repr(stmt.attributes[3]), 'Field(int, end_col_offset, quantifiers=[OPTIONAL])') + # Types that already existed when end positions were added in 3.8 keep + # them optional, so that code written before 3.8 can still build their + # nodes. A type introduced afterwards has no such callers and requires + # them: pattern was given required end positions when it was added + # (gh-88058) and type_param was corrected to match (gh-106145). + OPTIONAL_END_POSITIONS = frozenset({ + 'stmt', 'expr', 'excepthandler', 'arg', 'keyword', 'alias', + }) + + def test_end_positions_are_required_for_new_types(self): + for name, type_ in self.types.items(): + attributes = getattr(type_, 'attributes', None) + if not attributes: + continue + ends = [f for f in attributes + if f.name in ('end_lineno', 'end_col_offset')] + with self.subTest(type=name): + self.assertEqual(len(ends), 2) + optional = [f.opt for f in ends] + if name in self.OPTIONAL_END_POSITIONS: + self.assertEqual(optional, [True, True]) + else: + self.assertEqual( + optional, [False, False], + f'{name} postdates the 3.8 addition of end positions, ' + f'so end_lineno and end_col_offset should be declared ' + f'"int" rather than "int?"') + def test_constructor_fields(self): ehandler = self.types['excepthandler'] self.assertEqual(len(ehandler.types), 1)