gh-157695: Check that C API macros start with "Py" by vstinner · Pull Request #157696 · python/cpython

vstinner · 2026-09-17T18:58:28Z

Add "make check-capi-macros". Ignore existing macros which don't start with "Py", only prevent adding new names which don't respect this convention.

Run "make check-capi-macros" in the GitHub Action "Check if generated files are up to date" job.

Issue: The Python C API should only define macros with a name starting wit Py #157695

Add "make check-capi-macros". Ignore existing macros which don't start with "Py", only prevent adding new names which don't respect this convention. Run "make check-capi-macros" in the GitHub Action "Check if generated files are up to date" job.

ZeroIntensity

I think this is a great idea. For clarity, do we already run similar checks for actual symbols? (For example, it should be impossible to add a new function to our headers that isn't prefixed with Py.)

vstinner · 2026-09-17T23:26:12Z

I think this is a great idea. For clarity, do we already run similar checks for actual symbols? (For example, it should be impossible to add a new function to our headers that isn't prefixed with Py.)

We do already have make smelly which fails if Python exports a symbol which doesn't start with Py. The check is run by the CI.

So I'm not sure that it's useful to parse the public C API to check if PyAPI_FUNC() and PyAPI_DATA() use the Py prefix.

Last year, I found exported symbols which doesn't with Py prefix: issue gh-141376. It was a bug in the make smelly check (now fixed).

ZeroIntensity · 2026-09-17T23:43:58Z

We do already have make smelly which fails if Python exports a symbol which doesn't start with Py. The check is run by the CI.

Yeah, as long as we have something, then I'm happy.

vstinner · 2026-09-17T23:46:49Z

I wrote a quick patch to test "make smelly":

diff --git a/Include/pyerrors.h b/Include/pyerrors.h
index cfabbc5fe8d..f74534da7e6 100644
--- a/Include/pyerrors.h
+++ b/Include/pyerrors.h
@@ -326,6 +326,8 @@ PyAPI_FUNC(int) PyOS_snprintf(char *str, size_t size, const char  *format, ...)
 PyAPI_FUNC(int) PyOS_vsnprintf(char *str, size_t size, const char  *format, va_list va)
                         Py_GCC_ATTRIBUTE((format(printf, 3, 0)));
 
+PyAPI_FUNC(int) test_dummy_func(void);
+
 #ifndef Py_LIMITED_API
 #  define Py_CPYTHON_ERRORS_H
 #  include "cpython/pyerrors.h"
diff --git a/Python/errors.c b/Python/errors.c
index eb148998fc4..df7d3241dcc 100644
--- a/Python/errors.c
+++ b/Python/errors.c
@@ -2091,3 +2091,9 @@ PyErr_ProgramTextObject(PyObject *filename, int lineno)
 {
     return _PyErr_ProgramDecodedTextObject(filename, lineno, NULL);
 }
+
+int
+test_dummy_func(void)
+{
+    return 4;
+}

With this change, "make smelly" fails as expected with:

Found 1 smelly symbols in total!
    - test_dummy_func from libpython3.16td.a
make: *** [Makefile:3528: smelly] Error 1

ZeroIntensity · 2026-09-17T23:48:35Z

+    print('ERROR: the Python C API defines the following macros '
+          'with a name not starting with "Py":')
+    print()


Up to you, but it'd be nice to have colorful output with _colorize (the C API docs check script does this).

I'm not sure that it's needed to add colors to this script.

vstinner · 2026-09-18T00:08:08Z

In the issue gh-149685, @encukou was concerned about new macros defined by pyconfig.h which doesn't start with Py. Well, yes, pyconfig.h is included by Python.h, so it's also part of the (public) Python C API.

I updated my PR to parse also pyconfig.h.in.

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

ZeroIntensity · 2026-09-18T01:59:07Z

Is our configure script capable of generating HAVE_ macros with a _Py/Py prefix?

vstinner · 2026-09-18T02:22:28Z

Is our configure script capable of generating HAVE_ macros with a _Py/Py prefix?

Some recently added "HAVE" macros start with a _Py_ prefix. Example from configure.ac:

AS_VAR_IF([ac_cv_have_libgcc_eh_frame_registration], [yes], [
  AC_DEFINE([_Py_HAVE_LIBGCC_EH_FRAME_REGISTRATION], [1],
    [Define to 1 if libgcc __register_frame and __deregister_frame are linkable.])
])

On the other hand, high-level functions like AC_CHECK_FUNCS always generate HAVE_xxx macro if the function xxx is available. I don't know how to add a prefix to the macro name.

Python 3.16 adds the following macros to pyconfig.h.in compared to Python 3.15:

HAVE_ACOSPI
HAVE_ASINPI
HAVE_ATAN2PI
HAVE_ATANPI
HAVE_COSPI
HAVE_CURSES_DEFINE_KEY
HAVE_CURSES_HAS_MOUSE
HAVE_CURSES_IS_KEYPAD
HAVE_CURSES_IS_LEAVEOK
HAVE_CURSES_KEYOK
HAVE_CURSES_KEY_DEFINED
HAVE_CURSES_NEW_PRESCR
HAVE_CURSES_NOFILTER
HAVE_CURSES_SCR_DUMP
HAVE_CURSES_SCR_SET
HAVE_CURSES_SLK_ATTR_OFF
HAVE_CURSES_SLK_ATTR_ON
HAVE_CURSES_SLK_ATTR_SET
HAVE_CURSES_SLK_COLOR
HAVE_CURSES_TERM_ATTRS
HAVE_CURSES_USE_SCREEN
HAVE_CURSES_USE_WINDOW
HAVE_CURSES_WATTR_GET
HAVE_CURSES_WATTR_OFF
HAVE_CURSES_WATTR_ON
HAVE_CURSES_WATTR_SET
HAVE_CURSES_WCOLOR_SET
HAVE_FLOAT16
HAVE_GETDENTS64
HAVE_GETTID
HAVE_ICONV
HAVE_ICONV_H
HAVE_IF_INDEXTONAME
HAVE_IF_NAMETOINDEX
HAVE_PIDFD_GETFD
HAVE_PIDFD_OPEN
HAVE_PIDFD_SEND_SIGNAL
HAVE_RL_CHANGE_ENVIRONMENT
HAVE_SINPI
HAVE_TANPI
_ISOC23_SOURCE

HAVE_FLOAT16 can be easily modified to add a _Py prefix.

hugovk

We have two other scripts with somewhat similar functionality to check things against ignore lists, and they both also verify the opposite: that the ignore file has no stale entries.

If a stale entry is found, the script also fails. This makes sure we keep it clean and relevant.

Let's add a similar check here, it would find the unnecessary FVC_ALT_REPR. And can also identify the generated ones from opcode_ids.h.

For reference:

cpython/Doc/tools/check-warnings.py

Lines 237 to 251 in 659e262

    
           def fail_if_improved( 
        
               files_with_expected_nits: set[str], files_with_nits: set[str] 
        
           ) -> int: 
        
               """ 
        
               We may have fixed warnings in some files so that the files are now completely clean. 
        
               Good news! Let's add them to .nitignore to prevent regression. 
        
               """ 
        
               files_with_no_nits = files_with_expected_nits - files_with_nits 
        
               if files_with_no_nits: 
        
                   print("\nCongratulations! You improved:\n") 
        
                   for filename in sorted(files_with_no_nits): 
        
                       print(filename) 
        
                   print("\nPlease remove from Doc/tools/.nitignore\n") 
        
                   return -1 
        
               return 0

cpython/Tools/build/check_warnings.py

Lines 195 to 225 in 659e262

    
           def get_unexpected_improvements( 
        
               ignore_rules: set[IgnoreRule], 
        
               files_with_warnings: dict[str, list[CompileWarning]], 
        
           ) -> int: 
        
               """ 
        
               Returns failure status if the number of warnings for a file is greater 
        
               than the expected number of warnings for that file based on the ignore 
        
               rules 
        
               """ 
        
               unexpected_improvements = [] 
        
               for rule in ignore_rules: 
        
                   if ( 
        
                       not rule.ignore_all 
        
                       and rule.file_path not in files_with_warnings.keys() 
        
                   ): 
        
                       if rule.file_path not in files_with_warnings.keys(): 
        
                           unexpected_improvements.append((rule.file_path, rule.count, 0)) 
        
                       elif len(files_with_warnings[rule.file_path]) < rule.count: 
        
                           unexpected_improvements.append(( 
        
                               rule.file_path, 
        
                               rule.count, 
        
                               len(files_with_warnings[rule.file_path]), 
        
                           )) 
        
               if unexpected_improvements: 
        
                   print("Unexpected improvements:") 
        
                   for file in unexpected_improvements: 
        
                       print(f"{file[0]} expected {file[1]} warnings, found {file[2]}") 
        
                   return 1 
        
               return 0

Fix UNDEF_REGEX macro: replace fullmatch() with match()

vstinner · 2026-09-18T12:56:53Z

Ok, I made multiple changes to address reviews:

Cleanup regexs
Move the check fro Makefile to pre-commit
Check also if the ignored list is outdated
Parse also #undef in Include/ headers

I also added a check to make sure that the ignore list is sorted. Is it overkill?

I'm aware of Python 3.15 re.prefixprefix(), but my system Python is Python 3.14 and it's convenient for me to be able to use it :-)

vstinner · 2026-09-18T13:02:42Z

HAVE_FLOAT16 can be easily modified to add a _Py prefix.

I wrote PR gh-157752 to rename HAVE_FLOAT16 macro to _Py_HAVE_FLOAT16.

vstinner · 2026-09-18T13:08:18Z

I added the following header to the ignore list:

# Old C API macros with a name which doesn't start with "Py". If a macro
# is removed from the C API, it should be removed from this list as well.
#
# Adding a new macro to this ignore list requires the approval of the C API
# Working Group. Open an issue at:
# https://github.com/capi-workgroup/decisions/issues/

"Adding a new macro to this ignore list requires approval" is a bold rule. Do we need exceptions? Or can we decide what are these exceptions later?

AC_CHECK_FUNCS(), AC_CHECK_HEADERS() and PY_CHECK_FUNC() always define HAVE_xxx macros. Should we say that new HAVE_xxx macros defined by these functions are fine?

Or should we already add new functions defining _Py_HAVE_xxx macros instead?

hugovk

I also added a check to make sure that the ignore list is sorted. Is it overkill?

I think it's worth it, it's easier to manually search a sorted list, and it's a simple check.

(prek/pre-commit have a built-in sorter lint (file-contents-sorter) but that would require moving the ignore file's header into the .py)

hugovk · 2026-09-19T14:37:37Z

+
+    # Parse header files
+    include_dir = os.path.join(SRC_DIR, 'Include')
+    files = glob.glob(os.path.join(include_dir, '*.h'))


(Forgot to post this yesterday)

Include/opcode_ids.h contains generated opcodes, we could exclude these:

Suggested change

files = glob.glob(os.path.join(include_dir, '*.h'))

files = glob.glob(os.path.join(include_dir, '*.h'))

files.remove(os.path.join(include_dir, 'opcode_ids.h'))

And then remove the 245 generated names from check_capi_macros_ignored.txt. This will make maintenance easier so we don't need to add/remove future ones.

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

vstinner · 2026-09-19T17:44:16Z

Include/opcode_ids.h contains generated opcodes, we could exclude these: (...) And then remove the 245 generated names from check_capi_macros_ignored.txt. This will make maintenance easier so we don't need to add/remove future ones.

Even if it's generated, it's part of the public Python C API and so we should care about these macros. Ignoring them only because they are generated doesn't look correct to me.

vstinner · 2026-09-19T17:44:45Z

I updated the PR to address @hugovk's latest review.

vstinner · 2026-09-19T19:31:09Z

Even if it's generated, it's part of the public Python C API and so we should care about these macros. Ignoring them only because they are generated doesn't look correct to me.

Hum. I didn't take in account that opcode.h is not included by Python.h. Maybe we should/can ignore the following Include/ header files which are not included by Python.h:

opcode.h
opcode_ids.h
errcode.h
osdefs.h
structmember.h

I don't know. I'm now confused :-)

See details in my comment on the issue.

vstinner · 2026-09-20T10:32:23Z

Ok, I changed my mind. I excluded 7 header files which are not included by Python.h:

dynamic_annotations.h
errcode.h
opcode.h
opcode_ids.hh (generated)
osdefs.h
pyexpat.h
structmember.h

They define multiple macros without Py prefix.

The following header files are still parsed by the script even if they are not included by Python.h:

datetime.h
frameobject.h
marshal.h
py_curses.h
pydtrace.h

I wrote this script to list header files not included by Python.h:

import re, glob, sys, os

INCLUDE = re.compile(r'\s*#\s*include\s+["<]([a-z0-9_./]+)[">]')

os.chdir("Include")
headers = set(glob.glob("*.h"))
headers.discard("Python.h")
os.chdir("..")
with open("Include/Python.h") as fp:
    for line in fp:
        match = INCLUDE.match(line)
        if match:
            name = match.group(1)
            headers.discard(name)
        elif line.startswith("// "):
            continue
        elif "include" in line:
            print(f"failed to parse: {line!r}")
            sys.exit(1)

print("Not included by Python.h:")
for name in sorted(headers):
    print(f"- {name}")

vstinner · 2026-09-20T10:33:24Z

I also wrote PR gh-157821 to rename HAVE_ICONV macro to _Py_HAVE_ICONV.

hugovk

Thanks! Looks good, can always exclude generated ones later if we decide to do so.

vstinner requested a review from erlend-aasland as a code owner September 17, 2026 18:58

vstinner added the topic-C-API label Sep 17, 2026

vstinner requested review from AA-Turner, JacobCoffee, emmatyping, ezio-melotti, hugovk, itamaro and webknjaz as code owners September 17, 2026 18:58

bedevere-app Bot mentioned this pull request Sep 17, 2026

The Python C API should only define macros with a name starting wit Py #157695

Open

bedevere-app Bot added the awaiting core review label Sep 17, 2026

vstinner added 3 commits September 17, 2026 21:26

Add a comment to the ignore file

0d6621d

Remove duplicates

714f1f5

Fix error message

44eb795

ZeroIntensity reviewed Sep 17, 2026

View reviewed changes

Comment thread Misc/NEWS.d/next/C_API/2026-09-17-20-50-56.gh-issue-157695.06UH5M.rst Outdated

Comment thread Tools/build/check_capi_macros.py

Remove NEWS entry

819558a

vstinner added the skip news label Sep 17, 2026

ZeroIntensity reviewed Sep 17, 2026

View reviewed changes

Parse also pyconfig.h.in

c6af62b

vstinner and others added 2 commits September 18, 2026 02:10

Update Tools/build/check_capi_macros.py

998dd34

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

Update .github/workflows/build.yml

fa62aac

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

hugovk reviewed Sep 18, 2026

View reviewed changes

Comment thread Tools/build/check_capi_macros.py

Comment thread Tools/build/check_capi_macros.py

Comment thread Makefile.pre.in Outdated

Comment thread .github/workflows/build.yml Outdated

hugovk reviewed Sep 18, 2026

View reviewed changes

Comment thread Tools/build/check_capi_macros_ignored.txt Outdated

encukou reviewed Sep 18, 2026

View reviewed changes

Comment thread Tools/build/check_capi_macros.py Outdated

vstinner added 5 commits September 18, 2026 14:23

Cleanup regexs

6188980

Move the check fro Makefile to pre-commit

8c09a6c

Check also if the ignored list is outdated

7c8c77d

Fix UNDEF_REGEX macro: replace fullmatch() with match()

Parse also #undef in Include/ headers

80a2e2f

Check that the ignore list is sorted

d2347d9

hugovk reviewed Sep 19, 2026

View reviewed changes

vstinner and others added 3 commits September 19, 2026 19:27

Update Tools/build/check_capi_macros.py

a63e018

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Accept spaces in more places in regex

86514dc

Check for duplicated entires in the ignore list

dec9c88

vstinner added 2 commits September 19, 2026 21:04

Merge branch 'main' into check_capi_macros

d326d3f

Remove HAVE_FLOAT16 macro from ignore list

d7a9559

Exclude 7 header files

dd15660

hugovk approved these changes Sep 20, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 20, 2026

	def fail_if_improved(
	files_with_expected_nits: set[str], files_with_nits: set[str]
	) -> int:
	"""
	We may have fixed warnings in some files so that the files are now completely clean.
	Good news! Let's add them to .nitignore to prevent regression.
	"""
	files_with_no_nits = files_with_expected_nits - files_with_nits
	if files_with_no_nits:
	print("\nCongratulations! You improved:\n")
	for filename in sorted(files_with_no_nits):
	print(filename)
	print("\nPlease remove from Doc/tools/.nitignore\n")
	return -1
	return 0

	def get_unexpected_improvements(
	ignore_rules: set[IgnoreRule],
	files_with_warnings: dict[str, list[CompileWarning]],
	) -> int:
	"""
	Returns failure status if the number of warnings for a file is greater
	than the expected number of warnings for that file based on the ignore
	rules
	"""
	unexpected_improvements = []
	for rule in ignore_rules:
	if (
	not rule.ignore_all
	and rule.file_path not in files_with_warnings.keys()
	):
	if rule.file_path not in files_with_warnings.keys():
	unexpected_improvements.append((rule.file_path, rule.count, 0))
	elif len(files_with_warnings[rule.file_path]) < rule.count:
	unexpected_improvements.append((
	rule.file_path,
	rule.count,
	len(files_with_warnings[rule.file_path]),
	))

	if unexpected_improvements:
	print("Unexpected improvements:")
	for file in unexpected_improvements:
	print(f"{file[0]} expected {file[1]} warnings, found {file[2]}")
	return 1

	return 0

	files = glob.glob(os.path.join(include_dir, '*.h'))
	files = glob.glob(os.path.join(include_dir, '*.h'))
	files.remove(os.path.join(include_dir, 'opcode_ids.h'))

Uh oh!

Conversation

vstinner commented Sep 17, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vstinner commented Sep 17, 2026

Uh oh!

ZeroIntensity commented Sep 17, 2026

Uh oh!

vstinner commented Sep 17, 2026

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner commented Sep 18, 2026

Uh oh!

ZeroIntensity commented Sep 18, 2026

Uh oh!

vstinner commented Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vstinner commented Sep 18, 2026

Uh oh!

vstinner commented Sep 18, 2026

Uh oh!

vstinner commented Sep 18, 2026

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vstinner commented Sep 19, 2026

Uh oh!

vstinner commented Sep 19, 2026

Uh oh!

vstinner commented Sep 19, 2026

Uh oh!

vstinner commented Sep 20, 2026

Uh oh!

vstinner commented Sep 20, 2026

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vstinner commented Sep 17, 2026 •

edited by bedevere-app Bot

Loading