Infra: Add custom lexers for PEP 823 + 824

[cdce8p]

Even minimal syntax highlighting can improve the readability of code snippets and examples in a PEPs. For syntax changes the default Python lexer provided by pygments can sometimes fail to parse the code having to fall back to pure text.

This PR adds new custom lexers which extends the default one to support the examples in PEP 823 and 824. The sphinx directives for these will be .. code-block:: py823 and py824 respectively.

• PEP 823: None-aware access operators #4798
• PEP 824: None-coalescing operators #4799

	Before	After
823
824

[cdce8p]

@hugovk Would you mind taking a look at this change from an infra perspective? Is it fine to add the custom lexers within pep_sphinx_extensions/lexers or should they be moved somewhere else? As a followup it might be great to add the ASDL lexer from the CPython repo here as well. Just wasn't sure about the licensing which is why I didn't include it. https://github.com/python/cpython/blob/main/Doc/tools/extensions/lexers/asdl_lexer.py

Note: The custom lexers themselves don't attempt to be feature complete, they work well enough to enable syntax highlighting in the PEPs though.

[hugovk]

Hi! I'll come back to this, but a quick note: might be some overlap with #5084? cc @ZeroIntensity

[ZeroIntensity]

Yeah, the new keywords look like they can be covered by #5084.

I think it would also be a good idea to make this generic; rather than making a custom lexer for every new syntax PEP, we should add a way to customize code blocks if necessary (my PR lets you do .. code-block:: python+soft-keywords:my_keyword, for example).

[cdce8p]

While I do like a good general solution, I'm not sure it's the best option here. Syntax changes in itself are difficult to generalize. Sure there are different categories, like adding a new (soft-)keyword or operator, but how would you make sure it works in every case? And if it doesn't work and requires changes, you would either have to go back and check every PEP that nothing broke or add a small test framework as well.

Writing a custom lexer is fairly strait forward and only a few lines of code. This will be even easier once there are a few examples you just need to copy and modify. It's also important to keep in mind that they don't have to be perfect. It's enough if they work for the examples in the PEP and nothing more.