[3.15] gh-156939: Fix two bugs in PyBytesWriter (#158404)

[vstinner]

Fix PyBytesWriter_Format() error handling: restore the writer to its previous state on error. Add an unit test injecting MemoryError.

Fix PyBytesWriter_FinishWithSize() to return an empty bytearray when use_bytearray is set and size is 0, instead of always returning an empty bytes object.

Fix test_capi.test_bytes: always check writer finish result type, to make sure that _PyBytesWriter_CreateByteArray() always create bytearray objects.

(cherry picked from commit 24ea85a)

• Issue: C API: Detect buffer overflow in PyBytesWriter #156939

[vstinner]

Without this fix, bytearray.fromhex() and bytearray.__mod__() can return a bytes object instead of a bytearray object.

Current behavior (wrong):

$ ./python
>>> bytearray.fromhex('')
b''
>>> bytearray(b'') % ()
b''

Correct behavior (with this fix):

$ ./python
>>> bytearray.fromhex('')
bytearray(b'')
>>> bytearray(b'') % ()
bytearray(b'')

[vstinner]

@cmaloney: Would you mind to review this fix?

@hugovk: It would be nice to have this fix in Python 3.15.0 final release.

[vstinner]

I was curious so I checked the the Git history to see when the bug was introduced: the bug is as old as the introduction of the PyBytesWriter C API (one year ago), commit adb4140.

[cmaloney]

Looks good to me. Feels like maybe should have been a couple different NEWS entries for the different bugs

[vstinner]

@cmaloney:

Feels like maybe should have been a couple different NEWS entries for the different bugs

Oh right, I added a second NEWS entry.

[vstinner]

test_tkinter failed on "Tests / Windows (free-threading) / Build and test (x64, switch-case)" CI job, but it passed on other Windows CI jobs. It loooks like a flaky test.

FAIL: test_boundary_values_accepted (test.test_tkinter.test_simpledialog.QueryDialogTest.test_boundary_values_accepted)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_tkinter\test_simpledialog.py", line 246, in test_boundary_values_accepted
    self.assertEqual(d.result, 10)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: None != 10

[vstinner]

@hugovk Thanks for the merge.