gh-88661: Derive the gzip FNAME field like gunzip does

[v0ropaev]

Closes #88661.

An archive created as spam.tgz records spam.tgz as the name to decompress to, so gunzip on it produces a file called spam.tgz. Measured on main, reading FLG and the NUL-terminated FNAME straight out of the RFC 1952 header:

	main	this branch
tarfile.open("test.tgz", "w:gz")	b'test.tgz'	b'test.tar'
tarfile.open("s.tgz", "w|gz")	b's.tgz'	b's.tar'
tarfile.open("TEST.TGZ", "w:gz")	b'TEST.TGZ'	b'TEST.tar'
tarfile.open("test.tAr.Gz", "w:gz")	b'test.tAr.Gz'	b'test.tAr'
gzip.open("data.tgz", "wb")	b'data.tgz'	b'data.tar'
gzip.open("DATA.GZ", "wb")	b'DATA.GZ'	b'DATA'

Two independent code paths carry the same defect: GzipFile._write_gzip_header (used by gzip.open and by tarfile's w:gz) and _Stream._init_write_gz (w|gz). Both are fixed identically.

Following @serhiy-storchaka's direction

The command-line tool gunzip converts .tgz to .tar. It also ignores case, so it removes .GZ, .Gz, and .gZ, and converts .TgZ, .tGz, etc to .tar. I think this is a strong argument for option 1.

So the suffix is matched case-insensitively, and .tgz becomes .tar rather than being stripped. Matching the case-insensitivity is a wider change than the issue title suggests — gzip.open("DATA.GZ", "wb") now records DATA instead of DATA.GZ — so I am calling that out rather than letting it ride along.

The stem keeps its case and the suffix becomes lowercase .tar, which is what GNU make_ofname() does:

/* Make a special case for .tgz and .taz: */
strlwr(suff);
if (strequ(suff, ".tgz") || strequ(suff, ".taz")) {
    strcpy(suff, ".tar");
} else {
    *suff = '\0'; /* strip the z suffix */
}

.taz is included, which the issue does not mention. It sits in the same if as .tgz in make_ofname(), so implementing only half of the criterion Serhiy cited seemed worse than implementing it. The stdlib already treats the two alike — mimetypes.py:462 has '.tgz': '.tar.gz', '.taz': '.tar.gz'. .tz is not included: GNU handles it through the generic known_suffixes[] stripping, not the .tar special case.

The logic is duplicated rather than shared. _Stream.__init__ imports only zlib; gzip is pulled in lazily and only by TarFile.gzopen, so a shared helper in gzip.py would add a gzip import to the w|gz path for four lines. The duplication predates this change; both copies now say the same thing.

The comparison slices the original and lowercases only the tail rather than using fname.lower().endswith(...), because str.lower() can change length ('İ') and the slice index would then be wrong.

Not in scope, but noticed

python -m gzip -d (Lib/gzip.py:736) does if arg[-3:] != ".gz": sys.exit(...), so the CLI that describes itself as "act like gunzip" is still case-sensitive and refuses .tgz outright. That is a different behaviour — it rejects the file rather than misnaming a header field — so I left it alone. Happy to do it as a follow-up if you want the module internally consistent.

Verification

Run on a build of main (3.16.0a0):

./python.exe -m test test_gzip test_tarfile   →  run=863, skipped=5, SUCCESS

Reverting only Lib/gzip.py and Lib/tarfile.py to main while keeping the tests: run=863 failures=26, from two test methods (test_fname, test_metadata_name_suffix) whose subtests cover each suffix and case combination.

A mutation lowercasing the stem as well as the suffix — fname[:-4].lower() + b'.tar' — fails 2 tests, so the case-preserving half is pinned rather than incidental.

main only; I am not proposing a backport, since the change alters bytes in generated files. Misc/NEWS.d/next/Library/ entry added, plus a note under Porting to Python 3.16 rather than under Improved Modules, since the section is explicitly for "changes that may require changes to your code" and byte-for-byte comparisons of generated archives will shift.

Thanks to @miikka for laying out the options in the thread, including the case that GzipFile cannot know whether a .tgz really is a tarball — which is why this follows the file name like gunzip does rather than inspecting content.

• Issue: wrong FNAME in tarfile if tgz extension is used #88661

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34806519 | 📁 Comparing 171996d against main (69f98a5)

  🔍 Preview build  

13 files changed · ± 13 modified

± Modified

• c-api/init_config.html
• c-api/interp-lifecycle.html
• c-api/slice.html
• howto/free-threading-python.html
• library/argparse.html
• library/ctypes.html
• library/dis.html
• library/optparse.html
• library/turtle.html
• using/configure.html
• and 3 more...

[serhiy-storchaka]

.taz is for .tar.Z, but Python's gzip module does not support the .Z compression.

The reading side is now inconsistent with what we write:

$ python -m gzip -d spam.tgz
filename doesn't end in .gz: 'spam.tgz'

main() tests arg[-3:] != ".gz", so it rejects both spam.tgz and spam.GZ, which gunzip accepts. Could you fix it here?

[v0ropaev]

Both done, thank you — the .taz point is one I had backwards.

Dropped it. I had taken it from GNU make_ofname(), where .tgz and .taz sit in the same branch, without noticing that the branch exists because gunzip also reads .Z. This module does not, so a .taz here is not something it could have written or can read, and mapping it to .tar was wrong. Only .gz and .tgz now.

The command line uses the same logic. The suffix handling moved into one small helper that both _write_gzip_header and main() call, so:

python -m gzip -d spam.tgz   ->  spam.tar
python -m gzip -d spam.GZ    ->  spam
python -m gzip -d spam.taz   ->  filename doesn't end in .gz or .tgz: 'spam.taz'

tarfile._Stream._init_write_gz keeps its own copy of the two lines rather than importing gzip, since gzip is only pulled in lazily by TarFile.gzopen and the w|gz path does not otherwise need it.

One thing to flag rather than bury: test_decompress_infile_outfile_error asserted the old message text, so I changed the assertion. The message genuinely changed, it is not a test bent to fit — the same test now also covers .taz being refused, and there is a new one running .tgz, .TGZ and .GZ through the command line end to end.

./python.exe -m test test_gzip test_tarfile is 864 passing.

[v0ropaev]

The iOS job caught something my machine could not: the three names in the new command-line test differed only in case, so on a case-insensitive filesystem the second write landed on the first file and it failed with FileExistsError on testgzip.TGZ. They have distinct stems now. macOS and Linux were green, so this was only visible on the simulator.

[serhiy-storchaka]

Drop the mention of .taz -- it is irrelevant here.

Docstring are plain text; do not use the RST syntax.

[v0ropaev]

Done, plain text now and the .taz sentence is gone.

[serhiy-storchaka]

You can use the test.support.subTests() decorator.

[v0ropaev]

Switched all three loops over to it. The metadata one lost its two explicit upper-case rows in the process, it now runs every suffix against both the plain name and the upper-cased one, which covers a bit more than before.