Repository navigation
Bump to setuptools>=10.0.04 to allow fetching on shallow clone and remove fecth-depth setting from workflows - #4909
Conversation
|
@weiji14 Any comments? |
| # fetch all history so that setuptools-scm works | ||
| fetch-depth: 0 | ||
| fetch-depth: 0 # Fetch all history so that setuptools-scm works. | ||
| filter: blob:none # But skip file contents of old commits (~1 GB) that aren't needed. |
There was a problem hiding this comment.
Interesting, I'm aware of and have used sparse checkout for one repo, but didn't realize you can independently do a blobless clone before. I ran git-sizer locally and got these numbers:
Processing blobs: 47347
Processing trees: 24759
Processing commits: 4661
Matching commits to trees: 4661
Processing annotated tags: 2
Processing references: 150
| Name | Value | Level of concern |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size | | |
| * Commits | | |
| * Count | 4.66 k | |
| * Total size | 4.00 MiB | |
| * Trees | | |
| * Count | 24.8 k | |
| * Total size | 18.6 MiB | |
| * Total tree entries | 447 k | |
| * Blobs | | |
| * Count | 47.3 k | |
| * Total size | 1.47 GiB | |
| * Annotated tags | | |
| * Count | 2 | |
| * References | | |
| * Count | 150 | |
| * Branches | 21 | |
| * Tags | 32 | |
| * Remote-tracking refs | 96 | |
| * Git stash | 1 | |
| | | |
| Biggest objects | | |
| * Commits | | |
| * Maximum size [1] | 3.37 KiB | |
| * Maximum parents [2] | 2 | |
| * Trees | | |
| * Maximum entries [3] | 397 | |
| * Blobs | | |
| * Maximum size [4] | 43.1 MiB | **** |
| | | |
| History structure | | |
| * Maximum history depth | 3.56 k | |
| * Maximum tag depth [5] | 1 | |
| | | |
| Biggest checkouts | | |
| * Number of directories [6] | 7.67 k | *** |
| * Maximum path depth [6] | 5 | |
| * Maximum path length [6] | 91 B | |
| * Number of files [6] | 30.3 k | |
| * Total size of files [6] | 1.11 GiB | * |
| * Number of symlinks [6] | 1 | |
| * Number of submodules | 0 | |
Surprised that our blobs add up to almost 1.5 GiB already! I'm actually wondering if we should just setup for a treeless clone though to avoid downloading ~19MiB worth of trees as well?
There was a problem hiding this comment.
I've changed to filter: tree:0 for a treeless clone. It works.
There was a problem hiding this comment.
I've tried the pre_parse solution on the branch scm/fetch_on_shallow (commit 1098117). The manually triggered CI runs are available at https://github.com/GenericMappingTools/pygmt/actions/runs/35717238260.
I like this solution better for its simplicity (the fetch-depth: 0 setting is no longer needed). It also works for users who make a shallow clone of the repository.
So, I plan to use the pre_parse solution if no objections.
There was a problem hiding this comment.
Yeah let's just go with that scm/fetch_on_shallow branch solution, sounds much simpler.
weiji14
left a comment
There was a problem hiding this comment.
Nice, down from ~20s to <3s (depending on the OS) for the checkout step!
Docs CI failures should be unrelated as mentioned at #4914 (comment).
Theactions/checkoutstep takes 20 s for checking out the source code (see https://github.com/GenericMappingTools/pygmt/actions/runs/35558719234/job/106207316870 for an example). It's slow because we havefetch-depth: 0(meaning fetch all Git history), which is required to have setuptools-scm work (xref: https://setuptools-scm.readthedocs.io/latest/integrations/#troubleshooting).I asked Claude for help, and it suggested thefilter: blob:noneoption, which perform a blobless clone (downloads all commit history and folder structures, but skips downloading actual file contents until you need them) (xref: https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/).With this option flag, theactions/checkoutstep now takes 3 seconds only (https://github.com/GenericMappingTools/pygmt/actions/runs/35599672884/job/106332531269?pr=4909).It seems that setuptools-scm also provides such feature in pypa/setuptools-scm#1303. I haven't tried it yet, but it seems we just need to add following settings to
pyproject.toml:However, such a feature is only available after setuptools-scm v10.0.4, which was released in March, 2026. So, setting this configuration means we need to bump the minimum required version for setuptools-scm.
TODO:
pre_parse = "fetch_on_shallow"works