Skip to content

Bump to setuptools>=10.0.04 to allow fetching on shallow clone and remove fecth-depth setting from workflows - #4909

Merged
seisman merged 10 commits into
mainfrom
ci/faster-checkout
Sep 23, 2026
Merged

seisman merged 10 commits into
mainfrom
ci/faster-checkout

Conversation

@seisman

@seisman seisman commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

The actions/checkout step takes 20 s for checking out the source code (see https://github.com/GenericMappingTools/pygmt/actions/runs/35558719234/job/106207316870 for an example). It's slow because we have fetch-depth: 0 (meaning fetch all Git history), which is required to have setuptools-scm work (xref: https://setuptools-scm.readthedocs.io/latest/integrations/#troubleshooting).

I asked Claude for help, and it suggested the filter: blob:none option, which perform a blobless clone (downloads all commit history and folder structures, but skips downloading actual file contents until you need them) (xref: https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/).

With this option flag, the actions/checkout step now takes 3 seconds only (https://github.com/GenericMappingTools/pygmt/actions/runs/35599672884/job/106332531269?pr=4909).

It seems that setuptools-scm also provides such feature in pypa/setuptools-scm#1303. I haven't tried it yet, but it seems we just need to add following settings to pyproject.toml:

[tool.setuptools_scm.scm.git]
pre_parse = "fetch_on_shallow"

However, such a feature is only available after setuptools-scm v10.0.4, which was released in March, 2026. So, setting this configuration means we need to bump the minimum required version for setuptools-scm.

TODO:

  • Explore if pre_parse = "fetch_on_shallow" works
  • Decide which option we want to use
  • Apply the same changes to other workflow files

@seisman seisman added this to the 0.20.0 milestone Sep 21, 2026
@seisman seisman added the maintenance Boring but important stuff for the core devs label Sep 21, 2026
@seisman

seisman commented Sep 21, 2026

Copy link
Copy Markdown
Member Author

@weiji14 Any comments?

Comment thread .github/workflows/ci_tests.yaml Outdated
# fetch all history so that setuptools-scm works
fetch-depth: 0
fetch-depth: 0 # Fetch all history so that setuptools-scm works.
filter: blob:none # But skip file contents of old commits (~1 GB) that aren't needed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I'm aware of and have used sparse checkout for one repo, but didn't realize you can independently do a blobless clone before. I ran git-sizer locally and got these numbers:

Processing blobs: 47347
Processing trees: 24759
Processing commits: 4661
Matching commits to trees: 4661
Processing annotated tags: 2
Processing references: 150
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Commits                    |           |                                |
|   * Count                    |  4.66 k   |                                |
|   * Total size               |  4.00 MiB |                                |
| * Trees                      |           |                                |
|   * Count                    |  24.8 k   |                                |
|   * Total size               |  18.6 MiB |                                |
|   * Total tree entries       |   447 k   |                                |
| * Blobs                      |           |                                |
|   * Count                    |  47.3 k   |                                |
|   * Total size               |  1.47 GiB |                                |
| * Annotated tags             |           |                                |
|   * Count                    |     2     |                                |
| * References                 |           |                                |
|   * Count                    |   150     |                                |
|     * Branches               |    21     |                                |
|     * Tags                   |    32     |                                |
|     * Remote-tracking refs   |    96     |                                |
|     * Git stash              |     1     |                                |
|                              |           |                                |
| Biggest objects              |           |                                |
| * Commits                    |           |                                |
|   * Maximum size         [1] |  3.37 KiB |                                |
|   * Maximum parents      [2] |     2     |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [3] |   397     |                                |
| * Blobs                      |           |                                |
|   * Maximum size         [4] |  43.1 MiB | ****                           |
|                              |           |                                |
| History structure            |           |                                |
| * Maximum history depth      |  3.56 k   |                                |
| * Maximum tag depth      [5] |     1     |                                |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of directories  [6] |  7.67 k   | ***                            |
| * Maximum path depth     [6] |     5     |                                |
| * Maximum path length    [6] |    91 B   |                                |
| * Number of files        [6] |  30.3 k   |                                |
| * Total size of files    [6] |  1.11 GiB | *                              |
| * Number of symlinks     [6] |     1     |                                |
| * Number of submodules       |     0     |                                |

Surprised that our blobs add up to almost 1.5 GiB already! I'm actually wondering if we should just setup for a treeless clone though to avoid downloading ~19MiB worth of trees as well?

@seisman seisman Sep 22, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed to filter: tree:0 for a treeless clone. It works.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried the pre_parse solution on the branch scm/fetch_on_shallow (commit 1098117). The manually triggered CI runs are available at https://github.com/GenericMappingTools/pygmt/actions/runs/35717238260.

I like this solution better for its simplicity (the fetch-depth: 0 setting is no longer needed). It also works for users who make a shallow clone of the repository.

So, I plan to use the pre_parse solution if no objections.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah let's just go with that scm/fetch_on_shallow branch solution, sounds much simpler.

@seisman seisman added skip-changelog Skip adding Pull Request to changelog needs review This PR has higher priority and needs review. labels Sep 22, 2026
@seisman seisman changed the title WIP: CI: Add filter settings in the actions/checkout step to speedup checkout Bump to setuptools>=10.0.04 to allow fetching on shallow clone and remove fecth-depth setting from workflows Sep 23, 2026
@seisman
seisman marked this pull request as ready for review September 23, 2026 02:52
@seisman seisman added final review call This PR requires final review and approval from a second reviewer and removed needs review This PR has higher priority and needs review. labels Sep 23, 2026

@weiji14 weiji14 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, down from ~20s to <3s (depending on the OS) for the checkout step!

Docs CI failures should be unrelated as mentioned at #4914 (comment).

@seisman seisman removed the final review call This PR requires final review and approval from a second reviewer label Sep 23, 2026
@seisman
seisman merged commit 8f83ec9 into main Sep 23, 2026
27 of 32 checks passed
@seisman
seisman deleted the ci/faster-checkout branch September 23, 2026 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Boring but important stuff for the core devs skip-changelog Skip adding Pull Request to changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants