CI: Try to fix "Remote download is currently deactivated" failures by Esteban82 · Pull Request #9243 · GenericMappingTools/gmt

Esteban82 · 2026-10-02T12:59:48Z

This is an attempt to fix the CI runs where ~70 tests that use remote data fail at once with Remote download is currently deactivated. It may not fully solve it, so please check the workflow runs of this PR (the "GMT CI Caches" run and the Tests jobs) to confirm it works.

Related: #9179, #9197, #9200, #9223.

Written with Claude Code using Claude Sonnet 5, Opus 5, Sonnet 5.5 and Opus 5.5 (max effort);
the analysis and the final diff were reviewed independently with Claude Opus.

… faking them The gmt-cache artifact carries gmt_data_server.txt and gmt_hash_server.txt only in server/, while GMT reads them from the root of ~/.gmt/static. Without them the test run has to fetch the index live; whenever that fetch fails, which happens on some runs and not others, auto_download is latched off and every test that needs remote data fails. The blind touch from #9179 does not help: it creates zero-byte indexes, which GMT rejects and deletes. Copy them up from server/ and fail loudly if neither copy is usable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Put the two index files at the root of the artifact, where GMT reads them, so the test jobs no longer depend on fetching them live. Fail the job instead of uploading a partial cache: retry while gmt reports missing files (GMT <= 6.7 still exits 0 on a failed download), check the final layout, and refuse to upload nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

seisman · 2026-10-02T13:08:48Z

+          # The tests read the two index files from the root of ~/.gmt/static
+          cp ~/.gmt/static/server/gmt_data_server.txt ~/.gmt/static/server/gmt_hash_server.txt ~/.gmt/static/


From the output of this workflow run (https://github.com/GenericMappingTools/gmt/actions/runs/36334427941/job/108662441207), I don't think ~/.gmt/static/server/gmt_data_server.txt and ~/.gmt/static/server/gmt_hash_server.txt exist.

I think it should be:

mv ~/.gmt/gmt_data_server.txt ~/.gmt/gmt_hash_server.txt ~/.gmt/static/

I don't see any errors in your link. And I think that the files exist in both places: ~/.gmt/static/server/
(step 3, line 486 of the run you linked) and ~/.gmt/.

I don't see any errors in your link. And I think that the files exist in both places: ~/.gmt/static/server/ (step 3, line 486 of the run you linked) and ~/.gmt/.

The file list is displayed by the ls -lR ~/.gmt command at the end of this step. They exist in both places due to the mv commands above.

Actually, it's likely the workaround is no longer needed (at least with GMT 6.7.0). After setting GMT_DATA_SERVER to static, the files are downloaded to the expected directories:

$ gmt --version 6.7.0 $ gmt set GMT_DATA_SERVER static $ gmt which -Ga @earth_relief_01d_g gmtwhich [NOTICE]: Remote data courtesy of GMT data server static [http://static.generic-mapping-tools.org] gmtwhich [NOTICE]: SRTM15 Earth Relief at 1x1 arc degrees reduced by Gaussian Cartesian filtering (111.2 km fullwidth) [Tozer et al., 2019]. gmtwhich [NOTICE]: -> Download grid file [114K]: earth_relief_01d_g.grd /Users/seisman/.gmt/static/server/earth/earth_relief/earth_relief_01d_g.grd $ gmt which -Ga @needle.jpg /Users/seisman/.gmt/static/cache/needle.jpg

Could you please remove the old mv and the new cp commands, then we can know if it alreadys works without the workaround.

Ok, I made that change (in d1abd98).

Claude pointed out that gmt get -Dcache still needs gmt_hash_server.txt in server/ (gmtget.c:311), so I added a single cp for that (f8eb213).

joa-quim · 2026-10-02T13:11:24Z

Isn't this a duplicate of #9200 ?

Esteban82 · 2026-10-02T13:27:55Z

Isn't this a duplicate of #9200 ?

It tries to solve the same issue but it is not the same solution.

joa-quim · 2026-10-02T16:20:50Z

We should rerun the CIs a couple of times. In #9200 it also worked fine the first time, but than the failures came back.

As suggested in review: with GMT 6.7.0 and GMT_DATA_SERVER set to the name "static", the data and cache files are written under ~/.gmt/static directly, so the mv commands and the copy of the index files should not be needed. Use the name instead of the URL, since a URL is not treated as a ghost server and puts everything under ~/.gmt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gmt get -Dcache reads <USERDIR>/server/gmt_hash_server.txt (gmtget.c), but GMT writes the hash index to <USERDIR>/, so without this copy it fails with "Unable to access or read" even with the correct ~/.gmt/static layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Esteban82 · 2026-10-02T18:40:45Z

The last 2 commits failed, right? Should I revert them?

Reverts d1abd98 and f8eb213. With the name "static" the "Cache GMT data" job downloads nothing ("Remote download is currently deactivated" from the first dataset), as it did before #9223 switched to the URL. Restore the URL and the mv workaround. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

seisman · 2026-10-03T06:02:40Z

The last 2 commits failed, right? Should I revert them?

The failures are due to temporary internet connection issues, not because changes in the last two commits.

Reverts 72fc3f7. Review feedback is that the failed run was a temporary connection problem and not an effect of d1abd98 and f8eb213, so try them again. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

joa-quim · 2026-10-03T21:54:38Z

+          # gmt get -Dcache reads the hash index from server/, not from where GMT writes it
+          cp ~/.gmt/static/gmt_hash_server.txt ~/.gmt/static/server/
          download gmt get -Dcache || exit 1
-          # Workaroud for https://github.com/GenericMappingTools/gmt/issues/8437.


Isn't it bad to remove a fix for an old issue?

I agree that we should remove the old workaround.

seisman · 2026-10-04T08:06:52Z

+            return 1
+          }
+          download gmt which -Ga $data || exit 1
+          # gmt get -Dcache reads the hash index from server/, not from where GMT writes it


gmt get -Dcache reads the hash index from server/, not from where GMT writes it

I think this is a bug. Is it already fixed in the master branch or not?

I think that it is not fixed in master. I will take a look.

This was fixed by #9246. This can be removed once the conda-forge gmt includes this fix (6.7.0 does not).

seisman · 2026-10-04T08:07:06Z

+          # gmt get -Dcache reads the hash index from server/, not from where GMT writes it
+          cp ~/.gmt/static/gmt_hash_server.txt ~/.gmt/static/server/
          download gmt get -Dcache || exit 1
-          # Workaroud for https://github.com/GenericMappingTools/gmt/issues/8437.


I agree that we should remove the old workaround.

seisman · 2026-10-04T08:08:25Z

+          # Older gmt-cache artifacts carry the two index files only in server/
+          for f in gmt_data_server.txt gmt_hash_server.txt; do
+            [ -s ~/.gmt/static/$f ] || cp ~/.gmt/static/server/$f ~/.gmt/static/ || true
+            [ -s ~/.gmt/static/$f ] || { echo "::error::$f is missing or empty in the gmt-cache artifact"; exit 1; }
+            touch ~/.gmt/static/$f
+          done


I don't think this is necessary. The gmt-cache artifacts is refreshed every week and can also be refreshed manually. So we don't need any workarounds for the old artifacts.

Ok, so I should delete these lines, right?
The same workaround is also in tests.yml and build.yml. Should I delete it there too?

seisman · 2026-10-06T16:11:27Z

      - name: Download cached GMT remote data from GitHub Artifacts
        run: |
          gh run download -n gmt-cache -D ~/.gmt/static/
-          touch ~/.gmt/static/gmt_data_server.txt ~/.gmt/static/gmt_hash_server.txt


I think this line is still required. The gmt-cache artifact will expire in seven days. So that workflows will see these files older than 24 hours and will try to refresh them, which may cause failures.

Esteban82 and others added 2 commits October 2, 2026 09:36

Esteban82 added the maintenance Boring but important stuff for the core devs label Oct 2, 2026

Esteban82 requested review from joa-quim and seisman October 2, 2026 12:59

seisman reviewed Oct 2, 2026

View reviewed changes

Esteban82 and others added 2 commits October 2, 2026 13:31

Re-apply the experiment with GMT_DATA_SERVER=static in the cache job

3a19126

Reverts 72fc3f7. Review feedback is that the failed run was a temporary connection problem and not an effect of d1abd98 and f8eb213, so try them again. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

joa-quim reviewed Oct 3, 2026

View reviewed changes

seisman reviewed Oct 4, 2026

View reviewed changes

Esteban82 added 2 commits October 5, 2026 08:59

Remove old workaround.

de9b4b4

Merge branch 'master' into fix-ci-cache-v2

7d208dd

Esteban82 mentioned this pull request Oct 6, 2026

gmtget: Fix -Dcache not finding gmt_hash_server.txt #9246

Merged

Merge branch 'master' into fix-ci-cache-v2

e839623

seisman reviewed Oct 6, 2026

View reviewed changes

Restore touch command

7b77ae5

		# The tests read the two index files from the root of ~/.gmt/static
		cp ~/.gmt/static/server/gmt_data_server.txt ~/.gmt/static/server/gmt_hash_server.txt ~/.gmt/static/

Conversation

Esteban82 commented Oct 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Esteban82 Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joa-quim commented Oct 2, 2026

Uh oh!

Esteban82 commented Oct 2, 2026

Uh oh!

joa-quim commented Oct 2, 2026

Uh oh!

Esteban82 commented Oct 2, 2026

Uh oh!

seisman commented Oct 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Esteban82 Oct 2, 2026 •

edited

Loading