C-Path GitLab returns 403 to GitHub Actions, silently dropping the whole c-path registry

[matentzn]

The Rebuild registry files workflow cannot fetch https://gitlab.c-path.org/c-pathontology/mapping-commons/-/raw/main/registry.yml. The host answers HTTP 403 to the runner, so scripts/cli.py logs the error, continues, and every C-Path mapping set disappears from data/mapping-specifications.json. The same URL returns 200 from a laptop, which is why this never shows up locally.

The 403 is not GitLab's. server: awselb/2.0 on the response means it comes from the AWS load balancer / WAF in front of their GitLab, before the request ever reaches the application — so no GitLab token or deploy key will fix it. The block is tied to the source IP: identical request, identical User-Agent, 200 from a residential IP and 403 from an Azure-hosted GitHub runner. The most likely culprit is an AWS managed rule group (anonymous-IP / IP-reputation) that treats hosting-provider ranges as hostile.

This has been happening since at least 2026-05-01 — every scheduled monthly run (May 1, Jun 1, Jul 1, Aug 1) failed on exactly this line. The User-Agent override added in fef1a01 (_session.headers["User-Agent"]) was an attempt to fix it and did not help, because the block is not UA-based.

Note that seven of the eight C-Path sets are affected twice over: their mapping_set_ids are https://w3id.org/cpont/mappings/*, which 307-redirect straight back to gitlab.c-path.org. Even with registry.yml mirrored elsewhere, the TSV fetches would still be blocked. Only mondo_hasdbxref_hp.sssom.tsv (on purl.obolibrary.org) survives.

Two ways out, both requiring someone at C-Path:

1. WAF allow rule. Cleanest is a custom header: the workflow sends a shared secret header from GitHub Actions secrets, an AWS WAF rule allows any request carrying it, bypassing the managed rule group. An IP allowlist built from https://api.github.com/meta (actions) also works but means thousands of CIDRs and constant churn.
2. Mirror. Publish registry.yml and the seven TSVs to a location that isn't behind the WAF, and either point mapping-server.yml at the mirror or add mirror_from: to each entry in the C-Path registry.

First step is to ask C-Path infra to pull the AWS WAF logs for those blocked requests — that gives the exact rule name and settles which fix applies.

Examples

$ curl -sI https://gitlab.c-path.org/c-pathontology/mapping-commons/-/raw/main/registry.yml
HTTP/2 200          # from a laptop

From the runner (Rebuild registry files, run 30679487837, 2026-08-01):

Error: Error fetching registry file: https://gitlab.c-path.org/c-pathontology/mapping-commons/-/raw/main/registry.yml (HTTP 403)
error: recipe `data` failed on line 17 with exit code 1

Proof the 403 is served by the load balancer, not GitLab (reproducible locally by stripping the User-Agent, which trips a different WAF rule):

$ curl -sD- -A "" https://gitlab.c-path.org/c-pathontology/mapping-commons/-/raw/main/registry.yml
HTTP/2 403
server: awselb/2.0

Related

• Update all registry files #32 — the PR this produces: deletes all eight C-Path entries from data/mapping-specifications.json. Do not merge until this is resolved.