Repository navigation
Periodically find and queue missing compiler compatibility checks #804
Description
A package can finish publishing, but some of its compiler compatibility checks may never get queued. For example, the server might time out or restart while finding the checks to run. The package is available to users, but its list of supported compilers can remain incomplete indefinitely.
This happened with web-storage@5.0.1 in #800.
#801 makes follow-up scheduling faster and inserts jobs one at a time. That saves jobs already queued before an interruption, but it cannot recover checks it never reached. #803 improves solver performance, but faster solving alone does not repair missing checks.
So the basic problem remains: how do we detect and fix the unavoidable case where a compiler matrix job might not get queued for a published package? And how do we do it without introducing new state?
I think we can accomplish this with a regular, stateless sweep over the registry. The basic idea is to run a small repair command on a regular interval, for example through a systemd timer. Each run starts fresh and asks: Which published package versions still need compiler checks?
For each published package version and each supported compiler:
- If the package metadata already lists that exact compiler, skip it.
- If a matrix job already exists for that package version and compiler, skip it.
- Otherwise, try to find dependency versions that allow a build with that compiler.
- If a build plan is found, immediately insert a matrix job into the existing queue.
- If no plan is found, leave it alone and try again on a later sweep. A dependency may gain compiler support in the meantime.
The normal server executor still runs the compilation jobs and updates package metadata after successful compilation. The repair command does not compile packages or write compatibility metadata itself.
This survives interruptions itself. As a concrete example, suppose a sweep finds 100 missing checks and stops after queuing 40. The next sweep starts over, skips those 40 existing jobs, and finds the remaining checks. We do not need to remember where the previous sweep stopped.
This also finds old gaps, not just gaps caused by the latest publication or compiler release. It should examine all currently published versions, not only the newest version of each package.
The goal is eventual repair: as sweeps and compilation jobs complete, missing compatibility results are filled in wherever a build succeeds. A missing compiler entry is not proof that the package cannot compile with it, and a solved build plan is not proof of compatibility until compilation succeeds.
Side note: separating queueing from publication success
If publication finishes successfully, a later timeout while queuing compatibility checks currently makes the publish job fail. This is kind of misleading though; the compiler matrix is an eventually-correct best-effort thing, not a requirement for publishing. Compatibility scheduling should be outside the publication job's success/failure boundary and timeout. The periodic sweep can take responsibility for discovering that follow-up work.
The same principle applies to matrix jobs: once compilation succeeds and the compatibility metadata is successfully written/pushed, a later failure to schedule dependant checks should not turn that successful check into a failed job.
Activity
mikesol commentedon Oct 14, 2026
Thanks for writing this up! I think the stateless sweep is the right call. Two small things. First, could we say that a finished job counts as "already exists" in step 2, even if it failed? Otherwise, if I understand correctly, a package that fails to compile gets re-queued on every sweep. Second, packages with no build plan get re-solved on every sweep. My understanding is that #803 makes this cheap, but it'd still be good to measure a full sweep and adjust the interval accordingly. Nice work!
thomashoneyman commentedon Oct 14, 2026
Yeah, agreed on both. A job counts as “already exists” whether it’s pending, succeeded, or failed. The sweep would only add checks that never got a job in the first place (it won't retry failed compilations) — I can make that more explicit.
On the other hand, packages with no build plan indeed get re-solved on every sweep. I feel like that's the biggest weakness in this proposal, and I wasn't sure how bad that would be so I grabbed the current registry data and ran the equivalent of one of these sweeps over it. On a machine with a little (not much) more firepower than the registry machine, it took about 12 minutes to run and it repeated about 120,000 unsuccessful solves, so that feels like a dealbreaker.
I took a stab at an implementation so I could do that sweep and it feels pretty good minus the repeated solves. Two options come to mind:
Cacheeffect to put a hash of the solver inputs on disk for a given compatibility check, such that on future re-solves it's just loaded instead of actually solved. But this doesn't help with avoiding going through those 120k (and growing) entries, it just skips solving work altogether.This all assumes we're still happy considering no build plan to be a terminal outcome and we're not promising to keep re-checking all these packages as new dependency versions are published within their bounds.
thomashoneyman commentedon Oct 14, 2026
Well, I guess this has problems too -- we require jobs to have a build plan, and this wouldn't have one. So then we either relax that requirement to allow
null, wherenullmust always correspond with a failed job, or we decide to store this elsewhere.f-f commentedon Oct 14, 2026
I have been looking at this for the past few days and I am not really happy about it - some notes:
thomashoneyman commentedon Oct 14, 2026
I'd rephrase this as: matrix jobs can fail to start for a variety of reasons, some of which are unavoidable. In this case, it was timeouts; in another, maybe the machine goes down or restarts. I agree we should first try and make it so these situations are rare, and #801 and #803 help ensure that we don't hit the case that originally triggered all these misses. But it's not completely avoidable, so it still feels worth seeing if we can have some periodic job to check for dropped jobs and re-enqueue them.
Maybe we can use the existing system, in which case I agree with you! When I was originally looking for failed jobs I was looking at the database directly, and that led me to think this fixup likely should run on the registry machine itself — as opposed to those periodic jobs you linked, which are run in GitHub Actions and then submit jobs over the API. So that's the deciding factor IMO: if it must be run on-machine to see the information it needs, then
systemd; otherwise, the external job-starter would be better.I'm cautiously optimistic about splitting publish from the follow-on tasks, I think that's probably a nicer architecture. On the other hand, I'm really hesitant to do something like put a 2h timeout in place. That would block the registry for up to 2hrs on a single job, which feels unacceptable. Imagine running
spago publishand your job queues for hours before it can be picked up!Perhaps with both we could (a) split publish from the follow-on tasks, and (b) move those follow-on computations like solving and matrix cascade into a worker thread where if it happened to take 2hrs, that's OK, it's not blocking subsequent publishes.
I don't remember exactly how we chose 10 minutes, but IIRC we looked at the slowest compilation in the registry when we did a giant publish of lots of packages and even
unicodeanddeku(the slowest iirc) were well under this.I do like being loudly notified of failures! Alas, currently, no failures in the registry notify us at all, as we don't have any real monitoring besides the health of the machine itself. True, if we don't even fail loudly in the first place then we wouldn't be notified either so they're both in play, but a loud failure today would probably go unnoticed too if it's just in logs.
f-f commentedon Oct 14, 2026
I am not convinced yet that it's not avoidable 🙂
For the publishing job we have numerous checks in place so that, if for some reason the publish job was dropped halfway through, then we'd figure out the second time around where we were with it and pick it back up. Couldn't we do the same with the matrix-enqueueing job? (assuming that we split it in its own separate job type)
I.e. once it's its own job type, then queueing becomes a near-instant operation (that we can issue before marking the publish job as done, so that if the server restarts the operation can be repeated; once it's an independent job then it's basically an idempotent operation once we revert back to enqueueing all the jobs at once when the solving is done.
Not really - the trigger can still be external, then the server code can run on the machine all the checks it needs.
We need them to be in the queue due to the lock on the git repo. We chose 10 mins because it's a long enough time to achieve stuff and compile and so on, while being short enough that a publish waiting 10 mins while a job is running is not a big deal.
I guess we can keep the timeout to 10 mins but try to be more granular in what the jobs do, and eventually we can get to the point where nothing times out.
Yeah I think we should be worried about getting that stuff in place before we try to rearchitecture stuff 😄
thomashoneyman commentedon Oct 14, 2026
OK, this is true, if we're willing to make these kinds of things jobs. I guess i don't have a clear picture of what is meaningful enough a unit to be considered a "job". I've been thinking of that as like, the first-class registry operations (publish, unpublish, ...). How would you describe the new list of job types as you're thinking about this?
f-f commentedon Oct 14, 2026
We can split the jobs however granular we think we need them, and the main constraint we have is due to the lock we need to keep on the consistency of the git repo. The publish job can't really be split much further due to the fact that we need to do a bunch of stuff before we get to a point where we can write in the repo. Stuff that happens after we are done writing to the repo can be split. E.g. in the case of the publishing job we could have the docs publishing (Pursuit upload today, registry docs tomorrow) happen in a subsequent job that gets enqueued at the tail of the publishing job.
Splitting jobs is added ceremony, but it buys us the ability to interleave more urgent stuff in between and ultimately having the registry be more responsive. E.g. another publish job could slip in between a publish operation and its docs publishing.
This are the job kinds we have today:
registry-dev/lib/src/API/V1.purs
Lines 246 to 251 in f278f23
I am proposing that we add
EnqueueCompilerJobsForPackageandEnqueueDependantsForPackagejobs to the list. We'd enqueue those from any place where we callsolveForAllCompilersandsolveDependantsForCompilerrespectively, and never from the web API (just like matrix jobs today). The jobs would do little more than just call these functions really. It is important that we revert to adding things to the DB all at once after solving because this gives us the ability to rerun the job idempotently (by solving, checking if those solving jobs have been added to the DB, and enqueue them if not).If you'd like to add a recurring script that enqueues matrix jobs on demand (which I don't think we should do, because as I described above matrix jobs should be enqueued only when they solve) then we'd need to expose an endpoint to enqueue a single matrix job from the API.
thomashoneyman commentedon Oct 14, 2026
OK, I'm down with this. I am happier without the recurring job, and I don't mind adding more granular jobs (the extra ceremony there would be more complexity elsewhere anyway). Thanks for clarifying.