feat: declared background workers + frankenphp_get_worker_handle()

[nicolas-grekas]

Background workers run a script in a loop outside the HTTP request cycle, sharing the PHP runtime with the request threads. This is the smallest useful slice of #2398, rebuilt on the Server of #2499: the parallel Scope machinery is gone, a background worker attaches to a php_server through WithWorkerServerScope() like any other worker.

Declared with background in a worker block (php_server or global) or WithWorkerBackground() in Go. name is required, it is the script's identity; match is rejected; num >= 1, no lazy start here. $_SERVER['FRANKENPHP_WORKER'] now carries the name for every worker, HTTP ones included (the documented contract is to test its presence, not its value, as suggested on #2393), and $_SERVER['FRANKENPHP_WORKER_BACKGROUND'] is set in background workers so a script serving both roles can tell them apart with isset(). The lifecycle mirrors HTTP workers: re-run on a cooperative exit, restart with a capped quadratic backoff on a crash, max_consecutive_failures fails Init() during startup only. A crash past the ready point is paced by that backoff but never counted toward the cap, which is about a script that never boots: unlike an HTTP worker, paced by the traffic it needs before it can crash, a background worker reaches its ready point on its own and would otherwise spin. drain() now runs on shutdown, reboot and handler transitions, so a parked script wakes up instead of waiting out the force-kill grace period.

The script gets one handle, frankenphp_get_worker_handle(): resource, a stream that reaches EOF when the worker is drained. It is meant to carry control messages later, hence one handle rather than one per purpose. It is backed by a socket pair, not a pipe: on Windows PHP's php_select() only waits properly on sockets before 8.5, and the socket path is version-independent. Streams don't own the socket (php_sockop_close() would shutdown() it on Windows), so a run gets one stream, and closing it then fetching again yields a fresh one over the same socket without losing the drain signal; the read timeout is infinite so a blocking read parks as well as stream_select() does.

A worker counts as ready on its first wait on the handle (select cast or read), the background analog of frankenphp_handle_request(): Init() waits for it, ready_workers counts from it, and an exit before it is a boot failure. Fetching the handle is not the ready point, nothing forces a script to fetch it after bootstrapping.

Worker names are now scoped like paths: unique within a php_server or among global workers, so two blocks may each declare queue. The script sees the declared name; metrics and logs report a scoped worker as <server name>:<name>, with a numeric suffix on server names when two blocks resolve to the same one. The collision-driven renaming in the Caddy module is gone.

Deferred: lazy start (frankenphp_ensure_background_worker()), catch-all workers, shared-state APIs, and the orchestrator-style runtime API discussed in #2398.

Supersedes #2543 and #2398.

[henderkes]

Please rewrite the PR description to not be LLM slop reasoning with itself about what it did and why. I've tried reading this three times and I just can't.

[nicolas-grekas]

Sure, I'll let you know when I'm done, for now I just let it do the rebase 😅

[henderkes]

What happens here when a global background worker and a php_server scoped background worker share the same name and are both eligible for the same source file?

[henderkes]

found another one, anyway, have you tested this on windows?

[Copilot]

Pull request overview

Adds declared background PHP workers with graceful stop-stream handling and Caddy configuration support.

Changes:

• Adds background-worker lifecycle, validation, and thread allocation.
• Exposes frankenphp_get_worker_handle().
• Adds Caddy integration, documentation, fixtures, and tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
worker.go	Registers and validates background workers.
threadbackgroundworker.go	Implements background-worker lifecycle.
requestoptions.go	Rejects background workers for HTTP requests.
phpthread.go	Drains handlers during shutdown and transitions.
phpmainthread.go	Drains handlers during reboot.
options.go	Adds WithWorkerBackground().
frankenphp.go	Reserves background-worker threads.
frankenphp.c	Implements stop pipes and PHP API.
frankenphp.h	Declares C primitives.
frankenphp.stub.php	Declares the PHP function.
frankenphp_arginfo.h	Registers generated arginfo.
docs/config.md	Documents background configuration.
caddy/workerconfig.go	Parses background worker blocks.
caddy/config_test.go	Tests Caddy parsing and validation.
bgworker_test.go	Tests lifecycle, restart, scope, and validation.
testdata/bgworker/basic.php	Provides lifecycle fixture.
testdata/bgworker/crash.php	Provides restart fixture.
testdata/bgworker/early-return.php	Provides startup-failure fixture.
testdata/bgworker/named.php	Provides named-worker fixture.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[AlliBalliBaba]

Hmm I don't remember the specific reason why background workers are treated separately here. Wouldn't it make sense to just do this and count it into the general pool, like other workers:

if w.num <= 0 {
    if w.isBackgroundWorker {
		opt.workers[i].num = 1
    } else {
		opt.workers[i].num = maxProcs
    }
}

Worker thread count is already added on top of the general thread count. It would only overflow in case someone sets a general cap on global threads, in which case it should probably still honor that cap.

[nicolas-grekas]

The num_threads / max_threads budget exists for autoscaling HTTP workers, and background workers take no part in that: they don't scale, don't queue requests and never compete for a free thread. Counting them into the pool would change what the budget means depending on how many background workers a config declares: declare five and you silently get five fewer HTTP threads, so people would have to bump the budget just to keep the capacity they had, and the setting stops describing HTTP capacity. That's why they're reserved on top: the HTTP admission math is untouched and the totals are bumped afterwards (reservedThreads). Requiring an explicit num keeps that reservation visible in the config rather than defaulted.

[alexandre-daubois]

The reservation is right for num_threads, but max_threads auto is memory-derived and then floored back up to numThreads, so background workers silently push past that ceiling. Failing Init() on the total, instead of bumping max_threads, would keep it.

[nicolas-grekas]

auto is a memory heuristic that num_threads already overrides through the same floor; background threads are fixed threads and get the same treatment. Failing Init() would let a heuristic reject an explicit config, so they stay on top.

[AlliBalliBaba]

IMO it's a bit unfortunate that the ceiling exists in the first place, it would be better to have something like num_regular_threads and max_regular_threads, so people don't have to do maths with worker thread count.

But with the current logic all workers count toward that ceiling (they take away threads from the regular threads), so I think it makes more sense to be consistent when it comes to background workers, or we'll just make it even more confusing.

[nicolas-grekas]

Agreed on num_regular_threads / max_regular_threads being the shape that avoids the maths; that would be a separate change to the existing settings.

On consistency: HTTP workers count against the ceiling because they draw from the same pool, autoscale into it and compete with regular threads for it. Background threads never enter that pool: fixed count, no scaling, no requests. Counting them would make num_threads mean a different HTTP capacity depending on how many background workers a config declares.

What the review did change is the plumbing. calculateMaxThreads() used to bump the totals and subtract them back later; it now resolves num_threads / max_threads against the HTTP workers alone and returns the background threads separately, for Init() to add where a real total is needed. No addition-then-subtraction anywhere.

[AlliBalliBaba]

It looks like currently the handle is only used for shutdown. IIRC in the future you'd also want to use the handle to send messages or even requests.

Would it maybe be cleaner to have a separate handle for each? Makes the api look more like we're selecting over different channels, in other words:

frankenphp_get_shutdown_handle(); # instead of frankenphp_get_worker_handle
frankenphp_get_message_handle(); # future scope: can return a dedicated message
frankenphp_get_request_handle(); # future scope: can return a dedicated request object

[nicolas-grekas]

I'd rather keep one handle. In the prototype built on this primitive, shutdown, messages and requests all arrive on the same stream as typed messages, and the worker loop is a single stream_select() plus a dispatch on what was read; that worked well in practice. One handle per kind means selecting over N streams, N functions to document and keep in sync, and ordering questions between them (a message landing after shutdown was signalled on another stream). Fewer functions is also less API to get wrong. This PR only uses the EOF-on-drain part, but the handle is meant to carry the rest.

[AlliBalliBaba]

Hmm I think you're right since streams can only send strings.

What do you think about something like this? Abstracting things a bit allows us to do more in the future without BC breaks.

$worker = new \FrankenPHP\Worker(
    onMessage: fn(\FrankenPHP\Message $message) => ...,
    onRequest: fn(\FrankenPHP\Request $request) => ...,
    onShutdown: fn() => ...
);

$handle = $worker->getHandle();

while ($message = fgets($handle)) { # or the equivalent with stream_select
  $worker->handle($message);
}

The message can literally be "1", "2", "3", it will be handled internally and forwarded to onMessage() or onShutdown()

[nicolas-grekas]

That's pure PHP over the primitive, so it can ship as a package or a docs example and evolve without a FrankenPHP release; in the engine it freezes the callback signatures and the Message / Request shapes before anything uses them. A class can be added later, not removed.

BC-wise the primitive is the smaller surface: "lines, then EOF on drain", where a new kind of message is a new prefix. Three callback signatures and two classes are more to keep stable, not less.

Callbacks also take ownership of the loop, so anything a handler waits on has to be routed back through the dispatcher. The stream composes with Revolt, amphp, ReactPHP or a plain blocking read, none of which have to know about each other.

The tasks of #2636 are the first real message type here and they wanted functions: the loop drains the queue on each wake-up, since a line is a wake-up and not a count, and inside a task the worker does a stream_select() on the task's own stream to notice the sender giving up. A dispatcher handing out one message at a time makes both awkward. frankenphp_handle_request() is callback-shaped because a request has a beginning and an end; a background worker's loop owns the process lifetime.

Hand-rolled dispatch being easy to get wrong is fair, so I'd answer it with a documented loop, and a package if it earns its place.

[AlliBalliBaba]

The tasks of #2636 are the first real message type here and they wanted functions: the loop drains the queue on each wake-up, since a line is a wake-up and not a count, and inside a task the worker does a stream_select() on the task's own stream to notice the sender giving up. A dispatcher handing out one message at a time makes both awkward. frankenphp_handle_request() is callback-shaped because a request has a beginning and an end; a background worker's loop owns the process lifetime.

That's kind of the point, we're locking ourselves out of any changes/extensions to the api by requiring very specific steps to be followed (receiving literal "task" -> checking frankenphp_receive_task() -> receiving a stream -> passing the stream to frankenphp_update_task -> fclose)

$handle = frankenphp_get_worker_handle();
while ($message =  fgets($handle)) {
    if ($message === "task") {
        while ($task = frankenphp_receive_task()) {
            [$stream, $payload] = $task;
            frankenphp_update_task($stream, ['progress' => 50]);
            frankenphp_update_task($stream, ['result' => process($payload)]);
            fclose($stream);
        }
    }
}

It's not just about making the API less awkward, it's also about keeping control over how we handle what is sent in the streams. Doing it somehow like this still allows integrating the handle into amphp/react/etc with minimal surface.

$worker = new \FrankenPHP\Worker(onMessage: function(\FrankenPHP\Message $message){
   $message->respond(['result' => process($message->payload)]);
});
$handle = $worker->getHandle();
while ($message =  fgets($handle)) {
   $worker->handle($message);
}

Also allows us to just call exit() directly on shutdown if we want to.

[nicolas-grekas]

You're right that "task\n" is a bad idea, my mistake, and the pool case rules it out: a line is a wake-up, not a description. A plain "\n" would be enough, content unspecified.

That leaves the kind to the receive side, which may be where an object fits: frankenphp_receive([Task::class]) returning a Task. The list is the opt-in, so a script only ever sees kinds it knows, the runtime keeps the wire format and can answer for scripts that don't ask, and a new kind is a new class instead of a new function. Does that give you the room you're after? About callback-based approaches, they're doomed to fail: a handler that runs to completion holds one task at a time, so anything that keeps several open and selects across them has to escape it, which is the case multiplexing needs.

Mostly a #2636 discussion, but it belongs here too, since it confirms the shape of this PR: one handle, an opaque wake-up, EOF on drain, nothing parsed.

On exit(): EOF already ends the loop, so a script can return or exit on shutdown with no API for it.

[AlliBalliBaba]

Yes on one hand a Task class would be much cleaner, on the other hand, forwarding the message on the handler back to us leaves us in control of what to do.

For example when we get back "1", we poll for a message and call the onMessage handler with the message as argument. When we receive "2", we call the onShutdown handler and exit out of the script.

In the future we might want to do something like send "3" when there is a "tick" or "request" event. Gives us more control, eg. we can add a custom \FrankenPHP\Tick and throw an Exception if the handler is not implemented. The user does not need to worry about what "1", "2", "3" means.

Doing this via callback doesn't stop us from handling multiple tasks at a time, $worker->handle($message) just calls another function immediately. The event loop can call $worker->handle($message) before the callback is finished again no problem (unless I'm missing something)

[AlliBalliBaba]

Looking at how react/revolt incorporate handles, maybe we could even do something like this:

$worker = new \FrankenPHP\Worker(onTask: function(\FrankenPHP\Message $message){
   $message->respond(process($message->payload));
});
$handle = $worker->getHandle();

// --- ReactPHP ---
Loop::addReadStream($handle, function () use ($worker) {
    $worker->tick(); // handle just is a wakeup mechanism, we can decide what to do here non-blocking
});

[nicolas-grekas]

Two review-level items.

Name collision between a global and a php_server background worker: names are now scoped like paths, unique within a php_server or among global workers, so both start. Each script sees jobs in FRANKENPHP_WORKER, WithWorkerName() resolves within the request's server first, and metrics and logs report the scoped one as <server name>:jobs (server names get a numeric suffix when two blocks resolve to the same one). Two workers with the same name inside one php_server fail Init(). TestBackgroundWorkerOnServer and same_worker_name_in_two_servers cover both directions.

Windows: the Windows workflow runs the full suite on PRs and it passes here on 8.5.10, background worker tests included. It also surfaced that stream_select() on a pipe would have spun on 8.4 and older, since php_select() only waits properly on sockets before the GH-16889 fix, which is 8.5 only. The handle is therefore backed by a socket pair on every platform, which takes the version-independent Winsock path.

The branch is squashed to 3e93a24; sha references in earlier replies predate the squash.

[nicolas-grekas]

Since the replies above, a self-review pass amended into the single commit (2f9c5b6):

• a worker scoped to a Server never passed to WithServer() panicked on a nil map; Init() now rejects it
• failureCount reset at the ready point, like HTTP workers: a crash after it restarts right away, only boot failures count toward max_consecutive_failures
• the drain race check in setupScript also covers TransitionRequested
• Windows: PHP's socketpair() emulation listens on INADDR_ANY and accepts the first peer, so the pair is verified with getpeername/getsockname; SOCK_CLOEXEC where available
• a warning after 10s when a script has not waited on its handle, since Init() and Shutdown() wait for that point; docs say to block on the handle, feof() polling is not a wait
• max_threads is rejected on background workers instead of ignored, and the Caddyfile requires num
• untyped resource return in the stub, dead zend_unset_timeout() between requests removed, background threads no longer reported busy by the threads endpoint
• tests: parking on a blocking read, RestartWorkers() draining a parked script, the handle throwing outside a background worker, the new validations

CI: all test jobs pass. The Windows job's caddy-suite timeout (POSTed configuration isn't active, then an admin GET hangs) is the same flake main's nightly runs hit (33416988237, 33302318173) and passes on rerun. The Docker matrix fails on the Go 1.27 runtime/cgo link error that #2634 and main's nightly show too.

[alexandre-daubois]

$_SERVER['FRANKENPHP_WORKER'] moves from 1 to the worker name for every existing HTTP worker, and an unnamed worker now sees the absolute script path, so a script gating on === '1' stops detecting worker mode. The testdata/_executor.php change in this diff is that exact comparison, so calling the break out in docs/worker.md would help more than describing the new value under name alone.

[nicolas-grekas]

Called out in docs/worker.md now, with the isset() advice; the two symlink fixtures' messages fixed too.

[henderkes]

I'd be okay with this, if we weren't immediately introducing another implicit contract by setting FRANKENPHP_WORKER_BACKGROUND to literal 1.
Don't have a solution, but I suspect @AlliBalliBaba may have a smart idea

[nicolas-grekas]

We could add another var of course, eg FRANKENPHP_WORKER_NAME

[nicolas-grekas]

Fair, and the answer we just settled on for the wake-up line applies here too: presence is the contract, the value is not. The docs and the stub say so explicitly now, and the fixture reading it was changed to test presence, so nothing in the tree compares against 1.

If that still reads as too implicit, the alternative is a function rather than a variable, at the cost of API surface for a boot-time fact. Happy to go that way if you and @AlliBalliBaba prefer it.

[Copilot]

🟡 Changes recommended

Handle aliasing, unbounded crash loops, ambiguous metric identities, and extension-worker regressions remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 45/45 changed files
• Comments generated: 5
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Scoped extension dispatch drops request options, and background-worker configuration has inconsistent global and environment behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 47/47 changed files
• Comments generated: 3
• Review effort level: Balanced

[henderkes]

I'm largely happy with the API decisions.

Mostly architectural pointers, I'm a bit concerned with the amount of code duplication and extra code debt.

Also raising this from above here again: #2617 (comment)

[henderkes]

I'd be okay with this, if we weren't immediately introducing another implicit contract by setting FRANKENPHP_WORKER_BACKGROUND to literal 1.
Don't have a solution, but I suspect @AlliBalliBaba may have a smart idea

[nicolas-grekas]

Thanks, all items should be addressed now!

[henderkes]

Thanks, all items should be addressed now!

I'm not that fast in reviewing 😆. I hope to look again tomorrow.

blub