feat(microvm-worker): act on lifecycle hooks

[ParidelPooya]

Summary

Part 6 of the MicroVM port: the worker acts on the suspend, resume, and terminate lifecycle hooks, and suspends its own MicroVM when a session asks for it. It follows the JS worker in @aws/durable-execution-sdk-js-microvm-worker (aws/aws-durable-execution-sdk-js#972, #975, and #976).

Before this PR, the worker answered these hooks with HTTP 200 and did nothing else. It parsed autoSuspendIdleSeconds from the run hook payload and ignored it. So microvm_session(..., auto_suspend_on_idle=True) from #779 had no effect with the Python worker.

The suspend hook

1. The worker changes its state, and then answers HTTP 200 at once. A suspend hook that runs past its timeout terminates the MicroVM.
2. It refuses every job with HTTP 503 until the resume hook arrives, or for 30 seconds. A job accepted now would freeze with the MicroVM. After a 503, the durable function resumes the MicroVM and delivers the job again.
3. Running jobs freeze with the MicroVM and continue after a resume.
4. A run hook job that arrives during the refusal is deferred, and starts when the refusal ends. The durable function cannot deliver a run hook job again.

The terminate hook

1. The worker sets each running job's context.cancelled, stops its heartbeats, and fails its callback with the new MicrovmTerminatedError. The microvm operation then fails at once with MicrovmJobFailedError (MicrovmTerminatedError), instead of at the job's heartbeat timeout or its timeout.
2. It skips a job whose callback a heartbeat already found gone.
3. It answers HTTP 200 when the reports end, or after 5 seconds. The answer also waits for jobs that report their own outcome, for run hook requests whose body is still arriving, and for the failure reports of invalid requests. A second terminate hook waits for the same work.
4. From then on it refuses jobs with HTTP 503, fails a deferred or late run hook job with MicrovmTerminatedError, and never suspends again.

Hooks that do not come from Lambda

The MicroVM endpoint forwards a request to a hook path from any caller that holds an auth token for the MicroVM. Probes from the JS work found that Lambda's own hook calls carry a local Host and no x-amzn-requestid header, and forwarded requests carry both. So the worker acts on suspend, terminate, and resume only when the Host is localhost, 127.0.0.1, or [::1], with any port, and the request has no x-amzn-requestid. Any other request to these hooks gets HTTP 200 and a warning. The run hook does not use the check, because a false rejection there would lose every job.

Suspending when idle

When the run hook payload sets autoSuspendIdleSeconds:

1. The idle time starts when the run hook arrives without a job, when a job ends, and when the resume hook arrives. A job that starts cancels it.
2. When it ends, the worker calls SuspendMicrovm with the run hook's MicroVM ID, never one from a job request, and refuses jobs from that moment.
3. The refusal ends at the resume hook, or 30 seconds after the call returned.
4. The worker waits for the call for at most 10 seconds. A before-send handler on the client stops botocore's retries at that deadline. Without it, botocore kept retrying in the call's thread, and a probe measured an attempt 22.5 seconds after the first one.
5. A failed call has one of three outcomes:
   • Rejected on the first attempt, such as AccessDeniedException, or missing credentials. The refusal ends at once. The worker logs the lambda:SuspendMicrovm permission, and does not retry until a job ends.
   • Refused for now on the first attempt: a throttle, a 429, expired credentials, or a clock offset, at any HTTP status. The refusal ends at once, and the idle time starts again.
   • Unknown: a timeout, a conflict, a server error, a dropped connection, or any error after a botocore retry. The service may have suspended the MicroVM, so the refusal continues.

New options: create_microvms_client on start_microvm_worker and MicrovmWorkerListener.create. MicrovmWorkerListener.close() stops the idle time, and MicrovmWorker.close() calls it.

Dependency change

The worker now requires boto3>=1.43.35, up from 1.42.1. botocore 1.43.35 is the first release with the lambda-microvms client. The extras package already requires it.

Differences from JS

• The worker registers a job for the terminate hook before the job's thread starts, and the thread checks the mark under the lock before it starts the heartbeats. Requests and jobs run in threads, so a terminate hook can arrive in between. The JS worker registers synchronously, so it has no such gap.
• The state change for suspend and resume comes before the 200 answer. In JS the answer and the change run in one event loop turn. In Python another request thread could accept a job between them.
• A first-attempt throttle, expired credentials, or clock offset is "refused for now", and the idle time starts again. JS decides by HTTP status: such a 4xx is a rejection, and a 429 is an unknown outcome. A JS follow-up could adopt the Python behavior.
• The SuspendMicrovm deadline uses a daemon thread and a before-send handler, because botocore has no per-call deadline. JS passes an AbortSignal.

Not in this PR

• Examples and integration tests against the live service (item 7). They need the Python CI account to allow MicroVM resources.
• Custom subtypes, as in the earlier parts.

Testing

• Worker package: 385 tests pass. worker.py and self_suspend.py have 100% line and branch coverage. mypy and ruff are clean.
• Extras package: 259 tests pass.
• Repository: 5,089 tests pass.
• lifecycle_hooks_test.py runs a real worker over HTTP, with a fake clock in place of the worker's timers. It ports the cases of the JS lifecycle-hooks.test.ts and auto-suspend.test.ts, and adds Python cases: a terminate hook before a job's thread runs, a stale timer that fires after its cancel, threads and timers that cannot start, and the three call outcomes.
• self_suspend_test.py covers the outcome classification, and checks against a silent local service that no attempt starts after the deadline. With the before-send handler removed, that test fails.
• worker_composed_test.py runs a microvm_session whose real worker suspends itself between two jobs, with and without the resume hook. Without it, the second job gets 503 answers until the refusal ends, and the session succeeds.
• A sub-agent review approved the change with four non-blocking findings. All four are fixed in dbe36864.
• The last commit fixes test flakes in parallel runs on macOS. A dual-stack worker binds [::] with SO_REUSEADDR, and macOS lets that bind succeed on a port that another process holds on 127.0.0.1. Tests now connect to ::1. 80 of 80 runs with 8 workers passed, against 15 failures in 262 runs before. Linux refuses such a bind, so a MicroVM is not affected.
• Not run against the live service.