feat(extras): deliver large MicroVM jobs over HTTP

[ParidelPooya]

Summary

Part 4b of the MicroVM port: microvm() delivers a job over HTTP when it does not fit in the run hook. It follows the JavaScript operation in @aws/durable-execution-sdk-js-extras.

Before this change, a job whose run hook payload exceeded 4,096 code points failed with MicrovmDeliveryError before the launch. Now the operation picks the delivery:

1. A job whose run hook payload fits goes in the run hook, as before.
2. A larger job, or any job when config.request.path is set, goes over HTTP.

The choice runs after the callback exists, because the payload holds the callback ID. Replay follows the delivery that the launch checkpointed, not a new choice.

HTTP delivery

• The launch sends a runHookPayload with only version and region, because the worker rejects a payload without them. The MicroVM gets the ALL_INGRESS connector unless the config sets connectors.
• The launch step checkpoints {"microvmId", "endpoint", "delivery": "http"}, the same document as JS.
• A new <name>.request step creates an auth token with CreateMicrovmAuthToken and POSTs the job document to the endpoint. The body has the job fields at the top level, next to version, region, and microvmId.
• A failed delivery terminates the MicroVM and raises MicrovmDeliveryError.

Retries

The request step retries in two tiers, as in JS:

1. Inside one step attempt, send_job retries connection errors, timeouts, 429, and 5xx, with a delay from 250 ms to 4 s. It retries one 401 or 403 with a new auth token. Another 4xx, or a 3xx, raises MicrovmRequestRejectedError at once. The tier ends at the retry window (60 s by default, 1 s to 10 min), or 10 s before the invocation times out.
2. The step retry strategy. default_microvm_retry_strategy now retries MicrovmEndpointUnavailableError and MicrovmRequestUnauthorizedError, and does not retry MicrovmRequestRejectedError.

The 10-second reserve keeps time for the step to record its failure. boto3 cannot cancel a call. So a CreateMicrovmAuthToken call with a deadline runs in a daemon thread, and the step stops waiting for it at the reserve. Each POST runs in a daemon thread too, and the step waits for it at most 10 seconds, or until the reserve. A socket timeout alone does not bound a POST, because each byte received starts it again.

The POST uses urllib with no proxy handler, because the request goes to the MicroVM endpoint directly. MicrovmConfig.opener replaces it, for example for a network that reaches the endpoint only through a proxy. The default opener is built on first use. The path is sent exactly as written. It must be an RFC 3986 path without a . or .. segment, because the workers match the raw request path. The JS port has the same rule in aws/aws-durable-execution-sdk-js#1005. The POST adds its headers as unredirected headers. So an opener that follows a redirect sends no auth token on.

A request step that starts with less than 11 seconds left in the invocation fails at once with a retryable error that names the function timeout.

Differs from JS: redirects

The operation does not follow redirects. A 3xx fails the delivery at once, with the Location in the message. JS follows redirects, and that loses the job for a 301, 302, or 303: the POST becomes a GET without the job, and the GET's 2xx counts as accepted. A redirect to another host also receives the auth token. aws/aws-durable-execution-sdk-js#1001 proposes the same behavior for JS.

New public API

• MicrovmRequestConfig with path, port, and retry_window, and MicrovmConfig.request and MicrovmConfig.opener.
• DEFAULT_MICROVM_JOB_PATH and DEFAULT_MICROVM_PORT.
• MicrovmRequestRejectedError, MicrovmRequestUnauthorizedError, and MicrovmEndpointUnavailableError.

Invalid request settings raise the core's ValidationError before any durable operation starts.

Testing

• Package: 187 tests pass, with 100% line and branch coverage. mypy and ruff are clean.
• Repository: 4,955 tests pass, on the session branch that contains this branch.
• A sub-agent review applied 14 mutations, and 2 survived. Tests now cover both, plus the review's other findings. Each new test fails under the mutation it guards against: replay following the recomputed delivery, a POST without a total bound, following redirects, no path encoding, urlopen with proxies, and catching only Exception in the deadline thread.
• request_test.py runs send_job against a local HTTP server: the retry statuses, the delays and their cap, the token refresh, the retry window, the reserve, the request timeout, a server that answers one byte at a time, redirects, a caller's opener and a redirect it followed, with no auth token on the followed request, the path sent as written, the time-left check, and proxy settings in the environment.
• operation_test.py runs the operation on the local test runner with HTTP delivery: a large job, a configured route and port, a rejected job, a step retry after the first tier, a replay that keeps the launch's delivery in both directions, a replay that sends no second POST, a missing endpoint, and a botocore Stubber check of CreateMicrovmAuthToken against the service model.
• worker_composed_test.py sends a large job and a route job over HTTP to the real worker from feat(microvm-worker): add HTTP listener and server #767.
• Not run against the live service.