Skip to content

fix(microvm-worker): fail a job whose heartbeats cannot start - #771

Open
Vivek0712 wants to merge 1 commit into
aws:mainfrom
Vivek0712:fix/microvm-worker-heartbeat-start
Open

Vivek0712 wants to merge 1 commit into
aws:mainfrom
Vivek0712:fix/microvm-worker-heartbeat-start

Conversation

@Vivek0712

Copy link
Copy Markdown

Fixes #770. Two follow-ups to the listener in #767.

Changes

  1. A job whose heartbeats cannot start fails its callback. _run_job_and_report now catches a failure of Heartbeats.start, logs could not start the job's heartbeats, and reports the error through _report. Before, the job had its 202 (or the run hook its 200), the worker only logged job failed inside the worker, and the callback ID was recorded as finished, so the durable function waited for its heartbeat timeout. This matches what feat(microvm-worker): add HTTP listener and server #767 already does when the job's own thread cannot start.
  2. A Content-Length of more than 19 digits is invalid. _CONTENT_LENGTH is now [0-9]{1,19}. Before, a value of more than 4,300 digits matched, int() raised ValueError past Python's digit limit, and with Expect: 100-continue the error escaped handle_expect_100, so the client got no answer and stderr a traceback. Nineteen digits still exceed every body limit, so no valid request changes. With Expect, such a request now gets 100 Continue and then the 400 for the invalid header, the same path as any other invalid length.

Testing

  • test_fails_a_job_whose_heartbeats_cannot_start and test_answers_400_to_a_content_length_with_too_many_digits are new. Each fails when its fix is reverted.
  • 261 tests pass. worker.py keeps 100% line and branch coverage.
  • ruff format --check, ruff check, mypy and lintcommit.py pass.

Live check of #767

I also ran #767's listener on the live service before writing these. I built it into a MicroVM image, with a Python copy of the JS SDK's extras/e2e/microvm-app.ts, and pointed the JS SDK's run-e2e.mjs at that image, with the TypeScript durable function and the assertions unchanged. 11 scenarios passed: succeed, fail, crash, request-succeed, request-fail, auto-http, auto-http-cjk, run-hook-cjk, run-hook-emoji, session, parallel. terminated ends with MicrovmTimeoutError at the 300 s heartbeat timeout, as expected until the terminate hook action lands, and suspend-session waits for the suspend and resume actions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

🤖 Generated with Claude Code

A job that has its 202, or its 200 from the run hook, gets no second
delivery. When the heartbeat thread could not start, the worker only
logged the error and recorded the callback ID as finished, so the
durable function waited for its heartbeat timeout. The worker now fails
the callback, as it does when the job's own thread cannot start.

A Content-Length of more than 4,300 digits matched the header pattern,
and int() raised past Python's limit on digits. With Expect:
100-continue the error escaped the handler, and the client got no
answer. The pattern now allows at most 19 digits, so such a header is
invalid and gets HTTP 400.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Vivek0712
Vivek0712 deployed to ai-pr-review-runtime October 13, 2026 21:24 — with GitHub Actions Active

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I checked both fixes. The two new tests fail on main and pass with this change. worker.py stays at 100% coverage, and mypy is clean.

One nit, not blocking. Content-Length: 0000000000000000000005 is a valid header, and Node accepts it, but the 19-digit check rejects it with 400. If you want to keep it, count the digits after stripping leading zeros. No client I know sends this, so the current change is fine as is.

I also checked the JS worker. Node's parser answers 400 for an overflowing length with and without Expect. Its heartbeats run on timers, so JS needs neither change.

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — c8bb74e5 Deployed Oct 13, 2026 by Vivek0712 via notify / Generate notification summary #200
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(microvm-worker): a job whose heartbeats cannot start is never reported, and a long Content-Length with Expect gets no answer

2 participants