Skip to content

fix(server): treat a WebAssembly trap as terminal instead of reopening forever - #8

Merged
Patodo merged 2 commits into
mainfrom
fix/wasm-trap-is-terminal
Sep 19, 2026
Merged

Patodo merged 2 commits into
mainfrom
fix/wasm-trap-is-terminal

Conversation

@Patodo

@Patodo Patodo commented Sep 19, 2026

Copy link
Copy Markdown
Owner

修掉那次「越界会死」的根因。证据来自你机器上的致命栈,结论是:平台认出了 trap,但它的"恢复"本身就是 bug。

致命栈给出的关键信息

RuntimeError: memory access out of bounds
    at new e (...sql-wasm.js:75:184)                 ← 构造 Database
    at openMetaDbFromDisk (...worker.cjs:53349:63)   ← 重开 meta 库
    at getDb (...worker.cjs:53998:19)
    at cleanupDesktopActions (...worker.cjs:139541:20)
    at Timeout.cleanup ...                           ← 裸 setInterval(6h)

getDb() 只在 db === null 时才重开 ⇒ evict 已经发生过一次,即平台认出了 WASM 错误。所以问题不是"没认出来",而是重开那一刻又 trap。

根因:initSqlJs() 重复调用返回同一个模块实例

平台原有的恢复是「弃用 handle + 从磁盘重开」(app-db 里还会把缓存的 SqlJs 置空)。我实测:

const a = await initSqlJs(); const b = await initSqlJs();
a === b   // true

模块实例是同一个、而且已经被 trap 撕裂,所以进程内不存在任何可用的恢复——重开必然在 new SQL.Database(...) 上再 trap 一次。于是:

  • 一次 trap → 之后每一个碰库的请求都 500
  • 而 5 秒的请求日志 flush 是 try { … } catch { /* swallow */ },第一次 trap 就发生在它里面并被吞掉 ⇒ 进程带着坏模块又跑了 46 分钟没人知道;
  • 46 分钟后 6 小时的清理定时器(没有 catch)撞上同一个 trap → 未捕获异常 → 进程退出。

改了什么

① trap 一旦发生就是终局,不再假装恢复runtime-errors.tsapp-db.tsmeta-sqlite.ts

新增 markSqlJsRuntimeUnusable(err, scope):记录终局状态、打一行明确日志、让进程停止一次(退出码非 0),由监督者/看门狗拉起全新进程——对撕裂的模块实例,这是唯一存在的恢复。之后任何碰库的请求拿到 db_runtime_restart_required(503),而不是再去重开撞一次。重复 trap 不会再次触发停止。

② 碰库的定时器套上错误边界(新增 timer-guard.ts

一共 8 处:6 小时的 desktop/device action 清理、5 秒的请求日志 flush、30 秒的验证会话清理、60 秒的 app 库空闲关闭、两处 15 秒 SSE 心跳。guardTimerCallback 把 trap 送到上面同一条终局路径;普通定时器失败只记日志、不再静默消失;异步 reject 也会被捕获。

③ 那条吞异常的 catch 不再吞致命错误request-logger.ts)——正是它让故障隐身 46 分钟。

一条测试曾经保证了错的结论

tests/integration/app-db.test.ts 里有个用例断言"evict 后重开可用",它用 JS 抛出的 WebAssembly.RuntimeError 模拟 trap——这种异常不会撕裂 emscripten 模块,所以测试里重开确实可用,真机上必然不可用。我把它改成断言真实语义(终局 + 拒绝访问 + 只请求停止一次),并在注释里写清为什么旧断言是错的。

验证

  • packages/server-core27 文件 / 205 用例全过
  • packages/server:失败集合与本机改动前基线完全一致(8 个 Windows 环境类文件、14 条:可执行启动、rename EPERM、clone 夹具等),无新增失败
  • 新增用例:runtime-errors.test.ts(3)、timer-guard.test.ts(3)、改写后的 app-db.test.ts(5 全过)
  • 两个包 tsc 通过

未包含 / 仍需你那边确认

  • 没有在真机上复现原始 trap(我无法按需触发 WASM 越界),修的是恢复语义错误边界db 为何会第一次 trap,本轮没有定论——数据文件在崩溃时是完好的(integrity_check=ok、816 KB),所以不是数据损坏或体积问题。
  • 平台侧的整改只到"不再带着坏模块硬撑、不再静默";进程内真正自愈需要换进程,所以现场仍需一个监督者(你们现在的看门狗正好扮演这个角色,只是触发条件从"探活失败"会变成"进程退出了")。
  • 顺带一提:named-SQL 的 500 出参建议带上 code/details(平台是 Fastify 默认序列化),你们客户端也把 json.code 打出来——这次的 46 分钟静默里,客户端只保留了 HTTP 500 Internal Server Error

`localapp server run` asked the OS for an ephemeral port on every start, so the
address changed on each restart. Anything that has to name the Server — a saved
profile, a reverse proxy, a published container port, or a terminal that keeps
one process alive — could not pin it, and the port a running Server reported had
to be copied by hand into every client.

Default to 50524 and keep `--port` as the override, including `--port 0` for
callers that really want an ephemeral port. The container image already passes
an explicit port, so its behaviour is unchanged.

The foreground mode is also a first-class way to run a personal Server when a
fixed address matters or when the machine cannot register a scheduled task:
docs/local-runtime.md now says so next to the daemon path.

Verified on the packaged build: no arguments listens on 127.0.0.1:50524 with
/health answering 200, and `--port 55441` moves it there and releases 50524.
…g forever

A production Server died from `RuntimeError: memory access out of bounds` at
`openMetaDbFromDisk` -> `new SqlJs.Database(...)`. That stack is the clue: the
open only happens when the cached database was already evicted, so the trap had
been recognized before — the recovery was the bug.

`initSqlJs()` returns the same Emscripten module instance on every call, so once
a trap tears it, reopening the database in-process traps again on the very next
construction. The previous code evicted the handle and reopened anyway, which
turned one trap into every database request failing, and — because the 5s
request-log flush swallows its own errors — did so invisibly for 46 minutes,
until the 6h cleanup callback hit the same trap without a catch and killed the
process.

Record the runtime as terminal on the first trap, refuse later database access
with `db_runtime_restart_required` (503) instead of a doomed reopen, and stop the
process once so its supervisor starts a clean one — the only recovery that
exists for a torn module instance.

Timer callbacks that touch SQLite also had no error boundary at all: six of them
plus two SSE heartbeats. Route them through `guardTimerCallback`, which sends a
trap to the same terminal stop, keeps a genuine timer failure from vanishing, and
captures asynchronous rejections.

The previous app-db test asserted the opposite guarantee — it simulated the trap
with a JS-thrown RuntimeError, which leaves the module intact, so in-process
reopen appeared to work. It now asserts the real one.
@Patodo
Patodo merged commit 78c8146 into main Sep 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant