Skip to content

dev2-site: healthchecks for every app + redis container, and nginx->app keepalive - #153

Merged
ralyodio merged 3 commits into
masterfrom
feat/dev2-healthchecks-keepalive
Oct 6, 2026
Merged

ralyodio merged 3 commits into
masterfrom
feat/dev2-healthchecks-keepalive

Conversation

@ralyodio

@ralyodio ralyodio commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

healthchecks (dev2-site healthchecks <site>… | --all): watchdog-autoheal restarts what Docker marked unhealthy, but 47 of 230 dev2 containers had no healthcheck.

  • The probe runs inside the container with its own bun or node and accepts any HTTP status under 500, without following redirects. A blocked event loop fails it, a 404 or login redirect doesn't. Redis gets redis-cli ping.
  • The chosen test is saved in the site state. render_compose writes it back, and migrate runs the step after deploy.
  • Applied to dev2: 216 of 231 containers are healthy and 0 unhealthy. The 15 left are workers with no port, pairux media, test databases, tor, and 4 hand-made compose files that pick it up on their next deploy.
  • A first-run bug stripped our own check on a re-run (hqtui.com). It's fixed and regression-checked.

vhost keepalive:

  • Each plain loopback proxy_pass becomes an upstream with keepalive 16; keepalive_timeout 4s. That's under Node's 5s server keepAliveTimeout, so nginx never reuses a socket the app just closed.
  • A new $connection_upgrade_keepalive map (installed to conf.d by the tool) still upgrades websockets but no longer sends close.
  • Verified on hqtui.com: 5 pooled connections held after requests finish, versus a new TCP connection per request before. A vhost --all --dry-run over 78 sites showed only keepalive diffs.

🤖 Generated with Claude Code

ralyodio and others added 3 commits October 6, 2026 13:08
…ealthcheck

watchdog-autoheal restarts what Docker marked unhealthy, but 47 of 230
containers on dev2 had no healthcheck, so a wedged one stayed invisible.
The probe runs inside the container with the runtime it already has (bun
or node) and passes on any HTTP status under 500 without following
redirects, so a blocked event loop fails and a 404 or a login redirect
does not. Redis gets redis-cli ping. The chosen test is kept in the site
state and render_compose writes it back, so provision cannot drop it; the
migrate sequence runs it after deploy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ever report an empty read as fine

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… connections

healthchecks: the container reports the healthcheck we added, which read as
'already has one', so the plan was empty and health_compose removed our
marked block (hqtui.com lost its check on the first fleet run). A marked
block is ours and is re-planned.

vhost: every plain proxy_pass to a loopback port becomes an upstream with
keepalive 16 and keepalive_timeout 4s (under Node's 5s server timeout, so
nginx never reuses a socket the app just closed), and Connection comes
from a new $connection_upgrade_keepalive map (installed to conf.d by the
tool) that still upgrades websockets but no longer sends close on every
request. Before this nginx opened a new TCP connection per request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

ThreatCrush Security Scan

32 finding(s)

HIGH/CRITICAL: 7 | MEDIUM: 12 | LOW: 13

Severity Rule Location
HIGH py-ssrf-outbound-request dev2/dev2-site:151
HIGH py-ssrf-outbound-request dev2/dev2-site:1060
HIGH py-ssrf-outbound-request dev2/dev2-site:1771
HIGH sh-remote-script-execution root-ubuntu.sh:3248
HIGH sh-remote-script-execution root-ubuntu.sh:3249
HIGH sh-remote-script-execution root-ubuntu.sh:5137
HIGH sh-remote-script-execution root-ubuntu.sh:5141
MEDIUM sql-template-interpolation dev2/dev2-site:1286
MEDIUM sql-template-interpolation dev2/dev2-site:1363
MEDIUM sh-remote-script-execution root-ubuntu.sh:5314
MEDIUM redos-nested-quantifier src/domain-free.ts:56
MEDIUM redos-nested-quantifier src/emoji.ts:167
MEDIUM redos-nested-quantifier src/icon.ts:166
MEDIUM redos-nested-quantifier src/mail.ts:1047
MEDIUM sql-template-interpolation src/users-dump.ts:487
MEDIUM sql-string-concatenation src/users-dump.ts:507
MEDIUM sql-template-interpolation src/users-dump.ts:540
MEDIUM sql-string-concatenation src/users-dump.ts:574
MEDIUM redos-nested-quantifier src/wcag.ts:556
LOW secret-generic-credential src/credentials.ts:36
LOW secret-generic-credential src/credentials.ts:53
LOW secret-generic-credential src/credentials.ts:56
LOW secret-generic-credential src/user-export.ts:689
LOW secret-generic-credential src/user-export.ts:695
LOW secret-generic-api-key test/credentials.test.ts:208
LOW secret-generic-credential test/mail.test.ts:141
LOW secret-generic-credential test/proxy.test.ts:52
LOW secret-generic-credential test/proxy.test.ts:68
LOW secret-generic-credential test/shorten.test.ts:36
LOW secret-database-url test/users-dump.test.ts:108
LOW secret-database-url test/users-dump.test.ts:119
LOW secret-database-url test/users-dump.test.ts:120

Snippets are redacted; ThreatCrush never prints matched credential material.

@ralyodio
ralyodio merged commit 19ef5c8 into master Oct 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant