KeepAlive for web workflow

[grgrant]

Edit: Supersceded by comment below #11342 (comment) where I tracked down that mDNS is the issue creating the very slow circup sessions

I was troubleshooting some very slow circup sessions so I developed with Claude's help a version of WebWorkflow that honors KeepAllive. circup takes advantage of keep alive.

So the question for discussion is whether or not anyone has thought of enabling KeepAlive (persistent connections) in WebWorkflow and whether there is interest in me posting the PR for people to bang on or should we leave things as they are.

My implementation currently times out after 2s so only a running connection is kept alive.

The changed behavior is one connection could hold until it finished rather than the current interleaving of single requests where a web connection and circup can work concurrently.

I can't say I've tested all cases by any means but I've run for a few hours with the Keep Alive code and circup runs the same as usual and the filesystem operations in Web Workflow work fine. So far nothing I've tried seems to have regressed.

[tannewt]

I'm not exactly sure what this means. Could you explain what we'd be able to do now that we can't currently?

[grgrant]

Edit: While the keep alive does solve the timing problem there is a likely simpler fix. On a dual stack system like my Mac the mDNS query is asking for both A and AAAA records. Since CP workflow doesn't respond with a NSEC negative response for the AAAA portion it takes a 5s timeout. Since likely the mDNS is in the ESP-IDF code it may be more difficult to fix it there. So it may fall to just patching circup to do a single IPV4 resolve and cache the IPV4 address returned.

@tannewt I finally tracked down the smoking gun. Also tagging @FoamyGuy @tekktrik since they've touched circup the most recently for Adafruit. Also tagging @dhalbert because he most frequently touches the supervisor/web_workflow stuff.

Problem Description

Without implementing KeepAlive the current circup takes 52x longer to do a check for updates when a person uses circup --host hostname.local update --all vs using either hostname.lan or hostip. As an example the test output below shows almost 7 minutes vs 7-8 seconds for the other two versions. This is due to no KeepAlive and a reconnect for each operation of circup even though circup with no changes utilizes the KeepAlive version of the server I created. mDNS resolution is a time consuming process (5s per query/response) vs regular DNS (likely cached) and IP addresses (no resolution needed).

I should mention that I have frequently had an circup session using hostname.local take 11-14 minutes especially if there are actually files to update.

I've included below a set of queries with the exact same circup (same laptop session) against my KeepAlive version of Web Workflow. The hostname.local mDNS query took an extra 10 seconds (15 vs 5) to do the circup operation but almost noise compared to 415 seconds without KeepAlive on the Web Workflow API.

Alternative solution

Circup could be refactored -- to catch mDNS hostnames and resolve them once -- but doing that would only solve circup's slowdown with multiple mDNS operations. The next person's utility would behave slowly for operations via a mDNS hostname and fine for bare IP or DNS hostnames and they'd never know why without reading this issue.

circup running on current 10.3.0 CP without KeepAlive

Mac:repos bob$ host racksen.lan
racksen.lan has address 192.168.5.90
Mac:repos bob$ dns-sd -q racksen.local
DATE: ---Sun 13 Sep 2026---
12:54:53.571  ...STARTING...
Timestamp     A/R  Flags         IF  Name                          Type   Class  Rdata
12:54:53.805  Add  2             14  racksen.local.                Addr   IN     192.168.5.90

Mac:repos bob$ time circup --host racksen.local update
Found device unexpectedmaker_tinys2 at http://:somepass@racksen.local:80, running CircuitPython 10.3.0.
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	6m55.303s
user	0m0.443s
sys	0m0.177s

Mac:repos bob$ time circup --host racksen.local update --all # Rerun in case all made a diff
Found device unexpectedmaker_tinys2 at http://:somepass@racksen.local:80, running CircuitPython 10.3.0.
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	6m53.507s
user	0m0.362s
sys	0m0.165s

Mac:repos bob$ time circup --host 192.168.5.90 update --all
Found device unexpectedmaker_tinys2 at http://:somepass@192.168.5.90:80, running CircuitPython 10.3.0.
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	0m7.657s
user	0m0.348s
sys	0m0.140s

Mac:repos bob$ time circup --host racksen.lan update --all
Found device unexpectedmaker_tinys2 at http://:somepass@racksen.lan:80, running CircuitPython 10.3.0.
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	0m7.550s
user	0m0.363s
sys	0m0.145s

circup (same client) using KeepAlive Circuit Python

Mac:repos bob$ host provtest.lan
provtest.lan has address 192.168.5.215
Mac:repos bob$ time circup --host provtest.local update --all
Found device unexpectedmaker_tinys2 at http://:somepass@provtest.local:80, running CircuitPython 10.3.0-51-gcf1b161e03.
A newer version of CircuitPython (10.3.0) is available.
Get it here: https://circuitpython.org/board/unexpectedmaker_tinys2
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	0m16.106s
user	0m0.466s
sys	0m0.183s
Mac:repos bob$ time circup --host provtest.lan update --all
Found device unexpectedmaker_tinys2 at http://:somepass@provtest.lan:80, running CircuitPython 10.3.0-51-gcf1b161e03.
A newer version of CircuitPython (10.3.0) is available.
Get it here: https://circuitpython.org/board/unexpectedmaker_tinys2
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	0m5.353s
user	0m0.452s
sys	0m0.151s
Mac:repos bob$ time circup --host 192.168.5.215 update --all
Found device unexpectedmaker_tinys2 at http://:somepass@192.168.5.215:80, running CircuitPython 10.3.0-51-gcf1b161e03.
A newer version of CircuitPython (10.3.0) is available.
Get it here: https://circuitpython.org/board/unexpectedmaker_tinys2
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
None of the module[s] found on the device need an update.

real	0m5.302s
user	0m0.451s
sys	0m0.148s

mDNS query time measurement

Mac:repos bob$ for i in $(seq 10); do /usr/bin/time -p python3 -c "import socket; socket.getaddrinfo('racksen.local', 80)" 2>&1 | awk '/real/{print $2}'; done
3.49
5.05
5.05
5.05
5.05
5.05
5.04
5.05
5.05
5.05

[tannewt]

Since CP workflow doesn't respond with a NSEC negative response for the AAAA portion it takes a 5s timeout. Since likely the mDNS is in the ESP-IDF code it may be more difficult to fix it there.

We could fix this in the IDF. We have a forked version of the IDF under adafruit that we use in CP.

[grgrant]

@tannewt I'm testing a NSEC fix for the Adafruit fork of esp-protocols/.../mdns.c. Some notes (I know you're aware since you're the author of the changes to the fork):

• The upstream esp-protocols repo is way ahead of ours 1.13.0 and is a refactor of mdns.c into many separate files
• The maintainer of that repo david-cermak had actually created a PR #263 to fix Issue #258 in 2023 but never was merged into main. It is unknown why because no notes I could find said why it stalled and was closed.
• Claude and I backported the david-cermak code to our 1.4.0 mdns.c and Claude also found an uninitialized variable issue and record_length bug and fixed it.

I've verified a speedup of 1m23s to 2.5s by using the new NSEC version. I plan to put this version on a number of my continuously running sensors for a longer term test.

Would you like this it as a PR to the CircuitPython branch to carry now, and/or should I push the tested revival upstream to Cermak so we can drop the carry later if we sync back to upstream at some point. And if to our local should it be in the circuitpython repo or on the Adafruit fork of esp-protocols -- sorry I don't know the nuances of that.

Here are the two runs

10.3.1

Mac:board_inventory bob$ time circup --host wss3-1.local freeze
Found device waveshare_esp32_s3_zero at http://:somepass@wss3-1.local:80, running CircuitPython 10.3.1.
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
adafruit_connection_manager==3.1.8
adafruit_ntp==3.4.0
adafruit_pixelbuf==2.1.0
adafruit_requests==4.1.17
neopixel==6.4.2

real	1m23.228s
user	0m0.308s
sys	0m0.132s

10.3.1 plus NSEC code

Mac:board_inventory bob$ time circup --host wss3-2.local freeze
Found device waveshare_esp32_s3_zero at http://:sompass@wss3-2.local:80, running CircuitPython 10.3.1-dirty.
A newer version of CircuitPython (10.3.1) is available.
Get it here: https://circuitpython.org/board/waveshare_esp32_s3_zero
Using latest bundle for adafruit/Adafruit_CircuitPython_Bundle (20260910).
Using latest bundle for adafruit/CircuitPython_Community_Bundle (20260911).
adafruit_connection_manager==3.1.8
adafruit_ntp==3.4.0
adafruit_pixelbuf==2.1.0
adafruit_requests==4.1.17
neopixel==6.4.2

real	0m2.475s
user	0m0.266s
sys	0m0.113s

[tannewt]

I'm glad it is working better!

I'd definitely like to get us up to date on esp-protocols. Any idea if our one hostname change is fixed?

Maybe it is easiest for me to rebase adafruit/esp-protocols with my one change onto upstream and update CP to it. Then you can PR any additional fixes there.