NAS-144436 / 28.0.0-BETA.1 / truenas-appliance-runner: skip GRUB's menu, and show where a boot's time goes

[william-gr]

A follow-up to #21. The appliance took 80+ s to answer its API; this makes the boot say where its time goes and takes 10 s off it.

• Skip GRUB's menu. `start` presses Enter at GRUB over QMP (`send-key ret`) instead of waiting out its 10 s timeout. The kernel starts at ~5 s instead of ~14 s.
• Milestones on one clock. `tnctl wait --since` logs, in seconds since qemu started: the port accepting, the websocket answering, the login, `system.ready`, and when the guest's kernel started.
• A timestamped console. `tnctl tail-ts` copies the serial console to `console.ts.log` with each line stamped on that clock. The action prints it in a folded group after the boot, and `console` prefers it.
• `truenas-appliance boot-report`. Over ssh with a throwaway key: `systemd-analyze` (the critical chain and blame), this boot's middleware log, the journal's and dmesg's pauses, the kernel command line, the clocksource, and CPU steal and iowait in the appliance and on the runner.
• `enable-ssh --pubkey` for that key.

What it found, across three image builds: there is no steal at either level. The appliance uses about one core for the whole boot, so the time is serial work made slower by nested virtualization. The details are in iXsystems/github-truenas-runner's `docs/e2e-appliances.md`, whose boot test runs `boot-report`.

Tested: `test_tnctl.py` and shellcheck in CI. Three appliance image builds ran these scripts from this branch: 38293612187, 38296400372 and 38298978367.

[bugclerk]

Jira URL: https://ixsystems.atlassian.net/browse/NAS-144436

[github-actions]

LOW: The except here catches RPCError, KeyError, TypeError, ValueError, but RPC.call can also raise Closed or TimeoutError (lines 102/137/179). By this point system.ready has already succeeded. If the socket drops or system.info doesn't answer within 120 s, the exception escapes cmd_wait, start reports "the appliance's API never came up", and the boot fails because a diagnostic line failed. I can't name a realistic trigger, since the same socket has just answered, so this is LOW. Catching Exception (or adding Closed, OSError) would keep the uptime line from ever failing the boot.

[github-actions]

One finding: one LOW. No human review is needed.

• LOW: tnctl.py:315. The new uptime lookup in cmd_wait --since doesn't catch Closed/TimeoutError. If system.info fails after system.ready has already succeeded, start dies with "API never came up" because a diagnostic failed. I can't name a realistic trigger (inline).

What else I checked and found sound:

• The two new background jobs in start (tail-ts under nohup, and skip_grub's subshell) send their output to /dev/null. So the action's out="$(appliance.sh start)" doesn't wait on them.
• skip_grub exits once it has sent Enter, or after 60 s.
• tail-ts stops when qemu's pid goes away.
• start runs rm -rf "$DIR", so console.ts.log never carries lines from an earlier boot into a later one.
• revert calls wait without --since, so its behaviour is unchanged apart from the probe interval going from 2 s to 1 s.
• boot-report's side effects (ssh, a key, passwordless sudo for the admin) are documented in the README and the script header, and a revert to the baseline snapshot undoes them.
• The new tail-ts and the extended wait test assert the milestone order and the cleaned, timed lines.

[github-actions]

Approved. 1 finding(s): 1 LOW.

Nothing blocks and no human review is needed.

[bugclerk]

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.