perf(package): strip comments from the Apple runner source the npm package ships

[thymikee]

Summary

apple/runner/** ships to npm as Swift source, and the packager rewrote it only to remove
#if AGENT_DEVICE_RUNNER_UNIT_TESTS blocks — so every doc comment and design note was downloaded
on every install. scripts/package-apple-runner-source.mjs now also strips comments, through a new
scripts/strip-swift-comments.mjs.

A regex cannot do this safely: // and /* open a comment only in code position, a raw literal
(#"…"#) moves its own closing delimiter and interpolation opener with the # count, interpolation
segments hold code including further literals, extended regex literals (#/…/#) are a second
delimiter family that also opens on a # run, and Swift block comments nest. The module is a
lexical scanner over those states; a construct it cannot account for throws at packaging time
rather than shipping Swift that does not compile. A line whose only content was a comment
disappears; blank lines and every byte inside a literal survive. The unit-test block strip is
unchanged and still runs first, so which blocks it removes does not depend on comment removal.
5 files touched.

Bare /…/ regex literals are the one construct no scanner can resolve — the same / opens a
comment, divides, and starts a regex literal, and only the parse separates them. Where one could
begin (an expression position whose / is not followed by a space, a tab or )), packaging throws
by file and line instead of rewriting bytes it cannot prove are code. Divisions (width/2,
Double(3)/Double(4)), the recording scripts' #! shebang and (/) flow through untouched.

dist/apple/runner/ 555,907 B -> 484,009 B (-71,898 B, -12.9%); its Swift alone 441,196 B ->
369,298 B (-16.3%). Closes #2461.

Validation

Tested commit 9770a12.

• pnpm format, pnpm lint, pnpm typecheck, pnpm check:affected --run (which runs
  check:fallow --base origin/main), pnpm check:xctest-selection: all pass.
• Packaged runner builds: xcodebuild build-for-testing on dist/apple/runner/.../AgentDeviceRunner.xcodeproj
  reports ** TEST BUILD SUCCEEDED ** for both generic/platform=iOS Simulator and
  platform=macOS,arch=arm64. The two runtime-compiled recording scripts compile with
  xcrun swiftc (0 errors), and all 44 packaged Swift files pass xcrun swiftc -parse.
• Re-running the scanner over its own output removes 0 further comments in all 44 files.
• Every Swift snippet in the regex-literal regressions was checked against xcrun swiftc -parse
  (Swift 6.2.3) both as written and as the scanner leaves it, so the fixtures are valid Swift
  rather than a guess at the grammar.
• No device run: packaging-only change, and the packaged Swift is unchanged apart from comments.
  No xcodebuild test was run. Confirm the Bundle Size job for the one-time drop.

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.53 MB	4.46 MB	-74.3 kB
Package (unpacked)	4.53 MB	4.46 MB	-74.3 kB
Package (download)	1.34 MB	1.31 MB	-30.9 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	28.3 ms	28.1 ms	-0.2 ms
CLI --help	80.2 ms	76.1 ms	-4.2 ms

[thymikee]

One packaging safety gap at d95e6ab: valid Swift extended regex literals are silently rewritten. let pattern = #/foo//bar/# becomes let pattern = #/foo, because the scanner treats the regex contents as a line comment. Please preserve regex literals or reject unsupported regex syntax before rewriting, and add this regression. The current runner corpus has no such literals, so the existing compile evidence is consistent with this gap; the claimed fail-closed behavior still needs fixing.

[thymikee]

You were right, and the characterisation was exact: at d95e6ab the scanner only knew the #"
raw-string family, so #/foo//bar/# lost everything from its first inner // and shipped
let pattern = #/foo. Fixed in 9770a12.

What changed in scripts/strip-swift-comments.mjs

• Extended regex literals are now their own frame kind, opened by #+ followed by /. This is the
  #-run discrimination you flagged: # + quote is a raw string literal, # + / is a regex
  literal, and # + anything else stays a directive (#if, #available, the recording scripts'
  #!). The # count sets the terminator, so ##/…/## closes only on /##.
• Both forms are handled. A newline straight after the opener selects Swift's multi-line form,
  whose closing delimiter has to start its own line; the single-line form throws
  Unterminated regex literal at a newline instead of scanning on. \/ is copied as a pair, so an
  escaped slash cannot close the literal early.
• Bare /…/ takes your second option. A / in code position that could open one — an expression
  position, and not followed by a space, a tab, a newline or ), which Swift itself bars a regex
  literal from opening on — now throws
  Ambiguous bare regex literal or division in <file>:<line>, naming the file and line and
  suggesting #/…/# or a spaced operator. Nothing is guessed at. Expression position is decided
  from the emitted tail: a / after an operand (identifier, ), ], ", backtick, ?, !)
  divides, except after a keyword that can precede an expression, so return /x/ is refused while
  width/2 is not.
• scanCodeCharacter's slash handling moved into consumeSlash to stay under the fallow
  complexity threshold.

What the new regressions prove

scripts/__tests__/strip-swift-comments.test.ts (18 -> 25 tests) and
src/__tests__/apple-runner-package-source.test.ts (+1):

Regression	Before this commit
#/foo//bar/# survives byte for byte	shipped let pattern = #/foo
##/a//b/#c/## survives (the #-count terminator)	shipped let pounded = ##/a
multi-line #/ … /# holding //, /*e*/, a\/#b and a blank line survives	shipped the literal with two lines gutted
#/x/*y/# survives	threw Unterminated block comment — a false packaging failure
#/a//b/# // trailing keeps the literal, drops only the real comment	shipped let trailing = #/a
/foo//bar/ and return /x\/y/ throw, naming file and line	silently shipped let pattern = /foo
width/2, Double(3)/Double(4), width / 2, #!/usr/bin/env swift, (/) pass through	(unchanged — these are the no-false-fail-close side)
unterminated #/… and a multi-line literal closing mid-line throw	n/a

Every Swift snippet in those tests was checked against xcrun swiftc -parse (Swift 6.2.3) both as
written and as the scanner leaves it, so the grammar is measured rather than assumed. Two things
that came out of that and are encoded in the scanner: Swift honours \/ inside extended
delimiters (#/a\/#b/# parses), and it closes a multi-line literal at the first unescaped
delimiter but then requires it to start its own line — so a mid-line /# is a file that does not
compile either way, and packaging refuses it rather than stripping around it.

Re-validated on 9770a12

• pnpm format, pnpm lint, pnpm typecheck, pnpm check:affected --run (including
  check:fallow --base origin/main), pnpm check:xctest-selection: all pass.
• xcodebuild build-for-testing on the packaged project: ** TEST BUILD SUCCEEDED ** for both
  generic/platform=iOS Simulator and platform=macOS,arch=arm64. Build only — no xcodebuild test
  and no device or simulator run.
• All 44 packaged Swift files pass xcrun swiftc -parse; both runtime-compiled recording scripts
  compile with xcrun swiftc (0 errors); re-running the scanner over its own output removes 0
  further comments.
• You were also right that the corpus has no such literals, so the size numbers are unchanged and
  the PR body still holds: dist/apple/runner/ 555,907 B -> 484,009 B (-71,898 B, -12.9%), Swift
  441,196 B -> 369,298 B (-16.3%), 891 comments stripped. The compile evidence now rests on a
  scanner that handles the construct rather than on the corpus not containing it.

[thymikee]

The regex corruption is fixed at 9770a12: the original failing literal now survives unchanged, and the new tests cover delimiters, escapes and unsupported bare-regex syntax. No remaining code findings. The reported packaged iOS/macOS builds pass; marking ready for human review while the remaining smoke check finishes.

[thymikee]

Before this merges — I want to challenge the size of it, with measurements, because I think ~95% of the machinery is buying ~0.6% of the bytes.

I measured the shipped corpus (apple/runner/**/*.swift, 44 files):

construct the scanner handles	occurrences in the corpus
whole-line // comments	1,351 lines / 109,378 B
trailing // comments	~652 B (crude upper bound)
block comments /* */	0
extended regex literals #/…/#	0
multi-line string literals """	4, in 2 files
multi-line raw literals #"""	0

So 99.4% of the comment bytes are whole-line //, and the constructs that motivated most of the 425-line scanner — nested block comments, extended regex literals, the bare-/ expression-position heuristic with its 128-character lookbehind — cover cases that do not occur at all here.

A much smaller rule gets essentially the same saving: delete lines whose trimmed text starts with //, while tracking only """ literals so a line inside one is never deleted. That is one construct instead of six, roughly 30 lines instead of 425, and it never has to decide whether a / is a comment, a division or a regex — which is exactly where the corruption you caught lives. I checked the one hazard it would still have: zero whole-line // currently sit inside a multi-line literal.

What that trades away is real but small: the ~652 B of trailing comments stay, and doc comments on the same line as code stay. Call it 108.7 kB recovered instead of 109.4 kB.

The reason I think this matters more than the byte count: the current design can fail a publish on valid Swift. The bare-slash heuristic decides expression position from a 128-char lookbehind, and if it guesses wrong on some future file it throws rather than shipping. A rule that only ever deletes a line that is entirely a comment cannot corrupt code and cannot block a release — worst case it leaves a comment in.

Two smaller points, whichever way you go:

• If you keep the full scanner, consider making the bare-/ case a per-file bail-out (ship that file verbatim, comments intact) instead of a hard packaging failure. Same safety, no publish-blocking false positive.
• The 30.9 kB download / 4.2 ms --help win is real and I am not arguing against doing this at all — only against hand-rolling six constructs' worth of Swift grammar at publish time to get the last 0.6%.

Happy to cut the smaller version if you agree; I did not want to rewrite an approved PR on my own initiative.

[thymikee]

The current code verdict is unchanged, and checks are green. The smaller approach is worth considering, but tracking only triple-quoted strings is not enough to guarantee safety: a line starting with // can also be content inside a multiline regex literal. Before replacing the scanner, define the supported input and prove that other files are preserved unchanged. This is a design choice before merging, not a new failure in the reviewed head.