Full-message tokenization for droppedTokens telemetry blocks OpenCode for 46–77 seconds #448

Summary

Magic Context synchronously serializes and tokenizes the entire transformed message array twice to calculate droppedTokens telemetry. In a running OpenCode Desktop session, direct timing around these two calls measured approximately 77 seconds of tokenization in one transform and 46 seconds in another. The sidecar event-loop diagnostics recorded matching stalls.

This path remains present in the published 0.42.3 npm bundle. The runtime measurements below are from 0.42.0, with the previously reported SQL-reader/marker corrections applied locally and a temporary timing probe. This is separate from those SQL issues.

Expected behavior

Calculating a diagnostic metric should not synchronously block the shared OpenCode backend for tens of seconds. Exact token-budget enforcement should remain separate from inexpensive telemetry estimation.

Direct measurements

Captured September 14, 2026. Times below are EDT (UTC−4); each timestamp is the end of the measured call.

The first pair spent 77,057.84 ms inside the tokenizer, corresponding to a 77,443.63 ms sidecar event-loop stall reported at 12:52:02.268. The second pair spent 45,680.13 ms inside the tokenizer, corresponding to a 45,868.91 ms stall reported at 12:53:31.841. Small intervening cleanup operations account for much of the remaining interval.

The probe calls JSON.stringify once and the original estimateTokens once, measuring each with performance.now(). It preserves the original result and exceptions. It logs only slow calls (at least 250 ms), at most 20 records, with durations/counts only and no message content. These measurements localize the time to the synchronous estimateTokens call, including any GC occurring inside that interval; they do not distinguish individual tokenizer internals.

Verified published code

Package: @cortexkit/opencode-magic-context@0.42.3, downloaded from npm on September 14 without installing or executing it.

dist/index.js:33755–33756:

let droppedTokens = 0;
const tokensBeforeReductions = isCacheBustingPass || shouldApplyPendingOps
  ? estimateTokens(JSON.stringify(args.messages))
  : 0;

dist/index.js:33967–33968:

if (isCacheBustingPass) {
  droppedTokens = Math.max(0,
    tokensBeforeReductions - estimateTokens(JSON.stringify(args.messages)));
}

dist/index-wde5446v.js:6319–6332 still implements estimateTokens using:

activeTokenizer.encode(text, "all").length

The existing cheap fallback is estimateTokensHeuristically(text), which returns Math.ceil(text.length / 3.5). It is used only when the tokenizer is unavailable or fails, not to bound telemetry work.

The before/after delta is returned as droppedTokens, passed to recordPendingTransformDecision, and persisted as transform_decisions.dropped_tokens. These particular counts do not drive context-budget or scheduling decisions. Other similarly named values, such as undroppedTokens, are distinct and should not be changed as part of this fix.

Proposed narrow correction

Use the existing character-based estimate for both telemetry-only before/after counts, rather than invoking the exact tokenizer on the full serialized message array. Preserve the nonnegative delta calculation and leave estimateTokens and its real token-budget callers unchanged.

This deliberately changes the accuracy of dropped_tokens telemetry; it is not an exact-count-preserving optimization. Document that field as estimated. Serialization can remain for this narrow correction: it measured only 10–18 ms here, versus 23–43 seconds per tokenization call.

Suggested regression coverage:

• Exercise the real transform path with a multi-megabyte message array and pending operations/cache-busting enabled.
• Assert that these telemetry sites do not invoke the exact tokenizer.
• Verify that the emitted delta is nonnegative and uses the inexpensive estimate consistently before and after reductions.
• Verify unchanged transformed messages, applied operations, and actual token-budget decisions.
• Measure event-loop responsiveness outside the transform as well as elapsed telemetry work.