Repository navigation

Decompose _tokenize_chat_view into a staged _ChatViewTokenizer - #997

Merged

bradhilton merged 12 commits into

mainfrom

refactor/tokenize-chat-view

Sep 27, 2026

Merged

Decompose _tokenize_chat_view into a staged _ChatViewTokenizer#997
bradhilton merged 12 commits into
mainfrom
refactor/tokenize-chat-view

Commits on Sep 27, 2026
- Move _tokenize_chat_view into a _ChatViewTokenizer class
  bradhilton
  committed
  89dbb65
- Extract _ChatViewTokenizer._render_messages
  bradhilton
  committed
  ecdce4e
- Extract _ChatViewTokenizer._render_canonical_masks
  bradhilton
  committed
  c4379bd
- Extract _ChatViewTokenizer._substitute_exact_prefix
  bradhilton
  committed
  2f813c7
- Extract _ChatViewTokenizer._translate_masks
  bradhilton
  committed
  f8b4d2b
- Extract _ChatViewTokenizer._prepare_span_search
  bradhilton
  committed
  5eb451f
- Extract _ChatViewTokenizer._prove_marked_bounds
  bradhilton
  committed
  f01e1a3
- Extract _ChatViewTokenizer._prove_probed_bounds
  bradhilton
  committed
  e22eca8
- Extract _ChatViewTokenizer._tokenize_exact_length_stops
  bradhilton
  committed
  cb838b8
- Extract _ChatViewTokenizer._collect_replacements
  bradhilton
  committed
  2779a64
- Extract _ChatViewTokenizer._assemble
  bradhilton
  committed
  3c3d1a6
- Break the _ChatViewTokenizer render-cache reference cycle
  bradhilton
  committed
  b9f9da8