Vela 2.0 0.3B on Core ML: Swift engine, model store, parity check, guardrail + chatbot front-door demos

[Alex-Wengg]

What

Adds vLLM Semantic Router × KR Labs' Vela-2.0-0.3B (ModernBERT routing / safety / span model) on Core ML:

• Vela2Manager: a Swift port of the release's vela2_inference.py on a Core ML encoder. It covers the tokenizer with character offsets (a mmBERT/Gemma BPE built on LayaTokenizer), schema assembly, the fp32 readout heads (choice cosine + MLP, word × label spans), calibration (per-type temperatures, PII length rule + sparse gate) and the span decoder. Word units, span units and trimming are hand-written scanners using Python's \w / \s rules, because ICU's differ for combining marks, No digits and similar. Choice questions plus the first span question share one encoder pass. Sequences of ≤128 tokens run on the Neural Engine, longer ones on the GPU.
• Vela2ModelStore: pinned and checksummed FluidInference/vela-2.0-0.3b-coreml @ fb68b856 (fp16 multifunction encoder L128–L1024, heads, tokenizer, calibration, licences).
• Vela2Check parity: compares against fixtures from the Python engine.
• GuardrailDemo: a chat guardrail. Each outgoing message is screened (prompt attack, harm, route, fact-check) and its PII masked. Replies are checked against a source document, and unsupported claims are underlined. Each check shows whether it ran on the ANE or GPU and how long it took. --demo plays the scenarios once.
• FrontDoorDemo: 1,000 synthetic chatbot messages arrive in an inbox and fly into Answered (routed to a team, PII redacted), Blocked · jailbreak and Blocked · harmful. Includes Pause/Resume. The data is synthetic, from a seeded generator kept in model-lab.
• README section.

Verification (M5 Pro, macOS 27)

• Parity, Swift vs the Python engine, on 38 requests (PII, attack, harm, routing, hallucination; EN/DE/FR/ES/ZH/JA/HI/AR, URLs, e-mails):
  • 0 token / sequence mismatches
  • 0 / 120 choices differ (max |Δp| 0.00000)
  • 0 / 38 span sets differ (GPU). With the ANE for L128, choices are identical and span probabilities are within 0.003.
• Core ML encoder vs PyTorch: identical choices and span sets on 8 requests.
• Encoder timing: L128 3.5 ms on the ANE / 4.4 ms on the GPU; L256 5.0, L512 8.3, L1024 16.5 ms on the GPU. 99.4 % of ops are placed on the ANE, but it is only faster up to 128 tokens.
• FrontDoorDemo: 1,000 messages at ~90 msg/s end to end with the UI open (guard ~3.5 ms on the ANE, route + PII ~8.5 ms on the GPU).
  • harmful blocked 150/150
  • jailbreak 105/150
  • 16/700 benign false-blocked
  • topic 578/700 vs the generator's labels
• Model store: fresh Vela2ModelStore.ensure() download, all 8 assets checksum-verified, and the demo runs from the cache.

Not done

• No iOS run.
• swift test was not run locally (no Xcode).
• The detail sheets in both demos have not been checked by hand.
• Overlaps PR Decision 2.0 (Kai 0.6B, Eos 0.8B) on Core ML: runtime, model store, parity check, issue-triage demo #25 in Package.swift and README.md; whichever merges second needs a small rebase.

🤖 Generated with Claude Code