Repository navigation
Decision 2.0 (Kai 0.6B, Eos 0.8B) on Core ML: runtime, model store, parity check, issue-triage demo - #25
Merged
Conversation
…arity check
Decision2Manager answers every question of a request in one Core ML call (shared
prefix + packed question suffixes) for both the Qwen3 (Kai) and Qwen3.5 hybrid (Eos)
packages published at FluidInference/decision-2.0-{kai,eos}-coreml.
Decision2ModelStore pins and checksums both snapshots.
Decision2Check parity on typed-decisions TEST (400 requests, 2,000 decisions):
0 token mismatches vs the upstream encoder, identical probabilities to the Python
Core ML runtime, 5 (Kai) / 4 (Eos) near-tie flips vs upstream PyTorch.
Card example: Kai 16.5 ms, Eos 35.7 ms per 3-question request on an M5 Pro.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-0.6B; README section SwiftUI demo: a GitHub-style issue list (shown as a fictional repo with made-up author handles) labeled live by Kai, five decisions per issue in one Core ML call (type, owning workgroup, priority, needs-info, good first issue). Issues come from fetch-issues.sh (gh CLI) into the user's cache, never the repository. 67.6 ms per issue, 14.8 issues/s on an M5 Pro; labels identical to the Python Core ML reference on every compared issue. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s, 2-10 score levels); empty request returns no answers instead of crashing Review fixes for #25. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Alex-Wengg
added a commit
that referenced
this pull request
Oct 7, 2026
# Conflicts: # Package.swift # README.md
Alex-Wengg
added a commit
that referenced
this pull request
Oct 7, 2026
…ardrail + chatbot front-door demos (#26) ## What Adds vLLM Semantic Router × KR Labs' [Vela-2.0-0.3B](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) (ModernBERT routing / safety / span model) on Core ML: - **`Vela2Manager`**: a Swift port of the release's `vela2_inference.py` on a Core ML encoder. It covers the tokenizer with character offsets (a mmBERT/Gemma BPE built on `LayaTokenizer`), schema assembly, the fp32 readout heads (choice cosine + MLP, word × label spans), calibration (per-type temperatures, PII length rule + sparse gate) and the span decoder. Word units, span units and trimming are hand-written scanners using Python's `\w` / `\s` rules, because ICU's differ for combining marks, `No` digits and similar. Choice questions plus the first span question share one encoder pass. Sequences of ≤128 tokens run on the Neural Engine, longer ones on the GPU. - **`Vela2ModelStore`**: pinned and checksummed [FluidInference/vela-2.0-0.3b-coreml](https://huggingface.co/FluidInference/vela-2.0-0.3b-coreml) @ `fb68b856` (fp16 multifunction encoder L128–L1024, heads, tokenizer, calibration, licences). - **`Vela2Check parity`**: compares against fixtures from the Python engine. - **`GuardrailDemo`**: a chat guardrail. Each outgoing message is screened (prompt attack, harm, route, fact-check) and its PII masked. Replies are checked against a source document, and unsupported claims are underlined. Each check shows whether it ran on the ANE or GPU and how long it took. `--demo` plays the scenarios once. - **`FrontDoorDemo`**: 1,000 synthetic chatbot messages arrive in an inbox and fly into Answered (routed to a team, PII redacted), Blocked · jailbreak and Blocked · harmful. Includes Pause/Resume. The data is synthetic, from a seeded generator kept in model-lab. - README section. ## Verification (M5 Pro, macOS 27) - **Parity, Swift vs the Python engine, on 38 requests** (PII, attack, harm, routing, hallucination; EN/DE/FR/ES/ZH/JA/HI/AR, URLs, e-mails): - 0 token / sequence mismatches - 0 / 120 choices differ (max |Δp| 0.00000) - 0 / 38 span sets differ (GPU). With the ANE for L128, choices are identical and span probabilities are within 0.003. - **Core ML encoder vs PyTorch:** identical choices and span sets on 8 requests. - **Encoder timing:** L128 3.5 ms on the ANE / 4.4 ms on the GPU; L256 5.0, L512 8.3, L1024 16.5 ms on the GPU. 99.4 % of ops are placed on the ANE, but it is only faster up to 128 tokens. - **FrontDoorDemo:** 1,000 messages at ~90 msg/s end to end with the UI open (guard ~3.5 ms on the ANE, route + PII ~8.5 ms on the GPU). - harmful blocked 150/150 - jailbreak 105/150 - 16/700 benign false-blocked - topic 578/700 vs the generator's labels - **Model store:** fresh `Vela2ModelStore.ensure()` download, all 8 assets checksum-verified, and the demo runs from the cache. ## Not done - No iOS run. - `swift test` was not run locally (no Xcode). - The detail sheets in both demos have not been checked by hand. - Overlaps PR #25 in `Package.swift` and `README.md`; whichever merges second needs a small rebase. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds vLLM Semantic Router's Decision 2.0 models to FluidUse, running on the GPU through Core ML:
Decision2Manager: one manager for both packages, Kai 0.6B (Qwen3) and Eos 0.8B (Qwen3.5 hybrid). A request is a state (text or JSON) and any number of named choice / yes-no / score questions, all answered in one call. The shared prefix runs once, and each question's suffix continues from it exactly as in its own upstream row. Long requests are split into chunks that each repeat the prefix, on the function with the lowest measured cost. Kai's packaged Score offsets are applied.Decision2ModelStore:.ensure(.kai)/.ensure(.eos)download pinned, checksummed snapshots from FluidInference/decision-2.0-kai-coreml (73344136, 1.1 GB) and FluidInference/decision-2.0-eos-coreml (7bf83236, 1.4 GB).Decision2Check:parity <model dir> <fixtures.json>against the Python reference, andexample kai|eos.IssueTriageDemo: a SwiftUI demo that labels 1,000 GitHub issues live (type, owning workgroup, priority, needs-info, good first issue), with five decisions per issue in one call. The issues come fromfetch-issues.sh(gh CLI) into the user's cache, never into the repo. The UI shows a fictional repo name and made-up author handles.Verification (M5 Pro 24 GB, macOS 27)
Decision2Check parityon typed-decisions TEST (LocalLLaMA/typed-decisions, 400 requests, 2,000 decisions):All flips are near-ties (upstream top-2 margin ≤ 0.009). On the card example (3 questions), Kai takes 16.5 ms and Eos 35.7 ms in steady state;
warm()runs largest-first, so the first request is not penalized. A freshDecision2ModelStore.ensuredownload and the example run passed for both models. IssueTriageDemo ran 1,000 issues at 67.6 ms/issue (14.8 issues/s), with labels identical to the Python reference on every compared issue.Not done: no iOS run (the API is
@available(macOS 15, iOS 18)); the detail sheet in IssueTriageDemo (click an issue) has not been opened in a test run;swift testwas not run locally (no Xcode here).🤖 Generated with Claude Code