Repository navigation
核实并修复优化分析报告中的问题：流式解析、Serialization 写入、编码器并发与正确性 - #11
Merged
Merged
Conversation
ReerJSONEncoder.encode read the stored `options` without taking optionsLock, while every setter mutates it under the lock. Concurrent configuration changes could therefore race with encode and crash with a use-after-free of the userInfo dictionary. Take one locked snapshot at the start of encode (matching ReerJSONDecoder) and use it for both the encoder implementation and the output formatting flags. Co-authored-by: Cursor <cursoragent@cursor.com>
JSONStreamParser handed yyjson every byte from the read offset to the end of the buffer for each item. In non-INSITU mode yyjson copies its whole input, so K buffered items cost O(K * N) bytes of memcpy and the parse time grew quadratically (16000 NDJSON items: 120 ms). Parse a window sized from the previous item (at least 256 bytes) and double it only when the window cannot decide the value by itself: incomplete input, a value ending exactly at the window end, or an error. Once the window covers all available bytes the parser behaves exactly as before, so boundary deferral and error reporting are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
ReerJSONSerialization.data(withJSONObject:) was 11-60x slower than JSONSerialization. Profiling showed three causes: - yyjson_mut_obj_put scans the whole object on every insert, making wide dictionaries O(n^2). NSDictionary keys are unique, so obj_add is safe. - isValidJSONObject walked the full object graph before conversion walked it again. Validation now happens during conversion and still throws the same "Invalid JSON object" error. - Most of the remaining time was Swift dynamic casts from Any and NSString → String bridging copies. On Darwin, values are now dispatched with class checks on AnyObject, containers are read with CFDictionaryGetKeysAndValues / CFArrayGetValues, and strings are copied straight from CFString storage. Linux keeps the Any-based dispatch. Co-authored-by: Cursor <cursoragent@cursor.com>
SwiftPM warned about Info.plist, Models/instruments.json and Models/json_to_swift.rb on every build. None of them are loaded by the tests, so exclude them instead of bundling them as resources. Co-authored-by: Cursor <cursoragent@cursor.com>
Keyed containers insert with yyjson_mut_obj_put so that re-encoding a key replaces its value in place, but obj_put scans the whole object on every insert, making objects with many keys O(n^2) (2000 keys: 2.6 µs/key). Objects below 32 keys keep using obj_put. Once an object reaches 32 keys the encoder lazily builds a key set for it; new keys are appended with obj_add and only genuine duplicates fall back to obj_put, preserving the last-value-wins, original-position semantics. The index is shared with referencing encoders so superEncoder writes stay in sync. Co-authored-by: Cursor <cursoragent@cursor.com>
wrapStringKeyedDictValue restored codingPath only on the success path. If a value threw and the caller recovered (e.g. `try? container.encode`), the dictionary key stayed on the path, so later errors and .custom key strategies saw a wrong coding path ([dict, value] instead of [value], unlike Foundation). Pop the key in a defer. Co-authored-by: Cursor <cursoragent@cursor.com>
nestedContainer(keyedBy:forKey:) reused an existing object for the key,
but nestedUnkeyedContainer(forKey:) always inserted a new array, so a
second call for the same key replaced the first array and dropped its
elements ({"list":[2]} instead of Foundation's {"list":[1,2]}).
Look up and reuse an existing array like Foundation does.
The existing-value lookup scanned the object linearly, which made wide
objects built from nested containers O(n^2); objects with 32+ keys now
consult the key index first. nestedContainer and superEncoder also go
through the container's _key() helper instead of calling convertedKey
directly.
Co-authored-by: Cursor <cursoragent@cursor.com>
…rategy The .custom key encoding strategy was called with the encoder's current path plus the key, but nested keyed containers do not push their key onto the encoder, so converters saw ["leaf"] where Foundation passes ["outer", "inner", "leaf"]. Convert keys with the owning container's codingPath instead. Co-authored-by: Cursor <cursoragent@cursor.com>
yyjson reads an integer literal outside the Int64/UInt64 range as a double. integer() accepted any double exactly representable in T, so -9223372036854775809 rounded to -2^63 and silently decoded as Int64.min, while Foundation throws. Only accept the double for literals with a fraction or exponent; plain integer literals are parsed exactly from the raw text (which still covers Int128/UInt128). Co-authored-by: Cursor <cursoragent@cursor.com>
The decoder reads numbers as raw text (needed for Decimal, Int128 and
big integers), so every integer decode re-parsed its literal through the
general yyjson_read_number. Literals of the form -?[0-9]{1,18} (which
cannot overflow Int64) are now converted directly; everything else still
takes the existing path. decode [Int] 200k: 6.28 → 5.62 ms.
Co-authored-by: Cursor <cursoragent@cursor.com>
With .convertFromSnakeCase every keyed container converts all keys of the JSON object, including keys the type never reads. The Foundation algorithm used for this (split + lowercased + capitalized + joined) cost about 1 µs per key, so wide objects dominated decode time. Keys without inner underscores are now returned as is, and ASCII keys whose components are all letters or all digits are converted with byte operations. Other keys still use the original algorithm, which is kept as the reference and fuzz-tested against the fast path. The eager table is kept because snake-case conversion is not invertible, so requested keys cannot be mapped back reliably. Co-authored-by: Cursor <cursoragent@cursor.com>
Every decoded string was scanned for NUL bytes to choose between String(bytes:encoding:) and String(decoding:as:). For the valid UTF-8 yyjson hands back both produce the same String, embedded NULs included, so the scan only added an extra pass (decode [String] 100k x 8 B: 1.30 → 1.00 ms). stringWhenJSON5, which deliberately rejects NULs, is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
JSONValue/JSONObject/JSONArray.description produced JSON-looking text that did not parse back and lost data: - strings and keys were not escaped (a quote or newline broke the output) and were read with String(cString:), truncating at embedded NULs; - unsigned integers above Int64.max printed as negative numbers; - raw numbers (numberAsRaw / bigNumberAsRaw) printed as null; - non-finite doubles printed as inf/nan instead of Infinity/NaN. Render directly from the yyjson value into one byte buffer, keeping the existing ": " / ", " layout. This also removes the per-level [String] arrays and joins (638 KB document: 8.1 → 2.0 ms). Co-authored-by: Cursor <cursoragent@cursor.com>
…ocumentRef DocumentStorage and DocumentRef each carried a verbatim copy of the copying and in-place (INSITU + padding) parse logic, so a fix to one could silently miss the other. Move both into private helpers and keep only the ownership difference in the two types. Also drop the unused Span overload of DocumentRef.streamParse. No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>
Feeding a 300 KB item 7 bytes at a time re-parses it from the start on every call and took ~8 s; 997-byte chunks exercise the same boundaries. Co-authored-by: Cursor <cursoragent@cursor.com>
a3e08a1 routed every keyed-container insert through putKey, which reads yyjson_mut_obj_size on each call. That check is itself cheap, but on the common narrow-object path it is pure overhead: a fresh object never reaches the index, so every insert pays a size read and a branch for nothing. Re-encoding Twitter (its user objects carry ~40 keys, so they sit just above the old threshold of 32) regressed ~22%: 0.70 ms -> 0.87 ms. Count inserts in the container instead. Narrow objects stay on the plain yyjson_mut_obj_put fast path and never call putKey; only once a container has written threshold keys does it hand over to the shared index, which discovers the earlier keys lazily. Raise the threshold to 256 so that payloads in the tens-of-keys range never build a key set at all. Verified: Twitter re-encode back to 0.706/0.707/0.708 ms over three alternating rounds (main 0.701/0.720/0.712, pr-head 0.863/0.865/0.872). All 143 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
逐条核实了 DeepSeek harness 生成的《ReerJSON 优化分析报告》。每一项都先写测试确认问题是否真实存在，确认后再修复，一个修复一个提交；每个提交都附带修复前失败、修复后通过的回归测试（集中在
Tests/ReerJSONTests/RegressionTests.swift），以及修复前后的 A/B 性能数据。结论：
.custom键策略收到的 codingPath 不完整；-9223372036854775809会被静默解码成Int64.min。swiftlang/swift:nightly-6.4.x-noble验证了swift build，新增的回归测试在 Linux 上也全部通过。整体效果（
main对比本分支，交替运行 3 轮，取最小值）JSONStreamParser16000 条 NDJSON 一次喂入StreamingJSONLinesDecoder16000 条ReerJSONSerialization写 1000 键字典 ×10ReerJSONSerialization写 5000 行 × 6 键ReerJSONSerialization5000 行 sortedKeysJSONValue.description（638 KB）[Int]200k[String]100k × 8B[String: Int]验证方法
-O）。git worktree构建"修复前的提交"和"修复后"两个二进制，交替运行 3～7 轮，每个用例取所有轮次中的最小值，以消除约 ±5% 的运行间抖动。逐项结论
options未加锁obj_putO(n²)wrapStringKeyedDictValue的 codingPath 不恢复try?吞掉错误后，后续错误的路径是错的nestedUnkeyedContainer不复用已有数组_key()快路径.custom下多一次分配"不成立，两条路径分配相同.custom键策略的 codingPath 不完整NUMBER_AS_RAW二次解析Decimal精度约束，保持不变JSON.string的memchrdescription系列DocumentStorage/DocumentRef代码重复IncrementalParserO(N·K).sortedKeys整树深拷贝unboxInteger错误路径重复解析employee.bin/.DS_Store.gitignore忽略，从未进入 git各修复详情
1. C-1 编码器读取
options未加锁（a620811）确认：一个线程循环修改
userInfo/keyEncodingStrategy，另一个线程循环调用encode。独立复现程序在 release 构建下 5 次运行全部 SIGSEGV（exit 139），swift test -c release下测试进程 signal 11 崩溃。原因是encode无锁拷贝options，而 setter 在锁内替换userInfo字典，读线程用到了已经释放的旧字典。（TSan 在这个场景下没有报告，所以最终以崩溃复现为准。）修复：与
ReerJSONDecoder保持一致，新增optionsSnapshot()，在encode开头加锁取一次快照，并直接交给JSONEncoderImpl，outputFormatting也从同一份快照读取（顺带把每次 encode 的加锁次数从 3 次减到 1 次）。效果：复现程序 5/5 正常退出，测试 3/3 通过。性能持平：
2. P0-1 流式解析 O(K²)（aaf9695）
确认：复杂度测试中，条目数扩大 8 倍，耗时扩大 128～137 倍（jsonLines 和 jsonArray 两种模式都是）。
修复：没有采用报告建议的"切片 + INSITU"方案，而是用了更简单、语义完全不变的自适应窗口：每次只把
max(256, 上一条长度 × 2)字节交给 yyjson；如果窗口内无法确定结果（数据不完整、值恰好结束在窗口末尾，或者解析出错），就把窗口翻倍重试。当窗口覆盖整个剩余缓冲区时，行为与原实现完全一致，所以边界延迟产出、错误位置和错误信息都不变。新增了混合大小条目、分块喂入、数字恰好在缓冲区末尾、有效条目之后出现错误等测试。效果（单位 ms）：
3. P0-2 + C-6 ReerJSONSerialization 写路径（caddf5f）
确认：宽字典复杂度测试中，键数扩大 8 倍，耗时扩大 44 倍。实测问题比报告说的严重：6 键的普通行数据也比 Foundation 慢 11.6 倍（85 ms 对 7.3 ms）。用
sample采样后发现，主要热点不在 yyjson，而在 Swift 运行时对Any的动态类型转换（swift_conformsToProtocol、tryCast、dyld 协议一致性查找），以及 NSString 桥接成 String 时的拷贝。修复：
obj_put改为obj_add（NSDictionary 的键天然唯一）。isValidJSONObject预遍历；错误仍然是JSONError.invalidData("Invalid JSON object")，fragment 的 NaN 错误信息也保持原样。公开的isValidJSONObject(_:)行为不变。AnyObject，用 ObjC 类检查分派类型；容器用CFDictionaryGetKeysAndValues/CFArrayGetValues一次性取出全部元素；字符串优先走CFStringGetCStringPtr（仅 ASCII 存储时成功），否则用CFStringGetBytes写入栈上临时缓冲区；未配对代理项仍然退回到原来的 String 桥接，保证输出一致。Linux 上保留原来基于Any的分派逻辑。JSONSerialization的输出逐字节对比（包含非 ASCII、emoji、NSMutableString、嵌套、UInt64.max等）、各类非法对象的错误信息、未配对代理项。效果（单位 ms，×10 次）：
现在所有场景都快过 Foundation。另外，仓库自带的
SerializationMicroBenchmarkTests让 XCTest 总耗时从 11.3 秒降到 2.6 秒。4. C-13 Package.swift 警告（efa7385）
确认：每次构建都会输出
found 3 file(s) which are unhandled。这 3 个文件（Info.plist、instruments.json、json_to_swift.rb）都没有被任何测试引用。修复：加到
exclude里，而不是作为资源打包进测试 bundle。修复后构建没有这条警告。5. P2 编码器 keyed container 用
obj_put，O(n²)（a3e08a1）确认：复杂度测试中，键数扩大 8 倍，耗时扩大 59 倍。
修复：对象键数小于 32 时仍用
obj_put；达到 32 键后，为该对象懒建一个Set<String>键索引：新键走obj_add，只有真正重复时才回退到obj_put原地替换。这样保留了"后写覆盖、位置不变"的语义，也就是 skill 记录里提到的keyStrategyDuplicateKeys等 3 个测试依赖的行为。索引按需创建，窄对象不产生额外分配；referencing encoder（superEncoder）共享同一个索引。新增了 5/31/32/33/200 键下的重复键、superEncoder、"字典作为 single value 后再用 keyed container 追加"等测试。效果：
6. C-2 字典值抛错后 codingPath 残留（073e395）
确认：
encode(to:)里先try? container.encode(["inner": Thrower()], forKey: .dict)，再编码一个会抛错的值。ReerJSON 报告的错误路径是["dict", "value"]，Foundation 是["value"]。残留的路径还会传给.custom键策略。修复：改用
defer恢复路径，同时省掉一次数组拷贝。性能持平（[String: Int](1000)0.97x、[[String: String]]1.06x，都在噪声范围内）。7. C-4 + C-3
nestedUnkeyedContainer不复用已有数组（1447757）确认：对同一个 key 两次调用
nestedUnkeyedContainer并各写入一个元素，ReerJSON 输出{"list":[2]}，第一个数组的内容丢失；Foundation 输出{"list":[1,2]}。修复：与
nestedContainer一致，查找并复用同名的已有数组。另外，原来的"查找已有值"用的是yyjson_mut_obj_getn线性扫描，宽对象会退化成 O(n²)（测试中比值为 30）；现在对象达到 32 键后会先查键索引。C-3 提到的几处直接调用impl.convertedKey的地方也统一改为走_key()。8. 新发现：
.custom键策略收到的 codingPath 不完整（7ea4e78）确认：嵌套 container 里的键，Foundation 传给转换闭包的路径是
outer.inner.leaf、outer.leaf.0.leaf，ReerJSON 传入的只是leaf。原因是嵌套 container 不会把自己的 key 压入 encoder 的路径，而转换时用的是 encoder 的路径。按路径做键映射的用户会得到错误的结果。修复：改用 container 自己的
codingPath。开销不变（Payload.custom×10：95.0 → 93.8 ms）。9. 新发现：超出 64 位的整数被静默截断（5c9d699）
确认：在写 P1 的整数对照测试时发现，
-9223372036854775809解码为Int/Int64时没有报错，而是静默得到Int64.min；Foundation 会抛错。原因是 yyjson 把超出范围的整数读成 double（-2^63），而T(exactly:)恰好能精确表示这个 double。修复：原文是纯整数字面量（没有
./e/E）时，不接受 double 结果，改用T(String)精确解析（Int128也走这条路径）。这个分支不在整数解码的热路径上。10. P1 整数二次解析（8b78b8c）
确认：确认每次整数解码都会通过
yyjson_read_number重新解析原文。新增的对照测试覆盖 10 种整数类型 × 38 个边界字面量，逐一与 Foundation 对比，另外单独测试了 JSON5 的+1、0x1F。修复：原文形如
-?[0-9]{1,18}时直接手写解析（18 位十进制数不可能溢出 Int64），其他情况照旧走原路径。浮点路径没有改：去掉NUMBER_AS_RAW会让Decimal丢失精度，报告自己也指出了这一点。11. C-5 snake_case 键转换（45aa0c5）
确认：每行多 64 个不参与解码的键，耗时从 13.4 ms 涨到 352 ms，折合每个多余键约 1µs，主要花在 Foundation 算法的
split+capitalized+joined上。为什么不按报告建议改成按需查找：
convertFromSnakeCase不是双射（a_b和aB都会映射到aB），当前"转换后首个出现者胜出"的语义依赖完整的映射表，把请求的键反推成 snake_case 做不到可靠。修复：保留急切构建，只优化转换本身：没有内部下划线的键直接返回原串；纯 ASCII、且每一段全是字母或全是数字的键，用字节操作完成转换；其他情况回退到原算法（保留为
convertFromSnakeCaseReference）。新增测试包括：与 Foundation 对比 34 个刁钻键名的allKeys，以及 2 万个随机键的快路径和原算法逐一比对。12. C-9
JSON.string的memchr（9d2d283）确认：两个分支分别用
String(bytes:encoding:)和String(decoding:as:)，对合法 UTF-8 的结果完全相同（包括内嵌 NUL）；解码器不会启用allowInvalidUnicode，所以这个扫描是多余的。含\u0000的对照测试在修复前后都与 Foundation 一致。stringWhenJSON5遇到 NUL 返回 nil 是有意为之的行为，保持不变。13. C-10
JSONValue.description（8971da7）确认：报告说的
inf/nan只算半成立，因为 yyjson 在allowInfAndNaN下接受inf/nan。但实测发现了更严重的问题，修复前 4 处断言失败："或换行的内容会生成无法解析的输出；String(cString:)读取字符串，"a\u0000b"被截断成"a"；Int64.max的无符号整数输出为负数（18446744073709551615→-1）；numberAsRaw/bigNumberAsRaw解析出的数字输出为null。修复：直接遍历 yyjson 值，写入同一个字节缓冲区；正确转义字符串；非有限浮点数输出
Infinity/-Infinity/NaN（与库里 writer 一致）。输出格式（": "、", "）保持不变，原有的 description 测试全部通过。14. C-12 文档解析代码重复（47b7f9c）
把拷贝解析和就地解析（INSITU + padding）抽成两个私有函数，
DocumentStorage/DocumentRef只保留各自的所有权差异；同时删除了没有调用者的Span版streamParse。纯重构，流式解析基准全部持平（0.96～1.03x）。另外还有一个只改测试的提交（e29084a），把混合大小流式测试的分块大小从 7 调到 997 字节，测试耗时从约 8 秒降到 0.1 秒。
不修的项
IncrementalParser.read()是 O(N·K)：代码注释和JSONIncrementalReader的文档都已经写明这是有意的取舍，而且 P0-1 的窗口方案不适用于单个大文档。.sortedKeys整树深拷贝：实测 638 KB 文档普通写出 0.42 ms，sortedKeys 写出 1.29 ms，Foundation 12.96 ms。排序必须在可变 DOM 上进行，要去掉拷贝就得自己写一个 writer，投入产出不划算。unboxInteger失败时重复解析：只发生在抛错路径上，开销是微秒级。Span、~Copyable等新特性，这是有意为之。employee.bin/.DS_Store：git check-ignore显示两者分别被.gitignore的*.bin和.DS_Store规则忽略，从未提交过。报告第 3 节中已被报告自己实测证伪的两个想法（解码容器 class → struct、
withCString→yyObjGet）没有做。复现基准
基准包放在仓库之外（
../ReerJSONPerf），通过 path 依赖本仓库，没有提交到本 PR。复现方式：./ab.sh <修复前的提交> <分组> [轮数]，其中分组可选stream、serial、numbers、wide、snake、strings、desc、sorted。Package.swift / ab.sh / compare.py
Sources/PerfCheck/main.swift
C-1 并发复现程序（RaceRepro）