Repository navigation
perf(kvrec): 键索引修复逐键读取 O(n²)（fixes #24） - #25
Merged
Merged
Conversation
KeyValueRecord::get/has linearly scanned fields_ on every call. Real callers (e.g. go.neuralnet.cpp parse_nngames over a .nngames container) do per-key reads over the whole record, yielding O(n²) string compares: 80k keys took 33.7s, and full-dataset parse was 51% of verify time. Add a private key_index_ (key -> first-occurrence offset): - set() overloads emplace into the index (first occurrence wins) - parse()/parse_at() rebuild the index after populating fields_ - has()/get() are read-only lookups; concurrent const reads stay safe - duplicate keys with type mismatch keep the old "first key+type match" semantics via forward fallback scan from the first occurrence - nested Record get() rebuilds the sub-record's index The container byte layout is unchanged (downstream data identity contract); this only touches the in-memory read path. Benchmark (issue #24, clang -std=c++23 -O2): 80k per-key gets 33.7s -> 0.014s; scaling now linear. Fixes #24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题（#24）
KeyValueRecord::get/has对fields_做线性扫描。真实调用方（go.neuralnet.cpp 的parse_nngames）是逐键读取整个 record，导致 O(n²) 次字符串比较：80k 键 33.7s，全量.nngamesparse 占 verify 总时长 51%。修法（issue 建议方案 1：内存结构维护索引）
新增私有键索引
key_index_（key → fields_ 首次出现下标）：set()四个重载emplace维护索引（emplace 保留首个出现下标）parse()/parse_at()填充fields_后重建索引has()/get()只读查询 —— 并发 const 读取安全（无需 mutable 惰性构建）Record的get重建子记录索引容器字节布局零变化（
.nngames数据身份契约不受影响），只动内存读取路径。验证
issue #24 中的微基准（clang
-std=c++23 -O2）：耗时随 N 线性增长，二次复杂度消除。
现有测试全部通过：
kvrec_test、dataset_samples_test、model_spec_validation_test、rl_output_test、train_state_test。Fixes #24