Add ArenaCompactTree::from_zipper_cached, re-using shared subtries - #58
Add ArenaCompactTree::from_zipper_cached, re-using shared subtries#58imlvts wants to merge 4 commits into
Conversation
from_zipper walks every path of the source trie, so a subtrie that is
structurally shared (grafted in many places) is serialized once per
occurrence. from_zipper_cached keys a map on the source's shared_node_id
and hands the already-built Node back instead of walking the subtrie again.
A cached Node can be handed to any parent because every NodeId/LineId in it
addresses arena data that is already written, and all arena references point
backwards. The node itself still has to be copied into each sibling run --
the format requires siblings to be contiguous, so a shared subtrie cannot be
pointed at from two runs -- but the copy keeps pointing at the original line
data and children, so a repeated subtrie costs one node instead of a subtrie.
The traversal is the jumping catamorphism unrolled, rather than a call to
into_cata_jumping_cached, because that one only consults its cache one byte
below a fork. Checking at the fork we land on after jumping a chain is what
catches a subtrie grafted under a multi-byte path, and re-using it needs an
operation the generic jumping algebra can't express ("prefix an already
folded subtrie"), but which is just a line node here.
For the parts of a trie with nothing to re-use both builders emit the same
bytes; tests assert that byte-for-byte over chains, wide branches, values at
every step, and random tries, and assert equal content plus a >50x size drop
on shared ones.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of the 256^5 paths of length 5 carries a value. The source is built by grafting a root-valued leaf under all 256 bytes, five levels over, so it is nothing but shared nodes; the cached builder folds it into ~1300 nodes (37KB). from_zipper is not run on it -- it would walk 10^12 paths -- so the test checks the shape instead: a full child mask and no value at each of the five levels, the value and an empty mask at depth 5, and 729 sampled paths that resolve, agree with the source PathMap, carry nothing on their prefixes, and do not extend past depth 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
||
| /// Graft `leaf` under every prefix, `levels` times over, so each level | ||
| /// shares the whole trie built by the level below it | ||
| fn make_shared_map(prefixes: &[&[u8]], levels: usize, leaf: PathMap<u64>) -> PathMap<u64> { |
There was a problem hiding this comment.
This seems like an almost adversarially obscure function.
Prefixes should be a map. Setting at all leaves should be product.
Also consider using merkleize or int range instead.
There was a problem hiding this comment.
merkleize or int range won't work on the fully populated 5-byte example which has over 1 trillion paths.
I will try to find a better API.
There was a problem hiding this comment.
Why would int range not work? It does 2^128 paths instantly.
There was a problem hiding this comment.
Checked it, didn't know it was working like that already.
| let mut next = PathMap::<u64>::new(); | ||
| for prefix in prefixes { | ||
| let mut wz = next.write_zipper_at_path(prefix); | ||
| wz.graft(&map.read_zipper()); |
There was a problem hiding this comment.
This should probably use graft_map
There was a problem hiding this comment.
No. Using graft_map consumes the map, and it makes me think it will clone the Node -> no reuse of deepest maps. Moved two functions together for visibility.
| /// finding it would mean giving up `descend_until` and stepping byte by byte. | ||
| /// | ||
| /// See [`ArenaCompactTree::from_zipper_cached`] for the user-facing docs. | ||
| fn build_arena_tree_cached<V, Z, F>(mut z: Z, map_val: F) -> ArenaCompactTree<Vec<u8>> |
There was a problem hiding this comment.
This should really give us a pause about the caching morphisms. Raison d'être (1) feels like it should be expressible in the caching catamorphism, though I understand it is not at the moment, (2) cached acata should just provide an unsafe FnMut: this function also needs an explanation why caching and mutability can be combined.
There was a problem hiding this comment.
This was one of the reasons I postponed the implementation of this feature -- I tried using cached morphisms, and failed to express it cleanly with the existing API. I will experiment on the APIs to see what I can do.
3 participants
from_zipperwalks every path of the source trie, so a subtrie that isstructurally shared (grafted in many places) is serialized once per
occurrence.
from_zipper_cachedkeys a map on the source'sshared_node_idand hands the already-built
Nodeback instead of walking the subtrie again.A cached
Nodecan be handed to any parent because everyNodeId/LineIdin itaddresses arena data that is already written, and all arena references point
backwards. The node itself still has to be copied into each sibling run --
the format requires siblings to be contiguous, so a shared subtrie cannot be
pointed at from two runs -- but the copy keeps pointing at the original line
data and children, so a repeated subtrie costs one node instead of a subtrie.
The traversal is the jumping catamorphism unrolled, rather than a call to
into_cata_jumping_cached, because that one only consults its cache one bytebelow a fork. Checking at the fork we land on after jumping a chain is what
catches a subtrie grafted under a multi-byte path, and re-using it needs an
operation the generic jumping algebra can't express ("prefix an already
folded subtrie"), but which is just a line node here.
For the parts of a trie with nothing to re-use both builders emit the same
bytes; tests assert that byte-for-byte over chains, wide branches, values at
every step, and random tries, and assert equal content plus a >50x size drop
on shared ones.
There's a test which stores 256^5 (or 10^12) paths in 37KB serialized.