Repository navigation
arena: pack objects, refine string size classes and align external allocations - #98
Merged
Merged
Conversation
External allocations returned their payload right after a one byte class field, so large arena strings stored their headers at a misaligned address. Pad the header to the arena alignment like normal chunks. Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Variants, kvpairs, key/value lists, arrays and SDS headers only hold pointers, sizes, integers and doubles. Allocate them with that alignment instead of the raw allocation alignment, which includes long double and pads every 40 byte object to 48 bytes on x86_64 and arm64. Owned SDS strings used classes starting at 32 bytes of payload, so a three byte map key took 80 bytes. Small classes now grow in alignment steps of the whole allocation up to 128 bytes of payload, and larger classes grow by about 25 percent. Pooled blocks are no longer zeroed, since the SDS layer initializes what it uses. Bump allocations from the current chunk take an inlined fast path. Decoded log record batches use 27 to 31 percent less arena memory and 20 to 25 percent less CPU. Raw allocations keep the public alignment. Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes arena-backed CFL object graphs smaller and faster to build, and
fixes a misaligned allocation in the arena. The public API, struct layouts and
heap-backed behaviour are unchanged.
Motivation
Decoded telemetry records are mostly structure: variants, kvpairs, key/value
lists and short owned map keys. In an arena, a typical log record with six
short keys and a nested map took about 1,320 bytes on x86_64, roughly 14 times
its MessagePack size. Most of that was padding:
long double(16 bytes on x86_64and arm64), so 40-byte variants and kvpairs took 48 bytes;
three-byte key such as
msgtook 80 bytes.Changes
Object packing
Variants, kvpairs, key/value lists, arrays (struct and entry storage) and SDS
headers only hold pointers, sizes, 64-bit integers and doubles. They are now
allocated with the strictest of those alignments instead of the raw allocation
alignment. Compile-time checks verify that every CFL object type fits that
alignment on the target platform.
Finer SDS size classes
the complete allocation, so a short string uses at most one alignment unit
more than it needs. The number of small classes adapts to the platform
alignment, keeping the same coverage on 32-bit and 64-bit targets.
initializes the headers and the payload it writes. Variants and kvpairs are
still zeroed.
Allocation fast path
Bump allocations from the current chunk take an inlined fast path. Walking the
remaining chunks and reserving a new one moved to a separate slow path with
the same behaviour as before.
Fix: external allocation alignment
External allocations, used for large owned SDS strings, returned their
payload right after a one-byte class field. Large arena strings therefore
stored their headers at a misaligned address: undefined behaviour, reported by
UBSan, and a potential fault on strict-alignment targets. The external header
is now padded to the arena alignment, like normal chunks.
New benchmark
cfl-benchmark-arena-recordsbuilds batches of decoded log records (shortkeys, a nested map, an array, strings, integers and doubles). It reports
nanoseconds per record and used and reserved bytes per record, with options
for referenced or owned values, record edits (remove, insert, rename) and the
chunk size. See
benchmarks/README.md.API
No public header changes, and no symbols removed.
Internal allocation helpers, declared in
src/cfl_arena_internal.hand notpart of the public API:
void *cfl_arena_alloc_object(struct cfl_arena *arena, size_t size)void *cfl_arena_calloc_object(struct cfl_arena *arena, size_t count, size_t size)CFL_ARENA_SDS_MAX_PAYLOAD(1024) names the largest payload recycled throughthe SDS size classes.
The public raw allocators (
cfl_arena_malloc(),cfl_arena_calloc(),cfl_arena_memdup(),cfl_arena_strndup()) keep the documented alignment,including
long double. Tests check this after interleaved objectallocations.
Compatibility
No public struct layout, signature or return value changes, and heap-backed
values behave as before. Observable differences, none of which are documented
guarantees:
cfl_sds_alloc()on an arena-backed string reports the new, tighter classcapacity. It is always at least the string length.
cfl_arena_bytes_used(),cfl_arena_bytes_reserved())report smaller values for the same graph. External allocations carry a
slightly larger header because of the alignment fix.
as with heap strings.
Performance
Release build, x86_64, best of 5 runs.
cfl-benchmark-arena-records arena 512 3000:Existing benchmarks:
cfl-benchmark-variant-arena arena 1000 1000 8192cfl-benchmark-variant-mutable arena 100 100 10 8192cfl-benchmark-variant-mutable, random and bimodal payloadsbenchmarks/run_variant_matrix.sh(270 arena configurations): reserved bytesmedian −13%, time geometric mean −10%. The only configurations with more
reserved bytes are large strings with external storage disabled and 4 KiB
chunks, at up to +3.8%. Those configurations already leave about 40 percent
of reserved memory unused because of how oversized chunks are reused after
reset.
Downstream, Fluent Bit's per-chunk arena for decoded log records uses 25%
fewer bytes per flat record (1,320 → 990) and 18% fewer per Kubernetes record
(4,428 → 3,628).
Testing
New and updated cases in
tests/arena.c:terminator, monotonic class capacities, bounded waste, slot reuse and
bytes_usedaccounting;allocations verified intact;
storage and external storage, plus self-append and
cfl_sds_printf();SDS headers in arena storage, including external strings;
against a shadow model across resets;
reuse_sds_size_classestest now checks class invariantsinstead of fixed capacities.
The new size-class and alignment tests fail on the previous implementation.
Commands:
--leak-check=full.test function pointer casts, which also appear on
master.-Wall -Wextra -Wpedantic: no new warningscompared to
master.lib/cflwith this branch as inbuild-downstream, at each project'sorigin/master:integration scenarios that use CFL graphs pass normally and under Valgrind.
Not run locally: 32-bit targets (no 32-bit libc available). The size-class
math was checked for 4, 8 and 16-byte object alignment, and CI covers ARM64
and Windows.