Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions benchmarks/string-bench/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,11 @@ pub use serialized::*;

/// Session with the available string encodings and their canonicalize kernels
/// registered for benchmarking.
pub static SESSION: LazyLock<VortexSession> =
LazyLock::new(|| VortexSession::default().with_tokio());
pub static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
let session = VortexSession::default().with_tokio();
vortex_bench::enable_benchmark_editions(&session);
session
});

/// A named string column, canonicalized to a `VarBinViewArray`-backed array,
/// ready for either benchmark path.
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/string-bench/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ use vortex_bench::LogFormat;
use vortex_bench::create_output_writer;
use vortex_bench::display::DisplayFormat;
use vortex_bench::display::print_measurements_json;
use vortex_bench::enable_benchmark_editions;
use vortex_bench::measurements::CustomUnitMeasurement;
use vortex_bench::setup_logging_and_tracing_with_format;

Expand Down Expand Up @@ -282,6 +283,8 @@ fn run_vortex_column(
) -> Result<Vec<SerializedResult>> {
let runtime = CurrentThreadRuntime::new();
let session = VortexSession::default().with_handle(runtime.handle());
enable_benchmark_editions(&session);

let mut results = Vec::with_capacity(encoders.len());
for &encoder in encoders {
let result = runtime.block_on(bench_serialized_with_session(
Expand Down
8 changes: 5 additions & 3 deletions docs/specs/versioning/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,8 @@ supports all its members. New formats require a later edition, leaving the earli
to writers targeting older versions.

An _edition family_ groups editions for related components. Membership is cumulative within a
family: each later edition includes all earlier members.
family: each later edition includes all earlier members. The `unstable` family is the exception. It
has one undated edition that never freezes, and its members change in place.

The `core` family covers the default writer's formats, while optional features have independent
families. A writer can select one `core` edition and one `tensor` edition, for example, and use the
Expand Down Expand Up @@ -251,8 +252,9 @@ historical writing implementation.

A new format starts in a draft edition so it can be tested before its origin commits to reading it
indefinitely. Drafts have no recorded minimum version or frozen guarantee. A format intended for
`core` can progress from its own family to `preview` for broader testing, then to `core` for default
use. Promotion preserves its wire ID and interpretation. The
`core` can progress from the shared `unstable` edition to `preview` for broader testing, then to
`core` for default use. A defective ID can leave the `unstable` edition when its successor joins.
Promotion preserves its wire ID and interpretation. The
[registry instructions](editions.md#format-testing-and-promotion) cover promotion, freezing, and
recording the minimum version.

Expand Down
36 changes: 27 additions & 9 deletions docs/specs/versioning/editions.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,16 @@ or revision requires a new draft edition. Vortex-maintained draft formats are ex
compatible unless a defect blocks promotion into `core`, while independent plugin projects state
their own policy.

### `unstable`

- `array`: `fastlanes.bitpacked.v2`, `fastlanes.delta`, `fastlanes.for.v2`,
`vortex.decimal_byte_parts.v2`, `vortex.patched`, `vortex.union`
- `layout`: `vortex.list`
- `aggregate`: `vortex.bloom_filter.sbbf`, `vortex.sum_v2`

This edition is undated and never freezes. Its members change in place. The list layout is
distinct from the `vortex.list` array encoding already in core.

### `preview2026.08.0`

This edition currently adds no components.
Expand Down Expand Up @@ -113,17 +123,23 @@ For example, `core2026.08.0` declares `min`, `max`, `bounded_min`, `bounded_max`
statistics, however, do store sums, in a fixed legacy field governed by the enclosing format's
contract.

Aggregate serialization support alone does not make an aggregate an edition member. Editions
declare aggregate IDs intended for persisted zone maps. The `all_*` aggregates are in-memory pruning
predicates and do not need entries. Legacy file statistics also do not need entries.

## Format testing and promotion

A format intended for `core` starts in a dedicated edition family. Its first edition is a draft,
with no recorded `min_library_version` and no frozen compatibility guarantee. Even at this draft
stage, the format is expected to be complete. If testing reveals a defect whose correction changes
what readers must understand, the correction needs a new wire ID and a later edition.
A format intended for `core` starts in the shared `unstable` edition. This edition is undated, has
no recorded `min_library_version`, and never freezes, so it gives no compatibility guarantee. Even
at this stage, the format is expected to be complete. If testing reveals a defect whose correction
changes what readers must understand, the correction needs a new wire ID. Replace the defective ID
with its successor in the `unstable` declaration, for example `vortex.foo` with `vortex.foo_v2`.

After initial testing, the format can enter a new `preview` edition for broader opt-in use, then a
later `core` edition for default use. Promotion changes which editions permit the format while
preserving its contract and wire ID. The current `preview` edition is empty, so its first component
must go into a new edition.
preserving its contract and wire ID. The `unstable` and `preview` editions are registered but must
be enabled explicitly. The current `preview` edition is empty, so its first component must go into a
new edition.

## Freezing an edition

Expand All @@ -142,10 +158,12 @@ writers from choosing it, but readers must retain support because existing files

Default declarations live in `vortex-edition/src/declarations/`. Optional modules keep declarations
with their implementation code. The exported TOML records are under `vortex/editions/`, grouped by
family.
family. The undated `unstable` edition never freezes, so it has no record; only its `family.toml`
is exported.

1. For a new component, declare its own family and draft edition. For a revision, add a later
edition to the family that owns the earlier ID.
1. For a new core-maintained component, add it to the `unstable` edition. Optional modules use
their own families. For a revision in `unstable`, replace the earlier ID with its successor.
For a revision in another family, add a later edition to the family that owns the earlier ID.
2. To promote a tested format, add it to new `preview` and `core` editions without changing its ID
or contract.
3. When an edition freezes, record the first release of its origin that supports every permitted
Expand Down
7 changes: 5 additions & 2 deletions docs/specs/versioning/using-editions.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,11 @@ no components prevents the writer from serializing any component governed by edi
## Edition families

An _edition family_ groups editions for related formats. The `core` family covers the default
writer's formats. Optional features have their own families, such as `tensor` and `zstd`, so they
can add formats without changing an application's `core` selection.
writer's formats. Core-maintained formats not yet in `core` are tested in the shared `unstable` and
`preview` families, which must be enabled explicitly. The `unstable` family has one undated edition,
`unstable`, whose members change in place. Optional features have their own families,
such as `tensor` and `zstd`, so they can add formats without changing an application's `core`
selection.

A writer selects at most one edition per family. Selecting `core2026.08.0` and `tensor2026.04.0`
permits every component in either edition. Within one family, a later edition includes all earlier
Expand Down
47 changes: 16 additions & 31 deletions fuzz/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,20 +34,23 @@ pub use gpu::FuzzCompressGpu;
pub use gpu::run_compress_gpu;
pub use row::FuzzRowEncode;
pub use row::run_row_encode;
use vortex::editions::DEFAULT_PREVIEW_EDITION;
use vortex::editions::EditionSessionExt;
use vortex::editions::UNSTABLE;
use vortex_error::VortexExpect;
use vortex_session::VortexSession;

pub const FUZZ_ARRAY_MAX_LEN: usize = 2048;
pub const FUZZ_FILE_ARRAY_MAX_LEN: usize = 16_384;

fn enable_latest_core_edition(session: &vortex_session::VortexSession) {
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;
use vortex_error::VortexExpect;
use vortex_error::vortex_err;

session
.enable_edition(CORE_2026_08_3)
.map_err(|error| vortex_err!("{error}"))
.vortex_expect("latest core edition is registered");
/// Enable the `preview` and `unstable` editions in addition to the default editions, so the
/// fuzzer also exercises the components that await stabilization.
fn enable_fuzz_editions(session: &VortexSession) {
for edition in [DEFAULT_PREVIEW_EDITION, UNSTABLE] {
session
.enable_edition(edition)
.vortex_expect("fuzz edition is registered");
}
}

// Runtime initialization - platform-specific
Expand All @@ -73,7 +76,7 @@ mod native_runtime {
session = session.with::<vortex_cuda::CudaSession>();
vortex_cuda::initialize_cuda(&session.cuda_session());
}
super::enable_latest_core_edition(&session);
super::enable_fuzz_editions(&session);
session
});
}
Expand All @@ -96,7 +99,7 @@ mod wasm_runtime {

pub static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
let session = VortexSession::default().with_handle(WasmRuntime::handle());
super::enable_latest_core_edition(&session);
super::enable_fuzz_editions(&session);
session
});
}
Expand All @@ -115,28 +118,10 @@ mod wasi_runtime {

pub static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
let session = VortexSession::default();
super::enable_latest_core_edition(&session);
super::enable_fuzz_editions(&session);
session
});
}

#[cfg(all(target_arch = "wasm32", not(target_os = "unknown")))]
pub use wasi_runtime::SESSION;

#[cfg(test)]
mod tests {
use vortex::editions::CORE_2026_08_3;
use vortex::editions::EditionSessionExt;

use super::SESSION;

#[test]
fn fuzz_session_uses_latest_core_edition() {
assert!(
SESSION
.enabled_editions()
.editions()
.contains(&CORE_2026_08_3)
);
}
}
14 changes: 14 additions & 0 deletions vortex-bench/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ use tpch::benchmark::TpcHBenchmark;
pub use utils::file::*;
pub use utils::logging::*;
use vortex::compressor::BtrBlocksCompressorBuilder;
use vortex::editions::DEFAULT_PREVIEW_EDITION;
use vortex::editions::EditionSessionExt;
use vortex::editions::UNSTABLE;
use vortex::error::VortexExpect;
use vortex::error::vortex_err;
use vortex::file::VortexWriteOptions;
Expand Down Expand Up @@ -83,9 +86,20 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
pub static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
let session = VortexSession::default().with_tokio();
vortex_spatial::initialize(&session);
enable_benchmark_editions(&session);
session
});

/// Enable the `preview` and `unstable` editions in addition to the default editions, so
/// benchmarks also measure the components that await stabilization.
pub fn enable_benchmark_editions(session: &VortexSession) {
for edition in [DEFAULT_PREVIEW_EDITION, UNSTABLE] {
session
.enable_edition(edition)
.vortex_expect("benchmark edition is registered");
}
}

#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, Serialize, Deserialize)]
pub struct Target {
pub engine: Engine,
Expand Down
2 changes: 1 addition & 1 deletion vortex-edition/src/declarations/core/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pub static FAMILY: EditionFamily = EditionFamily {
wire representation that old readers either recognize or reject; several IDs may deserialize \
into one current in-memory array. Every core edition freezes, and a frozen edition carries a \
read-forever guarantee: a file written with it stays readable by every later Vortex release. \
New core objects first undergo testing in independently versioned families, then join preview for \
New core objects first undergo testing in the shared unstable family, then join preview for \
broad opt-in use, and finally join core with the same IDs and wire contracts. An edition may \
freeze in the release that cuts it; after that release version is known, the declaration is \
backfilled with it as the minimum. A frozen edition never changes.",
Expand Down
5 changes: 4 additions & 1 deletion vortex-edition/src/declarations/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,14 @@

pub mod core;
pub mod preview;
pub mod unstable;

use crate::EditionDeclaration;
use crate::EditionFamily;

/// The first-party edition families. Every family must be declared before its editions.
pub static EDITION_FAMILIES: &[&EditionFamily] = &[&core::FAMILY, &preview::FAMILY];
pub static EDITION_FAMILIES: &[&EditionFamily] =
&[&core::FAMILY, &preview::FAMILY, &unstable::FAMILY];

/// The first-party Vortex edition declarations.
pub static EDITION_DECLARATIONS: &[&EditionDeclaration] = &[
Expand All @@ -30,4 +32,5 @@ pub static EDITION_DECLARATIONS: &[&EditionDeclaration] = &[
&core::v2026_08_2::DECLARATION,
&core::v2026_08_3::DECLARATION,
&preview::v2026_08::DECLARATION,
&unstable::DECLARATION,
];
10 changes: 5 additions & 5 deletions vortex-edition/src/declarations/preview/mod.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright the Vortex contributors

//! The `preview` edition family: additive, opt-in components awaiting adoption into `core`.
//! The `preview` edition family: opt-in components awaiting adoption into `core`.

use crate::EditionFamily;

/// The `preview` family: stable opt-in components not yet available by default.
/// The `preview` family: tested opt-in components not yet available by default.
pub static FAMILY: EditionFamily = EditionFamily {
name: "preview",
origin: "vortex",
doc: "Additive, opt-in components maintained as part of Vortex but not yet adopted by the \
default core writer. Components enter preview only once their serialized contracts are ready for \
broad testing; independently evolving work remains in its own family until then.",
doc: "Opt-in components maintained as part of Vortex but not yet adopted by the default \
core writer. Components move from unstable into preview once their serialized contracts are \
ready for broad testing, then join core with the same IDs and wire contracts.",
};

pub mod v2026_08;
Expand Down
43 changes: 43 additions & 0 deletions vortex-edition/src/declarations/unstable/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright the Vortex contributors

//! The `unstable` edition family: core-maintained components undergoing focused testing.

use crate::Edition;
use crate::EditionDeclaration;
use crate::EditionFamily;
use crate::EditionId;
use crate::EditionMember;

/// The shared family for core-maintained components before promotion to preview.
pub static FAMILY: EditionFamily = EditionFamily {
name: "unstable",
origin: "vortex",
doc: "Opt-in components maintained as part of Vortex whose serialized contracts are ready \
for focused testing. The family has one undated edition that never freezes, so components join \
and leave it in place; a reader-visible correction still takes a new wire ID. Tested contracts \
move into preview for broad opt-in use, then into core for use by default. The unstable edition \
carries no read-forever guarantee.",
};

/// The single, undated edition of the `unstable` family.
pub const UNSTABLE: EditionId = EditionId::undated("unstable");

/// Built-in serialized components awaiting stabilization in [`UNSTABLE`].
pub static DECLARATION: EditionDeclaration = EditionDeclaration {
edition: Edition {
id: UNSTABLE,
min_library_version: None,
},
added: &[
EditionMember::array(&"fastlanes.bitpacked.v2"),
EditionMember::array(&"fastlanes.delta"),
EditionMember::array(&"fastlanes.for.v2"),
EditionMember::array(&"vortex.decimal_byte_parts.v2"),
EditionMember::array(&"vortex.patched"),
EditionMember::array(&"vortex.union"),
EditionMember::layout(&"vortex.list"),
EditionMember::aggregate(&"vortex.bloom_filter.sbbf"),
EditionMember::aggregate(&"vortex.sum_v2"),
],
};
Loading
Loading