D-QRY-2: active / static lists and membership history as mask programs

.claude/board/LATEST_STATE.md

@@ -85,6 +85,31 @@
 - Quack became consumable by git with lance-graph #1457
   (`ISS-LG-MASKRISC-PATH-NDARRAY`, resolved).
 
+## Lists (D-QRY-2)
+
+- `hubspo-query::list`: every list operation is a Quack filter over planes
+  the caller names, run by mask-risc (no row ids, no population loop).
+  - `refresh(predicate, prev, changed)` = `(prev ∧ ¬changed) ∨
+    (predicate ∧ changed)`: an active list re-reads the predicate only
+    where records changed. Deletes need no special case (the compiled
+    predicate carries `live`).
+  - static lists: `add`, `remove`, and `members` (`∧ live` on every read);
+  - history: `added` / `removed` between two stored versions.
+- Invariant this rests on: **ordinals are never reused** (records are
+  appended, the base is never rewritten; P-SPARSE-1). A store that reused
+  ordinals would have to clear a deleted record's bit in every static list.
+- `tests/list_oracle.rs`: 40 rounds of updates / inserts / deletes keep a
+  refreshed list equal to full recomputation and to a plain loop (added /
+  removed equal plain set differences, both > 100 over the run); a static
+  list equals a plain set through adds, removes and deletes; refresh keeps
+  stored bits outside the changed rows (> 1,000 of them disagree with the
+  predicate on purpose).
+  Disable runs 2026-10-10 (`cargo test -p hubspo-query --test list_oracle`):
+  refresh keeping stale bits (2 red), refresh re-reading every row (1 red),
+  `members` without `live` (1 red), `added` computing `removed` (1 red),
+  `remove` keeping rows (1 red). A `live` gate inside `add` stayed green
+  and was removed: `members` already applies it on every read.
+
 ## Harvest (D-ONT-1, public-catalog half)
 
 - `scripts/fetch-hubspot-specs.sh` → `.cache/` (gitignored), then

.claude/board/STATUS_BOARD.md

@@ -46,8 +46,8 @@
 
 | D-id | Deliverable | Size | Status | Evidence |
 |---|---|---|---|---|
-| D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | In PR | `hubspo-store::catalog` + `hubspo-query::filter`; 600 random filter sets equal a plain-loop oracle on mask-risc; #19 |
-| D-QRY-2 | Lists: active (incremental on version delta) + static (stored mask); membership history | L | Queued | P-LIST-1 2026-10-10: 2.6 ms at 10M; alloc ∝ fragments touched. P-SPARSE-1: 1.93 ms, 257 KiB flat at 10M |
+| D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | Shipped | `hubspo-store::catalog` + `hubspo-query::filter`; 600 random filter sets equal a plain-loop oracle on mask-risc; #19 |
+| D-QRY-2 | Lists: active (incremental on version delta) + static (stored mask); membership history | L | In PR | `hubspo-query::list` (refresh, static add/remove/members, added/removed); tests vs full recomputation. P-LIST-1 2.6 ms at 10M; P-SPARSE-1 1.93 ms, 257 KiB flat at 10M. Storage of list masks lands with D-STO-1 |
 | D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | Queued |  |
 | D-QRY-4 | Quack lowering of callcenter's `ParsedQuery` (the upstream DataFusion `parsed_query_to_plan` is an unimplemented stub) | M | Queued |  |
 | D-QRY-5 | Association-traversing filters ("contacts whose company has an open deal"): Quack `Semijoin` over edge lanes for one hop; multi-hop needs cypher-quack shapes that do not exist yet → upstream request first | L | Queued |  |

crates/hubspo-query/src/lib.rs

@@ -5,7 +5,10 @@
 //! - Zone: zone 2 (no serde, no wire formats).
 //! - Deliverables: D-QRY-1..7 (`.claude/plans/hubspo-integration-v1.md`).
 //! - [`filter`] (D-QRY-1): HubSpot filter groups → one Quack [`Filter`].
+//! - [`list`] (D-QRY-2): active / static lists and membership history as
+//!   mask programs.
 //!
 //! [`Filter`]: lance_graph_quack::Filter
 
 pub mod filter;
+pub mod list;

crates/hubspo-query/src/list.rs

@@ -0,0 +1,70 @@
+//! D-QRY-2: active lists, static lists and membership history as mask
+//! programs (H-ADR-5).
+//!
+//! A list's membership is a resident mask plane over record ordinals; the
+//! store owns its words. Everything here builds a Quack [`Filter`] over
+//! planes the caller names, so every list operation runs on mask-risc like
+//! any other query: no row ids are materialised and no Rust loop walks the
+//! population.
+//!
+//! - **Active list.** Membership is the compiled predicate
+//!   ([`crate::filter::compile`]). After a write, [`refresh`] re-evaluates the
+//!   predicate only where records changed and keeps the stored bits
+//!   elsewhere: `(prev ∧ ¬changed) ∨ (predicate ∧ changed)`. A deleted record
+//!   is a changed record whose live bit is clear, so it leaves the list with
+//!   no special case — the compiled predicate already conjoins `live`.
+//!   P-LIST-1 / P-SPARSE-1 measured the step at 0.85–2.6 ms up to 10M rows.
+//! - **Static list.** Membership is whatever was added. [`add`] and
+//!   [`remove`] take a plane of the records named in the request. A static
+//!   list's stored bits can outlive a deleted record, so readers go through
+//!   [`members`], which conjoins `live`. This relies on ordinals never being
+//!   reused (see [`add`]).
+//! - **History.** Between two stored versions of one list, [`added`] and
+//!   [`removed`] are the two differences.
+//!
+//! [`Filter`]: lance_graph_quack::Filter
+
+use lance_graph_quack::{Filter, Mask};
+
+/// The active list's membership after a write that touched `changed`.
+///
+/// `predicate` is the list's compiled filter (it includes `live`); `prev` is
+/// the stored membership; `changed` marks every record inserted, updated or
+/// deleted since `prev` was stored.
+pub fn refresh(predicate: &Filter, prev: Mask, changed: Mask) -> Filter {
+    Filter::or([
+        Filter::and([Filter::plane(prev), Filter::negate(Filter::plane(changed))]),
+        Filter::and([predicate.clone(), Filter::plane(changed)]),
+    ])
+}
+
+/// A static list after adding the records in `rows`.
+///
+/// Not gated on `live`: a bit stored for a deleted record is inert, because
+/// every read goes through [`members`]. That holds only while ordinals are
+/// never reused, which the store guarantees (records are appended, the base
+/// is never rewritten; P-SPARSE-1). A store that reused ordinals would have
+/// to clear a deleted record's bit in every static list first.
+pub fn add(prev: Mask, rows: Mask) -> Filter {
+    Filter::or([Filter::plane(prev), Filter::plane(rows)])
+}
+
+/// A static list after removing the records in `rows`.
+pub fn remove(prev: Mask, rows: Mask) -> Filter {
+    Filter::and([Filter::plane(prev), Filter::negate(Filter::plane(rows))])
+}
+
+/// A list's current members: its stored bits restricted to live records.
+pub fn members(list: Mask, live: Mask) -> Filter {
+    Filter::and([Filter::plane(list), Filter::plane(live)])
+}
+
+/// Records in `next` that were not in `prev`.
+pub fn added(prev: Mask, next: Mask) -> Filter {
+    Filter::and([Filter::plane(next), Filter::negate(Filter::plane(prev))])
+}
+
+/// Records in `prev` that are not in `next`.
+pub fn removed(prev: Mask, next: Mask) -> Filter {
+    Filter::and([Filter::plane(prev), Filter::negate(Filter::plane(next))])
+}