diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index a47ac8c..793846b 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -3,6 +3,12 @@ > Open blockers and cross-repo follow-ups. Append new entries at the top; > change only the `Status` line of an existing entry. +## ISS-LG-MASKRISC-PATH-NDARRAY — Quack could not be consumed by git + +- **Status:** RESOLVED 2026-10-10 by lance-graph #1457 (`e724d83`): mask-risc names ndarray by git, and lance-graph's workspaces patch it back to the sibling checkout. +- **Where:** lance-graph `crates/lance-graph-mask-risc/Cargo.toml` (`ndarray = { path = "../../../ndarray" }`). +- **What:** the relative path leaves the lance-graph checkout, so any git consumer of `lance-graph-quack` (which depends on mask-risc) fails with "no matching package named `ndarray`". It also breaks lance-graph's own CARGO-COMPUTE-SUBSTRATE rule 5. Fix upstream: mask-risc names `ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master" }`, and lance-graph's root adds `[patch."https://github.com/AdaWorldAPI/ndarray"] ndarray = { path = "../ndarray" }`, so its own builds keep the sibling checkout and one ndarray identity. Verified locally: HubSPO-rs builds and tests against the patched checkout with one ndarray package. + ## ISS-DIRSIM-HUBSPO-GAPS — dir-sim lacks what tickets, mail and documents need - **Status:** OPEN (upstream asks; not built here — STOP rule) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index be40c33..7ae7645 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -49,6 +49,42 @@ - CI: `.github/workflows/ci.yml` runs fmt, clippy (`--no-deps -D warnings`) and `cargo test --workspace` with debug info off. +## Filter compiler (D-QRY-1) + +- `hubspo-store::catalog`: the production `Registrar` + `Binder` for Quack. + One `Codebook` per coded field (label `i` → code `i`, unknown labels bind + to nothing), plane `0` of every table is its live plane, nullable fields + name a validity plane. Refuses duplicate fields / lanes / labels, a + codebook on an integer field, and live-as-validity. 4 unit tests. +- `hubspo-query::filter::compile(object, groups, &binder) -> Filter`: + HubSpot filter groups (AND within, OR across), operators EQ NEQ LT LTE GT + GTE BETWEEN IN NOT_IN HAS_PROPERTY NOT_HAS_PROPERTY; CONTAINS_TOKEN is + refused (text search is the Tantivy index). Values are HubSpot strings; + integers are decimal `i32` until D-ONT-6 fixes lane encodings (amounts, + timestamps). Three-valued over nullable properties: a record with no + value is kept by no comparison, NEQ and NOT_IN included — whether HubSpot + agrees is a value-oracle question (`ISS-ORACLE-PORTAL`). +- `tests/filter_oracle.rs`: 600 random filter-group sets over 5,000 + records (3 % deleted, 20–30 % missing values with random payloads left in + the lanes) are compiled, lowered and run on mask-risc; every kept-row set + equals a plain-loop reading (456 non-trivial, 122 empty, 22 all live). + Disable runs 2026-10-10 (`cargo test -p hubspo-query --test filter_oracle`): + + | disable | red | + |---|---| + | `sql_where` dropped | oracle + missing-value test | + | NOT_IN compiled as IN | oracle | + | BETWEEN upper bound exclusive | oracle | + | groups ANDed instead of ORed | oracle | + | live plane dropped | oracle + presence test | + | NOT_HAS_PROPERTY on a required property true | oracle + presence test | + + A seventh guard (`reads_value`, keeping presence-only properties off the + nullable list) stayed green and was deleted: `sql_where` gates only + comparison leaves, so it guarded nothing. +- Quack became consumable by git with lance-graph #1457 + (`ISS-LG-MASKRISC-PATH-NDARRAY`, resolved). + ## Harvest (D-ONT-1, public-catalog half) - `scripts/fetch-hubspot-specs.sh` → `.cache/` (gitignored), then diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index f5ecd3b..93170ef 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -27,7 +27,7 @@ | D-ONT-2 | OGAR: `ogar-from-schema` OpenAPI front end (queued upstream, `lib.rs:48`) — **requested from OGAR, not built in HubSPO-rs** | L | Queued | | | D-ONT-3 | Mint dossier: per candidate concept (H-ADR-2 table) the METHOD/STORAGE/STRUCTURE evidence + OGIT source + convergence argument; submit to the 5+3 codebook council | M | Shipped | ratified v3 `.claude/plans/hubspo-concept-mint-dossier-v3.md` (5+3 council, 2026-10-10) | | D-ONT-4 | OGAR PR: `ConceptDomain::Crm` (`0x05XX`), 15 concepts in `class_ids` + `CODEBOOK` + `ALL`, `HubSpoPort` (`0x000B`) + aliases, layout doc row; lance-graph `ogar_codebook` mirror rows + `AppPrefix` 0x0008–0x000B. `crm_actions` deferred to D-ACT-1 (dossier v3 §7) | M | Shipped | OGAR#330, lance-graph#1438 | -| D-ONT-5 | `hubspo-port`: `RbacPlug` binding; `HotPlug` over Crm ids asserted to fail `NoCapabilitiesFor` until D-ACT-1 (dossier v3 §7); deliberately-wrong-plug test proving each drift arm can fire | S | In PR | `hubspo-port`, 13 tests | +| D-ONT-5 | `hubspo-port`: `RbacPlug` binding; `HotPlug` over Crm ids asserted to fail `NoCapabilitiesFor` until D-ACT-1 (dossier v3 §7); deliberately-wrong-plug test proving each drift arm can fire | S | Shipped | `hubspo-port`, 13 tests (`04edf87` on main) | | D-ONT-6 | `hubspo-schema`: ClassViews for contact, company, deal, ticket, lead, product, line item, quote, activity kinds; standard property position tables via the generator into `hubspo-generated` | L | Queued | | | D-ONT-7 | Custom-object/-property design (H-ADR-3) + probe P-CUSTOM-1 (lane-per-property vs packed run; fold latency at 50 / 500 / 2,000 custom properties) | M | In progress | P-CUSTOM-1 2026-10-10: lanes 0.73–1.61× std, packed 15–35×; lane write 33–96 s vs 1.5 s | @@ -46,7 +46,7 @@ | D-id | Deliverable | Size | Status | Evidence | |---|---|---|---|---| -| D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | Queued | P-QDB-1: catalog + codebook `Binder` and nine bound programs proven against an oracle | +| D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | In PR | `hubspo-store::catalog` + `hubspo-query::filter`; 600 random filter sets equal a plain-loop oracle on mask-risc; #19 | | D-QRY-2 | Lists: active (incremental on version delta) + static (stored mask); membership history | L | Queued | P-LIST-1 2026-10-10: 2.6 ms at 10M; alloc ∝ fragments touched. P-SPARSE-1: 1.93 ms, 257 KiB flat at 10M | | D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | Queued | | | D-QRY-4 | Quack lowering of callcenter's `ParsedQuery` (the upstream DataFusion `parsed_query_to_plan` is an unimplemented stub) | M | Queued | | diff --git a/crates/hubspo-query/Cargo.toml b/crates/hubspo-query/Cargo.toml index a364ac8..fa5cbbc 100644 --- a/crates/hubspo-query/Cargo.toml +++ b/crates/hubspo-query/Cargo.toml @@ -10,3 +10,10 @@ publish = false [lints] workspace = true + +[dependencies] +lance-graph-quack.workspace = true + +[dev-dependencies] +hubspo-store = { path = "../hubspo-store" } +lance-graph-mask-risc.workspace = true diff --git a/crates/hubspo-query/src/filter.rs b/crates/hubspo-query/src/filter.rs new file mode 100644 index 0000000..a97d7d4 --- /dev/null +++ b/crates/hubspo-query/src/filter.rs @@ -0,0 +1,360 @@ +//! D-QRY-1: HubSpot filter groups compiled to one Quack [`Filter`]. +//! +//! HubSpot's search filter is a list of *filter groups*: filters inside a +//! group are ANDed, groups are ORed. Values arrive as strings, as in +//! HubSpot's API. [`compile`] binds every property name and literal once +//! through a [`Binder`] (the catalog in `hubspo-store`) and returns a +//! numeric filter; nothing textual survives it. The caller lowers it with +//! `lance_graph_quack::lower` and runs it on mask-risc. +//! +//! ```text +//! live ∧ (g₁ ∨ g₂ ∨ …), gᵢ = f₁ ∧ f₂ ∧ … +//! ``` +//! +//! ## Operators +//! +//! | operator | coded field (enumeration) | integer field | +//! |---|---|---| +//! | `EQ` / `NEQ` | code compare | `=` / `<>` | +//! | `LT` `LTE` `GT` `GTE` | refused | ordered compare | +//! | `BETWEEN` | refused | `value ≤ x ≤ highValue` | +//! | `IN` / `NOT_IN` | code set | integer set | +//! | `HAS_PROPERTY` / `NOT_HAS_PROPERTY` | validity plane | validity plane | +//! | `CONTAINS_TOKEN` / `NOT_CONTAINS_TOKEN` | refused: text search is the Tantivy index (H-ADR-6) | refused | +//! +//! ## Missing values +//! +//! A filter that reads a nullable property is SQL three-valued +//! (`Filter::sql_where`): a record with no value is kept by no comparison, +//! including `NEQ` and `NOT_IN`. Only `HAS_PROPERTY` / `NOT_HAS_PROPERTY` ask +//! about the missing value itself. Whether HubSpot keeps records with no +//! value under `NEQ` / `NOT_IN` is an open question for the value oracle +//! (`ISS-ORACLE-PORTAL`); the answer would change only this module. +//! +//! ## Literals +//! +//! - Coded fields: the literal must be one of the field's labels. An unknown +//! label is an error, not an empty result: nothing is minted at bind time. +//! - Integer fields: a decimal `i32`. Decimals, millisecond timestamps and +//! currency amounts need the lane encodings of D-ONT-6 (minor units, day +//! numbers) and are refused here until then. + +use lance_graph_quack::bind::{Binder, FieldKind}; +use lance_graph_quack::{Cmp, Col, Filter, Mask}; + +/// A HubSpot filter operator. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum Operator { + /// `EQ` + Eq, + /// `NEQ` + Neq, + /// `LT` + Lt, + /// `LTE` + Lte, + /// `GT` + Gt, + /// `GTE` + Gte, + /// `BETWEEN` (inclusive on both ends) + Between, + /// `IN` + In, + /// `NOT_IN` + NotIn, + /// `HAS_PROPERTY` + HasProperty, + /// `NOT_HAS_PROPERTY` + NotHasProperty, + /// `CONTAINS_TOKEN` + ContainsToken, + /// `NOT_CONTAINS_TOKEN` + NotContainsToken, +} + +impl Operator { + /// The operator HubSpot's API spells `name`. + pub fn from_hubspot(name: &str) -> Option { + Some(match name { + "EQ" => Self::Eq, + "NEQ" => Self::Neq, + "LT" => Self::Lt, + "LTE" => Self::Lte, + "GT" => Self::Gt, + "GTE" => Self::Gte, + "BETWEEN" => Self::Between, + "IN" => Self::In, + "NOT_IN" => Self::NotIn, + "HAS_PROPERTY" => Self::HasProperty, + "NOT_HAS_PROPERTY" => Self::NotHasProperty, + "CONTAINS_TOKEN" => Self::ContainsToken, + "NOT_CONTAINS_TOKEN" => Self::NotContainsToken, + _ => return None, + }) + } +} + +/// One HubSpot filter: `propertyName`, `operator`, `value` / `highValue` / +/// `values`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PropertyFilter { + /// `propertyName`. + pub property: String, + /// `operator`. + pub operator: Operator, + /// `value` (single-value operators and `BETWEEN`'s low end). + pub value: Option, + /// `highValue` (`BETWEEN`'s high end). + pub high_value: Option, + /// `values` (`IN` / `NOT_IN`). + pub values: Vec, +} + +impl PropertyFilter { + /// A filter with one value. + pub fn new(property: &str, operator: Operator, value: &str) -> Self { + Self { + property: property.into(), + operator, + value: Some(value.into()), + high_value: None, + values: Vec::new(), + } + } + + /// `HAS_PROPERTY` / `NOT_HAS_PROPERTY`. + pub fn presence(property: &str, operator: Operator) -> Self { + Self { + property: property.into(), + operator, + value: None, + high_value: None, + values: Vec::new(), + } + } + + /// `BETWEEN low AND high`. + pub fn between(property: &str, low: &str, high: &str) -> Self { + Self { + property: property.into(), + operator: Operator::Between, + value: Some(low.into()), + high_value: Some(high.into()), + values: Vec::new(), + } + } + + /// `IN` / `NOT_IN` over `values`. + pub fn set(property: &str, operator: Operator, values: &[&str]) -> Self { + Self { + property: property.into(), + operator, + value: None, + high_value: None, + values: values.iter().map(|v| (*v).to_string()).collect(), + } + } +} + +/// A filter group: its filters are ANDed. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct FilterGroup { + /// `filters`. + pub filters: Vec, +} + +/// Why a filter did not compile. Each names the HubSpot term that failed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum CompileError { + /// The object type has no table. + UnknownObject(String), + /// The object type has no such property. + UnknownProperty(String), + /// The property's domain has no such value. + UnknownValue { + /// Property. + property: String, + /// The literal. + value: String, + }, + /// The value is not a decimal `i32`. + NotAnInteger { + /// Property. + property: String, + /// The literal. + value: String, + }, + /// The operator does not apply to the property's kind. + OperatorNotSupported { + /// Property. + property: String, + /// Operator. + operator: Operator, + }, + /// Text search is not a Quack predicate; it is the Tantivy index. + TextSearch { + /// Property. + property: String, + }, + /// The operator needs a value (or `values`) that was not given. + MissingValue { + /// Property. + property: String, + /// Operator. + operator: Operator, + }, + /// A filter group with no filters. + EmptyGroup, +} + +/// Compile `groups` over the object type `object` into one numeric filter. +/// +/// No groups means every live record. +/// +/// # Errors +/// +/// The first [`CompileError`]; nothing is minted on failure. +pub fn compile( + object: &str, + groups: &[FilterGroup], + binder: &dyn Binder, +) -> Result { + let table = binder + .table(object) + .ok_or_else(|| CompileError::UnknownObject(object.into()))?; + let live_mask = binder.live(table); + let live = Filter::plane(live_mask); + if groups.is_empty() { + return Ok(live); + } + let mut nullable: Vec<(Col, Mask)> = Vec::new(); + let mut alternatives = Vec::with_capacity(groups.len()); + for group in groups { + if group.filters.is_empty() { + return Err(CompileError::EmptyGroup); + } + let mut leaves = Vec::with_capacity(group.filters.len()); + for f in &group.filters { + let field = binder + .field(table, &f.property) + .ok_or_else(|| CompileError::UnknownProperty(f.property.clone()))?; + let leaf = compile_one( + f, + field.col, + field.kind, + field.validity, + live_mask, + &|lit| binder.code(table, field.col, lit), + )?; + // `sql_where` gates only comparison leaves, so listing a + // property that only a presence filter reads changes nothing. + if let Some(valid) = field.validity + && !nullable.contains(&(field.col, valid)) + { + nullable.push((field.col, valid)); + } + leaves.push(leaf); + } + alternatives.push(Filter::and(leaves)); + } + Ok(Filter::and([live, Filter::or(alternatives)]).sql_where(&nullable)) +} + +fn compile_one( + f: &PropertyFilter, + col: Col, + kind: FieldKind, + validity: Option, + live: Mask, + code: &dyn Fn(&str) -> Option, +) -> Result { + let missing = || CompileError::MissingValue { + property: f.property.clone(), + operator: f.operator, + }; + let unsupported = || CompileError::OperatorNotSupported { + property: f.property.clone(), + operator: f.operator, + }; + let value = || f.value.as_deref().ok_or_else(missing); + let int = |lit: &str| { + lit.trim() + .parse::() + .map_err(|_| CompileError::NotAnInteger { + property: f.property.clone(), + value: lit.into(), + }) + }; + let coded = |lit: &str| { + code(lit).ok_or_else(|| CompileError::UnknownValue { + property: f.property.clone(), + value: lit.into(), + }) + }; + let leaf = |cmp| Ok(Filter::cmp(col, cmp)); + match (f.operator, kind) { + (Operator::HasProperty, _) => Ok(match validity { + Some(valid) => Filter::is_not_null(valid), + // Every live record has a value. Quack has no constant TRUE (an + // empty AND is refused at lowering), so restate `live`. + None => Filter::plane(live), + }), + (Operator::NotHasProperty, _) => Ok(match validity { + Some(valid) => Filter::is_null(valid), + // No live record lacks a value: FALSE, spelled `¬live`. + None => Filter::negate(Filter::plane(live)), + }), + (Operator::ContainsToken | Operator::NotContainsToken, _) => { + Err(CompileError::TextSearch { + property: f.property.clone(), + }) + } + (Operator::Eq, FieldKind::Code) => leaf(Cmp::EqU32(coded(value()?)?)), + (Operator::Neq, FieldKind::Code) => leaf(Cmp::NeU32(coded(value()?)?)), + (Operator::Eq, FieldKind::I32) => leaf(Cmp::EqI32(int(value()?)?)), + (Operator::Neq, FieldKind::I32) => leaf(Cmp::NeI32(int(value()?)?)), + (Operator::Lt, FieldKind::I32) => leaf(Cmp::LtI32(int(value()?)?)), + (Operator::Lte, FieldKind::I32) => leaf(Cmp::LeI32(int(value()?)?)), + (Operator::Gt, FieldKind::I32) => leaf(Cmp::GtI32(int(value()?)?)), + (Operator::Gte, FieldKind::I32) => leaf(Cmp::GeI32(int(value()?)?)), + (Operator::Between, FieldKind::I32) => { + let low = int(value()?)?; + let high = int(f.high_value.as_deref().ok_or_else(missing)?)?; + Ok(Filter::and([ + Filter::cmp(col, Cmp::GeI32(low)), + Filter::cmp(col, Cmp::LeI32(high)), + ])) + } + (Operator::In | Operator::NotIn, _) => { + if f.values.is_empty() { + return Err(missing()); + } + let set = match kind { + FieldKind::Code => Filter::in_u32( + col, + f.values + .iter() + .map(|v| coded(v)) + .collect::, _>>()?, + ), + FieldKind::I32 => Filter::in_i32( + col, + f.values + .iter() + .map(|v| int(v)) + .collect::, _>>()?, + ), + }; + Ok(if f.operator == Operator::In { + set + } else { + Filter::negate(set) + }) + } + ( + Operator::Lt | Operator::Lte | Operator::Gt | Operator::Gte | Operator::Between, + FieldKind::Code, + ) => Err(unsupported()), + } +} diff --git a/crates/hubspo-query/src/lib.rs b/crates/hubspo-query/src/lib.rs index 0bb97ca..70a9e39 100644 --- a/crates/hubspo-query/src/lib.rs +++ b/crates/hubspo-query/src/lib.rs @@ -4,5 +4,8 @@ //! //! - Zone: zone 2 (no serde, no wire formats). //! - Deliverables: D-QRY-1..7 (`.claude/plans/hubspo-integration-v1.md`). -//! - First real code lands in wave W3. Until then this crate is empty on -//! purpose; Wave 0 only fixes the workspace shape. +//! - [`filter`] (D-QRY-1): HubSpot filter groups → one Quack [`Filter`]. +//! +//! [`Filter`]: lance_graph_quack::Filter + +pub mod filter; diff --git a/crates/hubspo-query/tests/filter_oracle.rs b/crates/hubspo-query/tests/filter_oracle.rs new file mode 100644 index 0000000..1809611 --- /dev/null +++ b/crates/hubspo-query/tests/filter_oracle.rs @@ -0,0 +1,524 @@ +//! D-QRY-1 against a plain-loop oracle: random HubSpot filter groups are +//! compiled, lowered and run on mask-risc, and the kept rows must equal what +//! a row-by-row reading of the same filter keeps. + +use hubspo_query::filter::{CompileError, FilterGroup, Operator, PropertyFilter, compile}; +use hubspo_store::catalog::{Catalog, Codebook, FieldSpec}; +use lance_graph_mask_risc::{Foreign, LaneRef, Out, Planes, Scratch, execute_into, words_for}; +use lance_graph_quack::bind::{FieldKind, create_table}; +use lance_graph_quack::{Agg, Col, Mask, Query, lower}; + +const STAGES: &[&str] = &[ + "subscriber", + "lead", + "marketingqualifiedlead", + "salesqualifiedlead", + "opportunity", + "customer", +]; +const COUNTRIES: &[&str] = &["DE", "AT", "CH", "FR", "NL", "US", "GB", "PL"]; +const N: usize = 5_000; + +struct Rng(u64); +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn below(&mut self, n: u64) -> u64 { + self.next() % n + } + fn chance(&mut self, percent: u64) -> bool { + self.below(100) < percent + } +} + +/// The records as the oracle reads them; `None` is "no value". +struct Records { + live: Vec, + stage: Vec, + country: Vec>, + amount: Vec>, + created: Vec, +} + +/// The same records as resident lanes and planes. A missing value keeps a +/// random payload in its lane, so a compiler that reads it gets caught. +struct Resident { + stage: Vec, + country: Vec, + amount: Vec, + created: Vec, + masks: [Vec; 3], +} + +fn bits(n: usize, f: impl Fn(usize) -> bool) -> Vec { + let mut w = vec![0u64; words_for(n)]; + for i in 0..n { + if f(i) { + w[i / 64] |= 1 << (i % 64); + } + } + w +} + +fn records() -> (Records, Resident) { + let mut r = Rng(0xD_0001); + let mut rec = Records { + live: Vec::new(), + stage: Vec::new(), + country: Vec::new(), + amount: Vec::new(), + created: Vec::new(), + }; + let mut res = Resident { + stage: Vec::new(), + country: Vec::new(), + amount: Vec::new(), + created: Vec::new(), + masks: [Vec::new(), Vec::new(), Vec::new()], + }; + for _ in 0..N { + rec.live.push(!r.chance(3)); + let stage = r.below(STAGES.len() as u64) as u32; + rec.stage.push(stage); + res.stage.push(stage); + let country = r.below(COUNTRIES.len() as u64) as u32; + let has_country = !r.chance(20); + rec.country.push(has_country.then_some(country)); + res.country.push(country); + let amount = r.below(5_500) as i32 - 500; + let has_amount = !r.chance(30); + rec.amount.push(has_amount.then_some(amount)); + res.amount.push(amount); + let created = r.below(365) as i32; + rec.created.push(created); + res.created.push(created); + } + res.masks = [ + bits(N, |i| rec.live[i]), + bits(N, |i| rec.country[i].is_some()), + bits(N, |i| rec.amount[i].is_some()), + ]; + (rec, res) +} + +fn catalog() -> Catalog { + let mut c = Catalog::new(); + let t = create_table("contacts") + .width(16) + .register(&mut c) + .expect("register") + .id; + let field = |name: &str, col: u16, kind, validity, labels: Option<&[&str]>| FieldSpec { + name: name.into(), + col: Col(col), + kind, + validity, + codebook: labels.map(|l| Codebook::new(l.iter().copied()).expect("codebook")), + }; + for spec in [ + field("lifecyclestage", 0, FieldKind::Code, None, Some(STAGES)), + field( + "country", + 1, + FieldKind::Code, + Some(Mask(1)), + Some(COUNTRIES), + ), + field("amount", 2, FieldKind::I32, Some(Mask(2)), None), + field("createdate", 3, FieldKind::I32, None, None), + ] { + c.declare(t, spec).expect("declare"); + } + c +} + +// ---------------------------------------------------------------- oracle ---- + +/// One property of one record, as the oracle sees it. +enum Val { + Code(Option), + Int(Option), +} + +fn value(rec: &Records, property: &str, row: usize) -> Val { + match property { + "lifecyclestage" => Val::Code(Some(rec.stage[row])), + "country" => Val::Code(rec.country[row]), + "amount" => Val::Int(rec.amount[row]), + "createdate" => Val::Int(Some(rec.created[row])), + other => panic!("no property {other}"), + } +} + +fn label_code(property: &str, label: &str) -> u32 { + let list = if property == "lifecyclestage" { + STAGES + } else { + COUNTRIES + }; + list.iter().position(|l| *l == label).expect("label") as u32 +} + +/// HubSpot's reading of one filter on one record, with a missing value kept +/// by no comparison (the module's documented choice). +fn keeps(rec: &Records, f: &PropertyFilter, row: usize) -> bool { + let v = value(rec, &f.property, row); + let present = match &v { + Val::Code(x) => x.is_some(), + Val::Int(x) => x.is_some(), + }; + match f.operator { + Operator::HasProperty => return present, + Operator::NotHasProperty => return !present, + _ if !present => return false, + _ => {} + } + let one = || f.value.as_deref().expect("value"); + match v { + Val::Code(Some(x)) => { + let set: Vec = f + .values + .iter() + .map(|l| label_code(&f.property, l)) + .collect(); + match f.operator { + Operator::Eq => x == label_code(&f.property, one()), + Operator::Neq => x != label_code(&f.property, one()), + Operator::In => set.contains(&x), + Operator::NotIn => !set.contains(&x), + op => panic!("{op:?} on a coded property"), + } + } + Val::Int(Some(x)) => { + let n = |s: &str| s.parse::().expect("int"); + let set: Vec = f.values.iter().map(|s| n(s)).collect(); + match f.operator { + Operator::Eq => x == n(one()), + Operator::Neq => x != n(one()), + Operator::Lt => x < n(one()), + Operator::Lte => x <= n(one()), + Operator::Gt => x > n(one()), + Operator::Gte => x >= n(one()), + Operator::Between => { + x >= n(one()) && x <= n(f.high_value.as_deref().expect("high")) + } + Operator::In => set.contains(&x), + Operator::NotIn => !set.contains(&x), + op => panic!("{op:?} on an integer property"), + } + } + _ => unreachable!("missing values returned above"), + } +} + +fn oracle(rec: &Records, groups: &[FilterGroup]) -> Vec { + (0..N) + .filter(|&row| { + rec.live[row] + && (groups.is_empty() + || groups + .iter() + .any(|g| g.filters.iter().all(|f| keeps(rec, f, row)))) + }) + .collect() +} + +// ----------------------------------------------------------------- engine ---- + +fn engine(res: &Resident, cat: &Catalog, groups: &[FilterGroup]) -> Vec { + let filter = compile("contacts", groups, cat).expect("compile"); + let program = lower(&Query { + filter, + agg: Agg::Rows, + }) + .expect("lower"); + let masks: Vec<&[u64]> = res.masks.iter().map(Vec::as_slice).collect(); + let lanes = [ + LaneRef::U32(&res.stage), + LaneRef::U32(&res.country), + LaneRef::I32(&res.amount), + LaneRef::I32(&res.created), + ]; + let planes = Planes { + n_rows: N, + masks: &masks, + lanes: &lanes, + }; + let mut out = vec![0u64; words_for(N)]; + let mut scratch = Scratch::for_program(&program, N).expect("scratch"); + execute_into( + &program, + &planes, + &Foreign::NONE, + &mut scratch, + Out::Mask(&mut out), + ) + .expect("execute"); + lance_graph_mask_risc::materialize_rows(&out, N) +} + +// ------------------------------------------------------------- generator ---- + +fn random_filter(r: &mut Rng) -> PropertyFilter { + let pick = |r: &mut Rng, l: &[&'static str]| l[r.below(l.len() as u64) as usize]; + let int = |r: &mut Rng, lo: i64, hi: i64| (lo + r.below((hi - lo) as u64) as i64).to_string(); + match r.below(4) { + p @ (0 | 1) => { + let (name, labels) = if p == 0 { + ("lifecyclestage", STAGES) + } else { + ("country", COUNTRIES) + }; + match r.below(6) { + 0 => PropertyFilter::new(name, Operator::Eq, pick(r, labels)), + 1 => PropertyFilter::new(name, Operator::Neq, pick(r, labels)), + 2 | 3 => { + let k = 1 + r.below(3) as usize; + let vs: Vec<&str> = (0..k).map(|_| pick(r, labels)).collect(); + let op = if r.chance(50) { + Operator::In + } else { + Operator::NotIn + }; + PropertyFilter::set(name, op, &vs) + } + 4 => PropertyFilter::presence(name, Operator::HasProperty), + _ => PropertyFilter::presence(name, Operator::NotHasProperty), + } + } + p => { + let (name, lo, hi) = if p == 2 { + ("amount", -600, 5_100) + } else { + ("createdate", -5, 370) + }; + match r.below(10) { + 0 => PropertyFilter::new(name, Operator::Eq, &int(r, lo, hi)), + 1 => PropertyFilter::new(name, Operator::Neq, &int(r, lo, hi)), + 2 => PropertyFilter::new(name, Operator::Lt, &int(r, lo, hi)), + 3 => PropertyFilter::new(name, Operator::Lte, &int(r, lo, hi)), + 4 => PropertyFilter::new(name, Operator::Gt, &int(r, lo, hi)), + 5 => PropertyFilter::new(name, Operator::Gte, &int(r, lo, hi)), + 6 => { + let a = int(r, lo, hi); + let b = int(r, lo, hi); + PropertyFilter::between(name, &a, &b) + } + 7 => { + let vs: Vec = (0..3).map(|_| int(r, lo, hi)).collect(); + let vs: Vec<&str> = vs.iter().map(String::as_str).collect(); + let op = if r.chance(50) { + Operator::In + } else { + Operator::NotIn + }; + PropertyFilter::set(name, op, &vs) + } + 8 => PropertyFilter::presence(name, Operator::HasProperty), + _ => PropertyFilter::presence(name, Operator::NotHasProperty), + } + } + } +} + +fn random_groups(r: &mut Rng) -> Vec { + (0..1 + r.below(3)) + .map(|_| FilterGroup { + filters: (0..1 + r.below(4)).map(|_| random_filter(r)).collect(), + }) + .collect() +} + +// ------------------------------------------------------------------ tests ---- + +#[test] +fn random_filter_groups_keep_exactly_the_oracle_rows() { + let (rec, res) = records(); + let cat = catalog(); + let mut r = Rng(0xF1_17E5); + let (mut empty, mut full, mut nontrivial) = (0, 0, 0); + let live = oracle(&rec, &[]).len(); + for case in 0..600 { + let groups = random_groups(&mut r); + let want = oracle(&rec, &groups); + let got = engine(&res, &cat, &groups); + assert_eq!(got, want, "case {case}: {groups:?}"); + match want.len() { + 0 => empty += 1, + n if n == live => full += 1, + _ => nontrivial += 1, + } + } + // Anti-vacuity: most cases must select a proper, non-empty subset, or + // agreement would say little. + eprintln!("cases: {nontrivial} non-trivial, {empty} empty, {full} all live"); + assert!( + nontrivial > 450, + "only {nontrivial} non-trivial cases ({empty} empty, {full} full)" + ); +} + +#[test] +fn no_groups_is_every_live_record() { + let (rec, res) = records(); + let got = engine(&res, &catalog(), &[]); + assert_eq!(got.len(), rec.live.iter().filter(|l| **l).count()); + assert!(got.len() < N, "the fixture must delete some records"); +} + +#[test] +fn a_missing_value_is_kept_by_no_comparison() { + let (rec, res) = records(); + let cat = catalog(); + for f in [ + PropertyFilter::new("country", Operator::Neq, "DE"), + PropertyFilter::set("country", Operator::NotIn, &["DE", "AT"]), + PropertyFilter::new("amount", Operator::Neq, "0"), + PropertyFilter::set("amount", Operator::NotIn, &["1", "2"]), + ] { + let got = engine( + &res, + &cat, + &[FilterGroup { + filters: vec![f.clone()], + }], + ); + assert!(!got.is_empty(), "{f:?} keeps nothing"); + let missing = |row: usize| match f.property.as_str() { + "country" => rec.country[row].is_none(), + _ => rec.amount[row].is_none(), + }; + assert!( + got.iter().all(|&row| !missing(row)), + "{f:?} kept a record with no value" + ); + } +} + +#[test] +fn presence_on_a_required_property_is_all_or_nothing() { + let (rec, res) = records(); + let cat = catalog(); + let live = rec.live.iter().filter(|l| **l).count(); + let has = engine( + &res, + &cat, + &[FilterGroup { + filters: vec![PropertyFilter::presence( + "createdate", + Operator::HasProperty, + )], + }], + ); + let lacks = engine( + &res, + &cat, + &[FilterGroup { + filters: vec![PropertyFilter::presence( + "createdate", + Operator::NotHasProperty, + )], + }], + ); + assert_eq!((has.len(), lacks.len()), (live, 0)); +} + +#[test] +fn malformed_filters_are_refused_by_name() { + let cat = catalog(); + let one = |f: PropertyFilter| compile("contacts", &[FilterGroup { filters: vec![f] }], &cat); + assert_eq!( + compile("deals", &[], &cat), + Err(CompileError::UnknownObject("deals".into())) + ); + assert_eq!( + one(PropertyFilter::new("email", Operator::Eq, "a@b.c")), + Err(CompileError::UnknownProperty("email".into())) + ); + assert_eq!( + one(PropertyFilter::new( + "lifecyclestage", + Operator::Eq, + "evangelist" + )), + Err(CompileError::UnknownValue { + property: "lifecyclestage".into(), + value: "evangelist".into() + }) + ); + assert_eq!( + one(PropertyFilter::set("country", Operator::In, &["DE", "XX"])), + Err(CompileError::UnknownValue { + property: "country".into(), + value: "XX".into() + }) + ); + assert_eq!( + one(PropertyFilter::new("amount", Operator::Gt, "100.50")), + Err(CompileError::NotAnInteger { + property: "amount".into(), + value: "100.50".into() + }) + ); + assert_eq!( + one(PropertyFilter::new("lifecyclestage", Operator::Lt, "lead")), + Err(CompileError::OperatorNotSupported { + property: "lifecyclestage".into(), + operator: Operator::Lt + }) + ); + assert_eq!( + one(PropertyFilter::new("country", Operator::ContainsToken, "D")), + Err(CompileError::TextSearch { + property: "country".into() + }) + ); + assert_eq!( + one(PropertyFilter::set("amount", Operator::In, &[])), + Err(CompileError::MissingValue { + property: "amount".into(), + operator: Operator::In + }) + ); + assert_eq!( + one(PropertyFilter { + high_value: None, + ..PropertyFilter::between("amount", "1", "2") + }), + Err(CompileError::MissingValue { + property: "amount".into(), + operator: Operator::Between + }) + ); + assert_eq!( + compile("contacts", &[FilterGroup::default()], &cat), + Err(CompileError::EmptyGroup) + ); +} + +#[test] +fn hubspot_operator_names_round_trip() { + for name in [ + "EQ", + "NEQ", + "LT", + "LTE", + "GT", + "GTE", + "BETWEEN", + "IN", + "NOT_IN", + "HAS_PROPERTY", + "NOT_HAS_PROPERTY", + "CONTAINS_TOKEN", + "NOT_CONTAINS_TOKEN", + ] { + assert!(Operator::from_hubspot(name).is_some(), "{name}"); + } + assert_eq!(Operator::from_hubspot("eq"), None); +} diff --git a/crates/hubspo-store/Cargo.toml b/crates/hubspo-store/Cargo.toml index 3ac6a15..c08b64f 100644 --- a/crates/hubspo-store/Cargo.toml +++ b/crates/hubspo-store/Cargo.toml @@ -10,3 +10,6 @@ publish = false [lints] workspace = true + +[dependencies] +lance-graph-quack.workspace = true diff --git a/crates/hubspo-store/src/catalog.rs b/crates/hubspo-store/src/catalog.rs new file mode 100644 index 0000000..ef1b3a5 --- /dev/null +++ b/crates/hubspo-store/src/catalog.rs @@ -0,0 +1,358 @@ +//! The table and field catalog that Quack binds through (H-ADR-6). +//! +//! `lance-graph-quack::bind` names two seams and implements neither: +//! [`Registrar`] (who owns the table catalog) and [`Binder`] (names and text +//! literals → numeric identities). HubSPO-rs owns both here. Proven first in +//! probe P-QDB-1 against a plain-loop oracle. +//! +//! Text stops at this module. A coded field carries a [`Codebook`]: its +//! labels are numbered `0..n` in the order given, a literal binds to its +//! number once, and an unknown literal binds to nothing ([`Binder::code`] +//! returns `None`; nothing is minted at bind time). A coded field without a +//! codebook (a foreign key) never binds a text literal. +//! +//! Plane convention: every table's resident plane `0` is its live plane +//! ([`LIVE`]). A nullable field names its own validity plane. + +use std::collections::HashMap; + +use lance_graph_quack::bind::{Binder, BoundField, FieldKind, RegisterError, Registrar, TableId}; +use lance_graph_quack::{Col, Mask}; + +/// Every table's live plane: resident mask `0`. +pub const LIVE: Mask = Mask(0); + +/// Labels of one coded field, numbered in the order given. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Codebook { + labels: Vec, + codes: HashMap, +} + +impl Codebook { + /// A codebook over `labels`; label `i` gets code `i`. + /// + /// # Errors + /// + /// [`CatalogError::DuplicateLabel`] if a label repeats: two codes for one + /// label would make a binding depend on which one was looked up. + pub fn new>(labels: impl IntoIterator) -> Result { + let labels: Vec = labels.into_iter().map(Into::into).collect(); + let mut codes = HashMap::with_capacity(labels.len()); + for (i, label) in labels.iter().enumerate() { + let code = u32::try_from(i).map_err(|_| CatalogError::CodebookTooLarge)?; + if codes.insert(label.clone(), code).is_some() { + return Err(CatalogError::DuplicateLabel(label.clone())); + } + } + Ok(Self { labels, codes }) + } + + /// The code of `label`, if the codebook has it. + pub fn code(&self, label: &str) -> Option { + self.codes.get(label).copied() + } + + /// The label of `code`, if the codebook has it (hydration, last). + pub fn label(&self, code: u32) -> Option<&str> { + self.labels.get(code as usize).map(String::as_str) + } + + /// Number of labels. + pub fn len(&self) -> usize { + self.labels.len() + } + + /// Whether the codebook has no labels. + pub fn is_empty(&self) -> bool { + self.labels.is_empty() + } +} + +/// One field to declare on a table. +#[derive(Debug, Clone)] +pub struct FieldSpec { + /// The field's name, as filters spell it. + pub name: String, + /// The lane the executor reads. + pub col: Col, + /// What a literal for it must be. + pub kind: FieldKind, + /// The validity plane when the field is nullable. + pub validity: Option, + /// Labels for a coded field; `None` for a foreign key or an integer. + pub codebook: Option, +} + +/// Why a declaration was refused. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum CatalogError { + /// No table with this id. + UnknownTable(TableId), + /// The table already has a field with this name. + DuplicateField(String), + /// The table already has a field on this lane. + DuplicateColumn(Col), + /// A codebook on an integer field: integer literals never go through one. + CodebookOnInteger(String), + /// A field's validity plane is the live plane. + ValidityIsLive(String), + /// A label appears twice in one codebook. + DuplicateLabel(String), + /// More labels than `u32` codes. + CodebookTooLarge, +} + +#[derive(Debug, Clone)] +struct TableMeta { + name: String, + width: u32, + fields: Vec<(String, BoundField, Option)>, +} + +/// HubSPO-rs's catalog: the production [`Registrar`] and [`Binder`]. +#[derive(Debug, Clone, Default)] +pub struct Catalog { + tables: Vec, +} + +impl Catalog { + /// An empty catalog. + pub fn new() -> Self { + Self::default() + } + + /// Declare a field on a registered table. + /// + /// # Errors + /// + /// See [`CatalogError`]; nothing is changed on error. + pub fn declare(&mut self, table: TableId, spec: FieldSpec) -> Result<(), CatalogError> { + let meta = self + .tables + .get_mut(table.0 as usize) + .ok_or(CatalogError::UnknownTable(table))?; + if meta.fields.iter().any(|f| f.0 == spec.name) { + return Err(CatalogError::DuplicateField(spec.name)); + } + if meta.fields.iter().any(|f| f.1.col == spec.col) { + return Err(CatalogError::DuplicateColumn(spec.col)); + } + if spec.kind == FieldKind::I32 && spec.codebook.is_some() { + return Err(CatalogError::CodebookOnInteger(spec.name)); + } + if spec.validity == Some(LIVE) { + return Err(CatalogError::ValidityIsLive(spec.name)); + } + meta.fields.push(( + spec.name, + BoundField { + col: spec.col, + kind: spec.kind, + validity: spec.validity, + }, + spec.codebook, + )); + Ok(()) + } + + /// The codebook of a field, for hydrating codes back to labels. + pub fn codebook(&self, table: TableId, field: &str) -> Option<&Codebook> { + self.field_meta(table, field)?.2.as_ref() + } + + /// A table's declared row width in bytes. + pub fn width(&self, table: TableId) -> Option { + self.tables.get(table.0 as usize).map(|t| t.width) + } + + fn field_meta( + &self, + table: TableId, + field: &str, + ) -> Option<&(String, BoundField, Option)> { + self.tables + .get(table.0 as usize)? + .fields + .iter() + .find(|f| f.0 == field) + } +} + +impl Registrar for Catalog { + fn register(&mut self, name: &str, width: u32) -> Result { + if self.tables.iter().any(|t| t.name == name) { + return Err(RegisterError::Exists(name.into())); + } + // Lanes are 4-byte (`U32` / `I32`); a width that is not a whole + // number of lanes has no layout here. + if !width.is_multiple_of(4) { + return Err(RegisterError::Refused { + table: name.into(), + width, + }); + } + let id = u32::try_from(self.tables.len()).map_err(|_| RegisterError::Refused { + table: name.into(), + width, + })?; + self.tables.push(TableMeta { + name: name.into(), + width, + fields: Vec::new(), + }); + Ok(TableId(id)) + } +} + +impl Binder for Catalog { + fn table(&self, name: &str) -> Option { + self.tables + .iter() + .position(|t| t.name == name) + .and_then(|i| u32::try_from(i).ok()) + .map(TableId) + } + + fn live(&self, _table: TableId) -> Mask { + LIVE + } + + fn field(&self, table: TableId, name: &str) -> Option { + self.field_meta(table, name).map(|f| f.1) + } + + fn code(&self, table: TableId, field: Col, literal: &str) -> Option { + self.tables + .get(table.0 as usize)? + .fields + .iter() + .find(|f| f.1.col == field)? + .2 + .as_ref()? + .code(literal) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use lance_graph_quack::bind::create_table; + + fn contact() -> (Catalog, TableId) { + let mut c = Catalog::new(); + let t = create_table("contact") + .width(8) + .register(&mut c) + .expect("register") + .id; + c.declare( + t, + FieldSpec { + name: "lifecyclestage".into(), + col: Col(0), + kind: FieldKind::Code, + validity: None, + codebook: Some(Codebook::new(["lead", "customer"]).expect("codebook")), + }, + ) + .expect("declare"); + (c, t) + } + + #[test] + fn literals_bind_to_their_position_and_unknown_ones_to_nothing() { + let (c, t) = contact(); + assert_eq!(c.code(t, Col(0), "lead"), Some(0)); + assert_eq!(c.code(t, Col(0), "customer"), Some(1)); + assert_eq!(c.code(t, Col(0), "evangelist"), None); + assert_eq!( + c.codebook(t, "lifecyclestage").and_then(|b| b.label(1)), + Some("customer") + ); + } + + #[test] + fn a_field_without_a_codebook_binds_no_text() { + let (mut c, t) = contact(); + c.declare( + t, + FieldSpec { + name: "owner".into(), + col: Col(1), + kind: FieldKind::Code, + validity: Some(Mask(1)), + codebook: None, + }, + ) + .expect("declare"); + assert_eq!(c.code(t, Col(1), "lead"), None); + assert_eq!(c.field(t, "owner").map(|f| f.validity), Some(Some(Mask(1)))); + } + + #[test] + fn declarations_that_would_make_binding_ambiguous_are_refused() { + let (mut c, t) = contact(); + let spec = |name: &str, col: u16| FieldSpec { + name: name.into(), + col: Col(col), + kind: FieldKind::I32, + validity: None, + codebook: None, + }; + assert_eq!( + c.declare(t, spec("lifecyclestage", 5)), + Err(CatalogError::DuplicateField("lifecyclestage".into())) + ); + assert_eq!( + c.declare(t, spec("amount", 0)), + Err(CatalogError::DuplicateColumn(Col(0))) + ); + let mut coded = spec("amount", 2); + coded.codebook = Some(Codebook::new(["x"]).expect("codebook")); + assert_eq!( + c.declare(t, coded), + Err(CatalogError::CodebookOnInteger("amount".into())) + ); + let mut live = spec("amount", 2); + live.validity = Some(LIVE); + assert_eq!( + c.declare(t, live), + Err(CatalogError::ValidityIsLive("amount".into())) + ); + assert_eq!( + Codebook::new(["a", "b", "a"]), + Err(CatalogError::DuplicateLabel("a".into())) + ); + // The good declaration still goes through after the refusals. + assert_eq!(c.declare(t, spec("amount", 2)), Ok(())); + assert_eq!( + c.declare(TableId(9), spec("x", 3)), + Err(CatalogError::UnknownTable(TableId(9))) + ); + } + + #[test] + fn tables_register_once_and_only_in_whole_lanes() { + let (mut c, _) = contact(); + assert_eq!( + create_table("contact") + .width(8) + .register(&mut c) + .map(|r| r.id), + Err(RegisterError::Exists("contact".into())) + ); + assert!(matches!( + create_table("deal").width(6).register(&mut c), + Err(RegisterError::Refused { .. }) + )); + let deal = create_table("deal") + .width(12) + .register(&mut c) + .expect("deal") + .id; + assert_eq!(c.table("deal"), Some(deal)); + assert_eq!(c.width(deal), Some(12)); + assert_eq!(c.table("company"), None); + } +} diff --git a/crates/hubspo-store/src/lib.rs b/crates/hubspo-store/src/lib.rs index a5342fd..7ce19e2 100644 --- a/crates/hubspo-store/src/lib.rs +++ b/crates/hubspo-store/src/lib.rs @@ -4,5 +4,7 @@ //! //! - Zone: zone 2 (no serde, no wire formats). //! - Deliverables: D-STO-1..6 (`.claude/plans/hubspo-integration-v1.md`). -//! - First real code lands in wave W2. Until then this crate is empty on -//! purpose; Wave 0 only fixes the workspace shape. +//! - First module: [`catalog`], the table and field catalog every Quack read +//! binds through (H-ADR-6). Storage lands with D-STO-1. + +pub mod catalog;