From a9fa7ba5a9ead1e736ad7c2b39e560d20d50711e Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 15:44:51 -0800 Subject: [PATCH 001/162] diet: run as a deployed model The script declares the solve with problem.configure, deploys the model, waits for the first refresh and prints the report from the deployed model. Status and objective are read through the model's own relationships. The README covers the deployment config, the internal schema grants made before the first run, and re-running. --- diet/README.md | 52 ++++++++++++++++++++++++++++++++++++++++++++----- diet/diet.py | 52 ++++++++++++++++++++++++++++++++----------------- diet/runbook.md | 2 +- 3 files changed, 82 insertions(+), 24 deletions(-) diff --git a/diet/README.md b/diet/README.md index 1b2d6f57..f043514a 100644 --- a/diet/README.md +++ b/diet/README.md @@ -36,7 +36,7 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ## What's included - **Model**: two concepts (`Food`, `Nutrient`), a `Scenario` concept, per-food decision variables, nutrient-bound constraints, and a cost-minimizing objective — all in `diet.py`. -- **Runner**: `diet.py`, a single Python script that runs end-to-end against a Snowflake-connected RAI account. +- **Runner**: `diet.py`, a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/foods.csv` (foods with cost and per-nutrient content) and `data/nutrients.csv` (nutrient min/max bounds). - **Outputs**: per-scenario termination status, objective cost, and a table of the foods (and amounts) in each least-cost basket, printed to stdout. @@ -45,7 +45,8 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -73,15 +74,56 @@ Built using **prescriptive reasoning** (linear programming with continuous decis python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: diet_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .DIET # published views and tables + identity: + internal_schema: .DIET_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python diet.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .DIET_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .DIET_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .DIET_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .DIET_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .DIET_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .DIET_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .DIET_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .DIET_META TO APPLICATION ; ``` + The deploy creates the public schema `.DIET` if it doesn't exist. 5. Run: ```bash python diet.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .DIET_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory. 6. Expected output — a few lines confirm a successful run: diff --git a/diet/diet.py b/diet/diet.py index 7ab734e3..fbab3e41 100644 --- a/diet/diet.py +++ b/diet/diet.py @@ -16,7 +16,9 @@ - One solve handles all scaling levels; results extracted via model.select(). Run: - `python diet.py` + `python diet.py` deploys the model to the schema in raiconfig.yaml, waits for + its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints per-scenario termination status, objective value, and a table of foods @@ -108,26 +110,40 @@ # Solve (single solve for all scenarios) # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) -# Check solve status — engine-side IC avoids querying data to the client. -model.require(problem.termination_status() == "OPTIMAL") +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() -# Python-side solve info for formatted output. -si = problem.solve_info() -si.display() +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"diet solve status is {String:status}") +TotalCost = model.Relationship(f"diet total cost is {Float:cost}") +model.define(SolveStatus(problem.termination_status()), TotalCost(problem.objective_value())) # -------------------------------------------------- -# Extract results per scenario +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nDiet plan per scenario:") -model.select( - Scenario.scenario_name.alias("scenario"), - Food.name.alias("food"), - Food.cost, - x_amt.alias("amount"), -).where( - Food.x_amount(Scenario, x_amt), x_amt > 0.001 -).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal diet plan: termination status {status}") + print(f"• objective: {round(cost[0], 9)}\n") + + print("\nDiet plan per scenario:") + model.select( + Scenario.scenario_name.alias("scenario"), + Food.name.alias("food"), + Food.cost, + x_amt.alias("amount"), + ).where(Food.x_amount(Scenario, x_amt), x_amt > 0.001).inspect() diff --git a/diet/runbook.md b/diet/runbook.md index 7fa7118a..6fd61393 100644 --- a/diet/runbook.md +++ b/diet/runbook.md @@ -38,7 +38,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem What's the minimum-cost amount of each food (continuous, non-negative) that keeps every nutrient's total intake within its [min, max] bounds? Solve three requirement scenarios in one solve — bounds scaled to 80%, 100%, and 120% — with the scenario as a first-class concept and the amount decision indexed by (food, scenario). Minimize total cost across all three, and persist the amounts to the ontology. -**Response:** OPTIMAL (HiGHS), total objective $35.49 across the three scenarios. The amount decision is written back as `Food.x_amount(Scenario) -> amount` so the next step can read it; the solve is sub-second (18 simplex iterations). +**Response:** OPTIMAL (HiGHS), total objective $35.49 across the three scenarios. The amount decision is written back as `Food.x_amount(Scenario) -> amount` so the next step can read it; the solve is sub-second (18 simplex iterations). In `diet.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the per-scenario diet plan From 0548ea983f090ff33f05d26ce497b1b65a5b7cc7 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 15:44:51 -0800 Subject: [PATCH 002/162] factory_production: run as a deployed model The solve, with sensitivity analysis, is declared with problem.configure and runs on each refresh of the deployed model. The report reads the plan through Product.x_quantity and checks status, profit, shadow prices, reduced costs and basis status in Python, since model.require checks are not evaluated on a deployed model. --- factory_production/README.md | 54 ++++- factory_production/factory_production.py | 276 +++++++++++------------ 2 files changed, 184 insertions(+), 146 deletions(-) diff --git a/factory_production/README.md b/factory_production/README.md index 3cd4ea55..cb3d2af6 100644 --- a/factory_production/README.md +++ b/factory_production/README.md @@ -51,7 +51,8 @@ Manufacturing operations must decide how much of each product to make at each fa ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -80,15 +81,56 @@ Manufacturing operations must decide how much of each product to make at each fa python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: factory_production_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .FACTORY_PRODUCTION # published views and tables + identity: + internal_schema: .FACTORY_PRODUCTION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python factory_production.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .FACTORY_PRODUCTION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; ``` + The deploy creates the public schema `.FACTORY_PRODUCTION` if it doesn't exist. 5. Run: ```bash python factory_production.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .FACTORY_PRODUCTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory. 6. Expected output (model and solver display trimmed): ```text @@ -103,7 +145,7 @@ Manufacturing operations must decide how much of each product to make at each fa Factory capacity shadow prices (d profit / d hour): factory avail shadow_price - amazing_brewery 30.0 -0.0 + amazing_brewery 30.0 0.0 steel_factory 40.0 4200.0 Product reduced costs and basis status: @@ -111,7 +153,7 @@ Manufacturing operations must decide how much of each product to make at each fa amazing_brewery ales 2.0 NONBASIC_AT_UPPER amazing_brewery stouts 4.0 NONBASIC_AT_UPPER steel_factory bands 4.0 NONBASIC_AT_UPPER - steel_factory coils -0.0 BASIC + steel_factory coils 0.0 BASIC Most profit-sensitive capacity: steel_factory (d profit / d hour = +4200.00) diff --git a/factory_production/factory_production.py b/factory_production/factory_production.py index db7c3f47..c573fc1b 100644 --- a/factory_production/factory_production.py +++ b/factory_production/factory_production.py @@ -12,7 +12,7 @@ - Maximize total profit. A plain solve answers "what is the most profitable production plan?". Requesting -``solve(sensitivity=True)`` ALSO answers the marginal questions -- in one solve, +``sensitivity=True`` on the solve ALSO answers the marginal questions -- in one solve, read straight off the variable / constraint objects (the same attribute style as ``.name`` or ``.lower``): @@ -42,7 +42,9 @@ is derived automatically from its field names; a constraint's is declared.) Run: - `python factory_production.py` + `python factory_production.py` deploys the model to the schema in raiconfig.yaml, waits for + its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints the production plan, each factory's capacity shadow price, each product's @@ -52,7 +54,7 @@ from pathlib import Path from pandas import read_csv -from relationalai.semantics import Float, Integer, Model, String, std, sum +from relationalai.semantics import Float, Integer, Model, String, sum from relationalai.semantics.reasoners.prescriptive import Problem # -------------------------------------------------- @@ -131,13 +133,12 @@ # Variable: production quantity per product. The demand cap is the variable's UPPER # BOUND (not a separate constraint), so its marginal surfaces as the variable's -# reduced cost below. populate=False keeps the solution read explicit via values(). +# reduced cost below. quantity_var = problem.solve_for( Product.x_quantity, name=["qty", Product.factory.name, Product.name], lower=0, upper=Product.demand, - populate=False, ) # Constraint: each factory's total resource usage <= its available hours, over the @@ -158,149 +159,144 @@ # Objective: maximize total profit across all factories. problem.maximize(sum(Product.profit * Product.x_quantity)) -problem.display() -problem.solve("highs", time_limit_sec=60, sensitivity=True) -si = problem.solve_info() -si.display() - -assert si.termination_status == "OPTIMAL" -assert si.sensitivity is True -# Optimum: steel_factory fills all 40 hrs (bands to its 6000 cap = 30 hrs, coils takes -# the last 10 hrs = 1400 units); amazing_brewery makes both products to their demand -# caps in 25 of its 30 hrs. Profit = 192000 + 8000 = 200000. -assert si.objective_value is not None and abs(si.objective_value - 200000) < 0.01, ( - f"total profit changed: expected 200000, got {si.objective_value}" -) +# Each refresh of the deployed model runs this solve, with sensitivity analysis. +problem.configure("highs", time_limit_sec=60, sensitivity=True) -print(f"\nBaseline status: {si.termination_status}, total profit: {si.objective_value:.2f}") - -# --- The solved production plan ------------------------------------------------- -# Read every product's solved quantity once via values(0, ref) on the variable (no -# populate). Columns are aliased so the printed headers read as factory / product / -# quantity; this one frame feeds the plan table, the plan assertion, and the capacity -# summary below. -plan_ref = Float.ref() -plan_df = ( - model.select( - quantity_var.product.factory.name.alias("factory"), - quantity_var.product.name.alias("product"), - quantity_var.product.factory.avail.alias("avail"), - quantity_var.product.rate.alias("rate"), - plan_ref.alias("quantity"), - ) - .where(quantity_var.values(0, plan_ref)) - .to_df() - .sort_values(["factory", "product"], ignore_index=True) -) -print("\nProduction plan:") -produced = plan_df.loc[plan_df["quantity"] > 1e-6, ["factory", "product", "quantity"]] -print(produced.to_string(index=False)) - -# Pin the whole plan, not just the objective scalar: in general a matching objective is -# necessary but not sufficient, because a different product mix could reach the same -# profit. This LP's optimum happens to be unique (each factory's hours pin its plan), so -# pinning the quantities both guards against a regression and documents that uniqueness. -# Product names are unique across factories (asserted above), so the per-name lookup is a -# single row; tolerance is tight because the vertices here are exactly integral. -EXPECTED_PLAN = {"bands": 6000.0, "coils": 1400.0, "stouts": 1000.0, "ales": 2000.0} -for product_name, expected_qty in EXPECTED_PLAN.items(): - actual_qty = plan_df.loc[plan_df["product"] == product_name, "quantity"].sum() - assert abs(actual_qty - expected_qty) < 0.01, ( - f"{product_name}: expected {expected_qty}, got {actual_qty}" - ) +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() -# --- Capacity shadow prices: which factory to expand first ---------------------- -# A constraint carries its declared entity key as a back-pointer (cap.factory), so a -# shadow price joins to that factory's own data by KEY -- no name parsing, no pandas join. -cap_df = ( - model.select( - cap.factory.name.alias("factory"), - cap.factory.avail.alias("avail"), - cap.shadow_price.alias("shadow_price"), - ) - .to_df() - .sort_values("factory", ignore_index=True) -) -print("\nFactory capacity shadow prices (d profit / d hour):") -print(cap_df.to_string(index=False)) -# Maximize + a <= capacity => shadow_price >= 0: an extra hour can only help. The -# implication runs one way: a factory with idle hours prices at 0 (slack -- its -# bottleneck is demand, not capacity), and a positive price marks a binding capacity. -# steel_factory is full (binding) and carries the marginal value of an hour; every other -# factory has spare hours (slack) and prices at 0. -model.where(cap.factory.name == "steel_factory").require(cap.shadow_price > 1e-6) -model.where(cap.factory.name != "steel_factory").require( - std.math.abs(cap.shadow_price) < 1e-6 -) +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"factory production solve status is {String:status}") +TotalProfit = model.Relationship(f"factory production total profit is {Float:profit}") +model.define(SolveStatus(problem.termination_status()), TotalProfit(problem.objective_value())) -# --- Reduced costs: which products are demand-capped? --------------------------- -# Each marginal reads straight off the variable; the variable's back-pointer -# (quantity_var.product) joins it to the product's factory by ENTITY KEY. -rc_df = ( - model.select( - quantity_var.product.factory.name.alias("factory"), - quantity_var.product.name.alias("product"), - quantity_var.reduced_cost.alias("reduced_cost"), - quantity_var.basis_status.alias("basis_status"), +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + profit = model.select(TotalProfit).to_df().iloc[:, 0].tolist() + assert status == ["OPTIMAL"], f"termination status {status}" + # Optimum: steel_factory fills all 40 hrs (bands to its 6000 cap = 30 hrs, coils takes + # the last 10 hrs = 1400 units); amazing_brewery makes both products to their demand + # caps in 25 of its 30 hrs. Profit = 192000 + 8000 = 200000. + assert len(profit) == 1 and abs(profit[0] - 200000) < 0.01, ( + f"total profit changed: expected 200000, got {profit}" ) - .to_df() - .sort_values(["factory", "product"], ignore_index=True) -) -print("\nProduct reduced costs and basis status:") -print(rc_df.to_string(index=False)) -# A product pinned at its demand cap is NONBASIC_AT_UPPER with reduced_cost >= 0 (the -# extra profit from one more unit of allowed demand); the swing product that sets a -# binding factory's marginal price is BASIC at ~0. Here bands, stouts and ales sit at -# their demand caps; coils is the swing product in steel_factory. Assert BOTH halves of -# the lesson -- the reduced cost AND the basis status -- so a regression in either is loud. -for capped in ("bands", "stouts", "ales"): - model.where(quantity_var.product.name == capped).require( - quantity_var.reduced_cost > 1e-6 + + print(f"\nBaseline status: {status[0]}, total profit: {profit[0]:.2f}") + + # --- The solved production plan ------------------------------------------------- + # Read every product's solved quantity once from Product.x_quantity, which the solve + # populates. Columns are aliased so the printed headers read as factory / product / + # quantity; this one frame feeds the plan table, the plan assertion, and the capacity + # summary below. + plan_df = ( + model.select( + Product.factory.name.alias("factory"), + Product.name.alias("product"), + Product.factory.avail.alias("avail"), + Product.rate.alias("rate"), + Product.x_quantity.alias("quantity"), + ) + .to_df() + .sort_values(["factory", "product"], ignore_index=True) ) - model.where(quantity_var.product.name == capped).require( - quantity_var.basis_status == "NONBASIC_AT_UPPER" + print("\nProduction plan:") + produced = plan_df.loc[plan_df["quantity"] > 1e-6, ["factory", "product", "quantity"]] + print(produced.to_string(index=False)) + + # Pin the whole plan, not just the objective scalar: in general a matching objective is + # necessary but not sufficient, because a different product mix could reach the same + # profit. This LP's optimum happens to be unique (each factory's hours pin its plan), so + # pinning the quantities both guards against a regression and documents that uniqueness. + # Product names are unique across factories (asserted above), so the per-name lookup is a + # single row; tolerance is tight because the vertices here are exactly integral. + EXPECTED_PLAN = {"bands": 6000.0, "coils": 1400.0, "stouts": 1000.0, "ales": 2000.0} + for product_name, expected_qty in EXPECTED_PLAN.items(): + actual_qty = plan_df.loc[plan_df["product"] == product_name, "quantity"].sum() + assert abs(actual_qty - expected_qty) < 0.01, f"{product_name}: expected {expected_qty}, got {actual_qty}" + + # --- Capacity shadow prices: which factory to expand first ---------------------- + # A constraint carries its declared entity key as a back-pointer (cap.factory), so a + # shadow price joins to that factory's own data by KEY -- no name parsing, no pandas join. + cap_df = ( + model.select( + cap.factory.name.alias("factory"), + cap.factory.avail.alias("avail"), + cap.shadow_price.alias("shadow_price"), + ) + .to_df() + .sort_values("factory", ignore_index=True) ) -# coils is basic (interior, between 0 and its demand cap): reduced cost ~0, status BASIC. -# This coils read runs AFTER the requires above, and a query is what validates pending -# relational requires -- so the same select that fetches coils also checks them. Zero- -# checks use a HiGHS float tolerance (1e-4); the requires use 1e-6 (both are exact-0 here). -coils_df = ( - model.select( - quantity_var.reduced_cost.alias("reduced_cost"), - quantity_var.basis_status.alias("basis_status"), + assert len(cap_df) == len(EXPECTED_FACTORIES), "missing capacity shadow prices" + print("\nFactory capacity shadow prices (d profit / d hour):") + print(cap_df.to_string(index=False)) + # Maximize + a <= capacity => shadow_price >= 0: an extra hour can only help. The + # implication runs one way: a factory with idle hours prices at 0 (slack -- its + # bottleneck is demand, not capacity), and a positive price marks a binding capacity. + # steel_factory is full (binding) and carries the marginal value of an hour; every other + # factory has spare hours (slack) and prices at 0. + steel = cap_df["factory"] == "steel_factory" + assert (cap_df.loc[steel, "shadow_price"] > 1e-6).all() + assert (cap_df.loc[~steel, "shadow_price"].abs() < 1e-6).all() + + # --- Reduced costs: which products are demand-capped? --------------------------- + # Each marginal reads straight off the variable; the variable's back-pointer + # (quantity_var.product) joins it to the product's factory by ENTITY KEY. + rc_df = ( + model.select( + quantity_var.product.factory.name.alias("factory"), + quantity_var.product.name.alias("product"), + quantity_var.reduced_cost.alias("reduced_cost"), + quantity_var.basis_status.alias("basis_status"), + ) + .to_df() + .sort_values(["factory", "product"], ignore_index=True) ) - .where(quantity_var.product.name == "coils") - .to_df() -) -assert (coils_df["reduced_cost"].abs() < 1e-4).all() -assert (coils_df["basis_status"] == "BASIC").all() - -# --- Acting on it: which factory's capacity is the bottleneck? ------------------ -# With a maximize objective every capacity shadow price is >= 0, so the largest one is -# the factory whose marginal hour earns the most -- the capacity to expand first. -bottleneck = max(zip(cap_df["factory"], cap_df["shadow_price"]), key=lambda fp: fp[1]) -print( - f"\nMost profit-sensitive capacity: {bottleneck[0]} (d profit / d hour = {bottleneck[1]:+.2f})" -) + assert len(rc_df) == len(EXPECTED_PRODUCTS), "missing reduced costs" + print("\nProduct reduced costs and basis status:") + print(rc_df.to_string(index=False)) + # A product pinned at its demand cap is NONBASIC_AT_UPPER with reduced_cost >= 0 (the + # extra profit from one more unit of allowed demand); the swing product that sets a + # binding factory's marginal price is BASIC at ~0. Here bands, stouts and ales sit at + # their demand caps; coils is the swing product in steel_factory. Assert BOTH halves of + # the lesson -- the reduced cost AND the basis status -- so a regression in either is loud. + capped = rc_df["product"].isin(["bands", "stouts", "ales"]) + assert (rc_df.loc[capped, "reduced_cost"] > 1e-6).all() + assert (rc_df.loc[capped, "basis_status"] == "NONBASIC_AT_UPPER").all() + # coils is basic (interior, between 0 and its demand cap): reduced cost ~0, status BASIC. + # The zero-check uses a HiGHS float tolerance (1e-4); it is exact-0 here. + coils = rc_df["product"] == "coils" + assert (rc_df.loc[coils, "reduced_cost"].abs() < 1e-4).all() + assert (rc_df.loc[coils, "basis_status"] == "BASIC").all() + + # --- Acting on it: which factory's capacity is the bottleneck? ------------------ + # With a maximize objective every capacity shadow price is >= 0, so the largest one is + # the factory whose marginal hour earns the most -- the capacity to expand first. + bottleneck = max(zip(cap_df["factory"], cap_df["shadow_price"]), key=lambda fp: fp[1]) + print(f"\nMost profit-sensitive capacity: {bottleneck[0]} (d profit / d hour = {bottleneck[1]:+.2f})") + + # -------------------------------------------------- + # Summary -- capacity utilization per factory + # -------------------------------------------------- + # Ties the shadow prices above back to the plan: in this data the binding factory has + # zero idle hours and a positive shadow price, while idle hours mean slack and a zero + # price. Only idle => zero-price holds in general; a binding capacity can still price at + # zero under degeneracy, so read the prices, not just the idle column. + plan_df["hours"] = plan_df["quantity"] / plan_df["rate"] + util = plan_df.groupby("factory").agg(avail=("avail", "first"), hours_used=("hours", "sum")).reset_index() + util["idle"] = util["avail"] - util["hours_used"] + print("\n" + "=" * 50) + print("Factory Capacity Summary") + print("=" * 50) + print(util.to_string(index=False)) -# -------------------------------------------------- -# Summary -- capacity utilization per factory -# -------------------------------------------------- -# Ties the shadow prices above back to the plan: in this data the binding factory has -# zero idle hours and a positive shadow price, while idle hours mean slack and a zero -# price. Only idle => zero-price holds in general; a binding capacity can still price at -# zero under degeneracy, so read the prices, not just the idle column. -plan_df["hours"] = plan_df["quantity"] / plan_df["rate"] -util = ( - plan_df.groupby("factory") - .agg(avail=("avail", "first"), hours_used=("hours", "sum")) - .reset_index() -) -util["idle"] = util["avail"] - util["hours_used"] -print("\n" + "=" * 50) -print("Factory Capacity Summary") -print("=" * 50) -print(util.to_string(index=False)) # -------------------------------------------------- # Customize From eaa14ca2aa4de967be137050004d622c5d52e2b8 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 15:48:53 -0800 Subject: [PATCH 003/162] bom-reachability: run as a deployed model Graph results the report reads are declared with the model, the script deploys it and waits for the first refresh, and the report reads the deployed model. Assembly depth is computed from the enumerated paths and printed; it is no longer written back onto SKU, since a rule defined after the deploy is not part of the deployed model. --- bom-reachability/README.md | 35 ++- bom-reachability/bom_reachability.py | 320 ++++++++++++++------------- bom-reachability/runbook.md | 8 +- 3 files changed, 191 insertions(+), 172 deletions(-) diff --git a/bom-reachability/README.md b/bom-reachability/README.md index e1b8c934..8eedb007 100644 --- a/bom-reachability/README.md +++ b/bom-reachability/README.md @@ -35,24 +35,25 @@ This template turns a BOM into a dependency graph so you can trace those transit - A full transitive dependency tree per finished good, broken down by type (component vs raw material), from all-pairs `reachable(full=True)`. - A most-depended-on ranking of SKUs and a betweenness-centrality ranking that flags the structural bottleneck components. - An enumeration of the end-to-end assembly chains that build each finished good, reduced to the maximal (non-extendable) chains via path enumeration. -- An `assembly_depth` property persisted back onto each SKU, so the longest chain terminating at each finished good is queryable as ontology after the run. +- Each finished good's assembly depth: the length of the longest chain that builds it. Built on RelationalAI's graph reasoner: **reachability** for transitive dependencies, **betweenness centrality** for bottlenecks, and **path enumeration** for assembly chains. ## What's included -- **Model**: two concepts (`SKU` and `BillOfMaterials`) wired into a directed dependency graph, plus a derived `feeds` self-relationship on `SKU` and a persisted `assembly_depth` property. -- **Runner**: `bom_reachability.py` — a single self-contained Python script that runs reachability, betweenness centrality, and assembly-path enumeration end to end. +- **Model**: two concepts (`SKU` and `BillOfMaterials`) wired into a directed dependency graph, plus a derived `feeds` self-relationship on `SKU`. +- **Runner**: `bom_reachability.py` — a single self-contained Python script that deploys the model to your Snowflake account and prints the reachability, betweenness centrality, and assembly-path report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/skus.csv` (9 SKUs across 3 tiers) and `data/bill_of_materials.csv` (14 BOM entries with site-specific assembly). See *Sample data* below. -- **Outputs**: printed transitive-dependency lists per finished good, a most-depended-on and betweenness ranking, the maximal assembly chains, and an `assembly_depth` property written back onto the SKU ontology. +- **Outputs**: printed transitive-dependency lists per finished good, a most-depended-on and betweenness ranking, the maximal assembly chains, and each finished good's assembly depth. ## Prerequisites ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -87,18 +88,32 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen python -m pip install . ``` -4. **Configure Snowflake connection and RAI profile** +4. **Configure the Snowflake connection and the deployment** - ```bash - rai init + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .BOM_REACHABILITY # published views and tables + identity: + internal_schema: .BOM_REACHABILITY_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python bom_reachability.py` stops with an error naming the missing setting. + 5. **Run the template** ```bash python bom_reachability.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + 6. **Expected output** (a few lines confirm success): ```text @@ -139,7 +154,7 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen - **Primary identifiers**: `SKU` by `id`; `BillOfMaterials` by `id`. - **Important invariants**: each `BillOfMaterials` row points an output SKU at one input SKU ("depends on"); the BOM is acyclic (raw materials feed components feed finished goods, never back). -The model also derives a `feeds` self-relationship on `SKU` (input SKU feeds output SKU), built from the `BillOfMaterials` intermediary and used to enumerate assembly paths, plus a persisted `assembly_depth` property (the longest assembly chain terminating at each SKU). +The model also derives a `feeds` self-relationship on `SKU` (input SKU feeds output SKU), built from the `BillOfMaterials` intermediary and used to enumerate assembly paths. For the full concept and property definitions, see `bom_reachability.py`; `runbook.md` builds them step by step with the RAI skills. @@ -169,7 +184,7 @@ Betweenness centrality ranks components by how many shortest dependency paths pa ### 5. Enumerate assembly paths (PREVIEW, requires `relationalai>=1.15`) -Where reachability returns dependency *pairs*, path enumeration returns the actual *build sequences*. It walks the derived `feeds` edge to enumerate every assembly path; because the BOM is acyclic, this yields exactly the simple paths with no cycle risk. A maximal-paths view keeps only the longest non-extendable chains, and the longest assembly depth is persisted as `SKU.assembly_depth`. +Where reachability returns dependency *pairs*, path enumeration returns the actual *build sequences*. It walks the derived `feeds` edge to enumerate every assembly path; because the BOM is acyclic, this yields exactly the simple paths with no cycle risk. A maximal-paths view keeps only the longest non-extendable chains, and the deepest chain ending at each finished good gives its assembly depth. See `bom_reachability.py` for the implementation and `runbook.md` for the skill-driven reproduction. diff --git a/bom-reachability/bom_reachability.py b/bom-reachability/bom_reachability.py index 70afafba..8e5d407d 100644 --- a/bom-reachability/bom_reachability.py +++ b/bom-reachability/bom_reachability.py @@ -15,11 +15,11 @@ SKU->SKU "feeds into" edge from the BillOfMaterials intermediary (input feeds output), enumerates every assembly path with model.path(...).all().paths(), filters to the maximal (longest non-extendable) chains so sub-paths are - suppressed, and persists each finished good's longest assembly depth back onto - the SKU ontology. + suppressed, and reports each finished good's longest assembly depth. Run: - `python bom_reachability.py` + `python bom_reachability.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh, and prints the report from the deployed model. Output: Prints transitive dependency lists per finished product, a betweenness @@ -31,7 +31,7 @@ from pathlib import Path import pandas as pd -from relationalai.semantics import Float, Integer, Model, String, where +from relationalai.semantics import Float, Model, String, where from relationalai.semantics.reasoners.graph import Graph # -------------------------------------------------- @@ -87,96 +87,16 @@ edge_dst_relationship=BillOfMaterials.input_sku, ) -print("=== BOM Dependency Graph ===") -graph.num_nodes().inspect() -graph.num_edges().inspect() - -# -------------------------------------------------- -# Reachability: all transitive dependency pairs -# -------------------------------------------------- +# Graph results the report reads, declared here so they deploy with the model. +num_nodes = graph.num_nodes() +num_edges = graph.num_edges() +# Reachability: all transitive dependency pairs. reachable = graph.reachable(full=True) -src, dst = graph.Node.ref("src"), graph.Node.ref("dst") -all_deps_df = ( - where(reachable(src, dst)) - .select( - src.id.alias("product_id"), - src.name.alias("product_name"), - src.type.alias("product_type"), - dst.id.alias("dep_id"), - dst.name.alias("dep_name"), - dst.type.alias("dep_type"), - ) - .to_df() -) - -print(f"\nTotal reachable pairs: {len(all_deps_df)}") - -# -------------------------------------------------- -# Dependencies of each finished good -# -------------------------------------------------- - -finished = all_deps_df[all_deps_df["product_type"] == "FINISHED_GOOD"] -for product_id in sorted(finished["product_id"].unique()): - deps = finished[(finished["product_id"] == product_id) & (finished["dep_id"] != product_id)] - if len(deps) > 0: - product_name = deps["product_name"].iloc[0] - print(f"\n--- Dependencies of '{product_name}' ({product_id}) ---") - print(f" {len(deps)} transitive dependencies:") - for dep_type in ["COMPONENT", "RAW_MATERIAL"]: - type_deps = deps[deps["dep_type"] == dep_type] - if len(type_deps) > 0: - print(f"\n {dep_type}s ({len(type_deps)}):") - for _, row in type_deps.sort_values("dep_name").iterrows(): - print(f" - {row['dep_name']} ({row['dep_id']})") - -# -------------------------------------------------- -# Most depended-on components -# -------------------------------------------------- - -non_self = all_deps_df[all_deps_df["product_id"] != all_deps_df["dep_id"]] -if len(non_self) > 0: - dep_counts = ( - non_self.groupby(["dep_id", "dep_name", "dep_type"]) - .size() - .reset_index(name="depended_on_by") - .sort_values("depended_on_by", ascending=False) - ) - print("\n=== Most Depended-On SKUs ===") - print(dep_counts.to_string(index=False)) - -# -------------------------------------------------- -# Betweenness centrality: structural bottlenecks -# -------------------------------------------------- - +# Betweenness centrality: structural bottlenecks. betweenness = graph.betweenness_centrality() -node = graph.Node.ref("n") -score = Float.ref("s") -btw_df = ( - where(betweenness(node, score)) - .select( - node.id.alias("sku_id"), - node.name.alias("sku_name"), - node.type.alias("type"), - node.category.alias("category"), - score.alias("betweenness"), - ) - .to_df() - .sort_values("betweenness", ascending=False) - .reset_index(drop=True) -) - -print("\n=== Betweenness Centrality (bottleneck components) ===") -print(btw_df.to_string(index=False)) - -bottlenecks = btw_df[btw_df["betweenness"] > 0] -if len(bottlenecks) > 0: - top = bottlenecks.iloc[0] - print(f"\nTop bottleneck: {top['sku_name']} (betweenness={top['betweenness']:.4f})") - print(" Sits on the most dependency paths -- disruption here affects the most product lines.") - # -------------------------------------------------- # Assembly path enumeration # PREVIEW capability; requires relationalai>=1.15. @@ -188,8 +108,6 @@ # SKU feeds the output SKU), enumerate all such paths, then keep only the # maximal (longest, non-extendable) chains so that sub-paths are suppressed. -print("\n=== Assembly Path Enumeration (PREVIEW) ===") - # Binary SKU->SKU edge: input_sku "feeds into" output_sku. This is the # build-direction (bottom-up) reverse of the depends-on graph edge above. SKU.feeds = model.Relationship(f"{SKU} feeds into {SKU}", short_name="feeds") @@ -207,36 +125,8 @@ # allow extra headroom so deeper BOMs enumerate fully. MAX_ASSEMBLY_HOPS = 4 p = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS)).all().paths() -assembly_df = ( - model.where(p) - .select( - p.alias("path"), - p.nodes["index"].alias("step"), - SKU(p.nodes).id.alias("sku_id"), - SKU(p.nodes).name.alias("sku_name"), - ) - .to_df() -) -# Reassemble each path in pandas: group on the path id, order by step index. -assembly_df["step"] = assembly_df["step"].astype(int) -paths = [] -for path_id, grp in assembly_df.groupby("path"): - ordered = grp.sort_values("step") - ids = ordered["sku_id"].tolist() - names = ordered["sku_name"].tolist() - paths.append({"ids": ids, "names": names, "length": len(ids) - 1}) - -print(f"\n Enumerated {len(paths)} assembly path(s) (<= {MAX_ASSEMBLY_HOPS} hops).") - -# (a) All assembly paths, longest first. -print("\n All assembly chains (feeds-into order):") -for path in sorted(paths, key=lambda x: -x["length"]): - print(f" [{path['length']} hop] " + " -> ".join(path["names"])) - -# (b) Maximal paths: keep only the longest, non-extendable chains -- a chain -# that is NOT a contiguous sub-sequence (prefix or suffix) of any longer chain. -# Suppressing these sub-paths leaves just the end-to-end assembly routes. + def _is_contiguous_subsequence(short, long): """True if `short` appears as a contiguous run inside `long`.""" if len(short) >= len(long): @@ -244,48 +134,162 @@ def _is_contiguous_subsequence(short, long): return any(long[i:i + len(short)] == short for i in range(len(long) - len(short) + 1)) -maximal = [ - path - for path in paths - if not any( - other is not path and _is_contiguous_subsequence(path["ids"], other["ids"]) - for other in paths - ) -] - -print(f"\n Maximal assembly chains ({len(maximal)} of {len(paths)}, sub-paths suppressed):") -for path in sorted(maximal, key=lambda x: -x["length"]): - print(f" [{path['length']} hop] " + " -> ".join(path["names"])) - -# Persist each finished good's longest assembly depth back onto the SKU ontology. -# The terminal SKU of a maximal chain is the thing it builds; the deepest chain -# ending there is that finished good's assembly depth. -SKU.assembly_depth = model.Property(f"{SKU} has assembly depth {Integer:assembly_depth}") -depth_by_sku = {} -for path in maximal: - terminal = path["ids"][-1] - depth_by_sku[terminal] = max(depth_by_sku.get(terminal, 0), path["length"]) - -if depth_by_sku: - depth_rows = pd.DataFrame( - [{"sku_id": sku_id, "assembly_depth": depth} for sku_id, depth in depth_by_sku.items()] - ) - depth_data = model.data(depth_rows) - model.define(SKU.assembly_depth(depth_data.assembly_depth)).where( - SKU.id == depth_data.sku_id +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh. + model.deploy() + + print("=== BOM Dependency Graph ===") + num_nodes.inspect() + num_edges.inspect() + + src, dst = graph.Node.ref("src"), graph.Node.ref("dst") + all_deps_df = ( + where(reachable(src, dst)) + .select( + src.id.alias("product_id"), + src.name.alias("product_name"), + src.type.alias("product_type"), + dst.id.alias("dep_id"), + dst.name.alias("dep_name"), + dst.type.alias("dep_type"), + ) + .to_df() ) - print("\n Assembly depth persisted onto SKU (longest chain terminating at each):") - depth_df = ( - where(SKU.assembly_depth > 0) + print(f"\nTotal reachable pairs: {len(all_deps_df)}") + + # -------------------------------------------------- + # Dependencies of each finished good + # -------------------------------------------------- + + finished = all_deps_df[all_deps_df["product_type"] == "FINISHED_GOOD"] + for product_id in sorted(finished["product_id"].unique()): + deps = finished[(finished["product_id"] == product_id) & (finished["dep_id"] != product_id)] + if len(deps) > 0: + product_name = deps["product_name"].iloc[0] + print(f"\n--- Dependencies of '{product_name}' ({product_id}) ---") + print(f" {len(deps)} transitive dependencies:") + for dep_type in ["COMPONENT", "RAW_MATERIAL"]: + type_deps = deps[deps["dep_type"] == dep_type] + if len(type_deps) > 0: + print(f"\n {dep_type}s ({len(type_deps)}):") + for _, row in type_deps.sort_values("dep_name").iterrows(): + print(f" - {row['dep_name']} ({row['dep_id']})") + + # -------------------------------------------------- + # Most depended-on components + # -------------------------------------------------- + + non_self = all_deps_df[all_deps_df["product_id"] != all_deps_df["dep_id"]] + if len(non_self) > 0: + dep_counts = ( + non_self.groupby(["dep_id", "dep_name", "dep_type"]) + .size() + .reset_index(name="depended_on_by") + .sort_values("depended_on_by", ascending=False) + ) + print("\n=== Most Depended-On SKUs ===") + print(dep_counts.to_string(index=False)) + + # -------------------------------------------------- + # Betweenness centrality: structural bottlenecks + # -------------------------------------------------- + + node = graph.Node.ref("n") + score = Float.ref("s") + btw_df = ( + where(betweenness(node, score)) .select( - SKU.id.alias("sku_id"), - SKU.name.alias("sku_name"), - SKU.type.alias("type"), - SKU.assembly_depth.alias("assembly_depth"), + node.id.alias("sku_id"), + node.name.alias("sku_name"), + node.type.alias("type"), + node.category.alias("category"), + score.alias("betweenness"), ) .to_df() - .sort_values("assembly_depth", ascending=False) + .sort_values("betweenness", ascending=False) .reset_index(drop=True) ) - print(depth_df.to_string(index=False)) + + print("\n=== Betweenness Centrality (bottleneck components) ===") + print(btw_df.to_string(index=False)) + + bottlenecks = btw_df[btw_df["betweenness"] > 0] + if len(bottlenecks) > 0: + top = bottlenecks.iloc[0] + print(f"\nTop bottleneck: {top['sku_name']} (betweenness={top['betweenness']:.4f})") + print(" Sits on the most dependency paths -- disruption here affects the most product lines.") + + # -------------------------------------------------- + # Assembly path enumeration + # -------------------------------------------------- + + print("\n=== Assembly Path Enumeration (PREVIEW) ===") + + assembly_df = ( + model.where(p) + .select( + p.alias("path"), + p.nodes["index"].alias("step"), + SKU(p.nodes).id.alias("sku_id"), + SKU(p.nodes).name.alias("sku_name"), + ) + .to_df() + ) + + # Reassemble each path in pandas: group on the path id, order by step index. + assembly_df["step"] = assembly_df["step"].astype(int) + paths = [] + for path_id, grp in assembly_df.groupby("path"): + ordered = grp.sort_values("step") + ids = ordered["sku_id"].tolist() + names = ordered["sku_name"].tolist() + paths.append({"ids": ids, "names": names, "length": len(ids) - 1}) + + print(f"\n Enumerated {len(paths)} assembly path(s) (<= {MAX_ASSEMBLY_HOPS} hops).") + + # (a) All assembly paths, longest first. + print("\n All assembly chains (feeds-into order):") + for path in sorted(paths, key=lambda x: -x["length"]): + print(f" [{path['length']} hop] " + " -> ".join(path["names"])) + + # (b) Maximal paths: keep only the longest, non-extendable chains -- a chain + # that is NOT a contiguous sub-sequence (prefix or suffix) of any longer chain. + # Suppressing these sub-paths leaves just the end-to-end assembly routes. + maximal = [ + path + for path in paths + if not any(other is not path and _is_contiguous_subsequence(path["ids"], other["ids"]) for other in paths) + ] + + print(f"\n Maximal assembly chains ({len(maximal)} of {len(paths)}, sub-paths suppressed):") + for path in sorted(maximal, key=lambda x: -x["length"]): + print(f" [{path['length']} hop] " + " -> ".join(path["names"])) + + # Each finished good's assembly depth: the terminal SKU of a maximal chain is the + # thing it builds, and the deepest chain ending there is its assembly depth. + depth_by_sku = {} + for path in maximal: + terminal = path["ids"][-1] + depth_by_sku[terminal] = max(depth_by_sku.get(terminal, 0), path["length"]) + + if depth_by_sku: + sku_info = all_deps_df.drop_duplicates("product_id").set_index("product_id") + depth_df = pd.DataFrame( + [ + { + "sku_id": sku_id, + "sku_name": sku_info.at[sku_id, "product_name"], + "type": sku_info.at[sku_id, "product_type"], + "assembly_depth": depth, + } + for sku_id, depth in depth_by_sku.items() + if depth > 0 + ] + ).sort_values(["assembly_depth", "sku_id"], ascending=[False, True], ignore_index=True) + print("\n Assembly depth (longest chain terminating at each finished good):") + print(depth_df.to_string(index=False)) diff --git a/bom-reachability/runbook.md b/bom-reachability/runbook.md index ba164800..31f9edd9 100644 --- a/bom-reachability/runbook.md +++ b/bom-reachability/runbook.md @@ -10,10 +10,10 @@ BOM graph (9 SKUs across 3 tiers, 10 dependency edges from 14 site-specific BOM • transitive dependencies per finished good (reachable) -> 7 each • structural bottleneck component (betweenness centrality) -> Mobile Processor A15 = 4.0 • every raw-material to finished-good assembly chain (paths) -> 18 paths - • maximal chains + persist SKU.assembly_depth -> 8 maximal, depth 2 + • maximal chains + assembly depth per finished good -> 8 maximal, depth 2 ``` -Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` skill (named at the start of each prompt). They run in order in a single session — the assembly-path and depth steps read the `BillOfMaterials` relationships the build step established, and the final step persists `SKU.assembly_depth` back onto the ontology. +Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` skill (named at the start of each prompt). They run in order in a single session — the assembly-path and depth steps read the `BillOfMaterials` relationships the build step established. --- @@ -49,6 +49,6 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## 6. Maximal chains and assembly depth -**Prompt:** /rai-graph-analysis The full path set repeats every sub-chain. Reduce it to just the maximal end-to-end chains — the longest routes not contained inside a longer one — and persist each finished good's assembly depth as `SKU.assembly_depth` on the ontology (the length of the deepest chain that builds it). +**Prompt:** /rai-graph-analysis The full path set repeats every sub-chain. Reduce it to just the maximal end-to-end chains — the longest routes not contained inside a longer one — and report each finished good's assembly depth (the length of the deepest chain that builds it). -**Response:** 8 of the 18 paths are maximal; the other 10 are shorter sub-chains contained inside them. All 8 maximal chains are 2-hop raw-material → component → finished-good routes (e.g. Silicon Wafer 300mm → Mobile Processor A15 → ProTab T1). Both finished goods have an assembly depth of 2, persisted back onto the ontology as `SKU.assembly_depth` so a later query can rank products by build complexity without re-enumerating paths. +**Response:** 8 of the 18 paths are maximal; the other 10 are shorter sub-chains contained inside them. All 8 maximal chains are 2-hop raw-material → component → finished-good routes (e.g. Silicon Wafer 300mm → Mobile Processor A15 → ProTab T1). Both finished goods have an assembly depth of 2. From 852661d64c0dd6b0d085b2a6834fe6ed1034f4a8 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 15:48:53 -0800 Subject: [PATCH 004/162] diet, factory_production: note that teardown keeps the schemas --- diet/README.md | 2 +- factory_production/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/diet/README.md b/diet/README.md index f043514a..322e4b3f 100644 --- a/diet/README.md +++ b/diet/README.md @@ -123,7 +123,7 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ```sql CALL .DIET_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory. + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output — a few lines confirm a successful run: diff --git a/factory_production/README.md b/factory_production/README.md index cb3d2af6..0d2a80f9 100644 --- a/factory_production/README.md +++ b/factory_production/README.md @@ -130,7 +130,7 @@ Manufacturing operations must decide how much of each product to make at each fa ```sql CALL .FACTORY_PRODUCTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory. + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (model and solver display trimmed): ```text From b0fde5e02f5579584f16ccf81074f19fb0d92d43 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 15:59:39 -0800 Subject: [PATCH 005/162] bom-reachability: persist assembly depth with a deployed rule Define SKU.assembly_depth at module level as the longest assembly chain ending at each finished good, so it deploys with the model and stays queryable after the run. Restores the README and runbook text that describes the persisted property. --- bom-reachability/README.md | 10 +++--- bom-reachability/bom_reachability.py | 51 ++++++++++++++-------------- bom-reachability/runbook.md | 8 ++--- 3 files changed, 35 insertions(+), 34 deletions(-) diff --git a/bom-reachability/README.md b/bom-reachability/README.md index 8eedb007..d52b4ba2 100644 --- a/bom-reachability/README.md +++ b/bom-reachability/README.md @@ -35,17 +35,17 @@ This template turns a BOM into a dependency graph so you can trace those transit - A full transitive dependency tree per finished good, broken down by type (component vs raw material), from all-pairs `reachable(full=True)`. - A most-depended-on ranking of SKUs and a betweenness-centrality ranking that flags the structural bottleneck components. - An enumeration of the end-to-end assembly chains that build each finished good, reduced to the maximal (non-extendable) chains via path enumeration. -- Each finished good's assembly depth: the length of the longest chain that builds it. +- An `assembly_depth` property persisted back onto each SKU, so the longest chain terminating at each finished good is queryable as ontology after the run. Built on RelationalAI's graph reasoner: **reachability** for transitive dependencies, **betweenness centrality** for bottlenecks, and **path enumeration** for assembly chains. ## What's included -- **Model**: two concepts (`SKU` and `BillOfMaterials`) wired into a directed dependency graph, plus a derived `feeds` self-relationship on `SKU`. +- **Model**: two concepts (`SKU` and `BillOfMaterials`) wired into a directed dependency graph, plus a derived `feeds` self-relationship on `SKU` and a persisted `assembly_depth` property. - **Runner**: `bom_reachability.py` — a single self-contained Python script that deploys the model to your Snowflake account and prints the reachability, betweenness centrality, and assembly-path report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/skus.csv` (9 SKUs across 3 tiers) and `data/bill_of_materials.csv` (14 BOM entries with site-specific assembly). See *Sample data* below. -- **Outputs**: printed transitive-dependency lists per finished good, a most-depended-on and betweenness ranking, the maximal assembly chains, and each finished good's assembly depth. +- **Outputs**: printed transitive-dependency lists per finished good, a most-depended-on and betweenness ranking, the maximal assembly chains, and an `assembly_depth` property written back onto the SKU ontology. ## Prerequisites @@ -154,7 +154,7 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen - **Primary identifiers**: `SKU` by `id`; `BillOfMaterials` by `id`. - **Important invariants**: each `BillOfMaterials` row points an output SKU at one input SKU ("depends on"); the BOM is acyclic (raw materials feed components feed finished goods, never back). -The model also derives a `feeds` self-relationship on `SKU` (input SKU feeds output SKU), built from the `BillOfMaterials` intermediary and used to enumerate assembly paths. +The model also derives a `feeds` self-relationship on `SKU` (input SKU feeds output SKU), built from the `BillOfMaterials` intermediary and used to enumerate assembly paths, plus a persisted `assembly_depth` property (the longest assembly chain terminating at each SKU). For the full concept and property definitions, see `bom_reachability.py`; `runbook.md` builds them step by step with the RAI skills. @@ -184,7 +184,7 @@ Betweenness centrality ranks components by how many shortest dependency paths pa ### 5. Enumerate assembly paths (PREVIEW, requires `relationalai>=1.15`) -Where reachability returns dependency *pairs*, path enumeration returns the actual *build sequences*. It walks the derived `feeds` edge to enumerate every assembly path; because the BOM is acyclic, this yields exactly the simple paths with no cycle risk. A maximal-paths view keeps only the longest non-extendable chains, and the deepest chain ending at each finished good gives its assembly depth. +Where reachability returns dependency *pairs*, path enumeration returns the actual *build sequences*. It walks the derived `feeds` edge to enumerate every assembly path; because the BOM is acyclic, this yields exactly the simple paths with no cycle risk. A maximal-paths view keeps only the longest non-extendable chains, and the longest assembly depth is persisted as `SKU.assembly_depth`. See `bom_reachability.py` for the implementation and `runbook.md` for the skill-driven reproduction. diff --git a/bom-reachability/bom_reachability.py b/bom-reachability/bom_reachability.py index 8e5d407d..5e59159a 100644 --- a/bom-reachability/bom_reachability.py +++ b/bom-reachability/bom_reachability.py @@ -15,7 +15,8 @@ SKU->SKU "feeds into" edge from the BillOfMaterials intermediary (input feeds output), enumerates every assembly path with model.path(...).all().paths(), filters to the maximal (longest non-extendable) chains so sub-paths are - suppressed, and reports each finished good's longest assembly depth. + suppressed, and persists each finished good's longest assembly depth back onto + the SKU ontology. Run: `python bom_reachability.py` deploys the model to the schema in raiconfig.yaml, @@ -31,8 +32,9 @@ from pathlib import Path import pandas as pd -from relationalai.semantics import Float, Model, String, where +from relationalai.semantics import Float, Integer, Model, String, where from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.std import aggregates as aggs # -------------------------------------------------- # Configure inputs @@ -126,6 +128,15 @@ MAX_ASSEMBLY_HOPS = 4 p = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS)).all().paths() +# Persist each finished good's longest assembly depth onto the SKU ontology: the +# longest assembly chain that ends at it (max path length per final endpoint). +built = SKU.ref() +chain = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS), built).all().paths() +SKU.assembly_depth = model.Property(f"{SKU} has assembly depth {Integer:assembly_depth}") +model.where(chain, built.type == "FINISHED_GOOD").define( + built.assembly_depth(aggs.max(chain.length).per(built)) +) + def _is_contiguous_subsequence(short, long): """True if `short` appears as a contiguous run inside `long`.""" @@ -270,26 +281,16 @@ def _is_contiguous_subsequence(short, long): for path in sorted(maximal, key=lambda x: -x["length"]): print(f" [{path['length']} hop] " + " -> ".join(path["names"])) - # Each finished good's assembly depth: the terminal SKU of a maximal chain is the - # thing it builds, and the deepest chain ending there is its assembly depth. - depth_by_sku = {} - for path in maximal: - terminal = path["ids"][-1] - depth_by_sku[terminal] = max(depth_by_sku.get(terminal, 0), path["length"]) - - if depth_by_sku: - sku_info = all_deps_df.drop_duplicates("product_id").set_index("product_id") - depth_df = pd.DataFrame( - [ - { - "sku_id": sku_id, - "sku_name": sku_info.at[sku_id, "product_name"], - "type": sku_info.at[sku_id, "product_type"], - "assembly_depth": depth, - } - for sku_id, depth in depth_by_sku.items() - if depth > 0 - ] - ).sort_values(["assembly_depth", "sku_id"], ascending=[False, True], ignore_index=True) - print("\n Assembly depth (longest chain terminating at each finished good):") - print(depth_df.to_string(index=False)) + print("\n Assembly depth persisted onto SKU (longest chain terminating at each):") + depth_df = ( + where(SKU.assembly_depth > 0) + .select( + SKU.id.alias("sku_id"), + SKU.name.alias("sku_name"), + SKU.type.alias("type"), + SKU.assembly_depth.alias("assembly_depth"), + ) + .to_df() + .sort_values(["assembly_depth", "sku_id"], ascending=[False, True], ignore_index=True) + ) + print(depth_df.to_string(index=False)) diff --git a/bom-reachability/runbook.md b/bom-reachability/runbook.md index 31f9edd9..ba164800 100644 --- a/bom-reachability/runbook.md +++ b/bom-reachability/runbook.md @@ -10,10 +10,10 @@ BOM graph (9 SKUs across 3 tiers, 10 dependency edges from 14 site-specific BOM • transitive dependencies per finished good (reachable) -> 7 each • structural bottleneck component (betweenness centrality) -> Mobile Processor A15 = 4.0 • every raw-material to finished-good assembly chain (paths) -> 18 paths - • maximal chains + assembly depth per finished good -> 8 maximal, depth 2 + • maximal chains + persist SKU.assembly_depth -> 8 maximal, depth 2 ``` -Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` skill (named at the start of each prompt). They run in order in a single session — the assembly-path and depth steps read the `BillOfMaterials` relationships the build step established. +Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` skill (named at the start of each prompt). They run in order in a single session — the assembly-path and depth steps read the `BillOfMaterials` relationships the build step established, and the final step persists `SKU.assembly_depth` back onto the ontology. --- @@ -49,6 +49,6 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## 6. Maximal chains and assembly depth -**Prompt:** /rai-graph-analysis The full path set repeats every sub-chain. Reduce it to just the maximal end-to-end chains — the longest routes not contained inside a longer one — and report each finished good's assembly depth (the length of the deepest chain that builds it). +**Prompt:** /rai-graph-analysis The full path set repeats every sub-chain. Reduce it to just the maximal end-to-end chains — the longest routes not contained inside a longer one — and persist each finished good's assembly depth as `SKU.assembly_depth` on the ontology (the length of the deepest chain that builds it). -**Response:** 8 of the 18 paths are maximal; the other 10 are shorter sub-chains contained inside them. All 8 maximal chains are 2-hop raw-material → component → finished-good routes (e.g. Silicon Wafer 300mm → Mobile Processor A15 → ProTab T1). Both finished goods have an assembly depth of 2. +**Response:** 8 of the 18 paths are maximal; the other 10 are shorter sub-chains contained inside them. All 8 maximal chains are 2-hop raw-material → component → finished-good routes (e.g. Silicon Wafer 300mm → Mobile Processor A15 → ProTab T1). Both finished goods have an assembly depth of 2, persisted back onto the ontology as `SKU.assembly_depth` so a later query can rank products by build complexity without re-enumerating paths. From 9b1ddd59be0e4e1d87797c78b903fe84f25fbcc2 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:01:04 -0800 Subject: [PATCH 006/162] factory_production: check the returned factory and product keys The shadow-price and reduced-cost checks compare the returned key sets with the expected factories and products, so a missing or duplicated row fails instead of passing a row count. --- factory_production/factory_production.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/factory_production/factory_production.py b/factory_production/factory_production.py index c573fc1b..4fca9a2d 100644 --- a/factory_production/factory_production.py +++ b/factory_production/factory_production.py @@ -234,7 +234,7 @@ .to_df() .sort_values("factory", ignore_index=True) ) - assert len(cap_df) == len(EXPECTED_FACTORIES), "missing capacity shadow prices" + assert sorted(cap_df["factory"]) == sorted(EXPECTED_FACTORIES), "expected one shadow price per factory" print("\nFactory capacity shadow prices (d profit / d hour):") print(cap_df.to_string(index=False)) # Maximize + a <= capacity => shadow_price >= 0: an extra hour can only help. The @@ -259,7 +259,7 @@ .to_df() .sort_values(["factory", "product"], ignore_index=True) ) - assert len(rc_df) == len(EXPECTED_PRODUCTS), "missing reduced costs" + assert sorted(rc_df["product"]) == sorted(EXPECTED_PRODUCTS), "expected one reduced cost per product" print("\nProduct reduced costs and basis status:") print(rc_df.to_string(index=False)) # A product pinned at its demand cap is NONBASIC_AT_UPPER with reduced_cost >= 0 (the From 75b3c91f51fc69ac04f4835281957bae214e7542 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:07:37 -0800 Subject: [PATCH 007/162] ad_spend_allocation: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and objective through the model's own relationships and the allocation through Allocation.x_spend. The README covers the deployment config, the internal schema grants made before the first run, and re-running. --- ad_spend_allocation/README.md | 58 ++++++++++++++++++---- ad_spend_allocation/ad_spend_allocation.py | 55 ++++++++++++++------ 2 files changed, 89 insertions(+), 24 deletions(-) diff --git a/ad_spend_allocation/README.md b/ad_spend_allocation/README.md index f91eb00a..0bd998aa 100644 --- a/ad_spend_allocation/README.md +++ b/ad_spend_allocation/README.md @@ -40,7 +40,7 @@ This template also answers the follow-up question every planner asks: if we had ## What's included - **Model**: a single ontology with `Channel`, `Campaign`, `Effectiveness`, `Scenario`, and an `Allocation` decision concept — plus the prescriptive formulation (decision variables, constraints, objective) that runs on it. -- **Runner**: `ad_spend_allocation.py` — one Python script that loads the CSVs, builds the model, solves all scenarios at once, and prints the allocation table. +- **Runner**: `ad_spend_allocation.py` — one Python script that loads the CSVs, builds the model, deploys it to your Snowflake account, where each refresh solves all scenarios at once, and prints the allocation table from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: 5 channels with spend bounds, 3 campaigns with budgets, and the 15 channel-campaign conversion rates. See *Sample data* below. - **Outputs**: solver termination status, objective value, and a per-scenario table of non-trivial spend allocations printed to stdout; the spend and funding decisions are also written back to the ontology as queryable properties. @@ -50,7 +50,8 @@ This template also answers the follow-up question every planner asks: if we had ### Access - A Snowflake account with the RelationalAI Native App installed. -- A Snowflake user with permissions to access the RelationalAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools @@ -84,17 +85,56 @@ This template also answers the follow-up question every planner asks: if we had python -m pip install . ``` -4. Configure your RelationalAI connection: - - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: ad_spend_allocation_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .AD_SPEND_ALLOCATION # published views and tables + identity: + internal_schema: .AD_SPEND_ALLOCATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python ad_spend_allocation.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .AD_SPEND_ALLOCATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; + ``` + The deploy creates the public schema `.AD_SPEND_ALLOCATION` if it doesn't exist. -5. Run the template end-to-end: - +5. Run: ```bash python ad_spend_allocation.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .AD_SPEND_ALLOCATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output. A per-scenario allocation table confirms a successful run (trimmed to the first budget level here; the full run also prints the $45K and $55K scenarios): @@ -187,7 +227,7 @@ Focus on the first changes most users will make. - **Budget levels** — the three scenarios (`budget_35k`, `budget_45k`, `budget_55k`) are defined in the `scenario_data` block near the top of the script. Add, remove, or change levels there to sweep a different range. - **Spend bounds and budgets** — per-channel `min_spend` / `max_spend` live in `channels.csv`; per-campaign `budget` lives in `campaigns.csv`. These are usually the binding limits, so they are the first knobs to adjust when the total-budget cap stops mattering. -- **Solver settings** — the solve uses HiGHS with a 60-second time limit (`problem.solve("highs", time_limit_sec=60)`); adjust the limit for larger instances. +- **Solver settings** — the solve uses HiGHS with a 60-second time limit (`problem.configure("highs", time_limit_sec=60)`); adjust the limit for larger instances. ### Extend the model diff --git a/ad_spend_allocation/ad_spend_allocation.py b/ad_spend_allocation/ad_spend_allocation.py index d22c0cd8..52089b2d 100644 --- a/ad_spend_allocation/ad_spend_allocation.py +++ b/ad_spend_allocation/ad_spend_allocation.py @@ -21,7 +21,9 @@ - One solve handles all budget levels; results extracted via model.select(). Run: - `python ad_spend_allocation.py` + `python ad_spend_allocation.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints the solver termination status, objective value, and a table of @@ -171,21 +173,44 @@ # Solve (single solve for all scenarios) # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -problem.solve_info().display() +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"ad spend solve status is {String:status}") +TotalConversions = model.Relationship(f"ad spend total conversions is {Float:conversions}") +model.define(SolveStatus(problem.termination_status()), TotalConversions(problem.objective_value())) # -------------------------------------------------- -# Extract results per scenario +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nSpend allocation per scenario:") -model.select( - Scenario.name.alias("scenario"), - Allocation.effectiveness.channel.name.alias("channel"), - Allocation.effectiveness.campaign.name.alias("campaign"), - x_spend.alias("spend"), -).where( - Allocation.x_spend(Scenario, x_spend), x_spend > 0.001 -).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + conversions = model.select(TotalConversions).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal spend allocation: termination status {status}") + print(f"• objective: {round(conversions[0], 9)}\n") + + # -------------------------------------------------- + # Extract results per scenario + # -------------------------------------------------- + + print("\nSpend allocation per scenario:") + model.select( + Scenario.name.alias("scenario"), + Allocation.effectiveness.channel.name.alias("channel"), + Allocation.effectiveness.campaign.name.alias("campaign"), + x_spend.alias("spend"), + ).where(Allocation.x_spend(Scenario, x_spend), x_spend > 0.001).inspect() From bace2e2f9740ba99b7d6f79f8f286ccbe5c1e7c5 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:10:45 -0800 Subject: [PATCH 008/162] campaign_roi: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and total conversions through the model's own relationships. The README covers the deployment config, the internal schema grants made before the first run, and re-running. --- campaign_roi/README.md | 56 +++++++++++++++--- campaign_roi/campaign_roi.py | 107 ++++++++++++++++++++++------------- 2 files changed, 115 insertions(+), 48 deletions(-) diff --git a/campaign_roi/README.md b/campaign_roi/README.md index 8ae4bbad..d4a2762f 100644 --- a/campaign_roi/README.md +++ b/campaign_roi/README.md @@ -37,7 +37,7 @@ Every marketing team eventually faces the same question: the portfolio of campai ## What's included - **Model**: a continuous LP over a single `Campaign` concept, with per-campaign floor and cap bounds, a total-budget constraint, a paused-region share cap, and a conversion-maximizing objective. -- **Runner**: `campaign_roi.py` — a single Python script that runs end-to-end against a Snowflake-connected RAI account. +- **Runner**: `campaign_roi.py` — a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/campaigns.csv` — 12 campaigns across 5 regions with current budgets and empirical conversion rates. - **Outputs**: solver status, optimized-versus-current conversions and lift, a per-campaign reallocation table, and a regional-spend summary against the paused-region cap. @@ -47,7 +47,8 @@ Every marketing team eventually faces the same question: the portfolio of campai ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools @@ -81,17 +82,56 @@ Every marketing team eventually faces the same question: the portfolio of campai python -m pip install . ``` -4. Configure the Snowflake connection and RAI profile: - - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: campaign_roi_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .CAMPAIGN_ROI # published views and tables + identity: + internal_schema: .CAMPAIGN_ROI_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python campaign_roi.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .CAMPAIGN_ROI_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; + ``` + The deploy creates the public schema `.CAMPAIGN_ROI` if it doesn't exist. -5. Run the template: - +5. Run: ```bash python campaign_roi.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .CAMPAIGN_ROI_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (a few lines confirm a successful run): diff --git a/campaign_roi/campaign_roi.py b/campaign_roi/campaign_roi.py index 307b694d..e66d6ecd 100644 --- a/campaign_roi/campaign_roi.py +++ b/campaign_roi/campaign_roi.py @@ -13,7 +13,9 @@ - Maximize total expected conversions = sum(budget * conversion_rate). Run: - `python campaign_roi.py` + `python campaign_roi.py` deploys the model to the schema in raiconfig.yaml, waits for + its first refresh (which runs the solve), and prints the report from the deployed + model. Output: Prints solver termination status, total conversions vs current, @@ -98,42 +100,67 @@ # Solve and check solution # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -si = problem.solve_info() -si.display() - -print(f"\nStatus: {si.termination_status}") -print(f"Total expected conversions (optimized): {si.objective_value:,.1f}") - -current_total = (campaign_csv["current_budget"] * campaign_csv["conversion_rate"]).sum() -print(f"Total expected conversions (current): {current_total:,.1f}") -print(f"Lift: {(si.objective_value - current_total) / current_total * 100:+.1f}%") - -print("\nBudget reallocation per campaign (sorted by largest absolute change):") -results_df = model.select( - Campaign.name.alias("campaign"), - Campaign.region.alias("region"), - Campaign.current_budget.alias("current"), - Campaign.x_budget.alias("optimized"), - Campaign.conversion_rate.alias("rate"), -).to_df() -results_df["delta"] = results_df["optimized"] - results_df["current"] -results_df["multiple"] = results_df["optimized"] / results_df["current"] -results_df = results_df.sort_values("delta", key=abs, ascending=False).reset_index(drop=True) -print(results_df.to_string(index=False)) - -print("\nRegional spend (optimized):") -regional = ( - results_df.groupby("region")[["current", "optimized"]] - .sum() - .reset_index() - .sort_values("optimized", ascending=False) -) -regional["share_of_total"] = regional["optimized"] / TOTAL_BUDGET -print(regional.to_string(index=False)) -print( - f"\nPaused region '{PAUSED_REGION}' cap: " - f"{PAUSED_CAP_FRACTION * TOTAL_BUDGET:.1f} ({PAUSED_CAP_FRACTION:.0%} of ${TOTAL_BUDGET:,.0f}K total)" -) +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"campaign roi solve status is {String:status}") +TotalConversions = model.Relationship(f"campaign roi total conversions is {Float:conversions}") +model.define(SolveStatus(problem.termination_status()), TotalConversions(problem.objective_value())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + conversions = model.select(TotalConversions).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal budget reallocation: termination status {status}") + objective_value = conversions[0] + print(f"• objective: {round(objective_value, 9)}\n") + + print(f"\nStatus: {status[0]}") + print(f"Total expected conversions (optimized): {objective_value:,.1f}") + + current_total = (campaign_csv["current_budget"] * campaign_csv["conversion_rate"]).sum() + print(f"Total expected conversions (current): {current_total:,.1f}") + print(f"Lift: {(objective_value - current_total) / current_total * 100:+.1f}%") + + print("\nBudget reallocation per campaign (sorted by largest absolute change):") + results_df = model.select( + Campaign.name.alias("campaign"), + Campaign.region.alias("region"), + Campaign.current_budget.alias("current"), + Campaign.x_budget.alias("optimized"), + Campaign.conversion_rate.alias("rate"), + ).to_df() + results_df["delta"] = results_df["optimized"] - results_df["current"] + results_df["multiple"] = results_df["optimized"] / results_df["current"] + results_df = results_df.sort_values("delta", key=abs, ascending=False).reset_index(drop=True) + print(results_df.to_string(index=False)) + + print("\nRegional spend (optimized):") + regional = ( + results_df.groupby("region")[["current", "optimized"]] + .sum() + .reset_index() + .sort_values("optimized", ascending=False) + ) + regional["share_of_total"] = regional["optimized"] / TOTAL_BUDGET + print(regional.to_string(index=False)) + print( + f"\nPaused region '{PAUSED_REGION}' cap: " + f"{PAUSED_CAP_FRACTION * TOTAL_BUDGET:.1f} ({PAUSED_CAP_FRACTION:.0%} of ${TOTAL_BUDGET:,.0f}K total)" + ) From 5471eb6269c86d0d09d52a823c56cfddeec3307b Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:14:38 -0800 Subject: [PATCH 009/162] network_flow_planning: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and total cost through the model's own relationships. The README covers the deployment config, the internal schema grants made before the first run, and re-running. --- network_flow_planning/README.md | 58 +++++++++++--- .../network_flow_planning.py | 75 +++++++++++++------ 2 files changed, 100 insertions(+), 33 deletions(-) diff --git a/network_flow_planning/README.md b/network_flow_planning/README.md index 7dd71367..f5eb3692 100644 --- a/network_flow_planning/README.md +++ b/network_flow_planning/README.md @@ -49,7 +49,7 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. ## What's included -- `network_flow_planning.py` — main script (single end-to-end run) +- `network_flow_planning.py` — main script (deploys the model, then prints the report from the deployed model) - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/sites.csv` — 12 sites: 3 warehouses, 3 transit hubs, 3 fulfillment centers, 3 customers - `data/lanes.csv` — 17 directed lanes connecting the tiers, each with a cost-per-unit and capacity @@ -61,7 +61,8 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools @@ -95,17 +96,56 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. python -m pip install . ``` -4. Configure Snowflake connection and RAI profile: - - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: network_flow_planning_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .NETWORK_FLOW_PLANNING # published views and tables + identity: + internal_schema: .NETWORK_FLOW_PLANNING_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python network_flow_planning.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .NETWORK_FLOW_PLANNING_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; + ``` + The deploy creates the public schema `.NETWORK_FLOW_PLANNING` if it doesn't exist. -5. Run the template: - +5. Run: ```bash python network_flow_planning.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .NETWORK_FLOW_PLANNING_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (truncated): @@ -212,7 +252,7 @@ Replace the three CSVs in `data/` with your own: - For a live network, replace the three `data/` CSVs with `model.data(snowflake_table)` calls so the model reads sites, lanes, and demand directly from your warehouse tables. - The bundled data is 12 sites and 17 lanes; the MILP scales to whatever fits the prescriptive engine's solve budget. If solves grow slow, tighten lane capacities to prune the flow space or size up the engine. -- The template solves with HiGHS by default (`problem.solve("highs")`); swap to `"gurobi"` for larger integer programs if a Gurobi-enabled engine is available. +- The template solves with HiGHS by default (`problem.configure("highs", time_limit_sec=60)`); swap to `"gurobi"` for larger integer programs if a Gurobi-enabled engine is available. - Pin `relationalai` (see Prerequisites) so runs stay reproducible across environments. ## Troubleshooting diff --git a/network_flow_planning/network_flow_planning.py b/network_flow_planning/network_flow_planning.py index cc22ef41..f8ec72c2 100644 --- a/network_flow_planning/network_flow_planning.py +++ b/network_flow_planning/network_flow_planning.py @@ -16,7 +16,9 @@ - Minimize total cost (transport + fixed-cost FC opening). Run: - `python network_flow_planning.py` + `python network_flow_planning.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints solver termination status, total cost (transport + fixed @@ -185,26 +187,51 @@ # Solve and check solution # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -si = problem.solve_info() -si.display() - -print(f"\nStatus: {si.termination_status}") -print(f"Total cost: ${si.objective_value:,.2f}") - -print("\nOpened fulfillment centers:") -model.select( - Site.name.alias("fulfillment_center"), - Site.capacity.alias("capacity"), - Site.fixed_cost.alias("fixed_cost"), -).where(Site.x_open > 0.5).inspect() - -print("\nActive flows:") -model.select( - Lane.source.name.alias("from_site"), - Lane.dest.name.alias("to_site"), - Lane.x_flow.alias("flow"), - Lane.cost_per_unit.alias("unit_cost"), -).where(Lane.x_flow > 0.001).inspect() +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"network flow solve status is {String:status}") +TotalCost = model.Relationship(f"network flow total cost is {Float:cost}") +model.define(SolveStatus(problem.termination_status()), TotalCost(problem.objective_value())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal network plan: termination status {status}") + total_cost = cost[0] + print(f"• objective: {round(total_cost, 9)}\n") + + print(f"\nStatus: {status[0]}") + print(f"Total cost: ${total_cost:,.2f}") + + print("\nOpened fulfillment centers:") + model.select( + Site.name.alias("fulfillment_center"), + Site.capacity.alias("capacity"), + Site.fixed_cost.alias("fixed_cost"), + ).where(Site.x_open > 0.5).inspect() + + print("\nActive flows:") + model.select( + Lane.source.name.alias("from_site"), + Lane.dest.name.alias("to_site"), + Lane.x_flow.alias("flow"), + Lane.cost_per_unit.alias("unit_cost"), + ).where(Lane.x_flow > 0.001).inspect() From 0c46d58668db76d78cacf6bbf74f7346c88d2a6f Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:18:19 -0800 Subject: [PATCH 010/162] production_planning: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and objective through the model's own relationships and the plan through Production.x_quantity. The README covers the deployment config, the internal schema grants made before the first run, and re-running, and its expected output drops the solver-detail lines the script no longer prints. --- production_planning/README.md | 56 ++++++++++++++++++---- production_planning/production_planning.py | 55 +++++++++++++++------ 2 files changed, 88 insertions(+), 23 deletions(-) diff --git a/production_planning/README.md b/production_planning/README.md index 90fe8fa4..362a3974 100644 --- a/production_planning/README.md +++ b/production_planning/README.md @@ -36,7 +36,7 @@ This template finds the profit-maximizing production plan across a set of machin ## What's included -- `production_planning.py` -- Main script with the scenario model, constraints, objective, and result summary +- `production_planning.py` -- Main script with the scenario model, constraints, objective, and result summary; it deploys the model and prints the report from the deployed model - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/products.csv` -- Products with base demand and per-unit profit margins - `data/machines.csv` -- Machines with available hours per planning period @@ -47,7 +47,8 @@ This template finds the profit-maximizing production plan across a set of machin ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -75,23 +76,62 @@ This template finds the profit-maximizing production plan across a set of machin python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: production_planning_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .PRODUCTION_PLANNING # published views and tables + identity: + internal_schema: .PRODUCTION_PLANNING_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python production_planning.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .PRODUCTION_PLANNING_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; ``` + The deploy creates the public schema `.PRODUCTION_PLANNING` if it doesn't exist. 5. Run: ```bash python production_planning.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .PRODUCTION_PLANNING_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (all scenarios solved together in a single call): ```text Solve result: • status: OPTIMAL • objective: 44735.0 - • solver: HiGHS - • relative gap: 0.0 Production plan per scenario: scenario machine product quantity @@ -178,7 +218,7 @@ Focus on the first changes most users will make. ### Tune parameters - **Demand scenarios** — edit the `scenario_data` list at the top of the script to change the multipliers or add scenarios. -- **Solver time limit** — `time_limit_sec` (default `60`) on the `problem.solve(...)` call. +- **Solver time limit** — `time_limit_sec` (default `60`) on the `problem.configure(...)` call. - **Extend scenario analysis** to vary other parameters, such as machine availability or profit margins. ### Extend the model diff --git a/production_planning/production_planning.py b/production_planning/production_planning.py index d3b6fe4c..d22133e3 100644 --- a/production_planning/production_planning.py +++ b/production_planning/production_planning.py @@ -18,7 +18,9 @@ - One solve handles all demand levels; results extracted via model.select(). Run: - `python production_planning.py` + `python production_planning.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints the solver termination status, objective value, and a table of @@ -142,21 +144,44 @@ # Solve (single solve for all scenarios) # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -problem.solve_info().display() +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"production planning solve status is {String:status}") +TotalProfit = model.Relationship(f"production planning total profit is {Float:profit}") +model.define(SolveStatus(problem.termination_status()), TotalProfit(problem.objective_value())) # -------------------------------------------------- -# Extract results per scenario +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nProduction plan per scenario:") -model.select( - Scenario.name.alias("scenario"), - Production.rate.machine.name.alias("machine"), - Production.rate.product.name.alias("product"), - x_qty.alias("quantity"), -).where( - Production.x_quantity(Scenario, x_qty), x_qty > 0.001 -).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + profit = model.select(TotalProfit).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal production plan: termination status {status}") + print(f"• objective: {round(profit[0], 9)}\n") + + # -------------------------------------------------- + # Extract results per scenario + # -------------------------------------------------- + + print("\nProduction plan per scenario:") + model.select( + Scenario.name.alias("scenario"), + Production.rate.machine.name.alias("machine"), + Production.rate.product.name.alias("product"), + x_qty.alias("quantity"), + ).where(Production.x_quantity(Scenario, x_qty), x_qty > 0.001).inspect() From 79c79b832c77090a28b99eb5d024217c88f3e8b0 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:21:37 -0800 Subject: [PATCH 011/162] retail_markdown: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and total revenue through the model's own relationships and the schedules through the template's own variable properties. The README covers the deployment config, the internal schema grants made before the first run, and re-running. --- retail_markdown/README.md | 54 +++++++++++++++++--- retail_markdown/retail_markdown.py | 82 ++++++++++++++++++++---------- 2 files changed, 102 insertions(+), 34 deletions(-) diff --git a/retail_markdown/README.md b/retail_markdown/README.md index c2afc23a..1b053db4 100644 --- a/retail_markdown/README.md +++ b/retail_markdown/README.md @@ -38,7 +38,7 @@ This template finds the discount schedule that maximizes revenue across a multi- ## What's included -- `retail_markdown.py` -- Main script defining the MIP model with discount selection, sales tracking, and revenue optimization +- `retail_markdown.py` -- Main script defining the MIP model with discount selection, sales tracking, and revenue optimization; it deploys the model and prints the report from the deployed model - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/products.csv` -- Products with initial price, cost, inventory, base demand, and salvage rate - `data/discounts.csv` -- Discount levels with percentage and demand lift factor @@ -49,7 +49,8 @@ This template finds the discount schedule that maximizes revenue across a multi- ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -77,15 +78,56 @@ This template finds the discount schedule that maximizes revenue across a multi- python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: retail_markdown_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .RETAIL_MARKDOWN # published views and tables + identity: + internal_schema: .RETAIL_MARKDOWN_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python retail_markdown.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .RETAIL_MARKDOWN_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; ``` + The deploy creates the public schema `.RETAIL_MARKDOWN` if it doesn't exist. 5. Run: ```bash python retail_markdown.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .RETAIL_MARKDOWN_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output: ```text @@ -159,7 +201,7 @@ Focus on the first changes most users will make. - **Discount levels** — modify `discounts.csv` to add finer or coarser tiers with different demand lifts. - **Planning horizon** — add or remove rows in `weeks.csv`; the model scales with longer horizons. -- **Solver time limit** — `time_limit_sec` (default `60`) on the `problem.solve(...)` call. +- **Solver time limit** — `time_limit_sec` (default `60`) on the `problem.configure(...)` call. ### Extend the model diff --git a/retail_markdown/retail_markdown.py b/retail_markdown/retail_markdown.py index 2127d4f2..7fccdece 100644 --- a/retail_markdown/retail_markdown.py +++ b/retail_markdown/retail_markdown.py @@ -12,7 +12,9 @@ - Maximize total revenue from discounted sales plus salvage value of leftover inventory. Run: - `python retail_markdown.py` + `python retail_markdown.py` deploys the model to the schema in raiconfig.yaml, waits + for its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints the solver termination status, objective value (sales + salvage), and @@ -170,32 +172,56 @@ # Solve and check solution # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -si = problem.solve_info() -si.display() +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) -print(f"Status: {si.termination_status}") -print(f"Total revenue (sales + salvage): ${si.objective_value:.2f}") +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() -# Extract solution via model.select() — properties are populated after solve -print("\n=== Selected Discounts by Product-Week ===") -model.select( - Product.name.alias("product"), Week_ref.num.alias("week"), - Discount_ref.discount_pct.alias("discount_pct"), -).where( - Product.x_select(Week_ref, Discount_ref, selection_ref), selection_ref > 0.5 -).inspect() - -print("\n=== Sales by Product-Week ===") -model.select( - Product.name.alias("product"), Week_ref.num.alias("week"), - Discount_ref.discount_pct.alias("discount_pct"), sales_ref.alias("units_sold"), -).where(Product.x_sales(Week_ref, Discount_ref, sales_ref), sales_ref > 0.01).inspect() - -print("\n=== Cumulative Sales by Product-Week ===") -model.select( - Product.name.alias("product"), Week_ref.num.alias("week"), - cumulative_ref.alias("cumulative_sold"), -).where(Product.x_cuml_sales(Week_ref, cumulative_ref)).inspect() +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"retail markdown solve status is {String:status}") +TotalRevenue = model.Relationship(f"retail markdown total revenue is {Float:revenue}") +model.define(SolveStatus(problem.termination_status()), TotalRevenue(problem.objective_value())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + total_revenue = model.select(TotalRevenue).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal markdown plan: termination status {status}") + print(f"• objective: {round(total_revenue[0], 9)}\n") + + print(f"Status: {status[0]}") + print(f"Total revenue (sales + salvage): ${total_revenue[0]:.2f}") + + # Extract solution via model.select() — properties are populated after solve + print("\n=== Selected Discounts by Product-Week ===") + model.select( + Product.name.alias("product"), Week_ref.num.alias("week"), + Discount_ref.discount_pct.alias("discount_pct"), + ).where( + Product.x_select(Week_ref, Discount_ref, selection_ref), selection_ref > 0.5 + ).inspect() + + print("\n=== Sales by Product-Week ===") + model.select( + Product.name.alias("product"), Week_ref.num.alias("week"), + Discount_ref.discount_pct.alias("discount_pct"), sales_ref.alias("units_sold"), + ).where(Product.x_sales(Week_ref, Discount_ref, sales_ref), sales_ref > 0.01).inspect() + + print("\n=== Cumulative Sales by Product-Week ===") + model.select( + Product.name.alias("product"), Week_ref.num.alias("week"), + cumulative_ref.alias("cumulative_sold"), + ).where(Product.x_cuml_sales(Week_ref, cumulative_ref)).inspect() From 6be82ea79fc5f2b789d83b2d7f56a50f7e1bf354 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:25:28 -0800 Subject: [PATCH 012/162] supply_chain_transport: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and total cost through the model's own relationships and the plan through the template's own variable properties. The README covers the deployment config, the internal schema grants made before the first run, and re-running. --- supply_chain_transport/README.md | 52 +++++++++-- .../supply_chain_transport.py | 86 ++++++++++++------- 2 files changed, 103 insertions(+), 35 deletions(-) diff --git a/supply_chain_transport/README.md b/supply_chain_transport/README.md index 35afc880..cbbba084 100644 --- a/supply_chain_transport/README.md +++ b/supply_chain_transport/README.md @@ -38,7 +38,7 @@ The model demonstrates several advanced techniques: multi-period inventory flow ## What's included -- `supply_chain_transport.py` -- main script with ontology, formulation, and solver call +- `supply_chain_transport.py` -- main script with ontology, formulation, and solver configuration; it deploys the model and prints the report from the deployed model - `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself - `data/freight_groups.csv` -- 2 freight groups with inventory/transport/arrival windows - `pyproject.toml` -- Python package configuration @@ -47,7 +47,8 @@ The model demonstrates several advanced techniques: multi-period inventory flow ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -76,15 +77,56 @@ The model demonstrates several advanced techniques: multi-period inventory flow python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: supply_chain_transport_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .SUPPLY_CHAIN_TRANSPORT # published views and tables + identity: + internal_schema: .SUPPLY_CHAIN_TRANSPORT_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python supply_chain_transport.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .SUPPLY_CHAIN_TRANSPORT_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; ``` + The deploy creates the public schema `.SUPPLY_CHAIN_TRANSPORT` if it doesn't exist. 5. Run: ```bash python supply_chain_transport.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .SUPPLY_CHAIN_TRANSPORT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output: ```text diff --git a/supply_chain_transport/supply_chain_transport.py b/supply_chain_transport/supply_chain_transport.py index f70fabbb..dfb165d2 100644 --- a/supply_chain_transport/supply_chain_transport.py +++ b/supply_chain_transport/supply_chain_transport.py @@ -11,7 +11,9 @@ - Minimize total cost (inventory holding + TL fixed cost + LTL variable cost). Run: - `python supply_chain_transport.py` + `python supply_chain_transport.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from the + deployed model. Output: Prints the solver termination status, total cost, and tables showing inventory @@ -252,32 +254,56 @@ # Solve and check solution # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -si = problem.solve_info() -si.display() - -print(f"Status: {si.termination_status}") -print(f"Total cost: ${si.objective_value:.2f}") - -# Extract solution via model.select() — properties are populated after solve -print("\n=== Inventory Levels ===") -model.select( - FreightGroup.name.alias("freight_group"), time_period_ref.alias("day"), - x_inv.alias("inventory"), -).where(FreightGroup.x_inv(time_period_ref, x_inv)).inspect() - -print("\n=== Transport Quantities ===") -model.select( - TransportType.name.alias("type"), FreightGroup_ref.name.alias("freight_group"), - time_period_ref.alias("day"), x_qty_tra.alias("quantity"), -).where( - TransportType.x_qty_tra(FreightGroup_ref, time_period_ref, x_qty_tra), x_qty_tra > 0.001 -).inspect() - -print("\n=== Arrival Days ===") -model.select( - FreightGroup.name.alias("freight_group"), - FreightGroup.z_arr_day.alias("arrival_day"), -).inspect() +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"supply chain transport solve status is {String:status}") +TotalTransportCost = model.Relationship(f"supply chain transport total cost is {Float:cost}") +model.define(SolveStatus(problem.termination_status()), TotalTransportCost(problem.objective_value())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalTransportCost).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal transport plan: termination status {status}") + print(f"• objective: {round(cost[0], 9)}\n") + + print(f"Status: {status[0]}") + print(f"Total cost: ${cost[0]:.2f}") + + # Extract solution via model.select() — properties are populated after solve + print("\n=== Inventory Levels ===") + model.select( + FreightGroup.name.alias("freight_group"), time_period_ref.alias("day"), + x_inv.alias("inventory"), + ).where(FreightGroup.x_inv(time_period_ref, x_inv)).inspect() + + print("\n=== Transport Quantities ===") + model.select( + TransportType.name.alias("type"), FreightGroup_ref.name.alias("freight_group"), + time_period_ref.alias("day"), x_qty_tra.alias("quantity"), + ).where( + TransportType.x_qty_tra(FreightGroup_ref, time_period_ref, x_qty_tra), x_qty_tra > 0.001 + ).inspect() + + print("\n=== Arrival Days ===") + model.select( + FreightGroup.name.alias("freight_group"), + FreightGroup.z_arr_day.alias("arrival_day"), + ).inspect() From 92360311cdb5ce230238ee64d2c8c8c2bebad9ef Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:17:00 -0800 Subject: [PATCH 013/162] planogram_optimization: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and objective through the model's own relationships and keeps the demand-table lookup check. The problem.verify call is removed; the constraints are unchanged. The README covers the deployment config, the internal schema grants and re-running. --- planogram_optimization/README.md | 69 +++++-- .../planogram_optimization.py | 185 ++++++++++-------- planogram_optimization/runbook.md | 2 +- 3 files changed, 154 insertions(+), 102 deletions(-) diff --git a/planogram_optimization/README.md b/planogram_optimization/README.md index caf6275c..32aaca95 100644 --- a/planogram_optimization/README.md +++ b/planogram_optimization/README.md @@ -40,11 +40,11 @@ The same pattern applies to any predict-then-optimize problem where the predicti - An element-style decision-indexed table lookup binding `realized_demand` to the matching `PredictedDemand` row via an `implies` cascade - Shelf-length capacity (`sum(Sku.facings * Sku.width_cm).per(Shelf) <= Shelf.length_cm`) and per-category cardinality (`sum(Sku.active).per(Category)` in `[min_skus_active, max_skus_active]`) - A linear `sum(realized_demand)` objective the CSP maximizes -- Post-solve verification via `problem.verify()` for the capacity, cardinality, and active-coupling ICs, plus a Python dict lookup confirming `(sku_id, facings) -> demand_units` matches the input table +- A post-solve check in the report: a Python dict lookup confirming `(sku_id, facings) -> demand_units` matches the input table, and the solver status is `OPTIMAL` ## What's included -- `planogram_optimization.py` -- main script with ontology, decisions, constraints, and solver call +- `planogram_optimization.py` -- main script with ontology, decisions, constraints, and solver configuration; it deploys the model and prints the report from the deployed model - **Runbook**: `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/skus.csv` -- 18 SKUs across 4 categories (snacks, beverages, candy, household_paper) with width, max-facings, and pre-assigned shelf; `sku_id` must be unique, `category` must match a `categories.csv` name, and `assigned_shelf_id` must reference a `shelves.csv` id. The `brand` column is loaded but not used by the bundled CSP -- it is reserved for the brand-block contiguity customization described under "Customize this template" - `data/shelves.csv` -- 4 shelves (Top Eye-Level 100cm, Upper-Middle 80cm, Lower-Middle 80cm, Bottom 90cm); `id` must be unique and is the foreign-key target of `skus.assigned_shelf_id` @@ -56,7 +56,8 @@ The same pattern applies to any predict-then-optimize problem where the predicti ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -85,24 +86,62 @@ The same pattern applies to any predict-then-optimize problem where the predicti python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: planogram_optimization_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .PLANOGRAM_OPTIMIZATION # published views and tables + identity: + internal_schema: .PLANOGRAM_OPTIMIZATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python planogram_optimization.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .PLANOGRAM_OPTIMIZATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; ``` + The deploy creates the public schema `.PLANOGRAM_OPTIMIZATION` if it doesn't exist. 5. Run: ```bash python planogram_optimization.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .PLANOGRAM_OPTIMIZATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (the solver maximizes total predicted weekly demand; the exact selection may vary across solver versions if multiple allocations achieve the same optimal objective). The script first prints the formulation (~30 lines, omitted here for brevity), then the solve-result block, then the per-SKU allocation, shelf utilization, and category active counts: ```text Solve result: • status: OPTIMAL • objective: 1656 - • solve time: 1.4s - • num_points: 1 - • solver: MiniZinc_unknown Optimal facings per SKU: id name category shelf facings realized_demand @@ -140,7 +179,7 @@ The same pattern applies to any predict-then-optimize problem where the predicti 3 snacks 5 3 5 ``` - `MiniZinc_unknown` is the version string MiniZinc reports for itself today; the underlying solver binary is selected by the RAI Native App. Total weekly demand of `1656` units, 16 of 18 SKUs active. Two SKUs go inactive because the candy and household_paper categories each have 4 SKUs but `max_skus_active=3`: Mint Roll loses the candy cap (lowest predicted demand at every k), Paper Towels 6pk loses the household_paper cap (lowest predicted demand-per-cm). The bottom shelf is fully binding (90/90cm); the other three are within 1-3cm of capacity. + Total weekly demand of `1656` units, 16 of 18 SKUs active. Two SKUs go inactive because the candy and household_paper categories each have 4 SKUs but `max_skus_active=3`: Mint Roll loses the candy cap (lowest predicted demand at every k), Paper Towels 6pk loses the household_paper cap (lowest predicted demand-per-cm). The bottom shelf is fully binding (90/90cm); the other three are within 1-3cm of capacity. ## Template structure ```text @@ -181,14 +220,14 @@ For the full concept and property definitions, see `planogram_optimization.py`; The CSP picks integer facings per SKU; the predictive output binds each SKU's realized demand via a decision-indexed table lookup, and the solver maximizes total predicted demand under shelf and category limits. ```text -skus + shelves + categories + predicted-demand table → facings / realized_demand / active decisions → lookup + capacity + cardinality constraints → maximize demand → solve → verify +skus + shelves + categories + predicted-demand table → facings / realized_demand / active decisions → lookup + capacity + cardinality constraints → maximize demand → configure solve → deploy → check and report ``` **The decision-indexed table lookup is the predict-to-CSP hand-off.** For each SKU, `realized_demand` must equal the `PredictedDemand` row whose `facings_count` matches the chosen `facings`. Because equating a data property with a decision variable inside a `where` binding isn't a form the prescriptive rewriter lowers today, the lookup is written with an `implies` cascade — one half-reified equality per candidate facing count, of which only the row matching the chosen `facings` activates. This is the element-style lookup: no bilinearity, no big-M, no SOS2. **An `active` indicator couples to `facings` via two linear inequalities.** The per-category cardinality is a relational sum over a 0/1 `active` variable (a sum over a Boolean expression like `facings >= 1` is not a valid form). Two linear constraints pin `active = 1` whenever `facings >= 1` and `active = 0` otherwise, after which the per-category count reads as a plain `sum(active).per(Category)` with min/max bounds. Shelf capacity is likewise a plain relational inequality: facings times width, summed per shelf, cannot exceed the shelf length. -**Verification is split by constraint kind.** The `implies`-bodied lookup is solver-only — it goes to the solver but is not passed to `verify()`, which would return silently OK without evaluating it; instead a post-solve Python dict re-checks each `(sku_id, facings) -> demand_units` against the input table. The pure relational-arithmetic constraints (capacity, cardinality, active coupling) *are* re-evaluated by `verify()`, and a post-solve assertion confirms the solver reached `OPTIMAL`. +**Checks run in the report.** The `implies`-bodied lookup is solver-only; after the deploy, a Python dict re-checks each `(sku_id, facings) -> demand_units` against the input table, and an assertion confirms the solver reached `OPTIMAL`. The constraints (lookup, capacity, cardinality, active coupling) all go to the solver; the report does not re-evaluate them, and a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` reproduces the pipeline step by step with the RAI skills. @@ -223,7 +262,7 @@ For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` r - For a live catalog, replace the `read_csv(...)` loads with `model.data(snowflake_table)` calls so the SKUs, shelves, categories, and predicted-demand table read directly from your warehouse; the CSP is unchanged. - The vendored `predicted_demand_table.csv` becomes a materialized inference output on a schedule — refresh it from your demand model, and the pre-solve completeness guard (`_assert_demand_table_complete`) gates each run. -- The bundled data is 18 SKUs across 4 shelves; the CSP scales to whatever the constraint solver's budget allows. Raise `time_limit_sec` in the `problem.solve(...)` call for larger catalogs. +- The bundled data is 18 SKUs across 4 shelves; the CSP scales to whatever the constraint solver's budget allows. Raise `time_limit_sec` in the `problem.configure(...)` call for larger catalogs. - Pin `relationalai` (see Prerequisites) so runs stay reproducible across environments. ## Troubleshooting @@ -241,7 +280,7 @@ For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` r Import error or AttributeError on relationalai - Confirm your virtual environment is active: `which python` should point to `.venv`. -- Reinstall dependencies: `python -m pip install .`. The pinned version (`relationalai==2.1.1`) ships the `solve_info()`, `verify()`, and `where().require()` APIs this template uses; older versions lack them and produce attribute errors. +- Reinstall dependencies: `python -m pip install .`. The pinned version (`relationalai==2.1.1`) ships the `configure()`, `install_display_strings()`, and `where().require()` APIs this template uses; older versions lack them and produce attribute errors. - If you share a venv across templates, run `python -m pip install --upgrade --force-reinstall relationalai==2.1.1`. @@ -283,7 +322,7 @@ For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` r ### Core concepts - [Predict-then-optimize patterns](https://docs.relational.ai/) — feeding a predictive model's output into a prescriptive solve on one ontology. -- [Prescriptive reasoner](https://docs.relational.ai/) — the `Problem` API: integer decisions (`solve_for`), constraints (`satisfy`), objective (`maximize`), and `verify`. +- [Prescriptive reasoner](https://docs.relational.ai/) — the `Problem` API: integer decisions (`solve_for`), constraints (`satisfy`), objective (`maximize`), and `configure`. ### Language / modeling reference diff --git a/planogram_optimization/planogram_optimization.py b/planogram_optimization/planogram_optimization.py index b74da365..0eaf57d8 100644 --- a/planogram_optimization/planogram_optimization.py +++ b/planogram_optimization/planogram_optimization.py @@ -19,13 +19,15 @@ pair so the per-category active-SKU cardinality is expressible as `sum(Sku.active).per(Category)`). - Capacity, cardinality, and the active-iff-facings coupling are pure - relational arithmetic and are re-evaluated by `problem.verify()`. The - implies-bodied table lookup is solver-only; a post-solve check - re-validates `(sku_id, facings) -> demand_units` against the input - table via a Python dict lookup. + relational arithmetic constraints. The implies-bodied table lookup is + solver-only; a post-solve check re-validates + `(sku_id, facings) -> demand_units` against the input table via a Python + dict lookup. Run: - `python planogram_optimization.py` + `python planogram_optimization.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and + prints the report from the deployed model. Output: Prints the formulation, the solve-result block, the optimal facings per @@ -275,8 +277,7 @@ def _assert_shelves_cover_skus(skus_csv, shelves_csv): # - `Sku.facings <= Sku.max_facings * Sku.active` forces `facings == 0` when # `active == 0`. # Together they pin `active = 1` whenever `facings >= 1`, and `active = 0` -# otherwise. Pure relational arithmetic, so both ICs are re-evaluated by -# `problem.verify()` below. +# otherwise. active_lower_ic = model.where(Sku).require(Sku.facings >= Sku.active) active_upper_ic = model.where(Sku).require(Sku.facings <= Sku.max_facings * Sku.active) problem.satisfy(active_lower_ic) @@ -295,88 +296,100 @@ def _assert_shelves_cover_skus(skus_csv, shelves_csv): problem.maximize(sum(Sku.realized_demand)) # -------------------------------------------------- -# Solve and verify +# Solve # -------------------------------------------------- -problem.display() -problem.solve("minizinc", time_limit_sec=60) -problem.solve_info().display() - -# Re-check the relational arithmetic ICs in the returned solution. Never -# pass the implies-bodied table lookup (demand_lookup_ic) to verify() -- -# implies-bodied ICs are solver-only and verify() returns silently-OK -# without actually evaluating them. The lookup is verified post-solve via -# a Python dict lookup below. -problem.verify( - shelf_capacity_ic, - category_min_ic, - category_max_ic, - active_lower_ic, - active_upper_ic, -) -model.require(problem.termination_status() == "OPTIMAL") - -# Post-solve table-lookup check. Pull the chosen `(Sku.id, Sku.facings, -# Sku.realized_demand)` triples and confirm `realized_demand` matches the -# input table at the chosen `facings`. Catches the silent-failure mode -# where the implies cascade leaves `realized_demand` unconstrained for a -# missing (sku_id, k) row -- the pre-solve `_assert_demand_table_complete` -# guard already covers the bundled data, but this is defense in depth. -solution_df = model.select( - Sku.id.alias("sku_id"), - Sku.facings.alias("facings_count"), - Sku.realized_demand.alias("realized_demand"), -).to_df() -expected_lookup = { - (int(row.sku_id), int(row.facings_count)): int(row.demand_units) - for row in demand_csv.itertuples(index=False) -} -mismatches = [] -for row in solution_df.itertuples(index=False): - key = (int(row.sku_id), int(row.facings_count)) - expected = expected_lookup.get(key) - if expected is None or int(row.realized_demand) != expected: - mismatches.append((*key, int(row.realized_demand), expected)) -if mismatches: - raise AssertionError( - "Demand-lookup verification failed: Sku.realized_demand does not " - "match PredictedDemand.demand_units at the chosen facings count for " - f"{len(mismatches)} SKU(s). Pre-solve guards rule out the common " - "case (missing rows in predicted_demand_table.csv); reaching this " - "check means the implies cascade did not bind realized_demand for " - "the chosen `facings`. Inspect the formulation printed by " - "`problem.display()` to confirm the demand-lookup constraint expanded " - "as expected. Mismatches " - "(sku_id, facings, realized_demand, expected_demand):\n" - f"{mismatches}" - ) +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"planogram solve status is {String:status}") +TotalDemand = model.Relationship(f"planogram total demand is {Integer:demand}") +model.define(SolveStatus(problem.termination_status()), TotalDemand(problem.objective_value())) # -------------------------------------------------- -# Inspect solution +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nOptimal facings per SKU:") -model.select( - Sku.id, - Sku.name, - Sku.category, - Sku.shelf.name.alias("shelf"), - Sku.facings, - Sku.realized_demand, -).inspect() - -print("\nShelf utilization:") -model.select( - Shelf.id, - Shelf.name, - sum(Sku.facings * Sku.width_cm).per(Shelf).alias("used_cm"), - Shelf.length_cm, -).where(Sku.shelf == Shelf).inspect() - -print("\nCategory active counts:") -model.select( - Category.name, - sum(Sku.active).per(Category).alias("active_skus"), - Category.min_skus_active, - Category.max_skus_active, -).where(Sku.category == Category.name).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + demand = model.select(TotalDemand).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + assert status == ["OPTIMAL"], f"termination status {status}" + assert len(demand) == 1, f"expected one objective value, got {demand}" + print(f"• objective: {demand[0]}\n") + + # Post-solve table-lookup check. Pull the chosen `(Sku.id, Sku.facings, + # Sku.realized_demand)` triples and confirm `realized_demand` matches the + # input table at the chosen `facings`. Catches the silent-failure mode + # where the implies cascade leaves `realized_demand` unconstrained for a + # missing (sku_id, k) row -- the pre-solve `_assert_demand_table_complete` + # guard already covers the bundled data, but this is defense in depth. + solution_df = model.select( + Sku.id.alias("sku_id"), + Sku.facings.alias("facings_count"), + Sku.realized_demand.alias("realized_demand"), + ).to_df() + expected_lookup = { + (int(row.sku_id), int(row.facings_count)): int(row.demand_units) + for row in demand_csv.itertuples(index=False) + } + mismatches = [] + for row in solution_df.itertuples(index=False): + key = (int(row.sku_id), int(row.facings_count)) + expected = expected_lookup.get(key) + if expected is None or int(row.realized_demand) != expected: + mismatches.append((*key, int(row.realized_demand), expected)) + if mismatches: + raise AssertionError( + "Demand-lookup verification failed: Sku.realized_demand does not " + "match PredictedDemand.demand_units at the chosen facings count for " + f"{len(mismatches)} SKU(s). Pre-solve guards rule out the common " + "case (missing rows in predicted_demand_table.csv); reaching this " + "check means the implies cascade did not bind realized_demand for " + "the chosen `facings`. Inspect the formulation printed by " + "`problem.display()` to confirm the demand-lookup constraint expanded " + "as expected. Mismatches " + "(sku_id, facings, realized_demand, expected_demand):\n" + f"{mismatches}" + ) + + # ---------------------------------------------- + # Inspect solution + # ---------------------------------------------- + + print("\nOptimal facings per SKU:") + model.select( + Sku.id, + Sku.name, + Sku.category, + Sku.shelf.name.alias("shelf"), + Sku.facings, + Sku.realized_demand, + ).inspect() + + print("\nShelf utilization:") + model.select( + Shelf.id, + Shelf.name, + sum(Sku.facings * Sku.width_cm).per(Shelf).alias("used_cm"), + Shelf.length_cm, + ).where(Sku.shelf == Shelf).inspect() + + print("\nCategory active counts:") + model.select( + Category.name, + sum(Sku.active).per(Category).alias("active_skus"), + Category.min_skus_active, + Category.max_skus_active, + ).where(Sku.category == Category.name).inspect() diff --git a/planogram_optimization/runbook.md b/planogram_optimization/runbook.md index 419893b1..8799a26d 100644 --- a/planogram_optimization/runbook.md +++ b/planogram_optimization/runbook.md @@ -85,7 +85,7 @@ Each SKU has a concave demand curve over facing counts — demand rises with fac **Response** -Total predicted weekly demand around **1,600+ units** — the decision picks facings per SKU (binding each SKU's realized demand to its predicted-demand row via a table lookup), with shelf-length and category-cardinality constraints; `Sku.facings`, `Sku.realized_demand`, and `Sku.active` are written back. (MiniZinc runs to its time limit and returns a strong incumbent surfaced as OPTIMAL, so the exact objective and the specific SKUs dropped vary slightly run to run; the structure below is stable.) +Total predicted weekly demand around **1,600+ units** — the decision picks facings per SKU (binding each SKU's realized demand to its predicted-demand row via a table lookup), with shelf-length and category-cardinality constraints; `Sku.facings`, `Sku.realized_demand`, and `Sku.active` are written back. (MiniZinc runs to its time limit and returns a strong incumbent surfaced as OPTIMAL, so the exact objective and the specific SKUs dropped vary slightly run to run; the structure below is stable.) In `planogram_optimization.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ### 6. Read the planogram From 8d2f57609a494cd39ad095a3c90487aaf4573c91 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:20:04 -0800 Subject: [PATCH 014/162] pod_placement: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads status and objective through the model's own relationships and keeps the OPTIMAL check. The problem.verify call is removed; the constraints are unchanged. The README covers the deployment config, the internal schema grants and re-running. --- pod_placement/README.md | 67 +++++++++++++++----- pod_placement/pod_placement.py | 112 ++++++++++++++++++--------------- pod_placement/runbook.md | 2 +- 3 files changed, 115 insertions(+), 66 deletions(-) diff --git a/pod_placement/README.md b/pod_placement/README.md index d9cfc7c3..8b1a1a76 100644 --- a/pod_placement/README.md +++ b/pod_placement/README.md @@ -46,13 +46,13 @@ This template encodes that stack as a pure constraint satisfaction and optimizat - A reified-cardinality gang-placement IC `sum(Pod.placed).per(Deployment) == Deployment.replicas * Deployment.placed` - A pairwise topology rack-clique IC for distributed-training groups -- `xa + xb <= 1` for every (Pa, Pb, Na, Nb) tuple where the pods are in the same training group and the nodes are in different racks - A linear `maximize(sum(Deployment.placed))` objective the CSP optimizes -- Post-solve verification via `problem.verify()` confirming every IC holds in the returned solution, plus a `termination_status() == "OPTIMAL"` assertion +- A `SolveStatus` check in the report: the script stops unless the solver proved `OPTIMAL` ## What's included The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the optimum -- rack-clique forces the distributed-training group onto one rack, CPU / memory / GPU bin-packing then pin it to a single node at exact 100% utilization on all three resources, anti-affinity partitions the alpha-beta and gamma-delta tenant pods across disjoint node sets, deployment co-location affinity co-locates the cache pair, and the spread cap binds for every multi-replica non-overridden deployment. Swap in your own cluster topology and workload to apply the template to a real cluster. -- `pod_placement.py` -- main script with concepts, decisions, constraints, the solver call, and the post-solve inspection +- `pod_placement.py` -- main script with concepts, decisions, constraints, the solver configuration, and the post-solve inspection; it deploys the model and prints the report from the deployed model - **Runbook**: `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/nodes.csv` -- 8 nodes across 2 zones (`us-east-1a`, `us-east-1b`) and 4 racks; six general-purpose nodes (12000 millicores, 49152 MiB) and two GPU nodes (8000 millicores, 32768 MiB, 4 GPUs each). Total cluster: 88000 millicores, 360448 MiB, 8 GPUs - `data/tenants.csv` -- 4 tenants (`tenant_alpha`, `tenant_beta`, `tenant_gamma`, `tenant_delta`) @@ -67,7 +67,8 @@ The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the op ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -96,24 +97,62 @@ The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the op python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: pod_placement_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .POD_PLACEMENT # published views and tables + identity: + internal_schema: .POD_PLACEMENT_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python pod_placement.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .POD_PLACEMENT_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .POD_PLACEMENT_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .POD_PLACEMENT_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; ``` + The deploy creates the public schema `.POD_PLACEMENT` if it doesn't exist. 5. Run: ```bash python pod_placement.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .POD_PLACEMENT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (the solver maximizes the number of placed deployments; the specific (pod, node) tuples and per-node utilization may vary across solver versions when multiple placements tie at the optimum). The script first prints the formulation (a few thousand lines for the bundled data, omitted here for brevity), then the solve-result block, then the per-node utilization, the per-pod placement, and any unplaced deployments. A representative run looks like: ```text Solve result: • status: OPTIMAL • objective: 12 - • solve time: ~1s - • num_points: 1 - • solver: MiniZinc_unknown Per-node utilization: node_id node cpu_used cpu_cap memory_used gpu_used @@ -183,7 +222,7 @@ For the full concept and property definitions — including the three symmetric The CSP picks a node for each pod via a binary 2D assignment matrix; every other rule is plain relational arithmetic over that matrix, and the solver maximizes the number of placed deployments. ```text -nodes + pods + deployments + affinity / anti-affinity / training relations → 2D on_node matrix + placement indicators → bin-packing / anti-affinity / affinity / spread / gang / rack-clique constraints → maximize placed deployments → solve → verify +nodes + pods + deployments + affinity / anti-affinity / training relations → 2D on_node matrix + placement indicators → bin-packing / anti-affinity / affinity / spread / gang / rack-clique constraints → maximize placed deployments → configure solve → deploy → check and report ``` **A binary 2D matrix is the primary decision.** Each pod gets one binary variable per node (`Pod.on_node(Node, x)`), and a per-pod cardinality constraint pins each pod's row sum to a 0/1 placement indicator. An integer-valued node-id decision would read more naturally for the "different node" pairwise patterns, but per-node aggregates over it would mix a decision variable with a data property inside a `where` binding — a form the prescriptive rewriter doesn't lower today. The 2D matrix sidesteps that: every constraint becomes a plain relational sum over `x`. @@ -196,7 +235,7 @@ nodes + pods + deployments + affinity / anti-affinity / training relations → 2 **Failure-domain spread is a two-key aggregate.** With `max_per_zone` precomputed (default `ceil(replicas / num_zones)`, or the row's `max_per_zone_override`), the count of a deployment's pods per zone must stay under its cap. The override is a per-deployment escape hatch: the bundled data uses it on the distributed-training job, whose rack-clique requirement forces all its pods onto one rack — and therefore one zone — which would otherwise collide with the default spread cap. -Every constraint is pure relational arithmetic (no `implies` bodies), so `problem.verify()` re-evaluates all of them in the returned solution, and a post-solve assertion confirms the solver proved `OPTIMAL` rather than timing out. +Every constraint is pure relational arithmetic (no `implies` bodies), and an assertion in the report confirms the solver proved `OPTIMAL` rather than timing out. The report does not re-evaluate the constraints, and a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces the model step by step with the RAI skills. @@ -256,7 +295,7 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces Import error or AttributeError on relationalai - Confirm your virtual environment is active: `which python` should point to `.venv`. -- Reinstall dependencies: `python -m pip install .`. The pinned version (`relationalai==2.1.1`) ships the `solve_info()`, `verify()`, and `where().require()` APIs this template uses; older versions lack them and produce attribute errors. +- Reinstall dependencies: `python -m pip install .`. The pinned version (`relationalai==2.1.1`) ships the `configure()`, `install_display_strings()`, and `where().require()` APIs this template uses; older versions lack them and produce attribute errors. @@ -286,8 +325,8 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces
Solver hits the time limit (status is not `OPTIMAL`) and the script crashes -- The post-solve `model.require(problem.termination_status() == "OPTIMAL")` is a hard assertion: if you raise `time_limit_sec` and the solver returns a feasible-but-not-proven-optimal result (`SOLUTION_LIMIT` or `LOCALLY_SOLVED`), the script raises rather than print a partial solution. -- For a tutorial run on the bundled data this is the right posture (the oracle is "the solver proved 11/12"). For larger user data, either (a) raise `time_limit_sec` until the solver proves optimality, or (b) replace `model.require(...)` with a `print(...)` warning and let the inspection blocks run on the best feasible placement found. +- The report's `assert status == ["OPTIMAL"]` is a hard assertion: if you raise `time_limit_sec` and the solver returns a feasible-but-not-proven-optimal result (`SOLUTION_LIMIT` or `LOCALLY_SOLVED`), the script raises rather than print a partial solution. +- For a tutorial run on the bundled data this is the right posture (the oracle is "the solver proved 11/12"). For larger user data, either (a) raise `time_limit_sec` until the solver proves optimality, or (b) replace the assertion with a `print(...)` warning and let the inspection blocks run on the best feasible placement found.
diff --git a/pod_placement/pod_placement.py b/pod_placement/pod_placement.py index 22660f5e..03b44149 100644 --- a/pod_placement/pod_placement.py +++ b/pod_placement/pod_placement.py @@ -43,12 +43,12 @@ properties; ``TenantAntiAffinity`` and ``DeploymentAffinity`` are symmetric data relationships closed at definition time so the ICs can use either argument order. -- All ICs are pure relational arithmetic -- no ``implies`` bodies -- - so ``problem.verify()`` re-evaluates every one in the returned - solution. +- All ICs are pure relational arithmetic -- no ``implies`` bodies. Run: - `python pod_placement.py` + `python pod_placement.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report + from the deployed model. Output: Prints the formulation, the solve-result block, per-node @@ -408,56 +408,66 @@ def _compute_max_per_zone(replicas, override): problem.maximize(sum(Deployment.placed)) # -------------------------------------------------- -# Solve and verify +# Solve # -------------------------------------------------- -problem.display() -problem.solve("minizinc", time_limit_sec=60) -problem.solve_info().display() - -# Every IC is pure relational arithmetic, so verify() re-evaluates -# all nine in the returned solution. The post-solve -# `model.require(... == "OPTIMAL")` below is a hard assertion: if -# you raise `time_limit_sec` and the solver returns a feasible-but- -# not-proven-optimal result, the script will raise rather than print -# a partial solution. Soften to a `print(...)` warning if you want -# the inspection blocks to run on partial solutions. -problem.verify( - placement_coupling_ic, - gang_placement_ic, - cpu_capacity_ic, - memory_capacity_ic, - gpu_capacity_ic, - anti_affinity_ic, - affinity_ic, - spread_ic, - rack_clique_ic, -) -model.require(problem.termination_status() == "OPTIMAL") +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"pod placement solve status is {String:status}") +PlacedDeployments = model.Relationship(f"pod placement placed deployments is {Integer:placed}") +model.define(SolveStatus(problem.termination_status()), PlacedDeployments(problem.objective_value())) # -------------------------------------------------- -# Inspect results +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nPer-node utilization:") -model.select( - Node.id.alias("node_id"), - Node.name.alias("node"), - sum(Pod.cpu_millicores * x).where(Pod.on_node(Node, x)).per(Node).alias("cpu_used"), - Node.cpu_millicores.alias("cpu_cap"), - sum(Pod.memory_mib * x).where(Pod.on_node(Node, x)).per(Node).alias("memory_used"), - sum(Pod.gpu_units * x).where(Pod.on_node(Node, x)).per(Node).alias("gpu_used"), -).inspect() - -print("\nPlaced pods (pod_id -> node_id):") -model.select( - Pod.id.alias("pod_id"), - Node.id.alias("node_id"), -).where(Pod.on_node(Node, 1)).inspect() - -print("\nUnplaced deployments (if any):") -model.select( - Deployment.id.alias("deployment_id"), - Deployment.name.alias("deployment"), - Deployment.replicas.alias("replicas"), -).where(Deployment.placed == 0).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. This is a hard assertion: if + # you raise `time_limit_sec` and the solver returns a feasible-but-not-proven- + # optimal result, the script raises rather than print a partial solution. + # Soften to a `print(...)` warning if you want the inspection blocks to run + # on partial solutions. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + placed = model.select(PlacedDeployments).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + assert status == ["OPTIMAL"], f"termination status {status}" + assert len(placed) == 1, f"expected one objective value, got {placed}" + print(f"• objective: {placed[0]}\n") + + # ---------------------------------------------- + # Inspect results + # ---------------------------------------------- + + print("\nPer-node utilization:") + model.select( + Node.id.alias("node_id"), + Node.name.alias("node"), + sum(Pod.cpu_millicores * x).where(Pod.on_node(Node, x)).per(Node).alias("cpu_used"), + Node.cpu_millicores.alias("cpu_cap"), + sum(Pod.memory_mib * x).where(Pod.on_node(Node, x)).per(Node).alias("memory_used"), + sum(Pod.gpu_units * x).where(Pod.on_node(Node, x)).per(Node).alias("gpu_used"), + ).inspect() + + print("\nPlaced pods (pod_id -> node_id):") + model.select( + Pod.id.alias("pod_id"), + Node.id.alias("node_id"), + ).where(Pod.on_node(Node, 1)).inspect() + + print("\nUnplaced deployments (if any):") + model.select( + Deployment.id.alias("deployment_id"), + Deployment.name.alias("deployment"), + Deployment.replicas.alias("replicas"), + ).where(Deployment.placed == 0).inspect() diff --git a/pod_placement/runbook.md b/pod_placement/runbook.md index c9b5ba1c..fbf281f9 100644 --- a/pod_placement/runbook.md +++ b/pod_placement/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Assign pods to nodes to place as many deployments as possible, treating every rule as hard: a pod goes on at most one node; a deployment is placed only if all its replicas are; per node, total CPU, memory, and GPU requests can't exceed capacity; pods of anti-affine tenants can't share a node; affinity-paired deployments must share a node; each deployment's pods are spread so no zone holds more than its share; and the distributed-training pods must all land on the same rack. Maximize the number of placed deployments and persist the assignment to the ontology. -**Response:** OPTIMAL (MiniZinc constraint solver), objective **12 — all 12 deployments placed**. The model is a binary pod-to-node assignment matrix plus placed indicators (hundreds of variables and constraints across the six rule families); the assignment, `Pod.placed`, and `Deployment.placed` are written back to the ontology and re-verified against every rule. +**Response:** OPTIMAL (MiniZinc constraint solver), objective **12 — all 12 deployments placed**. The model is a binary pod-to-node assignment matrix plus placed indicators (hundreds of variables and constraints across the six rule families); the assignment, `Pod.placed`, and `Deployment.placed` are written back to the ontology. In `pod_placement.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the placement From 49e6280cfb539dd1aea51a21550f19d0f2bca12e Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:23:51 -0800 Subject: [PATCH 015/162] product_configurator: run as a deployed model The solve, with solution_limit for enumeration, is declared with problem.configure and runs on each refresh of the deployed model. The report keeps the per-solution reads and the empty-result branch, and prints the status and number of builds. The problem.verify call is removed; the constraints are unchanged. The README covers the deployment config, the internal schema grants and re-running. --- product_configurator/README.md | 66 +++++++++-- product_configurator/product_configurator.py | 116 +++++++++++-------- product_configurator/runbook.md | 2 +- 3 files changed, 121 insertions(+), 63 deletions(-) diff --git a/product_configurator/README.md b/product_configurator/README.md index 10ce43bd..1d4f1690 100644 --- a/product_configurator/README.md +++ b/product_configurator/README.md @@ -18,7 +18,7 @@ tags: Configurable products -- cars, industrial equipment, enterprise software, network gear -- come with hundreds or thousands of options grouped into slots (engine, transmission, trim, sound, wheels, and so on). Picking a buildable combination by hand is hard: options require or exclude other options, regional regulations remove some choices entirely, and the total price has to stay under a target. -A configurator UI rarely wants a *single* feasible build. A buyer's quote, a sales playbook, and a "show me my options under this ceiling" trade-off slider all need the *population* of feasible configurations -- the cheap-but-basic build, the loaded build at the price-ceiling boundary, the build that drops a low-utility option to free up budget elsewhere. This template formulates the configuration problem as a **constraint satisfaction** model using RelationalAI's **prescriptive** reasoning and runs the solver in multi-solution mode: pass `solution_limit=MAX_CONFIGURATIONS` to `problem.solve(...)`, then enumerate each feasible build via `Variable.values(solution_index, value)`. The output is pivoted to one row per build (one column per slot, plus a total in dollars, sorted ascending by price) -- ready to drop straight into a buyer-facing UI. +A configurator UI rarely wants a *single* feasible build. A buyer's quote, a sales playbook, and a "show me my options under this ceiling" trade-off slider all need the *population* of feasible configurations -- the cheap-but-basic build, the loaded build at the price-ceiling boundary, the build that drops a low-utility option to free up budget elsewhere. This template formulates the configuration problem as a **constraint satisfaction** model using RelationalAI's **prescriptive** reasoning and runs the solver in multi-solution mode: pass `solution_limit=MAX_CONFIGURATIONS` to `problem.configure(...)`, so each refresh of the deployed model enumerates the builds, then enumerate each feasible build via `Variable.values(solution_index, value)`. The output is pivoted to one row per build (one column per slot, plus a total in dollars, sorted ascending by price) -- ready to drop straight into a buyer-facing UI. For one target region, every returned build picks exactly one option per slot so that all feature-model rules are satisfied (implies / excludes), only region-allowed options appear, and the total price stays within a ceiling. Solver enumeration guarantees each returned build is *distinct* (differs on at least one option). When `solution_limit` is large enough to exhaust the search, the solver returns the full feasible set and reports status `OPTIMAL`; when the limit is tighter than the feasible set, the solver returns the first K it finds (status `SOLUTION_LIMIT`) and the specific subset depends on solver heuristics, so plan accordingly. @@ -36,8 +36,8 @@ The configurator scenario here is automotive trim, drawn from the public Renault - A constraint model with binary `Option.selected` decisions and four constraint families: per-slot exactly-one, implies, excludes, and price ceiling - A region-filtered decision domain so options not allowed in the target region simply don't appear as decisions - A pre-solve catalog validation pass that fails fast on two pathologies the region filter can hide: a slot with zero region-allowed options (the exactly-one IC would not bind), and an implies rule whose tail option is not allowed in the region (the implies IC would not bind) -- **Multi-solution enumeration as the primary code path**: `problem.solve(..., solution_limit=MAX_CONFIGURATIONS)` runs the search in enumeration mode and `Variable.values(solution_index, value)` surfaces every distinct feasible build; the bundled demo's `MAX_CONFIGURATIONS = 100` is set above the feasible-set size so the solver exhausts the search (status `OPTIMAL`), and a post-solve `pandas.pivot` collapses the per-option rows into one row per build (one column per slot, sorted ascending by total price) for buyer-facing display -- Post-solve sanity check via `problem.verify()` confirming every re-evaluable constraint holds against the first returned configuration (`verify` re-evaluates each named IC against the populated property -- the first solution -- not across every enumerated build, but the model itself enforces the constraints across every solution the solver returns) +- **Multi-solution enumeration as the primary code path**: `problem.configure(..., solution_limit=MAX_CONFIGURATIONS)` makes each refresh run the search in enumeration mode and `Variable.values(solution_index, value)` surfaces every distinct feasible build; the bundled demo's `MAX_CONFIGURATIONS = 100` is set above the feasible-set size so the solver exhausts the search (status `OPTIMAL`), and a post-solve `pandas.pivot` collapses the per-option rows into one row per build (one column per slot, sorted ascending by total price) for buyer-facing display +- A report that prints the solve status and the number of builds, and says so when no feasible build exists; the model itself enforces the constraints across every solution the solver returns ## What's included @@ -54,7 +54,8 @@ The configurator scenario here is automotive trim, drawn from the public Renault ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -83,15 +84,56 @@ The configurator scenario here is automotive trim, drawn from the public Renault python -m pip install . ``` -4. Configure (prompts for Snowflake account, role, and profile name): - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: product_configurator_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .PRODUCT_CONFIGURATOR # published views and tables + identity: + internal_schema: .PRODUCT_CONFIGURATOR_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python product_configurator.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .PRODUCT_CONFIGURATOR_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; ``` + The deploy creates the public schema `.PRODUCT_CONFIGURATOR` if it doesn't exist. 5. Run: ```bash python product_configurator.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .PRODUCT_CONFIGURATOR_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output. With `MAX_CONFIGURATIONS = 100` and `TARGET_REGION = "EU"`, the solver exhausts the search and returns every distinct feasible build (status `OPTIMAL`, 63 builds), pivoted to one row per configuration and sorted ascending by total dollars. A few representative lines (the script prints all 63): @@ -114,7 +156,7 @@ The configurator scenario here is automotive trim, drawn from the public Renault ```text product_configurator/ - product_configurator.py # Main script (ontology, decisions, constraints, solve, inspection) + product_configurator.py # Main script (ontology, decisions, constraints, solver configuration, deploy, inspection) data/ slots.csv # 6 slots (Engine, Transmission, Trim, Sound, Wheels, Roof) options.csv # 16 options across the 6 slots, each priced in integer cents @@ -155,21 +197,21 @@ For the full concept and property definitions, see `product_configurator.py`; `r The model declares a binary "selected" decision per region-allowed option, applies four constraint families, then enumerates every feasible build in one multi-solution solve. ```text -slots + options + implies + excludes + regional rules → region-filtered selected decisions → exactly-one + implies + excludes + price constraints → multi-solution solve → pivot to one row per build +slots + options + implies + excludes + regional rules → region-filtered selected decisions → exactly-one + implies + excludes + price constraints → multi-solution solve configured → deploy → pivot to one row per build ``` **1. Slots, options, and rule tables.** Slots and options are concepts; each option points to its slot, has a price in integer cents, and lists the regions it is allowed in. Each rule is a concept identified by the pair of option ids it links — `Implies` is directional (head -> tail), `Excludes` is symmetric. The compound-key field names are deliberately dissimilar (`head_id`/`tail_id`, `left_id`/`right_id`) because PyRel emits a "potential relationship typo" warning when two compound-key names differ by a single character. **2. The binary decision.** `Option.selected` is 0/1 and is scoped, via a `where` on region availability, to options allowed in the target region — options banned in the region simply never get a decision variable. The `solve_for` call returns a variable handle that step 4 uses to read per-solution outputs. -**3. The four constraint families.** Each is stored in a named variable so it can be re-checked by `problem.verify(...)`, and pushed to the solver with `problem.satisfy(...)`: +**3. The four constraint families.** Each is pushed to the solver with `problem.satisfy(...)`: - **Exactly-one** — the selected options per slot sum to 1, so every build fills each slot exactly once. - **Implies** — for each region-allowed rule, `selected[head] <= selected[tail]`: picking the head forces the tail. - **Excludes** — for each region-allowed pair, `selected[left] + selected[right] <= 1`: at most one of the pair. - **Price ceiling** — the price-weighted sum of selected options stays within `PRICE_CEILING_CENTS`. -**4. Solve in multi-solution mode, verify, and pivot.** Passing `solution_limit=MAX_CONFIGURATIONS` runs the search in enumeration mode, returning up to that many distinct feasible builds. `problem.verify()` re-checks the named constraints against the first returned solution (the constraint structure is shared across every solution the solver returns). The variable handle exposes a `.values(solution_index, value)` relationship over the per-solution outputs; binding the value slot to `1` surfaces just the picked options, and a `pandas.pivot` collapses the per-option rows into one row per build (one column per slot, plus a dollar total, sorted ascending by price) for buyer-facing display. +**4. Solve in multi-solution mode, deploy, and pivot.** Passing `solution_limit=MAX_CONFIGURATIONS` to `problem.configure(...)` makes each refresh of the deployed model run the search in enumeration mode, returning up to that many distinct feasible builds; the report prints the status and the number of builds, and does not re-evaluate the constraints. The variable handle exposes a `.values(solution_index, value)` relationship over the per-solution outputs; binding the value slot to `1` surfaces just the picked options, and a `pandas.pivot` collapses the per-option rows into one row per build (one column per slot, plus a dollar total, sorted ascending by price) for buyer-facing display. For the exact PyRel formulation, see `product_configurator.py`; `runbook.md` reproduces the model step by step with the RAI skills. @@ -248,7 +290,7 @@ For the exact PyRel formulation, see `product_configurator.py`; `runbook.md` rep
How many configurations will the solver return? -- Up to `MAX_CONFIGURATIONS` (100 by default) or however many feasible builds exist in the catalog, whichever is smaller. `solve_info().num_points` reports the actual count; `solve_info().status` reports `OPTIMAL` when the search has been exhausted (every distinct feasible build returned) and `SOLUTION_LIMIT` when the cap was hit before exhaustion (more builds available). +- Up to `MAX_CONFIGURATIONS` (100 by default) or however many feasible builds exist in the catalog, whichever is smaller. The report's `num_points` line shows the actual count; its `status` line shows `OPTIMAL` when the search has been exhausted (every distinct feasible build returned) and `SOLUTION_LIMIT` when the cap was hit before exhaustion (more builds available). - The bundled demo has 63 feasible builds in `EU` and `MAX_CONFIGURATIONS = 100` is large enough to enumerate them all, so status is `OPTIMAL` and the *set* of returned builds is deterministic (ordering is re-imposed by the post-solve sort by `total_$`). - When `MAX_CONFIGURATIONS` is tighter than the feasible set, the solver returns the first K it finds and the specific subset depends on MiniZinc's branching heuristics, so the *set* may shift across solver versions. Treat the `solution` column as a label, not a ranking. - Returned builds are guaranteed to be *distinct* (each differs on at least one option) but not maximally diverse -- two builds may share five of six slots and only differ on the cheapest. For broad spread, layer optimization passes or apply your own diversity filters in post-processing. diff --git a/product_configurator/product_configurator.py b/product_configurator/product_configurator.py index b6ed7f57..3cf244d3 100644 --- a/product_configurator/product_configurator.py +++ b/product_configurator/product_configurator.py @@ -9,10 +9,12 @@ playbook, and a trade-off slider all need the population of configurations. Run: - `python product_configurator.py` + `python product_configurator.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and + prints the report from the deployed model. Output: - Prints the configurator's solver info and every feasible build as a + Prints the solve status, the number of builds, and every feasible build as a pivot table (one row per build, one column per slot, sorted ascending by total price). """ @@ -147,9 +149,9 @@ def _validate_regional_catalog(slots_csv, options_csv, regional_csv, implies_csv problem = Problem(model, Integer) -# `populate=True` (default) writes the first solution back into Option.selected -# so problem.verify(...) can re-evaluate ICs against it; multi-solution output -# below goes through `selected_var.values(sol_idx, ...)` regardless. +# `populate=True` (default) writes the first solution back into Option.selected; +# multi-solution output below goes through `selected_var.values(sol_idx, ...)` +# regardless. # # `where=[Option.allowed_in(TARGET_REGION)]` filters the decision domain so # banned options never get a decision variable at all. The ICs below re-assert @@ -198,53 +200,67 @@ def _validate_regional_catalog(slots_csv, options_csv, regional_csv, implies_csv problem.satisfy(price_ic) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- -problem.display() +# Each refresh of the deployed model runs this solve. # `solution_limit=MAX_CONFIGURATIONS` asks the solver to enumerate up to that # many distinct feasible builds; query each one via `.values(sol_idx, val)`. -problem.solve("minizinc", time_limit_sec=60, solution_limit=MAX_CONFIGURATIONS) -si = problem.solve_info() -si.display() - -# `verify` re-evaluates each named IC against the populated property (the -# first solution). The model itself enforces the ICs across every solution. -problem.verify(exactly_one_ic, implies_ic, excludes_ic, price_ic) - -if si.num_points is None or si.num_points == 0: - print( - f"\nNo feasible build for region {TARGET_REGION!r} under the encoded " - "constraints. See the README troubleshooting section for likely causes." - ) -else: - # ---------------------------------------------- - # Inspect every feasible configuration - # ---------------------------------------------- - - # `selected_var.values(sol_idx, 1)` indexes the per-solution outputs and - # filters to the options the solver picked. Pivot to one row per build, - # one column per slot, sorted ascending by total dollars. - sol_idx = Integer.ref() - selections_df = ( - model.select( - sol_idx.alias("solution"), - selected_var.option.slot.name.alias("slot"), - selected_var.option.name.alias("option"), - selected_var.option.price_cents.alias("price_cents"), +problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_CONFIGURATIONS) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"product configurator solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and number of solutions of the latest refresh's solve. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + si = problem.solve_info() + print(f"\nSolve result:\n• status: {', '.join(status)}\n• num_points: {si.num_points}\n") + + if si.num_points is None or si.num_points == 0: + print( + f"\nNo feasible build for region {TARGET_REGION!r} under the encoded " + "constraints. See the README troubleshooting section for likely causes." + ) + else: + # ------------------------------------------ + # Inspect every feasible configuration + # ------------------------------------------ + + # `selected_var.values(sol_idx, 1)` indexes the per-solution outputs and + # filters to the options the solver picked. Pivot to one row per build, + # one column per slot, sorted ascending by total dollars. + sol_idx = Integer.ref() + selections_df = ( + model.select( + sol_idx.alias("solution"), + selected_var.option.slot.name.alias("slot"), + selected_var.option.name.alias("option"), + selected_var.option.price_cents.alias("price_cents"), + ) + .where(selected_var.values(sol_idx, 1)) + .to_df() + .astype({"solution": "int64", "price_cents": "int64"}) + ) + build_view = selections_df.pivot(index="solution", columns="slot", values="option") + build_view["total_$"] = selections_df.groupby("solution")["price_cents"].sum() // 100 + build_view = build_view.sort_values("total_$").reset_index() + build_view.columns.name = None + print( + f"\nFeasible builds for region {TARGET_REGION!r} " + f"(ceiling ${PRICE_CEILING_CENTS // 100:,}, up to {MAX_CONFIGURATIONS} per run):" ) - .where(selected_var.values(sol_idx, 1)) - .to_df() - .astype({"solution": "int64", "price_cents": "int64"}) - ) - build_view = selections_df.pivot(index="solution", columns="slot", values="option") - build_view["total_$"] = ( - selections_df.groupby("solution")["price_cents"].sum() // 100 - ) - build_view = build_view.sort_values("total_$").reset_index() - build_view.columns.name = None - print( - f"\nFeasible builds for region {TARGET_REGION!r} " - f"(ceiling ${PRICE_CEILING_CENTS // 100:,}, up to {MAX_CONFIGURATIONS} per run):" - ) - print(build_view.to_string(index=False)) + print(build_view.to_string(index=False)) diff --git a/product_configurator/runbook.md b/product_configurator/runbook.md index f56e8f37..80a29c95 100644 --- a/product_configurator/runbook.md +++ b/product_configurator/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Enumerate every legal configuration for the EU region that costs at most $20,000. Use a binary "pick this option" decision over the EU-allowed options, require exactly one option per slot, enforce the implies rules (if the head is picked, the tail must be) and the excludes rules (the two can't both be picked), and keep total price within the ceiling. There's no objective — return the full set of feasible builds in multi-solution mode. -**Response:** OPTIMAL — search exhausted (MiniZinc constraint solver in multi-solution mode). **63 distinct feasible builds** found. Because the EU filter drops the V6, the model has 15 binary option variables and 13 constraints (the V6/Manual exclusion drops out with the V6). The selection is `Option.selected`, with the first solution persisted to the ontology for verification. +**Response:** OPTIMAL — search exhausted (MiniZinc constraint solver in multi-solution mode). **63 distinct feasible builds** found. Because the EU filter drops the V6, the model has 15 binary option variables and 13 constraints (the V6/Manual exclusion drops out with the V6). The selection is `Option.selected`, with the first solution persisted to the ontology. In `product_configurator.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the configuration population From 6d1e9ba75c258dfc54d5a1bd5fee17d021634279 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:27:35 -0800 Subject: [PATCH 016/162] shift_assignment: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report reads the status through the model's own relationship and keeps the OPTIMAL check. The problem.verify call is removed; the constraints are unchanged. The README covers the deployment config, the internal schema grants and re-running. --- shift_assignment/README.md | 62 +++++++++++++++++++++++----- shift_assignment/runbook.md | 2 +- shift_assignment/shift_assignment.py | 52 ++++++++++++++++------- 3 files changed, 89 insertions(+), 27 deletions(-) diff --git a/shift_assignment/README.md b/shift_assignment/README.md index 87bd7810..c9cdd5f7 100644 --- a/shift_assignment/README.md +++ b/shift_assignment/README.md @@ -32,11 +32,11 @@ The template also demonstrates scenario analysis by sweeping over different mini - A constraint model that assigns workers to shifts respecting availability and capacity limits - Scenario analysis across multiple minimum-coverage levels (1, 2, 3 workers per shift) -- Post-solve verification via `problem.verify()` to confirm constraint satisfaction across all scenarios +- A report that prints the solve status and stops unless the solver returned `OPTIMAL` ## What's included -- `shift_assignment.py` -- main script with ontology, constraints, and scenario analysis +- `shift_assignment.py` -- main script with ontology, constraints, and scenario analysis; it deploys the model and prints the report from the deployed model - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/workers.csv` -- 10 workers with IDs and names - `data/shifts.csv` -- 3 shifts (Morning, Afternoon, Night) with capacity limits @@ -47,7 +47,8 @@ The template also demonstrates scenario analysis by sweeping over different mini ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -76,15 +77,56 @@ The template also demonstrates scenario analysis by sweeping over different mini python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: shift_assignment_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .SHIFT_ASSIGNMENT # published views and tables + identity: + internal_schema: .SHIFT_ASSIGNMENT_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python shift_assignment.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .SHIFT_ASSIGNMENT_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; ``` + The deploy creates the public schema `.SHIFT_ASSIGNMENT` if it doesn't exist. 5. Run: ```bash python shift_assignment.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .SHIFT_ASSIGNMENT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (a few representative rows confirm a successful run): ```text @@ -149,13 +191,13 @@ The pipeline loads workers, shifts, and availability pairs, adds a `Scenario` co ```text CSV inputs → load Worker / Shift / availability → add Scenario coverage levels → binary assignment variable (per worker-shift-scenario, scoped to availability) - → coverage + workload + capacity constraints → solve → verify → per-scenario roster + → coverage + workload + capacity constraints → configure solve → deploy → per-scenario roster ``` 1. **Load the data.** Workers and shifts come from their CSVs; availability pairs become a `Worker.available_for(Shift)` relationship that scopes which assignments are even possible. 2. **Set up the decision.** A binary `x_assign` variable indicates whether a worker takes a given shift in a given scenario. Each coverage level (`coverage_1` / `_2` / `_3`) is a `Scenario` with a `min_coverage`, and the variable is scoped to the availability relationship so unavailable pairs are never considered. -3. **Constrain the roster.** Three constraints govern every scenario: each shift meets its `min_coverage`, each worker takes no more than `max_shifts` shifts (default 1), and no shift exceeds its `capacity`. The constraints are named so they can be re-checked after solving. -4. **Solve and verify.** A single solve handles all scenarios simultaneously; this is a feasibility problem with no objective, so the solver returns any assignment satisfying every constraint. `problem.verify()` then re-fires the named constraints as integrity checks to confirm the solution holds. +3. **Constrain the roster.** Three constraints govern every scenario: each shift meets its `min_coverage`, each worker takes no more than `max_shifts` shifts (default 1), and no shift exceeds its `capacity`. +4. **Solve and deploy.** A single solve handles all scenarios simultaneously; this is a feasibility problem with no objective, so the solver returns any assignment satisfying every constraint. `problem.configure(...)` declares the solve, the script deploys the model, and each refresh of the deployed model runs the solve. The report checks the solve status; it does not re-evaluate the constraints, and a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. See `shift_assignment.py` for the implementation and `runbook.md` for the skill-driven reproduction. @@ -214,7 +256,7 @@ See `shift_assignment.py` for the implementation and `runbook.md` for the skill- MiniZinc solver not available - This template uses the MiniZinc constraint solver. Ensure the RAI Native App version supports MiniZinc. -- As an alternative, you can try switching to `"highs"` in the `problem.solve()` call, though HiGHS is designed for linear/MIP problems. +- As an alternative, you can try switching to `"highs"` in the `problem.configure()` call, though HiGHS is designed for linear/MIP problems.
diff --git a/shift_assignment/runbook.md b/shift_assignment/runbook.md index 66e31ea6..c8e93e3f 100644 --- a/shift_assignment/runbook.md +++ b/shift_assignment/runbook.md @@ -38,7 +38,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Is there a feasible assignment of workers to shifts where each shift has at least its minimum coverage and no more than its capacity, each worker works at most one shift, and no one is assigned to a shift they're unavailable for? Solve it for three minimum-coverage targets — 1, 2, and 3 workers per shift — in a single solve, with the coverage target as a first-class scenario concept and the assignment decision indexed by (worker, shift, scenario). Persist the assignments to the ontology. -**Response:** OPTIMAL (satisfiable) at all three coverage levels — 66 binary variables (22 availability pairs x 3 scenarios), 48 constraints. The assignment is written back as `Worker.x_assign(Shift, Scenario)`. This is a feasibility problem with no objective, so the solver returns one valid assignment per level — an alternative valid assignment may differ run to run. +**Response:** OPTIMAL (satisfiable) at all three coverage levels — 66 binary variables (22 availability pairs x 3 scenarios), 48 constraints. The assignment is written back as `Worker.x_assign(Shift, Scenario)`. This is a feasibility problem with no objective, so the solver returns one valid assignment per level — an alternative valid assignment may differ run to run. In `shift_assignment.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the staffing levels diff --git a/shift_assignment/shift_assignment.py b/shift_assignment/shift_assignment.py index 15cc395c..4b382c75 100644 --- a/shift_assignment/shift_assignment.py +++ b/shift_assignment/shift_assignment.py @@ -17,7 +17,9 @@ - One solve handles all coverage levels; results extracted via model.select(). Run: - `python shift_assignment.py` + `python shift_assignment.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report + from the deployed model. Output: Prints per-scenario termination status and a table of assignments, followed @@ -117,23 +119,41 @@ # Solve (single solve for all scenarios) # -------------------------------------------------- -problem.display() -problem.solve("minizinc", time_limit_sec=60) -problem.solve_info().display() +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60) -# Verify constraints hold in the solver's solution — fires ICs without a separate query. -problem.verify(coverage_ic, workload_ic, capacity_ic) -model.require(problem.termination_status() == "OPTIMAL") +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"shift assignment solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) # -------------------------------------------------- -# Extract results per scenario +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nAssignments per scenario:") -model.select( - Scenario.name.alias("scenario"), - Worker.name.alias("worker"), - Shift.name.alias("shift"), -).where( - Worker.x_assign(Shift, Scenario, assigned_ref), assigned_ref > 0 -).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status of the latest refresh's solve. A refresh records any termination + # status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}\n") + assert status == ["OPTIMAL"], f"termination status {status}" + + # ---------------------------------------------- + # Extract results per scenario + # ---------------------------------------------- + + print("\nAssignments per scenario:") + model.select( + Scenario.name.alias("scenario"), + Worker.name.alias("worker"), + Shift.name.alias("shift"), + ).where( + Worker.x_assign(Shift, Scenario, assigned_ref), assigned_ref > 0 + ).inspect() From 21b1aafc3039d150768b1e7bd9bddd8789014351 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:31:08 -0800 Subject: [PATCH 017/162] synthetic_eligibility_records: run as a deployed model The solve, with solution_limit for enumeration, is declared with problem.configure and runs on each refresh of the deployed model. The report keeps the per-solution reads (populate=False) and the empty-result branch, and prints the status and number of records. The README covers the deployment config, the internal schema grants and re-running. --- synthetic_eligibility_records/README.md | 68 ++++++++--- .../synthetic_eligibility_records.py | 109 +++++++++++------- 2 files changed, 119 insertions(+), 58 deletions(-) diff --git a/synthetic_eligibility_records/README.md b/synthetic_eligibility_records/README.md index 720df510..751ea8ed 100644 --- a/synthetic_eligibility_records/README.md +++ b/synthetic_eligibility_records/README.md @@ -21,7 +21,7 @@ tags: Healthcare payer engineering teams, RegTech rule-certification harnesses, and claim-engine fuzzers all need *batches* of internally consistent member eligibility records to test against. Real beneficiary data is gated behind PII rules; sampled production data carries cohort biases; hand-crafted fixtures drift out of sync with the regulations they are meant to exercise. The right alternative is a constrained generative model: declare the eligibility rules once, ask the solver for K records that satisfy every rule simultaneously. -Synthetic-data tooling consumers want a *batch* of K diverse records per solve, not one. A single record can't expose a CFD cascade or a network-attribution corner case; K records spread across age bands, plan types, and provider networks can. This template encodes member eligibility as a **constraint satisfaction** model using RelationalAI's **prescriptive** reasoning and runs the solver in multi-solution mode: pass `solution_limit=K` to `problem.solve(...)`, then enumerate each generated record via `Variable.values(solution_index, value)`. The output is one row per generated member -- ready to drop into a test fixture, a fuzzing oracle, or a coverage matrix. +Synthetic-data tooling consumers want a *batch* of K diverse records per solve, not one. A single record can't expose a CFD cascade or a network-attribution corner case; K records spread across age bands, plan types, and provider networks can. This template encodes member eligibility as a **constraint satisfaction** model using RelationalAI's **prescriptive** reasoning and runs the solver in multi-solution mode: pass `solution_limit=K` to `problem.configure(...)`, so each refresh of the deployed model enumerates the records, then enumerate each generated record via `Variable.values(solution_index, value)`. The output is one row per generated member -- ready to drop into a test fixture, a fuzzing oracle, or a coverage matrix. The rule structure here is drawn from the public [CMS Medicare](https://www.cms.gov/Medicare/Medicare) and [NCQA](https://www.ncqa.org/) regulatory shape: age-by-plan-type CFDs (over-65 must be on Medicare-Advantage; under-65 must not) and PCP-network attribution (the chosen primary-care provider must be in-network for the chosen plan). The same template structure -- decision-valued tuple per record, reference-data lookups via composition, multi-solution enumeration -- applies to any rule-driven synthetic-data domain: KYC member records (banking), tenant lease attributes (proptech), shipment manifests (logistics). @@ -39,13 +39,13 @@ The rule structure here is drawn from the public [CMS Medicare](https://www.cms. - A pair of CFD ICs encoding the two arms of the age-by-plan rule using the forbidden-pair `implies(age_bucket_id == AgeBucket.id, plan_id != Plan.id)` idiom -- safe under the CSP rewriter - A PCP-network attribution IC iterating over reference-data `(Plan, Provider)` tuples in different networks and forbidding the cross-network combination - A pre-solve dense-ID check on `plans.csv`, `providers.csv`, and `age_buckets.csv` so the solver's integer decision bounds line up with the reference rows the CFDs iterate over (sparse IDs would let the solver pick a value with no matching row, leaving the rules unconstrained for that record and silently dropping it from the post-solve display join) -- **Multi-solution enumeration as the primary code path**: `problem.solve(..., solution_limit=MAX_RECORDS)` runs the search in enumeration mode; `Variable.values(solution_index, value)` joins the three decision variables on a shared solution index to reconstruct each record +- **Multi-solution enumeration as the primary code path**: `problem.configure(..., solution_limit=MAX_RECORDS)` makes each refresh run the search in enumeration mode; `Variable.values(solution_index, value)` joins the three decision variables on a shared solution index to reconstruct each record - An empty-result branch driven by `solve_info().num_points`: when no feasible record exists, the script prints a diagnostic instead of hard-failing, which is the right shape for a reusable generator -- No `problem.verify()` call: every IC uses `implies`, which is solver-only -- passing implies-bodied ICs to `verify()` returns silently-OK without actually evaluating them, so the convention is that they must NOT be passed. The CFD and network-attribution invariants are directly visible in the expected-output block in the Quickstart: every record's `age_years` vs `plan_type` and every record's `network` vs `provider` are printed side-by-side +- No check re-evaluates the ICs: every IC uses `implies`, which is solver-only. The CFD and network-attribution invariants are directly visible in the expected-output block in the Quickstart: every record's `age_years` vs `plan_type` and every record's `network` vs `provider` are printed side-by-side ## What's included -- `synthetic_eligibility_records.py` -- main script with ontology, decisions, constraints, and solver call +- `synthetic_eligibility_records.py` -- main script with ontology, decisions, constraints, and solver configuration; it deploys the model and prints the report from the deployed model - `data/age_buckets.csv` -- 4 representative ages spanning the adult/senior split (2 under 65, 2 at or above) - `data/plans.csv` -- 3 plans (PPO, HMO, MedicareAdvantage) each on its own network - `data/providers.csv` -- 4 primary-care providers (1 PPO, 1 HMO, 2 Medicare) so each plan-network has at least one in-network PCP and the bundled K=8 enumeration spans all three plans @@ -55,7 +55,8 @@ The rule structure here is drawn from the public [CMS Medicare](https://www.cms. ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -83,24 +84,62 @@ The rule structure here is drawn from the public [CMS Medicare](https://www.cms. python -m pip install . ``` -4. Configure (prompts for Snowflake account, role, and profile name): - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: synthetic_eligibility_records_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .SYNTHETIC_ELIGIBILITY_RECORDS # published views and tables + identity: + internal_schema: .SYNTHETIC_ELIGIBILITY_RECORDS_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python synthetic_eligibility_records.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .SYNTHETIC_ELIGIBILITY_RECORDS_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; ``` + The deploy creates the public schema `.SYNTHETIC_ELIGIBILITY_RECORDS` if it doesn't exist. 5. Run: ```bash python synthetic_eligibility_records.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .SYNTHETIC_ELIGIBILITY_RECORDS_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output. With `MAX_RECORDS = 16` and a bundled feasible set of 8 records, the solver exhausts the search space and returns status `OPTIMAL` with all 8 records. Each row carries its `solution` index. Solver build strings, exact wall times, and per-solution ordering will vary; the structure of the output and the *set* of returned records is stable: ```text Solve result: • status: OPTIMAL - • objective: 0 - • solve time: 0.12s • num_points: 8 - • solver: MiniZinc_unknown Generated member records (up to 16 per run): solution age_years plan_type plan_network provider_network provider @@ -174,9 +213,10 @@ network_match_ic = model.where(Plan.network_id != Provider.network_id).require( **5. Multi-solution enumeration via `Variable.values(solution_index, value)`.** Capturing the variable subconcept from `solve_for(...)` exposes a `.values(sol_idx, val)` relationship that indexes the per-solution outputs. Binding the value slot directly to a reference Concept's `.id` walks the chosen ID back to that record's columns in one step: ```python -problem.solve("minizinc", time_limit_sec=60, solution_limit=MAX_RECORDS) +problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_RECORDS) + +# in the report, after model.deploy(): si = problem.solve_info() -si.display() sol_idx = Integer.ref() records_df = ( @@ -275,7 +315,7 @@ To extend to a fleet of N members, reintroduce a `Member` concept inside the mod
How many records will the solver return? -- Up to `MAX_RECORDS` (16 by default) or however many feasible records exist in the reference data, whichever is smaller. `solve_info().num_points` reports the actual count after the solve; `solve_info().status` reports `SOLUTION_LIMIT` when the limit was hit (more records available) and `OPTIMAL` when the search has been exhausted. +- Up to `MAX_RECORDS` (16 by default) or however many feasible records exist in the reference data, whichever is smaller. The report's `num_points` line shows the actual count after the solve; its `status` line shows `SOLUTION_LIMIT` when the limit was hit (more records available) and `OPTIMAL` when the search has been exhausted. - Solution ordering is not guaranteed across runs or solver versions; the *set* of returned records may also shift if MiniZinc's branching heuristics see new ties. Treat the `solution` column as a label, not a ranking. - The K returned records are guaranteed to be pairwise *distinct* on at least one decision (age bucket, plan, or provider) but not maximally diverse and not ranked. For broader spread, raise `MAX_RECORDS` past the size of the feasible set so the solver exhausts every distinct case, or add stratification buckets and re-solve per stratum. diff --git a/synthetic_eligibility_records/synthetic_eligibility_records.py b/synthetic_eligibility_records/synthetic_eligibility_records.py index 3da39c67..8b4836dc 100644 --- a/synthetic_eligibility_records/synthetic_eligibility_records.py +++ b/synthetic_eligibility_records/synthetic_eligibility_records.py @@ -11,7 +11,9 @@ want a batch of records per solve, not one. Run: - `python synthetic_eligibility_records.py` + `python synthetic_eligibility_records.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and + prints the report from the deployed model. Output: Prints the formulation, the solver's status block, and every generated @@ -202,55 +204,74 @@ def _warn_orphan_networks(plans_csv, providers_csv): problem.satisfy(non_senior_no_medicare_ic) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- -problem.display() +# Each refresh of the deployed model runs this solve. # `solution_limit=MAX_RECORDS` asks the solver to enumerate up to that # many distinct feasible records; query each one via `.values(idx, val)`. -problem.solve("minizinc", time_limit_sec=60, solution_limit=MAX_RECORDS) -si = problem.solve_info() -si.display() - -# No `problem.verify(...)` call: every IC here is implies-bodied, which is -# solver-only -- verify() would return silently-OK without re-evaluating. -# The invariants are visible by eye in the table below: every row prints -# `age_years` next to `plan_type` (CFD check) and `plan_network` next to -# `provider_network` (network-attribution check). -if si.num_points is None or si.num_points == 0: - print( - "\nThe solver returned no eligibility records. See the printed solve status " - "above and the troubleshooting section in the README for likely causes " - "(over-constrained reference data, mismatched plan/provider networks, or " - "the search budget expired before a record was found)." - ) +problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_RECORDS) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"synthetic eligibility solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) # -------------------------------------------------- -# Inspect every generated member record +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -# `Variable.values(sol_idx, value)` indexes the solver's outputs across -# every returned solution; binding the value slot to a reference Concept's -# `.id` walks the chosen ID back to that record's columns in one step. - -sol_idx = Integer.ref() -records_df = ( - model.select( - sol_idx.alias("solution"), - AgeBucket.age_years.alias("age_years"), - Plan.plan_type.alias("plan_type"), - Plan.network_id.alias("plan_network"), - Provider.network_id.alias("provider_network"), - Provider.name.alias("provider"), - ) - .where( - age_bucket_var.values(sol_idx, AgeBucket.id), - plan_id_var.values(sol_idx, Plan.id), - provider_id_var.values(sol_idx, Provider.id), +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and number of solutions of the latest refresh's solve. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + si = problem.solve_info() + print(f"\nSolve result:\n• status: {', '.join(status)}\n• num_points: {si.num_points}\n") + + # Every IC here is implies-bodied (solver-only), so the report does not + # re-evaluate them. The invariants are visible by eye in the table below: + # every row prints `age_years` next to `plan_type` (CFD check) and + # `plan_network` next to `provider_network` (network-attribution check). + if si.num_points is None or si.num_points == 0: + print( + "\nThe solver returned no eligibility records. See the printed solve status " + "above and the troubleshooting section in the README for likely causes " + "(over-constrained reference data, mismatched plan/provider networks, or " + "the search budget expired before a record was found)." + ) + + # ---------------------------------------------- + # Inspect every generated member record + # ---------------------------------------------- + + # `Variable.values(sol_idx, value)` indexes the solver's outputs across + # every returned solution; binding the value slot to a reference Concept's + # `.id` walks the chosen ID back to that record's columns in one step. + + sol_idx = Integer.ref() + records_df = ( + model.select( + sol_idx.alias("solution"), + AgeBucket.age_years.alias("age_years"), + Plan.plan_type.alias("plan_type"), + Plan.network_id.alias("plan_network"), + Provider.network_id.alias("provider_network"), + Provider.name.alias("provider"), + ) + .where( + age_bucket_var.values(sol_idx, AgeBucket.id), + plan_id_var.values(sol_idx, Plan.id), + provider_id_var.values(sol_idx, Provider.id), + ) + .to_df() + .sort_values("solution") + .reset_index(drop=True) ) - .to_df() - .sort_values("solution") - .reset_index(drop=True) -) -print(f"\nGenerated member records (up to {MAX_RECORDS} per run):") -print(records_df.to_string(index=False)) + print(f"\nGenerated member records (up to {MAX_RECORDS} per run):") + print(records_df.to_string(index=False)) From f4e3846fbda50917f7e3604ee14c390d68ee9b93 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:37:33 -0800 Subject: [PATCH 018/162] synthetic_order_lifecycle: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The derived event type is declared with model.Relationship at module level so it deploys with the model, and the report reads status through the model's own relationship and keeps the OPTIMAL check. The problem.verify call is removed; the constraints are unchanged. The README covers the deployment config and the internal schema grants. --- synthetic_order_lifecycle/README.md | 76 ++++++---- .../synthetic_order_lifecycle.py | 139 ++++++++++-------- 2 files changed, 126 insertions(+), 89 deletions(-) diff --git a/synthetic_order_lifecycle/README.md b/synthetic_order_lifecycle/README.md index f62aa885..591b038c 100644 --- a/synthetic_order_lifecycle/README.md +++ b/synthetic_order_lifecycle/README.md @@ -40,11 +40,11 @@ The same pattern applies to any test-data-generation problem where rows have to - An auxiliary `fill_qty` decision channeled to `qty` when the `FILL` indicator is 1 (else 0) via two `implies` so the per-order fill-conservation aggregate stays linear - Value-pinning: PLACE event's `qty` and `tick_price` pinned to the order's `original_qty` and `original_tick_price` via `implies` - Venue eligibility encoded as a relationship lookup against the event's chosen `venue_id` -- Post-solve verification via `problem.verify()` confirming every re-evaluable constraint in the returned trace (`implies`-bodied and `all_different`-bodied ICs are solver-side only and intentionally excluded -- see step 5) +- A report that prints the solve status and stops unless the solver returned `OPTIMAL` (the constraints are not re-evaluated against the returned trace -- see step 5) ## What's included -- `synthetic_order_lifecycle.py` -- main script with ontology, decisions, constraints, and solver call +- `synthetic_order_lifecycle.py` -- main script with ontology, decisions, constraints, and solver configuration; it deploys the model and prints the report from the deployed model - `data/symbols.csv` -- 5 tradable symbols (AAPL, MSFT, GOOG, NVDA, TSLA) - `data/venues.csv` -- 5 trading venues (NYSE, NASDAQ, ARCA, BATS, IEX) - `data/symbol_venues.csv` -- 13 (symbol, venue) eligible pairs out of 25 possible (sparser than full coverage so venue eligibility visibly binds) @@ -56,7 +56,8 @@ The same pattern applies to any test-data-generation problem where rows have to ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -85,24 +86,57 @@ The same pattern applies to any test-data-generation problem where rows have to python -m pip install . ``` -4. Configure (prompts for Snowflake account, role, and profile name): - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: synthetic_order_lifecycle_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .SYNTHETIC_ORDER_LIFECYCLE # published views and tables + identity: + internal_schema: .SYNTHETIC_ORDER_LIFECYCLE_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python synthetic_order_lifecycle.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .SYNTHETIC_ORDER_LIFECYCLE_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SYNTHETIC_ORDER_LIFECYCLE_META TO APPLICATION ; ``` + The deploy creates the public schema `.SYNTHETIC_ORDER_LIFECYCLE` if it doesn't exist. 5. Run: ```bash python synthetic_order_lifecycle.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output. The script prints the formulation (every variable and constraint, ~1,000 lines on the sample data, omitted here; each event's four type indicators all print as `is_type_` -- so `is_type_1` appears four times -- because variable names are keyed by event id), the solve-result block, the full event trace (all 36 rows; abridged below to the first AAPL block), and per-order fill totals. Rows are sorted by `(order_id, ts_ms)`; the `type` column reads each event's chosen `EventType` member name back from a derived enum-typed property (`OrderEvent.event_type.name`), with no display-side post-processing. Exact event types, timestamps, quantities, prices, and venues vary across runs and solver versions: +6. Expected output. The script prints the formulation (every variable and constraint, ~1,000 lines on the sample data, omitted here; each event's four type indicators all print as `is_type_` -- so `is_type_1` appears four times -- because variable names are keyed by event id), the solve-result block, the full event trace (all 36 rows; abridged below to the first AAPL block), and per-order fill totals. Rows are sorted by `(order_id, ts_ms)`; the `type` column reads each event's chosen `EventType` member name back from a derived enum-typed relationship (`OrderEvent.event_type.name`), with no display-side post-processing. Exact event types, timestamps, quantities, prices, and venues vary across runs and solver versions: ```text Solve result: • status: OPTIMAL - • solve time: 1.17s - • num_points: 1 - • solver: MiniZinc_unknown - • raw status string: SATISFIABLE Generated event trace (one row per slot, sorted by order then timestamp): order_id symbol event_id ts_ms type qty tick_price venue @@ -209,21 +243,13 @@ place_price_match_ic = model.where(OrderEvent.has_type(EventType.PLACE, place_pr ) ``` -**5. Solve and verify.** `implies` and `all_different` are solver-only. They go to `satisfy()` but must NOT be passed to `verify()` -- the relational engine cannot re-evaluate wire-format constraint relations and would return silently-OK regardless of whether the constraint actually holds. The remaining ICs are plain relational arithmetic and ARE re-evaluated by `verify()`: +**5. Solve and deploy.** Every IC goes to the solver through `satisfy()`; the report does not re-evaluate them against the returned trace. `problem.configure(...)` declares the solve, the script deploys the model, and each refresh of the deployed model runs the solve. The report checks the solve status through the model's own `SolveStatus` relationship; a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed trace: ```python -problem.solve("minizinc", time_limit_sec=60) -problem.solve_info().display() - -problem.verify( - type_sum_ic, - exactly_one_place_ic, - at_most_one_cancel_ic, - qty_upper_ic, - venue_ok_ic, - fill_sum_ic, -) -model.require(problem.termination_status() == "OPTIMAL") +problem.configure("minizinc", time_limit_sec=60) + +SolveStatus = model.Relationship(f"synthetic order lifecycle solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) ``` ## Customize this template @@ -254,7 +280,7 @@ model.require(problem.termination_status() == "OPTIMAL") Import error or AttributeError on relationalai - Confirm your virtual environment is active: `which python` should point to `.venv`. -- Reinstall dependencies: `python -m pip install .`. The pinned version (`relationalai==2.1.1`) ships enum members as constants across the DSL and prescriptive reasoning (alongside the `solve_info()`, `verify()`, and chained-`where().require()` APIs); older versions reject the enum-indexed decision variable and produce type or attribute errors. +- Reinstall dependencies: `python -m pip install .`. The pinned version (`relationalai==2.1.1`) ships enum members as constants across the DSL and prescriptive reasoning (alongside the `configure()`, `install_display_strings()`, and chained-`where().require()` APIs); older versions reject the enum-indexed decision variable and produce type or attribute errors. - If you share a venv across templates, run `python -m pip install --upgrade --force-reinstall relationalai==2.1.1`.
diff --git a/synthetic_order_lifecycle/synthetic_order_lifecycle.py b/synthetic_order_lifecycle/synthetic_order_lifecycle.py index e81605ba..21b57fd4 100644 --- a/synthetic_order_lifecycle/synthetic_order_lifecycle.py +++ b/synthetic_order_lifecycle/synthetic_order_lifecycle.py @@ -5,7 +5,7 @@ - Pre-allocate (order, event-slot) rows and load order, symbol, and venue reference data from CSV. - Model each event slot as a decision over type (PLACE / MODIFY / CANCEL / FILL), timestamp, venue, quantity, and tick price. - Encode MiFID II / Reg NMS-flavour sequencing rules: PLACE first, nothing after CANCEL, exactly one PLACE per order, fill-quantity conservation, venue eligibility per symbol. -- Solve as constraint satisfaction (MiniZinc) and verify the relational arithmetic ICs against the returned trace. +- Solve as constraint satisfaction (MiniZinc) and print the returned trace. Event types are modelled as a `model.Enum` (requires relationalai>=1.12): a single enum-indexed property carries one binary decision variable per @@ -16,11 +16,13 @@ $175.00), times are integer milliseconds, quantities are integer shares. Run: - `python synthetic_order_lifecycle.py` + `python synthetic_order_lifecycle.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and + prints the report from the deployed model. Output: Prints the formulation, the generated event trace (one row per slot), - the per-order filled-quantity totals, and post-solve verification. + and the per-order filled-quantity totals. """ from pathlib import Path @@ -265,35 +267,11 @@ class EventType(model.Enum): ) problem.satisfy(fill_sum_ic) -# -------------------------------------------------- -# Solve and check solution -# -------------------------------------------------- - -problem.display() -problem.solve("minizinc", time_limit_sec=60) -problem.solve_info().display() - -# Re-check the relational arithmetic ICs in the returned solution. The -# all_different- and implies-bodied ICs are solver-only and are NOT passed -# here -- verify() returns silently-OK on them without actually checking. -problem.verify( - type_sum_ic, - exactly_one_place_ic, - at_most_one_cancel_ic, - qty_upper_ic, - venue_ok_ic, - fill_sum_ic, -) -model.require(problem.termination_status() == "OPTIMAL") - -# Inspect the generated trace. - -print("\nGenerated event trace (one row per slot, sorted by order then timestamp):") -# Derived enum-typed property mapping each event to its chosen type: the +# Derived enum-typed relationship mapping each event to its chosen type: the # member whose (event, type) indicator the solver set to 1 -- exactly one # per event, guaranteed by the one-hot IC. The trace then reads the type # label off the member's built-in `.name`. -OrderEvent.event_type = model.Property( +OrderEvent.event_type = model.Relationship( f"{OrderEvent} has decided {EventType:event_type}" ) for member in EventType: @@ -302,40 +280,73 @@ class EventType(model.Enum): OrderEvent.has_type(member, chosen_ind), chosen_ind == 1 ) -trace_df = ( - model.select( - OrderEvent.order.id.alias("order_id"), - OrderEvent.order.symbol.name.alias("symbol"), - OrderEvent.event_id.alias("event_id"), - OrderEvent.ts_ms.alias("ts_ms"), - OrderEvent.event_type.name.alias("type"), - OrderEvent.qty.alias("qty"), - OrderEvent.tick_price.alias("tick_price"), - Venue.name.alias("venue"), +# -------------------------------------------------- +# Solve +# -------------------------------------------------- + +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"synthetic order lifecycle solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status of the latest refresh's solve. A refresh records any termination + # status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}\n") + assert status == ["OPTIMAL"], f"termination status {status}" + + # Inspect the generated trace. + + print("\nGenerated event trace (one row per slot, sorted by order then timestamp):") + trace_df = ( + model.select( + OrderEvent.order.id.alias("order_id"), + OrderEvent.order.symbol.name.alias("symbol"), + OrderEvent.event_id.alias("event_id"), + OrderEvent.ts_ms.alias("ts_ms"), + OrderEvent.event_type.name.alias("type"), + OrderEvent.qty.alias("qty"), + OrderEvent.tick_price.alias("tick_price"), + Venue.name.alias("venue"), + ) + .where(Venue.id(OrderEvent.venue_id)) + .to_df() ) - .where(Venue.id(OrderEvent.venue_id)) - .to_df() -) -trace_df = ( - trace_df.astype( - { - "order_id": "int64", - "event_id": "int64", - "ts_ms": "int64", - "qty": "int64", - "tick_price": "int64", - } + trace_df = ( + trace_df.astype( + { + "order_id": "int64", + "event_id": "int64", + "ts_ms": "int64", + "qty": "int64", + "tick_price": "int64", + } + ) + .sort_values(["order_id", "ts_ms"]) + .reset_index(drop=True)[ + ["order_id", "symbol", "event_id", "ts_ms", "type", "qty", "tick_price", "venue"] + ] ) - .sort_values(["order_id", "ts_ms"]) - .reset_index(drop=True)[ - ["order_id", "symbol", "event_id", "ts_ms", "type", "qty", "tick_price", "venue"] - ] -) -print(trace_df.to_string(index=False)) - -print("\nFilled quantity per order (cannot exceed Order.original_qty):") -model.select( - Order.id.alias("order_id"), - Order.original_qty.alias("original_qty"), - rai_sum(OrderEvent.fill_qty).per(Order).alias("filled_qty"), -).where(OrderEvent.order(Order)).inspect() + print(trace_df.to_string(index=False)) + + print("\nFilled quantity per order (cannot exceed Order.original_qty):") + model.select( + Order.id.alias("order_id"), + Order.original_qty.alias("original_qty"), + rai_sum(OrderEvent.fill_qty).per(Order).alias("filled_qty"), + ).where(OrderEvent.order(Order)).inspect() From 885781ce0cec8acfc464dba361d7a670ff2921c6 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:08:02 -0800 Subject: [PATCH 019/162] commercial_underwriting: run as a deployed model The script deploys the model, waits for the first refresh and prints the report from the deployed model. The README covers the deployment config and re-running. --- commercial_underwriting/README.md | 27 +- .../commercial_underwriting.py | 286 +++++++++--------- 2 files changed, 168 insertions(+), 145 deletions(-) diff --git a/commercial_underwriting/README.md b/commercial_underwriting/README.md index 87751f75..b67b87d5 100644 --- a/commercial_underwriting/README.md +++ b/commercial_underwriting/README.md @@ -50,7 +50,8 @@ This template shows how to express that rulebook as declarative logic on a four- ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -84,18 +85,34 @@ This template shows how to express that rulebook as declarative logic on a four- python -m pip install . ``` -4. Configure Snowflake connection and RAI profile: - - ```bash - rai init +4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .COMMERCIAL_UNDERWRITING # published views and tables + identity: + internal_schema: .COMMERCIAL_UNDERWRITING_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python commercial_underwriting.py` stops with an error naming the missing setting. + 5. Run the template: ```bash python commercial_underwriting.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing; it prints the report from the deployed model. A change to the model redeploys it. + + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + 6. Expected output (truncated): ```text diff --git a/commercial_underwriting/commercial_underwriting.py b/commercial_underwriting/commercial_underwriting.py index 9cb6b4de..8e167f1b 100644 --- a/commercial_underwriting/commercial_underwriting.py +++ b/commercial_underwriting/commercial_underwriting.py @@ -20,7 +20,9 @@ PyRel resolves dependencies automatically. Run: - `python commercial_underwriting.py` + `python commercial_underwriting.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from the + deployed model. Output: Prints the eligibility flags found at each level, the per-entity @@ -321,146 +323,150 @@ ).define(RiskTier_Standard(InsuredEntity)) # -------------------------------------------------- -# Reporting +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("=" * 70) -print("Stage 1: Location- and coverage-level eligibility flags") -print("=" * 70) - -poor_fp = model.where(Location.has_poor_fire_protection()).select( - Location.id.alias("location_id"), - Location.address.alias("address"), - Location.fire_protection_class.alias("fp_class"), -).to_df() -print(f"\nLocations with poor fire protection (FP > {MAX_FP_CLASS_FOR_ELIGIBILITY}): {len(poor_fp)}") -if not poor_fp.empty: - print(poor_fp.to_string(index=False)) - -old_constr = model.where(Location.has_old_construction()).select( - Location.id.alias("location_id"), - Location.address.alias("address"), - Location.year_built.alias("year_built"), -).to_df() -print(f"\nLocations with old construction (year_built < {MIN_YEAR_BUILT}): {len(old_constr)}") -if not old_constr.empty: - print(old_constr.to_string(index=False)) - -high_risk = model.where(Location.has_high_risk_occupancy()).select( - Location.id.alias("location_id"), - Location.address.alias("address"), - Location.occupancy_class.alias("occupancy"), -).to_df() -print(f"\nLocations with high-risk occupancy: {len(high_risk)}") -if not high_risk.empty: - print(high_risk.to_string(index=False)) - -underinsured = model.where(Coverage.is_underinsured()).select( - Coverage.id.alias("coverage_id"), - Coverage.location.address.alias("location_address"), - Coverage.coverage_limit.alias("limit"), - Coverage.location.replacement_value.alias("replacement_value"), -).to_df() -underinsured["itv_pct"] = ( - underinsured["limit"] / underinsured["replacement_value"] * 100 -).round(1) -print(f"\nUnderinsured BUILDING coverages (ITV < {int(ITV_THRESHOLD * 100)}%): {len(underinsured)}") -if not underinsured.empty: - print(underinsured.to_string(index=False)) - -print("\n" + "=" * 70) -print("Stage 2: Per-entity decline and marginal factors") -print("=" * 70) - - -def query_flag(rel, label): - """Return a list of insured-entity IDs that match a unary boolean Relationship.""" - df = model.where(rel()).select(InsuredEntity.id.alias("id")).to_df() - if df.empty: - return set(), label - return set(df["id"]), label - - -decline_factor_rels = [ - (InsuredEntity.has_restricted_industry, "restricted_industry"), - (InsuredEntity.has_insufficient_history, "insufficient_history"), - (InsuredEntity.has_unprotected_location, "unprotected_location"), - (InsuredEntity.has_old_construction_location, "old_construction_location"), -] -marginal_factor_rels = [ - (InsuredEntity.has_marginal_industry, "marginal_industry"), - (InsuredEntity.has_high_risk_occupancy_location, "high_risk_occupancy_location"), - (InsuredEntity.has_underinsured_coverage, "underinsured_coverage"), -] - -ie_df = model.select( - InsuredEntity.id.alias("id"), - InsuredEntity.name.alias("name"), - InsuredEntity.industry_code.alias("industry"), - InsuredEntity.years_in_business.alias("years"), -).to_df().sort_values("id").reset_index(drop=True) - -for rel, label in decline_factor_rels + marginal_factor_rels: - ids, _ = query_flag(rel, label) - ie_df[label] = ie_df["id"].isin(ids) - -print("\nFlag matrix (True = flag set):") -print(ie_df.to_string(index=False)) - -print("\n" + "=" * 70) -print("Stage 3: Risk-tier classification") -print("=" * 70) - -decline_ids = set( - model.where(RiskTier_Decline(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] -) -nonstandard_ids = set( - model.where(RiskTier_NonStandard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] -) -preferred_ids = set( - model.where(RiskTier_Preferred(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] -) -standard_ids = set( - model.where(RiskTier_Standard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] -) +if __name__ == "__main__": + # Deploy and wait for the first refresh. + model.deploy() + + print("=" * 70) + print("Stage 1: Location- and coverage-level eligibility flags") + print("=" * 70) + + poor_fp = model.where(Location.has_poor_fire_protection()).select( + Location.id.alias("location_id"), + Location.address.alias("address"), + Location.fire_protection_class.alias("fp_class"), + ).to_df() + print(f"\nLocations with poor fire protection (FP > {MAX_FP_CLASS_FOR_ELIGIBILITY}): {len(poor_fp)}") + if not poor_fp.empty: + print(poor_fp.to_string(index=False)) + + old_constr = model.where(Location.has_old_construction()).select( + Location.id.alias("location_id"), + Location.address.alias("address"), + Location.year_built.alias("year_built"), + ).to_df() + print(f"\nLocations with old construction (year_built < {MIN_YEAR_BUILT}): {len(old_constr)}") + if not old_constr.empty: + print(old_constr.to_string(index=False)) + + high_risk = model.where(Location.has_high_risk_occupancy()).select( + Location.id.alias("location_id"), + Location.address.alias("address"), + Location.occupancy_class.alias("occupancy"), + ).to_df() + print(f"\nLocations with high-risk occupancy: {len(high_risk)}") + if not high_risk.empty: + print(high_risk.to_string(index=False)) + + underinsured = model.where(Coverage.is_underinsured()).select( + Coverage.id.alias("coverage_id"), + Coverage.location.address.alias("location_address"), + Coverage.coverage_limit.alias("limit"), + Coverage.location.replacement_value.alias("replacement_value"), + ).to_df() + underinsured["itv_pct"] = ( + underinsured["limit"] / underinsured["replacement_value"] * 100 + ).round(1) + print(f"\nUnderinsured BUILDING coverages (ITV < {int(ITV_THRESHOLD * 100)}%): {len(underinsured)}") + if not underinsured.empty: + print(underinsured.to_string(index=False)) + + print("\n" + "=" * 70) + print("Stage 2: Per-entity decline and marginal factors") + print("=" * 70) + + + def query_flag(rel, label): + """Return a list of insured-entity IDs that match a unary boolean Relationship.""" + df = model.where(rel()).select(InsuredEntity.id.alias("id")).to_df() + if df.empty: + return set(), label + return set(df["id"]), label + + + decline_factor_rels = [ + (InsuredEntity.has_restricted_industry, "restricted_industry"), + (InsuredEntity.has_insufficient_history, "insufficient_history"), + (InsuredEntity.has_unprotected_location, "unprotected_location"), + (InsuredEntity.has_old_construction_location, "old_construction_location"), + ] + marginal_factor_rels = [ + (InsuredEntity.has_marginal_industry, "marginal_industry"), + (InsuredEntity.has_high_risk_occupancy_location, "high_risk_occupancy_location"), + (InsuredEntity.has_underinsured_coverage, "underinsured_coverage"), + ] + + ie_df = model.select( + InsuredEntity.id.alias("id"), + InsuredEntity.name.alias("name"), + InsuredEntity.industry_code.alias("industry"), + InsuredEntity.years_in_business.alias("years"), + ).to_df().sort_values("id").reset_index(drop=True) + + for rel, label in decline_factor_rels + marginal_factor_rels: + ids, _ = query_flag(rel, label) + ie_df[label] = ie_df["id"].isin(ids) + + print("\nFlag matrix (True = flag set):") + print(ie_df.to_string(index=False)) + + print("\n" + "=" * 70) + print("Stage 3: Risk-tier classification") + print("=" * 70) + + decline_ids = set( + model.where(RiskTier_Decline(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + nonstandard_ids = set( + model.where(RiskTier_NonStandard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + preferred_ids = set( + model.where(RiskTier_Preferred(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + standard_ids = set( + model.where(RiskTier_Standard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) -def classify(entity_id): - if entity_id in decline_ids: - return "DECLINE" - if entity_id in nonstandard_ids: - return "NON-STANDARD" - if entity_id in preferred_ids: - return "PREFERRED" - if entity_id in standard_ids: - return "STANDARD" - return "UNCLASSIFIED" - - -tier_df = ie_df[["id", "name", "industry", "years"]].copy() -tier_df["tier"] = tier_df["id"].apply(classify) -print("\nFinal risk tiers:") -print(tier_df.to_string(index=False)) - -# Sanity check: every entity must land in exactly one tier (mutual exclusivity). -print("\nTier counts:") -print(tier_df["tier"].value_counts().to_string()) - -unclassified = tier_df[tier_df["tier"] == "UNCLASSIFIED"] -if not unclassified.empty: - print(f"\nWARNING: {len(unclassified)} entities are unclassified — rule set is non-exhaustive.") - -# Verify mutual exclusivity: no entity in two tiers. -overlap = (decline_ids & nonstandard_ids) | (decline_ids & standard_ids) | (decline_ids & preferred_ids) \ - | (nonstandard_ids & standard_ids) | (nonstandard_ids & preferred_ids) | (standard_ids & preferred_ids) -if overlap: - print(f"\nWARNING: {len(overlap)} entities classified into multiple tiers: {sorted(overlap)}") - -# Aggregate premium by tier — the underwriting business view. -premium_df = model.where(Policy.insured_entity(InsuredEntity)).select( - InsuredEntity.id.alias("id"), - sum(Policy, Policy.total_premium).per(InsuredEntity).alias("premium"), -).to_df() -tier_df = tier_df.merge(premium_df, on="id") -print("\nPremium by tier:") -print(tier_df.groupby("tier")["premium"].agg(["count", "sum", "mean"]).to_string()) + def classify(entity_id): + if entity_id in decline_ids: + return "DECLINE" + if entity_id in nonstandard_ids: + return "NON-STANDARD" + if entity_id in preferred_ids: + return "PREFERRED" + if entity_id in standard_ids: + return "STANDARD" + return "UNCLASSIFIED" + + + tier_df = ie_df[["id", "name", "industry", "years"]].copy() + tier_df["tier"] = tier_df["id"].apply(classify) + print("\nFinal risk tiers:") + print(tier_df.to_string(index=False)) + + # Sanity check: every entity must land in exactly one tier (mutual exclusivity). + print("\nTier counts:") + print(tier_df["tier"].value_counts().to_string()) + + unclassified = tier_df[tier_df["tier"] == "UNCLASSIFIED"] + if not unclassified.empty: + print(f"\nWARNING: {len(unclassified)} entities are unclassified — rule set is non-exhaustive.") + + # Verify mutual exclusivity: no entity in two tiers. + overlap = (decline_ids & nonstandard_ids) | (decline_ids & standard_ids) | (decline_ids & preferred_ids) \ + | (nonstandard_ids & standard_ids) | (nonstandard_ids & preferred_ids) | (standard_ids & preferred_ids) + if overlap: + print(f"\nWARNING: {len(overlap)} entities classified into multiple tiers: {sorted(overlap)}") + + # Aggregate premium by tier — the underwriting business view. + premium_df = model.where(Policy.insured_entity(InsuredEntity)).select( + InsuredEntity.id.alias("id"), + sum(Policy, Policy.total_premium).per(InsuredEntity).alias("premium"), + ).to_df() + tier_df = tier_df.merge(premium_df, on="id") + print("\nPremium by tier:") + print(tier_df.groupby("tier")["premium"].agg(["count", "sum", "mean"]).to_string()) From 3d32af75d14fefc69478e187822fb8a36d95fc1e Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:20:29 -0800 Subject: [PATCH 020/162] shipment_compliance: run as a deployed model The script deploys the model, waits for the first refresh and prints the report from the deployed model. The README covers the deployment config and re-running. --- shipment_compliance/README.md | 24 ++++++-- shipment_compliance/shipment_compliance.py | 72 ++++++++++++---------- 2 files changed, 59 insertions(+), 37 deletions(-) diff --git a/shipment_compliance/README.md b/shipment_compliance/README.md index c467054a..6203abee 100644 --- a/shipment_compliance/README.md +++ b/shipment_compliance/README.md @@ -56,7 +56,8 @@ The four rules demonstrate different rule patterns: ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools - Python >= 3.10 @@ -85,15 +86,30 @@ The four rules demonstrate different rule patterns: python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + ```yaml + deployment: + role: + public_schema: .SHIPMENT_COMPLIANCE # published views and tables + identity: + internal_schema: .SHIPMENT_COMPLIANCE_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python shipment_compliance.py` stops with an error naming the missing setting. + 5. Run: ```bash python shipment_compliance.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing; it prints the report from the deployed model. A change to the model redeploys it. + + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (a few lines confirm a successful run): diff --git a/shipment_compliance/shipment_compliance.py b/shipment_compliance/shipment_compliance.py index 2854987e..9e84720f 100644 --- a/shipment_compliance/shipment_compliance.py +++ b/shipment_compliance/shipment_compliance.py @@ -20,7 +20,9 @@ No optimization solver is used. Rules are pure logic derivations. Run: - `python shipment_compliance.py` + `python shipment_compliance.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from + the deployed model. Output: Prints which entities match each compliance rule. @@ -194,37 +196,41 @@ class Priority(model.Enum): model.where(Demand.priority == Priority.URGENT).define(Demand.is_escalated()) # -------------------------------------------------- -# Query results +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("=== Rule 1: Late Shipments (delay_days > 0) ===\n") -model.select( - Shipment.id.alias("shipment_id"), - Shipment.sku.name.alias("sku"), - Shipment.supplier.name.alias("supplier"), - Shipment.delay_days.alias("delay_days"), -).where(Shipment.is_late()).inspect() - -print("\n=== Rule 2: At-Risk Shipments (undelivered + unreliable supplier) ===\n") -model.select( - Shipment.id.alias("shipment_id"), - Shipment.sku.name.alias("sku"), - Shipment.supplier.name.alias("supplier"), - Shipment.status.name.alias("status"), - Shipment.supplier.reliability_score.alias("reliability"), -).where(Shipment.is_at_risk()).inspect() - -print("\n=== Rule 3: Single-Sourced BOM Inputs (only 1 operation route) ===\n") -model.select( - BOM.id.alias("bom_id"), - BOM.input_sku.name.alias("input_sku"), - BOM.input_quantity.alias("qty"), -).where(BOM.is_single_sourced()).inspect() - -print("\n=== Rule 4: Escalated Demands (HIGH or URGENT priority) ===\n") -model.select( - Demand.id.alias("demand_id"), - Demand.sku.name.alias("sku"), - Demand.quantity.alias("quantity"), - Demand.priority.name.alias("priority"), -).where(Demand.is_escalated()).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh. + model.deploy() + + print("=== Rule 1: Late Shipments (delay_days > 0) ===\n") + model.select( + Shipment.id.alias("shipment_id"), + Shipment.sku.name.alias("sku"), + Shipment.supplier.name.alias("supplier"), + Shipment.delay_days.alias("delay_days"), + ).where(Shipment.is_late()).inspect() + + print("\n=== Rule 2: At-Risk Shipments (undelivered + unreliable supplier) ===\n") + model.select( + Shipment.id.alias("shipment_id"), + Shipment.sku.name.alias("sku"), + Shipment.supplier.name.alias("supplier"), + Shipment.status.name.alias("status"), + Shipment.supplier.reliability_score.alias("reliability"), + ).where(Shipment.is_at_risk()).inspect() + + print("\n=== Rule 3: Single-Sourced BOM Inputs (only 1 operation route) ===\n") + model.select( + BOM.id.alias("bom_id"), + BOM.input_sku.name.alias("input_sku"), + BOM.input_quantity.alias("qty"), + ).where(BOM.is_single_sourced()).inspect() + + print("\n=== Rule 4: Escalated Demands (HIGH or URGENT priority) ===\n") + model.select( + Demand.id.alias("demand_id"), + Demand.sku.name.alias("sku"), + Demand.quantity.alias("quantity"), + Demand.priority.name.alias("priority"), + ).where(Demand.is_escalated()).inspect() From 08b25fae9685d3a5d3a25fb179a5be68adb5382e Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:20:29 -0800 Subject: [PATCH 021/162] sample-template: run as a deployed model The runner deploys the model, waits for the first refresh and prints the item from the deployed model. The README covers the deployment config, re-running, and where a template declares its reasoning relative to the deploy. --- sample-template/README.md | 43 +++++++++++++++++++++++++++++++++---- sample-template/template.py | 6 +++++- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/sample-template/README.md b/sample-template/README.md index f548d0c3..7bdfb22e 100644 --- a/sample-template/README.md +++ b/sample-template/README.md @@ -47,7 +47,9 @@ layout without prescribing a business domain. ## Prerequisites - Python 3.10 or later. -- A RelationalAI account and configured credentials. +- A Snowflake account that has the RAI Native App installed. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This starter has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ## Quickstart @@ -69,12 +71,43 @@ layout without prescribing a business domain. python -m pip install . ``` +1. Configure the Snowflake connection and the deployment: run `rai init` to + create `raiconfig.yaml` with your Snowflake connection, set the connection + `role` to the model role, then add a `deployment` section. Give each template + its own schema pair so two deployed templates never share one; the deploy + creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .SAMPLE_TEMPLATE # published views and tables + identity: + internal_schema: .SAMPLE_TEMPLATE_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + ``` + + Without the `deployment` section, `python template.py` stops with an error + naming the missing setting. + 1. Run the renamed entrypoint: ```bash python template.py ``` + The script deploys the model, waits for its first refresh, and prints the + report from the deployed model. `rai models deploy` deploys without printing + the report. + + Running the script again without changing the model redeploys nothing; it + prints the item from the deployed model. A change to the model redeploys it. + + To remove the deployment, run `rai models teardown --force` from + this directory; it keeps the two schemas. + The example prints the item loaded by `model/source.py`. Your template should document a similarly small success signal. @@ -96,8 +129,9 @@ do not duplicate them in `template-docs.yaml`. ## How it works `model/schema.py` creates the model and stable vocabulary. Importing `model` -loads the single source in `model/source.py`. The runner then selects the -modeled records and prints them. +loads the single source in `model/source.py`. The runner then deploys the model, +waits for its first refresh, and selects the modeled records from the deployed +model and prints them. ```text source records -> model/source.py -> model/schema.py -> runner -> output @@ -108,7 +142,8 @@ source records -> model/source.py -> model/schema.py -> runner -> output 1. Replace `Item` with the concepts and relationships for the business problem. 1. Replace the in-memory record with local or remote source mappings. -1. Add reasoning, solves, queries, and reporting to the runner. +1. Declare reasoning and solves at the top level of the runner, before the + deploy, and put queries and reporting after `model.deploy()`. 1. Update every sidecar symbol, source path, walkthrough focus, and sample column after changing the code. diff --git a/sample-template/template.py b/sample-template/template.py index d17f2807..4e2a2764 100644 --- a/sample-template/template.py +++ b/sample-template/template.py @@ -7,7 +7,8 @@ - Keep queries, reasoning, solves, and reporting in the runner. Run: - `python template.py` + `python template.py` deploys the model to the schema in raiconfig.yaml, waits + for its first refresh, and prints the item from the deployed model. Output: A small table containing the bundled example item. @@ -23,6 +24,9 @@ def inspect_items(): def main() -> None: + # Deploy and wait for the first refresh. + model.deploy() + print(inspect_items().to_df()) From be86691d9546e6c68f616dcaeb8ab84c8f60b407 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:21:49 -0800 Subject: [PATCH 022/162] entity_resolution: run as a deployed model The graph, rules and reinsurance solve are declared with the model and run on each refresh of the deployed model; the report reads the cession plan from the deployed model. The solve status is published through a relationship, and the README and docs sidecar cover the deployment config and the configured solve. --- entity_resolution/README.md | 56 +++- entity_resolution/entity_resolution.py | 373 +++++++++++++------------ entity_resolution/template-docs.yaml | 15 +- 3 files changed, 255 insertions(+), 189 deletions(-) diff --git a/entity_resolution/README.md b/entity_resolution/README.md index b3369271..edb31b2a 100644 --- a/entity_resolution/README.md +++ b/entity_resolution/README.md @@ -33,8 +33,15 @@ constraints for your workflow. ## Quickstart Before you start, install Python 3.10 or later and get access to a Snowflake -account with the RAI Native App. Graph and Prescriptive reasoning are in Public -Preview; ask your RelationalAI support representative to enable Prescriptive +account with the RAI Native App. An account administrator must first do the +one-time model deployment setup on that account: follow +[Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) +through **Governed setup**, **PAT authentication**, and steps 1-2 of +**Additional setup for the prescriptive reasoner** (Quickstart step 4 below +covers this template's own schema). It gives you a model role, a programmatic +access token (PAT) stored as a secret, an external access integration, a PAT +user, and a warehouse; your Snowflake user must hold the model role. Graph and +Prescriptive reasoning are in Public Preview; ask your RelationalAI support representative to enable Prescriptive reasoning. Preview features are for evaluation and testing, not production applications. The template pins `relationalai==2.1.1` in `pyproject.toml` and uses HiGHS without a separate solver license. @@ -70,7 +77,48 @@ Use this sequence to run the bundled example: 4. **Configure your project** - Use the configuration builder in [Start building with PyRel](https://docs.relational.ai/get-started/start-building-with-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + Use the configuration builder in [Start building with PyRel](https://docs.relational.ai/get-started/start-building-with-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. Set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: entity_resolution_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .ENTITY_RESOLUTION # published views and tables + identity: + internal_schema: .ENTITY_RESOLUTION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + + Without the `deployment` section, `python entity_resolution.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .ENTITY_RESOLUTION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + ``` + + The deploy creates the public schema `.ENTITY_RESOLUTION` if it doesn't exist. 5. **Run the template** @@ -80,6 +128,8 @@ Use this sequence to run the bundled example: python entity_resolution.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + ```text Auto-resolved 51 records into 31 insured parties. Households over the limit after RESOLUTION: 4 diff --git a/entity_resolution/entity_resolution.py b/entity_resolution/entity_resolution.py index f9f61d28..59e78e7e 100644 --- a/entity_resolution/entity_resolution.py +++ b/entity_resolution/entity_resolution.py @@ -21,7 +21,9 @@ breach it still hides), the cession plan, and pairwise precision / recall / F1. Run: - `python entity_resolution.py` + `python entity_resolution.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from + the deployed model. Output: Prints blocking/banding stats, graph size, resolved-party and golden-record @@ -44,7 +46,7 @@ record_ids, review_df, ) -from relationalai.semantics import Float +from relationalai.semantics import Float, String from relationalai.semantics.reasoners.graph import Graph from relationalai.semantics.reasoners.prescriptive import Problem from relationalai.semantics.std import aggregates @@ -55,25 +57,8 @@ REINSURANCE_RATE = 0.12 REINSURANCE_BUDGET = 120_000.0 -n_records = len(record_ids) -all_pairs = n_records * (n_records - 1) // 2 -print("=" * 64) -print("Blocking & scoring") -print("=" * 64) -print(f"Records: {n_records}") -print(f"All possible pairs: {all_pairs}") -print( - f"Candidate pairs: {candidate_count} " - f"({100 * (1 - candidate_count / all_pairs):.1f}% fewer comparisons)" -) -print(f"Auto-merge matches: {len(matches_df)} (score >= {AUTO_MERGE})") -print( - f"Held for review: {len(review_df)} " - f"([{REVIEW_FLOOR}, {AUTO_MERGE}))" -) - -def cluster_accepted_matches() -> tuple[int, int]: +def cluster_accepted_matches(): """Cluster records connected by accepted match edges.""" graph = Graph(model, directed=False, weighted=False, node_concept=Record) @@ -90,9 +75,7 @@ def cluster_accepted_matches() -> tuple[int, int]: Record.entity_key(rep_node.record_id) ) - n_nodes = int(model.select(graph.num_nodes().alias("n")).to_df()["n"].iloc[0]) - n_edges = int(model.select(graph.num_edges().alias("n")).to_df()["n"].iloc[0]) - return n_nodes, n_edges + return graph.num_nodes(), graph.num_edges() def derive_exposure_and_breaches() -> None: @@ -133,7 +116,7 @@ def derive_exposure_and_breaches() -> None: def choose_reinsurance_cessions(): - """Solve the reinsurance-budget knapsack for breached parties.""" + """Declare the reinsurance-budget knapsack for breached parties.""" problem = Problem(model, Float) problem.solve_for( ResolvedParty.cede, @@ -155,173 +138,205 @@ def choose_reinsurance_cessions(): problem.maximize( aggregates.sum(ResolvedParty.excess * ResolvedParty.cede) ) - problem.solve("highs", time_limit_sec=60) - return problem.solve_info() + problem.configure("highs", time_limit_sec=60) + return problem -n_nodes, n_edges = cluster_accepted_matches() -print(f"Graph: {n_nodes} nodes, {n_edges} match edges") - +num_nodes, num_edges = cluster_accepted_matches() derive_exposure_and_breaches() -si = choose_reinsurance_cessions() - -resolved = model.select( - Record.record_id.alias("record_id"), - Record.entity_key.alias("entity_id"), - Record.source_system.alias("source_system"), - Record.full_name.alias("full_name"), - Record.created_at.alias("created_at"), - Record.coverage_amount.alias("coverage_amount"), -).to_df() -resolved["entity_id"] = resolved["entity_id"].astype(str) - -n_entities = resolved["entity_id"].nunique() -print("\n" + "=" * 64) -print("Resolved parties") -print("=" * 64) -print(f"Auto-resolved {n_records} records into {n_entities} insured parties.") - -resolved = resolved.sort_values(["entity_id", "created_at"]) -golden = resolved.groupby("entity_id").tail(1).set_index("entity_id") -sizes = resolved.groupby("entity_id").size() -print("\nMulti-record parties (golden record in CAPS):") -for entity_id in sizes[sizes > 1].index: - golden_record = golden.loc[entity_id] - members = resolved[resolved["entity_id"] == entity_id] - exposure = members["coverage_amount"].sum() +problem = choose_reinsurance_cessions() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"reinsurance solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + n_records = len(record_ids) + all_pairs = n_records * (n_records - 1) // 2 + print("=" * 64) + print("Blocking & scoring") + print("=" * 64) + print(f"Records: {n_records}") + print(f"All possible pairs: {all_pairs}") + print( + f"Candidate pairs: {candidate_count} " + f"({100 * (1 - candidate_count / all_pairs):.1f}% fewer comparisons)" + ) + print(f"Auto-merge matches: {len(matches_df)} (score >= {AUTO_MERGE})") print( - f" {golden_record['full_name'].upper():<22} " - f"{len(members)} policies total exposure ${exposure:>12,.0f}" + f"Held for review: {len(review_df)} " + f"([{REVIEW_FLOOR}, {AUTO_MERGE}))" ) -record_level_breaches = int( - (resolved["coverage_amount"] > ACCUMULATION_LIMIT).sum() -) -over = ( - model.select( - ResolvedParty.key.alias("id"), - ResolvedParty.total_exposure.alias("exposure"), + n_nodes = int(model.select(num_nodes.alias("n")).to_df()["n"].iloc[0]) + n_edges = int(model.select(num_edges.alias("n")).to_df()["n"].iloc[0]) + print(f"Graph: {n_nodes} nodes, {n_edges} match edges") + + resolved = model.select( + Record.record_id.alias("record_id"), + Record.entity_key.alias("entity_id"), + Record.source_system.alias("source_system"), + Record.full_name.alias("full_name"), + Record.created_at.alias("created_at"), + Record.coverage_amount.alias("coverage_amount"), + ).to_df() + resolved["entity_id"] = resolved["entity_id"].astype(str) + + n_entities = resolved["entity_id"].nunique() + print("\n" + "=" * 64) + print("Resolved parties") + print("=" * 64) + print(f"Auto-resolved {n_records} records into {n_entities} insured parties.") + + resolved = resolved.sort_values(["entity_id", "created_at"]) + golden = resolved.groupby("entity_id").tail(1).set_index("entity_id") + sizes = resolved.groupby("entity_id").size() + print("\nMulti-record parties (golden record in CAPS):") + for entity_id in sizes[sizes > 1].index: + golden_record = golden.loc[entity_id] + members = resolved[resolved["entity_id"] == entity_id] + exposure = members["coverage_amount"].sum() + print( + f" {golden_record['full_name'].upper():<22} " + f"{len(members)} policies total exposure ${exposure:>12,.0f}" + ) + + record_level_breaches = int( + (resolved["coverage_amount"] > ACCUMULATION_LIMIT).sum() ) - .where(ResolvedParty.is_over_limit()) - .to_df() -) -print("\n" + "=" * 64) -print(f"Accumulation control (limit ${ACCUMULATION_LIMIT:,.0f})") -print("=" * 64) -print( - "Policies over the limit at the RECORD level: " - f"{record_level_breaches}" -) -print(f"Households over the limit after RESOLUTION: {len(over)}") + over = ( + model.select( + ResolvedParty.key.alias("id"), + ResolvedParty.total_exposure.alias("exposure"), + ) + .where(ResolvedParty.is_over_limit()) + .to_df() + ) + print("\n" + "=" * 64) + print(f"Accumulation control (limit ${ACCUMULATION_LIMIT:,.0f})") + print("=" * 64) + print( + "Policies over the limit at the RECORD level: " + f"{record_level_breaches}" + ) + print(f"Households over the limit after RESOLUTION: {len(over)}") -print( - f"\nReinsurance cession plan (status {si.termination_status}, " - f"budget ${REINSURANCE_BUDGET:,.0f} at " - f"{REINSURANCE_RATE:.0%} rate on line):" -) -ceded = ( - model.select( - ResolvedParty.key.alias("id"), - ResolvedParty.total_exposure.alias("exposure"), - ResolvedParty.excess.alias("excess"), - ResolvedParty.premium.alias("premium"), - ResolvedParty.cede.alias("cede"), + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + print( + f"\nReinsurance cession plan (status {', '.join(status)}, " + f"budget ${REINSURANCE_BUDGET:,.0f} at " + f"{REINSURANCE_RATE:.0%} rate on line):" ) - .where(ResolvedParty.is_over_limit()) - .to_df() -) -ceded["id"] = ceded["id"].astype(str) -name_by_entity = golden["full_name"].to_dict() -ceded["name"] = ceded["id"].map(name_by_entity).fillna("(party)") -ceded = ceded.sort_values("excess", ascending=False) -total_premium = total_ceded = 0.0 -for _, row in ceded.iterrows(): - chosen = row["cede"] > 0.5 - mark = "CEDE " if chosen else "keep " - if chosen: - total_premium += row["premium"] - total_ceded += row["excess"] + ceded = ( + model.select( + ResolvedParty.key.alias("id"), + ResolvedParty.total_exposure.alias("exposure"), + ResolvedParty.excess.alias("excess"), + ResolvedParty.premium.alias("premium"), + ResolvedParty.cede.alias("cede"), + ) + .where(ResolvedParty.is_over_limit()) + .to_df() + ) + ceded["id"] = ceded["id"].astype(str) + name_by_entity = golden["full_name"].to_dict() + ceded["name"] = ceded["id"].map(name_by_entity).fillna("(party)") + ceded = ceded.sort_values("excess", ascending=False) + total_premium = total_ceded = 0.0 + for _, row in ceded.iterrows(): + chosen = row["cede"] > 0.5 + mark = "CEDE " if chosen else "keep " + if chosen: + total_premium += row["premium"] + total_ceded += row["excess"] + print( + f" {mark} {row['name']:<22} exposure " + f"${row['exposure']:>11,.0f} excess ${row['excess']:>9,.0f} " + f"premium ${row['premium']:>8,.0f}" + ) print( - f" {mark} {row['name']:<22} exposure " - f"${row['exposure']:>11,.0f} excess ${row['excess']:>9,.0f} " - f"premium ${row['premium']:>8,.0f}" + f" -> ceded ${total_ceded:,.0f} of excess exposure for " + f"${total_premium:,.0f} premium (of ${REINSURANCE_BUDGET:,.0f})" ) -print( - f" -> ceded ${total_ceded:,.0f} of excess exposure for " - f"${total_premium:,.0f} premium (of ${REINSURANCE_BUDGET:,.0f})" -) -auto_entity = dict(zip(resolved["record_id"], resolved["entity_id"])) -coverage_by_record = dict( - zip(resolved["record_id"], resolved["coverage_amount"]) -) -confirmed = {record_id: auto_entity[record_id] for record_id in record_ids} -for _, row in review_df.iterrows(): - rec_a, rec_b = int(row["rec_a"]), int(row["rec_b"]) - target, source = confirmed[rec_a], confirmed[rec_b] + auto_entity = dict(zip(resolved["record_id"], resolved["entity_id"])) + coverage_by_record = dict( + zip(resolved["record_id"], resolved["coverage_amount"]) + ) + confirmed = {record_id: auto_entity[record_id] for record_id in record_ids} + for _, row in review_df.iterrows(): + rec_a, rec_b = int(row["rec_a"]), int(row["rec_b"]) + target, source = confirmed[rec_a], confirmed[rec_b] + for record_id in record_ids: + if confirmed[record_id] == source: + confirmed[record_id] = target + confirmed_exposure: dict[str, float] = {} for record_id in record_ids: - if confirmed[record_id] == source: - confirmed[record_id] = target -confirmed_exposure: dict[str, float] = {} -for record_id in record_ids: - party_id = confirmed[record_id] - confirmed_exposure[party_id] = ( - confirmed_exposure.get(party_id, 0.0) - + coverage_by_record[record_id] + party_id = confirmed[record_id] + confirmed_exposure[party_id] = ( + confirmed_exposure.get(party_id, 0.0) + + coverage_by_record[record_id] + ) + confirmed_breaches = sum( + 1 for exposure in confirmed_exposure.values() + if exposure > ACCUMULATION_LIMIT + ) + print("\n" + "=" * 64) + print("Review queue") + print("=" * 64) + print(f"Pairs held for steward review: {len(review_df)}") + print( + "Breaches surfaced only if the review queue is confirmed: " + f"{confirmed_breaches - len(over)} " + f"(resolution would then show {confirmed_breaches} breached " + f"households, not {len(over)})" ) -confirmed_breaches = sum( - 1 for exposure in confirmed_exposure.values() - if exposure > ACCUMULATION_LIMIT -) -print("\n" + "=" * 64) -print("Review queue") -print("=" * 64) -print(f"Pairs held for steward review: {len(review_df)}") -print( - "Breaches surfaced only if the review queue is confirmed: " - f"{confirmed_breaches - len(over)} " - f"(resolution would then show {confirmed_breaches} breached " - f"households, not {len(over)})" -) -truth_df = load_ground_truth() -truth_map = dict( - zip(truth_df["record_id"], truth_df["true_entity_id"]) -) -true_pairs = { - (rec_a, rec_b) - for rec_a, rec_b in combinations(record_ids, 2) - if truth_map[rec_a] == truth_map[rec_b] -} -predicted_pairs = { - (rec_a, rec_b) - for rec_a, rec_b in combinations(record_ids, 2) - if auto_entity[rec_a] == auto_entity[rec_b] -} -true_positives = len(predicted_pairs & true_pairs) -false_positives = len(predicted_pairs - true_pairs) -false_negatives = len(true_pairs - predicted_pairs) -precision = ( - true_positives / (true_positives + false_positives) - if true_positives + false_positives - else 1.0 -) -recall = ( - true_positives / (true_positives + false_negatives) - if true_positives + false_negatives - else 1.0 -) -f1 = ( - 2 * precision * recall / (precision + recall) - if precision + recall - else 0.0 -) -print("\n" + "=" * 64) -print("Evaluation vs ground truth (pairwise, auto-resolution)") -print("=" * 64) -print(f" precision: {precision:.3f} recall: {recall:.3f} f1: {f1:.3f}") -print( - f" (true positives={true_positives}, false positives={false_positives}, " - f"false negatives={false_negatives}; the {false_negatives} miss is the " - "held review pair)" -) + truth_df = load_ground_truth() + truth_map = dict( + zip(truth_df["record_id"], truth_df["true_entity_id"]) + ) + true_pairs = { + (rec_a, rec_b) + for rec_a, rec_b in combinations(record_ids, 2) + if truth_map[rec_a] == truth_map[rec_b] + } + predicted_pairs = { + (rec_a, rec_b) + for rec_a, rec_b in combinations(record_ids, 2) + if auto_entity[rec_a] == auto_entity[rec_b] + } + true_positives = len(predicted_pairs & true_pairs) + false_positives = len(predicted_pairs - true_pairs) + false_negatives = len(true_pairs - predicted_pairs) + precision = ( + true_positives / (true_positives + false_positives) + if true_positives + false_positives + else 1.0 + ) + recall = ( + true_positives / (true_positives + false_negatives) + if true_positives + false_negatives + else 1.0 + ) + f1 = ( + 2 * precision * recall / (precision + recall) + if precision + recall + else 0.0 + ) + print("\n" + "=" * 64) + print("Evaluation vs ground truth (pairwise, auto-resolution)") + print("=" * 64) + print(f" precision: {precision:.3f} recall: {recall:.3f} f1: {f1:.3f}") + print( + f" (true positives={true_positives}, false positives={false_positives}, " + f"false negatives={false_negatives}; the {false_negatives} miss is the " + "held review pair)" + ) diff --git a/entity_resolution/template-docs.yaml b/entity_resolution/template-docs.yaml index 2ba78bdf..788e8d65 100644 --- a/entity_resolution/template-docs.yaml +++ b/entity_resolution/template-docs.yaml @@ -203,7 +203,7 @@ guided_walkthrough: focus: file: entity_resolution.py anchors: - - start: 'def cluster_accepted_matches() -> tuple[int, int]:' + - start: 'def cluster_accepted_matches():' end: Record.entity_key(rep_node.record_id) end_offset: 1 references: @@ -286,13 +286,14 @@ guided_walkthrough: party a yes-or-no choice to transfer its excess. problem.satisfy(...) adds a constraint that caps the total premium paid to the reinsurer at REINSURANCE_BUDGET. problem.maximize(...) aims to transfer the most - excess risk, and problem.solve(...) runs the solver, though a small - budget may leave some excess with the insurer. + excess risk, and problem.configure(...) chooses the solver and its time + limit. Each refresh of the deployed model runs that solve, though a + small budget may leave some excess with the insurer. focus: file: entity_resolution.py anchors: - start: 'def choose_reinsurance_cessions():' - end: return problem.solve_info() + end: return problem references: Problem(model, Float): anchor: @@ -317,10 +318,10 @@ guided_walkthrough: start: problem.maximize( end: aggregates.sum(ResolvedParty.excess * ResolvedParty.cede) end_offset: 1 - problem.solve(...): + problem.configure(...): anchor: - start: problem.solve("highs", time_limit_sec=60) - end: problem.solve("highs", time_limit_sec=60) + start: problem.configure("highs", time_limit_sec=60) + end: problem.configure("highs", time_limit_sec=60) links: - title: Add constraints href: /build/guides/reasoning/prescriptive/decision-problems/constraints/ From 8b3a2c3c0f946f9eb3743c5d3a438c1cec63e4b8 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:28:36 -0800 Subject: [PATCH 023/162] cybersecurity-attack-paths: run as a deployed model The script deploys the model, waits for the first refresh and prints the report from the deployed model. Asset.on_attack_path is now a rule over the kill-chain paths, declared as a Relationship because a Property defined from a path rule fails deployment planning. The README covers the deployment config. --- cybersecurity-attack-paths/README.md | 23 +- .../cybersecurity_attack_paths.py | 300 ++++++++++-------- 2 files changed, 178 insertions(+), 145 deletions(-) diff --git a/cybersecurity-attack-paths/README.md b/cybersecurity-attack-paths/README.md index 34086951..189792f0 100644 --- a/cybersecurity-attack-paths/README.md +++ b/cybersecurity-attack-paths/README.md @@ -48,7 +48,8 @@ This template enumerates multi-step attack paths across an enterprise asset grap ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -83,18 +84,32 @@ This template enumerates multi-step attack paths across an enterprise asset grap python -m pip install . ``` -4. **Configure Snowflake connection and RAI profile** +4. **Configure the Snowflake connection and the deployment** - ```bash - rai init + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .CYBERSECURITY_ATTACK_PATHS # published views and tables + identity: + internal_schema: .CYBERSECURITY_ATTACK_PATHS_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python cybersecurity_attack_paths.py` stops with an error naming the missing setting. + 5. **Run the template** ```bash python cybersecurity_attack_paths.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + Expected output starts with the kill-chains that reach a crown jewel: ```text diff --git a/cybersecurity-attack-paths/cybersecurity_attack_paths.py b/cybersecurity-attack-paths/cybersecurity_attack_paths.py index b1eeb353..02032513 100644 --- a/cybersecurity-attack-paths/cybersecurity_attack_paths.py +++ b/cybersecurity-attack-paths/cybersecurity_attack_paths.py @@ -20,7 +20,9 @@ crown-jewel-reaching chain. Run: - /opt/homebrew/bin/python3.11 cybersecurity_attack_paths.py + `python cybersecurity_attack_paths.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from the + deployed model. Output: Prints the kill-chain attack paths into crown jewels (with the technique at @@ -118,157 +120,173 @@ def load_csv(filename): # an internet-facing asset and dst to a crown jewel pins the threat model: an # externally reachable foothold that ends on a high-value asset. -print("=== Cybersecurity Attack Paths: kill-chain into crown jewels ===") - a, b, c, dst = Asset.ref(), Asset.ref(), Asset.ref(), Asset.ref() kill = model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, MAX_PIVOTS), dst).all().paths() -kill_df = ( - model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - ) - .select( - kill.alias("path_id"), - kill.nodes["index"].alias("step"), - Asset(kill.nodes).id.alias("asset_id"), - Asset(kill.nodes).name.alias("asset_name"), - ) - .to_df() -) -# Technique at each hop (the relationship label), as a separate projection. -hop_df = ( - model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - ) - .select( - kill.alias("path_id"), - kill.relationships["index"].alias("hop"), - kill.relationships["relationship"].alias("technique"), - ) - .to_df() -) - -kill_df["step"] = kill_df["step"].astype(int) -kill_df = kill_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) -hop_df["hop"] = hop_df["hop"].astype(int) -hop_df = hop_df.drop_duplicates(["path_id", "hop"]).sort_values(["path_id", "hop"]) - -# Reassemble each kill-chain: ordered asset names + the technique used at each hop. -# The technique is the relationship label as-is -- the dotted relationship name, -# e.g. "Asset.exploit_to". -chains = [] -for pid, g in kill_df.groupby("path_id"): - assets = list(g.sort_values("step")["asset_name"]) - techs = list(hop_df[hop_df["path_id"] == pid].sort_values("hop")["technique"]) - labelled = assets[0] - for nm, tech in zip(assets[1:], techs): - labelled += f" --[{tech}]--> {nm}" - chains.append({"path_id": pid, "hops": len(assets) - 1, "asset_ids": tuple( - g.sort_values("step")["asset_id"]), "labelled": labelled}) - -print(f"\n{len(chains)} kill-chain attack path(s) reach a crown jewel " - f"(exploit -> cred -> 1-{MAX_PIVOTS} pivots, from an internet-facing asset):") -for ch in sorted(chains, key=lambda c: c["hops"]): - print(f" [{ch['hops']} hops] {ch['labelled']}") - -# -------------------------------------------------- -# Point query: every route between one entry and one crown jewel -# -------------------------------------------------- -# Pin both endpoints by id and enumerate all simple routes between them over the -# technique-agnostic can_reach edge (any technique, 1..MAX_ROUTE_HOPS). This is the -# >=1.15-era native point query (still current) -- src/dst unified to specific assets inside .all().paths(). +# The point query's pattern: routes over the technique-agnostic can_reach edge from +# a pinned entry asset to a pinned crown jewel, 1..MAX_ROUTE_HOPS hops. src_pt, dst_pt = Asset.ref(), Asset.ref() route = model.path(src_pt.can_reach.repeat(1, MAX_ROUTE_HOPS), dst_pt).all().paths() -# Keep only SIMPLE routes (no asset revisited). .all().paths() has WALK semantics and -# no native simple-path mode, so drop node-repeating walks with a negated existential -# over positions -- expressed natively in the route's where() (no pandas). -i, j = Integer.ref(), Integer.ref() -route_df = ( - model.where( - route, - src_pt.id == ENTRY_ASSET, - dst_pt.id == TARGET_ASSET, - not_(where(i < j, route.nodes(i) == route.nodes(j))), - ) - .select( - route.alias("path_id"), - route.nodes["index"].alias("step"), - Asset(route.nodes).id.alias("asset_id"), - ) - .to_df() -) -route_df["step"] = route_df["step"].astype(int) -route_df = route_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) -routes = sorted( - {tuple(g.sort_values("step")["asset_id"]) for _, g in route_df.groupby("path_id")}, - key=len, -) -print(f"\n=== Point query: all routes from {ENTRY_ASSET} to {TARGET_ASSET} " - f"(any technique, <= {MAX_ROUTE_HOPS} hops, simple) ===") -print(f" {len(routes)} route(s):") -for seq in routes: - print(" " + " -> ".join(seq)) +# Asset.on_attack_path marks the assets that lie on a crown-jewel kill-chain, so a +# downstream query can pull them without re-enumerating paths. It is a +# Relationship, not a Property: a Property defined from a path fails deployment +# planning. +Asset.on_attack_path = model.Relationship(f"{Asset} on attack path {String:on_attack_path}") +on_path, pos = Asset.ref(), Integer.ref() +model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + on_path == kill.nodes(pos), +).define(on_path.on_attack_path("yes")) # -------------------------------------------------- -# Rank kill-chains by total asset exposure +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -# The riskiest chain is the one whose assets carry the most exposure overall -- the -# first remediation target. Exposure is a per-asset score (a stand-in for a CVSS / -# attack-surface metric) summed ALONG each kill-chain IN PYREL: -# aggregates.sum(Asset(kill.nodes).exposure_score).per(kill) sums the per-node value -# over each path's node sequence -- the core paths-only aggregation, reusing the same -# kill pattern and its internet-facing/crown-jewel endpoint filters. This is a -# position-sum (every node position, with repetition); it equals the distinct-asset -# sum because each kill-chain here is a simple path (no asset revisited). -totals = ( - model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - total := aggs.sum(Asset(kill.nodes).exposure_score).per(kill), + +if __name__ == "__main__": + # Deploy and wait for the first refresh. + model.deploy() + + print("=== Cybersecurity Attack Paths: kill-chain into crown jewels ===") + kill_df = ( + model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + ) + .select( + kill.alias("path_id"), + kill.nodes["index"].alias("step"), + Asset(kill.nodes).id.alias("asset_id"), + Asset(kill.nodes).name.alias("asset_name"), + ) + .to_df() + ) + # Technique at each hop (the relationship label), as a separate projection. + hop_df = ( + model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + ) + .select( + kill.alias("path_id"), + kill.relationships["index"].alias("hop"), + kill.relationships["relationship"].alias("technique"), + ) + .to_df() ) - .select(kill.alias("path_id"), total.alias("total_exposure")) - .to_df() -) -# Join total_exposure onto the assembled chains by path id (label string-join is pandas). -exposure_by_path = dict( - totals[["path_id", "total_exposure"]].itertuples(index=False, name=None) -) -for ch in chains: - ch["total_exposure"] = int(exposure_by_path[ch["path_id"]]) -print("\n=== Kill-chains ranked by total asset exposure (highest = remediate first) ===") -for ch in sorted(chains, key=lambda c: c["total_exposure"], reverse=True): - print(f" exposure {ch['total_exposure']:>3} ({ch['hops']} hops) {ch['labelled']}") + kill_df["step"] = kill_df["step"].astype(int) + kill_df = kill_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) + hop_df["hop"] = hop_df["hop"].astype(int) + hop_df = hop_df.drop_duplicates(["path_id", "hop"]).sort_values(["path_id", "hop"]) + + # Reassemble each kill-chain: ordered asset names + the technique used at each hop. + # The technique is the relationship label as-is -- the dotted relationship name, + # e.g. "Asset.exploit_to". + chains = [] + for pid, g in kill_df.groupby("path_id"): + assets = list(g.sort_values("step")["asset_name"]) + techs = list(hop_df[hop_df["path_id"] == pid].sort_values("hop")["technique"]) + labelled = assets[0] + for nm, tech in zip(assets[1:], techs): + labelled += f" --[{tech}]--> {nm}" + chains.append({"path_id": pid, "hops": len(assets) - 1, "asset_ids": tuple( + g.sort_values("step")["asset_id"]), "labelled": labelled}) + + print(f"\n{len(chains)} kill-chain attack path(s) reach a crown jewel " + f"(exploit -> cred -> 1-{MAX_PIVOTS} pivots, from an internet-facing asset):") + for ch in sorted(chains, key=lambda c: c["hops"]): + print(f" [{ch['hops']} hops] {ch['labelled']}") + + # -------------------------------------------------- + # Point query: every route between one entry and one crown jewel + # -------------------------------------------------- + # Pin both endpoints by id and enumerate all simple routes between them over the + # technique-agnostic can_reach edge (any technique, 1..MAX_ROUTE_HOPS). This is the + # >=1.15-era native point query (still current) -- src/dst unified to specific assets inside .all().paths(). + + i, j = Integer.ref(), Integer.ref() + # Keep only SIMPLE routes (no asset revisited). .all().paths() has WALK semantics and + # no native simple-path mode, so drop node-repeating walks with a negated existential + # over positions -- expressed natively in the route's where() (no pandas). + route_df = ( + model.where( + route, + src_pt.id == ENTRY_ASSET, + dst_pt.id == TARGET_ASSET, + not_(where(i < j, route.nodes(i) == route.nodes(j))), + ) + .select( + route.alias("path_id"), + route.nodes["index"].alias("step"), + Asset(route.nodes).id.alias("asset_id"), + ) + .to_df() + ) + route_df["step"] = route_df["step"].astype(int) + route_df = route_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) + routes = sorted( + {tuple(g.sort_values("step")["asset_id"]) for _, g in route_df.groupby("path_id")}, + key=len, + ) -# -------------------------------------------------- -# Persist: flag assets that lie on a crown-jewel attack path -# -------------------------------------------------- -# Bind the result back onto the ontology so a downstream query can pull the assets -# on a kill-chain without re-enumerating paths. - -Asset.on_attack_path = model.Property(f"{Asset} on attack path {String:on_attack_path}") -total_assets = len(model.select(Asset.id.alias("id")).to_df()) -on_path_ids = sorted({i for ch in chains for i in ch["asset_ids"]}) -flag_data = model.data(pd.DataFrame({"id": on_path_ids})) -fa = Asset.ref() -model.where(fa.id == flag_data["id"]).define(fa.on_attack_path("yes")) - -x = Asset.ref() -flagged = ( - model.where( - x.on_attack_path == "yes", + print(f"\n=== Point query: all routes from {ENTRY_ASSET} to {TARGET_ASSET} " + f"(any technique, <= {MAX_ROUTE_HOPS} hops, simple) ===") + print(f" {len(routes)} route(s):") + for seq in routes: + print(" " + " -> ".join(seq)) + + # -------------------------------------------------- + # Rank kill-chains by total asset exposure + # -------------------------------------------------- + # The riskiest chain is the one whose assets carry the most exposure overall -- the + # first remediation target. Exposure is a per-asset score (a stand-in for a CVSS / + # attack-surface metric) summed ALONG each kill-chain IN PYREL: + # aggregates.sum(Asset(kill.nodes).exposure_score).per(kill) sums the per-node value + # over each path's node sequence -- the core paths-only aggregation, reusing the same + # kill pattern and its internet-facing/crown-jewel endpoint filters. This is a + # position-sum (every node position, with repetition); it equals the distinct-asset + # sum because each kill-chain here is a simple path (no asset revisited). + totals = ( + model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + total := aggs.sum(Asset(kill.nodes).exposure_score).per(kill), + ) + .select(kill.alias("path_id"), total.alias("total_exposure")) + .to_df() + ) + # Join total_exposure onto the assembled chains by path id (label string-join is pandas). + exposure_by_path = dict( + totals[["path_id", "total_exposure"]].itertuples(index=False, name=None) + ) + for ch in chains: + ch["total_exposure"] = int(exposure_by_path[ch["path_id"]]) + + print("\n=== Kill-chains ranked by total asset exposure (highest = remediate first) ===") + for ch in sorted(chains, key=lambda c: c["total_exposure"], reverse=True): + print(f" exposure {ch['total_exposure']:>3} ({ch['hops']} hops) {ch['labelled']}") + + # -------------------------------------------------- + # Assets on a crown-jewel attack path + # -------------------------------------------------- + + total_assets = len(model.select(Asset.id.alias("id")).to_df()) + x = Asset.ref() + flagged = ( + model.where( + x.on_attack_path == "yes", + ) + .select(x.id.alias("id"), x.name.alias("name"), x.zone.alias("zone")) + .to_df() ) - .select(x.id.alias("id"), x.name.alias("name"), x.zone.alias("zone")) - .to_df() -) -print("\n=== Assets on a crown-jewel attack path (Asset.on_attack_path persisted) ===") -print(f" {len(flagged)} of {total_assets} assets:") -for _, r in flagged.sort_values("id").iterrows(): - print(f" {r['id']:<8} {r['name']:<24} ({r['zone']})") + print("\n=== Assets on a crown-jewel attack path (Asset.on_attack_path persisted) ===") + print(f" {len(flagged)} of {total_assets} assets:") + for _, r in flagged.sort_values("id").iterrows(): + print(f" {r['id']:<8} {r['name']:<24} ({r['zone']})") From cf076763f114c686c031fd4e16390b5c915ab3f8 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:28:36 -0800 Subject: [PATCH 024/162] it-dependency-mapping: run as a deployed model The script deploys the model, waits for the first refresh and prints the report from the deployed model. Feature.max_downstream_depth is declared as a Relationship because a Property defined from a path rule fails deployment planning. The README covers the deployment config. --- it-dependency-mapping/README.md | 23 +- .../it_dependency_mapping.py | 302 +++++++++--------- 2 files changed, 176 insertions(+), 149 deletions(-) diff --git a/it-dependency-mapping/README.md b/it-dependency-mapping/README.md index 1c7ad8f5..0354f812 100644 --- a/it-dependency-mapping/README.md +++ b/it-dependency-mapping/README.md @@ -53,7 +53,8 @@ Built using RelationalAI's **variable-length path enumeration** (`model.path(... ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -88,18 +89,32 @@ Built using RelationalAI's **variable-length path enumeration** (`model.path(... python -m pip install . ``` -4. **Configure Snowflake connection and RAI profile** +4. **Configure the Snowflake connection and the deployment** - ```bash - rai init + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .IT_DEPENDENCY_MAPPING # published views and tables + identity: + internal_schema: .IT_DEPENDENCY_MAPPING_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python it_dependency_mapping.py` stops with an error naming the missing setting. + 5. **Run the template** ```bash python it_dependency_mapping.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + 6. **Expected output** (a few lines confirm success): ```text diff --git a/it-dependency-mapping/it_dependency_mapping.py b/it-dependency-mapping/it_dependency_mapping.py index 5008e4d5..1a970705 100644 --- a/it-dependency-mapping/it_dependency_mapping.py +++ b/it-dependency-mapping/it_dependency_mapping.py @@ -15,7 +15,9 @@ deepest dependency chains, and the owners that sit along the longest chain. Run: - /opt/homebrew/bin/python3.11 it_dependency_mapping.py + `python it_dependency_mapping.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from the + deployed model. Output: Prints the total downstream path count, per-feature path counts and longest @@ -90,169 +92,179 @@ def load_csv(path): # per path -- p.length is its hop count, p.nodes is the ordered sequence of # Feature nodes it visits, and src is auto-unified with p.nodes(0) (the start). -print("=== IT Dependency Mapping: downstream paths ===") - src = Feature.ref() p = model.path(src, Feature.contributes_to.repeat(1, MAX_DEPTH)).all().paths() -# Project the node sequence of each path (for display-side reassembly). Keep this -# its own query: selecting p.length or a path aggregate alongside p.nodes fans the -# node rows out (known paths issue), so the per-path rollups below are SEPARATE -# model.where(...) queries. -paths_df = ( - model.where(p) - .select( - p.alias("path_id"), - p.nodes["index"].alias("step"), - Feature(p.nodes).id.alias("feature_id"), - Feature(p.nodes).name.alias("feature_name"), - ) - .to_df() -) -paths_df["step"] = paths_df["step"].astype(int) -# Projecting p.nodes can emit duplicate (path, step) rows -- dedupe to one node -# per step before reassembly. -paths_df = paths_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) - -# Reassemble each path: order steps by node index, join the visited features into -# an ordered chain. The hop count is the max node index (a path over N edges has -# nodes at indices 0..N). String-joining names into a chain label is a pandas job. -chains = ( - paths_df.groupby("path_id") - .agg( - hops=("step", "max"), - node_ids=("feature_id", lambda s: tuple(s)), - chain=("feature_name", lambda s: " -> ".join(s)), - ) - .reset_index() -) - -print(f"\nTotal downstream dependency paths (1-{MAX_DEPTH} hops): {len(chains)}") - -# -------------------------------------------------- -# Per-feature path counts and longest downstream depth (in-PyRel aggregates) -# -------------------------------------------------- - -feature_df = model.select( - Feature.id.alias("feature_id"), - Feature.name.alias("feature_name"), - Feature.owner.alias("owner"), - Feature.deploy_tier.alias("deploy_tier"), -).to_df() - -# Paths starting at each feature: aggregates.count(p).per(src) groups the -# enumerated paths by their (typed) source endpoint -- a path-level rollup keyed -# on src, computed IN PyRel rather than via a pandas groupby on the chains frame. -paths_started_df = ( - model.where(p, n := aggs.count(p).per(src)) - .select(src.id.alias("feature_id"), n.alias("paths_downstream")) - .to_df() -) -# Count column arrives as Int128 -- cast before pandas merge/arithmetic. -paths_started_df["paths_downstream"] = paths_started_df["paths_downstream"].astype(int) - # Persist each feature's longest downstream depth back to the ontology so a # downstream query can rank features by reach without re-enumerating paths. # aggregates.max(p.length).per(src) takes the max hop count over the paths rooted -# at each src and defines it directly onto the Feature property -- no pandas -# groupby and no model.data round-trip. Kept SEPARATE from the node projection. -Feature.max_downstream_depth = model.Property( +# at each src and defines it directly onto Feature -- no pandas groupby and no +# model.data round-trip. Kept SEPARATE from the node projection. It is a +# Relationship, not a Property: a Property defined from a path fails deployment +# planning. +Feature.max_downstream_depth = model.Relationship( f"{Feature} has {Integer:max_downstream_depth}" ) model.define(src.max_downstream_depth(aggs.max(p.length).per(src))).where(p) -# Read the persisted depth back for the display table. Features with no outgoing -# path (the leaf dashboards) have no max_downstream_depth row -- fill them as 0. -max_depth_df = model.select( - Feature.id.alias("feature_id"), - Feature.max_downstream_depth.alias("max_downstream_depth"), -).to_df() -max_depth_df["max_downstream_depth"] = max_depth_df["max_downstream_depth"].astype( - "Int64" -) +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- -# Build the per-feature display table from the query results above (NOT from a -# pandas groupby on chains): paths_downstream from the count query, depth from the -# persisted property. -feature_summary = feature_df.merge( - paths_started_df, on="feature_id", how="left" -).merge(max_depth_df, on="feature_id", how="left") -feature_summary["paths_downstream"] = ( - feature_summary["paths_downstream"].fillna(0).astype(int) -) -feature_summary["max_downstream_depth"] = ( - feature_summary["max_downstream_depth"].fillna(0).astype(int) -) +if __name__ == "__main__": + # Deploy and wait for the first refresh. + model.deploy() + + print("=== IT Dependency Mapping: downstream paths ===") + + # Project the node sequence of each path (for display-side reassembly). Keep this + # its own query: selecting p.length or a path aggregate alongside p.nodes fans the + # node rows out (known paths issue), so the per-path rollups below are SEPARATE + # model.where(...) queries. + paths_df = ( + model.where(p) + .select( + p.alias("path_id"), + p.nodes["index"].alias("step"), + Feature(p.nodes).id.alias("feature_id"), + Feature(p.nodes).name.alias("feature_name"), + ) + .to_df() + ) + paths_df["step"] = paths_df["step"].astype(int) + # Projecting p.nodes can emit duplicate (path, step) rows -- dedupe to one node + # per step before reassembly. + paths_df = paths_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) + + # Reassemble each path: order steps by node index, join the visited features into + # an ordered chain. The hop count is the max node index (a path over N edges has + # nodes at indices 0..N). String-joining names into a chain label is a pandas job. + chains = ( + paths_df.groupby("path_id") + .agg( + hops=("step", "max"), + node_ids=("feature_id", lambda s: tuple(s)), + chain=("feature_name", lambda s: " -> ".join(s)), + ) + .reset_index() + ) -print("\n=== Per-feature downstream reach ===") -print( - f" {'Feature':<30} {'Tier':<9} {'Paths':>6} {'Max Depth':>10}" -) -print(f" {'-' * 57}") -for _, r in feature_summary.sort_values( - ["max_downstream_depth", "paths_downstream"], ascending=False -).iterrows(): - print( - f" {r['feature_name']:<30} {r['deploy_tier']:<9} " - f"{r['paths_downstream']:>6} {r['max_downstream_depth']:>10}" + print(f"\nTotal downstream dependency paths (1-{MAX_DEPTH} hops): {len(chains)}") + + # -------------------------------------------------- + # Per-feature path counts and longest downstream depth (in-PyRel aggregates) + # -------------------------------------------------- + + feature_df = model.select( + Feature.id.alias("feature_id"), + Feature.name.alias("feature_name"), + Feature.owner.alias("owner"), + Feature.deploy_tier.alias("deploy_tier"), + ).to_df() + + # Paths starting at each feature: aggregates.count(p).per(src) groups the + # enumerated paths by their (typed) source endpoint -- a path-level rollup keyed + # on src, computed IN PyRel rather than via a pandas groupby on the chains frame. + paths_started_df = ( + model.where(p, n := aggs.count(p).per(src)) + .select(src.id.alias("feature_id"), n.alias("paths_downstream")) + .to_df() + ) + # Count column arrives as Int128 -- cast before pandas merge/arithmetic. + paths_started_df["paths_downstream"] = paths_started_df["paths_downstream"].astype(int) + + # Read the persisted depth back for the display table. Features with no outgoing + # path (the leaf dashboards) have no max_downstream_depth row -- fill them as 0. + max_depth_df = model.select( + Feature.id.alias("feature_id"), + Feature.max_downstream_depth.alias("max_downstream_depth"), + ).to_df() + max_depth_df["max_downstream_depth"] = max_depth_df["max_downstream_depth"].astype( + "Int64" ) -# -------------------------------------------------- -# Maximal chains: longest non-extendable dependency chains -# -------------------------------------------------- -# A path is maximal when its node sequence is not a contiguous sub-chain of any -# other enumerated path -- i.e. it cannot be extended upstream or downstream. -# This collapses the full path set to the small set of end-to-end chains. + # Build the per-feature display table from the query results above (NOT from a + # pandas groupby on chains): paths_downstream from the count query, depth from the + # persisted property. + feature_summary = feature_df.merge( + paths_started_df, on="feature_id", how="left" + ).merge(max_depth_df, on="feature_id", how="left") + feature_summary["paths_downstream"] = ( + feature_summary["paths_downstream"].fillna(0).astype(int) + ) + feature_summary["max_downstream_depth"] = ( + feature_summary["max_downstream_depth"].fillna(0).astype(int) + ) -all_sequences = list(chains["node_ids"]) + print("\n=== Per-feature downstream reach ===") + print( + f" {'Feature':<30} {'Tier':<9} {'Paths':>6} {'Max Depth':>10}" + ) + print(f" {'-' * 57}") + for _, r in feature_summary.sort_values( + ["max_downstream_depth", "paths_downstream"], ascending=False + ).iterrows(): + print( + f" {r['feature_name']:<30} {r['deploy_tier']:<9} " + f"{r['paths_downstream']:>6} {r['max_downstream_depth']:>10}" + ) + # -------------------------------------------------- + # Maximal chains: longest non-extendable dependency chains + # -------------------------------------------------- + # A path is maximal when its node sequence is not a contiguous sub-chain of any + # other enumerated path -- i.e. it cannot be extended upstream or downstream. + # This collapses the full path set to the small set of end-to-end chains. -def is_sub_chain(short, long): - """True if `short` is a contiguous sub-sequence of the longer `long`.""" - if len(short) >= len(long): - return False - return any( - long[i : i + len(short)] == short - for i in range(len(long) - len(short) + 1) - ) + all_sequences = list(chains["node_ids"]) -maximal = chains[ - chains["node_ids"].apply( - lambda seq: not any( - is_sub_chain(seq, other) for other in all_sequences if other != seq + def is_sub_chain(short, long): + """True if `short` is a contiguous sub-sequence of the longer `long`.""" + if len(short) >= len(long): + return False + return any( + long[i : i + len(short)] == short + for i in range(len(long) - len(short) + 1) ) - ) -].copy() -maximal = maximal.sort_values(["hops", "chain"], ascending=[False, True]) -print( - f"\n=== Maximal dependency chains ({len(maximal)} of {len(chains)} paths) ===" -) -for _, r in maximal.iterrows(): - print(f" {r['hops']} hops: {r['chain']}") -# -------------------------------------------------- -# Owners along the single longest chain (blast radius) -# -------------------------------------------------- -# The longest chain is the worst-case propagation path: a change at its root has -# to clear every owner along the way. Join owner metadata onto its features. - -owner_by_id = dict(zip(feature_df["feature_id"], feature_df["owner"])) -name_by_id = dict(zip(feature_df["feature_id"], feature_df["feature_name"])) -tier_by_id = dict(zip(feature_df["feature_id"], feature_df["deploy_tier"])) - -longest = maximal.iloc[0] -longest_ids = longest["node_ids"] -distinct_owners = len({owner_by_id[i] for i in longest_ids}) - -print(f"\n=== Longest dependency chain: {longest['hops']} hops ===") -print(f" Chain: {longest['chain']}") -print(f" Spans {len(longest_ids)} features and {distinct_owners} owners:") -for i in longest_ids: - print(f" - {name_by_id[i]:<30} owner={owner_by_id[i]:<14} tier={tier_by_id[i]}") -print( - f"\n A change at '{name_by_id[longest_ids[0]]}' propagates through " - f"{len(longest_ids) - 1} downstream feature(s) before reaching " - f"'{name_by_id[longest_ids[-1]]}'." -) + maximal = chains[ + chains["node_ids"].apply( + lambda seq: not any( + is_sub_chain(seq, other) for other in all_sequences if other != seq + ) + ) + ].copy() + maximal = maximal.sort_values(["hops", "chain"], ascending=[False, True]) + + print( + f"\n=== Maximal dependency chains ({len(maximal)} of {len(chains)} paths) ===" + ) + for _, r in maximal.iterrows(): + print(f" {r['hops']} hops: {r['chain']}") + + # -------------------------------------------------- + # Owners along the single longest chain (blast radius) + # -------------------------------------------------- + # The longest chain is the worst-case propagation path: a change at its root has + # to clear every owner along the way. Join owner metadata onto its features. + + owner_by_id = dict(zip(feature_df["feature_id"], feature_df["owner"])) + name_by_id = dict(zip(feature_df["feature_id"], feature_df["feature_name"])) + tier_by_id = dict(zip(feature_df["feature_id"], feature_df["deploy_tier"])) + + longest = maximal.iloc[0] + longest_ids = longest["node_ids"] + distinct_owners = len({owner_by_id[i] for i in longest_ids}) + + print(f"\n=== Longest dependency chain: {longest['hops']} hops ===") + print(f" Chain: {longest['chain']}") + print(f" Spans {len(longest_ids)} features and {distinct_owners} owners:") + for i in longest_ids: + print(f" - {name_by_id[i]:<30} owner={owner_by_id[i]:<14} tier={tier_by_id[i]}") + print( + f"\n A change at '{name_by_id[longest_ids[0]]}' propagates through " + f"{len(longest_ids) - 1} downstream feature(s) before reaching " + f"'{name_by_id[longest_ids[-1]]}'." + ) From 482c537aba6eeb3b5f0aa9e90be5eed2e7c2cf9e Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:07:59 -0800 Subject: [PATCH 025/162] transaction_screening_local: run as a deployed model The script calls model.deploy() on the in-memory DuckDB database before the report queries, which now run under __main__. The config drops auto_deploy, so a query never deploys; the README's How it works and troubleshooting text follow. --- transaction_screening_local/README.md | 8 ++- .../transaction_screening_local.py | 69 ++++++++++--------- 2 files changed, 42 insertions(+), 35 deletions(-) diff --git a/transaction_screening_local/README.md b/transaction_screening_local/README.md index fc575ea3..18577a67 100644 --- a/transaction_screening_local/README.md +++ b/transaction_screening_local/README.md @@ -89,6 +89,8 @@ Local DuckDB execution relies on deploy mode, which the package currently flags python transaction_screening_local.py ``` + The script deploys the model into the in-memory DuckDB database, waits for its first refresh, and prints the report from the deployed model. The database lasts only for the run, so every run deploys the model again. + 5. Expected output (abbreviated). The suspects and the investigation set confirm a successful run — 6 of 54 accounts flag, and the review set expands to 8: ```text @@ -145,11 +147,11 @@ For the full concept and property definitions, see `transaction_screening_local. ## How it works -The script starts by configuring a local DuckDB connection with a `deployment` section, so the model materializes and runs on your machine with no Snowflake connection. The section sets `public_schema` (`main`, where the model's views are published), `identity.internal_schema` (`main_internal`, where RelationalAI keeps the model's internal objects), and `auto_deploy`, which deploys the model before each query. It then loads the transfer ledger from CSV and derives an `Account` for every party seen at either end of a transfer. +The script starts by configuring a local DuckDB connection with a `deployment` section, so the model materializes and runs on your machine with no Snowflake connection. The section sets `public_schema` (`main`, where the model's views are published) and `identity.internal_schema` (`main_internal`, where RelationalAI keeps the model's internal objects). It then loads the transfer ledger from CSV and derives an `Account` for every party seen at either end of a transfer. Next it builds the `transfers_to` relationship, linking each sender to its receiver, one row per transfer. The classification rules follow as declarative derived relationships: an account is flagged as structuring when it sends a transfer just under the reporting threshold, and as a large sender when it sends a transfer over the large-transfer limit. An account is a suspect if either flag holds — expressed as two rules that together read as a logical OR. -Finally, connectivity comes from a self-join over `transfers_to` rather than a graph reasoner: an account sits near a suspect if it transacted directly with one in either direction. The investigation set — accounts under review — is the union of suspects and their direct counterparties. The queries at the bottom of the script summarize the network and print the review set. +Finally, connectivity comes from a self-join over `transfers_to` rather than a graph reasoner: an account sits near a suspect if it transacted directly with one in either direction. The investigation set — accounts under review — is the union of suspects and their direct counterparties. The bottom of the script deploys the model, then its queries summarize the network and print the review set. ```text transactions.csv → load → Account + transfers_to → classification rules → suspects → one-hop expansion → review set @@ -186,7 +188,7 @@ DuckDB tables need a three-part name. Reference them as `memory..
A query falls back to a Snowflake path, or reads an empty model relation -Make sure the configuration includes a `deployment` section with `public_schema`, `identity.internal_schema`, and `auto_deploy` set to true, so that the model is routed to the DuckDB executor and materialized before queries run. +Make sure the configuration includes a `deployment` section with `public_schema` and `identity.internal_schema`, so that the model is routed to the DuckDB executor, and that `model.deploy()` runs before the queries, so that the model is materialized when they read it.
diff --git a/transaction_screening_local/transaction_screening_local.py b/transaction_screening_local/transaction_screening_local.py index e795479c..7475b22e 100644 --- a/transaction_screening_local/transaction_screening_local.py +++ b/transaction_screening_local/transaction_screening_local.py @@ -15,7 +15,9 @@ The whole pipeline is declarative — PyRel resolves dependencies automatically. Run: - `python transaction_screening_local.py` + `python transaction_screening_local.py` deploys the model to the in-memory + DuckDB database, waits for its first refresh, and prints the report from + the deployed model. Output: Prints the network overview, per-account sent volume, the suspect accounts, @@ -39,17 +41,15 @@ LARGE_TRANSFER = 50000.0 # Local DuckDB config (no Snowflake): a duckdb connection plus a `deployment` -# section routes the model to the local DuckDB executor and materializes derived -# relations before queries. The section names the schema for the model's -# published views (`public_schema`), the schema for its internal objects -# (`identity.internal_schema`), and turns on `auto_deploy`. +# section routes the model to the local DuckDB executor. The section names the +# schema for the model's published views (`public_schema`) and the schema for +# its internal objects (`identity.internal_schema`). config = create_config( connections={"local": DuckDBConnection(path=":memory:")}, # or a file path, e.g. "./dev.duckdb" default_connection="local", deployment={ "public_schema": "main", "identity": {"internal_schema": "main_internal"}, - "auto_deploy": True, }, ) @@ -113,30 +113,35 @@ model.where(Account.near_suspect()).define(Account.under_review()) # -------------------------------------------------- -# Results +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("== Network overview ==") -model.select( - aggs.count(txn.id).alias("transactions"), - aggs.sum(txn.amount).alias("total_moved"), -).inspect() - -print("\n== Sent volume per account ==") -model.where(txn.src == Account.id).select( - Account.id.alias("account"), - aggs.sum(txn.amount).per(Account).alias("total_sent"), - aggs.count(txn.id).per(Account).alias("sent_count"), -).inspect() - -print("\n== Suspect accounts (rules) ==") -model.where(Account.is_suspect()).select(Account.id.alias("suspect")).inspect() - -print("\n== Suspect -> counterparty (relationship self-join) ==") -_counterparty = Account.ref() -model.where(Account.is_suspect(), Account.transfers_to(_counterparty)).select( - Account.id.alias("suspect"), - _counterparty.id.alias("counterparty"), -).inspect() - -print("\n== Investigation set (suspect or one hop from a suspect) ==") -model.where(Account.under_review()).select(Account.id.alias("flagged_for_review")).inspect() + +if __name__ == "__main__": + # Deploy into the DuckDB database and wait for the first refresh. + model.deploy() + + print("== Network overview ==") + model.select( + aggs.count(txn.id).alias("transactions"), + aggs.sum(txn.amount).alias("total_moved"), + ).inspect() + + print("\n== Sent volume per account ==") + model.where(txn.src == Account.id).select( + Account.id.alias("account"), + aggs.sum(txn.amount).per(Account).alias("total_sent"), + aggs.count(txn.id).per(Account).alias("sent_count"), + ).inspect() + + print("\n== Suspect accounts (rules) ==") + model.where(Account.is_suspect()).select(Account.id.alias("suspect")).inspect() + + print("\n== Suspect -> counterparty (relationship self-join) ==") + _counterparty = Account.ref() + model.where(Account.is_suspect(), Account.transfers_to(_counterparty)).select( + Account.id.alias("suspect"), + _counterparty.id.alias("counterparty"), + ).inspect() + + print("\n== Investigation set (suspect or one hop from a suspect) ==") + model.where(Account.under_review()).select(Account.id.alias("flagged_for_review")).inspect() From 22eb813faf4b74e3a8a418eaad7c393cc6e3d6df Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:19:26 -0800 Subject: [PATCH 026/162] money_laundering_motif_detection: run as a deployed model Each runner deploys the shared model name with its own motif Problem and reports from it; status comes from a per-runner SolveStatus relationship and the solution count from solve_info().num_points. The README says that running another runner redeploys and replaces the previous runner's motif. --- money_laundering_motif_detection/README.md | 56 +++++- .../motif_butterfly.py | 183 ++++++++++-------- .../motif_kyc_burst.py | 153 ++++++++------- .../motif_smurf_army.py | 153 ++++++++------- 4 files changed, 316 insertions(+), 229 deletions(-) diff --git a/money_laundering_motif_detection/README.md b/money_laundering_motif_detection/README.md index 92b0015d..698a2933 100644 --- a/money_laundering_motif_detection/README.md +++ b/money_laundering_motif_detection/README.md @@ -70,7 +70,7 @@ Each motif owns its own decisions and constraints. The bullets below summarise t - All chosen burst-tx within `BURST_WINDOW_MINUTES` of each other - Sub-threshold amount filter -Multi-solution enumeration via `problem.solve(..., solution_limit=K)` is available on every runner for batch analyst-triage workflows. The butterfly and KYC-burst runners exhaust to `OPTIMAL` on the bundled data; the smurf runner hits `SOLUTION_LIMIT` because pairwise-distinct beneficial-owner cohorts can swap any one chosen smurf for the dup-BO decoys (47 or 48), opening more cohorts than `MAX_SMURF_MOTIFS=5`. Raise the limit to enumerate further. +Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is available on every runner for batch analyst-triage workflows. The butterfly and KYC-burst runners exhaust to `OPTIMAL` on the bundled data; the smurf runner hits `SOLUTION_LIMIT` because pairwise-distinct beneficial-owner cohorts can swap any one chosen smurf for the dup-BO decoys (47 or 48), opening more cohorts than `MAX_SMURF_MOTIFS=5`. Raise the limit to enumerate further. ## What's included @@ -88,7 +88,8 @@ Multi-solution enumeration via `problem.solve(..., solution_limit=K)` is availab ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -116,17 +117,58 @@ Multi-solution enumeration via `problem.solve(..., solution_limit=K)` is availab python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: money_laundering_motif_detection_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .MONEY_LAUNDERING_MOTIF_DETECTION # published views and tables + identity: + internal_schema: .MONEY_LAUNDERING_MOTIF_DETECTION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, each runner stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .MONEY_LAUNDERING_MOTIF_DETECTION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; ``` + The deploy creates the public schema `.MONEY_LAUNDERING_MOTIF_DETECTION` if it doesn't exist. -5. Run any of the three motif detectors. They share the bundled data and run independently: +5. Run any of the three motif detectors. They share the bundled data and the configured schema pair: ```bash python motif_butterfly.py python motif_smurf_army.py python motif_kyc_burst.py ``` + Each runner deploys the model, waits for the first refresh, which runs that runner's solve, and prints its report from the deployed model. `rai models deploy --path motif_butterfly.py` (or another runner) deploys without printing the report. + + The three runners deploy the same model, `money_laundering_motif_detection`, each with only its own motif. Running a different runner than the last one redeploys, replacing the previous runner's motif, and its refresh solves the new motif; the deployed model then holds only that runner's motif. Running the same runner again without changing it redeploys nothing and does not solve again; it reports the latest refresh. A change to a runner or to its `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the same runner: + ```sql + CALL .MONEY_LAUNDERING_MOTIF_DETECTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output. The bundled dataset plants exactly the motifs each detector targets, plus dup-BO / out-of-window / over-amount decoys for each. Solver build strings and exact wall times will vary; the structure of the output is stable. @@ -253,7 +295,7 @@ See `model_setup.py` and the three motif runners for the implementation, and `ru ### Scale up / productionize -- Raise the solution limit on a real ledger. Each runner's `MAX_*_MOTIFS` is sized for the demo. On a production ledger with thousands of accounts you may want 100+ so the analyst inbox surfaces the full population of candidates. The `time_limit_sec` argument to `problem.solve` is your safety net. +- Raise the solution limit on a real ledger. Each runner's `MAX_*_MOTIFS` is sized for the demo. On a production ledger with thousands of accounts you may want 100+ so the analyst inbox surfaces the full population of candidates. The `time_limit_sec` argument to `problem.configure` is your safety net. - Pin `relationalai` to a known-good version for reproducible runs, and schedule the runners after each ledger refresh so candidate cases stay current for analyst triage. ## References diff --git a/money_laundering_motif_detection/motif_butterfly.py b/money_laundering_motif_detection/motif_butterfly.py index be9a0e42..2a5cf5d1 100644 --- a/money_laundering_motif_detection/motif_butterfly.py +++ b/money_laundering_motif_detection/motif_butterfly.py @@ -15,7 +15,9 @@ currency-transaction-report threshold so no leg triggers a CTR filing. Run: - `python motif_butterfly.py` + `python motif_butterfly.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), + and prints the report from the deployed model. Output: Prints the formulation, the solve-result block, and three @@ -26,7 +28,7 @@ import pandas as pd from model_setup import AMOUNT_THRESHOLD_DOLLARS, DATA_DIR, create_model -from relationalai.semantics import Integer, sum +from relationalai.semantics import Integer, String, sum from relationalai.semantics.reasoners.prescriptive import Problem # Runner-level parameters. @@ -209,86 +211,99 @@ ).require(Account.is_hub + H2.is_hub <= 1) problem.satisfy(same_bo_ic) -# Solve and report. -print("\n" + "=" * 70) -print("BUTTERFLY (scatter-gather with per-hub conservation)") -print("CSP technique: per-vertex aggregate equality over decision-selected edge subset") -print("=" * 70) -problem.display() -problem.solve("minizinc", time_limit_sec=60, solution_limit=MAX_BUTTERFLY_MOTIFS) -si = problem.solve_info() -si.display() - -# No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` -# calls above means the decision-variable values are NOT written back to -# the relational `is_motif` / `is_source` / `is_hub` / `is_dest` properties. -# Without those values in the relational layer, verify() cannot re-evaluate -# ICs that reference them and prints spurious "Requirements not met" -# warnings instead of doing real verification. The inspect tables below -# surface the solver's choices directly; manual inspection against the IC -# formulas is the verification path here. - -status = si.termination_status -motif_count = si.num_points or 0 -if status == "INFEASIBLE": - print( - "\nResult: no butterfly motifs exist under the encoded constraints. " - "The search space was exhausted with zero feasible motifs." - ) -elif status == "OPTIMAL": - print( - f"\nResult: {motif_count} butterfly motif(s) found and search exhausted " - "(no further motifs exist under the encoded constraints)." - ) -elif status == "SOLUTION_LIMIT": - print( - f"\nResult: {motif_count} butterfly motif(s) found; " - f"hit MAX_BUTTERFLY_MOTIFS={MAX_BUTTERFLY_MOTIFS}. " - "More may exist -- raise the limit to enumerate further." - ) -else: - print( - f"\nResult: search returned status={status} with {motif_count} motif(s); " - "did not finish. Raise time_limit_sec or check the README troubleshooting." - ) +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_BUTTERFLY_MOTIFS) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"butterfly solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) + +# Deploy, then report (`rai models deploy` skips this block). +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + print("\n" + "=" * 70) + print("BUTTERFLY (scatter-gather with per-hub conservation)") + print("CSP technique: per-vertex aggregate equality over decision-selected edge subset") + print("=" * 70) + problem.display() + + # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` + # calls above means the decision-variable values are NOT written back to + # the relational `is_motif` / `is_source` / `is_hub` / `is_dest` properties. + # Without those values in the relational layer, verify() cannot re-evaluate + # ICs that reference them and prints spurious "Requirements not met" + # warnings instead of doing real verification. The inspect tables below + # surface the solver's choices directly; manual inspection against the IC + # formulas is the verification path here. + + # Status of the latest refresh's solve and the number of solutions it found. + status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) + motif_count = problem.solve_info().num_points or 0 + print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") + if status == "INFEASIBLE": + print( + "\nResult: no butterfly motifs exist under the encoded constraints. " + "The search space was exhausted with zero feasible motifs." + ) + elif status == "OPTIMAL": + print( + f"\nResult: {motif_count} butterfly motif(s) found and search exhausted " + "(no further motifs exist under the encoded constraints)." + ) + elif status == "SOLUTION_LIMIT": + print( + f"\nResult: {motif_count} butterfly motif(s) found; " + f"hit MAX_BUTTERFLY_MOTIFS={MAX_BUTTERFLY_MOTIFS}. " + "More may exist -- raise the limit to enumerate further." + ) + else: + print( + f"\nResult: search returned status={status} with {motif_count} motif(s); " + "did not finish. Raise time_limit_sec or check the README troubleshooting." + ) + + print("\nButterfly: chosen motif transactions (one row per motif edge per solution):") + sol_idx = Integer.ref() + val = Integer.ref() + model.select( + sol_idx.alias("solution"), + is_motif_var.transaction.tx_id.alias("tx_id"), + is_motif_var.transaction.src.id.alias("src_account_id"), + is_motif_var.transaction.src.name.alias("src_name"), + is_motif_var.transaction.dst.id.alias("dst_account_id"), + is_motif_var.transaction.dst.name.alias("dst_name"), + is_motif_var.transaction.amount_dollars.alias("amount"), + is_motif_var.transaction.ts_minutes.alias("ts_min"), + ).where(is_motif_var.values(sol_idx, val), val == 1).inspect() + + print("\nButterfly: chosen motif hubs per solution (with shared beneficial owner):") + model.select( + sol_idx.alias("solution"), + is_hub_var.account.id.alias("hub_id"), + is_hub_var.account.name.alias("hub_name"), + is_hub_var.account.bo_id.alias("bo_id"), + ).where(is_hub_var.values(sol_idx, val), val == 1).inspect() -print("\nButterfly: chosen motif transactions (one row per motif edge per solution):") -sol_idx = Integer.ref() -val = Integer.ref() -model.select( - sol_idx.alias("solution"), - is_motif_var.transaction.tx_id.alias("tx_id"), - is_motif_var.transaction.src.id.alias("src_account_id"), - is_motif_var.transaction.src.name.alias("src_name"), - is_motif_var.transaction.dst.id.alias("dst_account_id"), - is_motif_var.transaction.dst.name.alias("dst_name"), - is_motif_var.transaction.amount_dollars.alias("amount"), - is_motif_var.transaction.ts_minutes.alias("ts_min"), -).where(is_motif_var.values(sol_idx, val), val == 1).inspect() - -print("\nButterfly: chosen motif hubs per solution (with shared beneficial owner):") -model.select( - sol_idx.alias("solution"), - is_hub_var.account.id.alias("hub_id"), - is_hub_var.account.name.alias("hub_name"), - is_hub_var.account.bo_id.alias("bo_id"), -).where(is_hub_var.values(sol_idx, val), val == 1).inspect() - -print("\nButterfly: chosen motif source and destination per solution:") -src_sol = Integer.ref() -src_val = Integer.ref() -dst_sol = Integer.ref() -dst_val = Integer.ref() -model.select( - src_sol.alias("solution"), - is_source_var.account.id.alias("source_id"), - is_source_var.account.name.alias("source_name"), - is_dest_var.account.id.alias("dest_id"), - is_dest_var.account.name.alias("dest_name"), -).where( - is_source_var.values(src_sol, src_val), - src_val == 1, - is_dest_var.values(dst_sol, dst_val), - dst_val == 1, - src_sol == dst_sol, -).inspect() + print("\nButterfly: chosen motif source and destination per solution:") + src_sol = Integer.ref() + src_val = Integer.ref() + dst_sol = Integer.ref() + dst_val = Integer.ref() + model.select( + src_sol.alias("solution"), + is_source_var.account.id.alias("source_id"), + is_source_var.account.name.alias("source_name"), + is_dest_var.account.id.alias("dest_id"), + is_dest_var.account.name.alias("dest_name"), + ).where( + is_source_var.values(src_sol, src_val), + src_val == 1, + is_dest_var.values(dst_sol, dst_val), + dst_val == 1, + src_sol == dst_sol, + ).inspect() diff --git a/money_laundering_motif_detection/motif_kyc_burst.py b/money_laundering_motif_detection/motif_kyc_burst.py index b9c49a9f..b33a1859 100644 --- a/money_laundering_motif_detection/motif_kyc_burst.py +++ b/money_laundering_motif_detection/motif_kyc_burst.py @@ -19,7 +19,9 @@ https://www.tookitaki.com/compliance-hub/smurfing-structuring-aml-detection-reporting)). Run: - `python motif_kyc_burst.py` + `python motif_kyc_burst.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), + and prints the report from the deployed model. Output: Prints the formulation, the solve-result block, and two @@ -29,7 +31,7 @@ """ from model_setup import AMOUNT_THRESHOLD_DOLLARS, create_model -from relationalai.semantics import Integer, sum +from relationalai.semantics import Integer, String, sum from relationalai.semantics.reasoners.prescriptive import Problem # Runner-level parameters. @@ -142,70 +144,83 @@ ).require(Transaction.is_burst_tx == 0) problem.satisfy(threshold_ic) -# Solve and report. -print("\n" + "=" * 70) -print(f"KYC-MIX BURST (fan-in to account id {BURST_TARGET_DESTINATION_ID})") -print("CSP technique: cardinality over chosen subset") -print("=" * 70) -problem.display() -problem.solve("minizinc", time_limit_sec=60, solution_limit=MAX_BURST_MOTIFS) -si = problem.solve_info() -si.display() - -# No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` -# calls above means the decision-variable values are NOT written back to -# the relational `is_burst` / `is_burst_tx` properties. Without those values -# in the relational layer, verify() cannot re-evaluate ICs that reference -# them and prints spurious "Requirements not met" warnings instead of -# doing real verification. The inspect tables below surface the solver's -# choices directly; manual inspection against the IC formulas is the -# verification path here. - -status = si.termination_status -motif_count = si.num_points or 0 -if status == "INFEASIBLE": - print( - "\nResult: no KYC-mix burst cohort exists under the encoded constraints " - f"for target id {BURST_TARGET_DESTINATION_ID}. The search space was " - "exhausted with zero feasible cohorts." - ) -elif status == "OPTIMAL": - print( - f"\nResult: {motif_count} KYC-mix burst cohort(s) found and search exhausted " - "(no further cohorts exist under the encoded constraints)." - ) -elif status == "SOLUTION_LIMIT": - print( - f"\nResult: {motif_count} KYC-mix burst cohort(s) found; " - f"hit MAX_BURST_MOTIFS={MAX_BURST_MOTIFS}. " - "More may exist -- raise the limit to enumerate further." - ) -else: - print( - f"\nResult: search returned status={status} with {motif_count} cohort(s); " - "did not finish. Raise time_limit_sec or check the README troubleshooting." - ) - -print("\nKYC-burst: chosen burst-tx per solution:") -sol_idx = Integer.ref() -val = Integer.ref() -model.select( - sol_idx.alias("solution"), - is_burst_tx_var.transaction.tx_id.alias("tx_id"), - is_burst_tx_var.transaction.src.id.alias("src_account_id"), - is_burst_tx_var.transaction.src.name.alias("src_name"), - is_burst_tx_var.transaction.src.kyc_tier.alias("src_kyc_tier"), - is_burst_tx_var.transaction.dst.name.alias("dst_name"), - is_burst_tx_var.transaction.amount_dollars.alias("amount"), - is_burst_tx_var.transaction.ts_minutes.alias("ts_min"), -).where(is_burst_tx_var.values(sol_idx, val), val == 1).inspect() - -print("\nKYC-burst: chosen burst accounts per solution (with KYC tier and jurisdiction):") -model.select( - sol_idx.alias("solution"), - is_burst_var.account.id.alias("burst_id"), - is_burst_var.account.name.alias("burst_name"), - is_burst_var.account.kyc_tier.alias("kyc_tier"), - is_burst_var.account.jurisdiction.alias("jurisdiction"), - is_burst_var.account.bo_id.alias("bo_id"), -).where(is_burst_var.values(sol_idx, val), val == 1).inspect() +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_BURST_MOTIFS) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"kyc burst solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) + +# Deploy, then report (`rai models deploy` skips this block). +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + print("\n" + "=" * 70) + print(f"KYC-MIX BURST (fan-in to account id {BURST_TARGET_DESTINATION_ID})") + print("CSP technique: cardinality over chosen subset") + print("=" * 70) + problem.display() + + # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` + # calls above means the decision-variable values are NOT written back to + # the relational `is_burst` / `is_burst_tx` properties. Without those values + # in the relational layer, verify() cannot re-evaluate ICs that reference + # them and prints spurious "Requirements not met" warnings instead of + # doing real verification. The inspect tables below surface the solver's + # choices directly; manual inspection against the IC formulas is the + # verification path here. + + # Status of the latest refresh's solve and the number of solutions it found. + status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) + motif_count = problem.solve_info().num_points or 0 + print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") + if status == "INFEASIBLE": + print( + "\nResult: no KYC-mix burst cohort exists under the encoded constraints " + f"for target id {BURST_TARGET_DESTINATION_ID}. The search space was " + "exhausted with zero feasible cohorts." + ) + elif status == "OPTIMAL": + print( + f"\nResult: {motif_count} KYC-mix burst cohort(s) found and search exhausted " + "(no further cohorts exist under the encoded constraints)." + ) + elif status == "SOLUTION_LIMIT": + print( + f"\nResult: {motif_count} KYC-mix burst cohort(s) found; " + f"hit MAX_BURST_MOTIFS={MAX_BURST_MOTIFS}. " + "More may exist -- raise the limit to enumerate further." + ) + else: + print( + f"\nResult: search returned status={status} with {motif_count} cohort(s); " + "did not finish. Raise time_limit_sec or check the README troubleshooting." + ) + + print("\nKYC-burst: chosen burst-tx per solution:") + sol_idx = Integer.ref() + val = Integer.ref() + model.select( + sol_idx.alias("solution"), + is_burst_tx_var.transaction.tx_id.alias("tx_id"), + is_burst_tx_var.transaction.src.id.alias("src_account_id"), + is_burst_tx_var.transaction.src.name.alias("src_name"), + is_burst_tx_var.transaction.src.kyc_tier.alias("src_kyc_tier"), + is_burst_tx_var.transaction.dst.name.alias("dst_name"), + is_burst_tx_var.transaction.amount_dollars.alias("amount"), + is_burst_tx_var.transaction.ts_minutes.alias("ts_min"), + ).where(is_burst_tx_var.values(sol_idx, val), val == 1).inspect() + + print("\nKYC-burst: chosen burst accounts per solution (with KYC tier and jurisdiction):") + model.select( + sol_idx.alias("solution"), + is_burst_var.account.id.alias("burst_id"), + is_burst_var.account.name.alias("burst_name"), + is_burst_var.account.kyc_tier.alias("kyc_tier"), + is_burst_var.account.jurisdiction.alias("jurisdiction"), + is_burst_var.account.bo_id.alias("bo_id"), + ).where(is_burst_var.values(sol_idx, val), val == 1).inspect() diff --git a/money_laundering_motif_detection/motif_smurf_army.py b/money_laundering_motif_detection/motif_smurf_army.py index 2ede918f..e4024a0c 100644 --- a/money_laundering_motif_detection/motif_smurf_army.py +++ b/money_laundering_motif_detection/motif_smurf_army.py @@ -22,7 +22,9 @@ arriving at the target merchant within a short coordinated burst. Run: - `python motif_smurf_army.py` + `python motif_smurf_army.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), + and prints the report from the deployed model. Output: Prints the formulation, the solve-result block, and two @@ -32,7 +34,7 @@ """ from model_setup import AMOUNT_THRESHOLD_DOLLARS, create_model -from relationalai.semantics import Integer, sum +from relationalai.semantics import Integer, String, sum from relationalai.semantics.reasoners.prescriptive import Problem # Runner-level parameters. @@ -167,70 +169,83 @@ ).require(Transaction.is_smurf_tx == 0) problem.satisfy(threshold_ic) -# Solve and report. -print("\n" + "=" * 70) -print(f"SMURF ARMY (fan-in to account id {SMURF_TARGET_DESTINATION_ID})") -print("CSP technique: pairwise distinctness + sum-target over chosen subset") -print("=" * 70) -problem.display() -problem.solve("minizinc", time_limit_sec=60, solution_limit=MAX_SMURF_MOTIFS) -si = problem.solve_info() -si.display() - -# No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` -# calls above means the decision-variable values are NOT written back to -# the relational `is_smurf` / `is_smurf_tx` properties. Without those values -# in the relational layer, verify() cannot re-evaluate ICs that reference -# them and prints spurious "Requirements not met" warnings instead of -# doing real verification. The inspect tables below surface the solver's -# choices directly; manual inspection against the IC formulas is the -# verification path here. - -status = si.termination_status -motif_count = si.num_points or 0 -if status == "INFEASIBLE": - print( - "\nResult: no smurf-army cohort exists under the encoded constraints " - f"for target id {SMURF_TARGET_DESTINATION_ID}. The search space was " - "exhausted with zero feasible cohorts." - ) -elif status == "OPTIMAL": - print( - f"\nResult: {motif_count} smurf-army cohort(s) found and search exhausted " - "(no further cohorts exist under the encoded constraints)." - ) -elif status == "SOLUTION_LIMIT": - print( - f"\nResult: {motif_count} smurf-army cohort(s) found; " - f"hit MAX_SMURF_MOTIFS={MAX_SMURF_MOTIFS}. " - "More may exist -- raise the limit to enumerate further." - ) -else: - print( - f"\nResult: search returned status={status} with {motif_count} cohort(s); " - "did not finish. Raise time_limit_sec or check the README troubleshooting." - ) - -print("\nSmurf army: chosen smurf-tx per solution:") -sol_idx = Integer.ref() -val = Integer.ref() -model.select( - sol_idx.alias("solution"), - is_smurf_tx_var.transaction.tx_id.alias("tx_id"), - is_smurf_tx_var.transaction.src.id.alias("src_account_id"), - is_smurf_tx_var.transaction.src.name.alias("src_name"), - is_smurf_tx_var.transaction.src.bo_id.alias("src_bo_id"), - is_smurf_tx_var.transaction.dst.name.alias("dst_name"), - is_smurf_tx_var.transaction.amount_dollars.alias("amount"), - is_smurf_tx_var.transaction.ts_minutes.alias("ts_min"), -).where(is_smurf_tx_var.values(sol_idx, val), val == 1).inspect() - -print("\nSmurf army: chosen smurf accounts per solution (with distinct BOs and jurisdictions):") -model.select( - sol_idx.alias("solution"), - is_smurf_var.account.id.alias("smurf_id"), - is_smurf_var.account.name.alias("smurf_name"), - is_smurf_var.account.bo_id.alias("bo_id"), - is_smurf_var.account.kyc_tier.alias("kyc_tier"), - is_smurf_var.account.jurisdiction.alias("jurisdiction"), -).where(is_smurf_var.values(sol_idx, val), val == 1).inspect() +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_SMURF_MOTIFS) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"smurf army solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) + +# Deploy, then report (`rai models deploy` skips this block). +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + print("\n" + "=" * 70) + print(f"SMURF ARMY (fan-in to account id {SMURF_TARGET_DESTINATION_ID})") + print("CSP technique: pairwise distinctness + sum-target over chosen subset") + print("=" * 70) + problem.display() + + # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` + # calls above means the decision-variable values are NOT written back to + # the relational `is_smurf` / `is_smurf_tx` properties. Without those values + # in the relational layer, verify() cannot re-evaluate ICs that reference + # them and prints spurious "Requirements not met" warnings instead of + # doing real verification. The inspect tables below surface the solver's + # choices directly; manual inspection against the IC formulas is the + # verification path here. + + # Status of the latest refresh's solve and the number of solutions it found. + status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) + motif_count = problem.solve_info().num_points or 0 + print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") + if status == "INFEASIBLE": + print( + "\nResult: no smurf-army cohort exists under the encoded constraints " + f"for target id {SMURF_TARGET_DESTINATION_ID}. The search space was " + "exhausted with zero feasible cohorts." + ) + elif status == "OPTIMAL": + print( + f"\nResult: {motif_count} smurf-army cohort(s) found and search exhausted " + "(no further cohorts exist under the encoded constraints)." + ) + elif status == "SOLUTION_LIMIT": + print( + f"\nResult: {motif_count} smurf-army cohort(s) found; " + f"hit MAX_SMURF_MOTIFS={MAX_SMURF_MOTIFS}. " + "More may exist -- raise the limit to enumerate further." + ) + else: + print( + f"\nResult: search returned status={status} with {motif_count} cohort(s); " + "did not finish. Raise time_limit_sec or check the README troubleshooting." + ) + + print("\nSmurf army: chosen smurf-tx per solution:") + sol_idx = Integer.ref() + val = Integer.ref() + model.select( + sol_idx.alias("solution"), + is_smurf_tx_var.transaction.tx_id.alias("tx_id"), + is_smurf_tx_var.transaction.src.id.alias("src_account_id"), + is_smurf_tx_var.transaction.src.name.alias("src_name"), + is_smurf_tx_var.transaction.src.bo_id.alias("src_bo_id"), + is_smurf_tx_var.transaction.dst.name.alias("dst_name"), + is_smurf_tx_var.transaction.amount_dollars.alias("amount"), + is_smurf_tx_var.transaction.ts_minutes.alias("ts_min"), + ).where(is_smurf_tx_var.values(sol_idx, val), val == 1).inspect() + + print("\nSmurf army: chosen smurf accounts per solution (with distinct BOs and jurisdictions):") + model.select( + sol_idx.alias("solution"), + is_smurf_var.account.id.alias("smurf_id"), + is_smurf_var.account.name.alias("smurf_name"), + is_smurf_var.account.bo_id.alias("bo_id"), + is_smurf_var.account.kyc_tier.alias("kyc_tier"), + is_smurf_var.account.jurisdiction.alias("jurisdiction"), + ).where(is_smurf_var.values(sol_idx, val), val == 1).inspect() From a3bd8773ea9d9896c91f52474afc62061bf9f0ef Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:29:05 -0800 Subject: [PATCH 027/162] simple-start: run as a deployed model The notebook declares the concepts, the connections relationship, the graph and its betweenness centrality first, then deploys in one model.deploy() cell, then runs the query cells. The note about waiting for the initial sync is gone: the deploy waits for the first refresh. The README covers the deployment config and the model role's read access to the source table. --- simple-start/README.md | 33 ++++++++----- simple-start/simple-start.ipynb | 88 ++++++++++++++++++++------------- 2 files changed, 77 insertions(+), 44 deletions(-) diff --git a/simple-start/README.md b/simple-start/README.md index 43a197fa..a3936d19 100644 --- a/simple-start/README.md +++ b/simple-start/README.md @@ -42,7 +42,8 @@ It walks through a small end-to-end example: create a Snowflake table, model it - Python >= 3.10 - A Snowflake account with the RelationalAI Native App installed -- A Snowflake user and role that can: +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role, where the role can also: - Create schemas and tables or write into a schema you control - Create and refresh a stream into the RelationalAI app, as prompted by the notebook @@ -75,16 +76,24 @@ It walks through a small end-to-end example: create a Snowflake table, model it python -m pip install . ``` -4. **Configure credentials** +4. **Configure the Snowflake connection and the deployment** - This notebook reads data from Snowflake and executes RelationalAI queries, so you need a working RelationalAI and Snowflake configuration. + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one, and keep both apart from `RAI_DEMO.SIMPLE_START`, which holds the source table; the deploy creates both schemas if they don't exist. - If you use the RelationalAI CLI, run: - - ```bash - rai init + ```yaml + deployment: + role: + public_schema: .SIMPLE_START_MODEL # published views and tables + identity: + internal_schema: .SIMPLE_START_MODEL_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, the notebook's `model.deploy()` cell stops with an error naming the missing setting. + If you have multiple profiles, set one explicitly: ```bash @@ -99,7 +108,7 @@ It walks through a small end-to-end example: create a Snowflake table, model it 6. **Run the template** - Open `simple-start.ipynb` and run the cells top-to-bottom or "Run All". + Open `simple-start.ipynb` and run the cells top-to-bottom or "Run All". The `model.deploy()` cell deploys the model and waits for its first refresh; the cells after it query the deployed model. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 7. **Expected output** @@ -115,9 +124,10 @@ At a high level, the notebook: 1. Creates and populates the `CONNECTIONS` table in Snowflake. 2. Defines `Station` and `Connection` concepts and loads them from the Snowflake source table. -3. Builds an undirected graph with `Station` as the node concept and the station connectivity relation as its edges. -4. Lists the resulting edges as a table. -5. Computes betweenness centrality and queries the scores into a pandas DataFrame. +3. Builds an undirected graph with `Station` as the node concept and the station connectivity relation as its edges, and declares betweenness centrality over it. +4. Deploys the model and waits for its first refresh. +5. Lists the resulting edges as a table. +6. Queries the betweenness centrality scores into a pandas DataFrame. > [!TIP] > At any point after step 2 you can run `inspect.schema(model)` to see exactly what's registered. See [Inspect the model schema](#inspect-the-model-schema) below. @@ -206,6 +216,7 @@ concept_or_none = inspect.to_concept(some_handle, default=None) - Replace `RAI_DEMO.SIMPLE_START.CONNECTIONS` with your own edge table. - Ensure your table has two columns that represent the endpoints of each edge. +- Make sure the model role can read your table (`SELECT` on it, `USAGE` on its database and schema): the deploy reads the table with the model role alone and stops with an error naming the table otherwise. **Extend the model:** diff --git a/simple-start/simple-start.ipynb b/simple-start/simple-start.ipynb index f98e897b..08605553 100644 --- a/simple-start/simple-start.ipynb +++ b/simple-start/simple-start.ipynb @@ -34,7 +34,7 @@ "\n", "If you haven't already done so, begin by installing the RelationalAI Native App in your Snowflake account. See [this page in the RelationalAI documentation](https://docs.relational.ai/manage/install) for instructions.\n", "\n", - "Next, [install RelationalAI's Python library into a local Python virtual environment](https://docs.relational.ai/build/setup/ide) and run `rai init` to connect your project to your Snowflake account.\n", + "Next, [install RelationalAI's Python library into a local Python virtual environment](https://docs.relational.ai/build/setup/ide) and run `rai init` to connect your project to your Snowflake account. Then add a `deployment` section to `raiconfig.yaml`, as described in this template's README, so the notebook can deploy the model.\n", "\n", "This notebook is written with the assumption that you're running it from such a virtual environment. To do that, navigate in your terminal to your project's root directory and run the following commands:\n", "\n", @@ -184,9 +184,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Simple Queries\n", - "\n", - "The following query counts the number of connections:" + "Next we'll define a `connections` relationship between `Station` entities." ] }, { @@ -195,23 +193,27 @@ "metadata": {}, "outputs": [], "source": [ - "# Count number of rows in the connections table:\n", - "model.select(aggregates.count(Connection)).to_df()" + "Station.connections = model.Relationship(f\"{Station} is connected to {Station:other_station}\")\n", + "\n", + "model.where(\n", + " Connection.src == Station\n", + ").define(\n", + " Station.connections(Connection.dst)\n", + ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "> [!NOTE]\n", - "> If this query does not return data the first time, wait for the initial sync to finish, then rerun the cell." + "### Graph Analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Next we'll define a `connections` relationship between `Station` entities." + "The power stations and connections between them form a graph. We can model a graph in RelationalAI by wrapping the model in a `Graph` object. Passing `node_concept=Station` makes `Station` the graph's `Node` concept, and we then define the graph's edges through its `Edge` concept." ] }, { @@ -220,20 +222,15 @@ "metadata": {}, "outputs": [], "source": [ - "Station.connections = model.Relationship(f\"{Station} is connected to {Station:other_station}\")\n", - "\n", - "model.where(\n", - " Connection.src == Station\n", - ").define(\n", - " Station.connections(Connection.dst)\n", - ")" + "graph = Graph(model, directed=False, weighted=False, node_concept=Station)\n", + "model.define(graph.Edge.new(src=Station, dst=Station.connections))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Now we can ask for a list of all the stations IDs:" + "**Betweenness centrality** is a common graph metric that assigns each station a score based on its position in the network. It’s available as a method on the `Graph` object:" ] }, { @@ -242,26 +239,35 @@ "metadata": {}, "outputs": [], "source": [ - "(\n", - " model.select(Station.id)\n", - " .to_df()\n", - " .sort_values(\"id\")\n", - " .reset_index(drop=True)\n", - ")" + "Node = graph.Node\n", + "Node.centrality = graph.betweenness_centrality()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Graph Analysis" + "### Deploy the Model\n", + "\n", + "The model is now fully declared. Deploying it creates the model's objects in the deployment schemas from `raiconfig.yaml` and waits for the first refresh, which loads the `CONNECTIONS` table and computes the graph metric. The queries below read from the deployed model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "model.deploy()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The power stations and connections between them form a graph. We can model a graph in RelationalAI by wrapping the model in a `Graph` object. Passing `node_concept=Station` makes `Station` the graph's `Node` concept, and we then define the graph's edges through its `Edge` concept." + "### Simple Queries\n", + "\n", + "The following query counts the number of connections:" ] }, { @@ -270,8 +276,29 @@ "metadata": {}, "outputs": [], "source": [ - "graph = Graph(model, directed=False, weighted=False, node_concept=Station)\n", - "model.define(graph.Edge.new(src=Station, dst=Station.connections))" + "# Count number of rows in the connections table:\n", + "model.select(aggregates.count(Connection)).to_df()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we can ask for a list of all the stations IDs:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "(\n", + " model.select(Station.id)\n", + " .to_df()\n", + " .sort_values(\"id\")\n", + " .reset_index(drop=True)\n", + ")" ] }, { @@ -298,9 +325,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The edge table shows which stations connect to which, but it doesn’t give an at-a-glance way to compare stations.\n", - "\n", - "**Betweenness centrality** is a common graph metric that assigns each station a score based on its position in the network. It’s available as a method on the `Graph` object:" + "The edge table shows which stations connect to which, but it doesn’t give an at-a-glance way to compare stations. The betweenness centrality scores defined above do:" ] }, { @@ -309,9 +334,6 @@ "metadata": {}, "outputs": [], "source": [ - "Node = graph.Node\n", - "Node.centrality = graph.betweenness_centrality()\n", - "\n", "(\n", " model.select(Node.id, Node.centrality.alias(\"betweenness_centrality\"))\n", " .to_df()\n", From 5da5b2a90174778bddae54cb67715bb5727e8817 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:24:22 -0800 Subject: [PATCH 028/162] financial_index_replication: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. Status and objective are read through the model's own relationships, and the script keeps accepting OPTIMAL or TIME_LIMIT. The README covers the deployment config, the internal schema grants and re-running; the docs sidecar anchors follow the configure call. --- financial_index_replication/README.md | 60 +++- .../financial_index_replication.py | 279 ++++++++++-------- financial_index_replication/runbook.md | 2 +- .../template-docs.yaml | 23 +- 4 files changed, 224 insertions(+), 140 deletions(-) diff --git a/financial_index_replication/README.md b/financial_index_replication/README.md index 314912ac..faf012b2 100644 --- a/financial_index_replication/README.md +++ b/financial_index_replication/README.md @@ -39,6 +39,15 @@ ask your RelationalAI support representative to enable it. Preview features are for evaluation and testing, not production applications. The template pins `relationalai==2.1.1` in `pyproject.toml`. +The script deploys the model to your Snowflake account, so an account +administrator must first complete the one-time model deployment setup there: +follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) +through **Governed setup**, **PAT authentication**, and steps 1-2 of +**Additional setup for the prescriptive reasoner** (step 4 below covers this +template's own schema). It gives you a model role, a programmatic access token +(PAT) stored as a secret, an external access integration, a PAT user, and a +warehouse. Your Snowflake user must hold the model role. + Use this sequence to run the bundled example: 1. **Download the template** @@ -70,7 +79,48 @@ Use this sequence to run the bundled example: 4. **Configure your project** - Use the configuration builder in [Start building with PyRel](https://docs.relational.ai/get-started/start-building-with-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + Use the configuration builder in [Start building with PyRel](https://docs.relational.ai/get-started/start-building-with-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. Set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: financial_index_replication_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .FINANCIAL_INDEX_REPLICATION # published views and tables + identity: + internal_schema: .FINANCIAL_INDEX_REPLICATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + + Without the `deployment` section, `python financial_index_replication.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .FINANCIAL_INDEX_REPLICATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + ``` + + The deploy creates the public schema `.FINANCIAL_INDEX_REPLICATION` if it doesn't exist. 5. **Run the template** @@ -80,6 +130,8 @@ Use this sequence to run the bundled example: python financial_index_replication.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + ```text Selected names: exactly 20 Status: OPTIMAL @@ -89,3 +141,9 @@ Use this sequence to run the bundled example: The solve selects exactly 20 names, though a selected stock may have zero weight. It writes `data/replica_returns.csv` for plotting. See `runbook.md` for the basket, sector exposures, tracking quality, and baseline comparison. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .FINANCIAL_INDEX_REPLICATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. diff --git a/financial_index_replication/financial_index_replication.py b/financial_index_replication/financial_index_replication.py index 3a21ef97..67534cbf 100644 --- a/financial_index_replication/financial_index_replication.py +++ b/financial_index_replication/financial_index_replication.py @@ -14,7 +14,9 @@ absolute tracking residuals. Run: - python financial_index_replication.py + `python financial_index_replication.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and prints + the report from the deployed model. Output: Prints the selected 20-stock basket, sector exposures, tracking quality @@ -36,7 +38,7 @@ stock_returns_csv, stocks_csv, ) -from relationalai.semantics import Float, sum +from relationalai.semantics import Float, String, sum from relationalai.semantics.reasoners.prescriptive import Problem # -------------------------------------------------- @@ -190,146 +192,163 @@ # Solve # -------------------------------------------------- -print("=" * 70) -print("FINANCIAL INDEX REPLICATION") -print("=" * 70) -print(f"Universe: {len(stocks_csv)} stocks") -print(f"Selected names: exactly {N_REPLICATION_NAMES}") -print(f"Max position: {MAX_WEIGHT:.0%}") -print(f"Sector active band: +/- {SECTOR_ACTIVE_BAND:.0%}") -print(f"Portfolio value: ${PORTFOLIO_VALUE:,.0f}") -print(f"Max ADV participation per name: {MAX_ADV_PARTICIPATION:.0%}") - -# Uncomment to print the formulated problem for debugging. -# problem.display() -problem.solve("highs", time_limit_sec=240) -si = problem.solve_info() -si.display() -assert si.termination_status == "OPTIMAL" or si.termination_status == "TIME_LIMIT", "Solver did not find an optimal solution." - -print(f"\nStatus: {si.termination_status}") -print(f"Objective: total absolute residual = {si.objective_value:.6f}") +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=240) + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"financial index replication solve status is {String:status}") +TotalResidual = model.Relationship( + f"financial index replication total absolute residual is {Float:residual}" +) +model.define(SolveStatus(problem.termination_status()), TotalResidual(problem.objective_value())) # -------------------------------------------------- -# Evaluate and report +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -solution_df = ( - model.select( - Stock.ticker.alias("ticker"), - Stock.sector.alias("sector"), - Stock.benchmark_weight.alias("benchmark_weight"), - Stock.previous_weight.alias("previous_weight"), - Stock.avg_dollar_volume.alias("avg_dollar_volume"), - selected.alias("selected"), - weight.alias("weight"), - ) - .where( - Stock.x_selected(selected), - Stock.x_weight(weight), - weight > 0.0001, +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + print("=" * 70) + print("FINANCIAL INDEX REPLICATION") + print("=" * 70) + print(f"Universe: {len(stocks_csv)} stocks") + print(f"Selected names: exactly {N_REPLICATION_NAMES}") + print(f"Max position: {MAX_WEIGHT:.0%}") + print(f"Sector active band: +/- {SECTOR_ACTIVE_BAND:.0%}") + print(f"Portfolio value: ${PORTFOLIO_VALUE:,.0f}") + print(f"Max ADV participation per name: {MAX_ADV_PARTICIPATION:.0%}") + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + residual = model.select(TotalResidual).to_df().iloc[:, 0].tolist() + assert status in (["OPTIMAL"], ["TIME_LIMIT"]), f"Solver did not find an optimal solution: {status}" + + print(f"\nStatus: {status[0]}") + print(f"Objective: total absolute residual = {residual[0]:.6f}") + + # -------------------------------------------------- + # Evaluate and report + # -------------------------------------------------- + + solution_df = ( + model.select( + Stock.ticker.alias("ticker"), + Stock.sector.alias("sector"), + Stock.benchmark_weight.alias("benchmark_weight"), + Stock.previous_weight.alias("previous_weight"), + Stock.avg_dollar_volume.alias("avg_dollar_volume"), + selected.alias("selected"), + weight.alias("weight"), + ) + .where( + Stock.x_selected(selected), + Stock.x_weight(weight), + weight > 0.0001, + ) + .to_df() + .sort_values("weight", ascending=False) ) - .to_df() - .sort_values("weight", ascending=False) -) -returns_wide = stock_returns_csv.pivot( - index="date", columns="ticker", values="return" -).sort_index() -index_series = index_returns_csv.set_index("date").sort_index() -weights = solution_df.set_index("ticker")["weight"] -portfolio_returns = returns_wide[weights.index].mul(weights, axis=1).sum(axis=1) -tracking_diff = portfolio_returns - index_series["index_return"] - -def annualized_tracking_error(): - monthly_rms = sqrt((tracking_diff ** 2).mean()) - return monthly_rms * sqrt(MONTHS_PER_YEAR) - -def mean_abs_tracking(): - return tracking_diff.abs().mean() - -print("\n=== Selected Replication Basket ===") -print( - solution_df[ - [ - "ticker", - "sector", - "weight", - "benchmark_weight", - "previous_weight", - "avg_dollar_volume", - ] - ].to_string( - index=False, - formatters={ - "weight": "{:.2%}".format, - "benchmark_weight": "{:.2%}".format, - "previous_weight": "{:.2%}".format, - "avg_dollar_volume": "${:,.0f}".format, - }, + returns_wide = stock_returns_csv.pivot( + index="date", columns="ticker", values="return" + ).sort_index() + index_series = index_returns_csv.set_index("date").sort_index() + weights = solution_df.set_index("ticker")["weight"] + portfolio_returns = returns_wide[weights.index].mul(weights, axis=1).sum(axis=1) + tracking_diff = portfolio_returns - index_series["index_return"] + + def annualized_tracking_error(): + monthly_rms = sqrt((tracking_diff ** 2).mean()) + return monthly_rms * sqrt(MONTHS_PER_YEAR) + + def mean_abs_tracking(): + return tracking_diff.abs().mean() + + print("\n=== Selected Replication Basket ===") + print( + solution_df[ + [ + "ticker", + "sector", + "weight", + "benchmark_weight", + "previous_weight", + "avg_dollar_volume", + ] + ].to_string( + index=False, + formatters={ + "weight": "{:.2%}".format, + "benchmark_weight": "{:.2%}".format, + "previous_weight": "{:.2%}".format, + "avg_dollar_volume": "${:,.0f}".format, + }, + ) ) -) -sector_report = ( - solution_df.groupby("sector", as_index=False)["weight"].sum() - .merge( - stocks_csv.groupby("sector", as_index=False)["benchmark_weight"].sum(), - on="sector", + sector_report = ( + solution_df.groupby("sector", as_index=False)["weight"].sum() + .merge( + stocks_csv.groupby("sector", as_index=False)["benchmark_weight"].sum(), + on="sector", + ) + ) + sector_report["active_weight"] = ( + sector_report["weight"] - sector_report["benchmark_weight"] ) -) -sector_report["active_weight"] = ( - sector_report["weight"] - sector_report["benchmark_weight"] -) -print("\n=== Sector Exposure ===") -print( - sector_report.sort_values("sector").to_string( - index=False, - formatters={ - "weight": "{:.2%}".format, - "benchmark_weight": "{:.2%}".format, - "active_weight": "{:+.2%}".format, - }, + print("\n=== Sector Exposure ===") + print( + sector_report.sort_values("sector").to_string( + index=False, + formatters={ + "weight": "{:.2%}".format, + "benchmark_weight": "{:.2%}".format, + "active_weight": "{:+.2%}".format, + }, + ) ) -) -turnover = ( - solution_df.set_index("ticker")["weight"] - .reindex(stocks_csv["ticker"], fill_value=0.0) - .sub(stocks_csv.set_index("ticker")["previous_weight"]) - .abs() - .sum() -) + turnover = ( + solution_df.set_index("ticker")["weight"] + .reindex(stocks_csv["ticker"], fill_value=0.0) + .sub(stocks_csv.set_index("ticker")["previous_weight"]) + .abs() + .sum() + ) -print("\n=== Tracking Quality ===") -print(f"Annualized tracking error: {annualized_tracking_error():.2%}") -print(f"Mean abs monthly residual: {mean_abs_tracking():.4%}") -print(f"Implied turnover: {turnover:.2%}") - -# Simple benchmark: equal-weight the 20 most correlated stocks. -top_corr_ids = ( - returns_wide - .corrwith(index_series["index_return"]) - .abs() - .sort_values(ascending=False) - .head(20) - .index -) -baseline_returns = returns_wide[top_corr_ids].mean(axis=1) -baseline_diff = baseline_returns - index_series["index_return"] + print("\n=== Tracking Quality ===") + print(f"Annualized tracking error: {annualized_tracking_error():.2%}") + print(f"Mean abs monthly residual: {mean_abs_tracking():.4%}") + print(f"Implied turnover: {turnover:.2%}") + + # Simple benchmark: equal-weight the 20 most correlated stocks. + top_corr_ids = ( + returns_wide + .corrwith(index_series["index_return"]) + .abs() + .sort_values(ascending=False) + .head(20) + .index + ) + baseline_returns = returns_wide[top_corr_ids].mean(axis=1) + baseline_diff = baseline_returns - index_series["index_return"] -baseline_tracking_error = sqrt((baseline_diff ** 2).mean()) * sqrt( - MONTHS_PER_YEAR -) + baseline_tracking_error = sqrt((baseline_diff ** 2).mean()) * sqrt( + MONTHS_PER_YEAR + ) -replica_returns_csv = DATA_DIR / "replica_returns.csv" -pd.DataFrame({ - "date": index_series.index, - "index_return": index_series["index_return"].values, - "replica_return": portfolio_returns.reindex(index_series.index).values, -}).to_csv(replica_returns_csv, index=False) - -print("\n=== Baseline Comparison ===") -print("Baseline: equal-weight top-20 stocks by full-history correlation") -print(f"Baseline annualized tracking error: {baseline_tracking_error:.2%}") -print(f"\nWrote benchmark-vs-replica returns to: {replica_returns_csv}") + replica_returns_csv = DATA_DIR / "replica_returns.csv" + pd.DataFrame({ + "date": index_series.index, + "index_return": index_series["index_return"].values, + "replica_return": portfolio_returns.reindex(index_series.index).values, + }).to_csv(replica_returns_csv, index=False) + + print("\n=== Baseline Comparison ===") + print("Baseline: equal-weight top-20 stocks by full-history correlation") + print(f"Baseline annualized tracking error: {baseline_tracking_error:.2%}") + print(f"\nWrote benchmark-vs-replica returns to: {replica_returns_csv}") diff --git a/financial_index_replication/runbook.md b/financial_index_replication/runbook.md index 7b5155e1..d502d2b2 100644 --- a/financial_index_replication/runbook.md +++ b/financial_index_replication/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Which 20 stocks, and at what weights, best replicate the benchmark's monthly returns? Select exactly 20 names (binary) and assign continuous weights that sum to 1, with at most 10% in any one name and a weight only if the name is selected; keep each sector within ±4% of its benchmark weight; respect a trading-capacity limit (the change from the previous weight, scaled by a $10 million portfolio value, can't exceed 5% of a stock's average dollar volume); and define each month's tracking residual as benchmark return minus basket return. Minimize the total absolute monthly tracking error, and persist the selection and weights to the ontology. -**Response:** `financial_index_replication.py` returns a 20-name basket. The decision is `Stock.x_selected` (exactly 20 names) plus `Stock.x_weight` (summing to 1) and per-month residual variables. This is a cardinality-constrained tracking MILP: depending on the solver it either solves to OPTIMAL or returns a strong feasible incumbent at the time limit — either way the exact names and weights can vary, so treat the tracking quality below, not a specific basket, as the result. +**Response:** `financial_index_replication.py` returns a 20-name basket. The decision is `Stock.x_selected` (exactly 20 names) plus `Stock.x_weight` (summing to 1) and per-month residual variables. This is a cardinality-constrained tracking MILP: depending on the solver it either solves to OPTIMAL or returns a strong feasible incumbent at the time limit — either way the exact names and weights can vary, so treat the tracking quality below, not a specific basket, as the result. In `financial_index_replication.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Assess tracking quality diff --git a/financial_index_replication/template-docs.yaml b/financial_index_replication/template-docs.yaml index 28d5ddf1..2dc4c529 100644 --- a/financial_index_replication/template-docs.yaml +++ b/financial_index_replication/template-docs.yaml @@ -282,10 +282,12 @@ guided_walkthrough: - id: solve-and-inspect-basket title: Choose the basket and check tracking explanation: |- - problem.solve(...) solves the decision problem under the portfolio - rules to choose stocks and weights. model.select(...) queries the model - for stocks with weights above 0.01% of the portfolio, so a selected - stock with zero weight does not appear in the result. + problem.configure(...) sets the solver and time limit that each refresh + of the deployed model uses when it solves the decision problem under + the portfolio rules. model.deploy() deploys the model and waits for the + first refresh, which chooses stocks and weights. model.select(...) + queries the model for stocks with weights above 0.01% of the portfolio, + so a selected stock with zero weight does not appear in the result. portfolio_returns calculates the basket’s monthly returns from those weights and stock returns, and tracking_diff compares them with the @@ -295,17 +297,22 @@ guided_walkthrough: focus: file: financial_index_replication.py anchors: - - start: problem.solve("highs", time_limit_sec=240) + - start: problem.configure("highs", time_limit_sec=240) end: return monthly_rms * sqrt(MONTHS_PER_YEAR) references: - problem.solve(...): + problem.configure(...): anchor: - start: problem.solve("highs", time_limit_sec=240) - end: problem.solve("highs", time_limit_sec=240) + start: problem.configure("highs", time_limit_sec=240) + end: problem.configure("highs", time_limit_sec=240) + model.deploy(): + anchor: + start: model.deploy() + end: model.deploy() model.select(...): anchor: start: model.select( end: .to_df() + occurrence: 3 portfolio_returns: anchor: start: returns_wide = stock_returns_csv.pivot( From ae173af974c06a16f8c54dd7ee913084450d6a05 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:27:18 -0800 Subject: [PATCH 029/162] cell_tower_coverage: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. Status and covered population are read through the model's own relationships, and the script keeps accepting OPTIMAL or TIME_LIMIT. The README covers the deployment config, the internal schema grants and re-running. --- cell_tower_coverage/README.md | 58 +++- cell_tower_coverage/cell_tower_coverage.py | 351 +++++++++++---------- cell_tower_coverage/runbook.md | 2 +- 3 files changed, 238 insertions(+), 173 deletions(-) diff --git a/cell_tower_coverage/README.md b/cell_tower_coverage/README.md index ee309257..4b0178d1 100644 --- a/cell_tower_coverage/README.md +++ b/cell_tower_coverage/README.md @@ -39,7 +39,7 @@ Telecommunications and infrastructure teams routinely have to decide where to bu ## What's included - **Model**: `cell_tower_coverage.py` builds the semantic model (three concepts, two coverage relationships), the mixed-integer optimization problem (three binary variable families, four constraints, one objective), the solve, and the reporting. -- **Runner**: `cell_tower_coverage.py` — a single Python script that runs end-to-end against a Snowflake-connected RAI account. +- **Runner**: `cell_tower_coverage.py` — a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: 6 candidate tower sites, 10 demand zones, and 19 feasible tower-zone coverage pairs. See *Sample data* below. - **Outputs**: stdout tables (selected sites, assigned zones, uncovered zones, coverage summary) plus `data/coverage_solution.csv`. @@ -49,7 +49,8 @@ Telecommunications and infrastructure teams routinely have to decide where to bu ### Access - A Snowflake account with the RelationalAI Native App installed. -- A Snowflake user with permissions to access the RelationalAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools @@ -83,18 +84,63 @@ Telecommunications and infrastructure teams routinely have to decide where to bu python -m pip install . ``` -4. Configure: +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: cell_tower_coverage_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .CELL_TOWER_COVERAGE # published views and tables + identity: + internal_schema: .CELL_TOWER_COVERAGE_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` - ```bash - rai init + Without the `deployment` section, `python cell_tower_coverage.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .CELL_TOWER_COVERAGE_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; ``` + The deploy creates the public schema `.CELL_TOWER_COVERAGE` if it doesn't exist. + 5. Run the template end-to-end: ```bash python cell_tower_coverage.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .CELL_TOWER_COVERAGE_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + 6. Expected output (a few lines confirm a successful run): ```text @@ -208,7 +254,7 @@ Focus on the first changes most users will make. - **Build budget** — `BUILD_BUDGET` (default `650,000`) caps total capital spend. - **Rollout size** — `MAX_NEW_TOWERS` (default `3`) caps how many sites the plan may select. -- **Solve budget** — `time_limit_sec` on the `problem.solve("highs", ...)` call (default `60` seconds). +- **Solve budget** — `time_limit_sec` on the `problem.configure("highs", ...)` call (default `60` seconds). ### Extend the model diff --git a/cell_tower_coverage/cell_tower_coverage.py b/cell_tower_coverage/cell_tower_coverage.py index cc12a5fc..e890ad2f 100644 --- a/cell_tower_coverage/cell_tower_coverage.py +++ b/cell_tower_coverage/cell_tower_coverage.py @@ -13,7 +13,9 @@ The optimization is modeled as a mixed-integer linear problem. Run: - python cell_tower_coverage.py + `python cell_tower_coverage.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from + the deployed model. Output: Prints the selected tower sites, assigned demand zones, tower utilization, @@ -201,188 +203,205 @@ # Solve # -------------------------------------------------- -print("=" * 70) -print("CELL TOWER COVERAGE") -print("=" * 70) -print(f"Candidate tower sites: {len(tower_sites_csv)}") -print(f"Demand zones: {len(demand_zones_csv)}") -print(f"Coverage pairs: {len(coverage_pairs_csv)}") -print(f"Build budget: ${BUILD_BUDGET:,.0f}") -print(f"Max new towers: {MAX_NEW_TOWERS}") - -# Uncomment to print the formulated problem for debugging. -# problem.display() -problem.solve("highs", time_limit_sec=60) -si = problem.solve_info() -si.display() -assert si.termination_status == "OPTIMAL" or si.termination_status == "TIME_LIMIT", "Solver did not find an optimal solution." - -print(f"\nStatus: {si.termination_status}") -print(f"Objective: covered population = {si.objective_value:,.0f}") +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"cell tower coverage solve status is {String:status}") +CoveredPopulation = model.Relationship( + f"cell tower coverage covered population is {Float:covered_population}" +) +model.define(SolveStatus(problem.termination_status()), CoveredPopulation(problem.objective_value())) # -------------------------------------------------- -# Evaluate and report +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -selected_sites_df = ( - model.select( - TowerSite.site_id.alias("site_id"), - TowerSite.name.alias("name"), - TowerSite.site_type.alias("site_type"), - TowerSite.region.alias("region"), - TowerSite.build_cost.alias("build_cost"), - TowerSite.capacity.alias("capacity"), - selected.alias("selected"), +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + print("=" * 70) + print("CELL TOWER COVERAGE") + print("=" * 70) + print(f"Candidate tower sites: {len(tower_sites_csv)}") + print(f"Demand zones: {len(demand_zones_csv)}") + print(f"Coverage pairs: {len(coverage_pairs_csv)}") + print(f"Build budget: ${BUILD_BUDGET:,.0f}") + print(f"Max new towers: {MAX_NEW_TOWERS}") + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + objective = model.select(CoveredPopulation).to_df().iloc[:, 0].tolist() + assert status in (["OPTIMAL"], ["TIME_LIMIT"]), f"Solver did not find an optimal solution: {status}" + + print(f"\nStatus: {status[0]}") + print(f"Objective: covered population = {objective[0]:,.0f}") + + # -------------------------------------------------- + # Evaluate and report + # -------------------------------------------------- + + selected_sites_df = ( + model.select( + TowerSite.site_id.alias("site_id"), + TowerSite.name.alias("name"), + TowerSite.site_type.alias("site_type"), + TowerSite.region.alias("region"), + TowerSite.build_cost.alias("build_cost"), + TowerSite.capacity.alias("capacity"), + selected.alias("selected"), + ) + .where( + TowerSite.x_selected(selected), + selected > 0.5, + ) + .to_df() + .sort_values(["region", "site_id"]) ) - .where( - TowerSite.x_selected(selected), - selected > 0.5, + selected_sites_df["build_cost"] = pd.to_numeric(selected_sites_df["build_cost"]) + selected_sites_df["capacity"] = pd.to_numeric(selected_sites_df["capacity"]) + selected_sites_df["selected"] = ( + pd.to_numeric(selected_sites_df["selected"]) > 0.5 + ).astype(int) + + zone_solution_df = ( + model.select( + DemandZone.zone_id.alias("zone_id"), + DemandZone.name.alias("name"), + DemandZone.region.alias("region"), + DemandZone.population.alias("population"), + covered.alias("covered"), + ) + .where(DemandZone.y_covered(covered)) + .to_df() + .sort_values(["covered", "population"], ascending=[False, False]) ) - .to_df() - .sort_values(["region", "site_id"]) -) -selected_sites_df["build_cost"] = pd.to_numeric(selected_sites_df["build_cost"]) -selected_sites_df["capacity"] = pd.to_numeric(selected_sites_df["capacity"]) -selected_sites_df["selected"] = ( - pd.to_numeric(selected_sites_df["selected"]) > 0.5 -).astype(int) - -zone_solution_df = ( - model.select( - DemandZone.zone_id.alias("zone_id"), - DemandZone.name.alias("name"), - DemandZone.region.alias("region"), - DemandZone.population.alias("population"), - covered.alias("covered"), + zone_solution_df["population"] = pd.to_numeric(zone_solution_df["population"]) + zone_solution_df["covered"] = ( + pd.to_numeric(zone_solution_df["covered"]) > 0.5 + ).astype(int) + + assignment_df = ( + model.select( + CoveragePair.site_id.alias("assigned_site_id"), + TowerSite.name.alias("assigned_site"), + CoveragePair.zone_id.alias("zone_id"), + assigned.alias("assigned"), + ) + .where( + CoveragePair.z_assigned(assigned), + CoveragePair.site(TowerSite), + assigned > 0.5, + ) + .to_df() + .sort_values(["assigned_site_id", "zone_id"]) ) - .where(DemandZone.y_covered(covered)) - .to_df() - .sort_values(["covered", "population"], ascending=[False, False]) -) -zone_solution_df["population"] = pd.to_numeric(zone_solution_df["population"]) -zone_solution_df["covered"] = ( - pd.to_numeric(zone_solution_df["covered"]) > 0.5 -).astype(int) - -assignment_df = ( - model.select( - CoveragePair.site_id.alias("assigned_site_id"), - TowerSite.name.alias("assigned_site"), - CoveragePair.zone_id.alias("zone_id"), - assigned.alias("assigned"), + assignment_df["assigned"] = ( + pd.to_numeric(assignment_df["assigned"]) > 0.5 + ).astype(int) + + zone_solution_df = zone_solution_df.merge( + assignment_df[["zone_id", "assigned_site_id", "assigned_site"]], + on="zone_id", + how="left", ) - .where( - CoveragePair.z_assigned(assigned), - CoveragePair.site(TowerSite), - assigned > 0.5, - ) - .to_df() - .sort_values(["assigned_site_id", "zone_id"]) -) -assignment_df["assigned"] = ( - pd.to_numeric(assignment_df["assigned"]) > 0.5 -).astype(int) - -zone_solution_df = zone_solution_df.merge( - assignment_df[["zone_id", "assigned_site_id", "assigned_site"]], - on="zone_id", - how="left", -) -zone_solution_df["assigned_site_id"] = zone_solution_df["assigned_site_id"].fillna("") -zone_solution_df["assigned_site"] = zone_solution_df["assigned_site"].fillna("") - -total_population = demand_zones_csv["population"].sum() -covered_population = zone_solution_df.loc[ - zone_solution_df["covered"] > 0.5, "population" -].sum() -uncovered_population = total_population - covered_population -selected_cost = selected_sites_df["build_cost"].sum() - -tower_load_df = ( - zone_solution_df[ - zone_solution_df["covered"] > 0.5 - ] - .groupby("assigned_site_id", as_index=False)["population"] - .sum() - .rename(columns={"population": "assigned_population"}) -) -selected_sites_df = selected_sites_df.merge( - tower_load_df, - left_on="site_id", - right_on="assigned_site_id", - how="left", -).drop(columns=["assigned_site_id"]) -selected_sites_df["assigned_population"] = ( - selected_sites_df["assigned_population"].fillna(0).astype(int) -) -selected_sites_df["utilization"] = ( - selected_sites_df["assigned_population"] / selected_sites_df["capacity"] -) - -print("\n=== Selected Tower Sites ===") -print( - selected_sites_df[ - [ - "site_id", - "name", - "site_type", - "region", - "build_cost", - "capacity", - "assigned_population", - "utilization", + zone_solution_df["assigned_site_id"] = zone_solution_df["assigned_site_id"].fillna("") + zone_solution_df["assigned_site"] = zone_solution_df["assigned_site"].fillna("") + + total_population = demand_zones_csv["population"].sum() + covered_population = zone_solution_df.loc[ + zone_solution_df["covered"] > 0.5, "population" + ].sum() + uncovered_population = total_population - covered_population + selected_cost = selected_sites_df["build_cost"].sum() + + tower_load_df = ( + zone_solution_df[ + zone_solution_df["covered"] > 0.5 ] - ].to_string( - index=False, - formatters={ - "build_cost": "${:,.0f}".format, - "capacity": "{:,.0f}".format, - "assigned_population": "{:,.0f}".format, - "utilization": "{:.1%}".format, - }, + .groupby("assigned_site_id", as_index=False)["population"] + .sum() + .rename(columns={"population": "assigned_population"}) + ) + selected_sites_df = selected_sites_df.merge( + tower_load_df, + left_on="site_id", + right_on="assigned_site_id", + how="left", + ).drop(columns=["assigned_site_id"]) + selected_sites_df["assigned_population"] = ( + selected_sites_df["assigned_population"].fillna(0).astype(int) + ) + selected_sites_df["utilization"] = ( + selected_sites_df["assigned_population"] / selected_sites_df["capacity"] ) -) -print("\n=== Assigned Demand Zones ===") -print( - zone_solution_df[ - zone_solution_df["covered"] > 0.5 - ][ - ["zone_id", "name", "region", "population", "assigned_site"] - ].to_string( - index=False, - formatters={"population": "{:,.0f}".format}, + print("\n=== Selected Tower Sites ===") + print( + selected_sites_df[ + [ + "site_id", + "name", + "site_type", + "region", + "build_cost", + "capacity", + "assigned_population", + "utilization", + ] + ].to_string( + index=False, + formatters={ + "build_cost": "${:,.0f}".format, + "capacity": "{:,.0f}".format, + "assigned_population": "{:,.0f}".format, + "utilization": "{:.1%}".format, + }, + ) ) -) -uncovered_zones_df = zone_solution_df[zone_solution_df["covered"] <= 0.5] -if not uncovered_zones_df.empty: - print("\n=== Uncovered Demand Zones ===") + print("\n=== Assigned Demand Zones ===") print( - uncovered_zones_df[ - ["zone_id", "name", "region", "population"] + zone_solution_df[ + zone_solution_df["covered"] > 0.5 + ][ + ["zone_id", "name", "region", "population", "assigned_site"] ].to_string( index=False, formatters={"population": "{:,.0f}".format}, ) ) -print("\n=== Coverage Summary ===") -print(f"Selected build cost: ${selected_cost:,.0f} of ${BUILD_BUDGET:,.0f}") -print(f"Covered population: {covered_population:,.0f} of {total_population:,.0f}") -print(f"Coverage rate: {covered_population / total_population:.2%}") -print(f"Uncovered population: {uncovered_population:,.0f}") - -coverage_solution_csv = DATA_DIR / "coverage_solution.csv" -zone_solution_df[ - [ - "zone_id", - "name", - "region", - "population", - "covered", - "assigned_site_id", - "assigned_site", - ] -].to_csv(coverage_solution_csv, index=False) -print(f"\nWrote coverage solution to: {coverage_solution_csv}") + uncovered_zones_df = zone_solution_df[zone_solution_df["covered"] <= 0.5] + if not uncovered_zones_df.empty: + print("\n=== Uncovered Demand Zones ===") + print( + uncovered_zones_df[ + ["zone_id", "name", "region", "population"] + ].to_string( + index=False, + formatters={"population": "{:,.0f}".format}, + ) + ) + + print("\n=== Coverage Summary ===") + print(f"Selected build cost: ${selected_cost:,.0f} of ${BUILD_BUDGET:,.0f}") + print(f"Covered population: {covered_population:,.0f} of {total_population:,.0f}") + print(f"Coverage rate: {covered_population / total_population:.2%}") + print(f"Uncovered population: {uncovered_population:,.0f}") + + coverage_solution_csv = DATA_DIR / "coverage_solution.csv" + zone_solution_df[ + [ + "zone_id", + "name", + "region", + "population", + "covered", + "assigned_site_id", + "assigned_site", + ] + ].to_csv(coverage_solution_csv, index=False) + print(f"\nWrote coverage solution to: {coverage_solution_csv}") diff --git a/cell_tower_coverage/runbook.md b/cell_tower_coverage/runbook.md index d141f97e..5ae930c5 100644 --- a/cell_tower_coverage/runbook.md +++ b/cell_tower_coverage/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Which sites should we build and which tower should serve each zone to cover the most people, building at most 3 towers within a $650,000 budget? Use a binary build decision per site, a binary covered decision per zone, and a binary serve decision per feasible site-zone pair. A zone counts as covered only if it's assigned to exactly one built tower, a tower can only serve zones if it's built, and the population a tower serves can't exceed its capacity. Maximize total covered population and persist the decisions to the ontology. -**Response:** OPTIMAL (HiGHS), covered population **90,000 of 102,900** (relative gap 0.0). The model has 3 binary decision families and 5 constraint families; build, cover, and serve decisions are written back as `TowerSite.x_selected`, `DemandZone.y_covered`, and `CoveragePair.z_assigned`. +**Response:** OPTIMAL (HiGHS), covered population **90,000 of 102,900** (relative gap 0.0). The model has 3 binary decision families and 5 constraint families; build, cover, and serve decisions are written back as `TowerSite.x_selected`, `DemandZone.y_covered`, and `CoveragePair.z_assigned`. In `cell_tower_coverage.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the build plan From 0de996b7044fc89e1304d5d252d95630e6a21364 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:29:49 -0800 Subject: [PATCH 030/162] test_data_generation: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. Status and total deviation are read through the model's own relationships, and the model.require status check becomes a Python check in the report. problem.display() is installed with the model. The README covers the deployment config, the internal schema grants and re-running. --- test_data_generation/README.md | 54 ++++++++++++- test_data_generation/test_data_generation.py | 83 +++++++++++++------- 2 files changed, 104 insertions(+), 33 deletions(-) diff --git a/test_data_generation/README.md b/test_data_generation/README.md index 56af106f..48a1e62f 100644 --- a/test_data_generation/README.md +++ b/test_data_generation/README.md @@ -47,7 +47,8 @@ After determining row counts, the script generates actual test data records with ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -75,15 +76,60 @@ After determining row counts, the script generates actual test data records with python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: test_data_generation_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .TEST_DATA_GENERATION # published views and tables + identity: + internal_schema: .TEST_DATA_GENERATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python test_data_generation.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .TEST_DATA_GENERATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + ``` + + The deploy creates the public schema `.TEST_DATA_GENERATION` if it doesn't exist. + 5. Run: ```bash python test_data_generation.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .TEST_DATA_GENERATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output: ```text diff --git a/test_data_generation/test_data_generation.py b/test_data_generation/test_data_generation.py index 17c7f0e0..112ad74f 100644 --- a/test_data_generation/test_data_generation.py +++ b/test_data_generation/test_data_generation.py @@ -10,7 +10,9 @@ - Generate synthetic records consistent with the solved row counts. Run: - `python test_data_generation.py` + `python test_data_generation.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from + the deployed model. Output: Prints the solver status, total weighted deviation, optimal row counts per @@ -189,30 +191,21 @@ problem.minimize(total_deviation) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- -problem.display() -problem.solve("highs", time_limit_sec=60) -model.require(problem.termination_status() == "OPTIMAL") -si = problem.solve_info() -si.display() +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=60) -print(f"Status: {si.termination_status}") -print(f"Total weighted deviation: {si.objective_value:.2f}") +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() -# Extract row counts -row_counts = {} -results_df = model.select(Table.table_name, Table.x_actual_rows, Table.target_rows).to_df() -for _, row in results_df.iterrows(): - actual = int(round(row['actual_rows'])) - target = int(row['target_rows']) - row_counts[row['table_name']] = {'actual': actual, 'target': target} - -print("\nOptimal row counts:") -for table, counts in row_counts.items(): - dev = abs(counts['actual'] - counts['target']) - print(f" {table}: {counts['actual']} rows (target: {counts['target']}, deviation: {dev})") +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"test data generation solve status is {String:status}") +TotalDeviation = model.Relationship( + f"test data generation total weighted deviation is {Float:total_deviation}" +) +model.define(SolveStatus(problem.termination_status()), TotalDeviation(problem.objective_value())) # -------------------------------------------------- # Phase 2: Generate actual test data (optional) @@ -273,12 +266,44 @@ def random_float(min_val, max_val): return generated -# Generate data and show samples -print("\n" + "=" * 50) -print("GENERATED TEST DATA") -print("=" * 50) +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- -generated_data = generate_test_data(row_counts) -for table, df in generated_data.items(): - print(f"\n{table}: {len(df)} rows") - print(df.head(3).to_string(index=False)) +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + deviation = model.select(TotalDeviation).to_df().iloc[:, 0].tolist() + assert status == ["OPTIMAL"], f"termination status {status}" + + print(f"Status: {status[0]}") + print(f"Total weighted deviation: {deviation[0]:.2f}") + + # Extract row counts + row_counts = {} + results_df = model.select(Table.table_name, Table.x_actual_rows, Table.target_rows).to_df() + for _, row in results_df.iterrows(): + actual = int(round(row['actual_rows'])) + target = int(row['target_rows']) + row_counts[row['table_name']] = {'actual': actual, 'target': target} + + print("\nOptimal row counts:") + for table, counts in row_counts.items(): + dev = abs(counts['actual'] - counts['target']) + print(f" {table}: {counts['actual']} rows (target: {counts['target']}, deviation: {dev})") + + # Generate data and show samples + print("\n" + "=" * 50) + print("GENERATED TEST DATA") + print("=" * 50) + + generated_data = generate_test_data(row_counts) + for table, df in generated_data.items(): + print(f"\n{table}: {len(df)} rows") + print(df.head(3).to_string(index=False)) From 5400728fe23c0289ad01092f8b9a29798179e49a Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:32:41 -0800 Subject: [PATCH 031/162] traveling_salesman: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. Status and tour distance are read through the model's own relationships. The model.require checks become Python checks in the report: the problem's variable and objective counts are read back and compared with the data, and the tour must leave every node exactly once. problem.display() is installed with the model. The README covers the deployment config, the internal schema grants and re-running. --- traveling_salesman/README.md | 56 +++++++++++++++++++-- traveling_salesman/runbook.md | 2 +- traveling_salesman/traveling_salesman.py | 62 +++++++++++++++++------- 3 files changed, 96 insertions(+), 24 deletions(-) diff --git a/traveling_salesman/README.md b/traveling_salesman/README.md index a4b6b6e8..fd589a53 100644 --- a/traveling_salesman/README.md +++ b/traveling_salesman/README.md @@ -48,7 +48,8 @@ Built using **prescriptive reasoning** (mixed-integer programming with the HiGHS ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -77,15 +78,60 @@ Built using **prescriptive reasoning** (mixed-integer programming with the HiGHS python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: traveling_salesman_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .TRAVELING_SALESMAN # published views and tables + identity: + internal_schema: .TRAVELING_SALESMAN_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python traveling_salesman.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .TRAVELING_SALESMAN_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + ``` + + The deploy creates the public schema `.TRAVELING_SALESMAN` if it doesn't exist. + 5. Run: ```bash python traveling_salesman.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .TRAVELING_SALESMAN_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output: ```text @@ -148,7 +194,7 @@ See `traveling_salesman.py` for the implementation, and `runbook.md` to reproduc ### Tune parameters -- The solve time cap is `time_limit_sec` (default 60) in the `problem.solve("highs", ...)` call. Lower it to accept a near-optimal tour sooner on larger instances. +- The solve time cap is `time_limit_sec` (default 60) in the `problem.configure("highs", ...)` call. Lower it to accept a near-optimal tour sooner on larger instances. ### Extend the model diff --git a/traveling_salesman/runbook.md b/traveling_salesman/runbook.md index b10eb956..4ba7c1fa 100644 --- a/traveling_salesman/runbook.md +++ b/traveling_salesman/runbook.md @@ -37,7 +37,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem What's the shortest round-trip that visits every node exactly once and returns to the start, as one connected tour with no disconnected subtours? Use a binary decision per edge for whether it's in the tour, require every node to have exactly one selected edge in and one out, and eliminate subtours with MTZ ordering variables — one integer position per node, anchored at node 1. Minimize total distance, and persist the edge selection to the ontology. -**Response:** OPTIMAL (HiGHS mixed-integer program), shortest tour distance **8.5** (relative gap 0.0). The model has 16 variables (12 binary edge selections + 4 integer MTZ positions) and 15 constraints; the edge decision is written back as `Edge.x` and the position as `Node.u`. +**Response:** OPTIMAL (HiGHS mixed-integer program), shortest tour distance **8.5** (relative gap 0.0). The model has 16 variables (12 binary edge selections + 4 integer MTZ positions) and 15 constraints; the edge decision is written back as `Edge.x` and the position as `Node.u`. In `traveling_salesman.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the optimal route diff --git a/traveling_salesman/traveling_salesman.py b/traveling_salesman/traveling_salesman.py index c6e1686b..43f70649 100644 --- a/traveling_salesman/traveling_salesman.py +++ b/traveling_salesman/traveling_salesman.py @@ -9,7 +9,9 @@ - Solve for the shortest Hamiltonian cycle and print the selected edges. Run: - `python traveling_salesman.py` + `python traveling_salesman.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from + the deployed model. Output: Prints the solver termination status, objective value (shortest tour @@ -19,7 +21,7 @@ from pathlib import Path from pandas import read_csv -from relationalai.semantics import Float, Integer, Model, count, sum +from relationalai.semantics import Float, Integer, Model, String, count, sum from relationalai.semantics.reasoners.prescriptive import Problem # -------------------------------------------------- @@ -91,21 +93,45 @@ )) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- -# Check model structure — engine-side ICs avoid querying data to the client. -model.require(problem.num_variables() == count(Edge) + count(Node)) -model.require(problem.num_min_objectives() == 1) -problem.display() - -# Solve with HiGHS (MILP branch-and-bound) -problem.solve("highs", time_limit_sec=60) -si_highs = problem.solve_info() -si_highs.display() -print(f"HiGHS: {si_highs.termination_status}, tour distance={si_highs.objective_value:.1f}") - -# Extract solution -model.require(count(Edge).where(Edge.x > 0.5) == count(Node)) -print("\nTour edges:") -model.select(Edge.i.alias("from"), Edge.j.alias("to")).where(Edge.x > 0.5).inspect() +# Each refresh of the deployed model runs this solve (HiGHS MILP branch-and-bound). +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, and the problem's size, published through the model. +SolveStatus = model.Relationship(f"tsp solve status is {String:status}") +TourDistance = model.Relationship(f"tsp tour distance is {Float:distance}") +model.define(SolveStatus(problem.termination_status()), TourDistance(problem.objective_value())) +num_variables = problem.num_variables() +num_min_objectives = problem.num_min_objectives() + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + # Check model structure: one variable per edge and per node, one minimization objective. + nodes = sorted(edge_csv["i"].unique().tolist()) + variables = model.select(num_variables).to_df().iloc[:, 0].tolist() + assert variables == [len(edge_csv) + len(nodes)], f"expected one variable per edge and node, got {variables}" + objectives = model.select(num_min_objectives).to_df().iloc[:, 0].tolist() + assert objectives == [1], f"expected one minimization objective, got {objectives}" + problem.display() + + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + distance = model.select(TourDistance).to_df().iloc[:, 0].tolist() + print(f"HiGHS: {status[0]}, tour distance={distance[0]:.1f}") + + # Extract solution: the tour leaves every node exactly once. + tour = model.select(Edge.i.alias("from"), Edge.j.alias("to")).where(Edge.x > 0.5).to_df() + assert sorted(tour["from"].tolist()) == nodes, f"tour does not leave every node exactly once: {tour}" + print("\nTour edges:") + print(tour) + print() From f9dd30bf7ffb4640319469251d835363b3bce2cb Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:35:27 -0800 Subject: [PATCH 032/162] water_allocation: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. Status and total cost are read through the model's own relationships and printed as before. problem.display() is installed with the model. The README covers the deployment config, the internal schema grants and re-running. --- water_allocation/README.md | 56 +++++++++++++++++++++++++--- water_allocation/runbook.md | 2 +- water_allocation/water_allocation.py | 48 +++++++++++++++++------- 3 files changed, 86 insertions(+), 20 deletions(-) diff --git a/water_allocation/README.md b/water_allocation/README.md index f7145cdb..253e75ba 100644 --- a/water_allocation/README.md +++ b/water_allocation/README.md @@ -38,7 +38,7 @@ Built using **prescriptive reasoning** (nonlinear program over continuous flow v ## What's included - **Model**: a single-stage nonlinear optimization on a shared ontology — `Source`, `User`, and `Connection` concepts wired to the bundled CSVs. -- **Runner**: `water_allocation.py` — a single Python script that builds the model, constraints, and objective and calls the solver against a Snowflake-connected RAI account. +- **Runner**: `water_allocation.py` — a single Python script that builds the model, constraints, and objective, deploys the model to your Snowflake account, where each refresh calls the solver, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: a small water-distribution network of sources, user groups, and the connections between them. See *Sample data* below. - **Outputs**: solver status, total cost, and the per-connection flow allocations. @@ -47,7 +47,8 @@ Built using **prescriptive reasoning** (nonlinear program over continuous flow v ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10. @@ -76,15 +77,60 @@ Built using **prescriptive reasoning** (nonlinear program over continuous flow v python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: water_allocation_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .WATER_ALLOCATION # published views and tables + identity: + internal_schema: .WATER_ALLOCATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python water_allocation.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .WATER_ALLOCATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + ``` + + The deploy creates the public schema `.WATER_ALLOCATION` if it doesn't exist. + 5. Run: ```bash python water_allocation.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```sql + CALL .WATER_ALLOCATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (a few lines confirm a successful run): ```text diff --git a/water_allocation/runbook.md b/water_allocation/runbook.md index a1b3213a..8d30455b 100644 --- a/water_allocation/runbook.md +++ b/water_allocation/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem What non-negative flow on each connection meets every user's demand at minimum total sourcing cost, given that each source's total outflow can't exceed its capacity and that transmission loss grows with utilization — the water delivered on a connection is flow times (1 minus loss_rate times flow over max_flow)? That delivered-water term is quadratic, so this needs a nonlinear solver. Persist the chosen flows to the ontology. -**Response:** LOCALLY_SOLVED (Ipopt nonlinear solver), total cost **$853.39**. 9 continuous flow variables (each bounded by its connection's max flow), 6 constraints (3 source-capacity, 3 nonlinear demand-satisfaction). Flows are written back as `Connection.x_flow`. +**Response:** LOCALLY_SOLVED (Ipopt nonlinear solver), total cost **$853.39**. 9 continuous flow variables (each bounded by its connection's max flow), 6 constraints (3 source-capacity, 3 nonlinear demand-satisfaction). Flows are written back as `Connection.x_flow`. In `water_allocation.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the allocation diff --git a/water_allocation/water_allocation.py b/water_allocation/water_allocation.py index 9cc6b2f2..e5d4aced 100644 --- a/water_allocation/water_allocation.py +++ b/water_allocation/water_allocation.py @@ -16,7 +16,9 @@ optimal solutions for smooth NLP problems with continuous variables. Run: - `python water_allocation.py` + `python water_allocation.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from + the deployed model. Output: Prints the solver termination status, objective value, and a table of @@ -110,20 +112,38 @@ problem.minimize(total_cost) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- -problem.display() -problem.solve("ipopt", time_limit_sec=60) -si = problem.solve_info() -si.display() +# Each refresh of the deployed model runs this solve. +problem.configure("ipopt", time_limit_sec=60) -print(f"Status: {si.termination_status}") # Ipopt returns LOCALLY_SOLVED -print(f"Total cost: ${si.objective_value:.2f}") +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() -print("\nFlow allocations:") -model.select( - Connection.source.name.alias("source"), - Connection.user.name.alias("user"), - Connection.x_flow -).where(Connection.x_flow > 0.001).inspect() +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"water allocation solve status is {String:status}") +TotalCost = model.Relationship(f"water allocation total cost is {Float:cost}") +model.define(SolveStatus(problem.termination_status()), TotalCost(problem.objective_value())) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- + +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() + print(f"Status: {status[0]}") # Ipopt returns LOCALLY_SOLVED + print(f"Total cost: ${cost[0]:.2f}") + + print("\nFlow allocations:") + model.select( + Connection.source.name.alias("source"), + Connection.user.name.alias("user"), + Connection.x_flow + ).where(Connection.x_flow > 0.001).inspect() From df5e0c68d24c2635ef0ca68c388b2190107593ab Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 17:04:27 -0800 Subject: [PATCH 033/162] warehouse_allocation: run as a deployed model The centrality, connected-component and cross-region route rules are declared with the model, so each refresh runs the graph analysis and then the solve declared with problem.configure. Status and holding cost are read through the model's own relationships, problem.display() is installed with the model, and the script prints every stage from the deployed model. The README covers the deployment config and the internal schema grants. --- warehouse_allocation/README.md | 50 +++++- warehouse_allocation/runbook.md | 2 +- warehouse_allocation/warehouse_allocation.py | 170 +++++++++++-------- 3 files changed, 143 insertions(+), 79 deletions(-) diff --git a/warehouse_allocation/README.md b/warehouse_allocation/README.md index 479e4d36..a3b15c3b 100644 --- a/warehouse_allocation/README.md +++ b/warehouse_allocation/README.md @@ -41,7 +41,7 @@ Built using **graph analysis** (eigenvector centrality, weakly connected compone ## What's included - **Model**: a two-stage pipeline (graph, then prescriptive) on a single shared ontology — `Site`, `Route`, and `Demand` concepts wired to the bundled CSVs, plus the centrality enrichment Stage 1 writes back. -- **Runner**: `warehouse_allocation.py` — a single Python script that runs both stages end-to-end against a Snowflake-connected RAI account. +- **Runner**: `warehouse_allocation.py` — a single Python script that deploys the model to your Snowflake account, where each refresh runs both stages, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: a small distribution network of warehouses and stores with routes and per-site demand. See *Sample data* below. - **Outputs**: stdout diagnostics (centrality ranking, connected-component summary, bridge routes, solver status) plus the inventory allocation plan as queryable `Site.x_inventory`. @@ -50,7 +50,8 @@ Built using **graph analysis** (eigenvector centrality, weakly connected compone ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10. @@ -79,15 +80,54 @@ Built using **graph analysis** (eigenvector centrality, weakly connected compone python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + + ```yaml + reasoners: + prescriptive: + name: warehouse_allocation_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .WAREHOUSE_ALLOCATION # published views and tables + identity: + internal_schema: .WAREHOUSE_ALLOCATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: ``` + Without the `deployment` section, `python warehouse_allocation.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .WAREHOUSE_ALLOCATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .WAREHOUSE_ALLOCATION_META TO APPLICATION ; + ``` + + The deploy creates the public schema `.WAREHOUSE_ALLOCATION` if it doesn't exist. + 5. Run: ```bash python warehouse_allocation.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. 6. Expected output (a few lines confirm a successful run): ```text diff --git a/warehouse_allocation/runbook.md b/warehouse_allocation/runbook.md index 9c529b44..7fc6c1d7 100644 --- a/warehouse_allocation/runbook.md +++ b/warehouse_allocation/runbook.md @@ -86,7 +86,7 @@ Eigenvector centrality (the "important neighbors make you important" measure) ra **Response** -OPTIMAL (HiGHS), total holding cost **$13,713.29**. 10 continuous allocation variables, 14 constraints (1 budget cap, 8 demand floors, 5 warehouse centrality floors); `Site.x_inventory` written back. +OPTIMAL (HiGHS), total holding cost **$13,713.29**. 10 continuous allocation variables, 14 constraints (1 budget cap, 8 demand floors, 5 warehouse centrality floors); `Site.x_inventory` written back. In `warehouse_allocation.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs the graph analysis, then the solve. ### 6. Read the allocation diff --git a/warehouse_allocation/warehouse_allocation.py b/warehouse_allocation/warehouse_allocation.py index cbe89d42..f674798a 100644 --- a/warehouse_allocation/warehouse_allocation.py +++ b/warehouse_allocation/warehouse_allocation.py @@ -21,7 +21,9 @@ data transfer between stages. Run: - `python warehouse_allocation.py` + `python warehouse_allocation.py` deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the graph analysis and the solve), and + prints the report from the deployed model. Output: Prints centrality ranking, weakly-connected-component summary, bridge @@ -106,51 +108,11 @@ # Eigenvector centrality: stored directly on Site (via node_concept) Site.centrality = graph.eigenvector_centrality() -# Display centrality ranking -print("=" * 50) -print("Stage 1a: Eigenvector Centrality") -print("=" * 50) - -model.select( - Site.name.alias("site"), - Site.type.alias("type"), - Site.region.alias("region"), - Site.centrality.alias("centrality"), -).inspect() - # -------------------------------------------------- # Stage 1b: Weakly Connected Components -- surface fragmentation # -------------------------------------------------- wcc = graph.weakly_connected_component() -site_ref = graph.Node.ref("site") -comp_ref = graph.Node.ref("component_id") - -wcc_df = ( - where(wcc(site_ref, comp_ref)) - .select( - site_ref.id.alias("site_id"), - site_ref.name.alias("site"), - site_ref.region.alias("region"), - comp_ref.id.alias("component_id"), - aggs.count(site_ref).per(comp_ref).alias("component_size"), - ) - .to_df() -) - -print("\n" + "=" * 50) -print("Stage 1b: Connected Components") -print("=" * 50) -num_components = wcc_df["component_id"].nunique() -if num_components == 1: - print(f"UNIFIED NETWORK: all {len(wcc_df)} sites in a single component") -else: - print(f"FRAGMENTED NETWORK: {num_components} separate components detected") - for comp_id in sorted(wcc_df["component_id"].unique()): - comp_df = wcc_df[wcc_df["component_id"] == comp_id] - size = int(comp_df["component_size"].iloc[0]) - regions = ", ".join(sorted(comp_df["region"].unique())) - print(f" Component {comp_id}: {size} sites ({regions})") # -------------------------------------------------- # Stage 1c: Bridge Routes -- cross-region connectors @@ -167,21 +129,6 @@ src.region != dst.region, ).define(Route.is_cross_region(route_ref, 1.0)) -bridge_df = model.select( - Route.id.alias("route_id"), - Route.source.name.alias("from_site"), - Route.source.region.alias("from_region"), - Route.dest.name.alias("to_site"), - Route.dest.region.alias("to_region"), -).where(Route.is_cross_region == 1.0).to_df() - -print("\n" + "=" * 50) -print(f"Stage 1c: Bridge Routes ({len(bridge_df)} cross-region connectors)") -print("=" * 50) -if not bridge_df.empty: - for _, row in bridge_df.iterrows(): - print(f" {row['from_site']} ({row['from_region']}) -> {row['to_site']} ({row['to_region']})") - # -------------------------------------------------- # Stage 2: Prescriptive -- allocate inventory across sites # -------------------------------------------------- @@ -218,23 +165,100 @@ # Solve # -------------------------------------------------- -print("\n" + "=" * 50) -print("Stage 2: Inventory Allocation") -print("=" * 50) +# Each refresh of the deployed model runs the graph analysis, then this solve. +problem.configure("highs", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = Relationship(f"warehouse allocation solve status is {String:status}") +TotalHoldingCost = Relationship(f"warehouse allocation total holding cost is {Float:holding_cost}") +model.define(SolveStatus(problem.termination_status()), TotalHoldingCost(problem.objective_value())) -problem.display() -problem.solve("highs", time_limit_sec=60) -si = problem.solve_info() -si.display() +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- -print(f"\nStatus: {si.termination_status}") -print(f"Total holding cost: ${si.objective_value:.2f}") +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the graph analysis and the solve. + model.deploy() + + # Display centrality ranking + print("=" * 50) + print("Stage 1a: Eigenvector Centrality") + print("=" * 50) + + model.select( + Site.name.alias("site"), + Site.type.alias("type"), + Site.region.alias("region"), + Site.centrality.alias("centrality"), + ).inspect() + + # Stage 1b: connected components + site_ref = graph.Node.ref("site") + comp_ref = graph.Node.ref("component_id") + + wcc_df = ( + where(wcc(site_ref, comp_ref)) + .select( + site_ref.id.alias("site_id"), + site_ref.name.alias("site"), + site_ref.region.alias("region"), + comp_ref.id.alias("component_id"), + aggs.count(site_ref).per(comp_ref).alias("component_size"), + ) + .to_df() + ) -print("\nAllocation plan:") -model.select( - Site.name.alias("site"), - Site.type.alias("type"), - Site.centrality.alias("centrality"), - Site.x_inventory.alias("allocated"), - Site.holding_cost.alias("unit_cost"), -).where(Site.x_inventory > 0.01).inspect() + print("\n" + "=" * 50) + print("Stage 1b: Connected Components") + print("=" * 50) + num_components = wcc_df["component_id"].nunique() + if num_components == 1: + print(f"UNIFIED NETWORK: all {len(wcc_df)} sites in a single component") + else: + print(f"FRAGMENTED NETWORK: {num_components} separate components detected") + for comp_id in sorted(wcc_df["component_id"].unique()): + comp_df = wcc_df[wcc_df["component_id"] == comp_id] + size = int(comp_df["component_size"].iloc[0]) + regions = ", ".join(sorted(comp_df["region"].unique())) + print(f" Component {comp_id}: {size} sites ({regions})") + + # Stage 1c: bridge routes + bridge_df = model.select( + Route.id.alias("route_id"), + Route.source.name.alias("from_site"), + Route.source.region.alias("from_region"), + Route.dest.name.alias("to_site"), + Route.dest.region.alias("to_region"), + ).where(Route.is_cross_region == 1.0).to_df() + + print("\n" + "=" * 50) + print(f"Stage 1c: Bridge Routes ({len(bridge_df)} cross-region connectors)") + print("=" * 50) + if not bridge_df.empty: + for _, row in bridge_df.iterrows(): + print(f" {row['from_site']} ({row['from_region']}) -> {row['to_site']} ({row['to_region']})") + + # Stage 2: allocation + print("\n" + "=" * 50) + print("Stage 2: Inventory Allocation") + print("=" * 50) + + problem.display() + + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + holding_cost = model.select(TotalHoldingCost).to_df().iloc[:, 0].tolist() + print(f"\nStatus: {status[0]}") + print(f"Total holding cost: ${holding_cost[0]:.2f}") + + print("\nAllocation plan:") + model.select( + Site.name.alias("site"), + Site.type.alias("type"), + Site.centrality.alias("centrality"), + Site.x_inventory.alias("allocated"), + Site.holding_cost.alias("unit_cost"), + ).where(Site.x_inventory > 0.01).inspect() From 991a7e85fad3a9a04b87a2eea67a73a875b28156 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 17:07:28 -0800 Subject: [PATCH 034/162] cybersecurity-attack-paths, it-dependency-mapping: name the release limit Say that the Property-from-path planning failure is a relationalai 2.1.1 limit, so the Relationship workaround can be revisited on a later release. --- cybersecurity-attack-paths/cybersecurity_attack_paths.py | 4 ++-- it-dependency-mapping/it_dependency_mapping.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cybersecurity-attack-paths/cybersecurity_attack_paths.py b/cybersecurity-attack-paths/cybersecurity_attack_paths.py index 02032513..09fe9904 100644 --- a/cybersecurity-attack-paths/cybersecurity_attack_paths.py +++ b/cybersecurity-attack-paths/cybersecurity_attack_paths.py @@ -130,8 +130,8 @@ def load_csv(filename): # Asset.on_attack_path marks the assets that lie on a crown-jewel kill-chain, so a # downstream query can pull them without re-enumerating paths. It is a -# Relationship, not a Property: a Property defined from a path fails deployment -# planning. +# Relationship, not a Property: on relationalai 2.1.1, deploying a Property +# written from a path rule fails planning. Asset.on_attack_path = model.Relationship(f"{Asset} on attack path {String:on_attack_path}") on_path, pos = Asset.ref(), Integer.ref() model.where( diff --git a/it-dependency-mapping/it_dependency_mapping.py b/it-dependency-mapping/it_dependency_mapping.py index 1a970705..08d0646a 100644 --- a/it-dependency-mapping/it_dependency_mapping.py +++ b/it-dependency-mapping/it_dependency_mapping.py @@ -100,8 +100,8 @@ def load_csv(path): # aggregates.max(p.length).per(src) takes the max hop count over the paths rooted # at each src and defines it directly onto Feature -- no pandas groupby and no # model.data round-trip. Kept SEPARATE from the node projection. It is a -# Relationship, not a Property: a Property defined from a path fails deployment -# planning. +# Relationship, not a Property: on relationalai 2.1.1, deploying a Property +# written from a path rule fails planning. Feature.max_downstream_depth = model.Relationship( f"{Feature} has {Integer:max_downstream_depth}" ) From 6118c9ca63a80651646fe30ba4259789521eed0d Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:06:49 -0800 Subject: [PATCH 035/162] disease-outbreak-prevention: run as a deployed model The script deploys the model, waits for the first refresh, which computes the graph metrics, and prints the report from the deployed model. The Streamlit app and model_setup.py are unchanged. The README covers the deployment config, running again, and teardown. --- disease-outbreak-prevention/README.md | 27 ++++++++++++++----- .../disease_outbreak_prevention_network.py | 8 ++++-- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/disease-outbreak-prevention/README.md b/disease-outbreak-prevention/README.md index 503d927d..5f10755d 100644 --- a/disease-outbreak-prevention/README.md +++ b/disease-outbreak-prevention/README.md @@ -39,7 +39,7 @@ Built using **graph analysis** (a directed, weighted graph with `Facility` as th ## What's included - **Model**: shared setup in `model_setup.py` — the `Facility` concept, the `FacilityConnection` edge concept (with `transfer_volume`, `contact_intensity`, and a derived `risk_weight`), and the directed weighted `Graph` built from them. -- **Runner**: `disease_outbreak_prevention_network.py` (command-line analysis with detailed output) and `app.py` (optional Streamlit web app), both driven by the same shared model. +- **Runner**: `disease_outbreak_prevention_network.py` (command-line analysis with detailed output, run from a model deployed to your Snowflake account) and `app.py` (optional Streamlit web app), both driven by the same shared model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/facilities.csv` and `data/connections.csv`. - **Outputs**: a ranked facility table, a top-priority breakdown, and network summary statistics printed to stdout; the Streamlit app adds an interactive network graph and CSV export. @@ -49,7 +49,8 @@ Built using **graph analysis** (a directed, weighted graph with `Facility` as th ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -89,12 +90,24 @@ You can customize the data and model as needed after you have it running end-to- python -m pip install . ``` -4. **Configure Snowflake connection and RAI profile** +4. **Configure the Snowflake connection and the deployment** - ```bash - rai init + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .DISEASE_OUTBREAK_PREVENTION # published views and tables + identity: + internal_schema: .DISEASE_OUTBREAK_PREVENTION_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python disease_outbreak_prevention_network.py` stops with an error naming the missing setting. + 5. **Run the template** **Option A: Command-line script** @@ -103,6 +116,8 @@ You can customize the data and model as needed after you have it running end-to- python disease_outbreak_prevention_network.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. Running the script again without changing the model redeploys nothing and does not refresh the model again; it reports the latest refresh. A change to the model redeploys, and the redeploy's refresh recomputes the results. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + **Option B: Interactive Streamlit app** ```bash @@ -179,7 +194,7 @@ CSV files → model_setup.create_model() → weighted centrality + degree metric 3. **Query and rank.** A single query pulls each facility's identity, type, region, centrality score, and in/out connection counts, then sorts descending by centrality to produce the ranked shortlist. -4. **Present results.** The CLI script prints the ranked table, a top-priority breakdown, network summary statistics, and response recommendations. The optional Streamlit app renders the same ranking as an interactive network graph with a filterable table and CSV export. +4. **Present results.** The CLI script deploys the model, waits for its first refresh, and prints the ranked table, a top-priority breakdown, network summary statistics, and response recommendations. The optional Streamlit app renders the same ranking as an interactive network graph with a filterable table and CSV export. See `model_setup.py` and `disease_outbreak_prevention_network.py` for the implementation and `runbook.md` for the skill-driven reproduction. diff --git a/disease-outbreak-prevention/disease_outbreak_prevention_network.py b/disease-outbreak-prevention/disease_outbreak_prevention_network.py index 1e1aecf8..1f231cc6 100644 --- a/disease-outbreak-prevention/disease_outbreak_prevention_network.py +++ b/disease-outbreak-prevention/disease_outbreak_prevention_network.py @@ -9,7 +9,9 @@ - Rank facilities by their centrality to optimize outbreak response. Run: - `python disease_outbreak_prevention_network.py` + `python disease_outbreak_prevention_network.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from the + deployed model. Output: Prints a ranked list of healthcare facilities by degree centrality, showing which @@ -47,7 +49,7 @@ # -------------------------------------------------- -# Query and display results +# Query and display results (`rai models deploy` skips the report) # -------------------------------------------------- def main() -> None: @@ -135,4 +137,6 @@ def main() -> None: if __name__ == "__main__": + # Deploy and wait for the first refresh, which computes the graph metrics. + model.deploy() main() From 529cd39e90ff572a58ebad70c448711935056978 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:10:40 -0800 Subject: [PATCH 036/162] humanitarian-aid-supply-chain: run as a deployed model The script deploys the model, waits for the first refresh, which computes the graph metrics, and prints the report from the deployed model. The Streamlit app and model_setup.py are unchanged. The README covers the deployment config and teardown. --- humanitarian-aid-supply-chain/README.md | 27 ++++++++++++++----- .../humanitarian_aid_supply_chain.py | 8 ++++-- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/humanitarian-aid-supply-chain/README.md b/humanitarian-aid-supply-chain/README.md index ef3edcb9..aae60b56 100644 --- a/humanitarian-aid-supply-chain/README.md +++ b/humanitarian-aid-supply-chain/README.md @@ -43,7 +43,7 @@ Built using **graph analysis** — PageRank (an iterative random-walk algorithm) ## What's included - **Shared model setup**: `model_setup.py` - Common model configuration and graph creation (used by both scripts) -- **Command-line script**: `humanitarian_aid_supply_chain.py` - CLI analysis script with detailed output +- **Command-line script**: `humanitarian_aid_supply_chain.py` - CLI analysis script with detailed output, run from a model deployed to your Snowflake account - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Interactive app**: `app.py` - Streamlit web application with visualizations and interactive analysis - **Data**: `data/distribution_points.csv` and `data/supply_routes.csv` @@ -53,7 +53,8 @@ Built using **graph analysis** — PageRank (an iterative random-walk algorithm) ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -93,12 +94,24 @@ You can customize the data and model as needed after you have it running end-to- python -m pip install . ``` -4. **Configure Snowflake connection and RAI profile** +4. **Configure the Snowflake connection and the deployment** - ```bash - rai init + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .HUMANITARIAN_AID_SUPPLY_CHAIN # published views and tables + identity: + internal_schema: .HUMANITARIAN_AID_SUPPLY_CHAIN_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python humanitarian_aid_supply_chain.py` stops with an error naming the missing setting. + 5. **Run the template** **Option A: Command-line script** @@ -107,6 +120,8 @@ You can customize the data and model as needed after you have it running end-to- python humanitarian_aid_supply_chain.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + **Option B: Interactive Streamlit app** ```bash @@ -189,7 +204,7 @@ CSV files → model_setup.create_model() → Calculate PageRank → Calculate De **Query and analyze strategic categories.** Both metrics are queried together into a DataFrame, then each distribution point is classified against the 70th-percentile thresholds on the two scores: critical coordination hubs (high on both), influential endpoints (high PageRank, lower degree), and network connectors (lower PageRank, high degree). -**Display strategic analysis and recommendations.** The CLI script prints a ranked table of all points with both metrics, the strategic-category breakdown, network-wide and regional statistics, and recommendations for response teams. The Streamlit app adds an interactive overview, a color-coded network visualization, filterable rankings with CSV export, and per-category detail. +**Display strategic analysis and recommendations.** The CLI script deploys the model, waits for its first refresh, and prints a ranked table of all points with both metrics, the strategic-category breakdown, network-wide and regional statistics, and recommendations for response teams. The Streamlit app adds an interactive overview, a color-coded network visualization, filterable rankings with CSV export, and per-category detail. See `model_setup.py` and `humanitarian_aid_supply_chain.py` for the implementation, and `runbook.md` for the skill-driven reproduction. diff --git a/humanitarian-aid-supply-chain/humanitarian_aid_supply_chain.py b/humanitarian-aid-supply-chain/humanitarian_aid_supply_chain.py index 8f85fcc6..4fcfcba5 100644 --- a/humanitarian-aid-supply-chain/humanitarian_aid_supply_chain.py +++ b/humanitarian-aid-supply-chain/humanitarian_aid_supply_chain.py @@ -10,7 +10,9 @@ - Analyze weighted, directed supply chains to prioritize humanitarian response. Run: - `python humanitarian_aid_supply_chain.py` + `python humanitarian_aid_supply_chain.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from the + deployed model. Output: Displays ranked distribution points showing both influence (PageRank) and connectivity @@ -50,7 +52,7 @@ # -------------------------------------------------- -# Query and display results +# Query and display results (`rai models deploy` skips the report) # -------------------------------------------------- def main() -> None: @@ -205,4 +207,6 @@ def main() -> None: if __name__ == "__main__": + # Deploy and wait for the first refresh, which computes the graph metrics. + model.deploy() main() From 88994149c382773bdb7f7f0e4d67e7ade1ddf758 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:13:46 -0800 Subject: [PATCH 037/162] wildlife-conservation-network: run as a deployed model The script deploys the model, waits for the first refresh, which computes the communities and centrality, and prints the report from the deployed model. The Streamlit app and model_setup.py are unchanged. The README covers the deployment config and teardown. --- wildlife-conservation-network/README.md | 27 ++++++++++++++----- .../wildlife_conservation_network.py | 8 ++++-- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/wildlife-conservation-network/README.md b/wildlife-conservation-network/README.md index 36967831..2bd1cf97 100644 --- a/wildlife-conservation-network/README.md +++ b/wildlife-conservation-network/README.md @@ -37,7 +37,7 @@ Built using **graph analysis** (Louvain community detection and degree centralit ## What's included - **Model**: a single shared ontology — `Organization` nodes and `Partnership` edges — plus the Louvain community and degree-centrality enrichment the graph reasoner writes back. Defined once in `model_setup.py` and reused by both runners. -- **Runner**: `wildlife_conservation_network.py` (CLI report) and `app.py` (interactive Streamlit visualization), both against a Snowflake-connected RAI account. +- **Runner**: `wildlife_conservation_network.py` (CLI report, run from a model deployed to your Snowflake account) and `app.py` (interactive Streamlit visualization), both against a Snowflake-connected RAI account. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: a small network of conservation organizations and the partnerships between them. See *Sample data* below. - **Outputs**: a per-organization table of community assignments and metrics, a per-community breakdown, and (in the app) an interactive network graph. @@ -47,7 +47,8 @@ Built using **graph analysis** (Louvain community detection and degree centralit ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- A role that can deploy models, set up once by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) (**Try it out**, or **Governed setup** for a shared account). This template has no prescriptive solve, so it needs no PAT setup. +- A Snowflake user that holds that role. ### Tools @@ -87,12 +88,24 @@ You can customize the data and model as needed after you have it running end-to- python -m pip install . ``` -4. **Configure Snowflake connection and RAI profile** +4. **Configure the Snowflake connection and the deployment** - ```bash - rai init + Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + + ```yaml + deployment: + role: + public_schema: .WILDLIFE_CONSERVATION_NETWORK # published views and tables + identity: + internal_schema: .WILDLIFE_CONSERVATION_NETWORK_META # refresh procedures and metadata + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard ``` + Without the `deployment` section, `python wildlife_conservation_network.py` stops with an error naming the missing setting. + 5. **Run the template** **Option A: Command-line script** @@ -101,6 +114,8 @@ You can customize the data and model as needed after you have it running end-to- python wildlife_conservation_network.py ``` + The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + **Option B: Interactive Streamlit app** ```bash @@ -173,7 +188,7 @@ Both runners share one model. `model_setup.py` builds the RelationalAI model con The graph reasoner then runs two algorithms over that graph. Louvain community detection partitions the network into collaboration clusters by optimizing modularity — iteratively grouping organizations to maximize connections within a cluster and minimize connections between clusters — and writes a community label onto each organization. Degree centrality normalizes each organization's partnership count to a 0-to-1 scale, which makes hubs comparable across clusters of different sizes; the highest-centrality organization inside a community is the one best placed to broker coordination there. -Reading those results back is a query joining the community label, centrality score, and raw partnership count per organization. The CLI script renders a per-organization table, a per-community breakdown (size, region, species focus, hub), and network-wide summary statistics. The optional Streamlit app is a visualization layer over the same query: an interactive network graph colored by community, expandable per-cluster detail, and cross-community connector analysis. +Reading those results back is a query joining the community label, centrality score, and raw partnership count per organization. The CLI script deploys the model, waits for its first refresh, and renders a per-organization table, a per-community breakdown (size, region, species focus, hub), and network-wide summary statistics. The optional Streamlit app is a visualization layer over the same query: an interactive network graph colored by community, expandable per-cluster detail, and cross-community connector analysis. See `model_setup.py` and `wildlife_conservation_network.py` for the implementation, and `runbook.md` to reproduce it step by step with the RAI skills. diff --git a/wildlife-conservation-network/wildlife_conservation_network.py b/wildlife-conservation-network/wildlife_conservation_network.py index 01c4b4ea..2067ceb9 100644 --- a/wildlife-conservation-network/wildlife_conservation_network.py +++ b/wildlife-conservation-network/wildlife_conservation_network.py @@ -10,7 +10,9 @@ - Analyze community structure to optimize resource sharing and identify coordination leaders. Run: - `python wildlife_conservation_network.py` + `python wildlife_conservation_network.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh, and prints the report from the + deployed model. Output: Prints community clusters of conservation organizations with centrality metrics, @@ -52,7 +54,7 @@ # -------------------------------------------------- -# Query and display results +# Query and display results (`rai models deploy` skips the report) # -------------------------------------------------- def main() -> None: @@ -167,4 +169,6 @@ def main() -> None: if __name__ == "__main__": + # Deploy and wait for the first refresh, which computes the graph metrics. + model.deploy() main() From e80b0e2e714cc51ce30ef3481cc2f803cdd24287 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:21:51 -0800 Subject: [PATCH 038/162] patient_cohort_recruitment: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The report checks the termination status in Python, since model.require is not evaluated on a deployed model, and no longer calls problem.verify or prints the solver details. The README covers the deployment config, the internal schema grants, and teardown. --- patient_cohort_recruitment/README.md | 67 ++++++--- .../patient_cohort_recruitment.py | 130 +++++++++--------- 2 files changed, 117 insertions(+), 80 deletions(-) diff --git a/patient_cohort_recruitment/README.md b/patient_cohort_recruitment/README.md index da9eee6f..e17d8575 100644 --- a/patient_cohort_recruitment/README.md +++ b/patient_cohort_recruitment/README.md @@ -40,13 +40,13 @@ The same shape recurs across knowledge-graph cohort and set-cover problems where - A constraint model with four binary decision streams targeting sub-concepts directly: `EligiblePatient.is_in_cohort` (eligible patients only), `CoverableGene.is_covered` (coverable kinase genes only), `CoverableTherapy.is_covered`, and `CoverableAdverseEvent.is_covered` - Per-axis coverage upper-bound ICs that link `is_covered` to in-cohort patient decisions (`Sub.is_covered <= sum(EligiblePatient.is_in_cohort).per(Sub)`), and lower-bound ICs (`sum(Sub.is_covered) >= MIN_*`) that force the cohort to span enough distinct values - Pre-solve Python invariants that catch silent-failure modes (duplicate keys, dangling foreign keys, missing kinase root, negative timestamps) before the solver runs -- Post-solve verification via `problem.verify()` confirming every IC holds in the returned solution, plus a `termination_status() == "OPTIMAL"` assertion +- A check in the report that stops unless the solve's `termination_status()` is `OPTIMAL` ## What's included -The bundled CSVs are illustrative, fully synthetic demo data (e.g. patient names `P_Alpha`...`P_Oscar`, fictional gene/therapy/AE labels) sized so the pipeline runs end-to-end in a few seconds; swap in your own ontology and patient KG to apply the template to real cohorts. +The bundled CSVs are illustrative, fully synthetic demo data (e.g. patient names `P_Alpha`...`P_Oscar`, fictional gene/therapy/AE labels) sized so the solve takes a few seconds; swap in your own ontology and patient KG to apply the template to real cohorts. -- `patient_cohort_recruitment.py` -- main script with concepts, the Graph closure, the rules, the decisions and constraints, and the solver call +- `patient_cohort_recruitment.py` -- main script with concepts, the Graph closure, the rules, the decisions and constraints, and the solve configuration - **Runbook**: `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/genes.csv` -- 10 sample genes: 7 in the kinase pathway sub-ontology (a root plus two intermediate sub-roots and four leaves) and 3 unrelated metabolism genes - `data/gene_is_a.csv` -- 8 `is_a` edges that lay out the kinase-pathway tree and a parallel unrelated tree @@ -62,7 +62,8 @@ The bundled CSVs are illustrative, fully synthetic demo data (e.g. patient names ### Access - A Snowflake account that has the RAI Native App installed. -- A Snowflake user with permissions to access the RAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools - Python >= 3.10 @@ -91,25 +92,59 @@ The bundled CSVs are illustrative, fully synthetic demo data (e.g. patient names python -m pip install . ``` -4. Configure: - ```bash - rai init +4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: patient_cohort_recruitment_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .PATIENT_COHORT_RECRUITMENT # published views and tables + identity: + internal_schema: .PATIENT_COHORT_RECRUITMENT_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python patient_cohort_recruitment.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .PATIENT_COHORT_RECRUITMENT_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PATIENT_COHORT_RECRUITMENT_META TO APPLICATION ; ``` + The deploy creates the public schema `.PATIENT_COHORT_RECRUITMENT` if it doesn't exist. 5. Run: ```bash python patient_cohort_recruitment.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + 6. Expected output — a few lines confirm a successful run (the script also prints the gene closure, the eligible-patient set, and the per-axis coverage tally): ```text Solve result: • status: OPTIMAL - • objective: 0 - • solve time: ~1s - • num_points: 1 - • solver: MiniZinc_unknown Selected cohort: patient_id patient_name age_years @@ -170,7 +205,7 @@ For the full concept and property definitions, see `patient_cohort_recruitment.p The pipeline runs three stages in order: Graph closes the ontology, Rules lift the closure to patient-level facts, and the CSP solver selects the cohort. ```text -gene ontology → graph closure (KinaseGene) → rules (eligibility + coverage sub-concepts) → CSP cohort selection → verify +gene ontology → graph closure (KinaseGene) → rules (eligibility + coverage sub-concepts) → CSP cohort selection → deploy → report ``` **Graph reasoner: one call to close the ontology.** The `is_a` CSV is in OMOP / SNOMED convention (child -> parent), but the graph is built with source and destination swapped, so reachability from the pathway root flows *downwards* through the subclass tree onto every descendant gene. The full transitive closure (every ancestor-descendant pair) is materialized as a sub-concept `KinaseGene extends Gene` — the genes reachable from the configured pathway root. @@ -179,7 +214,7 @@ gene ontology → graph closure (KinaseGene) → rules (eligibility + coverage s **Prescriptive reasoner: cohort selection as a CSP.** Decisions target the sub-concepts directly (`EligiblePatient.is_in_cohort`, `CoverableTherapy.is_covered`, and the gene/AE analogues), creating one binary variable per sub-concept row. The `Coverable*` sub-concepts are scoped to *eligible-patient* coverage, not any-patient coverage: a value covered only by ineligible patients would otherwise have no upper-bound constraint binding it and the solver could mark it covered for free. Scoping coverage to eligible patients, and scoping the decisions to `Coverable*`, ensures every `is_covered` decision has a real upper bound. -The CSP signature is coverage upper bound + per-pair lower bound + floor. For each coverable value, `is_covered` is bounded above by the number of in-cohort patients that cover it (an unsupported value can't be marked covered) and bounded below per pair by each covering in-cohort patient (any in-cohort patient covering it forces `is_covered` to 1). The two bounds pin the indicator to the actual coverage; the floor constraint (`sum(is_covered) >= MIN_*`) then forces the cohort to span at least the required number of distinct genes, therapies, and adverse events. Every constraint is pure relational arithmetic, so `problem.verify()` re-evaluates all of them in the returned solution. +The CSP signature is coverage upper bound + per-pair lower bound + floor. For each coverable value, `is_covered` is bounded above by the number of in-cohort patients that cover it (an unsupported value can't be marked covered) and bounded below per pair by each covering in-cohort patient (any in-cohort patient covering it forces `is_covered` to 1). The two bounds pin the indicator to the actual coverage; the floor constraint (`sum(is_covered) >= MIN_*`) then forces the cohort to span at least the required number of distinct genes, therapies, and adverse events. For the exact PyRel formulation, see `patient_cohort_recruitment.py`; `runbook.md` reproduces the three stages step by step with the RAI skills. @@ -199,7 +234,7 @@ For the exact PyRel formulation, see `patient_cohort_recruitment.py`; `runbook.m ### Extend the model -- **Move from feasibility to optimization.** This template is a satisfaction model — any cohort that hits the floors is correct. To rank, swap `problem.solve(...)` for `problem.maximize(sum(CoverableGene.is_covered) + sum(CoverableTherapy.is_covered) + sum(CoverableAdverseEvent.is_covered))` to find the cohort with the broadest joint span, or `problem.minimize(sum(EligiblePatient.is_in_cohort * EligiblePatient.age_years))` for a younger cohort. MiniZinc / Chuffed handles both. +- **Move from feasibility to optimization.** This template is a satisfaction model — any cohort that hits the floors is correct. To rank, add `problem.maximize(sum(CoverableGene.is_covered) + sum(CoverableTherapy.is_covered) + sum(CoverableAdverseEvent.is_covered))` to find the cohort with the broadest joint span, or `problem.minimize(sum(EligiblePatient.is_in_cohort * EligiblePatient.age_years))` for a younger cohort. MiniZinc / Chuffed handles both. - **Add patient-level eligibility rules** — minimum age, treatment-naive status, organ-function thresholds — by adding more conjuncts to the `EligiblePatient` definition (or by introducing further `extends=[Patient]` sub-concepts). Each extra rule narrows the eligible set; the CSP automatically drops decisions for newly-ineligible patients. - **Add cohort-level fairness rules.** A balanced-cohort study might require a minimum count from each of two demographic strata. Add a stratum property (`Patient.stratum`) and an IC `sum(EligiblePatient.is_in_cohort).per(EligiblePatient.stratum) >= MIN_PER_STRATUM` to enforce minimum representation per stratum. The decision-side aggregate must key on the sub-concept (`EligiblePatient`), not the parent (`Patient`). - **Keep aggregates on a single sub-concept.** Aggregations over a decision must reference the sub-concept the decision was scoped to — mixing parent and sub-concept references in a single aggregate triggers a TypeError. @@ -208,7 +243,7 @@ For the exact PyRel formulation, see `patient_cohort_recruitment.py`; `runbook.m - For a live patient store, replace the `read_csv(...)` loads with `model.data(snowflake_table)` calls so the ontology reads directly from your OMOP / FHIR tables; the rules and CSP are unchanged. - The pre-solve invariants (unique keys, foreign-key integrity, root presence, non-negative timestamps) become your first-pass data-quality gate on real feeds — keep them. -- The bundled data runs in seconds; larger cohorts scale to whatever the constraint solver's budget allows. Raise `time_limit_sec` in the `problem.solve(...)` call for bigger eligible sets. +- The bundled data runs in seconds; larger cohorts scale to whatever the constraint solver's budget allows. Raise `time_limit_sec` in the `problem.configure(...)` call for bigger eligible sets. - Pin `relationalai` (see Prerequisites) so runs stay reproducible across environments. ## Troubleshooting @@ -227,7 +262,7 @@ For the exact PyRel formulation, see `patient_cohort_recruitment.py`; `runbook.m Multiple feasible cohorts exist; which one does the solver return? - This is constraint satisfaction, not optimization. Any cohort that hits the floors is a correct answer; the solver is free to return different ones across runs. -- To enumerate cohorts (e.g., for clinical-team review), pass `solution_limit=N` to `problem.solve(...)` and iterate over `problem.num_points()` solutions. +- To enumerate cohorts (e.g., for clinical-team review), pass `solution_limit=N` to `problem.configure(...)` and iterate over `problem.num_points()` solutions. - To pin a single answer, switch to optimization -- e.g. `problem.maximize(sum(CoverableGene.is_covered) + sum(CoverableTherapy.is_covered) + sum(CoverableAdverseEvent.is_covered))` returns the cohort with the broadest joint span.
diff --git a/patient_cohort_recruitment/patient_cohort_recruitment.py b/patient_cohort_recruitment/patient_cohort_recruitment.py index d8da3c6d..821be28a 100644 --- a/patient_cohort_recruitment/patient_cohort_recruitment.py +++ b/patient_cohort_recruitment/patient_cohort_recruitment.py @@ -73,17 +73,16 @@ so the floor constrains the cohort to span at least ``MIN_GENES_COVERED`` / ``MIN_THERAPIES_COVERED`` / ``MIN_AES_COVERED`` distinct values. -- All ten ICs (cohort size + 3 upper + 3 lower + 3 floor) are pure - relational arithmetic, so ``problem.verify()`` re-evaluates every - one in the returned solution. Run: - `python patient_cohort_recruitment.py` + `python patient_cohort_recruitment.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and + prints the report from the deployed model. Output: - Prints the formulation, the kinase-pathway gene closure, the - eligible patient set, the chosen cohort, the per-axis coverage - tally, and post-solve constraint verification. + Prints the formulation, the solve status, the kinase-pathway gene closure, + the eligible patient set, the chosen cohort, and the per-axis coverage + tally. """ from pathlib import Path @@ -553,66 +552,69 @@ def _assert_nonneg_column(df, col, source): problem.satisfy(ae_min_ic) # -------------------------------------------------- -# Solve and verify +# Configure the solve # -------------------------------------------------- -problem.display() -problem.solve("minizinc", time_limit_sec=60) -problem.solve_info().display() - -# All ten ICs are pure relational arithmetic, so `verify` re-evaluates -# every one in the returned solution. -problem.verify( - cohort_size_ic, - gene_cover_ub_ic, - therapy_cover_ub_ic, - ae_cover_ub_ic, - gene_cover_lb_ic, - therapy_cover_lb_ic, - ae_cover_lb_ic, - gene_min_ic, - therapy_min_ic, - ae_min_ic, -) -model.require(problem.termination_status() == "OPTIMAL") +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=60) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"cohort solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) # -------------------------------------------------- -# Inspect results +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nKinase-pathway gene closure (reachable from KINASE_ROOT_GENE_ID):") -model.select( - KinaseGene.id.alias("gene_id"), - KinaseGene.name.alias("gene_name"), -).inspect() - -print("\nEligible patients (carry a kinase mutation and have a qualifying pair):") -model.select( - EligiblePatient.id.alias("patient_id"), - EligiblePatient.name.alias("patient_name"), -).inspect() - -print("\nSelected cohort:") -model.select( - EligiblePatient.id.alias("patient_id"), - EligiblePatient.name.alias("patient_name"), - EligiblePatient.age_years.alias("age_years"), -).where(EligiblePatient.is_in_cohort == 1).inspect() - -print("\nKinase-pathway genes covered by the cohort:") -model.select( - CoverableGene.id.alias("gene_id"), - CoverableGene.name.alias("gene_name"), -).where(CoverableGene.is_covered == 1).inspect() - -print("\nTherapies covered by the cohort:") -model.select( - CoverableTherapy.id.alias("therapy_id"), - CoverableTherapy.name.alias("therapy_name"), -).where(CoverableTherapy.is_covered == 1).inspect() - -print("\nAdverse events covered by the cohort:") -model.select( - CoverableAdverseEvent.id.alias("ae_id"), - CoverableAdverseEvent.term.alias("ae_term"), -).where(CoverableAdverseEvent.is_covered == 1).inspect() +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Status of the latest refresh's solve. A refresh records any termination + # status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No cohort: termination status {status}") + + print("\nKinase-pathway gene closure (reachable from KINASE_ROOT_GENE_ID):") + model.select( + KinaseGene.id.alias("gene_id"), + KinaseGene.name.alias("gene_name"), + ).inspect() + + print("\nEligible patients (carry a kinase mutation and have a qualifying pair):") + model.select( + EligiblePatient.id.alias("patient_id"), + EligiblePatient.name.alias("patient_name"), + ).inspect() + + print("\nSelected cohort:") + model.select( + EligiblePatient.id.alias("patient_id"), + EligiblePatient.name.alias("patient_name"), + EligiblePatient.age_years.alias("age_years"), + ).where(EligiblePatient.is_in_cohort == 1).inspect() + + print("\nKinase-pathway genes covered by the cohort:") + model.select( + CoverableGene.id.alias("gene_id"), + CoverableGene.name.alias("gene_name"), + ).where(CoverableGene.is_covered == 1).inspect() + + print("\nTherapies covered by the cohort:") + model.select( + CoverableTherapy.id.alias("therapy_id"), + CoverableTherapy.name.alias("therapy_name"), + ).where(CoverableTherapy.is_covered == 1).inspect() + + print("\nAdverse events covered by the cohort:") + model.select( + CoverableAdverseEvent.id.alias("ae_id"), + CoverableAdverseEvent.term.alias("ae_term"), + ).where(CoverableAdverseEvent.is_covered == 1).inspect() From 3788b9e8188647b261ec598fe0de8a3a5985709a Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 16:36:01 -0800 Subject: [PATCH 039/162] book_slate_recommendation: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. The feasibility check and the report run after the deploy and read the status and objective through the model's own relationships. The report accepts OPTIMAL or TIME_LIMIT with a slate, since the bundled slice ends at the 180 s time limit. problem.verify and the solver details are no longer printed. The README covers the deployment config, the internal schema grants, and teardown. --- book_slate_recommendation/README.md | 62 ++- .../book_slate_recommendation.py | 467 +++++++++--------- 2 files changed, 288 insertions(+), 241 deletions(-) diff --git a/book_slate_recommendation/README.md b/book_slate_recommendation/README.md index 1071f584..a75c7b8b 100644 --- a/book_slate_recommendation/README.md +++ b/book_slate_recommendation/README.md @@ -42,7 +42,7 @@ This template composes the slate as a single optimization decision rather than a ## What's included - **Model**: a three-stage pipeline (knowledge-graph path walks, then graph analysis, then prescriptive optimization) on a single shared ontology, wired to the bundled CSVs. -- **Runner**: `book_slate_recommendation.py` — a single Python script that runs end-to-end against a Snowflake-connected RAI account. +- **Runner**: `book_slate_recommendation.py` — a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: a deterministic Open Library (CC0 public domain) slice of roughly 59 books, 52 authors, and 12 subjects, with synthetic users and read events layered on top. See *Sample data* below. - **Outputs**: the formulation, the solve-result block, the chosen slate (with per-book `path_count_total` and `triangle_count`), the per-user subject distribution, per-pick explanation-path support, and diagnostic candidate-set and structural-embeddedness tables. @@ -52,7 +52,8 @@ This template composes the slate as a single optimization decision rather than a ### Access - A Snowflake account with the RelationalAI Native App installed. -- A Snowflake user with permissions to access the RelationalAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. ### Tools @@ -86,11 +87,44 @@ This template composes the slate as a single optimization decision rather than a python -m pip install . ``` -4. Configure (prompts for Snowflake account, role, and profile name): - - ```bash - rai init +4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: book_slate_recommendation_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .BOOK_SLATE_RECOMMENDATION # published views and tables + identity: + internal_schema: .BOOK_SLATE_RECOMMENDATION_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python book_slate_recommendation.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .BOOK_SLATE_RECOMMENDATION_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .BOOK_SLATE_RECOMMENDATION_META TO APPLICATION ; ``` + The deploy creates the public schema `.BOOK_SLATE_RECOMMENDATION` if it doesn't exist. 5. Run the template end-to-end: @@ -98,14 +132,16 @@ This template composes the slate as a single optimization decision rather than a python book_slate_recommendation.py ``` -6. Expected output (a few lines confirm a successful run). Solver build strings and exact wall times vary, but `status: OPTIMAL` and the chosen slate are stable for the bundled slice: + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + +6. Expected output (a few lines confirm a successful run). The status is `OPTIMAL` when the solver proves the slate optimal within `time_limit_sec`, and `TIME_LIMIT` when the limit ends the search first with the best slate found, as on the bundled slice with the default 180 seconds. The objective and the chosen slate can differ between runs: ```text Solve result: - • status: OPTIMAL - • solve time: a few seconds on the bundled slice - • num_points: 1 - • solver: MiniZinc + • status: TIME_LIMIT + • objective: 542 Final slate per user (K = 3, slot 1 = top of row): user_id slot book_id path_count_total triangle_count @@ -230,7 +266,7 @@ See *Troubleshooting* for the joint-feasibility relationships between these dial
Solve returns INFEASIBLE / the final slate is empty -The runner enforces ten constraints and requires an `OPTIMAL` status. If the data is too sparse or too clustered, the joint constraints can be infeasible. The dominant causes on customer slices are: +The runner enforces ten constraints and requires an `OPTIMAL` or `TIME_LIMIT` status with a slate. If the data is too sparse or too clustered, the joint constraints can be infeasible. The dominant causes on customer slices are: 1. **Too few candidates after the already-read exclusion.** Each user needs at least `SLATE_SIZE_K` candidates in their 2-hop reach after already-read books are excluded. Recommendable candidates come from length-2 `read_Book -> similar_Book` walks, so a user whose reads have no `similar_to` neighbors will be infeasible. Inspect the candidate-set diagnostic printed at the end of the runner. 2. **Disjoint floors.** `FRESHNESS_FLOOR` and `ORIGINALS_FLOOR` are independent. A user with no candidate that is both fresh and in-house must spend `FRESHNESS_FLOOR + ORIGINALS_FLOOR` distinct picks, not the maximum of the two. @@ -239,7 +275,7 @@ The runner enforces ten constraints and requires an `OPTIMAL` status. If the dat 5. **Missing edges.** Every picked book must have at least one `Book.about(Subject)` and one `Book.written_by(Author)` row, or the diversity and uniqueness caps silently exempt it. 6. **Hero pin.** Each user needs at least one candidate whose book has `triangle_count >= HERO_EMBEDDEDNESS_THRESHOLD`, or there is no eligible book for slot 1. Inspect the structural-embeddedness diagnostic table, then either lower `HERO_EMBEDDEDNESS_THRESHOLD` or densify the similarity input. -The runner runs a Python-level pre-solve assertion just before `problem.solve(...)` that materializes the candidate set, anti-joins against the read history, and raises a `ValueError` naming the affected users per shortfall, with a strategy block keyed by which condition fired. It also prints an unread-candidate-count table before solving. +After the deploy, the runner runs a Python-level assertion that materializes the candidate set, anti-joins against the read history, and raises a `ValueError` naming the affected users per shortfall, with a strategy block keyed by which condition fired. It also prints an unread-candidate-count table before the solve result.
diff --git a/book_slate_recommendation/book_slate_recommendation.py b/book_slate_recommendation/book_slate_recommendation.py index c02034ca..64b7cd9d 100644 --- a/book_slate_recommendation/book_slate_recommendation.py +++ b/book_slate_recommendation/book_slate_recommendation.py @@ -43,7 +43,9 @@ Customers fetch larger slices by re-running with ``--size md``/``--size lg``. Run: - `python book_slate_recommendation.py` + `python book_slate_recommendation.py` deploys the model to the schema in + raiconfig.yaml, waits for its first refresh (which runs the solve), and + prints the report from the deployed model. Output: Prints the formulation, the solve-result block, the chosen slate @@ -686,75 +688,19 @@ def _assert_in_set(df, col, allowed, source): problem.maximize(sum((SLATE_SIZE_K + 1 - Candidate.slot) * Candidate.path_count_total)) # -------------------------------------------------- -# Solve and check solution +# Configure the solve # -------------------------------------------------- -problem.display() - -# Pre-solve floor assertion: per-user floor ICs anchor on Candidate -# rows, so a user with no candidates left after exclude_read is -# silently skipped instead of flagged as infeasible. Compute the -# unread candidate set per user explicitly and refuse to solve if -# any user fails any of the per-IC feasibility necessary conditions -# below -- sparse customer reach hits a clear Python-level error -# rather than a quiet INFEASIBLE solve. -candidate_df = model.select( - Candidate.user_id.alias("user_id"), - Candidate.book_id.alias("book_id"), - Candidate.path_count_total.alias("path_count_total"), -).to_df() -candidate_df["path_count_total"] = candidate_df["path_count_total"].astype(int) -unread = candidate_df.merge( - reads_csv[["user_id", "book_id"]], - on=["user_id", "book_id"], - how="left", - indicator=True, -) -unread = unread[unread["_merge"] == "left_only"][["user_id", "book_id", "path_count_total"]] -# Pull triangle counts so the hero-eligible feasibility check can -# count books meeting HERO_EMBEDDEDNESS_THRESHOLD per user. -triangle_df = model.select( - Book.id.alias("book_id"), - Book.triangle_count.alias("triangle_count"), -).to_df() -# PyRel returns Int128Array-backed columns; cast to native int so -# pandas comparisons against Python ints work in the feasibility -# checks below. -triangle_df["book_id"] = triangle_df["book_id"].astype(int) -triangle_df["triangle_count"] = triangle_df["triangle_count"].astype(int) -unread_with_book = unread.merge( - books_csv[["id", "age_days", "in_house"]].rename(columns={"id": "book_id"}), - on="book_id", - how="left", -).merge(triangle_df, on="book_id", how="left") -unread_authors = unread.merge( - book_authors_csv[["book_id", "author_id"]], - on="book_id", - how="left", -) -unread_subjects = unread.merge( - book_subjects_csv[["book_id", "subject_id"]], - on="book_id", - how="left", -) -unread_counts = unread_with_book.groupby("user_id").size() -fresh_counts = ( - unread_with_book[unread_with_book["age_days"] <= FRESH_WINDOW_DAYS].groupby("user_id").size() -) -in_house_counts = unread_with_book[unread_with_book["in_house"] == 1].groupby("user_id").size() -hero_counts = ( - unread_with_book[unread_with_book["triangle_count"] >= HERO_EMBEDDEDNESS_THRESHOLD] - .groupby("user_id") - .size() -) -distinct_author_counts = unread_authors.groupby("user_id")["author_id"].nunique() -distinct_subject_counts = unread_subjects.groupby("user_id")["subject_id"].nunique() -# cold_start_ic caps weakly-explained picks at COLD_START_CAP per user; -# the slate must be filled with K - COLD_START_CAP strongly-explained -# (path_count_total >= WEAK_EXPLANATION_THRESHOLD) candidates at minimum. -strong_counts = ( - unread[unread["path_count_total"] >= WEAK_EXPLANATION_THRESHOLD].groupby("user_id").size() -) +# Each refresh of the deployed model runs this solve. +problem.configure("minizinc", time_limit_sec=180) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"slate solve status is {String:status}") +TotalSupport = model.Relationship(f"slate total position-weighted path support is {Integer:support}") +model.define(SolveStatus(problem.termination_status()), TotalSupport(problem.objective_value())) # explanation_ic requires sum((K+1-slot)*path_count_total) >= EXPLANATION_FLOOR. @@ -773,180 +719,245 @@ def _max_explanation_score(group): return int(total) -max_explanation = unread.groupby("user_id")["path_count_total"].apply(_max_explanation_score) - -all_users = set(users_csv["id"]) -absent_from_unread = sorted(all_users - set(unread_counts.index)) -short_total = sorted(unread_counts[unread_counts < SLATE_SIZE_K].index.tolist()) -short_fresh = sorted(list(all_users - set(fresh_counts[fresh_counts >= FRESHNESS_FLOOR].index))) -short_in_house = sorted( - list(all_users - set(in_house_counts[in_house_counts >= ORIGINALS_FLOOR].index)) -) -short_hero = sorted(list(all_users - set(hero_counts[hero_counts >= 1].index))) -short_authors = sorted( - list(all_users - set(distinct_author_counts[distinct_author_counts >= SLATE_SIZE_K].index)) -) -# Subject-span feasibility: with K picks and MAX_PER_SUBJECT cap, the user's -# pool must span at least ceil(K / MAX_PER_SUBJECT) distinct subjects. -min_subjects = -(-SLATE_SIZE_K // MAX_PER_SUBJECT) # ceil division -short_subjects = sorted( - list(all_users - set(distinct_subject_counts[distinct_subject_counts >= min_subjects].index)) -) -# Cold-start feasibility: at least K - COLD_START_CAP strongly-explained picks -# are required. If a user has fewer strong candidates, cold_start_ic forces -# infeasibility. -min_strong = max(0, SLATE_SIZE_K - COLD_START_CAP) -short_strong = ( - sorted(list(all_users - set(strong_counts[strong_counts >= min_strong].index))) - if min_strong > 0 - else [] -) -short_explanation = sorted( - list(all_users - set(max_explanation[max_explanation >= EXPLANATION_FLOOR].index)) -) -if ( - absent_from_unread - or short_total - or short_fresh - or short_in_house - or short_hero - or short_authors - or short_subjects - or short_strong - or short_explanation -): - raise ValueError( - "Per-user candidate floor violated:\n" - f" users absent from unread candidate table: {absent_from_unread}\n" - f" users with < SLATE_SIZE_K={SLATE_SIZE_K} unread candidates: {short_total}\n" - f" users with < FRESHNESS_FLOOR={FRESHNESS_FLOOR} unread fresh " - f"(age_days <= {FRESH_WINDOW_DAYS}) candidates: {short_fresh}\n" - f" users with < ORIGINALS_FLOOR={ORIGINALS_FLOOR} unread in-house " - f"candidates: {short_in_house}\n" - f" users with no unread hero-eligible (triangle_count >= " - f"{HERO_EMBEDDEDNESS_THRESHOLD}) candidates: {short_hero}\n" - f" users with < SLATE_SIZE_K={SLATE_SIZE_K} distinct unread authors: {short_authors}\n" - f" users with < ceil(K/MAX_PER_SUBJECT)={min_subjects} distinct unread " - f"subjects: {short_subjects}\n" - f" users with < K-COLD_START_CAP={min_strong} strongly-explained " - f"(path_count_total >= {WEAK_EXPLANATION_THRESHOLD}) unread candidates: " - f"{short_strong}\n" - f" users whose max achievable position-weighted explanation score < " - f"EXPLANATION_FLOOR={EXPLANATION_FLOOR}: {short_explanation}\n" - "Pick a strategy by which condition fired:\n" - " reach (absent_from_unread, short_total): densify Book.similar_to or " - "the read history.\n" - " per-IC floor (short_fresh, short_in_house, short_hero, short_subjects, " - "short_strong): lower the corresponding constant, or refresh the data slice " - "with denser coverage in that dimension.\n" - " short_authors: lower SLATE_SIZE_K or use a corpus with more distinct " - "authors per user reach.\n" - " short_explanation: lower EXPLANATION_FLOOR, or expand the candidate pool " - "so per-user path_count_total distributions reach the threshold." - ) - -print("\nUnread candidate count per user (pre-solve diagnostic):") -print(unread_counts.sort_index().to_string()) - -problem.solve("minizinc", time_limit_sec=180) -problem.solve_info().display() - -problem.verify( - slate_size_ic, - slot_uniqueness_ic, - exclude_read_ic, - author_uniqueness_ic, - subject_span_ic, - freshness_ic, - originals_ic, - cold_start_ic, - hero_pin_ic, - explanation_ic, -) -model.require(problem.termination_status() == "OPTIMAL") - # -------------------------------------------------- -# Inspect results +# Deploy, then report (`rai models deploy` skips this block) # -------------------------------------------------- -print("\nUsers in this run:") -model.select( - User.id.alias("user_id"), - User.name.alias("user"), -).inspect() - -# Headline output: the chosen slate per user, ordered by slot -# (1=hero/top of row, K=bottom). -print(f"\nFinal slate per user (K = {SLATE_SIZE_K}, slot 1 = top of row):") -b_pick = Book.ref() -tc_pick = Integer.ref() -slate_df = ( - model.select( +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + problem.display() + + # Per-user floor assertion: per-user floor ICs anchor on Candidate + # rows, so a user with no candidates left after exclude_read is + # silently skipped instead of flagged as infeasible. Compute the + # unread candidate set per user explicitly and refuse to report if + # any user fails any of the per-IC feasibility necessary conditions + # below -- sparse customer reach hits a clear Python-level error + # rather than a quiet INFEASIBLE solve. + candidate_df = model.select( Candidate.user_id.alias("user_id"), - Candidate.slot.alias("slot"), Candidate.book_id.alias("book_id"), Candidate.path_count_total.alias("path_count_total"), - tc_pick.alias("triangle_count"), + ).to_df() + candidate_df["path_count_total"] = candidate_df["path_count_total"].astype(int) + unread = candidate_df.merge( + reads_csv[["user_id", "book_id"]], + on=["user_id", "book_id"], + how="left", + indicator=True, ) - .where( - Candidate.slot <= SLATE_SIZE_K, - b_pick.id == Candidate.book_id, - b_pick.triangle_count == tc_pick, + unread = unread[unread["_merge"] == "left_only"][["user_id", "book_id", "path_count_total"]] + # Pull triangle counts so the hero-eligible feasibility check can + # count books meeting HERO_EMBEDDEDNESS_THRESHOLD per user. + triangle_df = model.select( + Book.id.alias("book_id"), + Book.triangle_count.alias("triangle_count"), + ).to_df() + # PyRel returns Int128Array-backed columns; cast to native int so + # pandas comparisons against Python ints work in the feasibility + # checks below. + triangle_df["book_id"] = triangle_df["book_id"].astype(int) + triangle_df["triangle_count"] = triangle_df["triangle_count"].astype(int) + unread_with_book = unread.merge( + books_csv[["id", "age_days", "in_house"]].rename(columns={"id": "book_id"}), + on="book_id", + how="left", + ).merge(triangle_df, on="book_id", how="left") + unread_authors = unread.merge( + book_authors_csv[["book_id", "author_id"]], + on="book_id", + how="left", ) - .to_df() -) -print(slate_df.sort_values(["user_id", "slot"]).to_string(index=False)) - -print("\nSubject distribution per user's slate:") -u_disp = User.ref() -b_disp = Book.ref() -s_disp = Subject.ref() -model.select( - u_disp.id.alias("user_id"), - s_disp.name.alias("subject"), - count(Candidate) - .per(u_disp, s_disp) - .where( - Candidate.user_id == u_disp.id, - Candidate.book_id == b_disp.id, - Book.about(b_disp, s_disp), - Candidate.slot <= SLATE_SIZE_K, + unread_subjects = unread.merge( + book_subjects_csv[["book_id", "subject_id"]], + on="book_id", + how="left", + ) + unread_counts = unread_with_book.groupby("user_id").size() + fresh_counts = ( + unread_with_book[unread_with_book["age_days"] <= FRESH_WINDOW_DAYS].groupby("user_id").size() + ) + in_house_counts = unread_with_book[unread_with_book["in_house"] == 1].groupby("user_id").size() + hero_counts = ( + unread_with_book[unread_with_book["triangle_count"] >= HERO_EMBEDDEDNESS_THRESHOLD] + .groupby("user_id") + .size() ) - .alias("n_picked"), -).inspect() + distinct_author_counts = unread_authors.groupby("user_id")["author_id"].nunique() + distinct_subject_counts = unread_subjects.groupby("user_id")["subject_id"].nunique() + # cold_start_ic caps weakly-explained picks at COLD_START_CAP per user; + # the slate must be filled with K - COLD_START_CAP strongly-explained + # (path_count_total >= WEAK_EXPLANATION_THRESHOLD) candidates at minimum. + strong_counts = ( + unread[unread["path_count_total"] >= WEAK_EXPLANATION_THRESHOLD].groupby("user_id").size() + ) + + + max_explanation = unread.groupby("user_id")["path_count_total"].apply(_max_explanation_score) -print("\nExplanation-path support per picked item (ordered by slot):") -expl_df = ( + all_users = set(users_csv["id"]) + absent_from_unread = sorted(all_users - set(unread_counts.index)) + short_total = sorted(unread_counts[unread_counts < SLATE_SIZE_K].index.tolist()) + short_fresh = sorted(list(all_users - set(fresh_counts[fresh_counts >= FRESHNESS_FLOOR].index))) + short_in_house = sorted( + list(all_users - set(in_house_counts[in_house_counts >= ORIGINALS_FLOOR].index)) + ) + short_hero = sorted(list(all_users - set(hero_counts[hero_counts >= 1].index))) + short_authors = sorted( + list(all_users - set(distinct_author_counts[distinct_author_counts >= SLATE_SIZE_K].index)) + ) + # Subject-span feasibility: with K picks and MAX_PER_SUBJECT cap, the user's + # pool must span at least ceil(K / MAX_PER_SUBJECT) distinct subjects. + min_subjects = -(-SLATE_SIZE_K // MAX_PER_SUBJECT) # ceil division + short_subjects = sorted( + list(all_users - set(distinct_subject_counts[distinct_subject_counts >= min_subjects].index)) + ) + # Cold-start feasibility: at least K - COLD_START_CAP strongly-explained picks + # are required. If a user has fewer strong candidates, cold_start_ic forces + # infeasibility. + min_strong = max(0, SLATE_SIZE_K - COLD_START_CAP) + short_strong = ( + sorted(list(all_users - set(strong_counts[strong_counts >= min_strong].index))) + if min_strong > 0 + else [] + ) + short_explanation = sorted( + list(all_users - set(max_explanation[max_explanation >= EXPLANATION_FLOOR].index)) + ) + if ( + absent_from_unread + or short_total + or short_fresh + or short_in_house + or short_hero + or short_authors + or short_subjects + or short_strong + or short_explanation + ): + raise ValueError( + "Per-user candidate floor violated:\n" + f" users absent from unread candidate table: {absent_from_unread}\n" + f" users with < SLATE_SIZE_K={SLATE_SIZE_K} unread candidates: {short_total}\n" + f" users with < FRESHNESS_FLOOR={FRESHNESS_FLOOR} unread fresh " + f"(age_days <= {FRESH_WINDOW_DAYS}) candidates: {short_fresh}\n" + f" users with < ORIGINALS_FLOOR={ORIGINALS_FLOOR} unread in-house " + f"candidates: {short_in_house}\n" + f" users with no unread hero-eligible (triangle_count >= " + f"{HERO_EMBEDDEDNESS_THRESHOLD}) candidates: {short_hero}\n" + f" users with < SLATE_SIZE_K={SLATE_SIZE_K} distinct unread authors: {short_authors}\n" + f" users with < ceil(K/MAX_PER_SUBJECT)={min_subjects} distinct unread " + f"subjects: {short_subjects}\n" + f" users with < K-COLD_START_CAP={min_strong} strongly-explained " + f"(path_count_total >= {WEAK_EXPLANATION_THRESHOLD}) unread candidates: " + f"{short_strong}\n" + f" users whose max achievable position-weighted explanation score < " + f"EXPLANATION_FLOOR={EXPLANATION_FLOOR}: {short_explanation}\n" + "Pick a strategy by which condition fired:\n" + " reach (absent_from_unread, short_total): densify Book.similar_to or " + "the read history.\n" + " per-IC floor (short_fresh, short_in_house, short_hero, short_subjects, " + "short_strong): lower the corresponding constant, or refresh the data slice " + "with denser coverage in that dimension.\n" + " short_authors: lower SLATE_SIZE_K or use a corpus with more distinct " + "authors per user reach.\n" + " short_explanation: lower EXPLANATION_FLOOR, or expand the candidate pool " + "so per-user path_count_total distributions reach the threshold." + ) + + print("\nUnread candidate count per user (feasibility diagnostic):") + print(unread_counts.sort_index().to_string()) + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. The time limit can end the + # search with the best slate found, which the report accepts. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + support = model.select(TotalSupport).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status not in (["OPTIMAL"], ["TIME_LIMIT"]) or not support: + raise SystemExit(f"No slate: termination status {status}, objective {support}") + print(f"• objective: {support[0]}") + + print("\nUsers in this run:") + model.select( + User.id.alias("user_id"), + User.name.alias("user"), + ).inspect() + + # Headline output: the chosen slate per user, ordered by slot + # (1=hero/top of row, K=bottom). + print(f"\nFinal slate per user (K = {SLATE_SIZE_K}, slot 1 = top of row):") + b_pick = Book.ref() + tc_pick = Integer.ref() + slate_df = ( + model.select( + Candidate.user_id.alias("user_id"), + Candidate.slot.alias("slot"), + Candidate.book_id.alias("book_id"), + Candidate.path_count_total.alias("path_count_total"), + tc_pick.alias("triangle_count"), + ) + .where( + Candidate.slot <= SLATE_SIZE_K, + b_pick.id == Candidate.book_id, + b_pick.triangle_count == tc_pick, + ) + .to_df() + ) + print(slate_df.sort_values(["user_id", "slot"]).to_string(index=False)) + + print("\nSubject distribution per user's slate:") + u_disp = User.ref() + b_disp = Book.ref() + s_disp = Subject.ref() + model.select( + u_disp.id.alias("user_id"), + s_disp.name.alias("subject"), + count(Candidate) + .per(u_disp, s_disp) + .where( + Candidate.user_id == u_disp.id, + Candidate.book_id == b_disp.id, + Book.about(b_disp, s_disp), + Candidate.slot <= SLATE_SIZE_K, + ) + .alias("n_picked"), + ).inspect() + + print("\nExplanation-path support per picked item (ordered by slot):") + expl_df = ( + model.select( + Candidate.user_id.alias("user_id"), + Candidate.slot.alias("slot"), + Candidate.book_id.alias("book_id"), + Candidate.path_count_via_kg_walk.alias("paths_via_kg_walk"), + Candidate.path_count_via_author.alias("paths_via_author"), + Candidate.path_count_via_subject.alias("paths_via_subject"), + ) + .where(Candidate.slot <= SLATE_SIZE_K) + .to_df() + ) + print(expl_df.sort_values(["user_id", "slot"]).to_string(index=False)) + + # Diagnostics: candidate set sizing and structural-embeddedness map. + + print( + f"\nCandidate set per user (Books reachable within {MAX_HOPS} hops over the heterogeneous KG):" + ) model.select( Candidate.user_id.alias("user_id"), - Candidate.slot.alias("slot"), Candidate.book_id.alias("book_id"), + Candidate.path_count_total.alias("path_count_total"), Candidate.path_count_via_kg_walk.alias("paths_via_kg_walk"), Candidate.path_count_via_author.alias("paths_via_author"), Candidate.path_count_via_subject.alias("paths_via_subject"), - ) - .where(Candidate.slot <= SLATE_SIZE_K) - .to_df() -) -print(expl_df.sort_values(["user_id", "slot"]).to_string(index=False)) + ).inspect() -# Diagnostics: candidate set sizing and structural-embeddedness map. - -print( - f"\nCandidate set per user (Books reachable within {MAX_HOPS} hops over the heterogeneous KG):" -) -model.select( - Candidate.user_id.alias("user_id"), - Candidate.book_id.alias("book_id"), - Candidate.path_count_total.alias("path_count_total"), - Candidate.path_count_via_kg_walk.alias("paths_via_kg_walk"), - Candidate.path_count_via_author.alias("paths_via_author"), - Candidate.path_count_via_subject.alias("paths_via_subject"), -).inspect() - -print("\nBook structural embeddedness (similar_to triangle counts):") -model.select( - Book.id.alias("book_id"), - Book.title.alias("title"), - Book.triangle_count.alias("triangle_count"), -).inspect() + print("\nBook structural embeddedness (similar_to triangle counts):") + model.select( + Book.id.alias("book_id"), + Book.title.alias("title"), + Book.triangle_count.alias("triangle_count"), + ).inspect() From 85a20d45ee5598d5e1b2d5f0f71896b995622e71 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 17:07:19 -0800 Subject: [PATCH 040/162] defect_root_cause: run as a deployed model The solve is declared with problem.configure and runs on each refresh of the deployed model. DiagnosisResult is derived by rules from the named root causes, so it deploys with the model, and the plant-wide failure rate used by the contrast score is read from units.csv. The formulation audit and every query run after the deploy; the audit checks the problem size relations and the grounded coverage constraints in Python. The README covers the deployment config, the internal schema grants, and teardown. --- defect_root_cause/README.md | 54 +++- defect_root_cause/defect_root_cause.py | 364 ++++++++++++++++--------- 2 files changed, 283 insertions(+), 135 deletions(-) diff --git a/defect_root_cause/README.md b/defect_root_cause/README.md index 90bd9cdf..8192c9bf 100644 --- a/defect_root_cause/README.md +++ b/defect_root_cause/README.md @@ -45,7 +45,7 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo ## What's included -- **Model and pipeline** (`defect_root_cause.py`): a single script that defines the ontology and runs all three reasoning stages end to end. +- **Model and pipeline** (`defect_root_cause.py`): a single script that defines the ontology and all three reasoning stages, deploys the model to your Snowflake account, and prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Data generator** (`generate_data.py`): rebuilds the sample corpus deterministically from a fixed seed. The generated files are already included, so you do not need to run it to use the template. - **Sample data** (`data/`): ten CSV files describing the product structure, the material lots and their genealogy, the finished units and their test results, the process history, and the list of candidate causes. @@ -56,7 +56,8 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo ### Access - A Snowflake account with the RelationalAI Native App installed. -- A Snowflake user with permission to access the RelationalAI Native App. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. - A prescriptive engine for the optimization step. The template solves with the open-source HiGHS solver, so no additional solver license is required. ### Tools @@ -91,11 +92,44 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo python -m pip install . ``` -4. Configure your Snowflake connection and RelationalAI profile: - - ```bash - rai init +4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + ```yaml + reasoners: + prescriptive: + name: defect_root_cause_solver # created on first deploy if it doesn't exist + deployment: + role: + public_schema: .DEFECT_ROOT_CAUSE # published views and tables + identity: + internal_schema: .DEFECT_ROOT_CAUSE_META # solver inputs, results, refresh procedures + schedules: + standard: + interval_s: 0 # refresh only on deploy or on demand + schedule: standard + pat_secret: # these four come from the setup page + pat_eai: + pat_user: + pat_warehouse: + ``` + Without the `deployment` section, `python defect_root_cause.py` stops with an error naming the missing setting. + + Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + ```sql + -- as the model role + CREATE SCHEMA IF NOT EXISTS .DEFECT_ROOT_CAUSE_META; + + -- as an administrator + GRANT CALLER USAGE ON DATABASE TO APPLICATION ; + GRANT CALLER USAGE ON SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; + GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; + GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; + GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; + GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; + GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; ``` + The deploy creates the public schema `.DEFECT_ROOT_CAUSE` if it doesn't exist. 5. Run the template: @@ -103,6 +137,10 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo python defect_root_cause.py ``` + The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + + To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + 6. Confirm success. The run ends with the diagnosis; the final lines should look like this: ```text @@ -148,7 +186,7 @@ The model represents a manufacturing genealogy: finished units built from materi The genealogy is the backbone: a built `Lot` consumes its input lots, so following those links from a finished unit reaches every material beneath it. -- **Key entities**: `SKU`, `Lot`, `Unit`, `Machine`, and `Factor` (a candidate root cause — any lot, machine, or shift, put on a common footing); plus a `DiagnosisResult` summary the prescriptive stage writes back. +- **Key entities**: `SKU`, `Lot`, `Unit`, `Machine`, and `Factor` (a candidate root cause — any lot, machine, or shift, put on a common footing); plus a `DiagnosisResult` summary derived from the prescriptive stage's result. - **Primary identifiers**: every concept carries a string `id` (for example `LOT::SP-0423` on `Factor`); the genealogy and process history are threaded by relationships (`Lot consumes Lot`, `Unit consumes Lot`, `Unit ran on Machine`) rather than shared keys. - **Important invariants**: `Unit.defective` is `0` or `1`; `Factor.lift` is the touched-unit defect rate over the plant baseline (≈ 1.0 for non-causal high-volume factors); the `is_root_cause` decision variable is binary; a defective unit is either explained by a named cause it touches or marked unexplained. @@ -215,7 +253,7 @@ LOT::SA-PCBA-L15 LOT 93 16 0.172043 3.028926 LOT::CP-PCB-L03 LOT 332 31 0.093373 1.643900 LOT::RM-CU-L02 LOT 803 71 0.088418 1.556663 -Pre-solve audit: 142 coverage constraints grounded (one per defective unit). +Formulation audit: 142 coverage constraints grounded (one per defective unit). Diagnosis solve: OPTIMAL objective=46.32 Root-cause diagnosis: 2 factors explain 120/142 defective units (85% coverage) factor label kind defects_on_factor lift diff --git a/defect_root_cause/defect_root_cause.py b/defect_root_cause/defect_root_cause.py index 3b69b0cd..438b7113 100644 --- a/defect_root_cause/defect_root_cause.py +++ b/defect_root_cause/defect_root_cause.py @@ -31,7 +31,9 @@ enrichments as first-class properties -- the accretive enrichment pattern. Run: - python defect_root_cause.py + python defect_root_cause.py deploys the model to the schema in raiconfig.yaml, + waits for its first refresh (which runs the solve), and prints the report from + the deployed model. Output: Prints per-stage diagnostics -- genealogy reachability validation, the @@ -168,7 +170,8 @@ model.define(Lot.lookup(id=gd.PARENT_LOT_ID).consumes(Lot.lookup(id=gd.CHILD_LOT_ID))) # Units, their final-test result, and the top-tier lots they consume. -ud = model.data(read_csv(DATA_DIR / "units.csv")) +units_csv = read_csv(DATA_DIR / "units.csv") +ud = model.data(units_csv) model.define( u := Unit.new(id=ud.UNIT_ID), u.sku(SKU.lookup(id=ud.SKU_ID)), @@ -195,26 +198,6 @@ sf = model.data(fac[fac.KIND == "SHIFT"]) model.define(f := Factor.new(id=sf.FACTOR_ID), f.kind(sf.KIND), f.label(sf.LABEL), f.shift_name(sf.REF_ID)) -n_units = model.select(Unit.id).to_df() -n_def = model.where(Unit.is_defective()).select(Unit.id).to_df() -print(f"Loaded {len(n_units)} units, {len(n_def)} defective ({len(n_def) / len(n_units):.2%} final-test failure rate)") - -# Descriptive: when did the spike start? A real investigation begins on the -# timeline -- failures by build week -- before touching genealogy. The onset is -# the first week whose rate runs well above the opening week's baseline; it -# tells the rest of the chain which window's changes to interrogate. -wk = model.select(Unit.build_week.alias("week"), Unit.defective.alias("defective")).to_df() -wk["defective"] = wk["defective"].astype(int) -timeline = wk.groupby("week").agg(units=("defective", "size"), defects=("defective", "sum")) -timeline["rate"] = timeline["defects"] / timeline["units"] -opening = timeline["rate"].iloc[0] -onset = next((int(w) for w, r in timeline["rate"].items() if r > 2 * opening), None) -print("\nWhen did the spike start? Final-test failure rate by build week:") -for week, r in timeline.iterrows(): - mark = " <- spike onset" if onset == int(week) else "" - print(f" week {int(week)}: {int(r['units']):>4} units {r['rate']:>5.1%}{mark}") -print(f"Spike onset at week {onset}: the chain interrogates what entered the line then.") - # -------------------------------------------------- # Stage 1 -- Graph: backward reachability over the lot genealogy # -------------------------------------------------- @@ -224,9 +207,8 @@ graph = Graph(model, directed=True, weighted=False, node_concept=Lot) parent, child = Lot.ref(), Lot.ref() model.where(parent.consumes(child)).define(graph.Edge.new(src=parent, dst=child)) -n_nodes = int(model.select(graph.num_nodes().alias("n")).to_df()["n"].iloc[0]) -n_edges = int(model.select(graph.num_edges().alias("n")).to_df()["n"].iloc[0]) -print(f"\nGenealogy graph: {n_nodes} lot nodes, {n_edges} parent->child edges") +num_nodes = graph.num_nodes() +num_edges = graph.num_edges() # Transitive closure: which lots each top-tier lot reaches. reach = graph.reachable(full=True) @@ -249,30 +231,12 @@ model.where((u := Unit.ref()).shift == (f := Factor.ref()).shift_name).define(Factor.touches(f, u)) -# Naive view: rank candidate factors by the raw count of defective units they -# touch. High-volume factors dominate -- a near-universal housing lot and the -# busiest placement line top the list purely because they touch almost -# everything. That is exactly the trap the contrast stage corrects. Genealogy -# closure puts lots and machines on the same footing: a deep lot is "touched by" -# every finished unit that carries it. -incidence = model.where((f := Factor.ref()).touches(u := Unit.ref())).select( - f.id.alias("factor"), f.kind.alias("kind"), u.defective.alias("defective") -).to_df() -incidence["defective"] = incidence["defective"].astype(int) -top_raw = ( - incidence.groupby(["factor", "kind"]) - .agg(units=("defective", "size"), defects=("defective", "sum")) - .reset_index().sort_values("defects", ascending=False).head(6).reset_index(drop=True) -) -print(f"\nBackward reachability built {len(incidence):,} (factor, unit) incidence facts.") -print("Top candidate factors by RAW defective units touched (pre-contrast -- note the high-volume bias):") -print(top_raw.to_string(index=False)) - # -------------------------------------------------- # Stage 2 -- Rules: contrast scoring of candidate factors # -------------------------------------------------- -BASELINE_RATE = len(n_def) / len(n_units) +# Plant-wide failure rate, from the input data. +BASELINE_RATE = int((units_csv["DEFECTIVE"] == 1).sum()) / len(units_csv) # Per-factor incidence: how many units it touches, and how many of those failed. Factor.touched_count = model.Property(f"{Factor} touches {Integer:touched_count}") @@ -301,18 +265,6 @@ Factor.defect_count >= MIN_DEFECTS ).define(Factor.is_suspect(Factor)) -suspects = model.where(Factor.is_suspect(f := Factor.ref())).select( - f.id.alias("factor"), - f.kind.alias("kind"), - f.touched_count.alias("units"), - f.defect_count.alias("defects"), - f.defect_rate.alias("rate"), - f.lift.alias("lift"), -).to_df().sort_values("lift", ascending=False).reset_index(drop=True) -print(f"\nContrast scoring: {len(suspects)} suspect factors (baseline failure rate {BASELINE_RATE:.2%}, " - f"lift>={LIFT_THRESHOLD}, support>={MIN_SUPPORT}, defects>={MIN_DEFECTS})") -print(suspects.to_string(index=False)) - # -------------------------------------------------- # Stage 3 -- Prescriptive: minimal-diagnosis set-cover MILP # -------------------------------------------------- @@ -354,81 +306,239 @@ aggs.sum(Factor.is_root_cause * Factor.select_cost) + W_UNEXPLAINED * aggs.sum(Unit.unexplained) ) -# Pre-solve audit: variables and objective registered, and -- the easy thing to -# get wrong -- one coverage constraint grounded per defective unit. -model.require(problem.num_variables() > 0) -model.require(problem.num_constraints() > 0) -model.require(problem.num_min_objectives() == 1) -n_cover = len(model.select(cover).to_df()) -assert n_cover == len(n_def), f"coverage grounded {n_cover} rows, expected {len(n_def)}" -print(f"\nPre-solve audit: {n_cover} coverage constraints grounded (one per defective unit).") - -problem.solve("highs", time_limit_sec=120) -si = problem.solve_info() -print(f"\nDiagnosis solve: {si.termination_status} objective={si.objective_value}") - -# Read back the diagnosis (populate=True wrote the solution onto is_root_cause). -diagnosis = model.where((f := Factor.ref()).is_root_cause > 0.5).select( - f.id.alias("factor"), - f.label.alias("label"), - f.kind.alias("kind"), - f.defect_count.alias("defects_on_factor"), - f.lift.alias("lift"), -).to_df().sort_values("lift", ascending=False).reset_index(drop=True) - -explained = model.where( - Unit.is_defective(), - (f := Factor.ref()).is_root_cause > 0.5, - f.touches(Unit) -).select(Unit.id.alias("unit")).to_df() -n_defective = len(n_def) -n_explained = explained["unit"].nunique() - -# Persist the headline as queryable ontology: a DiagnosisResult singleton. +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=120) + +# Problem-structure relations the formulation audit reads, declared here so they +# deploy with the model. +num_vars = problem.num_variables() +num_cons = problem.num_constraints() +num_min_objs = problem.num_min_objectives() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"defect diagnosis solve status is {String:status}") +DiagnosisCost = model.Relationship(f"defect diagnosis cost is {Float:cost}") +model.define( + SolveStatus(problem.termination_status()), DiagnosisCost(problem.objective_value()) +) + +# Persist the headline as queryable ontology: a DiagnosisResult singleton derived +# from the named root causes (populate=True writes the solution onto is_root_cause). DiagnosisResult = model.Concept("DiagnosisResult", identify_by={"id": Integer}) DiagnosisResult.n_defective = model.Property(f"{DiagnosisResult} has {Integer:n_defective}") DiagnosisResult.n_explained = model.Property(f"{DiagnosisResult} has {Integer:n_explained}") DiagnosisResult.n_root_causes = model.Property(f"{DiagnosisResult} has {Integer:n_root_causes}") DiagnosisResult.coverage = model.Property(f"{DiagnosisResult} has {Float:coverage}") +model.define(DiagnosisResult.new(id=1)) model.define( - dr := DiagnosisResult.new(id=1), - dr.n_defective(n_defective), - dr.n_explained(n_explained), - dr.n_root_causes(len(diagnosis)), - dr.coverage(n_explained / n_defective), + DiagnosisResult.n_defective( + aggs.count(Unit).per(DiagnosisResult).where(Unit.is_defective()) + ) ) +model.define( + DiagnosisResult.n_explained( + aggs.count(Unit) + .per(DiagnosisResult) + .where( + Unit.is_defective(), + (f := Factor.ref()).is_root_cause > 0.5, + f.touches(Unit), + ) + ) +) +model.define( + DiagnosisResult.n_root_causes( + aggs.count(Factor).per(DiagnosisResult).where(Factor.is_root_cause > 0.5) + ) +) +model.define( + DiagnosisResult.coverage( + floats.float(DiagnosisResult.n_explained) + / floats.float(DiagnosisResult.n_defective) + ) +) + +# -------------------------------------------------- +# Deploy, then report (`rai models deploy` skips this block) +# -------------------------------------------------- -print(f"\nRoot-cause diagnosis: {len(diagnosis)} factors explain {n_explained}/{n_defective} " - f"defective units ({n_explained / n_defective:.0%} coverage)") -print(diagnosis.to_string(index=False)) - -# Corroborating evidence per named cause, the way a real RCA write-up reads: -# the defect signature it concentrates (a paste fault should present as cold -# solder, a reflow fault as solder bridging), plus a kind-specific tell -- -# supplier and receipt date for a lot, calibration age for a machine. All of it -# is read back from ontology already loaded; the diagnosis is the hypothesis, -# this is the supporting evidence an engineer would confirm physically. -sig = model.where( - (f := Factor.ref()).is_root_cause > 0.5, - f.touches(u := Unit.ref()), - u.is_defective() -).select(f.id.alias("factor"), u.defect_type.alias("defect_type")).to_df() -print("\nCorroborating evidence per root cause:") -for _, row in diagnosis.iterrows(): - fid, kind, label = row["factor"], row["kind"], row["label"] - types = sig[sig["factor"] == fid]["defect_type"] - dom = types.value_counts() - signature = f"{dom.iloc[0] / len(types):.0%} {dom.index[0]}" - ref = fid.split("::", 1)[1] - if kind == "LOT": - tell_df = model.where( - (lo := Lot.ref()).id == ref, - lo.supplier(sp := Supplier.ref()) - ).select(sp.name.alias("supplier"), lo.received_date.alias("recv")).to_df() - tell = f"supplier {tell_df['supplier'].iloc[0]}, received {tell_df['recv'].iloc[0]}" - else: - tell_df = model.where((mm := Machine.ref()).id == ref).select( - mm.calibration_age_days.alias("age") - ).to_df() - tell = f"{int(tell_df['age'].iloc[0])} days since last calibration" - print(f" {label:<26} {signature:<18} {tell}") +if __name__ == "__main__": + # Deploy and wait for the first refresh, which runs the solve. + model.deploy() + + n_units = model.select(Unit.id).to_df() + n_def = model.where(Unit.is_defective()).select(Unit.id).to_df() + print( + f"Loaded {len(n_units)} units, {len(n_def)} defective ({len(n_def) / len(n_units):.2%} final-test failure rate)" + ) + + # Descriptive: when did the spike start? A real investigation begins on the + # timeline -- failures by build week -- before touching genealogy. The onset is + # the first week whose rate runs well above the opening week's baseline; it + # tells the rest of the chain which window's changes to interrogate. + wk = model.select( + Unit.build_week.alias("week"), Unit.defective.alias("defective") + ).to_df() + wk["defective"] = wk["defective"].astype(int) + timeline = wk.groupby("week").agg( + units=("defective", "size"), defects=("defective", "sum") + ) + timeline["rate"] = timeline["defects"] / timeline["units"] + opening = timeline["rate"].iloc[0] + onset = next((int(w) for w, r in timeline["rate"].items() if r > 2 * opening), None) + print("\nWhen did the spike start? Final-test failure rate by build week:") + for week, r in timeline.iterrows(): + mark = " <- spike onset" if onset == int(week) else "" + print( + f" week {int(week)}: {int(r['units']):>4} units {r['rate']:>5.1%}{mark}" + ) + print( + f"Spike onset at week {onset}: the chain interrogates what entered the line then." + ) + + n_nodes = int(model.select(num_nodes.alias("n")).to_df()["n"].iloc[0]) + n_edges = int(model.select(num_edges.alias("n")).to_df()["n"].iloc[0]) + print(f"\nGenealogy graph: {n_nodes} lot nodes, {n_edges} parent->child edges") + + # Naive view: rank candidate factors by the raw count of defective units they + # touch. High-volume factors dominate -- a near-universal housing lot and the + # busiest placement line top the list purely because they touch almost + # everything. That is exactly the trap the contrast stage corrects. Genealogy + # closure puts lots and machines on the same footing: a deep lot is "touched by" + # every finished unit that carries it. + incidence = ( + model.where((f := Factor.ref()).touches(u := Unit.ref())) + .select( + f.id.alias("factor"), f.kind.alias("kind"), u.defective.alias("defective") + ) + .to_df() + ) + incidence["defective"] = incidence["defective"].astype(int) + top_raw = ( + incidence.groupby(["factor", "kind"]) + .agg(units=("defective", "size"), defects=("defective", "sum")) + .reset_index() + .sort_values("defects", ascending=False) + .head(6) + .reset_index(drop=True) + ) + print( + f"\nBackward reachability built {len(incidence):,} (factor, unit) incidence facts." + ) + print( + "Top candidate factors by RAW defective units touched (pre-contrast -- note the high-volume bias):" + ) + print(top_raw.to_string(index=False)) + + suspects = ( + model.where(Factor.is_suspect(f := Factor.ref())) + .select( + f.id.alias("factor"), + f.kind.alias("kind"), + f.touched_count.alias("units"), + f.defect_count.alias("defects"), + f.defect_rate.alias("rate"), + f.lift.alias("lift"), + ) + .to_df() + .sort_values("lift", ascending=False) + .reset_index(drop=True) + ) + print( + f"\nContrast scoring: {len(suspects)} suspect factors (baseline failure rate {BASELINE_RATE:.2%}, " + f"lift>={LIFT_THRESHOLD}, support>={MIN_SUPPORT}, defects>={MIN_DEFECTS})" + ) + print(suspects.to_string(index=False)) + + # Formulation audit: variables and objective registered, and -- the easy thing to + # get wrong -- one coverage constraint grounded per defective unit. + n_vars = model.select(num_vars).to_df().iloc[:, 0].tolist() + n_cons = model.select(num_cons).to_df().iloc[:, 0].tolist() + n_objs = model.select(num_min_objs).to_df().iloc[:, 0].tolist() + assert n_vars[0] > 0, f"no variables: {n_vars}" + assert n_cons[0] > 0, f"no constraints: {n_cons}" + assert n_objs == [1], f"expected one minimization objective, got {n_objs}" + n_cover = len(model.select(cover).to_df()) + assert n_cover == len(n_def), ( + f"coverage grounded {n_cover} rows, expected {len(n_def)}" + ) + print( + f"\nFormulation audit: {n_cover} coverage constraints grounded (one per defective unit)." + ) + + # Status and objective of the latest refresh's solve. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(DiagnosisCost).to_df().iloc[:, 0].tolist() + print( + f"\nDiagnosis solve: {', '.join(status)} objective={round(cost[0], 9) if cost else None}" + ) + + # Read back the diagnosis (populate=True writes the solution onto is_root_cause). + diagnosis = ( + model.where((f := Factor.ref()).is_root_cause > 0.5) + .select( + f.id.alias("factor"), + f.label.alias("label"), + f.kind.alias("kind"), + f.defect_count.alias("defects_on_factor"), + f.lift.alias("lift"), + ) + .to_df() + .sort_values("lift", ascending=False) + .reset_index(drop=True) + ) + + # The headline, read back from the DiagnosisResult singleton. + headline = model.select( + DiagnosisResult.n_defective.alias("n_defective"), + DiagnosisResult.n_explained.alias("n_explained"), + DiagnosisResult.n_root_causes.alias("n_root_causes"), + DiagnosisResult.coverage.alias("coverage"), + ).to_df() + n_defective = int(headline["n_defective"].iloc[0]) + n_explained = int(headline["n_explained"].iloc[0]) + print( + f"\nRoot-cause diagnosis: {int(headline['n_root_causes'].iloc[0])} factors explain {n_explained}/{n_defective} " + f"defective units ({headline['coverage'].iloc[0]:.0%} coverage)" + ) + print(diagnosis.to_string(index=False)) + + # Corroborating evidence per named cause, the way a real RCA write-up reads: + # the defect signature it concentrates (a paste fault should present as cold + # solder, a reflow fault as solder bridging), plus a kind-specific tell -- + # supplier and receipt date for a lot, calibration age for a machine. All of it + # is read back from ontology already loaded; the diagnosis is the hypothesis, + # this is the supporting evidence an engineer would confirm physically. + sig = ( + model.where( + (f := Factor.ref()).is_root_cause > 0.5, + f.touches(u := Unit.ref()), + u.is_defective(), + ) + .select(f.id.alias("factor"), u.defect_type.alias("defect_type")) + .to_df() + ) + print("\nCorroborating evidence per root cause:") + for _, row in diagnosis.iterrows(): + fid, kind, label = row["factor"], row["kind"], row["label"] + types = sig[sig["factor"] == fid]["defect_type"] + dom = types.value_counts() + signature = f"{dom.iloc[0] / len(types):.0%} {dom.index[0]}" + ref = fid.split("::", 1)[1] + if kind == "LOT": + tell_df = ( + model.where( + (lo := Lot.ref()).id == ref, lo.supplier(sp := Supplier.ref()) + ) + .select(sp.name.alias("supplier"), lo.received_date.alias("recv")) + .to_df() + ) + tell = f"supplier {tell_df['supplier'].iloc[0]}, received {tell_df['recv'].iloc[0]}" + else: + tell_df = ( + model.where((mm := Machine.ref()).id == ref) + .select(mm.calibration_age_days.alias("age")) + .to_df() + ) + tell = f"{int(tell_df['age'].iloc[0])} days since last calibration" + print(f" {label:<26} {signature:<18} {tell}") From 6a5602dd59d8792c132ef9c75ba93deaa5dca44c Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 17:09:26 -0800 Subject: [PATCH 041/162] book_slate_recommendation: describe the deployed status check The runner no longer verifies constraints after the solve, and it accepts TIME_LIMIT with a slate, as the bundled slice ends there. --- book_slate_recommendation/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book_slate_recommendation/README.md b/book_slate_recommendation/README.md index a75c7b8b..9469c7cc 100644 --- a/book_slate_recommendation/README.md +++ b/book_slate_recommendation/README.md @@ -210,7 +210,7 @@ The pipeline runs in three stages over the shared ontology. Its architectural ce **Stage 2 — Graph analysis (hero-slot embeddedness).** A `Graph` is built from `Book.similar_to`, and triangle count returns the per-book count of similarity triangles each book participates in, stored as `Book.triangle_count`. Triangle count is graph-derived: it depends on the topology of the similarity graph, not on a per-book scalar that could be supplied externally. Stage 3 uses it to pin the hero slot to a densely embedded book. -**Stage 3 — Prescriptive constraint program (slot assignment).** Each candidate's slot is a decision in `1..K+1`, where slots `1..K` are slate positions (1 = hero, K = bottom of row) and slot K+1 is the unpicked sentinel. The K+1 encoding lets the position weight `(K + 1 - slot)` be zero for unpicked candidates, so no auxiliary picked/unpicked indicator is needed. The model enforces ten constraints — cardinality (exactly K picks per user), slot uniqueness, already-read exclusion, author uniqueness, subject-concentration cap, freshness floor, originals-exposure floor, cold-start cap, hero pin, and explanation-path floor — under a position-weighted engagement-decay objective. The objective weights each pick by `(K + 1 - slot) * path_count_total`, largest at slot 1, so the solver places the highest-path-support candidates at the top of the row — the canonical position-decay engagement model. After the solve, the runner verifies every constraint, requires an `OPTIMAL` status, and prints the chosen slate, subject distribution, and per-pick explanation support. +**Stage 3 — Prescriptive constraint program (slot assignment).** Each candidate's slot is a decision in `1..K+1`, where slots `1..K` are slate positions (1 = hero, K = bottom of row) and slot K+1 is the unpicked sentinel. The K+1 encoding lets the position weight `(K + 1 - slot)` be zero for unpicked candidates, so no auxiliary picked/unpicked indicator is needed. The model enforces ten constraints — cardinality (exactly K picks per user), slot uniqueness, already-read exclusion, author uniqueness, subject-concentration cap, freshness floor, originals-exposure floor, cold-start cap, hero pin, and explanation-path floor — under a position-weighted engagement-decay objective. The objective weights each pick by `(K + 1 - slot) * path_count_total`, largest at slot 1, so the solver places the highest-path-support candidates at the top of the row — the canonical position-decay engagement model. After the deploy's refresh runs the solve, the runner requires an `OPTIMAL` or `TIME_LIMIT` status and prints the chosen slate, subject distribution, and per-pick explanation support. See `book_slate_recommendation.py` for the implementation and `runbook.md` for the skill-driven reproduction. From e35e438e77e2d22605f29c229aeed5af168616d5 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:05:08 -0800 Subject: [PATCH 042/162] diet: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/diet_plan.py (`python -m queries.diet_plan`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- diet/README.md | 144 ++++++++++++++++++---------- diet/{diet.py => model/__init__.py} | 43 ++------- diet/queries/diet_plan.py | 28 ++++++ diet/runbook.md | 4 +- 4 files changed, 132 insertions(+), 87 deletions(-) rename diet/{diet.py => model/__init__.py} (74%) create mode 100644 diet/queries/diet_plan.py diff --git a/diet/README.md b/diet/README.md index 322e4b3f..37795a38 100644 --- a/diet/README.md +++ b/diet/README.md @@ -35,8 +35,8 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ## What's included -- **Model**: two concepts (`Food`, `Nutrient`), a `Scenario` concept, per-food decision variables, nutrient-bound constraints, and a cost-minimizing objective — all in `diet.py`. -- **Runner**: `diet.py`, a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. +- **Model**: two concepts (`Food`, `Nutrient`), a `Scenario` concept, per-food decision variables, nutrient-bound constraints, and a cost-minimizing objective — all in `model/__init__.py`, which `rai models deploy --path model` deploys to your Snowflake account, where each refresh runs the solve. +- **Query**: `queries/diet_plan.py`, which prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/foods.csv` (foods with cost and per-nutrient content) and `data/nutrients.csv` (nutrient min/max bounds). - **Outputs**: per-scenario termination status, objective cost, and a table of the foods (and amounts) in each least-cost basket, printed to stdout. @@ -45,7 +45,7 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -53,28 +53,61 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .DIET_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .DIET_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .DIET_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .DIET_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .DIET_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .DIET_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .DIET_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .DIET_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/diet.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/diet.zip unzip diet.zip cd diet ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -93,51 +126,63 @@ Built using **prescriptive reasoning** (linear programming with continuous decis pat_user: pat_warehouse: ``` - Without the `deployment` section, `python diet.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .DIET_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .DIET_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .DIET_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .DIET_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .DIET_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .DIET_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .DIET_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .DIET_META TO APPLICATION ; +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: + + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.DIET` if it doesn't exist. -5. Run: + The deploy creates the public schema `.DIET` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + ```bash - python diet.py + python -m queries.diet_plan ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```text + Solve result: + • status: OPTIMAL + • objective: 35.486583333 + + Diet plan per scenario: + scenario food cost amount + 0 baseline hamburger 2.49 0.604514 + 1 baseline icecream 1.59 2.591319 + 2 baseline milk 0.89 6.970139 + 3 scaling_120pct hamburger 2.49 0.725417 + 4 scaling_120pct icecream 1.59 3.109583 + 5 scaling_120pct milk 0.89 8.364167 + 6 scaling_80pct hamburger 2.49 0.483611 + 7 scaling_80pct icecream 1.59 2.073056 + 8 scaling_80pct milk 0.89 5.576111 + ``` + + The formulation printed above the solve result is trimmed here. The report checks the solve's termination status and stops unless it is `OPTIMAL`. The least-cost basket is the same three foods in every scenario, scaled with the requirements, for a per-scenario cost of $9.46 / $11.83 / $14.19 at 0.8 / 1.0 / 1.2 times the bounds. See `runbook.md` for the step-by-step walkthrough. + + To solve again without changing the model, refresh the deployed model, then run the query again: + ```sql CALL .DIET_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output — a few lines confirm a successful run: +7. **Tear down the deployment** - ```text - Solve result: - • status: OPTIMAL - • objective: 35.49 + Preview what the teardown removes, then remove it: - Diet plan per scenario (0.8 / 1.0 / 1.2 demand): the same baseline - basket — hamburger + icecream + milk — scaled per scenario, for a - per-scenario cost of $9.46 / $11.83 / $14.19. + ```bash + rai models teardown + rai models teardown --force ``` - The full per-scenario diet plan prints above; see `runbook.md` for the complete log. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -146,13 +191,16 @@ Built using **prescriptive reasoning** (linear programming with continuous decis ├── README.md # this file ├── runbook.md # step-by-step analyst walkthrough ├── pyproject.toml # dependencies -├── diet.py # main script (model, constraints, scenarios, solve) +├── model/ +│ └── __init__.py # the deployed model (concepts, constraints, scenarios, solve) +├── queries/ +│ └── diet_plan.py # report from the deployed model └── data/ ├── foods.csv # foods with cost and per-nutrient content └── nutrients.csv # nutrient min/max bounds ``` -**Start here**: run `python diet.py` for the full model and scenario solve end to end, or follow `runbook.md` to rebuild it step by step. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.diet_plan` for the full model and scenario solve end to end, or follow `runbook.md` to rebuild it step by step. ## Sample data @@ -169,7 +217,7 @@ The model is small and self-contained: two source concepts plus a `Scenario` con - **Primary identifiers**: `Food.name` and `Nutrient.name` (both strings); `Scenario.scenario_name` (string). - **Important invariants**: nutrient `min` and `max` bounds are non-negative and `min <= max`; each food's decision amount is non-negative (`lower=0`); each food's per-nutrient content is keyed by a nutrient that exists in `nutrients.csv`. -For the full concept and property definitions, see `diet.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -187,7 +235,7 @@ foods.csv + nutrients.csv → concepts → decision variables → constraints + 4. **Scenario analysis.** A first-class `Scenario` concept scales every nutrient bound by a factor (0.8 / 1.0 / 1.2 here), so tighter and looser requirements solve together in a single solve and their costs are directly comparable. -The prescriptive decision variable appears in the model schema alongside the source-data properties — a unified view of everything the model knows, including variables added by `solve_for()`. See `diet.py` for the implementation and `runbook.md` for the skill-driven reproduction. +The prescriptive decision variable appears in the model schema alongside the source-data properties — a unified view of everything the model knows, including variables added by `solve_for()`. See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -200,7 +248,7 @@ Focus on the first changes most users will make. ### Tune parameters -- Edit the scenario rows in `diet.py` (the `("scaling_80pct", 0.8)`, `("baseline", 1.0)`, `("scaling_120pct", 1.2)` tuples) to test different scaling factors, or add rows for finer resolution. +- Edit the scenario rows in `model/__init__.py` (the `("scaling_80pct", 0.8)`, `("baseline", 1.0)`, `("scaling_120pct", 1.2)` tuples) to test different scaling factors, or add rows for finer resolution. - Adjust the solver time limit (`time_limit_sec`) if you scale up to many foods and nutrients. ### Extend the model @@ -223,9 +271,9 @@ The nutritional bounds may be too tight for the available foods. Check that at l
-rai init fails or connection errors +Connection errors -Ensure your Snowflake credentials are configured correctly and that the RAI Native App is installed on your account. Run `rai init` again and verify the connection settings. +Ensure your Snowflake credentials are configured correctly and that the RAI Native App is installed on your account. Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings.
diff --git a/diet/diet.py b/diet/model/__init__.py similarity index 74% rename from diet/diet.py rename to diet/model/__init__.py index fbab3e41..0d9eb191 100644 --- a/diet/diet.py +++ b/diet/model/__init__.py @@ -1,6 +1,6 @@ -"""Diet optimization (prescriptive optimization) template. +"""Diet optimization (prescriptive optimization) template: the model. -This script demonstrates a classic diet linear optimization problem in RelationalAI: +This package declares a classic diet linear optimization problem in RelationalAI: - Map sample data from CSVs describing foods, nutrients, and per-food nutrient quantities. - Model foods and nutrients as *concepts* with typed properties. @@ -16,13 +16,9 @@ - One solve handles all scaling levels; results extracted via model.select(). Run: - `python diet.py` deploys the model to the schema in raiconfig.yaml, waits for - its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints per-scenario termination status, objective value, and a table of foods - with non-trivial amounts, followed by a scenario analysis summary. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.diet_plan` then prints the report from the deployed model. """ from pathlib import Path @@ -35,7 +31,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & map data @@ -120,30 +116,3 @@ SolveStatus = model.Relationship(f"diet solve status is {String:status}") TotalCost = model.Relationship(f"diet total cost is {Float:cost}") model.define(SolveStatus(problem.termination_status()), TotalCost(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal diet plan: termination status {status}") - print(f"• objective: {round(cost[0], 9)}\n") - - print("\nDiet plan per scenario:") - model.select( - Scenario.scenario_name.alias("scenario"), - Food.name.alias("food"), - Food.cost, - x_amt.alias("amount"), - ).where(Food.x_amount(Scenario, x_amt), x_amt > 0.001).inspect() diff --git a/diet/queries/diet_plan.py b/diet/queries/diet_plan.py new file mode 100644 index 00000000..aa68bf85 --- /dev/null +++ b/diet/queries/diet_plan.py @@ -0,0 +1,28 @@ +"""Diet plan report: reads the deployed diet model. + +Run `python -m queries.diet_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and objective of the latest refresh's solve, and +the foods (and amounts) in each scenario's least-cost basket. +""" + +from model import Food, Scenario, SolveStatus, TotalCost, model, problem, x_amt + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal diet plan: termination status {status}") + print(f"• objective: {round(cost[0], 9)}\n") + + print("\nDiet plan per scenario:") + model.select( + Scenario.scenario_name.alias("scenario"), + Food.name.alias("food"), + Food.cost, + x_amt.alias("amount"), + ).where(Food.x_amount(Scenario, x_amt), x_amt > 0.001).inspect() diff --git a/diet/runbook.md b/diet/runbook.md index 6fd61393..92166c33 100644 --- a/diet/runbook.md +++ b/diet/runbook.md @@ -38,7 +38,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem What's the minimum-cost amount of each food (continuous, non-negative) that keeps every nutrient's total intake within its [min, max] bounds? Solve three requirement scenarios in one solve — bounds scaled to 80%, 100%, and 120% — with the scenario as a first-class concept and the amount decision indexed by (food, scenario). Minimize total cost across all three, and persist the amounts to the ontology. -**Response:** OPTIMAL (HiGHS), total objective $35.49 across the three scenarios. The amount decision is written back as `Food.x_amount(Scenario) -> amount` so the next step can read it; the solve is sub-second (18 simplex iterations). In `diet.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** OPTIMAL (HiGHS), total objective $35.49 across the three scenarios. The amount decision is written back as `Food.x_amount(Scenario) -> amount` so the next step can read it; the solve is sub-second (18 simplex iterations). In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the per-scenario diet plan @@ -48,4 +48,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 9 foods with cost and nutrient content, 4 nutrients with min/max bounds. Full model in `diet.py`. +Bundled CSVs in `data/`: 9 foods with cost and nutrient content, 4 nutrients with min/max bounds. Full model in `model/__init__.py`; the report in `queries/diet_plan.py`. From 154f0c5624a2fb78a29fed81c1cac3c69701534c Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:05:23 -0800 Subject: [PATCH 043/162] factory_production: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/plan_and_sensitivity.py (`python -m queries.plan_and_sensitivity`), which reads the plan, shadow prices, reduced costs and basis statuses from the deployed model. Its checks and the model's input-data guards raise explicitly, and the plan check verifies one row per product before comparing quantities. Rewrite the README Quickstart as the seven deployment steps. --- factory_production/README.md | 132 +++++--- factory_production/factory_production.py | 311 ------------------ factory_production/model/__init__.py | 179 ++++++++++ .../queries/plan_and_sensitivity.py | 158 +++++++++ factory_production/runbook.md | 2 +- 5 files changed, 426 insertions(+), 356 deletions(-) delete mode 100644 factory_production/factory_production.py create mode 100644 factory_production/model/__init__.py create mode 100644 factory_production/queries/plan_and_sensitivity.py diff --git a/factory_production/README.md b/factory_production/README.md index 0d2a80f9..a6e22f3e 100644 --- a/factory_production/README.md +++ b/factory_production/README.md @@ -41,7 +41,8 @@ Manufacturing operations must decide how much of each product to make at each fa ## What's included -- `factory_production.py` -- Main script defining the optimization model, constraints, objective, and sensitivity read-back +- `model/__init__.py` -- the deployed model: concepts, data, constraints, objective, and the solve with sensitivity analysis, deployed with `rai models deploy --path model` +- `queries/plan_and_sensitivity.py` -- the report from the deployed model: production plan, shadow prices, reduced costs, basis statuses, and capacity utilization - `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself - `data/factories.csv` -- Factory names and available resource-hours - `data/products.csv` -- Products with factory assignment, production rate, profit, and demand cap @@ -51,7 +52,7 @@ Manufacturing operations must decide how much of each product to make at each fa ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -60,28 +61,61 @@ Manufacturing operations must decide how much of each product to make at each fa ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .FACTORY_PRODUCTION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/factory_production.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/factory_production.zip unzip factory_production.zip cd factory_production ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -100,39 +134,27 @@ Manufacturing operations must decide how much of each product to make at each fa pat_user: pat_warehouse: ``` - Without the `deployment` section, `python factory_production.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .FACTORY_PRODUCTION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .FACTORY_PRODUCTION_META TO APPLICATION ; - ``` - The deploy creates the public schema `.FACTORY_PRODUCTION` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python factory_production.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .FACTORY_PRODUCTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema `.FACTORY_PRODUCTION` if it doesn't exist and waits for the model's first refresh, which runs the solve with sensitivity analysis. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the production plan and its sensitivity report from the deployed model: + + ```bash + python -m queries.plan_and_sensitivity ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output (model and solver display trimmed): ```text Baseline status: OPTIMAL, total profit: 200000.00 @@ -165,7 +187,26 @@ Manufacturing operations must decide how much of each product to make at each fa steel_factory 40.0 40.0 0.0 ``` - Reading the result: `steel_factory` fills all 40 hours (it is the binding factory, so its capacity prices at **+4200/hour** -- the per-hour profit of the swing product `coils`, i.e. its 30/unit profit times its 140 units/hour rate). `amazing_brewery` meets all demand in 25 of its 30 hours, so its capacity is **slack** and prices at **0** -- its bottleneck is demand, not capacity. Each of these demand-capped products (`bands`, `stouts`, `ales`) carries a positive reduced cost here: the profit you would gain per extra unit of demand allowed. + The formulation printed above the plan is trimmed here. The report checks the solve's status, the plan, and the marginals against the bundled data's optimum, and stops with an error if any of them differs. + + Reading the result: `steel_factory` fills all 40 hours (it is the binding factory, so its capacity prices at +4200/hour -- the per-hour profit of the swing product `coils`, i.e. its 30/unit profit times its 140 units/hour rate). `amazing_brewery` meets all demand in 25 of its 30 hours, so its capacity is slack and prices at 0 -- its bottleneck is demand, not capacity. Each of these demand-capped products (`bands`, `stouts`, `ales`) carries a positive reduced cost here: the profit you would gain per extra unit of demand allowed. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .FACTORY_PRODUCTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -174,13 +215,16 @@ Manufacturing operations must decide how much of each product to make at each fa ├── README.md # this file ├── runbook.md # step-by-step analyst walkthrough ├── pyproject.toml # dependencies -├── factory_production.py # main script (LP model + sensitivity read-back) +├── model/ +│ └── __init__.py # the deployed model (LP + sensitivity solve) +├── queries/ +│ └── plan_and_sensitivity.py # report and sensitivity read-back from the deployed model └── data/ ├── factories.csv # factory names and available resource-hours └── products.csv # products with factory, rate, profit, demand cap ``` -**Start here**: run `python factory_production.py` for the full solve and sensitivity report end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.plan_and_sensitivity` for the full solve and sensitivity report end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -189,7 +233,7 @@ The bundled data is a small, hand-checkable product-mix setting -- two factories - **`data/factories.csv`** (2 rows) -- `steel_factory` (40 available hours) and `amazing_brewery` (30 available hours). - **`data/products.csv`** (4 rows) -- `bands` and `coils` at `steel_factory`, `stouts` and `ales` at `amazing_brewery`, each with a production `rate` (units per hour), a unit `profit`, and a `demand` cap. -Product names are unique across factories, so the plan can be keyed by product name alone. The script asserts the expected factory and product names on load, so an edit to `data/` fails loudly rather than silently passing the name-gated sensitivity checks on zero rows. +Product names are unique across factories, so the plan can be keyed by product name alone. `model/__init__.py` checks the expected factory and product names on load, so an edit to `data/` fails loudly rather than silently passing the name-gated sensitivity checks on zero rows. ## Model overview @@ -199,7 +243,7 @@ The model has two concepts and one relationship linking them; the decision varia - **Primary identifiers**: `Factory` by `name`; `Product` by `name` plus `factory_name` (a composite identity). - **Important invariants**: production quantities are non-negative and capped at each product's demand; each factory's total resource usage stays within its available hours; profit and rate are positive. -For the full concept and property definitions, see `factory_production.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -225,7 +269,7 @@ Because the objective is a maximization, the signs mirror a minimize-cost model: Sensitivity marginals are exact for a linear program. They describe the rate of change at the current optimum -- the range over which that rate holds is not reported (there is no RHS/coefficient ranging) -- and a large, discrete change (adding a factory, removing a product) is a structural change best answered by re-solving. -See `factory_production.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` and `queries/plan_and_sensitivity.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -233,7 +277,7 @@ See `factory_production.py` for the implementation and `runbook.md` for the skil - Replace `data/factories.csv` (columns `name`, `avail`) and `data/products.csv` (columns `factory_name`, `name`, `rate`, `profit`, `demand`) with your own rows. The model and the per-factory marginals pick up new factories and products automatically. - Keep product names unique across factories, or key the plan by the composite `(factory_name, name)` instead of by name alone. -- Update the `EXPECTED_FACTORIES` / `EXPECTED_PRODUCTS` guard sets at the top of the script (or remove them) so the load-time assertions match your data. +- Update the `EXPECTED_FACTORIES` / `EXPECTED_PRODUCTS` guard sets in `model/__init__.py` (or remove them) so the load-time checks match your data, and update or remove the report's checks of the bundled optimum in `queries/plan_and_sensitivity.py`. ### Tune parameters @@ -265,9 +309,9 @@ Usually that factory has idle resource-hours -- slack capacity, so an extra hour
-rai init fails or connection errors +Connection errors -Ensure your Snowflake credentials are configured correctly and that the RAI Native App is installed on your account. Run `rai init` again and verify the connection settings. +Ensure your Snowflake credentials are configured correctly and that the RAI Native App is installed on your account. Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings.
diff --git a/factory_production/factory_production.py b/factory_production/factory_production.py deleted file mode 100644 index 4fca9a2d..00000000 --- a/factory_production/factory_production.py +++ /dev/null @@ -1,311 +0,0 @@ -"""Factory production (prescriptive optimization) template. - -This script demonstrates a product-mix linear program in RelationalAI that -maximizes total profit subject to per-factory resource capacity, then reads back -the *marginals* a planner asks next: - -- Load sample CSVs describing factories (resource availability) and the products - each factory makes (production rate, unit profit, demand cap). -- Model them as *concepts* with typed properties. -- Choose a non-negative production quantity per product, capped by its demand. -- Limit each factory's total resource usage to its available hours. -- Maximize total profit. - -A plain solve answers "what is the most profitable production plan?". Requesting -``sensitivity=True`` on the solve ALSO answers the marginal questions -- in one solve, -read straight off the variable / constraint objects (the same attribute style as -``.name`` or ``.lower``): - -- **Shadow price** of a factory's capacity (``cap.shadow_price``): how much total - profit moves per extra hour of that factory. A capacity with idle hours prices at - zero (it is not the bottleneck); a positive price flags a binding capacity worth - expanding -- so this ranks which factory to expand first. (The implication runs one - way: slack => zero price, and a positive price => binding; a binding capacity *can* - still price at zero under degeneracy, though none does in this data.) -- **Reduced cost** of a product (``quantity_var.reduced_cost``) and its **basis - status** (``quantity_var.basis_status``): a product held at its demand cap shows a - positive reduced cost here (the extra profit per unit of demand allowed); the swing - product that sets the binding factory's marginal price is BASIC at ~0. - -Sign convention (the mirror image of a minimize-cost model): with a MAXIMIZE -objective, a binding ``<=`` capacity prices >= 0 and a product at its demand -*upper* bound has reduced cost >= 0. (Contrast the ``supplier_reliability`` -template, which minimizes cost: there a binding ``<=`` capacity prices <= 0.) Note -the two sensitivity objects come from two different modeling choices: capacity is a -*constraint* (its marginal is a shadow price), while the demand cap is a variable -*upper bound* (its marginal is a reduced cost). - -A constraint declared with ``keyed_by={"factory": Factory}`` carries an entity -back-pointer to what it grounds over (``cap.factory``), so a marginal joins to that -entity's own data by ENTITY KEY (``cap.factory.avail``) -- never by parsing the -constraint name string. (A variable's back-pointer, such as ``quantity_var.product``, -is derived automatically from its field names; a constraint's is declared.) - -Run: - `python factory_production.py` deploys the model to the schema in raiconfig.yaml, waits for - its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints the production plan, each factory's capacity shadow price, each product's - reduced cost and basis status, and a per-factory capacity-utilization summary. -""" - -from pathlib import Path - -from pandas import read_csv -from relationalai.semantics import Float, Integer, Model, String, sum -from relationalai.semantics.reasoners.prescriptive import Problem - -# -------------------------------------------------- -# Configure inputs -# -------------------------------------------------- - -DATA_DIR = Path(__file__).parent / "data" - -# -------------------------------------------------- -# Define semantic model & load data -# -------------------------------------------------- - -model = Model("factory_production") -Concept, Property = model.Concept, model.Property - -# Factory concept: factories with total resource availability (hours). -Factory = Concept("Factory", identify_by={"name": String}) -Factory.avail = Property(f"{Factory} has {Float:avail}") - -# Load factories from CSV. -factory_csv = read_csv(DATA_DIR / "factories.csv") -model.define(Factory.new(model.data(factory_csv).to_schema())) - -# Product concept: products with production rate, unit profit, demand cap, and parent factory. -Product = Concept("Product", identify_by={"name": String, "factory_name": String}) -Product.factory = Property(f"{Product} is produced by {Factory}") -Product.rate = Property(f"{Product} has {Float:rate}") -Product.profit = Property(f"{Product} has {Float:profit}") -Product.demand = Property(f"{Product} has {Integer:demand}") - -# Load products from CSV and link each product to its factory. -product_csv = read_csv(DATA_DIR / "products.csv") -product_data = model.data(product_csv) -model.define( - p := Product.new(name=product_data.name, factory_name=product_data.factory_name), - p.rate(product_data.rate), - p.profit(product_data.profit), - p.demand(product_data.demand), -) -model.define(Product.factory(Factory)).where( - Product.factory_name(product_data.factory_name), - Factory.name(product_data.factory_name), -) - -# Guard the reference data. Every sensitivity assertion below is gated by a hard-coded -# factory/product name, so if data/ is edited those asserts could silently pass on zero -# rows. Fail loudly here instead, with a clear message, the moment the data drifts. -EXPECTED_FACTORIES = {"steel_factory", "amazing_brewery"} -EXPECTED_PRODUCTS = {"bands", "coils", "stouts", "ales"} -assert set(factory_csv["name"]) == EXPECTED_FACTORIES, ( - f"factories.csv changed: {set(factory_csv['name'])} != {EXPECTED_FACTORIES}" -) -assert set(product_csv["name"]) == EXPECTED_PRODUCTS, ( - f"products.csv changed: {set(product_csv['name'])} != {EXPECTED_PRODUCTS}" -) -# Product names are unique across factories, so the plan is keyed by name alone below. -assert product_csv["name"].is_unique, ( - f"product names are not unique across factories: {list(product_csv['name'])}" -) - -# -------------------------------------------------- -# Model the decision problem -# -------------------------------------------------- - -# Variable: quantity[product] = amount produced (0 .. demand cap). -Product.x_quantity = Property(f"{Product} has {Float:quantity}") - -# -------------------------------------------------- -# Baseline solve with sensitivity analysis -# -------------------------------------------------- -# One product-mix LP over ALL factories. They share no resources, so the joint -# optimum is just each factory solved independently -- but a single solve lets us -# read every factory's marginals side by side off the captured handles. - -problem = Problem(model, Float) - -# Variable: production quantity per product. The demand cap is the variable's UPPER -# BOUND (not a separate constraint), so its marginal surfaces as the variable's -# reduced cost below. -quantity_var = problem.solve_for( - Product.x_quantity, - name=["qty", Product.factory.name, Product.name], - lower=0, - upper=Product.demand, -) - -# Constraint: each factory's total resource usage <= its available hours, over the -# factories that make products. Captured as a handle; ``keyed_by`` declares each -# instance's ENTITY KEY, so its shadow price reads back through that key (cap.factory), -# never by parsing the constraint name string; the per-factory name is a readable label. -cap = problem.satisfy( - model.where(Product.factory == Factory).require( - sum(Product.x_quantity / Product.rate) - .where(Product.factory == Factory) - .per(Factory) - <= Factory.avail - ), - name=["cap", Factory.name], - keyed_by={"factory": Factory}, -) - -# Objective: maximize total profit across all factories. -problem.maximize(sum(Product.profit * Product.x_quantity)) - -# Each refresh of the deployed model runs this solve, with sensitivity analysis. -problem.configure("highs", time_limit_sec=60, sensitivity=True) - -# Deploy the rules behind problem.display() with the model. -problem.install_display_strings() - -# The solve's status and objective, published through the model's own relationships. -SolveStatus = model.Relationship(f"factory production solve status is {String:status}") -TotalProfit = model.Relationship(f"factory production total profit is {Float:profit}") -model.define(SolveStatus(problem.termination_status()), TotalProfit(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - profit = model.select(TotalProfit).to_df().iloc[:, 0].tolist() - assert status == ["OPTIMAL"], f"termination status {status}" - # Optimum: steel_factory fills all 40 hrs (bands to its 6000 cap = 30 hrs, coils takes - # the last 10 hrs = 1400 units); amazing_brewery makes both products to their demand - # caps in 25 of its 30 hrs. Profit = 192000 + 8000 = 200000. - assert len(profit) == 1 and abs(profit[0] - 200000) < 0.01, ( - f"total profit changed: expected 200000, got {profit}" - ) - - print(f"\nBaseline status: {status[0]}, total profit: {profit[0]:.2f}") - - # --- The solved production plan ------------------------------------------------- - # Read every product's solved quantity once from Product.x_quantity, which the solve - # populates. Columns are aliased so the printed headers read as factory / product / - # quantity; this one frame feeds the plan table, the plan assertion, and the capacity - # summary below. - plan_df = ( - model.select( - Product.factory.name.alias("factory"), - Product.name.alias("product"), - Product.factory.avail.alias("avail"), - Product.rate.alias("rate"), - Product.x_quantity.alias("quantity"), - ) - .to_df() - .sort_values(["factory", "product"], ignore_index=True) - ) - print("\nProduction plan:") - produced = plan_df.loc[plan_df["quantity"] > 1e-6, ["factory", "product", "quantity"]] - print(produced.to_string(index=False)) - - # Pin the whole plan, not just the objective scalar: in general a matching objective is - # necessary but not sufficient, because a different product mix could reach the same - # profit. This LP's optimum happens to be unique (each factory's hours pin its plan), so - # pinning the quantities both guards against a regression and documents that uniqueness. - # Product names are unique across factories (asserted above), so the per-name lookup is a - # single row; tolerance is tight because the vertices here are exactly integral. - EXPECTED_PLAN = {"bands": 6000.0, "coils": 1400.0, "stouts": 1000.0, "ales": 2000.0} - for product_name, expected_qty in EXPECTED_PLAN.items(): - actual_qty = plan_df.loc[plan_df["product"] == product_name, "quantity"].sum() - assert abs(actual_qty - expected_qty) < 0.01, f"{product_name}: expected {expected_qty}, got {actual_qty}" - - # --- Capacity shadow prices: which factory to expand first ---------------------- - # A constraint carries its declared entity key as a back-pointer (cap.factory), so a - # shadow price joins to that factory's own data by KEY -- no name parsing, no pandas join. - cap_df = ( - model.select( - cap.factory.name.alias("factory"), - cap.factory.avail.alias("avail"), - cap.shadow_price.alias("shadow_price"), - ) - .to_df() - .sort_values("factory", ignore_index=True) - ) - assert sorted(cap_df["factory"]) == sorted(EXPECTED_FACTORIES), "expected one shadow price per factory" - print("\nFactory capacity shadow prices (d profit / d hour):") - print(cap_df.to_string(index=False)) - # Maximize + a <= capacity => shadow_price >= 0: an extra hour can only help. The - # implication runs one way: a factory with idle hours prices at 0 (slack -- its - # bottleneck is demand, not capacity), and a positive price marks a binding capacity. - # steel_factory is full (binding) and carries the marginal value of an hour; every other - # factory has spare hours (slack) and prices at 0. - steel = cap_df["factory"] == "steel_factory" - assert (cap_df.loc[steel, "shadow_price"] > 1e-6).all() - assert (cap_df.loc[~steel, "shadow_price"].abs() < 1e-6).all() - - # --- Reduced costs: which products are demand-capped? --------------------------- - # Each marginal reads straight off the variable; the variable's back-pointer - # (quantity_var.product) joins it to the product's factory by ENTITY KEY. - rc_df = ( - model.select( - quantity_var.product.factory.name.alias("factory"), - quantity_var.product.name.alias("product"), - quantity_var.reduced_cost.alias("reduced_cost"), - quantity_var.basis_status.alias("basis_status"), - ) - .to_df() - .sort_values(["factory", "product"], ignore_index=True) - ) - assert sorted(rc_df["product"]) == sorted(EXPECTED_PRODUCTS), "expected one reduced cost per product" - print("\nProduct reduced costs and basis status:") - print(rc_df.to_string(index=False)) - # A product pinned at its demand cap is NONBASIC_AT_UPPER with reduced_cost >= 0 (the - # extra profit from one more unit of allowed demand); the swing product that sets a - # binding factory's marginal price is BASIC at ~0. Here bands, stouts and ales sit at - # their demand caps; coils is the swing product in steel_factory. Assert BOTH halves of - # the lesson -- the reduced cost AND the basis status -- so a regression in either is loud. - capped = rc_df["product"].isin(["bands", "stouts", "ales"]) - assert (rc_df.loc[capped, "reduced_cost"] > 1e-6).all() - assert (rc_df.loc[capped, "basis_status"] == "NONBASIC_AT_UPPER").all() - # coils is basic (interior, between 0 and its demand cap): reduced cost ~0, status BASIC. - # The zero-check uses a HiGHS float tolerance (1e-4); it is exact-0 here. - coils = rc_df["product"] == "coils" - assert (rc_df.loc[coils, "reduced_cost"].abs() < 1e-4).all() - assert (rc_df.loc[coils, "basis_status"] == "BASIC").all() - - # --- Acting on it: which factory's capacity is the bottleneck? ------------------ - # With a maximize objective every capacity shadow price is >= 0, so the largest one is - # the factory whose marginal hour earns the most -- the capacity to expand first. - bottleneck = max(zip(cap_df["factory"], cap_df["shadow_price"]), key=lambda fp: fp[1]) - print(f"\nMost profit-sensitive capacity: {bottleneck[0]} (d profit / d hour = {bottleneck[1]:+.2f})") - - # -------------------------------------------------- - # Summary -- capacity utilization per factory - # -------------------------------------------------- - # Ties the shadow prices above back to the plan: in this data the binding factory has - # zero idle hours and a positive shadow price, while idle hours mean slack and a zero - # price. Only idle => zero-price holds in general; a binding capacity can still price at - # zero under degeneracy, so read the prices, not just the idle column. - plan_df["hours"] = plan_df["quantity"] / plan_df["rate"] - util = plan_df.groupby("factory").agg(avail=("avail", "first"), hours_used=("hours", "sum")).reset_index() - util["idle"] = util["avail"] - util["hours_used"] - print("\n" + "=" * 50) - print("Factory Capacity Summary") - print("=" * 50) - print(util.to_string(index=False)) - - -# -------------------------------------------------- -# Customize -# -------------------------------------------------- -# Try editing data/ and re-running: -# - Raise amazing_brewery's demand caps (products.csv) until its 30 hrs bind: its -# capacity shadow price jumps from 0 to positive once capacity becomes the bottleneck. -# - Lower steel_factory's avail (factories.csv): coils (the swing product) shrinks but -# stays the swing down to just above 30 hrs, so the shadow price holds at 4200. At -# exactly 30 hrs coils hits zero -- a degenerate breakpoint where the marginal is -# one-sided -- and below 30 hrs bands can no longer fill its 6000 demand, so bands -# becomes the swing and the price rises to 5000 (its own profit-per-hour, 25 x 200). diff --git a/factory_production/model/__init__.py b/factory_production/model/__init__.py new file mode 100644 index 00000000..29cb22cf --- /dev/null +++ b/factory_production/model/__init__.py @@ -0,0 +1,179 @@ +"""Factory production (prescriptive optimization) template: the model. + +This package declares a product-mix linear program in RelationalAI that +maximizes total profit subject to per-factory resource capacity, then reads back +the *marginals* a planner asks next: + +- Load sample CSVs describing factories (resource availability) and the products + each factory makes (production rate, unit profit, demand cap). +- Model them as *concepts* with typed properties. +- Choose a non-negative production quantity per product, capped by its demand. +- Limit each factory's total resource usage to its available hours. +- Maximize total profit. + +A plain solve answers "what is the most profitable production plan?". Requesting +``sensitivity=True`` on the solve ALSO answers the marginal questions -- in one solve, +read straight off the variable / constraint objects (the same attribute style as +``.name`` or ``.lower``): + +- **Shadow price** of a factory's capacity (``cap.shadow_price``): how much total + profit moves per extra hour of that factory. A capacity with idle hours prices at + zero (it is not the bottleneck); a positive price flags a binding capacity worth + expanding -- so this ranks which factory to expand first. (The implication runs one + way: slack => zero price, and a positive price => binding; a binding capacity *can* + still price at zero under degeneracy, though none does in this data.) +- **Reduced cost** of a product (``quantity_var.reduced_cost``) and its **basis + status** (``quantity_var.basis_status``): a product held at its demand cap shows a + positive reduced cost here (the extra profit per unit of demand allowed); the swing + product that sets the binding factory's marginal price is BASIC at ~0. + +Sign convention (the mirror image of a minimize-cost model): with a MAXIMIZE +objective, a binding ``<=`` capacity prices >= 0 and a product at its demand +*upper* bound has reduced cost >= 0. (Contrast the ``supplier_reliability`` +template, which minimizes cost: there a binding ``<=`` capacity prices <= 0.) Note +the two sensitivity objects come from two different modeling choices: capacity is a +*constraint* (its marginal is a shadow price), while the demand cap is a variable +*upper bound* (its marginal is a reduced cost). + +A constraint declared with ``keyed_by={"factory": Factory}`` carries an entity +back-pointer to what it grounds over (``cap.factory``), so a marginal joins to that +entity's own data by ENTITY KEY (``cap.factory.avail``) -- never by parsing the +constraint name string. (A variable's back-pointer, such as ``quantity_var.product``, +is derived automatically from its field names; a constraint's is declared.) + +Run: + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. Then + `python -m queries.plan_and_sensitivity` prints the production plan, each factory's + capacity shadow price, each product's reduced cost and basis status, and a + per-factory capacity-utilization summary from the deployed model. +""" + +from pathlib import Path + +from pandas import read_csv +from relationalai.semantics import Float, Integer, Model, String, sum +from relationalai.semantics.reasoners.prescriptive import Problem + +# -------------------------------------------------- +# Configure inputs +# -------------------------------------------------- + +DATA_DIR = Path(__file__).resolve().parents[1] / "data" + +# -------------------------------------------------- +# Define semantic model & load data +# -------------------------------------------------- + +model = Model("factory_production") +Concept, Property = model.Concept, model.Property + +# Factory concept: factories with total resource availability (hours). +Factory = Concept("Factory", identify_by={"name": String}) +Factory.avail = Property(f"{Factory} has {Float:avail}") + +# Load factories from CSV. +factory_csv = read_csv(DATA_DIR / "factories.csv") +model.define(Factory.new(model.data(factory_csv).to_schema())) + +# Product concept: products with production rate, unit profit, demand cap, and parent factory. +Product = Concept("Product", identify_by={"name": String, "factory_name": String}) +Product.factory = Property(f"{Product} is produced by {Factory}") +Product.rate = Property(f"{Product} has {Float:rate}") +Product.profit = Property(f"{Product} has {Float:profit}") +Product.demand = Property(f"{Product} has {Integer:demand}") + +# Load products from CSV and link each product to its factory. +product_csv = read_csv(DATA_DIR / "products.csv") +product_data = model.data(product_csv) +model.define( + p := Product.new(name=product_data.name, factory_name=product_data.factory_name), + p.rate(product_data.rate), + p.profit(product_data.profit), + p.demand(product_data.demand), +) +model.define(Product.factory(Factory)).where( + Product.factory_name(product_data.factory_name), + Factory.name(product_data.factory_name), +) + +# Guard the reference data. Every sensitivity check in the report is gated by a hard-coded +# factory/product name, so if data/ is edited those checks could silently pass on zero +# rows. Fail loudly here instead, with a clear message, the moment the data drifts. +EXPECTED_FACTORIES = {"steel_factory", "amazing_brewery"} +EXPECTED_PRODUCTS = {"bands", "coils", "stouts", "ales"} +if set(factory_csv["name"]) != EXPECTED_FACTORIES: + raise ValueError(f"factories.csv changed: {set(factory_csv['name'])} != {EXPECTED_FACTORIES}") +if set(product_csv["name"]) != EXPECTED_PRODUCTS: + raise ValueError(f"products.csv changed: {set(product_csv['name'])} != {EXPECTED_PRODUCTS}") +# Product names are unique across factories, so the report keys the plan by name alone. +if not product_csv["name"].is_unique: + raise ValueError(f"product names are not unique across factories: {list(product_csv['name'])}") + +# -------------------------------------------------- +# Model the decision problem +# -------------------------------------------------- + +# Variable: quantity[product] = amount produced (0 .. demand cap). +Product.x_quantity = Property(f"{Product} has {Float:quantity}") + +# -------------------------------------------------- +# Baseline solve with sensitivity analysis +# -------------------------------------------------- +# One product-mix LP over ALL factories. They share no resources, so the joint +# optimum is just each factory solved independently -- but a single solve lets us +# read every factory's marginals side by side off the captured handles. + +problem = Problem(model, Float) + +# Variable: production quantity per product. The demand cap is the variable's UPPER +# BOUND (not a separate constraint), so its marginal surfaces as the variable's +# reduced cost in the report. +quantity_var = problem.solve_for( + Product.x_quantity, + name=["qty", Product.factory.name, Product.name], + lower=0, + upper=Product.demand, +) + +# Constraint: each factory's total resource usage <= its available hours, over the +# factories that make products. Captured as a handle; ``keyed_by`` declares each +# instance's ENTITY KEY, so its shadow price reads back through that key (cap.factory), +# never by parsing the constraint name string; the per-factory name is a readable label. +cap = problem.satisfy( + model.where(Product.factory == Factory).require( + sum(Product.x_quantity / Product.rate) + .where(Product.factory == Factory) + .per(Factory) + <= Factory.avail + ), + name=["cap", Factory.name], + keyed_by={"factory": Factory}, +) + +# Objective: maximize total profit across all factories. +problem.maximize(sum(Product.profit * Product.x_quantity)) + +# Each refresh of the deployed model runs this solve, with sensitivity analysis. +problem.configure("highs", time_limit_sec=60, sensitivity=True) + +# Deploy the rules behind problem.display() with the model. +problem.install_display_strings() + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"factory production solve status is {String:status}") +TotalProfit = model.Relationship(f"factory production total profit is {Float:profit}") +model.define(SolveStatus(problem.termination_status()), TotalProfit(problem.objective_value())) + + +# -------------------------------------------------- +# Customize +# -------------------------------------------------- +# Try editing data/, redeploying, and re-running the report: +# - Raise amazing_brewery's demand caps (products.csv) until its 30 hrs bind: its +# capacity shadow price jumps from 0 to positive once capacity becomes the bottleneck. +# - Lower steel_factory's avail (factories.csv): coils (the swing product) shrinks but +# stays the swing down to just above 30 hrs, so the shadow price holds at 4200. At +# exactly 30 hrs coils hits zero -- a degenerate breakpoint where the marginal is +# one-sided -- and below 30 hrs bands can no longer fill its 6000 demand, so bands +# becomes the swing and the price rises to 5000 (its own profit-per-hour, 25 x 200). diff --git a/factory_production/queries/plan_and_sensitivity.py b/factory_production/queries/plan_and_sensitivity.py new file mode 100644 index 00000000..b0e2b7e8 --- /dev/null +++ b/factory_production/queries/plan_and_sensitivity.py @@ -0,0 +1,158 @@ +"""Production plan and sensitivity report: reads the deployed factory_production model. + +Run `python -m queries.plan_and_sensitivity` after `rai models deploy --path model`. Prints +the formulation, the status and total profit of the latest refresh's solve, the production +plan, each factory's capacity shadow price, each product's reduced cost and basis status, +and a per-factory capacity-utilization summary. Each section is checked against the bundled +data's known optimum; a failed check stops the report with an error. +""" + +from model import ( + EXPECTED_FACTORIES, + EXPECTED_PRODUCTS, + Product, + SolveStatus, + TotalProfit, + cap, + model, + problem, + quantity_var, +) + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + profit = model.select(TotalProfit).to_df().iloc[:, 0].tolist() + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal production plan: termination status {status}") + # Optimum: steel_factory fills all 40 hrs (bands to its 6000 cap = 30 hrs, coils takes + # the last 10 hrs = 1400 units); amazing_brewery makes both products to their demand + # caps in 25 of its 30 hrs. Profit = 192000 + 8000 = 200000. + if len(profit) != 1 or abs(profit[0] - 200000) >= 0.01: + raise SystemExit(f"total profit changed: expected 200000, got {profit}") + + print(f"\nBaseline status: {status[0]}, total profit: {profit[0]:.2f}") + + # --- The solved production plan ------------------------------------------------- + # Read populated quantities once for the plan checks and capacity summary. + plan_df = ( + model.select( + Product.factory.name.alias("factory"), + Product.name.alias("product"), + Product.factory.avail.alias("avail"), + Product.rate.alias("rate"), + Product.x_quantity.alias("quantity"), + ) + .to_df() + .sort_values(["factory", "product"], ignore_index=True) + ) + print("\nProduction plan:") + produced = plan_df.loc[plan_df["quantity"] > 1e-6, ["factory", "product", "quantity"]] + print(produced.to_string(index=False)) + + # Pin the whole plan, not just the objective scalar: in general a matching objective is + # necessary but not sufficient, because a different product mix could reach the same + # profit. This LP's optimum happens to be unique (each factory's hours pin its plan), so + # pinning the quantities both guards against a regression and documents that uniqueness. + # Check the plan's key set first (one row per product), then each quantity; tolerance is + # tight because the vertices here are exactly integral. + EXPECTED_PLAN = {"bands": 6000.0, "coils": 1400.0, "stouts": 1000.0, "ales": 2000.0} + if sorted(plan_df["product"]) != sorted(EXPECTED_PLAN): + raise SystemExit(f"expected one plan row per product, got {sorted(plan_df['product'])}") + for product_name, expected_qty in EXPECTED_PLAN.items(): + actual_qty = plan_df.loc[plan_df["product"] == product_name, "quantity"].iloc[0] + if abs(actual_qty - expected_qty) >= 0.01: + raise SystemExit(f"{product_name}: expected {expected_qty}, got {actual_qty}") + + # --- Capacity shadow prices: which factory to expand first ---------------------- + # A constraint carries its declared entity key as a back-pointer (cap.factory), so a + # shadow price joins to that factory's own data by KEY -- no name parsing, no pandas join. + cap_df = ( + model.select( + cap.factory.name.alias("factory"), + cap.factory.avail.alias("avail"), + cap.shadow_price.alias("shadow_price"), + ) + .to_df() + .sort_values("factory", ignore_index=True) + ) + if sorted(cap_df["factory"]) != sorted(EXPECTED_FACTORIES): + raise SystemExit(f"expected one shadow price per factory, got {sorted(cap_df['factory'])}") + print("\nFactory capacity shadow prices (d profit / d hour):") + print(cap_df.to_string(index=False)) + # Maximize + a <= capacity => shadow_price >= 0: an extra hour can only help. The + # implication runs one way: a factory with idle hours prices at 0 (slack -- its + # bottleneck is demand, not capacity), and a positive price marks a binding capacity. + # steel_factory is full (binding) and carries the marginal value of an hour; every other + # factory has spare hours (slack) and prices at 0. + steel = cap_df["factory"] == "steel_factory" + if not (cap_df.loc[steel, "shadow_price"] > 1e-6).all(): + raise SystemExit("steel_factory capacity should be binding with a positive shadow price") + if not (cap_df.loc[~steel, "shadow_price"].abs() < 1e-6).all(): + raise SystemExit("factories with idle hours should price their capacity at 0") + + # --- Reduced costs: which products are demand-capped? --------------------------- + # Each marginal reads straight off the variable; the variable's back-pointer + # (quantity_var.product) joins it to the product's factory by ENTITY KEY. + rc_df = ( + model.select( + quantity_var.product.factory.name.alias("factory"), + quantity_var.product.name.alias("product"), + quantity_var.reduced_cost.alias("reduced_cost"), + quantity_var.basis_status.alias("basis_status"), + ) + .to_df() + .sort_values(["factory", "product"], ignore_index=True) + ) + if sorted(rc_df["product"]) != sorted(EXPECTED_PRODUCTS): + raise SystemExit(f"expected one reduced cost per product, got {sorted(rc_df['product'])}") + print("\nProduct reduced costs and basis status:") + print(rc_df.to_string(index=False)) + # A product pinned at its demand cap is NONBASIC_AT_UPPER with reduced_cost >= 0 (the + # extra profit from one more unit of allowed demand); the swing product that sets a + # binding factory's marginal price is BASIC at ~0. Here bands, stouts and ales sit at + # their demand caps; coils is the swing product in steel_factory. Check BOTH halves of + # the lesson -- the reduced cost AND the basis status -- so a regression in either is loud. + capped = rc_df["product"].isin(["bands", "stouts", "ales"]) + if not (rc_df.loc[capped, "reduced_cost"] > 1e-6).all(): + raise SystemExit("demand-capped products should have a positive reduced cost") + if not (rc_df.loc[capped, "basis_status"] == "NONBASIC_AT_UPPER").all(): + raise SystemExit("demand-capped products should be NONBASIC_AT_UPPER") + # coils is basic (interior, between 0 and its demand cap): reduced cost ~0, status BASIC. + # The zero-check uses a HiGHS float tolerance (1e-4); it is exact-0 here. + coils = rc_df["product"] == "coils" + if not (rc_df.loc[coils, "reduced_cost"].abs() < 1e-4).all(): + raise SystemExit("coils, the swing product, should have a reduced cost of ~0") + if not (rc_df.loc[coils, "basis_status"] == "BASIC").all(): + raise SystemExit("coils, the swing product, should be BASIC") + + # --- Acting on it: which factory's capacity is the bottleneck? ------------------ + # With a maximize objective every capacity shadow price is >= 0, so the largest one is + # the factory whose marginal hour earns the most -- the capacity to expand first. + bottleneck = max(zip(cap_df["factory"], cap_df["shadow_price"]), key=lambda fp: fp[1]) + print( + f"\nMost profit-sensitive capacity: {bottleneck[0]} " + f"(d profit / d hour = {bottleneck[1]:+.2f})" + ) + + # -------------------------------------------------- + # Summary -- capacity utilization per factory + # -------------------------------------------------- + # Ties the shadow prices above back to the plan: in this data the binding factory has + # zero idle hours and a positive shadow price, while idle hours mean slack and a zero + # price. Only idle => zero-price holds in general; a binding capacity can still price at + # zero under degeneracy, so read the prices, not just the idle column. + plan_df["hours"] = plan_df["quantity"] / plan_df["rate"] + util = ( + plan_df.groupby("factory") + .agg(avail=("avail", "first"), hours_used=("hours", "sum")) + .reset_index() + ) + util["idle"] = util["avail"] - util["hours_used"] + print("\n" + "=" * 50) + print("Factory Capacity Summary") + print("=" * 50) + print(util.to_string(index=False)) diff --git a/factory_production/runbook.md b/factory_production/runbook.md index ca0e9a06..728b0487 100644 --- a/factory_production/runbook.md +++ b/factory_production/runbook.md @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 2 factories with resource-hours, 4 products with rate / profit / demand. Full model in `factory_production.py`. +Bundled CSVs in `data/`: 2 factories with resource-hours, 4 products with rate / profit / demand. Full model in `model/__init__.py`; the report in `queries/plan_and_sensitivity.py`. From 6f7dc41d03f31a87ceced8588c0129ceae9bb488 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:11:10 -0800 Subject: [PATCH 044/162] entity_resolution: deploy the model package and query it Move the graph, rules-based and prescriptive declarations into model/reasoning.py, re-exported from model/ (deployed with `rai models deploy --path model`), and the report into queries/resolution_report.py (`python -m queries.resolution_report`). The report stops unless the reinsurance solve is OPTIMAL. Rewrite the README Quickstart as the seven deployment steps, move the setup-page link into a Prerequisites section, and point the walkthrough at model/reasoning.py with page.deployment set. --- entity_resolution/README.md | 97 +++-- entity_resolution/entity_resolution.py | 342 ------------------ entity_resolution/model/__init__.py | 24 +- entity_resolution/model/reasoning.py | 129 +++++++ entity_resolution/model/source.py | 2 +- .../queries/resolution_report.py | 198 ++++++++++ entity_resolution/runbook.md | 12 +- entity_resolution/template-docs.yaml | 7 +- 8 files changed, 419 insertions(+), 392 deletions(-) delete mode 100644 entity_resolution/entity_resolution.py create mode 100644 entity_resolution/model/reasoning.py create mode 100644 entity_resolution/queries/resolution_report.py diff --git a/entity_resolution/README.md b/entity_resolution/README.md index edb31b2a..9a8d2492 100644 --- a/entity_resolution/README.md +++ b/entity_resolution/README.md @@ -30,23 +30,35 @@ a configurable limit, and selects reinsurance cessions within a premium budget. Adapt the matching fields, thresholds, exposure rules, and decision constraints for your workflow. +## Prerequisites + +- A Snowflake account that has the RAI Native App installed. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. + ## Quickstart -Before you start, install Python 3.10 or later and get access to a Snowflake -account with the RAI Native App. An account administrator must first do the -one-time model deployment setup on that account: follow -[Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) -through **Governed setup**, **PAT authentication**, and steps 1-2 of -**Additional setup for the prescriptive reasoner** (Quickstart step 4 below -covers this template's own schema). It gives you a model role, a programmatic -access token (PAT) stored as a secret, an external access integration, a PAT -user, and a warehouse; your Snowflake user must hold the model role. Graph and -Prescriptive reasoning are in Public Preview; ask your RelationalAI support representative to enable Prescriptive -reasoning. Preview features are for evaluation and testing, not production -applications. The template pins `relationalai==2.1.1` in `pyproject.toml` and -uses HiGHS without a separate solver license. - -Use this sequence to run the bundled example: +Before you start, install Python 3.10 or later and complete the model deployment setup listed under Prerequisites. Graph and Prescriptive reasoning are in Public Preview; ask your RelationalAI support representative to enable Prescriptive reasoning. Preview features are for evaluation and testing, not production applications. The template pins `relationalai==2.1.1` in `pyproject.toml` and uses HiGHS without a separate solver license. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .ENTITY_RESOLUTION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: 1. **Download the template** @@ -77,7 +89,9 @@ Use this sequence to run the bundled example: 4. **Configure your project** - Use the configuration builder in [Start building with PyRel](https://docs.relational.ai/get-started/start-building-with-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. Set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: ```yaml reasoners: @@ -98,38 +112,24 @@ Use this sequence to run the bundled example: pat_warehouse: ``` - Without the `deployment` section, `python entity_resolution.py` stops with an error naming the missing setting. +5. **Deploy the model** - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .ENTITY_RESOLUTION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .ENTITY_RESOLUTION_META TO APPLICATION ; + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.ENTITY_RESOLUTION` if it doesn't exist. + The deploy creates the public schema `.ENTITY_RESOLUTION` if it doesn't exist and waits for the model's first refresh, which resolves the records and runs the reinsurance solve. Each refresh of the deployed model runs them again. -5. **Run the template** +6. **Query the results** - Run the bundled script from the template directory: + Print the resolution report from the deployed model: ```bash - python entity_resolution.py + python -m queries.resolution_report ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. - ```text Auto-resolved 51 records into 31 insured parties. Households over the limit after RESOLUTION: 4 @@ -138,4 +138,23 @@ Use this sequence to run the bundled example: No single policy breaches the $1M limit, but resolution surfaces four over-limit households and the optimizer cedes the most excess it can afford - within budget. See `runbook.md` for the full workflow. + within budget. The lines above are excerpts of the full report, which checks the + reinsurance solve's status and stops unless it is `OPTIMAL`. See `runbook.md` + for the full workflow. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .ENTITY_RESOLUTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. diff --git a/entity_resolution/entity_resolution.py b/entity_resolution/entity_resolution.py deleted file mode 100644 index 59e78e7e..00000000 --- a/entity_resolution/entity_resolution.py +++ /dev/null @@ -1,342 +0,0 @@ -"""Entity Resolution (graph + rules-based + prescriptive reasoning) template. - -Resolution is the step that makes the downstream reasoning correct. An insurer's -policyholder records are scattered across policy systems (auto, home, life) and -an acquired book, so the same insured looks like several customers -- and their -combined exposure to the carrier is invisible until the records are resolved. -This template resolves them, then aggregates and acts on the resolved exposure: - -- Pre-process (pandas): normalize fields, block, and score candidate pairs. - Scores split into two bands -- AUTO_MERGE (merge automatically) and REVIEW - (held for a steward) -- so high-precision automation and human review coexist. -- Stage 1 -- Graph: weakly-connected-components over the auto-merged match edges - clusters records into one insured party, closing transitive chains. -- Stage 2 -- Rules-based: confidence tiers on matches, a duplicate flag, and the - payoff -- total exposure per resolved party and an accumulation-limit breach - flag. At the record level no policy looks dangerous; resolved, households breach. -- Stage 3 -- Prescriptive: with a finite reinsurance budget, choose which breached - households to cede to reinsurance (a knapsack) to transfer the most excess - exposure off the book. -- Stage 4: the record-level vs resolved-level contrast, the review queue (and the - breach it still hides), the cession plan, and pairwise precision / recall / F1. - -Run: - `python entity_resolution.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from - the deployed model. - -Output: - Prints blocking/banding stats, graph size, resolved-party and golden-record - summary, the accumulation contrast, the reinsurance cession plan, the review - queue, and pairwise precision / recall / F1 against the ground-truth labels. -""" - -from itertools import combinations - -from model import ( - AUTO_MERGE, - REVIEW_FLOOR, - CandidateMatch, - Record, - ResolvedParty, - candidate_count, - load_ground_truth, - matches_df, - model, - record_ids, - review_df, -) -from relationalai.semantics import Float, String -from relationalai.semantics.reasoners.graph import Graph -from relationalai.semantics.reasoners.prescriptive import Problem -from relationalai.semantics.std import aggregates - -HIGH_TIER = 0.90 - -ACCUMULATION_LIMIT = 1_000_000.0 -REINSURANCE_RATE = 0.12 -REINSURANCE_BUDGET = 120_000.0 - - -def cluster_accepted_matches(): - """Cluster records connected by accepted match edges.""" - graph = Graph(model, directed=False, weighted=False, node_concept=Record) - - match_ref = CandidateMatch.ref() - rec_a, rec_b = Record.ref(), Record.ref() - model.where(match_ref.rec_a == rec_a, match_ref.rec_b == rec_b).define( - graph.Edge.new(src=rec_a, dst=rec_b) - ) - - graph.Node.entity_id = graph.weakly_connected_component() - - rep_node = Record.ref() - model.where(Record.entity_id == rep_node).define( - Record.entity_key(rep_node.record_id) - ) - - return graph.num_nodes(), graph.num_edges() - - -def derive_exposure_and_breaches() -> None: - """Derive match tiers, resolved membership, exposure, and breaches.""" - model.define(CandidateMatch.confidence_tier("HIGH")).where( - CandidateMatch.score >= HIGH_TIER - ) - model.define(CandidateMatch.confidence_tier("MEDIUM")).where( - CandidateMatch.score < HIGH_TIER - ) - - records_per_entity = aggregates.count(Record).per(Record.entity_key) - model.define(Record.is_duplicate(Record)).where(records_per_entity >= 2) - - model.define(ResolvedParty.new(key=Record.entity_key)) - record, party = Record.ref(), ResolvedParty.ref() - model.where(record.entity_key == party.key).define( - record.resolved_party(party) - ) - - party_record = Record.ref() - model.where(party_record.entity_key == ResolvedParty.key).define( - ResolvedParty.total_exposure( - aggregates.sum(party_record.coverage_amount).per(ResolvedParty) - ) - ) - model.define(ResolvedParty.is_over_limit(ResolvedParty)).where( - ResolvedParty.total_exposure > ACCUMULATION_LIMIT - ) - model.where(ResolvedParty.total_exposure > ACCUMULATION_LIMIT).define( - ResolvedParty.excess( - ResolvedParty.total_exposure - ACCUMULATION_LIMIT - ) - ) - model.define( - ResolvedParty.premium(ResolvedParty.excess * REINSURANCE_RATE) - ) - - -def choose_reinsurance_cessions(): - """Declare the reinsurance-budget knapsack for breached parties.""" - problem = Problem(model, Float) - problem.solve_for( - ResolvedParty.cede, - where=[ResolvedParty.is_over_limit()], - name=["cede", ResolvedParty.key], - type="bin", - lower=0.0, - upper=1.0, - ) - problem.satisfy( - model.require( - aggregates.sum( - ResolvedParty.premium * ResolvedParty.cede - ) - <= REINSURANCE_BUDGET - ), - name=["reinsurance_budget"], - ) - problem.maximize( - aggregates.sum(ResolvedParty.excess * ResolvedParty.cede) - ) - problem.configure("highs", time_limit_sec=60) - return problem - - -num_nodes, num_edges = cluster_accepted_matches() -derive_exposure_and_breaches() -problem = choose_reinsurance_cessions() - -# The solve's status, published through the model's own relationship. -SolveStatus = model.Relationship(f"reinsurance solve status is {String:status}") -model.define(SolveStatus(problem.termination_status())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - n_records = len(record_ids) - all_pairs = n_records * (n_records - 1) // 2 - print("=" * 64) - print("Blocking & scoring") - print("=" * 64) - print(f"Records: {n_records}") - print(f"All possible pairs: {all_pairs}") - print( - f"Candidate pairs: {candidate_count} " - f"({100 * (1 - candidate_count / all_pairs):.1f}% fewer comparisons)" - ) - print(f"Auto-merge matches: {len(matches_df)} (score >= {AUTO_MERGE})") - print( - f"Held for review: {len(review_df)} " - f"([{REVIEW_FLOOR}, {AUTO_MERGE}))" - ) - - n_nodes = int(model.select(num_nodes.alias("n")).to_df()["n"].iloc[0]) - n_edges = int(model.select(num_edges.alias("n")).to_df()["n"].iloc[0]) - print(f"Graph: {n_nodes} nodes, {n_edges} match edges") - - resolved = model.select( - Record.record_id.alias("record_id"), - Record.entity_key.alias("entity_id"), - Record.source_system.alias("source_system"), - Record.full_name.alias("full_name"), - Record.created_at.alias("created_at"), - Record.coverage_amount.alias("coverage_amount"), - ).to_df() - resolved["entity_id"] = resolved["entity_id"].astype(str) - - n_entities = resolved["entity_id"].nunique() - print("\n" + "=" * 64) - print("Resolved parties") - print("=" * 64) - print(f"Auto-resolved {n_records} records into {n_entities} insured parties.") - - resolved = resolved.sort_values(["entity_id", "created_at"]) - golden = resolved.groupby("entity_id").tail(1).set_index("entity_id") - sizes = resolved.groupby("entity_id").size() - print("\nMulti-record parties (golden record in CAPS):") - for entity_id in sizes[sizes > 1].index: - golden_record = golden.loc[entity_id] - members = resolved[resolved["entity_id"] == entity_id] - exposure = members["coverage_amount"].sum() - print( - f" {golden_record['full_name'].upper():<22} " - f"{len(members)} policies total exposure ${exposure:>12,.0f}" - ) - - record_level_breaches = int( - (resolved["coverage_amount"] > ACCUMULATION_LIMIT).sum() - ) - over = ( - model.select( - ResolvedParty.key.alias("id"), - ResolvedParty.total_exposure.alias("exposure"), - ) - .where(ResolvedParty.is_over_limit()) - .to_df() - ) - print("\n" + "=" * 64) - print(f"Accumulation control (limit ${ACCUMULATION_LIMIT:,.0f})") - print("=" * 64) - print( - "Policies over the limit at the RECORD level: " - f"{record_level_breaches}" - ) - print(f"Households over the limit after RESOLUTION: {len(over)}") - - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - print( - f"\nReinsurance cession plan (status {', '.join(status)}, " - f"budget ${REINSURANCE_BUDGET:,.0f} at " - f"{REINSURANCE_RATE:.0%} rate on line):" - ) - ceded = ( - model.select( - ResolvedParty.key.alias("id"), - ResolvedParty.total_exposure.alias("exposure"), - ResolvedParty.excess.alias("excess"), - ResolvedParty.premium.alias("premium"), - ResolvedParty.cede.alias("cede"), - ) - .where(ResolvedParty.is_over_limit()) - .to_df() - ) - ceded["id"] = ceded["id"].astype(str) - name_by_entity = golden["full_name"].to_dict() - ceded["name"] = ceded["id"].map(name_by_entity).fillna("(party)") - ceded = ceded.sort_values("excess", ascending=False) - total_premium = total_ceded = 0.0 - for _, row in ceded.iterrows(): - chosen = row["cede"] > 0.5 - mark = "CEDE " if chosen else "keep " - if chosen: - total_premium += row["premium"] - total_ceded += row["excess"] - print( - f" {mark} {row['name']:<22} exposure " - f"${row['exposure']:>11,.0f} excess ${row['excess']:>9,.0f} " - f"premium ${row['premium']:>8,.0f}" - ) - print( - f" -> ceded ${total_ceded:,.0f} of excess exposure for " - f"${total_premium:,.0f} premium (of ${REINSURANCE_BUDGET:,.0f})" - ) - - auto_entity = dict(zip(resolved["record_id"], resolved["entity_id"])) - coverage_by_record = dict( - zip(resolved["record_id"], resolved["coverage_amount"]) - ) - confirmed = {record_id: auto_entity[record_id] for record_id in record_ids} - for _, row in review_df.iterrows(): - rec_a, rec_b = int(row["rec_a"]), int(row["rec_b"]) - target, source = confirmed[rec_a], confirmed[rec_b] - for record_id in record_ids: - if confirmed[record_id] == source: - confirmed[record_id] = target - confirmed_exposure: dict[str, float] = {} - for record_id in record_ids: - party_id = confirmed[record_id] - confirmed_exposure[party_id] = ( - confirmed_exposure.get(party_id, 0.0) - + coverage_by_record[record_id] - ) - confirmed_breaches = sum( - 1 for exposure in confirmed_exposure.values() - if exposure > ACCUMULATION_LIMIT - ) - print("\n" + "=" * 64) - print("Review queue") - print("=" * 64) - print(f"Pairs held for steward review: {len(review_df)}") - print( - "Breaches surfaced only if the review queue is confirmed: " - f"{confirmed_breaches - len(over)} " - f"(resolution would then show {confirmed_breaches} breached " - f"households, not {len(over)})" - ) - - truth_df = load_ground_truth() - truth_map = dict( - zip(truth_df["record_id"], truth_df["true_entity_id"]) - ) - true_pairs = { - (rec_a, rec_b) - for rec_a, rec_b in combinations(record_ids, 2) - if truth_map[rec_a] == truth_map[rec_b] - } - predicted_pairs = { - (rec_a, rec_b) - for rec_a, rec_b in combinations(record_ids, 2) - if auto_entity[rec_a] == auto_entity[rec_b] - } - true_positives = len(predicted_pairs & true_pairs) - false_positives = len(predicted_pairs - true_pairs) - false_negatives = len(true_pairs - predicted_pairs) - precision = ( - true_positives / (true_positives + false_positives) - if true_positives + false_positives - else 1.0 - ) - recall = ( - true_positives / (true_positives + false_negatives) - if true_positives + false_negatives - else 1.0 - ) - f1 = ( - 2 * precision * recall / (precision + recall) - if precision + recall - else 0.0 - ) - print("\n" + "=" * 64) - print("Evaluation vs ground truth (pairwise, auto-resolution)") - print("=" * 64) - print(f" precision: {precision:.3f} recall: {recall:.3f} f1: {f1:.3f}") - print( - f" (true positives={true_positives}, false positives={false_positives}, " - f"false negatives={false_negatives}; the {false_negatives} miss is the " - "held review pair)" - ) diff --git a/entity_resolution/model/__init__.py b/entity_resolution/model/__init__.py index 85a003d0..d47ae33b 100644 --- a/entity_resolution/model/__init__.py +++ b/entity_resolution/model/__init__.py @@ -1,4 +1,7 @@ -"""Entity-resolution model and loaded source facts.""" +"""Entity-resolution model: loaded source facts and the reasoning over them. + +`rai models deploy --path model` deploys everything this package declares. +""" from .schema import CandidateMatch, Record, ResolvedParty, ReviewPair, model from .source import ( @@ -12,17 +15,36 @@ review_df, ) +# isort: split +# The reasoning declares rules over the facts loaded above. +from .reasoning import ( + ACCUMULATION_LIMIT, + REINSURANCE_BUDGET, + REINSURANCE_RATE, + SolveStatus, + num_edges, + num_nodes, + problem, +) + __all__ = [ + "ACCUMULATION_LIMIT", "AUTO_MERGE", + "REINSURANCE_BUDGET", + "REINSURANCE_RATE", "REVIEW_FLOOR", "CandidateMatch", "Record", "ResolvedParty", "ReviewPair", + "SolveStatus", "candidate_count", "load_ground_truth", "matches_df", "model", + "num_edges", + "num_nodes", + "problem", "record_ids", "records_df", "review_df", diff --git a/entity_resolution/model/reasoning.py b/entity_resolution/model/reasoning.py new file mode 100644 index 00000000..b7ca1252 --- /dev/null +++ b/entity_resolution/model/reasoning.py @@ -0,0 +1,129 @@ +"""Graph, rules-based, and prescriptive reasoning for entity resolution. + +Resolution is the step that makes the downstream reasoning correct. An insurer's +policyholder records are scattered across policy systems (auto, home, life) and +an acquired book, so the same insured looks like several customers -- and their +combined exposure to the carrier is invisible until the records are resolved. +`model/source.py` normalizes, blocks, and scores candidate pairs into two bands -- +AUTO_MERGE (merge automatically) and REVIEW (held for a steward). This module +declares the reasoning over them: + +- Stage 1 -- Graph: weakly-connected-components over the auto-merged match edges + clusters records into one insured party, closing transitive chains. +- Stage 2 -- Rules-based: confidence tiers on matches, a duplicate flag, and the + payoff -- total exposure per resolved party and an accumulation-limit breach + flag. At the record level no policy looks dangerous; resolved, households breach. +- Stage 3 -- Prescriptive: with a finite reinsurance budget, choose which breached + households to cede to reinsurance (a knapsack) to transfer the most excess + exposure off the book. Each refresh of the deployed model runs the solve. + +`queries/resolution_report.py` prints Stage 4 from the deployed model: the +record-level vs resolved-level contrast, the review queue (and the breach it still +hides), the cession plan, and pairwise precision / recall / F1. +""" + +from relationalai.semantics import Float, String +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.reasoners.prescriptive import Problem +from relationalai.semantics.std import aggregates + +from .schema import CandidateMatch, Record, ResolvedParty, model + +HIGH_TIER = 0.90 + +ACCUMULATION_LIMIT = 1_000_000.0 +REINSURANCE_RATE = 0.12 +REINSURANCE_BUDGET = 120_000.0 + + +def cluster_accepted_matches(): + """Cluster records connected by accepted match edges.""" + graph = Graph(model, directed=False, weighted=False, node_concept=Record) + + match_ref = CandidateMatch.ref() + rec_a, rec_b = Record.ref(), Record.ref() + model.where(match_ref.rec_a == rec_a, match_ref.rec_b == rec_b).define( + graph.Edge.new(src=rec_a, dst=rec_b) + ) + + graph.Node.entity_id = graph.weakly_connected_component() + + rep_node = Record.ref() + model.where(Record.entity_id == rep_node).define( + Record.entity_key(rep_node.record_id) + ) + + return graph.num_nodes(), graph.num_edges() + + +def derive_exposure_and_breaches() -> None: + """Derive match tiers, resolved membership, exposure, and breaches.""" + model.define(CandidateMatch.confidence_tier("HIGH")).where( + CandidateMatch.score >= HIGH_TIER + ) + model.define(CandidateMatch.confidence_tier("MEDIUM")).where( + CandidateMatch.score < HIGH_TIER + ) + + records_per_entity = aggregates.count(Record).per(Record.entity_key) + model.define(Record.is_duplicate(Record)).where(records_per_entity >= 2) + + model.define(ResolvedParty.new(key=Record.entity_key)) + record, party = Record.ref(), ResolvedParty.ref() + model.where(record.entity_key == party.key).define( + record.resolved_party(party) + ) + + party_record = Record.ref() + model.where(party_record.entity_key == ResolvedParty.key).define( + ResolvedParty.total_exposure( + aggregates.sum(party_record.coverage_amount).per(ResolvedParty) + ) + ) + model.define(ResolvedParty.is_over_limit(ResolvedParty)).where( + ResolvedParty.total_exposure > ACCUMULATION_LIMIT + ) + model.where(ResolvedParty.total_exposure > ACCUMULATION_LIMIT).define( + ResolvedParty.excess( + ResolvedParty.total_exposure - ACCUMULATION_LIMIT + ) + ) + model.define( + ResolvedParty.premium(ResolvedParty.excess * REINSURANCE_RATE) + ) + + +def choose_reinsurance_cessions(): + """Declare the reinsurance-budget knapsack for breached parties.""" + problem = Problem(model, Float) + problem.solve_for( + ResolvedParty.cede, + where=[ResolvedParty.is_over_limit()], + name=["cede", ResolvedParty.key], + type="bin", + lower=0.0, + upper=1.0, + ) + problem.satisfy( + model.require( + aggregates.sum( + ResolvedParty.premium * ResolvedParty.cede + ) + <= REINSURANCE_BUDGET + ), + name=["reinsurance_budget"], + ) + problem.maximize( + aggregates.sum(ResolvedParty.excess * ResolvedParty.cede) + ) + problem.configure("highs", time_limit_sec=60) + return problem + + +num_nodes, num_edges = cluster_accepted_matches() +derive_exposure_and_breaches() +problem = choose_reinsurance_cessions() + +# The solve's status, published through the model's own relationship. +SolveStatus = model.Relationship(f"reinsurance solve status is {String:status}") +model.define(SolveStatus(problem.termination_status())) diff --git a/entity_resolution/model/source.py b/entity_resolution/model/source.py index 1d7c7445..ae61323a 100644 --- a/entity_resolution/model/source.py +++ b/entity_resolution/model/source.py @@ -265,7 +265,7 @@ def load_candidate_facts(matches: DataFrame, reviews: DataFrame) -> None: def load_ground_truth() -> DataFrame: - """Read the optional labels used by the runner's evaluation section.""" + """Read the optional labels used by the report's evaluation section.""" return read_csv(DATA_DIR / "ground_truth.csv") diff --git a/entity_resolution/queries/resolution_report.py b/entity_resolution/queries/resolution_report.py new file mode 100644 index 00000000..1350bc82 --- /dev/null +++ b/entity_resolution/queries/resolution_report.py @@ -0,0 +1,198 @@ +"""Resolution report: reads the deployed entity_resolution model. + +Run `python -m queries.resolution_report` after `rai models deploy --path model`. +Prints blocking/banding stats, graph size, resolved-party and golden-record +summary, the accumulation contrast, the reinsurance cession plan, the review +queue, and pairwise precision / recall / F1 against the ground-truth labels. +""" + +from itertools import combinations + +from model import ( + ACCUMULATION_LIMIT, + AUTO_MERGE, + REINSURANCE_BUDGET, + REINSURANCE_RATE, + REVIEW_FLOOR, + Record, + ResolvedParty, + SolveStatus, + candidate_count, + load_ground_truth, + matches_df, + model, + num_edges, + num_nodes, + record_ids, + review_df, +) + +if __name__ == "__main__": + n_records = len(record_ids) + all_pairs = n_records * (n_records - 1) // 2 + print("=" * 64) + print("Blocking & scoring") + print("=" * 64) + print(f"Records: {n_records}") + print(f"All possible pairs: {all_pairs}") + print( + f"Candidate pairs: {candidate_count} " + f"({100 * (1 - candidate_count / all_pairs):.1f}% fewer comparisons)" + ) + print(f"Auto-merge matches: {len(matches_df)} (score >= {AUTO_MERGE})") + print(f"Held for review: {len(review_df)} ([{REVIEW_FLOOR}, {AUTO_MERGE}))") + + n_nodes = int(model.select(num_nodes.alias("n")).to_df()["n"].iloc[0]) + n_edges = int(model.select(num_edges.alias("n")).to_df()["n"].iloc[0]) + print(f"Graph: {n_nodes} nodes, {n_edges} match edges") + + resolved = model.select( + Record.record_id.alias("record_id"), + Record.entity_key.alias("entity_id"), + Record.source_system.alias("source_system"), + Record.full_name.alias("full_name"), + Record.created_at.alias("created_at"), + Record.coverage_amount.alias("coverage_amount"), + ).to_df() + resolved["entity_id"] = resolved["entity_id"].astype(str) + + n_entities = resolved["entity_id"].nunique() + print("\n" + "=" * 64) + print("Resolved parties") + print("=" * 64) + print(f"Auto-resolved {n_records} records into {n_entities} insured parties.") + + resolved = resolved.sort_values(["entity_id", "created_at"]) + golden = resolved.groupby("entity_id").tail(1).set_index("entity_id") + sizes = resolved.groupby("entity_id").size() + print("\nMulti-record parties (golden record in CAPS):") + for entity_id in sizes[sizes > 1].index: + golden_record = golden.loc[entity_id] + members = resolved[resolved["entity_id"] == entity_id] + exposure = members["coverage_amount"].sum() + print( + f" {golden_record['full_name'].upper():<22} " + f"{len(members)} policies total exposure ${exposure:>12,.0f}" + ) + + record_level_breaches = int((resolved["coverage_amount"] > ACCUMULATION_LIMIT).sum()) + over = ( + model.select( + ResolvedParty.key.alias("id"), + ResolvedParty.total_exposure.alias("exposure"), + ) + .where(ResolvedParty.is_over_limit()) + .to_df() + ) + print("\n" + "=" * 64) + print(f"Accumulation control (limit ${ACCUMULATION_LIMIT:,.0f})") + print("=" * 64) + print(f"Policies over the limit at the RECORD level: {record_level_breaches}") + print(f"Households over the limit after RESOLUTION: {len(over)}") + + # Status of the latest refresh's solve. A refresh records any termination + # status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + print( + f"\nReinsurance cession plan (status {', '.join(status)}, " + f"budget ${REINSURANCE_BUDGET:,.0f} at " + f"{REINSURANCE_RATE:.0%} rate on line):" + ) + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal cession plan: termination status {status}") + ceded = ( + model.select( + ResolvedParty.key.alias("id"), + ResolvedParty.total_exposure.alias("exposure"), + ResolvedParty.excess.alias("excess"), + ResolvedParty.premium.alias("premium"), + ResolvedParty.cede.alias("cede"), + ) + .where(ResolvedParty.is_over_limit()) + .to_df() + ) + ceded["id"] = ceded["id"].astype(str) + name_by_entity = golden["full_name"].to_dict() + ceded["name"] = ceded["id"].map(name_by_entity).fillna("(party)") + ceded = ceded.sort_values("excess", ascending=False) + total_premium = total_ceded = 0.0 + for _, row in ceded.iterrows(): + chosen = row["cede"] > 0.5 + mark = "CEDE " if chosen else "keep " + if chosen: + total_premium += row["premium"] + total_ceded += row["excess"] + print( + f" {mark} {row['name']:<22} exposure " + f"${row['exposure']:>11,.0f} excess ${row['excess']:>9,.0f} " + f"premium ${row['premium']:>8,.0f}" + ) + print( + f" -> ceded ${total_ceded:,.0f} of excess exposure for " + f"${total_premium:,.0f} premium (of ${REINSURANCE_BUDGET:,.0f})" + ) + + auto_entity = dict(zip(resolved["record_id"], resolved["entity_id"])) + coverage_by_record = dict(zip(resolved["record_id"], resolved["coverage_amount"])) + confirmed = {record_id: auto_entity[record_id] for record_id in record_ids} + for _, row in review_df.iterrows(): + rec_a, rec_b = int(row["rec_a"]), int(row["rec_b"]) + target, source = confirmed[rec_a], confirmed[rec_b] + for record_id in record_ids: + if confirmed[record_id] == source: + confirmed[record_id] = target + confirmed_exposure: dict[str, float] = {} + for record_id in record_ids: + party_id = confirmed[record_id] + confirmed_exposure[party_id] = ( + confirmed_exposure.get(party_id, 0.0) + coverage_by_record[record_id] + ) + confirmed_breaches = sum( + 1 for exposure in confirmed_exposure.values() if exposure > ACCUMULATION_LIMIT + ) + print("\n" + "=" * 64) + print("Review queue") + print("=" * 64) + print(f"Pairs held for steward review: {len(review_df)}") + print( + "Breaches surfaced only if the review queue is confirmed: " + f"{confirmed_breaches - len(over)} " + f"(resolution would then show {confirmed_breaches} breached " + f"households, not {len(over)})" + ) + + truth_df = load_ground_truth() + truth_map = dict(zip(truth_df["record_id"], truth_df["true_entity_id"])) + true_pairs = { + (rec_a, rec_b) + for rec_a, rec_b in combinations(record_ids, 2) + if truth_map[rec_a] == truth_map[rec_b] + } + predicted_pairs = { + (rec_a, rec_b) + for rec_a, rec_b in combinations(record_ids, 2) + if auto_entity[rec_a] == auto_entity[rec_b] + } + true_positives = len(predicted_pairs & true_pairs) + false_positives = len(predicted_pairs - true_pairs) + false_negatives = len(true_pairs - predicted_pairs) + precision = ( + true_positives / (true_positives + false_positives) + if true_positives + false_positives + else 1.0 + ) + recall = ( + true_positives / (true_positives + false_negatives) + if true_positives + false_negatives + else 1.0 + ) + f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0 + print("\n" + "=" * 64) + print("Evaluation vs ground truth (pairwise, auto-resolution)") + print("=" * 64) + print(f" precision: {precision:.3f} recall: {recall:.3f} f1: {f1:.3f}") + print( + f" (true positives={true_positives}, false positives={false_positives}, " + f"false negatives={false_negatives}; the {false_negatives} miss is the " + "held review pair)" + ) diff --git a/entity_resolution/runbook.md b/entity_resolution/runbook.md index e5dbb31a..2f64128e 100644 --- a/entity_resolution/runbook.md +++ b/entity_resolution/runbook.md @@ -38,7 +38,7 @@ record level, and cedes $927k of excess exposure to reinsurance for $111k. > **How to use this walkthrough.** Each section below is a Prompt that an analyst pastes into a fresh agent session loaded with the named `/rai-*` skill. Prompts are designed to run **in order, in a single session** — every step relies on enrichments the previous steps wrote back to the shared ontology, so the agent inherits accumulated model state across prompts. -Keep the shared declarations in `model/schema.py`, source preparation and fact loading in `model/source.py`, and graph, rules-based, prescriptive, query, and reporting logic in `entity_resolution.py`. +Keep the shared declarations in `model/schema.py`, source preparation and fact loading in `model/source.py`, graph, rules-based, and prescriptive reasoning in `model/reasoning.py`, and the queries and reporting that read the deployed model in `queries/resolution_report.py`. ### 1. Build the ontology @@ -81,7 +81,7 @@ Blocking reduces 1,275 possible pairs to 28 candidates. 25 score >= 0.70 and loa **Prompt** ``` -/rai-graph-analysis In entity_resolution.py, determine which records refer to the same insured party. Treat each CandidateMatch as an undirected link and group records connected directly or through a chain of matches into one party (connected components). Weakly-connected-components returns the representative record, so also derive a stable integer party key from it, and report how many parties the 51 records auto-resolve into. +/rai-graph-analysis In model/reasoning.py, determine which records refer to the same insured party. Treat each CandidateMatch as an undirected link and group records connected directly or through a chain of matches into one party (connected components). Weakly-connected-components returns the representative record, so also derive a stable integer party key from it, and report how many parties the 51 records auto-resolve into. ``` **Response** @@ -105,7 +105,7 @@ Weakly-connected-components over 51 nodes and 25 auto-merge edges resolves the r **Prompt** ``` -/rai-pyrel Declare ResolvedParty and its stable properties and relationships in model/schema.py. In entity_resolution.py, create one ResolvedParty per party key, relate each Record to its party, total the coverage across its policies, and flag parties whose total exposure exceeds the $1,000,000 accumulation limit. How many policies breach the limit at the record level versus how many households breach it after resolution? +/rai-pyrel Declare ResolvedParty and its stable properties and relationships in model/schema.py. In model/reasoning.py, create one ResolvedParty per party key, relate each Record to its party, total the coverage across its policies, and flag parties whose total exposure exceeds the $1,000,000 accumulation limit. How many policies breach the limit at the record level versus how many households breach it after resolution? ``` **Response** @@ -117,7 +117,7 @@ A `ResolvedParty` per party key carries `total_exposure` (summed `coverage_amoun **Prompt** ``` -/rai-prescriptive-problem In entity_resolution.py, choose reinsurance cessions for the breached parties. We can spend a $120,000 reinsurance premium this period, at a 12% rate on line, to cede the excess exposure of breached households (premium = 12% of the amount over the limit). Which households should we cede to transfer the most excess exposure off the book? Each breached household is cede-or-not (binary), total premium must stay within budget, and we maximize the excess exposure ceded. +/rai-prescriptive-problem In model/reasoning.py, choose reinsurance cessions for the breached parties. We can spend a $120,000 reinsurance premium this period, at a 12% rate on line, to cede the excess exposure of breached households (premium = 12% of the amount over the limit). Which households should we cede to transfer the most excess exposure off the book? Each breached household is cede-or-not (binary), total premium must stay within budget, and we maximize the excess exposure ceded. ``` **Response** @@ -129,7 +129,7 @@ A binary `ResolvedParty.cede` over the 4 breached households; budget constraint **Prompt** ``` -/rai-pyrel Using the source frames exported by model/__init__.py and the queries in entity_resolution.py, report how many matches are held in the review queue and whether confirming them changes the accumulation picture. Also score the auto-resolution against the data/ground_truth.csv labels: pairwise precision, recall, and F1. +/rai-pyrel In queries/resolution_report.py, using the source frames exported by model/__init__.py and queries of the deployed model, report how many matches are held in the review queue and whether confirming them changes the accumulation picture. Also score the auto-resolution against the data/ground_truth.csv labels: pairwise precision, recall, and F1. ``` **Response** @@ -138,4 +138,4 @@ A binary `ResolvedParty.cede` over the 4 breached households; budget constraint ## Data -Bundled CSVs in `data/`: `records.csv` (51 policy records across AUTO / HOME / LIFE / LEGACY, covering 30 people, with per-policy `coverage_amount`) and `ground_truth.csv` (record_id → true party label). Three cases are placed deliberately: a transitive chain (`M. O'Brien`) that only clustering resolves, a same-name trap (two distinct Chicago "John Smith" policyholders) that scoring keeps apart, and a review-band duplicate (`Ethan Brooks` / `E. Brooks`) whose hidden combined exposure breaches the limit only once confirmed. The dataset is fixed and hand-curated so the walkthrough's numbers are stable. `model/source.py` reads and loads the data; `entity_resolution.py` runs the reasoning and reporting chain end to end. +Bundled CSVs in `data/`: `records.csv` (51 policy records across AUTO / HOME / LIFE / LEGACY, covering 30 people, with per-policy `coverage_amount`) and `ground_truth.csv` (record_id → true party label). Three cases are placed deliberately: a transitive chain (`M. O'Brien`) that only clustering resolves, a same-name trap (two distinct Chicago "John Smith" policyholders) that scoring keeps apart, and a review-band duplicate (`Ethan Brooks` / `E. Brooks`) whose hidden combined exposure breaches the limit only once confirmed. The dataset is fixed and hand-curated so the walkthrough's numbers are stable. `model/source.py` reads and loads the data; `model/reasoning.py` declares the reasoning chain, and `queries/resolution_report.py` reports from the deployed model. diff --git a/entity_resolution/template-docs.yaml b/entity_resolution/template-docs.yaml index 788e8d65..9717043b 100644 --- a/entity_resolution/template-docs.yaml +++ b/entity_resolution/template-docs.yaml @@ -6,6 +6,7 @@ page: - what-this-template-is-for omit_from_readme: [] download_and_run_from_readme: true + deployment: true model_explorer: title: Explore the model @@ -201,7 +202,7 @@ guided_walkthrough: representative ID, giving later calculations a key for the insured party. focus: - file: entity_resolution.py + file: model/reasoning.py anchors: - start: 'def cluster_accepted_matches():' end: Record.entity_key(rep_node.record_id) @@ -242,7 +243,7 @@ guided_walkthrough: ResolvedParty.excess(...) measures how much coverage exceeds it for the reinsurance decision. focus: - file: entity_resolution.py + file: model/reasoning.py anchors: - start: model.define(ResolvedParty.new(key=Record.entity_key)) end: ResolvedParty.total_exposure - ACCUMULATION_LIMIT @@ -290,7 +291,7 @@ guided_walkthrough: limit. Each refresh of the deployed model runs that solve, though a small budget may leave some excess with the insurer. focus: - file: entity_resolution.py + file: model/reasoning.py anchors: - start: 'def choose_reinsurance_cessions():' end: return problem From a0c1f68c437bfb7e13a6fe4a12bd17cac2c0a853 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:12:03 -0800 Subject: [PATCH 045/162] diet, factory_production, entity_resolution: name the public schema by its config key The generated docs page rejects angle-bracket placeholders in inline code in Quickstart prose, so the deploy step refers to the public schema named in raiconfig.yaml instead. --- diet/README.md | 2 +- entity_resolution/README.md | 2 +- factory_production/README.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/diet/README.md b/diet/README.md index 37795a38..f17809df 100644 --- a/diet/README.md +++ b/diet/README.md @@ -135,7 +135,7 @@ Use this sequence to deploy the model and query it: rai models deploy --path model ``` - The deploy creates the public schema `.DIET` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. diff --git a/entity_resolution/README.md b/entity_resolution/README.md index 9a8d2492..45e492c8 100644 --- a/entity_resolution/README.md +++ b/entity_resolution/README.md @@ -120,7 +120,7 @@ Use this sequence to deploy the model and query it: rai models deploy --path model ``` - The deploy creates the public schema `.ENTITY_RESOLUTION` if it doesn't exist and waits for the model's first refresh, which resolves the records and runs the reinsurance solve. Each refresh of the deployed model runs them again. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which resolves the records and runs the reinsurance solve. Each refresh of the deployed model runs them again. 6. **Query the results** diff --git a/factory_production/README.md b/factory_production/README.md index a6e22f3e..10ca9f3f 100644 --- a/factory_production/README.md +++ b/factory_production/README.md @@ -143,7 +143,7 @@ Use this sequence to deploy the model and query it: rai models deploy --path model ``` - The deploy creates the public schema `.FACTORY_PRODUCTION` if it doesn't exist and waits for the model's first refresh, which runs the solve with sensitivity analysis. Each refresh of the deployed model runs the solve again. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve with sensitivity analysis. Each refresh of the deployed model runs the solve again. Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. From 52a42d7a8703b5fccfd03068c8711b50c3d005bc Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:05:00 -0800 Subject: [PATCH 046/162] bom-reachability: deploy the model package and query the deployment Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/dependency_report.py, and rewrite the README Quickstart as deploy, query, and tear down. Assembly depth again covers every SKU that ends an assembly chain (feeds nothing further), as the original Python computation did, rather than only finished goods. On the bundled data both are the two finished goods. --- bom-reachability/README.md | 75 +++-- bom-reachability/bom_reachability.py | 296 ------------------ bom-reachability/model/__init__.py | 133 ++++++++ bom-reachability/queries/dependency_report.py | 169 ++++++++++ 4 files changed, 354 insertions(+), 319 deletions(-) delete mode 100644 bom-reachability/bom_reachability.py create mode 100644 bom-reachability/model/__init__.py create mode 100644 bom-reachability/queries/dependency_report.py diff --git a/bom-reachability/README.md b/bom-reachability/README.md index d52b4ba2..770cb693 100644 --- a/bom-reachability/README.md +++ b/bom-reachability/README.md @@ -35,14 +35,15 @@ This template turns a BOM into a dependency graph so you can trace those transit - A full transitive dependency tree per finished good, broken down by type (component vs raw material), from all-pairs `reachable(full=True)`. - A most-depended-on ranking of SKUs and a betweenness-centrality ranking that flags the structural bottleneck components. - An enumeration of the end-to-end assembly chains that build each finished good, reduced to the maximal (non-extendable) chains via path enumeration. -- An `assembly_depth` property persisted back onto each SKU, so the longest chain terminating at each finished good is queryable as ontology after the run. +- An `assembly_depth` property persisted back onto each SKU that ends an assembly chain (the finished goods, in the sample data), so the longest chain terminating there is queryable as ontology after the deploy. Built on RelationalAI's graph reasoner: **reachability** for transitive dependencies, **betweenness centrality** for bottlenecks, and **path enumeration** for assembly chains. ## What's included -- **Model**: two concepts (`SKU` and `BillOfMaterials`) wired into a directed dependency graph, plus a derived `feeds` self-relationship on `SKU` and a persisted `assembly_depth` property. -- **Runner**: `bom_reachability.py` — a single self-contained Python script that deploys the model to your Snowflake account and prints the reachability, betweenness centrality, and assembly-path report from the deployed model. +- **Ontology**: two concepts (`SKU` and `BillOfMaterials`) wired into a directed dependency graph, plus a derived `feeds` self-relationship on `SKU` and a persisted `assembly_depth` property. +- **Model**: `model/__init__.py` — the concepts, data, graph, and rules that `rai models deploy --path model` deploys to your Snowflake account. +- **Query**: `queries/dependency_report.py` — prints the reachability, betweenness centrality, and assembly-path report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/skus.csv` (9 SKUs across 3 tiers) and `data/bill_of_materials.csv` (14 BOM entries with site-specific assembly). See *Sample data* below. - **Outputs**: printed transitive-dependency lists per finished good, a most-depended-on and betweenness ranking, the maximal assembly chains, and an `assembly_depth` property written back onto the SKU ontology. @@ -63,34 +64,42 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen ## Quickstart -1. Download and extract this template: +Before you start, install Python 3.10 or later and get a Snowflake role that can deploy models, as described under Access. This template has no prescriptive solve, so it needs no setup beyond that role. + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/bom-reachability.zip). Unzip it and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/bom-reachability.zip unzip bom-reachability.zip cd bom-reachability ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. **Create and activate a virtual environment** + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. **Install dependencies** +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: ```bash python -m pip install . ``` -4. **Configure the Snowflake connection and the deployment** +4. **Configure your project** - Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist: ```yaml deployment: @@ -104,17 +113,23 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen schedule: standard ``` - Without the `deployment` section, `python bom_reachability.py` stops with an error naming the missing setting. +5. **Deploy the model** -5. **Run the template** + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: ```bash - python bom_reachability.py + rai models deploy --path model ``` - The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The deploy waits for the model's first refresh, which computes the reachability, betweenness, and assembly-path results. Without the `deployment` section, it stops with an error naming the missing setting. + +6. **Query the results** -6. **Expected output** (a few lines confirm success): + Print the report from the deployed model: + + ```bash + python -m queries.dependency_report + ``` ```text === BOM Dependency Graph === @@ -127,6 +142,17 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen Both finished goods resolve to 7 transitive dependencies, the processor is the single most disruptive component, and each finished good lands at assembly depth 2. The full printout and a step-by-step walkthrough are in `runbook.md`. +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The teardown removes the deployed objects and keeps the two schemas. + ## Template structure ```text @@ -134,13 +160,16 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen ├── README.md ├── runbook.md # paste-able graph-analysis walkthrough (RAI skills) ├── pyproject.toml # dependencies -├── bom_reachability.py # self-contained runner (reachability + betweenness + assembly paths) +├── model/ +│ └── __init__.py # concepts, data, graph, and rules (reachability + betweenness + assembly paths) +├── queries/ +│ └── dependency_report.py # report printed from the deployed model └── data/ ├── skus.csv # 9 SKUs across 3 tiers (raw material / component / finished good) └── bill_of_materials.csv # 14 site-specific BOM entries (output SKU requires input SKU) ``` -**Start here**: run `python bom_reachability.py` for the full analysis end to end (reachability, betweenness centrality, and assembly-path enumeration), or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.dependency_report` for the full analysis end to end (reachability, betweenness centrality, and assembly-path enumeration), or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -154,9 +183,9 @@ Built on RelationalAI's graph reasoner: **reachability** for transitive dependen - **Primary identifiers**: `SKU` by `id`; `BillOfMaterials` by `id`. - **Important invariants**: each `BillOfMaterials` row points an output SKU at one input SKU ("depends on"); the BOM is acyclic (raw materials feed components feed finished goods, never back). -The model also derives a `feeds` self-relationship on `SKU` (input SKU feeds output SKU), built from the `BillOfMaterials` intermediary and used to enumerate assembly paths, plus a persisted `assembly_depth` property (the longest assembly chain terminating at each SKU). +The model also derives a `feeds` self-relationship on `SKU` (input SKU feeds output SKU), built from the `BillOfMaterials` intermediary and used to enumerate assembly paths, plus a persisted `assembly_depth` property (the longest assembly chain terminating at each SKU that feeds nothing further). -For the full concept and property definitions, see `bom_reachability.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -186,7 +215,7 @@ Betweenness centrality ranks components by how many shortest dependency paths pa Where reachability returns dependency *pairs*, path enumeration returns the actual *build sequences*. It walks the derived `feeds` edge to enumerate every assembly path; because the BOM is acyclic, this yields exactly the simple paths with no cycle risk. A maximal-paths view keeps only the longest non-extendable chains, and the longest assembly depth is persisted as `SKU.assembly_depth`. -See `bom_reachability.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation, `queries/dependency_report.py` for the report, and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -211,7 +240,7 @@ Focus on the first changes most users will make. - Swap the CSV loads for `model.data(snowflake_table)` calls to run against Snowflake-backed tables instead of the bundled files. - Size the engine to the graph: reachability and betweenness scale with node and edge counts, and path enumeration grows with assembly depth and branching, so wide or deep BOMs want a larger engine. -- Pin `relationalai` to a known-good version for reproducible runs, and schedule the script (cron, orchestrator, or Snowflake task) to refresh results as the BOM changes. +- Pin `relationalai` to a known-good version for reproducible runs, and give the `standard` schedule in `raiconfig.yaml` a nonzero `interval_s` to refresh results on a timer as the BOM changes. ## Troubleshooting diff --git a/bom-reachability/bom_reachability.py b/bom-reachability/bom_reachability.py deleted file mode 100644 index 5e59159a..00000000 --- a/bom-reachability/bom_reachability.py +++ /dev/null @@ -1,296 +0,0 @@ -"""BOM Reachability (graph analysis) template. - -Answers: - "What components does Product X transitively depend on?" - "Which components are bottleneck dependencies across products?" - -Data: supply chain (SKUs + BillOfMaterials). -Graph: directed, unweighted. SKU nodes, BOM rows as edges (output -> input). - Edge direction: output_sku -> input_sku ("depends on"). -Algorithms: reachable(full=True) for transitive dependency tracing, - betweenness_centrality() for identifying structural bottlenecks. - -Assembly path enumeration (PREVIEW, requires relationalai>=1.15): enumerate the - bottom-up assembly chains that build each finished good. Derives a binary - SKU->SKU "feeds into" edge from the BillOfMaterials intermediary (input feeds - output), enumerates every assembly path with model.path(...).all().paths(), - filters to the maximal (longest non-extendable) chains so sub-paths are - suppressed, and persists each finished good's longest assembly depth back onto - the SKU ontology. - -Run: - `python bom_reachability.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh, and prints the report from the deployed model. - -Output: - Prints transitive dependency lists per finished product, a betweenness - centrality ranking that flags structural bottleneck components, and the - maximal raw-material -> finished-good assembly chains with per-finished-good - assembly depth. -""" - -from pathlib import Path - -import pandas as pd -from relationalai.semantics import Float, Integer, Model, String, where -from relationalai.semantics.reasoners.graph import Graph -from relationalai.semantics.std import aggregates as aggs - -# -------------------------------------------------- -# Configure inputs -# -------------------------------------------------- - -DATA_DIR = Path(__file__).parent / "data" - -# -------------------------------------------------- -# Define semantic model & load data -# -------------------------------------------------- - -model = Model("bom_reachability") - -# SKU concept -SKU = model.Concept("SKU", identify_by={"id": String}) -SKU.name = model.Property(f"{SKU} has {String:name}") -SKU.type = model.Property(f"{SKU} has type {String:type}") -SKU.category = model.Property(f"{SKU} in {String:category}") - -sku_data = model.data(pd.read_csv(DATA_DIR / "skus.csv")) -model.define(SKU.new(id=sku_data["ID"])) -where(SKU.id == sku_data["ID"]).define( - SKU.name(sku_data["NAME"]), - SKU.type(sku_data["TYPE"]), - SKU.category(sku_data["CATEGORY"]), -) - -# BillOfMaterials concept -BillOfMaterials = model.Concept("BillOfMaterials", identify_by={"id": String}) -BillOfMaterials.output_sku = model.Relationship(f"{BillOfMaterials} produces {SKU}") -BillOfMaterials.input_sku = model.Relationship(f"{BillOfMaterials} requires {SKU}") - -bom_data = model.data(pd.read_csv(DATA_DIR / "bill_of_materials.csv")) -model.define(BillOfMaterials.new(id=bom_data["ID"])) -where(BillOfMaterials.id == bom_data["ID"]).define( - BillOfMaterials.output_sku(SKU.lookup(id=bom_data["OUTPUT_SKU_ID"])), - BillOfMaterials.input_sku(SKU.lookup(id=bom_data["INPUT_SKU_ID"])), -) - -# -------------------------------------------------- -# Build graph: SKU nodes, BOM edges (directed) -# Edge: output_sku -> input_sku ("depends on") -# -------------------------------------------------- - -graph = Graph( - model, - directed=True, - weighted=False, - node_concept=SKU, - edge_concept=BillOfMaterials, - edge_src_relationship=BillOfMaterials.output_sku, - edge_dst_relationship=BillOfMaterials.input_sku, -) - -# Graph results the report reads, declared here so they deploy with the model. -num_nodes = graph.num_nodes() -num_edges = graph.num_edges() - -# Reachability: all transitive dependency pairs. -reachable = graph.reachable(full=True) - -# Betweenness centrality: structural bottlenecks. -betweenness = graph.betweenness_centrality() - -# -------------------------------------------------- -# Assembly path enumeration -# PREVIEW capability; requires relationalai>=1.15. -# -------------------------------------------------- -# Where betweenness scores a single *node*, this enumerates the full *chains* -# that build each finished good: every bottom-up assembly path from a raw -# material up through its components to the finished good. We derive a binary -# SKU->SKU "feeds into" edge from the BillOfMaterials intermediary (the input -# SKU feeds the output SKU), enumerate all such paths, then keep only the -# maximal (longest, non-extendable) chains so that sub-paths are suppressed. - -# Binary SKU->SKU edge: input_sku "feeds into" output_sku. This is the -# build-direction (bottom-up) reverse of the depends-on graph edge above. -SKU.feeds = model.Relationship(f"{SKU} feeds into {SKU}", short_name="feeds") -bom_ref = BillOfMaterials.ref() -sku_in, sku_out = SKU.ref(), SKU.ref() -model.where( - bom_ref.input_sku(sku_in), - bom_ref.output_sku(sku_out), -).define(sku_in.feeds(sku_out)) - -# Enumerate every assembly chain. The BOM is a DAG (raw materials -> components -# -> finished goods, never back), so .all().paths() already yields simple paths -- -# there is no cycle risk, and the repeat bound only needs to cover the BOM depth. -# Deepest chain here is raw_material -> component -> finished_good (2 hops); we -# allow extra headroom so deeper BOMs enumerate fully. -MAX_ASSEMBLY_HOPS = 4 -p = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS)).all().paths() - -# Persist each finished good's longest assembly depth onto the SKU ontology: the -# longest assembly chain that ends at it (max path length per final endpoint). -built = SKU.ref() -chain = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS), built).all().paths() -SKU.assembly_depth = model.Property(f"{SKU} has assembly depth {Integer:assembly_depth}") -model.where(chain, built.type == "FINISHED_GOOD").define( - built.assembly_depth(aggs.max(chain.length).per(built)) -) - - -def _is_contiguous_subsequence(short, long): - """True if `short` appears as a contiguous run inside `long`.""" - if len(short) >= len(long): - return False - return any(long[i:i + len(short)] == short for i in range(len(long) - len(short) + 1)) - - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh. - model.deploy() - - print("=== BOM Dependency Graph ===") - num_nodes.inspect() - num_edges.inspect() - - src, dst = graph.Node.ref("src"), graph.Node.ref("dst") - all_deps_df = ( - where(reachable(src, dst)) - .select( - src.id.alias("product_id"), - src.name.alias("product_name"), - src.type.alias("product_type"), - dst.id.alias("dep_id"), - dst.name.alias("dep_name"), - dst.type.alias("dep_type"), - ) - .to_df() - ) - - print(f"\nTotal reachable pairs: {len(all_deps_df)}") - - # -------------------------------------------------- - # Dependencies of each finished good - # -------------------------------------------------- - - finished = all_deps_df[all_deps_df["product_type"] == "FINISHED_GOOD"] - for product_id in sorted(finished["product_id"].unique()): - deps = finished[(finished["product_id"] == product_id) & (finished["dep_id"] != product_id)] - if len(deps) > 0: - product_name = deps["product_name"].iloc[0] - print(f"\n--- Dependencies of '{product_name}' ({product_id}) ---") - print(f" {len(deps)} transitive dependencies:") - for dep_type in ["COMPONENT", "RAW_MATERIAL"]: - type_deps = deps[deps["dep_type"] == dep_type] - if len(type_deps) > 0: - print(f"\n {dep_type}s ({len(type_deps)}):") - for _, row in type_deps.sort_values("dep_name").iterrows(): - print(f" - {row['dep_name']} ({row['dep_id']})") - - # -------------------------------------------------- - # Most depended-on components - # -------------------------------------------------- - - non_self = all_deps_df[all_deps_df["product_id"] != all_deps_df["dep_id"]] - if len(non_self) > 0: - dep_counts = ( - non_self.groupby(["dep_id", "dep_name", "dep_type"]) - .size() - .reset_index(name="depended_on_by") - .sort_values("depended_on_by", ascending=False) - ) - print("\n=== Most Depended-On SKUs ===") - print(dep_counts.to_string(index=False)) - - # -------------------------------------------------- - # Betweenness centrality: structural bottlenecks - # -------------------------------------------------- - - node = graph.Node.ref("n") - score = Float.ref("s") - btw_df = ( - where(betweenness(node, score)) - .select( - node.id.alias("sku_id"), - node.name.alias("sku_name"), - node.type.alias("type"), - node.category.alias("category"), - score.alias("betweenness"), - ) - .to_df() - .sort_values("betweenness", ascending=False) - .reset_index(drop=True) - ) - - print("\n=== Betweenness Centrality (bottleneck components) ===") - print(btw_df.to_string(index=False)) - - bottlenecks = btw_df[btw_df["betweenness"] > 0] - if len(bottlenecks) > 0: - top = bottlenecks.iloc[0] - print(f"\nTop bottleneck: {top['sku_name']} (betweenness={top['betweenness']:.4f})") - print(" Sits on the most dependency paths -- disruption here affects the most product lines.") - - # -------------------------------------------------- - # Assembly path enumeration - # -------------------------------------------------- - - print("\n=== Assembly Path Enumeration (PREVIEW) ===") - - assembly_df = ( - model.where(p) - .select( - p.alias("path"), - p.nodes["index"].alias("step"), - SKU(p.nodes).id.alias("sku_id"), - SKU(p.nodes).name.alias("sku_name"), - ) - .to_df() - ) - - # Reassemble each path in pandas: group on the path id, order by step index. - assembly_df["step"] = assembly_df["step"].astype(int) - paths = [] - for path_id, grp in assembly_df.groupby("path"): - ordered = grp.sort_values("step") - ids = ordered["sku_id"].tolist() - names = ordered["sku_name"].tolist() - paths.append({"ids": ids, "names": names, "length": len(ids) - 1}) - - print(f"\n Enumerated {len(paths)} assembly path(s) (<= {MAX_ASSEMBLY_HOPS} hops).") - - # (a) All assembly paths, longest first. - print("\n All assembly chains (feeds-into order):") - for path in sorted(paths, key=lambda x: -x["length"]): - print(f" [{path['length']} hop] " + " -> ".join(path["names"])) - - # (b) Maximal paths: keep only the longest, non-extendable chains -- a chain - # that is NOT a contiguous sub-sequence (prefix or suffix) of any longer chain. - # Suppressing these sub-paths leaves just the end-to-end assembly routes. - maximal = [ - path - for path in paths - if not any(other is not path and _is_contiguous_subsequence(path["ids"], other["ids"]) for other in paths) - ] - - print(f"\n Maximal assembly chains ({len(maximal)} of {len(paths)}, sub-paths suppressed):") - for path in sorted(maximal, key=lambda x: -x["length"]): - print(f" [{path['length']} hop] " + " -> ".join(path["names"])) - - print("\n Assembly depth persisted onto SKU (longest chain terminating at each):") - depth_df = ( - where(SKU.assembly_depth > 0) - .select( - SKU.id.alias("sku_id"), - SKU.name.alias("sku_name"), - SKU.type.alias("type"), - SKU.assembly_depth.alias("assembly_depth"), - ) - .to_df() - .sort_values(["assembly_depth", "sku_id"], ascending=[False, True], ignore_index=True) - ) - print(depth_df.to_string(index=False)) diff --git a/bom-reachability/model/__init__.py b/bom-reachability/model/__init__.py new file mode 100644 index 00000000..1a277c7b --- /dev/null +++ b/bom-reachability/model/__init__.py @@ -0,0 +1,133 @@ +"""BOM Reachability (graph analysis) template. + +Answers: + "What components does Product X transitively depend on?" + "Which components are bottleneck dependencies across products?" + +Data: supply chain (SKUs + BillOfMaterials). +Graph: directed, unweighted. SKU nodes, BOM rows as edges (output -> input). + Edge direction: output_sku -> input_sku ("depends on"). +Algorithms: reachable(full=True) for transitive dependency tracing, + betweenness_centrality() for identifying structural bottlenecks. + +Assembly path enumeration (PREVIEW, requires relationalai>=1.15): enumerate the + bottom-up assembly chains that build each finished good. Derives a binary + SKU->SKU "feeds into" edge from the BillOfMaterials intermediary (input feeds + output), enumerates every assembly path with model.path(...).all().paths(), + filters to the maximal (longest non-extendable) chains so sub-paths are + suppressed, and persists the longest assembly depth of each SKU that ends a + chain (feeds nothing further) back onto the SKU ontology. + +Deploy: + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh; `python -m + queries.dependency_report` then prints the report from the deployed model. +""" + +from pathlib import Path + +import pandas as pd +from relationalai.semantics import Integer, Model, String, not_, where +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.std import aggregates as aggs + +# -------------------------------------------------- +# Configure inputs +# -------------------------------------------------- + +DATA_DIR = Path(__file__).resolve().parents[1] / "data" + +# -------------------------------------------------- +# Define semantic model & load data +# -------------------------------------------------- + +model = Model("bom_reachability") + +# SKU concept +SKU = model.Concept("SKU", identify_by={"id": String}) +SKU.name = model.Property(f"{SKU} has {String:name}") +SKU.type = model.Property(f"{SKU} has type {String:type}") +SKU.category = model.Property(f"{SKU} in {String:category}") + +sku_data = model.data(pd.read_csv(DATA_DIR / "skus.csv")) +model.define(SKU.new(id=sku_data["ID"])) +where(SKU.id == sku_data["ID"]).define( + SKU.name(sku_data["NAME"]), + SKU.type(sku_data["TYPE"]), + SKU.category(sku_data["CATEGORY"]), +) + +# BillOfMaterials concept +BillOfMaterials = model.Concept("BillOfMaterials", identify_by={"id": String}) +BillOfMaterials.output_sku = model.Relationship(f"{BillOfMaterials} produces {SKU}") +BillOfMaterials.input_sku = model.Relationship(f"{BillOfMaterials} requires {SKU}") + +bom_data = model.data(pd.read_csv(DATA_DIR / "bill_of_materials.csv")) +model.define(BillOfMaterials.new(id=bom_data["ID"])) +where(BillOfMaterials.id == bom_data["ID"]).define( + BillOfMaterials.output_sku(SKU.lookup(id=bom_data["OUTPUT_SKU_ID"])), + BillOfMaterials.input_sku(SKU.lookup(id=bom_data["INPUT_SKU_ID"])), +) + +# -------------------------------------------------- +# Build graph: SKU nodes, BOM edges (directed) +# Edge: output_sku -> input_sku ("depends on") +# -------------------------------------------------- + +graph = Graph( + model, + directed=True, + weighted=False, + node_concept=SKU, + edge_concept=BillOfMaterials, + edge_src_relationship=BillOfMaterials.output_sku, + edge_dst_relationship=BillOfMaterials.input_sku, +) + +# Graph results the report reads, declared here so they deploy with the model. +num_nodes = graph.num_nodes() +num_edges = graph.num_edges() + +# Reachability: all transitive dependency pairs. +reachable = graph.reachable(full=True) + +# Betweenness centrality: structural bottlenecks. +betweenness = graph.betweenness_centrality() + +# -------------------------------------------------- +# Assembly path enumeration +# PREVIEW capability; requires relationalai>=1.15. +# -------------------------------------------------- +# Where betweenness scores a single *node*, this enumerates the full *chains* +# that build each finished good: every bottom-up assembly path from a raw +# material up through its components to the finished good. We derive a binary +# SKU->SKU "feeds into" edge from the BillOfMaterials intermediary (the input +# SKU feeds the output SKU), enumerate all such paths, then keep only the +# maximal (longest, non-extendable) chains so that sub-paths are suppressed. + +# Binary SKU->SKU edge: input_sku "feeds into" output_sku. This is the +# build-direction (bottom-up) reverse of the depends-on graph edge above. +SKU.feeds = model.Relationship(f"{SKU} feeds into {SKU}", short_name="feeds") +bom_ref = BillOfMaterials.ref() +sku_in, sku_out = SKU.ref(), SKU.ref() +model.where( + bom_ref.input_sku(sku_in), + bom_ref.output_sku(sku_out), +).define(sku_in.feeds(sku_out)) + +# Enumerate every assembly chain. The BOM is a DAG (raw materials -> components +# -> finished goods, never back), so .all().paths() already yields simple paths -- +# there is no cycle risk, and the repeat bound only needs to cover the BOM depth. +# Deepest chain here is raw_material -> component -> finished_good (2 hops); we +# allow extra headroom so deeper BOMs enumerate fully. +MAX_ASSEMBLY_HOPS = 4 +p = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS)).all().paths() + +# Persist an assembly depth onto every SKU that ends an assembly chain (it feeds +# nothing further): the length of the longest chain that ends at it. +built, next_sku = SKU.ref(), SKU.ref() +chain = model.path(SKU.feeds.repeat(1, MAX_ASSEMBLY_HOPS), built).all().paths() +SKU.assembly_depth = model.Property(f"{SKU} has assembly depth {Integer:assembly_depth}") +model.where(chain, not_(built.feeds(next_sku))).define( + built.assembly_depth(aggs.max(chain.length).per(built)) +) diff --git a/bom-reachability/queries/dependency_report.py b/bom-reachability/queries/dependency_report.py new file mode 100644 index 00000000..f55cc258 --- /dev/null +++ b/bom-reachability/queries/dependency_report.py @@ -0,0 +1,169 @@ +"""Dependency, bottleneck, and assembly-chain report from the deployed BOM model. + +Run after `rai models deploy --path model`: + python -m queries.dependency_report +""" + +from model import ( + MAX_ASSEMBLY_HOPS, + SKU, + betweenness, + graph, + model, + num_edges, + num_nodes, + p, + reachable, +) +from relationalai.semantics import Float, where + + +def _is_contiguous_subsequence(short, long): + """True if `short` appears as a contiguous run inside `long`.""" + if len(short) >= len(long): + return False + return any(long[i:i + len(short)] == short for i in range(len(long) - len(short) + 1)) + + +if __name__ == "__main__": + print("=== BOM Dependency Graph ===") + num_nodes.inspect() + num_edges.inspect() + + src, dst = graph.Node.ref("src"), graph.Node.ref("dst") + all_deps_df = ( + where(reachable(src, dst)) + .select( + src.id.alias("product_id"), + src.name.alias("product_name"), + src.type.alias("product_type"), + dst.id.alias("dep_id"), + dst.name.alias("dep_name"), + dst.type.alias("dep_type"), + ) + .to_df() + ) + + print(f"\nTotal reachable pairs: {len(all_deps_df)}") + + # -------------------------------------------------- + # Dependencies of each finished good + # -------------------------------------------------- + + finished = all_deps_df[all_deps_df["product_type"] == "FINISHED_GOOD"] + for product_id in sorted(finished["product_id"].unique()): + deps = finished[(finished["product_id"] == product_id) & (finished["dep_id"] != product_id)] + if len(deps) > 0: + product_name = deps["product_name"].iloc[0] + print(f"\n--- Dependencies of '{product_name}' ({product_id}) ---") + print(f" {len(deps)} transitive dependencies:") + for dep_type in ["COMPONENT", "RAW_MATERIAL"]: + type_deps = deps[deps["dep_type"] == dep_type] + if len(type_deps) > 0: + print(f"\n {dep_type}s ({len(type_deps)}):") + for _, row in type_deps.sort_values("dep_name").iterrows(): + print(f" - {row['dep_name']} ({row['dep_id']})") + + # -------------------------------------------------- + # Most depended-on components + # -------------------------------------------------- + + non_self = all_deps_df[all_deps_df["product_id"] != all_deps_df["dep_id"]] + if len(non_self) > 0: + dep_counts = ( + non_self.groupby(["dep_id", "dep_name", "dep_type"]) + .size() + .reset_index(name="depended_on_by") + .sort_values("depended_on_by", ascending=False) + ) + print("\n=== Most Depended-On SKUs ===") + print(dep_counts.to_string(index=False)) + + # -------------------------------------------------- + # Betweenness centrality: structural bottlenecks + # -------------------------------------------------- + + node = graph.Node.ref("n") + score = Float.ref("s") + btw_df = ( + where(betweenness(node, score)) + .select( + node.id.alias("sku_id"), + node.name.alias("sku_name"), + node.type.alias("type"), + node.category.alias("category"), + score.alias("betweenness"), + ) + .to_df() + .sort_values("betweenness", ascending=False) + .reset_index(drop=True) + ) + + print("\n=== Betweenness Centrality (bottleneck components) ===") + print(btw_df.to_string(index=False)) + + bottlenecks = btw_df[btw_df["betweenness"] > 0] + if len(bottlenecks) > 0: + top = bottlenecks.iloc[0] + print(f"\nTop bottleneck: {top['sku_name']} (betweenness={top['betweenness']:.4f})") + print(" Sits on the most dependency paths -- disruption here affects the most product lines.") + + # -------------------------------------------------- + # Assembly path enumeration + # -------------------------------------------------- + + print("\n=== Assembly Path Enumeration (PREVIEW) ===") + + assembly_df = ( + model.where(p) + .select( + p.alias("path"), + p.nodes["index"].alias("step"), + SKU(p.nodes).id.alias("sku_id"), + SKU(p.nodes).name.alias("sku_name"), + ) + .to_df() + ) + + # Reassemble each path in pandas: group on the path id, order by step index. + assembly_df["step"] = assembly_df["step"].astype(int) + paths = [] + for path_id, grp in assembly_df.groupby("path"): + ordered = grp.sort_values("step") + ids = ordered["sku_id"].tolist() + names = ordered["sku_name"].tolist() + paths.append({"ids": ids, "names": names, "length": len(ids) - 1}) + + print(f"\n Enumerated {len(paths)} assembly path(s) (<= {MAX_ASSEMBLY_HOPS} hops).") + + # (a) All assembly paths, longest first. + print("\n All assembly chains (feeds-into order):") + for path in sorted(paths, key=lambda x: -x["length"]): + print(f" [{path['length']} hop] " + " -> ".join(path["names"])) + + # (b) Maximal paths: keep only the longest, non-extendable chains -- a chain + # that is NOT a contiguous sub-sequence (prefix or suffix) of any longer chain. + # Suppressing these sub-paths leaves just the end-to-end assembly routes. + maximal = [ + path + for path in paths + if not any(other is not path and _is_contiguous_subsequence(path["ids"], other["ids"]) for other in paths) + ] + + print(f"\n Maximal assembly chains ({len(maximal)} of {len(paths)}, sub-paths suppressed):") + for path in sorted(maximal, key=lambda x: -x["length"]): + print(f" [{path['length']} hop] " + " -> ".join(path["names"])) + + print("\n Assembly depth persisted onto SKU (longest chain terminating at each):") + depth_df = ( + where(SKU.assembly_depth > 0) + .select( + SKU.id.alias("sku_id"), + SKU.name.alias("sku_name"), + SKU.type.alias("type"), + SKU.assembly_depth.alias("assembly_depth"), + ) + .to_df() + .sort_values(["assembly_depth", "sku_id"], ascending=[False, True], ignore_index=True) + ) + print(depth_df.to_string(index=False)) From 84b35c6c2e1f7a93298557c9bef9aae94ca4c30f Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:12:40 -0800 Subject: [PATCH 047/162] money_laundering_motif_detection: deploy one model with all three motifs The shared ontology and the three motifs move into one model package (model/ontology.py, butterfly.py, kyc_burst.py, smurf_army.py), deployed with `rai models deploy --path model`. Each motif keeps its constraints and solver settings as its own named Problem, so every refresh solves all three. Each report moves to queries/_motifs.py, and the README Quickstart becomes deploy, query, and tear down. Previously the three runners deployed the same model name with only their own motif, so running a different runner replaced the deployed motif. --- money_laundering_motif_detection/README.md | 206 +++++++++++------- .../model/__init__.py | 13 ++ .../butterfly.py} | 120 +--------- .../kyc_burst.py} | 95 +------- .../model/ontology.py | 54 +++++ .../smurf_army.py} | 95 +------- .../model_setup.py | 69 ------ .../queries/butterfly_motifs.py | 100 +++++++++ .../queries/kyc_burst_motifs.py | 83 +++++++ .../queries/smurf_army_motifs.py | 83 +++++++ money_laundering_motif_detection/runbook.md | 6 +- 11 files changed, 492 insertions(+), 432 deletions(-) create mode 100644 money_laundering_motif_detection/model/__init__.py rename money_laundering_motif_detection/{motif_butterfly.py => model/butterfly.py} (65%) rename money_laundering_motif_detection/{motif_kyc_burst.py => model/kyc_burst.py} (62%) create mode 100644 money_laundering_motif_detection/model/ontology.py rename money_laundering_motif_detection/{motif_smurf_army.py => model/smurf_army.py} (66%) delete mode 100644 money_laundering_motif_detection/model_setup.py create mode 100644 money_laundering_motif_detection/queries/butterfly_motifs.py create mode 100644 money_laundering_motif_detection/queries/kyc_burst_motifs.py create mode 100644 money_laundering_motif_detection/queries/smurf_army_motifs.py diff --git a/money_laundering_motif_detection/README.md b/money_laundering_motif_detection/README.md index 698a2933..adadeac8 100644 --- a/money_laundering_motif_detection/README.md +++ b/money_laundering_motif_detection/README.md @@ -18,13 +18,13 @@ tags: Banking compliance and financial-crime teams hunt for "layering" -- the obfuscation phase of money laundering, where a launderer routes funds across a coordinated cohort of intermediary accounts so no single transaction triggers a [FinCEN currency-transaction-report](https://www.ecfr.gov/current/title-31/subtitle-B/chapter-X/part-1010/subpart-C/section-1010.311) (CTR) filing. Launderers hide these schemes behind ordinary-looking transaction shapes, padding them with decoy edges and lookalike account clusters so the fraudulent cohort blends into the noise. What separates a real motif from its lookalikes is a condition that must hold jointly across the whole cohort the detector picks, not any single account or transaction on its own. -**This template detects three classes of layering motif on one shared transaction ledger using constraint-satisfaction (CSP) reasoning, which enforces such joint conditions across a solver-chosen subset of accounts and edges.** The three motifs run as independent scripts over a shared ontology: +**This template detects three classes of layering motif on one shared transaction ledger using constraint-satisfaction (CSP) reasoning, which enforces such joint conditions across a solver-chosen subset of accounts and edges.** The three motifs share one ontology and deploy together as one model, with one report each: -| Motif | Joint condition it enforces | Topology | Runner | +| Motif | Joint condition it enforces | Topology | Model / report | |---|---|---|---| -| **Butterfly** | Per-hub flow balance: each chosen hub's incoming dollars equal its outgoing dollars | Scatter-gather (one source -> K hubs -> one destination) | `motif_butterfly.py` | -| **Smurf army** | Pairwise-distinct beneficial owners across the chosen cohort, plus a sum-to-target within a tight time window | Fan-in (K sources -> one destination) | `motif_smurf_army.py` | -| **KYC-mix burst** | Cardinality: at least a floor number of the chosen accounts are retail-tier | Fan-in within a tight time window | `motif_kyc_burst.py` | +| **Butterfly** | Per-hub flow balance: each chosen hub's incoming dollars equal its outgoing dollars | Scatter-gather (one source -> K hubs -> one destination) | `model/butterfly.py` / `queries/butterfly_motifs.py` | +| **Smurf army** | Pairwise-distinct beneficial owners across the chosen cohort, plus a sum-to-target within a tight time window | Fan-in (K sources -> one destination) | `model/smurf_army.py` / `queries/smurf_army_motifs.py` | +| **KYC-mix burst** | Cardinality: at least a floor number of the chosen accounts are retail-tier | Fan-in within a tight time window | `model/kyc_burst.py` / `queries/kyc_burst_motifs.py` | Each motif is grounded in a documented anti-money-laundering (AML) typology -- the eight-pattern taxonomy from [IBM AMLworld](https://arxiv.org/abs/2306.16424) (Altman et al., NeurIPS 2023) and the structuring / smurf-army typology from FATF and FinCEN (31 CFR § 1010.311). The same joint-condition patterns generalize beyond banking: per-hub flow balance surfaces collusion rings in marketplace fraud, pairwise distinctness surfaces identity-cohort scams in subscription services, and cardinality floors surface compliance-mix violations in regulatory audits. @@ -37,16 +37,16 @@ Each motif is grounded in a documented anti-money-laundering (AML) typology -- t ## What you'll build -A shared ontology in `model_setup.py` and three independent CSP runners, each adding its own decision properties and constraints to the same `Account` / `Transaction` model. Every runner solves a separate `Problem` and prints the analyst-facing motif tables. +A shared ontology in `model/ontology.py` and three CSP motifs, each adding its own decision properties and constraints to the same `Account` / `Transaction` model. Each motif configures a separate, named `Problem`; every refresh of the deployed model solves all three, and each motif's report prints its analyst-facing tables. -Shared ontology (in `model_setup.py`): +Shared ontology (in `model/ontology.py`): - `Account` concept with `bo_id` (beneficial owner cluster), `kyc_tier`, and `jurisdiction` properties - `Transaction` concept with `src`, `dst`, `amount_dollars`, `ts_minutes` Each motif owns its own decisions and constraints. The bullets below summarise the integrity constraints per motif, with the load-bearing joint condition in bold; the "How it works" section walks each one with code. -- **Butterfly** (`motif_butterfly.py`): +- **Butterfly** (`model/butterfly.py`): - Decisions: `Transaction.is_motif`, `Account.is_source` / `is_hub` / `is_dest` - Per-account flow conservation in *count* (source out-degree K, hub in/out-degree 1, dest in-degree K) - Global motif-edge count (= 2K) -- closes a gap in the per-account ICs above for accounts with one-sided traffic (no outgoing or no incoming rows), where the per-account IC isn't instantiated and the role variables would otherwise be free @@ -55,7 +55,7 @@ Each motif owns its own decisions and constraints. The bullets below summarise t - Same-beneficial-owner pairwise filter over chosen hub pairs - Sub-threshold amount filter -- **Smurf army** (`motif_smurf_army.py`): +- **Smurf army** (`model/smurf_army.py`): - Decisions: `Account.is_smurf`, `Transaction.is_smurf_tx` - Exactly N smurfs sending to a single target merchant - **Pairwise distinct beneficial owners** across chosen smurfs @@ -63,22 +63,24 @@ Each motif owns its own decisions and constraints. The bullets below summarise t - All chosen smurf-tx within `SMURF_WINDOW_MINUTES` of each other - Sub-threshold amount filter -- **KYC-mix burst** (`motif_kyc_burst.py`): +- **KYC-mix burst** (`model/kyc_burst.py`): - Decisions: `Account.is_burst`, `Transaction.is_burst_tx` - Exactly N accounts in a coordinated burst to a single target merchant - **`>= RETAIL_FLOOR` retail-tier accounts among chosen** -- a cardinality constraint over the *selected* subset - All chosen burst-tx within `BURST_WINDOW_MINUTES` of each other - Sub-threshold amount filter -Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is available on every runner for batch analyst-triage workflows. The butterfly and KYC-burst runners exhaust to `OPTIMAL` on the bundled data; the smurf runner hits `SOLUTION_LIMIT` because pairwise-distinct beneficial-owner cohorts can swap any one chosen smurf for the dup-BO decoys (47 or 48), opening more cohorts than `MAX_SMURF_MOTIFS=5`. Raise the limit to enumerate further. +Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is available on every motif for batch analyst-triage workflows. The butterfly and KYC-burst motifs exhaust to `OPTIMAL` on the bundled data; the smurf motif hits `SOLUTION_LIMIT` because pairwise-distinct beneficial-owner cohorts can swap any one chosen smurf for the dup-BO decoys (47 or 48), opening more cohorts than `MAX_SMURF_MOTIFS=5`. Raise the limit to enumerate further. ## What's included -- `model_setup.py` -- shared ontology (`Account`, `Transaction`) + CSV load. Every motif runner imports `create_model()` and adds its own decisions and constraints on top. -- `motif_butterfly.py` -- per-hub flow conservation over a decision-selected edge subset. -- `motif_smurf_army.py` -- pairwise distinct beneficial owners + sum-target + tight time window. -- `motif_kyc_burst.py` -- cardinality on the chosen subset + tight time window. -- **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. +- `model/` -- the model that `rai models deploy --path model` deploys: + - `ontology.py` -- shared ontology (`Account`, `Transaction`) + CSV load. Every motif module imports the model from here and adds its own decisions and constraints on top. + - `butterfly.py` -- per-hub flow conservation over a decision-selected edge subset. + - `smurf_army.py` -- pairwise distinct beneficial owners + sum-target + tight time window. + - `kyc_burst.py` -- cardinality on the chosen subset + tight time window. +- `queries/` -- one report per motif, printed from the deployed model: `butterfly_motifs.py`, `smurf_army_motifs.py`, `kyc_burst_motifs.py`. +- **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the code itself. - `data/accounts.csv` -- 90 accounts across 69 beneficial-owner clusters, including two butterfly-cluster hubs (`bo_id=100`, `bo_id=200`), a five-account smurf cohort with pairwise-distinct BOs (`bo_id=1701..1705`), a five-account KYC-mix burst cohort (4 retail + 1 business across 4 jurisdictions), plus dup-BO / out-of-window / over-amount decoys for each motif and 30 unrelated noise accounts. - `data/transactions.csv` -- 138 directed transactions: 12 butterfly motif edges, 5 smurf motif edges, 5 KYC-burst motif edges, named decoys / cross-cluster traffic, and 60 noise transactions deterministically generated via `data/generate.py`. - `data/generate.py` -- the deterministic generator that produced the bundled CSVs. Re-run only if you change the planted-motif design or want to regenerate at a different size; the template ships its outputs so users don't need to run it. @@ -88,7 +90,7 @@ Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is ava ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart's setup block covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -96,28 +98,61 @@ Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is ava ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later, and have an account administrator complete the one-time model deployment setup described under Access. + +Before the first deploy, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .MONEY_LAUNDERING_MOTIF_DETECTION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; +``` + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/money_laundering_motif_detection.zip). Unzip it and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/money_laundering_motif_detection.zip unzip money_laundering_motif_detection.zip cd money_laundering_motif_detection ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -136,43 +171,27 @@ Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is ava pat_user: pat_warehouse: ``` - Without the `deployment` section, each runner stops with an error naming the missing setting. - - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .MONEY_LAUNDERING_MOTIF_DETECTION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .MONEY_LAUNDERING_MOTIF_DETECTION_META TO APPLICATION ; - ``` - The deploy creates the public schema `.MONEY_LAUNDERING_MOTIF_DETECTION` if it doesn't exist. -5. Run any of the three motif detectors. They share the bundled data and the configured schema pair: +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: + ```bash - python motif_butterfly.py - python motif_smurf_army.py - python motif_kyc_burst.py + rai models deploy --path model ``` - Each runner deploys the model, waits for the first refresh, which runs that runner's solve, and prints its report from the deployed model. `rai models deploy --path motif_butterfly.py` (or another runner) deploys without printing the report. - The three runners deploy the same model, `money_laundering_motif_detection`, each with only its own motif. Running a different runner than the last one redeploys, replacing the previous runner's motif, and its refresh solves the new motif; the deployed model then holds only that runner's motif. Running the same runner again without changing it redeploys nothing and does not solve again; it reports the latest refresh. A change to a runner or to its `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the same runner: - ```sql - CALL .MONEY_LAUNDERING_MOTIF_DETECTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); - ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The model holds all three motifs. The deploy creates the public schema `.MONEY_LAUNDERING_MOTIF_DETECTION` if it doesn't exist and waits for the first refresh, which runs the three motif solves. Without the `deployment` section, it stops with an error naming the missing setting. + + Deploying again without changing the model redeploys nothing and does not solve again; the queries report the latest refresh. A change to a motif module or to its `problem.configure(...)` call redeploys, and the redeploy's refresh solves all three motifs. To solve again without a change, refresh the deployed model in Snowflake with `CALL .MONEY_LAUNDERING_MOTIF_DETECTION_META._INTERNAL_REFRESH_MODEL('standard', NULL);`, then run the queries again. -6. Expected output. The bundled dataset plants exactly the motifs each detector targets, plus dup-BO / out-of-window / over-amount decoys for each. Solver build strings and exact wall times will vary; the structure of the output is stable. +6. **Query the results** + + Print each motif's report from the deployed model. The bundled dataset plants exactly the motifs each detector targets, plus dup-BO, out-of-window, and over-amount decoys for each. Solver build strings and exact wall times will vary; the structure of the output is stable: + + ```bash + python -m queries.butterfly_motifs + ``` - `motif_butterfly.py` finds the two planted scatter-gathers (cluster `100` -> `WireRecipientCorp`, cluster `200` -> `OffshoreLLC`) and exhausts the search: ```text Solve result: • status: OPTIMAL @@ -187,7 +206,12 @@ Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is ava 11 1 12 9 HubAccB3 10 OffshoreLLC 7780 44 ``` - `motif_smurf_army.py` hits `MAX_SMURF_MOTIFS=5` because the bundled data admits multiple sum-target cohorts: the planted five-distinct-BO cohort plus cohorts that swap one of the dup-BO decoys (47 or 48, both `bo_id=1706`) in for one of the planted smurfs. `distinct_bo_ic` is the IC that prevents the {47, 48, + three planted} cohort -- which sums in-range -- from being returned. The first cohort enumerated by the solver is typically the planted one: + The butterfly search finds the two planted scatter-gathers (cluster `100` -> `WireRecipientCorp`, cluster `200` -> `OffshoreLLC`) and exhausts the search. + + ```bash + python -m queries.smurf_army_motifs + ``` + ```text Solve result: • status: SOLUTION_LIMIT @@ -206,7 +230,12 @@ Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is ava ... ``` - `motif_kyc_burst.py` finds two valid 5-account cohorts that satisfy the retail floor (4 retail, all within 60 min) -- the planted cohort with Brookline Industries plus an alternative-business swap (Newland Capital Inc): + The smurf-army search hits `MAX_SMURF_MOTIFS=5` because the bundled data admits multiple sum-target cohorts: the planted five-distinct-BO cohort plus cohorts that swap one of the dup-BO decoys (47 or 48, both `bo_id=1706`) in for one of the planted smurfs. `distinct_bo_ic` is the constraint that prevents the 47, 48, and three-planted cohort, which sums in range, from being returned. The first cohort the solver enumerates is typically the planted one. + + ```bash + python -m queries.kyc_burst_motifs + ``` + ```text Solve result: • status: OPTIMAL @@ -226,22 +255,41 @@ Multi-solution enumeration via `problem.configure(..., solution_limit=K)` is ava 9 1 57 Newland Capital Inc business Germany 2206 ``` + The KYC-mix burst search finds two valid 5-account cohorts that satisfy the retail floor (4 retail, all within 60 min): the planted cohort with Brookline Industries plus an alternative-business swap (Newland Capital Inc). + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The teardown removes the deployed objects and keeps the two schemas. + ## Template structure ```text . ├── README.md ├── pyproject.toml -├── model_setup.py # shared ontology + data load -├── motif_butterfly.py # per-hub flow conservation -├── motif_smurf_army.py # pairwise distinct BOs + sum-target -├── motif_kyc_burst.py # cardinality on chosen subset +├── model/ +│ ├── __init__.py # the deployed model: ontology + all three motifs +│ ├── ontology.py # shared ontology + data load +│ ├── butterfly.py # per-hub flow conservation +│ ├── smurf_army.py # pairwise distinct BOs + sum-target +│ └── kyc_burst.py # cardinality on chosen subset +├── queries/ +│ ├── butterfly_motifs.py # butterfly report +│ ├── smurf_army_motifs.py # smurf-army report +│ └── kyc_burst_motifs.py # KYC-mix burst report └── data/ ├── accounts.csv ├── transactions.csv └── generate.py # deterministic generator (one-time, not runtime) ``` -**Start here**: run `python motif_butterfly.py` (or `motif_smurf_army.py` / `motif_kyc_burst.py`) for a motif detector end to end, or follow `runbook.md` to reproduce the template step by step with the RAI skills. Each runner is self-contained -- it imports the shared ontology from `model_setup.py`, loads the bundled data, and solves its own motif independently. +**Start here**: deploy with `rai models deploy --path model`, then run `python -m queries.butterfly_motifs` (or `smurf_army_motifs` / `kyc_burst_motifs`) for a motif report, or follow `runbook.md` to reproduce the template step by step with the RAI skills. Each motif module adds its own decisions and constraints to the shared ontology in `model/ontology.py`, and each motif is solved as its own `Problem`. ## Sample data @@ -252,17 +300,17 @@ The bundled CSVs are synthetic and tuned so each of the three motifs is visible ## Model overview -All three runners share one ontology, defined in `model_setup.py`: `Account` vertices and `Transaction` directed edges. Each runner adds its own decision-valued properties (which account or transaction is part of the motif) on top of these base concepts at solve time. +All three motifs share one ontology, defined in `model/ontology.py`: `Account` vertices and `Transaction` directed edges. Each motif adds its own decision-valued properties (which account or transaction is part of the motif) on top of these base concepts. - **Key entities**: `Account` (a bank account) and `Transaction` (a directed transfer between two accounts). - **Primary identifiers**: integer `id` on `Account`; integer `tx_id` on `Transaction`. - **Important invariants**: `Transaction.src` and `Transaction.dst` reference existing accounts; `amount_dollars` is positive; layering edges sit under the `AMOUNT_THRESHOLD_DOLLARS` CTR line (default `$10,000`); `ts_minutes` is non-negative. -`Account` carries `name`, `bo_id` (beneficial-owner cluster), `kyc_tier`, and `jurisdiction`; `Transaction` carries `src`, `dst`, `amount_dollars`, and `ts_minutes`. Each runner then adds its own motif-specific decision properties on top -- e.g. `Account.is_hub` / `is_smurf` / `is_burst` and `Transaction.is_motif` / `is_smurf_tx` / `is_burst_tx` -- populated by the solver. For the full concept and property definitions, see `model_setup.py`; `runbook.md` builds them step by step with the RAI skills. +`Account` carries `name`, `bo_id` (beneficial-owner cluster), `kyc_tier`, and `jurisdiction`; `Transaction` carries `src`, `dst`, `amount_dollars`, and `ts_minutes`. Each motif then adds its own motif-specific decision properties on top -- e.g. `Account.is_hub` / `is_smurf` / `is_burst` and `Transaction.is_motif` / `is_smurf_tx` / `is_burst_tx` -- populated by the solver. For the full concept and property definitions, see `model/ontology.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works -The three motifs share an `Account` / `Transaction` ontology defined in `model_setup.py`. Each runner adds its own decision-valued properties, builds its own `Problem`, and constrains it with a different *class* of CSP technique. The "What you'll build" section above lists every constraint per motif; below is the load-bearing joint condition for each, with the part that rules / paths / graph reasoning cannot express called out explicitly. +The three motifs share an `Account` / `Transaction` ontology defined in `model/ontology.py`. Each motif adds its own decision-valued properties, builds its own `Problem`, and constrains it with a different *class* of CSP technique. The "What you'll build" section above lists every constraint per motif; below is the load-bearing joint condition for each, with the part that rules / paths / graph reasoning cannot express called out explicitly. **Butterfly: per-vertex aggregate equality over a decision-selected edge subset.** For every account the solver picks as a hub, the dollars it receives via motif edges must equal what it forwards, within `CONSERVATION_TOLERANCE_DOLLARS`. The constraint is arithmetic over a *decision-selected subset* of edges -- it cannot be evaluated until the solver has chosen which transactions are in the motif and which accounts are hubs. It is written in big-M form so it is active when an account is a hub and vacuous otherwise, with the M coefficient computed from the data. (Big-M rather than half-reification: a half-reified `implies` introduces a free Boolean auxiliary per non-hub account that the solver treats as search space, returning thousands of trivially-distinct solutions; big-M with a data-computed bound has no auxiliary, so enumeration stays clean and the solver exhausts after the data's actual motifs.) A path enumeration sees one walk at a time and never the joint condition; a rules-only encoding can sum-per-vertex but cannot bind that sum to "the chosen subset of edges, where the chosen accounts are hubs." @@ -270,22 +318,22 @@ The three motifs share an `Account` / `Transaction` ontology defined in `model_s **KYC-mix burst: cardinality over a decision-selected vertex subset.** Among the N chosen burst accounts, at least `RETAIL_FLOOR` must be retail-tier. Rules can label each account's tier; only CSP enforces the count *over the selected subset*. The launder-grade burst signature is "K accounts with a retail-tier floor transacting together in a tight window." A graph-pattern matcher can find the time-window cluster but has no way to express "and at least M of the chosen K are retail-tier." -**Solver call and enumeration.** Every runner calls the MiniZinc solver with a `solution_limit`. The variable subconcept returned by `solve_for(...)` exposes a per-solution values relationship; filtering on the chosen rows surfaces what the solver picked, since the populated property reflects only the first solution. These are pure satisfaction problems with no objective, so `status: OPTIMAL` means the solver enumerated all feasible motifs up to `solution_limit` -- not an optimisation verdict -- while `SOLUTION_LIMIT` means the limit was hit before enumeration exhausted (raise `MAX_*_MOTIFS` to surface more). +**Solver call and enumeration.** Every motif configures the MiniZinc solver with a `solution_limit`. The variable subconcept returned by `solve_for(...)` exposes a per-solution values relationship; filtering on the chosen rows surfaces what the solver picked, since the populated property reflects only the first solution. These are pure satisfaction problems with no objective, so `status: OPTIMAL` means the solver enumerated all feasible motifs up to `solution_limit` -- not an optimisation verdict -- while `SOLUTION_LIMIT` means the limit was hit before enumeration exhausted (raise `MAX_*_MOTIFS` to surface more). -See `model_setup.py` and the three motif runners for the implementation, and `runbook.md` for the skill-driven reproduction. +See `model/` for the implementation, `queries/` for the reports, and `runbook.md` for the skill-driven reproduction. ## Customize this template ### Use your own data -- Replace the two CSV files with your own accounts and transactions; every runner picks them up automatically. The shared ontology in `model_setup.py` reads columns by name (`id`, `name`, `bo_id`, `kyc_tier`, `jurisdiction`, `tx_id`, `src_id`, `dst_id`, `amount_dollars`, `ts_minutes`). Drop a column you don't have by deleting the corresponding `Account. = model.Property(...)` line and any constraint that references it. -- Point the smurf and burst runners at a different target merchant by editing `SMURF_TARGET_DESTINATION_ID` in `motif_smurf_army.py` and `BURST_TARGET_DESTINATION_ID` in `motif_kyc_burst.py`. On a real ledger you would set these from your watchlist of known-launder destinations. +- Replace the two CSV files with your own accounts and transactions; every motif picks them up on the next deploy. The shared ontology in `model/ontology.py` reads columns by name (`id`, `name`, `bo_id`, `kyc_tier`, `jurisdiction`, `tx_id`, `src_id`, `dst_id`, `amount_dollars`, `ts_minutes`). Drop a column you don't have by deleting the corresponding `Account. = model.Property(...)` line and any constraint that references it. +- Point the smurf and burst motifs at a different target merchant by editing `SMURF_TARGET_DESTINATION_ID` in `model/smurf_army.py` and `BURST_TARGET_DESTINATION_ID` in `model/kyc_burst.py`. On a real ledger you would set these from your watchlist of known-launder destinations. -**Test against real public data.** The bundled CSVs are synthetic and tuned to make the three motifs visible. For production-style validation, point the runners at a slice of [IBM AMLworld HI-Small / LI-Small](https://github.com/IBM/AML-Data) (CDLA-Sharing-1.0; the dataset paper is Altman et al., NeurIPS 2023). AMLworld natively labels the eight-pattern taxonomy used in the butterfly motif (scatter-gather), and a 500-2000 transaction slice will solve in seconds. KYC tiers and jurisdictions are not in AMLworld natively; the smurf-army and KYC-burst runners would need either synthetic augmentation columns or a different vertex category mapped from AMLworld's `Bank` / `Payment Format` columns. +**Test against real public data.** The bundled CSVs are synthetic and tuned to make the three motifs visible. For production-style validation, point the model at a slice of [IBM AMLworld HI-Small / LI-Small](https://github.com/IBM/AML-Data) (CDLA-Sharing-1.0; the dataset paper is Altman et al., NeurIPS 2023). AMLworld natively labels the eight-pattern taxonomy used in the butterfly motif (scatter-gather), and a 500-2000 transaction slice will solve in seconds. KYC tiers and jurisdictions are not in AMLworld natively; the smurf-army and KYC-burst motifs would need either synthetic augmentation columns or a different vertex category mapped from AMLworld's `Bank` / `Payment Format` columns. ### Tune parameters -- Edit the per-runner constants to tune the thresholds. The FinCEN CTR threshold (`AMOUNT_THRESHOLD_DOLLARS = 10_000`) lives in `model_setup.py`. Per-hub conservation tolerance, smurf target dollars, smurf window, burst window, and retail floor are at the top of each motif's runner. +- Edit the per-motif constants to tune the thresholds, then deploy again. The FinCEN CTR threshold (`AMOUNT_THRESHOLD_DOLLARS = 10_000`) lives in `model/ontology.py`. Per-hub conservation tolerance, smurf target dollars, smurf window, burst window, and retail floor are at the top of each motif's module in `model/`. - Sweep the smurf target by parameterizing `SMURF_TARGET_DOLLARS` and re-running; alternatively, replace the equality with a minimum cumulative deposit constraint (`>= SMURF_TARGET_DOLLARS - tol`) if your watchlist tracks "at least $X laundered through this merchant" rather than a known exact total. ### Extend the model @@ -295,15 +343,15 @@ See `model_setup.py` and the three motif runners for the implementation, and `ru ### Scale up / productionize -- Raise the solution limit on a real ledger. Each runner's `MAX_*_MOTIFS` is sized for the demo. On a production ledger with thousands of accounts you may want 100+ so the analyst inbox surfaces the full population of candidates. The `time_limit_sec` argument to `problem.configure` is your safety net. -- Pin `relationalai` to a known-good version for reproducible runs, and schedule the runners after each ledger refresh so candidate cases stay current for analyst triage. +- Raise the solution limit on a real ledger. Each motif's `MAX_*_MOTIFS` is sized for the demo. On a production ledger with thousands of accounts you may want 100+ so the analyst inbox surfaces the full population of candidates. The `time_limit_sec` argument to `problem.configure` is your safety net. +- Pin `relationalai` to a known-good version for reproducible runs, and give the `standard` schedule in `raiconfig.yaml` a nonzero `interval_s` so the deployed model re-solves on a timer and candidate cases stay current for analyst triage. ## References - [Altman et al., *Realistic Synthetic Financial Transactions for Anti-Money Laundering Models*, NeurIPS 2023](https://arxiv.org/abs/2306.16424). The IBM AMLworld paper. Defines the canonical eight-pattern AML topology taxonomy (fan-in, fan-out, scatter-gather, gather-scatter, cycle, bipartite, stack, random) and explicitly notes "total in equals total out" for the conservation patterns -- the property the butterfly motif here encodes. Public dataset releases at [IBM/AML-Data](https://github.com/IBM/AML-Data) under CDLA-Sharing-1.0. - [Starnini et al., *Smurf-Based Anti-Money Laundering in Time-Evolving Transaction Networks*, ECML PKDD 2021](https://link.springer.com/chapter/10.1007/978-3-030-86514-6_11). Time-window plus flow-balance signal in real-world transaction graphs (>180M transactions, >31M bank accounts). - [Pareja et al., *The Shape of Money Laundering: Subgraph Representation*, arXiv:2404.19109](https://arxiv.org/pdf/2404.19109). Subgraph-classification benchmark on Elliptic2; multi-leg / multi-account topologies. -- [FATF (Financial Action Task Force) Typologies Reports](https://www.fatf-gafi.org/en/publications/Methodsandtrends.html) -- canonical typology source for placement / layering / integration stages, including the structuring and smurf-army patterns the motif runners encode. +- [FATF (Financial Action Task Force) Typologies Reports](https://www.fatf-gafi.org/en/publications/Methodsandtrends.html) -- canonical typology source for placement / layering / integration stages, including the structuring and smurf-army patterns the motifs encode. - [FinCEN Currency Transaction Report (CTR) regulation, 31 CFR § 1010.311](https://www.ecfr.gov/current/title-31/subtitle-B/chapter-X/part-1010/subpart-C/section-1010.311) -- the $10K reporting threshold whose evasion drives the structuring typology. - [Tookitaki, *Smurfing and Structuring: AML Detection and Reporting*](https://www.tookitaki.com/compliance-hub/smurfing-structuring-aml-detection-reporting). Practitioner guidance on the KYC-tier-diverse smurf-army profile that the KYC-mix burst motif encodes. @@ -322,20 +370,20 @@ See `model_setup.py` and the three motif runners for the implementation, and `ru
- How many motifs / cohorts will each runner return? + How many motifs / cohorts will each motif return? -- Up to `MAX_*_MOTIFS` (10/5/5 by default for butterfly/smurf/burst) or however many feasible motifs exist in the data, whichever is smaller. `solve_info().num_points` reports the actual count after the solve. On the bundled data, butterfly exhausts at 2 motifs and KYC-burst at 2 cohorts; the smurf runner hits the limit (5) because pairwise-distinct-BO cohorts can swap any chosen smurf for a dup-BO decoy that still sums in-range -- the `distinct_bo_ic` is the constraint that prevents the dup-BO cohort, and seeing it bind is the point of running the smurf demo. +- Up to `MAX_*_MOTIFS` (10/5/5 by default for butterfly/smurf/burst) or however many feasible motifs exist in the data, whichever is smaller. `solve_info().num_points` reports the actual count after the solve. On the bundled data, butterfly exhausts at 2 motifs and KYC-burst at 2 cohorts; the smurf motif hits the limit (5) because pairwise-distinct-BO cohorts can swap any chosen smurf for a dup-BO decoy that still sums in-range -- the `distinct_bo_ic` is the constraint that prevents the dup-BO cohort, and seeing it bind is the point of running the smurf demo. - Solution ordering is not guaranteed across runs or solver versions; the *set* of motifs is, but solution index 0 may swap with solution index 1 between runs. Treat the `solution` column as a label, not a ranking. - To get a ranked answer (e.g. surface the largest scheme first), switch to optimisation -- `problem.maximize(sum(Transaction.is_motif * Transaction.amount_dollars))` returns the motif with the largest total laundered amount under `solution_limit=1`. For a top-K ranked list, run an iterative exclusion-cut loop (re-solve after forbidding each previous motif's edge set) or sort the enumerated motifs in post-processing.
- Import error for relationalai or model_setup + Import error for relationalai or model - Confirm your virtual environment is active: `which python` should point to `.venv`. - Reinstall dependencies: `python -m pip install .`. -- The motif runners import from `model_setup` as a sibling module. Run them from inside the template directory (so the directory is on Python's path) or `python -m motif_butterfly` from the parent. +- The reports import the `model` package from the template directory. Run `rai models deploy --path model` and `python -m queries.` from inside the template directory, so the directory is on Python's path.
@@ -363,7 +411,7 @@ See `model_setup.py` and the three motif runners for the implementation, and `ru ### Reasoner reference -- [Prescriptive reasoner](https://docs.relational.ai/) -- the `Problem` API, decision variables, integrity constraints, and multi-solution enumeration with `solution_limit`, as all three motif runners use. +- [Prescriptive reasoner](https://docs.relational.ai/) -- the `Problem` API, decision variables, integrity constraints, and multi-solution enumeration with `solution_limit`, as all three motifs use. ## Support diff --git a/money_laundering_motif_detection/model/__init__.py b/money_laundering_motif_detection/model/__init__.py new file mode 100644 index 00000000..844aef4e --- /dev/null +++ b/money_laundering_motif_detection/model/__init__.py @@ -0,0 +1,13 @@ +"""Money-laundering motif detection: one model, three motif solves. + +The shared ontology (`ontology.py`) and the three motifs (`butterfly.py`, +`kyc_burst.py`, `smurf_army.py`) form one model. `rai models deploy --path +model` deploys it, and each refresh of the deployment runs all three solves. +Each `queries/_motifs.py` prints one motif's report from the deployed +model. +""" + +from . import butterfly, kyc_burst, smurf_army +from .ontology import Account, Transaction, model + +__all__ = ["Account", "Transaction", "butterfly", "kyc_burst", "model", "smurf_army"] diff --git a/money_laundering_motif_detection/motif_butterfly.py b/money_laundering_motif_detection/model/butterfly.py similarity index 65% rename from money_laundering_motif_detection/motif_butterfly.py rename to money_laundering_motif_detection/model/butterfly.py index 2a5cf5d1..adaa5987 100644 --- a/money_laundering_motif_detection/motif_butterfly.py +++ b/money_laundering_motif_detection/model/butterfly.py @@ -14,24 +14,18 @@ single destination. Every per-leg amount stays under the FinCEN $10K currency-transaction-report threshold so no leg triggers a CTR filing. -Run: - `python motif_butterfly.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), - and prints the report from the deployed model. - -Output: - Prints the formulation, the solve-result block, and three - per-solution inspect tables: the chosen motif transactions - (one row per motif edge), the K hubs and their shared - beneficial owner, and the source / destination assignments. +Deploy: + Part of the model in `model/`, deployed with `rai models deploy --path model`; + each refresh of the deployment runs this solve. `python -m queries.butterfly_motifs` + prints its report from the deployed model. """ -import pandas as pd -from model_setup import AMOUNT_THRESHOLD_DOLLARS, DATA_DIR, create_model from relationalai.semantics import Integer, String, sum from relationalai.semantics.reasoners.prescriptive import Problem -# Runner-level parameters. +from .ontology import AMOUNT_THRESHOLD_DOLLARS, Account, Transaction, model, tx_csv + +# Motif parameters. # K = 3 is the smallest hub count that's clearly a fan-out-then-fan-in pattern # rather than a direct payment chain. K = 3 @@ -44,21 +38,15 @@ # your own ledger. MAX_BUTTERFLY_MOTIFS = 10 -model, Account, Transaction = create_model() - # Tightest big-M coefficient for the per-hub conservation IC below. The big-M # form makes that IC active when `is_hub == 1` and vacuous when `is_hub == 0`; # M needs to bound |sum_in - sum_out| over decision-selected motif edges, # which is at most the per-account total of incoming or outgoing amounts. -# Re-load transactions here to compute that bound; only the butterfly motif -# needs this post-load aggregation, so it's local rather than wired through -# `create_model()`'s return. -_tx_csv = pd.read_csv(DATA_DIR / "transactions.csv") CONSERVATION_BIG_M = ( int( max( - _tx_csv.groupby("dst_id")["amount_dollars"].sum().max(), - _tx_csv.groupby("src_id")["amount_dollars"].sum().max(), + tx_csv.groupby("dst_id")["amount_dollars"].sum().max(), + tx_csv.groupby("src_id")["amount_dollars"].sum().max(), ) ) + AMOUNT_THRESHOLD_DOLLARS @@ -70,7 +58,8 @@ Account.is_hub = model.Property(f"{Account} is hub if {Integer:is_hub}") Account.is_dest = model.Property(f"{Account} is dest if {Integer:is_dest}") -problem = Problem(model, Integer) +# The model configures three Problems, one per motif, so each has a unique name. +problem = Problem(model, Integer, name="butterfly") # Each role decision spans the full Account scope -- not narrowed via # `where=[...]` on the `solve_for(...)` call. Tighter scoping leaves the @@ -220,90 +209,3 @@ # The solve's status, published through the model's own relationship. SolveStatus = model.Relationship(f"butterfly solve status is {String:status}") model.define(SolveStatus(problem.termination_status())) - -# Deploy, then report (`rai models deploy` skips this block). -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - print("\n" + "=" * 70) - print("BUTTERFLY (scatter-gather with per-hub conservation)") - print("CSP technique: per-vertex aggregate equality over decision-selected edge subset") - print("=" * 70) - problem.display() - - # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` - # calls above means the decision-variable values are NOT written back to - # the relational `is_motif` / `is_source` / `is_hub` / `is_dest` properties. - # Without those values in the relational layer, verify() cannot re-evaluate - # ICs that reference them and prints spurious "Requirements not met" - # warnings instead of doing real verification. The inspect tables below - # surface the solver's choices directly; manual inspection against the IC - # formulas is the verification path here. - - # Status of the latest refresh's solve and the number of solutions it found. - status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) - motif_count = problem.solve_info().num_points or 0 - print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") - if status == "INFEASIBLE": - print( - "\nResult: no butterfly motifs exist under the encoded constraints. " - "The search space was exhausted with zero feasible motifs." - ) - elif status == "OPTIMAL": - print( - f"\nResult: {motif_count} butterfly motif(s) found and search exhausted " - "(no further motifs exist under the encoded constraints)." - ) - elif status == "SOLUTION_LIMIT": - print( - f"\nResult: {motif_count} butterfly motif(s) found; " - f"hit MAX_BUTTERFLY_MOTIFS={MAX_BUTTERFLY_MOTIFS}. " - "More may exist -- raise the limit to enumerate further." - ) - else: - print( - f"\nResult: search returned status={status} with {motif_count} motif(s); " - "did not finish. Raise time_limit_sec or check the README troubleshooting." - ) - - print("\nButterfly: chosen motif transactions (one row per motif edge per solution):") - sol_idx = Integer.ref() - val = Integer.ref() - model.select( - sol_idx.alias("solution"), - is_motif_var.transaction.tx_id.alias("tx_id"), - is_motif_var.transaction.src.id.alias("src_account_id"), - is_motif_var.transaction.src.name.alias("src_name"), - is_motif_var.transaction.dst.id.alias("dst_account_id"), - is_motif_var.transaction.dst.name.alias("dst_name"), - is_motif_var.transaction.amount_dollars.alias("amount"), - is_motif_var.transaction.ts_minutes.alias("ts_min"), - ).where(is_motif_var.values(sol_idx, val), val == 1).inspect() - - print("\nButterfly: chosen motif hubs per solution (with shared beneficial owner):") - model.select( - sol_idx.alias("solution"), - is_hub_var.account.id.alias("hub_id"), - is_hub_var.account.name.alias("hub_name"), - is_hub_var.account.bo_id.alias("bo_id"), - ).where(is_hub_var.values(sol_idx, val), val == 1).inspect() - - print("\nButterfly: chosen motif source and destination per solution:") - src_sol = Integer.ref() - src_val = Integer.ref() - dst_sol = Integer.ref() - dst_val = Integer.ref() - model.select( - src_sol.alias("solution"), - is_source_var.account.id.alias("source_id"), - is_source_var.account.name.alias("source_name"), - is_dest_var.account.id.alias("dest_id"), - is_dest_var.account.name.alias("dest_name"), - ).where( - is_source_var.values(src_sol, src_val), - src_val == 1, - is_dest_var.values(dst_sol, dst_val), - dst_val == 1, - src_sol == dst_sol, - ).inspect() diff --git a/money_laundering_motif_detection/motif_kyc_burst.py b/money_laundering_motif_detection/model/kyc_burst.py similarity index 62% rename from money_laundering_motif_detection/motif_kyc_burst.py rename to money_laundering_motif_detection/model/kyc_burst.py index b33a1859..280b5f4b 100644 --- a/money_laundering_motif_detection/motif_kyc_burst.py +++ b/money_laundering_motif_detection/model/kyc_burst.py @@ -18,23 +18,18 @@ mules to defeat single-tier-pattern alerts ([Tookitaki AML guidance]( https://www.tookitaki.com/compliance-hub/smurfing-structuring-aml-detection-reporting)). -Run: - `python motif_kyc_burst.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), - and prints the report from the deployed model. - -Output: - Prints the formulation, the solve-result block, and two - per-solution inspect tables: the chosen burst transactions - and the chosen burst accounts annotated with KYC tier and - jurisdiction so the retail-floor mix is visible. +Deploy: + Part of the model in `model/`, deployed with `rai models deploy --path model`; + each refresh of the deployment runs this solve. `python -m queries.kyc_burst_motifs` + prints its report from the deployed model. """ -from model_setup import AMOUNT_THRESHOLD_DOLLARS, create_model from relationalai.semantics import Integer, String, sum from relationalai.semantics.reasoners.prescriptive import Problem -# Runner-level parameters. +from .ontology import AMOUNT_THRESHOLD_DOLLARS, Account, Transaction, model + +# Motif parameters. # Target destination receiving the coordinated burst. Bundled data plants # Continental Settlements LLC (id 51). BURST_TARGET_DESTINATION_ID = 51 @@ -51,13 +46,12 @@ # Solver solution-limit: enumerate up to this many distinct burst cohorts. MAX_BURST_MOTIFS = 5 -model, Account, Transaction = create_model() - # Decision-valued properties for the KYC-burst motif. Account.is_burst = model.Property(f"{Account} is burst if {Integer:is_burst}") Transaction.is_burst_tx = model.Property(f"{Transaction} is burst tx if {Integer:is_burst_tx}") -problem = Problem(model, Integer) +# The model configures three Problems, one per motif, so each has a unique name. +problem = Problem(model, Integer, name="kyc_burst") # Full-Account scope for is_burst and full-Transaction scope for is_burst_tx # -- not narrowed via `where=[...]` on the `solve_for(...)` call. Tight scoping @@ -153,74 +147,3 @@ # The solve's status, published through the model's own relationship. SolveStatus = model.Relationship(f"kyc burst solve status is {String:status}") model.define(SolveStatus(problem.termination_status())) - -# Deploy, then report (`rai models deploy` skips this block). -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - print("\n" + "=" * 70) - print(f"KYC-MIX BURST (fan-in to account id {BURST_TARGET_DESTINATION_ID})") - print("CSP technique: cardinality over chosen subset") - print("=" * 70) - problem.display() - - # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` - # calls above means the decision-variable values are NOT written back to - # the relational `is_burst` / `is_burst_tx` properties. Without those values - # in the relational layer, verify() cannot re-evaluate ICs that reference - # them and prints spurious "Requirements not met" warnings instead of - # doing real verification. The inspect tables below surface the solver's - # choices directly; manual inspection against the IC formulas is the - # verification path here. - - # Status of the latest refresh's solve and the number of solutions it found. - status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) - motif_count = problem.solve_info().num_points or 0 - print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") - if status == "INFEASIBLE": - print( - "\nResult: no KYC-mix burst cohort exists under the encoded constraints " - f"for target id {BURST_TARGET_DESTINATION_ID}. The search space was " - "exhausted with zero feasible cohorts." - ) - elif status == "OPTIMAL": - print( - f"\nResult: {motif_count} KYC-mix burst cohort(s) found and search exhausted " - "(no further cohorts exist under the encoded constraints)." - ) - elif status == "SOLUTION_LIMIT": - print( - f"\nResult: {motif_count} KYC-mix burst cohort(s) found; " - f"hit MAX_BURST_MOTIFS={MAX_BURST_MOTIFS}. " - "More may exist -- raise the limit to enumerate further." - ) - else: - print( - f"\nResult: search returned status={status} with {motif_count} cohort(s); " - "did not finish. Raise time_limit_sec or check the README troubleshooting." - ) - - print("\nKYC-burst: chosen burst-tx per solution:") - sol_idx = Integer.ref() - val = Integer.ref() - model.select( - sol_idx.alias("solution"), - is_burst_tx_var.transaction.tx_id.alias("tx_id"), - is_burst_tx_var.transaction.src.id.alias("src_account_id"), - is_burst_tx_var.transaction.src.name.alias("src_name"), - is_burst_tx_var.transaction.src.kyc_tier.alias("src_kyc_tier"), - is_burst_tx_var.transaction.dst.name.alias("dst_name"), - is_burst_tx_var.transaction.amount_dollars.alias("amount"), - is_burst_tx_var.transaction.ts_minutes.alias("ts_min"), - ).where(is_burst_tx_var.values(sol_idx, val), val == 1).inspect() - - print("\nKYC-burst: chosen burst accounts per solution (with KYC tier and jurisdiction):") - model.select( - sol_idx.alias("solution"), - is_burst_var.account.id.alias("burst_id"), - is_burst_var.account.name.alias("burst_name"), - is_burst_var.account.kyc_tier.alias("kyc_tier"), - is_burst_var.account.jurisdiction.alias("jurisdiction"), - is_burst_var.account.bo_id.alias("bo_id"), - ).where(is_burst_var.values(sol_idx, val), val == 1).inspect() diff --git a/money_laundering_motif_detection/model/ontology.py b/money_laundering_motif_detection/model/ontology.py new file mode 100644 index 00000000..6e345b80 --- /dev/null +++ b/money_laundering_motif_detection/model/ontology.py @@ -0,0 +1,54 @@ +"""Shared ontology for the three money-laundering motifs. + +This module owns the model, the ontology, and the data load that all three +motifs share -- accounts, transactions, beneficial-owner clusters, KYC tier, +jurisdiction. Each motif module (`butterfly.py`, `smurf_army.py`, +`kyc_burst.py`) imports the model from here and adds its own decision +properties, constraints, and configured Problem. +""" + +from pathlib import Path + +import pandas as pd +from relationalai.semantics import Integer, Model, String + +DATA_DIR = Path(__file__).resolve().parents[1] / "data" + +# Currency-transaction-report (CTR) threshold from FinCEN 31 CFR 1010.311. +# Layering schemes split deposits into amounts that stay under this line so +# no single transaction triggers a CTR filing. +AMOUNT_THRESHOLD_DOLLARS = 10_000 + +accounts_csv = pd.read_csv(DATA_DIR / "accounts.csv") +tx_csv = pd.read_csv(DATA_DIR / "transactions.csv") + +model = Model("money_laundering_motif_detection") + +# Concept: bank account. +Account = model.Concept("Account", identify_by={"id": Integer}) +Account.name = model.Property(f"{Account} has {String:name}") +Account.bo_id = model.Property(f"{Account} owned by {Integer:bo_id}") +Account.kyc_tier = model.Property(f"{Account} has {String:kyc_tier}") +Account.jurisdiction = model.Property(f"{Account} from {String:jurisdiction}") +model.define(Account.new(model.data(accounts_csv).to_schema())) + +# Concept: transaction (a directed edge in the graph). +Transaction = model.Concept("Transaction", identify_by={"tx_id": Integer}) +Transaction.src = model.Property(f"{Transaction} from {Account:src}") +Transaction.dst = model.Property(f"{Transaction} to {Account:dst}") +Transaction.amount_dollars = model.Property(f"{Transaction} has {Integer:amount_dollars}") +Transaction.ts_minutes = model.Property(f"{Transaction} occurs at {Integer:ts_minutes}") +tx_data = model.data(tx_csv) +model.define( + t := Transaction.new(tx_id=tx_data.tx_id), + t.amount_dollars(tx_data.amount_dollars), + t.ts_minutes(tx_data.ts_minutes), +) +model.define(Transaction.src(Account)).where( + Transaction.tx_id(tx_data.tx_id), + Account.id(tx_data.src_id), +) +model.define(Transaction.dst(Account)).where( + Transaction.tx_id(tx_data.tx_id), + Account.id(tx_data.dst_id), +) diff --git a/money_laundering_motif_detection/motif_smurf_army.py b/money_laundering_motif_detection/model/smurf_army.py similarity index 66% rename from money_laundering_motif_detection/motif_smurf_army.py rename to money_laundering_motif_detection/model/smurf_army.py index e4024a0c..332fb198 100644 --- a/money_laundering_motif_detection/motif_smurf_army.py +++ b/money_laundering_motif_detection/model/smurf_army.py @@ -21,23 +21,18 @@ a known total ($SMURF_TARGET_DOLLARS) into under-threshold deposits all arriving at the target merchant within a short coordinated burst. -Run: - `python motif_smurf_army.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), - and prints the report from the deployed model. - -Output: - Prints the formulation, the solve-result block, and two - per-solution inspect tables: the chosen smurf transactions - and the chosen smurf accounts annotated with their distinct - beneficial owners and jurisdictions. +Deploy: + Part of the model in `model/`, deployed with `rai models deploy --path model`; + each refresh of the deployment runs this solve. `python -m queries.smurf_army_motifs` + prints its report from the deployed model. """ -from model_setup import AMOUNT_THRESHOLD_DOLLARS, create_model from relationalai.semantics import Integer, String, sum from relationalai.semantics.reasoners.prescriptive import Problem -# Runner-level parameters. +from .ontology import AMOUNT_THRESHOLD_DOLLARS, Account, Transaction, model + +# Motif parameters. # The target merchant the smurfs deposit into. In production this could be a # known launder destination from a watchlist; for the demo it's the bundled # data's planted target ("Pacific Receivables Inc", id 41). @@ -55,13 +50,12 @@ # Solver solution-limit: enumerate up to this many distinct smurf cohorts. MAX_SMURF_MOTIFS = 5 -model, Account, Transaction = create_model() - # Decision-valued properties for the smurf-army motif. Account.is_smurf = model.Property(f"{Account} is smurf if {Integer:is_smurf}") Transaction.is_smurf_tx = model.Property(f"{Transaction} is smurf tx if {Integer:is_smurf_tx}") -problem = Problem(model, Integer) +# The model configures three Problems, one per motif, so each has a unique name. +problem = Problem(model, Integer, name="smurf_army") # Full-Account scope for is_smurf and full-Transaction scope for is_smurf_tx # -- not narrowed via `where=[...]` on the `solve_for(...)` call. Tight scoping @@ -178,74 +172,3 @@ # The solve's status, published through the model's own relationship. SolveStatus = model.Relationship(f"smurf army solve status is {String:status}") model.define(SolveStatus(problem.termination_status())) - -# Deploy, then report (`rai models deploy` skips this block). -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - print("\n" + "=" * 70) - print(f"SMURF ARMY (fan-in to account id {SMURF_TARGET_DESTINATION_ID})") - print("CSP technique: pairwise distinctness + sum-target over chosen subset") - print("=" * 70) - problem.display() - - # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` - # calls above means the decision-variable values are NOT written back to - # the relational `is_smurf` / `is_smurf_tx` properties. Without those values - # in the relational layer, verify() cannot re-evaluate ICs that reference - # them and prints spurious "Requirements not met" warnings instead of - # doing real verification. The inspect tables below surface the solver's - # choices directly; manual inspection against the IC formulas is the - # verification path here. - - # Status of the latest refresh's solve and the number of solutions it found. - status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) - motif_count = problem.solve_info().num_points or 0 - print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") - if status == "INFEASIBLE": - print( - "\nResult: no smurf-army cohort exists under the encoded constraints " - f"for target id {SMURF_TARGET_DESTINATION_ID}. The search space was " - "exhausted with zero feasible cohorts." - ) - elif status == "OPTIMAL": - print( - f"\nResult: {motif_count} smurf-army cohort(s) found and search exhausted " - "(no further cohorts exist under the encoded constraints)." - ) - elif status == "SOLUTION_LIMIT": - print( - f"\nResult: {motif_count} smurf-army cohort(s) found; " - f"hit MAX_SMURF_MOTIFS={MAX_SMURF_MOTIFS}. " - "More may exist -- raise the limit to enumerate further." - ) - else: - print( - f"\nResult: search returned status={status} with {motif_count} cohort(s); " - "did not finish. Raise time_limit_sec or check the README troubleshooting." - ) - - print("\nSmurf army: chosen smurf-tx per solution:") - sol_idx = Integer.ref() - val = Integer.ref() - model.select( - sol_idx.alias("solution"), - is_smurf_tx_var.transaction.tx_id.alias("tx_id"), - is_smurf_tx_var.transaction.src.id.alias("src_account_id"), - is_smurf_tx_var.transaction.src.name.alias("src_name"), - is_smurf_tx_var.transaction.src.bo_id.alias("src_bo_id"), - is_smurf_tx_var.transaction.dst.name.alias("dst_name"), - is_smurf_tx_var.transaction.amount_dollars.alias("amount"), - is_smurf_tx_var.transaction.ts_minutes.alias("ts_min"), - ).where(is_smurf_tx_var.values(sol_idx, val), val == 1).inspect() - - print("\nSmurf army: chosen smurf accounts per solution (with distinct BOs and jurisdictions):") - model.select( - sol_idx.alias("solution"), - is_smurf_var.account.id.alias("smurf_id"), - is_smurf_var.account.name.alias("smurf_name"), - is_smurf_var.account.bo_id.alias("bo_id"), - is_smurf_var.account.kyc_tier.alias("kyc_tier"), - is_smurf_var.account.jurisdiction.alias("jurisdiction"), - ).where(is_smurf_var.values(sol_idx, val), val == 1).inspect() diff --git a/money_laundering_motif_detection/model_setup.py b/money_laundering_motif_detection/model_setup.py deleted file mode 100644 index e5098c67..00000000 --- a/money_laundering_motif_detection/model_setup.py +++ /dev/null @@ -1,69 +0,0 @@ -"""Shared model setup for the three money-laundering motif-detection runners. - -This module owns the ontology and data load that all three motifs share -- -accounts, transactions, beneficial-owner clusters, KYC tier, jurisdiction. -Each motif runner (`motif_butterfly.py`, `motif_smurf_army.py`, -`motif_kyc_burst.py`) imports `create_model()` from here and then adds its -own decision properties + constraints + solve. -""" - -from pathlib import Path - -import pandas as pd -from relationalai.semantics import Integer, Model, String - -DATA_DIR = Path(__file__).parent / "data" - -# Currency-transaction-report (CTR) threshold from FinCEN 31 CFR 1010.311. -# Layering schemes split deposits into amounts that stay under this line so -# no single transaction triggers a CTR filing. -AMOUNT_THRESHOLD_DOLLARS = 10_000 - - -def create_model(): - """Build the shared semantic model: accounts and transactions. - - Returns - ------- - model : Model - The Semantics model container. - Account : Concept - Account concept with name, bo_id, kyc_tier, jurisdiction properties. - Transaction : Concept - Transaction concept with src, dst, amount_dollars, ts_minutes. - """ - accounts_csv = pd.read_csv(DATA_DIR / "accounts.csv") - tx_csv = pd.read_csv(DATA_DIR / "transactions.csv") - - model = Model("money_laundering_motif_detection") - - # Concept: bank account. - Account = model.Concept("Account", identify_by={"id": Integer}) - Account.name = model.Property(f"{Account} has {String:name}") - Account.bo_id = model.Property(f"{Account} owned by {Integer:bo_id}") - Account.kyc_tier = model.Property(f"{Account} has {String:kyc_tier}") - Account.jurisdiction = model.Property(f"{Account} from {String:jurisdiction}") - model.define(Account.new(model.data(accounts_csv).to_schema())) - - # Concept: transaction (a directed edge in the graph). - Transaction = model.Concept("Transaction", identify_by={"tx_id": Integer}) - Transaction.src = model.Property(f"{Transaction} from {Account:src}") - Transaction.dst = model.Property(f"{Transaction} to {Account:dst}") - Transaction.amount_dollars = model.Property(f"{Transaction} has {Integer:amount_dollars}") - Transaction.ts_minutes = model.Property(f"{Transaction} occurs at {Integer:ts_minutes}") - tx_data = model.data(tx_csv) - model.define( - t := Transaction.new(tx_id=tx_data.tx_id), - t.amount_dollars(tx_data.amount_dollars), - t.ts_minutes(tx_data.ts_minutes), - ) - model.define(Transaction.src(Account)).where( - Transaction.tx_id(tx_data.tx_id), - Account.id(tx_data.src_id), - ) - model.define(Transaction.dst(Account)).where( - Transaction.tx_id(tx_data.tx_id), - Account.id(tx_data.dst_id), - ) - - return model, Account, Transaction diff --git a/money_laundering_motif_detection/queries/butterfly_motifs.py b/money_laundering_motif_detection/queries/butterfly_motifs.py new file mode 100644 index 00000000..e37a2b2b --- /dev/null +++ b/money_laundering_motif_detection/queries/butterfly_motifs.py @@ -0,0 +1,100 @@ +"""Report from the deployed model: butterfly (scatter-gather) motifs. + +Run after `rai models deploy --path model`: + python -m queries.butterfly_motifs +""" + +from model import model +from model.butterfly import ( + MAX_BUTTERFLY_MOTIFS, + SolveStatus, + is_dest_var, + is_hub_var, + is_motif_var, + is_source_var, + problem, +) +from relationalai.semantics import Integer + +if __name__ == "__main__": + print("\n" + "=" * 70) + print("BUTTERFLY (scatter-gather with per-hub conservation)") + print("CSP technique: per-vertex aggregate equality over decision-selected edge subset") + print("=" * 70) + problem.display() + + # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` + # calls in `model/butterfly.py` means the decision-variable values are NOT written back to + # the relational `is_motif` / `is_source` / `is_hub` / `is_dest` properties. + # Without those values in the relational layer, verify() cannot re-evaluate + # ICs that reference them and prints spurious "Requirements not met" + # warnings instead of doing real verification. The inspect tables below + # surface the solver's choices directly; manual inspection against the IC + # formulas is the verification path here. + + # Status of the latest refresh's solve and the number of solutions it found. + status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) + motif_count = problem.solve_info().num_points or 0 + print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") + if status == "INFEASIBLE": + print( + "\nResult: no butterfly motifs exist under the encoded constraints. " + "The search space was exhausted with zero feasible motifs." + ) + elif status == "OPTIMAL": + print( + f"\nResult: {motif_count} butterfly motif(s) found and search exhausted " + "(no further motifs exist under the encoded constraints)." + ) + elif status == "SOLUTION_LIMIT": + print( + f"\nResult: {motif_count} butterfly motif(s) found; " + f"hit MAX_BUTTERFLY_MOTIFS={MAX_BUTTERFLY_MOTIFS}. " + "More may exist -- raise the limit to enumerate further." + ) + else: + print( + f"\nResult: search returned status={status} with {motif_count} motif(s); " + "did not finish. Raise time_limit_sec or check the README troubleshooting." + ) + + print("\nButterfly: chosen motif transactions (one row per motif edge per solution):") + sol_idx = Integer.ref() + val = Integer.ref() + model.select( + sol_idx.alias("solution"), + is_motif_var.transaction.tx_id.alias("tx_id"), + is_motif_var.transaction.src.id.alias("src_account_id"), + is_motif_var.transaction.src.name.alias("src_name"), + is_motif_var.transaction.dst.id.alias("dst_account_id"), + is_motif_var.transaction.dst.name.alias("dst_name"), + is_motif_var.transaction.amount_dollars.alias("amount"), + is_motif_var.transaction.ts_minutes.alias("ts_min"), + ).where(is_motif_var.values(sol_idx, val), val == 1).inspect() + + print("\nButterfly: chosen motif hubs per solution (with shared beneficial owner):") + model.select( + sol_idx.alias("solution"), + is_hub_var.account.id.alias("hub_id"), + is_hub_var.account.name.alias("hub_name"), + is_hub_var.account.bo_id.alias("bo_id"), + ).where(is_hub_var.values(sol_idx, val), val == 1).inspect() + + print("\nButterfly: chosen motif source and destination per solution:") + src_sol = Integer.ref() + src_val = Integer.ref() + dst_sol = Integer.ref() + dst_val = Integer.ref() + model.select( + src_sol.alias("solution"), + is_source_var.account.id.alias("source_id"), + is_source_var.account.name.alias("source_name"), + is_dest_var.account.id.alias("dest_id"), + is_dest_var.account.name.alias("dest_name"), + ).where( + is_source_var.values(src_sol, src_val), + src_val == 1, + is_dest_var.values(dst_sol, dst_val), + dst_val == 1, + src_sol == dst_sol, + ).inspect() diff --git a/money_laundering_motif_detection/queries/kyc_burst_motifs.py b/money_laundering_motif_detection/queries/kyc_burst_motifs.py new file mode 100644 index 00000000..6358964e --- /dev/null +++ b/money_laundering_motif_detection/queries/kyc_burst_motifs.py @@ -0,0 +1,83 @@ +"""Report from the deployed model: KYC-mix burst cohorts. + +Run after `rai models deploy --path model`: + python -m queries.kyc_burst_motifs +""" + +from model import model +from model.kyc_burst import ( + BURST_TARGET_DESTINATION_ID, + MAX_BURST_MOTIFS, + SolveStatus, + is_burst_tx_var, + is_burst_var, + problem, +) +from relationalai.semantics import Integer + +if __name__ == "__main__": + print("\n" + "=" * 70) + print(f"KYC-MIX BURST (fan-in to account id {BURST_TARGET_DESTINATION_ID})") + print("CSP technique: cardinality over chosen subset") + print("=" * 70) + problem.display() + + # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` + # calls in `model/kyc_burst.py` means the decision-variable values are NOT written back to + # the relational `is_burst` / `is_burst_tx` properties. Without those values + # in the relational layer, verify() cannot re-evaluate ICs that reference + # them and prints spurious "Requirements not met" warnings instead of + # doing real verification. The inspect tables below surface the solver's + # choices directly; manual inspection against the IC formulas is the + # verification path here. + + # Status of the latest refresh's solve and the number of solutions it found. + status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) + motif_count = problem.solve_info().num_points or 0 + print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") + if status == "INFEASIBLE": + print( + "\nResult: no KYC-mix burst cohort exists under the encoded constraints " + f"for target id {BURST_TARGET_DESTINATION_ID}. The search space was " + "exhausted with zero feasible cohorts." + ) + elif status == "OPTIMAL": + print( + f"\nResult: {motif_count} KYC-mix burst cohort(s) found and search exhausted " + "(no further cohorts exist under the encoded constraints)." + ) + elif status == "SOLUTION_LIMIT": + print( + f"\nResult: {motif_count} KYC-mix burst cohort(s) found; " + f"hit MAX_BURST_MOTIFS={MAX_BURST_MOTIFS}. " + "More may exist -- raise the limit to enumerate further." + ) + else: + print( + f"\nResult: search returned status={status} with {motif_count} cohort(s); " + "did not finish. Raise time_limit_sec or check the README troubleshooting." + ) + + print("\nKYC-burst: chosen burst-tx per solution:") + sol_idx = Integer.ref() + val = Integer.ref() + model.select( + sol_idx.alias("solution"), + is_burst_tx_var.transaction.tx_id.alias("tx_id"), + is_burst_tx_var.transaction.src.id.alias("src_account_id"), + is_burst_tx_var.transaction.src.name.alias("src_name"), + is_burst_tx_var.transaction.src.kyc_tier.alias("src_kyc_tier"), + is_burst_tx_var.transaction.dst.name.alias("dst_name"), + is_burst_tx_var.transaction.amount_dollars.alias("amount"), + is_burst_tx_var.transaction.ts_minutes.alias("ts_min"), + ).where(is_burst_tx_var.values(sol_idx, val), val == 1).inspect() + + print("\nKYC-burst: chosen burst accounts per solution (with KYC tier and jurisdiction):") + model.select( + sol_idx.alias("solution"), + is_burst_var.account.id.alias("burst_id"), + is_burst_var.account.name.alias("burst_name"), + is_burst_var.account.kyc_tier.alias("kyc_tier"), + is_burst_var.account.jurisdiction.alias("jurisdiction"), + is_burst_var.account.bo_id.alias("bo_id"), + ).where(is_burst_var.values(sol_idx, val), val == 1).inspect() diff --git a/money_laundering_motif_detection/queries/smurf_army_motifs.py b/money_laundering_motif_detection/queries/smurf_army_motifs.py new file mode 100644 index 00000000..8b6522cf --- /dev/null +++ b/money_laundering_motif_detection/queries/smurf_army_motifs.py @@ -0,0 +1,83 @@ +"""Report from the deployed model: smurf-army cohorts. + +Run after `rai models deploy --path model`: + python -m queries.smurf_army_motifs +""" + +from model import model +from model.smurf_army import ( + MAX_SMURF_MOTIFS, + SMURF_TARGET_DESTINATION_ID, + SolveStatus, + is_smurf_tx_var, + is_smurf_var, + problem, +) +from relationalai.semantics import Integer + +if __name__ == "__main__": + print("\n" + "=" * 70) + print(f"SMURF ARMY (fan-in to account id {SMURF_TARGET_DESTINATION_ID})") + print("CSP technique: pairwise distinctness + sum-target over chosen subset") + print("=" * 70) + problem.display() + + # No `problem.verify(...)` call: `populate=False` on the `solve_for(...)` + # calls in `model/smurf_army.py` means the decision-variable values are NOT written back to + # the relational `is_smurf` / `is_smurf_tx` properties. Without those values + # in the relational layer, verify() cannot re-evaluate ICs that reference + # them and prints spurious "Requirements not met" warnings instead of + # doing real verification. The inspect tables below surface the solver's + # choices directly; manual inspection against the IC formulas is the + # verification path here. + + # Status of the latest refresh's solve and the number of solutions it found. + status = ", ".join(model.select(SolveStatus).to_df().iloc[:, 0]) + motif_count = problem.solve_info().num_points or 0 + print(f"\nSolve result:\n• status: {status}\n• num_points: {motif_count}\n") + if status == "INFEASIBLE": + print( + "\nResult: no smurf-army cohort exists under the encoded constraints " + f"for target id {SMURF_TARGET_DESTINATION_ID}. The search space was " + "exhausted with zero feasible cohorts." + ) + elif status == "OPTIMAL": + print( + f"\nResult: {motif_count} smurf-army cohort(s) found and search exhausted " + "(no further cohorts exist under the encoded constraints)." + ) + elif status == "SOLUTION_LIMIT": + print( + f"\nResult: {motif_count} smurf-army cohort(s) found; " + f"hit MAX_SMURF_MOTIFS={MAX_SMURF_MOTIFS}. " + "More may exist -- raise the limit to enumerate further." + ) + else: + print( + f"\nResult: search returned status={status} with {motif_count} cohort(s); " + "did not finish. Raise time_limit_sec or check the README troubleshooting." + ) + + print("\nSmurf army: chosen smurf-tx per solution:") + sol_idx = Integer.ref() + val = Integer.ref() + model.select( + sol_idx.alias("solution"), + is_smurf_tx_var.transaction.tx_id.alias("tx_id"), + is_smurf_tx_var.transaction.src.id.alias("src_account_id"), + is_smurf_tx_var.transaction.src.name.alias("src_name"), + is_smurf_tx_var.transaction.src.bo_id.alias("src_bo_id"), + is_smurf_tx_var.transaction.dst.name.alias("dst_name"), + is_smurf_tx_var.transaction.amount_dollars.alias("amount"), + is_smurf_tx_var.transaction.ts_minutes.alias("ts_min"), + ).where(is_smurf_tx_var.values(sol_idx, val), val == 1).inspect() + + print("\nSmurf army: chosen smurf accounts per solution (with distinct BOs and jurisdictions):") + model.select( + sol_idx.alias("solution"), + is_smurf_var.account.id.alias("smurf_id"), + is_smurf_var.account.name.alias("smurf_name"), + is_smurf_var.account.bo_id.alias("bo_id"), + is_smurf_var.account.kyc_tier.alias("kyc_tier"), + is_smurf_var.account.jurisdiction.alias("jurisdiction"), + ).where(is_smurf_var.values(sol_idx, val), val == 1).inspect() diff --git a/money_laundering_motif_detection/runbook.md b/money_laundering_motif_detection/runbook.md index a42fc378..bc539dbb 100644 --- a/money_laundering_motif_detection/runbook.md +++ b/money_laundering_motif_detection/runbook.md @@ -19,7 +19,7 @@ shared ledger: 90 accounts (beneficial owner, KYC tier, jurisdiction) · 138 tra • each motif is a joint constraint a rules/path query alone can't express; the planted decoys are correctly excluded ``` -Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` skill (named at the start of each prompt). They run in order in a single session — the three motif detectors all read the shared `Account`/`Transaction` ledger the build step created. (In the template these are three runner scripts — `motif_butterfly.py`, `motif_smurf_army.py`, `motif_kyc_burst.py` — sharing `model_setup.py`.) +Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` skill (named at the start of each prompt). They run in order in a single session — the three motif detectors all read the shared `Account`/`Transaction` ledger the build step created. (In the template these are three motif modules — `model/butterfly.py`, `model/smurf_army.py`, `model/kyc_burst.py` — sharing `model/ontology.py` in one deployed model, with one report each in `queries/`.) --- @@ -51,8 +51,8 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Find KYC-mix bursts: exactly 5 accounts that each send a sub-$10,000 transfer into the same target within a 60-minute window, where at least 4 of the 5 are retail-tier accounts. Enumerate the cohorts. -**Response:** The detector returns KYC-burst cohorts — 5 sub-$10k transfers into one flagged target within the window, at least 4 from retail-tier accounts. The **at-least-4-retail cardinality over the chosen subset** is the joint condition a per-account rule can't capture. How many cohorts surface depends on how tightly the target and window are scoped (and the solution-limit cap): pinned to the planted target the shipped script surfaces a small set, while a looser target scope admits many more — so treat this motif as "find the bursts at this target," not a fixed count. +**Response:** The detector returns KYC-burst cohorts — 5 sub-$10k transfers into one flagged target within the window, at least 4 from retail-tier accounts. The **at-least-4-retail cardinality over the chosen subset** is the joint condition a per-account rule can't capture. How many cohorts surface depends on how tightly the target and window are scoped (and the solution-limit cap): pinned to the planted target the shipped model surfaces a small set, while a looser target scope admits many more — so treat this motif as "find the bursts at this target," not a fixed count. ## Data -Bundled CSVs in `data/`: 90 accounts, 138 transactions (deterministically generated by `data/generate.py`, whose output ships with the template). The motif parameters (hub count, target sum, windows, KYC floor, solution limits) live in the runner scripts. Full models in `model_setup.py` plus the three `motif_*.py` runners. +Bundled CSVs in `data/`: 90 accounts, 138 transactions (deterministically generated by `data/generate.py`, whose output ships with the template). The motif parameters (hub count, target sum, windows, KYC floor, solution limits) live in the motif modules. Full model in `model/`: `ontology.py` plus the three motif modules. From 1cdf2f8a2c28646920116b409735def40eec1c96 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:18:46 -0800 Subject: [PATCH 048/162] money_laundering_motif_detection: keep placeholders out of Quickstart prose The docs page rejects angle brackets in Quickstart text, so the deploy step names the public schema by its config key and the refresh call moves to the query step's closing SQL block. --- money_laundering_motif_detection/README.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/money_laundering_motif_detection/README.md b/money_laundering_motif_detection/README.md index adadeac8..2df71215 100644 --- a/money_laundering_motif_detection/README.md +++ b/money_laundering_motif_detection/README.md @@ -180,9 +180,9 @@ Run these steps from a terminal: rai models deploy --path model ``` - The model holds all three motifs. The deploy creates the public schema `.MONEY_LAUNDERING_MOTIF_DETECTION` if it doesn't exist and waits for the first refresh, which runs the three motif solves. Without the `deployment` section, it stops with an error naming the missing setting. + The model holds all three motifs. The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the first refresh, which runs the three motif solves. - Deploying again without changing the model redeploys nothing and does not solve again; the queries report the latest refresh. A change to a motif module or to its `problem.configure(...)` call redeploys, and the redeploy's refresh solves all three motifs. To solve again without a change, refresh the deployed model in Snowflake with `CALL .MONEY_LAUNDERING_MOTIF_DETECTION_META._INTERNAL_REFRESH_MODEL('standard', NULL);`, then run the queries again. + Deploying again without changing the model applies no updates and does not solve again; the queries report the latest refresh. A change to a motif module or to its `problem.configure(...)` call redeploys, and the redeploy's refresh solves all three motifs. 6. **Query the results** @@ -206,7 +206,7 @@ Run these steps from a terminal: 11 1 12 9 HubAccB3 10 OffshoreLLC 7780 44 ``` - The butterfly search finds the two planted scatter-gathers (cluster `100` -> `WireRecipientCorp`, cluster `200` -> `OffshoreLLC`) and exhausts the search. + The butterfly search finds the two planted scatter-gathers (cluster `100` sending to `WireRecipientCorp`, cluster `200` sending to `OffshoreLLC`) and exhausts the search. ```bash python -m queries.smurf_army_motifs @@ -257,6 +257,12 @@ Run these steps from a terminal: The KYC-mix burst search finds two valid 5-account cohorts that satisfy the retail floor (4 retail, all within 60 min): the planted cohort with Brookline Industries plus an alternative-business swap (Newland Capital Inc). + To solve again without changing the model, refresh the deployed model, then run the queries again: + + ```sql + CALL .MONEY_LAUNDERING_MOTIF_DETECTION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + 7. **Tear down the deployment** Preview what the teardown removes, then remove it: From e312e890be89bd35aaaeb837f633514650e6acc1 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:28:51 -0800 Subject: [PATCH 049/162] ad_spend_allocation: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/spend_allocation.py (`python -m queries.spend_allocation`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- ad_spend_allocation/README.md | 142 ++++++++++++------ .../__init__.py} | 47 +----- .../queries/spend_allocation.py | 40 +++++ ad_spend_allocation/runbook.md | 2 +- 4 files changed, 139 insertions(+), 92 deletions(-) rename ad_spend_allocation/{ad_spend_allocation.py => model/__init__.py} (80%) create mode 100644 ad_spend_allocation/queries/spend_allocation.py diff --git a/ad_spend_allocation/README.md b/ad_spend_allocation/README.md index 0bd998aa..4af7ad83 100644 --- a/ad_spend_allocation/README.md +++ b/ad_spend_allocation/README.md @@ -40,7 +40,8 @@ This template also answers the follow-up question every planner asks: if we had ## What's included - **Model**: a single ontology with `Channel`, `Campaign`, `Effectiveness`, `Scenario`, and an `Allocation` decision concept — plus the prescriptive formulation (decision variables, constraints, objective) that runs on it. -- **Runner**: `ad_spend_allocation.py` — one Python script that loads the CSVs, builds the model, deploys it to your Snowflake account, where each refresh solves all scenarios at once, and prints the allocation table from the deployed model. +- **Deployment**: `model/__init__.py` loads the CSVs, builds the model, and configures the solve; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh solves all scenarios at once. +- **Query**: `queries/spend_allocation.py` prints the allocation table from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: 5 channels with spend bounds, 3 campaigns with budgets, and the 15 channel-campaign conversion rates. See *Sample data* below. - **Outputs**: solver termination status, objective value, and a per-scenario table of non-trivial spend allocations printed to stdout; the spend and funding decisions are also written back to the ontology as queryable properties. @@ -50,7 +51,7 @@ This template also answers the follow-up question every planner asks: if we had ### Access - A Snowflake account with the RelationalAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -60,18 +61,40 @@ This template also answers the follow-up question every planner asks: if we had ## Quickstart -1. Download the template and extract it: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .AD_SPEND_ALLOCATION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/ad_spend_allocation.zip), unzip it, and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/ad_spend_allocation.zip unzip ad_spend_allocation.zip cd ad_spend_allocation ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. Create a virtual environment and activate it: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv @@ -79,13 +102,20 @@ This template also answers the follow-up question every planner asks: if we had python -m pip install --upgrade pip ``` -3. Install dependencies: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -104,52 +134,61 @@ This template also answers the follow-up question every planner asks: if we had pat_user: pat_warehouse: ``` - Without the `deployment` section, `python ad_spend_allocation.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .AD_SPEND_ALLOCATION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .AD_SPEND_ALLOCATION_META TO APPLICATION ; +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: + + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.AD_SPEND_ALLOCATION` if it doesn't exist. -5. Run: + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + ```bash - python ad_spend_allocation.py + python -m queries.spend_allocation ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```text + Solve result: + • status: OPTIMAL + • objective: 9740.0 + + Spend allocation per scenario: + scenario channel campaign spend + 0 budget_35k Email Brand_Awareness 2000.0 + 1 budget_35k Email Seasonal_Sale 2000.0 + 2 budget_35k Search Product_Launch 10000.0 + 3 budget_35k Search Seasonal_Sale 8000.0 + 4 budget_35k Social Brand_Awareness 3000.0 + 5 budget_35k Video Product_Launch 10000.0 + ... + ``` + + The formulation printed above the solve result is trimmed here, and so are the budget_45k and budget_55k scenarios that the full report prints. All three budgets invest heavily in Search and Video (highest ROI channels). The $35K budget activates 6 channel-campaign pairs. The $45K and $55K budgets add Search Brand_Awareness ($5K) and increase Social Brand_Awareness to $8K — diminishing returns mean the extra $10K from $45K to $55K produces no new activations. The report checks the solve's termination status and stops unless it is OPTIMAL. See `runbook.md` for the full printout and a step-by-step walkthrough. + + To solve again without changing the model, refresh the deployed model, then run the query again: + ```sql CALL .AD_SPEND_ALLOCATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output. A per-scenario allocation table confirms a successful run (trimmed to the first budget level here; the full run also prints the $45K and $55K scenarios): +7. **Tear down the deployment** - ```text - Spend allocation per scenario: - scenario channel campaign spend - budget_35k Email Brand_Awareness 2000.0 - budget_35k Email Seasonal_Sale 2000.0 - budget_35k Search Product_Launch 10000.0 - budget_35k Search Seasonal_Sale 8000.0 - budget_35k Social Brand_Awareness 3000.0 - budget_35k Video Product_Launch 10000.0 + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - All three budgets invest heavily in Search and Video (highest ROI channels). The $35K budget activates 6 channel-campaign pairs. The $45K and $55K budgets add Search Brand_Awareness ($5K) and increase Social Brand_Awareness to $8K — diminishing returns mean the extra $10K from $45K to $55K produces no new activations. See `runbook.md` for the full printout and a step-by-step walkthrough. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -157,7 +196,10 @@ The tree below shows the top-level layout: ```text ad_spend_allocation/ -├── ad_spend_allocation.py # Main script (ontology, formulation, single-solve scenario sweep) +├── model/ +│ └── __init__.py # the deployed model (ontology, formulation, scenario sweep) +├── queries/ +│ └── spend_allocation.py # report from the deployed model ├── data/ │ ├── channels.csv # 5 channels with min/max spend and ROI coefficient │ ├── campaigns.csv # 3 campaigns with budget and target conversions @@ -167,7 +209,7 @@ ad_spend_allocation/ └── pyproject.toml # dependencies ``` -**Start here**: run `python ad_spend_allocation.py` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.spend_allocation` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -177,7 +219,7 @@ The bundled data is synthetic and illustrative — designed to teach the optimiz - **`campaigns.csv`** (3 rows) — the campaigns (Brand_Awareness, Product_Launch, Seasonal_Sale), each with a budget and a target-conversions figure. - **`effectiveness.csv`** (15 rows) — one row per channel-campaign pair, giving the conversion rate (conversions per dollar spent) for that pair. -The three budget levels ($35K, $45K, $55K) that drive the scenario sweep are defined in the script, not in a CSV. +The three budget levels ($35K, $45K, $55K) that drive the scenario sweep are defined in `model/__init__.py`, not in a CSV. ## Model overview @@ -187,11 +229,11 @@ One ontology holds the inputs and the decision variables. The three source CSVs - **Primary identifiers**: integer `id` on `Channel` and `Campaign`; a composite `channel_id` + `campaign_id` on `Effectiveness`; a string `name` on `Scenario`; and the linked `Effectiveness` on `Allocation`. - **Important invariants**: spend is non-negative; `x_active` is binary (0/1); spend on a pair sits within its channel's min/max only when the pair is active; per-campaign spend stays within the campaign budget; every campaign has at least one funded channel; and total spend stays within the scenario's total budget. -For the full concept and property definitions, see `ad_spend_allocation.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works -The script loads the CSVs into concepts, defines the decision variables, adds the constraints and objective, and solves all budget levels in one call. +The model loads the CSVs into concepts, defines the decision variables, adds the constraints and objective, and solves all budget levels in one solve. 1. **Define the ontology.** Channels, campaigns, and their per-pair effectiveness (conversion rates) load into concepts, with the effectiveness rows linking each channel to each campaign. @@ -205,7 +247,7 @@ The script loads the CSVs into concepts, defines the decision variables, adds th 6. **Solve once for all scenarios.** A single HiGHS solve covers all three budget levels; results are extracted per scenario and printed as a table. -See `ad_spend_allocation.py` for the implementation and `runbook.md` for the skill-driven reproduction. The end-to-end flow: +See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. The end-to-end flow: ```text CSV inputs → load into concepts → decision variables (spend + funding, per scenario) @@ -225,7 +267,7 @@ Focus on the first changes most users will make. ### Tune parameters -- **Budget levels** — the three scenarios (`budget_35k`, `budget_45k`, `budget_55k`) are defined in the `scenario_data` block near the top of the script. Add, remove, or change levels there to sweep a different range. +- **Budget levels** — the three scenarios (`budget_35k`, `budget_45k`, `budget_55k`) are defined in the `scenario_data` block near the top of `model/__init__.py`. Add, remove, or change levels there to sweep a different range. - **Spend bounds and budgets** — per-channel `min_spend` / `max_spend` live in `channels.csv`; per-campaign `budget` lives in `campaigns.csv`. These are usually the binding limits, so they are the first knobs to adjust when the total-budget cap stops mattering. - **Solver settings** — the solve uses HiGHS with a 60-second time limit (`problem.configure("highs", time_limit_sec=60)`); adjust the limit for larger instances. @@ -264,7 +306,7 @@ Focus on the first changes most users will make.
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI / Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/ad_spend_allocation/ad_spend_allocation.py b/ad_spend_allocation/model/__init__.py similarity index 80% rename from ad_spend_allocation/ad_spend_allocation.py rename to ad_spend_allocation/model/__init__.py index 52089b2d..38b0cd0f 100644 --- a/ad_spend_allocation/ad_spend_allocation.py +++ b/ad_spend_allocation/model/__init__.py @@ -1,6 +1,6 @@ -"""Ad spend allocation (prescriptive optimization) template. +"""Ad spend allocation (prescriptive optimization) template: the model. -This script demonstrates a mixed-integer linear optimization (MILP) workflow +This package demonstrates a mixed-integer linear optimization (MILP) workflow in RelationalAI: - Load sample CSVs describing marketing channels, campaigns, and channel-campaign @@ -21,13 +21,9 @@ - One solve handles all budget levels; results extracted via model.select(). Run: - `python ad_spend_allocation.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints the solver termination status, objective value, and a table of - non-trivial allocations for each budget scenario. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.spend_allocation` then prints the report from the deployed model. """ from pathlib import Path @@ -40,7 +36,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -183,34 +179,3 @@ SolveStatus = model.Relationship(f"ad spend solve status is {String:status}") TotalConversions = model.Relationship(f"ad spend total conversions is {Float:conversions}") model.define(SolveStatus(problem.termination_status()), TotalConversions(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - conversions = model.select(TotalConversions).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal spend allocation: termination status {status}") - print(f"• objective: {round(conversions[0], 9)}\n") - - # -------------------------------------------------- - # Extract results per scenario - # -------------------------------------------------- - - print("\nSpend allocation per scenario:") - model.select( - Scenario.name.alias("scenario"), - Allocation.effectiveness.channel.name.alias("channel"), - Allocation.effectiveness.campaign.name.alias("campaign"), - x_spend.alias("spend"), - ).where(Allocation.x_spend(Scenario, x_spend), x_spend > 0.001).inspect() diff --git a/ad_spend_allocation/queries/spend_allocation.py b/ad_spend_allocation/queries/spend_allocation.py new file mode 100644 index 00000000..1baa0fd9 --- /dev/null +++ b/ad_spend_allocation/queries/spend_allocation.py @@ -0,0 +1,40 @@ +"""Spend allocation report: reads the deployed ad_spend_allocation model. + +Run `python -m queries.spend_allocation` after `rai models deploy --path model`. Prints the +formulation, the termination status and objective of the latest refresh's solve, and a table +of the non-trivial spend allocations for each budget scenario. +""" + +from model import ( + Allocation, + Scenario, + SolveStatus, + TotalConversions, + model, + problem, + x_spend, +) + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + conversions = model.select(TotalConversions).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal spend allocation: termination status {status}") + print(f"• objective: {round(conversions[0], 9)}\n") + + # -------------------------------------------------- + # Extract results per scenario + # -------------------------------------------------- + + print("\nSpend allocation per scenario:") + model.select( + Scenario.name.alias("scenario"), + Allocation.effectiveness.channel.name.alias("channel"), + Allocation.effectiveness.campaign.name.alias("campaign"), + x_spend.alias("spend"), + ).where(Allocation.x_spend(Scenario, x_spend), x_spend > 0.001).inspect() diff --git a/ad_spend_allocation/runbook.md b/ad_spend_allocation/runbook.md index 0a718c40..b3feba0a 100644 --- a/ad_spend_allocation/runbook.md +++ b/ad_spend_allocation/runbook.md @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 5 channels, 3 campaigns, 15 effectiveness rows. The three budget levels ($35k, $45k, $55k) are defined in the script. Full model in `ad_spend_allocation.py`. +Bundled CSVs in `data/`: 5 channels, 3 campaigns, 15 effectiveness rows. The three budget levels ($35k, $45k, $55k) are defined in the model. Full model in `model/__init__.py`; the report in `queries/spend_allocation.py`. From 11fdec09470c30bee34c269bf57b133102e9e6ec Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:29:07 -0800 Subject: [PATCH 050/162] campaign_roi: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/budget_reallocation.py (`python -m queries.budget_reallocation`). The current-budget conversion total is computed from the deployed campaigns instead of the CSV. Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- campaign_roi/README.md | 131 ++++++++++++------ .../{campaign_roi.py => model/__init__.py} | 73 +--------- campaign_roi/queries/budget_reallocation.py | 66 +++++++++ campaign_roi/runbook.md | 2 +- 4 files changed, 160 insertions(+), 112 deletions(-) rename campaign_roi/{campaign_roi.py => model/__init__.py} (57%) create mode 100644 campaign_roi/queries/budget_reallocation.py diff --git a/campaign_roi/README.md b/campaign_roi/README.md index d4a2762f..6da9440b 100644 --- a/campaign_roi/README.md +++ b/campaign_roi/README.md @@ -37,7 +37,8 @@ Every marketing team eventually faces the same question: the portfolio of campai ## What's included - **Model**: a continuous LP over a single `Campaign` concept, with per-campaign floor and cap bounds, a total-budget constraint, a paused-region share cap, and a conversion-maximizing objective. -- **Runner**: `campaign_roi.py` — a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. +- **Deployment**: `model/__init__.py` holds the model; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve. +- **Query**: `queries/budget_reallocation.py` prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: `data/campaigns.csv` — 12 campaigns across 5 regions with current budgets and empirical conversion rates. - **Outputs**: solver status, optimized-versus-current conversions and lift, a per-campaign reallocation table, and a regional-spend summary against the paused-region cap. @@ -47,7 +48,7 @@ Every marketing team eventually faces the same question: the portfolio of campai ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -57,32 +58,61 @@ Every marketing team eventually faces the same question: the portfolio of campai ## Quickstart -1. Download the ZIP file for this template and extract it: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .CAMPAIGN_ROI_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/campaign_roi.zip), unzip it, and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/campaign_roi.zip unzip campaign_roi.zip cd campaign_roi ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. Create and activate a virtual environment: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. Install dependencies: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -101,50 +131,58 @@ Every marketing team eventually faces the same question: the portfolio of campai pat_user: pat_warehouse: ``` - Without the `deployment` section, `python campaign_roi.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .CAMPAIGN_ROI_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .CAMPAIGN_ROI_META TO APPLICATION ; - ``` - The deploy creates the public schema `.CAMPAIGN_ROI` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python campaign_roi.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .CAMPAIGN_ROI_META._INTERNAL_REFRESH_MODEL('standard', NULL); - ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. -6. Expected output (a few lines confirm a successful run): +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.budget_reallocation + ``` ```text + Solve result: + • status: OPTIMAL + • objective: 3151.1 + Status: OPTIMAL Total expected conversions (optimized): 3,151.1 Total expected conversions (current): 2,691.0 Lift: +17.1% - + ... Paused region 'WEST' cap: 565.0 (50% of $1,130K total) ``` - The optimum hits the WEST regional cap exactly (50% of $1,130K = $565K), reaches the 3x cap on three high-performing campaigns (`WEST_Premium_Upgrade`, `CENTRAL_Retention_3`, `SOUTH_WinBack_6`), and pushes six low-performing campaigns down to the 10% floor. Despite holding total spend constant, conversions increase by 17.1%. The full printout, including the per-campaign reallocation table and regional-spend summary, is in `runbook.md`. + The formulation, the per-campaign reallocation table, and the regional spend table are trimmed here. The optimum hits the WEST regional cap exactly (50% of $1,130K = $565K), reaches the 3x cap on three high-performing campaigns (`WEST_Premium_Upgrade`, `CENTRAL_Retention_3`, `SOUTH_WinBack_6`), and pushes six low-performing campaigns down to the 10% floor. Despite holding total spend constant, conversions increase by 17.1%. The report checks the solve's termination status and stops unless it is OPTIMAL. The full printout, including the per-campaign reallocation table and regional-spend summary, is in `runbook.md`. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .CAMPAIGN_ROI_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -153,12 +191,15 @@ Every marketing team eventually faces the same question: the portfolio of campai ├── README.md # this file ├── pyproject.toml # dependencies ├── runbook.md # analyst-facing walkthrough -├── campaign_roi.py # main script (end-to-end) +├── model/ +│ └── __init__.py # the deployed model +├── queries/ +│ └── budget_reallocation.py # report from the deployed model └── data/ └── campaigns.csv # 12 campaigns across 5 regions ``` -**Start here**: run `python campaign_roi.py` for the full reallocation end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.budget_reallocation` for the full reallocation end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -182,11 +223,11 @@ The model is a single concept: one `Campaign` per row of `campaigns.csv`, enrich - **Primary identifiers**: integer `id` on `Campaign`, loaded from `data/campaigns.csv`. - **Important invariants**: `current_budget` and `conversion_rate` are non-negative; the optimized budget `x_budget` stays between the per-campaign floor and cap; total spend does not exceed `TOTAL_BUDGET`; and the paused region's spend does not exceed its share cap. -For the full concept and property definitions, see `campaign_roi.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works -The script loads the campaigns into a single `Campaign` concept, defines one continuous decision variable per campaign, adds the floor, cap, total-budget, and regional-cap constraints, maximizes conversions, and prints the reallocation. +The model loads the campaigns into a single `Campaign` concept, defines one continuous decision variable per campaign, adds the floor, cap, total-budget, and regional-cap constraints, and maximizes conversions; the report prints the reallocation. ### 1. Decision variable @@ -200,7 +241,7 @@ The per-campaign floor and cap are derived from each campaign's current spend: t The objective maximizes total expected conversions — the sum over campaigns of budget times conversion rate. -See `campaign_roi.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -210,7 +251,7 @@ Replace `data/campaigns.csv` with your own. Required columns: `id`, `name`, `reg ### Tune parameters -All parameters live at the top of the script under `# Configure inputs`: +All parameters live at the top of `model/__init__.py` under `# Configure inputs`: | Parameter | Default | Effect | |---|---|---| diff --git a/campaign_roi/campaign_roi.py b/campaign_roi/model/__init__.py similarity index 57% rename from campaign_roi/campaign_roi.py rename to campaign_roi/model/__init__.py index e66d6ecd..b5143ed4 100644 --- a/campaign_roi/campaign_roi.py +++ b/campaign_roi/model/__init__.py @@ -1,6 +1,6 @@ -"""Campaign ROI (prescriptive optimization) template. +"""Campaign ROI (prescriptive optimization) template: the model. -This script demonstrates a marketing budget reallocation LP in RelationalAI: +This package demonstrates a marketing budget reallocation LP in RelationalAI: - Load sample CSV describing marketing campaigns across regions, each with a current budget and a conversion rate. @@ -13,14 +13,9 @@ - Maximize total expected conversions = sum(budget * conversion_rate). Run: - `python campaign_roi.py` deploys the model to the schema in raiconfig.yaml, waits for - its first refresh (which runs the solve), and prints the report from the deployed - model. - -Output: - Prints solver termination status, total conversions vs current, - optimized budget vs current per campaign, and regional spend summary - against the paused-region cap. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.budget_reallocation` then prints the report from the deployed model. """ from pathlib import Path @@ -33,7 +28,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" TOTAL_BUDGET = 1130.0 # $K — same as current total; pure reallocation, no budget cut FLOOR_FRACTION = 0.1 # no campaign drops below 10% of current spend @@ -97,7 +92,7 @@ problem.maximize(sum(Campaign.x_budget * Campaign.conversion_rate)) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- # Each refresh of the deployed model runs this solve. @@ -110,57 +105,3 @@ SolveStatus = model.Relationship(f"campaign roi solve status is {String:status}") TotalConversions = model.Relationship(f"campaign roi total conversions is {Float:conversions}") model.define(SolveStatus(problem.termination_status()), TotalConversions(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - conversions = model.select(TotalConversions).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal budget reallocation: termination status {status}") - objective_value = conversions[0] - print(f"• objective: {round(objective_value, 9)}\n") - - print(f"\nStatus: {status[0]}") - print(f"Total expected conversions (optimized): {objective_value:,.1f}") - - current_total = (campaign_csv["current_budget"] * campaign_csv["conversion_rate"]).sum() - print(f"Total expected conversions (current): {current_total:,.1f}") - print(f"Lift: {(objective_value - current_total) / current_total * 100:+.1f}%") - - print("\nBudget reallocation per campaign (sorted by largest absolute change):") - results_df = model.select( - Campaign.name.alias("campaign"), - Campaign.region.alias("region"), - Campaign.current_budget.alias("current"), - Campaign.x_budget.alias("optimized"), - Campaign.conversion_rate.alias("rate"), - ).to_df() - results_df["delta"] = results_df["optimized"] - results_df["current"] - results_df["multiple"] = results_df["optimized"] / results_df["current"] - results_df = results_df.sort_values("delta", key=abs, ascending=False).reset_index(drop=True) - print(results_df.to_string(index=False)) - - print("\nRegional spend (optimized):") - regional = ( - results_df.groupby("region")[["current", "optimized"]] - .sum() - .reset_index() - .sort_values("optimized", ascending=False) - ) - regional["share_of_total"] = regional["optimized"] / TOTAL_BUDGET - print(regional.to_string(index=False)) - print( - f"\nPaused region '{PAUSED_REGION}' cap: " - f"{PAUSED_CAP_FRACTION * TOTAL_BUDGET:.1f} ({PAUSED_CAP_FRACTION:.0%} of ${TOTAL_BUDGET:,.0f}K total)" - ) diff --git a/campaign_roi/queries/budget_reallocation.py b/campaign_roi/queries/budget_reallocation.py new file mode 100644 index 00000000..afe28978 --- /dev/null +++ b/campaign_roi/queries/budget_reallocation.py @@ -0,0 +1,66 @@ +"""Budget reallocation report: reads the deployed campaign_roi model. + +Run `python -m queries.budget_reallocation` after `rai models deploy --path model`. Prints the +formulation, the termination status and total expected conversions of the latest refresh's +solve against the current budgets, the optimized budget per campaign, and the regional spend +against the paused-region cap. +""" + +from model import ( + PAUSED_CAP_FRACTION, + PAUSED_REGION, + TOTAL_BUDGET, + Campaign, + SolveStatus, + TotalConversions, + model, + problem, +) + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + conversions = model.select(TotalConversions).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal budget reallocation: termination status {status}") + objective_value = conversions[0] + print(f"• objective: {round(objective_value, 9)}\n") + + print(f"\nStatus: {status[0]}") + print(f"Total expected conversions (optimized): {objective_value:,.1f}") + + results_df = model.select( + Campaign.name.alias("campaign"), + Campaign.region.alias("region"), + Campaign.current_budget.alias("current"), + Campaign.x_budget.alias("optimized"), + Campaign.conversion_rate.alias("rate"), + ).to_df() + + current_total = (results_df["current"] * results_df["rate"]).sum() + print(f"Total expected conversions (current): {current_total:,.1f}") + print(f"Lift: {(objective_value - current_total) / current_total * 100:+.1f}%") + + print("\nBudget reallocation per campaign (sorted by largest absolute change):") + results_df["delta"] = results_df["optimized"] - results_df["current"] + results_df["multiple"] = results_df["optimized"] / results_df["current"] + results_df = results_df.sort_values("delta", key=abs, ascending=False).reset_index(drop=True) + print(results_df.to_string(index=False)) + + print("\nRegional spend (optimized):") + regional = ( + results_df.groupby("region")[["current", "optimized"]] + .sum() + .reset_index() + .sort_values("optimized", ascending=False) + ) + regional["share_of_total"] = regional["optimized"] / TOTAL_BUDGET + print(regional.to_string(index=False)) + print( + f"\nPaused region '{PAUSED_REGION}' cap: " + f"{PAUSED_CAP_FRACTION * TOTAL_BUDGET:.1f} ({PAUSED_CAP_FRACTION:.0%} of ${TOTAL_BUDGET:,.0f}K total)" + ) diff --git a/campaign_roi/runbook.md b/campaign_roi/runbook.md index d41f52ad..37eb0180 100644 --- a/campaign_roi/runbook.md +++ b/campaign_roi/runbook.md @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSV in `data/`: 12 campaigns with region, current budget, and conversion rate. The budget bounds (10%–300%), total-spend cap, and WEST regional cap are constants in the script. Full model in `campaign_roi.py`. +Bundled CSV in `data/`: 12 campaigns with region, current budget, and conversion rate. The budget bounds (10%–300%), total-spend cap, and WEST regional cap are constants in the model. Full model in `model/__init__.py`; the report in `queries/budget_reallocation.py`. From 6fb8d2a071aedebc037b878dece1733a825a6a16 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:31:14 -0800 Subject: [PATCH 051/162] cell_tower_coverage: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/coverage_report.py (`python -m queries.coverage_report`). The report takes the site, zone and pair counts and the total population from the deployed model, checks the termination status with an explicit raise, and still prints the absolute path of data/coverage_solution.csv. Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- cell_tower_coverage/README.md | 143 +++++++----- .../__init__.py} | 210 +---------------- .../queries/coverage_report.py | 215 ++++++++++++++++++ cell_tower_coverage/runbook.md | 4 +- 4 files changed, 312 insertions(+), 260 deletions(-) rename cell_tower_coverage/{cell_tower_coverage.py => model/__init__.py} (51%) create mode 100644 cell_tower_coverage/queries/coverage_report.py diff --git a/cell_tower_coverage/README.md b/cell_tower_coverage/README.md index 4b0178d1..7e32725c 100644 --- a/cell_tower_coverage/README.md +++ b/cell_tower_coverage/README.md @@ -38,8 +38,8 @@ Telecommunications and infrastructure teams routinely have to decide where to bu ## What's included -- **Model**: `cell_tower_coverage.py` builds the semantic model (three concepts, two coverage relationships), the mixed-integer optimization problem (three binary variable families, four constraints, one objective), the solve, and the reporting. -- **Runner**: `cell_tower_coverage.py` — a single Python script that deploys the model to your Snowflake account, where each refresh runs the solve, and prints the report from the deployed model. +- **Model**: `model/__init__.py` builds the semantic model (three concepts, two coverage relationships), the mixed-integer optimization problem (three binary variable families, four constraints, one objective), and the solve; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve. +- **Query**: `queries/coverage_report.py` prints the report from the deployed model and writes `data/coverage_solution.csv`. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: 6 candidate tower sites, 10 demand zones, and 19 feasible tower-zone coverage pairs. See *Sample data* below. - **Outputs**: stdout tables (selected sites, assigned zones, uncovered zones, coverage summary) plus `data/coverage_solution.csv`. @@ -49,7 +49,7 @@ Telecommunications and infrastructure teams routinely have to decide where to bu ### Access - A Snowflake account with the RelationalAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -59,18 +59,40 @@ Telecommunications and infrastructure teams routinely have to decide where to bu ## Quickstart -1. Download the template and extract it: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .CELL_TOWER_COVERAGE_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/cell_tower_coverage.zip), unzip it, and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/cell_tower_coverage.zip unzip cell_tower_coverage.zip cd cell_tower_coverage ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. Create a virtual environment and activate it: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv @@ -78,13 +100,19 @@ Telecommunications and infrastructure teams routinely have to decide where to bu python -m pip install --upgrade pip ``` -3. Install dependencies: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: ```yaml reasoners: @@ -105,43 +133,25 @@ Telecommunications and infrastructure teams routinely have to decide where to bu pat_warehouse: ``` - Without the `deployment` section, `python cell_tower_coverage.py` stops with an error naming the missing setting. +5. **Deploy the model** - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .CELL_TOWER_COVERAGE_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .CELL_TOWER_COVERAGE_META TO APPLICATION ; + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.CELL_TOWER_COVERAGE` if it doesn't exist. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. -5. Run the template end-to-end: + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. - ```bash - python cell_tower_coverage.py - ``` +6. **Query the results** - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + Print the report from the deployed model: - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .CELL_TOWER_COVERAGE_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ```bash + python -m queries.coverage_report ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. - -6. Expected output (a few lines confirm a successful run): ```text ====================================================================== @@ -154,32 +164,57 @@ Telecommunications and infrastructure teams routinely have to decide where to bu Max new towers: 3 Status: OPTIMAL - Objective: covered population = ... + Objective: covered population = 90,000 + + ... === Coverage Summary === - Selected build cost: ... - Covered population: ... - Coverage rate: ... + Selected build cost: $610,000 of $650,000 + Covered population: 90,000 of 102,900 + Coverage rate: 87.46% + Uncovered population: 12,900 - Wrote coverage solution to: data/coverage_solution.csv + Wrote coverage solution to: /path/to/cell_tower_coverage/data/coverage_solution.csv ``` + The selected-sites, assigned-zones, and uncovered-zones tables are trimmed here. Three towers (T04, T02, and T06) cover 90,000 of 102,900 people for $610,000 of the $650,000 budget, and zones Z04 and Z10 stay uncovered. The report also writes the per-zone solution to `data/coverage_solution.csv` in the template directory, and it stops unless the termination status is OPTIMAL or TIME_LIMIT. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .CELL_TOWER_COVERAGE_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. + ## Template structure ```text cell_tower_coverage/ - cell_tower_coverage.py # Main script: model, solve, report + model/ + __init__.py # the deployed model (concepts, constraints, solve) + queries/ + coverage_report.py # report from the deployed model data/ tower_sites.csv # 6 candidate build sites demand_zones.csv # 10 population demand zones coverage_pairs.csv # 19 feasible tower-zone coverage pairs - coverage_solution.csv # written by the script after solving + coverage_solution.csv # written by the query README.md # this file runbook.md # paste-testable walkthrough (RAI skills) pyproject.toml # dependencies ``` -**Start here**: run `python cell_tower_coverage.py` for the full model, solve, and report end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.coverage_report` for the full model, solve, and report end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -196,15 +231,15 @@ Three concepts describe the planning problem; the optimization adds one binary d - **Key entities**: `TowerSite` — a candidate infrastructure location that may be built (site type, build cost, serving capacity); `DemandZone` — a population area that may be covered by a selected tower; `CoveragePair` — a feasible tower-zone service relationship, where only paired zones can be served and only by their paired towers. - **Primary identifiers**: `site_id` on `TowerSite`, `zone_id` on `DemandZone`, and a composite `site_id` + `zone_id` key on `CoveragePair`. -- **Important invariants**: build costs, capacities, and populations are non-negative; every demand zone is reachable by at least one coverage pair (the script raises an error otherwise); the decision variables (`selected`, `covered`, `assigned`) are binary; total build cost stays at or under the budget and the number of selected towers stays at or under the cap. +- **Important invariants**: build costs, capacities, and populations are non-negative; every demand zone is reachable by at least one coverage pair (the deploy stops with an error otherwise); the decision variables (`selected`, `covered`, `assigned`) are binary; total build cost stays at or under the budget and the number of selected towers stays at or under the cap. `CoveragePair` links each pair to its `TowerSite` and `DemandZone`, so only zones that appear as a pair can be served, and only by the towers they are paired with. -For the full concept and property definitions, see `cell_tower_coverage.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works -The pipeline runs in one script, from CSV inputs to a solved plan and an exported solution: +The pipeline runs from CSV inputs to a solved plan and an exported solution: ```text CSV inputs -> load and validate -> concepts and coverage pairs -> decision variables -> constraints and objective -> solve (HiGHS) -> report and export @@ -212,7 +247,7 @@ CSV inputs -> load and validate -> concepts and coverage pairs -> decision varia ### 1. Load and validate the planning data -The script reads the three CSVs and runs feasibility checks before building the model: each file is non-empty, the parameters are positive, coverage pairs reference known site and zone IDs, and every demand zone is reachable by at least one pair. Any violation raises a clear error rather than producing a silently wrong plan. +The model reads the three CSVs and runs feasibility checks before building it: each file is non-empty, the parameters are positive, coverage pairs reference known site and zone IDs, and every demand zone is reachable by at least one pair. Any violation raises a clear error rather than producing a silently wrong plan. ### 2. Define selection, coverage, and assignment variables @@ -236,9 +271,9 @@ Two portfolio constraints bound the plan: total build cost stays at or under `BU ### 7. Maximize covered population -The objective rewards covering high-population zones, weighting each covered zone by its population. Because each zone has a single binary covered variable and each covered zone has exactly one assignment, the model handles overlapping tower coverage without double-counting population or overloading selected towers. After the solve, the script builds report tables (selected sites with utilization, assigned zones, uncovered zones, coverage summary) and writes `data/coverage_solution.csv`. +The objective rewards covering high-population zones, weighting each covered zone by its population. Because each zone has a single binary covered variable and each covered zone has exactly one assignment, the model handles overlapping tower coverage without double-counting population or overloading selected towers. After the solve, the query builds report tables (selected sites with utilization, assigned zones, uncovered zones, coverage summary) and writes `data/coverage_solution.csv`. -See `cell_tower_coverage.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -247,7 +282,7 @@ Focus on the first changes most users will make. ### Use your own data - Replace the CSVs in `data/` with your own; keep the column names listed in *Sample data* above. -- Regenerate coverage pairs with your preferred distance, signal-strength, or engineering threshold, and make sure every demand zone appears in at least one pair (the script requires it). +- Regenerate coverage pairs with your preferred distance, signal-strength, or engineering threshold, and make sure every demand zone appears in at least one pair (the model requires it). - Adjust the `capacity` column in `tower_sites.csv` to reflect real serving capacity, and the `build_cost` column to reflect real capital cost. ### Tune parameters @@ -299,7 +334,7 @@ Focus on the first changes most users will make.
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/cell_tower_coverage/cell_tower_coverage.py b/cell_tower_coverage/model/__init__.py similarity index 51% rename from cell_tower_coverage/cell_tower_coverage.py rename to cell_tower_coverage/model/__init__.py index e890ad2f..39dbb3f4 100644 --- a/cell_tower_coverage/cell_tower_coverage.py +++ b/cell_tower_coverage/model/__init__.py @@ -1,6 +1,6 @@ -"""Cell tower coverage (prescriptive optimization) template. +"""Cell tower coverage (prescriptive optimization) template: the model. -This script demonstrates maximum coverage planning in RelationalAI: +This package demonstrates maximum coverage planning in RelationalAI: - Load sample demand zones, candidate tower sites, and tower-zone coverage pairs. - Choose tower sites to build under a fixed capital budget and tower-count limit. @@ -13,14 +13,9 @@ The optimization is modeled as a mixed-integer linear problem. Run: - `python cell_tower_coverage.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from - the deployed model. - -Output: - Prints the selected tower sites, assigned demand zones, tower utilization, - coverage summary, and writes data/coverage_solution.csv for downstream - mapping or reporting. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.coverage_report` then prints the report from the deployed model. """ from pathlib import Path @@ -36,7 +31,7 @@ BUILD_BUDGET = 650_000 MAX_NEW_TOWERS = 3 -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model and load data @@ -212,196 +207,3 @@ f"cell tower coverage covered population is {Float:covered_population}" ) model.define(SolveStatus(problem.termination_status()), CoveredPopulation(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - print("=" * 70) - print("CELL TOWER COVERAGE") - print("=" * 70) - print(f"Candidate tower sites: {len(tower_sites_csv)}") - print(f"Demand zones: {len(demand_zones_csv)}") - print(f"Coverage pairs: {len(coverage_pairs_csv)}") - print(f"Build budget: ${BUILD_BUDGET:,.0f}") - print(f"Max new towers: {MAX_NEW_TOWERS}") - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - objective = model.select(CoveredPopulation).to_df().iloc[:, 0].tolist() - assert status in (["OPTIMAL"], ["TIME_LIMIT"]), f"Solver did not find an optimal solution: {status}" - - print(f"\nStatus: {status[0]}") - print(f"Objective: covered population = {objective[0]:,.0f}") - - # -------------------------------------------------- - # Evaluate and report - # -------------------------------------------------- - - selected_sites_df = ( - model.select( - TowerSite.site_id.alias("site_id"), - TowerSite.name.alias("name"), - TowerSite.site_type.alias("site_type"), - TowerSite.region.alias("region"), - TowerSite.build_cost.alias("build_cost"), - TowerSite.capacity.alias("capacity"), - selected.alias("selected"), - ) - .where( - TowerSite.x_selected(selected), - selected > 0.5, - ) - .to_df() - .sort_values(["region", "site_id"]) - ) - selected_sites_df["build_cost"] = pd.to_numeric(selected_sites_df["build_cost"]) - selected_sites_df["capacity"] = pd.to_numeric(selected_sites_df["capacity"]) - selected_sites_df["selected"] = ( - pd.to_numeric(selected_sites_df["selected"]) > 0.5 - ).astype(int) - - zone_solution_df = ( - model.select( - DemandZone.zone_id.alias("zone_id"), - DemandZone.name.alias("name"), - DemandZone.region.alias("region"), - DemandZone.population.alias("population"), - covered.alias("covered"), - ) - .where(DemandZone.y_covered(covered)) - .to_df() - .sort_values(["covered", "population"], ascending=[False, False]) - ) - zone_solution_df["population"] = pd.to_numeric(zone_solution_df["population"]) - zone_solution_df["covered"] = ( - pd.to_numeric(zone_solution_df["covered"]) > 0.5 - ).astype(int) - - assignment_df = ( - model.select( - CoveragePair.site_id.alias("assigned_site_id"), - TowerSite.name.alias("assigned_site"), - CoveragePair.zone_id.alias("zone_id"), - assigned.alias("assigned"), - ) - .where( - CoveragePair.z_assigned(assigned), - CoveragePair.site(TowerSite), - assigned > 0.5, - ) - .to_df() - .sort_values(["assigned_site_id", "zone_id"]) - ) - assignment_df["assigned"] = ( - pd.to_numeric(assignment_df["assigned"]) > 0.5 - ).astype(int) - - zone_solution_df = zone_solution_df.merge( - assignment_df[["zone_id", "assigned_site_id", "assigned_site"]], - on="zone_id", - how="left", - ) - zone_solution_df["assigned_site_id"] = zone_solution_df["assigned_site_id"].fillna("") - zone_solution_df["assigned_site"] = zone_solution_df["assigned_site"].fillna("") - - total_population = demand_zones_csv["population"].sum() - covered_population = zone_solution_df.loc[ - zone_solution_df["covered"] > 0.5, "population" - ].sum() - uncovered_population = total_population - covered_population - selected_cost = selected_sites_df["build_cost"].sum() - - tower_load_df = ( - zone_solution_df[ - zone_solution_df["covered"] > 0.5 - ] - .groupby("assigned_site_id", as_index=False)["population"] - .sum() - .rename(columns={"population": "assigned_population"}) - ) - selected_sites_df = selected_sites_df.merge( - tower_load_df, - left_on="site_id", - right_on="assigned_site_id", - how="left", - ).drop(columns=["assigned_site_id"]) - selected_sites_df["assigned_population"] = ( - selected_sites_df["assigned_population"].fillna(0).astype(int) - ) - selected_sites_df["utilization"] = ( - selected_sites_df["assigned_population"] / selected_sites_df["capacity"] - ) - - print("\n=== Selected Tower Sites ===") - print( - selected_sites_df[ - [ - "site_id", - "name", - "site_type", - "region", - "build_cost", - "capacity", - "assigned_population", - "utilization", - ] - ].to_string( - index=False, - formatters={ - "build_cost": "${:,.0f}".format, - "capacity": "{:,.0f}".format, - "assigned_population": "{:,.0f}".format, - "utilization": "{:.1%}".format, - }, - ) - ) - - print("\n=== Assigned Demand Zones ===") - print( - zone_solution_df[ - zone_solution_df["covered"] > 0.5 - ][ - ["zone_id", "name", "region", "population", "assigned_site"] - ].to_string( - index=False, - formatters={"population": "{:,.0f}".format}, - ) - ) - - uncovered_zones_df = zone_solution_df[zone_solution_df["covered"] <= 0.5] - if not uncovered_zones_df.empty: - print("\n=== Uncovered Demand Zones ===") - print( - uncovered_zones_df[ - ["zone_id", "name", "region", "population"] - ].to_string( - index=False, - formatters={"population": "{:,.0f}".format}, - ) - ) - - print("\n=== Coverage Summary ===") - print(f"Selected build cost: ${selected_cost:,.0f} of ${BUILD_BUDGET:,.0f}") - print(f"Covered population: {covered_population:,.0f} of {total_population:,.0f}") - print(f"Coverage rate: {covered_population / total_population:.2%}") - print(f"Uncovered population: {uncovered_population:,.0f}") - - coverage_solution_csv = DATA_DIR / "coverage_solution.csv" - zone_solution_df[ - [ - "zone_id", - "name", - "region", - "population", - "covered", - "assigned_site_id", - "assigned_site", - ] - ].to_csv(coverage_solution_csv, index=False) - print(f"\nWrote coverage solution to: {coverage_solution_csv}") diff --git a/cell_tower_coverage/queries/coverage_report.py b/cell_tower_coverage/queries/coverage_report.py new file mode 100644 index 00000000..8b8cf306 --- /dev/null +++ b/cell_tower_coverage/queries/coverage_report.py @@ -0,0 +1,215 @@ +"""Coverage report: reads the deployed cell_tower_coverage model. + +Run `python -m queries.coverage_report` after `rai models deploy --path model`. Prints the +selected tower sites, assigned demand zones, tower utilization and coverage summary of the +latest refresh's solve, and writes data/coverage_solution.csv for downstream mapping or +reporting. +""" + +import pandas as pd +from model import ( + BUILD_BUDGET, + DATA_DIR, + MAX_NEW_TOWERS, + CoveragePair, + CoveredPopulation, + DemandZone, + SolveStatus, + TowerSite, + assigned, + covered, + model, + selected, +) + +if __name__ == "__main__": + # Entity counts and total population come from the deployed model. + sites_df = model.select(TowerSite.site_id).to_df() + zones_df = model.select(DemandZone.zone_id, DemandZone.population).to_df() + pairs_df = model.select(CoveragePair.site_id, CoveragePair.zone_id).to_df() + + print("=" * 70) + print("CELL TOWER COVERAGE") + print("=" * 70) + print(f"Candidate tower sites: {len(sites_df)}") + print(f"Demand zones: {len(zones_df)}") + print(f"Coverage pairs: {len(pairs_df)}") + print(f"Build budget: ${BUILD_BUDGET:,.0f}") + print(f"Max new towers: {MAX_NEW_TOWERS}") + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + objective = model.select(CoveredPopulation).to_df().iloc[:, 0].tolist() + if status not in (["OPTIMAL"], ["TIME_LIMIT"]): + raise SystemExit(f"Solver did not find an optimal solution: {status}") + + print(f"\nStatus: {status[0]}") + print(f"Objective: covered population = {objective[0]:,.0f}") + + # -------------------------------------------------- + # Evaluate and report + # -------------------------------------------------- + + selected_sites_df = ( + model.select( + TowerSite.site_id.alias("site_id"), + TowerSite.name.alias("name"), + TowerSite.site_type.alias("site_type"), + TowerSite.region.alias("region"), + TowerSite.build_cost.alias("build_cost"), + TowerSite.capacity.alias("capacity"), + selected.alias("selected"), + ) + .where( + TowerSite.x_selected(selected), + selected > 0.5, + ) + .to_df() + .sort_values(["region", "site_id"]) + ) + selected_sites_df["build_cost"] = pd.to_numeric(selected_sites_df["build_cost"]) + selected_sites_df["capacity"] = pd.to_numeric(selected_sites_df["capacity"]) + selected_sites_df["selected"] = ( + pd.to_numeric(selected_sites_df["selected"]) > 0.5 + ).astype(int) + + zone_solution_df = ( + model.select( + DemandZone.zone_id.alias("zone_id"), + DemandZone.name.alias("name"), + DemandZone.region.alias("region"), + DemandZone.population.alias("population"), + covered.alias("covered"), + ) + .where(DemandZone.y_covered(covered)) + .to_df() + .sort_values(["covered", "population"], ascending=[False, False]) + ) + zone_solution_df["population"] = pd.to_numeric(zone_solution_df["population"]) + zone_solution_df["covered"] = ( + pd.to_numeric(zone_solution_df["covered"]) > 0.5 + ).astype(int) + + assignment_df = ( + model.select( + CoveragePair.site_id.alias("assigned_site_id"), + TowerSite.name.alias("assigned_site"), + CoveragePair.zone_id.alias("zone_id"), + assigned.alias("assigned"), + ) + .where( + CoveragePair.z_assigned(assigned), + CoveragePair.site(TowerSite), + assigned > 0.5, + ) + .to_df() + .sort_values(["assigned_site_id", "zone_id"]) + ) + assignment_df["assigned"] = ( + pd.to_numeric(assignment_df["assigned"]) > 0.5 + ).astype(int) + + zone_solution_df = zone_solution_df.merge( + assignment_df[["zone_id", "assigned_site_id", "assigned_site"]], + on="zone_id", + how="left", + ) + zone_solution_df["assigned_site_id"] = zone_solution_df["assigned_site_id"].fillna("") + zone_solution_df["assigned_site"] = zone_solution_df["assigned_site"].fillna("") + + total_population = pd.to_numeric(zones_df["population"]).sum() + covered_population = zone_solution_df.loc[ + zone_solution_df["covered"] > 0.5, "population" + ].sum() + uncovered_population = total_population - covered_population + selected_cost = selected_sites_df["build_cost"].sum() + + tower_load_df = ( + zone_solution_df[ + zone_solution_df["covered"] > 0.5 + ] + .groupby("assigned_site_id", as_index=False)["population"] + .sum() + .rename(columns={"population": "assigned_population"}) + ) + selected_sites_df = selected_sites_df.merge( + tower_load_df, + left_on="site_id", + right_on="assigned_site_id", + how="left", + ).drop(columns=["assigned_site_id"]) + selected_sites_df["assigned_population"] = ( + selected_sites_df["assigned_population"].fillna(0).astype(int) + ) + selected_sites_df["utilization"] = ( + selected_sites_df["assigned_population"] / selected_sites_df["capacity"] + ) + + print("\n=== Selected Tower Sites ===") + print( + selected_sites_df[ + [ + "site_id", + "name", + "site_type", + "region", + "build_cost", + "capacity", + "assigned_population", + "utilization", + ] + ].to_string( + index=False, + formatters={ + "build_cost": "${:,.0f}".format, + "capacity": "{:,.0f}".format, + "assigned_population": "{:,.0f}".format, + "utilization": "{:.1%}".format, + }, + ) + ) + + print("\n=== Assigned Demand Zones ===") + print( + zone_solution_df[ + zone_solution_df["covered"] > 0.5 + ][ + ["zone_id", "name", "region", "population", "assigned_site"] + ].to_string( + index=False, + formatters={"population": "{:,.0f}".format}, + ) + ) + + uncovered_zones_df = zone_solution_df[zone_solution_df["covered"] <= 0.5] + if not uncovered_zones_df.empty: + print("\n=== Uncovered Demand Zones ===") + print( + uncovered_zones_df[ + ["zone_id", "name", "region", "population"] + ].to_string( + index=False, + formatters={"population": "{:,.0f}".format}, + ) + ) + + print("\n=== Coverage Summary ===") + print(f"Selected build cost: ${selected_cost:,.0f} of ${BUILD_BUDGET:,.0f}") + print(f"Covered population: {covered_population:,.0f} of {total_population:,.0f}") + print(f"Coverage rate: {covered_population / total_population:.2%}") + print(f"Uncovered population: {uncovered_population:,.0f}") + + coverage_solution_csv = DATA_DIR / "coverage_solution.csv" + zone_solution_df[ + [ + "zone_id", + "name", + "region", + "population", + "covered", + "assigned_site_id", + "assigned_site", + ] + ].to_csv(coverage_solution_csv, index=False) + print(f"\nWrote coverage solution to: {coverage_solution_csv}") diff --git a/cell_tower_coverage/runbook.md b/cell_tower_coverage/runbook.md index 5ae930c5..af67b020 100644 --- a/cell_tower_coverage/runbook.md +++ b/cell_tower_coverage/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Which sites should we build and which tower should serve each zone to cover the most people, building at most 3 towers within a $650,000 budget? Use a binary build decision per site, a binary covered decision per zone, and a binary serve decision per feasible site-zone pair. A zone counts as covered only if it's assigned to exactly one built tower, a tower can only serve zones if it's built, and the population a tower serves can't exceed its capacity. Maximize total covered population and persist the decisions to the ontology. -**Response:** OPTIMAL (HiGHS), covered population **90,000 of 102,900** (relative gap 0.0). The model has 3 binary decision families and 5 constraint families; build, cover, and serve decisions are written back as `TowerSite.x_selected`, `DemandZone.y_covered`, and `CoveragePair.z_assigned`. In `cell_tower_coverage.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** OPTIMAL (HiGHS), covered population **90,000 of 102,900** (relative gap 0.0). The model has 3 binary decision families and 5 constraint families; build, cover, and serve decisions are written back as `TowerSite.x_selected`, `DemandZone.y_covered`, and `CoveragePair.z_assigned`. In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the build plan @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 6 tower sites, 10 demand zones, 19 coverage pairs. The budget ($650k) and tower cap (3) are constants in the script. Full model in `cell_tower_coverage.py`. +Bundled CSVs in `data/`: 6 tower sites, 10 demand zones, 19 coverage pairs. The budget ($650k) and tower cap (3) are constants in the model. Full model in `model/__init__.py`; the report in `queries/coverage_report.py`. From caf9e80492e4175d30684abf924a6e845c9577be Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:33:54 -0800 Subject: [PATCH 052/162] network_flow_planning: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/network_plan.py (`python -m queries.network_plan`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- network_flow_planning/README.md | 149 +++++++++++------- .../__init__.py} | 57 +------ network_flow_planning/queries/network_plan.py | 39 +++++ network_flow_planning/runbook.md | 2 +- 4 files changed, 143 insertions(+), 104 deletions(-) rename network_flow_planning/{network_flow_planning.py => model/__init__.py} (77%) create mode 100644 network_flow_planning/queries/network_plan.py diff --git a/network_flow_planning/README.md b/network_flow_planning/README.md index f5eb3692..30150f1a 100644 --- a/network_flow_planning/README.md +++ b/network_flow_planning/README.md @@ -49,7 +49,8 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. ## What's included -- `network_flow_planning.py` — main script (deploys the model, then prints the report from the deployed model) +- `model/__init__.py` — the model; `rai models deploy --path model` deploys it +- `queries/network_plan.py` — prints the report from the deployed model - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/sites.csv` — 12 sites: 3 warehouses, 3 transit hubs, 3 fulfillment centers, 3 customers - `data/lanes.csv` — 17 directed lanes connecting the tiers, each with a cost-per-unit and capacity @@ -61,7 +62,7 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -71,32 +72,61 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. ## Quickstart -1. Download the ZIP file for this template and extract it: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .NETWORK_FLOW_PLANNING_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/network_flow_planning.zip), unzip it, and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/network_flow_planning.zip unzip network_flow_planning.zip cd network_flow_planning ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. Create and activate a virtual environment: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. Install dependencies: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -115,61 +145,71 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. pat_user: pat_warehouse: ``` - Without the `deployment` section, `python network_flow_planning.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .NETWORK_FLOW_PLANNING_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .NETWORK_FLOW_PLANNING_META TO APPLICATION ; - ``` - The deploy creates the public schema `.NETWORK_FLOW_PLANNING` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python network_flow_planning.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .NETWORK_FLOW_PLANNING_META._INTERNAL_REFRESH_MODEL('standard', NULL); - ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. -6. Expected output (truncated): +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.network_plan + ``` ```text + Solve result: + • status: OPTIMAL + • objective: 5280.0 + Status: OPTIMAL Total cost: $5,280.00 + Opened fulfillment centers: fulfillment_center capacity fixed_cost - FC_Northeast 350.0 1500.0 - FC_Southwest 200.0 1200.0 + 0 FC_Northeast 350.0 1500.0 + 1 FC_Southwest 200.0 1200.0 + Active flows: from_site to_site flow unit_cost - Chicago_Warehouse Memphis_Hub 200.0 2.0 - Dallas_Warehouse KansasCity_Hub 250.0 2.5 - Memphis_Hub FC_Northeast 200.0 1.5 - KansasCity_Hub FC_Northeast 50.0 2.0 - KansasCity_Hub FC_Southwest 200.0 1.5 - FC_Northeast Customer_NYC 180.0 1.0 - FC_Northeast Customer_LA 70.0 5.0 - FC_Southwest Customer_Houston 150.0 1.0 - FC_Southwest Customer_LA 50.0 3.5 + 0 Chicago_Warehouse Memphis_Hub 200.0 2.0 + 1 Dallas_Warehouse KansasCity_Hub 250.0 2.5 + 2 FC_Northeast Customer_LA 70.0 5.0 + 3 FC_Northeast Customer_NYC 180.0 1.0 + 4 FC_Southwest Customer_Houston 150.0 1.0 + 5 FC_Southwest Customer_LA 50.0 3.5 + 6 KansasCity_Hub FC_Northeast 50.0 2.0 + 7 KansasCity_Hub FC_Southwest 200.0 1.5 + 8 Memphis_Hub FC_Northeast 200.0 1.5 + ``` + + The formulation printed above the solve result is trimmed here. The optimizer opens the two largest fulfillment centers and routes LA's demand split across both — the savings from not opening FC_West ($1,000 fixed) outweigh the extra transport cost incurred by routing 120 units of LA demand through the more expensive FC_Northeast and FC_Southwest paths. The report checks the solve's termination status and stops unless it is OPTIMAL. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .NETWORK_FLOW_PLANNING_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - The optimizer opens the two largest fulfillment centers and routes LA's demand split across both — the savings from not opening FC_West ($1,000 fixed) outweigh the extra transport cost incurred by routing 120 units of LA demand through the more expensive FC_Northeast and FC_Southwest paths. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -177,14 +217,17 @@ The objective minimizes transport cost plus fixed-cost FC opening cost. . ├── README.md # this file ├── pyproject.toml # dependencies -├── network_flow_planning.py # main script (end-to-end) +├── model/ +│ └── __init__.py # the deployed model +├── queries/ +│ └── network_plan.py # report from the deployed model └── data/ ├── sites.csv # 12 sites across 4 types ├── lanes.csv # 17 directed transport lanes └── demand.csv # 3 customer demands ``` -**Start here:** run `python network_flow_planning.py` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here:** deploy with `rai models deploy --path model` and run `python -m queries.network_plan` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -208,7 +251,7 @@ The model uses three concepts. A single `Site` concept carries all four network - **Primary identifiers**: integer `id` on each of `Site`, `Lane`, and `Demand`, loaded from the corresponding CSV. - **Important invariants**: `inventory`, `capacity`, `fixed_cost`, `cost_per_unit`, and `quantity` are non-negative; only fulfillment centers carry a positive `fixed_cost` (that is what scopes the binary open decision); every `Lane.source` / `Lane.dest` and every `Demand.site` resolves to a real `Site`. -For the full concept and property definitions, see `network_flow_planning.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -220,7 +263,7 @@ The model runs as a single MILP that decides flow on every lane and which fulfil **Objective.** Minimize transport cost (per-unit lane cost times flow, summed over lanes) plus the fixed cost of every opened fulfillment center. The two cost terms live on different concepts (`Lane` and `Site`), so the objective adds one sum per concept. -For the exact PyRel formulation, see `network_flow_planning.py`; `runbook.md` reproduces the model step by step with the RAI skills. +For the exact PyRel formulation, see `model/__init__.py`; `runbook.md` reproduces the model step by step with the RAI skills. ```text CSV inputs → load Site / Lane / Demand → flow + open decisions → role-based constraints → minimize transport + fixed cost → solve → export diff --git a/network_flow_planning/network_flow_planning.py b/network_flow_planning/model/__init__.py similarity index 77% rename from network_flow_planning/network_flow_planning.py rename to network_flow_planning/model/__init__.py index f8ec72c2..c0b6399c 100644 --- a/network_flow_planning/network_flow_planning.py +++ b/network_flow_planning/model/__init__.py @@ -1,6 +1,6 @@ -"""Network Flow Planning (prescriptive optimization) template. +"""Network Flow Planning (prescriptive optimization) template: the model. -This script demonstrates a multi-tier network flow problem with fixed-cost +This package demonstrates a multi-tier network flow problem with fixed-cost facility opening in RelationalAI: - Load sample CSVs describing sites (warehouses, hubs, fulfillment centers, @@ -16,14 +16,9 @@ - Minimize total cost (transport + fixed-cost FC opening). Run: - `python network_flow_planning.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints solver termination status, total cost (transport + fixed - components), the set of opened fulfillment centers, and the active flow - plan across the network. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.network_plan` then prints the report from the deployed model. """ from pathlib import Path @@ -36,7 +31,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -184,7 +179,7 @@ ) # -------------------------------------------------- -# Solve and check solution +# Solve # -------------------------------------------------- # Each refresh of the deployed model runs this solve. @@ -197,41 +192,3 @@ SolveStatus = model.Relationship(f"network flow solve status is {String:status}") TotalCost = model.Relationship(f"network flow total cost is {Float:cost}") model.define(SolveStatus(problem.termination_status()), TotalCost(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal network plan: termination status {status}") - total_cost = cost[0] - print(f"• objective: {round(total_cost, 9)}\n") - - print(f"\nStatus: {status[0]}") - print(f"Total cost: ${total_cost:,.2f}") - - print("\nOpened fulfillment centers:") - model.select( - Site.name.alias("fulfillment_center"), - Site.capacity.alias("capacity"), - Site.fixed_cost.alias("fixed_cost"), - ).where(Site.x_open > 0.5).inspect() - - print("\nActive flows:") - model.select( - Lane.source.name.alias("from_site"), - Lane.dest.name.alias("to_site"), - Lane.x_flow.alias("flow"), - Lane.cost_per_unit.alias("unit_cost"), - ).where(Lane.x_flow > 0.001).inspect() diff --git a/network_flow_planning/queries/network_plan.py b/network_flow_planning/queries/network_plan.py new file mode 100644 index 00000000..dacb8940 --- /dev/null +++ b/network_flow_planning/queries/network_plan.py @@ -0,0 +1,39 @@ +"""Network plan report: reads the deployed network_flow_planning model. + +Run `python -m queries.network_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and total cost of the latest refresh's solve, the +opened fulfillment centers, and the active flow on each lane. +""" + +from model import Lane, Site, SolveStatus, TotalCost, model, problem + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal network plan: termination status {status}") + total_cost = cost[0] + print(f"• objective: {round(total_cost, 9)}\n") + + print(f"\nStatus: {status[0]}") + print(f"Total cost: ${total_cost:,.2f}") + + print("\nOpened fulfillment centers:") + model.select( + Site.name.alias("fulfillment_center"), + Site.capacity.alias("capacity"), + Site.fixed_cost.alias("fixed_cost"), + ).where(Site.x_open > 0.5).inspect() + + print("\nActive flows:") + model.select( + Lane.source.name.alias("from_site"), + Lane.dest.name.alias("to_site"), + Lane.x_flow.alias("flow"), + Lane.cost_per_unit.alias("unit_cost"), + ).where(Lane.x_flow > 0.001).inspect() diff --git a/network_flow_planning/runbook.md b/network_flow_planning/runbook.md index 8b6f8a09..e18430b6 100644 --- a/network_flow_planning/runbook.md +++ b/network_flow_planning/runbook.md @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 12 sites, 17 lanes, 3 demands. Full model in `network_flow_planning.py`. +Bundled CSVs in `data/`: 12 sites, 17 lanes, 3 demands. Full model in `model/__init__.py`; the report in `queries/network_plan.py`. From b3a504be26d7e474ae35510d96a6dfa9dfd9e83b Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:37:09 -0800 Subject: [PATCH 053/162] planogram_optimization: split into a deployed model package and a query Move the model and its input-data guards into model/ (deployed with `rai models deploy --path model`) and the report into queries/facings_plan.py (`python -m queries.facings_plan`). The report's status check raises explicitly, and its demand-lookup check compares against the deployed demand table. Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- planogram_optimization/README.md | 132 ++++++++++++------ .../__init__.py} | 105 ++------------ .../queries/facings_plan.py | 103 ++++++++++++++ planogram_optimization/runbook.md | 4 +- 4 files changed, 201 insertions(+), 143 deletions(-) rename planogram_optimization/{planogram_optimization.py => model/__init__.py} (75%) create mode 100644 planogram_optimization/queries/facings_plan.py diff --git a/planogram_optimization/README.md b/planogram_optimization/README.md index 32aaca95..a47fa7c3 100644 --- a/planogram_optimization/README.md +++ b/planogram_optimization/README.md @@ -25,7 +25,7 @@ This template uses **Predictive** + **Prescriptive** reasoning: it solves the pe The same pattern applies to any predict-then-optimize problem where the prediction is a per-decision-value lookup table: workforce demand by staffing level, conversion rate by ad-spend bucket, throughput by line speed. > [!IMPORTANT] -> The bundled runner ships with a vendored `predicted_demand_table.csv` -- a stand-in for the output of any per-(SKU, k) demand regressor -- so the predict->CSP hand-off pattern can be inspected and run without a training arm. Wiring in a real regressor is documented under "Customize this template"; the CSP shape is unchanged. +> The bundled model ships with a vendored `predicted_demand_table.csv` -- a stand-in for the output of any per-(SKU, k) demand regressor -- so the predict->CSP hand-off pattern can be inspected and run without a training arm. Wiring in a real regressor is documented under "Customize this template"; the CSP shape is unchanged. ## Who this is for @@ -40,11 +40,12 @@ The same pattern applies to any predict-then-optimize problem where the predicti - An element-style decision-indexed table lookup binding `realized_demand` to the matching `PredictedDemand` row via an `implies` cascade - Shelf-length capacity (`sum(Sku.facings * Sku.width_cm).per(Shelf) <= Shelf.length_cm`) and per-category cardinality (`sum(Sku.active).per(Category)` in `[min_skus_active, max_skus_active]`) - A linear `sum(realized_demand)` objective the CSP maximizes -- A post-solve check in the report: a Python dict lookup confirming `(sku_id, facings) -> demand_units` matches the input table, and the solver status is `OPTIMAL` +- A post-solve check in the report: a Python dict lookup confirming `(sku_id, facings) -> demand_units` matches the deployed demand table, and the solver status is `OPTIMAL` ## What's included -- `planogram_optimization.py` -- main script with ontology, decisions, constraints, and solver configuration; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- ontology, decisions, constraints, and solver configuration; `rai models deploy --path model` deploys it +- `queries/facings_plan.py` -- prints the report from the deployed model - **Runbook**: `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/skus.csv` -- 18 SKUs across 4 categories (snacks, beverages, candy, household_paper) with width, max-facings, and pre-assigned shelf; `sku_id` must be unique, `category` must match a `categories.csv` name, and `assigned_shelf_id` must reference a `shelves.csv` id. The `brand` column is loaded but not used by the bundled CSP -- it is reserved for the brand-block contiguity customization described under "Customize this template" - `data/shelves.csv` -- 4 shelves (Top Eye-Level 100cm, Upper-Middle 80cm, Lower-Middle 80cm, Bottom 90cm); `id` must be unique and is the foreign-key target of `skus.assigned_shelf_id` @@ -56,7 +57,7 @@ The same pattern applies to any predict-then-optimize problem where the predicti ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -65,28 +66,61 @@ The same pattern applies to any predict-then-optimize problem where the predicti ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .PLANOGRAM_OPTIMIZATION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/planogram_optimization.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/planogram_optimization.zip unzip planogram_optimization.zip cd planogram_optimization ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -105,39 +139,27 @@ The same pattern applies to any predict-then-optimize problem where the predicti pat_user: pat_warehouse: ``` - Without the `deployment` section, `python planogram_optimization.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .PLANOGRAM_OPTIMIZATION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PLANOGRAM_OPTIMIZATION_META TO APPLICATION ; - ``` - The deploy creates the public schema `.PLANOGRAM_OPTIMIZATION` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python planogram_optimization.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .PLANOGRAM_OPTIMIZATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.facings_plan ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output (the solver maximizes total predicted weekly demand; the exact selection may vary across solver versions if multiple allocations achieve the same optimal objective). The script first prints the formulation (~30 lines, omitted here for brevity), then the solve-result block, then the per-SKU allocation, shelf utilization, and category active counts: ```text Solve result: • status: OPTIMAL @@ -179,14 +201,36 @@ The same pattern applies to any predict-then-optimize problem where the predicti 3 snacks 5 3 5 ``` + The formulation printed first (about 30 lines) is omitted here. The solver maximizes total predicted weekly demand, and the exact selection may vary across solver versions if several allocations reach the same optimal objective. The report stops unless the solve is OPTIMAL and each SKU's realized demand matches the demand table at its chosen facings. + Total weekly demand of `1656` units, 16 of 18 SKUs active. Two SKUs go inactive because the candy and household_paper categories each have 4 SKUs but `max_skus_active=3`: Mint Roll loses the candy cap (lowest predicted demand at every k), Paper Towels 6pk loses the household_paper cap (lowest predicted demand-per-cm). The bottom shelf is fully binding (90/90cm); the other three are within 1-3cm of capacity. + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .PLANOGRAM_OPTIMIZATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. + ## Template structure ```text . ├── README.md ├── pyproject.toml -├── planogram_optimization.py +├── model/ +│ └── __init__.py +├── queries/ +│ └── facings_plan.py └── data/ ├── skus.csv ├── shelves.csv @@ -194,7 +238,7 @@ The same pattern applies to any predict-then-optimize problem where the predicti └── predicted_demand_table.csv ``` -**Start here**: run `python planogram_optimization.py` for the full predict-then-optimize run end to end — it validates the data, binds each SKU's demand from the prediction table, solves the facing-count CSP, and prints the allocation — or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.facings_plan` for the full predict-then-optimize run end to end — the deploy validates the data, binds each SKU's demand from the prediction table, and solves the facing-count CSP, and the query prints the allocation — or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -213,7 +257,7 @@ The model has four concepts. The predictive output (`PredictedDemand`) is a data - **Primary identifiers**: integer `id` on `Sku` and `Shelf`; string `name` on `Category`; composite `(sku_id, facings_count)` on `PredictedDemand`. - **Important invariants**: `width_cm`, `max_facings`, and `length_cm` are non-negative integers; every `Sku.category` matches a `Category`, and every `Sku.assigned_shelf_id` matches a `Shelf`; `PredictedDemand` covers every `(sku_id, facings_count)` over `[0, max_facings]` with `demand_units = 0` at `facings_count = 0`; the CSP decisions `facings`, `realized_demand`, and `active` are integers (`active` is 0/1). -For the full concept and property definitions, see `planogram_optimization.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -227,9 +271,9 @@ skus + shelves + categories + predicted-demand table → facings / realized_dema **An `active` indicator couples to `facings` via two linear inequalities.** The per-category cardinality is a relational sum over a 0/1 `active` variable (a sum over a Boolean expression like `facings >= 1` is not a valid form). Two linear constraints pin `active = 1` whenever `facings >= 1` and `active = 0` otherwise, after which the per-category count reads as a plain `sum(active).per(Category)` with min/max bounds. Shelf capacity is likewise a plain relational inequality: facings times width, summed per shelf, cannot exceed the shelf length. -**Checks run in the report.** The `implies`-bodied lookup is solver-only; after the deploy, a Python dict re-checks each `(sku_id, facings) -> demand_units` against the input table, and an assertion confirms the solver reached `OPTIMAL`. The constraints (lookup, capacity, cardinality, active coupling) all go to the solver; the report does not re-evaluate them, and a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. +**Checks run in the report.** The `implies`-bodied lookup is solver-only; after the deploy, a Python dict re-checks each `(sku_id, facings) -> demand_units` against the deployed demand table, and the report stops unless the solver reached `OPTIMAL`. The constraints (lookup, capacity, cardinality, active coupling) all go to the solver; the report does not re-evaluate them, and a refresh started outside the report (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. -For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` reproduces the pipeline step by step with the RAI skills. +For the exact PyRel formulation, see `model/__init__.py`; `runbook.md` reproduces the pipeline step by step with the RAI skills. ## Customize this template @@ -288,7 +332,7 @@ For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` r
FileNotFoundError on a CSV -- The script resolves data paths as `Path(__file__).parent / "data"`. Run `python planogram_optimization.py` from the unzipped template root, not from a parent directory. +- Run `rai models deploy --path model` and `python -m queries.facings_plan` from the unzipped template root, not from a parent directory. - Confirm `data/` contains `skus.csv`, `shelves.csv`, `categories.csv`, and `predicted_demand_table.csv`.
@@ -296,7 +340,7 @@ For the exact PyRel formulation, see `planogram_optimization.py`; `runbook.md` r
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/planogram_optimization/planogram_optimization.py b/planogram_optimization/model/__init__.py similarity index 75% rename from planogram_optimization/planogram_optimization.py rename to planogram_optimization/model/__init__.py index 0eaf57d8..6f630786 100644 --- a/planogram_optimization/planogram_optimization.py +++ b/planogram_optimization/model/__init__.py @@ -1,6 +1,6 @@ -"""Planogram Optimization (predict-then-optimize) template. +"""Planogram Optimization (predict-then-optimize) template: the model. -This script demonstrates a predictive + prescriptive shelf-allocation problem +This package demonstrates a predictive + prescriptive shelf-allocation problem in RelationalAI: - Decide integer facing counts per SKU on each shelf. @@ -20,18 +20,13 @@ `sum(Sku.active).per(Category)`). - Capacity, cardinality, and the active-iff-facings coupling are pure relational arithmetic constraints. The implies-bodied table lookup is - solver-only; a post-solve check re-validates - `(sku_id, facings) -> demand_units` against the input table via a Python - dict lookup. + solver-only; the report re-validates `(sku_id, facings) -> demand_units` + against the deployed demand table via a Python dict lookup. Run: - `python planogram_optimization.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), and - prints the report from the deployed model. - -Output: - Prints the formulation, the solve-result block, the optimal facings per - SKU, per-shelf utilization, and per-category active counts. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.facings_plan` then prints the report from the deployed model. """ from pathlib import Path @@ -44,7 +39,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Pre-solve data invariants @@ -309,87 +304,3 @@ def _assert_shelves_cover_skus(skus_csv, shelves_csv): SolveStatus = model.Relationship(f"planogram solve status is {String:status}") TotalDemand = model.Relationship(f"planogram total demand is {Integer:demand}") model.define(SolveStatus(problem.termination_status()), TotalDemand(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - demand = model.select(TotalDemand).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - assert status == ["OPTIMAL"], f"termination status {status}" - assert len(demand) == 1, f"expected one objective value, got {demand}" - print(f"• objective: {demand[0]}\n") - - # Post-solve table-lookup check. Pull the chosen `(Sku.id, Sku.facings, - # Sku.realized_demand)` triples and confirm `realized_demand` matches the - # input table at the chosen `facings`. Catches the silent-failure mode - # where the implies cascade leaves `realized_demand` unconstrained for a - # missing (sku_id, k) row -- the pre-solve `_assert_demand_table_complete` - # guard already covers the bundled data, but this is defense in depth. - solution_df = model.select( - Sku.id.alias("sku_id"), - Sku.facings.alias("facings_count"), - Sku.realized_demand.alias("realized_demand"), - ).to_df() - expected_lookup = { - (int(row.sku_id), int(row.facings_count)): int(row.demand_units) - for row in demand_csv.itertuples(index=False) - } - mismatches = [] - for row in solution_df.itertuples(index=False): - key = (int(row.sku_id), int(row.facings_count)) - expected = expected_lookup.get(key) - if expected is None or int(row.realized_demand) != expected: - mismatches.append((*key, int(row.realized_demand), expected)) - if mismatches: - raise AssertionError( - "Demand-lookup verification failed: Sku.realized_demand does not " - "match PredictedDemand.demand_units at the chosen facings count for " - f"{len(mismatches)} SKU(s). Pre-solve guards rule out the common " - "case (missing rows in predicted_demand_table.csv); reaching this " - "check means the implies cascade did not bind realized_demand for " - "the chosen `facings`. Inspect the formulation printed by " - "`problem.display()` to confirm the demand-lookup constraint expanded " - "as expected. Mismatches " - "(sku_id, facings, realized_demand, expected_demand):\n" - f"{mismatches}" - ) - - # ---------------------------------------------- - # Inspect solution - # ---------------------------------------------- - - print("\nOptimal facings per SKU:") - model.select( - Sku.id, - Sku.name, - Sku.category, - Sku.shelf.name.alias("shelf"), - Sku.facings, - Sku.realized_demand, - ).inspect() - - print("\nShelf utilization:") - model.select( - Shelf.id, - Shelf.name, - sum(Sku.facings * Sku.width_cm).per(Shelf).alias("used_cm"), - Shelf.length_cm, - ).where(Sku.shelf == Shelf).inspect() - - print("\nCategory active counts:") - model.select( - Category.name, - sum(Sku.active).per(Category).alias("active_skus"), - Category.min_skus_active, - Category.max_skus_active, - ).where(Sku.category == Category.name).inspect() diff --git a/planogram_optimization/queries/facings_plan.py b/planogram_optimization/queries/facings_plan.py new file mode 100644 index 00000000..b26fe340 --- /dev/null +++ b/planogram_optimization/queries/facings_plan.py @@ -0,0 +1,103 @@ +"""Facings plan report: reads the deployed planogram_optimization model. + +Run `python -m queries.facings_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and objective of the latest refresh's solve, the optimal +facings per SKU, per-shelf utilization, and per-category active counts. It first checks that +each SKU's realized demand matches the predicted-demand table at its chosen facings. +""" + +from model import ( + Category, + PredictedDemand, + Shelf, + Sku, + SolveStatus, + TotalDemand, + model, + problem, +) +from relationalai.semantics import sum + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + demand = model.select(TotalDemand).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"termination status {status}") + if len(demand) != 1: + raise SystemExit(f"expected one objective value, got {demand}") + print(f"• objective: {demand[0]}\n") + + # Post-solve table-lookup check. Pull the chosen `(Sku.id, Sku.facings, + # Sku.realized_demand)` triples and confirm `realized_demand` matches the + # deployed demand table at the chosen `facings`. Catches the silent-failure mode + # where the implies cascade leaves `realized_demand` unconstrained for a + # missing (sku_id, k) row -- the pre-solve `_assert_demand_table_complete` + # guard already covers the bundled data, but this is defense in depth. + solution_df = model.select( + Sku.id.alias("sku_id"), + Sku.facings.alias("facings_count"), + Sku.realized_demand.alias("realized_demand"), + ).to_df() + demand_table_df = model.select( + PredictedDemand.sku_id.alias("sku_id"), + PredictedDemand.facings_count.alias("facings_count"), + PredictedDemand.demand_units.alias("demand_units"), + ).to_df() + expected_lookup = { + (int(row.sku_id), int(row.facings_count)): int(row.demand_units) + for row in demand_table_df.itertuples(index=False) + } + mismatches = [] + for row in solution_df.itertuples(index=False): + key = (int(row.sku_id), int(row.facings_count)) + expected = expected_lookup.get(key) + if expected is None or int(row.realized_demand) != expected: + mismatches.append((*key, int(row.realized_demand), expected)) + if mismatches: + raise SystemExit( + "Demand-lookup verification failed: Sku.realized_demand does not " + "match PredictedDemand.demand_units at the chosen facings count for " + f"{len(mismatches)} SKU(s). Pre-solve guards rule out the common " + "case (missing rows in predicted_demand_table.csv); reaching this " + "check means the implies cascade did not bind realized_demand for " + "the chosen `facings`. Inspect the formulation printed by " + "`problem.display()` to confirm the demand-lookup constraint expanded " + "as expected. Mismatches " + "(sku_id, facings, realized_demand, expected_demand):\n" + f"{mismatches}" + ) + + # ---------------------------------------------- + # Inspect solution + # ---------------------------------------------- + + print("\nOptimal facings per SKU:") + model.select( + Sku.id, + Sku.name, + Sku.category, + Sku.shelf.name.alias("shelf"), + Sku.facings, + Sku.realized_demand, + ).inspect() + + print("\nShelf utilization:") + model.select( + Shelf.id, + Shelf.name, + sum(Sku.facings * Sku.width_cm).per(Shelf).alias("used_cm"), + Shelf.length_cm, + ).where(Sku.shelf == Shelf).inspect() + + print("\nCategory active counts:") + model.select( + Category.name, + sum(Sku.active).per(Category).alias("active_skus"), + Category.min_skus_active, + Category.max_skus_active, + ).where(Sku.category == Category.name).inspect() diff --git a/planogram_optimization/runbook.md b/planogram_optimization/runbook.md index 8799a26d..91c41343 100644 --- a/planogram_optimization/runbook.md +++ b/planogram_optimization/runbook.md @@ -85,7 +85,7 @@ Each SKU has a concave demand curve over facing counts — demand rises with fac **Response** -Total predicted weekly demand around **1,600+ units** — the decision picks facings per SKU (binding each SKU's realized demand to its predicted-demand row via a table lookup), with shelf-length and category-cardinality constraints; `Sku.facings`, `Sku.realized_demand`, and `Sku.active` are written back. (MiniZinc runs to its time limit and returns a strong incumbent surfaced as OPTIMAL, so the exact objective and the specific SKUs dropped vary slightly run to run; the structure below is stable.) In `planogram_optimization.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +Total predicted weekly demand around **1,600+ units** — the decision picks facings per SKU (binding each SKU's realized demand to its predicted-demand row via a table lookup), with shelf-length and category-cardinality constraints; `Sku.facings`, `Sku.realized_demand`, and `Sku.active` are written back. (MiniZinc runs to its time limit and returns a strong incumbent surfaced as OPTIMAL, so the exact objective and the specific SKUs dropped vary slightly run to run; the structure below is stable.) In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ### 6. Read the planogram @@ -101,4 +101,4 @@ Total predicted weekly demand around **1,600+ units** — the decision picks fac ## Data -Bundled CSVs in `data/`: 18 SKUs, 4 shelves, 4 categories, 73 predicted-demand rows. The predicted demand is supplied as a table (standing in for a demand regression — see the template's customize section to swap in a trained model). Full chain in `planogram_optimization.py`. +Bundled CSVs in `data/`: 18 SKUs, 4 shelves, 4 categories, 73 predicted-demand rows. The predicted demand is supplied as a table (standing in for a demand regression — see the template's customize section to swap in a trained model). Full chain in `model/__init__.py`; the report in `queries/facings_plan.py`. From e6e8de4af9a49d27f56d691109ffe4af10a02106 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:40:15 -0800 Subject: [PATCH 054/162] pod_placement: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/placement_report.py (`python -m queries.placement_report`). The report requires a proven OPTIMAL and stops with an explicit raise instead of an assert; the README troubleshooting entry describes how to relax it. Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- pod_placement/README.md | 174 ++++++++++++------ .../{pod_placement.py => model/__init__.py} | 67 +------ pod_placement/queries/placement_report.py | 59 ++++++ pod_placement/runbook.md | 4 +- 4 files changed, 180 insertions(+), 124 deletions(-) rename pod_placement/{pod_placement.py => model/__init__.py} (86%) create mode 100644 pod_placement/queries/placement_report.py diff --git a/pod_placement/README.md b/pod_placement/README.md index 8b1a1a76..3ad5d1d3 100644 --- a/pod_placement/README.md +++ b/pod_placement/README.md @@ -46,13 +46,14 @@ This template encodes that stack as a pure constraint satisfaction and optimizat - A reified-cardinality gang-placement IC `sum(Pod.placed).per(Deployment) == Deployment.replicas * Deployment.placed` - A pairwise topology rack-clique IC for distributed-training groups -- `xa + xb <= 1` for every (Pa, Pb, Na, Nb) tuple where the pods are in the same training group and the nodes are in different racks - A linear `maximize(sum(Deployment.placed))` objective the CSP optimizes -- A `SolveStatus` check in the report: the script stops unless the solver proved `OPTIMAL` +- A `SolveStatus` check in the report: the report stops unless the solver proved `OPTIMAL` ## What's included The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the optimum -- rack-clique forces the distributed-training group onto one rack, CPU / memory / GPU bin-packing then pin it to a single node at exact 100% utilization on all three resources, anti-affinity partitions the alpha-beta and gamma-delta tenant pods across disjoint node sets, deployment co-location affinity co-locates the cache pair, and the spread cap binds for every multi-replica non-overridden deployment. Swap in your own cluster topology and workload to apply the template to a real cluster. -- `pod_placement.py` -- main script with concepts, decisions, constraints, the solver configuration, and the post-solve inspection; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- concepts, decisions, constraints, and the solver configuration; `rai models deploy --path model` deploys it +- `queries/placement_report.py` -- prints the report from the deployed model - **Runbook**: `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/nodes.csv` -- 8 nodes across 2 zones (`us-east-1a`, `us-east-1b`) and 4 racks; six general-purpose nodes (12000 millicores, 49152 MiB) and two GPU nodes (8000 millicores, 32768 MiB, 4 GPUs each). Total cluster: 88000 millicores, 360448 MiB, 8 GPUs - `data/tenants.csv` -- 4 tenants (`tenant_alpha`, `tenant_beta`, `tenant_gamma`, `tenant_delta`) @@ -67,7 +68,7 @@ The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the op ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -76,28 +77,61 @@ The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the op ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .POD_PLACEMENT_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/pod_placement.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/pod_placement.zip unzip pod_placement.zip cd pod_placement ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -116,70 +150,88 @@ The bundled CSVs are illustrative demo data tuned so multiple ICs bind at the op pat_user: pat_warehouse: ``` - Without the `deployment` section, `python pod_placement.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .POD_PLACEMENT_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .POD_PLACEMENT_META TO APPLICATION ; - ``` - The deploy creates the public schema `.POD_PLACEMENT` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python pod_placement.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .POD_PLACEMENT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.placement_report ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output (the solver maximizes the number of placed deployments; the specific (pod, node) tuples and per-node utilization may vary across solver versions when multiple placements tie at the optimum). The script first prints the formulation (a few thousand lines for the bundled data, omitted here for brevity), then the solve-result block, then the per-node utilization, the per-pod placement, and any unplaced deployments. A representative run looks like: ```text Solve result: • status: OPTIMAL • objective: 12 Per-node utilization: - node_id node cpu_used cpu_cap memory_used gpu_used - 0 1 gen-a1-n1 5500 12000 11264 0 - 1 2 gen-a1-n2 11000 12000 22528 0 - 2 3 gen-a2-n1 8500 12000 17408 0 - 3 4 gen-a2-n2 3500 12000 7168 0 - 4 5 gpu-b1-n1 5500 8000 11264 0 - 5 6 gen-b1-n2 11000 12000 22528 0 - 6 7 gpu-b2-n1 8000 8000 32768 4 - 7 8 gen-b2-n2 11000 12000 22528 0 + node_id node cpu_used cpu_cap memory_used gpu_used + 0 1 gen-a1-n1 4500 12000 9216 0 + 1 2 gen-a1-n2 8500 12000 17408 0 + 2 3 gen-a2-n1 11000 12000 22528 0 + 3 4 gen-a2-n2 4500 12000 9216 0 + 4 5 gpu-b1-n1 4500 8000 9216 0 + 5 6 gen-b1-n2 12000 12000 24576 0 + 6 7 gpu-b2-n1 8000 8000 32768 4 + 7 8 gen-b2-n2 11000 12000 22528 0 Placed pods (pod_id -> node_id): pod_id node_id ... Unplaced deployments (if any): - (empty) + Empty DataFrame + Columns: [deployment_id, deployment, replicas] + Index: [] + ``` + + The formulation (a few thousand lines for the bundled data) is omitted here, and so are all but the header of the placement table. The solver maximizes the number of placed deployments, and the specific placement and per-node utilization may vary across solver versions when several placements tie at the optimum. The report stops unless the solver proved OPTIMAL. + + The optimum places all `12` deployments and binds four ICs at the chosen GPU node: rack-clique pins `ml_gamma_train`'s 4 pods onto a single rack, and CPU / memory / GPU bin-packing then pin them all to one node within that rack at exactly 100% utilization on every resource (`gpu-b2-n1` here: 4 × 2000m = 8000m CPU = the node's CPU cap, 4 × 8192 MiB = 32768 MiB memory = the node's memory cap, 4 × 1 = 4 GPUs = the node's GPU cap -- a tight three-resource fit). + + The training group could equivalently land on `gpu-b1-n1` (same shape, other GPU rack) and the solver may tie-break either way across versions; what's solver-invariant is "all 4 pods on the same rack on a single node, 100% utilized on three resources". + + The `max_per_zone_override = 4` in `deployments.csv` is what unlocks this: without it, the default spread cap `ceil(4 / 2) = 2` plus the all-or-nothing gang rule plus the rack-clique requirement plus the GPU racks all sitting in `us-east-1b` (a rack belongs to exactly one zone, so all-pods-on-one-rack implies all-pods-in-one-zone) would make `ml_gamma_train` fully unplaced and the objective `11`. The override is the operator's explicit acceptance of a wider blast radius for the training group: a single-zone outage takes the whole training job, which is the standard tradeoff for an NVLink-tight ML deployment. + + The other bound counts that remain solver-invariant across the optimum: each multi-replica non-overridden deployment lands at exactly `ceil(replicas / 2)` pods per zone (the spread cap binds for `web_*` at 4+4, `api_*` at 2+2, `db_gamma` at 3+3), the affinity-paired caches (`cache_alpha`, `cache_delta`) co-locate on a single node, and alpha/beta and gamma/delta tenant pods partition the cluster into disjoint node sets. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .POD_PLACEMENT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - The optimum places all `12` deployments and binds **four** ICs at the chosen GPU node: rack-clique pins `ml_gamma_train`'s 4 pods onto a single rack, and CPU / memory / GPU bin-packing then pin them all to one node within that rack at exactly 100% utilization on every resource (`gpu-b2-n1` here: 4 × 2000m = 8000m CPU = the node's CPU cap, 4 × 8192 MiB = 32768 MiB memory = the node's memory cap, 4 × 1 = 4 GPUs = the node's GPU cap -- a tight three-resource fit). The training group could equivalently land on `gpu-b1-n1` (same shape, other GPU rack) and the solver may tie-break either way across versions; what's solver-invariant is "all 4 pods on the same rack on a single node, 100% utilized on three resources". The `max_per_zone_override = 4` in `deployments.csv` is what unlocks this: without it, the default spread cap `ceil(4 / 2) = 2` plus the all-or-nothing gang rule plus the rack-clique requirement plus the GPU racks all sitting in `us-east-1b` (a rack belongs to exactly one zone, so all-pods-on-one-rack implies all-pods-in-one-zone) would make `ml_gamma_train` fully unplaced and the objective `11`. The override is the operator's explicit acceptance of a wider blast radius for the training group: a single-zone outage takes the whole training job, which is the standard tradeoff for an NVLink-tight ML deployment. The other bound counts that remain solver-invariant across the optimum: each multi-replica non-overridden deployment lands at exactly `ceil(replicas / 2)` pods per zone (the spread cap binds for `web_*` at 4+4, `api_*` at 2+2, `db_gamma` at 3+3), the affinity-paired caches (`cache_alpha`, `cache_delta`) co-locate on a single node, and alpha/beta and gamma/delta tenant pods partition the cluster into disjoint node sets. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure ```text pod_placement/ - pod_placement.py # Main script (concepts, decisions, constraints, solve, inspection) + model/ + __init__.py # the deployed model (concepts, decisions, constraints, solve) + queries/ + placement_report.py # report from the deployed model data/ nodes.csv # 8 nodes across 2 zones and 4 racks tenants.csv # 4 tenants @@ -193,7 +245,7 @@ pod_placement/ pyproject.toml # dependencies ``` -**Start here**: run `python pod_placement.py` for the full formulation, solve, and post-solve inspection, or follow `runbook.md` to rebuild it step by step. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.placement_report` for the full formulation, solve, and post-solve inspection, or follow `runbook.md` to rebuild it step by step. ## Sample data @@ -209,13 +261,13 @@ The bundled CSVs are illustrative demo data for one small cluster, tuned so seve ## Model overview -The script builds one semantic model from the CSVs, then layers the decision variables and constraints on top. +The model builds one semantic model from the CSVs, then layers the decision variables and constraints on top. - **Key entities**: `Node` — a compute host in the cluster, with its resource capacities and topology placement; `Pod` — a single workload replica to be placed on a node; `Deployment` — a group of replica pods owned by one tenant; `Tenant` — an account whose deployments may be anti-affine to another tenant's; `Zone` and `Rack` — failure-domain and NVLink-island concepts, with `Rack.zone` linking each rack to its containing zone. - **Primary identifiers**: integer `id` on `Node`, `Pod`, `Deployment`, `Tenant`; string `name` on `Zone` and `Rack` (both derived from the unique `zone` / `rack` columns of `nodes.csv`). - **Important invariants**: node capacities are positive; pod row counts per `deployment_id` equal that deployment's `replicas` (the gang IC pins them); each rack belongs to exactly one zone; `max_per_zone_override`, when present, is a positive integer no greater than the row's `replicas`. -For the full concept and property definitions — including the three symmetric data relationships (`TenantAntiAffinity`, `DeploymentAffinity`, `DistributedTraining`) — see `pod_placement.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions — including the three symmetric data relationships (`TenantAntiAffinity`, `DeploymentAffinity`, `DistributedTraining`) — see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -235,9 +287,9 @@ nodes + pods + deployments + affinity / anti-affinity / training relations → 2 **Failure-domain spread is a two-key aggregate.** With `max_per_zone` precomputed (default `ceil(replicas / num_zones)`, or the row's `max_per_zone_override`), the count of a deployment's pods per zone must stay under its cap. The override is a per-deployment escape hatch: the bundled data uses it on the distributed-training job, whose rack-clique requirement forces all its pods onto one rack — and therefore one zone — which would otherwise collide with the default spread cap. -Every constraint is pure relational arithmetic (no `implies` bodies), and an assertion in the report confirms the solver proved `OPTIMAL` rather than timing out. The report does not re-evaluate the constraints, and a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. +Every constraint is pure relational arithmetic (no `implies` bodies), and the report stops unless the solver proved `OPTIMAL` rather than timing out. The report does not re-evaluate the constraints, and a refresh started outside the report (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. -For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces the model step by step with the RAI skills. +For the exact PyRel formulation, see `model/__init__.py`; `runbook.md` reproduces the model step by step with the RAI skills. ## Customize this template @@ -253,7 +305,7 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces ### Extend the model -- **Tier-aware placement.** Mark pods with `gpu_units > 0` and add an IC forcing them onto GPU-capable nodes only -- e.g. `model.where(Pod.on_node(Node, 1), Pod.gpu_units >= 1).require(Node.gpu_units >= 1)` (matches the `model.where(...).require(...)` chaining used by every IC in `pod_placement.py`). Conversely, mark some nodes as "general-purpose only" and forbid GPU workloads there. +- **Tier-aware placement.** Mark pods with `gpu_units > 0` and add an IC forcing them onto GPU-capable nodes only -- e.g. `model.where(Pod.on_node(Node, 1), Pod.gpu_units >= 1).require(Node.gpu_units >= 1)` (matches the `model.where(...).require(...)` chaining used by every IC in `model/__init__.py`). Conversely, mark some nodes as "general-purpose only" and forbid GPU workloads there. - **Migration cost under churn.** Add a `Pod.current_node` data property reflecting the existing assignment, then introduce a per-pod "moved" indicator (`moved = (1 - Pod.on_node(current_node, x))`) and minimize the total move count -- multi-solution then enumerates the N least-disruptive placements. - **NVLink-clique with explicit edges.** The bundled `data/distributed_training.csv` lists every pair in the training group, encoding rack-clique implicitly via the pairwise IC. To model a true edge-restricted clique (e.g. NVLink-2 connectivity between specific GPU pairs only), pre-compute the set of `(node_a, node_b)` pairs that lack a direct NVLink edge in Python and load them as a `NoNVLink` data relationship; then drop the `Na.rack != Nb.rack` clause from `rack_clique_ic` and replace it with `NoNVLink(Na, Nb)` to forbid co-placement on any pair of hosts without a direct NVLink path. - **Spot / preemptible tier.** Mark a subset of pods as preemptible (`Pod.is_preemptible == 1`) and let them displace lower-priority workloads when capacity is tight. Express as a per-node tiered capacity: preemptible pods consume "remaining" capacity, on-demand pods consume the primary allotment. @@ -261,7 +313,7 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces ### Scale up / productionize - **Multi-cluster (federated placement).** Add a `Cluster` concept above `Zone` and an outer `Pod.on_cluster(Cluster)` decision; spread / anti-affinity then have to lift to cluster-level constraints. Karmada-style multi-cluster placement is the standard precedent. -- The bundled cluster is small (8 nodes, 50 pods); the encoding scales to whatever the MiniZinc / Chuffed solver proves optimal within `time_limit_sec`. Raise the limit for larger clusters, or soften the post-solve `OPTIMAL` assertion (see *Troubleshooting*) to accept the best feasible placement found. +- The bundled cluster is small (8 nodes, 50 pods); the encoding scales to whatever the MiniZinc / Chuffed solver proves optimal within `time_limit_sec`. Raise the limit for larger clusters, or soften the report's `OPTIMAL` check (see *Troubleshooting*) to accept the best feasible placement found. ## Troubleshooting @@ -287,7 +339,7 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces The `Pod.deployment.tenant` chained navigation produces a runtime error on inspection - Multi-hop property navigation inside `model.select(...)` combined with a decision-variable filter (`where(Pod.on_node(Node, 1))`) can produce a query the engine refuses to evaluate within its runtime budget. The bundled inspection avoids the issue by keeping each query to single-hop navigation (`Node.id`, `Pod.id`). -- If you need richer post-solve reporting (e.g. a `(pod, deployment_name, tenant_name, node, zone, rack)` table), pull the placement into pandas and join in Python: `placements = model.select(Pod.id, Node.id).where(Pod.on_node(Node, 1)).to_df()`, then `placements.merge(pods_csv, on="pod_id").merge(deployments_csv, on="deployment_id")...`. +- If you need richer post-solve reporting (e.g. a `(pod, deployment_name, tenant_name, node, zone, rack)` table), pull the placement into pandas in the query module and join in Python: `placements = model.select(Pod.id, Node.id).where(Pod.on_node(Node, 1)).to_df()`, then `placements.merge(pods_csv, on="pod_id").merge(deployments_csv, on="deployment_id")...`. @@ -302,7 +354,7 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces
FileNotFoundError on a CSV -- The script resolves data paths as `Path(__file__).parent / "data"`. Run `python pod_placement.py` from the unzipped template root, not from a parent directory. +- Run `rai models deploy --path model` and `python -m queries.placement_report` from the unzipped template root, not from a parent directory. - Confirm `data/` contains `nodes.csv`, `tenants.csv`, `tenant_anti_affinity.csv`, `deployments.csv`, `deployment_affinity.csv`, `pods.csv`, and `distributed_training.csv`.
@@ -310,7 +362,7 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. - If you have multiple profiles, set `export RAI_PROFILE=`.
@@ -323,10 +375,10 @@ For the exact PyRel formulation, see `pod_placement.py`; `runbook.md` reproduces
- Solver hits the time limit (status is not `OPTIMAL`) and the script crashes + Solver hits the time limit (status is not `OPTIMAL`) and the report stops -- The report's `assert status == ["OPTIMAL"]` is a hard assertion: if you raise `time_limit_sec` and the solver returns a feasible-but-not-proven-optimal result (`SOLUTION_LIMIT` or `LOCALLY_SOLVED`), the script raises rather than print a partial solution. -- For a tutorial run on the bundled data this is the right posture (the oracle is "the solver proved 11/12"). For larger user data, either (a) raise `time_limit_sec` until the solver proves optimality, or (b) replace the assertion with a `print(...)` warning and let the inspection blocks run on the best feasible placement found. +- The report's `if status != ["OPTIMAL"]` check raises `SystemExit`: if you raise `time_limit_sec` and the solver returns a feasible-but-not-proven-optimal result (`SOLUTION_LIMIT` or `LOCALLY_SOLVED`), the report stops rather than print a partial solution. +- For a tutorial run on the bundled data this is the right posture (the oracle is "the solver proved 11/12"). For larger user data, either (a) raise `time_limit_sec` until the solver proves optimality, or (b) replace the check in `queries/placement_report.py` with a `print(...)` warning and let the inspection blocks run on the best feasible placement found.
diff --git a/pod_placement/pod_placement.py b/pod_placement/model/__init__.py similarity index 86% rename from pod_placement/pod_placement.py rename to pod_placement/model/__init__.py index 03b44149..97886ebe 100644 --- a/pod_placement/pod_placement.py +++ b/pod_placement/model/__init__.py @@ -1,6 +1,6 @@ -"""Pod placement (pure CSP) template. +"""Pod placement (pure CSP) template: the model. -This script demonstrates a multi-tenant Kubernetes-style pod-placement +This package demonstrates a multi-tenant Kubernetes-style pod-placement problem in RelationalAI: - Assign each ``Pod`` to a ``Node`` subject to per-node CPU, memory, @@ -46,14 +46,9 @@ - All ICs are pure relational arithmetic -- no ``implies`` bodies. Run: - `python pod_placement.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report - from the deployed model. - -Output: - Prints the formulation, the solve-result block, per-node - utilisation, the chosen (pod, node) placements, and any - unplaced deployments. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.placement_report` then prints the report from the deployed model. """ from math import ceil @@ -63,7 +58,7 @@ from relationalai.semantics import Integer, Model, String, sum from relationalai.semantics.reasoners.prescriptive import Problem -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Load CSVs and pre-compute per-deployment failure-domain spread caps. @@ -421,53 +416,3 @@ def _compute_max_per_zone(replicas, override): SolveStatus = model.Relationship(f"pod placement solve status is {String:status}") PlacedDeployments = model.Relationship(f"pod placement placed deployments is {Integer:placed}") model.define(SolveStatus(problem.termination_status()), PlacedDeployments(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. This is a hard assertion: if - # you raise `time_limit_sec` and the solver returns a feasible-but-not-proven- - # optimal result, the script raises rather than print a partial solution. - # Soften to a `print(...)` warning if you want the inspection blocks to run - # on partial solutions. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - placed = model.select(PlacedDeployments).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - assert status == ["OPTIMAL"], f"termination status {status}" - assert len(placed) == 1, f"expected one objective value, got {placed}" - print(f"• objective: {placed[0]}\n") - - # ---------------------------------------------- - # Inspect results - # ---------------------------------------------- - - print("\nPer-node utilization:") - model.select( - Node.id.alias("node_id"), - Node.name.alias("node"), - sum(Pod.cpu_millicores * x).where(Pod.on_node(Node, x)).per(Node).alias("cpu_used"), - Node.cpu_millicores.alias("cpu_cap"), - sum(Pod.memory_mib * x).where(Pod.on_node(Node, x)).per(Node).alias("memory_used"), - sum(Pod.gpu_units * x).where(Pod.on_node(Node, x)).per(Node).alias("gpu_used"), - ).inspect() - - print("\nPlaced pods (pod_id -> node_id):") - model.select( - Pod.id.alias("pod_id"), - Node.id.alias("node_id"), - ).where(Pod.on_node(Node, 1)).inspect() - - print("\nUnplaced deployments (if any):") - model.select( - Deployment.id.alias("deployment_id"), - Deployment.name.alias("deployment"), - Deployment.replicas.alias("replicas"), - ).where(Deployment.placed == 0).inspect() diff --git a/pod_placement/queries/placement_report.py b/pod_placement/queries/placement_report.py new file mode 100644 index 00000000..d83c2ca9 --- /dev/null +++ b/pod_placement/queries/placement_report.py @@ -0,0 +1,59 @@ +"""Placement report: reads the deployed pod_placement model. + +Run `python -m queries.placement_report` after `rai models deploy --path model`. Prints the +formulation, the termination status and objective of the latest refresh's solve, per-node +utilization, the chosen (pod, node) placements, and any unplaced deployments. +""" + +from model import ( + Deployment, + Node, + PlacedDeployments, + Pod, + SolveStatus, + model, + problem, + x, +) +from relationalai.semantics import sum + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status. The report requires a proven OPTIMAL. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + placed = model.select(PlacedDeployments).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"termination status {status}") + if len(placed) != 1: + raise SystemExit(f"expected one objective value, got {placed}") + print(f"• objective: {placed[0]}\n") + + # ---------------------------------------------- + # Inspect results + # ---------------------------------------------- + + print("\nPer-node utilization:") + model.select( + Node.id.alias("node_id"), + Node.name.alias("node"), + sum(Pod.cpu_millicores * x).where(Pod.on_node(Node, x)).per(Node).alias("cpu_used"), + Node.cpu_millicores.alias("cpu_cap"), + sum(Pod.memory_mib * x).where(Pod.on_node(Node, x)).per(Node).alias("memory_used"), + sum(Pod.gpu_units * x).where(Pod.on_node(Node, x)).per(Node).alias("gpu_used"), + ).inspect() + + print("\nPlaced pods (pod_id -> node_id):") + model.select( + Pod.id.alias("pod_id"), + Node.id.alias("node_id"), + ).where(Pod.on_node(Node, 1)).inspect() + + print("\nUnplaced deployments (if any):") + model.select( + Deployment.id.alias("deployment_id"), + Deployment.name.alias("deployment"), + Deployment.replicas.alias("replicas"), + ).where(Deployment.placed == 0).inspect() diff --git a/pod_placement/runbook.md b/pod_placement/runbook.md index fbf281f9..aef56c07 100644 --- a/pod_placement/runbook.md +++ b/pod_placement/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Assign pods to nodes to place as many deployments as possible, treating every rule as hard: a pod goes on at most one node; a deployment is placed only if all its replicas are; per node, total CPU, memory, and GPU requests can't exceed capacity; pods of anti-affine tenants can't share a node; affinity-paired deployments must share a node; each deployment's pods are spread so no zone holds more than its share; and the distributed-training pods must all land on the same rack. Maximize the number of placed deployments and persist the assignment to the ontology. -**Response:** OPTIMAL (MiniZinc constraint solver), objective **12 — all 12 deployments placed**. The model is a binary pod-to-node assignment matrix plus placed indicators (hundreds of variables and constraints across the six rule families); the assignment, `Pod.placed`, and `Deployment.placed` are written back to the ontology. In `pod_placement.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** OPTIMAL (MiniZinc constraint solver), objective **12 — all 12 deployments placed**. The model is a binary pod-to-node assignment matrix plus placed indicators (hundreds of variables and constraints across the six rule families); the assignment, `Pod.placed`, and `Deployment.placed` are written back to the ontology. In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the placement @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 8 nodes, 50 pods, 12 deployments, 4 tenants, plus the anti-affinity, affinity, and distributed-training rule files. Full model in `pod_placement.py`. +Bundled CSVs in `data/`: 8 nodes, 50 pods, 12 deployments, 4 tenants, plus the anti-affinity, affinity, and distributed-training rule files. Full model in `model/__init__.py`; the report in `queries/placement_report.py`. From aa21beae7255bf9f3dccb2b9cb93a643b3fcd5d3 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:42:38 -0800 Subject: [PATCH 055/162] product_configurator: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/feasible_builds.py (`python -m queries.feasible_builds`). The report now stops on a termination status other than OPTIMAL, SOLUTION_LIMIT or INFEASIBLE, the statuses the README describes. Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- product_configurator/README.md | 139 ++++++++++++------ .../__init__.py} | 67 +-------- .../queries/feasible_builds.py | 65 ++++++++ product_configurator/runbook.md | 4 +- 4 files changed, 163 insertions(+), 112 deletions(-) rename product_configurator/{product_configurator.py => model/__init__.py} (76%) create mode 100644 product_configurator/queries/feasible_builds.py diff --git a/product_configurator/README.md b/product_configurator/README.md index 1d4f1690..9617b060 100644 --- a/product_configurator/README.md +++ b/product_configurator/README.md @@ -41,7 +41,8 @@ The configurator scenario here is automotive trim, drawn from the public Renault ## What's included -- `product_configurator.py` -- main script with ontology, decisions, constraints, and solver call +- `model/__init__.py` -- ontology, decisions, constraints, and solver configuration; `rai models deploy --path model` deploys it +- `queries/feasible_builds.py` -- prints the report from the deployed model - **Runbook**: `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/slots.csv` -- 6 slots (Engine, Transmission, Trim, Sound, Wheels, Roof) - `data/options.csv` -- 16 options across the 6 slots, each with a price in integer cents @@ -54,7 +55,7 @@ The configurator scenario here is automotive trim, drawn from the public Renault ### Access - A Snowflake account that has the RAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. ### Tools @@ -63,28 +64,61 @@ The configurator scenario here is automotive trim, drawn from the public Renault ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .PRODUCT_CONFIGURATOR_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/product_configurator.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/product_configurator.zip unzip product_configurator.zip cd product_configurator ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -103,39 +137,26 @@ The configurator scenario here is automotive trim, drawn from the public Renault pat_user: pat_warehouse: ``` - Without the `deployment` section, `python product_configurator.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .PRODUCT_CONFIGURATOR_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PRODUCT_CONFIGURATOR_META TO APPLICATION ; - ``` - The deploy creates the public schema `.PRODUCT_CONFIGURATOR` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python product_configurator.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .PRODUCT_CONFIGURATOR_META._INTERNAL_REFRESH_MODEL('standard', NULL); - ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. -6. Expected output. With `MAX_CONFIGURATIONS = 100` and `TARGET_REGION = "EU"`, the solver exhausts the search and returns every distinct feasible build (status `OPTIMAL`, 63 builds), pivoted to one row per configuration and sorted ascending by total dollars. A few representative lines (the script prints all 63): +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.feasible_builds + ``` ```text Solve result: @@ -144,19 +165,39 @@ The configurator scenario here is automotive trim, drawn from the public Renault Feasible builds for region 'EU' (ceiling $20,000, up to 100 per run): solution Engine Roof Sound Transmission Trim Wheels total_$ - 60 1.6L Inline-4 Steel Roof Standard Sound Manual Base Trim 16-inch Alloy 1500 - 59 1.6L Inline-4 Steel Roof Standard Sound Manual Base Trim 18-inch Sport 2700 - ... 60 more builds omitted (script prints all 63) ... - 2 2.0L Turbo Panoramic Glass Premium Audio DCT Premium Trim 18-inch Sport 18200 + 40 1.6L Inline-4 Steel Roof Standard Sound Manual Base Trim 16-inch Alloy 1500 + 39 1.6L Inline-4 Steel Roof Standard Sound Manual Base Trim 18-inch Sport 2700 + ... 60 more builds omitted (the report prints all 63) ... + 1 2.0L Turbo Panoramic Glass Premium Audio DCT Premium Trim 18-inch Sport 18200 + ``` + + The formulation printed above the solve result is trimmed here. The solution column is the solver's index of each build and is not stable across solves, so the report sorts the builds by total price. With `TARGET_REGION = "EU"` the V6 engine is unavailable, and the 63 returned builds span $1,500-$18,200 -- every legal combination across the six slots, satisfying every implies/excludes rule. Lower `MAX_CONFIGURATIONS` to cap how many the solver returns (status flips to `SOLUTION_LIMIT` once hit). The full 63-row printout and a step-by-step walkthrough are in `runbook.md`. The report stops on a status other than OPTIMAL, SOLUTION_LIMIT, or INFEASIBLE. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .PRODUCT_CONFIGURATOR_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - With `TARGET_REGION = "EU"` the V6 engine is unavailable, and the 63 returned builds span $1,500-$18,200 -- every legal combination across the six slots, satisfying every implies/excludes rule. Lower `MAX_CONFIGURATIONS` to cap how many the solver returns (status flips to `SOLUTION_LIMIT` once hit). The full 63-row printout and a step-by-step walkthrough are in `runbook.md`. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure ```text product_configurator/ - product_configurator.py # Main script (ontology, decisions, constraints, solver configuration, deploy, inspection) + model/ + __init__.py # the deployed model (ontology, decisions, constraints, solver configuration) + queries/ + feasible_builds.py # report from the deployed model data/ slots.csv # 6 slots (Engine, Transmission, Trim, Sound, Wheels, Roof) options.csv # 16 options across the 6 slots, each priced in integer cents @@ -168,7 +209,7 @@ product_configurator/ pyproject.toml # dependencies ``` -**Start here**: run `python product_configurator.py` to enumerate every feasible build end to end, or follow `runbook.md` to rebuild it step by step. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.feasible_builds` to enumerate every feasible build end to end, or follow `runbook.md` to rebuild it step by step. ## Sample data @@ -182,7 +223,7 @@ The bundled CSVs are illustrative demo data for an automotive trim configurator, ## Model overview -The script builds a small ontology from the CSVs, then layers the binary decision and the four constraint families on top. +The model builds a small ontology from the CSVs, then layers the binary decision and the four constraint families on top. - **Key entities**: `Slot` — a configurable position on the product (e.g. engine), filled by exactly one option per build; `Option` — a choice that can fill a slot, with a price and region availability; plus the `Implies` and `Excludes` rule tables that constrain which options can combine. - **Primary identifiers**: integer `id` on `Slot` and `Option`; composite key on each rule table (`head_id` + `tail_id` on `Implies`, `left_id` + `right_id` on `Excludes`). @@ -190,7 +231,7 @@ The script builds a small ontology from the CSVs, then layers the binary decisio The two rule tables — `Implies(head_id, tail_id)` (selecting the head requires the tail) and `Excludes(left_id, right_id)` (at most one of the pair) — are concepts identified by the pair of option ids they link, and the constraint families read them directly. -For the full concept and property definitions, see `product_configurator.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -211,16 +252,16 @@ slots + options + implies + excludes + regional rules → region-filtered select - **Excludes** — for each region-allowed pair, `selected[left] + selected[right] <= 1`: at most one of the pair. - **Price ceiling** — the price-weighted sum of selected options stays within `PRICE_CEILING_CENTS`. -**4. Solve in multi-solution mode, deploy, and pivot.** Passing `solution_limit=MAX_CONFIGURATIONS` to `problem.configure(...)` makes each refresh of the deployed model run the search in enumeration mode, returning up to that many distinct feasible builds; the report prints the status and the number of builds, and does not re-evaluate the constraints. The variable handle exposes a `.values(solution_index, value)` relationship over the per-solution outputs; binding the value slot to `1` surfaces just the picked options, and a `pandas.pivot` collapses the per-option rows into one row per build (one column per slot, plus a dollar total, sorted ascending by price) for buyer-facing display. +**4. Solve in multi-solution mode, deploy, and pivot.** Passing `solution_limit=MAX_CONFIGURATIONS` to `problem.configure(...)` makes each refresh of the deployed model run the search in enumeration mode, returning up to that many distinct feasible builds; the report prints the status and the number of builds, stops on a status other than `OPTIMAL`, `SOLUTION_LIMIT`, or `INFEASIBLE`, and does not re-evaluate the constraints. The variable handle exposes a `.values(solution_index, value)` relationship over the per-solution outputs; binding the value slot to `1` surfaces just the picked options, and a `pandas.pivot` collapses the per-option rows into one row per build (one column per slot, plus a dollar total, sorted ascending by price) for buyer-facing display. -For the exact PyRel formulation, see `product_configurator.py`; `runbook.md` reproduces the model step by step with the RAI skills. +For the exact PyRel formulation, see `model/__init__.py`; `runbook.md` reproduces the model step by step with the RAI skills. ## Customize this template ### Use your own data - Replace the five CSV files with your slots, options, implies, excludes, and regional_rules tables. The constraint structure does not change. -- **Add a new region** by adding rows to `regional_rules.csv` for the new region and changing `TARGET_REGION` in the runner. +- **Add a new region** by adding rows to `regional_rules.csv` for the new region and changing `TARGET_REGION` in `model/__init__.py`. ### Tune parameters @@ -250,7 +291,7 @@ For the exact PyRel formulation, see `product_configurator.py`; `runbook.md` rep
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/product_configurator/product_configurator.py b/product_configurator/model/__init__.py similarity index 76% rename from product_configurator/product_configurator.py rename to product_configurator/model/__init__.py index 3cf244d3..17fab011 100644 --- a/product_configurator/product_configurator.py +++ b/product_configurator/model/__init__.py @@ -1,4 +1,4 @@ -"""Product Configurator (constraint satisfaction, multi-solution) template. +"""Product Configurator (constraint satisfaction, multi-solution) template: the model. For one target region, enumerate every feasible build of a configurable product: pick one option per slot subject to implies/excludes feature-model @@ -9,14 +9,9 @@ playbook, and a trade-off slider all need the population of configurations. Run: - `python product_configurator.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), and - prints the report from the deployed model. - -Output: - Prints the solve status, the number of builds, and every feasible build as a - pivot table (one row per build, one column per slot, sorted ascending - by total price). + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve. + `python -m queries.feasible_builds` then prints the report from the deployed model. """ from pathlib import Path @@ -37,7 +32,7 @@ # production catalog and the solver returns SOLUTION_LIMIT when it's hit. MAX_CONFIGURATIONS = 100 -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -156,7 +151,7 @@ def _validate_regional_catalog(slots_csv, options_csv, regional_csv, implies_csv # `where=[Option.allowed_in(TARGET_REGION)]` filters the decision domain so # banned options never get a decision variable at all. The ICs below re-assert # the region filter so they stay correctly scoped if you later parameterize -# this script over multiple regions in one solve. +# this model over multiple regions in one solve. selected_var = problem.solve_for( Option.selected, type="bin", @@ -214,53 +209,3 @@ def _validate_regional_catalog(slots_csv, options_csv, regional_csv, implies_csv # The solve's status, published through the model's own relationship. SolveStatus = model.Relationship(f"product configurator solve status is {String:status}") model.define(SolveStatus(problem.termination_status())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and number of solutions of the latest refresh's solve. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - si = problem.solve_info() - print(f"\nSolve result:\n• status: {', '.join(status)}\n• num_points: {si.num_points}\n") - - if si.num_points is None or si.num_points == 0: - print( - f"\nNo feasible build for region {TARGET_REGION!r} under the encoded " - "constraints. See the README troubleshooting section for likely causes." - ) - else: - # ------------------------------------------ - # Inspect every feasible configuration - # ------------------------------------------ - - # `selected_var.values(sol_idx, 1)` indexes the per-solution outputs and - # filters to the options the solver picked. Pivot to one row per build, - # one column per slot, sorted ascending by total dollars. - sol_idx = Integer.ref() - selections_df = ( - model.select( - sol_idx.alias("solution"), - selected_var.option.slot.name.alias("slot"), - selected_var.option.name.alias("option"), - selected_var.option.price_cents.alias("price_cents"), - ) - .where(selected_var.values(sol_idx, 1)) - .to_df() - .astype({"solution": "int64", "price_cents": "int64"}) - ) - build_view = selections_df.pivot(index="solution", columns="slot", values="option") - build_view["total_$"] = selections_df.groupby("solution")["price_cents"].sum() // 100 - build_view = build_view.sort_values("total_$").reset_index() - build_view.columns.name = None - print( - f"\nFeasible builds for region {TARGET_REGION!r} " - f"(ceiling ${PRICE_CEILING_CENTS // 100:,}, up to {MAX_CONFIGURATIONS} per run):" - ) - print(build_view.to_string(index=False)) diff --git a/product_configurator/queries/feasible_builds.py b/product_configurator/queries/feasible_builds.py new file mode 100644 index 00000000..c4805573 --- /dev/null +++ b/product_configurator/queries/feasible_builds.py @@ -0,0 +1,65 @@ +"""Feasible builds report: reads the deployed product_configurator model. + +Run `python -m queries.feasible_builds` after `rai models deploy --path model`. Prints the +formulation, the solve status and the number of builds, and every feasible build as a pivot +table (one row per build, one column per slot, sorted ascending by total price). The report +stops on a status other than OPTIMAL (every build enumerated), SOLUTION_LIMIT (the cap was +hit) or INFEASIBLE (no build exists, reported below). +""" + +from model import ( + MAX_CONFIGURATIONS, + PRICE_CEILING_CENTS, + TARGET_REGION, + SolveStatus, + model, + problem, + selected_var, +) +from relationalai.semantics import Integer + +if __name__ == "__main__": + problem.display() + + # Status and number of solutions of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + if status not in (["OPTIMAL"], ["SOLUTION_LIMIT"], ["INFEASIBLE"]): + raise SystemExit(f"Unexpected termination status {status}") + si = problem.solve_info() + print(f"\nSolve result:\n• status: {', '.join(status)}\n• num_points: {si.num_points}\n") + + if si.num_points is None or si.num_points == 0: + print( + f"\nNo feasible build for region {TARGET_REGION!r} under the encoded " + "constraints. See the README troubleshooting section for likely causes." + ) + else: + # ------------------------------------------ + # Inspect every feasible configuration + # ------------------------------------------ + + # `selected_var.values(sol_idx, 1)` indexes the per-solution outputs and + # filters to the options the solver picked. Pivot to one row per build, + # one column per slot, sorted ascending by total dollars. + sol_idx = Integer.ref() + selections_df = ( + model.select( + sol_idx.alias("solution"), + selected_var.option.slot.name.alias("slot"), + selected_var.option.name.alias("option"), + selected_var.option.price_cents.alias("price_cents"), + ) + .where(selected_var.values(sol_idx, 1)) + .to_df() + .astype({"solution": "int64", "price_cents": "int64"}) + ) + build_view = selections_df.pivot(index="solution", columns="slot", values="option") + build_view["total_$"] = selections_df.groupby("solution")["price_cents"].sum() // 100 + build_view = build_view.sort_values("total_$").reset_index() + build_view.columns.name = None + print( + f"\nFeasible builds for region {TARGET_REGION!r} " + f"(ceiling ${PRICE_CEILING_CENTS // 100:,}, up to {MAX_CONFIGURATIONS} per run):" + ) + print(build_view.to_string(index=False)) diff --git a/product_configurator/runbook.md b/product_configurator/runbook.md index 80a29c95..c831b936 100644 --- a/product_configurator/runbook.md +++ b/product_configurator/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Enumerate every legal configuration for the EU region that costs at most $20,000. Use a binary "pick this option" decision over the EU-allowed options, require exactly one option per slot, enforce the implies rules (if the head is picked, the tail must be) and the excludes rules (the two can't both be picked), and keep total price within the ceiling. There's no objective — return the full set of feasible builds in multi-solution mode. -**Response:** OPTIMAL — search exhausted (MiniZinc constraint solver in multi-solution mode). **63 distinct feasible builds** found. Because the EU filter drops the V6, the model has 15 binary option variables and 13 constraints (the V6/Manual exclusion drops out with the V6). The selection is `Option.selected`, with the first solution persisted to the ontology. In `product_configurator.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** OPTIMAL — search exhausted (MiniZinc constraint solver in multi-solution mode). **63 distinct feasible builds** found. Because the EU filter drops the V6, the model has 15 binary option variables and 13 constraints (the V6/Manual exclusion drops out with the V6). The selection is `Option.selected`, with the first solution persisted to the ontology. In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the configuration population @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 6 slots, 16 options, 4 implies rules, 3 excludes rules, regional availability. The target region (EU) and price ceiling ($20,000) are constants in the script. Full model in `product_configurator.py`. +Bundled CSVs in `data/`: 6 slots, 16 options, 4 implies rules, 3 excludes rules, regional availability. The target region (EU) and price ceiling ($20,000) are constants in the model. Full model in `model/__init__.py`; the report in `queries/feasible_builds.py`. From 6a65b6d870cfb47feec501e62294c2b04fb70ad6 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:29:07 -0800 Subject: [PATCH 056/162] traveling_salesman: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/tour.py (`python -m queries.tour`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. The report's size check compares the solver's variable count with the deployed model's distinct edges and nodes instead of CSV row counts, its checks raise explicit errors, and it stops unless the solve status is OPTIMAL. --- traveling_salesman/README.md | 133 +++++++++++------- .../__init__.py} | 47 ++----- traveling_salesman/queries/tour.py | 52 +++++++ traveling_salesman/runbook.md | 4 +- 4 files changed, 150 insertions(+), 86 deletions(-) rename traveling_salesman/{traveling_salesman.py => model/__init__.py} (67%) create mode 100644 traveling_salesman/queries/tour.py diff --git a/traveling_salesman/README.md b/traveling_salesman/README.md index fd589a53..ef7e1ca3 100644 --- a/traveling_salesman/README.md +++ b/traveling_salesman/README.md @@ -39,7 +39,8 @@ Built using **prescriptive reasoning** (mixed-integer programming with the HiGHS ## What's included -- `traveling_salesman.py` -- main script with ontology, MTZ formulation, and solver call +- `model/__init__.py` -- ontology, MTZ formulation, and solver configuration; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/tour.py` -- prints the report from the deployed model: the solve status, the tour distance, and the tour edges - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/edges.csv` -- 12 directed edges between 4 nodes with distances - `pyproject.toml` -- Python package configuration @@ -57,28 +58,60 @@ Built using **prescriptive reasoning** (mixed-integer programming with the HiGHS ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .TRAVELING_SALESMAN_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/traveling_salesman.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/traveling_salesman.zip unzip traveling_salesman.zip cd traveling_salesman ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: ```yaml reasoners: @@ -99,65 +132,71 @@ Built using **prescriptive reasoning** (mixed-integer programming with the HiGHS pat_warehouse: ``` - Without the `deployment` section, `python traveling_salesman.py` stops with an error naming the missing setting. +5. **Deploy the model** - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .TRAVELING_SALESMAN_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .TRAVELING_SALESMAN_META TO APPLICATION ; + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.TRAVELING_SALESMAN` if it doesn't exist. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: -5. Run: ```bash - python traveling_salesman.py + python -m queries.tour + ``` + + ```text + HiGHS: OPTIMAL, tour distance=8.5 + + Tour edges: + from to + 0 1 2 + 1 2 4 + 2 3 1 + 3 4 3 ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + The formulation printed above the status line is trimmed here. Before printing, the report checks the problem's size against the deployed model's distinct edges and nodes (one variable per edge and per node, one minimization objective), and it stops unless the termination status is `OPTIMAL`. The tour is 1, 2, 4, 3 and back to 1, for a total distance of 8.5. See `runbook.md` for the step-by-step walkthrough. + + To solve again without changing the model, refresh the deployed model, then run the query again: + ```sql CALL .TRAVELING_SALESMAN_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output: - ```text - Status: OPTIMAL - Tour distance: 8.50 +7. **Tear down the deployment** - Tour edges: - from to - 1 2 - 2 4 - 4 3 - 3 1 + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. + ## Template structure ```text traveling_salesman/ ├── README.md # this file ├── pyproject.toml # dependencies -├── traveling_salesman.py # main script (ontology, MTZ formulation, solve) +├── model/ +│ └── __init__.py # the deployed model (ontology, MTZ formulation, solve) +├── queries/ +│ └── tour.py # report from the deployed model └── data/ └── edges.csv # 12 directed edges between 4 nodes, with distances ``` -**Start here:** run `python traveling_salesman.py` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here:** deploy with `rai models deploy --path model` and run `python -m queries.tour` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -171,11 +210,11 @@ The model has two concepts: the `Edge` rows loaded from CSV, and the `Node` set - **Primary identifiers**: an `Edge` is identified by its endpoint pair `(i, j)`; a `Node` is identified by its index `v`. - **Important invariants**: distances are non-negative; every node needs exactly one in-edge and one out-edge; the selected edges must form a single cycle (no subtours). -For the full concept and property definitions, see `traveling_salesman.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works -The script loads the directed edges with their distances from CSV, then derives the `Node` set from the edge endpoints so nodes never need their own file. On the ontology it lays down the decision variables the solver will fill in: a binary variable per edge (1 if the edge is on the tour) and an integer ordering value per node used for subtour elimination. +The model loads the directed edges with their distances from CSV, then derives the `Node` set from the edge endpoints so nodes never need their own file. On the ontology it lays down the decision variables the solver will fill in: a binary variable per edge (1 if the edge is on the tour) and an integer ordering value per node used for subtour elimination. Two families of constraints shape the tour. Degree constraints require exactly one incoming and one outgoing selected edge at every node, so the route enters and leaves each stop once. Miller-Tucker-Zemlin (MTZ) constraints tie the ordering values together: whenever an edge is selected, the destination's order must exceed the source's, which is what rules out disconnected subtours and forces a single connected cycle. The objective minimizes the total distance of the selected edges, and the HiGHS solver returns the optimal tour. @@ -183,13 +222,13 @@ Two families of constraints shape the tour. Degree constraints require exactly o edges.csv → load edges → derive nodes → edge + ordering variables → degree + MTZ constraints → minimize distance → optimal tour ``` -See `traveling_salesman.py` for the implementation, and `runbook.md` to reproduce it step by step with the RAI skills. +See `model/__init__.py` for the implementation, and `runbook.md` to reproduce it step by step with the RAI skills. ## Customize this template ### Use your own data -- Replace `data/edges.csv` with your own city-to-city distance matrix, as a list of directed edges with the columns `i`, `j`, and `dist`, or change the `read_csv(...)` path in the script. +- Replace `data/edges.csv` with your own city-to-city distance matrix, as a list of directed edges with the columns `i`, `j`, and `dist`, or change the `read_csv(...)` path in `model/__init__.py`. - Provide an edge in both directions for every pair of nodes so the graph stays strongly connected. Asymmetric costs are supported: give the forward and reverse edges different `dist` values. ### Tune parameters @@ -227,7 +266,7 @@ See `traveling_salesman.py` for the implementation, and `runbook.md` to reproduc
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4). - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/traveling_salesman/traveling_salesman.py b/traveling_salesman/model/__init__.py similarity index 67% rename from traveling_salesman/traveling_salesman.py rename to traveling_salesman/model/__init__.py index 43f70649..3a738e65 100644 --- a/traveling_salesman/traveling_salesman.py +++ b/traveling_salesman/model/__init__.py @@ -1,6 +1,6 @@ -"""Traveling salesman (prescriptive optimization) template. +"""Traveling salesman (prescriptive optimization) template: the model. -This script demonstrates a mixed-integer linear optimization (MILP) problem +This package demonstrates a mixed-integer linear optimization (MILP) problem in RelationalAI: - Load a directed distance matrix from CSV. @@ -9,13 +9,9 @@ - Solve for the shortest Hamiltonian cycle and print the selected edges. Run: - `python traveling_salesman.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from - the deployed model. - -Output: - Prints the solver termination status, objective value (shortest tour - distance), and a table of selected edges. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.tour` then prints the report from the deployed model. """ from pathlib import Path @@ -28,7 +24,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -54,6 +50,10 @@ node_count = model.Relationship(f"node count is {Integer}") model.define(node_count(count(Node))) +# Rule: edge count, the number of distinct (i, j) edges +edge_count = model.Relationship(f"edge count is {Integer}") +model.define(edge_count(count(Edge))) + # -------------------------------------------------- # Model the decision problem # -------------------------------------------------- @@ -108,30 +108,3 @@ model.define(SolveStatus(problem.termination_status()), TourDistance(problem.objective_value())) num_variables = problem.num_variables() num_min_objectives = problem.num_min_objectives() - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - # Check model structure: one variable per edge and per node, one minimization objective. - nodes = sorted(edge_csv["i"].unique().tolist()) - variables = model.select(num_variables).to_df().iloc[:, 0].tolist() - assert variables == [len(edge_csv) + len(nodes)], f"expected one variable per edge and node, got {variables}" - objectives = model.select(num_min_objectives).to_df().iloc[:, 0].tolist() - assert objectives == [1], f"expected one minimization objective, got {objectives}" - problem.display() - - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - distance = model.select(TourDistance).to_df().iloc[:, 0].tolist() - print(f"HiGHS: {status[0]}, tour distance={distance[0]:.1f}") - - # Extract solution: the tour leaves every node exactly once. - tour = model.select(Edge.i.alias("from"), Edge.j.alias("to")).where(Edge.x > 0.5).to_df() - assert sorted(tour["from"].tolist()) == nodes, f"tour does not leave every node exactly once: {tour}" - print("\nTour edges:") - print(tour) - print() diff --git a/traveling_salesman/queries/tour.py b/traveling_salesman/queries/tour.py new file mode 100644 index 00000000..c62af62a --- /dev/null +++ b/traveling_salesman/queries/tour.py @@ -0,0 +1,52 @@ +"""Tour report: reads the deployed traveling_salesman model. + +Run `python -m queries.tour` after `rai models deploy --path model`. Checks the problem's +size against the deployed edges and nodes, prints the formulation, the termination status +and tour distance of the latest refresh's solve, and the edges of the shortest tour. A +failed check stops the report with an error. +""" + +from model import ( + Edge, + Node, + SolveStatus, + TourDistance, + edge_count, + model, + node_count, + num_min_objectives, + num_variables, + problem, +) + +if __name__ == "__main__": + # Check model structure: one variable per edge and per node, one minimization objective. + variables = model.select(num_variables).to_df().iloc[:, 0].tolist() + edges = model.select(edge_count).to_df().iloc[:, 0].tolist() + nodes_count = model.select(node_count).to_df().iloc[:, 0].tolist() + if len(edges) != 1 or len(nodes_count) != 1 or variables != [edges[0] + nodes_count[0]]: + raise SystemExit( + f"expected one variable per edge and node, got {variables} for edges {edges}, " + f"nodes {nodes_count}" + ) + objectives = model.select(num_min_objectives).to_df().iloc[:, 0].tolist() + if objectives != [1]: + raise SystemExit(f"expected one minimization objective, got {objectives}") + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + distance = model.select(TourDistance).to_df().iloc[:, 0].tolist() + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal tour: termination status {status}") + print(f"HiGHS: {status[0]}, tour distance={distance[0]:.1f}") + + # Extract solution: the tour leaves every node exactly once. + nodes = sorted(model.select(Node.v).to_df().iloc[:, 0].tolist()) + tour = model.select(Edge.i.alias("from"), Edge.j.alias("to")).where(Edge.x > 0.5).to_df() + if sorted(tour["from"].tolist()) != nodes: + raise SystemExit(f"tour does not leave every node exactly once: {tour}") + print("\nTour edges:") + print(tour) + print() diff --git a/traveling_salesman/runbook.md b/traveling_salesman/runbook.md index 4ba7c1fa..5514ae08 100644 --- a/traveling_salesman/runbook.md +++ b/traveling_salesman/runbook.md @@ -37,7 +37,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem What's the shortest round-trip that visits every node exactly once and returns to the start, as one connected tour with no disconnected subtours? Use a binary decision per edge for whether it's in the tour, require every node to have exactly one selected edge in and one out, and eliminate subtours with MTZ ordering variables — one integer position per node, anchored at node 1. Minimize total distance, and persist the edge selection to the ontology. -**Response:** OPTIMAL (HiGHS mixed-integer program), shortest tour distance **8.5** (relative gap 0.0). The model has 16 variables (12 binary edge selections + 4 integer MTZ positions) and 15 constraints; the edge decision is written back as `Edge.x` and the position as `Node.u`. In `traveling_salesman.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** OPTIMAL (HiGHS mixed-integer program), shortest tour distance **8.5** (relative gap 0.0). The model has 16 variables (12 binary edge selections + 4 integer MTZ positions) and 15 constraints; the edge decision is written back as `Edge.x` and the position as `Node.u`. In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the optimal route @@ -47,4 +47,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSV in `data/`: 12 directed edges (distances among 4 nodes). Full model in `traveling_salesman.py`. +Bundled CSV in `data/`: 12 directed edges (distances among 4 nodes). Full model in `model/__init__.py`; the report in `queries/tour.py`. From cd8356e0a8cf3b26db8d56601f271d4dd78f3464 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:29:08 -0800 Subject: [PATCH 057/162] water_allocation: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/flow_allocations.py (`python -m queries.flow_allocations`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. The report now stops unless the solve status is OPTIMAL or LOCALLY_SOLVED. --- water_allocation/README.md | 132 ++++++++++++------ .../__init__.py} | 38 +---- water_allocation/queries/flow_allocations.py | 27 ++++ water_allocation/runbook.md | 4 +- 4 files changed, 122 insertions(+), 79 deletions(-) rename water_allocation/{water_allocation.py => model/__init__.py} (79%) create mode 100644 water_allocation/queries/flow_allocations.py diff --git a/water_allocation/README.md b/water_allocation/README.md index 253e75ba..519a7121 100644 --- a/water_allocation/README.md +++ b/water_allocation/README.md @@ -38,7 +38,8 @@ Built using **prescriptive reasoning** (nonlinear program over continuous flow v ## What's included - **Model**: a single-stage nonlinear optimization on a shared ontology — `Source`, `User`, and `Connection` concepts wired to the bundled CSVs. -- **Runner**: `water_allocation.py` — a single Python script that builds the model, constraints, and objective, deploys the model to your Snowflake account, where each refresh calls the solver, and prints the report from the deployed model. +- **Deployment**: `model/__init__.py` — builds the model, constraints, and objective; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh calls the solver. +- **Query**: `queries/flow_allocations.py` — prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: a small water-distribution network of sources, user groups, and the connections between them. See *Sample data* below. - **Outputs**: solver status, total cost, and the per-connection flow allocations. @@ -56,28 +57,60 @@ Built using **prescriptive reasoning** (nonlinear program over continuous flow v ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .WATER_ALLOCATION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/water_allocation.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/water_allocation.zip unzip water_allocation.zip cd water_allocation ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: ```yaml reasoners: @@ -98,58 +131,67 @@ Built using **prescriptive reasoning** (nonlinear program over continuous flow v pat_warehouse: ``` - Without the `deployment` section, `python water_allocation.py` stops with an error naming the missing setting. +5. **Deploy the model** - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .WATER_ALLOCATION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .WATER_ALLOCATION_META TO APPLICATION ; + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.WATER_ALLOCATION` if it doesn't exist. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. -5. Run: - ```bash - python water_allocation.py - ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .WATER_ALLOCATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.flow_allocations ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output (a few lines confirm a successful run): ```text Status: LOCALLY_SOLVED Total cost: $853.39 Flow allocations: - Reservoir_A Municipal 400.00 - Reservoir_B Agricultural 182.06 - ... + source user flow + 0 Reservoir_A Agricultural 352.769782 + 1 Reservoir_A Industrial 247.230225 + 2 Reservoir_A Municipal 400.000004 + 3 Reservoir_B Agricultural 182.063671 + 4 Reservoir_B Industrial 177.934136 + 5 Reservoir_B Municipal 228.989014 + ``` + + The formulation printed above the status line is trimmed here. The report stops unless the termination status is `OPTIMAL` or `LOCALLY_SOLVED`; Ipopt returns `LOCALLY_SOLVED` for a nonlinear program it solved to a local optimum. See `runbook.md` for the step-by-step walkthrough. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .WATER_ALLOCATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - The full printout and a step-by-step walkthrough are in `runbook.md`. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure ```text water_allocation/ - water_allocation.py # Main script (network model, constraints, solver call) + model/ + __init__.py # The deployed model (network model, constraints, solver call) + queries/ + flow_allocations.py # Report from the deployed model data/ sources.csv # Water sources with capacity and cost per unit users.csv # User groups with demand and priority @@ -159,7 +201,7 @@ water_allocation/ pyproject.toml # dependencies ``` -**Start here**: run `python water_allocation.py` for the full solve end to end, or follow `runbook.md` to rebuild it step by step. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.flow_allocations` for the full solve end to end, or follow `runbook.md` to rebuild it step by step. ## Sample data @@ -177,13 +219,13 @@ A single shared ontology holds the network. The `Connection` concept carries the - **Primary identifiers**: integer `id` on `Source` and `User`; `Connection` is identified by its `source` and `user` endpoints. - **Important invariants**: `capacity`, `demand`, `max_flow`, and `cost_per_unit` are non-negative; `loss_rate` is a fraction in `[0, 1]`; each connection's flow is continuous and bounded between 0 and its `max_flow`. -For the full concept and property definitions, see `water_allocation.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works This is a network-flow problem: source-capacity limits, per-connection flow bounds, and demand constraints whose nonlinear loss term makes it quadratic, solved with the Ipopt nonlinear solver. -The script loads three concepts from CSV — sources with a capacity and an extraction cost, users with a demand, and connections that link a source to a user with a maximum flow and a loss rate. Each connection then gets a continuous flow decision variable, bounded between zero and its maximum flow, that the solver assigns. +The model loads three concepts from CSV — sources with a capacity and an extraction cost, users with a demand, and connections that link a source to a user with a maximum flow and a loss rate. Each connection then gets a continuous flow decision variable, bounded between zero and its maximum flow, that the solver assigns. Two families of constraints shape the plan. A source-capacity constraint holds the total outflow from each source within its capacity. A demand constraint requires each user's effective inflow — the flow delivered after losses — to meet its demand. The loss term is where the nonlinearity enters: the fraction lost on a connection grows with how hard the route is being pushed, so effective delivery falls as a connection approaches its maximum flow. That utilization-dependent loss makes the demand constraint quadratic, which is why the model needs the Ipopt nonlinear solver rather than a linear one. The objective minimizes total extraction cost across all flows, so the solver leans on cheaper sources and lightly loaded routes. @@ -191,7 +233,7 @@ Two families of constraints shape the plan. A source-capacity constraint holds t sources/users/connections CSVs → per-connection flow variables → capacity + loss-adjusted demand constraints → minimize cost → allocation plan ``` -See `water_allocation.py` for the implementation, and `runbook.md` to reproduce it step by step with the RAI skills. +See `model/__init__.py` for the implementation, and `runbook.md` to reproduce it step by step with the RAI skills. ## Customize this template @@ -243,7 +285,7 @@ Make sure you activated the virtual environment and ran `python -m pip install .
Connection or authentication errors -Run `rai init` to configure your Snowflake connection. Verify that your account has the RAI Native App installed and that your user has the required permissions. +Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. Verify that your account has the RAI Native App installed and that your user has the required permissions.
## Learn more diff --git a/water_allocation/water_allocation.py b/water_allocation/model/__init__.py similarity index 79% rename from water_allocation/water_allocation.py rename to water_allocation/model/__init__.py index e5d4aced..eb4bb3a4 100644 --- a/water_allocation/water_allocation.py +++ b/water_allocation/model/__init__.py @@ -1,6 +1,6 @@ -"""Water Allocation (nonlinear optimization) template. +"""Water Allocation (nonlinear optimization) template: the model. -This script demonstrates a nonlinear optimization model solved with Ipopt +This package demonstrates a nonlinear optimization model solved with Ipopt in RelationalAI: - Load sample CSVs describing water sources, users (demand points), and connections. @@ -16,13 +16,9 @@ optimal solutions for smooth NLP problems with continuous variables. Run: - `python water_allocation.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from - the deployed model. - -Output: - Prints the solver termination status, objective value, and a table of - non-trivial flow allocations. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.flow_allocations` then prints the report from the deployed model. """ from pathlib import Path @@ -35,7 +31,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -125,25 +121,3 @@ SolveStatus = model.Relationship(f"water allocation solve status is {String:status}") TotalCost = model.Relationship(f"water allocation total cost is {Float:cost}") model.define(SolveStatus(problem.termination_status()), TotalCost(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() - print(f"Status: {status[0]}") # Ipopt returns LOCALLY_SOLVED - print(f"Total cost: ${cost[0]:.2f}") - - print("\nFlow allocations:") - model.select( - Connection.source.name.alias("source"), - Connection.user.name.alias("user"), - Connection.x_flow - ).where(Connection.x_flow > 0.001).inspect() diff --git a/water_allocation/queries/flow_allocations.py b/water_allocation/queries/flow_allocations.py new file mode 100644 index 00000000..5bfeecdd --- /dev/null +++ b/water_allocation/queries/flow_allocations.py @@ -0,0 +1,27 @@ +"""Flow allocation report: reads the deployed water_allocation model. + +Run `python -m queries.flow_allocations` after `rai models deploy --path model`. Prints the +formulation, the termination status and total cost of the latest refresh's solve, and the +non-trivial flow on each connection. +""" + +from model import Connection, SolveStatus, TotalCost, model, problem + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. Ipopt returns LOCALLY_SOLVED. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalCost).to_df().iloc[:, 0].tolist() + if status not in (["OPTIMAL"], ["LOCALLY_SOLVED"]): + raise SystemExit(f"No usable water allocation: termination status {status}") + print(f"Status: {status[0]}") + print(f"Total cost: ${cost[0]:.2f}") + + print("\nFlow allocations:") + model.select( + Connection.source.name.alias("source"), + Connection.user.name.alias("user"), + Connection.x_flow + ).where(Connection.x_flow > 0.001).inspect() diff --git a/water_allocation/runbook.md b/water_allocation/runbook.md index 8d30455b..46e3ff30 100644 --- a/water_allocation/runbook.md +++ b/water_allocation/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem What non-negative flow on each connection meets every user's demand at minimum total sourcing cost, given that each source's total outflow can't exceed its capacity and that transmission loss grows with utilization — the water delivered on a connection is flow times (1 minus loss_rate times flow over max_flow)? That delivered-water term is quadratic, so this needs a nonlinear solver. Persist the chosen flows to the ontology. -**Response:** LOCALLY_SOLVED (Ipopt nonlinear solver), total cost **$853.39**. 9 continuous flow variables (each bounded by its connection's max flow), 6 constraints (3 source-capacity, 3 nonlinear demand-satisfaction). Flows are written back as `Connection.x_flow`. In `water_allocation.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** LOCALLY_SOLVED (Ipopt nonlinear solver), total cost **$853.39**. 9 continuous flow variables (each bounded by its connection's max flow), 6 constraints (3 source-capacity, 3 nonlinear demand-satisfaction). Flows are written back as `Connection.x_flow`. In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the allocation @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 3 sources with capacity and cost, 3 users with demand, 9 connections with max-flow and loss-rate. Full model in `water_allocation.py`. +Bundled CSVs in `data/`: 3 sources with capacity and cost, 3 users with demand, 9 connections with max-flow and loss-rate. Full model in `model/__init__.py`; the report in `queries/flow_allocations.py`. From 0293513888ee9fa5427dca89160e1bb6ca8cc04b Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:31:09 -0800 Subject: [PATCH 058/162] production_planning: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/production_plan.py (`python -m queries.production_plan`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- production_planning/README.md | 126 ++++++++++++------ .../__init__.py} | 48 +------ .../queries/production_plan.py | 32 +++++ production_planning/runbook.md | 2 +- 4 files changed, 123 insertions(+), 85 deletions(-) rename production_planning/{production_planning.py => model/__init__.py} (75%) create mode 100644 production_planning/queries/production_plan.py diff --git a/production_planning/README.md b/production_planning/README.md index 362a3974..14a6ac16 100644 --- a/production_planning/README.md +++ b/production_planning/README.md @@ -36,7 +36,8 @@ This template finds the profit-maximizing production plan across a set of machin ## What's included -- `production_planning.py` -- Main script with the scenario model, constraints, objective, and result summary; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- The scenario model, constraints, and objective; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/production_plan.py` -- Prints the report from the deployed model: the solve's status and objective and the production plan for each scenario - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/products.csv` -- Products with base demand and per-unit profit margins - `data/machines.csv` -- Machines with available hours per planning period @@ -55,28 +56,61 @@ This template finds the profit-maximizing production plan across a set of machin ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .PRODUCTION_PLANNING_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/production_planning.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/production_planning.zip unzip production_planning.zip cd production_planning ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -95,39 +129,27 @@ This template finds the profit-maximizing production plan across a set of machin pat_user: pat_warehouse: ``` - Without the `deployment` section, `python production_planning.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .PRODUCTION_PLANNING_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .PRODUCTION_PLANNING_META TO APPLICATION ; - ``` - The deploy creates the public schema `.PRODUCTION_PLANNING` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python production_planning.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .PRODUCTION_PLANNING_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.production_plan ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output (all scenarios solved together in a single call): ```text Solve result: • status: OPTIMAL @@ -152,7 +174,24 @@ This template finds the profit-maximizing production plan across a set of machin 14 low_demand Machine_3 Widget_B 1.0 ``` - The single objective (`44735.0`) is the combined profit across all three scenarios. Per-scenario profit is $15,020 (0.8x), $14,945 (1.0x), and $14,770 (1.1x) — profit *falls* as the demand floor rises, because a looser floor leaves more capacity for the highest-margin mix. + The formulation printed above the solve result is trimmed here. The report checks the solve's termination status and stops unless it is `OPTIMAL`. All three scenarios are solved together in a single call. The single objective (44735.0) is the combined profit across all three scenarios. Per-scenario profit is $15,020 (0.8x), $14,945 (1.0x), and $14,770 (1.1x): profit falls as the demand floor rises, because a looser floor leaves more capacity for the highest-margin mix. See `runbook.md` for the step-by-step walkthrough. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .PRODUCTION_PLANNING_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -160,15 +199,18 @@ This template finds the profit-maximizing production plan across a set of machin production_planning/ ├── README.md # this file ├── pyproject.toml # dependencies -├── production_planning.py # main script (single solve over all scenarios) ├── runbook.md # analyst-facing walkthrough +├── model/ +│ └── __init__.py # the deployed model (single solve over all scenarios) +├── queries/ +│ └── production_plan.py # report from the deployed model └── data/ ├── products.csv # products with base demand and per-unit profit ├── machines.csv # machines with available hours per period └── production_rates.csv # hours per unit for each machine-product pair ``` -**Start here**: run `python production_planning.py` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.production_plan` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -178,7 +220,7 @@ The bundled data is small and illustrative — three CSVs describing a plant wit - **`machines.csv`** — one row per machine, with `hours_available` per planning period. - **`production_rates.csv`** — one row per machine-product pair, with `hours_per_unit` (machine time to make one unit). A missing pair means that machine cannot make that product. -The three demand scenarios (`low_demand` 0.8, `baseline` 1.0, `high_demand` 1.1) are defined inline in the script, not loaded from a CSV. +The three demand scenarios (`low_demand` 0.8, `baseline` 1.0, `high_demand` 1.1) are defined inline in `model/__init__.py`, not loaded from a CSV. ## Model overview @@ -186,7 +228,7 @@ The three demand scenarios (`low_demand` 0.8, `baseline` 1.0, `high_demand` 1.1) - **Primary identifiers**: integer `id` on `Product` and `Machine`; string `name` on `Scenario`. `ProductionRate` and `Production` are identified structurally by the entities they link. - **Important invariants**: `demand`, `profit`, `hours_available`, and `hours_per_unit` are non-negative; the decision variable `x_quantity` is a non-negative integer; total machine hours used cannot exceed `hours_available`; total units produced must meet demand scaled by the scenario multiplier. -For the full concept and property definitions, see `production_planning.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -203,7 +245,7 @@ CSV inputs → load Product / Machine / ProductionRate → add Scenario multipli 3. **Constrain the plan.** For each machine and scenario, total hours used cannot exceed the machine's available hours. For each product and scenario, total units produced must meet that product's demand scaled by the scenario's multiplier. 4. **Maximize profit.** The objective sums per-unit profit across every production assignment, and the solver returns the profit-maximizing plan for all scenarios at once. -See `production_planning.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -217,7 +259,7 @@ Focus on the first changes most users will make. ### Tune parameters -- **Demand scenarios** — edit the `scenario_data` list at the top of the script to change the multipliers or add scenarios. +- **Demand scenarios** — edit the `scenario_data` list in `model/__init__.py` to change the multipliers or add scenarios. - **Solver time limit** — `time_limit_sec` (default `60`) on the `problem.configure(...)` call. - **Extend scenario analysis** to vary other parameters, such as machine availability or profit margins. @@ -249,7 +291,7 @@ Integer programming is harder than continuous optimization. For large instances,
Connection or authentication errors -Run `rai init` to configure your Snowflake connection. Verify that the RAI Native App is installed and your user has the required permissions. +Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. Verify that the RAI Native App is installed and your user has the required permissions.
diff --git a/production_planning/production_planning.py b/production_planning/model/__init__.py similarity index 75% rename from production_planning/production_planning.py rename to production_planning/model/__init__.py index d22133e3..d9a1f8f3 100644 --- a/production_planning/production_planning.py +++ b/production_planning/model/__init__.py @@ -1,6 +1,6 @@ -"""Production Planning (prescriptive optimization) template. +"""Production Planning (prescriptive optimization) template: the model. -This script demonstrates a mixed-integer linear optimization (MILP) workflow +This package demonstrates a mixed-integer linear optimization (MILP) workflow in RelationalAI: - Load sample CSVs describing products, machines, and machine-product production rates. @@ -18,14 +18,9 @@ - One solve handles all demand levels; results extracted via model.select(). Run: - `python production_planning.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints the solver termination status, objective value, and a table of - non-trivial production quantities for each scenario, followed by a scenario - summary table. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.production_plan` then prints the report from the deployed model. """ from pathlib import Path @@ -38,7 +33,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -154,34 +149,3 @@ SolveStatus = model.Relationship(f"production planning solve status is {String:status}") TotalProfit = model.Relationship(f"production planning total profit is {Float:profit}") model.define(SolveStatus(problem.termination_status()), TotalProfit(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - profit = model.select(TotalProfit).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal production plan: termination status {status}") - print(f"• objective: {round(profit[0], 9)}\n") - - # -------------------------------------------------- - # Extract results per scenario - # -------------------------------------------------- - - print("\nProduction plan per scenario:") - model.select( - Scenario.name.alias("scenario"), - Production.rate.machine.name.alias("machine"), - Production.rate.product.name.alias("product"), - x_qty.alias("quantity"), - ).where(Production.x_quantity(Scenario, x_qty), x_qty > 0.001).inspect() diff --git a/production_planning/queries/production_plan.py b/production_planning/queries/production_plan.py new file mode 100644 index 00000000..ddb0f193 --- /dev/null +++ b/production_planning/queries/production_plan.py @@ -0,0 +1,32 @@ +"""Production plan report: reads the deployed production_planning model. + +Run `python -m queries.production_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and objective of the latest refresh's solve, and the +non-trivial production quantities for each scenario. +""" + +from model import Production, Scenario, SolveStatus, TotalProfit, model, problem, x_qty + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + profit = model.select(TotalProfit).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal production plan: termination status {status}") + print(f"• objective: {round(profit[0], 9)}\n") + + # -------------------------------------------------- + # Extract results per scenario + # -------------------------------------------------- + + print("\nProduction plan per scenario:") + model.select( + Scenario.name.alias("scenario"), + Production.rate.machine.name.alias("machine"), + Production.rate.product.name.alias("product"), + x_qty.alias("quantity"), + ).where(Production.x_quantity(Scenario, x_qty), x_qty > 0.001).inspect() diff --git a/production_planning/runbook.md b/production_planning/runbook.md index d8877389..16142137 100644 --- a/production_planning/runbook.md +++ b/production_planning/runbook.md @@ -50,4 +50,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 3 products with demand and profit, 3 machines with hours, 9 machine-product rates. The three demand scenarios (0.8, 1.0, 1.1) are defined in the script. Full model in `production_planning.py`. +Bundled CSVs in `data/`: 3 products with demand and profit, 3 machines with hours, 9 machine-product rates. The three demand scenarios (0.8, 1.0, 1.1) are defined in `model/__init__.py`. Full model in `model/__init__.py`; the report in `queries/production_plan.py`. From 7b4257f7999ed53acb1b6dc1a26ea00a14c8d37d Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:33:44 -0800 Subject: [PATCH 059/162] shift_assignment: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/shift_assignments.py (`python -m queries.shift_assignments`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. The report's status check raises an explicit error instead of using assert. --- shift_assignment/README.md | 146 +++++++++++------- .../__init__.py} | 45 +----- shift_assignment/queries/shift_assignments.py | 31 ++++ shift_assignment/runbook.md | 4 +- 4 files changed, 129 insertions(+), 97 deletions(-) rename shift_assignment/{shift_assignment.py => model/__init__.py} (74%) create mode 100644 shift_assignment/queries/shift_assignments.py diff --git a/shift_assignment/README.md b/shift_assignment/README.md index c9cdd5f7..b73ff17a 100644 --- a/shift_assignment/README.md +++ b/shift_assignment/README.md @@ -36,7 +36,8 @@ The template also demonstrates scenario analysis by sweeping over different mini ## What's included -- `shift_assignment.py` -- main script with ontology, constraints, and scenario analysis; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- ontology, constraints, and scenario analysis; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/shift_assignments.py` -- prints the report from the deployed model: the solve status and the assignments for each scenario - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/workers.csv` -- 10 workers with IDs and names - `data/shifts.csv` -- 3 shifts (Morning, Afternoon, Night) with capacity limits @@ -56,28 +57,61 @@ The template also demonstrates scenario analysis by sweeping over different mini ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .SHIFT_ASSIGNMENT_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/shift_assignment.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/shift_assignment.zip unzip shift_assignment.zip cd shift_assignment ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -96,60 +130,57 @@ The template also demonstrates scenario analysis by sweeping over different mini pat_user: pat_warehouse: ``` - Without the `deployment` section, `python shift_assignment.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .SHIFT_ASSIGNMENT_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SHIFT_ASSIGNMENT_META TO APPLICATION ; +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: + + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.SHIFT_ASSIGNMENT` if it doesn't exist. -5. Run: + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + ```bash - python shift_assignment.py + python -m queries.shift_assignments ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: + ```text + Solve result: + • status: OPTIMAL + + Assignments per scenario: + scenario worker shift + 0 coverage_1 Alice Afternoon + 1 coverage_1 Bob Morning + 2 coverage_1 Carlos Afternoon + ... + ``` + + The formulation printed above the solve result is trimmed here, and the roster is cut after three rows. The three scenarios sweep `min_coverage` from 1 to 3 workers per shift. This is a feasibility problem with no objective, so the solver returns any assignment that satisfies the constraints, and the specific worker-to-shift roster varies from run to run. What is stable is feasibility: the report stops unless the termination status is `OPTIMAL`, and each scenario's roster gives every shift its minimum coverage with no worker over their shift limit. See `runbook.md` for the step-by-step walkthrough. + + To solve again without changing the model, refresh the deployed model, then run the query again: + ```sql CALL .SHIFT_ASSIGNMENT_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output (a few representative rows confirm a successful run): - ```text - Assignments per scenario: - scenario worker shift - coverage_1 Alice Morning - coverage_1 Bob Night - coverage_1 Carlos Afternoon - ... - coverage_2 Alice Afternoon - coverage_2 Bob Morning - ... - coverage_3 Alice Morning - coverage_3 Bob Morning - ... +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - The three scenarios sweep `min_coverage` from 1 to 3 workers per shift. This - is a feasibility problem with no objective, so the solver returns *any* - assignment that satisfies the constraints — the specific worker-to-shift - roster varies run to run. What is stable is feasibility: each scenario returns - an `OPTIMAL` status with every shift meeting its minimum coverage and no worker - over their shift limit. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -157,14 +188,17 @@ The template also demonstrates scenario analysis by sweeping over different mini . ├── README.md # this file ├── pyproject.toml # dependencies -├── shift_assignment.py # main script: ontology, constraints, scenario sweep +├── model/ +│ └── __init__.py # the deployed model: ontology, constraints, scenario sweep +├── queries/ +│ └── shift_assignments.py # report from the deployed model └── data/ ├── workers.csv # 10 workers (id, name) ├── shifts.csv # 3 shifts (id, name, capacity) └── availability.csv # worker-to-shift availability pairs ``` -**Start here**: run `python shift_assignment.py` for the full scenario sweep end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.shift_assignments` for the full scenario sweep end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -182,7 +216,7 @@ The model has three concepts plus a `Scenario` concept that parameterizes the co - **Primary identifiers**: `Worker.id` and `Shift.id` are integers; `Scenario.name` is a string. - **Important invariants**: `Shift.capacity` is a positive integer; each scenario's `min_coverage` must be no larger than the smallest shift capacity or that scenario is infeasible; each worker takes at most `max_shifts` shifts (default 1); a worker can only be assigned to a shift they are available for. -For the full concept and property definitions, see `shift_assignment.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -197,9 +231,9 @@ CSV inputs → load Worker / Shift / availability → add Scenario coverage leve 1. **Load the data.** Workers and shifts come from their CSVs; availability pairs become a `Worker.available_for(Shift)` relationship that scopes which assignments are even possible. 2. **Set up the decision.** A binary `x_assign` variable indicates whether a worker takes a given shift in a given scenario. Each coverage level (`coverage_1` / `_2` / `_3`) is a `Scenario` with a `min_coverage`, and the variable is scoped to the availability relationship so unavailable pairs are never considered. 3. **Constrain the roster.** Three constraints govern every scenario: each shift meets its `min_coverage`, each worker takes no more than `max_shifts` shifts (default 1), and no shift exceeds its `capacity`. -4. **Solve and deploy.** A single solve handles all scenarios simultaneously; this is a feasibility problem with no objective, so the solver returns any assignment satisfying every constraint. `problem.configure(...)` declares the solve, the script deploys the model, and each refresh of the deployed model runs the solve. The report checks the solve status; it does not re-evaluate the constraints, and a refresh started outside the script (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. +4. **Solve and deploy.** A single solve handles all scenarios simultaneously; this is a feasibility problem with no objective, so the solver returns any assignment satisfying every constraint. `problem.configure(...)` declares the solve, `rai models deploy --path model` deploys the model, and each refresh of the deployed model runs the solve. The report checks the solve status; it does not re-evaluate the constraints, and a refresh started outside the query (`rai models deploy`, the refresh procedure, a schedule) runs no checks, so read `SolveStatus` before relying on a refreshed solution. -See `shift_assignment.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -247,7 +281,7 @@ See `shift_assignment.py` for the implementation and `runbook.md` for the skill-
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4). - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/shift_assignment/shift_assignment.py b/shift_assignment/model/__init__.py similarity index 74% rename from shift_assignment/shift_assignment.py rename to shift_assignment/model/__init__.py index 4b382c75..7414f7e5 100644 --- a/shift_assignment/shift_assignment.py +++ b/shift_assignment/model/__init__.py @@ -1,6 +1,6 @@ -"""Shift Assignment (prescriptive optimization) template. +"""Shift Assignment (prescriptive optimization) template: the model. -This script demonstrates a constraint satisfaction / feasibility problem in RelationalAI: +This package demonstrates a constraint satisfaction / feasibility problem in RelationalAI: - Load sample CSVs describing workers, shifts, and worker-shift availability. - Model workers and shifts as *concepts* with typed properties and an availability @@ -17,13 +17,9 @@ - One solve handles all coverage levels; results extracted via model.select(). Run: - `python shift_assignment.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report - from the deployed model. - -Output: - Prints per-scenario termination status and a table of assignments, followed - by a scenario analysis summary. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.shift_assignments` then prints the report from the deployed model. """ from pathlib import Path @@ -38,7 +34,7 @@ # Define semantic model & load data # -------------------------------------------------- -data_dir = Path(__file__).parent / "data" +data_dir = Path(__file__).resolve().parents[1] / "data" # Concept: workers Worker = model.Concept("Worker", identify_by={"id": Integer}) @@ -128,32 +124,3 @@ # The solve's status, published through the model's own relationship. SolveStatus = model.Relationship(f"shift assignment solve status is {String:status}") model.define(SolveStatus(problem.termination_status())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status of the latest refresh's solve. A refresh records any termination - # status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}\n") - assert status == ["OPTIMAL"], f"termination status {status}" - - # ---------------------------------------------- - # Extract results per scenario - # ---------------------------------------------- - - print("\nAssignments per scenario:") - model.select( - Scenario.name.alias("scenario"), - Worker.name.alias("worker"), - Shift.name.alias("shift"), - ).where( - Worker.x_assign(Shift, Scenario, assigned_ref), assigned_ref > 0 - ).inspect() diff --git a/shift_assignment/queries/shift_assignments.py b/shift_assignment/queries/shift_assignments.py new file mode 100644 index 00000000..d6ae34c7 --- /dev/null +++ b/shift_assignment/queries/shift_assignments.py @@ -0,0 +1,31 @@ +"""Shift assignment report: reads the deployed shift_assignment model. + +Run `python -m queries.shift_assignments` after `rai models deploy --path model`. Prints the +formulation, the termination status of the latest refresh's solve, and the worker assigned +to each shift in every coverage scenario. +""" + +from model import Scenario, Shift, SolveStatus, Worker, assigned_ref, model, problem + +if __name__ == "__main__": + problem.display() + + # Status of the latest refresh's solve. A refresh records any termination + # status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}\n") + if status != ["OPTIMAL"]: + raise SystemExit(f"No feasible shift assignment: termination status {status}") + + # ---------------------------------------------- + # Extract results per scenario + # ---------------------------------------------- + + print("\nAssignments per scenario:") + model.select( + Scenario.name.alias("scenario"), + Worker.name.alias("worker"), + Shift.name.alias("shift"), + ).where( + Worker.x_assign(Shift, Scenario, assigned_ref), assigned_ref > 0 + ).inspect() diff --git a/shift_assignment/runbook.md b/shift_assignment/runbook.md index c8e93e3f..e854af86 100644 --- a/shift_assignment/runbook.md +++ b/shift_assignment/runbook.md @@ -38,7 +38,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Is there a feasible assignment of workers to shifts where each shift has at least its minimum coverage and no more than its capacity, each worker works at most one shift, and no one is assigned to a shift they're unavailable for? Solve it for three minimum-coverage targets — 1, 2, and 3 workers per shift — in a single solve, with the coverage target as a first-class scenario concept and the assignment decision indexed by (worker, shift, scenario). Persist the assignments to the ontology. -**Response:** OPTIMAL (satisfiable) at all three coverage levels — 66 binary variables (22 availability pairs x 3 scenarios), 48 constraints. The assignment is written back as `Worker.x_assign(Shift, Scenario)`. This is a feasibility problem with no objective, so the solver returns one valid assignment per level — an alternative valid assignment may differ run to run. In `shift_assignment.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** OPTIMAL (satisfiable) at all three coverage levels — 66 binary variables (22 availability pairs x 3 scenarios), 48 constraints. The assignment is written back as `Worker.x_assign(Shift, Scenario)`. This is a feasibility problem with no objective, so the solver returns one valid assignment per level — an alternative valid assignment may differ run to run. In `model/__init__.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Read the staffing levels @@ -48,4 +48,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 10 workers, 3 shifts with capacities, 22 availability pairs. The three coverage scenarios (1, 2, 3) are defined in the script. Full model in `shift_assignment.py`. +Bundled CSVs in `data/`: 10 workers, 3 shifts with capacities, 22 availability pairs. The three coverage scenarios (1, 2, 3) are defined in `model/__init__.py`. Full model in `model/__init__.py`; the report in `queries/shift_assignments.py`. From dfe3adb13eecf7a9d2c771c8c7e13a325156954e Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:36:37 -0800 Subject: [PATCH 060/162] retail_markdown: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/markdown_plan.py (`python -m queries.markdown_plan`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- retail_markdown/README.md | 138 ++++++++++++------ .../{retail_markdown.py => model/__init__.py} | 59 +------- retail_markdown/queries/markdown_plan.py | 56 +++++++ retail_markdown/runbook.md | 2 +- 4 files changed, 158 insertions(+), 97 deletions(-) rename retail_markdown/{retail_markdown.py => model/__init__.py} (74%) create mode 100644 retail_markdown/queries/markdown_plan.py diff --git a/retail_markdown/README.md b/retail_markdown/README.md index 1b053db4..7a124158 100644 --- a/retail_markdown/README.md +++ b/retail_markdown/README.md @@ -38,7 +38,8 @@ This template finds the discount schedule that maximizes revenue across a multi- ## What's included -- `retail_markdown.py` -- Main script defining the MIP model with discount selection, sales tracking, and revenue optimization; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- The MIP model with discount selection, sales tracking, and revenue optimization; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/markdown_plan.py` -- Prints the report from the deployed model: the solve's status and objective and the discount, sales, and cumulative-sales tables - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/products.csv` -- Products with initial price, cost, inventory, base demand, and salvage rate - `data/discounts.csv` -- Discount levels with percentage and demand lift factor @@ -57,28 +58,61 @@ This template finds the discount schedule that maximizes revenue across a multi- ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .RETAIL_MARKDOWN_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/retail_markdown.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/retail_markdown.zip unzip retail_markdown.zip cd retail_markdown ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -97,47 +131,62 @@ This template finds the discount schedule that maximizes revenue across a multi- pat_user: pat_warehouse: ``` - Without the `deployment` section, `python retail_markdown.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .RETAIL_MARKDOWN_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .RETAIL_MARKDOWN_META TO APPLICATION ; - ``` - The deploy creates the public schema `.RETAIL_MARKDOWN` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python retail_markdown.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .RETAIL_MARKDOWN_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.markdown_plan ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output: ```text + Solve result: + • status: OPTIMAL + • objective: 23374.65 + Status: OPTIMAL Total revenue (sales + salvage): $23374.65 + + === Selected Discounts by Product-Week === + product week discount_pct + 0 Jacket 1 20.0 + 1 Jacket 2 20.0 + 2 Jacket 3 20.0 + 3 Jacket 4 20.0 + ... + ``` + + The formulation printed above the solve result is trimmed here, and the selected discounts are cut after four rows; the report also prints the units sold and the cumulative sales for each product and week. The report checks the solve's termination status and stops unless it is `OPTIMAL`. Discounts start shallow (20 percent) and deepen to 30 percent later in the season; no product needs the 50 percent tier. See `runbook.md` for the full discount, sales, and cumulative-sales schedule. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .RETAIL_MARKDOWN_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - Discounts start shallow (20%) and deepen to 30% later in the season; no - product needs the 50% tier. See `runbook.md` for the full discount, sales, - and cumulative-sales schedule. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -145,15 +194,18 @@ This template finds the discount schedule that maximizes revenue across a multi- retail_markdown/ ├── README.md # this file ├── pyproject.toml # dependencies -├── retail_markdown.py # main script (MIP model, solve, result tables) ├── runbook.md # analyst-facing walkthrough +├── model/ +│ └── __init__.py # the deployed model (MIP, solve) +├── queries/ +│ └── markdown_plan.py # report from the deployed model (result tables) └── data/ ├── products.csv # products with price, cost, inventory, base demand, salvage rate ├── discounts.csv # discount levels with percentage and demand lift └── weeks.csv # planning weeks with seasonal demand multipliers ``` -**Start here**: run `python retail_markdown.py` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.markdown_plan` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -169,7 +221,7 @@ The bundled data is small and illustrative — a short seasonal clearance across - **Primary identifiers**: string `name` on `Product`; integer `level` on `Discount`; integer `num` on `Week`. - **Important invariants**: exactly one discount level is active per product-week; discounts can only deepen over successive weeks (price ladder); cumulative sales never exceed `initial_inventory`; `discount_pct`, `demand_lift`, and `demand_multiplier` are non-negative; the selection variable is binary and sales variables are non-negative. -For the full concept and property definitions, see `retail_markdown.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -185,7 +237,7 @@ CSV inputs → load Product / Discount / Week → decision variables (discount c 3. **Constrain the schedule.** Exactly one discount level is active per product-week; discounts can only deepen from one week to the next (the price ladder); and cumulative sales can never exceed starting inventory. 4. **Maximize revenue.** The objective adds sales revenue — discounted price times units sold — to the salvage value of unsold units at the end of the horizon, and the solver returns the revenue-maximizing discount schedule. -See `retail_markdown.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation and `runbook.md` for the skill-driven reproduction. ## Customize this template @@ -229,9 +281,9 @@ Mixed-integer programs can be computationally expensive. Reduce the number of pr
-rai init fails or connection errors +Connection errors -Ensure your Snowflake credentials are configured correctly and that the RAI Native App is installed on your account. Run `rai init` again and verify the connection settings. +Ensure your Snowflake credentials are configured correctly and that the RAI Native App is installed on your account. Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings.
diff --git a/retail_markdown/retail_markdown.py b/retail_markdown/model/__init__.py similarity index 74% rename from retail_markdown/retail_markdown.py rename to retail_markdown/model/__init__.py index 7fccdece..af4d4064 100644 --- a/retail_markdown/retail_markdown.py +++ b/retail_markdown/model/__init__.py @@ -1,6 +1,6 @@ -"""Retail markdown (prescriptive optimization) template. +"""Retail markdown (prescriptive optimization) template: the model. -This script demonstrates a retail markdown mixed-integer linear optimization (MILP) +This package demonstrates a retail markdown mixed-integer linear optimization (MILP) workflow in RelationalAI: - Load sample CSVs describing products, discount levels, and weekly demand multipliers. @@ -12,13 +12,9 @@ - Maximize total revenue from discounted sales plus salvage value of leftover inventory. Run: - `python retail_markdown.py` deploys the model to the schema in raiconfig.yaml, waits - for its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints the solver termination status, objective value (sales + salvage), and - three tables showing selected discounts, sales, and cumulative sales. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.markdown_plan` then prints the report from the deployed model. """ from pathlib import Path @@ -33,7 +29,7 @@ # Define semantic model & load data # -------------------------------------------------- -data_dir = Path(__file__).parent / "data" +data_dir = Path(__file__).resolve().parents[1] / "data" # Concept: products with inventory and demand info Product = model.Concept("Product", identify_by={"name": String}) @@ -182,46 +178,3 @@ SolveStatus = model.Relationship(f"retail markdown solve status is {String:status}") TotalRevenue = model.Relationship(f"retail markdown total revenue is {Float:revenue}") model.define(SolveStatus(problem.termination_status()), TotalRevenue(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - total_revenue = model.select(TotalRevenue).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal markdown plan: termination status {status}") - print(f"• objective: {round(total_revenue[0], 9)}\n") - - print(f"Status: {status[0]}") - print(f"Total revenue (sales + salvage): ${total_revenue[0]:.2f}") - - # Extract solution via model.select() — properties are populated after solve - print("\n=== Selected Discounts by Product-Week ===") - model.select( - Product.name.alias("product"), Week_ref.num.alias("week"), - Discount_ref.discount_pct.alias("discount_pct"), - ).where( - Product.x_select(Week_ref, Discount_ref, selection_ref), selection_ref > 0.5 - ).inspect() - - print("\n=== Sales by Product-Week ===") - model.select( - Product.name.alias("product"), Week_ref.num.alias("week"), - Discount_ref.discount_pct.alias("discount_pct"), sales_ref.alias("units_sold"), - ).where(Product.x_sales(Week_ref, Discount_ref, sales_ref), sales_ref > 0.01).inspect() - - print("\n=== Cumulative Sales by Product-Week ===") - model.select( - Product.name.alias("product"), Week_ref.num.alias("week"), - cumulative_ref.alias("cumulative_sold"), - ).where(Product.x_cuml_sales(Week_ref, cumulative_ref)).inspect() diff --git a/retail_markdown/queries/markdown_plan.py b/retail_markdown/queries/markdown_plan.py new file mode 100644 index 00000000..8e261fab --- /dev/null +++ b/retail_markdown/queries/markdown_plan.py @@ -0,0 +1,56 @@ +"""Markdown plan report: reads the deployed retail_markdown model. + +Run `python -m queries.markdown_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and objective (sales plus salvage) of the latest +refresh's solve, and tables of the selected discounts, the units sold, and the cumulative +sales for each product and week. +""" + +from model import ( + Discount_ref, + Product, + SolveStatus, + TotalRevenue, + Week_ref, + cumulative_ref, + model, + problem, + sales_ref, + selection_ref, +) + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + total_revenue = model.select(TotalRevenue).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal markdown plan: termination status {status}") + print(f"• objective: {round(total_revenue[0], 9)}\n") + + print(f"Status: {status[0]}") + print(f"Total revenue (sales + salvage): ${total_revenue[0]:.2f}") + + # Extract the solution with model.select(); the solved values are populated. + print("\n=== Selected Discounts by Product-Week ===") + model.select( + Product.name.alias("product"), Week_ref.num.alias("week"), + Discount_ref.discount_pct.alias("discount_pct"), + ).where( + Product.x_select(Week_ref, Discount_ref, selection_ref), selection_ref > 0.5 + ).inspect() + + print("\n=== Sales by Product-Week ===") + model.select( + Product.name.alias("product"), Week_ref.num.alias("week"), + Discount_ref.discount_pct.alias("discount_pct"), sales_ref.alias("units_sold"), + ).where(Product.x_sales(Week_ref, Discount_ref, sales_ref), sales_ref > 0.01).inspect() + + print("\n=== Cumulative Sales by Product-Week ===") + model.select( + Product.name.alias("product"), Week_ref.num.alias("week"), + cumulative_ref.alias("cumulative_sold"), + ).where(Product.x_cuml_sales(Week_ref, cumulative_ref)).inspect() diff --git a/retail_markdown/runbook.md b/retail_markdown/runbook.md index dc475d8f..c3f29f57 100644 --- a/retail_markdown/runbook.md +++ b/retail_markdown/runbook.md @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 4 products, 5 discount levels, 4 weeks. Full model in `retail_markdown.py`. +Bundled CSVs in `data/`: 4 products, 5 discount levels, 4 weeks. Full model in `model/__init__.py`; the report in `queries/markdown_plan.py`. From 12f5b2fedaedfe15b7ab149570e69bd7d3d200e8 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:39:45 -0800 Subject: [PATCH 061/162] supply_chain_transport: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/transport_plan.py (`python -m queries.transport_plan`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. --- supply_chain_transport/README.md | 138 ++++++++++++------ .../__init__.py} | 59 +------- .../queries/transport_plan.py | 55 +++++++ supply_chain_transport/runbook.md | 2 +- 4 files changed, 154 insertions(+), 100 deletions(-) rename supply_chain_transport/{supply_chain_transport.py => model/__init__.py} (82%) create mode 100644 supply_chain_transport/queries/transport_plan.py diff --git a/supply_chain_transport/README.md b/supply_chain_transport/README.md index cbbba084..2dc22378 100644 --- a/supply_chain_transport/README.md +++ b/supply_chain_transport/README.md @@ -38,7 +38,8 @@ The model demonstrates several advanced techniques: multi-period inventory flow ## What's included -- `supply_chain_transport.py` -- main script with ontology, formulation, and solver configuration; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- ontology, formulation, and solver configuration; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/transport_plan.py` -- prints the report from the deployed model: the solve status and cost and the inventory, transport, and arrival tables - `runbook.md` -- a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself - `data/freight_groups.csv` -- 2 freight groups with inventory/transport/arrival windows - `pyproject.toml` -- Python package configuration @@ -56,28 +57,61 @@ The model demonstrates several advanced techniques: multi-period inventory flow ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .SUPPLY_CHAIN_TRANSPORT_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/supply_chain_transport.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/supply_chain_transport.zip unzip supply_chain_transport.zip cd supply_chain_transport ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -96,78 +130,90 @@ The model demonstrates several advanced techniques: multi-period inventory flow pat_user: pat_warehouse: ``` - Without the `deployment` section, `python supply_chain_transport.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .SUPPLY_CHAIN_TRANSPORT_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SUPPLY_CHAIN_TRANSPORT_META TO APPLICATION ; - ``` - The deploy creates the public schema `.SUPPLY_CHAIN_TRANSPORT` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python supply_chain_transport.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .SUPPLY_CHAIN_TRANSPORT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.transport_plan ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output: ```text + Solve result: + • status: OPTIMAL + • objective: 1444.0 + Status: OPTIMAL Total cost: $1444.00 === Transport Quantities === - type freight_group day quantity - ltl fg1 2 4000.0 - ltl fg2 2 5000.0 + type freight_group day quantity + 0 ltl fg1 2 4000.0 + 1 ltl fg2 2 5000.0 + ``` + + The formulation printed above the solve result is trimmed here, and so are the inventory-level and arrival-day tables that the report also prints. The report checks the solve's termination status and stops unless it is `OPTIMAL`. Both freight groups ship less-than-truckload on day 2 (zero truckload trucks); see `runbook.md` for the full inventory and arrival-day tables. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .SUPPLY_CHAIN_TRANSPORT_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - Both freight groups ship less-than-truckload on day 2 (zero truckload trucks); see `runbook.md` for the full inventory and arrival-day tables. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure ```text . ├── README.md ├── pyproject.toml -├── supply_chain_transport.py +├── model/ +│ └── __init__.py +├── queries/ +│ └── transport_plan.py └── data/ └── freight_groups.csv ``` -**Start here**: run `python supply_chain_transport.py` for the full formulation and solve end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.transport_plan` for the full formulation and solve end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data -The bundled data is synthetic and illustrative -- a minimal two-group example designed to teach the formulation on a Snowflake-connected RAI account, not to match a specific shipper's freight book. The transport modes and cost segments are defined inline in the script, not loaded from CSV. +The bundled data is synthetic and illustrative -- a minimal two-group example designed to teach the formulation on a Snowflake-connected RAI account, not to match a specific shipper's freight book. The transport modes and cost segments are defined inline in `model/__init__.py`, not loaded from CSV. - **`freight_groups.csv`** (2 rows) -- one row per freight group, giving its inventory window (`inv_start_t` to `inv_end_t`), transport window (`tra_start_t` to `tra_end_t`), arrival window (`arr_start_t` to `arr_end_t`), and starting inventory weight (`inv_start`). ## Model overview -The formulation is built on three concepts: the freight groups loaded from CSV, and the transport types and LTL cost segments defined inline in the script. +The formulation is built on three concepts: the freight groups loaded from CSV, and the transport types and LTL cost segments defined inline in `model/__init__.py`. - **Key entities**: `FreightGroup` — a batch of freight with its own inventory, transport, and arrival time windows (the optimization solves for its per-day inventory, shipment quantities, and arrival day); `TransportType` — a shipping mode (truckload or less-than-truckload), with its transit time; `LTLSegment` — a breakpoint in the piecewise-linear less-than-truckload cost curve. - **Primary identifiers**: `name` on `FreightGroup` and `TransportType`; integer `seg` on `LTLSegment`. - **Important invariants**: window start days are less than or equal to window end days; `inv_start` weights are non-negative; transport-mode indicators and segment activation variables are binary; each freight group ships all inventory out by the end of its inventory window. -For the full concept and property definitions, see `supply_chain_transport.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -185,7 +231,7 @@ CSV freight groups + inline modes/segments → decision variables → inventory **5. Minimize total cost.** The objective combines inventory holding cost, TL fixed cost, and piecewise LTL variable cost, and the MIP is solved end to end. -For the implementation, see `supply_chain_transport.py`; to reproduce it step by step with the RAI skills, follow `runbook.md`. +For the implementation, see `model/__init__.py`; to reproduce it step by step with the RAI skills, follow `runbook.md`. ## Customize this template @@ -198,7 +244,7 @@ Focus on the first changes most users will make. ### Tune parameters -- **Cost parameters** -- change `inv_cost`, `tl_tra_cost`, or the LTL segment costs and limits in the script. +- **Cost parameters** -- change `inv_cost`, `tl_tra_cost`, or the LTL segment costs and limits in `model/__init__.py`. - **Transit times** -- adjust the inline `transit_time` values on the `tl` and `ltl` transport types. ### Extend the model @@ -235,7 +281,7 @@ Focus on the first changes most users will make.
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4). - If you have multiple profiles, set `export RAI_PROFILE=`.
diff --git a/supply_chain_transport/supply_chain_transport.py b/supply_chain_transport/model/__init__.py similarity index 82% rename from supply_chain_transport/supply_chain_transport.py rename to supply_chain_transport/model/__init__.py index dfb165d2..2a138f90 100644 --- a/supply_chain_transport/supply_chain_transport.py +++ b/supply_chain_transport/model/__init__.py @@ -1,6 +1,6 @@ -"""Supply Chain Transport (prescriptive optimization) template. +"""Supply Chain Transport (prescriptive optimization) template: the model. -This script demonstrates a multi-mode transportation optimization model in RelationalAI: +This package demonstrates a multi-mode transportation optimization model in RelationalAI: - Load sample CSVs describing freight groups with inventory and transport time windows. - Model freight groups, transport types (TL/LTL), and LTL cost segments as *concepts* @@ -11,13 +11,9 @@ - Minimize total cost (inventory holding + TL fixed cost + LTL variable cost). Run: - `python supply_chain_transport.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from the - deployed model. - -Output: - Prints the solver termination status, total cost, and tables showing inventory - levels, transport quantities, and arrival days. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.transport_plan` then prints the report from the deployed model. """ from pathlib import Path @@ -30,7 +26,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" dep_start = 1 dep_end = 4 @@ -264,46 +260,3 @@ SolveStatus = model.Relationship(f"supply chain transport solve status is {String:status}") TotalTransportCost = model.Relationship(f"supply chain transport total cost is {Float:cost}") model.define(SolveStatus(problem.termination_status()), TotalTransportCost(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - cost = model.select(TotalTransportCost).to_df().iloc[:, 0].tolist() - print(f"\nSolve result:\n• status: {', '.join(status)}") - if status != ["OPTIMAL"]: - raise SystemExit(f"No optimal transport plan: termination status {status}") - print(f"• objective: {round(cost[0], 9)}\n") - - print(f"Status: {status[0]}") - print(f"Total cost: ${cost[0]:.2f}") - - # Extract solution via model.select() — properties are populated after solve - print("\n=== Inventory Levels ===") - model.select( - FreightGroup.name.alias("freight_group"), time_period_ref.alias("day"), - x_inv.alias("inventory"), - ).where(FreightGroup.x_inv(time_period_ref, x_inv)).inspect() - - print("\n=== Transport Quantities ===") - model.select( - TransportType.name.alias("type"), FreightGroup_ref.name.alias("freight_group"), - time_period_ref.alias("day"), x_qty_tra.alias("quantity"), - ).where( - TransportType.x_qty_tra(FreightGroup_ref, time_period_ref, x_qty_tra), x_qty_tra > 0.001 - ).inspect() - - print("\n=== Arrival Days ===") - model.select( - FreightGroup.name.alias("freight_group"), - FreightGroup.z_arr_day.alias("arrival_day"), - ).inspect() diff --git a/supply_chain_transport/queries/transport_plan.py b/supply_chain_transport/queries/transport_plan.py new file mode 100644 index 00000000..b4f49b3d --- /dev/null +++ b/supply_chain_transport/queries/transport_plan.py @@ -0,0 +1,55 @@ +"""Transport plan report: reads the deployed supply_chain_transport model. + +Run `python -m queries.transport_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and total cost of the latest refresh's solve, and +tables of the inventory levels, the transport quantities, and the arrival days. +""" + +from model import ( + FreightGroup, + FreightGroup_ref, + SolveStatus, + TotalTransportCost, + TransportType, + model, + problem, + time_period_ref, + x_inv, + x_qty_tra, +) + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(TotalTransportCost).to_df().iloc[:, 0].tolist() + print(f"\nSolve result:\n• status: {', '.join(status)}") + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal transport plan: termination status {status}") + print(f"• objective: {round(cost[0], 9)}\n") + + print(f"Status: {status[0]}") + print(f"Total cost: ${cost[0]:.2f}") + + # Extract the solution with model.select(); the solved values are populated. + print("\n=== Inventory Levels ===") + model.select( + FreightGroup.name.alias("freight_group"), time_period_ref.alias("day"), + x_inv.alias("inventory"), + ).where(FreightGroup.x_inv(time_period_ref, x_inv)).inspect() + + print("\n=== Transport Quantities ===") + model.select( + TransportType.name.alias("type"), FreightGroup_ref.name.alias("freight_group"), + time_period_ref.alias("day"), x_qty_tra.alias("quantity"), + ).where( + TransportType.x_qty_tra(FreightGroup_ref, time_period_ref, x_qty_tra), x_qty_tra > 0.001 + ).inspect() + + print("\n=== Arrival Days ===") + model.select( + FreightGroup.name.alias("freight_group"), + FreightGroup.z_arr_day.alias("arrival_day"), + ).inspect() diff --git a/supply_chain_transport/runbook.md b/supply_chain_transport/runbook.md index 24b29a0f..5be35291 100644 --- a/supply_chain_transport/runbook.md +++ b/supply_chain_transport/runbook.md @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSV in `data/`: 2 freight groups with day-windows and weights. The transport modes and LTL rate tiers are defined in the script, along with TL cost/capacity and the holding rate. Full model in `supply_chain_transport.py`. +Bundled CSV in `data/`: 2 freight groups with day-windows and weights. The transport modes and LTL rate tiers are defined in `model/__init__.py`, along with TL cost/capacity and the holding rate. Full model in `model/__init__.py`; the report in `queries/transport_plan.py`. From c7b031ec1fd5f79301661658c9d6f5e6d9489b9d Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:42:15 -0800 Subject: [PATCH 062/162] synthetic_eligibility_records: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report into queries/eligibility_records.py (`python -m queries.eligibility_records`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down. The report no longer carries the comment about solver-only implies constraints, which only applied to the removed verify call. --- synthetic_eligibility_records/README.md | 146 +++++++++++------- .../__init__.py} | 72 +-------- .../queries/eligibility_records.py | 70 +++++++++ 3 files changed, 169 insertions(+), 119 deletions(-) rename synthetic_eligibility_records/{synthetic_eligibility_records.py => model/__init__.py} (75%) create mode 100644 synthetic_eligibility_records/queries/eligibility_records.py diff --git a/synthetic_eligibility_records/README.md b/synthetic_eligibility_records/README.md index 751ea8ed..57dd400c 100644 --- a/synthetic_eligibility_records/README.md +++ b/synthetic_eligibility_records/README.md @@ -40,12 +40,13 @@ The rule structure here is drawn from the public [CMS Medicare](https://www.cms. - A PCP-network attribution IC iterating over reference-data `(Plan, Provider)` tuples in different networks and forbidding the cross-network combination - A pre-solve dense-ID check on `plans.csv`, `providers.csv`, and `age_buckets.csv` so the solver's integer decision bounds line up with the reference rows the CFDs iterate over (sparse IDs would let the solver pick a value with no matching row, leaving the rules unconstrained for that record and silently dropping it from the post-solve display join) - **Multi-solution enumeration as the primary code path**: `problem.configure(..., solution_limit=MAX_RECORDS)` makes each refresh run the search in enumeration mode; `Variable.values(solution_index, value)` joins the three decision variables on a shared solution index to reconstruct each record -- An empty-result branch driven by `solve_info().num_points`: when no feasible record exists, the script prints a diagnostic instead of hard-failing, which is the right shape for a reusable generator +- An empty-result branch driven by `solve_info().num_points`: when no feasible record exists, the report prints a diagnostic instead of hard-failing, which is the right shape for a reusable generator - No check re-evaluates the ICs: every IC uses `implies`, which is solver-only. The CFD and network-attribution invariants are directly visible in the expected-output block in the Quickstart: every record's `age_years` vs `plan_type` and every record's `network` vs `provider` are printed side-by-side ## What's included -- `synthetic_eligibility_records.py` -- main script with ontology, decisions, constraints, and solver configuration; it deploys the model and prints the report from the deployed model +- `model/__init__.py` -- ontology, decisions, constraints, and solver configuration; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/eligibility_records.py` -- prints the report from the deployed model: the solve status, the number of solutions, and every generated member record - `data/age_buckets.csv` -- 4 representative ages spanning the adult/senior split (2 under 65, 2 at or above) - `data/plans.csv` -- 3 plans (PPO, HMO, MedicareAdvantage) each on its own network - `data/providers.csv` -- 4 primary-care providers (1 PPO, 1 HMO, 2 Medicare) so each plan-network has at least one in-network PCP and the bundled K=8 enumeration spans all three plans @@ -63,28 +64,61 @@ The rule structure here is drawn from the public [CMS Medicare](https://www.cms. ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .SYNTHETIC_ELIGIBILITY_RECORDS_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/synthetic_eligibility_records.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/synthetic_eligibility_records.zip unzip synthetic_eligibility_records.zip cd synthetic_eligibility_records ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -103,64 +137,72 @@ The rule structure here is drawn from the public [CMS Medicare](https://www.cms. pat_user: pat_warehouse: ``` - Without the `deployment` section, `python synthetic_eligibility_records.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .SYNTHETIC_ELIGIBILITY_RECORDS_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .SYNTHETIC_ELIGIBILITY_RECORDS_META TO APPLICATION ; - ``` - The deploy creates the public schema `.SYNTHETIC_ELIGIBILITY_RECORDS` if it doesn't exist. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python synthetic_eligibility_records.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .SYNTHETIC_ELIGIBILITY_RECORDS_META._INTERNAL_REFRESH_MODEL('standard', NULL); + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. + + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. + +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.eligibility_records ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output. With `MAX_RECORDS = 16` and a bundled feasible set of 8 records, the solver exhausts the search space and returns status `OPTIMAL` with all 8 records. Each row carries its `solution` index. Solver build strings, exact wall times, and per-solution ordering will vary; the structure of the output and the *set* of returned records is stable: ```text Solve result: • status: OPTIMAL • num_points: 8 Generated member records (up to 16 per run): - solution age_years plan_type plan_network provider_network provider - 0 78 MedicareAdvantage 3 3 Dr_Senior_B - 1 68 MedicareAdvantage 3 3 Dr_Senior_B - 2 78 MedicareAdvantage 3 3 Dr_Senior_A - 3 68 MedicareAdvantage 3 3 Dr_Senior_A - 4 28 HMO 2 2 Dr_East_HMO - 5 50 HMO 2 2 Dr_East_HMO - 6 50 PPO 1 1 Dr_North_PPO - 7 28 PPO 1 1 Dr_North_PPO + solution age_years plan_type plan_network provider_network provider + 0 78 MedicareAdvantage 3 3 Dr_Senior_B + 1 68 MedicareAdvantage 3 3 Dr_Senior_B + 2 78 MedicareAdvantage 3 3 Dr_Senior_A + 3 68 MedicareAdvantage 3 3 Dr_Senior_A + 4 50 HMO 2 2 Dr_East_HMO + 5 28 HMO 2 2 Dr_East_HMO + 6 50 PPO 1 1 Dr_North_PPO + 7 28 PPO 1 1 Dr_North_PPO + ``` + + The formulation printed above the solve result is trimmed here. With `MAX_RECORDS = 16` and a bundled feasible set of 8 records, the solver exhausts the search space and returns status `OPTIMAL` with all 8 records. Each row carries its `solution` index. Solver build strings, exact wall times, and per-solution ordering vary from run to run; the structure of the output and the set of returned records is stable. Each row is one full member: an age of 65 or more always pairs with `MedicareAdvantage`, and `plan_network` always equals `provider_network`. On a real catalog the feasible set typically exceeds `MAX_RECORDS`, and the solver returns status `SOLUTION_LIMIT` once the cap is hit. The report prints the status either way, and prints a diagnostic when the solver returned no records. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .SYNTHETIC_ELIGIBILITY_RECORDS_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` - Each row is one full member. The constraint encoding is visible by eye: `age_years >= 65` always pairs with `MedicareAdvantage`, and `plan_network` always equals `provider_network`. On a real catalog the feasible set typically exceeds `MAX_RECORDS`; the solver returns status `SOLUTION_LIMIT` once the cap is hit. + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure ```text . ├── README.md ├── pyproject.toml -├── synthetic_eligibility_records.py +├── model/ +│ └── __init__.py +├── queries/ +│ └── eligibility_records.py └── data/ ├── age_buckets.csv ├── plans.csv @@ -215,7 +257,7 @@ network_match_ic = model.where(Plan.network_id != Provider.network_id).require( ```python problem.configure("minizinc", time_limit_sec=60, solution_limit=MAX_RECORDS) -# in the report, after model.deploy(): +# in the report, after the deploy: si = problem.solve_info() sol_idx = Integer.ref() @@ -245,7 +287,7 @@ To extend to a fleet of N members, reintroduce a `Member` concept inside the mod ## Customize this template -- **Use your own plans and providers** by replacing the two CSV files. The constraint structure does not change; the integer ID columns stay required (the script uses them for the `plan_id` / `provider_id` decision domains) and IDs must remain dense and contiguous (the pre-solve check enforces this). +- **Use your own plans and providers** by replacing the two CSV files. The constraint structure does not change; the integer ID columns stay required (the model uses them for the `plan_id` / `provider_id` decision domains) and IDs must remain dense and contiguous (the pre-solve check enforces this). - **Raise the solution limit on a real catalog.** The bundled `MAX_RECORDS = 16` is sized so the solver exhausts the small demo feasible set; production test suites typically want 100--10,000 records per solve. `time_limit_sec` is your safety net -- enumeration stops when either the limit or the budget is reached. - **Adjust the seniority gate** by changing `SENIOR_THRESHOLD_YEARS` (currently 65, the CMS Medicare threshold). Both arms of the age-by-plan CFD read this constant directly. - **Add a dependent-count decision** by introducing a `num_dependents` scalar decision bounded by 0 and a per-plan `max_dependents` cap. Extend `plans.csv` with a `max_dependents` column, declare `Plan.max_dependents = model.Property(f"{Plan} has {Integer:max_dependents}")`, add `num_dependents = model.Relationship(f"{Integer:num_dependents}")` and a `problem.solve_for(num_dependents, ...)` call, then encode the cap with the same forbidden-pair idiom: `model.where(Plan.max_dependents >= 0).require(implies(plan_id == Plan.id, num_dependents <= Plan.max_dependents))`. @@ -267,7 +309,7 @@ To extend to a fleet of N members, reintroduce a `Member` concept inside the mod
Authentication or configuration errors -- Run `rai init` to create or update your RelationalAI/Snowflake configuration. +- Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4). - If you have multiple profiles, set `export RAI_PROFILE=`.
@@ -283,7 +325,7 @@ To extend to a fleet of N members, reintroduce a `Member` concept inside the mod
Solver returns INFEASIBLE / no feasible eligibility records -- Check the solve status the script prints. `INFEASIBLE` means the reference data admits no record; `UNKNOWN` or `TIME_LIMIT` means the budget expired before a record was found (raise `time_limit_sec` or shrink the decision domains). +- Check the solve status the report prints. `INFEASIBLE` means the reference data admits no record; `UNKNOWN` or `TIME_LIMIT` means the budget expired before a record was found (raise `time_limit_sec` or shrink the decision domains). - For genuine infeasibility: each solution picks one age bucket, so the model is infeasible only when *every* bucket lands on a side of the senior threshold that has no compatible plan-and-provider combination. Confirm there is at least one Medicare-Advantage plan whose network has a provider iff any age bucket has `age_years >= SENIOR_THRESHOLD_YEARS`, and at least one non-Medicare plan whose network has a provider iff any bucket has `age_years < SENIOR_THRESHOLD_YEARS`. The pre-solve coverage check also warns on either-direction asymmetry between `plans.csv` and `providers.csv`; read the startup warnings before assuming the data is sound.
@@ -307,7 +349,7 @@ To extend to a fleet of N members, reintroduce a `Member` concept inside the mod
Warning: provider network(s) [...] have no plans in plans.csv -- The script's symmetric coverage check found a `network_id` value in `providers.csv` that does not appear in any `plans.csv` row. The PCP-network-attribution IC forbids cross-network `(plan, provider)` combinations, so providers in those networks can never be matched to any plan and will never appear in a generated record. The model is still solvable -- this is a warning, not an error. +- The model's symmetric coverage check found a `network_id` value in `providers.csv` that does not appear in any `plans.csv` row. The PCP-network-attribution IC forbids cross-network `(plan, provider)` combinations, so providers in those networks can never be matched to any plan and will never appear in a generated record. The model is still solvable -- this is a warning, not an error. - Add a plan on the listed network(s) to make those providers reachable, or remove the dead provider rows from `providers.csv`.
@@ -326,6 +368,6 @@ To extend to a fleet of N members, reintroduce a `Member` concept inside the mod - `model.where(...)` filters at relational time only -- decision variables are not legal inside it. The rewriter raises this error when it encounters a decision-valued comparison in a `where` clause. - Move the decision condition into `implies` and use a tautological relational filter (or a real one) to scope any reference-data Concepts the IC needs. For example, replace `model.where(Plan.id == plan_id).require(num_dependents <= Plan.max_dependents)` with `model.where(Plan.max_dependents >= 0).require(implies(plan_id == Plan.id, num_dependents <= Plan.max_dependents))`. -- See the three constraint definitions in `synthetic_eligibility_records.py` (`network_match_ic`, `senior_must_medicare_ic`, `non_senior_no_medicare_ic`) for the canonical idiom. +- See the three constraint definitions in `model/__init__.py` (`network_match_ic`, `senior_must_medicare_ic`, `non_senior_no_medicare_ic`) for the canonical idiom.
diff --git a/synthetic_eligibility_records/synthetic_eligibility_records.py b/synthetic_eligibility_records/model/__init__.py similarity index 75% rename from synthetic_eligibility_records/synthetic_eligibility_records.py rename to synthetic_eligibility_records/model/__init__.py index 8b4836dc..54ffbc25 100644 --- a/synthetic_eligibility_records/synthetic_eligibility_records.py +++ b/synthetic_eligibility_records/model/__init__.py @@ -1,4 +1,4 @@ -"""Synthetic Eligibility Records (constrained generation, multi-solution) template. +"""Synthetic Eligibility Records (constrained generation, multi-solution) template: the model. Generate K distinct, internally consistent member eligibility records per solve. Each record is a tuple of three categorical decisions (age bucket, @@ -11,14 +11,9 @@ want a batch of records per solve, not one. Run: - `python synthetic_eligibility_records.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), and - prints the report from the deployed model. - -Output: - Prints the formulation, the solver's status block, and every generated - member record (one row per solution). Prints a no-records diagnostic - when the reference data is over-constrained. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.eligibility_records` then prints the report from the deployed model. """ from pathlib import Path @@ -40,7 +35,7 @@ # to the K records your downstream test fixture wants per solve. MAX_RECORDS = 16 -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # -------------------------------------------------- # Define semantic model & load data @@ -218,60 +213,3 @@ def _warn_orphan_networks(plans_csv, providers_csv): # The solve's status, published through the model's own relationship. SolveStatus = model.Relationship(f"synthetic eligibility solve status is {String:status}") model.define(SolveStatus(problem.termination_status())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and number of solutions of the latest refresh's solve. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - si = problem.solve_info() - print(f"\nSolve result:\n• status: {', '.join(status)}\n• num_points: {si.num_points}\n") - - # Every IC here is implies-bodied (solver-only), so the report does not - # re-evaluate them. The invariants are visible by eye in the table below: - # every row prints `age_years` next to `plan_type` (CFD check) and - # `plan_network` next to `provider_network` (network-attribution check). - if si.num_points is None or si.num_points == 0: - print( - "\nThe solver returned no eligibility records. See the printed solve status " - "above and the troubleshooting section in the README for likely causes " - "(over-constrained reference data, mismatched plan/provider networks, or " - "the search budget expired before a record was found)." - ) - - # ---------------------------------------------- - # Inspect every generated member record - # ---------------------------------------------- - - # `Variable.values(sol_idx, value)` indexes the solver's outputs across - # every returned solution; binding the value slot to a reference Concept's - # `.id` walks the chosen ID back to that record's columns in one step. - - sol_idx = Integer.ref() - records_df = ( - model.select( - sol_idx.alias("solution"), - AgeBucket.age_years.alias("age_years"), - Plan.plan_type.alias("plan_type"), - Plan.network_id.alias("plan_network"), - Provider.network_id.alias("provider_network"), - Provider.name.alias("provider"), - ) - .where( - age_bucket_var.values(sol_idx, AgeBucket.id), - plan_id_var.values(sol_idx, Plan.id), - provider_id_var.values(sol_idx, Provider.id), - ) - .to_df() - .sort_values("solution") - .reset_index(drop=True) - ) - print(f"\nGenerated member records (up to {MAX_RECORDS} per run):") - print(records_df.to_string(index=False)) diff --git a/synthetic_eligibility_records/queries/eligibility_records.py b/synthetic_eligibility_records/queries/eligibility_records.py new file mode 100644 index 00000000..016da0bb --- /dev/null +++ b/synthetic_eligibility_records/queries/eligibility_records.py @@ -0,0 +1,70 @@ +"""Eligibility records report: reads the deployed synthetic_eligibility_records model. + +Run `python -m queries.eligibility_records` after `rai models deploy --path model`. Prints +the formulation, the termination status and number of solutions of the latest refresh's +solve, and every generated member record (one row per solution). Prints a no-records +diagnostic when the reference data is over-constrained. +""" + +from model import ( + MAX_RECORDS, + AgeBucket, + Plan, + Provider, + SolveStatus, + age_bucket_var, + model, + plan_id_var, + problem, + provider_id_var, +) +from relationalai.semantics import Integer + +if __name__ == "__main__": + problem.display() + + # Status and number of solutions of the latest refresh's solve. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + si = problem.solve_info() + print(f"\nSolve result:\n• status: {', '.join(status)}\n• num_points: {si.num_points}\n") + + if si.num_points is None or si.num_points == 0: + print( + "\nThe solver returned no eligibility records. See the printed solve status " + "above and the troubleshooting section in the README for likely causes " + "(over-constrained reference data, mismatched plan/provider networks, or " + "the search budget expired before a record was found)." + ) + + # ---------------------------------------------- + # Inspect every generated member record + # ---------------------------------------------- + + # `Variable.values(sol_idx, value)` indexes the solver's outputs across + # every returned solution; binding the value slot to a reference Concept's + # `.id` walks the chosen ID back to that record's columns in one step. + # Every row prints `age_years` next to `plan_type` and `plan_network` next to + # `provider_network`, so the age-by-plan rule and the network attribution + # can be read off the table. + + sol_idx = Integer.ref() + records_df = ( + model.select( + sol_idx.alias("solution"), + AgeBucket.age_years.alias("age_years"), + Plan.plan_type.alias("plan_type"), + Plan.network_id.alias("plan_network"), + Provider.network_id.alias("provider_network"), + Provider.name.alias("provider"), + ) + .where( + age_bucket_var.values(sol_idx, AgeBucket.id), + plan_id_var.values(sol_idx, Plan.id), + provider_id_var.values(sol_idx, Provider.id), + ) + .to_df() + .sort_values("solution") + .reset_index(drop=True) + ) + print(f"\nGenerated member records (up to {MAX_RECORDS} per run):") + print(records_df.to_string(index=False)) From ffe35be60d882a7b056a96efdfee72185db0e037 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:45:00 -0800 Subject: [PATCH 063/162] test_data_generation: split into a deployed model package and a query Move the model into model/ (deployed with `rai models deploy --path model`) and the report, including the record generator that only the report uses, into queries/test_data_plan.py (`python -m queries.test_data_plan`). Rewrite the README Quickstart as the seven deployment steps: download, environment, install, configure, deploy, query, tear down, and replace the expected output with the current run. The report's status check raises an explicit error instead of using assert. --- test_data_generation/README.md | 147 ++++++++++++------ .../__init__.py} | 128 +-------------- .../queries/test_data_plan.py | 106 +++++++++++++ 3 files changed, 210 insertions(+), 171 deletions(-) rename test_data_generation/{test_data_generation.py => model/__init__.py} (54%) create mode 100644 test_data_generation/queries/test_data_plan.py diff --git a/test_data_generation/README.md b/test_data_generation/README.md index 48a1e62f..0e45bf93 100644 --- a/test_data_generation/README.md +++ b/test_data_generation/README.md @@ -20,7 +20,7 @@ Generating realistic test databases requires careful coordination of row counts This template uses **Prescriptive** reasoning to find optimal row counts for each table in a test database. It encodes referential integrity constraints, cardinality bounds, mandatory participation rules, and coverage requirements as a linear program, then minimizes weighted deviation from target row counts. Higher-priority tables stay closer to their targets. -After determining row counts, the script generates actual test data records with realistic values -- emails, dates, prices, and proper foreign key references -- giving you a complete, constraint-consistent test dataset. +After determining row counts, the report generates actual test data records with realistic values -- emails, dates, prices, and proper foreign key references -- giving you a complete, constraint-consistent test dataset. ## Who this is for @@ -37,7 +37,8 @@ After determining row counts, the script generates actual test data records with ## What's included -- `test_data_generation.py` -- Main script for row count optimization and data generation +- `model/__init__.py` -- Row count optimization model; `rai models deploy --path model` deploys it to your Snowflake account, where each refresh runs the solve +- `queries/test_data_plan.py` -- Prints the report from the deployed model: the optimal row counts and the generated test data - `data/testgen_schema.csv` -- Table and column definitions with types and foreign keys - `data/testgen_constraints.csv` -- Cardinality bounds, coverage, and participation rules - `data/testgen_targets.csv` -- Target, minimum, and maximum row counts per table with priorities @@ -55,28 +56,60 @@ After determining row counts, the script generates actual test data records with ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .TEST_DATA_GENERATION_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/test_data_generation.zip), unzip it, and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/test_data_generation.zip unzip test_data_generation.zip cd test_data_generation ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: + ```bash python -m pip install . ``` -4. Configure: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: ```yaml reasoners: @@ -97,50 +130,35 @@ After determining row counts, the script generates actual test data records with pat_warehouse: ``` - Without the `deployment` section, `python test_data_generation.py` stops with an error naming the missing setting. +5. **Deploy the model** - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .TEST_DATA_GENERATION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .TEST_DATA_GENERATION_META TO APPLICATION ; + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.TEST_DATA_GENERATION` if it doesn't exist. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. -5. Run: - ```bash - python test_data_generation.py - ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: - ```sql - CALL .TEST_DATA_GENERATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); +6. **Query the results** + + Print the report from the deployed model: + + ```bash + python -m queries.test_data_plan ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. -6. Expected output: ```text Status: OPTIMAL - Total weighted deviation: 1200.00 + Total weighted deviation: 18000.00 Optimal row counts: Customer: 100 rows (target: 100, deviation: 0) + Order: 1200 rows (target: 800, deviation: 400) + OrderLine: 3600 rows (target: 5000, deviation: 1400) Product: 500 rows (target: 500, deviation: 0) - Order: 800 rows (target: 800, deviation: 0) - OrderLine: 2400 rows (target: 5000, deviation: 2600) Supplier: 20 rows (target: 20, deviation: 0) SupplierProduct: 1500 rows (target: 1500, deviation: 0) @@ -149,24 +167,53 @@ After determining row counts, the script generates actual test data records with ================================================== Customer: 100 rows - customer_id email region created_date - 1 user1_abcd@gmail.com North 2020-03-15 + customer_id email region created_date + 1 user1_qahf@gmail.com Central 2018-12-23 + 2 user2_pacg@gmail.com Central 2020-03-25 + 3 user3_sskl@outlook.com North 2019-10-16 Product: 500 rows - product_id name category price - 1 Electronics Item 1 Clothing 42.99 + product_id name category price + 1 Books Item 1 Electronics 947.63 + 2 Clothing Item 2 Clothing 265.37 + 3 Electronics Item 3 Clothing 240.62 + + Order: 1200 rows + order_id customer_id order_date status + 1 77 2024-11-19 pending + 2 32 2023-02-15 cancelled + 3 57 2023-08-28 shipped + ... + ``` + + The formulation printed above the status line is trimmed here, and so are the sample rows of the remaining tables (OrderLine, Supplier, SupplierProduct) that the report also prints. The report stops unless the termination status is `OPTIMAL`. A table can finish away from its target when the foreign-key bounds are tight: here Order and OrderLine move off their targets (800 and 5000 rows) to 1200 and 3600 rows, for a total weighted deviation of 18000. The generated rows come from a fixed random seed, so the sample is repeatable. + + To solve again without changing the model, refresh the deployed model, then run the query again: - Order: 800 rows - order_id customer_id order_date status - 1 47 2023-06-12 delivered + ```sql + CALL .TEST_DATA_GENERATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force ``` + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. + ## Template structure ```text . ├── README.md ├── pyproject.toml -├── test_data_generation.py +├── model/ +│ └── __init__.py +├── queries/ +│ └── test_data_plan.py └── data/ ├── testgen_constraints.csv ├── testgen_schema.csv @@ -177,7 +224,7 @@ After determining row counts, the script generates actual test data records with ### 1. Load schema metadata -The script reads three CSV files describing the database schema: table definitions with column types and foreign keys, constraints (cardinality bounds, coverage, mandatory participation), and row count targets with priorities: +The model reads three CSV files describing the database schema: table definitions with column types and foreign keys, constraints (cardinality bounds, coverage, mandatory participation), and row count targets with priorities: ```python schema_df = read_csv(data_dir / "testgen_schema.csv") @@ -222,7 +269,7 @@ problem.minimize(total_deviation) ### 5. Generate test records -After solving, the script generates actual data records with realistic values using the computed row counts: +After the deploy, the report generates actual data records with realistic values using the computed row counts: ```python generated_data = generate_test_data(row_counts) @@ -258,5 +305,5 @@ Make sure you activated the virtual environment and ran `python -m pip install .
Connection or authentication errors -Run `rai init` to configure your Snowflake connection. Verify that your account has the RAI Native App installed and that your user has the required permissions. +Re-create `raiconfig.yaml` with the configuration builder (Quickstart step 4) and verify the connection settings. Verify that your account has the RAI Native App installed and that your user has the required permissions.
diff --git a/test_data_generation/test_data_generation.py b/test_data_generation/model/__init__.py similarity index 54% rename from test_data_generation/test_data_generation.py rename to test_data_generation/model/__init__.py index 112ad74f..728f78ee 100644 --- a/test_data_generation/test_data_generation.py +++ b/test_data_generation/model/__init__.py @@ -1,31 +1,23 @@ -"""Test Data Generation (prescriptive optimization) template. +"""Test Data Generation (prescriptive optimization) template: the model. -This script demonstrates an LP-based test data sizing workflow in RelationalAI: +This package demonstrates an LP-based test data sizing workflow in RelationalAI: - Load schema definitions, referential integrity constraints, and row-count targets from CSV. - Model database tables as *concepts* with target, min, and max row counts. - Solve an LP to choose feasible table row counts close to targets, respecting foreign key relationships, cardinality bounds, and coverage requirements. -- Generate synthetic records consistent with the solved row counts. Run: - `python test_data_generation.py` deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from - the deployed model. - -Output: - Prints the solver status, total weighted deviation, optimal row counts per - table, and a small sample of generated rows for each table. + `rai models deploy --path model` deploys this model to the schema in raiconfig.yaml + and waits for its first refresh, which runs the solve. + `python -m queries.test_data_plan` then prints the report from the deployed model. """ -import random -import string as string_module -from datetime import date, timedelta from pathlib import Path import pandas as pd -from pandas import DataFrame, read_csv +from pandas import read_csv from relationalai.semantics import Float, Integer, Model, String from relationalai.semantics import sum as rai_sum from relationalai.semantics.reasoners.prescriptive import Problem @@ -37,7 +29,7 @@ # Define semantic model & load data # -------------------------------------------------- -data_dir = Path(__file__).parent / "data" +data_dir = Path(__file__).resolve().parents[1] / "data" scale_factor = 1.0 # Load schema and constraint data @@ -70,11 +62,6 @@ Table.x_actual_rows = Property(f"{Table} has actual {Float:actual_rows}") Table.x_deviation = Property(f"{Table} has {Float:deviation}") -# Build lookup for table objects by name -table_objs = {} -for _, row in targets_df.iterrows(): - table_objs[row['table_name']] = row - # Extract FK relationships from schema fk_df = schema_df[schema_df['is_foreign_key']].copy() cardinality_constraints = constraints_df[ @@ -206,104 +193,3 @@ f"test data generation total weighted deviation is {Float:total_deviation}" ) model.define(SolveStatus(problem.termination_status()), TotalDeviation(problem.objective_value())) - -# -------------------------------------------------- -# Phase 2: Generate actual test data (optional) -# -------------------------------------------------- - -def generate_test_data(row_counts, seed=42): - """Generate actual test data records based on optimal row counts.""" - random.seed(seed) - - def random_email(i): - domains = ['gmail.com', 'yahoo.com', 'outlook.com', 'company.com'] - return f"user{i}_{''.join(random.choices(string_module.ascii_lowercase, k=4))}@{random.choice(domains)}" - - def random_date(start_year=2020, end_year=2024): - start = date(start_year, 1, 1) - end = date(end_year, 12, 31) - delta = (end - start).days - return start + timedelta(days=random.randint(0, delta)) - - def random_float(min_val, max_val): - return round(random.uniform(min_val, max_val), 2) - - generated = {} - - # Customer table - n_customers = row_counts.get('Customer', {}).get('actual', 100) - regions = ['North', 'South', 'East', 'West', 'Central'] - customers = [{'customer_id': i, 'email': random_email(i), 'region': random.choice(regions), 'created_date': random_date(2018, 2023)} for i in range(1, n_customers + 1)] - generated['Customer'] = DataFrame(customers) - - # Product table - n_products = row_counts.get('Product', {}).get('actual', 50) - categories = ['Electronics', 'Clothing', 'Home', 'Sports', 'Books'] - products = [{'product_id': i, 'name': f"{random.choice(categories)} Item {i}", 'category': random.choice(categories), 'price': random_float(0.99, 999.99)} for i in range(1, n_products + 1)] - generated['Product'] = DataFrame(products) - - # Order table - n_orders = row_counts.get('Order', {}).get('actual', 500) - statuses = ['pending', 'shipped', 'delivered', 'cancelled'] - orders = [{'order_id': i, 'customer_id': random.randint(1, max(1, n_customers)), 'order_date': random_date(2023, 2024), 'status': random.choice(statuses)} for i in range(1, n_orders + 1)] - generated['Order'] = DataFrame(orders) - - # OrderLine table - n_orderlines = row_counts.get('OrderLine', {}).get('actual', 1500) - orderlines = [{'orderline_id': i, 'order_id': random.randint(1, max(1, n_orders)), 'product_id': random.randint(1, max(1, n_products)), 'quantity': random.randint(1, 10), 'unit_price': random_float(0.99, 999.99)} for i in range(1, n_orderlines + 1)] - generated['OrderLine'] = DataFrame(orderlines) - - # Supplier table - n_suppliers = row_counts.get('Supplier', {}).get('actual', 50) - countries = ['USA', 'China', 'Germany', 'Japan', 'UK'] - suppliers = [{'supplier_id': i, 'name': f"Supplier_{i}", 'country': random.choice(countries), 'reliability_score': random_float(50.0, 100.0)} for i in range(1, n_suppliers + 1)] - generated['Supplier'] = DataFrame(suppliers) - - # SupplierProduct table - n_supplier_products = row_counts.get('SupplierProduct', {}).get('actual', 200) - supplier_products = [{'supplierproduct_id': i, 'supplier_id': random.randint(1, max(1, n_suppliers)), 'product_id': random.randint(1, max(1, n_products)), 'lead_time_days': random.randint(1, 90), 'unit_cost': random_float(0.50, 500.0)} for i in range(1, n_supplier_products + 1)] - generated['SupplierProduct'] = DataFrame(supplier_products) - - return generated - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - problem.display() - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - deviation = model.select(TotalDeviation).to_df().iloc[:, 0].tolist() - assert status == ["OPTIMAL"], f"termination status {status}" - - print(f"Status: {status[0]}") - print(f"Total weighted deviation: {deviation[0]:.2f}") - - # Extract row counts - row_counts = {} - results_df = model.select(Table.table_name, Table.x_actual_rows, Table.target_rows).to_df() - for _, row in results_df.iterrows(): - actual = int(round(row['actual_rows'])) - target = int(row['target_rows']) - row_counts[row['table_name']] = {'actual': actual, 'target': target} - - print("\nOptimal row counts:") - for table, counts in row_counts.items(): - dev = abs(counts['actual'] - counts['target']) - print(f" {table}: {counts['actual']} rows (target: {counts['target']}, deviation: {dev})") - - # Generate data and show samples - print("\n" + "=" * 50) - print("GENERATED TEST DATA") - print("=" * 50) - - generated_data = generate_test_data(row_counts) - for table, df in generated_data.items(): - print(f"\n{table}: {len(df)} rows") - print(df.head(3).to_string(index=False)) diff --git a/test_data_generation/queries/test_data_plan.py b/test_data_generation/queries/test_data_plan.py new file mode 100644 index 00000000..4ccaa5f5 --- /dev/null +++ b/test_data_generation/queries/test_data_plan.py @@ -0,0 +1,106 @@ +"""Test data plan report: reads the deployed test_data_generation model. + +Run `python -m queries.test_data_plan` after `rai models deploy --path model`. Prints the +formulation, the termination status and total weighted deviation of the latest refresh's +solve, the optimal row count per table, and a small sample of generated rows for each table. +""" + +import random +import string as string_module +from datetime import date, timedelta + +from model import SolveStatus, Table, TotalDeviation, model, problem +from pandas import DataFrame + + +def generate_test_data(row_counts, seed=42): + """Generate actual test data records based on optimal row counts.""" + random.seed(seed) + + def random_email(i): + domains = ['gmail.com', 'yahoo.com', 'outlook.com', 'company.com'] + return f"user{i}_{''.join(random.choices(string_module.ascii_lowercase, k=4))}@{random.choice(domains)}" + + def random_date(start_year=2020, end_year=2024): + start = date(start_year, 1, 1) + end = date(end_year, 12, 31) + delta = (end - start).days + return start + timedelta(days=random.randint(0, delta)) + + def random_float(min_val, max_val): + return round(random.uniform(min_val, max_val), 2) + + generated = {} + + # Customer table + n_customers = row_counts.get('Customer', {}).get('actual', 100) + regions = ['North', 'South', 'East', 'West', 'Central'] + customers = [{'customer_id': i, 'email': random_email(i), 'region': random.choice(regions), 'created_date': random_date(2018, 2023)} for i in range(1, n_customers + 1)] + generated['Customer'] = DataFrame(customers) + + # Product table + n_products = row_counts.get('Product', {}).get('actual', 50) + categories = ['Electronics', 'Clothing', 'Home', 'Sports', 'Books'] + products = [{'product_id': i, 'name': f"{random.choice(categories)} Item {i}", 'category': random.choice(categories), 'price': random_float(0.99, 999.99)} for i in range(1, n_products + 1)] + generated['Product'] = DataFrame(products) + + # Order table + n_orders = row_counts.get('Order', {}).get('actual', 500) + statuses = ['pending', 'shipped', 'delivered', 'cancelled'] + orders = [{'order_id': i, 'customer_id': random.randint(1, max(1, n_customers)), 'order_date': random_date(2023, 2024), 'status': random.choice(statuses)} for i in range(1, n_orders + 1)] + generated['Order'] = DataFrame(orders) + + # OrderLine table + n_orderlines = row_counts.get('OrderLine', {}).get('actual', 1500) + orderlines = [{'orderline_id': i, 'order_id': random.randint(1, max(1, n_orders)), 'product_id': random.randint(1, max(1, n_products)), 'quantity': random.randint(1, 10), 'unit_price': random_float(0.99, 999.99)} for i in range(1, n_orderlines + 1)] + generated['OrderLine'] = DataFrame(orderlines) + + # Supplier table + n_suppliers = row_counts.get('Supplier', {}).get('actual', 50) + countries = ['USA', 'China', 'Germany', 'Japan', 'UK'] + suppliers = [{'supplier_id': i, 'name': f"Supplier_{i}", 'country': random.choice(countries), 'reliability_score': random_float(50.0, 100.0)} for i in range(1, n_suppliers + 1)] + generated['Supplier'] = DataFrame(suppliers) + + # SupplierProduct table + n_supplier_products = row_counts.get('SupplierProduct', {}).get('actual', 200) + supplier_products = [{'supplierproduct_id': i, 'supplier_id': random.randint(1, max(1, n_suppliers)), 'product_id': random.randint(1, max(1, n_products)), 'lead_time_days': random.randint(1, 90), 'unit_cost': random_float(0.50, 500.0)} for i in range(1, n_supplier_products + 1)] + generated['SupplierProduct'] = DataFrame(supplier_products) + + return generated + + +if __name__ == "__main__": + problem.display() + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + deviation = model.select(TotalDeviation).to_df().iloc[:, 0].tolist() + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal row counts: termination status {status}") + + print(f"Status: {status[0]}") + print(f"Total weighted deviation: {deviation[0]:.2f}") + + # Extract row counts + row_counts = {} + results_df = model.select(Table.table_name, Table.x_actual_rows, Table.target_rows).to_df() + for _, row in results_df.iterrows(): + actual = int(round(row['actual_rows'])) + target = int(row['target_rows']) + row_counts[row['table_name']] = {'actual': actual, 'target': target} + + print("\nOptimal row counts:") + for table, counts in row_counts.items(): + dev = abs(counts['actual'] - counts['target']) + print(f" {table}: {counts['actual']} rows (target: {counts['target']}, deviation: {dev})") + + # Generate data and show samples + print("\n" + "=" * 50) + print("GENERATED TEST DATA") + print("=" * 50) + + generated_data = generate_test_data(row_counts) + for table, df in generated_data.items(): + print(f"\n{table}: {len(df)} rows") + print(df.head(3).to_string(index=False)) From d6cc10b72677a65c9f3b097e45a0c71932ad6007 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:26:12 -0800 Subject: [PATCH 064/162] cybersecurity-attack-paths: deploy the model package and query it Move the asset graph, path patterns, and rules into model/ and the report into queries/attack_path_report.py, run with rai models deploy --path model and python -m queries.attack_path_report. Rewrite the Quickstart to the deployed-template steps, and say precisely when the on_attack_path rule cannot be a Property. --- cybersecurity-attack-paths/README.md | 69 +++-- .../cybersecurity_attack_paths.py | 292 ------------------ cybersecurity-attack-paths/model/__init__.py | 135 ++++++++ .../queries/attack_path_report.py | 170 ++++++++++ 4 files changed, 354 insertions(+), 312 deletions(-) delete mode 100644 cybersecurity-attack-paths/cybersecurity_attack_paths.py create mode 100644 cybersecurity-attack-paths/model/__init__.py create mode 100644 cybersecurity-attack-paths/queries/attack_path_report.py diff --git a/cybersecurity-attack-paths/README.md b/cybersecurity-attack-paths/README.md index 189792f0..a4e9a67d 100644 --- a/cybersecurity-attack-paths/README.md +++ b/cybersecurity-attack-paths/README.md @@ -39,7 +39,8 @@ This template enumerates multi-step attack paths across an enterprise asset grap ## What's included -- **Self-contained script**: `cybersecurity_attack_paths.py` runs the full analysis end-to-end +- **Model**: `model/__init__.py` — the asset graph, technique edges, path patterns, and rules that `rai models deploy --path model` deploys to your Snowflake account. +- **Query**: `queries/attack_path_report.py` — prints the kill-chains, point-query routes, exposure ranking, and flagged assets from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Data**: `data/assets.csv` (12 assets) and `data/attack_steps.csv` (16 technique-tagged edges) @@ -59,34 +60,42 @@ This template enumerates multi-step attack paths across an enterprise asset grap ## Quickstart -1. Download and extract this template: +Before you start, install Python 3.10 or later and get a Snowflake role that can deploy models, as described under Access. This template has no prescriptive solve, so it needs no setup beyond that role. + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/cybersecurity-attack-paths.zip). Unzip it and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/cybersecurity-attack-paths.zip unzip cybersecurity-attack-paths.zip cd cybersecurity-attack-paths ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. **Create and activate a virtual environment** + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. **Install dependencies** +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: ```bash python -m pip install . ``` -4. **Configure the Snowflake connection and the deployment** +4. **Configure your project** - Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist: ```yaml deployment: @@ -100,39 +109,59 @@ This template enumerates multi-step attack paths across an enterprise asset grap schedule: standard ``` - Without the `deployment` section, `python cybersecurity_attack_paths.py` stops with an error naming the missing setting. +5. **Deploy the model** -5. **Run the template** + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: ```bash - python cybersecurity_attack_paths.py + rai models deploy --path model ``` - The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The deploy waits for the model's first refresh, which enumerates the kill-chains and flags the assets on them. Without the `deployment` section, it stops with an error naming the missing setting. + +6. **Query the results** - Expected output starts with the kill-chains that reach a crown jewel: + Print the report from the deployed model: + + ```bash + python -m queries.attack_path_report + ``` ```text 3 kill-chain attack path(s) reach a crown jewel (exploit -> cred -> 1-2 pivots, from an internet-facing asset): [3 hops] VPN Gateway --[Asset.exploit_to]--> Jump Host --[Asset.cred_to]--> File Server --[Asset.pivot_to]--> Customer Database ``` - See the runbook for the full output. + The lines above are the start of the report. The full report continues with the routes between one entry point and one crown jewel, the kill-chains ranked by total exposure, and the assets flagged on a crown-jewel attack path. See `runbook.md` for the full output. + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The teardown removes the deployed objects and keeps the two schemas. ## Template structure ```text cybersecurity-attack-paths/ -├── cybersecurity_attack_paths.py ├── pyproject.toml ├── README.md ├── runbook.md +├── model/ +│ └── __init__.py # assets, technique edges, path patterns, and rules +├── queries/ +│ └── attack_path_report.py # report printed from the deployed model └── data/ ├── assets.csv └── attack_steps.csv ``` -**Start here**: run `python cybersecurity_attack_paths.py` for the full analysis end to end, or follow `runbook.md` to rebuild it step by step. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.attack_path_report` for the full analysis end to end, or follow `runbook.md` to rebuild it step by step. ## Sample data @@ -146,7 +175,7 @@ cybersecurity-attack-paths/ - **Primary identifiers**: `Asset` by `id`. - **Important invariants**: the three technique edges are directed (`src` reaches `dst`, not the reverse); `internet_facing` and `crown_jewel` are `yes`/`no` flags that pin the kill-chain endpoints; `on_attack_path` is set only on assets that lie on a crown-jewel-reaching chain. -For the full concept and property definitions — including the four directed self-relationships (`exploit_to`, `cred_to`, `pivot_to`, and the technique-agnostic `can_reach` union edge) — see `cybersecurity_attack_paths.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions — including the four directed self-relationships (`exploit_to`, `cred_to`, `pivot_to`, and the technique-agnostic `can_reach` union edge) — see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -161,7 +190,7 @@ The centerpiece is a multi-edge path pattern (which needs `relationalai>=1.21`) A separate point query pins both endpoints by id to enumerate every route between a chosen entry point and a chosen crown jewel over the union edge. The kill-chains are then ranked by the exposure summed along each one. Finally, the assets lying on any crown-jewel chain are flagged back onto the ontology as `Asset.on_attack_path`, so a later query can pull them without re-enumerating paths. -See `cybersecurity_attack_paths.py` for the implementation and `runbook.md` to reproduce it step by step with the RAI skills. +See `model/__init__.py` for the implementation, `queries/attack_path_report.py` for the report, and `runbook.md` to reproduce it step by step with the RAI skills. ## Customize this template diff --git a/cybersecurity-attack-paths/cybersecurity_attack_paths.py b/cybersecurity-attack-paths/cybersecurity_attack_paths.py deleted file mode 100644 index 09fe9904..00000000 --- a/cybersecurity-attack-paths/cybersecurity_attack_paths.py +++ /dev/null @@ -1,292 +0,0 @@ -"""Cybersecurity attack-path analysis (graph paths) template. - -Enumerates multi-step attack chains across an enterprise asset graph by composing -distinct attacker techniques in series -- a capability unlocked by multi-edge path -patterns (relationalai>=1.21): - -- Loads Asset nodes (hosts, services, accounts) and three DISTINCT directed edges - between them, one per technique: exploit_to (vulnerability exploitation), - cred_to (credential reuse), pivot_to (network lateral movement). -- Enumerates kill-chain attack paths with a multi-relationship sequence: - ``model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, N), dst)`` -- the - classic order (exploit a perimeter host, reuse credentials inward, then pivot to - a crown jewel). The edge order is enforced by the pattern; a single unioned - "can move" edge or a flat join cannot express "exploit FIRST, then creds, then - pivots". ``p.relationships`` labels the technique used at each hop. -- Runs a point query between one named internet-facing entry and one named crown - jewel over a derived ``can_reach`` edge, enumerating every route between them. -- Scores each kill-chain by the asset exposure summed along it, and persists - ``Asset.on_attack_path`` back to the ontology for the assets that lie on a - crown-jewel-reaching chain. - -Run: - `python cybersecurity_attack_paths.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh, and prints the report from the - deployed model. - -Output: - Prints the kill-chain attack paths into crown jewels (with the technique at - each hop), every route between a chosen entry point and a chosen crown jewel, - the kill-chains ranked by total asset exposure, and the assets flagged - on a crown-jewel attack path. -""" - -from pathlib import Path - -import pandas as pd -from relationalai.semantics import Integer, Model, String, not_, where -from relationalai.semantics.std import aggregates as aggs - -# -------------------------------------------------- -# Configure inputs -# -------------------------------------------------- - -DATA_DIR = Path(__file__).parent / "data" - -# Lateral-movement depth: how many pivot_to hops the kill-chain may take after the -# initial exploit + credential reuse. The estate is small, so a low cap suffices. -MAX_PIVOTS = 2 -# Maximum length (in edges) of a route in the point query between two named assets. -MAX_ROUTE_HOPS = 6 -# The point query's endpoints: a specific internet-facing entry and a crown jewel. -ENTRY_ASSET = "web-01" -TARGET_ASSET = "db-01" - - -def load_csv(filename): - return pd.read_csv(DATA_DIR / filename) - - -# -------------------------------------------------- -# Define semantic model & load data -# -------------------------------------------------- - -model = Model("cybersecurity_attack_paths") - -# Asset concept: a host, service, or account in the enterprise estate. -Asset = model.Concept("Asset", identify_by={"id": String}) -Asset.name = model.Property(f"{Asset} has {String:name}") -Asset.zone = model.Property(f"{Asset} sits in {String:zone}") -Asset.internet_facing = model.Property(f"{Asset} is internet facing {String:internet_facing}") -Asset.crown_jewel = model.Property(f"{Asset} is crown jewel {String:crown_jewel}") -Asset.exposure_score = model.Property(f"{Asset} has exposure score {Integer:exposure_score}") - -# One directed self-relationship per attacker technique. An edge means "an attacker -# on the source asset can reach the destination asset by this technique". -Asset.exploit_to = model.Relationship(f"{Asset} exploits to {Asset}", short_name="exploit_to") -Asset.cred_to = model.Relationship(f"{Asset} reuses credentials to {Asset}", short_name="cred_to") -Asset.pivot_to = model.Relationship(f"{Asset} pivots to {Asset}", short_name="pivot_to") -# Technique-agnostic union edge: an attacker can move from src to dst by SOME -# technique. Used for the point query (any route between two named assets). -Asset.can_reach = model.Relationship(f"{Asset} can reach {Asset}", short_name="can_reach") - -# Load asset data. -asset_data = model.data(load_csv("assets.csv")) -model.define(Asset.new(id=asset_data["id"])) -model.where(Asset.id == asset_data["id"]).define( - Asset.name(asset_data["name"]), - Asset.zone(asset_data["zone"]), - Asset.internet_facing(asset_data["internet_facing"]), - Asset.crown_jewel(asset_data["crown_jewel"]), - Asset.exposure_score(asset_data["exposure_score"]), -) - -# Load attack steps. Each row is one technique-specific edge; split by technique to -# populate the three typed edges, and load every row into the can_reach union. -steps_df = load_csv("attack_steps.csv") -for technique, rel in [("exploit", "exploit_to"), ("cred", "cred_to"), ("pivot", "pivot_to")]: - sub = model.data(steps_df[steps_df["technique"] == technique][["src", "dst"]]) - u, v = Asset.ref(), Asset.ref() - model.where( - u.id == sub["src"], - v.id == sub["dst"], - ).define(getattr(u, rel)(v)) - -all_steps = model.data(steps_df[["src", "dst"]]) -ru, rv = Asset.ref(), Asset.ref() -model.where( - ru.id == all_steps["src"], - rv.id == all_steps["dst"], -).define(ru.can_reach(rv)) - -# -------------------------------------------------- -# Paths: kill-chain attack paths (multi-relationship sequence) -# PREVIEW capability; requires relationalai>=1.21. -# -------------------------------------------------- -# model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, MAX_PIVOTS), dst) is a -# MULTI-EDGE pattern: distinct relationships in series. It matches the kill-chain -# order -- one exploit hop, then one credential-reuse hop, then 1..MAX_PIVOTS -# lateral pivots -- ending at the explicit dst endpoint. Filtering the source to -# an internet-facing asset and dst to a crown jewel pins the threat model: an -# externally reachable foothold that ends on a high-value asset. - -a, b, c, dst = Asset.ref(), Asset.ref(), Asset.ref(), Asset.ref() -kill = model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, MAX_PIVOTS), dst).all().paths() - -# The point query's pattern: routes over the technique-agnostic can_reach edge from -# a pinned entry asset to a pinned crown jewel, 1..MAX_ROUTE_HOPS hops. -src_pt, dst_pt = Asset.ref(), Asset.ref() -route = model.path(src_pt.can_reach.repeat(1, MAX_ROUTE_HOPS), dst_pt).all().paths() - -# Asset.on_attack_path marks the assets that lie on a crown-jewel kill-chain, so a -# downstream query can pull them without re-enumerating paths. It is a -# Relationship, not a Property: on relationalai 2.1.1, deploying a Property -# written from a path rule fails planning. -Asset.on_attack_path = model.Relationship(f"{Asset} on attack path {String:on_attack_path}") -on_path, pos = Asset.ref(), Integer.ref() -model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - on_path == kill.nodes(pos), -).define(on_path.on_attack_path("yes")) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh. - model.deploy() - - print("=== Cybersecurity Attack Paths: kill-chain into crown jewels ===") - kill_df = ( - model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - ) - .select( - kill.alias("path_id"), - kill.nodes["index"].alias("step"), - Asset(kill.nodes).id.alias("asset_id"), - Asset(kill.nodes).name.alias("asset_name"), - ) - .to_df() - ) - # Technique at each hop (the relationship label), as a separate projection. - hop_df = ( - model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - ) - .select( - kill.alias("path_id"), - kill.relationships["index"].alias("hop"), - kill.relationships["relationship"].alias("technique"), - ) - .to_df() - ) - - kill_df["step"] = kill_df["step"].astype(int) - kill_df = kill_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) - hop_df["hop"] = hop_df["hop"].astype(int) - hop_df = hop_df.drop_duplicates(["path_id", "hop"]).sort_values(["path_id", "hop"]) - - # Reassemble each kill-chain: ordered asset names + the technique used at each hop. - # The technique is the relationship label as-is -- the dotted relationship name, - # e.g. "Asset.exploit_to". - chains = [] - for pid, g in kill_df.groupby("path_id"): - assets = list(g.sort_values("step")["asset_name"]) - techs = list(hop_df[hop_df["path_id"] == pid].sort_values("hop")["technique"]) - labelled = assets[0] - for nm, tech in zip(assets[1:], techs): - labelled += f" --[{tech}]--> {nm}" - chains.append({"path_id": pid, "hops": len(assets) - 1, "asset_ids": tuple( - g.sort_values("step")["asset_id"]), "labelled": labelled}) - - print(f"\n{len(chains)} kill-chain attack path(s) reach a crown jewel " - f"(exploit -> cred -> 1-{MAX_PIVOTS} pivots, from an internet-facing asset):") - for ch in sorted(chains, key=lambda c: c["hops"]): - print(f" [{ch['hops']} hops] {ch['labelled']}") - - # -------------------------------------------------- - # Point query: every route between one entry and one crown jewel - # -------------------------------------------------- - # Pin both endpoints by id and enumerate all simple routes between them over the - # technique-agnostic can_reach edge (any technique, 1..MAX_ROUTE_HOPS). This is the - # >=1.15-era native point query (still current) -- src/dst unified to specific assets inside .all().paths(). - - i, j = Integer.ref(), Integer.ref() - # Keep only SIMPLE routes (no asset revisited). .all().paths() has WALK semantics and - # no native simple-path mode, so drop node-repeating walks with a negated existential - # over positions -- expressed natively in the route's where() (no pandas). - route_df = ( - model.where( - route, - src_pt.id == ENTRY_ASSET, - dst_pt.id == TARGET_ASSET, - not_(where(i < j, route.nodes(i) == route.nodes(j))), - ) - .select( - route.alias("path_id"), - route.nodes["index"].alias("step"), - Asset(route.nodes).id.alias("asset_id"), - ) - .to_df() - ) - route_df["step"] = route_df["step"].astype(int) - route_df = route_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) - routes = sorted( - {tuple(g.sort_values("step")["asset_id"]) for _, g in route_df.groupby("path_id")}, - key=len, - ) - - print(f"\n=== Point query: all routes from {ENTRY_ASSET} to {TARGET_ASSET} " - f"(any technique, <= {MAX_ROUTE_HOPS} hops, simple) ===") - print(f" {len(routes)} route(s):") - for seq in routes: - print(" " + " -> ".join(seq)) - - # -------------------------------------------------- - # Rank kill-chains by total asset exposure - # -------------------------------------------------- - # The riskiest chain is the one whose assets carry the most exposure overall -- the - # first remediation target. Exposure is a per-asset score (a stand-in for a CVSS / - # attack-surface metric) summed ALONG each kill-chain IN PYREL: - # aggregates.sum(Asset(kill.nodes).exposure_score).per(kill) sums the per-node value - # over each path's node sequence -- the core paths-only aggregation, reusing the same - # kill pattern and its internet-facing/crown-jewel endpoint filters. This is a - # position-sum (every node position, with repetition); it equals the distinct-asset - # sum because each kill-chain here is a simple path (no asset revisited). - totals = ( - model.where( - kill, - a.internet_facing == "yes", - dst.crown_jewel == "yes", - total := aggs.sum(Asset(kill.nodes).exposure_score).per(kill), - ) - .select(kill.alias("path_id"), total.alias("total_exposure")) - .to_df() - ) - # Join total_exposure onto the assembled chains by path id (label string-join is pandas). - exposure_by_path = dict( - totals[["path_id", "total_exposure"]].itertuples(index=False, name=None) - ) - for ch in chains: - ch["total_exposure"] = int(exposure_by_path[ch["path_id"]]) - - print("\n=== Kill-chains ranked by total asset exposure (highest = remediate first) ===") - for ch in sorted(chains, key=lambda c: c["total_exposure"], reverse=True): - print(f" exposure {ch['total_exposure']:>3} ({ch['hops']} hops) {ch['labelled']}") - - # -------------------------------------------------- - # Assets on a crown-jewel attack path - # -------------------------------------------------- - - total_assets = len(model.select(Asset.id.alias("id")).to_df()) - x = Asset.ref() - flagged = ( - model.where( - x.on_attack_path == "yes", - ) - .select(x.id.alias("id"), x.name.alias("name"), x.zone.alias("zone")) - .to_df() - ) - - print("\n=== Assets on a crown-jewel attack path (Asset.on_attack_path persisted) ===") - print(f" {len(flagged)} of {total_assets} assets:") - for _, r in flagged.sort_values("id").iterrows(): - print(f" {r['id']:<8} {r['name']:<24} ({r['zone']})") diff --git a/cybersecurity-attack-paths/model/__init__.py b/cybersecurity-attack-paths/model/__init__.py new file mode 100644 index 00000000..f0f76b69 --- /dev/null +++ b/cybersecurity-attack-paths/model/__init__.py @@ -0,0 +1,135 @@ +"""Cybersecurity attack-path analysis (graph paths) template. + +Enumerates multi-step attack chains across an enterprise asset graph by composing +distinct attacker techniques in series -- a capability unlocked by multi-edge path +patterns (relationalai>=1.21): + +- Loads Asset nodes (hosts, services, accounts) and three DISTINCT directed edges + between them, one per technique: exploit_to (vulnerability exploitation), + cred_to (credential reuse), pivot_to (network lateral movement). +- Enumerates kill-chain attack paths with a multi-relationship sequence: + ``model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, N), dst)`` -- the + classic order (exploit a perimeter host, reuse credentials inward, then pivot to + a crown jewel). The edge order is enforced by the pattern; a single unioned + "can move" edge or a flat join cannot express "exploit FIRST, then creds, then + pivots". ``p.relationships`` labels the technique used at each hop. +- Runs a point query between one named internet-facing entry and one named crown + jewel over a derived ``can_reach`` edge, enumerating every route between them. +- Scores each kill-chain by the asset exposure summed along it, and persists + ``Asset.on_attack_path`` back to the ontology for the assets that lie on a + crown-jewel-reaching chain. + +Run: + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh; `python -m + queries.attack_path_report` then prints the report from the deployed model. +""" + +from pathlib import Path + +import pandas as pd +from relationalai.semantics import Integer, Model, String + +# -------------------------------------------------- +# Configure inputs +# -------------------------------------------------- + +DATA_DIR = Path(__file__).resolve().parents[1] / "data" + +# Lateral-movement depth: how many pivot_to hops the kill-chain may take after the +# initial exploit + credential reuse. The estate is small, so a low cap suffices. +MAX_PIVOTS = 2 +# Maximum length (in edges) of a route in the point query between two named assets. +MAX_ROUTE_HOPS = 6 +# The point query's endpoints: a specific internet-facing entry and a crown jewel. +ENTRY_ASSET = "web-01" +TARGET_ASSET = "db-01" + + +def load_csv(filename): + return pd.read_csv(DATA_DIR / filename) + + +# -------------------------------------------------- +# Define semantic model & load data +# -------------------------------------------------- + +model = Model("cybersecurity_attack_paths") + +# Asset concept: a host, service, or account in the enterprise estate. +Asset = model.Concept("Asset", identify_by={"id": String}) +Asset.name = model.Property(f"{Asset} has {String:name}") +Asset.zone = model.Property(f"{Asset} sits in {String:zone}") +Asset.internet_facing = model.Property(f"{Asset} is internet facing {String:internet_facing}") +Asset.crown_jewel = model.Property(f"{Asset} is crown jewel {String:crown_jewel}") +Asset.exposure_score = model.Property(f"{Asset} has exposure score {Integer:exposure_score}") + +# One directed self-relationship per attacker technique. An edge means "an attacker +# on the source asset can reach the destination asset by this technique". +Asset.exploit_to = model.Relationship(f"{Asset} exploits to {Asset}", short_name="exploit_to") +Asset.cred_to = model.Relationship(f"{Asset} reuses credentials to {Asset}", short_name="cred_to") +Asset.pivot_to = model.Relationship(f"{Asset} pivots to {Asset}", short_name="pivot_to") +# Technique-agnostic union edge: an attacker can move from src to dst by SOME +# technique. Used for the point query (any route between two named assets). +Asset.can_reach = model.Relationship(f"{Asset} can reach {Asset}", short_name="can_reach") + +# Load asset data. +asset_data = model.data(load_csv("assets.csv")) +model.define(Asset.new(id=asset_data["id"])) +model.where(Asset.id == asset_data["id"]).define( + Asset.name(asset_data["name"]), + Asset.zone(asset_data["zone"]), + Asset.internet_facing(asset_data["internet_facing"]), + Asset.crown_jewel(asset_data["crown_jewel"]), + Asset.exposure_score(asset_data["exposure_score"]), +) + +# Load attack steps. Each row is one technique-specific edge; split by technique to +# populate the three typed edges, and load every row into the can_reach union. +steps_df = load_csv("attack_steps.csv") +for technique, rel in [("exploit", "exploit_to"), ("cred", "cred_to"), ("pivot", "pivot_to")]: + sub = model.data(steps_df[steps_df["technique"] == technique][["src", "dst"]]) + u, v = Asset.ref(), Asset.ref() + model.where( + u.id == sub["src"], + v.id == sub["dst"], + ).define(getattr(u, rel)(v)) + +all_steps = model.data(steps_df[["src", "dst"]]) +ru, rv = Asset.ref(), Asset.ref() +model.where( + ru.id == all_steps["src"], + rv.id == all_steps["dst"], +).define(ru.can_reach(rv)) + +# -------------------------------------------------- +# Paths: kill-chain attack paths (multi-relationship sequence) +# PREVIEW capability; requires relationalai>=1.21. +# -------------------------------------------------- +# model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, MAX_PIVOTS), dst) is a +# MULTI-EDGE pattern: distinct relationships in series. It matches the kill-chain +# order -- one exploit hop, then one credential-reuse hop, then 1..MAX_PIVOTS +# lateral pivots -- ending at the explicit dst endpoint. Filtering the source to +# an internet-facing asset and dst to a crown jewel pins the threat model: an +# externally reachable foothold that ends on a high-value asset. + +a, b, c, dst = Asset.ref(), Asset.ref(), Asset.ref(), Asset.ref() +kill = model.path(a.exploit_to, b.cred_to, c.pivot_to.repeat(1, MAX_PIVOTS), dst).all().paths() + +# The point query's pattern: routes over the technique-agnostic can_reach edge from +# a pinned entry asset to a pinned crown jewel, 1..MAX_ROUTE_HOPS hops. +src_pt, dst_pt = Asset.ref(), Asset.ref() +route = model.path(src_pt.can_reach.repeat(1, MAX_ROUTE_HOPS), dst_pt).all().paths() + +# Asset.on_attack_path marks the assets that lie on a crown-jewel kill-chain, so a +# downstream query can pull them without re-enumerating paths. It is a +# Relationship, not a Property: deploying this rule as a Property fails planning +# on relationalai 2.1.1. +Asset.on_attack_path = model.Relationship(f"{Asset} on attack path {String:on_attack_path}") +on_path, pos = Asset.ref(), Integer.ref() +model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + on_path == kill.nodes(pos), +).define(on_path.on_attack_path("yes")) diff --git a/cybersecurity-attack-paths/queries/attack_path_report.py b/cybersecurity-attack-paths/queries/attack_path_report.py new file mode 100644 index 00000000..a9cc4c76 --- /dev/null +++ b/cybersecurity-attack-paths/queries/attack_path_report.py @@ -0,0 +1,170 @@ +"""Attack-path report from the deployed model. + +Run after `rai models deploy --path model`: + python -m queries.attack_path_report + +Prints the kill-chain attack paths into crown jewels (with the technique at each +hop), every route between a chosen entry point and a chosen crown jewel, the +kill-chains ranked by total asset exposure, and the assets flagged on a +crown-jewel attack path. +""" + +from model import ( + ENTRY_ASSET, + MAX_PIVOTS, + MAX_ROUTE_HOPS, + TARGET_ASSET, + Asset, + a, + dst, + dst_pt, + kill, + model, + route, + src_pt, +) +from relationalai.semantics import Integer, not_, where +from relationalai.semantics.std import aggregates as aggs + +if __name__ == "__main__": + print("=== Cybersecurity Attack Paths: kill-chain into crown jewels ===") + kill_df = ( + model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + ) + .select( + kill.alias("path_id"), + kill.nodes["index"].alias("step"), + Asset(kill.nodes).id.alias("asset_id"), + Asset(kill.nodes).name.alias("asset_name"), + ) + .to_df() + ) + # Technique at each hop (the relationship label), as a separate projection. + hop_df = ( + model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + ) + .select( + kill.alias("path_id"), + kill.relationships["index"].alias("hop"), + kill.relationships["relationship"].alias("technique"), + ) + .to_df() + ) + + kill_df["step"] = kill_df["step"].astype(int) + kill_df = kill_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) + hop_df["hop"] = hop_df["hop"].astype(int) + hop_df = hop_df.drop_duplicates(["path_id", "hop"]).sort_values(["path_id", "hop"]) + + # Reassemble each kill-chain: ordered asset names + the technique used at each hop. + # The technique is the relationship label as-is -- the dotted relationship name, + # e.g. "Asset.exploit_to". + chains = [] + for pid, g in kill_df.groupby("path_id"): + assets = list(g.sort_values("step")["asset_name"]) + techs = list(hop_df[hop_df["path_id"] == pid].sort_values("hop")["technique"]) + labelled = assets[0] + for nm, tech in zip(assets[1:], techs): + labelled += f" --[{tech}]--> {nm}" + chains.append({"path_id": pid, "hops": len(assets) - 1, "asset_ids": tuple( + g.sort_values("step")["asset_id"]), "labelled": labelled}) + + print(f"\n{len(chains)} kill-chain attack path(s) reach a crown jewel " + f"(exploit -> cred -> 1-{MAX_PIVOTS} pivots, from an internet-facing asset):") + for ch in sorted(chains, key=lambda c: c["hops"]): + print(f" [{ch['hops']} hops] {ch['labelled']}") + + # -------------------------------------------------- + # Point query: every route between one entry and one crown jewel + # -------------------------------------------------- + # Pin both endpoints by id and enumerate all simple routes between them over the + # technique-agnostic can_reach edge (any technique, 1..MAX_ROUTE_HOPS). This is the + # >=1.15-era native point query (still current) -- src/dst unified to specific assets inside .all().paths(). + + i, j = Integer.ref(), Integer.ref() + # Keep only SIMPLE routes (no asset revisited). .all().paths() has WALK semantics and + # no native simple-path mode, so drop node-repeating walks with a negated existential + # over positions -- expressed natively in the route's where() (no pandas). + route_df = ( + model.where( + route, + src_pt.id == ENTRY_ASSET, + dst_pt.id == TARGET_ASSET, + not_(where(i < j, route.nodes(i) == route.nodes(j))), + ) + .select( + route.alias("path_id"), + route.nodes["index"].alias("step"), + Asset(route.nodes).id.alias("asset_id"), + ) + .to_df() + ) + route_df["step"] = route_df["step"].astype(int) + route_df = route_df.drop_duplicates(["path_id", "step"]).sort_values(["path_id", "step"]) + routes = sorted( + {tuple(g.sort_values("step")["asset_id"]) for _, g in route_df.groupby("path_id")}, + key=len, + ) + + print(f"\n=== Point query: all routes from {ENTRY_ASSET} to {TARGET_ASSET} " + f"(any technique, <= {MAX_ROUTE_HOPS} hops, simple) ===") + print(f" {len(routes)} route(s):") + for seq in routes: + print(" " + " -> ".join(seq)) + + # -------------------------------------------------- + # Rank kill-chains by total asset exposure + # -------------------------------------------------- + # The riskiest chain is the one whose assets carry the most exposure overall -- the + # first remediation target. Exposure is a per-asset score (a stand-in for a CVSS / + # attack-surface metric) summed ALONG each kill-chain IN PYREL: + # aggregates.sum(Asset(kill.nodes).exposure_score).per(kill) sums the per-node value + # over each path's node sequence -- the core paths-only aggregation, reusing the same + # kill pattern and its internet-facing/crown-jewel endpoint filters. This is a + # position-sum (every node position, with repetition); it equals the distinct-asset + # sum because each kill-chain here is a simple path (no asset revisited). + totals = ( + model.where( + kill, + a.internet_facing == "yes", + dst.crown_jewel == "yes", + total := aggs.sum(Asset(kill.nodes).exposure_score).per(kill), + ) + .select(kill.alias("path_id"), total.alias("total_exposure")) + .to_df() + ) + # Join total_exposure onto the assembled chains by path id (label string-join is pandas). + exposure_by_path = dict( + totals[["path_id", "total_exposure"]].itertuples(index=False, name=None) + ) + for ch in chains: + ch["total_exposure"] = int(exposure_by_path[ch["path_id"]]) + + print("\n=== Kill-chains ranked by total asset exposure (highest = remediate first) ===") + for ch in sorted(chains, key=lambda c: c["total_exposure"], reverse=True): + print(f" exposure {ch['total_exposure']:>3} ({ch['hops']} hops) {ch['labelled']}") + + # -------------------------------------------------- + # Assets on a crown-jewel attack path + # -------------------------------------------------- + + total_assets = len(model.select(Asset.id.alias("id")).to_df()) + x = Asset.ref() + flagged = ( + model.where( + x.on_attack_path == "yes", + ) + .select(x.id.alias("id"), x.name.alias("name"), x.zone.alias("zone")) + .to_df() + ) + + print("\n=== Assets on a crown-jewel attack path (Asset.on_attack_path persisted) ===") + print(f" {len(flagged)} of {total_assets} assets:") + for _, r in flagged.sort_values("id").iterrows(): + print(f" {r['id']:<8} {r['name']:<24} ({r['zone']})") From 33a51b4bb5682ef5a6e0469fa5e1f4a2f18c39aa Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:29:49 -0800 Subject: [PATCH 065/162] it-dependency-mapping: deploy the model package and query it Move the features, dependency edges, path pattern, and depth rule into model/ and the report into queries/dependency_chain_report.py, run with rai models deploy --path model and python -m queries.dependency_chain_report. Rewrite the Quickstart to the deployed-template steps. The depth rule is a relationship, not a property: deploying it as a Property fails planning on relationalai 2.1.1, and the README now says relationship. --- it-dependency-mapping/README.md | 73 +++++++---- it-dependency-mapping/model/__init__.py | 103 +++++++++++++++ .../dependency_chain_report.py} | 117 ++---------------- 3 files changed, 161 insertions(+), 132 deletions(-) create mode 100644 it-dependency-mapping/model/__init__.py rename it-dependency-mapping/{it_dependency_mapping.py => queries/dependency_chain_report.py} (56%) diff --git a/it-dependency-mapping/README.md b/it-dependency-mapping/README.md index 0354f812..c4eea2bb 100644 --- a/it-dependency-mapping/README.md +++ b/it-dependency-mapping/README.md @@ -44,7 +44,8 @@ Built using RelationalAI's **variable-length path enumeration** (`model.path(... ## What's included -- **Self-contained script**: `it_dependency_mapping.py` -- Runs the full analysis end-to-end +- **Model**: `model/__init__.py` -- the features, dependency edges, path pattern, and depth rule that `rai models deploy --path model` deploys to your Snowflake account. +- **Query**: `queries/dependency_chain_report.py` -- prints the path counts, maximal chains, and owner blast radius from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Data**: `data/features.csv` (14 features across tiers) and `data/dependencies.csv` (15 dependency edges) @@ -64,34 +65,42 @@ Built using RelationalAI's **variable-length path enumeration** (`model.path(... ## Quickstart -1. Download and extract this template: +Before you start, install Python 3.10 or later and get a Snowflake role that can deploy models, as described under Access. This template has no prescriptive solve, so it needs no setup beyond that role. + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/it-dependency-mapping.zip). Unzip it and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/it-dependency-mapping.zip unzip it-dependency-mapping.zip cd it-dependency-mapping ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. **Create and activate a virtual environment** + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. **Install dependencies** +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: ```bash python -m pip install . ``` -4. **Configure the Snowflake connection and the deployment** +4. **Configure your project** - Run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist: ```yaml deployment: @@ -105,17 +114,23 @@ Built using RelationalAI's **variable-length path enumeration** (`model.path(... schedule: standard ``` - Without the `deployment` section, `python it_dependency_mapping.py` stops with an error naming the missing setting. +5. **Deploy the model** -5. **Run the template** + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: ```bash - python it_dependency_mapping.py + rai models deploy --path model ``` - The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + The deploy waits for the model's first refresh, which enumerates the downstream paths and computes each feature's longest downstream depth. Without the `deployment` section, it stops with an error naming the missing setting. + +6. **Query the results** -6. **Expected output** (a few lines confirm success): + Print the report from the deployed model: + + ```bash + python -m queries.dependency_chain_report + ``` ```text Total downstream dependency paths (1-8 hops): 46 @@ -124,21 +139,35 @@ Built using RelationalAI's **variable-length path enumeration** (`model.path(... === Longest dependency chain: 5 hops === ``` - The full printout -- per-feature reach, all six maximal chains, and the owners along the longest chain -- is in the `## Expected output` section below. + The lines above are excerpts of the report. The full printout, with per-feature reach, all six maximal chains, and the owners along the longest chain, is in the Expected output section below. + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The teardown removes the deployed objects and keeps the two schemas. ## Template structure ```text it-dependency-mapping/ -├── it_dependency_mapping.py # Self-contained analysis script ├── pyproject.toml # Dependencies and project metadata ├── README.md # This file +├── model/ +│ └── __init__.py # features, dependency edges, path pattern, and depth rule +├── queries/ +│ └── dependency_chain_report.py # report printed from the deployed model └── data/ ├── features.csv # 14 features (id, name, owner, deploy_tier) └── dependencies.csv # 15 dependency edges (from_feature, to_feature) ``` -**Start here**: run `python it_dependency_mapping.py` for the full analysis end to end, or follow `runbook.md` to rebuild it step by step. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.dependency_chain_report` for the full analysis end to end, or follow `runbook.md` to rebuild it step by step. ## Sample data @@ -168,7 +197,7 @@ A single concept, `Feature`, represents every node in the estate, and one self-r - **Primary identifiers**: the string `id` on `Feature`. - **Important invariants**: the `contributes_to` graph is acyclic — a feature does not depend on itself, so every enumerated traversal path is a simple chain. -For the full concept and property definitions, see `it_dependency_mapping.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -180,11 +209,11 @@ CSV files --> Define Feature + contributes_to --> Enumerate downstream paths --> **Enumerate downstream paths.** A variable-length traversal of 1 to `MAX_DEPTH` `contributes_to` edges, enumerated with `.all().paths()`, walks every downstream dependency path. Each path arrives as one row per visited node; grouping by path reassembles the ordered chain, and the hop count comes from the maximum node index. (Selecting the path length alongside the nodes would fan the rows out, so the code derives hops from the index and dedupes before reassembly.) -**Persist longest downstream depth.** The longest path starting at each feature is its downstream depth. RelationalAI computes it in PyRel — a max over path length per source endpoint — and defines it straight onto the feature as a first-class `max_downstream_depth` property, with no DataFrame round-trip, so later queries can rank features by reach without re-enumerating paths. +**Persist longest downstream depth.** The longest path starting at each feature is its downstream depth. RelationalAI computes it in PyRel — a max over path length per source endpoint — and defines it straight onto the feature as a first-class `max_downstream_depth` relationship, with no DataFrame round-trip, so later queries can rank features by reach without re-enumerating paths. **Reduce to maximal chains.** The full path set contains every sub-chain. A path is *maximal* when its node sequence is not a contiguous sub-chain of any other enumerated path -- it cannot be extended upstream or downstream. Filtering to maximal chains leaves only the end-to-end propagation paths. The single longest maximal chain is the worst-case blast radius: joining owner metadata onto its features shows every person a change at the root would have to clear before it reaches the final consumer. -See `it_dependency_mapping.py` for the implementation, and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation, `queries/dependency_chain_report.py` for the report, and `runbook.md` for the skill-driven reproduction. ## Expected output @@ -221,7 +250,7 @@ The deepest chain runs five hops from a raw source all the way to a downstream d ### Tune parameters -- `MAX_DEPTH` (default `8`, at the top of `it_dependency_mapping.py`) bounds the traversal length. Raise it if your estate has longer chains than the sample; lower it to cap enumeration on a denser graph. +- `MAX_DEPTH` (default `8`, at the top of `model/__init__.py`) bounds the traversal length. Raise it if your estate has longer chains than the sample; lower it to cap enumeration on a denser graph. ### Extend the model diff --git a/it-dependency-mapping/model/__init__.py b/it-dependency-mapping/model/__init__.py new file mode 100644 index 00000000..08adf6ab --- /dev/null +++ b/it-dependency-mapping/model/__init__.py @@ -0,0 +1,103 @@ +"""IT dependency mapping (graph paths) template. + +Maps the downstream dependency structure of a software / data-pipeline estate +by enumerating variable-length traversal paths over an acyclic dependency DAG: + +- Loads Feature nodes (services, pipelines, jobs, dashboards) and a + contributes_to self-relationship (feeds / contributes to) from CSV. +- Enumerates every downstream dependency path with + ``model.path(Feature.contributes_to.repeat(1, N)).all().paths()`` -- because the + estate is acyclic, every enumerated path is simple. +- Reduces the full path set to its maximal chains (longest non-extendable + dependency chains, with shorter sub-chains dropped). +- Persists each feature's longest downstream depth back to the ontology as + Feature.max_downstream_depth, then reports per-feature path counts, the + deepest dependency chains, and the owners that sit along the longest chain. + +Run: + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh; `python -m + queries.dependency_chain_report` then prints the report from the deployed model. +""" + +from pathlib import Path + +import pandas as pd +from relationalai.semantics import Integer, Model, String +from relationalai.semantics.std import aggregates as aggs + +# -------------------------------------------------- +# Configure inputs +# -------------------------------------------------- + +DATA_DIR = Path(__file__).resolve().parents[1] / "data" + +# Maximum traversal depth (in edges) when enumerating downstream paths. The +# estate is acyclic, so this only needs to cover the longest real chain. +MAX_DEPTH = 8 + + +def load_csv(path): + return pd.read_csv(path) + + +# -------------------------------------------------- +# Define semantic model & load data +# -------------------------------------------------- + +model = Model("it_dependency_mapping") + +# Feature concept: a service, data pipeline, feature job, or dashboard. +Feature = model.Concept("Feature", identify_by={"id": String}) +Feature.name = model.Property(f"{Feature} has {String:name}") +Feature.owner = model.Property(f"{Feature} owned by {String:owner}") +Feature.deploy_tier = model.Property(f"{Feature} has deploy tier {String:deploy_tier}") + +# Feature.contributes_to: a self-relationship -- the upstream feature feeds / +# contributes to the downstream feature, forming an acyclic dependency DAG. +Feature.contributes_to = model.Relationship( + f"{Feature} contributes to {Feature}", short_name="contributes_to" +) + +# Load feature data from CSV. +feature_data = model.data(load_csv(DATA_DIR / "features.csv")) +model.define(Feature.new(id=feature_data["id"])) +model.where(Feature.id == feature_data["id"]).define( + Feature.name(feature_data["name"]), + Feature.owner(feature_data["owner"]), + Feature.deploy_tier(feature_data["deploy_tier"]), +) + +# Load dependency edges from CSV (upstream feature -> downstream feature). +dependency_data = model.data(load_csv(DATA_DIR / "dependencies.csv")) +upstream, downstream = Feature.ref(), Feature.ref() +model.where( + upstream.id == dependency_data["from_feature"], + downstream.id == dependency_data["to_feature"], +).define(upstream.contributes_to(downstream)) + +# -------------------------------------------------- +# Paths: enumerate downstream dependency paths +# PREVIEW capability; requires relationalai>=1.15. +# -------------------------------------------------- +# model.path(src, Feature.contributes_to.repeat(1, MAX_DEPTH)) describes a +# variable-length traversal of 1..MAX_DEPTH contributes_to edges starting from a +# typed source endpoint `src`. .all().paths() enumerates every such path; because +# contributes_to is acyclic, each path is simple. The result is one PathTraversal +# per path -- p.length is its hop count, p.nodes is the ordered sequence of +# Feature nodes it visits, and src is auto-unified with p.nodes(0) (the start). + +src = Feature.ref() +p = model.path(src, Feature.contributes_to.repeat(1, MAX_DEPTH)).all().paths() + +# Persist each feature's longest downstream depth back to the ontology so a +# downstream query can rank features by reach without re-enumerating paths. +# aggregates.max(p.length).per(src) takes the max hop count over the paths rooted +# at each src and defines it directly onto Feature -- no pandas groupby and no +# model.data round-trip. Kept SEPARATE from the node projection. It is a +# Relationship, not a Property: deploying this rule as a Property fails planning +# on relationalai 2.1.1. +Feature.max_downstream_depth = model.Relationship( + f"{Feature} has {Integer:max_downstream_depth}" +) +model.define(src.max_downstream_depth(aggs.max(p.length).per(src))).where(p) diff --git a/it-dependency-mapping/it_dependency_mapping.py b/it-dependency-mapping/queries/dependency_chain_report.py similarity index 56% rename from it-dependency-mapping/it_dependency_mapping.py rename to it-dependency-mapping/queries/dependency_chain_report.py index 08d0646a..c820d447 100644 --- a/it-dependency-mapping/it_dependency_mapping.py +++ b/it-dependency-mapping/queries/dependency_chain_report.py @@ -1,120 +1,17 @@ -"""IT dependency mapping (graph paths) template. +"""Downstream dependency report from the deployed IT estate model. -Maps the downstream dependency structure of a software / data-pipeline estate -by enumerating variable-length traversal paths over an acyclic dependency DAG: +Run after `rai models deploy --path model`: + python -m queries.dependency_chain_report -- Loads Feature nodes (services, pipelines, jobs, dashboards) and a - contributes_to self-relationship (feeds / contributes to) from CSV. -- Enumerates every downstream dependency path with - ``model.path(Feature.contributes_to.repeat(1, N)).all().paths()`` -- because the - estate is acyclic, every enumerated path is simple. -- Reduces the full path set to its maximal chains (longest non-extendable - dependency chains, with shorter sub-chains dropped). -- Persists each feature's longest downstream depth back to the ontology as - Feature.max_downstream_depth, then reports per-feature path counts, the - deepest dependency chains, and the owners that sit along the longest chain. - -Run: - `python it_dependency_mapping.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh, and prints the report from the - deployed model. - -Output: - Prints the total downstream path count, per-feature path counts and longest - downstream depth, the deepest maximal dependency chains, and the chain of - owners along the single longest chain (the upgrade / incident blast radius). +Prints the total downstream path count, per-feature path counts and longest +downstream depth, the deepest maximal dependency chains, and the chain of owners +along the single longest chain (the upgrade / incident blast radius). """ -from pathlib import Path - -import pandas as pd -from relationalai.semantics import Integer, Model, String +from model import MAX_DEPTH, Feature, model, p, src from relationalai.semantics.std import aggregates as aggs -# -------------------------------------------------- -# Configure inputs -# -------------------------------------------------- - -DATA_DIR = Path(__file__).parent / "data" - -# Maximum traversal depth (in edges) when enumerating downstream paths. The -# estate is acyclic, so this only needs to cover the longest real chain. -MAX_DEPTH = 8 - - -def load_csv(path): - return pd.read_csv(path) - - -# -------------------------------------------------- -# Define semantic model & load data -# -------------------------------------------------- - -model = Model("it_dependency_mapping") - -# Feature concept: a service, data pipeline, feature job, or dashboard. -Feature = model.Concept("Feature", identify_by={"id": String}) -Feature.name = model.Property(f"{Feature} has {String:name}") -Feature.owner = model.Property(f"{Feature} owned by {String:owner}") -Feature.deploy_tier = model.Property(f"{Feature} has deploy tier {String:deploy_tier}") - -# Feature.contributes_to: a self-relationship -- the upstream feature feeds / -# contributes to the downstream feature, forming an acyclic dependency DAG. -Feature.contributes_to = model.Relationship( - f"{Feature} contributes to {Feature}", short_name="contributes_to" -) - -# Load feature data from CSV. -feature_data = model.data(load_csv(DATA_DIR / "features.csv")) -model.define(Feature.new(id=feature_data["id"])) -model.where(Feature.id == feature_data["id"]).define( - Feature.name(feature_data["name"]), - Feature.owner(feature_data["owner"]), - Feature.deploy_tier(feature_data["deploy_tier"]), -) - -# Load dependency edges from CSV (upstream feature -> downstream feature). -dependency_data = model.data(load_csv(DATA_DIR / "dependencies.csv")) -upstream, downstream = Feature.ref(), Feature.ref() -model.where( - upstream.id == dependency_data["from_feature"], - downstream.id == dependency_data["to_feature"], -).define(upstream.contributes_to(downstream)) - -# -------------------------------------------------- -# Paths: enumerate downstream dependency paths -# PREVIEW capability; requires relationalai>=1.15. -# -------------------------------------------------- -# model.path(src, Feature.contributes_to.repeat(1, MAX_DEPTH)) describes a -# variable-length traversal of 1..MAX_DEPTH contributes_to edges starting from a -# typed source endpoint `src`. .all().paths() enumerates every such path; because -# contributes_to is acyclic, each path is simple. The result is one PathTraversal -# per path -- p.length is its hop count, p.nodes is the ordered sequence of -# Feature nodes it visits, and src is auto-unified with p.nodes(0) (the start). - -src = Feature.ref() -p = model.path(src, Feature.contributes_to.repeat(1, MAX_DEPTH)).all().paths() - -# Persist each feature's longest downstream depth back to the ontology so a -# downstream query can rank features by reach without re-enumerating paths. -# aggregates.max(p.length).per(src) takes the max hop count over the paths rooted -# at each src and defines it directly onto Feature -- no pandas groupby and no -# model.data round-trip. Kept SEPARATE from the node projection. It is a -# Relationship, not a Property: on relationalai 2.1.1, deploying a Property -# written from a path rule fails planning. -Feature.max_downstream_depth = model.Relationship( - f"{Feature} has {Integer:max_downstream_depth}" -) -model.define(src.max_downstream_depth(aggs.max(p.length).per(src))).where(p) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - if __name__ == "__main__": - # Deploy and wait for the first refresh. - model.deploy() - print("=== IT Dependency Mapping: downstream paths ===") # Project the node sequence of each path (for display-side reassembly). Keep this From 959d6691ad9c926c8b4ff90d2c934e6705b0fd68 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:31:13 -0800 Subject: [PATCH 066/162] shipment_compliance: deploy the model package and query it Move the concepts, enums, and four rules into model/ and the report into queries/compliance_report.py, run with rai models deploy --path model and python -m queries.compliance_report. Rewrite the Quickstart to the deployed-template steps. --- shipment_compliance/README.md | 78 +++++++++++++------ .../__init__.py} | 53 ++----------- .../queries/compliance_report.py | 43 ++++++++++ shipment_compliance/runbook.md | 2 +- 4 files changed, 104 insertions(+), 72 deletions(-) rename shipment_compliance/{shipment_compliance.py => model/__init__.py} (79%) create mode 100644 shipment_compliance/queries/compliance_report.py diff --git a/shipment_compliance/README.md b/shipment_compliance/README.md index 6203abee..8fffa722 100644 --- a/shipment_compliance/README.md +++ b/shipment_compliance/README.md @@ -42,7 +42,8 @@ The four rules demonstrate different rule patterns: ## What's included -- `shipment_compliance.py` -- Main script defining the data model and four rules +- **Model**: `model/__init__.py` -- the data model and four rules that `rai models deploy --path model` deploys to your Snowflake account. +- **Query**: `queries/compliance_report.py` -- prints the entities each rule flags from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - `data/suppliers.csv` -- Supplier names and reliability scores - `data/skus.csv` -- Product and component catalog @@ -65,28 +66,43 @@ The four rules demonstrate different rule patterns: ## Quickstart -1. Download ZIP: +Before you start, install Python 3.10 or later and get a Snowflake role that can deploy models, as described under Access. This template has no prescriptive solve, so it needs no setup beyond that role. + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/shipment_compliance.zip). Unzip it and enter the template directory: + ```bash - curl -O https://docs.relational.ai/templates/zips/v1/shipment_compliance.zip unzip shipment_compliance.zip cd shipment_compliance ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. -2. Create venv: +2. **Create a Python environment** + + Create and activate a virtual environment, then update pip: + ```bash python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` -3. Install: +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: + ```bash python -m pip install . ``` -4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist: + ```yaml deployment: role: @@ -99,23 +115,25 @@ The four rules demonstrate different rule patterns: schedule: standard ``` - Without the `deployment` section, `python shipment_compliance.py` stops with an error naming the missing setting. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: -5. Run: ```bash - python shipment_compliance.py + rai models deploy --path model ``` - The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. - Running the script again without changing the model redeploys nothing; it prints the report from the deployed model. A change to the model redeploys it. + The deploy waits for the model's first refresh, which evaluates the four rules. Without the `deployment` section, it stops with an error naming the missing setting. - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + Deploying again without changing the model applies no updates. A change to the model redeploys it. -6. Expected output (a few lines confirm a successful run): +6. **Query the results** - The script prints the entities each rule flags. On the bundled data the four - rules catch **2 late shipments**, **6 at-risk shipments**, **2 single-sourced - BOM inputs**, and **4 escalated demands**: + Print the report from the deployed model: + + ```bash + python -m queries.compliance_report + ``` ```text === Rule 1: Late Shipments (delay_days > 0) === @@ -131,7 +149,18 @@ The four rules demonstrate different rule patterns: (4 rows) ``` - See `runbook.md` for the full record-by-record walkthrough of each flag. + The report prints the entities each rule flags, under that rule's heading. On the bundled data the four rules catch 2 late shipments, 6 at-risk shipments, 2 single-sourced BOM inputs, and 4 escalated demands. See `runbook.md` for the full record-by-record walkthrough of each flag. + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The teardown removes the deployed objects and keeps the two schemas. ## Template structure @@ -139,7 +168,10 @@ The four rules demonstrate different rule patterns: . ├── README.md # this file ├── pyproject.toml # dependencies -├── shipment_compliance.py # main script: data model + four derived rules +├── model/ +│ └── __init__.py # data model + four derived rules +├── queries/ +│ └── compliance_report.py # report printed from the deployed model ├── runbook.md # analyst-facing paste-testable walkthrough └── data/ ├── suppliers.csv # 5 suppliers (id, name, reliability_score) @@ -150,7 +182,7 @@ The four rules demonstrate different rule patterns: └── demands.csv # 8 demand orders (SKU, quantity, priority) ``` -**Start here**: run `python shipment_compliance.py` for the full run end to end — authoring all four rules and printing the flagged records — or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here**: deploy with `rai models deploy --path model` and run `python -m queries.compliance_report` for the full run end to end — the four rules and the flagged records — or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -173,7 +205,7 @@ Six concepts model a small supply chain, linked by SKU and supplier references. - **Primary identifiers**: an integer `id` on every concept. - **Important invariants**: `Supplier.reliability_score` is a fraction in `[0, 1]`; `Shipment.status` is a `ShipmentStatus` member (`PENDING` / `IN_TRANSIT` / `DELIVERED`); `Demand.priority` is a `Priority` member (`LOW` / `STANDARD` / `HIGH` / `URGENT`); `delay_days`, `quantity`, and `input_quantity` are non-negative. Each rule adds a derived boolean flag back onto its concept: `Shipment.is_late` (Rule 1), `Shipment.is_at_risk` (Rule 2), `BillOfMaterials.is_single_sourced` (Rule 3), and `Demand.is_escalated` (Rule 4). -For the full concept and property definitions, see `shipment_compliance.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -188,7 +220,7 @@ CSV inputs → load Supplier / SKU / Shipment / Operation / BOM / Demand 2. **Define four rule patterns.** Each rule uses the `model.where(...).define(...)` pattern to set a boolean flag, and together they show four shapes: a simple threshold (Rule 1: `delay_days > 0`), a cross-entity join (Rule 2: undelivered shipments from suppliers scoring below `0.8`), an aggregation (Rule 3: BOM inputs produced by exactly one operation route), and OR semantics via multiple `define()` calls on the same relationship (Rule 4: `HIGH` or `URGENT` priority). 3. **Query the flags.** Each rule is queried with `model.select(...).where(Concept.rule_flag())` to display the matching entities. -See `shipment_compliance.py` for the implementation and `runbook.md` for the skill-driven reproduction. +See `model/__init__.py` for the implementation, `queries/compliance_report.py` for the report, and `runbook.md` for the skill-driven reproduction. ## Customize this template diff --git a/shipment_compliance/shipment_compliance.py b/shipment_compliance/model/__init__.py similarity index 79% rename from shipment_compliance/shipment_compliance.py rename to shipment_compliance/model/__init__.py index 9e84720f..5d09d0d3 100644 --- a/shipment_compliance/shipment_compliance.py +++ b/shipment_compliance/model/__init__.py @@ -19,13 +19,10 @@ No optimization solver is used. Rules are pure logic derivations. - Run: - `python shipment_compliance.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh, and prints the report from - the deployed model. - - Output: - Prints which entities match each compliance rule. +Run: + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh; `python -m + queries.compliance_report` then prints the report from the deployed model. """ from pathlib import Path @@ -41,7 +38,7 @@ # Define semantic model & load data # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # Supplier concept: companies that supply parts. Supplier = Concept("Supplier", identify_by={"id": Integer}) @@ -194,43 +191,3 @@ class Priority(model.Enum): Demand.is_escalated = Relationship(f"{Demand} is escalated") model.where(Demand.priority == Priority.HIGH).define(Demand.is_escalated()) model.where(Demand.priority == Priority.URGENT).define(Demand.is_escalated()) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh. - model.deploy() - - print("=== Rule 1: Late Shipments (delay_days > 0) ===\n") - model.select( - Shipment.id.alias("shipment_id"), - Shipment.sku.name.alias("sku"), - Shipment.supplier.name.alias("supplier"), - Shipment.delay_days.alias("delay_days"), - ).where(Shipment.is_late()).inspect() - - print("\n=== Rule 2: At-Risk Shipments (undelivered + unreliable supplier) ===\n") - model.select( - Shipment.id.alias("shipment_id"), - Shipment.sku.name.alias("sku"), - Shipment.supplier.name.alias("supplier"), - Shipment.status.name.alias("status"), - Shipment.supplier.reliability_score.alias("reliability"), - ).where(Shipment.is_at_risk()).inspect() - - print("\n=== Rule 3: Single-Sourced BOM Inputs (only 1 operation route) ===\n") - model.select( - BOM.id.alias("bom_id"), - BOM.input_sku.name.alias("input_sku"), - BOM.input_quantity.alias("qty"), - ).where(BOM.is_single_sourced()).inspect() - - print("\n=== Rule 4: Escalated Demands (HIGH or URGENT priority) ===\n") - model.select( - Demand.id.alias("demand_id"), - Demand.sku.name.alias("sku"), - Demand.quantity.alias("quantity"), - Demand.priority.name.alias("priority"), - ).where(Demand.is_escalated()).inspect() diff --git a/shipment_compliance/queries/compliance_report.py b/shipment_compliance/queries/compliance_report.py new file mode 100644 index 00000000..a554ea3e --- /dev/null +++ b/shipment_compliance/queries/compliance_report.py @@ -0,0 +1,43 @@ +"""Compliance report: reads the deployed shipment-compliance model. + +Run after `rai models deploy --path model`: + python -m queries.compliance_report + +Prints which entities match each compliance rule: late shipments, at-risk +shipments, single-sourced BOM inputs, and escalated demands. +""" + +from model import BOM, Demand, Shipment, model + +if __name__ == "__main__": + print("=== Rule 1: Late Shipments (delay_days > 0) ===\n") + model.select( + Shipment.id.alias("shipment_id"), + Shipment.sku.name.alias("sku"), + Shipment.supplier.name.alias("supplier"), + Shipment.delay_days.alias("delay_days"), + ).where(Shipment.is_late()).inspect() + + print("\n=== Rule 2: At-Risk Shipments (undelivered + unreliable supplier) ===\n") + model.select( + Shipment.id.alias("shipment_id"), + Shipment.sku.name.alias("sku"), + Shipment.supplier.name.alias("supplier"), + Shipment.status.name.alias("status"), + Shipment.supplier.reliability_score.alias("reliability"), + ).where(Shipment.is_at_risk()).inspect() + + print("\n=== Rule 3: Single-Sourced BOM Inputs (only 1 operation route) ===\n") + model.select( + BOM.id.alias("bom_id"), + BOM.input_sku.name.alias("input_sku"), + BOM.input_quantity.alias("qty"), + ).where(BOM.is_single_sourced()).inspect() + + print("\n=== Rule 4: Escalated Demands (HIGH or URGENT priority) ===\n") + model.select( + Demand.id.alias("demand_id"), + Demand.sku.name.alias("sku"), + Demand.quantity.alias("quantity"), + Demand.priority.name.alias("priority"), + ).where(Demand.is_escalated()).inspect() diff --git a/shipment_compliance/runbook.md b/shipment_compliance/runbook.md index c58b543c..34124f79 100644 --- a/shipment_compliance/runbook.md +++ b/shipment_compliance/runbook.md @@ -38,4 +38,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 5 suppliers, 6 SKUs, 12 shipments, 7 operations, 6 BOM lines, 8 demand orders. Full model in `shipment_compliance.py`. +Bundled CSVs in `data/`: 5 suppliers, 6 SKUs, 12 shipments, 7 operations, 6 BOM lines, 8 demand orders. Full model in `model/__init__.py`. From 75eb6f7beaf0312025129ffead52e3dbd9e6d433 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:31:38 -0800 Subject: [PATCH 067/162] shipment_compliance: say model, not script, in the module docstring --- shipment_compliance/model/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/shipment_compliance/model/__init__.py b/shipment_compliance/model/__init__.py index 5d09d0d3..2a194d6f 100644 --- a/shipment_compliance/model/__init__.py +++ b/shipment_compliance/model/__init__.py @@ -1,6 +1,6 @@ """Shipment Compliance (logic reasoning) template. -This script demonstrates derived business rules in RelationalAI: +This model declares derived business rules in RelationalAI: - Load sample CSVs describing suppliers, SKUs, shipments, operations, BOMs, and demand. - Define four rules as derived Relationships (boolean flags) on existing concepts. From abbdfc8da1931e5e82d8d3641bb3e5c39af55775 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:37:16 -0800 Subject: [PATCH 068/162] financial_index_replication: deploy the model package and query it Move the decision problem, solver settings, and status relationships into model/reasoning.py and the report into queries/replication_report.py, run with rai models deploy --path model and python -m queries.replication_report. The report now stops on any termination status other than OPTIMAL or TIME_LIMIT, and the universe check counts distinct tickers. The sidecar sets page.deployment, and the walkthrough's solve step splits into the solver settings (model/reasoning.py) and the tracking check (queries/replication_report.py). The README Quickstart follows the deployed-template steps. --- financial_index_replication/README.md | 99 ++--- .../financial_index_replication.py | 354 ------------------ financial_index_replication/model/__init__.py | 30 +- .../model/reasoning.py | 181 +++++++++ .../queries/replication_report.py | 177 +++++++++ financial_index_replication/runbook.md | 4 +- .../template-docs.yaml | 65 ++-- 7 files changed, 485 insertions(+), 425 deletions(-) delete mode 100644 financial_index_replication/financial_index_replication.py create mode 100644 financial_index_replication/model/reasoning.py create mode 100644 financial_index_replication/queries/replication_report.py diff --git a/financial_index_replication/README.md b/financial_index_replication/README.md index faf012b2..2aeaefee 100644 --- a/financial_index_replication/README.md +++ b/financial_index_replication/README.md @@ -31,24 +31,35 @@ with a simple alternative. Adapt the data and limits for another index strategy, but treat the example as a starting point rather than a production portfolio. +## Prerequisites + +- A Snowflake account that has the RAI Native App installed. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- A Snowflake user that holds the model role. + ## Quickstart -Before you start, install Python 3.10 or later and get access to a Snowflake -account with the RAI Native App. Prescriptive reasoning is in Public Preview; -ask your RelationalAI support representative to enable it. Preview features are -for evaluation and testing, not production applications. The template pins -`relationalai==2.1.1` in `pyproject.toml`. +Before you start, install Python 3.10 or later and complete the access prerequisites above. Prescriptive reasoning is in Public Preview; ask your RelationalAI support representative to enable it. Preview features are for evaluation and testing, not production applications. The template pins `relationalai==2.1.1` in `pyproject.toml`. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: -The script deploys the model to your Snowflake account, so an account -administrator must first complete the one-time model deployment setup there: -follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) -through **Governed setup**, **PAT authentication**, and steps 1-2 of -**Additional setup for the prescriptive reasoner** (step 4 below covers this -template's own schema). It gives you a model role, a programmatic access token -(PAT) stored as a secret, an external access integration, a PAT user, and a -warehouse. Your Snowflake user must hold the model role. +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .FINANCIAL_INDEX_REPLICATION_META; -Use this sequence to run the bundled example: +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: 1. **Download the template** @@ -79,12 +90,14 @@ Use this sequence to run the bundled example: 4. **Configure your project** - Use the configuration builder in [Start building with PyRel](https://docs.relational.ai/get-started/start-building-with-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. Set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: ```yaml reasoners: prescriptive: - name: financial_index_replication_solver # created on first deploy if it doesn't exist + name: financial_index_replication_solver # created on first deploy if it doesn't exist deployment: role: public_schema: .FINANCIAL_INDEX_REPLICATION # published views and tables @@ -100,50 +113,48 @@ Use this sequence to run the bundled example: pat_warehouse: ``` - Without the `deployment` section, `python financial_index_replication.py` stops with an error naming the missing setting. +5. **Deploy the model** - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .FINANCIAL_INDEX_REPLICATION_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .FINANCIAL_INDEX_REPLICATION_META TO APPLICATION ; + ```bash + rai models deploy --path model ``` - The deploy creates the public schema `.FINANCIAL_INDEX_REPLICATION` if it doesn't exist. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. -5. **Run the template** + Deploying again without changing the model applies no updates and does not solve again. A change to the model or to `problem.configure(...)` redeploys it. - Run the bundled script from the template directory: +6. **Query the results** + + Print the report from the deployed model: ```bash - python financial_index_replication.py + python -m queries.replication_report ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. - ```text Selected names: exactly 20 - Status: OPTIMAL + Status: TIME_LIMIT Wrote benchmark-vs-replica returns to: data/replica_returns.csv ``` - The solve selects exactly 20 names, though a selected stock may have zero - weight. It writes `data/replica_returns.csv` for plotting. See `runbook.md` - for the basket, sector exposures, tracking quality, and baseline comparison. + The solve selects exactly 20 names, though a selected stock may have zero weight. The solve ends as `OPTIMAL` or at its 240 second time limit (`TIME_LIMIT`) with the best basket found so far. The report accepts both, stops on any other termination status, and writes `data/replica_returns.csv` for plotting. See `runbook.md` for the basket, sector exposures, tracking quality, and baseline comparison. + + To solve again without changing the model, refresh the deployed model, then run the query again: - Running the script again without changing the model redeploys nothing and does not solve again; it reports the latest refresh. A change to the model or to `problem.configure(...)` redeploys, and the redeploy's refresh solves. To solve again without a change, refresh the deployed model, then run the script: ```sql CALL .FINANCIAL_INDEX_REPLICATION_META._INTERNAL_REFRESH_MODEL('standard', NULL); ``` - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. + diff --git a/financial_index_replication/financial_index_replication.py b/financial_index_replication/financial_index_replication.py deleted file mode 100644 index 67534cbf..00000000 --- a/financial_index_replication/financial_index_replication.py +++ /dev/null @@ -1,354 +0,0 @@ -"""Financial index replication (prescriptive optimization) template. - -This script demonstrates sparse index replication in RelationalAI: - -- Load sample monthly returns for an S&P 500-like benchmark and 50 stocks. -- Choose exactly 20 names and their portfolio weights. -- Minimize full-history tracking residuals versus the benchmark. -- Enforce long-only weights, max position size, sector neutrality, and - per-name ADV participation limits. -- Report full-history tracking error and compare against a simple correlation - baseline. - -The optimization is modeled as a mixed-integer linear problem that minimizes -absolute tracking residuals. - -Run: - `python financial_index_replication.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh (which runs the solve), and prints - the report from the deployed model. - -Output: - Prints the selected 20-stock basket, sector exposures, tracking quality - metrics, and a top-correlation baseline comparison. Writes - data/replica_returns.csv with date, index_return, and replica_return - columns for downstream plotting. -""" - -from math import sqrt - -import pandas as pd -from model import ( - DATA_DIR, - ReturnMonth, - Sector, - Stock, - index_returns_csv, - model, - stock_returns_csv, - stocks_csv, -) -from relationalai.semantics import Float, String, sum -from relationalai.semantics.reasoners.prescriptive import Problem - -# -------------------------------------------------- -# Configure replication policy -# -------------------------------------------------- - -N_REPLICATION_NAMES = 20 -MAX_WEIGHT = 0.10 -SECTOR_ACTIVE_BAND = 0.04 -PORTFOLIO_VALUE = 10_000_000 -MAX_ADV_PARTICIPATION = 0.05 -MONTHS_PER_YEAR = 12 - -# Basic feasibility checks before building and solving the optimization model. -eligible_stock_count = len(stocks_csv) -if eligible_stock_count < N_REPLICATION_NAMES: - raise ValueError( - f"N_REPLICATION_NAMES={N_REPLICATION_NAMES} requires at least that many " - f"eligible stocks, but only {eligible_stock_count} are available." - ) -if N_REPLICATION_NAMES * MAX_WEIGHT < 1.0: - raise ValueError( - f"N_REPLICATION_NAMES * MAX_WEIGHT must be at least 1.0 for full " - f"investment, got {N_REPLICATION_NAMES * MAX_WEIGHT:.2f}." - ) - -# -------------------------------------------------- -# Model the decision problem -# -------------------------------------------------- - -Stock.x_selected = model.Property(f"{Stock} selected if {Float:selected}") -Stock.x_weight = model.Property(f"{Stock} has replication weight {Float:weight}") -ReturnMonth.x_pos_error = model.Property( - f"{ReturnMonth} has positive tracking residual {Float:pos_error}" -) -ReturnMonth.x_neg_error = model.Property( - f"{ReturnMonth} has negative tracking residual {Float:neg_error}" -) - -selected = Float.ref("selected") -weight = Float.ref("weight") -pos_error = Float.ref("pos_error") -neg_error = Float.ref("neg_error") -index_return = Float.ref("index_return") -stock_return = Float.ref("stock_return") - -problem = Problem(model, Float) - -problem.solve_for( - Stock.x_selected(selected), - type="bin", - name=["selected", Stock.ticker], -) -problem.solve_for( - Stock.x_weight(weight), - type="cont", - lower=0, - upper=MAX_WEIGHT, - name=["weight", Stock.ticker], -) -problem.solve_for( - ReturnMonth.x_pos_error(pos_error), - type="cont", - lower=0, - name=["pos_error", ReturnMonth.date], -) -problem.solve_for( - ReturnMonth.x_neg_error(neg_error), - type="cont", - lower=0, - name=["neg_error", ReturnMonth.date], -) - -# Select exactly N names and invest all capital. -problem.satisfy( - model.where(Stock.x_selected(selected)).require( - sum(Stock, selected) == N_REPLICATION_NAMES - ) -) -# Full-investment constraint: portfolio weights must sum to 100%. -problem.satisfy(model.where(Stock.x_weight(weight)).require(sum(Stock, weight) == 1.0)) - -# A stock can carry weight only when selected. -problem.satisfy( - model.where( - Stock.x_weight(weight), - Stock.x_selected(selected), - ).require(weight <= MAX_WEIGHT * selected) -) - -# Keep sector exposures close to benchmark sector weights. -problem.satisfy( - model.where( - Stock.x_weight(weight), - Stock.sector_ref(Sector), - ).require( - sum(Stock, weight).per(Sector) <= Sector.benchmark_weight + SECTOR_ACTIVE_BAND - ) -) -problem.satisfy( - model.where( - Stock.x_weight(weight), - Stock.sector_ref(Sector), - ).require( - sum(Stock, weight).per(Sector) >= Sector.benchmark_weight - SECTOR_ACTIVE_BAND - ) -) - -# Limit per-name buy and sell dollars as a fraction of average dollar volume. -problem.satisfy( - model.where( - Stock.x_weight(weight), - ).require( - PORTFOLIO_VALUE * (weight - Stock.previous_weight) - <= MAX_ADV_PARTICIPATION * Stock.avg_dollar_volume - ) -) -problem.satisfy( - model.where( - Stock.x_weight(weight), - ).require( - PORTFOLIO_VALUE * (Stock.previous_weight - weight) - <= MAX_ADV_PARTICIPATION * Stock.avg_dollar_volume - ) -) - -# Monthly benchmark residual: -# index_return[t] - sum_i weight[i] * stock_return[i,t] = pos_error[t] - neg_error[t] -problem.satisfy( - model.where( - ReturnMonth.index_return(index_return), - ReturnMonth.x_pos_error(pos_error), - ReturnMonth.x_neg_error(neg_error), - Stock.x_weight(weight), - Stock.monthly_return(ReturnMonth, stock_return), - ).require( - index_return - sum(Stock, stock_return * weight).per(ReturnMonth) - == pos_error - neg_error - ) -) - -# Minimize total absolute tracking residual. -problem.minimize( - sum(ReturnMonth, pos_error + neg_error).where( - ReturnMonth.x_pos_error(pos_error), - ReturnMonth.x_neg_error(neg_error), - ) -) - -# -------------------------------------------------- -# Solve -# -------------------------------------------------- - -# Each refresh of the deployed model runs this solve. -problem.configure("highs", time_limit_sec=240) - -# The solve's status and objective, published through the model's own relationships. -SolveStatus = model.Relationship(f"financial index replication solve status is {String:status}") -TotalResidual = model.Relationship( - f"financial index replication total absolute residual is {Float:residual}" -) -model.define(SolveStatus(problem.termination_status()), TotalResidual(problem.objective_value())) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - print("=" * 70) - print("FINANCIAL INDEX REPLICATION") - print("=" * 70) - print(f"Universe: {len(stocks_csv)} stocks") - print(f"Selected names: exactly {N_REPLICATION_NAMES}") - print(f"Max position: {MAX_WEIGHT:.0%}") - print(f"Sector active band: +/- {SECTOR_ACTIVE_BAND:.0%}") - print(f"Portfolio value: ${PORTFOLIO_VALUE:,.0f}") - print(f"Max ADV participation per name: {MAX_ADV_PARTICIPATION:.0%}") - - # Status and objective of the latest refresh's solve. A refresh records any - # termination status, so the report checks it. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - residual = model.select(TotalResidual).to_df().iloc[:, 0].tolist() - assert status in (["OPTIMAL"], ["TIME_LIMIT"]), f"Solver did not find an optimal solution: {status}" - - print(f"\nStatus: {status[0]}") - print(f"Objective: total absolute residual = {residual[0]:.6f}") - - # -------------------------------------------------- - # Evaluate and report - # -------------------------------------------------- - - solution_df = ( - model.select( - Stock.ticker.alias("ticker"), - Stock.sector.alias("sector"), - Stock.benchmark_weight.alias("benchmark_weight"), - Stock.previous_weight.alias("previous_weight"), - Stock.avg_dollar_volume.alias("avg_dollar_volume"), - selected.alias("selected"), - weight.alias("weight"), - ) - .where( - Stock.x_selected(selected), - Stock.x_weight(weight), - weight > 0.0001, - ) - .to_df() - .sort_values("weight", ascending=False) - ) - - returns_wide = stock_returns_csv.pivot( - index="date", columns="ticker", values="return" - ).sort_index() - index_series = index_returns_csv.set_index("date").sort_index() - weights = solution_df.set_index("ticker")["weight"] - portfolio_returns = returns_wide[weights.index].mul(weights, axis=1).sum(axis=1) - tracking_diff = portfolio_returns - index_series["index_return"] - - def annualized_tracking_error(): - monthly_rms = sqrt((tracking_diff ** 2).mean()) - return monthly_rms * sqrt(MONTHS_PER_YEAR) - - def mean_abs_tracking(): - return tracking_diff.abs().mean() - - print("\n=== Selected Replication Basket ===") - print( - solution_df[ - [ - "ticker", - "sector", - "weight", - "benchmark_weight", - "previous_weight", - "avg_dollar_volume", - ] - ].to_string( - index=False, - formatters={ - "weight": "{:.2%}".format, - "benchmark_weight": "{:.2%}".format, - "previous_weight": "{:.2%}".format, - "avg_dollar_volume": "${:,.0f}".format, - }, - ) - ) - - sector_report = ( - solution_df.groupby("sector", as_index=False)["weight"].sum() - .merge( - stocks_csv.groupby("sector", as_index=False)["benchmark_weight"].sum(), - on="sector", - ) - ) - sector_report["active_weight"] = ( - sector_report["weight"] - sector_report["benchmark_weight"] - ) - - print("\n=== Sector Exposure ===") - print( - sector_report.sort_values("sector").to_string( - index=False, - formatters={ - "weight": "{:.2%}".format, - "benchmark_weight": "{:.2%}".format, - "active_weight": "{:+.2%}".format, - }, - ) - ) - - turnover = ( - solution_df.set_index("ticker")["weight"] - .reindex(stocks_csv["ticker"], fill_value=0.0) - .sub(stocks_csv.set_index("ticker")["previous_weight"]) - .abs() - .sum() - ) - - print("\n=== Tracking Quality ===") - print(f"Annualized tracking error: {annualized_tracking_error():.2%}") - print(f"Mean abs monthly residual: {mean_abs_tracking():.4%}") - print(f"Implied turnover: {turnover:.2%}") - - # Simple benchmark: equal-weight the 20 most correlated stocks. - top_corr_ids = ( - returns_wide - .corrwith(index_series["index_return"]) - .abs() - .sort_values(ascending=False) - .head(20) - .index - ) - baseline_returns = returns_wide[top_corr_ids].mean(axis=1) - baseline_diff = baseline_returns - index_series["index_return"] - - baseline_tracking_error = sqrt((baseline_diff ** 2).mean()) * sqrt( - MONTHS_PER_YEAR - ) - - replica_returns_csv = DATA_DIR / "replica_returns.csv" - pd.DataFrame({ - "date": index_series.index, - "index_return": index_series["index_return"].values, - "replica_return": portfolio_returns.reindex(index_series.index).values, - }).to_csv(replica_returns_csv, index=False) - - print("\n=== Baseline Comparison ===") - print("Baseline: equal-weight top-20 stocks by full-history correlation") - print(f"Baseline annualized tracking error: {baseline_tracking_error:.2%}") - print(f"\nWrote benchmark-vs-replica returns to: {replica_returns_csv}") diff --git a/financial_index_replication/model/__init__.py b/financial_index_replication/model/__init__.py index 21a65cea..1eba857b 100644 --- a/financial_index_replication/model/__init__.py +++ b/financial_index_replication/model/__init__.py @@ -1,15 +1,43 @@ -"""Financial index replication model and bundled sample data.""" +"""Financial index replication model: bundled sample data and the decision problem. + +`rai models deploy --path model` deploys everything this package declares. +""" from .schema import ReturnMonth, Sector, Stock, model from .source import DATA_DIR, index_returns_csv, stock_returns_csv, stocks_csv +# isort: split +# The decision problem declares variables and constraints over the facts loaded above. +from .reasoning import ( + MAX_ADV_PARTICIPATION, + MAX_WEIGHT, + MONTHS_PER_YEAR, + N_REPLICATION_NAMES, + PORTFOLIO_VALUE, + SECTOR_ACTIVE_BAND, + SolveStatus, + TotalResidual, + selected, + weight, +) + __all__ = [ "DATA_DIR", + "MAX_ADV_PARTICIPATION", + "MAX_WEIGHT", + "MONTHS_PER_YEAR", + "N_REPLICATION_NAMES", + "PORTFOLIO_VALUE", + "SECTOR_ACTIVE_BAND", "ReturnMonth", "Sector", + "SolveStatus", "Stock", + "TotalResidual", "index_returns_csv", "model", + "selected", "stock_returns_csv", "stocks_csv", + "weight", ] diff --git a/financial_index_replication/model/reasoning.py b/financial_index_replication/model/reasoning.py new file mode 100644 index 00000000..91e40160 --- /dev/null +++ b/financial_index_replication/model/reasoning.py @@ -0,0 +1,181 @@ +"""Sparse index replication: decision problem and solver settings. + +Declares, at module level, everything the deployment needs beyond the source +facts in `model/source.py`: + +- Choose exactly 20 names and their portfolio weights. +- Minimize full-history tracking residuals versus the benchmark. +- Enforce long-only weights, max position size, sector neutrality, and + per-name ADV participation limits. + +The optimization is a mixed-integer linear problem that minimizes absolute +tracking residuals. Each refresh of the deployed model runs the solve. +`queries/replication_report.py` prints the report from the deployed model. +""" + +from relationalai.semantics import Float, String, sum +from relationalai.semantics.reasoners.prescriptive import Problem + +from .schema import ReturnMonth, Sector, Stock, model +from .source import stocks_csv + +# -------------------------------------------------- +# Configure replication policy +# -------------------------------------------------- + +N_REPLICATION_NAMES = 20 +MAX_WEIGHT = 0.10 +SECTOR_ACTIVE_BAND = 0.04 +PORTFOLIO_VALUE = 10_000_000 +MAX_ADV_PARTICIPATION = 0.05 +MONTHS_PER_YEAR = 12 + +# Basic feasibility checks before building and solving the optimization model. +eligible_stock_count = stocks_csv["ticker"].nunique() +if eligible_stock_count < N_REPLICATION_NAMES: + raise ValueError( + f"N_REPLICATION_NAMES={N_REPLICATION_NAMES} requires at least that many " + f"eligible stocks, but only {eligible_stock_count} are available." + ) +if N_REPLICATION_NAMES * MAX_WEIGHT < 1.0: + raise ValueError( + f"N_REPLICATION_NAMES * MAX_WEIGHT must be at least 1.0 for full " + f"investment, got {N_REPLICATION_NAMES * MAX_WEIGHT:.2f}." + ) + +# -------------------------------------------------- +# Model the decision problem +# -------------------------------------------------- + +Stock.x_selected = model.Property(f"{Stock} selected if {Float:selected}") +Stock.x_weight = model.Property(f"{Stock} has replication weight {Float:weight}") +ReturnMonth.x_pos_error = model.Property( + f"{ReturnMonth} has positive tracking residual {Float:pos_error}" +) +ReturnMonth.x_neg_error = model.Property( + f"{ReturnMonth} has negative tracking residual {Float:neg_error}" +) + +selected = Float.ref("selected") +weight = Float.ref("weight") +pos_error = Float.ref("pos_error") +neg_error = Float.ref("neg_error") +index_return = Float.ref("index_return") +stock_return = Float.ref("stock_return") + +problem = Problem(model, Float) + +problem.solve_for( + Stock.x_selected(selected), + type="bin", + name=["selected", Stock.ticker], +) +problem.solve_for( + Stock.x_weight(weight), + type="cont", + lower=0, + upper=MAX_WEIGHT, + name=["weight", Stock.ticker], +) +problem.solve_for( + ReturnMonth.x_pos_error(pos_error), + type="cont", + lower=0, + name=["pos_error", ReturnMonth.date], +) +problem.solve_for( + ReturnMonth.x_neg_error(neg_error), + type="cont", + lower=0, + name=["neg_error", ReturnMonth.date], +) + +# Select exactly N names and invest all capital. +problem.satisfy( + model.where(Stock.x_selected(selected)).require( + sum(Stock, selected) == N_REPLICATION_NAMES + ) +) +# Full-investment constraint: portfolio weights must sum to 100%. +problem.satisfy(model.where(Stock.x_weight(weight)).require(sum(Stock, weight) == 1.0)) + +# A stock can carry weight only when selected. +problem.satisfy( + model.where( + Stock.x_weight(weight), + Stock.x_selected(selected), + ).require(weight <= MAX_WEIGHT * selected) +) + +# Keep sector exposures close to benchmark sector weights. +problem.satisfy( + model.where( + Stock.x_weight(weight), + Stock.sector_ref(Sector), + ).require( + sum(Stock, weight).per(Sector) <= Sector.benchmark_weight + SECTOR_ACTIVE_BAND + ) +) +problem.satisfy( + model.where( + Stock.x_weight(weight), + Stock.sector_ref(Sector), + ).require( + sum(Stock, weight).per(Sector) >= Sector.benchmark_weight - SECTOR_ACTIVE_BAND + ) +) + +# Limit per-name buy and sell dollars as a fraction of average dollar volume. +problem.satisfy( + model.where( + Stock.x_weight(weight), + ).require( + PORTFOLIO_VALUE * (weight - Stock.previous_weight) + <= MAX_ADV_PARTICIPATION * Stock.avg_dollar_volume + ) +) +problem.satisfy( + model.where( + Stock.x_weight(weight), + ).require( + PORTFOLIO_VALUE * (Stock.previous_weight - weight) + <= MAX_ADV_PARTICIPATION * Stock.avg_dollar_volume + ) +) + +# Monthly benchmark residual: +# index_return[t] - sum_i weight[i] * stock_return[i,t] = pos_error[t] - neg_error[t] +problem.satisfy( + model.where( + ReturnMonth.index_return(index_return), + ReturnMonth.x_pos_error(pos_error), + ReturnMonth.x_neg_error(neg_error), + Stock.x_weight(weight), + Stock.monthly_return(ReturnMonth, stock_return), + ).require( + index_return - sum(Stock, stock_return * weight).per(ReturnMonth) + == pos_error - neg_error + ) +) + +# Minimize total absolute tracking residual. +problem.minimize( + sum(ReturnMonth, pos_error + neg_error).where( + ReturnMonth.x_pos_error(pos_error), + ReturnMonth.x_neg_error(neg_error), + ) +) + +# -------------------------------------------------- +# Solve +# -------------------------------------------------- + +# Each refresh of the deployed model runs this solve. +problem.configure("highs", time_limit_sec=240) + +# The solve's status and objective, published through the model's own relationships. +SolveStatus = model.Relationship(f"financial index replication solve status is {String:status}") +TotalResidual = model.Relationship( + f"financial index replication total absolute residual is {Float:residual}" +) +model.define(SolveStatus(problem.termination_status()), TotalResidual(problem.objective_value())) diff --git a/financial_index_replication/queries/replication_report.py b/financial_index_replication/queries/replication_report.py new file mode 100644 index 00000000..90cc3a6e --- /dev/null +++ b/financial_index_replication/queries/replication_report.py @@ -0,0 +1,177 @@ +"""Replication report: reads the deployed index-replication model. + +Run after `rai models deploy --path model`: + python -m queries.replication_report + +Prints the selected 20-stock basket, sector exposures, tracking quality metrics, +and a top-correlation baseline comparison. Writes data/replica_returns.csv with +date, index_return, and replica_return columns for downstream plotting. +""" + +from math import sqrt + +import pandas as pd +from model import ( + DATA_DIR, + MAX_ADV_PARTICIPATION, + MAX_WEIGHT, + MONTHS_PER_YEAR, + N_REPLICATION_NAMES, + PORTFOLIO_VALUE, + SECTOR_ACTIVE_BAND, + SolveStatus, + Stock, + TotalResidual, + index_returns_csv, + model, + selected, + stock_returns_csv, + stocks_csv, + weight, +) + +if __name__ == "__main__": + print("=" * 70) + print("FINANCIAL INDEX REPLICATION") + print("=" * 70) + n_stocks = len(model.select(Stock.ticker).to_df()) + print(f"Universe: {n_stocks} stocks") + print(f"Selected names: exactly {N_REPLICATION_NAMES}") + print(f"Max position: {MAX_WEIGHT:.0%}") + print(f"Sector active band: +/- {SECTOR_ACTIVE_BAND:.0%}") + print(f"Portfolio value: ${PORTFOLIO_VALUE:,.0f}") + print(f"Max ADV participation per name: {MAX_ADV_PARTICIPATION:.0%}") + + # Status and objective of the latest refresh's solve. A refresh records any + # termination status, so the report checks it. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + residual = model.select(TotalResidual).to_df().iloc[:, 0].tolist() + if status not in (["OPTIMAL"], ["TIME_LIMIT"]): + raise SystemExit(f"Solver did not find an optimal solution: {status}") + + print(f"\nStatus: {status[0]}") + print(f"Objective: total absolute residual = {residual[0]:.6f}") + + # -------------------------------------------------- + # Evaluate and report + # -------------------------------------------------- + + solution_df = ( + model.select( + Stock.ticker.alias("ticker"), + Stock.sector.alias("sector"), + Stock.benchmark_weight.alias("benchmark_weight"), + Stock.previous_weight.alias("previous_weight"), + Stock.avg_dollar_volume.alias("avg_dollar_volume"), + selected.alias("selected"), + weight.alias("weight"), + ) + .where( + Stock.x_selected(selected), + Stock.x_weight(weight), + weight > 0.0001, + ) + .to_df() + .sort_values("weight", ascending=False) + ) + + returns_wide = stock_returns_csv.pivot( + index="date", columns="ticker", values="return" + ).sort_index() + index_series = index_returns_csv.set_index("date").sort_index() + weights = solution_df.set_index("ticker")["weight"] + portfolio_returns = returns_wide[weights.index].mul(weights, axis=1).sum(axis=1) + tracking_diff = portfolio_returns - index_series["index_return"] + + def annualized_tracking_error(): + monthly_rms = sqrt((tracking_diff ** 2).mean()) + return monthly_rms * sqrt(MONTHS_PER_YEAR) + + def mean_abs_tracking(): + return tracking_diff.abs().mean() + + print("\n=== Selected Replication Basket ===") + print( + solution_df[ + [ + "ticker", + "sector", + "weight", + "benchmark_weight", + "previous_weight", + "avg_dollar_volume", + ] + ].to_string( + index=False, + formatters={ + "weight": "{:.2%}".format, + "benchmark_weight": "{:.2%}".format, + "previous_weight": "{:.2%}".format, + "avg_dollar_volume": "${:,.0f}".format, + }, + ) + ) + + sector_report = ( + solution_df.groupby("sector", as_index=False)["weight"].sum() + .merge( + stocks_csv.groupby("sector", as_index=False)["benchmark_weight"].sum(), + on="sector", + ) + ) + sector_report["active_weight"] = ( + sector_report["weight"] - sector_report["benchmark_weight"] + ) + + print("\n=== Sector Exposure ===") + print( + sector_report.sort_values("sector").to_string( + index=False, + formatters={ + "weight": "{:.2%}".format, + "benchmark_weight": "{:.2%}".format, + "active_weight": "{:+.2%}".format, + }, + ) + ) + + turnover = ( + solution_df.set_index("ticker")["weight"] + .reindex(stocks_csv["ticker"], fill_value=0.0) + .sub(stocks_csv.set_index("ticker")["previous_weight"]) + .abs() + .sum() + ) + + print("\n=== Tracking Quality ===") + print(f"Annualized tracking error: {annualized_tracking_error():.2%}") + print(f"Mean abs monthly residual: {mean_abs_tracking():.4%}") + print(f"Implied turnover: {turnover:.2%}") + + # Simple benchmark: equal-weight the 20 most correlated stocks. + top_corr_ids = ( + returns_wide + .corrwith(index_series["index_return"]) + .abs() + .sort_values(ascending=False) + .head(20) + .index + ) + baseline_returns = returns_wide[top_corr_ids].mean(axis=1) + baseline_diff = baseline_returns - index_series["index_return"] + + baseline_tracking_error = sqrt((baseline_diff ** 2).mean()) * sqrt( + MONTHS_PER_YEAR + ) + + replica_returns_csv = DATA_DIR / "replica_returns.csv" + pd.DataFrame({ + "date": index_series.index, + "index_return": index_series["index_return"].values, + "replica_return": portfolio_returns.reindex(index_series.index).values, + }).to_csv(replica_returns_csv, index=False) + + print("\n=== Baseline Comparison ===") + print("Baseline: equal-weight top-20 stocks by full-history correlation") + print(f"Baseline annualized tracking error: {baseline_tracking_error:.2%}") + print(f"\nWrote benchmark-vs-replica returns to: {replica_returns_csv}") diff --git a/financial_index_replication/runbook.md b/financial_index_replication/runbook.md index d502d2b2..8cb9a58a 100644 --- a/financial_index_replication/runbook.md +++ b/financial_index_replication/runbook.md @@ -39,7 +39,7 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` **Prompt:** /rai-prescriptive-problem Which 20 stocks, and at what weights, best replicate the benchmark's monthly returns? Select exactly 20 names (binary) and assign continuous weights that sum to 1, with at most 10% in any one name and a weight only if the name is selected; keep each sector within ±4% of its benchmark weight; respect a trading-capacity limit (the change from the previous weight, scaled by a $10 million portfolio value, can't exceed 5% of a stock's average dollar volume); and define each month's tracking residual as benchmark return minus basket return. Minimize the total absolute monthly tracking error, and persist the selection and weights to the ontology. -**Response:** `financial_index_replication.py` returns a 20-name basket. The decision is `Stock.x_selected` (exactly 20 names) plus `Stock.x_weight` (summing to 1) and per-month residual variables. This is a cardinality-constrained tracking MILP: depending on the solver it either solves to OPTIMAL or returns a strong feasible incumbent at the time limit — either way the exact names and weights can vary, so treat the tracking quality below, not a specific basket, as the result. In `financial_index_replication.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. +**Response:** `model/reasoning.py` returns a 20-name basket. The decision is `Stock.x_selected` (exactly 20 names) plus `Stock.x_weight` (summing to 1) and per-month residual variables. This is a cardinality-constrained tracking MILP: depending on the solver it either solves to OPTIMAL or returns a strong feasible incumbent at the time limit — either way the exact names and weights can vary, so treat the tracking quality below, not a specific basket, as the result. In `model/reasoning.py` the solve is declared with `problem.configure(...)`, so each refresh of the deployed model runs it. ## 4. Assess tracking quality @@ -49,4 +49,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 50 stocks, 42 monthly index returns, 2,100 stock-month returns. The policy parameters (20 names, 10% max weight, ±4% sector band, ADV limit) are constants in `financial_index_replication.py`; the run writes a `replica_returns.csv` of monthly basket vs benchmark returns. The reusable model is split between `model/schema.py` and `model/source.py`, while the runner owns the decision problem, solve, result queries, and reporting. +Bundled CSVs in `data/`: 50 stocks, 42 monthly index returns, 2,100 stock-month returns. The policy parameters (20 names, 10% max weight, ±4% sector band, ADV limit) are constants in `model/reasoning.py`; the report writes a `replica_returns.csv` of monthly basket vs benchmark returns. The reusable model is split between `model/schema.py` and `model/source.py`, while `model/reasoning.py` declares the decision problem and the solve, and `queries/replication_report.py` reads the deployed model for the result queries and reporting. diff --git a/financial_index_replication/template-docs.yaml b/financial_index_replication/template-docs.yaml index 2dc4c529..d53586dd 100644 --- a/financial_index_replication/template-docs.yaml +++ b/financial_index_replication/template-docs.yaml @@ -6,6 +6,7 @@ page: - what-this-template-is-for omit_from_readme: [] download_and_run_from_readme: true + deployment: true model_explorer: title: Explore the model @@ -136,7 +137,7 @@ guided_walkthrough: monthly tracking fields prepare the model to compare the basket’s returns with the index in the next step. focus: - file: financial_index_replication.py + file: model/reasoning.py anchors: - start: Stock.x_selected = model.Property end: name=["weight", Stock.ticker], @@ -181,7 +182,7 @@ guided_walkthrough: the index. This comparison uses past returns and does not guarantee the basket will follow the index as closely in future months. focus: - file: financial_index_replication.py + file: model/reasoning.py anchors: - start: '# Monthly benchmark residual:' end: sum(ReturnMonth, pos_error + neg_error).where( @@ -217,7 +218,7 @@ guided_walkthrough: selected stock to have zero weight, so the model may invest in fewer than 20 stocks. focus: - file: financial_index_replication.py + file: model/reasoning.py anchors: - start: '# Select exactly N names and invest all capital.' end: ).require(weight <= MAX_WEIGHT * selected) @@ -258,7 +259,7 @@ guided_walkthrough: value. These constraints keep the sector mix near the index while limiting the trades needed to move from the previous portfolio. focus: - file: financial_index_replication.py + file: model/reasoning.py anchors: - start: '# Keep sector exposures close to benchmark sector weights.' end: PORTFOLIO_VALUE * (Stock.previous_weight - weight) @@ -279,15 +280,42 @@ guided_walkthrough: href: /build/guides/reasoning/prescriptive/decision-problems/constraints/ label: constraints - - id: solve-and-inspect-basket - title: Choose the basket and check tracking + - id: configure-the-solve + title: Run the solve on each refresh explanation: |- problem.configure(...) sets the solver and time limit that each refresh of the deployed model uses when it solves the decision problem under - the portfolio rules. model.deploy() deploys the model and waits for the - first refresh, which chooses stocks and weights. model.select(...) - queries the model for stocks with weights above 0.01% of the portfolio, - so a selected stock with zero weight does not appear in the result. + the portfolio rules. Deploying the model with rai models deploy --path + model waits for the first refresh, which chooses stocks and weights. + + SolveStatus and TotalResidual publish the solve's termination status + and total tracking residual through the model's own relationships, so + the report can read them after the deployment. + focus: + file: model/reasoning.py + anchors: + - start: problem.configure("highs", time_limit_sec=240) + end: model.define(SolveStatus(problem.termination_status()), TotalResidual(problem.objective_value())) + references: + problem.configure(...): + anchor: + start: problem.configure("highs", time_limit_sec=240) + end: problem.configure("highs", time_limit_sec=240) + SolveStatus: + symbol: SolveStatus + TotalResidual: + symbol: TotalResidual + links: + - title: Solve a decision problem + href: /build/guides/reasoning/prescriptive/decision-problems/solve/ + label: solves the decision problem + + - id: inspect-basket + title: Check tracking against the index + explanation: |- + model.select(...) queries the model for stocks with weights above 0.01% + of the portfolio, so a selected stock with zero weight does not appear + in the result. portfolio_returns calculates the basket’s monthly returns from those weights and stock returns, and tracking_diff compares them with the @@ -295,24 +323,16 @@ guided_walkthrough: as a yearly tracking measure, showing how closely this basket followed the index in the sample months. focus: - file: financial_index_replication.py + file: queries/replication_report.py anchors: - - start: problem.configure("highs", time_limit_sec=240) + - start: solution_df = ( end: return monthly_rms * sqrt(MONTHS_PER_YEAR) references: - problem.configure(...): - anchor: - start: problem.configure("highs", time_limit_sec=240) - end: problem.configure("highs", time_limit_sec=240) - model.deploy(): - anchor: - start: model.deploy() - end: model.deploy() model.select(...): anchor: start: model.select( end: .to_df() - occurrence: 3 + occurrence: 4 portfolio_returns: anchor: start: returns_wide = stock_returns_csv.pivot( @@ -326,9 +346,6 @@ guided_walkthrough: start: 'def annualized_tracking_error():' end: return monthly_rms * sqrt(MONTHS_PER_YEAR) links: - - title: Solve a decision problem - href: /build/guides/reasoning/prescriptive/decision-problems/solve/ - label: solves the decision problem - title: Query a model href: /build/guides/modeling/query-a-model/ label: queries the model From b7721b6a3af154d3b79e0c2b4f66bff6539adf92 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:39:13 -0800 Subject: [PATCH 069/162] commercial_underwriting: deploy the model package and query it Move the hierarchy, eligibility rules, and risk tiers into model/ and the report into queries/underwriting_report.py, run with rai models deploy --path model and python -m queries.underwriting_report. The report now stops, instead of printing a warning, when an entity lands in no tier or in two tiers. The README Quickstart follows the deployed-template steps. --- commercial_underwriting/README.md | 83 ++++++--- .../__init__.py} | 166 +---------------- .../queries/underwriting_report.py | 167 ++++++++++++++++++ commercial_underwriting/runbook.md | 2 +- 4 files changed, 230 insertions(+), 188 deletions(-) rename commercial_underwriting/{commercial_underwriting.py => model/__init__.py} (66%) create mode 100644 commercial_underwriting/queries/underwriting_report.py diff --git a/commercial_underwriting/README.md b/commercial_underwriting/README.md index b67b87d5..69d840e1 100644 --- a/commercial_underwriting/README.md +++ b/commercial_underwriting/README.md @@ -40,7 +40,8 @@ This template shows how to express that rulebook as declarative logic on a four- ## What's included - **Model**: the four-level `InsuredEntity` / `Policy` / `Location` / `Coverage` ontology, eight lower-level eligibility rules, two entity-level rollup rules (decline factors, marginal factors), and four mutually exclusive risk-tier subtype concepts — all as declarative derived Relationships. -- **Runner**: `commercial_underwriting.py` — a single Python script that loads the CSVs, applies the rules, and prints results end to end. +- **Model**: `model/__init__.py` — loads the CSVs and declares the rules and risk tiers that `rai models deploy --path model` deploys to your Snowflake account. +- **Query**: `queries/underwriting_report.py` — prints the eligibility flags, flag matrix, tier classification, and premium by tier from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Sample data**: four CSVs under `data/` — 8 insured businesses across 5 industries (`insured_entities.csv`), 8 commercial property policies one per insured (`policies.csv`), 11 scheduled locations (`locations.csv`), and 33 coverage lines at every location (`coverages.csv`). - **Outputs**: the printed eligibility flag matrix, the final per-entity tier classification, and the premium-by-tier business rollup. @@ -60,32 +61,42 @@ This template shows how to express that rulebook as declarative logic on a four- ## Quickstart -1. Download the ZIP file for this template and extract it: +Before you start, install Python 3.10 or later and get a Snowflake role that can deploy models, as described under Access. This template has no prescriptive solve, so it needs no setup beyond that role. + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/commercial_underwriting.zip). Unzip it and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/commercial_underwriting.zip unzip commercial_underwriting.zip cd commercial_underwriting ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. Create and activate a virtual environment: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. Install dependencies: +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: ```bash python -m pip install . ``` -4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist: ```yaml deployment: @@ -99,21 +110,25 @@ This template shows how to express that rulebook as declarative logic on a four- schedule: standard ``` - Without the `deployment` section, `python commercial_underwriting.py` stops with an error naming the missing setting. +5. **Deploy the model** -5. Run the template: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: ```bash - python commercial_underwriting.py + rai models deploy --path model ``` - The script deploys the model, waits for its first refresh, and prints the report from the deployed model. `rai models deploy` deploys without printing the report. + The deploy waits for the model's first refresh, which evaluates the eligibility rules and the risk-tier classification. Without the `deployment` section, it stops with an error naming the missing setting. - Running the script again without changing the model redeploys nothing; it prints the report from the deployed model. A change to the model redeploys it. + Deploying again without changing the model applies no updates. A change to the model redeploys it. - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. +6. **Query the results** -6. Expected output (truncated): + Print the report from the deployed model: + + ```bash + python -m queries.underwriting_report + ``` ```text Stage 1: Location- and coverage-level eligibility flags @@ -155,7 +170,18 @@ This template shows how to express that rulebook as declarative logic on a four- STANDARD 1 18000.0 18000.000000 ``` - Each entity falls into exactly one tier — the four tier concepts partition the `InsuredEntity` set. The premium-by-tier rollup is the underwriting business view: about $150K of bound premium is associated with submissions that today's rules would decline; about $236K is non-standard and could be repriced. + Each entity falls into exactly one tier, because the four tier concepts partition the insured entities, and the report stops if any entity lands in no tier or in two. The premium-by-tier rollup is the underwriting business view: about $150K of bound premium is associated with submissions that today's rules would decline; about $236K is non-standard and could be repriced. + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The teardown removes the deployed objects and keeps the two schemas. ## Template structure @@ -163,7 +189,10 @@ This template shows how to express that rulebook as declarative logic on a four- . ├── README.md # this file ├── pyproject.toml # dependencies -├── commercial_underwriting.py # main script (end-to-end) +├── model/ +│ └── __init__.py # hierarchy, rules, and risk tiers +├── queries/ +│ └── underwriting_report.py # report printed from the deployed model └── data/ ├── insured_entities.csv # top of the hierarchy ├── policies.csv # one policy per insured @@ -171,7 +200,7 @@ This template shows how to express that rulebook as declarative logic on a four- └── coverages.csv # coverage lines at each location ``` -**Start here:** run `python commercial_underwriting.py` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here:** deploy with `rai models deploy --path model` and run `python -m queries.underwriting_report` for the full run end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -194,7 +223,7 @@ The eight insureds are designed so each tier is represented and every rule fires - **Primary identifiers**: each level has an integer `id`; the hierarchy is threaded by relationships (`Policy.insured_entity`, `Location.policy`, `Coverage.location`) rather than by shared keys. - **Important invariants**: every policy belongs to exactly one insured entity, every location to one policy, every coverage to one location; the ITV check applies to `BUILDING` coverages only; the four risk tiers partition the `InsuredEntity` set (each entity lands in exactly one). -For the full concept and property definitions, see `commercial_underwriting.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions, see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -206,17 +235,17 @@ Those lower-level flags then roll up. Entity-level rollup rules traverse the `In InsuredEntity -> Policy -> Location -> Coverage ``` -See `commercial_underwriting.py` for the implementation and `runbook.md` to reproduce it step by step with the RAI skills. +See `model/__init__.py` for the implementation, `queries/underwriting_report.py` for the report, and `runbook.md` to reproduce it step by step with the RAI skills. ## Customize this template ### Use your own data -Replace the four CSVs with your own underwriting submissions. Required columns are listed in the *Model overview* section. The script's `Concept.new(...)` and `Relationship` definitions match the CSV column names exactly — keep the same headers or update the script's `model.data(...)` calls to match. +Replace the four CSVs with your own underwriting submissions. Required columns are listed in the *Model overview* section. The `Concept.new(...)` and `Relationship` definitions in `model/__init__.py` match the CSV column names exactly — keep the same headers or update the `model.data(...)` calls to match. ### Tune parameters -All thresholds are at the top of the script under `# Configure inputs`: +All thresholds are at the top of `model/__init__.py` under `# Configure inputs`: | Parameter | Default | Effect | |---|---|---| @@ -240,8 +269,8 @@ All thresholds are at the top of the script under `# Configure inputs`: ### Scale up / productionize - **Point at Snowflake tables.** Swap the CSV `model.data(...)` calls for `model.Table(...)` references to your submission tables; the ontology, rules, and tier classification are unchanged. Size the RAI engine to the submission volume — rules-based classification is lightweight, so a small engine handles large books. -- **Schedule triage runs.** Wire the script into your submission-intake pipeline (nightly batch or on-submission trigger) so every new account is scored as it arrives, and persist the per-entity tier and flag matrix for downstream underwriting review. -- **Pin dependencies for reproducibility.** Keep the `relationalai` version pinned in `pyproject.toml` so rule outcomes stay stable across runs, and version-control your threshold constants (top of the script) as auditable underwriting parameters. +- **Schedule triage runs.** Redeploy the model from your submission-intake pipeline (nightly batch or on-submission trigger) so every new account is scored as it arrives, and persist the per-entity tier and flag matrix for downstream underwriting review. +- **Pin dependencies for reproducibility.** Keep the `relationalai` version pinned in `pyproject.toml` so rule outcomes stay stable across runs, and version-control your threshold constants (top of `model/__init__.py`) as auditable underwriting parameters. ## Troubleshooting @@ -260,13 +289,13 @@ All thresholds are at the top of the script under `# Configure inputs`:
An entity isn't classified into any tier. - The script verifies mutual exclusivity and prints a warning if any entity is unclassified. The most likely cause is a rule condition that doesn't match the data — verify with `model.select(InsuredEntity.id).where(InsuredEntity.has_decline_factor()).to_df()` and the analogous `has_marginal_factor` query that those flags fire as expected. + The report verifies mutual exclusivity and stops with an error if any entity is unclassified. The most likely cause is a rule condition that doesn't match the data — verify with `model.select(InsuredEntity.id).where(InsuredEntity.has_decline_factor()).to_df()` and the analogous `has_marginal_factor` query that those flags fire as expected.
Two tiers fire for the same entity. - This indicates the tier conditions are not mutually exclusive. The script prints a warning if it finds any overlap. Check the `model.not_()` clauses on the lower-priority tier rules (NonStandard / Preferred / Standard) — each must explicitly exclude the higher-priority tier's flag. + This indicates the tier conditions are not mutually exclusive. The report stops with an error if it finds any overlap. Check the `model.not_()` clauses on the lower-priority tier rules (NonStandard / Preferred / Standard) — each must explicitly exclude the higher-priority tier's flag.
diff --git a/commercial_underwriting/commercial_underwriting.py b/commercial_underwriting/model/__init__.py similarity index 66% rename from commercial_underwriting/commercial_underwriting.py rename to commercial_underwriting/model/__init__.py index 8e167f1b..0e69b819 100644 --- a/commercial_underwriting/commercial_underwriting.py +++ b/commercial_underwriting/model/__init__.py @@ -1,6 +1,6 @@ """Commercial Underwriting (rules-based reasoning) template. -This script demonstrates a rules-based underwriting workflow in RelationalAI +This model declares a rules-based underwriting workflow in RelationalAI on a four-level commercial property/casualty hierarchy: InsuredEntity -> Policy -> Location -> Coverage @@ -20,26 +20,21 @@ PyRel resolves dependencies automatically. Run: - `python commercial_underwriting.py` deploys the model to the schema in - raiconfig.yaml, waits for its first refresh, and prints the report from the - deployed model. - -Output: - Prints the eligibility flags found at each level, the per-entity - decline reasons, the per-entity marginal factors, and the final - risk-tier classification table. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh; `python -m + queries.underwriting_report` then prints the report from the deployed model. """ from pathlib import Path from pandas import read_csv -from relationalai.semantics import Float, Integer, Model, String, sum +from relationalai.semantics import Float, Integer, Model, String # -------------------------------------------------- # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # Eligibility thresholds (typical commercial-lines underwriting guardrails). MIN_YEARS_IN_BUSINESS = 3 # below this -> insufficient history -> decline @@ -321,152 +316,3 @@ model.not_(InsuredEntity.has_marginal_factor()), model.not_(RiskTier_Preferred(InsuredEntity)), ).define(RiskTier_Standard(InsuredEntity)) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh. - model.deploy() - - print("=" * 70) - print("Stage 1: Location- and coverage-level eligibility flags") - print("=" * 70) - - poor_fp = model.where(Location.has_poor_fire_protection()).select( - Location.id.alias("location_id"), - Location.address.alias("address"), - Location.fire_protection_class.alias("fp_class"), - ).to_df() - print(f"\nLocations with poor fire protection (FP > {MAX_FP_CLASS_FOR_ELIGIBILITY}): {len(poor_fp)}") - if not poor_fp.empty: - print(poor_fp.to_string(index=False)) - - old_constr = model.where(Location.has_old_construction()).select( - Location.id.alias("location_id"), - Location.address.alias("address"), - Location.year_built.alias("year_built"), - ).to_df() - print(f"\nLocations with old construction (year_built < {MIN_YEAR_BUILT}): {len(old_constr)}") - if not old_constr.empty: - print(old_constr.to_string(index=False)) - - high_risk = model.where(Location.has_high_risk_occupancy()).select( - Location.id.alias("location_id"), - Location.address.alias("address"), - Location.occupancy_class.alias("occupancy"), - ).to_df() - print(f"\nLocations with high-risk occupancy: {len(high_risk)}") - if not high_risk.empty: - print(high_risk.to_string(index=False)) - - underinsured = model.where(Coverage.is_underinsured()).select( - Coverage.id.alias("coverage_id"), - Coverage.location.address.alias("location_address"), - Coverage.coverage_limit.alias("limit"), - Coverage.location.replacement_value.alias("replacement_value"), - ).to_df() - underinsured["itv_pct"] = ( - underinsured["limit"] / underinsured["replacement_value"] * 100 - ).round(1) - print(f"\nUnderinsured BUILDING coverages (ITV < {int(ITV_THRESHOLD * 100)}%): {len(underinsured)}") - if not underinsured.empty: - print(underinsured.to_string(index=False)) - - print("\n" + "=" * 70) - print("Stage 2: Per-entity decline and marginal factors") - print("=" * 70) - - - def query_flag(rel, label): - """Return a list of insured-entity IDs that match a unary boolean Relationship.""" - df = model.where(rel()).select(InsuredEntity.id.alias("id")).to_df() - if df.empty: - return set(), label - return set(df["id"]), label - - - decline_factor_rels = [ - (InsuredEntity.has_restricted_industry, "restricted_industry"), - (InsuredEntity.has_insufficient_history, "insufficient_history"), - (InsuredEntity.has_unprotected_location, "unprotected_location"), - (InsuredEntity.has_old_construction_location, "old_construction_location"), - ] - marginal_factor_rels = [ - (InsuredEntity.has_marginal_industry, "marginal_industry"), - (InsuredEntity.has_high_risk_occupancy_location, "high_risk_occupancy_location"), - (InsuredEntity.has_underinsured_coverage, "underinsured_coverage"), - ] - - ie_df = model.select( - InsuredEntity.id.alias("id"), - InsuredEntity.name.alias("name"), - InsuredEntity.industry_code.alias("industry"), - InsuredEntity.years_in_business.alias("years"), - ).to_df().sort_values("id").reset_index(drop=True) - - for rel, label in decline_factor_rels + marginal_factor_rels: - ids, _ = query_flag(rel, label) - ie_df[label] = ie_df["id"].isin(ids) - - print("\nFlag matrix (True = flag set):") - print(ie_df.to_string(index=False)) - - print("\n" + "=" * 70) - print("Stage 3: Risk-tier classification") - print("=" * 70) - - decline_ids = set( - model.where(RiskTier_Decline(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] - ) - nonstandard_ids = set( - model.where(RiskTier_NonStandard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] - ) - preferred_ids = set( - model.where(RiskTier_Preferred(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] - ) - standard_ids = set( - model.where(RiskTier_Standard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] - ) - - - def classify(entity_id): - if entity_id in decline_ids: - return "DECLINE" - if entity_id in nonstandard_ids: - return "NON-STANDARD" - if entity_id in preferred_ids: - return "PREFERRED" - if entity_id in standard_ids: - return "STANDARD" - return "UNCLASSIFIED" - - - tier_df = ie_df[["id", "name", "industry", "years"]].copy() - tier_df["tier"] = tier_df["id"].apply(classify) - print("\nFinal risk tiers:") - print(tier_df.to_string(index=False)) - - # Sanity check: every entity must land in exactly one tier (mutual exclusivity). - print("\nTier counts:") - print(tier_df["tier"].value_counts().to_string()) - - unclassified = tier_df[tier_df["tier"] == "UNCLASSIFIED"] - if not unclassified.empty: - print(f"\nWARNING: {len(unclassified)} entities are unclassified — rule set is non-exhaustive.") - - # Verify mutual exclusivity: no entity in two tiers. - overlap = (decline_ids & nonstandard_ids) | (decline_ids & standard_ids) | (decline_ids & preferred_ids) \ - | (nonstandard_ids & standard_ids) | (nonstandard_ids & preferred_ids) | (standard_ids & preferred_ids) - if overlap: - print(f"\nWARNING: {len(overlap)} entities classified into multiple tiers: {sorted(overlap)}") - - # Aggregate premium by tier — the underwriting business view. - premium_df = model.where(Policy.insured_entity(InsuredEntity)).select( - InsuredEntity.id.alias("id"), - sum(Policy, Policy.total_premium).per(InsuredEntity).alias("premium"), - ).to_df() - tier_df = tier_df.merge(premium_df, on="id") - print("\nPremium by tier:") - print(tier_df.groupby("tier")["premium"].agg(["count", "sum", "mean"]).to_string()) diff --git a/commercial_underwriting/queries/underwriting_report.py b/commercial_underwriting/queries/underwriting_report.py new file mode 100644 index 00000000..4d056b39 --- /dev/null +++ b/commercial_underwriting/queries/underwriting_report.py @@ -0,0 +1,167 @@ +"""Underwriting report: reads the deployed commercial-underwriting model. + +Run after `rai models deploy --path model`: + python -m queries.underwriting_report + +Prints the eligibility flags found at each level, the per-entity decline reasons +and marginal factors, the final risk-tier classification table, and the premium +by tier. +""" + +from model import ( + ITV_THRESHOLD, + MAX_FP_CLASS_FOR_ELIGIBILITY, + MIN_YEAR_BUILT, + Coverage, + InsuredEntity, + Location, + Policy, + RiskTier_Decline, + RiskTier_NonStandard, + RiskTier_Preferred, + RiskTier_Standard, + model, +) +from relationalai.semantics import sum + +if __name__ == "__main__": + print("=" * 70) + print("Stage 1: Location- and coverage-level eligibility flags") + print("=" * 70) + + poor_fp = model.where(Location.has_poor_fire_protection()).select( + Location.id.alias("location_id"), + Location.address.alias("address"), + Location.fire_protection_class.alias("fp_class"), + ).to_df() + print(f"\nLocations with poor fire protection (FP > {MAX_FP_CLASS_FOR_ELIGIBILITY}): {len(poor_fp)}") + if not poor_fp.empty: + print(poor_fp.to_string(index=False)) + + old_constr = model.where(Location.has_old_construction()).select( + Location.id.alias("location_id"), + Location.address.alias("address"), + Location.year_built.alias("year_built"), + ).to_df() + print(f"\nLocations with old construction (year_built < {MIN_YEAR_BUILT}): {len(old_constr)}") + if not old_constr.empty: + print(old_constr.to_string(index=False)) + + high_risk = model.where(Location.has_high_risk_occupancy()).select( + Location.id.alias("location_id"), + Location.address.alias("address"), + Location.occupancy_class.alias("occupancy"), + ).to_df() + print(f"\nLocations with high-risk occupancy: {len(high_risk)}") + if not high_risk.empty: + print(high_risk.to_string(index=False)) + + underinsured = model.where(Coverage.is_underinsured()).select( + Coverage.id.alias("coverage_id"), + Coverage.location.address.alias("location_address"), + Coverage.coverage_limit.alias("limit"), + Coverage.location.replacement_value.alias("replacement_value"), + ).to_df() + underinsured["itv_pct"] = ( + underinsured["limit"] / underinsured["replacement_value"] * 100 + ).round(1) + print(f"\nUnderinsured BUILDING coverages (ITV < {int(ITV_THRESHOLD * 100)}%): {len(underinsured)}") + if not underinsured.empty: + print(underinsured.to_string(index=False)) + + print("\n" + "=" * 70) + print("Stage 2: Per-entity decline and marginal factors") + print("=" * 70) + + + def query_flag(rel, label): + """Return a list of insured-entity IDs that match a unary boolean Relationship.""" + df = model.where(rel()).select(InsuredEntity.id.alias("id")).to_df() + if df.empty: + return set(), label + return set(df["id"]), label + + + decline_factor_rels = [ + (InsuredEntity.has_restricted_industry, "restricted_industry"), + (InsuredEntity.has_insufficient_history, "insufficient_history"), + (InsuredEntity.has_unprotected_location, "unprotected_location"), + (InsuredEntity.has_old_construction_location, "old_construction_location"), + ] + marginal_factor_rels = [ + (InsuredEntity.has_marginal_industry, "marginal_industry"), + (InsuredEntity.has_high_risk_occupancy_location, "high_risk_occupancy_location"), + (InsuredEntity.has_underinsured_coverage, "underinsured_coverage"), + ] + + ie_df = model.select( + InsuredEntity.id.alias("id"), + InsuredEntity.name.alias("name"), + InsuredEntity.industry_code.alias("industry"), + InsuredEntity.years_in_business.alias("years"), + ).to_df().sort_values("id").reset_index(drop=True) + + for rel, label in decline_factor_rels + marginal_factor_rels: + ids, _ = query_flag(rel, label) + ie_df[label] = ie_df["id"].isin(ids) + + print("\nFlag matrix (True = flag set):") + print(ie_df.to_string(index=False)) + + print("\n" + "=" * 70) + print("Stage 3: Risk-tier classification") + print("=" * 70) + + decline_ids = set( + model.where(RiskTier_Decline(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + nonstandard_ids = set( + model.where(RiskTier_NonStandard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + preferred_ids = set( + model.where(RiskTier_Preferred(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + standard_ids = set( + model.where(RiskTier_Standard(InsuredEntity)).select(InsuredEntity.id.alias("id")).to_df()["id"] + ) + + + def classify(entity_id): + if entity_id in decline_ids: + return "DECLINE" + if entity_id in nonstandard_ids: + return "NON-STANDARD" + if entity_id in preferred_ids: + return "PREFERRED" + if entity_id in standard_ids: + return "STANDARD" + return "UNCLASSIFIED" + + + tier_df = ie_df[["id", "name", "industry", "years"]].copy() + tier_df["tier"] = tier_df["id"].apply(classify) + print("\nFinal risk tiers:") + print(tier_df.to_string(index=False)) + + # Sanity check: every entity must land in exactly one tier (mutual exclusivity). + print("\nTier counts:") + print(tier_df["tier"].value_counts().to_string()) + + unclassified = tier_df[tier_df["tier"] == "UNCLASSIFIED"] + if not unclassified.empty: + raise SystemExit(f"{len(unclassified)} entities are unclassified: the rule set is non-exhaustive.") + + # Verify mutual exclusivity: no entity in two tiers. + overlap = (decline_ids & nonstandard_ids) | (decline_ids & standard_ids) | (decline_ids & preferred_ids) \ + | (nonstandard_ids & standard_ids) | (nonstandard_ids & preferred_ids) | (standard_ids & preferred_ids) + if overlap: + raise SystemExit(f"{len(overlap)} entities classified into multiple tiers: {sorted(overlap)}") + + # Aggregate premium by tier — the underwriting business view. + premium_df = model.where(Policy.insured_entity(InsuredEntity)).select( + InsuredEntity.id.alias("id"), + sum(Policy, Policy.total_premium).per(InsuredEntity).alias("premium"), + ).to_df() + tier_df = tier_df.merge(premium_df, on="id") + print("\nPremium by tier:") + print(tier_df.groupby("tier")["premium"].agg(["count", "sum", "mean"]).to_string()) diff --git a/commercial_underwriting/runbook.md b/commercial_underwriting/runbook.md index e8d19d01..9ac5f92e 100644 --- a/commercial_underwriting/runbook.md +++ b/commercial_underwriting/runbook.md @@ -38,4 +38,4 @@ Each prompt is pasted into a fresh agent session loaded with the named `/rai-*` ## Data -Bundled CSVs in `data/`: 8 insured entities, 8 policies, 11 locations, 33 coverages. Full model in `commercial_underwriting.py`. +Bundled CSVs in `data/`: 8 insured entities, 8 policies, 11 locations, 33 coverages. Full model in `model/__init__.py`. From 16ac91e1e074f335dca98e5f522c6da2c36b7992 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:44:22 -0800 Subject: [PATCH 070/162] defect_root_cause: deploy the model package and query it Move the ontology and the graph, rules, and prescriptive stages into model/ and the report into queries/root_cause_report.py, run with rai models deploy --path model and python -m queries.root_cause_report. The plant-wide baseline failure rate is now a model fact derived from the distinct units in the model (PlantBaseline), not a ratio of CSV rows, so the lift of each factor follows the modeled units. The report stops unless the solve terminated OPTIMAL, and its formulation checks raise instead of asserting. The README Quickstart follows the deployed-template steps. --- defect_root_cause/README.md | 117 +++++---- .../__init__.py} | 225 ++---------------- .../queries/root_cause_report.py | 215 +++++++++++++++++ defect_root_cause/runbook.md | 2 +- 4 files changed, 317 insertions(+), 242 deletions(-) rename defect_root_cause/{defect_root_cause.py => model/__init__.py} (65%) create mode 100644 defect_root_cause/queries/root_cause_report.py diff --git a/defect_root_cause/README.md b/defect_root_cause/README.md index 8192c9bf..781990c0 100644 --- a/defect_root_cause/README.md +++ b/defect_root_cause/README.md @@ -45,7 +45,8 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo ## What's included -- **Model and pipeline** (`defect_root_cause.py`): a single script that defines the ontology and all three reasoning stages, deploys the model to your Snowflake account, and prints the report from the deployed model. +- **Model** (`model/__init__.py`): the ontology and all three reasoning stages, which `rai models deploy --path model` deploys to your Snowflake account. +- **Query** (`queries/root_cause_report.py`): prints the report from the deployed model. - **Runbook**: `runbook.md` — a paste-testable walkthrough that reproduces the template step by step with the RAI skills; as important a reference as the script itself. - **Data generator** (`generate_data.py`): rebuilds the sample corpus deterministically from a fixed seed. The generated files are already included, so you do not need to run it to use the template. - **Sample data** (`data/`): ten CSV files describing the product structure, the material lots and their genealogy, the finished units and their test results, the process history, and the list of candidate causes. @@ -56,43 +57,72 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo ### Access - A Snowflake account with the RelationalAI Native App installed. -- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (Quickstart step 4 below covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. +- The one-time model deployment setup on that account, done by an account administrator: follow [Set up model deployment](https://private.relational.ai/early-access/model-deploy-setup) through **Governed setup**, **PAT authentication**, and steps 1-2 of **Additional setup for the prescriptive reasoner** (the Quickstart covers this template's own schema). It gives you a model role, a programmatic access token (PAT) stored as a secret, an external access integration, a PAT user, and a warehouse. - A Snowflake user that holds the model role. - A prescriptive engine for the optimization step. The template solves with the open-source HiGHS solver, so no additional solver license is required. ### Tools - Python 3.10 or later. -- The RelationalAI Python library, installed with the steps below and configured with `rai init`. +- The RelationalAI Python library and the `rai` CLI, installed by the Quickstart's `python -m pip install .`. ## Quickstart -1. Download and extract this template: +Before you start, install Python 3.10 or later and complete the access prerequisites above: a Snowflake account with the RAI Native App, the one-time model deployment setup, and a Snowflake user that holds the model role. + +Before the first deploy, create this template's internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: + +```sql +-- as the model role +CREATE SCHEMA IF NOT EXISTS .DEFECT_ROOT_CAUSE_META; + +-- as an administrator +GRANT CALLER USAGE ON DATABASE TO APPLICATION ; +GRANT CALLER USAGE ON SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; +``` + +Use this sequence to deploy the model and query it: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/defect_root_cause.zip), unzip it, and enter the template directory: ```bash - curl -O https://docs.relational.ai/templates/zips/v1/defect_root_cause.zip unzip defect_root_cause.zip cd defect_root_cause ``` - > [!TIP] - > You can also download the template ZIP using the "Download ZIP" button at the top of this page. +2. **Create a Python environment** -2. Create and activate a virtual environment: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate - python -m pip install -U pip + python -m pip install --upgrade pip ``` -3. Install dependencies: +3. **Install the template** + + Install the dependencies pinned in `pyproject.toml`: ```bash python -m pip install . ``` -4. Configure the Snowflake connection and the deployment: run `rai init` to create `raiconfig.yaml` with your Snowflake connection, set the connection `role` to the model role, then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory and verify your connection. + + Set the connection `role` to the model role, then add the prescriptive reasoner and a `deployment` section. Give each template its own schema pair so two deployed templates never share one: + ```yaml reasoners: prescriptive: @@ -111,37 +141,24 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo pat_user: pat_warehouse: ``` - Without the `deployment` section, `python defect_root_cause.py` stops with an error naming the missing setting. - Before the first run, create the internal schema as the model role, and have an administrator grant the RAI Native App caller access to it. The internal schema holds only this model's objects, so the grants cover all of it, including the solver's result tables that the deploy creates: - ```sql - -- as the model role - CREATE SCHEMA IF NOT EXISTS .DEFECT_ROOT_CAUSE_META; - - -- as an administrator - GRANT CALLER USAGE ON DATABASE TO APPLICATION ; - GRANT CALLER USAGE ON SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - GRANT CALLER USAGE ON WAREHOUSE TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL DYNAMIC TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - GRANT INHERITED CALLER SELECT ON ALL VIEWS IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - GRANT INHERITED CALLER INSERT, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - GRANT CALLER CREATE TABLE, CREATE STAGE ON SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - GRANT INHERITED CALLER READ, WRITE ON ALL STAGES IN SCHEMA .DEFECT_ROOT_CAUSE_META TO APPLICATION ; - ``` - The deploy creates the public schema `.DEFECT_ROOT_CAUSE` if it doesn't exist. +5. **Deploy the model** -5. Run the template: + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: ```bash - python defect_root_cause.py + rai models deploy --path model ``` - The script deploys the model, waits for the first refresh, which runs the solve, and prints the report from the deployed model. Each refresh of the deployed model runs the solve again; `rai models deploy` deploys without printing the report. + The deploy creates the public schema named in `raiconfig.yaml` if it doesn't exist and waits for the model's first refresh, which runs the solve. Each refresh of the deployed model runs the solve again. - To remove the deployment, run `rai models teardown --force` from this directory; it keeps the two schemas. +6. **Query the results** -6. Confirm success. The run ends with the diagnosis; the final lines should look like this: + Print the report from the deployed model: + + ```bash + python -m queries.root_cause_report + ``` ```text Root-cause diagnosis: 2 factors explain 120/142 defective units (85% coverage) @@ -150,7 +167,24 @@ At a high level, this exercises three RelationalAI capabilities on one shared mo MACHINE::REF-02 Reflow oven 2 (REFLOW) MACHINE 73 2.167304 ``` - The full annotated output appears in the [Sample output](#sample-output) section below. + The lines above are the end of the report. The report checks the solve's termination status and stops unless it is OPTIMAL. The full annotated output appears in the Sample output section below. + + To solve again without changing the model, refresh the deployed model, then run the query again: + + ```sql + CALL .DEFECT_ROOT_CAUSE_META._INTERNAL_REFRESH_MODEL('standard', NULL); + ``` + +7. **Tear down the deployment** + + Preview what the teardown removes, then remove it: + + ```bash + rai models teardown + rai models teardown --force + ``` + + The first command lists the model's objects without dropping anything; the second drops them and keeps the two schemas. ## Template structure @@ -159,7 +193,10 @@ defect_root_cause/ ├── README.md ├── runbook.md # analyst-facing, paste-testable walkthrough ├── pyproject.toml -├── defect_root_cause.py # the three-stage pipeline +├── model/ +│ └── __init__.py # ontology and the three reasoning stages +├── queries/ +│ └── root_cause_report.py # report printed from the deployed model ├── generate_data.py # regenerates the corpus (CSVs already included) └── data/ ├── skus.csv # 20 SKUs across four tiers @@ -174,7 +211,7 @@ defect_root_cause/ └── factors.csv # 118 candidate causes (lots, machines, and shifts) ``` -**Start here:** run `python defect_root_cause.py` for the whole pipeline end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. +**Start here:** deploy with `rai models deploy --path model` and run `python -m queries.root_cause_report` for the whole pipeline end to end, or follow `runbook.md` to reproduce it step by step with the RAI skills. ## Sample data @@ -186,11 +223,11 @@ The model represents a manufacturing genealogy: finished units built from materi The genealogy is the backbone: a built `Lot` consumes its input lots, so following those links from a finished unit reaches every material beneath it. -- **Key entities**: `SKU`, `Lot`, `Unit`, `Machine`, and `Factor` (a candidate root cause — any lot, machine, or shift, put on a common footing); plus a `DiagnosisResult` summary derived from the prescriptive stage's result. +- **Key entities**: `SKU`, `Lot`, `Unit`, `Machine`, and `Factor` (a candidate root cause — any lot, machine, or shift, put on a common footing); plus a `PlantBaseline` singleton holding the plant-wide failure rate (defective units over the distinct units in the model) and a `DiagnosisResult` summary derived from the prescriptive stage's result. - **Primary identifiers**: every concept carries a string `id` (for example `LOT::SP-0423` on `Factor`); the genealogy and process history are threaded by relationships (`Lot consumes Lot`, `Unit consumes Lot`, `Unit ran on Machine`) rather than shared keys. - **Important invariants**: `Unit.defective` is `0` or `1`; `Factor.lift` is the touched-unit defect rate over the plant baseline (≈ 1.0 for non-causal high-volume factors); the `is_root_cause` decision variable is binary; a defective unit is either explained by a named cause it touches or marked unexplained. -For the full concept and property definitions — including the relations the reasoning traces (`Lot consumes Lot`, `Unit consumes Lot`, `Unit ran on Machine`, and the derived `Factor touches Unit` incidence) — see `defect_root_cause.py`; `runbook.md` builds them step by step with the RAI skills. +For the full concept and property definitions — including the relations the reasoning traces (`Lot consumes Lot`, `Unit consumes Lot`, `Unit ran on Machine`, and the derived `Factor touches Unit` incidence) — see `model/__init__.py`; `runbook.md` builds them step by step with the RAI skills. ## How it works @@ -216,7 +253,7 @@ Each candidate is scored by how concentrated defects are among the units it touc A binary decision variable names each suspect a root cause, and a binary slack lets a defective unit be left unexplained. The coverage constraint requires every defective unit to be explained by a named cause it touches or else marked unexplained. The objective trades off three things: a cost for each named cause (so the diagnosis stays small), a penalty for naming a cause that also touches many good units (so it stays specific), and a cost for leaving a defect unexplained. Together these make the optimizer prefer the single deep root over the several proximate sub-assemblies that merely carry it. -See `defect_root_cause.py` for the implementation and `runbook.md` to reproduce each stage step by step with the RAI skills. +See `model/__init__.py` for the implementation, `queries/root_cause_report.py` for the report, and `runbook.md` to reproduce each stage step by step with the RAI skills. ## Sample output diff --git a/defect_root_cause/defect_root_cause.py b/defect_root_cause/model/__init__.py similarity index 65% rename from defect_root_cause/defect_root_cause.py rename to defect_root_cause/model/__init__.py index 438b7113..97c6b38d 100644 --- a/defect_root_cause/defect_root_cause.py +++ b/defect_root_cause/model/__init__.py @@ -31,16 +31,9 @@ enrichments as first-class properties -- the accretive enrichment pattern. Run: - python defect_root_cause.py deploys the model to the schema in raiconfig.yaml, - waits for its first refresh (which runs the solve), and prints the report from - the deployed model. - -Output: - Prints per-stage diagnostics -- genealogy reachability validation, the - contrast-scored suspect factors with lift, the MILP termination status, - and a final DiagnosisResult singleton row (defective units, units - explained, coverage, and the ranked root-cause factors) showing the - diagnosis as queryable ontology. + `rai models deploy --path model` deploys this model to the schema in + raiconfig.yaml and waits for its first refresh, which runs the solve; `python + -m queries.root_cause_report` then prints the report from the deployed model. """ from pathlib import Path @@ -61,7 +54,7 @@ # Configure inputs # -------------------------------------------------- -DATA_DIR = Path(__file__).parent / "data" +DATA_DIR = Path(__file__).resolve().parents[1] / "data" # Stage 2 contrast-scoring thresholds. A factor is a suspect only if defects # among the units it touches run materially above the plant-wide baseline @@ -170,8 +163,7 @@ model.define(Lot.lookup(id=gd.PARENT_LOT_ID).consumes(Lot.lookup(id=gd.CHILD_LOT_ID))) # Units, their final-test result, and the top-tier lots they consume. -units_csv = read_csv(DATA_DIR / "units.csv") -ud = model.data(units_csv) +ud = model.data(read_csv(DATA_DIR / "units.csv")) model.define( u := Unit.new(id=ud.UNIT_ID), u.sku(SKU.lookup(id=ud.SKU_ID)), @@ -235,8 +227,21 @@ # Stage 2 -- Rules: contrast scoring of candidate factors # -------------------------------------------------- -# Plant-wide failure rate, from the input data. -BASELINE_RATE = int((units_csv["DEFECTIVE"] == 1).sum()) / len(units_csv) +# Plant-wide failure rate: defective units over the distinct units in the model. +PlantBaseline = model.Concept("PlantBaseline", identify_by={"id": Integer}) +PlantBaseline.n_units = model.Property(f"{PlantBaseline} has {Integer:n_units}") +PlantBaseline.n_defective = model.Property(f"{PlantBaseline} has {Integer:n_defective}") +PlantBaseline.failure_rate = model.Property(f"{PlantBaseline} has {Float:failure_rate}") +model.define(PlantBaseline.new(id=1)) +model.define(PlantBaseline.n_units(aggs.count(Unit).per(PlantBaseline))) +model.define( + PlantBaseline.n_defective(aggs.count(Unit).per(PlantBaseline).where(Unit.is_defective())) +) +model.define( + PlantBaseline.failure_rate( + floats.float(PlantBaseline.n_defective) / floats.float(PlantBaseline.n_units) + ) +) # Per-factor incidence: how many units it touches, and how many of those failed. Factor.touched_count = model.Property(f"{Factor} touches {Integer:touched_count}") @@ -255,7 +260,10 @@ Factor.defect_rate(floats.float(Factor.defect_count) / floats.float(Factor.touched_count)) ) model.where(Factor.defect_count > 0).define( - Factor.lift((floats.float(Factor.defect_count) / floats.float(Factor.touched_count)) / BASELINE_RATE) + Factor.lift( + (floats.float(Factor.defect_count) / floats.float(Factor.touched_count)) + / PlantBaseline.failure_rate + ) ) # A factor is a suspect only when defect lift, support, and defect count all # clear threshold -- screening out high-coverage, low-lift trunk factors. @@ -357,188 +365,3 @@ / floats.float(DiagnosisResult.n_defective) ) ) - -# -------------------------------------------------- -# Deploy, then report (`rai models deploy` skips this block) -# -------------------------------------------------- - -if __name__ == "__main__": - # Deploy and wait for the first refresh, which runs the solve. - model.deploy() - - n_units = model.select(Unit.id).to_df() - n_def = model.where(Unit.is_defective()).select(Unit.id).to_df() - print( - f"Loaded {len(n_units)} units, {len(n_def)} defective ({len(n_def) / len(n_units):.2%} final-test failure rate)" - ) - - # Descriptive: when did the spike start? A real investigation begins on the - # timeline -- failures by build week -- before touching genealogy. The onset is - # the first week whose rate runs well above the opening week's baseline; it - # tells the rest of the chain which window's changes to interrogate. - wk = model.select( - Unit.build_week.alias("week"), Unit.defective.alias("defective") - ).to_df() - wk["defective"] = wk["defective"].astype(int) - timeline = wk.groupby("week").agg( - units=("defective", "size"), defects=("defective", "sum") - ) - timeline["rate"] = timeline["defects"] / timeline["units"] - opening = timeline["rate"].iloc[0] - onset = next((int(w) for w, r in timeline["rate"].items() if r > 2 * opening), None) - print("\nWhen did the spike start? Final-test failure rate by build week:") - for week, r in timeline.iterrows(): - mark = " <- spike onset" if onset == int(week) else "" - print( - f" week {int(week)}: {int(r['units']):>4} units {r['rate']:>5.1%}{mark}" - ) - print( - f"Spike onset at week {onset}: the chain interrogates what entered the line then." - ) - - n_nodes = int(model.select(num_nodes.alias("n")).to_df()["n"].iloc[0]) - n_edges = int(model.select(num_edges.alias("n")).to_df()["n"].iloc[0]) - print(f"\nGenealogy graph: {n_nodes} lot nodes, {n_edges} parent->child edges") - - # Naive view: rank candidate factors by the raw count of defective units they - # touch. High-volume factors dominate -- a near-universal housing lot and the - # busiest placement line top the list purely because they touch almost - # everything. That is exactly the trap the contrast stage corrects. Genealogy - # closure puts lots and machines on the same footing: a deep lot is "touched by" - # every finished unit that carries it. - incidence = ( - model.where((f := Factor.ref()).touches(u := Unit.ref())) - .select( - f.id.alias("factor"), f.kind.alias("kind"), u.defective.alias("defective") - ) - .to_df() - ) - incidence["defective"] = incidence["defective"].astype(int) - top_raw = ( - incidence.groupby(["factor", "kind"]) - .agg(units=("defective", "size"), defects=("defective", "sum")) - .reset_index() - .sort_values("defects", ascending=False) - .head(6) - .reset_index(drop=True) - ) - print( - f"\nBackward reachability built {len(incidence):,} (factor, unit) incidence facts." - ) - print( - "Top candidate factors by RAW defective units touched (pre-contrast -- note the high-volume bias):" - ) - print(top_raw.to_string(index=False)) - - suspects = ( - model.where(Factor.is_suspect(f := Factor.ref())) - .select( - f.id.alias("factor"), - f.kind.alias("kind"), - f.touched_count.alias("units"), - f.defect_count.alias("defects"), - f.defect_rate.alias("rate"), - f.lift.alias("lift"), - ) - .to_df() - .sort_values("lift", ascending=False) - .reset_index(drop=True) - ) - print( - f"\nContrast scoring: {len(suspects)} suspect factors (baseline failure rate {BASELINE_RATE:.2%}, " - f"lift>={LIFT_THRESHOLD}, support>={MIN_SUPPORT}, defects>={MIN_DEFECTS})" - ) - print(suspects.to_string(index=False)) - - # Formulation audit: variables and objective registered, and -- the easy thing to - # get wrong -- one coverage constraint grounded per defective unit. - n_vars = model.select(num_vars).to_df().iloc[:, 0].tolist() - n_cons = model.select(num_cons).to_df().iloc[:, 0].tolist() - n_objs = model.select(num_min_objs).to_df().iloc[:, 0].tolist() - assert n_vars[0] > 0, f"no variables: {n_vars}" - assert n_cons[0] > 0, f"no constraints: {n_cons}" - assert n_objs == [1], f"expected one minimization objective, got {n_objs}" - n_cover = len(model.select(cover).to_df()) - assert n_cover == len(n_def), ( - f"coverage grounded {n_cover} rows, expected {len(n_def)}" - ) - print( - f"\nFormulation audit: {n_cover} coverage constraints grounded (one per defective unit)." - ) - - # Status and objective of the latest refresh's solve. - status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() - cost = model.select(DiagnosisCost).to_df().iloc[:, 0].tolist() - print( - f"\nDiagnosis solve: {', '.join(status)} objective={round(cost[0], 9) if cost else None}" - ) - - # Read back the diagnosis (populate=True writes the solution onto is_root_cause). - diagnosis = ( - model.where((f := Factor.ref()).is_root_cause > 0.5) - .select( - f.id.alias("factor"), - f.label.alias("label"), - f.kind.alias("kind"), - f.defect_count.alias("defects_on_factor"), - f.lift.alias("lift"), - ) - .to_df() - .sort_values("lift", ascending=False) - .reset_index(drop=True) - ) - - # The headline, read back from the DiagnosisResult singleton. - headline = model.select( - DiagnosisResult.n_defective.alias("n_defective"), - DiagnosisResult.n_explained.alias("n_explained"), - DiagnosisResult.n_root_causes.alias("n_root_causes"), - DiagnosisResult.coverage.alias("coverage"), - ).to_df() - n_defective = int(headline["n_defective"].iloc[0]) - n_explained = int(headline["n_explained"].iloc[0]) - print( - f"\nRoot-cause diagnosis: {int(headline['n_root_causes'].iloc[0])} factors explain {n_explained}/{n_defective} " - f"defective units ({headline['coverage'].iloc[0]:.0%} coverage)" - ) - print(diagnosis.to_string(index=False)) - - # Corroborating evidence per named cause, the way a real RCA write-up reads: - # the defect signature it concentrates (a paste fault should present as cold - # solder, a reflow fault as solder bridging), plus a kind-specific tell -- - # supplier and receipt date for a lot, calibration age for a machine. All of it - # is read back from ontology already loaded; the diagnosis is the hypothesis, - # this is the supporting evidence an engineer would confirm physically. - sig = ( - model.where( - (f := Factor.ref()).is_root_cause > 0.5, - f.touches(u := Unit.ref()), - u.is_defective(), - ) - .select(f.id.alias("factor"), u.defect_type.alias("defect_type")) - .to_df() - ) - print("\nCorroborating evidence per root cause:") - for _, row in diagnosis.iterrows(): - fid, kind, label = row["factor"], row["kind"], row["label"] - types = sig[sig["factor"] == fid]["defect_type"] - dom = types.value_counts() - signature = f"{dom.iloc[0] / len(types):.0%} {dom.index[0]}" - ref = fid.split("::", 1)[1] - if kind == "LOT": - tell_df = ( - model.where( - (lo := Lot.ref()).id == ref, lo.supplier(sp := Supplier.ref()) - ) - .select(sp.name.alias("supplier"), lo.received_date.alias("recv")) - .to_df() - ) - tell = f"supplier {tell_df['supplier'].iloc[0]}, received {tell_df['recv'].iloc[0]}" - else: - tell_df = ( - model.where((mm := Machine.ref()).id == ref) - .select(mm.calibration_age_days.alias("age")) - .to_df() - ) - tell = f"{int(tell_df['age'].iloc[0])} days since last calibration" - print(f" {label:<26} {signature:<18} {tell}") diff --git a/defect_root_cause/queries/root_cause_report.py b/defect_root_cause/queries/root_cause_report.py new file mode 100644 index 00000000..76e3e621 --- /dev/null +++ b/defect_root_cause/queries/root_cause_report.py @@ -0,0 +1,215 @@ +"""Root-cause report: reads the deployed defect-diagnosis model. + +Run after `rai models deploy --path model`: + python -m queries.root_cause_report + +Prints per-stage diagnostics -- genealogy reachability validation, the +contrast-scored suspect factors with lift, the solve's termination status, and the +DiagnosisResult headline (defective units, units explained, coverage) with the +ranked root-cause factors, each with corroborating evidence. +""" + +from model import ( + LIFT_THRESHOLD, + MIN_DEFECTS, + MIN_SUPPORT, + DiagnosisCost, + DiagnosisResult, + Factor, + Lot, + Machine, + PlantBaseline, + SolveStatus, + Supplier, + Unit, + cover, + model, + num_cons, + num_edges, + num_min_objs, + num_nodes, + num_vars, +) + +if __name__ == "__main__": + n_units = model.select(Unit.id).to_df() + n_def = model.where(Unit.is_defective()).select(Unit.id).to_df() + print( + f"Loaded {len(n_units)} units, {len(n_def)} defective ({len(n_def) / len(n_units):.2%} final-test failure rate)" + ) + + # Descriptive: when did the spike start? A real investigation begins on the + # timeline -- failures by build week -- before touching genealogy. The onset is + # the first week whose rate runs well above the opening week's baseline; it + # tells the rest of the chain which window's changes to interrogate. + wk = model.select( + Unit.build_week.alias("week"), Unit.defective.alias("defective") + ).to_df() + wk["defective"] = wk["defective"].astype(int) + timeline = wk.groupby("week").agg( + units=("defective", "size"), defects=("defective", "sum") + ) + timeline["rate"] = timeline["defects"] / timeline["units"] + opening = timeline["rate"].iloc[0] + onset = next((int(w) for w, r in timeline["rate"].items() if r > 2 * opening), None) + print("\nWhen did the spike start? Final-test failure rate by build week:") + for week, r in timeline.iterrows(): + mark = " <- spike onset" if onset == int(week) else "" + print( + f" week {int(week)}: {int(r['units']):>4} units {r['rate']:>5.1%}{mark}" + ) + print( + f"Spike onset at week {onset}: the chain interrogates what entered the line then." + ) + + n_nodes = int(model.select(num_nodes.alias("n")).to_df()["n"].iloc[0]) + n_edges = int(model.select(num_edges.alias("n")).to_df()["n"].iloc[0]) + print(f"\nGenealogy graph: {n_nodes} lot nodes, {n_edges} parent->child edges") + + # Naive view: rank candidate factors by the raw count of defective units they + # touch. High-volume factors dominate -- a near-universal housing lot and the + # busiest placement line top the list purely because they touch almost + # everything. That is exactly the trap the contrast stage corrects. Genealogy + # closure puts lots and machines on the same footing: a deep lot is "touched by" + # every finished unit that carries it. + incidence = ( + model.where((f := Factor.ref()).touches(u := Unit.ref())) + .select( + f.id.alias("factor"), f.kind.alias("kind"), u.defective.alias("defective") + ) + .to_df() + ) + incidence["defective"] = incidence["defective"].astype(int) + top_raw = ( + incidence.groupby(["factor", "kind"]) + .agg(units=("defective", "size"), defects=("defective", "sum")) + .reset_index() + .sort_values("defects", ascending=False) + .head(6) + .reset_index(drop=True) + ) + print( + f"\nBackward reachability built {len(incidence):,} (factor, unit) incidence facts." + ) + print( + "Top candidate factors by RAW defective units touched (pre-contrast -- note the high-volume bias):" + ) + print(top_raw.to_string(index=False)) + + suspects = ( + model.where(Factor.is_suspect(f := Factor.ref())) + .select( + f.id.alias("factor"), + f.kind.alias("kind"), + f.touched_count.alias("units"), + f.defect_count.alias("defects"), + f.defect_rate.alias("rate"), + f.lift.alias("lift"), + ) + .to_df() + .sort_values("lift", ascending=False) + .reset_index(drop=True) + ) + baseline_rate = float(model.select(PlantBaseline.failure_rate).to_df().iloc[0, 0]) + print( + f"\nContrast scoring: {len(suspects)} suspect factors (baseline failure rate {baseline_rate:.2%}, " + f"lift>={LIFT_THRESHOLD}, support>={MIN_SUPPORT}, defects>={MIN_DEFECTS})" + ) + print(suspects.to_string(index=False)) + + # Formulation audit: variables and objective registered, and -- the easy thing to + # get wrong -- one coverage constraint grounded per defective unit. + n_vars = model.select(num_vars).to_df().iloc[:, 0].tolist() + n_cons = model.select(num_cons).to_df().iloc[:, 0].tolist() + n_objs = model.select(num_min_objs).to_df().iloc[:, 0].tolist() + if not n_vars or n_vars[0] <= 0: + raise SystemExit(f"no variables: {n_vars}") + if not n_cons or n_cons[0] <= 0: + raise SystemExit(f"no constraints: {n_cons}") + if n_objs != [1]: + raise SystemExit(f"expected one minimization objective, got {n_objs}") + n_cover = len(model.select(cover).to_df()) + if n_cover != len(n_def): + raise SystemExit(f"coverage grounded {n_cover} rows, expected {len(n_def)}") + print( + f"\nFormulation audit: {n_cover} coverage constraints grounded (one per defective unit)." + ) + + # Status and objective of the latest refresh's solve. + status = model.select(SolveStatus).to_df().iloc[:, 0].tolist() + cost = model.select(DiagnosisCost).to_df().iloc[:, 0].tolist() + print( + f"\nDiagnosis solve: {', '.join(status)} objective={round(cost[0], 9) if cost else None}" + ) + if status != ["OPTIMAL"]: + raise SystemExit(f"No optimal diagnosis: termination status {status}") + + # Read back the diagnosis (populate=True writes the solution onto is_root_cause). + diagnosis = ( + model.where((f := Factor.ref()).is_root_cause > 0.5) + .select( + f.id.alias("factor"), + f.label.alias("label"), + f.kind.alias("kind"), + f.defect_count.alias("defects_on_factor"), + f.lift.alias("lift"), + ) + .to_df() + .sort_values("lift", ascending=False) + .reset_index(drop=True) + ) + + # The headline, read back from the DiagnosisResult singleton. + headline = model.select( + DiagnosisResult.n_defective.alias("n_defective"), + DiagnosisResult.n_explained.alias("n_explained"), + DiagnosisResult.n_root_causes.alias("n_root_causes"), + DiagnosisResult.coverage.alias("coverage"), + ).to_df() + n_defective = int(headline["n_defective"].iloc[0]) + n_explained = int(headline["n_explained"].iloc[0]) + print( + f"\nRoot-cause diagnosis: {int(headline['n_root_causes'].iloc[0])} factors explain {n_explained}/{n_defective} " + f"defective units ({headline['coverage'].iloc[0]:.0%} coverage)" + ) + print(diagnosis.to_string(index=False)) + + # Corroborating evidence per named cause, the way a real RCA write-up reads: + # the defect signature it concentrates (a paste fault should present as cold + # solder, a reflow fault as solder bridging), plus a kind-specific tell -- + # supplier and receipt date for a lot, calibration age for a machine. All of it + # is read back from ontology already loaded; the diagnosis is the hypothesis, + # this is the supporting evidence an engineer would confirm physically. + sig = ( + model.where( + (f := Factor.ref()).is_root_cause > 0.5, + f.touches(u := Unit.ref()), + u.is_defective(), + ) + .select(f.id.alias("factor"), u.defect_type.alias("defect_type")) + .to_df() + ) + print("\nCorroborating evidence per root cause:") + for _, row in diagnosis.iterrows(): + fid, kind, label = row["factor"], row["kind"], row["label"] + types = sig[sig["factor"] == fid]["defect_type"] + dom = types.value_counts() + signature = f"{dom.iloc[0] / len(types):.0%} {dom.index[0]}" + ref = fid.split("::", 1)[1] + if kind == "LOT": + tell_df = ( + model.where( + (lo := Lot.ref()).id == ref, lo.supplier(sp := Supplier.ref()) + ) + .select(sp.name.alias("supplier"), lo.received_date.alias("recv")) + .to_df() + ) + tell = f"supplier {tell_df['supplier'].iloc[0]}, received {tell_df['recv'].iloc[0]}" + else: + tell_df = ( + model.where((mm := Machine.ref()).id == ref) + .select(mm.calibration_age_days.alias("age")) + .to_df() + ) + tell = f"{int(tell_df['age'].iloc[0])} days since last calibration" + print(f" {label:<26} {signature:<18} {tell}") diff --git a/defect_root_cause/runbook.md b/defect_root_cause/runbook.md index fe316a58..5b96aa67 100644 --- a/defect_root_cause/runbook.md +++ b/defect_root_cause/runbook.md @@ -144,4 +144,4 @@ Ontology gains a singleton `DiagnosisResult` with `n_defective` (142), `n_explai ## Data -Bundled CSVs in `data/`: 20 SKUs across four tiers, 26 bill-of-materials edges, 8 suppliers, 10 machines, 105 lots, 162 lot-genealogy edges, 2,500 units (142 defective) stamped with build week, 10,000 unit-to-lot consumption rows, 10,000 unit-process rows, and 118 candidate factors. `generate_data.py` regenerates them deterministically from a seed, with two planted root causes that share a week-2 incident onset (contaminated paste lot `SP-0423`, received at the onset; reflow oven `REF-02`, drifting at the onset) and several decoys (a near-universal housing lot, a high-volume placement line, the day shift). Contaminated boards reach only units built on or after the onset, so the timeline is a real signal. All stages run end-to-end via `defect_root_cause.py`. +Bundled CSVs in `data/`: 20 SKUs across four tiers, 26 bill-of-materials edges, 8 suppliers, 10 machines, 105 lots, 162 lot-genealogy edges, 2,500 units (142 defective) stamped with build week, 10,000 unit-to-lot consumption rows, 10,000 unit-process rows, and 118 candidate factors. `generate_data.py` regenerates them deterministically from a seed, with two planted root causes that share a week-2 incident onset (contaminated paste lot `SP-0423`, received at the onset; reflow oven `REF-02`, drifting at the onset) and several decoys (a near-universal housing lot, a high-volume placement line, the day shift). Contaminated boards reach only units built on or after the onset, so the timeline is a real signal. All stages deploy from `model/__init__.py` and report through `queries/root_cause_report.py`. From 5502f2e1270743ef3e9b930332c94c09b76e3f1f Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:46:01 -0800 Subject: [PATCH 071/162] sample-template: split the starter into a deployed model and a query The starter keeps model/schema.py and model/source.py, and its runner moves to queries/item_report.py, run with rai models deploy --path model and python -m queries.item_report. The sidecar sets page.deployment and points the last walkthrough step at the query module, and the README Quickstart follows the deployed-template steps so new templates start from that shape. --- sample-template/README.md | 104 +++++++++++++++++-------- sample-template/model/__init__.py | 5 +- sample-template/queries/item_report.py | 23 ++++++ sample-template/template-docs.yaml | 12 +-- sample-template/template.py | 34 -------- 5 files changed, 104 insertions(+), 74 deletions(-) create mode 100644 sample-template/queries/item_report.py delete mode 100644 sample-template/template.py diff --git a/sample-template/README.md b/sample-template/README.md index 7bdfb22e..4856a375 100644 --- a/sample-template/README.md +++ b/sample-template/README.md @@ -24,13 +24,15 @@ layout without prescribing a business domain. - Contributors creating a new template. - Authors who need a consistent boundary between schema, source loading, and - runner logic. + queries. ## What you'll build - A stable schema in `model/schema.py`. - Source reads and base-fact mappings in `model/source.py`. -- A runner that owns reasoning, solves, queries, and reporting. +- Reasoning and solves declared at module level in the model package, so one + deploy publishes them. +- A query module that reads the deployed model and prints the report. - A `template-docs.yaml` sidecar for the generated documentation experience. ## What's included @@ -39,8 +41,8 @@ layout without prescribing a business domain. and relationships. - **Sources**: `model/source.py` reads or constructs source data and maps base facts into the shared model. -- **Runner**: `template.py` imports the model package and owns executable - reasoning and reporting. +- **Query**: `queries/item_report.py` imports the model package, reads the + deployed model, and prints the report. - **Documentation metadata**: `template-docs.yaml` selects the model subset, walkthrough, guide links, and optional sample tables for the docs site. @@ -53,29 +55,42 @@ layout without prescribing a business domain. ## Quickstart -1. Copy this directory to a new root-level template directory and rename it: +Before you start, install Python 3.10 or later and get a Snowflake role that can deploy models, as described under Prerequisites. This starter has no prescriptive solve, so it needs no setup beyond that role. + +Run these steps from a terminal: + +1. **Download the template** + + [Download the ZIP](https://docs.relational.ai/templates/zips/v1/sample-template.zip). Unzip it and enter the template directory: ```bash - cp -R sample-template your_template_name - cd your_template_name + unzip sample-template.zip + cd sample-template ``` -1. Update the package metadata, runner name, source files, README, and - `template-docs.yaml`. +2. **Create a Python environment** -1. Create an environment and install the template: + Create and activate a virtual environment, then update pip: ```bash python -m venv .venv source .venv/bin/activate + python -m pip install --upgrade pip + ``` + +3. **Install the template** + + Install the template's dependencies, including the `rai` CLI, in the active environment: + + ```bash python -m pip install . ``` -1. Configure the Snowflake connection and the deployment: run `rai init` to - create `raiconfig.yaml` with your Snowflake connection, set the connection - `role` to the model role, then add a `deployment` section. Give each template - its own schema pair so two deployed templates never share one; the deploy - creates both schemas if they don't exist. +4. **Configure your project** + + Use the configuration builder in [Install and run PyRel](https://docs.relational.ai/get-started/install-and-run-pyrel/#configure-your-project) to create `raiconfig.yaml` in the template directory, with the model role as the connection `role`. + + Then add a `deployment` section. Give each template its own schema pair so two deployed templates never share one; the deploy creates both schemas if they don't exist: ```yaml deployment: @@ -89,27 +104,43 @@ layout without prescribing a business domain. schedule: standard ``` - Without the `deployment` section, `python template.py` stops with an error - naming the missing setting. +5. **Deploy the model** + + Deploy the model in `model/` to the schemas in `raiconfig.yaml`: + + ```bash + rai models deploy --path model + ``` + + The deploy waits for the model's first refresh. Without the `deployment` section, it stops with an error naming the missing setting. + + Deploying again without changing the model applies no updates. A change to the model redeploys it. + +6. **Query the results** -1. Run the renamed entrypoint: + Print the report from the deployed model: ```bash - python template.py + python -m queries.item_report ``` - The script deploys the model, waits for its first refresh, and prints the - report from the deployed model. `rai models deploy` deploys without printing - the report. + ```text + id name + 0 1 Example item + ``` + + The example prints the item loaded by `model/source.py`. Your template should document a similarly small success signal. + +7. **Tear down the deployment** - Running the script again without changing the model redeploys nothing; it - prints the item from the deployed model. A change to the model redeploys it. + Preview what the teardown removes, then remove it: - To remove the deployment, run `rai models teardown --force` from - this directory; it keeps the two schemas. + ```bash + rai models teardown + rai models teardown --force + ``` -The example prints the item loaded by `model/source.py`. Your template should -document a similarly small success signal. + The teardown removes the deployed objects and keeps the two schemas. ## Sample data @@ -129,21 +160,26 @@ do not duplicate them in `template-docs.yaml`. ## How it works `model/schema.py` creates the model and stable vocabulary. Importing `model` -loads the single source in `model/source.py`. The runner then deploys the model, -waits for its first refresh, and selects the modeled records from the deployed -model and prints them. +loads the single source in `model/source.py`. `rai models deploy --path model` +deploys the model and waits for its first refresh. The query module then selects +the modeled records from the deployed model and prints them. ```text -source records -> model/source.py -> model/schema.py -> runner -> output +source records -> model/source.py -> model/schema.py -> deploy -> queries/item_report.py -> output ``` ## Customize this template +1. Copy this directory to a new root-level template directory and rename it + (`cp -R sample-template your_template_name`). Update the package metadata, + the name of the query module, the Quickstart download link and directory + name, the README, and `template-docs.yaml`. 1. Replace `Item` with the concepts and relationships for the business problem. 1. Replace the in-memory record with local or remote source mappings. -1. Declare reasoning and solves at the top level of the runner, before the - deploy, and put queries and reporting after `model.deploy()`. +1. Declare reasoning and solves at the top level of the `model/` package, which + the deploy publishes, and put queries and reporting in modules under + `queries/` that only read the deployed model. 1. Update every sidecar symbol, source path, walkthrough focus, and sample column after changing the code. diff --git a/sample-template/model/__init__.py b/sample-template/model/__init__.py index 0c3b2e65..04366e12 100644 --- a/sample-template/model/__init__.py +++ b/sample-template/model/__init__.py @@ -1,4 +1,7 @@ -"""Shared model exports for the starter template.""" +"""Shared model exports for the starter template. + +`rai models deploy --path model` deploys everything this package declares. +""" from .schema import Item, model from .source import load_sample_data diff --git a/sample-template/queries/item_report.py b/sample-template/queries/item_report.py new file mode 100644 index 00000000..62815393 --- /dev/null +++ b/sample-template/queries/item_report.py @@ -0,0 +1,23 @@ +"""Item report: reads the deployed starter model. + +Run after `rai models deploy --path model`: + python -m queries.item_report + +Prints a small table containing the bundled example item. +""" + +from model import Item, model + + +def inspect_items(): + """Select the items loaded by the model package.""" + item = Item.ref() + return model.select(item.id, item.name) + + +def main() -> None: + print(inspect_items().to_df()) + + +if __name__ == "__main__": + main() diff --git a/sample-template/template-docs.yaml b/sample-template/template-docs.yaml index 87fea905..cd0d78b4 100644 --- a/sample-template/template-docs.yaml +++ b/sample-template/template-docs.yaml @@ -7,12 +7,14 @@ page: omit_from_readme: - model-overview - how-it-works + download_and_run_from_readme: true + deployment: true model_explorer: title: How the starter model fits together description: >- The starter separates stable schema declarations from source mappings and - keeps executable queries and reporting in the runner. + keeps executable queries and reporting in a query module. concepts: - symbol: Item description: A generic business record to replace with a domain concept. @@ -23,7 +25,7 @@ guided_walkthrough: title: Follow the starter flow description: >- These three steps show where a template declares its model, loads base - facts, and inspects results. + facts, and queries the deployed model. steps: - id: define-schema title: Define the stable schema @@ -59,10 +61,10 @@ guided_walkthrough: - id: inspect-results title: Inspect modeled records explanation: >- - inspect_items queries the model and returns the records printed by the - runner. + inspect_items queries the deployed model and returns the records printed + by the query module. focus: - file: template.py + file: queries/item_report.py symbols: - inspect_items references: diff --git a/sample-template/template.py b/sample-template/template.py deleted file mode 100644 index 4e2a2764..00000000 --- a/sample-template/template.py +++ /dev/null @@ -1,34 +0,0 @@ -"""Starter RelationalAI template. - -This script demonstrates the standard template package layout: - -- Declare stable concepts in model/schema.py. -- Load base facts in model/source.py. -- Keep queries, reasoning, solves, and reporting in the runner. - -Run: - `python template.py` deploys the model to the schema in raiconfig.yaml, waits - for its first refresh, and prints the item from the deployed model. - -Output: - A small table containing the bundled example item. -""" - -from model import Item, model - - -def inspect_items(): - """Select the items loaded by the model package.""" - item = Item.ref() - return model.select(item.id, item.name) - - -def main() -> None: - # Deploy and wait for the first refresh. - model.deploy() - - print(inspect_items().to_df()) - - -if __name__ == "__main__": - main() From fc73649f0556f3da59fa0239efbaed39939edae6 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:52:11 -0800 Subject: [PATCH 072/162] commercial_underwriting: keep the tier checks as warnings The report printed a warning for an unclassified or doubly classified entity before the move; it does so again instead of stopping. --- commercial_underwriting/README.md | 6 +++--- commercial_underwriting/queries/underwriting_report.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/commercial_underwriting/README.md b/commercial_underwriting/README.md index 69d840e1..84799680 100644 --- a/commercial_underwriting/README.md +++ b/commercial_underwriting/README.md @@ -170,7 +170,7 @@ Run these steps from a terminal: STANDARD 1 18000.0 18000.000000 ``` - Each entity falls into exactly one tier, because the four tier concepts partition the insured entities, and the report stops if any entity lands in no tier or in two. The premium-by-tier rollup is the underwriting business view: about $150K of bound premium is associated with submissions that today's rules would decline; about $236K is non-standard and could be repriced. + Each entity falls into exactly one tier, because the four tier concepts partition the insured entities, and the report prints a warning if any entity lands in no tier or in two. The premium-by-tier rollup is the underwriting business view: about $150K of bound premium is associated with submissions that today's rules would decline; about $236K is non-standard and could be repriced. 7. **Tear down the deployment** @@ -289,13 +289,13 @@ All thresholds are at the top of `model/__init__.py` under `# Configure inputs`:
An entity isn't classified into any tier. - The report verifies mutual exclusivity and stops with an error if any entity is unclassified. The most likely cause is a rule condition that doesn't match the data — verify with `model.select(InsuredEntity.id).where(InsuredEntity.has_decline_factor()).to_df()` and the analogous `has_marginal_factor` query that those flags fire as expected. + The report verifies mutual exclusivity and prints a warning if any entity is unclassified. The most likely cause is a rule condition that doesn't match the data — verify with `model.select(InsuredEntity.id).where(InsuredEntity.has_decline_factor()).to_df()` and the analogous `has_marginal_factor` query that those flags fire as expected.
Two tiers fire for the same entity. - This indicates the tier conditions are not mutually exclusive. The report stops with an error if it finds any overlap. Check the `model.not_()` clauses on the lower-priority tier rules (NonStandard / Preferred / Standard) — each must explicitly exclude the higher-priority tier's flag. + This indicates the tier conditions are not mutually exclusive. The report prints a warning if it finds any overlap. Check the `model.not_()` clauses on the lower-priority tier rules (NonStandard / Preferred / Standard) — each must explicitly exclude the higher-priority tier's flag.
diff --git a/commercial_underwriting/queries/underwriting_report.py b/commercial_underwriting/queries/underwriting_report.py index 4d056b39..126679b5 100644 --- a/commercial_underwriting/queries/underwriting_report.py +++ b/commercial_underwriting/queries/underwriting_report.py @@ -149,13 +149,13 @@ def classify(entity_id): unclassified = tier_df[tier_df["tier"] == "UNCLASSIFIED"] if not unclassified.empty: - raise SystemExit(f"{len(unclassified)} entities are unclassified: the rule set is non-exhaustive.") + print(f"\nWARNING: {len(unclassified)} entities are unclassified — rule set is non-exhaustive.") # Verify mutual exclusivity: no entity in two tiers. overlap = (decline_ids & nonstandard_ids) | (decline_ids & standard_ids) | (decline_ids & preferred_ids) \ | (nonstandard_ids & standard_ids) | (nonstandard_ids & preferred_ids) | (standard_ids & preferred_ids) if overlap: - raise SystemExit(f"{len(overlap)} entities classified into multiple tiers: {sorted(overlap)}") + print(f"\nWARNING: {len(overlap)} entities classified into multiple tiers: {sorted(overlap)}") # Aggregate premium by tier — the underwriting business view. premium_df = model.where(Policy.insured_entity(InsuredEntity)).select( From 77cf5698fbd221f49bd01164936b69b6235a0bb3 Mon Sep 17 00:00:00 2001 From: Chris Coey Date: Fri, 13 Nov 2026 18:52:21 -0800 Subject: [PATCH 073/162] Describe the model package and queries layout in the contributor docs The starter template now deploys its model package and reads it from queries/ modules, so the root README, CONTRIBUTING, the authoring skill and the prompts describe that layout instead of a single runner script, and the sidecar reference documents page.deployment. --- .agents/skills/template-authoring/SKILL.md | 18 +++++++++------ .../resources/template-docs.md | 9 +++++++- .../prompts/cleanup-template-code.prompt.md | 3 ++- .../prompts/create-template-readme.prompt.md | 2 +- .github/prompts/review-template.prompt.md | 2 +- CONTRIBUTING.md | 22 +++++++++++-------- README.md | 6 ++--- sample-template/template-docs.yaml | 2 +- 8 files changed, 40 insertions(+), 24 deletions(-) diff --git a/.agents/skills/template-authoring/SKILL.md b/.agents/skills/template-authoring/SKILL.md index 2ef97b70..495ca2da 100644 --- a/.agents/skills/template-authoring/SKILL.md +++ b/.agents/skills/template-authoring/SKILL.md @@ -24,7 +24,7 @@ removed while the template remains. ## Required workflow -1. Read the entire template directory, including its runner, `pyproject.toml`, +1. Read the entire template directory, including its queries, `pyproject.toml`, README, runbook, data files, and existing sidecar. 1. Preserve the entrypoint, solver choices, constraints, thresholds, output labels, and data semantics unless the request explicitly changes them. @@ -38,7 +38,8 @@ removed while the template remains. │ ├── __init__.py │ ├── schema.py │ └── source.py - ├── .py + ├── queries/ + │ └── .py ├── runbook.md ├── pyproject.toml └── data/ @@ -49,12 +50,15 @@ removed while the template remains. relationships with direct `model.Concept`, `model.Property`, and `model.Relationship` calls. 1. Put source reads, normalization, and base-fact mappings in - `model/source.py`. Keep solver declarations, derived reasoning, result - queries, and reporting in the runner. + `model/source.py`. `rai models deploy --path model` deploys what the + package declares, so put solver declarations and derived reasoning in the + package too (for example `model/reasoning.py`). Keep result queries and + reporting in `queries/.py`, which imports from `model`, declares + nothing, and runs as `python -m queries.`. 1. Re-export the model and stable schema symbols from `model/__init__.py`. A single-source template may load its source on import. A multi-source - template must expose explicit loader functions and let each runner choose - the correct one. + template must expose explicit loader functions and let each entry point + choose the correct one. 1. Keep README commands, code links, `runbook.md`, and `template-docs.yaml` aligned with the final paths and symbols. 1. Follow [the sidecar contract](./resources/template-docs.md) and select only @@ -73,7 +77,7 @@ provides the model explorer and walkthrough. - Explain the problem before implementation details. - For a long-form README, put the solution flow in **What you'll build** and - use **What's included** to identify the schema, source, runner, runbook, + use **What's included** to identify the schema, source, queries, runbook, sample data, and outputs. - When the sidecar sets `page.download_and_run_from_readme: true`, a README may contain only **What this template is for** and **Quickstart**, plus an diff --git a/.agents/skills/template-authoring/resources/template-docs.md b/.agents/skills/template-authoring/resources/template-docs.md index 400b6cc4..c0b949a8 100644 --- a/.agents/skills/template-authoring/resources/template-docs.md +++ b/.agents/skills/template-authoring/resources/template-docs.md @@ -49,6 +49,13 @@ sample_data: install, configure with the Start building with PyRel configuration builder, and run. Keep any representative output directly after its runner command. Templates without this flag retain their existing rendering. +- `deployment: true` (optional, requires `download_and_run_from_readme: true`) + marks a template that deploys its model. The Quickstart then has seven ordered + steps: download, Python environment, install, configure, deploy (one `bash` + block containing exactly `rai models deploy --path model`), query (one `bash` + block per report containing exactly `python -m queries.`, each optionally + followed by a `text` output block), and tear down (`rai models teardown`, then + `rai models teardown --force`). - `omit_readme_preamble: true` (optional) excludes the text before the first README `##` heading from the enhanced HTML and Markdown pages. Use it when the README opens with links to its own generated model explorer and @@ -149,7 +156,7 @@ guide links: ```yaml focus: - file: runner.py + file: queries/report.py anchors: - start: "# Stage 3: Optimize" end: "# Stage 4: Report" diff --git a/.github/prompts/cleanup-template-code.prompt.md b/.github/prompts/cleanup-template-code.prompt.md index c773fc7b..cf2eb4d7 100644 --- a/.github/prompts/cleanup-template-code.prompt.md +++ b/.github/prompts/cleanup-template-code.prompt.md @@ -20,7 +20,8 @@ dependencies. For a model-package refactor: - Move model creation and stable declarations to `model/schema.py`. - Move reads, normalization, and base-fact mappings to `model/source.py`. - Re-export stable symbols through `model/__init__.py`. -- Keep derived reasoning, solves, queries, and reporting in the runner. +- Keep derived reasoning and solver declarations in the model package, and + result queries and reporting in `queries/.py`. - Update README, runbook, and any existing `template-docs.yaml` references. - Do not force an existing README-only template to adopt the sidecar unless the request explicitly includes migration. diff --git a/.github/prompts/create-template-readme.prompt.md b/.github/prompts/create-template-readme.prompt.md index 2a93d913..2c704683 100644 --- a/.github/prompts/create-template-readme.prompt.md +++ b/.github/prompts/create-template-readme.prompt.md @@ -20,7 +20,7 @@ for a new template, a strict `template-docs.yaml` in the same change. - Keep the README complete and runnable from a fresh environment. - Begin with the business problem and outcome. - For a long-form README, keep the solution stages in **What you'll build**. - Name the model schema, source mapping, runner, runbook, sample data, and + Name the model schema, source mapping, queries, runbook, sample data, and outputs in **What's included**. - For a short enhanced README, use **What this template is for** and **Quickstart** with an opening link to its docs-page model and walkthrough. diff --git a/.github/prompts/review-template.prompt.md b/.github/prompts/review-template.prompt.md index 9cd8d471..1c7f5b36 100644 --- a/.github/prompts/review-template.prompt.md +++ b/.github/prompts/review-template.prompt.md @@ -18,7 +18,7 @@ Check: - Required files, entrypoint, pinned dependencies, and complete sample data. - README commands, paths, headings, expected output, and troubleshooting. -- `model/schema.py`, `model/source.py`, `model/__init__.py`, and runner +- `model/schema.py`, `model/source.py`, `model/__init__.py`, and `queries/` responsibility boundaries when the template uses the standard layout. - `runbook.md` consistency with the final code paths. - Every existing `template-docs.yaml` key, symbol, source, reference, anchor, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 341fc7e3..624e17f9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -53,8 +53,8 @@ This repository uses Ruff to lint Python template code. ## Add a new template 1. Copy `sample-template/` to a new root-level directory named for your template. -2. Rename the main runner, package metadata, and starter identifiers so they match the folder name. -3. Implement the schema, source mappings, runner logic, sample data, and outputs. +2. Rename the query module, package metadata, and starter identifiers so they match the folder name. +3. Implement the schema, source mappings, reasoning, queries, sample data, and outputs. 4. Replace the starter README content and update every field in `template-docs.yaml`. 5. Run focused lint, compilation, policy, and index checks. 6. Run the template from a fresh environment and verify that the README Quickstart works as written. @@ -69,15 +69,19 @@ At a minimum, make sure your template includes: - `model/schema.py` for model creation and stable declarations - `model/source.py` for source reads, normalization, and base-fact mappings - `model/__init__.py` for stable exports -- a runnable entrypoint such as `.py` or a notebook +- `queries/.py` modules that read the deployed model, run as `python -m queries.` - `runbook.md` that uses the same paths, symbols, and commands - `data/` when the template reads local sample files - sample data and output paths that match both the code and the README -Keep solver declarations, derived reasoning, result queries, and reporting in -the runner. A template with one source mode may load it when `model` is -imported. A template with multiple source modes must expose explicit loader -functions and let each runner choose one. +`rai models deploy --path model` deploys what the `model` package declares, +so solver declarations and derived reasoning belong in the package too (for +example `model/reasoning.py`). Keep result queries and reporting in +`queries/.py`: each module imports from `model`, declares nothing, and +runs its report under `if __name__ == "__main__":`. A template with one +source mode may load it when `model` is imported. A template with multiple +source modes must expose explicit loader functions and let each entry point +choose one. ## Generated documentation sidecars @@ -129,7 +133,7 @@ commands, expected results, and any required data attribution in Quickstart. The README and generated run steps must agree. For a longer README, use **What's included** to name the schema, source, -runner, runbook, sample data, and outputs. Use small Markdown tables for +queries, runbook, sample data, and outputs. Use small Markdown tables for representative sample rows where helpful. > [!TIP] @@ -186,7 +190,7 @@ Before opening a PR, make sure you can complete this checklist from a clean envi ruff check */ ``` -1. Compile the changed runner and model package: +1. Compile the changed model package and queries: ```bash python -m compileall -q sample-template diff --git a/README.md b/README.md index 2bd9d025..fb7463c4 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Each template lives in its own root-level folder. Within a folder, you will usua - `README.md` with the problem statement, prerequisites, and run instructions - `pyproject.toml` for template-local dependencies -- a main runner such as `