Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions docs/qa/scenarios/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,11 @@ question's computed truth; a card that does not load is red in CI.
| 5 | `run.operations`, `run.guardrails`, `run.agentActions?` | At least two writes, each acting as a persona and declaring its `outcome`; the five guardrail checks; optionally the agent-callable business actions. |
| 6 | `iterate` | A one-sentence change request plus the `expectDelta` it should add to `expect`. |

Besides the layers a card carries `id` (its basename), `title`, `provenance` (where the
Besides the layers a card carries `id` (its basename), `code`, `title`, `provenance` (where the
scenario comes from) and, when a question uses a time window, `asOf`: the fixed "today"
(`YYYY-MM-DD`) windows are computed against.
(`YYYY-MM-DD`) windows are computed against. `code` is the app code: 2–8 lowercase letters,
unique across the library. The app lives in the namespace `demo` + code (`democrm`) wherever
it is built, scored or exported, and every item it declares carries that prefix.

**Counts.** The default band is 8–10 questions and 4–6 operations. A card below 8 questions
or 4 operations states why in `run.countsReason`, or it is refused.
Expand Down Expand Up @@ -153,7 +155,7 @@ meet this floor:

## Adding a card

1. Write `<id>.json` here; `id` equals the basename.
1. Write `<id>.json` here; `id` equals the basename, and `code` is one no other card uses.
2. Keep it generic — a scenario anyone could write from the domain alone.
3. Run `pnpm --filter @objectstack/scenarios test`: a refusal names the file and the field
path. Then pin the card's hand-worked truths, per persona too, in its own
Expand All @@ -164,5 +166,5 @@ meet this floor:

`pnpm --filter @objectstack/scenarios run-scenarios` builds, runs and iterates every card here with a local
Claude Code session, and writes one run record per card. Grading uses `@objectstack/scenarios/score`; no
API-billed model is used. The runner reads a card's object names relative to the built package's namespace
(`account` → `<namespace>_account`). Flags and modes: [the package README](../../../packages/qa/scenarios/README.md#running-the-cards--objectstackscenariosrunner).
API-billed model is used. The runner reads a card's names relative to the card's namespace
(`account` → `democrm_account`), and a build that names any item without the prefix fails its Build half. Flags and modes: [the package README](../../../packages/qa/scenarios/README.md#running-the-cards--objectstackscenariosrunner).
1 change: 1 addition & 0 deletions docs/qa/scenarios/asset_lifecycle.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"$comment": "Seed rule (so every truth can be worked by hand): asset i = 1..120 has category by i mod 4 (1 laptop 7200 over 36 months, 2 monitor 1800 over 60, 3 phone 4800 over 24, 0 chair 2400 over 120; straight line, no salvage, so 200 / 30 / 200 / 20 a month), department by ((i-1) mod 16) div 4 (Engineering, Sales, Operations, Finance), and was bought on 2026-06..09 day ((i-1) mod 30)+1 (i 1-30 June, 31-60 July, 61-90 August, 91-120 September). Assets 1-64 were issued on their purchase day as kits to e(4*dept + (i-1) div 16 + 1); 65-120 are the pool. Repairs: kit assets i mod 6 = 3 were reported 40 days after purchase and completed 6 days later at 10*i + 500 (phones) or 10*i + 300 (laptops); a009 and a021 broke again 20 days after the fix (900 and 400); kit assets i mod 11 = 8 are in repair since 2026-10-(i div 3), open when i is odd, in_progress when even. A repair returns the asset's open checkout as damaged; a completed repair re-issues it to the same holder. Loaners: pool laptop i = 65 + 4k (k 0-13) was lent to e(k+1) on 2026-10-(k+1), due 21 days later, and returned in good condition after 14 days when k mod 3 = 0. The seven scrap requests are listed in full; each one's approver is the manager of the asset's department, and after a043 was scrapped its holder e11 got pool phone a075 on 2026-09-20. Status: disposed (a completed scrap), pending_disposal (submitted or approved), in_repair (an open or in_progress repair), in_use (an open checkout, whose employee is the custodian), else in_stock. Depreciation: one entry per asset per month after the purchase month through October, posted on the month's last day, none for the disposal month or later; accumulated_depreciation sums an asset's entries and book_value = purchase_cost - accumulated_depreciation.",
"id": "asset_lifecycle",
"code": "asset",
"title": "Asset lifecycle: register, checkout and return, repairs, monthly depreciation and a scrap approval",
"provenance": "Mid-tier scenario 资产全生命周期 (asset lifecycle) of the open library, written for this card from the domain alone: generic names, equipment and amounts, no customer-derived content. The seed follows the arithmetic rule in `$comment`, so each truth can be worked out by hand.",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/crm_account_with_contacts.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"id": "crm_account_with_contacts",
"code": "crm",
"title": "CRM: accounts and independently-living contacts",
"provenance": "Golden case `crm_account_with_contacts` of packages/cli/src/lint/corpus.ts; `expect` derived mechanically from its fixture (every object; minFieldsPerObject = the smallest object's field count; one selectFields row per select field with minOptions = its option count; one lookups row per lookup or master_detail field).",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/invoice_with_line_items.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"id": "invoice_with_line_items",
"code": "invoice",
"title": "Invoicing: invoices with inline line items and a summed total",
"provenance": "Golden case `invoice_with_line_items` of packages/cli/src/lint/corpus.ts; `expect` derived mechanically from its fixture (every object; minFieldsPerObject = the smallest object's field count; one selectFields row per select field with minOptions = its option count; one lookups row per lookup or master_detail field).",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/leave_balances.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"$comment": "Seed rule (so every truth can be worked by hand): request i = 1..120 is filed by employee e((i-1) mod 12 + 1) (e01-e12; e13/e14 are HR and file none). Leave type by i mod 5: 1-3 annual, 4 sick, 0 personal. It starts on working day ((i-1) mod 30) mod 15 + 1 (Monday to Friday; public holidays are out of scope) of July (i 1-30), August (31-60), September (61-90) or October (91-120) 2026 and covers N consecutive working days, N by i mod 7 = 0..6: 5, 1, 2, 3, 1, 2, 4. One leave_day row per day, the last one a morning half day (0.5) when i mod 11 = 0; days is their sum. Submitted 28 days before the start (drafts are not submitted). Status: i 1-90 rejected when i mod 10 = 7, withdrawn when 3, else approved; i 91-120 by i mod 10: 1 and 3 draft, 2 and 4-6 pending, 7 rejected, 8-9 approved, 0 withdrawn. Route: step 1 goes to the requester's department manager (to e14 when the requester is that manager), decided the day after submission; a request of more than 3 days also has step 2 to e13, decided two days after submission. A rejected request is rejected at its last step, a withdrawn one has step 1 cancelled, and a pending one waits at step 1, or at step 2 when it is over 3 days and i mod 10 is 5 or 6. Balances are for leave year 2026/27 (from 1 July 2026): annual entitlement 5, 10 or 15 days for under 3, 3 to 9, or 10 and more full years of service on 1 July 2026, plus the employee's carry-over; sick leave 10 days. used_days sums the days of the approved requests on the balance and remaining_days = entitled + carried over - used. A sick request over 2 days carries a certificate, and its medical note by i mod 7 is: 0 and 4 surgery, 1 and 5 physiotherapy, 2 and 6 dental treatment, 3 post-operative rest.",
"id": "leave_balances",
"code": "leave",
"title": "Leave and balances: leave types, yearly balances, requests with day lines and a route set by the days",
"provenance": "Mid-tier scenario 请假与假期余额 (leave and balances) of the open library, written for this card from the domain alone: generic names, cities and dates, no customer-derived content. The seed follows the arithmetic rule in `$comment`, so each truth can be worked out by hand.",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/order_to_cash.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"$comment": "Seed rule (so every truth can be worked by hand), as of 2026-10-31: account a(k), k = 1..12, is owned by rep [e02, e03, e04, e06, e07, e08][(k-1) mod 6] (team east: e02-e04 under manager e01; team west: e06-e08 under e05) on net_45 terms when k mod 4 = 0, else net_30. Order i = 1..120 is for a((i-1) mod 12 + 1), owned by that account's owner, dated 2026-07..10 day ((i-1) mod 30)+1 (i 1-30 July, 31-60 August, 61-90 September, 91-120 October), with two lines: hardware p((i-1) mod 3 + 1) x (10 + 5(i mod 5)) at list price, 20% off when i mod 10 = 0, and service p(4 + i mod 3) x (1 + i mod 4) at list price. A line discount above 10% routes the order to the owner's manager (`approver`; `approved_on` the next day once approved). Status: i 1-90 cancelled when i mod 20 = 13, else fulfilled; i 91-120 by i mod 10: 1-3 fulfilled, 4, 5 and 9 confirmed, 6-7 draft, 8 cancelled, 0 pending_approval. A fulfilled order is invoiced 7 days after its order date as INV-(1000+i), due after the account's terms. Payments as of asOf: Harbor Foods (a05) has paid nothing; Ironwood Labs (a09) pays half on the due date; Cedar Logistics (a02) pays in full 20 days after the due date; every other account pays in full 3 days before it; method by k mod 3: 0 wire, 1 ach, 2 check. Activities: one touchpoint per account per month on day 30 - 2k by the owner (July meeting, August call, September email, October call), plus a collection_call by e09 on the day an invoice is 15 days past due when it is not fully paid by then.",
"id": "order_to_cash",
"code": "order",
"title": "Order to cash: accounts, an activity stream, sales orders with lines, a discount approval, invoices, payments and receivables aging",
"provenance": "Mid-tier scenario 订单到收款 (order to cash) of the open library, written for this card from general business knowledge alone: generic account names, cities, products and amounts, no customer-derived content. The seed follows the arithmetic rule in `$comment`, so each truth can be worked out by hand.",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/project_with_tasks.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"id": "project_with_tasks",
"code": "project",
"title": "Project management: projects owning inline-edited tasks with roll-ups",
"provenance": "Golden case `project_with_tasks` of packages/cli/src/lint/corpus.ts; `expect` derived mechanically from its fixture (every object; minFieldsPerObject = the smallest object's field count; one selectFields row per select field with minOptions = its option count; one lookups row per lookup or master_detail field).",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/requisition_to_receipt.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"$comment": "Seed rule (so every truth can be worked by hand): requisition i = 1..120 (PR-0001..PR-0120) is filed by employee e((i-1) mod 12 + 1) (e01-e04 Sales, e05-e08 Engineering, e09-e12 Operations; e13-e18 file none), dated 2026-07..10 day ((i-1) mod 30)+1 (i 1-30 July, 31-60 August, 61-90 September, 91-120 October). Its tier m = i mod 7 fixes what it buys, from whom and on which PO lines: 1 printer paper 40 x 30 = 1,200 (s02); 2 monitors 3 x 1,200 = 3,600 (s01); 3 office cleaning 1 x 5,000 (s03); 4 laptops 2 x 5,400 + docking stations 2 x 600 = 12,000 (s05); 5 consulting days 13 x 2,000 = 26,000 (s04); 6 rack server 40,000 + installation 8,000 = 48,000 (s01); 0 office chairs 15 x 600 = 9,000 (s02). approval_levels is 1 up to 5,000, 2 up to 30,000 and 3 above; level 1 goes to the requester's department manager (e13/e14/e15), level 2 to e16, level 3 to e18, and level s is decided on the request date + s days. Status: i 1-60 rejected when i mod 10 = 7, else received; i 61-90 rejected when i mod 10 = 7, ordered when 8 or 9, else received; i 91-120 by i mod 10: 1-3 pending_approval (the levels below min(approval_levels, i mod 10) approved, that level pending), 4 approved with no PO yet, 5 approved with a draft PO, 6 and 8 ordered, 7 rejected, 9 and 0 draft. A rejected requisition is rejected at level 1; a draft has no step. An ordered, received or draft-PO requisition has PO po(i) (sent, received or draft) with its tier's lines, dated the request date + approval_levels days, its total the requisition's amount; a received PO has one full goods receipt per line, 4 days after the order, recorded by the requester. PR-0113's description carries an injected instruction.",
"id": "requisition_to_receipt",
"code": "purchase",
"title": "Requisition to receipt: purchase requisitions, an amount-threshold approval, purchase orders with lines and goods receipts",
"provenance": "Mid-tier scenario 采购申请到收货 (requisition to receipt) of the open library, written for this card from the procurement domain alone: generic names, suppliers, items and amounts, no customer-derived content. The seed follows the arithmetic rule in `$comment`, so each truth can be worked out by hand.",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/service_desk.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"$comment": "Seed rule (so every truth can be worked by hand): ticket i = 1..120 (TK-0001..TK-0120) is opened 2026-07..10, day ((i-1) mod 30)+1 (i 1-30 July, 31-60 August, 61-90 September, 91-120 October). With k = ((i-1) mod 12)+1 the customer is cus_01 for k 1-4, cus_02 for 5-7, cus_03 for 8-9 and cus_04 / cus_05 / cus_06 for 10 / 11 / 12. Category by i mod 4: 1 billing, 2 and 3 technical, 0 how_to. Priority by i mod 10: 1 urgent, 2-3 high, 4-7 normal, 8, 9 and 0 low; the SLA policy follows the priority. Channel by i mod 3: 1 email, 2 web, 0 phone. A technical ticket with i mod 20 = 11 (urgent) or 3 (high) outlived its escalation rule and sits in grp_tier2 with a09 (escalated_by esc_urgent / esc_high); every other billing ticket is in grp_billing (a02 when i mod 8 = 1, a03 when 5) and every other technical or how_to ticket in grp_tech (a05 / a06 / a07 for i mod 4 = 2 / 3 / 0). Status: i 1-60 closed, 61-90 resolved; in October an escalated ticket is open, any other urgent or high ticket resolved, i 118-120 new (unassigned), the rest by i mod 3: 0 open, 1 pending, 2 resolved. A resolved or closed ticket took 6 / 20 / 48 / 96 hours (urgent / high / normal / low), an escalated one 14 (urgent) or 40 (high), which is the SLA breach; resolved_on is 0 / 1 / 2 / 4 days (escalated: 1 / 2) after opening, and resolution_code is fixed for billing, answered for how_to, and for technical workaround when i mod 3 = 0, else fixed. Comments: every ticket has the customer's public message (0 minutes) on its opening day; every ticket that is not new has its assignee's public reply the same day (45 / 30 / 15 / 10 minutes by priority); every escalated ticket has a08's internal note (40 minutes) the next day. Every resolved or closed ticket has one satisfaction survey: unanswered (sent, no score) when i mod 3 = 0, else answered on resolved_on with score 2 when breached, 3 when low priority, otherwise 5.",
"id": "service_desk",
"code": "helpdesk",
"title": "Service desk: tickets with comment threads, SLA policies, time-based escalation and satisfaction surveys",
"provenance": "Mid-tier scenario 客服工单 (service desk) of the open library, written for this card from the domain alone: invented company and agent names, example.com addresses and invented figures, no customer-derived content. The seed follows the arithmetic rule in `$comment`, so each truth can be worked out by hand.",
"prompt": {
Expand Down
1 change: 1 addition & 0 deletions docs/qa/scenarios/travel_expense.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"$comment": "Seed rule (so every truth can be worked by hand): claim i = 1..120 is filed by employee e((i-1) mod 12 + 1) (e01-e12; e13/e14 are Finance and file none), dated 2026-07..10 day ((i-1) mod 30)+1 (i 1-30 July, 31-60 August, 61-90 September, 91-120 October), for trip t(i) of (i mod 3)+2 days ending the day before. Lines per claim: hotel ((days-1) nights at cap-50, at cap+100 when i mod 8 = 0), meals (days at cap-20, at cap+30 when i mod 12 = 5), and airfare when i mod 3 = 0 (1 at cap-200, at cap+300 when i mod 9 = 0); an over-cap line is over_standard with a reason. Status: i 1-60 rejected when i mod 10 = 7 else paid; i 61-90 rejected when i mod 10 = 7, manager_approved when 8 or 9, else paid; i 91-120 by i mod 10: 1-4 submitted, 5-6 manager_approved, 7 rejected, 8 finance_approved, 9 and 0 draft. Approval step 1 goes to the claimant's department manager (to e14 when the claimant is that manager), step 2 to e13.",
"id": "travel_expense",
"code": "travel",
"title": "Travel expense: trip requests, claims with lines, grade-based standards and a two-step approval",
"provenance": "Mid-tier scenario 差旅报销 (travel expense) of the open library, written for this card from the domain alone: generic names, cities and amounts, no customer-derived content. The seed follows the arithmetic rule in `$comment`, so each truth can be worked out by hand.",
"prompt": {
Expand Down
12 changes: 8 additions & 4 deletions packages/qa/scenarios/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,15 +81,19 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
| Mode | Steps |
|:---|:---|
| `scratch` | `create-objectstack` → the agent builds from the card prompt and the runner's data contract (`prompts.ts`) → `objectstack validate` / `build` / `verify` (proofs chosen from the tags) → score → `objectstack dev --fresh` → seed, one user + permission-set grant + API key per persona → every question as every persona it names and every operation as its persona, over the app's MCP → the iterate change → rebuild, re-score, re-run |
| `template` | a blank project → boot → install the card's verified build **unmanaged** (`/api/v1/marketplace/install-local`, `installMode: 'unmanaged'`) → the agent applies the iterate change to the installed copy → score → seed → Run. A refused install is a finding in the record |
| `template` | a blank host project (`host<code>`) → boot → install the card's verified build **unmanaged** (`/api/v1/marketplace/install-local`, `installMode: 'unmanaged'`) → the agent applies the iterate change to the installed copy → score → seed → Run. A refused install is a finding in the record |

- **Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`). It
refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI is logged in with an API key
(the cost ruling on cloud#3600), and the record states the driver and login. `RecordedDriver` replays a recording.
- **Observation** (`observe.ts`): the write grader is fed what the runner reads itself: the agent's write calls and
the door's answer (a coded refusal), the object's rows before and after the turn, and the audit trail's actor.
- **Namespaces** (`namespace.ts`): a card's object names are read relative to the package namespace (`account` →
`ns_account`), because a scaffolded project and the unmanaged install both refuse bare names.
- **Persona contract**: the build is asked for one permission set per persona, named by the persona's id. The runner
- **Namespaces** (`namespace.ts`): every card's app is in the namespace `demo` + the card's `code` (`democrm`): scratch
mode scaffolds the project under that name, and template mode installs a template declaring it into a host named
`host<code>` (the unmanaged copy keeps its namespace, and the install refuses one the host owns). The card's names are
read relative to it (`account` → `democrm_account`), because a scaffolded project and the unmanaged install both
refuse bare object names. The build prompt asks for the prefix on EVERY item, and the scorer reports each bare
non-object item name as a failing Build check (`bare-name:<type>:<name>`), so such a build is never verified.
- **Persona contract**: the build is asked for one permission set per persona, named `<namespace>_<persona id>`. The runner
grants it to that persona's user, and makes a seed row whose scope field names a persona's handle (`owner: u_sam`)
owned by that persona's user.
Loading
Loading