Skip to content

test(vendors): regression tests + AT_RISK_THRESHOLD + live-risk sort (follow-up to #119)#129

Open
chitcommit wants to merge 2 commits into
mainfrom
claude/vendor-spend-followup
Open

test(vendors): regression tests + AT_RISK_THRESHOLD + live-risk sort (follow-up to #119)#129
chitcommit wants to merge 2 commits into
mainfrom
claude/vendor-spend-followup

Conversation

@chitcommit

@chitcommit chitcommit commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Why

Follow-up to the merged vendor spend-control feature (#119). A self-review surfaced gaps that weren't blocking but are worth closing — chiefly that the two bugs Codex caught on #119 (PATCH-metadata persistence, at_risk-before-limit) shipped without a regression test. This PR adds that coverage plus two small polish fixes. No behavior change to the risk math.

What

Tests (the main point)

  • tests/mcp-vendors.test.tshermetic regression (mocks getDb, same approach as tests/mcp.test.ts, so it runs everywhere). Pins the query_vendors fix: fetch → recompute risk live → filter at_risk → sort by live score → then slice, so a high-risk vendor is never paged out by a small limit, and ordering uses live risk even when the stored risk_score is null/stale. Also covers get_vendor_risk level aggregation.
  • tests/routes/vendors.spec.ts — real-Neon route spec (describe.skipIf(!DATABASE_URL), per the repo's no-mocks rule; cc_vendors is created by the vitest globalSetup since 0019_ is in ADDITIVE_PREFIXES). Covers POST create, PATCH metadata persistence (the exact feat(finance): vendor spend control — cc_vendors + deterministic spend-risk #119 bug), at_risk filtering (failed in / healthy out), the documented clobbering upsert, and /summary.

Polish

  • AT_RISK_THRESHOLD constant in vendor-risk.ts replaces the duplicated >= 50 magic number across the route, both MCP tools, and the cron sweep.
  • GET /api/vendors now sorts by live risk, not the stale stored risk_score column. (The MCP query_vendors path already did after feat(finance): vendor spend control — cc_vendors + deterministic spend-risk #119's fix; the HTTP route still ordered on the column.)
  • Documented POST's FULL-representation upsert — on vendor_name conflict every column is overwritten and omitted fields fall back to defaults (mtd_spend→0, payment_status→'unknown'), so a partial re-POST silently wipes spend data. Use PATCH for partial updates. (Doc comment only; behavior unchanged to avoid breaking the merged contract.)

Testing

  • npm run typecheck — clean.
  • npx vitest run tests/mcp-vendors.test.ts tests/mcp.test.ts tests/lib/vendor-risk.spec.ts tests/routes/vendors.spec.ts57 passed, 5 skipped (the route spec skips without DATABASE_URL, consistent with every other integration spec in the repo; it runs wherever a Neon branch is configured).

Not addressed here (intentionally)

  • Non-UUID :id → HTTP 500 (Postgres uuid cast) instead of 400/404 — consistent with the existing obligations routes; left alone rather than diverging in one place.
  • Live billing-API ingestion (still the chittyagent-finance cross-repo lane).

https://claude.ai/code/session_015mkdG1VYH3AdqLe4E3i9H6


Generated by Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Vendor “at-risk” classification now uses a single shared threshold across endpoints and MCP tools, ensuring consistent filtering and summary results.
    • Vendor listings are sorted by live risk calculations rather than any previously stored score.
    • Risk evaluation now recomputes missing values when stored data is null/stale.
  • Documentation

    • Clarified that POST fully overwrites vendor fields (with defaults), while PATCH is for partial updates.
  • Tests

    • Added MCP vendor tooling regression coverage and expanded end-to-end vendor route integration tests.

Follow-up to the merged vendor spend-control feature (#119), closing gaps found
in self-review:

- tests/mcp-vendors.test.ts — hermetic regression (mocked getDb) pinning the
  at_risk-before-limit fix: filter → sort by live risk → slice, and live risk
  recompute when the stored risk_score is stale/null. Runs everywhere.
- tests/routes/vendors.spec.ts — real-Neon route spec (DATABASE_URL-gated, per
  the no-mocks rule) covering POST upsert, PATCH metadata persistence (the bug
  Codex flagged), at_risk filtering, and /summary.
- AT_RISK_THRESHOLD constant replaces the duplicated `>= 50` magic number across
  the route, MCP tools, and cron sweep.
- GET /api/vendors now sorts by LIVE risk, not the stale stored risk_score
  column (the MCP path already did after the #119 fixes; the HTTP route didn't).
- Document POST's FULL-representation (clobbering) upsert semantics so partial
  re-POSTs don't silently wipe spend data — use PATCH for partial updates.

https://claude.ai/code/session_015mkdG1VYH3AdqLe4E3i9H6
@github-actions

Copy link
Copy Markdown
  1. @coderabbitai review
  2. @copilot review
  3. @codex review
  4. @claude review
    Adversarial review request: evaluate security, policy bypass paths, regression risk, and merge-gating bypass attempts.

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 09983729-7c86-4800-87b0-63276c728404

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Extracts the hardcoded at-risk score cutoff into AT_RISK_THRESHOLD, updates cron, MCP, and vendor route checks to use it, and adds MCP unit tests plus vendor route integration tests.

Changes

AT_RISK_THRESHOLD Extraction and Consumer Updates

Layer / File(s) Summary
Constant definition
src/lib/vendor-risk.ts
Exports AT_RISK_THRESHOLD = 50 with a comment tying it to the urgencyLevel('high') boundary.
Cron and MCP consumers
src/lib/cron.ts, src/routes/mcp.ts
Replaces hardcoded 50 comparisons in sweepVendorRisk, query_vendors, and get_vendor_risk with AT_RISK_THRESHOLD.
Vendor route behavior
src/routes/vendors.ts
Uses AT_RISK_THRESHOLD in list and summary at-risk checks, sorts vendors by live risk, clarifies POST full-representation semantics, and overwrites metadata on conflict.
MCP and vendor route tests
tests/mcp-vendors.test.ts, tests/routes/vendors.spec.ts
Adds hermetic MCP tests and real-DB vendor route integration tests covering at-risk filtering, live-risk ordering, metadata persistence, full upsert behavior, and summary output.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 No more magic fifty in the air,
A shared threshold hops in with care.
MCP and vendors now speak the same way,
With tests to keep the risks in play. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is clearly related to the PR and captures the main changes: vendor regression tests, centralized risk threshold, and live-risk sorting.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/vendor-spend-followup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 29, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
chittycommand b347c8a Jun 30 2026, 03:27 PM

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/routes/vendors.ts`:
- Around line 110-113: The POST upsert behavior in the vendor create route is
documented as a full overwrite, but the conflict path does not replace metadata,
so existing metadata can remain stale after re-posting. Update the upsert logic
in the vendors route so the ON CONFLICT branch also assigns metadata from the
incoming payload, and adjust the surrounding POST docs/comments to match the
actual behavior of the create/upsert flow.

In `@tests/mcp-vendors.test.ts`:
- Around line 49-51: Update the MCP test helper to validate the structured JSON
response shape instead of the legacy text payload. In the helper that reads
`res.json()` and returns `data`, assert that `json.result.content[0].type` is
`json` and parse the payload from `content[0].json` rather than
`content[0].text`. Keep the helper aligned with the `src/routes/mcp.ts` contract
so the suite fails unless the MCP tool returns `content: [{ type: "json", json:
... }]`.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0eab47fa-7781-4b25-a594-f4d848d4496c

📥 Commits

Reviewing files that changed from the base of the PR and between efeb9c2 and e6f5a25.

📒 Files selected for processing (6)
  • src/lib/cron.ts
  • src/lib/vendor-risk.ts
  • src/routes/mcp.ts
  • src/routes/vendors.ts
  • tests/mcp-vendors.test.ts
  • tests/routes/vendors.spec.ts

Comment thread src/routes/vendors.ts
Comment thread tests/mcp-vendors.test.ts
Comment on lines +49 to +51
const json = (await res.json()) as Record<string, unknown>;
const result = json.result as { content: Array<{ text: string }>; isError?: boolean };
return { isError: result.isError === true, data: JSON.parse(result.content[0].text) };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Assert the structured MCP JSON payload here.

Lines 50-51 lock this helper to the legacy content[0].text shape, so the suite will still pass against a non-compliant MCP handler and will fail once src/routes/mcp.ts is corrected to the documented content: [{ type: "json", json: ... }] contract. Read content[0].json and assert type === 'json' instead.

As per coding guidelines, src/routes/mcp.ts MCP tools must return structured JSON using content: [{ type: "json", json: ... }] format.

Suggested fix
-    const result = json.result as { content: Array<{ text: string }>; isError?: boolean };
-    return { isError: result.isError === true, data: JSON.parse(result.content[0].text) };
+    const result = json.result as {
+      content: Array<{ type: 'json'; json: unknown }>;
+      isError?: boolean;
+    };
+    expect(result.content[0]?.type).toBe('json');
+    return { isError: result.isError === true, data: result.content[0]?.json };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const json = (await res.json()) as Record<string, unknown>;
const result = json.result as { content: Array<{ text: string }>; isError?: boolean };
return { isError: result.isError === true, data: JSON.parse(result.content[0].text) };
const json = (await res.json()) as Record<string, unknown>;
const result = json.result as {
content: Array<{ type: 'json'; json: unknown }>;
isError?: boolean;
};
expect(result.content[0]?.type).toBe('json');
return { isError: result.isError === true, data: result.content[0]?.json };
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/mcp-vendors.test.ts` around lines 49 - 51, Update the MCP test helper
to validate the structured JSON response shape instead of the legacy text
payload. In the helper that reads `res.json()` and returns `data`, assert that
`json.result.content[0].type` is `json` and parse the payload from
`content[0].json` rather than `content[0].text`. Keep the helper aligned with
the `src/routes/mcp.ts` contract so the suite fails unless the MCP tool returns
`content: [{ type: "json", json: ... }]`.

Source: Coding guidelines

ON CONFLICT was the one column omitted from the full-overwrite upsert, so a
re-POST kept stale metadata — contradicting the documented "full representation"
contract. Add `metadata = EXCLUDED.metadata` and assert the clobber (set then
reset) in the route spec.

https://claude.ai/code/session_015mkdG1VYH3AdqLe4E3i9H6
@github-actions

Copy link
Copy Markdown
  1. @coderabbitai review
  2. @copilot review
  3. @codex review
  4. @claude review
    Adversarial review request: evaluate security, policy bypass paths, regression risk, and merge-gating bypass attempts.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

@github-actions[bot] Reviewing this PR now with an adversarial focus on security, policy-bypass paths, regression risk, and merge-gating bypass attempts. I’m replying to the latest comment only, since I don’t have the full earlier conversation context.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

Thanks @coderabbitai — both findings reviewed:

  • metadata not replaced on POST upsert (vendors.ts) — fixed in b347c8a. Good catch: metadata was the one column missing from the ON CONFLICT DO UPDATE SET list, which contradicted the documented full-overwrite contract. Added metadata = EXCLUDED.metadata, and the route spec now asserts set-then-reset on re-POST.

  • MCP test should assert content[0].json — skipping, with reason. The shipped src/routes/mcp.ts handler returns content: [{ type: 'text', text: JSON.stringify(result) }] (the tools/call path), and the existing tests/mcp.test.ts already asserts that type: 'text' shape. So the helper is intentionally aligned with the actual contract. Applying the suggestion would assert type: 'json' and read content[0].json, which would make the suite fail against the real handler. CLAUDE.md's "structured JSON { type: "json" } where applicable" is aspirational wording, not what the server emits today — migrating to type: 'json' is a separate change touching mcp.ts + every MCP consumer + mcp.test.ts, out of scope for this test PR.

Separately, the red build / gates / dependency-audit checks are not from this PR — they're a pre-existing, repo-wide npm audit --audit-level=high failure (hono ≤4.12.24 + js-yaml advisories published since #119 merged); main itself fails the same gate. Tracking that remediation separately.


Generated by Claude Code

@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@chitcommit Thanks for the detailed follow-up.

Agreed on both points:

  • The metadata = EXCLUDED.metadata change in src/routes/vendors.ts cleanly fixes the POST full-overwrite inconsistency.
  • Your MCP test rationale makes sense — given the current src/routes/mcp.ts contract returns type: 'text' with JSON encoded in text, the helper should match that shipped behavior, not an aspirational type: 'json' shape.

Also understood on the dependency-audit failures being pre-existing on main, not introduced by this PR.

If you want, I can open a small follow-up issue for the separate MCP response-shape migration (type: 'text'type: 'json') so it’s tracked independently of this PR.

(•ᴗ•)🐇


✏️ Learnings added
Learnt from: chitcommit
URL: https://github.com/chittyos/chittycommand/pull/129

Timestamp: 2026-06-30T15:27:36.710Z
Learning: In this repository, the current `tools/call` handler in `src/routes/mcp.ts` returns MCP tool results as `content: [{ type: 'text', text: JSON.stringify(result) }]`, and `tests/mcp.test.ts` intentionally asserts that `type: 'text'` contract. A migration to structured `type: 'json'` responses would be a separate cross-cutting change affecting `src/routes/mcp.ts`, MCP consumers, and related tests.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants