feat: pluggable transaction classifier, and a sync that reports its progress

[suiramdev]

Summary

Two commits. The first replaces the semantic tail of packages/api/src/categorisation — the n-gram model, its trainer and the SIRENE/NAF pass — with a model-agnostic TransactionClassifier, a first jev provider behind TRANSACTION_CLASSIFIER, and a cache keyed by merchant signature. The second gives the sync a voice: a sync_run row records the phase and counters of the run a user has in flight, Budget polls it, and the lists refresh every hundred transactions. Read resolve.ts › classifyPending and sync-progress.ts › claimSyncRun first.

Motivation

• The shipped classifier resolved almost nothing, and its weights and SIRENE cache cost a weekly job and 3 000 lines.
• Categories were decided twice — by the pipeline, then again at read time — so a filter and a label could disagree.
• A first sync showed No transactions for minutes: the page looked finished and empty while the work ran.
• Nothing stopped two tabs starting the same provider read.

Drawbacks

• An unresolved credit now reads uncategorised, not other-income. Sign-split totals do not move; only the label does.
• The classifier is opt-in, so an instance with none configured loses the old model without a replacement.
• The classification cache is one row per signature for the whole instance: a hash, no user column, abstentions retried after 30 days.
• The status poll runs at 1.5 s while a sync is live, and stops when none is.

Prior art

• The provider registry copies packages/email/src/registry.ts: one factory, a Result, a startup refusal.
• The classification store copies the swallow idiom of user-override.ts: a cache or progress failure never fails a sync.
• Progress as a claimed database row is the usual shape for work a second request must observe.

Notes

• Migrations 20260919120000_merchant_classification and 20260920090000_sync_run; the first backfills resolutionStage.
• New optional variables: TRANSACTION_CLASSIFIER, TYPESAFE_API_KEY, TYPESAFE_MODEL. jev without a key refuses at startup.
• Removed: train:model, data/model-weights.json, the SIRENE client and cache, and the SIRENE steps of generate-data.
• Visual change: a progress panel above Budget, and placeholder rows in place of the empty states during a first sync. Both states, and the measurements taken on them in a browser, are in a comment below.
• bun test packages/api/src apps/web/src: 441 pass, 1 fail — merchant-key.test.ts › drops deferred and immediate debit markers needs data/place-tokens.json and fails the same way on dev.
• Docs: contributing/{categorisation,data-model}, guides/bank-connections, integrations/procedures, self-hosting/configuration, docs/engineering/{api,web-app,categorisation,data-pipeline}.md.
• The reviewer agent ran over each half and returned approved, empty findings. It caught an IBAN in the classifier payload and a superseded reporter that could release a live run's lock; both are fixed.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: a97f7975-98b0-43c4-950d-edf021c15860

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[suiramdev]

Verification against the dev stack (OrbStack, real Postgres, real provider refusal). Screenshots do not upload from the CLI, so each state is described with what was measured on it in the browser.

Importing. Importing transactions from your banks on the left of the panel, the detail on the right — Connecting to your banks, then Account 2 of 3, then 55 transactions so far — and a travelling bar under both. The bar is indeterminate on purpose: the indicator's own x measured 525 → 751 → 960 → 1066, then wrapped to -38, on the declared 1.4 s loop.

Categorising. Sorting transactions into categories with 30 of 120, and a determinate bar: aria-valuenow=25, indicator width 0.250 of the track. At 96 of 120 it read 0.80.

Transactions stream in. 120 rows inserted while the run was live: the totals went €0.00 → €226.71 and the rows rendered, with no reload and no navigation.

Empty states. With a live run and no history, Transactions renders 29 skeletons and Recurring 15, and neither shows its empty state. With 100 transactions present and a filter that matches none, both show the real empty state and zero skeletons — the loading state follows firstTransactionDate, not the filtered list length.

Locks. A second syncAccounts during a live run answered {"started":false} and left the row untouched. A row aged six minutes was reclaimed (started:true, counters reset) and ran to FINISHED. A reporter whose row had been reclaimed wrote nothing at all: its transactionsImported(999) and its finished() both landed as no-ops while the live run kept its counters and its lock.

Counters. 100 candidate rows of which 40 match an MCC rule → sync_run ends at categorised: 100, categorisedTotal: 100, while the procedure returns the 40 it resolved and 40 rows carry resolutionStage = mcc.

Classifier. Through budget.recategorise with TRANSACTION_CLASSIFIER=jev: netflix com resolved to streaming at confidence 1 (answeredBy jev-1.13.0), stage model; the second pass read cached-model with zero provider calls and no new cache row; a nonsense descriptor abstained and was not asked again. spotify and edf never reached the classifier — the dictionary answered first.

Motion. The panel enters with opacity 0→1, translateY(-8px)→0 and blur(4px)→0, sampled mid-flight at translateY(-7.08px); it leaves on a shorter ease-out. motion-reduce stops the loop and the width transition, and every state carries a text cue as well as the bar.