perf(rpc): batch goals bulkAnalytics ClickHouse queries for unfiltered goals

[FindMalek]

Closes #679

Problem

goalsRouter.bulkAnalytics fires 2 ClickHouse queries per goal (a completion count + a getTotalWebsiteUsers denominator) inside a Promise.all. A website with 20 goals loading the goals dashboard issues ~40 concurrent ClickHouse round-trips on every request.

Fix

Goals with no filters at all (neither request-level dashboard filters nor a goal-specific filter) get grouped by their effective start date and counted together:

• one shared getTotalWebsiteUsers call per date bucket (previously one per goal)
• one processGoalsConversionCountsBatch query per date bucket that matches every batched goal's step condition and returns per-goal completion counts via GROUP BY (previously one processGoalConversionCount query per goal)

Any goal carrying a filter keeps the exact original one-query-per-goal path, unchanged.

Why the scope stops at "no filters"

I traced buildIdentifiedEventStream (the shared query builder also used by the funnels feature) and found it only threads filter conditions into the match condition for the step at array index 0 — correct for its actual use case (a funnel's entry filter gating step 1), but if I'd batched filtered goals together as "steps," any goal past index 0 would silently have its filter dropped, corrupting its completion count.

With an empty filter list that branch never fires, so every step's match condition is identical in shape regardless of array position — which is what makes batching provably safe in that case. I didn't modify buildIdentifiedEventStream itself, since it's shared with funnels and a subtle bug there would have a much larger blast radius than this fix is worth.

Testing

• New packages/rpc/src/lib/analytics-utils-goals-batch.test.ts: asserts the batched query issues exactly one ClickHouse call for N goals (instead of N), correctly maps results back by position, handles zero goals, and asserts the generated SQL never carries per-step filter gating.
• Added to packages/rpc/package.json's explicit test file list.
• Full packages/rpc suite: 182 pass, 0 fail, 14 skip.
• tsc --noEmit and biome check clean on all touched files.
• Full monorepo turbo run check-types and turbo run test (triggered by the pre-push hook) both pass.

---

Summary by cubic

Closes #679 by batching ClickHouse queries for unfiltered goals in goalsRouter.bulkAnalytics. Previously each goal fired 2 queries (a completion count and the users denominator), so 20 goals meant ~40 round-trips per dashboard load. Goals with no filters now get grouped by effective start date, chunked by 255 goals per query, and counted in one query per chunk with a shared denominator call; a failed batched query falls back to the original per-goal path for just that chunk. Goals with filters keep the original per-goal behavior unchanged.

Behavior notes

• Batched queries are capped at 255 goals because step numbers are encoded as UInt8 in ClickHouse and wrap past that limit; grouping and chunking live in the pure groupGoalsForBulkAnalytics helper.
• Batching is scoped to zero-filter goals only because buildIdentifiedEventStream only threads filters into the first step; batching filtered goals would silently corrupt their counts, so that shared query builder is left untouched.

^{Written for commit dcef465. Summary will update on new commits.}

[FindMalek]

@izadoesdev this is ready for review whenever you have a chance, CI should be green aside from the Vercel preview checks, which need a team member to authorize the deploy (outside my permissions as an external contributor)

[vercel]

@FindMalek is attempting to deploy a commit to the Databuddy OSS Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
documentation	Skipped		Aug 27, 2026 4:41pm

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 82fcdf6a-524d-4fcc-94c7-a0d15d096bba

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

The PR batches filter-free goal conversion queries by effective start date while retaining the original individual path for filtered goals and adding per-goal fallback after batch failures.

• Caps each batch at 255 goals and renumbers steps per chunk to prevent UInt8 identifier overflow.
• Falls back to isolated per-goal queries when a combined query fails.
• Adds coverage for batching, grouping, date buckets, filtering, and result construction.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

The 255-goal chunking keeps generated step identifiers within the UInt8 range, and failed batches now retry each goal independently without dropping healthy sibling results.

Important Files Changed

Filename	Overview
packages/rpc/src/lib/analytics-utils.ts	Adds shared goal-result construction and a batched ClickHouse completion-count query.
packages/rpc/src/lib/goals-bulk-analytics-grouping.ts	Separates filtered goals and chunks filter-free goals by effective start date with a 255-goal limit.
packages/rpc/src/routers/goals.ts	Integrates grouped batch execution and restores per-goal isolation through fallback queries.
packages/rpc/src/lib/analytics-utils-goals-batch.test.ts	Covers batched query count, positional result mapping, empty input, filter exclusion, and result calculations.
packages/rpc/src/lib/goals-bulk-analytics-grouping.test.ts	Covers chunking, filtered-goal routing, request filters, and effective-date grouping.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[bulkAnalytics request] --> B{Request or goal filters?}
  B -->|Yes| C[Run goal individually]
  B -->|No| D[Group by effective start date]
  D --> E[Split into chunks of at most 255]
  E --> F[Shared users query and batched completion query]
  F -->|Success| G[Map per-step counts back to goals]
  F -->|Failure| H[Fall back to isolated per-goal queries]
  C --> I[Return analytics by goal ID]
  G --> I
  H --> I

Loading

_{Reviews (2): Last reviewed commit: "fix(rpc): cap goal batch size and isolat..." | Re-trigger Greptile}

[FindMalek]

Fixed both — capped batches at 255 goals (extracted the grouping into a tested pure function), and combined-query failures now fall back to per-goal queries for that bucket instead of failing every goal in it.

[FindMalek]

@greptile review