Skip to content

perf(rpc): goals bulkAnalytics fires 2 ClickHouse queries per goal (N+1) #679

Description

@FindMalek

Problem

goalsRouter.bulkAnalytics (packages/rpc/src/routers/goals.ts) fires two ClickHouse queries per goal — one for the conversion count (processGoalConversionCount) and one for the total-website-users denominator (getTotalWebsiteUsers) — inside a Promise.all over every requested goal.

For a website with N goals, loading the goals dashboard issues up to 2×N concurrent ClickHouse round-trips on every request. A website with 20 goals fires ~40 queries where the actual underlying data (unique visitor counts scoped by date range) could largely be answered by a small, fixed number of queries.

Root cause

Each goal independently computes:

  1. getTotalWebsiteUsers(websiteId, effectiveStartDate, endDate, combinedFilters) — the entrant denominator
  2. processGoalConversionCount(step, combinedFilters, params) — the completion count for that goal's single step

Most goals on a given website share the same date range and carry no goal-specific filters, so steps 1 and 2 above are frequently computing the same denominator and issuing structurally identical completion queries that differ only in which single event/pageview they're matching — exactly the shape ClickHouse can answer in one query via GROUP BY.

Proposed fix

Group goals by (a) whether they carry any filters (request-level or goal-level) and (b) their effective start date:

  • Goals with no filters at all in the same date bucket can be counted in a single batched query — one shared getTotalWebsiteUsers call plus one query that matches all their step conditions and returns per-goal completion counts via GROUP BY, instead of N×2 queries.
  • Goals with filters keep the existing one-query-per-goal path unchanged, since the shared event-stream query builder (buildIdentifiedEventStream) only threads filter conditions through the step at array index 0 (correct for its actual funnel-entry-filter use case) — batching filtered goals together would silently apply another goal's filter to the wrong goal.

This bounds the fix to the case where it's provably correct without touching the shared funnel query builder, while still collapsing the common case (dashboards with mostly unfiltered goals) from O(N) queries to O(distinct date ranges).

Will open a PR shortly with the batched implementation, a regression test asserting the query count, and no change in output for existing behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions