Repository navigation
feat: ClickHouse as the store, for horizontal scale-out - #22
Merged
Merged
Conversation
…ront of it With [clickhouse] set, logs are written to and read from ClickHouse instead of files in the data directory, so any number of instances behind one load balancer answer for all of them. The Loki API, OTLP ingest and every client are unchanged. - otel_logs is the OpenTelemetry Collector's table, as cboxdk/laravel-telemetry-store creates it, plus a Stream column holding the stream labels ingest computed; otel_log_streams, kept by a materialized view, answers label names, values and series without scanning the logs. - Records keep their verbatim resource attributes in ClickHouse mode, so ResourceAttributes is what other readers expect, and a row reads back as the record ingest produced. - The log engines take a LogStore trait; ClickHouse selects on time range, stream matchers and line filters, and the rest of LogQL runs on the rows as before. All 62 cases of the LogQL conformance corpus answer as Loki does on ClickHouse, run when TELEMETRYD_TEST_CLICKHOUSE_URL is set. Traces and metrics are still local. Queries that aggregate over a wide range still read every row; pushing aggregation into SQL comes next. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
count_over_time, rate, bytes_over_time and bytes_rate over nothing but |= and != line filters, grouped by stream labels, are counted by ClickHouse into buckets as wide as gcd(step, range), laid on the windows' own bounds. Every window is a whole number of buckets, so its value is theirs summed, with the same staleness markers emit gives from lines. This is every log volume chart, which read every line of its range before. A grouped label an attribute could stand in for, or a row with no stream, is grouped the old way, one record at a time, so the answer is exact. 13 of the 62 conformance queries take this path and all still answer as Loki. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…first A query's stream matchers are resolved against otel_log_streams, exactly, by the matcher code every store uses, and reach otel_logs as ServiceName IN (...) AND StreamHash IN (...): a UInt64 column instead of a map, five times faster over a billion rows, and an app list that lets the sort key prune even without an app matcher. - Log volume groups by StreamHash and reads the body only for the bytes functions; the labels it groups by come from the resolved streams. - AttributeLabels records the label names a row's attributes give it, so a grouped count finds the rows an attribute regroups with hasAny instead of reading their maps. - A limited scan reads windows from the end it starts at, the first sized from how densely the selected streams wrote there, each four times the last, so the newest hundred lines of a day read seconds of it. The 62 conformance queries still answer as Loki. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
Holds ClickHouse to the local store on what the Loki corpus does not reach: resource attributes over OTLP, attributes standing in for a grouped label, detected_level, offsets, and limited reads cut through bursts of a thousand lines sharing one timestamp, in both directions. Lines that share a timestamp are compared as a set: their order is insertion order on one instance and has no meaning across several. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
/ready pings ClickHouse and answers 503 with the reason when it does not answer, so a load balancer sends writes to an instance that can store them. /status carries every request by operation and outcome, the time spent, rows and bytes written and the last error; /metrics the same as telemetryd_clickhouse_* counters. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
Stream resolution, windowed newest-first reads and keyset pages move out of the log store into clickhouse::scan, generic over the row type, for traces to share. The schema now also creates otel_traces (the Collector's columns plus the stream columns, sorted by app and time so search reads newest first), the trace-id index the Collector keeps beside it, the trace streams summary and the attribute keys per hour. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
otel_log_volume, kept by a materialized view, holds lines and bytes per stream per second per set of attribute label names. A count with no line filter reads it instead of the lines: a day of two billion lines, grouped by level at a five-minute step, went from past the 30 s timeout to 27 ms. Exact, not approximate: - A second is (s, s+1], half open like a window, so whole seconds lie in one bucket; seconds a bucket bound splits are counted from the lines. - The rollup says whether any line in the range has an attribute that would regroup it; only then are such lines read and grouped one by one. - telemetryd_meta records from when the rollup holds every line. A table that held rows before the view existed is counted from the lines before that moment; the latest value written wins, so a backfill can move it. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
… front of it Spans are written to otel_traces as the Collector and the store package lay it out — ResourceAttributes and SpanAttributes split from the span's attributes and put back together exactly, kind and status as they spell them — plus the stream columns. The Tempo engines take a SpanStore trait: - TraceQL conditions are handed down as data and become necessary SQL conditions: name, status, kind and duration directly; attributes by value with the key matched on its letters, which covers relaxed lookups, behind a has(mapValues(...)) a Bloom index can skip granules by; resource.x conditions also narrow the streams read. The predicate still decides. - A trace by id reads the trace-id index first, as the Collector does, then only the stretch of otel_traces the trace lies in. - Search reads its matched traces whole with TraceId IN (...). The 22 Tempo conformance cases answer as Tempo on ClickHouse, and a new parity test holds 37 queries — events, links with trace state, relaxed and resource attributes, numeric comparisons, limits, tag values, TraceQL metrics — to the local store's answers exactly. Test helpers for ClickHouse move to tests/common. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…l store does With [clickhouse] set, metric samples from OTLP, remote_write and the span metrics generator are written to ClickHouse and PromQL reads them there. - The PromQL engine takes a MetricStore trait; the local store implements it by delegation, unchanged. - Samples go into a Null ingest table and two materialized views keep telemetryd_samples, sorted by metric, series and time, and telemetryd_series, every series per day with its labels. Matchers are resolved against the series exactly, in Rust; samples are read by name and series hash, inside the query's windows only, in RowBinary. - A value JSON cannot carry - a staleness marker, any other NaN, an infinity - is stored as a flag beside it and reads back bit for bit. - Counter summaries are declined, so the engine folds the samples itself. - The PromQL conformance corpus answers as Prometheus on ClickHouse, and a parity test holds 85 answers - histograms, resets, gauges, special values, windows past an hour at bounds inside a second, label APIs - identical to the local store's. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
- TTLs come from retention.logs/traces/metrics, not a ClickHouse setting of their own, and follow them on start and reload (MODIFY TTL with materialize_ttl_after_modify = 0, so nothing already written is rewritten). Up to ten years; DateTime ends in 2106. - clickhouse.storage_policy, cold_after and cold_volume: logs and spans move their older days to the policy's cold volume — object storage or slower disks, still queryable — before retention deletes them. Summaries and rollups stay hot. - otel_log_volume_1m beside the per-second rollup. A count answers whole minutes from it, the whole seconds of minutes a bound splits from the second rollup, and only the seconds a bound splits from the lines, so a log volume chart over ninety days reads thousands of rows. Long lists of split units go to ClickHouse a few hundred at a time instead of making the count fall back to reading every line. - relay.upstream with [clickhouse] is refused: there is nothing local to forward. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
The local tail is fed by the process's own ingest, which behind a load balancer is a fraction of the lines. With ClickHouse as the store a tail asks ClickHouse each second for what matched since it last asked, reaching back ten seconds for lines that landed late, and shows each line once. A busy second is read on from where the last ask stopped; a tail more than thirty seconds behind skips ahead and reports the skip as dropped, as Loki's does. Loki's delay_for is honoured. Also: the warning about memory for open segments is not given in ClickHouse mode, which has none. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
/status and telemetryd_app_* read series and rows per app from the streams summaries, and apportion each table's compressed bytes by row share, as the local store apportions a segment's. They were empty with ClickHouse as the store, which has no local segments to read them from. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
A rate over 90 days read every sample of it. Now telemetryd summarises each series per hour and per day into telemetryd_folds - the StreamFold a sealed local segment carries, made with the local store's own add and merge - and a window of whole days and hours is answered from those, its uneven ends read as samples, every window of a query in one read. - The maintenance loop summarises hours five minutes after they close, a day's partition per read, and days once their 24 hours are done; a lease in telemetryd_meta keeps instances from doing the same work, and a fold made twice is kept once by its Version. - telemetryd_sample_counts, kept by a view on every insert, says what each hour holds. A bucket is trusted only where its counts and its folds agree over the query's series; late data makes it disagree, so it is read as samples until the next pass, which looks back three days, makes it again. - Tables take their TTL from a retention passed in, with ttl_statements to change it later; statements(config, retention) is the new signature. - The parity tests compare within 1e-9 relative, the standard the PromQL corpus holds every road to, and cover three days at hour and day steps, late data, and its repair. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
With clickhouse.cluster set every table is created ON CLUSTER with a Replicated engine and replicated_deduplication_window = 0: a rollup's blocks are computed, two batches can count one second of one stream to the same number, and a replica's default deduplication would drop the second. A new test runs two instances in front of different replicas of one cluster, both creating the schema at once, writes logs and spans through one and holds the other's answers to its: lines, log volume, labels, search, trace by id. Run with TELEMETRYD_TEST_CLICKHOUSE_CLUSTER and TELEMETRYD_TEST_CLICKHOUSE_REPLICAS. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
scripts/clickhouse/compose.yml runs a Keeper and two replicas, pinned by digest; a new CI job brings it up and runs the server tests with the ClickHouse variables set, so the Loki and Tempo corpora, the parity tests, the cross-instance tail and the cluster test all gate a merge. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
A cookbook page for the ClickHouse store: setup, what lives where and which tables answer what, retention and cold tiers for ninety days and more, space and query timings measured on two billion lines, replicated clusters, what stays per instance (series limits, span-metrics pairing), delivery semantics, rows other writers left, and why relay and ClickHouse do not combine. [clickhouse] joins the configuration reference. The scope statements change with the product: one node is still the default and needs nothing else, and outgrowing it no longer means outgrowing telemetryd. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
Metric tables are made through schema::Layout and listed in AGING, so their TTL follows retention.metrics, moves raw samples to the cold volume with the rest, and changes on reload; app_usage counts metric series and samples; the cluster test proves metrics written through one replica read through the other. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
- Decide from the windows alone whether summaries can help before asking ClickHouse which are complete: a chart of five-minute windows holds no whole hour and now costs nothing extra. - No FINAL on the summaries a query reads: a fold made again is kept by its Version here, and in the completeness sums an extra row only makes a bucket read as samples until the table merges. - A counter's rate fetches its six fields, not the gauge ones. A month of 42,500 series: p95 over the month 1.5 s, per-route increase 0.15 s, per-route rate chart at 1 h 1.4 s and at 1 d 0.17 s. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
Measured on 600 million spans a day: filtered searches over 24 hours went from refused or past the timeout to 60–230 ms, an unfiltered one to 150 ms, a 95th-percentile duration over an hour from refused to 120 ms. - Search asks the store for the traces to show. ClickHouse reads matching spans newest first and stops once the limit'th latest-starting trace found starts after everything read so far — no trace still unread can start later — with starts bounded from the trace-id index, and drops any found trace that cannot reach the top. Only those are read whole, chunked, and only over the stretch the index says they cover. The local store names every trace its scan read, as before. - TraceQL conditions are stated twice, necessary and sufficient, following Rust's lookups: exact keys decide alone, a relaxed key that might answer is unknown, numbers held as text are certain only as plain decimals, nil and negations are explicit. Search selects by the necessary form. - TraceQL metrics — rate, count, sum, avg, min, max, quantiles and histograms of duration, by intrinsics, resource and span attributes — are aggregated in ClickHouse over the spans the sufficient forms take; spans only the necessary forms take come back and are counted one by one, so the answer is the local store's. Quantiles fall back to scanning when any span is undecided, since quantiles do not add. The trace parity test grows to 49 queries, 16 of them metrics, compared within a relative 1e-9 where summation order moves the last bits. The cookbook page carries the trace timings. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
/ready answers 503 naming ClickHouse, a write is refused with 503 and Retry-After so the sender keeps it, a read is refused: nothing is accepted that was not stored. Runs without a ClickHouse. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
The cookbook page names the metric tables and why they are not the Collector's, the folds and how late samples are handled, Prometheus API timings on 3.7 billion samples over thirty days, and what stays expensive. The ClickHouse CI job also runs the query crate's tests, where the PromQL corpus's ClickHouse variant lives. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
Security: - Reads run readonly = 2 with max_execution_time and cancel_http_readonly_queries_on_client_close, so ClickHouse stops what telemetryd stopped waiting for, and a read never carries write rights. - Plain http to a ClickHouse on another machine is refused unless clickhouse.allow_plaintext says otherwise, as relay.upstream is; a URL with credentials or parameters is refused, pointing at user and password. - Clients — senders, dashboards, unauthenticated /ready — get a short reason; the host, the SQL and ClickHouse's own text go to the log and /status. Robustness: - One ureq agent per client: connections and TLS sessions are reused instead of made per request. - /ready runs an authenticated query in the configured database, reused for a second, so a wrong password or a missing database takes an instance out of the balancer instead of leaving it ready and refusing every write. Operators: - A reload refuses every [clickhouse] change by name, as it does the other keys a restart is needed for; a TTL ClickHouse refuses is logged as an error, and one not managed (create_schema = false) as a warning. - Startup failures from ClickHouse say so, instead of blaming a locked data directory; with create_schema = false a missing table stops startup, named. - telemetryd clickhouse schema prints every statement for a DBA; telemetryd clickhouse check checks the account, database, tables, system.parts, cluster and storage policy before a deployment. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
Two instances starting together against different replicas: the second's CREATE DATABASE ON CLUSTER can return before its own replica has run the first's, and a CREATE TABLE in it is then refused. Wait, up to a minute, for the database to appear here. The cluster test passes five runs in five. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…r, day partitions
- completeness and repair take argMax(Seen, Version) per series and bucket, so a
refold that has not merged yet is not counted twice
- the name index is UInt32 and index 0 (not found) is skipped, no UInt16 wrap
- repair refolds only the incomplete series of each day (whole day past 5k series)
- completeness checks whole days first, then only the hours a plan still needs
- the fold lease is read back before it is trusted and renewed after each day
- the raw read behind a fold is capped; past it the window answers from samples
- counts are partitioned by day and folds by (tier, day): no over-retention
- explicit 'UTC' in date conversions; a Starts column of type DateTime('UTC')
- selectors cap at 1M series and switch to a series subquery past 10k
Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…regex, resolved keys - search_traces no longer caps the window read in SQL: spans the residual predicate turns down no longer end a search early (regression test). - A regex is decided in ClickHouse only when RE2 reads it as Rust does; \w \d \s \b are ASCII in RE2, so such patterns are left to the residual. - Undecided rows in an aggregate are capped (100k); past that, or for quantiles, the scan path answers instead. - of_traces reads each trace's own time bounds from the trace-id table and fetches spans by (TraceId, Timestamp) in chunks, ordered and limited. - otel_trace_keys (+ MV, trace_keys_since marker): the hourly attribute keys per service, so relaxed and unscoped fields resolve to the keys that exist instead of every sanitize-equal spelling; unscoped group-bys push down. - otel_traces_trace_id_ts is an AggregatingMergeTree keeping min Start / max End per trace, partitioned by day (new deployments). Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
- existing_columns qualified its columns: the alias name shadowed system.columns.name, so every ADD COLUMN ran again on each start - CREATE DATABASE only when the database is missing - logs planner rewrite; stream matcher regexes the same in RE2 and Rust - unreachable test: a stand-in ClickHouse for create_schema=false Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
- /status: ClickHouse requests and last error as objects; app usage read off the runtime and cached for 60 s; /metrics gauges off the runtime - telemetryd status shows the ClickHouse store and a recent failure instead of a local disk budget that guards nothing - startup says it stores in ClickHouse, and where - TELEMETRYD_TEST_CLICKHOUSE_REQUIRED=1 turns a skip into a failure; CI sets it. README and BUILD-STATUS say how to run them - otel_trace_keys in the cluster test's replica sync Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
- every table and view, partitions, the fold lease, the meta table - clickhouse check/schema, startup refusal, same config on every instance - upgrades keep existing layouts; otel_trace_keys backfill - storage policy on existing tables, move_factor, metric samples move too - 18.7 bytes a line, sizing, monitoring metrics and alerts, /status shape - grants, readonly reads and max_execution_time, TLS, no mTLS - delivery: any refused insert is a 503 with Retry-After - migration via export --to, BACKUP DATABASE, WebSocket tail behind a balancer - threat model: query input to SQL, credentials, the database as the boundary Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…ls.ca_file One window over the whole range read some 70M rows for a page of 100: ClickHouse reads every part in the range at once before the limit stops it. Windows sized from the hourly rate find the page in about 1M rows (errors over 24h of 2B lines: 7.5 s -> 0.05 s). Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…ile refused The global permit was taken before the share was checked, so a dozen refused requests from a noisy client held permits for the moment it took to refuse them, and a quiet client found the queue full: the soak's "the quiet client is never refused" failed intermittently, on main too. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
… skips granules The token index skipped granules only on tokens a pattern holds whole, so |= "9f8e7d6c" read every granule. A 4-gram bloom filter skips on any substring of four characters or more: on 20M lines it skipped 2541 of 2544 granules for an id the token index skipped none of, 2439 of 2544 for key=k49999 against 2148, and is smaller (58 MB against 63). Existing tables get it for new parts; their token index stays. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…it found Growing fourfold from a window sized for the unfiltered rate took a dozen windows to find a page for a rare term, each a request reading its skip indexes again. The next window is now sized from the matches the range read so far held, half again, and never less than fourfold. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
![cbox-agents[bot]](https://avatars.githubusercontent.com/u/19936443?s=60&v=4){kind=small}
There was a problem hiding this comment.
Talking to Cbox
- Reply under a finding to ask Cbox about it.
/cbox fixin reply to a finding: Cbox fixes it on this pull request's branch, or, when it can't push there, on a new branch whose push waits for a confirm in Cbox./cbox reviewreviews this pull request again now./cbox <question>asks Cbox about this pull request./cbox remember <rule>proposes a rule for Cbox's reviews; an owner or admin approves it in Cbox./cbox configshows the review settings this pull request uses and where each comes from./cbox helpshows this list.
Only people who can push to this repository and have linked their GitHub account in Cbox can use /cbox.
| cursor = last + 1; | ||
| write_meta(client, HOURS_CURSOR, &cursor.to_string())?; | ||
| // Renewed as it goes: a pass over weeks of backlog outlasts one lease. | ||
| write_meta(client, LEASE, &format!("{owner}@{}", now + LEASE_NANOS))?; |
There was a problem hiding this comment.
The lease "renewal" reuses the pass's fixed now, so it never extends past the lease's first expiry. Not blocking.
Suggested change
| write_meta(client, LEASE, &format!("{owner}@{}", now + LEASE_NANOS))?; | |
| write_meta(client, LEASE, &format!("{owner}@{}", crate::now_nanos() + LEASE_NANOS))?; |
A filter decided here, not in ClickHouse, kept one line in a hundred and the scan still asked for pages of a hundred: | logfmt | status >= 500 over an hour made some 96 round trips for its page. Pages are now what is still needed times the rate the predicate kept so far, half again, within 100..10k (1.8 s -> 0.16 s on 2B lines). Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
telemetryd clickhouse migrate [--dry-run] [--backfill] [--materialize-indexes] - finds tables laid out differently from what this version creates by probing ClickHouse with the statements an instance would run, and rebuilds them while writes continue: new layout under a temporary name, EXCHANGE TABLES, old partitions copied back through a staging table, checked, and MOVE PARTITION'd in, each recorded as staged and moved so an interrupted run resumes without copying one twice - drops objects earlier versions created, by name only - adds missing indexes; materializes them on partitions older than the index (ensure now records when it adds one) and then drops the index they replace - fills in rollups younger than their rows, a day at a time, by difference per key, so a day done twice adds nothing - one run at a time (lease, --take-over); NDJSON with --json telemetryd clickhouse import --from-data-dir DIR - reads a stopped node's segments and WAL and writes the records through the ClickHouse write path, in windows of time halved until they fit a batch, never cutting through a timestamp; resumable; asks the fold maker to summarise imported hours again export: a full page no longer ends partway through a timestamp, which silently skipped the rest of it (a scrape writes thousands of samples at one). Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
On a cluster with more than one shard each table is <name>_local on every
shard, with <name> a Distributed table over them, so every read and write
names the table as before; the views are on the local tables and each
shard rolls up its own rows. One shard and one server are laid out as
before.
- Spread: logs and their rollups at random, for scans over every shard's
disks; spans and the trace-id index by trace id, so a lookup asks one
shard; every metric table by series, so samples, series, counts and a
fold's versions meet on one shard; telemetryd_meta one copy everywhere
- Writes acknowledged once every shard holds its rows
(distributed_foreground_insert); reads and writes to each shard's first
replica in order, so an instance reads what it wrote; subqueries run on
each shard over its own (distributed_product_mode = local); a trace-id
read asks one shard (optimize_skip_unused_shards)
- Refused at startup: no {shard}/{replica} macros, several replicas
without internal_replication, tables laid out for another number of
shards (one vs several, or a recorded count that has changed)
- system.parts for what each app holds summed over one replica per shard
- telemetryd clickhouse schema asks the server, or --shards one|many;
check reports the layout
- DDL waits for the end of the statement (wait_end_of_query): ON CLUSTER
answered with its first server's row, so the next statement could run
before this one had everywhere, and a later server's failure went unread
- Tests: scripts/clickhouse/sharded (2 shards x 2 replicas), the whole
ClickHouse suite again on it (TELEMETRYD_TEST_CLICKHOUSE_URL_CLUSTER),
tests/clickhouse_shards.rs; unique test database names (two tests
sharing a microsecond shared a database)
Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
The body index goes on each shard's local table; migrate leaves a layout over several shards alone, which only this version creates. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
…rded An earlier version never laid tables out over several shards, so the old layouts are built on the server alone, and on a sharded cluster migrate is held to saying there is nothing older to bring up. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
There was a problem hiding this comment.
0 of 1 addressed.
Talking to Cbox
- Reply under a finding to ask Cbox about it.
/cbox fixin reply to a finding: Cbox fixes it on this pull request's branch, or, when it can't push there, on a new branch whose push waits for a confirm in Cbox./cbox reviewreviews this pull request again now./cbox <question>asks Cbox about this pull request./cbox remember <rule>proposes a rule for Cbox's reviews; an owner or admin approves it in Cbox./cbox configshows the review settings this pull request uses and where each comes from./cbox helpshows this list.
Only people who can push to this repository and have linked their GitHub account in Cbox can use /cbox.
| ) else { | ||
| return Ok(records); | ||
| }; | ||
| if first == last { |
There was a problem hiding this comment.
When a full page is one timestamp from end to end, all_at reads every record at that nanosecond with no cap, which bypasses MAX_LIMIT. Not…
In the at scan, set abort_over to a bound such as MAX_LIMIT * 10 so the request fails with BadRequest instead of materialising an unbounded timestamp.
tower_http's timeout answered a bare 503, with no body and no Retry-After. With ClickHouse as the store an insert waits up to clickhouse.timeout, 30 s by default like server.request_timeout, so with Keeper down every sender got a 503 that did not say when to come back. The limit is now our own middleware, answering Unavailable like every other refusal to wait. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
A runbook for a fleet on ClickHouse: ClickHouse, a replica or Keeper down (drilled on the test cluster), a disk filling, too many parts, slow queries, upgrading telemetryd and ClickHouse, rotating the password, backup and restore on a cluster, the cluster's server config, and what to watch in ClickHouse itself. Plus every command and flag on one page, and the body index named for what it is now. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
There was a problem hiding this comment.
0 of 1 addressed.
Talking to Cbox
- Reply under a finding to ask Cbox about it.
/cbox fixin reply to a finding: Cbox fixes it on this pull request's branch, or, when it can't push there, on a new branch whose push waits for a confirm in Cbox./cbox reviewreviews this pull request again now./cbox <question>asks Cbox about this pull request./cbox remember <rule>proposes a rule for Cbox's reviews; an owner or admin approves it in Cbox./cbox configshows the review settings this pull request uses and where each comes from./cbox helpshows this list.
Only people who can push to this repository and have linked their GitHub account in Cbox can use /cbox.
| done.folds_written += roll_hours(client, (cursor, last), None, now)?; | ||
| done.hours += last - cursor + 1; | ||
| cursor = last + 1; | ||
| write_meta(client, HOURS_CURSOR, &cursor.to_string())?; |
There was a problem hiding this comment.
roll_up writes its own in-memory cursor after each day and overwrites a rewind that refold_from made in the meantime. Not a blocker.
Before writing HOURS_CURSOR, and the same for DAYS_CURSOR, re-read it and write min(stored, cursor) only when the stored value has not moved below the value this pass started from; or have refold_from record a separate pending-rewind key that roll_up honours.
| let value = super::metrics::joined_value(f64::from_bits(read(16)), row[24]); | ||
| samples.entry(read(0)).or_default().push((read(8), value)); | ||
| held += 1; | ||
| if held > MAX_RAW_SAMPLES { |
There was a problem hiding this comment.
The fold shortcut's raw read buffers up to a fixed 20M samples and ignores limits.max_query_samples and whole_reads_allowed. Not a blocker.
Pass the query's resolved sample limit (or the whole_reads_allowed budget) into metric_folds::fold_windows, and stop at the lower of that and MAX_RAW_SAMPLES.
- export: a full page that is one timestamp end to end read every record at that nanosecond, past MAX_LIMIT. Now up to ten pages' worth, and refused with limit_exceeded beyond, rather than cut short. - folds: roll_up wrote its hours and days cursors over a rewind refold_from made meanwhile (an import), so the imported hours were never folded again. It now stops the pass when the stored cursor moved below it. - folds: the fold shortcut's raw read held up to a fixed 20M samples whatever limits.max_query_samples said. It now declines past the query's own ceiling, and the ordinary read refuses with the usual reason. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
There was a problem hiding this comment.
2 of 2 addressed.
Talking to Cbox
- Reply under a finding to ask Cbox about it.
/cbox fixin reply to a finding: Cbox fixes it on this pull request's branch, or, when it can't push there, on a new branch whose push waits for a confirm in Cbox./cbox reviewreviews this pull request again now./cbox <question>asks Cbox about this pull request./cbox remember <rule>proposes a rule for Cbox's reviews; an owner or admin approves it in Cbox./cbox configshows the review settings this pull request uses and where each comes from./cbox helpshows this list.
Only people who can push to this repository and have linked their GitHub account in Cbox can use /cbox.
| limit: "records at one timestamp", | ||
| detail: format!( | ||
| "more than {AT_ONE_TIMESTAMP} records share the timestamp {last}, and a \ | ||
| page cannot end partway through one; narrow the export by selector" |
There was a problem hiding this comment.
The refusal tells the client to narrow the export by selector, but ExportParams has no selector. The range can then never be exported. Not a…
Either let the export continue part-way through a timestamp (for example with an offset or cursor parameter), or remove the selector advice and tell the user to narrow start/end, which is the only lever the endpoint has.
…-ups - folds: an import rewound the fold cursors itself, and a pass in flight could write its progress over the rewind in the moment between its check and its write. An import now leaves a request under a key of its own, and only the lease holder moves the cursors, taking every pending request before it reads them. The single-node import test now imports behind a fleet that has already summarised, and fails without this. - export: the refusal past two million records at one timestamp told the client to narrow by selector, which the export API does not take. It now says what it is. - PromQL: the fold shortcut's sample ceiling is shared by a query's selectors, so together they hold no more than one read could. Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
sylvesterdamgaard
added a commit
that referenced
this pull request
Oct 25, 2026
Everything since 0.72.2: - ClickHouse as an optional store for logs, traces and metrics (#22): stateless instances behind a load balancer over one database, on one server, a replicated cluster or several shards. Loki, Tempo and Prometheus APIs unchanged. Retention as TTL, with hot and cold volumes. - telemetryd clickhouse schema, check, migrate and import. - Docs: scaling out on ClickHouse, operating it, and a command-line reference. - Fixes found on the way: relay fairness under load, a timed-out request now says when to retry, and export no longer skips records sharing one timestamp at a page boundary. - tokio 1.54.1 (RUSTSEC-2026-0386), and dependency bumps (#16, #17, #18, #20). Claude-Session: https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ClickHouse as an optional store for logs, traces and metrics, so several telemetryd instances can run behind one load balancer as stateless ingest and query nodes over one central database. The Loki, Tempo and Prometheus APIs stay the same, so laravel-telemetry and laravel-telemetry-ui need no changes.
When
[clickhouse]is not set, nothing changes: the local store works exactly as before.How
[clickhouse]config:url,database,user,password,cluster,create_schema,storage_policy,cold_after,cold_volume,allow_plaintext,timeout.retention.*.clusteris set. The log and span stream summaries, the log volume rollups (1 s and 1 m), the trace-id index and the trace attribute keys are all kept up by materialized views. Metrics live in samples, series and folds; the folds are coordinated through a lease.readonly=2andmax_execution_time, and is cancelled when the client goes away./readyis 503 when ClickHouse doesn't answer, and writes are refused with 503 plusRetry-After.telemetryd_clickhouse_*, and/statusandtelemetryd statusshow the store.telemetryd clickhouse schemaandtelemetryd clickhouse check.docs/cookbook/scale-out-on-clickhouse.md.docs/cookbook/operate-clickhouse.md: ClickHouse, a replica or Keeper down, a disk filling, too many parts, slow queries, upgrades, password rotation, backup and restore, cluster config, and what to watch in ClickHouse.docs/configuration/cli.md, every command and flag.[clickhouse]section in reference.md.Measured (laptop, ClickHouse 25.8 in Docker)
Tests
TELEMETRYD_TEST_CLICKHOUSE_REQUIRED=1makes a skip fail.Shards, migration and import
clusterof more than one shard, each table is<name>_localper shard with aDistributedtable under the old name. Logs are spread withrand()for scan parallelism. Spans are keyed byTraceId(one shard per trace). Samples, series, counts and folds are keyed bySeriesHash, so a series meets on one shard.telemetryd_metais one replication group across all servers. Writes are forwarded synchronously, so a 200 still means stored. Startup refuses a cluster withoutinternal_replication, without{shard}/{replica}macros, or with a changed shard count. The whole ClickHouse suite also runs on 2 shards × 2 replicas in CI.telemetryd clickhouse migrate:EXCHANGE TABLES, then copy back partition by partition, verified, with a lease, and resumable.--backfillfills rollups for rows older than their completeness markers, exactly once.--materialize-indexesbuilds the new body index on old partitions, then drops the token index it replaces.telemetryd clickhouse import --from-data-dir: copies a stopped single node's segments and WAL into ClickHouse through the write path, resumably. It also fixes an export bug where a full page could stop partway through records sharing one timestamp, so the next page skipped the rest.Also fixed
|= "<id>"now skips granules; before, it read all of them.wait_end_of_query).server.request_timeoutwas a bare503with noRetry-After. With Keeper down, every insert hit it. It now answers like every other refusal to wait.Known limits
export --to; there is no in-place rebalancing.https://claude.ai/code/session_01LtsaMBeethcCYvKx8PTBYL