Skip to content

Fix data loss, wrong values and monitor gaps - #4

Merged
Akshayab merged 1 commit into
security-hardeningfrom
correctness-fixes
Nov 17, 2026
Merged

Akshayab merged 1 commit into
security-hardeningfrom
correctness-fixes

Conversation

@Akshayab

Copy link
Copy Markdown
Collaborator

What and why

Fixes data loss, duplicate data, wrong chart values and monitor gaps found in a code review. Stacked on #3 (security hardening).

Intake (no lost or duplicated data)

  • Cumulative OTLP counters and histograms: the new reference value is applied only after the data is stored, so a failed store followed by a retry keeps the increment.
  • Stream keys include every resource, scope and data-point attribute, independent of order. Late samples are ignored. Histogram buckets are subtracted only when the layout matches.
  • Negative exponential-histogram buckets are kept, and disk usage keeps its device tag.
  • ClickHouse insert failures return 503 (retry) instead of 400. Multi-chunk PostgreSQL writes run in one transaction, and series metadata is marked written only after it succeeds.
  • Datadog Agent sketches keep their bins as weighted points, so distribution counts, averages and percentiles are correct. This adds a weight column, with a migration on both storages.

Monitors

  • With notify_no_data, groups that stop reporting go to No Data. The no-data timer starts at the newest data point.
  • evaluation_delay is applied to the query window.
  • Updates are validated as merged, so empty queries are rejected, and the priority is kept.
  • Log monitors grouped by env and span monitors grouped by span fields now evaluate.
  • percentile(N), change and pct_change are computed as written.
  • State is saved with a compare-and-set, so only one process announces each transition. Evaluation errors after a create or update are logged.

Queries and API

  • Formulas combine only matching groups (ungrouped series apply to all groups) and share one bucket size.
  • Data is selected from the exact start time. Numeric filters ignore text values.
  • Facets and charts find flat attribute keys.
  • Totals and timeseries are computed separately.
  • Percentiles use the same interpolation on PostgreSQL and ClickHouse.
  • The APM backfill is limited by span id, and query_index follows the source formula.
  • Dashboard updates lock the row and validate structure. Dashboard and monitor lists page when asked and are unchanged otherwise.

UI, SDKs, server

  • The dashboard page reloads when its id changes, so it can no longer save over another dashboard.
  • Precision-0 numbers keep their zeros.
  • Live tail pages with the cursor and cancels stale requests.
  • The Node exporter treats a reset response as a retryable failure instead of crashing the app.
  • Flask spans end when the response body is finished.
  • SIGTERM drains requests, flushes rollups and closes the database before exit.

Not changed in this PR: merging the PostgreSQL and ClickHouse query code, and replacing any types, are larger refactors and will follow separately.

How it was tested

  • pnpm run format:check, typechecks and unit tests pass (59 server tests, 10 new with the exact inputs from the review; Node SDK 6/6 with a reset test; Python SDK 6/6 with a streaming test)
  • scripts/e2e.sh passes (for intake, query or storage changes): on PostgreSQL and ClickHouse
  • Docs and the configuration reference are updated (for new settings or behavior): no new settings

Also checked on a running server (PostgreSQL): numeric filter with a text value, flat attribute sum, total plus timeseries, log monitor grouped by env, empty-query rejection, priority kept, dashboard validation, monitor paging, and the earlier log-search checks.

~ written by ⠕ Replit

Intake: cumulative OTLP references are applied only after the data is stored; stream
keys include all resource, scope and point attributes; late samples are ignored;
histograms are subtracted by matching bucket layout; negative exponential buckets
are kept; disk usage keeps its device tag. ClickHouse insert errors return 503;
multi-chunk PostgreSQL writes are one transaction; series metadata is marked written
only after success. Datadog Agent sketches keep their bins as weighted points.

Monitors: groups that stop reporting go to No Data when notify_no_data is set; the
no-data timer starts at the newest data point; evaluation_delay is applied; updates
are validated as merged (no empty queries) and keep priority; log env and span group
fields work; percentile(N), change and pct_change are computed; state is saved with a
compare-and-set so one process announces each transition.

Queries: formulas combine only matching groups and share one bucket size; data is
selected from the exact start time; numeric filters ignore text values; facets and
charts find flat attribute keys; totals and timeseries are computed separately;
percentiles use the same interpolation on PostgreSQL and ClickHouse; the APM
backfill is limited by span id; query_index follows the source formula.

API and UI: dashboard updates lock the row and validate structure; list routes page
when asked; the dashboard page reloads when its id changes; precision-0 numbers keep
their zeros; live tail pages with the cursor and cancels stale requests.

SDKs and server: the Node exporter treats a reset response as a retryable failure;
Flask spans end when the response body is finished; SIGTERM drains requests, flushes
rollups and closes the database before exit. Errors after monitor create and update
are logged.

Assisted-by: Replit
@Akshayab
Akshayab merged commit 76883d1 into security-hardening Nov 17, 2026
13 checks passed
@Akshayab
Akshayab deleted the correctness-fixes branch November 17, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant