DataCat requires PostgreSQL 14+ and a single Node.js process. The production process
serves the API, built UI, /install.sh, /downloads, and telemetry intake. No hosted
account or external service is needed.
Start with Ubuntu 24.04, at least 2 CPU cores and 4 GB RAM for a small installation. Capacity depends on ingestion volume, retention and query workload; monitor disk usage and size PostgreSQL accordingly.
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg git openssl postgresql postgresql-contrib python3-venv
# Install current Node 22 LTS from the official NodeSource repository.
curl -fsSL https://deb.nodesource.com/setup_22.x -o /tmp/nodesource-setup.sh
# Review this script before running it.
sudo -E bash /tmp/nodesource-setup.sh
sudo apt-get install -y nodejs
sudo corepack enable
corepack prepare pnpm@10.11.0 --activate
node --version
pnpm --version
git clone <your-fork-or-upstream-url> datacat
cd datacatUse Node 22.15+ (or Node 24) for the current stack and SDK. Building the agent requires Go 1.26.11+ (https://go.dev/doc/install).
Install Docker Engine and the Compose plugin using
https://docs.docker.com/engine/install/ubuntu/ (not the obsolete docker-compose v1).
The host Node/PostgreSQL installation above is unnecessary when using containers.
cp .env.example .env
sed -i "s/^POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$(openssl rand -hex 24)/" .env
sed -i "s/^SESSION_SECRET=.*/SESSION_SECRET=$(openssl rand -hex 32)/" .env
chmod 600 .env
mkdir -p downloads
# Edit PUBLIC_URL in .env if using a domain or a remote host.
docker compose up -d --build
docker compose psCompose stores PostgreSQL data in postgres-data, waits for database readiness, and
binds DataCat to 127.0.0.1:3000 by default. Use a reverse proxy for HTTPS. Only set
BIND_ADDRESS=0.0.0.0 if you have appropriate firewall rules. Never expose PostgreSQL.
docker compose down preserves data; down -v destroys data.
After setup, create an API key in the UI and add it as DATACAT_AGENT_API_KEY in .env.
The optional container agent measures its container, not the full host:
docker compose --profile agent up -d --buildFor full host visibility, install the host agent in section 4 instead.
DataCat keeps users, organizations, API keys, monitors and dashboards in PostgreSQL either way.
Spans, logs and metrics go to the storage you choose with DATACAT_STORAGE:
postgres (default) |
clickhouse |
|
|---|---|---|
| What you run | PostgreSQL only | PostgreSQL + ClickHouse |
| Best for | Small and medium installs; fewest moving parts | Large data volumes and long time ranges |
| Disk for 102M spans (measured) | 63.7 GB | 4.6 GB |
| 7-day aggregations on 102M spans | seconds to minutes; some 7-day pages run out of memory | 9–20× faster |
| Narrow lookups (one service's errors) | fastest (milliseconds) | slower (hundreds of ms) |
| Under overload | slows down, keeps all data | stays fast, may reset a few connections |
| Retention | hourly deletes of expired rows | table TTLs (drops whole days) |
| Replication | standard PostgreSQL replication | ClickHouse replicas with Keeper |
To use ClickHouse with Compose:
sed -i "s/^DATACAT_STORAGE=.*/DATACAT_STORAGE=clickhouse/" .env
sed -i "s/^CLICKHOUSE_PASSWORD=.*/CLICKHOUSE_PASSWORD=$(openssl rand -hex 24)/" .env
docker compose --profile clickhouse up -d --buildFrom source, run ClickHouse 24.8 or newer yourself and set DATACAT_STORAGE=clickhouse
and DATACAT_CLICKHOUSE_URL (plus user and password if set). DataCat creates its tables
on startup. Choose before you send data: switching storage on an existing install starts
with empty telemetry and does not move data that is already stored.
Create a non-superuser PostgreSQL owner and database. createuser --pwprompt asks for
a password without echoing it; choose your own strong password.
sudo -u postgres createuser --pwprompt datacat
sudo -u postgres createdb --owner=datacat datacat
pnpm install --frozen-lockfile
BASE_PATH=/ pnpm run build:selfhost
cp .env.example .env
chmod 600 .envEdit .env: set DATABASE_URL using your database password (URL-encode special
characters), PUBLIC_URL=http://localhost:3000 for initial local testing, and set
SESSION_SECRET to at least 32 random characters. This writes a random secret directly:
sed -i "s/^SESSION_SECRET=.*/SESSION_SECRET=$(openssl rand -hex 32)/" .env
set -a
. ./.env
set +a
NODE_ENV=production pnpm startDo not run as root. The schema is automatically created/upgraded, inside a transaction
with an advisory lock, before the listener opens. The DB user needs CREATE/ALTER
permissions on its own schema. Failed migrations abort startup rather than serving a
half-initialized app. datacat_migrations records applied versions. Existing databases
receive the nullable password hash column without deleting telemetry. Back up before
every upgrade; do not use development drizzle-kit push --force in production.
Create a dedicated datacat OS user, put the checkout at /srv/datacat, and store the
required environment variables in /etc/datacat/server.env owned by root, mode 0600.
Use an absolute path to the installed Node executable. Example
/etc/systemd/system/datacat.service:
[Unit]
Description=DataCat observability server
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
User=datacat
Group=datacat
WorkingDirectory=/srv/datacat
Environment=NODE_ENV=production
EnvironmentFile=/etc/datacat/server.env
ExecStart=/usr/bin/node --enable-source-maps /srv/datacat/artifacts/api-server/dist/index.mjs
Restart=on-failure
NoNewPrivileges=true
PrivateTmp=true
[Install]
WantedBy=multi-user.targetThen sudo systemctl daemon-reload && sudo systemctl enable --now datacat.
Use curl -fsS http://localhost:3000/api/healthz as a readiness check.
All server configuration lives in environment variables. The server does not
automatically load .env; Compose does, and source users can source it as above or
use systemd EnvironmentFile.
| Variable | Default / purpose |
|---|---|
DATABASE_URL |
Required. PostgreSQL connection URL. |
PORT |
3000, HTTP listener. In Compose this controls the published host port; the container listens on 3000. |
PUBLIC_URL |
Canonical external origin, e.g. https://observe.example.com. Used for invite links, installer and in-app documentation. Required when NODE_ENV=production; in development it falls back to the request origin. No path prefix. |
SESSION_SECRET |
Required in production, at least 32 random characters. Signs session cookies. Changing it signs browser sessions out. |
DATACAT_RETENTION_DAYS |
15, positive telemetry retention window. Legacy RETENTION_DAYS is accepted with lower priority. With PostgreSQL, cleanup runs hourly; with ClickHouse it is a table TTL. |
DATACAT_STORAGE |
postgres. Telemetry storage: postgres or clickhouse (see "Choosing telemetry storage"). App data always stays in PostgreSQL. |
DATACAT_CLICKHOUSE_URL |
http://127.0.0.1:8123. ClickHouse HTTP endpoint, used when DATACAT_STORAGE=clickhouse. |
DATACAT_CLICKHOUSE_DB / DATACAT_CLICKHOUSE_USER / DATACAT_CLICKHOUSE_PASSWORD |
datacat / default / empty. ClickHouse database (created on startup) and credentials. |
DATACAT_CLICKHOUSE_CLUSTER |
Empty. Optional ClickHouse cluster name: tables are created as replicated tables ON CLUSTER (needs ClickHouse Keeper). |
DATACAT_BACKGROUND_JOBS |
true. Set false on extra API processes that share one database so monitors are evaluated and notifications sent by exactly one process. |
DATACAT_MAX_BODY_MB |
16. Maximum intake request body (compressed); larger requests get 413. |
DATACAT_MAX_DECOMPRESSED_MB |
64. Maximum decompressed intake payload (protects against gzip/zstd bombs); 413 when exceeded. |
DATACAT_MAX_ATTRIBUTE_CHARS / DATACAT_MAX_LOG_MESSAGE_CHARS |
25000 / 262144. Longer span/log attribute values and log messages are truncated before storage (records keep at most 500 attributes). |
DATACAT_REDACTION |
on. Masks card numbers, bearer tokens, secret key/value pairs, AWS keys, JWTs, private keys and secret-named attributes in logs and spans before storage; off disables. See bench/security/README.md. |
DATACAT_REDACT_EMAILS / DATACAT_REDACT_PATTERNS / DATACAT_REDACT_KEYS |
false / empty / empty. Also mask emails; extra regexes separated by ;;; extra attribute names to mask (comma-separated). |
DATACAT_SETUP_TOKEN |
Empty. Token required to complete first-run setup. When empty, a random token is generated and printed in the log at startup. |
DATACAT_CUMULATIVE_STREAMS_MAX |
500000 (minimum 1000). How many cumulative OTLP metric streams are remembered for delta conversion; the oldest are forgotten past this. |
DATACAT_TRUST_PROXY |
Unset (trust no proxy). Set to the number of reverse proxies in front of DataCat (usually 1), or an Express trust-proxy value, so client addresses and HTTPS are read from X-Forwarded-*. Leave unset when DataCat is reached directly. |
DATACAT_ALLOW_PRIVATE_WEBHOOKS |
false. Webhook and Slack channels may not target loopback, private, link-local or metadata addresses, and redirects are not followed. Set true for receivers on your own network. |
DATACAT_CLICKHOUSE_ASYNC_INSERT |
1. ClickHouse async inserts; each request still waits for the insert to be acknowledged. |
DATACAT_STATS_FLUSH_MS |
10000. How often APM rollups are written (PostgreSQL storage). |
NODE_ENV |
Set to production; development enables pretty logging. |
LOG_LEVEL |
info. |
DATACAT_UI_DIR |
Absolute built UI directory; defaults to artifacts/web/dist relative to the API bundle. |
DATACAT_DOWNLOADS_DIR |
Absolute release asset directory; defaults to artifacts/api-server/downloads. Compose uses /app/downloads. |
DATACAT_DOWNLOAD_URL |
Optional base URL containing agent binaries and SHA256SUMS; defaults to ${PUBLIC_URL}/downloads. Can also be set when running the installer. |
API_PROXY |
UI development only: API origin for Vite's /api proxy. |
BASE_PATH |
UI build base, / for supported single-origin deployments. |
Compose-only settings: POSTGRES_PASSWORD (required), CLICKHOUSE_PASSWORD (with the clickhouse profile), BIND_ADDRESS (127.0.0.1),
DATACAT_AGENT_API_KEY (optional agent profile).
There are no required webhook/SMTP environment variables. Configure webhook and Slack incoming-webhook channels per organization in the UI. Email delivery is not implemented; monitor mentions without a configured channel are recorded in-app. Webhook destination URLs and optional headers are sensitive database contents.
- Visit
PUBLIC_URL. With zero users, DataCat shows Create your admin account + organization. Enter the organization, email, and a password of 12–256 characters. Setup also asks for the setup token:DATACAT_SETUP_TOKENif you set it, otherwise a random token the server prints in its log at startup (docker compose logs datacat | grep "setup token"). - The first account is an administrator; setup is locked thereafter.
- Under Organization Settings → API Keys, create a key for agents and SDKs. Only admins can create, view or revoke keys and notification channels. Application keys are shown once, when created.
- Under Users, invite an email and copy its link. Share it using a private channel. Opening the link lets a new member set their password. Links expire in 30 days and are consumed once; they are not permanent bearer-login URLs.
- Members log in with email/password at
/login. Sign out using the account menu.
Passwords use randomly salted scrypt, never plaintext. Opaque sessions live in
PostgreSQL, expire in seven days, and use HTTP-only SameSite=Lax cookies. HTTPS
PUBLIC_URL enables Secure cookies; explicit HTTP URLs support local deployments.
Local login is throttled per IP (set DATACAT_TRUST_PROXY behind a reverse proxy so the
real client address is used). Cookie-authenticated changes must come from DataCat's own origin.
There is currently no email/password-reset service; keep administrator credentials
safe and follow your organization's controlled account recovery procedures.
Terminate TLS with Caddy/nginx/your load balancer and proxy to 127.0.0.1:3000.
Set PUBLIC_URL=https://observe.example.com and forward the original scheme and host.
The API trusts proxy headers, so restrict direct network access to trusted proxies.
Example Caddyfile:
observe.example.com {
reverse_proxy 127.0.0.1:3000
}Download the matching release assets and SHA256SUMS from your project's GitHub
release into downloads/ (Compose) or artifacts/api-server/downloads/ (source).
To build your own releases with Go 1.26.11+:
DESTDIR="$PWD/dist/downloads" bash agent/release.sh
# Includes linux/amd64, linux/arm64, darwin/arm64 and darwin/amd64.
mkdir -p artifacts/api-server/downloads
cp dist/downloads/* artifacts/api-server/downloads/
# For Compose instead:
mkdir -p downloads
cp dist/downloads/* downloads/TARGETS=linux/arm64 restricts builds; DESTDIR can be outside the repository.
The script runs unit tests and produces SHA-256 checksums. Tag workflow
.github/workflows/release.yml also packages the Node SDK and Python sdist/wheel.
Tags should correspond to the versions declared in the agent manifest and SDK metadata.
On the machine you want to observe, put an API key in the environment using a protected secret store (do not put literal keys in shell history):
curl -fsSL https://observe.example.com/install.sh -o /tmp/datacat-install.sh
# Review /tmp/datacat-install.sh, then:
DATACAT_API_KEY="$YOUR_API_KEY" sh /tmp/datacat-install.shThe installer detects OS/architecture, verifies the binary, installs under
/opt/datacat-agent, writes mode-0600 /etc/datacat/datacat.yaml, and starts a service
on systemd Linux. macOS uses the matching Darwin binary but requires manual startup (or
your own launchd service). Existing configuration is preserved.
Verification: the server computes the SHA-256 of each agent binary it hosts and writes it
into the installer, so the checksum does not come from the same download as the binary.
With DATACAT_DOWNLOAD_URL pointing elsewhere, the installer uses that source's
SHA256SUMS and refuses plain-HTTP downloads from another host unless you pin
DATACAT_AGENT_SHA256. Releases built by this project's workflow carry a signed build
attestation; with the GitHub CLI installed, DATACAT_VERIFY_REPO=<owner>/<repo> makes
the installer check it (gh attestation verify). Serve the installer over HTTPS.
Host metrics appear within a minute. Add log paths/services under logs in the YAML
and restart only your own installed agent. OTLP ports 4317/4318 and compatible
tracer/StatsD ports bind locally by default; never expose them to untrusted networks.
See agent configuration.
Node 22.15+ is required by the current library. SDKs need not be published to npm:
# On the DataCat build machine; put the resulting .tgz in the served downloads directory.
cd sdks/node
npm pack --pack-destination ../../artifacts/api-server/downloads
# In your application directory:
npm install https://observe.example.com/downloads/datacat-trace-0.1.0.tgz
DATACAT_SERVICE=checkout node -r @datacat/trace/init app.jsInitialize before importing frameworks and loggers. With a local agent, export goes
to http://localhost:4318. For agentless use:
DATACAT_SERVICE=checkout DATACAT_URL=https://observe.example.com \
DATACAT_API_KEY="$YOUR_API_KEY" node -r @datacat/trace/init app.jsDATACAT_URL is the origin without /otlp. Structured Pino/Winston/Bunyan logs
ship with trace correlation; console.log is not automatically shipped. See the
Node SDK guide.
Use Python 3.10+ and a virtual environment. Installing directly from source does not require a public package registry release:
python3 -m venv .venv
. .venv/bin/activate
pip install '/path/to/datacat/sdks/python[instrumentation]'
DATACAT_SERVICE=checkout datacat-run python app.pyOr build a wheel (pip install build && python -m build sdks/python) and serve it
alongside the agent, then pip install that wheel URL. Install instrumentation
extras/framework dependencies appropriate for your app. For agentless use:
DATACAT_SERVICE=checkout DATACAT_URL=https://observe.example.com \
DATACAT_API_KEY="$YOUR_API_KEY" datacat-run python app.pySet your root logger to INFO to include INFO messages. See the Python SDK guide. Confirm services in APM, records in Logs, and metrics in Metrics Explorer.
- In Organization Settings → Notification Channels, create a webhook named
oncallwith your HTTPS receiver URL. Optional headers support receiver credentials. - In Monitors → New Monitor, choose a metric query such as
avg(last_5m):avg:system.cpu.user{*} > 80(CPU metrics are percentages; pick a metric your agent actually emits in Metrics Explorer). Set warning/critical thresholds. - Add
@webhook-oncallto the message and save. Slack channels use@slack-name. - Trigger the condition in a test environment. The evaluator checks every 30 seconds, records events and sends JSON containing monitor/state/value/tags. Check notification history and Events for delivery failures.
- Back up with
pg_dumpand testpg_restoreinto a separate DB. Backups include session tokens, API keys, invite tokens and webhook credentials: encrypt them. - Persist PostgreSQL and release downloads. Do not rebuild database state from logs.
- Before upgrades, back up; rebuild UI/API together, restart the DataCat service, and
verify
/api/healthz. Migrations are additive; do not assume binary downgrades are safe. - Retention deletes metrics, logs and spans hourly; use PostgreSQL vacuum/monitoring and disk alerts. Events and configuration are not currently automatically expired.
- No cron or SMTP service is required.
- Never run test tooling against a production database.
scripts/e2e.shrequires an empty, separately deployed test instance.