Restore anonymous usage telemetry

.cargo/config.toml

@@ -2,6 +2,10 @@
 rustflags = ["--cfg", "tokio_unstable"]
 rustdocflags = ["--cfg", "tokio_unstable"]
 
+# Keeps tests and `cargo run` from sending telemetry. A value already set in the environment wins.
+[env]
+QW_DISABLE_TELEMETRY = "1"
+
 [target.x86_64-unknown-linux-gnu]
 # x86-64-v3 requires AVX2, BMI1/BMI2, FMA — available on Intel Haswell (2013)+
 # and AMD Zen 1 (2017)+. Covers all current-gen AWS and GCP instance families.

.github/workflows/ci.yml

@@ -18,6 +18,7 @@ permissions:
 
 env:
   CARGO_INCREMENTAL: 0
+  QW_DISABLE_TELEMETRY: 1
   QW_TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@localhost:5432/quickwit-metastore-dev
   RUST_BACKTRACE: 1
   RUSTDOCFLAGS: -Dwarnings -Arustdoc::private_intra_doc_links

.github/workflows/coverage.yml

@@ -16,6 +16,7 @@ env:
   AWS_SECRET_ACCESS_KEY: "placeholder"
   CARGO_INCREMENTAL: 0
   PUBSUB_EMULATOR_HOST: "localhost:8681"
+  QW_DISABLE_TELEMETRY: 1
   QW_S3_ENDPOINT: "http://localhost:4566" # Services are exposed as localhost because we are not running coverage in a container.
   QW_S3_FORCE_PATH_STYLE_ACCESS: 1
   QW_TEST_DATABASE_URL: postgres://quickwit-dev:quickwit-dev@localhost:5432/quickwit-metastore-dev

.github/workflows/full-tests.yml

@@ -55,6 +55,8 @@ jobs:
     timeout-minutes: 60
     permissions:
       contents: read
+    env:
+      QW_DISABLE_TELEMETRY: 1
     steps:
       - name: Free disk space
         run: |

.github/workflows/ui-ci.yml

@@ -59,6 +59,7 @@ jobs:
           --health-retries 5
     env:
       CARGO_INCREMENTAL: 0
+      QW_DISABLE_TELEMETRY: 1
       RUST_BACKTRACE: 1
       RUSTFLAGS: -Dwarnings --cfg tokio_unstable
       RUSTDOCFLAGS: -Dwarnings -Arustdoc::private_intra_doc_links

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - Azure Blob Storage: support custom endpoints via `endpoint` and `endpoint_suffix` configuration options for sovereign clouds (#6624)
+- Restore anonymous usage telemetry, removed in 0.9.0. It is on by default: set `QW_DISABLE_TELEMETRY=1` to turn it off. `docs/telemetry.md` lists what is sent and where it goes (#6927)
 
 ### Fixed
 - (Jaeger) Query resource attributes when Jaeger request carries tags

LICENSE-3rdparty.csv

@@ -2,6 +2,7 @@ Component,Origin,License,Copyright
 RustyXML,https://github.com/Florob/RustyXML,MIT OR Apache-2.0,Florian Zeitz <florob@babelmonkeys.de>
 addr2line,https://github.com/gimli-rs/addr2line,Apache-2.0 OR MIT,The addr2line Authors
 adler2,https://github.com/oyvindln/adler2,0BSD OR MIT OR Apache-2.0,"Jonas Schievink <jonasschievink@gmail.com>, oyvindln <oyvindln@users.noreply.github.com>"
+advapi32-sys,https://github.com/retep998/winapi-rs,MIT,Peter Atashian <retep998@gmail.com>
 aead,https://github.com/RustCrypto/traits,MIT OR Apache-2.0,RustCrypto Developers
 aes,https://github.com/RustCrypto/block-ciphers,MIT OR Apache-2.0,RustCrypto Developers
 aes-siv,https://github.com/RustCrypto/AEADs,Apache-2.0 OR MIT,RustCrypto Developers
@@ -448,6 +449,7 @@ nom,https://github.com/rust-bakery/nom,MIT,contact@geoffroycouprie.com
 nom-language,https://github.com/rust-bakery/nom,MIT,contact@geoffroycouprie.com
 nonzero_ext,https://github.com/antifuchs/nonzero_ext,Apache-2.0,Andreas Fuchs <asf@boinkor.net>
 normalize-line-endings,https://github.com/derekdreery/normalize-line-endings,Apache-2.0,Richard Dodd <richdodj@gmail.com>
+ntapi,https://github.com/MSxDOS/ntapi,Apache-2.0 OR MIT,MSxDOS <melcodos@gmail.com>
 nu-ansi-term,https://github.com/nushell/nu-ansi-term,MIT,"ogham@bsago.me, Ryan Scheel (Havvy) <ryan.havvy@gmail.com>, Josh Triplett <josh@joshtriplett.org>, The Nushell Project Developers"
 num,https://github.com/rust-num/num,MIT OR Apache-2.0,The Rust Project Developers
 num-bigint,https://github.com/rust-num/num-bigint,MIT OR Apache-2.0,The Rust Project Developers
@@ -847,6 +849,7 @@ unwrap-infallible,https://github.com/mzabaluev/unwrap-infallible,MIT OR Apache-2
 ureq-proto,https://github.com/algesten/ureq-proto,MIT OR Apache-2.0,Martin Algesten <martin@algesten.se>
 url,https://github.com/servo/rust-url,MIT OR Apache-2.0,The rust-url developers
 urlencoding,https://github.com/kornelski/rust_urlencoding,MIT,"Kornel <kornel@geekhood.net>, Bertram Truong <b@bertramtruong.com>"
+username,https://pijul.org/darcs/user,MIT OR Apache-2.0,Pierre-Étienne Meunier <pierre-etienne.meunier@aalto.fi>
 utf8-ranges,https://github.com/BurntSushi/utf8-ranges,Unlicense OR MIT,Andrew Gallant <jamslam@gmail.com>
 utf8-width,https://github.com/magiclen/utf8-width,MIT,Magic Len <len@magiclen.org>
 utf8-zero,https://github.com/algesten/utf8-zero,MIT OR Apache-2.0,"Simon Sapin <simon.sapin@exyr.org>, Martin Algesten <martin@algesten.se>"
@@ -884,6 +887,7 @@ web-time,https://github.com/daxpedda/web-time,MIT OR Apache-2.0,The web-time Aut
 webpki-root-certs,https://github.com/rustls/webpki-roots,CDLA-Permissive-2.0,The webpki-root-certs Authors
 webpki-roots,https://github.com/rustls/webpki-roots,CDLA-Permissive-2.0,The webpki-roots Authors
 whoami,https://github.com/ardaku/whoami,Apache-2.0 OR BSL-1.0 OR MIT,The whoami Authors
+winapi,https://github.com/retep998/winapi-rs,MIT,Peter Atashian <retep998@gmail.com>
 winapi,https://github.com/retep998/winapi-rs,MIT OR Apache-2.0,Peter Atashian <retep998@gmail.com>
 winapi-i686-pc-windows-gnu,https://github.com/retep998/winapi-rs,MIT OR Apache-2.0,Peter Atashian <retep998@gmail.com>
 winapi-util,https://github.com/BurntSushi/winapi-util,Unlicense OR MIT,Andrew Gallant <jamslam@gmail.com>

docs/reference/cli.md

@@ -855,6 +855,14 @@ Specifies the path to the [quickwit config](../configuration/node-config.md). Co
 
 `export QW_CONFIG=config/quickwit.yaml`
 
+### QW_DISABLE_TELEMETRY
+
+Disables [telemetry](../telemetry.md) when set to any non-empty value other than `false`, `no` or `0`.
+
+*Example*
+
+`QW_DISABLE_TELEMETRY=1 quickwit --help`
+
 ### QW_POSTGRES_SKIP_MIGRATIONS
 
 Don't run database migrations (but verify that migrations were run successfully before, and no that unknown migration was run).

docs/telemetry.md

@@ -0,0 +1,85 @@
+---
+title: Telemetry
+sidebar_position: 12
+---
+
+Quickwit collects anonymous usage data to learn which versions are running, on which platforms, and which features are in use. Quickwit is maintained by Datadog, Inc., which receives and stores this data.
+
+## Disabling telemetry
+
+Telemetry is on by default. To turn it off, set `QW_DISABLE_TELEMETRY` before starting Quickwit:
+
+```bash
+export QW_DISABLE_TELEMETRY=1
+```
+
+Any non-empty value other than `false`, `no` or `0` disables telemetry.
+
+To check whether telemetry is on, run `quickwit --help`. While it is on, the output includes a `Telemetry: enabled` line:
+
+```bash
+quickwit --help
+Sub-second search & analytics engine on cloud storage.
+  Find more information at https://quickwit.io/docs
+
+Telemetry: enabled
+```
+
+The line disappears when telemetry is off.
+
+Only `quickwit run` sends telemetry. Other CLI commands never do.
+
+## What is sent
+
+Each time a node starts, it sends a request right away. After that, it sends at most one request per minute, plus one last request with the pending events when it shuts down. Each request holds up to 10 events.
+
+Events, each with the time it happened, in seconds since the Unix epoch:
+
+- `install`: the node found no install ID in its data directory and created one. This happens with a new data directory, but also on the first start after upgrading from a release without install IDs, and on the first start after telemetry is turned on.
+- `run_command`: the node started.
+- `end_command`: the node stopped, with its exit code.
+- `running`: the node is still up. Sent every 12 hours.
+- `ui_index_page_load`: the node served the Quickwit UI page. Browsers request it when someone opens or reloads the UI, and so does any other client that requests a UI URL, such as an uptime check.
+
+Each request also includes:
+
+- The version of the request format.
+- The Quickwit version, and how the binary was built: the commit, the target platform, and whether it was built from a release tag.
+- The services the node runs, such as `indexer`, `searcher` or `metastore`.
+- The features in use:
+  - the metastore type, file-backed or PostgreSQL, if the node runs the metastore or a metastore read replica;
+  - where the default index root URI points: S3, Azure Blob Storage, Google Cloud Storage or the local file system;
+  - whether the S3 storage config targets an S3-compatible store, through a custom endpoint or a flavor such as MinIO, Garage, DigitalOcean Spaces or Google Cloud Storage;
+  - whether the OTLP endpoints are enabled on an indexer, and the Jaeger endpoint on a searcher;
+  - whether ingest API v1 and v2 are enabled;
+  - whether the searcher uses AWS Lambda for leaf searches or a local split cache;
+  - whether document clustering is configured;
+  - whether the REST or gRPC server uses TLS.
+- The OS and the CPU architecture.
+- The number of CPUs and the amount of memory available to the node, as ranges such as `5-8` CPUs or `8-16GiB`.
+- Whether the node runs on Kubernetes, based on whether `KUBERNETES_SERVICE_HOST` is set.
+- A session ID, generated at random each time the node starts.
+- An install ID, generated at random when the node starts with telemetry on and finds none in its data directory, then stored there in the `telemetry_install_id` file. It is never created while telemetry is off.
+- An MD5 hash of the host name and the user name, used to count distinct installations from older releases, which have no install ID. The relay replaces it with a keyed hash before storage. See below.
+- The number of events dropped because too many were queued.
+
+`running` and `end_command` events also include how many requests each API received since the previous report, as ranges such as `10-99`: search and bulk ingestion on the Elasticsearch-compatible API, other Elasticsearch-compatible requests, search and ingestion on the native API, OTLP, and Jaeger. Searches from the Quickwit UI count as native searches.
+
+Quickwit never sends index or field names, documents, queries, configuration values or credentials.
+
+## Where the data goes
+
+Quickwit sends the data over HTTPS to `telemetry.quickwit.io`, which Datadog operates on AWS.
+
+- The request goes through Amazon CloudFront, which looks up the country and city of your IP address and passes them on. The IP address itself goes no further.
+- A small relay checks the request against the format above, rejects anything else, and keeps only the fields listed on this page.
+- The relay replaces the host hash with a keyed hash, computed with a secret that only the relay knows, so the stored value can't be matched against guessed host names and user names.
+- The relay forwards the events, with the country and city, to Datadog, where they are stored as logs. Your IP address is not stored.
+
+For nodes running in the cloud, the country and city are usually those of the data center.
+
+Releases 0.6.0 to 0.8.x send an older version of this data to the same address: the events other than `install`, each with its time, without the API counts, plus the version, services, OS, architecture, Kubernetes flag, session ID, host hash, number of dropped events, and the metastore, OTLP and Jaeger features. Their AWS Lambda indexer and searcher send `run_command` with only the `aws_lambda` feature.
+
+Releases before 0.6.0 also send data to the same address, from every CLI command: an event when a command ends, with its exit code, and an event when an index is created, ingested into, deleted or garbage collected, or when services start, with their names (in 0.1.x, with whether peer addresses were given). Each event has its time, and each request includes the number of dropped events, the version, OS, architecture, Kubernetes flag, session ID and host hash.
+
+To turn telemetry off in releases before 0.9.0, set `QW_DISABLE_TELEMETRY` to any value, or `DISABLE_QUICKWIT_TELEMETRY` in 0.1.x.