From b8aa7d033e9ac907081973705808d8497fc07d5d Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 13:19:35 +0530 Subject: [PATCH] docs(ui-capture): privacy, sessions, network bodies, ingest keys and limits Four guides and a reference page for the UI Testing section, written for @keploy/capture-sdk 3.1.0: - ui-capture/privacy: what each privacy mode masks and what every mode still records (URL paths, data-testid as written, and under balanced query values and the body fields no rule covers), selectors, editors, Do Not Track and Global Privacy Control (on by default since 3.1.0; Brave and DuckDuckGo send GPC by default), headers, credential names and bodies, values in URLs, secretRoutes and keepRoutes, piiPatterns, PrivacyEngine, what each batch carries, and what the dashboard, REST and MCP show. - ui-capture/sessions-and-sampling: one session per page load, every way a session ends, continuous capture, /complete, sampling per page load, custom session ids, pause and hidden tabs, batching and retries, and the server's session states. - ui-capture/network-and-bodies: which calls are recorded and from when (not before rrweb has loaded), how bodies are read, the never-read marker, large-body notes, worker redaction and not-captured markers, GraphQL terms, maskNetworkBodies, streams. - ui-capture/ingest-keys: what a key can do, creating one in the dashboard or the API, one app per environment, rotation, revocation within about 30 seconds, and who can manage keys. - ui-capture/limits: every limit in the browser and on the server. The sidebar gets a Guides category and a Limits and sizes reference item, and the overview links the new pages. Signed-off-by: slayerjain --- .../running-keploy/ui-capture.md | 6 +- .../version-4.0.0/ui-capture/ingest-keys.md | 123 ++++++++++ .../version-4.0.0/ui-capture/limits.md | 77 ++++++ .../ui-capture/network-and-bodies.md | 145 +++++++++++ .../version-4.0.0/ui-capture/privacy.md | 229 ++++++++++++++++++ .../ui-capture/sessions-and-sampling.md | 134 ++++++++++ .../version-4.0.0-sidebars.json | 17 ++ 7 files changed, 730 insertions(+), 1 deletion(-) create mode 100644 versioned_docs/version-4.0.0/ui-capture/ingest-keys.md create mode 100644 versioned_docs/version-4.0.0/ui-capture/limits.md create mode 100644 versioned_docs/version-4.0.0/ui-capture/network-and-bodies.md create mode 100644 versioned_docs/version-4.0.0/ui-capture/privacy.md create mode 100644 versioned_docs/version-4.0.0/ui-capture/sessions-and-sampling.md diff --git a/versioned_docs/version-4.0.0/running-keploy/ui-capture.md b/versioned_docs/version-4.0.0/running-keploy/ui-capture.md index 1aa8dc34a..cd7668117 100644 --- a/versioned_docs/version-4.0.0/running-keploy/ui-capture.md +++ b/versioned_docs/version-4.0.0/running-keploy/ui-capture.md @@ -36,7 +36,7 @@ UI Testing is in beta. Recording visits and reading them as flows and journeys w ## How UI capture works {#how-ui-capture-works} -1. **An ingest key** connects the SDK to one Keploy app. It can only post recordings into that app, so it is safe to ship in your page's JavaScript. +1. **An ingest key** connects the SDK to one Keploy app. It can only post recordings into that app, so it is safe to ship in your page's JavaScript. See [Ingest keys and environments](../ui-capture/ingest-keys.md). 2. **The capture SDK** runs in your users' browsers. It records the page with [rrweb](https://github.com/rrweb-io/rrweb) and every `fetch` and `XMLHttpRequest` call the page makes, masks personal data in the browser, and posts the recording to Keploy's `/ui/v1/ingest` in batches. 3. **Keploy ends a visit** after 30 minutes without activity, or at once when the visit is completed with `POST /ui/v1/sessions//complete`. It then extracts the visit's flows, usually within a few minutes. 4. **Each page of a visit is its own flow**, a route change in a single-page app included, and Keploy chains a visit's flows into its journey. Every journey figure counts sessions: one page load of your app, its in-app navigation included. @@ -230,6 +230,8 @@ Masking happens in the browser, before anything is sent, so Keploy's server neve The dashboard shows a flow's recorded URLs and calls to people on your team who can view the app. The [REST API and MCP](#what-an-agent-sees) return less. +[Privacy and masking](../ui-capture/privacy.md) covers every rule, and what each mode still records. + ## Limitations {#limitations} - **A visit ends after 30 minutes without activity.** A visitor who comes back to the same tab after that is not recorded again until the page reloads, unless you set `continuous: true`, which starts a new session at the visitor's next click, tap or key press. @@ -239,6 +241,8 @@ The dashboard shows a flow's recorded URLs and calls to people on your team who - **Sampling is per page load, not per user.** The SDK carries no user identity, so journey figures are shares of sessions, not of people. - **Replaying journeys as Playwright tests is in beta and not runnable yet.** See [What works today](#what-works-today). +[Sessions and sampling](../ui-capture/sessions-and-sampling.md) explains when a session starts and ends, and [Limits and sizes](../ui-capture/limits.md) lists every limit. + ## Troubleshooting {#troubleshooting} An SDK error or warning names a `KUI_E_` code, and [UI Capture errors and warnings](../ui-capture/errors.md) explains each one: what happened, why, and how to fix it. diff --git a/versioned_docs/version-4.0.0/ui-capture/ingest-keys.md b/versioned_docs/version-4.0.0/ui-capture/ingest-keys.md new file mode 100644 index 000000000..88bce210a --- /dev/null +++ b/versioned_docs/version-4.0.0/ui-capture/ingest-keys.md @@ -0,0 +1,123 @@ +--- +id: ingest-keys +title: "UI capture ingest keys: create, rotate and revoke them, one app per environment" +sidebar_label: Ingest keys and environments +description: What a Keploy UI ingest key can do, how to create, rotate and revoke one in the dashboard or the API, and why each environment gets its own app. +tags: + - UI testing + - UI capture + - ingest key +keywords: + - UI ingest key + - capture SDK apiKey + - rotate ingest key + - revoke ingest key + - uiCaptureSetup + - environments +--- + +import ProductTier from '@site/src/components/ProductTier'; + + + +The capture SDK sends what it records with an **ingest key**, the `apiKey` you pass to `KeployCapture.init()`. This page says what an ingest key can do, how to create, rotate and revoke one in the dashboard or over the API, who may, and why each environment you record should have an app of its own. + +## What an ingest key can do {#what-an-ingest-key-can-do} + +An ingest key is issued for one Keploy app. It can post recordings into that app and do nothing else: no reads, and no other API. That is what makes it safe to ship in your page's JavaScript, where every visitor can read it. + +- It works only on the SDK's two routes: `POST /ui/v1/ingest`, where the SDK posts its batches, and `POST /ui/v1/sessions//complete`, which [ends a visit at once](./sessions-and-sampling.md#end-a-visit-at-once). The SDK sends it as the `X-API-Key` header. A script or a CI job may send `Authorization: Bearer kep_…` instead, but from a page only `X-API-Key` passes the routes' CORS rules. +- A live personal access token, or any other live Keploy key that is not an ingest key, is refused there with `403` and "this API key is not a UI ingest key", and an ingest key is refused everywhere else. Never put another kind of Keploy credential in `apiKey`. +- A key records into the app it was created for, whatever page sends it. +- A key belongs to its app, not to the person who created it: it keeps working when that person leaves your organization, and stops working when the app is deleted. + +## Create one {#create-one} + +### In the dashboard {#create-in-the-dashboard} + +1. Open **UI Testing → UI Flows** and select your app in the header. An app with no flows and no test runs shows the setup card on its own; otherwise the card is under the **Setup** tab. +2. Under **1. Create an ingest key**, optionally type a name of up to 100 characters, such as `production web`. Without one, the key is named `web` and today's date, such as `web 2026-11-20`. +3. Click **Create ingest key**. If you signed in more than 15 minutes ago, Keploy first asks you to **Confirm it's you**: click **Sign in again** and sign in. +4. Copy the key. **Copy this key now — it is shown only once.** + +The setup card after Create ingest key: the new key in a box that says it is shown only once, the install command, an init snippet with the endpoint and the key filled in, and the key listed under Ingest keys + +A key created in the dashboard does not expire: its row under **Ingest keys** says `expires never`. To create one that expires, use the API. The [dashboard quickstart](./quickstart-dashboard.md#create-an-ingest-key) puts a new key into a sample shop. + +### With the REST API or MCP {#create-with-the-api} + +```bash +curl -X POST https://api.keploy.io/client/v1/apps//ui/ingest-keys \ + -H "Authorization: Bearer kep_YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"name": "production web", "ttlDays": 365}' +``` + +- **`name`** is optional, at most 100 bytes once spaces at either end are trimmed. Without one, the key is named `ui-capture` and today's date in UTC. `Web UI Session`, in any case, is reserved and refused. +- **`ttlDays`** is optional: the key's lifetime, from 1 to 365 days. Without it, the key does not expire. +- **The response's `data`** carries the key as `apiKey`, the `endpoint` the SDK posts to, the SDK's install command and an `init` snippet with the key in it, and the next steps. `apiKey` is in this response only, so store it before you go on. +- **Every call creates a new key.** A stored key's secret cannot be returned again, so list and revoke the keys you no longer use. +- A `409` whose `error.code` is `UI_INGEST_KEY_ACCOUNT_CLAIMED` means the key was taken back because the app's service account was being removed at the same moment: send the request again. + +An AI agent does the same with the MCP tool `uiCaptureSetup`. The [Overview](../running-keploy/ui-capture.md#set-up-ui-capture-from-an-ai-agent-mcp) shows how to connect one. + +A key's expiry is checked on every request the SDK makes, so a key stops working the moment it expires. A page that sends an expired key is refused with `401`, and its capture ends with [`KUI_E_INGEST_KEY_REJECTED`](./errors.md#kui-e-ingest-key-rejected). + +## One app per environment {#one-app-per-environment} + +Keploy builds an app's journeys from every session the app receives: each journey's share is a share of all the app's sessions in the last 30 days. Visits from a staging site, a preview deploy or a CI job that record into your production app count in those shares as much as real visitors do. + +So give each environment you record an app of its own, such as `storefront-web` for production and `storefront-web-staging` for staging, and give each environment's build the ingest key of its app. Pick each app's name with care: an app's name cannot be changed once it is created. + +## Rotate a key {#rotate-a-key} + +An app can have several active keys at once, so you can replace one without a gap in recording: + +1. Create a new key for the same app. +2. Deploy your app with the new key. +3. Wait until the old key is no longer used. Tabs that loaded your page before the deploy still send the old key until they reload. The key list shows each key's last use (**last used** in the dashboard, `lastUsedAt` over the API), which Keploy updates at most once every 5 minutes. +4. Revoke the old key. + +Each key in the list also shows its prefix (`keyPrefix`), the start of the key: it tells you which key a page ships. + +## Revoke a key {#revoke-a-key} + +**In the dashboard**, open the setup card: under the **Setup** tab once the app has flows or test runs. Under **Ingest keys**, click the trash icon next to the key. The row asks you to confirm: **Pages using it stop recording.** Click **Revoke**. + +An ingest key's row asking to confirm the revoke, with the line Pages using it stop recording and the Revoke and Cancel buttons + +**Over the API**, delete it by its id from the key list: + +```bash +curl https://api.keploy.io/client/v1/apps//ui/ingest-keys \ + -H "Authorization: Bearer kep_YOUR_API_KEY" + +curl -X DELETE https://api.keploy.io/client/v1/apps//ui/ingest-keys/ \ + -H "Authorization: Bearer kep_YOUR_API_KEY" +``` + +The answer's `data` is `{"id": "", "revoked": true}`. An id that is not one of this app's ingest keys is answered `404`. The list never returns a key's secret, and keeps revoked keys, marked `revoked`. An agent uses the MCP tools `listUIIngestKeys` and `revokeUIIngestKey`. + +Pages that use a revoked key are refused **within about 30 seconds**: each Keploy server keeps its answer for a key for up to 30 seconds. Their capture then ends with [`KUI_E_INGEST_KEY_REJECTED`](./errors.md#kui-e-ingest-key-rejected). Deleting the app deletes its keys too. + +## Who can manage keys {#who-can-manage-keys} + +| To | Permission on the app | API key scope, over the REST API or MCP | +| ------------------ | --------------------- | --------------------------------------- | +| Create a key | `testgen:run` | `write` | +| Revoke a key | `testgen:run` | `write` | +| List an app's keys | `app:view` | `read` | + +In the dashboard, creating or revoking a key also needs a sign-in less than 15 minutes old: past that, Keploy asks you to **Confirm it's you** first. + +Organization admins and the app's App Admins have `testgen:run`. A plain member does not, unless an admin grants it. + +No service-account key can manage ingest keys. Listing them is refused to every service account, and creating or revoking one needs `testgen:run`, which none of the roles a service account can hold grants (`ATG_RUNNER`, `ITG_RUNNER`, `UI_READER`, `MOCK_READER`). Use your own API key, or the dashboard. + +## Related + +- [Dashboard quickstart](./quickstart-dashboard.md) +- [Sessions and sampling](./sessions-and-sampling.md) +- [Limits and sizes](./limits.md) +- [Keploy Public API](../running-keploy/public-api.md) +- [`KUI_E_INGEST_KEY_REJECTED`](./errors.md#kui-e-ingest-key-rejected) diff --git a/versioned_docs/version-4.0.0/ui-capture/limits.md b/versioned_docs/version-4.0.0/ui-capture/limits.md new file mode 100644 index 000000000..8e5fbd6b6 --- /dev/null +++ b/versioned_docs/version-4.0.0/ui-capture/limits.md @@ -0,0 +1,77 @@ +--- +id: limits +title: "UI capture limits and sizes: the SDK's and the server's" +sidebar_label: Limits and sizes +description: Every size, count, rate and time limit of Keploy UI capture, in the capture SDK and on Keploy's server, and what happens past each one. +tags: + - UI testing + - UI capture + - limits + - reference +keywords: + - UI capture limits + - capture SDK maxBodySize + - ingest rate limit + - session size limit + - batch size + - journey plan window +--- + +import ProductTier from '@site/src/components/ProductTier'; + + + +Every limit of Keploy UI capture, and what happens past it: first those the capture SDK applies in the browser, for SDK 3.1.0, then those Keploy's server applies. Where a limit is an SDK option you can set, the table names the option and gives its default. + +## In the browser {#in-the-browser} + +| What | Limit | Past it | +| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| A recorded body, `maxBodySize` | 524,288 UTF-16 code units (512 KB) by default | Text is cut and marked `...[truncated]`. Cut JSON is recorded as `[UNPARSEABLE_BODY_NOT_REDACTED]`. A response whose `Content-Length` is larger is not downloaded: `[Response body too large: N bytes]`. See [Large bodies](./network-and-bodies.md#large-bodies). | +| Events in one request, `batchSize` | 50 by default, and at most 5,000 whatever you set | The rest go in the next request. | +| One request's size | 10 MiB | The batch is cut there. An event that is over 10 MiB on its own is dropped, with [`KUI_E_EVENT_OVERSIZED`](./errors.md#kui-e-event-oversized) once per capture. | +| Time between requests, `batchInterval` | 5,000 ms by default | Each request carries one batch, so a backlog drains one batch at a time. | +| Events held unsent, `maxQueueSize` | 5,000 events by default, and 16 MiB | The oldest events are dropped; when a batch that failed to send goes back into a full queue, the newest queued events are dropped instead. | +| The last request, as the page unloads | 32 KiB: half the 64 KiB of `keepalive` requests a browser lets a page have in flight | What does not fit is lost. | +| Redacting a body on the page's main thread | 64 KiB of each body where the page has no body worker; 64 KiB in all as the page leaves or `stop()` ends the capture | The rest of the body is cut, under `[Body cut: K of N characters kept, unverified]`, where `K` is what was kept: 65,536 characters where the page has no worker. | +| Bodies held for redaction | 8 Mi UTF-16 units (16 MiB) of copies | The next call's bodies are recorded as `[Body cut: 0 of N characters kept, unverified]`. | +| A session, `maxSessionDuration` | 600,000 ms (10 minutes) by default | The session ends. With `continuous: true`, the next one starts at the visitor's next click, tap or key press. | +| `stop()` waiting for bodies | 1 second | A body still open is recorded cut, marked `...[truncated]`. | +| A request that fails with `429`, a `5xx` or a network error | 3 attempts, up to 1 s and then up to 2 s apart | The batch goes back to the front of the queue. Each later batch gets one attempt, after a wait that doubles from 2 s to 60 s, each wait lengthened at random by up to half. | +| A session id from `sessionIdGenerator` | 1 to 200 characters of letters, digits, `.`, `-` and `_`, with no empty dot-separated part | The capture is refused with [`KUI_E_SESSION_ID_INVALID`](./errors.md#kui-e-session-id-invalid). | +| The page's URL, as the SDK reads it for credentials | 64 KiB of path, query and route, and 32 credential values | The page's URL is recorded as its origin and `/[REDACTED]`. A credential past those bounds is not held, so a link, a request or a text that carries it again is recorded with it. A value a `privacy.secretRoutes` route names in the URL's own path is always held. | +| Refused `init()` calls, as `onError` hears them | At most 16 waiting per error code; hand-overs paced from 50 ms, doubling to 30 s | A refusal that comes while 16 wait resolves its `init()` without calling `onError`. | + +## On the server {#on-the-server} + +These are the limits of Keploy's API server, which Keploy Cloud and a self-hosted Keploy both run. + +| What | Limit | Past it | +| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| One batch the SDK posts | 10 MiB, as sent and decoded, and 5,000 events | `413` for the size, `400` for the count. The SDK keeps its batches inside both. | +| Ingest requests | 6,000 a minute per ingest key, counted by each Keploy server | `429`. The SDK retries on its own backoff. | +| One session | 256 MiB of events, and 10,000 batches | `413` with the reason `session_full`: see [When a session ends](./sessions-and-sampling.md#when-a-session-ends). | +| A session with no new batch | 30 minutes | The server completes the session; it checks every minute. `uiCaptureStatus` reports the window as `idleAfterSeconds`. | +| Extracting a session's flows | 8 attempts, over about 2 hours | The session is marked `FAILED`. | +| A revoked key, or a deleted app's key | Up to 30 seconds on each server | The key is refused with `401`. | +| An ingest key's name | 100 characters in the dashboard's box; 100 bytes, spaces at either end trimmed, over the API | Over the API, refused with `400`. | +| An ingest key's lifetime, `ttlDays` | 1 to 365 days, or no expiry when left out | Refused with `400`. | +| A key's last use, `lastUsedAt` | Updated at most once every 5 minutes | | +| The sessions a journey plan reads | Those of the last 30 days, at most 50,000 | The plan reads only the newest, and says so in a note. | +| A journey plan's reuse | Up to 24 hours, or until the sessions processed since reach a tenth of those it read (at least one; never for a plan that stopped at 50,000) | The next read computes a new plan. | +| A stored journey plan | Kept 14 days | | +| `uiJourneys` with `refresh=true` | A new plan at most once a minute per app; it needs `testgen:run` on the app | | +| Journeys selected for testing | The most taken, until they cover 80% of the moves between pages, and at most 20 | | +| `uiFlowGraph` over REST and MCP | The 100 heaviest links, their nodes (at most 200), and 5 flow ids per link | What is left out is counted. | +| `listUIFlows` | 50 flows a page by default, at most 200 | | +| `uiCaptureStatus`'s recent sessions | The 10 most recent | | +| The REST API, `/client/v1` | 100 requests a minute per API key, counted by each Keploy server | `429`. `X-RateLimit-Limit` shows the limit in force. | +| The MCP endpoint, `/client/v1/mcp` | A request with a Keploy API key (`kep_…`) is not limited there, but each REST call its tools make counts against the key's 100 a minute. Any other request: 60 a minute per IP address | `429`. | + +## Related + +- [Sessions and sampling](./sessions-and-sampling.md) +- [Network calls and bodies](./network-and-bodies.md) +- [Ingest keys and environments](./ingest-keys.md) +- [UI Capture errors and warnings](./errors.md) +- [Keploy Public API](../running-keploy/public-api.md) diff --git a/versioned_docs/version-4.0.0/ui-capture/network-and-bodies.md b/versioned_docs/version-4.0.0/ui-capture/network-and-bodies.md new file mode 100644 index 000000000..d02bbdea5 --- /dev/null +++ b/versioned_docs/version-4.0.0/ui-capture/network-and-bodies.md @@ -0,0 +1,145 @@ +--- +id: network-and-bodies +title: "UI capture network calls: what the SDK records of requests and bodies" +sidebar_label: Network calls and bodies +description: Which network calls the Keploy capture SDK records, when it starts, how it reads and redacts request and response bodies, and the markers it records. +tags: + - UI testing + - UI capture + - network capture +keywords: + - UI capture network + - capture SDK + - fetch recording + - XMLHttpRequest + - response body + - maxBodySize + - maskNetworkBodies + - GraphQL requestTerm +--- + +import ProductTier from '@site/src/components/ProductTier'; + + + +Beside what a visitor does, the capture SDK records the network calls your page makes. Each flow lists the calls its steps made, and they are what a replay of the flow needs. This page says which calls are recorded and from when, how the SDK reads a body without changing what your page gets, and what it records when it cannot keep a body whole. It describes SDK 3.1.0. What is masked in a call is on [Privacy and masking](./privacy.md#network-calls). + +## What a recorded call holds {#what-a-recorded-call-holds} + +The SDK records every `fetch` and `XMLHttpRequest` call the page makes, each with: + +- its method and URL; +- its request and response headers and bodies, redacted; +- its status and duration; +- whether it was a `fetch` or an `XMLHttpRequest`; +- `requestTerm`, which names the [GraphQL](#graphql) operation a request runs. + +Only those two APIs are recorded: a WebSocket's messages, an `EventSource` stream and `navigator.sendBeacon` are not. The SDK's own requests to your `endpoint` are never recorded. + +A framework's own requests are recorded like your app's. In the [Next.js sample shop](https://github.com/keploy/samples-typescript/tree/main/ui-capture-next-shop), a `` fetches its page's server component payload (`GET /products?_rsc=…`) as it prefetches and when it is followed, and those calls appear in the recording beside the shop's own. A fetch a server component makes runs on the server, not in the browser, so it is not recorded. + +## Start capture before your first call {#start-capture-before-your-first-call} + +The SDK starts recording the page's calls only once it has loaded rrweb. `init()` waits for `import("rrweb")` before it patches `fetch` and `XMLHttpRequest`, and your bundler normally puts rrweb in a chunk of its own, fetched over the network. **A call the page makes before that chunk has loaded is not recorded at all**, in a hidden tab as in a visible one. Calling `init()` as early in your app's startup as you can narrows that window; nothing closes it. + +So a page's first calls are often missing from its recording. The [React sample shop](https://github.com/keploy/samples-typescript/tree/main/ui-capture-react-shop) imports its `src/keploy.ts` before anything else, and the first page's first request, `GET /api/featured`, is still usually missing. The shop does not hold its own requests back until the capture is recording, because that would make every visit wait for rrweb. + +An `init()` called while the browser is offline is the exception: it patches `fetch` and `XMLHttpRequest` at once, and sends nothing it records until rrweb has loaded once the browser is back online. If rrweb then cannot load, the capture is refused with [`KUI_E_DOM_RECORDER_UNAVAILABLE`](./errors.md#kui-e-dom-recorder-unavailable), and what it recorded meanwhile is dropped. On a page a service worker controls, `init()` asks for rrweb at once even offline, so precache rrweb's chunk with the rest of your app: a worker that cannot answer that request gets the capture refused, and refused again until the page is reloaded. + +## How the SDK reads a body {#how-the-sdk-reads-a-body} + +With network capture on, `fetch()` resolves when the response's headers arrive, as it does without the SDK, and the page gets the network's own `Response`. A streamed response, such as server-sent events over `fetch`, NDJSON or an LLM's tokens, streams to the page as it arrives. The SDK reads nothing of a body until the page does, and records what the page read. (Under a fetch polyfill whose responses have no `body`, such as whatwg-fetch, which Mirage JS and pretender use, the SDK reads a copy of each response whole instead.) + +- **Read whole**, with `text()`, `json()`, `arrayBuffer()`, `bytes()` or `blob()`: the browser's own method answers, and the SDK records what your app got. A body read with `formData()`, or with a `json()` your app put on `Response.prototype`, is recorded from a copy taken just before, read only as far as `maxBodySize`. +- **Read through `body`**, with a stream reader: the SDK reads the network once for each read the page makes, nothing ahead of it. A page that stops reading early is recorded with exactly what it read, and one that cancels or aborts stops the download at once. A body your app reads only through a clone's `body` is recorded as not read: monitoring tools such as Datadog RUM and Sentry read their own clone, and the recording stays your app's own read. +- **Never read**: the SDK never reads it either. It is recorded as `[Response body not read by the page]`, with its status and headers, once the page's `Response` and its clones are garbage-collected, or when the tab is hidden, the page leaves, the capture pauses or `stop()` ends it. A replay sends that text as the body. A body a browser API reads outside JavaScript, such as `cache.put()` or `WebAssembly.instantiateStreaming()`, is recorded as not read too. A mobile browser may discard a hidden tab without another event, so a body the page first reads after such a tab is shown again stays recorded as not read. + +Before SDK 3.1.0, a body the page never read was read and recorded whole. + +A body that fails part-way, because of an abort, a dropped connection or the page navigating away, is recorded with what had arrived, cut and marked `...[truncated]`; for a body the page was reading whole, with nothing of it. Each recorded body, whole or cut, is redacted the same way. + +## Large bodies {#large-bodies} + +`maxBodySize`, 524,288 by default (512 KB), bounds how much of a request or response body is recorded. It counts UTF-16 code units, the length of a JavaScript string: one unit per byte for ASCII text, and fewer units than bytes for text with other characters. Past it: + +- **Text** is cut at the bound and marked `...[truncated]`. +- **A cut body that was JSON** no longer parses, and no fragment can be redacted safely, so `[UNPARSEABLE_BODY_NOT_REDACTED]` is recorded in its place: never a partly redacted body, and never one in the clear. +- **A response whose `Content-Length` is over the bound** is never downloaded, and is recorded as `[Response body too large: N bytes]`. + +A body the SDK does not read as text is recorded as a note that names what it was. There are eight: + +| Note | Recorded for | +| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| `[Blob: N bytes, truncated]` | A `fetch` Blob over the bound. | +| `[Blob: N bytes]` | Any Blob sent with `XMLHttpRequest`, which would have to be read inside `send()`. | +| `[Binary: N bytes]` | A buffer over the bound, or one that is not UTF-8 text. A buffer that is text and fits is recorded as that text. | +| `[Response body too large: N bytes]` | A response whose `Content-Length` is over the bound, so the SDK never downloads it. `N` is the server's figure. | +| `[Response body not read by the page]` | A response body your app never read. A replay sends this text as the body. | +| `[ReadableStream]` | A streaming request body, which reading would consume out from under your app. | +| `[ response]` | An `XMLHttpRequest` whose `responseType` makes `responseText` throw. | +| `[Unreadable body]` | A body that could not be read at all, or a `multipart/form-data` body that does not parse into its fields. It is never passed through. | + +A file inside `FormData` is not one of these: the form is recorded, and the file's field reads `[File: name, N bytes]`. The file's bytes are never read. + +Lowering `maxBodySize` to save bandwidth gives you `[UNPARSEABLE_BODY_NOT_REDACTED]` for most large JSON, not a shorter body. + +## Redaction off the main thread {#redaction-off-the-main-thread} + +The page has its body first. The SDK then hands the body to a Web Worker, which redacts it off your page's main thread, in a task of its own once the page's work allows. Your bundler emits the worker, `body-worker.js`, beside your app. + +**Where the page has no worker**, the SDK redacts bodies on the main thread, and only up to 64 KiB (65,536 characters) of each: a longer body is recorded cut, under the line `[Body cut: 65536 of N characters kept, unverified]`, followed by its first 65,536 characters redacted, or by `[UNPARSEABLE_BODY_NOT_REDACTED]` for cut JSON. That is the case under webpack 4, under esbuild unless you copy `dist/body-worker.js` beside your bundle, when your build's assets are served from another origin, and under a Content-Security-Policy that does not allow `worker-src 'self'`. When the page leaves, or `stop()` ends the capture, bodies still in the worker are recorded on the main thread the same way, up to 64 KiB in all, each one past that cut under `[Body cut: K of N characters kept, unverified]`, where `K` is what was kept. + +**What the SDK holds for redaction is bounded.** While the copies of bodies waiting to be redacted come to 8 Mi UTF-16 units (16 MiB) or more, the next call's bodies are not kept: the call is recorded with each body as `[Body cut: 0 of N characters kept, unverified]`. With bodies of the full default `maxBodySize`, that happens to a healthy page past 16 calls in a burst before the page is idle, or past 8 calls a slow worker has not answered. + +**A body the SDK could not redact is recorded as not captured**, never in the clear: the cut line with nothing kept, then a line that says why. + +``` +[Body cut: 0 of N characters kept, unverified] +[Body not captured: the page left or the capture stopped before redaction was ready] + +[Body cut: 0 of N characters kept, unverified] +[Body not captured: the redaction code could not load] + +[Body cut: 0 of 0 characters kept, unverified] +[Body not captured: the page took the body's stream before the redaction code had loaded] +``` + +- The first is a body the page left on, or that `stop()` found, before the SDK's redaction code had arrived. `init()` asks for that code at once, beside rrweb. +- The second is a page where that code could not load at all: a Content-Security-Policy that refuses it, a `404` after a deploy, or the SDK's `dist/index.mjs` served alone. The SDK also logs [`KUI_E_BODY_WORKER_UNAVAILABLE`](./errors.md#kui-e-body-worker-unavailable) once per page. +- The third is a body your app took through `res.body` before the code had arrived, and never read whole. + +A cut or not-captured body is not the response the server sent, and a replay never serves it as one: a replayed test that reaches it is reported `unproven`. + +## GraphQL {#graphql} + +Every recorded call carries `requestTerm`, in every privacy mode, `"strict"` included, unless computing it failed (below). For a GraphQL request it names the operation the body runs: its `operationName` and a hash of its document with every string and number blanked, such as `graphql:GetApps#…`, or `graphql:GetApps#pq` for a persisted query. It is `""` for a body that is not GraphQL, and `body:*` for a body that is not text, is longer than 4 Mi characters, or that the SDK could not read or hash. It is computed from the body before redaction and never holds a variable or a literal. + +The code that computes it is a chunk the SDK loads the first time a body could be a GraphQL request, so a page that sends none never loads it. Where that code cannot load, the call is recorded with `body:*`, and where computing a term fails, it is recorded without one. Either way the SDK logs [`KUI_E_REQUEST_TERM_FAILED`](./errors.md#kui-e-request-term-failed) for the first such call in a capture. The page's own request is never affected. + +Keploy's server keys the flows it stores without the term, so GraphQL calls with one method and URL are answered in the order they were recorded, whatever operation each runs. + +## Mask every body {#mask-every-body} + +`privacy.maskNetworkBodies: true` records every request and response body as `[MASKED]`, decided without the SDK reading it: the page's `Response` is never read, cloned or relayed, and the SDK's body code is never loaded. A request with no body or an empty one, and a response to a `HEAD`, or with status `204`, `205` or `304`, or `Content-Length: 0`, is recorded with no body. A response that does not declare its length is `[MASKED]`, even one that turns out empty. + +One body is still read: a `Request` object's own, as in `fetch(new Request(url, { body }))`, which is copied and read only as far as its `requestTerm` needs. + +`privacyMode: "strict"` sets `maskNetworkBodies`. Setting `privacy.maskNetworkBodies: false` beside it is refused with [`KUI_E_PRIVACY_CONFLICT`](./errors.md#kui-e-privacy-conflict) before anything is recorded. + +## Streams and uploads {#streams-and-uploads} + +- **A streamed response** reaches the page as the network delivers it, with or without the SDK. +- **A stream passed as a request's `body`** to `fetch()` is not read at all: it is recorded as `[ReadableStream]` and pulled exactly as without the SDK. +- **A streamed upload built into a `Request`**, as in `new Request(url, { body: stream, duplex: "half" })`, is pulled further than the network alone would pull it: when the server answers before the upload ends, the SDK has read on into the stream, about `maxBodySize` bytes of it for ASCII text, up to 4 Mi characters for a stream that starts as a JSON object, which could be a GraphQL request, and all of a `multipart/form-data` body. No browser API says how much of a body the network consumed, so the SDK reads its own copy until it has what the recording keeps. +- **`KeployCapture.stop()`** waits up to one second for calls already answered to finish their bodies, and records any still open then, such as an endless stream, cut at that point and marked `...[truncated]`. A body the page never read is recorded as `[Response body not read by the page]`. `KeployCapture.pause()` records them the same way at once, with no wait. + +## Turn network capture off {#turn-network-capture-off} + +Pass `networkCapture: false` to `init()` to record the page only: the SDK then does not patch `fetch` or `XMLHttpRequest` and does not load its body code, and the flows it records have no network calls. + +## Related + +- [Privacy and masking](./privacy.md) +- [Sessions and sampling](./sessions-and-sampling.md) +- [Limits and sizes](./limits.md) +- [`KUI_E_BODY_WORKER_UNAVAILABLE`](./errors.md#kui-e-body-worker-unavailable) and the other warnings diff --git a/versioned_docs/version-4.0.0/ui-capture/privacy.md b/versioned_docs/version-4.0.0/ui-capture/privacy.md new file mode 100644 index 000000000..8de07a04f --- /dev/null +++ b/versioned_docs/version-4.0.0/ui-capture/privacy.md @@ -0,0 +1,229 @@ +--- +id: privacy +title: "UI capture privacy: what the SDK masks and what it still records" +sidebar_label: Privacy and masking +description: What the Keploy capture SDK masks in the browser in each privacy mode, what it still records, and how to mask, block and declare more. +tags: + - UI testing + - UI capture + - privacy + - masking +keywords: + - UI capture privacy + - capture SDK masking + - privacyMode + - strict mode + - Do Not Track + - Global Privacy Control + - honorPrivacySignals + - secretRoutes + - PII redaction +--- + +import ProductTier from '@site/src/components/ProductTier'; + + + +The capture SDK, `@keploy/capture-sdk`, masks personal data in your visitors' browsers before anything is sent. This page is for the developer who sets it up and for the privacy or security reviewer who signs it off: what each privacy mode masks, what every mode still records, and how to mask, block or declare more. It describes SDK 3.1.0. The [package's README on npm](https://www.npmjs.com/package/@keploy/capture-sdk) states every rule in full. + +## Where masking happens {#where-masking-happens} + +Masking happens in the browser, before the SDK sends anything. Nothing can be unmasked later, because Keploy's server never receives the original. The SDK sends what it records only to the `endpoint` you configure; [What leaves the browser](#what-leaves-the-browser) lists what each request carries. + +## Choose a privacy mode {#choose-a-privacy-mode} + +`privacyMode` sets what is recorded of the page. A masked value is recorded as asterisks, one per character, so its length is kept. + +| What the visitor typed or sees | `"balanced"` (default) | `"strict"` | `"off"` | +| ------------------------------------------------------------------------------------------------------------------------------ | ---------------------- | ---------- | ------- | +| `password` and `hidden` inputs | masked | masked | masked | +| `color`, `date`, `datetime-local`, `email`, `file`, `month`, `number`, `range`, `search`, `tel`, `text`, `time`, `url`, `week` | masked | masked | kept | +| `