diff --git a/docs/configuration/exporting/robusta-pro-features.rst b/docs/configuration/exporting/robusta-pro-features.rst index 466e23e53..cc159fd7e 100644 --- a/docs/configuration/exporting/robusta-pro-features.rst +++ b/docs/configuration/exporting/robusta-pro-features.rst @@ -19,6 +19,7 @@ Data Export and Reporting * :doc:`Alert Reporting API `: Get aggregated statistics and counts for different alert types * :doc:`Send Events API `: Send alerts, incidents, and changes from any monitoring source via a single webhook endpoint * :doc:`RBAC Configuration API `: Programmatically manage role-based access control configurations +* :doc:`Triggered Workflows `: Run a saved Holmes investigation from any system that can send a webhook Getting Started --------------- diff --git a/docs/configuration/exporting/send-events-api.rst b/docs/configuration/exporting/send-events-api.rst index 06ea255dc..720a56873 100644 --- a/docs/configuration/exporting/send-events-api.rst +++ b/docs/configuration/exporting/send-events-api.rst @@ -5,6 +5,8 @@ Send alerts from your monitoring system to Robusta through a single webhook endp This is the recommended ingestion path for new integrations. The legacy :doc:`Send Alerts API ` remains available for existing customers. +Alerts ingested here feed :doc:`Alert Triage `, which investigates them with AI and can group related alerts into :doc:`incidents `. + .. toctree:: :maxdepth: 1 :hidden: @@ -49,7 +51,7 @@ Query Parameters * - ``account_id`` - Your Robusta account ID, found in ``generated_values.yaml``. * - ``cluster`` - - Optional. The cluster to associate the alert with. When set, it overrides any cluster found in the alert payload and is used for the resulting alert investigation. When omitted, the cluster is taken from the payload if present, otherwise the alert is recorded under the ``external`` cluster. Use this when your monitoring system cannot add a cluster label to the alert itself. + - Optional. The cluster to associate the alert with. When set, it overrides any cluster found in the alert payload and is used for the resulting alert investigation. When omitted, the cluster is taken from the payload if present, otherwise the alert is recorded under the ``external`` cluster. Use this when your monitoring system cannot add a cluster label to the alert itself. The same parameter works on Triggered Workflows — see :ref:`Targeting a Cluster Per Request `. Authentication -------------- diff --git a/docs/how-it-works/architecture.rst b/docs/how-it-works/architecture.rst index 0cef5e512..3f4e33f43 100644 --- a/docs/how-it-works/architecture.rst +++ b/docs/how-it-works/architecture.rst @@ -27,7 +27,7 @@ The Robusta Platform is a centralized place to control your SRE agents and chat The Platform receives investigation results from HolmesGPT and provides: - AI-powered root cause analysis results for every alert -- Centralized alert management and triage +- Centralized alert management and :doc:`triage ` - Historical alert analysis and timelines - HolmesGPT Slack and Teams bot — tag the AI agent like a teammate to investigate issues on demand diff --git a/docs/how-it-works/oss-vs-saas.rst b/docs/how-it-works/oss-vs-saas.rst index b70f3c96f..7a227a76b 100644 --- a/docs/how-it-works/oss-vs-saas.rst +++ b/docs/how-it-works/oss-vs-saas.rst @@ -38,7 +38,7 @@ The Robusta Platform (SaaS or Self-Hosted) adds on top: - A web interface - HolmesGPT bots for Slack and Microsoft Teams -- Automatic alert triage and grouping +- Automatic :doc:`alert triage ` and :doc:`incident grouping ` - Background agents that proactively investigate and surface issues Robusta Classic diff --git a/docs/index.rst b/docs/index.rst index daebf5e55..c0ce989c1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,6 +16,16 @@ setup-robusta/index +.. toctree:: + :maxdepth: 4 + :caption: Platform + :hidden: + + Overview + platform/alert-triage + platform/incident-grouping + platform/triggered-workflows + .. toctree:: :maxdepth: 4 :caption: HTTP APIs @@ -67,7 +77,7 @@ Robusta is an AI-powered SRE agent that automatically investigates alerts and fi **How it works:** -* **Automatic investigation** — every alert is analyzed with AI-powered root cause analysis +* **Automatic investigation** — every alert is analyzed with AI-powered root cause analysis. See :doc:`Alert Triage `. * **Your data sources** — HolmesGPT connects to your existing monitoring, ITSM, cloud tools, and MCP servers to gather evidence * **Chat with your agent** — tag HolmesGPT in Slack or Teams to investigate issues on demand * **Centralized control** — the `Robusta Platform `_ gives you a single place to manage your SRE agents, triage alerts, and review investigation timelines diff --git a/docs/notification-routing/notification-grouping.rst b/docs/notification-routing/notification-grouping.rst index 69953a65e..727622809 100644 --- a/docs/notification-routing/notification-grouping.rst +++ b/docs/notification-routing/notification-grouping.rst @@ -5,6 +5,10 @@ Notification Grouping (Slack Only) Reduce alert noise by grouping related notifications into Slack threads. Instead of flooding channels with individual alerts, Robusta creates summary messages with threaded details. +.. note:: + + This page covers grouping of **Slack messages** by the in-cluster Robusta runner, using fixed fields such as ``namespace`` or ``severity``. To group alerts by their **root cause** into incidents, see :doc:`/platform/incident-grouping`. The two are independent and can be used together. + .. image:: /images/notification-grouping.png :width: 600px :align: center diff --git a/docs/platform/alert-triage.rst b/docs/platform/alert-triage.rst new file mode 100644 index 000000000..a63ebc41b --- /dev/null +++ b/docs/platform/alert-triage.rst @@ -0,0 +1,154 @@ +.. _alert-triage: + +Alert Triage +============= + +Alert Triage investigates your alerts with AI and tells you which ones matter. For each alert, Holmes gathers evidence from your connected data sources and returns a root cause, a suggested fix, a priority, and the team that should own it. + +Open **Alerts → Triage** in the Robusta UI to see triaged alerts and the backlog of alerts waiting to be investigated. + +The Triage Pipeline +------------------- + +Alerts move through the pipeline shown at the top of the Triage page: + +1. **Alert sources** — alerts arrive from AlertManager, Datadog, PagerDuty, and other sources via the :doc:`Send Events API `. +2. **Queued** — a new alert is queued for investigation. +3. **Investigating** — Holmes is running. +4. **Triage** — Holmes has written its findings back to the alert. +5. **Priority** — the alert is bucketed as Urgent, High, Medium, or Low. +6. **Destinations** — the findings are delivered wherever you configured. + +**Queued**, **Investigating**, **Backlog**, and the **Priority** buckets are clickable filters. Clicking **Alert sources**, **Triage**, or **Destinations** instead opens **Pipeline Settings** on the matching tab. + +What Holmes Produces +-------------------- + +Every completed investigation records: + +* **What happened** — a summary and a short timeline of events. +* **Root cause analysis** — the reasoning and evidence behind the conclusion. +* **How to fix** — the recommended remediation. +* **Priority** — Holmes classifies urgency internally as ``Urgent``, ``Not Urgent``, ``Noise``, or ``Duplicate``. The UI surfaces this as the **Priority** column: ``Urgent``, ``High``, ``Medium``, or ``Low``. When the alert is linked to an incident, the incident's own priority is shown instead. +* **Team** — the team that should own the alert, with the reason for the assignment. + +Holmes checks for duplicates first. When an alert is a duplicate of one it already investigated, it says so and stops instead of repeating the work. + +Enabling Automatic Triage +------------------------- + +Automatic triage is **off by default**. Turn it on under **Alerts → Triage → Settings → Triage Settings** — the **Settings** button in the page header opens the **Pipeline Settings** dialog. + +.. list-table:: + :widths: 30 15 55 + :header-rows: 1 + + * - Setting + - Default + - Description + * - **Triage Automatically** + - ``Off`` + - Investigates every newly firing alert without anyone clicking. When off, you can still investigate alerts manually. + * - **Daily auto-investigation limit** + - ``20`` + - Maximum automatic investigations in a rolling 24-hour window. ``20`` is also the maximum during beta. + * - **Default agent** + - *(none)* + - The agent that investigates alerts with no cluster of their own. See `Choosing the Agent`_. + * - **Create incidents from investigations** + - ``Off`` + - Lets investigations group related alerts into incidents. See :doc:`/platform/incident-grouping`. + +.. note:: + + **Create incidents from investigations** is independent of **Triage Automatically**. With automatic triage off, manual investigations still create and update incidents. + +Investigating Manually +---------------------- + +Click **Investigate** on any alert, or select several and investigate them in bulk. Manual investigations ignore both the **Triage Automatically** switch and the daily limit, and they do not count toward it. They still need an eligible agent. + +**Re-investigate** discards the previous findings and runs again. While it runs, the alert temporarily moves back to the backlog. + +Choosing the Agent +------------------ + +Investigations run on the agent belonging to the alert's cluster. Two cases need a **Default agent**: + +* Alerts that arrive without a cluster are recorded under the cluster ``external``. +* Alerts whose cluster has no Holmes agent connected. + +In both cases Robusta reroutes the investigation to the **Default agent**. Set one if alerts are queued but never investigated. + +The default agent only decides *where* the investigation runs — the alert keeps its own cluster for display and filtering. + +.. warning:: + + With no **Default agent** configured, these alerts are skipped rather than left queued, and the two cases record different reasons: + + * Alerts with no cluster of their own are marked ``Skipped`` with the reason ``no eligible cluster (external/unknown, no fallback)``. + * Alerts on a cluster with no connected agent are marked ``Skipped`` with the reason ``cluster does not support realtime conversations``. + +The second reason is also what you get when the agent on that cluster is running a Holmes too old to support realtime conversations. Check whether the cluster has a connected agent before upgrading anything. + +Tailoring the Results +--------------------- + +Under **Pipeline Settings**: + +* **Triage Settings** — the switches above. +* **Teams** — the teams Holmes may assign alerts to, each with a description. Without any, everything is assigned to a synthetic ``General`` team. +* **Destinations** — a two-step setup, and both steps are required: + + 1. **Give Holmes a messaging tool.** The tab shows the connection status of **Slack** and **Microsoft Teams**. Any other MCP server or data source that can send messages works too, and is configured under Data Sources. + 2. **Tell Holmes what to do after the investigation.** Free-text instructions appended to every investigation prompt, for both automatic and manual runs — which channel to notify and when, or extra context about your environment. + +.. warning:: + + Instructions alone deliver nothing. With no messaging tool connected, Holmes has no way to send the findings and reports no error. + +Urgency classification follows a built-in set of rules: + +* ``Urgent`` — active customer impact, data loss risk, or security exposure. +* ``Not Urgent`` — a real issue that can wait, such as a non-critical workload with no immediate user impact. +* ``Noise`` — no meaningful business impact: false positives, test or staging workloads, and known-benign conditions. + +These rules are not editable from the UI. + +Alert Statuses +-------------- + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Status + - Meaning + * - **Not triaged** + - In the backlog. No investigation has run. + * - **Queued** + - Waiting for an agent to pick it up. + * - **Investigating** + - Holmes is running. + * - **Triaged** + - Findings are available, or the alert was linked to an existing incident. + * - **Skipped** + - Deliberately not investigated. The reason is shown next to the status. + +Skip reasons you may see: + +* ``daily limit reached`` — the rolling 24-hour budget is spent. The reason shows the count and the limit. +* ``no eligible cluster (external/unknown, no fallback)`` — no **Default agent** is set. +* ``cluster does not support realtime conversations`` — that cluster has no connected agent, or its agent is too old. +* ``auto-investigation disabled for the account`` — **Triage Automatically** was switched off after the alert had already been queued. + +Alerts attached to an existing incident by a matching rule are marked triaged without an investigation. See :doc:`/platform/incident-grouping`. + +Limitations +----------- + +* **Alerts that arrive while Triage Automatically is off are never queued.** They stay ``Not triaged`` in the backlog rather than being marked ``Skipped``. Investigate them manually to run them. +* **Only new alerts are triaged automatically.** A recurrence that reuses an existing alert record does not queue another automatic investigation. +* **The daily limit is a rolling 24-hour window**, not a calendar day, and it counts investigations started rather than investigations that succeeded. +* **The limit applies to automatic runs only.** Manual investigations are never blocked by it. +* **Two of the three setup checklist steps reflect the last 30 days.** *Connect an alert source* and *Investigate your first alert* are ticked from recent activity, not from whether the step was ever done. *Enable Auto Triage* reflects the current setting. diff --git a/docs/platform/incident-grouping.rst b/docs/platform/incident-grouping.rst new file mode 100644 index 000000000..864c99232 --- /dev/null +++ b/docs/platform/incident-grouping.rst @@ -0,0 +1,103 @@ +.. _incident-grouping: + +Incident Grouping +================== + +Robusta groups related alerts and investigations into **incidents**, so one underlying problem appears as a single item instead of many. + +Grouping is **AI-based**. Holmes compares a new finding against the incidents that are already open and decides whether it shares the same *root cause* — not merely the same alert name, resource, or symptom. Two different alerts caused by the same failing dependency belong to one incident; the same alert firing twice for two unrelated reasons belongs to two. + +Open **Alerts → Incidents** to browse them. + +What an Incident Contains +------------------------- + +Each incident gets a stable ``INC-###`` identifier and holds: + +* **Title** and **summary** of the underlying problem. +* **Likely cause** and **suggested fix**. +* **Priority**. +* **Workloads** and **environments** affected — these accumulate as more findings attach. +* The alerts and investigation runs attached to it. An attached run records why it was attached; an attached alert shows a reason only when an auto-attach rule matched it, in which case it is tagged **Rule-attached**. + +When an attaching finding adds something new — a further detail about the root cause, or a workload or environment the incident did not already cover — Holmes updates the summary so it still describes the whole problem. A routine recurrence that adds nothing new leaves the summary as it is; only the attached findings and the affected workload and environment lists grow. + +Enabling Incident Grouping +-------------------------- + +Grouping is off by default and is enabled separately for each source of findings: + +* **Alert triage** — **Alerts → Triage → Settings → Triage Settings → Create incidents from investigations**. The **Settings** button opens the **Pipeline Settings** dialog. +* **Triggered workflows** — the **Create incidents from this workflow** switch on each workflow. See :doc:`/platform/triggered-workflows`. + +Enabling it for one does not enable it for the other. + +Recurrences and Episodes +------------------------ + +An incident tracks **one problem, not a workload's whole history**. Holmes opens a new incident rather than reusing an old one when: + +* the previous incident was already resolved or the system recovered, +* days passed with no occurrences, or +* the investigation points at a different cause. + +So an OOMKill on Monday and an OOMKill on Wednesday on the same deployment may legitimately be two incidents if the causes differ. When Holmes is unsure, it opens a new incident — a wrong merge hides a real problem, while duplicates are easy to merge afterwards. + +Auto-Attaching Recurring Incidents +---------------------------------- + +When Holmes creates an incident it can also write a **matcher** — a simple rule describing which future events belong to that incident — so the same known failure is not investigated repeatedly. A later event matching the rule is attached immediately, with no AI investigation and without consuming your daily investigation budget. + +Matchers are opt-in per feature, under **Settings → Feature Flags**: + +* **AI Triage → Auto-attach recurring incidents** — for alert triage. +* **Triggered Workflows → Auto-attach recurring incidents** — for triggered workflows. + +.. note:: + + These are two separate flags with the same label, and they do not gate identically: + + * **Triggered workflows** — rule-based attaching also requires **Create incidents from this workflow** on the workflow itself. + * **Alert triage** — the feature flag alone is enough. Matching runs before any other check, so alerts keep attaching to existing incidents even with **Create incidents from investigations** turned off. With that switch off Holmes no longer writes new matchers from alert investigations, but matchers already stored on open incidents still match. + +Matchers are deliberately conservative: + +* A matcher is a fixed set of conditions on the event's fields — never executable code. +* Holmes is instructed to match on stable identity (alert name, namespace, workload, exact error signature) and never on per-firing values such as pod hashes, timestamps, or trace IDs. +* If a matcher does not fit an event, or if **more than one** incident matches, the event falls through to a normal AI investigation instead of guessing. +* Only the 50 most recently updated open incidents are scanned. Attaching a finding refreshes an incident's position, so actively recurring incidents stay in the window, but long-dormant open incidents can drop out of it. + +Alerts attached this way are marked as rule-attached. Clicking **Investigate** on such an alert also attaches it instantly rather than starting an investigation — that is expected. + +.. warning:: + + A matcher that is too broad will silently pull unrelated events into the wrong incident, and there is no re-evaluation to undo it. + + Where you can see the rule depends on the incident's source. On triggered-workflow incidents it is shown read-only in the **Auto-attach rule** panel on the incident page. On alert incidents it is not displayed; the only view of it is the rule named in each **Rule-attached** alert's tag. + + A matcher cannot be edited or deleted from the UI. To stop a bad one, close the incident — only open incidents are scanned — or turn off the corresponding **Auto-attach recurring incidents** flag. + +At Scale +-------- + +For AI grouping, Holmes is shown the most recently updated open incidents — up to 40, trimmed further to fit the prompt — with incidents from the same cluster listed first. When more exist, it is told to search the rest before concluding that a finding is new. Keeping resolved incidents closed keeps grouping accurate. + +Other Kinds of Grouping +----------------------- + +"Grouping" means different things at different layers of Robusta. They coexist and solve different problems: + +.. list-table:: + :widths: 30 70 + :header-rows: 1 + + * - Feature + - What it does + * - **Incident grouping** (this page) + - AI decides that findings share a root cause and merges them into a persistent ``INC-###`` incident with a summary, cause, and fix. Runs in the Robusta platform and works with any alert source. + * - :doc:`Notification grouping ` + - Threads Slack notifications together so a channel is not flooded. Runs in the in-cluster Robusta runner, is configured in Helm values, groups by fixed fields such as ``namespace`` or ``severity``, and does not analyze causes. + +Notification grouping is a delivery-layer feature and is not replaced by incident grouping; the two can be used together. + +Alerts that have not been linked to an incident may still be collapsed in the alert list as duplicates of one another. That duplicate detection is also AI-based — it is the first step of each investigation — but it only compares against a small set of recently investigated alerts, and it produces a pointer to a group leader rather than a persistent incident with a summary, cause, and fix. diff --git a/docs/platform/index.rst b/docs/platform/index.rst new file mode 100644 index 000000000..97c2635e9 --- /dev/null +++ b/docs/platform/index.rst @@ -0,0 +1,38 @@ +:hide-toc: + +Platform +========= + +.. toctree:: + :maxdepth: 1 + :hidden: + + alert-triage + incident-grouping + triggered-workflows + +The Robusta Platform is where your alerts are investigated, grouped, and acted on. These pages cover the AI features that run there. + +.. grid:: 1 1 2 3 + :gutter: 3 + + .. grid-item-card:: :octicon:`pulse;1em;` Alert Triage + :class-card: sd-bg-light sd-bg-text-light + :link: alert-triage + :link-type: doc + + AI investigates your alerts and returns a root cause, a fix, a priority, and an owning team + + .. grid-item-card:: :octicon:`organization;1em;` Incident Grouping + :class-card: sd-bg-light sd-bg-text-light + :link: incident-grouping + :link-type: doc + + Related alerts and investigations are grouped by root cause into a single incident + + .. grid-item-card:: :octicon:`cpu;1em;` Triggered Workflows + :class-card: sd-bg-light sd-bg-text-light + :link: triggered-workflows + :link-type: doc + + Run a saved Holmes investigation from any system that can send a webhook diff --git a/docs/platform/triggered-workflows.rst b/docs/platform/triggered-workflows.rst new file mode 100644 index 000000000..2ffb883f4 --- /dev/null +++ b/docs/platform/triggered-workflows.rst @@ -0,0 +1,248 @@ +.. _triggered-workflows: + +Triggered Workflows +==================== + +Triggered Workflows let you define a reusable Holmes investigation and run it from any system that can send a webhook — PagerDuty, Jira, Sentry, GitHub Actions, AlertManager, or your own scripts. + +Each workflow is a saved prompt plus a target agent. When a webhook arrives, Holmes runs the prompt against the incoming payload and delivers the findings wherever the prompt tells it to. + +How It Works +------------ + +1. You create a workflow in the Robusta UI and get back a webhook URL. +2. Your external system POSTs an event to that URL. +3. Holmes investigates using the tools enabled on the target agent. +4. Holmes delivers the report as instructed in your prompt — a Jira comment, a Slack message, an HTTP callback, or all three. + +Robusta does not ship a separate delivery mechanism. **Delivery instructions live in the prompt itself**, and Holmes uses whichever toolsets and MCP servers are already enabled on the agent. + +Creating a Workflow +------------------- + +In the Robusta UI, open **Automate → Triggered workflows** and click **Create workflow** — the **+** button at the top of the workflow list, or the **Create workflow** button on the intro screen. Start from a template or write your own. + +.. list-table:: + :widths: 25 10 65 + :header-rows: 1 + + * - Field + - Required + - Description + * - **Title** + - Yes + - Names the workflow and titles each investigation it creates. + * - **Prompt** + - Yes + - What Holmes should investigate. End it with a delivery instruction, e.g. *"post the findings as a comment on the same Jira issue"*. + * - **Enabled** + - No + - Defaults to on. A disabled workflow still accepts webhooks, but every event is skipped instead of investigated. + * - **Agent** + - Yes + - The cluster whose Holmes runs the investigation. Agents running an older Holmes are marked ``(outdated)`` and cannot be selected. + * - **AI Model** + - No + - Overrides the default model for this workflow. + * - **Max daily investigations** + - Yes + - Rolling 24-hour cap. Defaults to ``20``, which is also the maximum during beta. + * - **Create incidents from this workflow** + - No + - Groups repeated findings into incidents instead of re-investigating them. See :doc:`/platform/incident-grouping`. + * - **Filters** + - No + - Restricts the workflow to matching payloads. See `Filtering Events`_. + +Webhook URL +----------- + +Every workflow gets its own URL. Copy it from the workflow page under **Send Events**. + +.. robusta-code:: + + POST https://api.robusta.dev/webhooks?account_id=&workflow_id= + +Query Parameters +---------------- + +.. list-table:: + :widths: 20 70 + :header-rows: 1 + + * - Parameter + - Description + * - ``account_id`` + - Your Robusta account ID, found in ``generated_values.yaml``. + * - ``workflow_id`` + - The workflow to run. Repeat the parameter (``&workflow_id=a&workflow_id=b``) or comma-separate values (``&workflow_id=a,b``) to trigger several workflows from one event. Each value must be a valid UUID. + * - ``cluster`` + - Optional. Runs the investigation on this agent instead of the one configured on the workflow. See `Targeting a Cluster Per Request`_. + * - ``type`` + - Optional. Categorizes the event: ``alert``, ``incident``, ``change``, or ``event``. Defaults to ``event``, and any other value is rejected. It does not affect workflow routing — an event carrying a ``workflow_id`` always goes to workflows. See `Limitations`_. + * - ``token`` + - Optional. Your API key, as an alternative to the ``Authorization`` header. Ignored if an ``Authorization`` header is present. See the warning under `Authentication`_. + +Authentication +-------------- + +Send a Robusta API key with the ``Alerts: Write`` permission. Create one in the Robusta UI under **Settings → API Keys → New API Key**, and use the full ``hint.key`` string exactly as the UI displays it. + +.. code-block:: + + Authorization: Bearer + +The ``Authorization`` header is the preferred method. For systems that cannot set headers, append the key as a query parameter instead: + +.. robusta-code:: + + POST https://api.robusta.dev/webhooks?account_id=&workflow_id=&token= + +.. warning:: + + Use ``token`` only when the sending system cannot set headers. A key in the query string is recorded wherever URLs are — proxy and access logs, browser history, and observability tooling. Give such keys the ``Alerts: Write`` permission and nothing more, and rotate the key in **Settings → API Keys** if the URL is ever exposed. + +Example Request +--------------- + +.. robusta-code:: bash + + curl --location --request POST \ + 'https://api.robusta.dev/webhooks?account_id=ACCOUNT_ID&workflow_id=WORKFLOW_ID' \ + --header 'Authorization: Bearer API_KEY' \ + --header 'Content-Type: application/json' \ + --data-raw '{ "incident": { "severity": "P1", "title": "Checkout latency" } }' + +Response +-------- + +A successful request returns ``200`` with the ID of the stored event: + +.. code-block:: json + + { "id": "8f1b...e21" } + +Errors: + +* ``400`` — missing ``account_id``, invalid ``type``, or a ``workflow_id`` that is not a UUID. Malformed IDs are rejected before authentication, and no workflow in the request runs. +* ``401`` — missing, malformed, or unknown API key. +* ``403`` — valid API key without the ``Alerts: Write`` permission. +* ``429`` — rate limit exceeded (300 requests per 5-minute window per account). +* ``503`` — transient storage failure. Retry. + +.. warning:: + + ``200`` only means the event was accepted for processing. Workflows that are disabled, filtered out, over their daily limit, or referenced by an unknown ID are all reported as ``200`` and skipped later. Check the **Events** table on the workflow page to see what actually happened. + +.. _triggered-workflows-cluster-param: + +Targeting a Cluster Per Request +------------------------------- + +Every workflow has an **Agent** — the cluster whose Holmes runs the investigation. Add ``cluster=`` to the webhook URL to override it for a single request: + +.. robusta-code:: + + POST https://api.robusta.dev/webhooks?account_id=&workflow_id=&cluster=prod-us-east-1 + +This lets one workflow definition serve many clusters. Instead of duplicating a workflow per cluster, configure the webhook in each cluster's monitoring system with a different ``cluster`` value. + +* ``cluster`` always wins over the workflow's configured **Agent**. +* When ``cluster`` is omitted, the workflow's **Agent** is used. +* When neither is set, the event is skipped as ``misconfigured: missing cluster_name``. +* The value is the cluster name as it appears in the Robusta UI. It is not validated — a typo produces an investigation that never runs. + +The same parameter works on alert webhooks, where it sets the cluster of the resulting alert. See the :doc:`Send Events API `. + +Filtering Events +---------------- + +A workflow with no filter investigates every event it receives. To narrow it down, set both **Key** and **Value matches regex** under **Filters**. + +Robusta walks the payload and collects the value of every key with that exact name, at any nesting depth, including inside arrays. If any collected value matches the regex, the workflow runs. + +With key ``severity`` and regex ``^(P1|P2)$``: + +.. code-block:: json + + { "incident": { "severity": "P1" } } + +runs the workflow, while + +.. code-block:: json + + { "incident": { "severity": "P3" } } + +skips it. + +Filter behavior: + +* **Both fields are required together.** Setting only one blocks saving. +* **Key names match exactly and are case-sensitive.** There is no dot-path or wildcard syntax — ``severity`` matches a ``severity`` key anywhere in the payload, including one you did not intend. +* **The regex is unanchored.** ``P1`` also matches ``P10`` and ``NOT-P1``. Use ``^…$`` for an exact match. +* **Patterns are evaluated as Python regular expressions.** The form validates using the browser's regex engine, so a pattern that is valid in JavaScript but not in Python is silently treated as never matching. +* **Only scalar values are compared.** Booleans become ``true``/``false``, numbers their text form, ``null`` becomes ``null``. Objects and arrays never match. +* **A filtered workflow needs valid JSON.** If the body is not parseable JSON, there is nothing to walk and the workflow is skipped. + +Delivering Results +------------------ + +Tell Holmes how to deliver the report at the end of your prompt. For example: + +.. code-block:: text + + A Jira webhook payload is available as the trigger context. Read the issue key, + summary and description, investigate the described problem using the available + tools, and post the findings back as a comment on the same Jira issue. + Add a comment only; do not edit any other field. + +Holmes can only deliver through tools that are enabled on the target agent — Slack through your Robusta notification settings, Jira, PagerDuty and GitHub through their MCP servers, and so on. If the prompt has no delivery instruction, or the necessary integration is not enabled, the report exists only inside the run's conversation in the Robusta UI. + +Daily Investigation Limit +------------------------- + +Each workflow has a **Max daily investigations** cap over a rolling 24 hours, set to ``20`` by default. The cap counts only events that actually start an investigation; once it is reached, further events that would need one are skipped until the window rolls forward. Events that an incident matcher attaches to an existing incident do not consume the budget and are not blocked by it — see :doc:`/platform/incident-grouping`. + +The workflow page shows **Investigations Today** against the limit. + +Monitoring Runs +--------------- + +The workflow page shows total and daily investigation counts, a 7-day chart, the most recent run, and how many events were skipped. + +**Events** lists incoming webhooks, including ones that were not investigated, with an outcome, the time received, and a preview of the payload. Clicking a row opens the event's full payload; for events that produced an investigation, the row's **Investigation** link opens the full conversation, exactly like a Holmes chat. + +.. list-table:: + :widths: 35 65 + :header-rows: 1 + + * - Outcome + - Meaning + * - ``Investigated`` / ``Investigating`` + - The event produced a run. + * - ``Investigation failed`` / ``Investigation timed out`` / ``Investigation stopped`` + - The investigation started but did not finish successfully. + * - ``Attached to incident`` + - An auto-attach rule matched, so the event joined an existing incident instead of being investigated. Only appears when **Create incidents from this workflow** is on. See :doc:`/platform/incident-grouping`. + * - ``not found`` + - No workflow with that ID exists on this account. Usually a deleted workflow whose webhook is still configured somewhere. + * - ``disabled`` + - The workflow exists but is switched off. + * - ``filtered`` + - The payload did not match the workflow's filter. + * - ``misconfigured: missing cluster_name`` + - No **Agent** on the workflow and no ``cluster`` parameter on the request. + * - ``LLM investigation limit reached`` + - The daily cap was hit. The note shows the count and the limit. + * - ``Parse error`` / ``Did not investigate`` + - The event could not be parsed, or produced no run and no specific skip reason. + +When webhooks arrive for a workflow that no longer exists, a red **N events with no workflow** entry appears at the top of the workflow list and opens a view of them. It covers only events received in the last 24 hours. + +Limitations +----------- + +* **A webhook with a** ``workflow_id`` **is not parsed as an alert.** Even with ``type=alert``, an event carrying a workflow ID is only fanned out to workflows — no alert or issue is created from it. Use two separate webhook calls if you need both. +* **Investigations are shared and unattributed.** Runs do not use the creator's personal integrations, and every account member with access to the cluster can see them. +* **Deleting a workflow deletes its investigations.** Webhooks still pointing at that ID then show up as ``not found``. +* **The Events table shows only the most recent 200 events, and the page loads only the most recent 100 investigations.** That capped run list drives the 7-day chart and the **Last Investigation** and **Investigations Today** boxes, while **Total Investigations** and **Events Skipped** are exact. On busy workflows the numbers will not agree.