ROB-669: Add Platform docs for Alert Triage, Incident Grouping and Triggered Workflows

docs/configuration/exporting/robusta-pro-features.rst

@@ -19,6 +19,7 @@ Data Export and Reporting
 * :doc:`Alert Reporting API <alert-statistics-api>`: Get aggregated statistics and counts for different alert types
 * :doc:`Send Events API <send-events-api>`: Send alerts, incidents, and changes from any monitoring source via a single webhook endpoint
 * :doc:`RBAC Configuration API <rbac-api>`: Programmatically manage role-based access control configurations
+* :doc:`Triggered Workflows </platform/triggered-workflows>`: Run a saved Holmes investigation from any system that can send a webhook
 
 Getting Started
 ---------------

docs/configuration/exporting/send-events-api.rst

@@ -5,6 +5,8 @@ Send alerts from your monitoring system to Robusta through a single webhook endp
 
 This is the recommended ingestion path for new integrations. The legacy :doc:`Send Alerts API </configuration/exporting/send-alerts-api>` remains available for existing customers.
 
+Alerts ingested here feed :doc:`Alert Triage </platform/alert-triage>`, which investigates them with AI and can group related alerts into :doc:`incidents </platform/incident-grouping>`.
+
 .. toctree::
    :maxdepth: 1
    :hidden:
@@ -49,7 +51,7 @@ Query Parameters
    * - ``account_id``
      - Your Robusta account ID, found in ``generated_values.yaml``.
    * - ``cluster``
-     - Optional. The cluster to associate the alert with. When set, it overrides any cluster found in the alert payload and is used for the resulting alert investigation. When omitted, the cluster is taken from the payload if present, otherwise the alert is recorded under the ``external`` cluster. Use this when your monitoring system cannot add a cluster label to the alert itself.
+     - Optional. The cluster to associate the alert with. When set, it overrides any cluster found in the alert payload and is used for the resulting alert investigation. When omitted, the cluster is taken from the payload if present, otherwise the alert is recorded under the ``external`` cluster. Use this when your monitoring system cannot add a cluster label to the alert itself. The same parameter works on Triggered Workflows — see :ref:`Targeting a Cluster Per Request <triggered-workflows-cluster-param>`.
 
 Authentication
 --------------

docs/how-it-works/architecture.rst

@@ -27,7 +27,7 @@ The Robusta Platform is a centralized place to control your SRE agents and chat
 The Platform receives investigation results from HolmesGPT and provides:
 
 - AI-powered root cause analysis results for every alert
-- Centralized alert management and triage
+- Centralized alert management and :doc:`triage </platform/alert-triage>`
 - Historical alert analysis and timelines
 - HolmesGPT Slack and Teams bot — tag the AI agent like a teammate to investigate issues on demand
 

docs/how-it-works/oss-vs-saas.rst

@@ -38,7 +38,7 @@ The Robusta Platform (SaaS or Self-Hosted) adds on top:
 
 - A web interface
 - HolmesGPT bots for Slack and Microsoft Teams
-- Automatic alert triage and grouping
+- Automatic :doc:`alert triage </platform/alert-triage>` and :doc:`incident grouping </platform/incident-grouping>`
 - Background agents that proactively investigate and surface issues
 
 Robusta Classic

docs/index.rst

@@ -16,6 +16,16 @@
 
    setup-robusta/index
 
+.. toctree::
+   :maxdepth: 4
+   :caption: Platform
+   :hidden:
+
+   Overview <platform/index>
+   platform/alert-triage
+   platform/incident-grouping
+   platform/triggered-workflows
+
 .. toctree::
    :maxdepth: 4
    :caption: HTTP APIs
@@ -67,7 +77,7 @@ Robusta is an AI-powered SRE agent that automatically investigates alerts and fi
 
 **How it works:**
 
-* **Automatic investigation** — every alert is analyzed with AI-powered root cause analysis
+* **Automatic investigation** — every alert is analyzed with AI-powered root cause analysis. See :doc:`Alert Triage <platform/alert-triage>`.
 * **Your data sources** — HolmesGPT connects to your existing monitoring, ITSM, cloud tools, and MCP servers to gather evidence
 * **Chat with your agent** — tag HolmesGPT in Slack or Teams to investigate issues on demand
 * **Centralized control** — the `Robusta Platform <https://home.robusta.dev>`_ gives you a single place to manage your SRE agents, triage alerts, and review investigation timelines

docs/notification-routing/notification-grouping.rst

@@ -5,6 +5,10 @@ Notification Grouping (Slack Only)
 
 Reduce alert noise by grouping related notifications into Slack threads. Instead of flooding channels with individual alerts, Robusta creates summary messages with threaded details.
 
+.. note::
+
+    This page covers grouping of **Slack messages** by the in-cluster Robusta runner, using fixed fields such as ``namespace`` or ``severity``. To group alerts by their **root cause** into incidents, see :doc:`/platform/incident-grouping`. The two are independent and can be used together.
+
 .. image:: /images/notification-grouping.png
    :width: 600px
    :align: center

docs/platform/alert-triage.rst

@@ -0,0 +1,154 @@
+.. _alert-triage:
+
+Alert Triage
+=============
+
+Alert Triage investigates your alerts with AI and tells you which ones matter. For each alert, Holmes gathers evidence from your connected data sources and returns a root cause, a suggested fix, a priority, and the team that should own it.
+
+Open **Alerts → Triage** in the Robusta UI to see triaged alerts and the backlog of alerts waiting to be investigated.
+
+The Triage Pipeline
+-------------------
+
+Alerts move through the pipeline shown at the top of the Triage page:
+
+1. **Alert sources** — alerts arrive from AlertManager, Datadog, PagerDuty, and other sources via the :doc:`Send Events API </configuration/exporting/send-events-api>`.
+2. **Queued** — a new alert is queued for investigation.
+3. **Investigating** — Holmes is running.
+4. **Triage** — Holmes has written its findings back to the alert.
+5. **Priority** — the alert is bucketed as Urgent, High, Medium, or Low.
+6. **Destinations** — the findings are delivered wherever you configured.
+
+**Queued**, **Investigating**, **Backlog**, and the **Priority** buckets are clickable filters. Clicking **Alert sources**, **Triage**, or **Destinations** instead opens **Pipeline Settings** on the matching tab.
+
+What Holmes Produces
+--------------------
+
+Every completed investigation records:
+
+* **What happened** — a summary and a short timeline of events.
+* **Root cause analysis** — the reasoning and evidence behind the conclusion.
+* **How to fix** — the recommended remediation.
+* **Priority** — Holmes classifies urgency internally as ``Urgent``, ``Not Urgent``, ``Noise``, or ``Duplicate``. The UI surfaces this as the **Priority** column: ``Urgent``, ``High``, ``Medium``, or ``Low``. When the alert is linked to an incident, the incident's own priority is shown instead.
+* **Team** — the team that should own the alert, with the reason for the assignment.
+
+Holmes checks for duplicates first. When an alert is a duplicate of one it already investigated, it says so and stops instead of repeating the work.
+
+Enabling Automatic Triage
+-------------------------
+
+Automatic triage is **off by default**. Turn it on under **Alerts → Triage → Settings → Triage Settings** — the **Settings** button in the page header opens the **Pipeline Settings** dialog.
+
+.. list-table::
+   :widths: 30 15 55
+   :header-rows: 1
+
+   * - Setting
+     - Default
+     - Description
+   * - **Triage Automatically**
+     - ``Off``
+     - Investigates every newly firing alert without anyone clicking. When off, you can still investigate alerts manually.
+   * - **Daily auto-investigation limit**
+     - ``20``
+     - Maximum automatic investigations in a rolling 24-hour window. ``20`` is also the maximum during beta.
+   * - **Default agent**
+     - *(none)*
+     - The agent that investigates alerts with no cluster of their own. See `Choosing the Agent`_.
+   * - **Create incidents from investigations**
+     - ``Off``
+     - Lets investigations group related alerts into incidents. See :doc:`/platform/incident-grouping`.
+
+.. note::
+
+    **Create incidents from investigations** is independent of **Triage Automatically**. With automatic triage off, manual investigations still create and update incidents.
+
+Investigating Manually
+----------------------
+
+Click **Investigate** on any alert, or select several and investigate them in bulk. Manual investigations ignore both the **Triage Automatically** switch and the daily limit, and they do not count toward it. They still need an eligible agent.
+
+**Re-investigate** discards the previous findings and runs again. While it runs, the alert temporarily moves back to the backlog.
+
+Choosing the Agent
+------------------
+
+Investigations run on the agent belonging to the alert's cluster. Two cases need a **Default agent**:
+
+* Alerts that arrive without a cluster are recorded under the cluster ``external``.
+* Alerts whose cluster has no Holmes agent connected.
+
+In both cases Robusta reroutes the investigation to the **Default agent**. Set one if alerts are queued but never investigated.
+
+The default agent only decides *where* the investigation runs — the alert keeps its own cluster for display and filtering.
+
+.. warning::
+
+    With no **Default agent** configured, these alerts are skipped rather than left queued, and the two cases record different reasons:
+
+    * Alerts with no cluster of their own are marked ``Skipped`` with the reason ``no eligible cluster (external/unknown, no fallback)``.
+    * Alerts on a cluster with no connected agent are marked ``Skipped`` with the reason ``cluster <name> does not support realtime conversations``.
+
+The second reason is also what you get when the agent on that cluster is running a Holmes too old to support realtime conversations. Check whether the cluster has a connected agent before upgrading anything.
+
+Tailoring the Results
+---------------------
+
+Under **Pipeline Settings**:
+
+* **Triage Settings** — the switches above.
+* **Teams** — the teams Holmes may assign alerts to, each with a description. Without any, everything is assigned to a synthetic ``General`` team.
+* **Destinations** — a two-step setup, and both steps are required:
+
+  1. **Give Holmes a messaging tool.** The tab shows the connection status of **Slack** and **Microsoft Teams**. Any other MCP server or data source that can send messages works too, and is configured under Data Sources.
+  2. **Tell Holmes what to do after the investigation.** Free-text instructions appended to every investigation prompt, for both automatic and manual runs — which channel to notify and when, or extra context about your environment.
+
+.. warning::
+
+    Instructions alone deliver nothing. With no messaging tool connected, Holmes has no way to send the findings and reports no error.
+
+Urgency classification follows a built-in set of rules:
+
+* ``Urgent`` — active customer impact, data loss risk, or security exposure.
+* ``Not Urgent`` — a real issue that can wait, such as a non-critical workload with no immediate user impact.
+* ``Noise`` — no meaningful business impact: false positives, test or staging workloads, and known-benign conditions.
+
+These rules are not editable from the UI.
+
+Alert Statuses
+--------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Status
+     - Meaning
+   * - **Not triaged**
+     - In the backlog. No investigation has run.
+   * - **Queued**
+     - Waiting for an agent to pick it up.
+   * - **Investigating**
+     - Holmes is running.
+   * - **Triaged**
+     - Findings are available, or the alert was linked to an existing incident.
+   * - **Skipped**
+     - Deliberately not investigated. The reason is shown next to the status.
+
+Skip reasons you may see:
+
+* ``daily limit reached`` — the rolling 24-hour budget is spent. The reason shows the count and the limit.
+* ``no eligible cluster (external/unknown, no fallback)`` — no **Default agent** is set.
+* ``cluster <name> does not support realtime conversations`` — that cluster has no connected agent, or its agent is too old.
+* ``auto-investigation disabled for the account`` — **Triage Automatically** was switched off after the alert had already been queued.
+
+Alerts attached to an existing incident by a matching rule are marked triaged without an investigation. See :doc:`/platform/incident-grouping`.
+
+Limitations
+-----------
+
+* **Alerts that arrive while Triage Automatically is off are never queued.** They stay ``Not triaged`` in the backlog rather than being marked ``Skipped``. Investigate them manually to run them.
+* **Only new alerts are triaged automatically.** A recurrence that reuses an existing alert record does not queue another automatic investigation.
+* **The daily limit is a rolling 24-hour window**, not a calendar day, and it counts investigations started rather than investigations that succeeded.
+* **The limit applies to automatic runs only.** Manual investigations are never blocked by it.
+* **Two of the three setup checklist steps reflect the last 30 days.** *Connect an alert source* and *Investigate your first alert* are ticked from recent activity, not from whether the step was ever done. *Enable Auto Triage* reflects the current setting.

docs/platform/incident-grouping.rst

@@ -0,0 +1,103 @@
+.. _incident-grouping:
+
+Incident Grouping
+==================
+
+Robusta groups related alerts and investigations into **incidents**, so one underlying problem appears as a single item instead of many.
+
+Grouping is **AI-based**. Holmes compares a new finding against the incidents that are already open and decides whether it shares the same *root cause* — not merely the same alert name, resource, or symptom. Two different alerts caused by the same failing dependency belong to one incident; the same alert firing twice for two unrelated reasons belongs to two.
+
+Open **Alerts → Incidents** to browse them.
+
+What an Incident Contains
+-------------------------
+
+Each incident gets a stable ``INC-###`` identifier and holds:
+
+* **Title** and **summary** of the underlying problem.
+* **Likely cause** and **suggested fix**.
+* **Priority**.
+* **Workloads** and **environments** affected — these accumulate as more findings attach.
+* The alerts and investigation runs attached to it. An attached run records why it was attached; an attached alert shows a reason only when an auto-attach rule matched it, in which case it is tagged **Rule-attached**.
+
+When an attaching finding adds something new — a further detail about the root cause, or a workload or environment the incident did not already cover — Holmes updates the summary so it still describes the whole problem. A routine recurrence that adds nothing new leaves the summary as it is; only the attached findings and the affected workload and environment lists grow.
+
+Enabling Incident Grouping
+--------------------------
+
+Grouping is off by default and is enabled separately for each source of findings:
+
+* **Alert triage** — **Alerts → Triage → Settings → Triage Settings → Create incidents from investigations**. The **Settings** button opens the **Pipeline Settings** dialog.
+* **Triggered workflows** — the **Create incidents from this workflow** switch on each workflow. See :doc:`/platform/triggered-workflows`.
+
+Enabling it for one does not enable it for the other.
+
+Recurrences and Episodes
+------------------------
+
+An incident tracks **one problem, not a workload's whole history**. Holmes opens a new incident rather than reusing an old one when:
+
+* the previous incident was already resolved or the system recovered,
+* days passed with no occurrences, or
+* the investigation points at a different cause.
+
+So an OOMKill on Monday and an OOMKill on Wednesday on the same deployment may legitimately be two incidents if the causes differ. When Holmes is unsure, it opens a new incident — a wrong merge hides a real problem, while duplicates are easy to merge afterwards.
+
+Auto-Attaching Recurring Incidents
+----------------------------------
+
+When Holmes creates an incident it can also write a **matcher** — a simple rule describing which future events belong to that incident — so the same known failure is not investigated repeatedly. A later event matching the rule is attached immediately, with no AI investigation and without consuming your daily investigation budget.
+
+Matchers are opt-in per feature, under **Settings → Feature Flags**:
+
+* **AI Triage → Auto-attach recurring incidents** — for alert triage.
+* **Triggered Workflows → Auto-attach recurring incidents** — for triggered workflows.
+
+.. note::
+
+    These are two separate flags with the same label, and they do not gate identically:
+
+    * **Triggered workflows** — rule-based attaching also requires **Create incidents from this workflow** on the workflow itself.
+    * **Alert triage** — the feature flag alone is enough. Matching runs before any other check, so alerts keep attaching to existing incidents even with **Create incidents from investigations** turned off. With that switch off Holmes no longer writes new matchers from alert investigations, but matchers already stored on open incidents still match.
+
+Matchers are deliberately conservative:
+
+* A matcher is a fixed set of conditions on the event's fields — never executable code.
+* Holmes is instructed to match on stable identity (alert name, namespace, workload, exact error signature) and never on per-firing values such as pod hashes, timestamps, or trace IDs.
+* If a matcher does not fit an event, or if **more than one** incident matches, the event falls through to a normal AI investigation instead of guessing.
+* Only the 50 most recently updated open incidents are scanned. Attaching a finding refreshes an incident's position, so actively recurring incidents stay in the window, but long-dormant open incidents can drop out of it.
+
+Alerts attached this way are marked as rule-attached. Clicking **Investigate** on such an alert also attaches it instantly rather than starting an investigation — that is expected.
+
+.. warning::
+
+    A matcher that is too broad will silently pull unrelated events into the wrong incident, and there is no re-evaluation to undo it.
+
+    Where you can see the rule depends on the incident's source. On triggered-workflow incidents it is shown read-only in the **Auto-attach rule** panel on the incident page. On alert incidents it is not displayed; the only view of it is the rule named in each **Rule-attached** alert's tag.
+
+    A matcher cannot be edited or deleted from the UI. To stop a bad one, close the incident — only open incidents are scanned — or turn off the corresponding **Auto-attach recurring incidents** flag.
+
+At Scale
+--------
+
+For AI grouping, Holmes is shown the most recently updated open incidents — up to 40, trimmed further to fit the prompt — with incidents from the same cluster listed first. When more exist, it is told to search the rest before concluding that a finding is new. Keeping resolved incidents closed keeps grouping accurate.
+
+Other Kinds of Grouping
+-----------------------
+
+"Grouping" means different things at different layers of Robusta. They coexist and solve different problems:
+
+.. list-table::
+   :widths: 30 70
+   :header-rows: 1
+
+   * - Feature
+     - What it does
+   * - **Incident grouping** (this page)
+     - AI decides that findings share a root cause and merges them into a persistent ``INC-###`` incident with a summary, cause, and fix. Runs in the Robusta platform and works with any alert source.
+   * - :doc:`Notification grouping </notification-routing/notification-grouping>`
+     - Threads Slack notifications together so a channel is not flooded. Runs in the in-cluster Robusta runner, is configured in Helm values, groups by fixed fields such as ``namespace`` or ``severity``, and does not analyze causes.
+
+Notification grouping is a delivery-layer feature and is not replaced by incident grouping; the two can be used together.
+
+Alerts that have not been linked to an incident may still be collapsed in the alert list as duplicates of one another. That duplicate detection is also AI-based — it is the first step of each investigation — but it only compares against a small set of recently investigated alerts, and it produces a pointer to a group leader rather than a persistent incident with a summary, cause, and fix.