From c3b9cd9d62b3011a97fa1b42ceb1e9fa83e493b8 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Wed, 2 Sep 2026 13:38:03 +0100 Subject: [PATCH] Publish the 1 September matrix: 101/114, one execution date MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first clean full matrix under the current base prompt. 114 cells, six experiments, nineteen scenarios, every row executed on 1 September — which is what #66 asks for before publishing resumes, and the export confirms it: no carried rows, no provenance warnings, one execution date across the file. Replaces a snapshot from 24 August that was measured under the previous base prompt and produced by a local sweep rather than a CI run, so the published numbers now link to a workflow run a reader can open. Published by hand from the run's own artifacts rather than by the workflow. `publish-results` was skipped on 1 September because `EVALS_PUBLISH` was `false`; re-running that job today exported correctly — the log shows the same 114 rows and the same 101/13 split — and then failed to push. A re-run checks out the SHA the original run used, and main has moved six merges since, so the push was a non-fast-forward and the rebase-and-retry loop did not recover it. The artifacts are the same either way; only the commit path differs. Filed separately. What the run says: claude-code-sonnet-5 19/19 codex-gpt-5.6 18/19 codex-gpt-5.6-no-skills 18/19 claude-code-sonnet-5-no-skills 17/19 codex-gpt-5.4-mini-no-skills 15/19 codex-gpt-5.4-mini 14/19 Skills delta +2, 0 and -1. The weak model's figure has now read -3, -2 and -1 on successive clean measurements, so the sign has replicated and the magnitude has not. Thirteen failures, and the largest cluster is real: four of six arms failed `verification-001-stripe-express`, three of them on the same check. Triaged before publishing, because that is the shape a scorer defect makes. It is not one — exactly one run configured the Hookdeck source with the secret sitting in its own `.env`, and the four that failed used placeholders like `whsec_xxx` and `whsec_replace_me`. That is the failure this scenario was built to catch. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK --- apps/web/src/data/eval-results.json | 14300 ++++-------------- results/index.json | 10 + results/latest.json | 14306 ++++--------------- results/runs/2026-09-01T14-10-11-000Z.json | 8438 +++++++++++ results/totals.json | 8 +- 5 files changed, 14615 insertions(+), 22447 deletions(-) create mode 100644 results/runs/2026-09-01T14-10-11-000Z.json diff --git a/apps/web/src/data/eval-results.json b/apps/web/src/data/eval-results.json index 66f7739..f972f26 100644 --- a/apps/web/src/data/eval-results.json +++ b/apps/web/src/data/eval-results.json @@ -9,7 +9,8 @@ "reasoningEffort": "high" }, "eval": "benchmark-alerting-001-delivery-alerts", - "ranAt": "2026-08-19T14:18:38.238Z", + "ranAt": "2026-09-01T08:05:52.679Z", + "runId": "33484972784", "stage": "build", "product": [ "event-gateway" @@ -18,15 +19,7 @@ "alerting" ], "suite": "benchmark", - "skills": { - "available": [ - "hookdeck", - "event-gateway" - ], - "loaded": [ - "event-gateway" - ] - }, + "gatedBy": "discovery", "passed": true, "checks": [ { @@ -46,465 +39,92 @@ "passed": true } ], - "attempts": 1, + "skills": { + "available": [ + "hookdeck", + "event-gateway" + ], + "loaded": [ + "event-gateway" + ] + }, "docs": { "calls": [ { "source": "shell_fetch", - "query": "curl -s https://hookdeck.com/docs/api/issue-triggers.md", + "query": "curl -s https://hookdeck.com/docs/issue-triggers.md 2>&1 | head -c 6000", + "pages": [ + { + "url": "https://hookdeck.com/docs/issue-triggers.md" + } + ], + "resultChars": 6000 + }, + { + "source": "shell_fetch", + "query": "curl -s https://hookdeck.com/docs/api/issue-triggers.md 2>&1 | head -c 8000", "pages": [ { "url": "https://hookdeck.com/docs/api/issue-triggers.md" } ], - "resultChars": 66518 + "resultChars": 8000 + }, + { + "source": "shell_fetch", + "query": "curl -s https://hookdeck.com/docs/api/notifications.md 2>&1 | head -c 6000", + "pages": [ + { + "url": "https://hookdeck.com/docs/api/notifications.md" + } + ], + "resultChars": 6000 }, { "source": "shell_fetch", - "query": "curl -s https://hookdeck.com/docs/issue-triggers.md 2>&1 | head -300", + "query": "curl -s https://hookdeck.com/docs/issue-triggers.md 2>&1 | sed -n '150,320p'", "pages": [ { "url": "https://hookdeck.com/docs/issue-triggers.md" } ], - "resultChars": 9388 + "resultChars": 3032 }, { "source": "shell_fetch", - "query": "echo \"=== existing issue triggers ===\"; curl -s https://api.hookdeck.com/2025-07-01/issue-triggers -H \"Authorization: Bearer $HOOKDECK_API_KEY\" | head -c 3000\necho\necho \"=== notifications docs ===\"\ncurl -s https://hookdeck.com/docs/api/notifications.md 2>&1 | head -c 100\necho\necho \"=== issues doc channels section ===\"\ncurl -s https://hookdeck.com/docs/issues.md 2>&1 | grep -n -i -A5 \"channel\"", + "query": "curl -s https://hookdeck.com/docs/issue-triggers.md 2>&1 | sed -n '110,150p'", "pages": [ { - "url": "https://hookdeck.com/docs/api/notifications.md" - }, + "url": "https://hookdeck.com/docs/issue-triggers.md" + } + ], + "resultChars": 962 + }, + { + "source": "shell_fetch", + "query": "curl -s https://hookdeck.com/docs/issues.md 2>&1 | head -c 5000", + "pages": [ { "url": "https://hookdeck.com/docs/issues.md" } ], - "resultChars": 2272 + "resultChars": 4999 }, { "source": "shell_fetch", - "query": "curl -s https://api.hookdeck.com/2025-07-01/issue-triggers -H \"Authorization: Bearer $HOOKDECK_API_KEY\"\necho\necho \"=== project notification settings ===\"\ncurl -s https://hookdeck.com/docs/issues.md 2>&1 | sed -n '1,40p'", + "query": "curl -s https://hookdeck.com/docs/issues.md 2>&1 | sed -n '/Configure notifications/,/^## /p' | head -c 3000", "pages": [ { "url": "https://hookdeck.com/docs/issues.md" } ], - "resultChars": 3193 + "resultChars": 3000 } ] }, - "toolCalls": [ - { - "endpoint": "Skill", - "body": { - "skill": "event-gateway" - }, - "name": "unknown", - "loadedSkills": [ - "event-gateway" - ], - "result": "Launching skill: event-gateway", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "hookdeck whoami", - "description": "Show current Hookdeck org/project context" - }, - "name": "shell", - "command": "hookdeck whoami", - "error": "Exit code 1\nYou aren't authenticated yet.\n\nNo terminal is attached, so browser sign-in cannot run.\n\nAuthenticate without a terminal using one of:\n hookdeck ci --api-key (or set HOOKDECK_API_KEY)\n hookdeck login --cli-key \n\nOr run `hookdeck login` in an interactive terminal.", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "echo \"HOOKDECK_API_KEY set: ${HOOKDECK_API_KEY:+yes}\"; hookdeck ci --api-key \"$HOOKDECK_API_KEY\" 2>&1; hookdeck whoami 2>&1", - "description": "Authenticate CLI with API key and show identity" - }, - "name": "shell", - "command": "echo \"HOOKDECK_API_KEY set: ${HOOKDECK_API_KEY:+yes}\"; hookdeck ci --api-key \"$HOOKDECK_API_KEY\" 2>&1; hookdeck whoami 2>&1", - "result": "HOOKDECK_API_KEY set: yes\ntime=\"Wed, 19 Aug 2026 14:15:41 UTC\" level=info msg=\"The Hookdeck CLI is configured on project evals-ci in organization Automated Testing\n\"\n\nUsing profile default (use -p flag to use a different config profile)\n\nLogged in as 8e127ae8abb5 () on project evals-ci in organization Automated Testing\nProject type: Gateway", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "hookdeck gateway connection list 2>&1", - "description": "List connections in project" - }, - "name": "shell", - "command": "hookdeck gateway connection list 2>&1", - "result": "Found 1 connection(s):\n\norders-to-fulfilment\n ID: web_BFL5GHb0FGi4\n Source: orders (src_8daq3ap8g6nc7g) [WEBHOOK]\n Destination: fulfilment-api (des_ilE1tIK8Koli) [HTTP]\n Status: active", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "cat /tmp/sandbox-9ec775af/.claude/skills/event-gateway/references/monitoring-debugging.md 2>&1 | head -200", - "description": "Read monitoring/debugging reference" - }, - "name": "shell", - "command": "cat /tmp/sandbox-9ec775af/.claude/skills/event-gateway/references/monitoring-debugging.md 2>&1 | head -200", - "result": "# Monitoring & Debugging\n\n## Contents\n\n- [Before querying or running metrics](#before-querying-or-running-metrics)\n- [Monitoring](#monitoring)\n - [Event Lifecycle](#event-lifecycle)\n - [Data Model](#data-model)\n - [Event Statuses](#event-statuses)\n - [Where to look](#where-to-look)\n- [Debugging](#debugging)\n - [Troubleshooting Flowchart](#troubleshooting-flowchart)\n - [Issues and Notifications](#issues-and-notifications)\n - [Replay](#replay)\n- [CLI: list and inspect](#cli-querying)\n- [CLI: metrics](#cli-metrics)\n- [Documentation](#documentation)\n\nHow to monitor webhook deliveries, debug failures, and replay events. For monitoring and debugging, the **CLI** is usually the right first touch (TUI, list/get, metrics, retry). The **API** is valid for querying and for automation driven from application or test code. See **[SKILL.md — Development & Operations](../SKILL.md#development--operations)** for full CLI vs API vs Dashboard guidance. Querying (events, requests, attempts, metrics) can also be done via the API — see the [API inspect docs](https://hookdeck.com/docs/api/inspect.md).\n\n**Prerequisite:** Before any `hookdeck …` commands in this file, [Hookdeck CLI installed](cli-workflows.md#hookdeck-cli-prerequisite).\n\n## Before querying or running metrics {#before-querying-or-running-metrics}\n\nAll list, inspect, and metrics commands are scoped to the **current [organization and project](https://hookdeck.com/docs/projects)** (organization = top-level account; project = where your sources, connections, and destinations live).\n\n- [ ] Run `hookdeck whoami` and **show the user the output**\n- [ ] **Unless** the user has very clearly identified org/project and whoami is an exact match, ask them to confirm before running queries or metrics\n- [ ] If not correct, use `hookdeck project list` to see options, then `hookdeck project use `\n- [ ] Run `hookdeck whoami` again, show the output, and (unless clear match) ask the user to confirm again before running queries or metrics\n\n**Common mistake:** Running a metrics query without verifying org/project — data can come from the wrong organization (e.g. \"Automated Testing\" instead of \"prod\"). **Correct approach:** Run `hookdeck whoami` and show the output. If the user hasn't clearly specified org/project or it doesn't match, ask them to confirm; if not correct, switch and confirm again. Only then run the query.\n\n## Monitoring {#monitoring}\n\nEvent lifecycle, data model, event statuses, and where to look (CLI TUI, Web Console, Dashboard).\n\n### Event Lifecycle\n\n1. Webhook provider sends a request to your Source URL (`https://hkdk.events/xxx`)\n2. Hookdeck creates a [Request](https://hookdeck.com/docs/requests) (the raw inbound data)\n3. Each Connection on that Source creates an [Event](https://hookdeck.com/docs/events) (processed, routed delivery)\n4. Hookdeck delivers the Event to the Destination, creating an [Attempt](https://hookdeck.com/docs/events)\n5. If delivery fails (non-2xx), Hookdeck retries according to the Connection's retry rules\n\n### Data Model\n\n| Entity | What it is |\n|--------|------------|\n| [Request](https://hookdeck.com/docs/requests) | Raw inbound webhook received by a Source |\n| [Event](https://hookdeck.com/docs/events) | Processed delivery routed through a Connection |\n| [Attempt](https://hookdeck.com/docs/events) | A single delivery try to a Destination |\n\nOne Request can produce multiple Events (fan-out). Each Event has one or more Attempts.\n\n### Event Statuses\n\n| Status | Meaning |\n|--------|---------|\n| `SUCCESSFUL` | Destination returned a 2xx response |\n| `FAILED` | All retry attempts exhausted, destination still returning non-2xx |\n| `QUEUED` | Scheduled for delivery or retry |\n| `HOLD` | Paused by a delay rule or manual hold |\n\n### Where to look\n\n#### CLI TUI\n\nBuilt into `hookdeck listen`. View events in real time, inspect request/response details, replay events -- all in the terminal.\n\n#### Web Console\n\nWhen not using an Event Gateway project, the CLI provides a link to a web interface for event inspection and replay.\n\n#### Dashboard\n\nWhen using an Event Gateway project, the [Hookdeck Dashboard](https://dashboard.hookdeck.com) provides:\n\n- **Events view**: Inspect Event payloads, headers, response, and Attempts\n- **Requests view**: See raw inbound webhooks before routing\n- **Issues**: Track patterns of failures (delivery issues, transformation errors, backpressure)\n- **Replay**: Re-deliver individual Events or bulk replay filtered sets\n- **Bookmarks**: Save representative Requests for repeated testing\n\n## Debugging {#debugging}\n\nTroubleshooting flowchart, issues and notifications, and replay.\n\n### Troubleshooting Flowchart\n\n**Events not arriving?**\n\n1. Is the provider sending to the correct Source URL?\n2. Is `hookdeck listen` running?\n3. If Source Authentication is configured, are the credentials correct? Failed auth returns `403` and no Event is created.\n\n**Events arriving but delivery failing?**\n\n1. Check the Attempt response in the CLI TUI or web UI\n2. Is your handler running and listening on the correct port?\n3. Is the endpoint path correct (e.g., `/webhooks` vs `/webhook`)?\n4. Is the handler returning a 2xx status? Non-2xx triggers retries.\n\n**Signature verification failing in your handler?**\n\n1. Is `HOOKDECK_WEBHOOK_SECRET` set correctly?\n2. Are you using the raw request body (not parsed JSON)?\n3. Are you using base64 encoding (not hex)?\n4. See [verification-code.md](verification-code.md) for debugging code\n\n### Issues and Notifications\n\n[Issues](https://hookdeck.com/docs/issues) are automatically created when problems are detected:\n\n| Issue type | Trigger |\n|------------|---------|\n| Delivery | Consecutive failed delivery attempts |\n| Transformation | JavaScript errors in transform rules |\n| Backpressure | Events queuing faster than they can be delivered |\n\nIssue lifecycle: `OPENED` -> `ACKNOWLEDGED` -> `RESOLVED` or `IGNORED`\n\n[Issue triggers](https://hookdeck.com/docs/issue-triggers) control what creates issues. [Notifications](https://hookdeck.com/docs/issue-triggers) can be sent to Email, Slack, PagerDuty, Microsoft Teams, or a webhook.\n\n### Replay\n\n**Single event replay**: Use the CLI TUI, web console, or Dashboard to replay any event.\n\n**Bulk retry via API**:\n\n```sh\ncurl -X POST https://api.hookdeck.com/$API_VERSION/bulk/events/retry \\\n -H \"Authorization: Bearer $HOOKDECK_API_KEY\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"query\": {\"webhook_id\": \"web_xxx\", \"status\": \"FAILED\"}}'\n```\n\n**[Bookmarks](https://hookdeck.com/docs/bookmarks)**: Save a Request as a bookmark in the Dashboard, then replay it whenever you need to re-test.\n\n## CLI: list and inspect {#cli-querying}\n\nWhen to use: detailed search, inspecting or retrying individual request, event, or attempt. From the terminal you can list, inspect, and retry without opening the Dashboard. Order reflects data flow: request → events → attempts.\n\n```sh\nhookdeck gateway request list\nhookdeck gateway request get req_xxx\nhookdeck gateway request retry req_xxx\nhookdeck gateway event list\nhookdeck gateway event get evt_xxx\nhookdeck gateway event retry evt_xxx\nhookdeck gateway attempt list --event-id evt_xxx\nhookdeck gateway attempt get att_xxx\n```\n\nSee [Request commands](https://hookdeck.com/docs/cli/request.md), [Event commands](https://hookdeck.com/docs/cli/event.md), and [Attempt commands](https://hookdeck.com/docs/cli/attempt.md) for full options. For aggregated metrics (volume, failure rate, backlog), see [CLI: metrics](#cli-metrics) in this file.\n\n## CLI: metrics {#cli-metrics}\n\nWhen to use: event volume, failure rates, which connections had the most events, backlog. Questions like *which connections had the most events?*, *what's our failure rate?*, or *where is backlog building?* are answered from the terminal with the metrics subcommands below. For listing or inspecting individual requests, events, or attempts, see [CLI: list and inspect](#cli-querying). Metrics over time are also available in the [Dashboard](https://dashboard.hookdeck.com) ([Metrics page](https://dashboard.hookdeck.com/metrics) and Source/Connection/Destination pages). The CLI uses `hookdeck gateway metrics` and its subcommands. All CLI commands require a date range (`--start`, `--end`, ISO 8601) and at least one `--measures` value; optional filters include `--granularity`, `--dimensions`, `--source-id`, `--destination-id`, `--connection-id`, and `--status`. See [Metrics](https://hookdeck.com/docs/metrics) and the [CLI metrics reference](https://hookdeck.com/docs/cli/metrics.md) for full reference.\n\n| Subcommand | Purpose |\n|------------|---------|\n| `metrics events` | Event volume/failure rates plus queue depth, pending, and issue-grouped views (via measures/dimensions) |\n| `metrics requests` | Request acceptance vs rejection counts |\n| `metrics attempts` | Delivery latency and success/failure |\n| `metrics transformations` | Transformation run counts and error rate |\n\n**Example commands (use cases):**\n\nEvent volume and failure rate over time:\n\n```sh\nhookdeck gateway metrics events --start 2026-02-01T00:00:00Z --end 2026-02-25T00:00:00Z --granularity 1d --measures count,failed_count,error_rate\n```\n\nRequest acceptance vs rejection:\n\n```sh\nhookdeck gateway metrics requests --start 2026-02-01T00:00:00Z --end 2026-02-25T00:00:00Z --measures count,accepted_count,rejected_count\n```\n\nDelivery latency (attempts):\n\n```sh\nhookdeck gateway metrics attempts --start 2026-02-01T00:00:00Z --end 2026-02-25T00:00:00Z --measures response_latency_avg,response_latency_p95\n```\n\nQueue backlog (use `events` with queue measures and optional `--destination-id`):\n\n```sh\nhookdeck gateway metrics events --start 2026-02-01T00:00:00Z --end 2026-02-25T00:00:00Z --measures max_depth,max_age --destination-id des_xxx\n```\n\nPending events over time (use `events` with `pending` measure):\n\n```sh\nhookdeck gateway metrics events --start 2026-02-01T00:00:00Z --end 2026-02-25T00:00:00Z --granularity 1h --measures pending,count\n```\n\nEvents grouped by issue (use `events` with `--dimensions issue_id` and `--issue-id`):", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "grep -n -i \"issue\" /tmp/sandbox-9ec775af/.claude/skills/event-gateway/references/cli-workflows.md", - "description": "Search cli-workflows reference for issue trigger commands" - }, - "name": "shell", - "command": "grep -n -i \"issue\" /tmp/sandbox-9ec775af/.claude/skills/event-gateway/references/cli-workflows.md", - "result": "(Bash completed with no output)", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "hookdeck gateway --help 2>&1 | grep -i issue; echo \"---\"; hookdeck gateway issue-trigger --help 2>&1", - "description": "Check CLI for issue-trigger subcommands" - }, - "name": "shell", - "command": "hookdeck gateway --help 2>&1 | grep -i issue; echo \"---\"; hookdeck gateway issue-trigger --help 2>&1", - "result": " issue Manage your issues [BETA]\n---\nCommands for managing Event Gateway sources, destinations, connections,\ntransformations, events, requests, metrics, and MCP server.\n\nThe gateway command group provides full access to all Event Gateway resources.\n\nUsage:\n hookdeck gateway [command]\n\nExamples:\n # List connections\n hookdeck gateway connection list\n\n # Create a source\n hookdeck gateway source create --name my-source --type WEBHOOK\n\n # Query event metrics\n hookdeck gateway metrics events --start 2026-01-01T00:00:00Z --end 2026-02-01T00:00:00Z\n\n # Start the MCP server for AI agent access\n hookdeck gateway mcp\n\nAvailable Commands:\n attempt Inspect delivery attempts [BETA]\n connection Manage your connections [BETA]\n destination Manage your destinations [BETA]\n event Inspect and manage events [BETA]\n issue Manage your issues [BETA]\n mcp Start an MCP server for AI agent access to Hookdeck [BETA]\n metrics Query Event Gateway metrics [BETA]\n request Inspect and manage requests [BETA]\n source Manage your sources [BETA]\n transformation Manage your transformations [BETA]\n\nFlags:\n -h, --help help for gateway\n\nGlobal Flags:\n --color string turn on/off color output (on, off, auto)\n --device-name string device name\n --hookdeck-config string path to CLI config file (default is $HOME/.config/hookdeck/config.toml)\n --insecure Allow invalid TLS certificates\n --log-level string log level (debug, info, warn, error) (default \"info\")\n -p, --profile string profile name (default \"default\")\n\nUse \"hookdeck gateway [command] --help\" for more information about a command.", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "hookdeck gateway issue --help 2>&1", - "description": "Check issue subcommands" - }, - "name": "shell", - "command": "hookdeck gateway issue --help 2>&1", - "result": "Manage Hookdeck issues.\n\nIssues are automatically created when delivery failures, transformation errors,\nor backpressure conditions are detected. Use these commands to list, inspect,\nupdate the status of, or dismiss issues.\n\n[BETA] This feature is in beta. Please share bugs and feedback via:\nhttps://github.com/hookdeck/hookdeck-cli/issues\n\nUsage:\n hookdeck gateway issue [command]\n\nAliases:\n issue, issues\n\nAvailable Commands:\n count Count issues\n dismiss Dismiss an issue\n get Get issue details\n list List issues\n update Update a issue by ID\n\nFlags:\n -h, --help help for issue\n\nGlobal Flags:\n --color string turn on/off color output (on, off, auto)\n --device-name string device name\n --hookdeck-config string path to CLI config file (default is $HOME/.config/hookdeck/config.toml)\n --insecure Allow invalid TLS certificates\n --log-level string log level (debug, info, warn, error) (default \"info\")\n -p, --profile string profile name (default \"default\")\n\nUse \"hookdeck gateway issue [command] --help\" for more information about a command.", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "cat /tmp/sandbox-9ec775af/.claude/skills/event-gateway/references/referencing-docs.md", - "description": "Read how to fetch docs" - }, - "name": "shell", - "command": "cat /tmp/sandbox-9ec775af/.claude/skills/event-gateway/references/referencing-docs.md", - "result": "# Referencing Hookdeck Documentation\n\nHookdeck documentation is the source of truth. Always verify claims against official docs before making Hookdeck-related assertions.\n\n## Fetching Documentation as Markdown\n\nMost Hookdeck doc pages are available as markdown by appending `.md`:\n\n```\ncurl https://hookdeck.com/docs/connections.md\ncurl https://hookdeck.com/docs/filters.md\ncurl https://hookdeck.com/docs/authentication.md\n```\n\n## Use Case Guides\n\nEach use case has an overview and a quickstart:\n\n| Use case | Overview | Quickstart |\n|----------|----------|------------|\n| Receive webhooks | `/docs/use-cases/receive-webhooks.md` | `/docs/use-cases/receive-webhooks/quickstart.md` |\n| Third-party routing | `/docs/use-cases/third-party-routing.md` | `/docs/use-cases/third-party-routing/quickstart.md` |\n| Asynchronous APIs | `/docs/use-cases/asynchronous-apis.md` | `/docs/use-cases/asynchronous-apis/quickstart.md` |\n| Test & debug localhost | `/docs/use-cases/test-debug-localhost.md` | `/docs/use-cases/test-debug-localhost/quickstart.md` |\n\n## Core Reference\n\n| Topic | Path |\n|-------|------|\n| Getting started | `/docs/hookdeck-basics.md` |\n| Sources | `/docs/sources.md` |\n| Destinations | `/docs/destinations.md` |\n| Connections | `/docs/connections.md` |\n| Filters | `/docs/filters.md` |\n| Transformations | `/docs/transformations.md` |\n| Retries | `/docs/retries.md` |\n| Deduplication | `/docs/deduplication.md` |\n| Authentication | `/docs/authentication.md` |\n| Events & Attempts | `/docs/events.md` |\n| Requests | `/docs/requests.md` |\n| Issues | `/docs/issues.md` |\n| Metrics | `/docs/metrics.md` |\n| Bookmarks | `/docs/bookmarks.md` |\n\n## Production (going live)\n\n| Topic | Path | Notes |\n|-------|------|--------|\n| **Receive webhooks → production** | `/docs/use-cases/receive-webhooks/quickstart#deliver-to-your-production-webhook-endpoint` | Deliver to production endpoint; use with Destinations, Retries, Issue triggers for full checklist |\n| **Destinations** | `/docs/destinations` | Set max delivery rate (rate limiting) on destinations |\n| **Retries** | `/docs/retries` | Configure automatic retries |\n| **Issue triggers** | `/docs/issue-triggers` | Define which issues trigger alerts and notifications |\n| **Issues & Notifications** | `/docs/issues` | Notifications, notification channels |\n| **Projects** | `/docs/projects` | Create a new project (e.g. for separate production environment) |\n\nHookdeck docs are the source of truth; point users and agents to these paths for production details.\n\n## CLI Reference\n\n| Page | Path |\n|------|------|\n| CLI overview (about + install + commands) | `/docs/cli.md` |\n| Listen command | `/docs/cli/listen.md` |\n| Connection commands | `/docs/cli/connection.md` |\n| Project commands | `/docs/cli/project.md` |\n| Source commands | `/docs/cli/source.md` |\n| Destination commands | `/docs/cli/destination.md` |\n| Transformation commands | `/docs/cli/transformation.md` |\n| Request commands | `/docs/cli/request.md` |\n| Event commands | `/docs/cli/event.md` |\n| Attempt commands | `/docs/cli/attempt.md` |\n| Metrics commands | `/docs/cli/metrics.md` |\n\n## API Reference\n\nThe API docs are split into focused pages:\n\n| Page | Path |\n|------|------|\n| API overview (auth, pagination, errors, rate limits) | `/docs/api.md` |\n| Connections | `/docs/api/connections.md` |\n| Sources | `/docs/api/sources.md` |\n| Destinations | `/docs/api/destinations.md` |\n| Rules | `/docs/api/rules.md` |\n| Transformations | `/docs/api/transformations.md` |\n| Issue triggers | `/docs/api/issue-triggers.md` |\n| Notifications | `/docs/api/notifications.md` |\n| Publish | `/docs/api/publish.md` |\n| Inspect (requests, events, attempts, bookmarks, issues, metrics) | `/docs/api/inspect.md` |\n| Bulk operations (retry, cancel, ignored events, requests) | `/docs/api/bulk.md` |\n\n## OpenAPI Specification\n\nThe latest Event Gateway API spec:\n\n```\nhttps://api.hookdeck.com/latest/openapi\n```\n\nThe `servers` section contains the current API version.\n\n## Best Practices\n\n1. Don't guess doc URLs -- use the paths listed above\n2. All paths are relative to `https://hookdeck.com`\n3. Docs are the source of truth for syntax, flags, and schemas\n4. This skill provides decision guidance and patterns that docs don't cover", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "curl -s https://hookdeck.com/docs/api/issue-triggers.md", - "description": "Fetch issue triggers API docs" - }, - "name": "shell", - "command": "curl -s https://hookdeck.com/docs/api/issue-triggers.md", - "result": " 404 - Hookdeck\n\n\n

\nThe page you are looking for has been
deleted or does not exist.\n

Home
", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "curl -s https://hookdeck.com/docs/issue-triggers.md 2>&1 | head -300", - "description": "Fetch issue triggers doc page" - }, - "name": "shell", - "command": "curl -s https://hookdeck.com/docs/issue-triggers.md 2>&1 | head -300", - "result": "# Issue Triggers\n\n![Issue Triggers](./images/issue-triggers.png)\n\nIssue triggers are the mechanism within Hookdeck to define which problems should trigger an issue (and a corresponding notification) and under which circumstances.\n\n## Behavior\n\nEffective use of issue triggers relies on an understanding of their behavior and their relationship with the [Issues](/docs/issues) they generate.\n\n### Generated Issues\n\nIssues generated by an issue trigger are internally associated with that issue trigger.\n\n* You can view an issue trigger's associated issues by [inspecting](#inspect-an-issue-trigger) it.\n* [Deleting](#delete-an-issue-trigger) an issue trigger will also delete all associated issues.\n* [Updating](#update-an-issue-trigger) an issue trigger will only affect generated issues after the update. Existing associated issues will be unaffected.\n\n### Notification Channels\n\nSpecify which channels (email, Slack, Discord, Microsoft Teams, Better Stack Uptime, incident.io, etc) should be used to notify you when an issue is triggered. To add a Notification Channel, read [Projects Integrations](/docs/projects#integrations).\n\n### Trigger Overlap\n\nIt is possible to have multiple issue triggers of a given type that overlap in their coverage. In such cases, Hookdeck will attempt to select the most specific trigger and apply its logic.\n\nFor instance, if you had two Delivery triggers and a given connection occurs in the `connections` list of both triggers, the longer of the two lists would be considered more specific and only the more specific trigger would run for that connection. Similarly, a Request trigger configured with a specific rejection cause will take priority over one configured with `'*'` (all causes).\n\n## Issue Types\n\nIssues are currently organized into four types -- delivery issues, transformation issues, backpressure issues, and request issues.\n\n### Delivery Issues\n\nDelivery Issues describe issues related to the attempted delivery of requests, featuring two properties:\n\n* `strategy` : 'first_attempt_failure' or 'last_attempt_failure'\n* `connections` : a list of connections for which the trigger should be applied\n\n### Transformation Issues\n\nTransformation Issues relate to issues that may occur in the data manipulation step of a request:\n\n* `log_level` : the log level where the issue should be opened -- 'warn', 'error', or 'fatal'\n* `transformations` : a list of transformations for which the trigger should be applied\n\n### Backpressure Issues\n\nBackpressure Issues are caused when the estimated queue time (600,000ms, or 10 minutes, by default) exceeds the configured `delay` threshold. This allows for integration-specific alerts for impending queue processing issues.\n\n* `delay` : the queue backpressure, in milliseconds, that should trigger an issue\n* `destinations` : a list of destinations for which the trigger should be applied\n\n### Request Issues\n\nRequest Issues are triggered when incoming webhook requests are rejected by a source. This allows you to monitor and respond to problems with inbound request ingestion before they impact your event pipeline.\n\n* `rejection_causes` : an array of specific rejection causes to match on, or `'*'` to match all rejection causes\n* `sources` : a list of sources for which the trigger should be applied, a pattern to match on source names, or `'*'` for all sources\n\nThe possible rejection causes are:\n\n| Rejection cause | Description |\n| --- | --- |\n| `SOURCE_DISABLED` | The source is disabled and not accepting requests |\n| `NO_CONNECTION` | The source has no connections configured |\n| `VERIFICATION_FAILED` | The request failed the source's verification check |\n| `UNSUPPORTED_HTTP_METHOD` | The request used an HTTP method not supported by the source |\n| `UNSUPPORTED_CONTENT_TYPE` | The request's content type is not supported |\n| `UNPARSABLE_JSON` | The request body contains invalid JSON |\n| `PAYLOAD_TOO_LARGE` | The request payload exceeds the maximum allowed size |\n| `INGESTION_FATAL` | A fatal error occurred during request ingestion |\n| `UNKNOWN` | The request was rejected for an unknown reason |\n\n## Default Triggers\n\nEach project is initialized with one issue trigger of each type:\n\n* The default Delivery issue trigger will alert on the first attempt failure of all connections.\n* The default Transformation issue trigger will alert for events on the `warn` log level of all transformations.\n* The default Backpressure issue trigger will alert when the estimated queue time exceeds the default `delay` of 600000ms (10 minutes) for all destinations.\n* The default Request issue trigger will alert on all rejection causes for all sources.\n\n## View issue triggers\n\nViewing issue triggers gives you an overview of the issue types you'll receive notifications about when they occur in your project. Each issue trigger corresponds to a specific issue type and the details necessary to trigger a notification.\n\n### Dashboard\n1. Open the [Issue Triggers](https://dashboard.hookdeck.com/issue-triggers) page to see a list of all issue triggers, grouped by type.\n\n### API\n`GET /2025-07-01/issue-triggers`\n\n**Response example**\n```json\n{\n \"pagination\": {\n \"order_by\": \"created_at\",\n \"dir\": \"desc\",\n \"limit\": 100\n },\n \"count\": 1,\n \"models\": [\n {\n \"id\": \"it_O9Um6FAQMQvxiP\",\n \"team_id\": \"tm_lbhzBKgFOUnB\",\n \"name\": null,\n \"type\": \"backpressure\",\n \"configs\": {\n \"delay\": 600000,\n \"destinations\": \"*\"\n },\n \"disabled_at\": null,\n \"updated_at\": \"2026-01-14T13:35:52.746Z\",\n \"created_at\": \"2026-01-14T13:35:52.806Z\",\n \"channels\": {\n \"email\": {}\n }\n }\n ]\n}\n```\n\n## Create an issue trigger\n\nCreating an issue trigger tells Hookdeck the circumstances under which an issue notification should be sent.\n\n### Dashboard\n1. Open the [Issues Triggers](https://dashboard.hookdeck.com/issue-triggers) page to see a list of all issue triggers, grouped by type.\n2. Determine the desired trigger type and click the + Create button in the corresponding section.\n3. Change the desired parameters for the given trigger's configuration.\n4. Click the Create Trigger button.\n\n### API\n`POST /2025-07-01/issue-triggers`\n\n**Request body example**\n```json\n{\n \"type\": \"delivery\",\n \"configs\": {\n \"strategy\": \"first_attempt\",\n \"connections\": \"*\"\n },\n \"channels\": {\n \"email\": {}\n }\n}\n```\n\n**Response example**\n```json\n{\n \"type\": \"delivery\",\n \"configs\": {\n \"strategy\": \"first_attempt\",\n \"connections\": \"*\"\n },\n \"channels\": {\n \"email\": {}\n },\n \"id\": \"it_BbDIsKCTbG8qck\",\n \"team_id\": \"tm_lbhzBKgFOUnB\",\n \"created_at\": \"2026-01-14T13:36:41.490Z\",\n \"name\": null,\n \"disabled_at\": null,\n \"updated_at\": \"2026-01-14T13:36:41.491Z\"\n}\n```\n\n## Inspect an issue trigger\n\nInspecting an issue trigger gives you access to its context and activation configuration.\n\n### Dashboard\n1. [Find your issue trigger](#view-issue-triggers) on the Issue Triggers page.\n2. Click the issue trigger's description.From here you can view all issues associated with an issue trigger, [update the parameters](#update-an-issue-trigger) of the selected trigger, and define filters to better target and refine your notification strategy.\n\n### API\n`GET /2025-07-01/issue-triggers/:id`\n\n**Response example**\n```json\n{\n \"id\": \"it_O9Um6FAQMQvxiP\",\n \"team_id\": \"tm_lbhzBKgFOUnB\",\n \"name\": null,\n \"type\": \"backpressure\",\n \"configs\": {\n \"delay\": 600000,\n \"destinations\": \"*\"\n },\n \"disabled_at\": null,\n \"updated_at\": \"2026-01-14T13:35:52.746Z\",\n \"created_at\": \"2026-01-14T13:35:52.806Z\",\n \"channels\": {\n \"email\": {}\n }\n}\n```\n\n## Update an issue trigger\n\nUpdating an issue trigger's configuration lets you define the activation context for when notifications get sent out.\n\n> Updates made to an issue trigger apply only to notifications and issues generated after the update. Issues associated with the trigger prior to any changes will be unaffected.\n\n### Dashboard\n1. [Find your issue trigger](#view-issue-triggers) on the Issue Triggers page.\n2. Click the issue trigger's description to [view the configuration](#inspect-an-issue-trigger).\n3. Change the desired parameters for the given trigger's configuration.\n4. Click Save for the configuration section to save the changes.\n\n### API\n`PUT /2025-07-01/issue-triggers/:id`\n\n**Request body example**\n```json\n{\n \"type\": \"delivery\",\n \"configs\": {\n \"strategy\": \"first_attempt\",\n \"connections\": \"*\"\n },\n \"channels\": {\n \"email\": {}\n }\n}\n```\n\n**Response example**\n```json\n{\n \"id\": \"it_O9Um6FAQMQvxiP\",\n \"team_id\": \"tm_lbhzBKgFOUnB\",\n \"name\": null,\n \"type\": \"backpressure\",\n \"configs\": {\n \"strategy\": \"first_attempt\",\n \"connections\": \"*\"\n },\n \"disabled_at\": null,\n \"updated_at\": \"2026-01-14T13:36:41.518Z\",\n \"created_at\": \"2026-01-14T13:35:52.806Z\",\n \"channels\": {\n \"email\": {}\n }\n}\n```\n\n## Delete an issue trigger\n\nDeleting an issue trigger prevents it from generating notifications.\n\n> Deleting an issue trigger will also permanently delete all associated issues.\n\n### Dashboard\n1. [Find your issue trigger](#view-issue-triggers) on the Issue Triggers page.\n2. Click the issue trigger's description to [view its configuration](#inspect-an-issue-trigger).\n3. Click Delete at the bottom of the page.\n4. Click Delete again to confirm.\n\n### API\n`DELETE /2025-07-01/issue-triggers/:id`\n\n**Response example**\n```json\n{\n \"id\": \"it_O9Um6FAQMQvxiP\"\n}\n```", - "ts": 0 - }, - { - "endpoint": "Bash", - "body": { - "command": "echo \"=== existing issue triggers ===\"; curl -s https://api.hookdeck.com/2025-07-01/issue-triggers -H \"Authorization: Bearer $HOOKDECK_API_KEY\" | head -c 3000\necho\necho \"=== notifications docs ===\"\ncurl -s https://hookdeck.com/docs/api/notifications.md 2>&1 | head -c 100\necho\necho \"=== issues doc channels section ===\"\ncurl -s https://hookdeck.com/docs/issues.md 2>&1 | grep -n -i -A5 \"channel\"", - "description": "Check existing issue triggers and notification docs" - }, - "name": "shell", - "command": "echo \"=== existing issue triggers ===\"; curl -s https://api.hookdeck.com/2025-07-01/issue-triggers -H \"Authorization: Bearer $HOOKDECK_API_KEY\" | head -c 3000\necho\necho \"=== notifications docs ===\"\ncurl -s https://hookdeck.com/docs/api/notifications.md 2>&1 | head -c 100\necho\necho \"=== issues doc channels section ===\"\ncurl -s https://hookdeck.com/docs/issues.md 2>&1 | grep -n -i -A5 \"channel\"", - "result": "=== existing issue triggers ===\n{\"pagination\":{\"order_by\":\"created_at\",\"dir\":\"desc\",\"limit\":100},\"count\":3,\"models\":[{\"id\":\"it_ayecSXxpWrAxfC\",\"team_id\":\"tm_c6MTDfxZm4tP\",\"name\":null,\"type\":\"transformation\",\"configs\":{\"log_level\":\"warn\",\"transformations\":\"*\"},\"disabled_at\":null,\"updated_at\":\"2026-08-07T10:46:14.027Z\",\"created_at\":\"2026-08-07T10:46:14.347Z\",\"channels\":{\"email\":{}}},{\"id\":\"it_MG6NeNJn9TxeRa\",\"team_id\":\"tm_c6MTDfxZm4tP\",\"name\":null,\"type\":\"request\",\"configs\":{\"sources\":\"*\",\"rejection_causes\":\"*\",\"excluded_rejection_causes\":[\"NO_CONNECTION\",\"UNSUPPORTED_HTTP_METHOD\"]},\"disabled_at\":null,\"updated_at\":\"2026-08-07T10:46:14.027Z\",\"created_at\":\"2026-08-07T10:46:14.347Z\",\"channels\":{\"email\":{}}},{\"id\":\"it_T7Z6hXLnlhazwM\",\"team_id\":\"tm_c6MTDfxZm4tP\",\"name\":null,\"type\":\"backpressure\",\"configs\":{\"delay\":600000,\"destinations\":\"*\"},\"disabled_at\":null,\"updated_at\":\"2026-08-07T10:46:14.027Z\",\"created_at\":\"2026-08-07T10:46:14.347Z\",\"channels\":{\"email\":{}}}]}\n=== notifications docs ===\n 404 - Hookdeck

Issue Triggers

\"Issue

Issue triggers are the mechanism within Hookdeck to define which problems should trigger an issue (and a corresponding notification) and under which circumstances.

Behavior

Effective use of issue triggers relies on an understanding of their behavior and their relationship with the Issues they generate.

Generated Issues

Issues generated by an issue trigger are internally associated with that issue trigger.

  • You can view an issue trigger's associated issues by inspecting it.
  • Deleting an issue trigger will also delete all associated issues.
  • Updating an issue trigger will only affect generated issues after the update. Existing associated issues will be unaffected.

Notification Channels

Specify which channels (email, Slack, Discord, Microsoft Teams, Better Stack Uptime, incident.io, etc) should be used to notify you when an issue is triggered. To add a Notification Channel, read Projects Integrations.

Trigger Overlap

It is possible to have multiple issue triggers of a given type that overlap in their coverage. In such cases, Hookdeck will attempt to select the most specific trigger and apply its logic.

For instance, if you had two Delivery triggers and a given connection occurs in the connections list of both triggers, the longer of the two lists would be considered more specific and only the more specific trigger would run for that connection. Similarly, a Request trigger configured with a specific rejection cause will take priority over one configured with '*' (all causes).

Issue Types

Issues are currently organized into four types -- delivery issues, transformation issues, backpressure issues, and request issues.

Delivery Issues

Delivery Issues describe issues related to the attempted delivery of requests, featuring two properties:

  • strategy : 'first_attempt_failure' or 'last_attempt_failure'
  • connections : a list of connections for which the trigger should be applied

Transformation Issues

Transformation Issues relate to issues that may occur in the data manipulation step of a request:

  • log_level : the log level where the issue should be opened -- 'warn', 'error', or 'fatal'
  • transformations : a list of transformations for which the trigger should be applied

Backpressure Issues

Backpressure Issues are caused when the estimated queue time (600,000ms, or 10 minutes, by default) exceeds the configured delay threshold. This allows for integration-specific alerts for impending queue processing issues.

  • delay : the queue backpressure, in milliseconds, that should trigger an issue
  • destinations : a list of destinations for which the trigger should be applied

Request Issues

Request Issues are triggered when incoming webhook requests are rejected by a source. This allows you to monitor and respond to problems with inbound request ingestion before they impact your event pipeline.

  • rejection_causes : an array of specific rejection causes to match on, or '*' to match all rejection causes
  • sources : a list of sources for which the trigger should be applied, a pattern to match on source names, or '*' for all sources

The possible rejection causes are:

Rejection causeDescription
SOURCE_DISABLEDThe source is disabled and not accepting requests
NO_CONNECTIONThe source has no connections configured
VERIFICATION_FAILEDThe request failed the source's verification check
UNSUPPORTED_HTTP_METHODThe request used an HTTP method not supported by the source
UNSUPPORTED_CONTENT_TYPEThe request's content type is not supported
UNPARSABLE_JSONThe request body contains invalid JSON
PAYLOAD_TOO_LARGEThe request payload exceeds the maximum allowed size
INGESTION_FATALA fatal error occurred during request ingestion
UNKNOWNThe request was rejected for an unknown reason

Default Triggers

Each project is initialized with one issue trigger of each type:

  • The default Delivery issue trigger will alert on the first attempt failure of all connections.
  • The default Transformation issue trigger will alert for events on the warn log level of all transformations.
  • The default Backpressure issue trigger will alert when the estimated queue time exceeds the default delay of 600000ms (10 minutes) for all destinations.
  • The default Request issue trigger will alert on all rejection causes for all sources.

View issue triggers

Viewing issue triggers gives you an overview of the issue types you'll receive notifications about when they occur in your project. Each issue trigger corresponds to a specific issue type and the details necessary to trigger a notification.

  1. Open the Issue Triggers page to see a list of all issue triggers, grouped by type.
GET
/2025-07-01/issue-triggers
Response example
JSON 
{\n295:  \"type\": \"delivery\",\n297:    \"strategy\": \"first_attempt\",\n298:    \"connections\": \"*\"\n304:  \"type\": \"delivery\",\n306:    \"strategy\": \"first_attempt\",\n307:    \"connections\": \"*\"\n318:}

Inspect an issue trigger

Inspecting an issue trigger gives you access to its context and activation configuration.

  1. Find your issue trigger on the Issue Triggers page.
  2. Click the issue trigger's description.

From here you can view all issues associated with an issue trigger, update the parameters of the selected trigger, and define filters to better target and refine your notification strategy.

GET
/2025-07-01/issue-triggers/:id
Response example
JSON 
{\n334:  \"type\": \"delivery\",\n336:    \"strategy\": \"first_attempt\",\n337:    \"connections\": \"*\"\n348:    \"strategy\": \"first_attempt\",\n349:    \"connections\": \"*\"\n359:}
<- Previous Issues & Notifications Next -> Metrics