Document the queue module

SUMMARY.md

@@ -153,6 +153,12 @@
   * [Drivers](modules/multi-factor-auth/drivers/README.md)
     * [Email](modules/multi-factor-auth/drivers/email.md)
     * [Authenticator](modules/multi-factor-auth/drivers/authenticator.md)
+* [Queue](modules/queue.md)
+  * [Tasks](modules/queue/tasks.md)
+  * [Queues](modules/queue/queues.md)
+  * [Workers](modules/queue/workers.md)
+  * [Process Managers](modules/queue/process-managers.md)
+  * [Admin](modules/queue/admin.md)
 * [Subscription](modules/subscription.md)
 * [Webhooks](modules/webhooks.md)
   * [Handlers](modules/webhooks/handlers.md)

modules/find-replace.md

@@ -64,7 +64,7 @@ A revert writes the previous value only when the live cell is still exactly the
 
 Admin searches run as a queue job. The queue is `default` unless `FIND_REPLACE_QUEUE` names another alias or queue class, such as `priority` or `App\Queue\Queues\Reports`. A blank value also uses `default`.
 
-`nails queue:work` listens to `default`. Pass `--queue` when the search uses another queue.
+`nails queue:work` listens to `default`. Pass `--queue` when the search uses another queue. See [Queue](queue.md) for [running workers](queue/workers.md) and keeping them alive with a [process manager](queue/process-managers.md).
 
 The Find & Replace index shows a warning when that queue has no live worker. A worker is live when its heartbeat is within `QUEUE_WORKER_HEARTBEAT_STALE` seconds (300 by default) and it was started for that queue. This is the same check as the queue dashboard alert. A stale worker, or a worker listening to a different queue, leaves the warning in place. The form can still be submitted; the search stays pending until a worker picks it up. If `FIND_REPLACE_QUEUE` does not name a queue, the index says so and the search is not saved.
 

modules/queue.md

@@ -0,0 +1,85 @@
+---
+description: >-
+  A database-backed job queue. Push work from a request, run it in a
+  long-lived worker, and retry it when it fails.
+---
+
+# Queue
+
+The queue module moves slow or fragile work out of the request. The app pushes a job, a worker process picks it up and runs it, and a failed job is retried with a growing delay. Jobs live in MySQL, so there is no broker to install.
+
+```bash
+composer require nails/module-queue
+nails db:migrate
+```
+
+The module needs `nails/module-console` for the worker commands and [`nails/module-cron`](cron.md) to schedule its cleanup. Reserving a job uses `SELECT … FOR UPDATE SKIP LOCKED`, so the database must be MySQL 8.0 or newer, or MariaDB 10.6 or newer.
+
+The migration creates two tables, prefixed with `NAILS_DB_PREFIX` as usual:
+
+| Table | Holds |
+|---|---|
+| `nails_queue_job` | Every job, from the moment it is pushed until it is rotated out |
+| `nails_queue_worker` | One row per running worker, with its last heartbeat |
+
+Both tables are hidden from [Find & Replace](find-replace/exclusions.md).
+
+## How it fits together
+
+| Piece | What it is |
+|---|---|
+| [Task](queue/tasks.md) | A class with a `run()` method. It is the work. |
+| Data | The JSON payload stored with a job and handed to `run()`. |
+| [Queue](queue/queues.md) | A named stream of jobs. A worker listens to one or more queues. |
+| [Worker](queue/workers.md) | The `nails queue:work` process. It runs one job at a time. |
+| [Manager](queue/tasks.md#pushing-a-job) | The service the app uses to push jobs. Workers use it to reserve them. |
+
+A job moves through four statuses:
+
+| Status | Meaning |
+|---|---|
+| `PENDING` | Waiting for a worker. A job with a future `available_at` is _scheduled_ and is still `PENDING`. |
+| `RUNNING` | A worker has reserved it and is running the task. |
+| `COMPLETE` | The task returned without throwing. |
+| `FAILED` | The task threw and has no retries left. The errors are kept on the job. |
+
+A failed attempt that still has retries left goes back to `PENDING` with a later `available_at`.
+
+1. The app calls `push()` with a task, a payload, and optionally a queue and a start time.
+2. A [worker](queue/workers.md) reserves the oldest job that is `PENDING` and due, in the queues it was started for.
+3. The worker calls `run()` on the task with the payload.
+4. The job becomes `COMPLETE`, is rescheduled for a retry, or becomes `FAILED`.
+5. [`queue:clean`](queue/workers.md#maintenance), run by cron, removes dead workers, recovers stuck jobs, and deletes old rows.
+
+{% hint style="warning" %}
+Nothing runs a job until a worker is running. Jobs sit in `PENDING` and the admin dashboard shows a warning. Keep a worker alive with a [process manager](queue/process-managers.md). Each [queue](queue/queues.md) a worker should serve is named when it starts, and the built-in `priority` queue needs a [worker of its own](queue/queues.md#priority).
+{% endhint %}
+
+## Pages
+
+* [Tasks](queue/tasks.md): write a task, push a job, and how retries work
+* [Queues](queue/queues.md): the built-in queues, your own queues, and aliases
+* [Workers](queue/workers.md): run workers, read their log, stop them, and the maintenance command
+* [Process managers](queue/process-managers.md): keep workers running with systemd or Supervisor
+* [Admin](queue/admin.md): the overview screen, the dashboard alert, and the API behind them
+
+## Configuration
+
+Set these as [config values](../getting-started/configuration.md), usually environment variables. All are optional.
+
+| Config | Default | Effect |
+|---|---|---|
+| `QUEUE_WORKER_WAIT_TIME` | `500` | Milliseconds an idle worker first sleeps between polls. The wait doubles on each empty poll, up to 5,000 ms, and starts again after a job. A random 0 to 200 ms is added each time. |
+| `QUEUE_WORKER_REFRESH_INTERVAL` | `300` | Seconds between calls to each queue's `refresh()` hook. |
+| `QUEUE_WORKER_HEARTBEAT_DEBOUNCE` | `5` | Minimum seconds between heartbeat writes from one worker. |
+| `QUEUE_WORKER_HEARTBEAT_STALE` | `300` | Seconds without a heartbeat before a worker counts as dead. |
+| `QUEUE_JOB_ROTATE_COMPLETE_DAYS` | `7` | Days to keep `COMPLETE` jobs. `0` keeps them forever. |
+| `QUEUE_JOB_ROTATE_FAILED_DAYS` | `30` | Days to keep `FAILED` jobs. `0` keeps them forever. |
+
+{% hint style="danger" %}
+A worker records its heartbeat between jobs, not during one. If a single job can run for longer than `QUEUE_WORKER_HEARTBEAT_STALE`, the worker looks dead, [`queue:clean`](queue/workers.md#maintenance) deletes it, and the job is put back in `PENDING` while it is still running. Another worker then runs it a second time. Set `QUEUE_WORKER_HEARTBEAT_STALE` higher than your longest job.
+{% endhint %}
+
+## Overriding
+
+The Manager, both models, both resources, and the `Data` factory can each be replaced by defining a class of the same name under `App\Queue`: `App\Queue\Service\Manager`, `App\Queue\Model\Job`, `App\Queue\Factory\Data`, and so on. See [Overloading](../key-concepts/factory/overloading.md).

modules/queue/admin.md

@@ -0,0 +1,87 @@
+---
+description: >-
+  The queue overview screen, the dashboard warning for unworked queues, and
+  the API behind them.
+---
+
+# Admin
+
+**Queue → Overview** shows what the queue is doing now: how many jobs are in each state, which workers are alive, and which jobs failed. It needs `nails/module-admin`.
+
+The screen loads its data from the [API](#api) and asks again every ten seconds, so it can stay open on a second monitor.
+
+## Permission
+
+| Permission | Class | Allows |
+|---|---|---|
+| Can view queue overview | `Nails\Queue\Admin\Permission\Overview\View` | See the screen and its data, and retry failed jobs |
+
+Grant it on a user group, as described in [User Permissions](../admin/user-permissions.md). The one permission covers retrying a failed job as well as looking at it.
+
+## Overview
+
+### Totals
+
+A row of tiles across the top:
+
+| Tile | Meaning |
+|---|---|
+| Workers (total, active, stale) | Worker rows. A worker is stale when its heartbeat is older than `QUEUE_WORKER_HEARTBEAT_STALE`. |
+| Jobs (total) | Every job in every status. |
+| Jobs pending | `PENDING` jobs that are due and waiting for a worker. |
+| Jobs scheduled | `PENDING` jobs whose start time is still in the future. |
+| Jobs running, complete, failed | Jobs in that status. Complete jobs disappear after [rotation](workers.md#maintenance). |
+| Oldest queued age | How long the oldest due `PENDING` job has been waiting. A figure that keeps growing means the workers are not keeping up, or are not running. |
+| Avg. latency (24h) | The average time between a job becoming available and a worker starting it. |
+| Avg. duration (24h) | The average time a job takes to run, for jobs that finished. |
+| Throughput (1h, 24h) | Jobs finished in that period, both `COMPLETE` and `FAILED`. |
+
+### Queues
+
+One row per [discovered queue](queues.md#discovery), with its job counts by status, its average latency and duration over the last hour, and its throughput over the last hour and the last 24 hours. A queue you have not [put in the discovery namespace](queues.md#discovery) is missing here.
+
+### Workers
+
+Every row in the worker table, with its ID, token, queues, how long it has been running, and when it last reported. A stale worker is highlighted. It is either dead or in the middle of a job longer than `QUEUE_WORKER_HEARTBEAT_STALE`; see [the warning](../queue.md#configuration) on that setting. `queue:clean` removes it.
+
+### Failed jobs
+
+The latest 100 `FAILED` jobs, newest first. Each shows its queue, task, payload, when it failed, the number of attempts, and the message from the last error. **Error Trace** opens the stack trace of the last error.
+
+**Retry** puts the job back in `PENDING`. It waits the usual [backoff](tasks.md#retries) for its next attempt number before it runs, so it is not instant. A retried job gets one more attempt: its retries are already used up, so if it fails again it is `FAILED` straight away. To give a task more tries, raise its `getMaxRetries()`.
+
+The old errors stay on the job.
+
+## Dashboard alert
+
+The admin dashboard shows the warning **Some queues are not being processed** when a [discovered queue](queues.md#discovery) has no live worker. It lists the queues without one. A worker is live when its heartbeat is newer than `QUEUE_WORKER_HEARTBEAT_STALE` and it was started for that queue's class.
+
+It is the same check [Find & Replace](../find-replace.md#queue) uses to warn about its own queue. The check is also available to your code:
+
+```php
+/** @var \Nails\Queue\Service\Manager $manager */
+$manager = Factory::service('Manager', \Nails\Queue\Constants::MODULE_SLUG);
+
+$manager->unworkedQueues();          //  Queue[]: discovered queues with no live worker
+$manager->isProcessed('priority');   //  bool: does a live worker cover this queue?
+```
+
+## API
+
+The screen reads and writes through two routes. Both need a logged-in user with the permission above. See [API](../api/README.md) for how routes and responses work.
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/api/queue/overview` | Everything on the screen |
+| `POST` | `/api/queue/overview/retry` | Retry a failed job |
+
+`GET` returns four keys in `data`:
+
+| Key | Holds |
+|---|---|
+| `kpis` | The tiles: a `label`, `value`, `hint`, and `type` (`number`, `age`, or `duration`) each |
+| `queues` | One entry per discovered queue, with `label`, `jobs`, `latency`, `duration`, and `throughput` |
+| `workers` | One entry per worker, with `id`, `token`, `queues`, `created`, `heartbeat`, and `stale` |
+| `failed` | Up to 100 failed jobs, with `id`, `queue`, `task`, `payload`, `finished`, `attempts`, and `errors` |
+
+`POST` takes `job_id`. It answers `400` if `job_id` is missing or the job is not `FAILED`, and `404` if there is no such job.