-
Notifications
You must be signed in to change notification settings - Fork 328
Clarify Schedule overlap and buffering behavior #5288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -187,10 +187,10 @@ The following options are available: | |
| - `BufferOne`: Starts the Workflow Execution as soon as the current one completes. | ||
| The buffer is limited to one. | ||
| If another Workflow Execution is supposed to start, but one is already in the buffer, only the one in the buffer eventually starts. | ||
| - `BufferAll`: Allows an unlimited number of Workflows to buffer. | ||
| They are started sequentially. | ||
| - `BufferAll`: Buffers Actions while another Workflow Execution is running. | ||
| They are started sequentially, subject to the Catchup Window and internal buffer limits. | ||
| - `CancelOther`: Cancels the running Workflow Execution, and then starts the new one after the old one completes cancellation. | ||
| - `TerminateOther`: Terminates the running Workflow Execution and starts the new one immediately. | ||
| - `TerminateOther`: Terminates the running Workflow Execution and then starts the new one without waiting for Workflow cooperation. | ||
| - `AllowAll` Starts any number of concurrent Workflow Executions. | ||
| With this policy (and only this policy), more than one Workflow Execution, started by the Schedule, can run simultaneously. | ||
|
|
||
|
|
@@ -208,23 +208,117 @@ This affects regularly scheduled Actions, manual triggers, and Backfills: | |
|
|
||
| - With `Skip`, each new scheduled Action is skipped while the previous scheduled Workflow Execution remains Paused. | ||
| - With `BufferOne`, one Action can wait behind the Paused Workflow Execution. Additional Actions are skipped while the buffer is full. | ||
| - With `BufferAll`, every matching Action is buffered and won't start until the Paused Workflow Execution closes. | ||
| - With `BufferAll`, matching Actions are buffered, subject to the Catchup Window and internal buffer limits, and won't start until the Paused Workflow Execution closes. | ||
| - With `AllowAll`, new Workflow Executions can start even while the earlier Workflow Execution is Paused. | ||
| - With `CancelOther`, the Schedule requests cancellation of the Paused Workflow Execution and starts the next Workflow Execution only after the canceled Workflow Execution closes. If the Paused Workflow Execution can't process cancellation until it is unpaused, the next start still waits. | ||
| - With `TerminateOther`, the Schedule terminates the Paused Workflow Execution and starts the next Workflow Execution immediately. | ||
| - With `TerminateOther`, the Schedule terminates the Paused Workflow Execution and then starts the next Workflow Execution without waiting for Workflow cooperation. | ||
|
|
||
| If you pause a Workflow Execution for investigation and don't want the Schedule to skip or buffer future Actions, pause the Schedule too. | ||
| When you're ready to resume the automation, unpause the Workflow Execution and then unpause the Schedule. | ||
| If Actions were skipped while the Workflow Execution was Paused, use [Backfill](#backfill) to run them intentionally with the Overlap Policy you want. | ||
|
|
||
| #### Catchup Window | ||
|
|
||
| The Temporal Service might be down or unavailable at the time when a Schedule should take an Action. | ||
| When it comes back up, the Catchup Window controls which missed Actions should be taken at that point. | ||
| The Temporal Service might be down or unavailable, or an Overlap Policy might delay an Action past its scheduled time. | ||
| The Catchup Window controls how late an automated Action can be taken. | ||
| The default is one year, meaning Actions will be taken unless over one year late. | ||
| If your Actions are more time-sensitive, you can set the Catchup Window to a smaller value (minimum ten seconds), accepting that an outage longer than the window could lead to missed Actions. | ||
| If your Actions are more time-sensitive, you can set the Catchup Window to a smaller value (minimum ten seconds), accepting that an outage or other delay longer than the window could lead to missed Actions. | ||
| The window is measured from the jitter-adjusted Action time, if jitter is configured. | ||
| Manual triggers and Backfills are not restricted by the Catchup Window. | ||
| (But you can always [Backfill](#backfill).) | ||
|
|
||
| #### Common pitfalls | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Move this roughly 90-line, six-heading block into a dedicated Schedule overlap page and link to it here. The change increases this page from 16 to 22 headings, well beyond the repository's target of fewer than 15, while some of its missed-Action diagnosis also overlaps AGENTS.md reference: AGENTS.md:L246-L252 Useful? React with 👍 / 👎.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yeah, this probably worth it's own page
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. could to add jitter as well? |
||
|
|
||
| A scheduled time does not guarantee that a Workflow Execution starts. | ||
| An Action can be skipped by an Overlap Policy, expire outside the Catchup Window, exceed an internal buffer limit, or fail when the Service attempts to start the Workflow Execution. | ||
|
|
||
| ##### `Skip` is the default Overlap Policy | ||
|
|
||
| With `Skip`, every Action that overlaps an open Workflow Execution is intentionally discarded. | ||
| Skipped Actions are not retained for later execution. | ||
|
|
||
| For example, suppose a Schedule runs every five minutes and its first Workflow Execution takes 22 minutes: | ||
|
|
||
| ```text | ||
| 09:00 starts | ||
| 09:05 overlaps and is skipped | ||
| 09:10 overlaps and is skipped | ||
| 09:15 overlaps and is skipped | ||
| 09:20 overlaps and is skipped | ||
| 09:22 the 09:00 Workflow Execution closes | ||
| 09:25 starts | ||
| ``` | ||
|
|
||
| Only the `09:00` and `09:25` Actions start. | ||
| Use `BufferOne` or `BufferAll` if overlapping Actions must be retained, or `AllowAll` if they can run concurrently. | ||
|
|
||
| ##### `BufferOne` preserves the oldest waiting Action | ||
|
|
||
| `BufferOne` retains the first Action that overlaps an open Workflow Execution. | ||
| It does not replace that Action with the most recent one. | ||
| Additional Actions are skipped while the buffer is occupied. | ||
|
|
||
| ```text | ||
| 09:00 starts | ||
| 09:05 is buffered | ||
| 09:10 is skipped because the buffer is occupied | ||
| 09:15 is skipped because the buffer is occupied | ||
| 09:20 is skipped because the buffer is occupied | ||
| 09:22 the 09:00 Workflow Execution closes | ||
| 09:22 the 09:05 Action becomes eligible to start | ||
| ``` | ||
|
|
||
| The buffered Action must still be inside its Catchup Window when it becomes eligible on the current Scheduler implementation. | ||
| Older Scheduler implementations differ in this edge case and might still attempt to start an Action after it has waited beyond the window. | ||
| Configure the Catchup Window to include the maximum expected overlap delay. | ||
|
|
||
| ##### `BufferAll` can create a backlog | ||
|
|
||
| `BufferAll` starts only one non-overlapping Action after each preceding Workflow Execution closes. | ||
| If Workflow Executions take longer than the Schedule interval, the backlog grows faster than it drains. | ||
| Buffered Actions are subject to an internal safety limit of approximately 1,000 per Schedule by default. | ||
| The effective limit can vary by deployment and Scheduler implementation. | ||
| Actions can also expire outside the Catchup Window, so `BufferAll` should not be treated as an unlimited durable queue. | ||
|
|
||
| For example, with a five-minute Catchup Window: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think the example matched the description above. seems like you'd want to show something that does not include a short catchup window. buffer all is probably the best overlap policy to use when demonstrating the behavior of skip due to catchup window and overlap policy. |
||
|
|
||
| ```text | ||
| 09:00 starts | ||
| 09:05 is buffered | ||
| 09:10 is buffered | ||
| 09:15 is buffered | ||
| 09:20 is buffered | ||
| 09:22 the 09:00 Workflow Execution closes | ||
| ``` | ||
|
|
||
| At `09:22`, only the `09:20` Action is still inside the five-minute window. | ||
| On the current Scheduler implementation, the `09:05`, `09:10`, and `09:15` Actions expire, and the `09:20` Action starts. | ||
|
|
||
| ##### Cancellation is asynchronous | ||
|
|
||
| `CancelOther` requests cancellation and waits for the running Workflow Execution to close. | ||
| The Workflow must process the cancellation before the replacement can start. | ||
| If more Actions become due while cancellation is pending, the most recent waiting Action is selected after the running Workflow Execution closes. | ||
|
|
||
| ```text | ||
| 09:00 starts | ||
| 09:05 requests cancellation of 09:00 and waits | ||
| 09:10 becomes the newest waiting Action | ||
| 09:15 becomes the newest waiting Action | ||
| 09:20 becomes the newest waiting Action | ||
| 09:22 the 09:00 Workflow Execution closes | ||
| 09:22 the 09:20 Action becomes eligible to start | ||
| ``` | ||
|
|
||
| A slow or cancellation-resistant Workflow can therefore delay the replacement beyond its Catchup Window. | ||
| `TerminateOther` does not require Workflow cooperation, but task processing and closure-notification latency can still delay the replacement. | ||
|
|
||
| ##### `AllowAll` can create unbounded concurrency | ||
|
|
||
| `AllowAll` produces the fewest policy-related misses because every Action can start concurrently. | ||
| However, when Workflow Executions take longer than the Schedule interval, concurrency continually increases until executions begin to close. | ||
| Ensure that Workers, downstream systems, and Namespace limits can handle the maximum expected concurrency. | ||
|
|
||
| #### Pause-on-failure | ||
|
|
||
| If this policy is set, a Workflow Execution started by a Schedule that ends with a failure or timeout (but not Cancellation or Termination) causes the Schedule to automatically pause. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📝 [vale] <Temporal.Headings> reported by reviewdog 🐶
'Catchup Window' should use sentence-style capitalization.