Repository navigation
Fix product page production capacity at 3 tasks - #2327
Merged
Merged
Conversation
Autoscaling on p95 response time over 10 second periods was flapping between 3 and 6 tasks every few minutes. Product page traffic is low enough that a single slow request breaches the threshold, and each scale-in stopped the warm tasks and kept the newly started cold ones, whose first requests are slow to render. That caused most of the slow requests we see. Setting max_capacity equal to min_capacity matches dev and staging and stops the scaling policies from changing the task count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The setting implements the intended capacity limit, with the loss of automatic scale-out explicitly acknowledged.
No open issues 🎉
What changed in this PR
Constrains production product-page autoscaling to three tasks to reduce cold-start latency caused by scaling churn.
Changes:
- Reduces
max_capacityfrom 9 to 3, matchingmin_capacityand preventing automatic scale-out.
| File | Description |
|---|---|
| infra/deployments/forms/tfvars/production.tfvars | Sets production product-page maximum capacity to three tasks. |
Review effort: Balanced
DavidBiddle
approved these changes
Oct 19, 2026
DavidBiddle
left a comment
DavidBiddle
left a comment
Contributor
There was a problem hiding this comment.
Looks good and the change is well reasoned. I think the tradeoff of not being able to autoscale is acceptable given the current low traffic and the current use cases of this app.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this pull request solve?
Trello card: https://trello.com/c/PrXP6eBv/879-enable-alerts-for-product-page-slos
Fixes the slow product page requests that breach the 1000ms latency SLO.
Cause: autoscaling keeps scaling production between 3 and 6 tasks every few minutes. Each scale-in stops warm tasks and keeps new cold ones, and a cold task's first page renders take 1.4–3s.
Evidence (last 4 weeks):
Why it flaps: the alarms use p95 response time over 10 second periods. With our low traffic, a single slow request is enough to trigger a scale-out.
Fix: set
max_capacityto matchmin_capacity(3), as dev and staging already do. The scaling policies can then no longer change the task count.This is a good enough fix for now: