fix(exporters): keep limiting after the first dropped alert - #928
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe HTTP exporter now enforces ChangesAlert rate-limit enforcement
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: 🔵 Low · up to The per-minute limit now suppresses every alert beyond the configured cap, and rule and malware alerts share that quota; a burst from one category can temporarily suppress alerts from the other until the window resets. The PR is mergeable with explicit owner awareness of this bounded tradeoff. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. (1 skipped: 1 unsupported.) ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
matthyx
left a comment
There was a problem hiding this comment.
Reviewed the logic in pkg/exporters/http_exporter.go independently of the PR description's own AI-review claim.
Correctness: looks right. Traced admitAlert() by hand against the bug it fixes:
- Old code:
count > max && !isNotifiedconflated "drop this alert" and "send the notice" into one bool. The first drop setisNotified, which flipped the condition tofalsefor every later alert in the window — so the limiter dropped exactly one alert per minute and let the rest through. - New code separates
admitted(false for every send past the limit) fromnotify(true only once, on the first refusal), decided underalertMetrics's mutex for the whole operation. Verified: window reset now counts the triggering alert (fixes the oldmax+1effective limit),sendAlertLimitReachedno longer double-setsisNotifiedor readscount/startTimeunlocked, andMaxAlertsPerMinute <= 0correctly short-circuits to "no limit" (matchesValidate()'s zero→default behavior for the common path, and still guards a hand-built exporter with a negative value). - Existing
TestSendRuleAlertRateReachedstill passes under the new semantics (traced by hand: 1st call admitted, 2nd call not admitted + notify). New tests inalert_limit_test.gocover the keep-limiting-after-drop behavior, window reset, zero-means-no-limit, and a 20×20 concurrent-sender case pinning admitted==limit and notified==1 — good coverage for a mutex-guarded counter. - Confirmed no other call sites reference the old
shouldSendLimitAlertname (searchedmain); the rename is clean.
Blockers before this can merge:
- DCO check is failing (
action_required). The commit444b0432has noSigned-off-bytrailer. This needsgit commit --amend -s(and a force-push) to pass. - PR is still a draft — needs to be marked "Ready for review" before it's mergeable regardless of review state.
No code changes requested — once the DCO sign-off is added and the PR is marked ready, this looks good to merge from a correctness standpoint.
shouldSendLimitAlert returned `count > max && !isNotified`, which is one
value carrying two decisions. The first alert past the limit is dropped
and sets isNotified; that makes the condition false for every later
alert in the same minute, so they are all sent. The limiter therefore
dropped exactly one alert per minute and let the rest through.
admitAlert returns the two decisions separately: admitted is false for
every alert past the limit, notify is true only for the first one.
Three smaller things fixed with it:
- ReportAlertSuppressed(..., "rate_limit") now fires for every dropped
alert instead of once per minute, so the metric becomes usable.
- The window reset no longer swallows the alert that triggers it. The
old code returned early without counting, so the effective limit was
max + 1.
- sendAlertLimitReached no longer reads count and startTime without
holding the lock.
A limit of zero or less means no limit. Validate() already replaces a
zero with the default, so this only guards an exporter built by hand.
Documented in docs/features/alert-rate-limit.md.
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
444b043 to
7571aae
Compare
matthyx
left a comment
There was a problem hiding this comment.
Re-checked: both blockers from my previous review are resolved.
- DCO now passes — the commit carries a
Signed-off-bytrailer. - PR is no longer a draft.
Code is unchanged since my last pass (http_exporter.go diff is identical) and I already traced admitAlert() by hand against the original bug — the fix is correct, the mutex covers the whole decision, and the new tests (including the concurrent-sender case) exercise the right behavior. CodeRabbit's automated review also came back clean with no actionable comments.
Approving.
Summary
The per-minute alert limit stops limiting after its first drop, so it drops exactly one alert
per minute and sends every later one. In practice there is no limit.
shouldSendLimitAlertreturnscount > max && !isNotified— one value carrying two decisions.The first alert past the limit is dropped and sets
isNotified, which makes the condition falsefor every later alert in that minute, so they are all sent.
Ticket
None — this repository has no ticket link.
Changes
admitAlertreplacesshouldSendLimitAlertand returns the two decisions separately:admittedis false for every alert past the limit,notifyis true only for the first one.AlertLimitReachednotice only on the first.
ReportAlertSuppressed(..., "rate_limit")now fires for everydropped alert instead of once per minute, so that metric becomes usable.
Validate()already replaces a zero with the default,so this only guards an exporter built by hand.
without counting, so the effective limit was
max + 1.sendAlertLimitReachedno longer sets the notified flag —admitAlertowns that decision —and no longer reads
countandstartTimewithout the lock.docs/features/alert-rate-limit.md.Testing
go build ./pkg/exporters/forGOOS=linux→ pass. New tests inpkg/exporters/alert_limit_test.go: the limiter keeps limiting after its first drop, the windowresets, zero means no limit, and twenty concurrent senders admit exactly the limit while the
notice is decided once. Run on Linux with
-race→ pass, together with the existingTestSendRuleAlertRateReached.AI Review
Local review with
armosec-shared-rules:code-review-standards(Opus, high effort) over thesibling change in the private agent, which carries the identical fix. Verdict: the limiter logic
is correct, the mutex covers the whole decision, and all call sites are consistent. Its one
finding was about the consequence rather than the code — a real limit can starve one alert kind
when consumers share an exporter — which is why
ReportAlertSuppressednow counts every drop.AI-skills: armosec-shared-rules:agent-dispatch-policy
Summary by CodeRabbit
Bug Fixes
Documentation