Ignore Datadog tracer transport noise by default - #10
Merged
Conversation
The tracer logs its own transport failures at ERROR level: lost traces and stats-payload errors when the POST to the local agent gets a 502 or times out during an agent restart or node churn. These are external and transient, and the verdict is knowable in advance, but they still open investigations. Normalization cannot collapse them. The agent endpoint path varies between occurrences of a single outage, and the tracer appends its own "(occurred: ...)" clause in a date format the leading-timestamp rule does not match, so each occurrence hashes to a distinct fingerprint and is investigated on its own. Add "Datadog Tracer" to the shipped ignore_patterns, with tests covering the drop, the fingerprint divergence that motivates it, and the fact that the pattern does not suppress application errors that merely mention the vendor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
"Datadog Tracer"is added to the shippedignore_patterns, so the tracer's own transport errors no longer open investigations.Why?
The tracer logs its own failures at ERROR level — lost traces, and stats-payload errors — when its POST to the local agent gets a 502 or times out during an agent restart or node churn. They are external, transient, and the verdict is knowable in advance, but each one still costs a full agent run to reach it.
Better normalization would not fix this. Within a single underlying outage the agent endpoint path varies (
/v0.4/tracesvs/v0.6/stats), and the tracer appends its own(occurred: ...)clause in a date format the leading-timestamp rule does not match. Both variations survive into the hash, so one outage fingerprints as several distinct errors and each is investigated separately. A message-pattern ignore is the only thing that collapses them.Anyone running the Datadog source against a service that ships the tracer will hit this, so it belongs in the default config rather than in each operator's local file.
Validation
pytest— 52 passedruff check .— all checks passedconfig.ymlpaymentssample serviceThree tests cover it: the drop itself, the fingerprint divergence that motivates it, and a guard that the pattern does not suppress application errors merely mentioning the vendor.
Risk and operations
Suppression is the risk worth naming: this pattern is deliberately anchored to the tracer's own log prefix rather than the vendor name, so an application error like
failed to publish metric to Datadogis still investigated. That boundary is pinned by a test.Operators upgrading an existing deployment need no migration —
ignore_patternsis read from their localconfig.yml, so they pick this up by adding the line themselves. Filtering happens before fingerprinting, so previously recorded errors are unaffected.