Repository navigation
[BUGFIX] Prometheus: ignore empty series selectors in label variables - #862
Conversation
"Add Series Selector" adds an empty series selector to a Prometheus
label values or label names variable. The plugins sent it to Prometheus
as an empty match[], which Prometheus rejects ("unknown position: parse
error: unexpected end of input"). So "Run Query" in the variable editor
showed this error while the field was empty, a variable saved with an
empty selector had no values on dashboards, and opening it in the
dashboard's "Edit variables" drawer showed a loading spinner that never
ended.
The selectors that are empty, contain only whitespace, or become empty
after variable interpolation are now left out. Without any match[],
Prometheus does not filter the result, as when the variable has no
selector. Non-empty selectors are sent unchanged. The model docs now
say that empty entries are ignored.
Closes perses/perses#3198
Signed-off-by: Raj Zalavadia <rzalavad@redhat.com>
AntoineThebaud
left a comment
There was a problem hiding this comment.
Thanks for the fix! Nice catch for this situation we didn't detect until now:
In the dashboard's "Edit variables" drawer, such a variable never showed its form: the form's preview re-sent the failing request on each mount, which put the drawer back on its spinner
I'd actually prefer that we reject querying & saving while there is an empty series selector set, rather than silently ignoring it, as it makes no sense saving such selector anyway & it would make things more explicit. So concretely it would mean rejecting empty strings for matchers both on CUE and TS side. But as we are very close to the end of the current release cycle, I'd rather have your fix as is than nothing, so let's keep this for later.
Description
Closes perses/perses#3198.
"Add Series Selector" adds an empty "Series Selector" field to a Prometheus label values or label names variable. Both plugins sent every selector to Prometheus as a
match[]value, the empty one included, and Prometheus rejects the whole request:So the variable preview showed this error when "Run Query" was clicked with the field still empty, and a variable saved with an empty selector had no values on dashboards. In the dashboard's "Edit variables" drawer, such a variable never showed its form: the form's preview re-sent the failing request on each mount, which put the drawer back on its spinner (16 to 19 times in 5 s against the Prometheus demo). And once a variable depending on it had been added or changed there ("Add" or "Apply" in the variable form, before the drawer's own "Apply"), the drawer usually stayed on its spinner whichever variable was opened.
Both plugins now leave out the selectors that are empty, contain only whitespace, or become empty after the variable interpolation they already did (e.g.
$selectorwith a text variable set to""), through a new internal helper,interpolateMatchers(prometheus/src/plugins/interpolation.ts, not exported from the package). Withoutmatch[], Prometheus does not filter, so the preview shows the same values as before the selector was added. Non-empty selectors are sent unchanged. The Loki label variables of this repository already drop an empty selector, and Grafana only sendsmatch[]when the selector is not empty. Doing this in the plugins rather than inMatcherEditoralso covers saved dashboards, Dashboard-as-Code and selectors that only become empty after interpolation. The Prometheus client is left as is: it is a lower-level API exported by the package, also used by the metrics finder, and it sends everymatch[]value it is given. There is no schema, spec or public API change, and saved dashboards stay valid.docs/prometheus/model.mdnow describesmatchersand says that empty entries are ignored.The issue suggests not sending a query while the field is empty. Since perses/perses#3590, adding the field no longer sends a request by itself: the request is sent on "Run Query" and when a dashboard loads the variable, and skipping it there would leave the preview on "No results" and the variable without values. So the request is still sent, without the empty selectors.
Behavior change on dashboard load: a list variable without a default value and without a value in the URL is
nulluntil its options arrive, and perses/shared does not make the variables that use it wait. A variable whose selector is only that variable (e.g.$metric, as the Grafana migration produces forlabel_values($metric, instance)) is now requested once withoutmatch[]instead of failing with a 400, so each such load fetches the label's whole value list, which can be large for a high-cardinality label (the plugin sends nolimit). When that response arrives before the parent's options, the variable lists the unfiltered values until its filtered request returns, and keeps the value it took from them if the filtered list is empty, as it already does when its parent switches to a value without matches. A variable that uses it can then be requested once more, with that value. Making these variables wait in perses/shared would avoid these requests.Tests: new unit tests for both plugins check the
match[]values of the single request sent to Prometheus (URL query string for label values, form body for label names): no selector, empty and whitespace-only selectors, a selector that is empty or only whitespace after interpolation, and interpolated non-empty selectors, which are sent unchanged and in order. Without the fix, the 4 tests with empty selectors fail.This only removes the empty-selector cause. A variable that fails for another reason (e.g. the selector
{}, which Prometheus rejects with "match[] must contain at least one non-empty matcher", or a datasource that is down) still loops when it is opened in the drawer, and still blocks the form once a variable depending on it was added or changed there. Both need a fix in perses/shared.Screenshots
No screenshot: this build was not run in a Perses UI. Instead, the issue's steps were run in the real
VariableEditorForm, with the plugins' own editors, against the Prometheus demo (a scratch component test, not part of this PR). Before, "Add Series Selector" then "Run Query" showedunknown position: parse error: unexpected end of inputfor both plugins. Now the preview lists the same values as before the selector was added, and once a selector is typed, "Run Query" filters them, as it did before this change.Both plugins were run with the real Prometheus client against a real Prometheus (the public demo, 3.13.0), before and after this change (label values of
joband label names, one request each):match[][''],[' '],['\t\n'],['$sel']withselset to""parse error: unexpected end of inputmatch[], all values['up{job="$job"}', '']withjobset toprometheusmatch[]=up{job="prometheus"}is sent, filtered values[' up{job="prometheus"} ']The drawer was checked against the same Prometheus with the real dashboard
VariableProviderandVariablecomponents, and the realuseResolveListVariableValuesandVariableEditorFormin a copy of the drawer'sVariableEditorFormWithContext: a variable with an empty selector now shows its form at once, after one request withoutmatch[](before: the spinner for the whole 5 s, and 17 to 20 requests). With a variable depending on it added or changed in the drawer,useResolveListVariableValuesnow finishes loading (before: still loading after 5 s).Checklist
[<catalog_entry>] <commit message>naming convention using one of thefollowing
catalog_entryvalues:FEATURE,ENHANCEMENT,BUGFIX,BREAKINGCHANGE,DOC,IGNORE.UI Changes