Skip to content

[BUGFIX] Prometheus: ignore empty series selectors in label variables - #862

Merged
AntoineThebaud merged 1 commit into
perses:mainfrom
rajusem:fix/prometheus-label-variables-empty-matcher
Oct 6, 2026
Merged

AntoineThebaud merged 1 commit into
perses:mainfrom
rajusem:fix/prometheus-label-variables-empty-matcher

Conversation

@rajusem

@rajusem rajusem commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Description

Closes perses/perses#3198.

"Add Series Selector" adds an empty "Series Selector" field to a Prometheus label values or label names variable. Both plugins sent every selector to Prometheus as a match[] value, the empty one included, and Prometheus rejects the whole request:

GET /api/v1/label/job/values?match[]=
400 {"status":"error","errorType":"bad_data","error":"unknown position: parse error: unexpected end of input"}

So the variable preview showed this error when "Run Query" was clicked with the field still empty, and a variable saved with an empty selector had no values on dashboards. In the dashboard's "Edit variables" drawer, such a variable never showed its form: the form's preview re-sent the failing request on each mount, which put the drawer back on its spinner (16 to 19 times in 5 s against the Prometheus demo). And once a variable depending on it had been added or changed there ("Add" or "Apply" in the variable form, before the drawer's own "Apply"), the drawer usually stayed on its spinner whichever variable was opened.

Both plugins now leave out the selectors that are empty, contain only whitespace, or become empty after the variable interpolation they already did (e.g. $selector with a text variable set to ""), through a new internal helper, interpolateMatchers (prometheus/src/plugins/interpolation.ts, not exported from the package). Without match[], Prometheus does not filter, so the preview shows the same values as before the selector was added. Non-empty selectors are sent unchanged. The Loki label variables of this repository already drop an empty selector, and Grafana only sends match[] when the selector is not empty. Doing this in the plugins rather than in MatcherEditor also covers saved dashboards, Dashboard-as-Code and selectors that only become empty after interpolation. The Prometheus client is left as is: it is a lower-level API exported by the package, also used by the metrics finder, and it sends every match[] value it is given. There is no schema, spec or public API change, and saved dashboards stay valid. docs/prometheus/model.md now describes matchers and says that empty entries are ignored.

The issue suggests not sending a query while the field is empty. Since perses/perses#3590, adding the field no longer sends a request by itself: the request is sent on "Run Query" and when a dashboard loads the variable, and skipping it there would leave the preview on "No results" and the variable without values. So the request is still sent, without the empty selectors.

Behavior change on dashboard load: a list variable without a default value and without a value in the URL is null until its options arrive, and perses/shared does not make the variables that use it wait. A variable whose selector is only that variable (e.g. $metric, as the Grafana migration produces for label_values($metric, instance)) is now requested once without match[] instead of failing with a 400, so each such load fetches the label's whole value list, which can be large for a high-cardinality label (the plugin sends no limit). When that response arrives before the parent's options, the variable lists the unfiltered values until its filtered request returns, and keeps the value it took from them if the filtered list is empty, as it already does when its parent switches to a value without matches. A variable that uses it can then be requested once more, with that value. Making these variables wait in perses/shared would avoid these requests.

Tests: new unit tests for both plugins check the match[] values of the single request sent to Prometheus (URL query string for label values, form body for label names): no selector, empty and whitespace-only selectors, a selector that is empty or only whitespace after interpolation, and interpolated non-empty selectors, which are sent unchanged and in order. Without the fix, the 4 tests with empty selectors fail.

This only removes the empty-selector cause. A variable that fails for another reason (e.g. the selector {}, which Prometheus rejects with "match[] must contain at least one non-empty matcher", or a datasource that is down) still loops when it is opened in the drawer, and still blocks the form once a variable depending on it was added or changed there. Both need a fix in perses/shared.

Screenshots

No screenshot: this build was not run in a Perses UI. Instead, the issue's steps were run in the real VariableEditorForm, with the plugins' own editors, against the Prometheus demo (a scratch component test, not part of this PR). Before, "Add Series Selector" then "Run Query" showed unknown position: parse error: unexpected end of input for both plugins. Now the preview lists the same values as before the selector was added, and once a selector is typed, "Run Query" filters them, as it did before this change.

Both plugins were run with the real Prometheus client against a real Prometheus (the public demo, 3.13.0), before and after this change (label values of job and label names, one request each):

Series selectors Before After
none all values, no match[] same request and result
[''], [' '], ['\t\n'], ['$sel'] with sel set to "" 400 parse error: unexpected end of input no match[], all values
['up{job="$job"}', ''] with job set to prometheus 400 only match[]=up{job="prometheus"} is sent, filtered values
[' up{job="prometheus"} '] filtered values same request (spaces kept) and result

The drawer was checked against the same Prometheus with the real dashboard VariableProvider and Variable components, and the real useResolveListVariableValues and VariableEditorForm in a copy of the drawer's VariableEditorFormWithContext: a variable with an empty selector now shows its form at once, after one request without match[] (before: the spinner for the whole 5 s, and 17 to 20 requests). With a variable depending on it added or changed in the drawer, useResolveListVariableValues now finishes loading (before: still loading after 5 s).

Checklist

  • Pull request has a descriptive title and context useful to a reviewer.
  • Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
    following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
  • All commits have DCO signoffs.

UI Changes

  • Changes that impact the UI include screenshots and/or screencasts of the relevant changes. (No screenshot: this build was not run in a Perses UI; checked against a real Prometheus instead, see above.)
  • Code follows the UI guidelines. (N/A)

"Add Series Selector" adds an empty series selector to a Prometheus
label values or label names variable. The plugins sent it to Prometheus
as an empty match[], which Prometheus rejects ("unknown position: parse
error: unexpected end of input"). So "Run Query" in the variable editor
showed this error while the field was empty, a variable saved with an
empty selector had no values on dashboards, and opening it in the
dashboard's "Edit variables" drawer showed a loading spinner that never
ended.

The selectors that are empty, contain only whitespace, or become empty
after variable interpolation are now left out. Without any match[],
Prometheus does not filter the result, as when the variable has no
selector. Non-empty selectors are sent unchanged. The model docs now
say that empty entries are ignored.

Closes perses/perses#3198

Signed-off-by: Raj Zalavadia <rzalavad@redhat.com>
@rajusem
rajusem requested review from a team and AntoineThebaud as code owners October 6, 2026 04:06
@rajusem
rajusem requested review from jgbernalp and removed request for a team October 6, 2026 04:06

@AntoineThebaud AntoineThebaud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! Nice catch for this situation we didn't detect until now:

In the dashboard's "Edit variables" drawer, such a variable never showed its form: the form's preview re-sent the failing request on each mount, which put the drawer back on its spinner

I'd actually prefer that we reject querying & saving while there is an empty series selector set, rather than silently ignoring it, as it makes no sense saving such selector anyway & it would make things more explicit. So concretely it would mean rejecting empty strings for matchers both on CUE and TS side. But as we are very close to the end of the current release cycle, I'd rather have your fix as is than nothing, so let's keep this for later.

@AntoineThebaud
AntoineThebaud added this pull request to the merge queue Oct 6, 2026
Merged via the queue into perses:main with commit ef6baf7 Oct 6, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ListVariable: Clicking on Add Series Selector throws an unhandled exception

2 participants