Skip to content

fix: preserve alias in metric tests for all generic test helpers - #1908

Open
CTWalk wants to merge 1 commit into
evidentlyai:mainfrom
CTWalk:metric-test-alias-parity
Open

fix: preserve alias in metric tests for all generic test helpers#1908
CTWalk wants to merge 1 commit into
evidentlyai:mainfrom
CTWalk:metric-test-alias-parity

Conversation

@CTWalk

@CTWalk CTWalk commented Jul 30, 2026

Copy link
Copy Markdown

Thanks for maintaining Evidently. This is a small consistency fix in the
generic test helpers; the reproduction and reasoning are below.

Related issue: #1907

What

gte() forwards alias= into its metric test. The seven sibling helpers
(eq, not_eq, gt, lt, lte, is_in, not_in) also accept alias=,
but forward it only to DescriptorTest. When one of them is used as a
metric-level test in a Report, the configured alias is silently dropped.

Why it matters

MetricTest.alias is the custom name used to build the test description:

# src/evidently/core/metric_types.py
description = f"{self.alias or value.display_name}: {result.description}"

For seven of the eight helpers that field can never be set through the public
helper, so a name the user configured never reaches the report.

Reproduce

On main (a4aa4c2, 0.7.21):

import pandas as pd

from evidently import Report
from evidently.metrics import MeanValue
from evidently.tests import gt, gte

data = pd.DataFrame({"score": [0.0, 1.0]})
for helper in (gte, gt):
    test = helper(1, alias="quality gate")
    snapshot = Report([MeanValue(column="score", tests=[test])]).run(data)
    print(helper.__name__, test.metric.alias)
    print(" ", snapshot.tests_results[0].description)

Before:

gte quality gate
  quality gate: Actual value 0.500 < 1.000
gt None
  Mean value of 'score': Actual value 0.500 < 1.000

After:

gte quality gate
  quality gate: Actual value 0.500 < 1.000
gt quality gate
  quality gate: Actual value 0.500 < 1.000

Change

  • src/evidently/tests/aliases.py — pass the already accepted alias
    argument into the metric-test constructor in the seven helpers. gte is
    untouched and serves as the reference behavior. ruff format rewraps five
    of those calls onto multiple lines; the only semantic change is
    alias=alias.
  • tests/future/tests/test_generic_tests_creation.py — one parameterized
    regression over all eight helpers, asserting the alias survives on both the
    metric and the descriptor.

No new API, parameter, state, or dependency. alias defaults to None, so
behavior is unchanged for every caller that does not pass it.

Checks

Run locally against this branch on Python 3.12:

  • pytest tests/future/tests/test_generic_tests_creation.py7 failed, 8 passed before the source change, 15 passed after. Reverting only the
    source change brings the seven failures back, so the new test does pin the
    behavior.
  • All non-UI tests/future1626 passed, 6 skipped, no failures.
  • ruff check and ruff format --check on both files, using the repository's
    ruff.toml and pinned ruff==0.3.7.

Every other alias= usage in the test tree is descriptor-level, so no
existing test exercises the changed metric path.

Not run locally: tests/future/test_ui (needs sqlalchemy), the legacy
tests/ tree, and mypy.

One open question

The @overload signatures in evidently/tests/aliases.py currently declare
alias only on the DescriptorTest-returning forms, so a typed
gt(1, alias="quality gate") resolves to the descriptor overload. This PR
deliberately leaves the overloads alone — the same gap already applies to
gte(), so nothing gets worse — but if you would like the metric-shaped
overloads to declare alias as well, I am happy to extend this PR.

The generic test helpers accept alias=, but only gte() forwards it into
the metric-level test. eq(), not_eq(), gt(), lt(), lte(), is_in(), and
not_in() forward it only to DescriptorTest, so an alias configured on a
Report metric test was silently dropped and the result description fell
back to the metric display name.

MetricTest.alias is the value used to build that description in
core/metric_types.py ("{self.alias or value.display_name}: ..."), so for
seven of the eight helpers the field could not be set through the public
helper at all.

Pass the already accepted alias into the seven metric constructors.
alias defaults to None, so behavior is unchanged for callers that do not
pass it. gte() is untouched and serves as the reference.

Add a parameterized regression over all eight helpers, asserting the
alias survives on both the metric and the descriptor.

Fixes evidentlyai#1907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant