StepEfficiency can return a negative score when a step contains multiple redundant tool calls

[usegitdivyansh]

Affected package

ai-evaluation (Python)

Package version

ai-evaluation==1.1.0

What happened?

StepEfficiency is documented to return a score from 0.0 to 1.0, but it can return a negative value when a single trajectory step contains more than one redundant tool call.

The cause is a denominator mismatch in the redundancy calculation. redundant_count is incremented once per redundant tool call (the inner loop runs over step.tool_calls), but redundancy_ratio divides that count by total_steps, the number of steps:

redundancy_ratio = 1.0 - (redundant_count / total_steps) if total_steps > 0 else 1.0

Because a single step can hold several tool calls, redundant_count can exceed total_steps. When it does, redundant_count / total_steps is greater than 1, so redundancy_ratio becomes negative, and since it is not clamped, the final score drops below 0.

Worth noting that the failure calculation immediately below uses the consistent denominator (failed_calls / total_calls, both per tool call), which suggests the redundancy block was meant to divide by the number of tool calls as well.

This also propagates to TrajectoryScore, which composes StepEfficiency at 30 percent weight, so a negative efficiency score pulls the composed score down too.

Minimal reproduction

from fi.evals.metrics.agents.metrics import StepEfficiency
from fi.evals.metrics.agents.types import (
    AgentTrajectoryInput, AgentStep, ToolCall, TaskDefinition,
)

# One step that makes 5 identical calls (4 of them redundant)
step = AgentStep(
    step_number=1,
    tool_calls=[
        ToolCall(name="search", arguments={"q": "x"}, success=True)
        for _ in range(5)
    ],
    is_final=True,
)

inp = AgentTrajectoryInput(
    trajectory=[step],
    task=TaskDefinition(description="search"),
)

result = StepEfficiency().compute_one(inp)
print(result["output"])   # -0.2
print(result["details"])  # {'total_steps': 1, 'redundant_steps': 4, 'failed_calls': 0}

Error output / stack trace

No exception is raised. The metric silently returns an out of range value:

-0.2
{'total_steps': 1, 'redundant_steps': 4, 'failed_calls': 0}

Environment

ai-evaluation==1.1.0
Python 3.13
Ubuntu 24.04
Local metrics only (no cloud / gateway)

Anything else?

Happy to open a PR. There look to be two reasonable ways to fix it, and I would rather confirm the intended semantics before sending code:

1. Divide redundant_count by total_calls instead of total_steps. This matches the failure calculation right below it and keeps the ratio in [0, 1] by construction.

2. Keep the "per step" denominator but clamp redundancy_ratio to [0, 1].

These give different scores for the same trajectory, since option 1 measures redundancy as a fraction of calls while option 2 measures it as a fraction of steps. Which behavior do you intend? Once confirmed, I will send a PR with the fix and tests covering the multiple-calls-per-step case (the current test_redundant_steps only uses one call per step, so this path is untested).

[abhiramArise]

Agree — option 1. The failure calc's failed_calls / total_calls is the intended pattern; total_steps in the redundancy line looks like a leftover from before multi-call-per-step support was added. Per-call denominator keeps the ratio bounded by construction, no clamp needed. Happy to review the PR once you send it — make sure the new test covers ≥2 redundant calls within a single step (your repro above) alongside the existing single-call-per-step case.

[usegitdivyansh]

Agree — option 1. The failure calc's failed_calls / total_calls is the intended pattern; total_steps in the redundancy line looks like a leftover from before multi-call-per-step support was added. Per-call denominator keeps the ratio bounded by construction, no clamp needed. Happy to review the PR once you send it — make sure the new test covers ≥2 redundant calls within a single step (your repro above) alongside the existing single-call-per-step case.

Thanks for the quick response. Option 1 it is, no clamp. I'll send a PR that changes the redundancy denominator to total_calls and adds a test with multiple redundant calls in a single step, keeping the existing single-call-per-step test. Will reference this issue.