agent-simulation: reshape JobMetrics for the measured metric set - #1695
Open
theomonnom wants to merge 28 commits into
Open
agent-simulation: reshape JobMetrics for the measured metric set#1695theomonnom wants to merge 28 commits into
theomonnom wants to merge 28 commits into
Conversation
The simulation worker's metrics were reworked; this brings the wire format in line with what it can actually measure. Added: - conciseness, conversation_progression and its four issue flags (unnecessary_tool_calls, information_loss, redundant_statements, poor_question_quality), judged over the authored dialog. The flags are first-class, and reach experience_score only through the derived progression rating, so they are never double-counted. - entity_recognition + its pooling pair on both STT and TTS: key entities are LLM-extracted from the dialog and scored by occurrence survival over the WER alignment, in both directions (did the agent hear ours, did the caller hear the agent's). - response_latency_samples, so the percentile pool size is visible - cut-ins are excluded from the percentiles and counted separately. - Turn.conciseness, and conciseness/conversation_progression means on RunMetrics. Removed (reserved by number and name): - keyterm_* on STT, replaced by the entity_* methodology. - speech_rate_wpm and its pooling stats: a real speaking rate needs a prosody model, and the word-count proxy was misleading. - silence_total_ms, derivable from the turn edges. - false_interruption_unrecovered_count: recovery is no longer tracked. - LLM.conciseness_score: conciseness is a dialog judgment, not an LLM-node timing, and now lives on JobMetrics/Turn. Renamed unanswered_persona_turns to unanswered_turns, keeping field 12; the worker emits both names during the transition so the bump is a no-op.
|
|
Théo Monnom seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Nothing aggregates them: the run aggregation pools group fields, and the only JobMetrics-level scalars it means by hand are accuracy_score, experience_score and scenario_pass_rate. Both would have shipped empty, and experience_score already carries conciseness and conversation_progression at run level. If run-level judged aggregates are wanted later they should land together with the per-dimension flag rates, once something computes them. Also trims the comments on the fields this branch adds to what the names do not already say.
The weights are editorial and change with the metric set; spelling them out on the wire format only guarantees they go stale.
The proto says what a field is and when it is available; how the worker derives it, with which thresholds and weights, belongs with the worker. One of these was already stale - the progression derivation changed after it was written.
Derivable from data already on the wire: the turn rows carry response_latency_ms and flag cut-ins, so the clean-handoff count is a subtraction the client can do itself.
jobs_total, jobs_measured and jobs_simulator_fault are all counts the client can take off the jobs it already has in the same response. jobs_simulator_fault was also wrong: it is derived from early_termination alone, while the worker withholds scoring on three signals - one of which never reaches the proto. Counting measured jobs whose composites are absent gives the true number, whatever the worker decided.
Nothing consumes these metrics yet, so there is no wire compatibility to protect and the freed numbers can be reused.
Listing a group's fields next to the group only guarantees the list goes stale; jumping to the definition shows them.
Same rule throughout: say what a field is and when it is available, not how the worker computes it or anything the name already gives. - the three headline scores shared one absence rule, now stated once - dropped the scoring recipes (turn_taking penalties, run-WER pooling) and the per-field aggregation notes the Turn block already covers - dropped 'word error rate' next to wer and friends - stated the 0/1-per-turn rule once instead of on every count - fixed an unfinished sentence and replaced the em dashes
Turn carried four group submessages where 28 of 40 fields were structurally always empty: a turn has no wer, no overlap_ratio, no percentiles. It now carries the flat set a turn actually measures, and the 0/1 counts become bools. response_latency_ms moves onto Turn, which is the only scope that ever set it, leaving Conversation a pure aggregate. RunMetrics goes with it. A per-job worker cannot compute a cross-job aggregate, so it existed only because the service recomputed it on every read - from the very jobs it returns in the same response. The client can take those means itself; the worker sends metrics and the service stores them.
A run summary should not have to pull every job and every turn to show one number. Now that Turn is flat, the reused groups are pure aggregates, so the message no longer drags per-turn fields with it.
…fields A turn field and the group that aggregates it now share a name, so the relationship is readable: stt_delay fed transcription_latency_ms on the group but transcription_delay_ms on the turn, hiding that they are the same measurement. Removed what nothing writes: ttfa_ms (the agent reports no time-to-first- audio), naturalness_score and enunciation_score (no audio judge exists), and audio_judge_model, which named that judge.
Every measured field on Turn now carries its ConversationTurn name with the unit suffix changed (stt_delay_s -> stt_delay_ms), and the aggregates follow the same names so a turn value and its mean or percentile read as the same measurement.
A turn field and its aggregate now share a name, so describing both says the same thing twice. Only the availability markers stay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.