Skip to content

fix(personaplex): avoid timing discarded audio decode - #1783

Closed
chaofengw-nv wants to merge 2 commits into
NVIDIA:mainfrom
chaofengw-nv:fix/personaplex-reference-unused-decode-20261017
Closed

chaofengw-nv wants to merge 2 commits into
NVIDIA:mainfrom
chaofengw-nv:fix/personaplex-reference-unused-decode-20261017

Conversation

@chaofengw-nv

@chaofengw-nv chaofengw-nv commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Background

Fixes #1782. PersonaPlex's Native benchmark decodes each emitted audio frame twice inside the measured task. Only the first result is returned; the auxiliary decode is discarded. This adds codec work to Native Task P50 that the TRTMC task does not perform.

Exit Criteria

Perform one acoustic decode per returned frame in the active AIPerf Native benchmark, preserve the official warmup and quality oracle, and verify the returned audio plus operation count with a CPU regression. Real GPU latency must be remeasured before using a new Native P50.

Implementation

Remove the unused auxiliary decode from the AIPerf reference adapter. Keep the auxiliary codec for the existing pinned official warmup, avoiding a separate warmup implementation. The official accuracy oracle, generation configuration, requests and quality gates are unchanged.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Validation

Commands and Results

  • CPU, CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python -m pytest -c pyproject.toml -q -p no:cacheprovider families/personaplex/tests/test_reference_decode_work.py: 1 passed. The test runs the real adapter invocation with deterministic codec/generator stand-ins, checks the returned waveform and two-frame budget, and requires two primary decodes and zero discarded decodes.
  • Original adapter negative control using the identical test: 1 failed, because it performs two discarded decodes (2 != 0). Returned audio assertions already pass on the original path.
  • GB300 original-request replay using the active adapter: FP16 and FP32 each completed all 20 requests. Every PCM SHA256 is present and matches its frozen counterpart (20/20 for each precision), and all 20 output-frame counts match. These are frame counts, not a token-ID comparison.
  • Fresh Native Task P50 (model_call_ms, n=20 each): FP16 2106.749 ms, FP32 3278.766 ms. Each run emitted 2405 frames in total. Historical controls are retained separately; these timings retain the original numerical policy.
  • ruff check --config ruff.toml on both changed files, python -m tools.model_ci validate, python -m tools.test_impact --base github/main --validate, and git diff --check: passed.

Hardware, Environment, and Revisions

  • Head 0e0ea393ee9649597da6bef04c76285acf0af6ee; base 9643800b3b5a951382c87ae1ebfbbeb24b1a8090.
  • CPU regression: x86-64, PyTorch 2.12, GPU access disabled; no checkpoint or GPU kernel was used.
  • Native replay: NVIDIA GB300, ARM64, CUDA 13, PyTorch 2.12.0+cu130; checkpoint revision fdaf4090a61cb315c138a1faee287ffd6c716309. The exercised adapter SHA256 5cf0414991d59e94eea0c47edff1b72d20a0554a4b55ec1e23f67f9918667fed matches this head.

Not Run / Remaining Gaps

Protected exact-head premerge, other platforms, and repeated latency runs are not qualified. The fresh timings are measurements from one full 20-request run at each precision; no confidence interval or full-model TRTMC accuracy claim is made. Historical Native P50 values include the extra decode and remain labeled separately. Numerical-precision alignment is separate work.

Contributor Self-Review

  • I have completed a self-review of this change.

Reviewed the measured call site, dataflow of the discarded value, the unchanged official warmup and accuracy oracle, the two-file family-local scope, CPU negative control, test assertions, and DCO.

Notes For Future Readers

This corrects benchmark work accounting rather than optimizing the model. Compare newly measured Native timings with TRTMC only after confirming the original requests and real output behavior. Do not rewrite historical measurements or interpret the CPU stand-ins as full-model accuracy proof.

Risk level

  • Low
  • Medium
  • High

The removed result has no consumer, and the auxiliary codec does not feed the generator or returned waveform. Original-request GPU replay confirms identical PCM for both exercised precisions.

Decode each returned speech frame once in the Native benchmark. The auxiliary codec output was discarded but its decode was included in task latency; keep the existing official warmup and quality oracle unchanged.

Add a CPU regression that verifies the returned waveform and counts work for a bounded generation request.

Refs: NVIDIA#1782
Signed-off-by: chaofengw <chaofengw@nvidia.com>
@chaofengw-nv chaofengw-nv self-assigned this Oct 17, 2026
@coderabbitai

coderabbitai Bot commented Oct 17, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Keep this change on the active AIPerf reference path. The legacy benchmark
entry point depends on the retired qualification harness and is outside
this fix.

Refs: NVIDIA#1782
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Oct 20, 2026
Decode each returned audio frame once while retaining the official warmup and accuracy oracle.

Closes NVIDIA#1782
Refs NVIDIA#1783

Signed-off-by: chaofengw <chaofengw@nvidia.com>
@chaofengw-nv

Copy link
Copy Markdown
Collaborator Author

Superseded by #1809. This change and its regression coverage are preserved in commit 70e49b3 of the consolidated PR. Original issue links and acceptance criteria remain tracked there.

chaofengw-nv added a commit that referenced this pull request Oct 21, 2026
* fix(personaplex): avoid timing discarded audio decode

Decode each returned audio frame once while retaining the official warmup and accuracy oracle.

Closes #1782
Refs #1783

Signed-off-by: chaofengw <chaofengw@nvidia.com>

* feat(personaplex): score spoken QA with VoiceBench

Collect spoken QA accuracy and task timing from the same inference outputs using the pinned VoiceBench questions, official voice prefix and offline transcription. Preserve explicit unanswered and ambiguous responses and the paired statistical verdict.

Signed-off-by: chaofengw <chaofengw@nvidia.com>

* fix(personaplex): feed both sides the same 24 kHz audio

VoiceBench and LibriSpeech questions are 16 kHz. Native resampled them with
librosa (soxr HQ) while TRTMC used its own windowed-sinc resampler, so the
two sides answered slightly different inputs. Resample each suite once with
SciPy polyphase (audio_sample_rate: 24000) and make the Native reference
reject audio it would have to resample. librosa no longer touches the model
input; the speech-parity suite moves to version 2.

Refs #1749

Signed-off-by: chaofengw <chaofengw@nvidia.com>

---------

Signed-off-by: chaofengw <chaofengw@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] PersonaPlex Native benchmark times discarded audio decoding

1 participant