Skip to content

[None][test] Add InferenceMAX-style GSM8K accuracy eval mode - #18738

Draft
zheyuf wants to merge 1 commit into
NVIDIA:mainfrom
zheyuf:zheyu/test/gsm8k-inferencemax-main
Draft

[None][test] Add InferenceMAX-style GSM8K accuracy eval mode#18738
zheyuf wants to merge 1 commit into
NVIDIA:mainfrom
zheyuf:zheyu/test/gsm8k-inferencemax-main

Conversation

@zheyuf

@zheyuf zheyuf commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an InferenceMAX-style GSM8K accuracy eval mode so model accuracy can be validated under the exact protocol published at inferencex.semianalysis.com/evaluation (SemiAnalysisAI/InferenceX), directly comparable to the vLLM/SGLang reference numbers there.

The existing in-repo gsm8k task measures several points below InferenceMAX's leaderboard numbers (MiniMax-M3-NVFP4: ~90.5 vs ~95.5) because it applies no chat template, runs no thinking, and caps output at 256 tokens.

The new mode is model-agnostic and easy to extend to other InferenceMAX-tracked models (DSR1, GPT-OSS, Kimi, GLM, ...). This is a port of the side-branch commit feat/m3_with_msa #16711 onto main.

What's in the PR

  • tensorrt_llm/evaluate: GSM8KInferenceMax evaluator + lm_eval_tasks/gsm8k_inferencemax/gsm8k_inferencemax.yaml (mirrors InferenceX utils/evals/gsm8k.yaml: chat template, 5-shot multiturn, #### [number] format instruction, max_gen_toks: 12288, strict #### N extraction). LmEvalEvaluator gains shuffle_dataset / fewshot_random_seed so the task reproduces the lm-eval CLI's exemplar selection (unshuffled train split, sampler seed 1234); defaults for every other task are unchanged.
  • trtllm-eval gsm8k_inferencemax CLI subcommand.
  • Accuracy harness: GSM8KInferenceMax task in accuracy_core.py; reference rows may now carry an explicit threshold (replaces the hypothesis-test threshold with a fixed floor, reported as such) and a free-text reference_note (provenance of the reference score). Both are optional and no existing row uses them.
  • TestMiniMaxM3::test_nvfp4 is parametrized by eval_mode: default = MMLU + GSM8K as before; inferencemax = chat-format GSM8K at 16k context with max_batch_size=64 (InferenceMAX evaluates a serving endpoint at client concurrency 64).
  • Test lists: the QA list (llm_function_core.txt) runs the inferencemax variants (TIMEOUT 180, thinking output dominates in eager mode); pre-merge l0_dgx_b200 keeps eval_mode=default. Existing waives are renamed to the new ids.

Accuracy thresholds (references/gsm8k_inferencemax.yaml)

Reference scores are the published InferenceMAX vLLM B200 FP4 numbers (2026-07); a strict explicit floor is used to guard model quality during bring-up / perf work.

Test Reference (source) Threshold
test_nvfp4[use_msa=False-eval_mode=inferencemax] 95.46, vLLM B200 FP4 spec-off mean [94.84-95.91] n=13 94.0
test_nvfp4[use_msa=True-eval_mode=inferencemax] 95.46, same row, FP8 KV cache 94.0

Side-branch measurements (4x B200, TP4/EP4, 2026-07-22): 95.53 (triton, BF16 KV) and 95.68 (MSA, FP8 KV); run-to-run SD 0.26, per-run lm-eval stderr 0.58. InferenceMAX's own gate for this task is 90.0.

Differences from the side-branch commit

Test coverage

  • accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=*-eval_mode=inferencemax] (QA list).
  • accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=True-eval_mode=default] (pre-merge l0_dgx_b200, currently waived on main under nvbugs/6601633; waive renamed).

To reviewers

  • As on the side branch, the QA list switches the MiniMax-M3 test_nvfp4 entries to InferenceMAX mode instead of adding the variants alongside the default ones (to keep QA time flat). Happy to keep both if preferred.
  • The mode has only been measured on the side branch so far; CI (/bot run) on this PR is the first run on main.

PR Checklist

  • PR title follows the [JIRA/NVBUG/None][type] format
  • Commits are signed off (DCO)
  • pre-commit clean (isort / yapf / test-list validators)
  • CI (/bot run)

Adds a `gsm8k_inferencemax` lm-eval task and evaluator that reproduce the
public InferenceMAX (SemiAnalysisAI/InferenceX) GSM8K protocol: chat
template with 5-shot multiturn exemplars, an explicit "#### [number]"
answer-format instruction, a 12288-token generation budget for thinking
models, strict "#### N" extraction, and the lm-eval CLI's exemplar
selection (unshuffled train split, sampler seed 1234). Scores are directly
comparable to inferencex.semianalysis.com/evaluation, unlike the
completion-format `gsm8k` task, which runs several points lower on
chat/thinking-tuned models.

The accuracy harness gains a matching `GSM8KInferenceMax` task plus
optional `threshold` / `reference_note` fields on reference rows, so a
benchmark-style explicit floor with published provenance can replace the
hypothesis-test threshold. `TestMiniMaxM3::test_nvfp4` is parametrized by
`eval_mode` (default = MMLU + GSM8K, inferencemax = chat GSM8K at 16k
context); the QA list runs the inferencemax variants, pre-merge keeps the
default variant.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant