[None][test] Add InferenceMAX-style GSM8K accuracy eval mode - #18738
Draft
zheyuf wants to merge 1 commit into
Draft
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode#18738zheyuf wants to merge 1 commit into
zheyuf wants to merge 1 commit into
Conversation
Adds a `gsm8k_inferencemax` lm-eval task and evaluator that reproduce the public InferenceMAX (SemiAnalysisAI/InferenceX) GSM8K protocol: chat template with 5-shot multiturn exemplars, an explicit "#### [number]" answer-format instruction, a 12288-token generation budget for thinking models, strict "#### N" extraction, and the lm-eval CLI's exemplar selection (unshuffled train split, sampler seed 1234). Scores are directly comparable to inferencex.semianalysis.com/evaluation, unlike the completion-format `gsm8k` task, which runs several points lower on chat/thinking-tuned models. The accuracy harness gains a matching `GSM8KInferenceMax` task plus optional `threshold` / `reference_note` fields on reference rows, so a benchmark-style explicit floor with published provenance can replace the hypothesis-test threshold. `TestMiniMaxM3::test_nvfp4` is parametrized by `eval_mode` (default = MMLU + GSM8K, inferencemax = chat GSM8K at 16k context); the QA list runs the inferencemax variants, pre-merge keeps the default variant. Ported from the feat/m3_with_msa side branch (NVIDIA#16711). Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an InferenceMAX-style GSM8K accuracy eval mode so model accuracy can be validated under the exact protocol published at inferencex.semianalysis.com/evaluation (SemiAnalysisAI/InferenceX), directly comparable to the vLLM/SGLang reference numbers there.
The existing in-repo
gsm8ktask measures several points below InferenceMAX's leaderboard numbers (MiniMax-M3-NVFP4: ~90.5 vs ~95.5) because it applies no chat template, runs no thinking, and caps output at 256 tokens.The new mode is model-agnostic and easy to extend to other InferenceMAX-tracked models (DSR1, GPT-OSS, Kimi, GLM, ...). This is a port of the side-branch commit
feat/m3_with_msa#16711 ontomain.What's in the PR
tensorrt_llm/evaluate:GSM8KInferenceMaxevaluator +lm_eval_tasks/gsm8k_inferencemax/gsm8k_inferencemax.yaml(mirrors InferenceXutils/evals/gsm8k.yaml: chat template, 5-shot multiturn,#### [number]format instruction,max_gen_toks: 12288, strict#### Nextraction).LmEvalEvaluatorgainsshuffle_dataset/fewshot_random_seedso the task reproduces the lm-eval CLI's exemplar selection (unshuffled train split, sampler seed 1234); defaults for every other task are unchanged.trtllm-eval gsm8k_inferencemaxCLI subcommand.GSM8KInferenceMaxtask inaccuracy_core.py; reference rows may now carry an explicitthreshold(replaces the hypothesis-test threshold with a fixed floor, reported as such) and a free-textreference_note(provenance of the reference score). Both are optional and no existing row uses them.TestMiniMaxM3::test_nvfp4is parametrized byeval_mode:default= MMLU + GSM8K as before;inferencemax= chat-format GSM8K at 16k context withmax_batch_size=64(InferenceMAX evaluates a serving endpoint at client concurrency 64).llm_function_core.txt) runs theinferencemaxvariants (TIMEOUT 180, thinking output dominates in eager mode); pre-mergel0_dgx_b200keepseval_mode=default. Existing waives are renamed to the new ids.Accuracy thresholds (
references/gsm8k_inferencemax.yaml)Reference scores are the published InferenceMAX vLLM B200 FP4 numbers (2026-07); a strict explicit floor is used to guard model quality during bring-up / perf work.
test_nvfp4[use_msa=False-eval_mode=inferencemax]test_nvfp4[use_msa=True-eval_mode=inferencemax]Side-branch measurements (4x B200, TP4/EP4, 2026-07-22): 95.53 (triton, BF16 KV) and 95.68 (MSA, FP8 KV); run-to-run SD 0.26, per-run lm-eval stderr 0.58. InferenceMAX's own gate for this task is 90.0.
Differences from the side-branch commit
test_nvfp4_eagle3[...-eval_mode=inferencemax]) and its reference row (published vLLM B200 FP4 Eagle3 mean 95.49) are not in this PR:test_nvfp4_eagle3does not exist onmainyet. They follow with [TRTLLM-14093][feat] Eagle3 support for MiniMax-M3 on the MSA backend #16021 (Eagle3 for MiniMax-M3 on the MSA backend), which will pick upeval_modeon top of this PR.Test coverage
accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=*-eval_mode=inferencemax](QA list).accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=True-eval_mode=default](pre-mergel0_dgx_b200, currently waived onmainunder nvbugs/6601633; waive renamed).To reviewers
test_nvfp4entries to InferenceMAX mode instead of adding the variants alongside the default ones (to keep QA time flat). Happy to keep both if preferred./bot run) on this PR is the first run onmain.PR Checklist
[JIRA/NVBUG/None][type]format/bot run)