Align GenAI perf metrics schema

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

Why

winml perf --runtime winml-genai was reporting LLM timing as a small set of aggregate fields, while classic winml perf used a different report surface. That made it hard to answer basic questions consistently across runtimes:

• Which part of startup was model/session load vs. native model creation?
• Which latency number is request-level TTFT vs. model-compute TTFT?
• Are prefill/decode throughput values warmup-excluded?
• Can RAM/VRAM fields be consumed consistently for classic and GenAI reports?
• Can external benchmark projections be built later without baking tool-specific shapes into the core report?

This PR makes GenAI perf emit a canonical phase-based schema and aligns the shared metadata/memory surfaces with classic perf, while keeping Ollama/OpenVINO-style projections out of the benchmark core.

What changed

GenAI report schema

Adds a canonical top-level GenAI JSON shape:

• schema_version
• benchmark_info
• load
• requests
• aggregate
• optional memory
• optional hw_monitor

The old GenAI-only accuracy placeholder is removed. Accuracy/eval belongs in eval/reporting flows, not raw perf output.

Load/startup metrics

GenAI load now records a breakdown around the onnxruntime-genai load path:

• session_load_duration_ms: end-to-end GenaiSession.load() wall time
• ep_registration_duration_ms: WinML EP plugin registration time when required
• bundle_prepare_duration_ms: derived bundle preparation / compile-or-rewrite time
• native_load_duration_ms: native og.Config + og.Model + og.Tokenizer creation time
• config_create_duration_ms: og.Config(...) creation time
• model_create_duration_ms: og.Model(...) creation time
• tokenizer_create_duration_ms: og.Tokenizer(...) creation time
• weight_upload_duration_ms: currently null because ORT GenAI does not expose exact native weight upload telemetry
• weight_upload_estimate_duration_ms: currently uses model_create_duration_ms
• weight_upload_estimate_source: documents the estimate source explicitly

This avoids reporting a misleading exact weight-upload number while still exposing the closest observable boundary.

Per-request GenAI timing

Each warmup and timed generation is preserved under requests, with kind and index, so consumers can inspect raw request-level behavior. The aggregate block excludes warmup requests.

Per request, timing is split into:

• prompt template application
• tokenization
• generator creation
• prefill / prompt processing
• first-token decode
• steady-state decode token steps
• generated sequence fetch
• detokenization

Derived request metrics include:

• request_ttft_duration_ms: template + tokenization + generator creation + prefill + first token
• model_ttft_duration_ms: prefill + first token
• response_eval_duration_ms: first token + decode token steps
• model_compute_duration_ms: prefill + response eval
• request_duration_ms: full request wall-clock boundary covered by the measured sub-phases
• prefill_tokens_per_second
• steady_state_decode_tokens_per_second
• response_eval_tokens_per_second
• steady_state_tpot_ms

Aggregates

aggregate reports warmup-excluded statistics for the timed requests, including mean/min/max/percentiles/stddev where applicable, plus:

• warmup_excluded
• warmup_request_count
• timed_request_count
• cold_start_ttft_duration_ms
• cold_start_total_duration_ms

Monitor and memory route correctness

GenAI now resolves the effective bundle route before monitor and memory sampling are initialized. This avoids using requested CLI values like device=config or a no-op EP override to select unrelated hardware.

• hw_monitor uses the proven effective accelerator route when one exists; otherwise it records CPU/RAM only.
• GenAI VRAM fields are emitted only when the effective route proves a specific accelerator adapter.
• GPU console display prefers hw_monitor["adapter"] over aggregate hw_monitor["gpu"], matching classic perf behavior on multi-GPU systems.

Classic / GenAI alignment

Shared report surfaces are aligned where the concepts are common:

• Classic perf now emits top-level schema_version: 2.
• Classic perf now includes benchmark_info.runtime: "winml".
• GenAI benchmark_info now includes model_id and running_model_path.
• GenAI monitor output uses hw_monitor, matching classic perf naming.
• GenAI and classic memory reports use shared RAM/VRAM field names for common checkpoints and deltas.

Checkpoint maxima are named explicitly as checkpoint maxima, not continuous peaks:

• rss_checkpoint_peak_mb
• vram_local_checkpoint_peak_mb
• vram_shared_checkpoint_peak_mb

These are the maximum of baseline / after-compile / after-inference samples. They do not claim to be continuously sampled memory peaks.

Intentional non-goals

• No Ollama/OpenVINO projection block is added to core perf JSON. Those mappings should be done by the report layer.
• No exact weight_upload_duration_ms is invented without native telemetry.
• No continuous memory peak is claimed from checkpoint-only sampling.
• No GenAI accuracy block is emitted from perf.
• GenAI keeps its phase-oriented load / requests / aggregate blocks instead of forcing classic latency_ms / throughput fields into the core schema.

Validation

• uv run --no-sync pytest tests\unit\commands\test_perf_cli.py tests\unit\commands\test_perf_genai.py tests\unit\session\test_genai_session.py -q — 364 passed
• uv run --no-sync ruff check src\winml\modelkit\commands\perf.py src\winml\modelkit\commands\_perf_genai.py src\winml\modelkit\session\genai_session.py tests\unit\commands\test_perf_cli.py tests\unit\commands\test_perf_genai.py tests\unit\session\test_genai_session.py — passed

[xieofxie]

Reviewed the latest revision; prior findings are addressed and no new issues remain.