Add dalpha-llm-router submission

[junilhong-dalpha]

Submit dalpha-llm-router

Affiliation: Dalpha · Website: https://dalpha.so · Type: closed-source (external_prediction_file)

Router

Dalpha LLM Router classifies each query with a zero-shot classifier (no training, no fine-tuning) and routes it with a fixed table over a 4-model pool:

Model	Setting
gemini-3.8-flash	thinking_level=low
gemini-3-flash-preview	thinking off
gpt-6-luna	reasoning_effort=high
anthropic/claude-haiku-5.5	default (called through OpenRouter)

The routing table was calibrated on an external set of 1,830 items drawn from public benchmarks (MMLU-Pro, SuperGPQA, MedMCQA, ARC-Challenge, PubMedQA, TriviaQA, NarrativeQA, GSM8K, MATH, HumanEval, SuperGLUE RTE/BoolQ), deduplicated against RouterArena prompts. A cheaper model is used for a query class only when it is statistically non-inferior to the most accurate model on that class in the external set.

Evaluation-only compliance

• No RouterArena data (queries, labels, or splits) was used to build, fit, or tune any component.
• Prompts were sent to the models unmodified; the prediction files pass check_config_prediction_files.py for both full and robustness.
• Costs use provider list prices, added to model_cost/model_cost.json for the three models not yet listed. Model names are registered in universal_model_names.py.
• The calibration data and fitting procedure are kept private but can be shared with the maintainers on request.
• 4 of the 8,400 queries (NarrativeQA) were refused by the provider's content filter and returned no answer; they are left as-is and scored as 0.

Local results (full, 8,400 queries)

Metric	Value
Accuracy	81.1%
Cost per 1K queries	$0.89
RouterArena score (local)	76.87
Robustness (decision agreement on perturbed split)	0.979

Files

• router_inference/config/dalpha-llm-router.json
• router_inference/predictions/dalpha-llm-router.json
• router_inference/predictions/dalpha-llm-router-robustness.json
• model_cost/model_cost.json, universal_model_names.py (three new entries)

Leaderboard row: router name "Dalpha LLM Router", link https://dalpha.so, affiliation Dalpha, closed-source.

[junilhong-dalpha]

/evaluate

[github-actions]

Router Evaluation Results

Router: dalpha-llm-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7686
Accuracy	81.09%
Total Cost	$7.446648
Avg Cost per Query	$0.000887
Avg Cost per 1K Queries	$0.8865
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.9786

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.0849
Opt.Cost (Cost Efficiency)	0.2340
Opt.Acc (Accuracy vs Optimal)	0.9429

---

Evaluation completed by RouterArena automated workflow

[junilhong-dalpha]

Thanks for the automated run. Some details on how the router was built, plus two disclosures.

How the router is built (evaluation-only rule). Routing is decided from the prompt text alone: a zero-shot classifier reads the prompt and returns a task type; the router does not match instruction headers or prompt templates and does not read RouterArena's config files or label files anywhere. Every parameter in the submitted router was derived from an external calibration set that does not overlap with RouterArena: the base model (highest accuracy on the external set) and the task-to-model table (a cheaper model replaces the base only where it is statistically non-inferior on the external set). The external set is 1,830 items from MMLU-Pro, SuperGPQA, MedMCQA, ARC-Challenge, PubMedQA (pqa_labeled), TriviaQA, NarrativeQA (validation), GSM8K, MATH, HumanEval and SuperGLUE RTE/BoolQ, deduplicated against the 8,400 RouterArena prompts. During development we ran the README's local evaluation on the full split end-to-end for a handful of frozen configurations, each specified entirely from the external set, and submitted one of them. We can provide the external-set manifest (SHA-256 per item), the classifier labels and the per-item results for all candidate models privately on request.

Classifier cost (disclosure). Each query makes one call to the classifier (TypeSafe Jev; three questions in a single call, output tokens are free). Measured over the 8,400 queries: 6,864,653 input tokens at $0.042 per million, i.e. $0.0343 per 1K queries on top of the reported $0.8865. Because the evaluation script recomputes each row's cost from its token_usage at the selected model's price, this overhead is not in the prediction file; including it moves the Arena score from 76.86 to 76.79. We are happy to account for it in whatever form you prefer (for example a separate price entry for the classifier, or a note on the leaderboard row).

The 4 abnormal entries. All four are NarrativeQA passages that Gemini 3 Flash Preview refuses at the prompt level (PROHIBITED_CONTENT), which safety settings cannot override; we retried each three times with the same result. They are left in the file as refusals and scored as 0, since the refusal is a property of the model the router chose. If you would rather see a provider-refusal fallback to the base model, we can add it and regenerate, but we did not want to change routing behaviour after seeing the benchmark result.