Large Language Models Systematically Favor Popular Options:
Evidence and Mitigation Across MCQs
Abdelrahman Abdallah · Mohammed Ali · Bhawna Piryani · Mahmoud Abdalla · Adam Jatowt
LLMs recall facts about famous entities better than obscure ones. PopMCQ asks what happens when familiarity and truth disagree: it renders the same question six ways, changing only how popular the distractors are, while the question and the correct answer stay fixed.
They disagree often. Under the hardest setting, models pick a popular-but-wrong option 66% of the time, and correctness correlates with the chosen option's popularity at ρ = −0.89.
PopDebias fixes most of it at inference time — no fine-tuning, no extra forward passes, label-free at test time. Averaged over 22 models and 4 datasets it lifts S2 accuracy from 43.8% → 75.7%.
| 🧩 What it is | 📍 Where | |
|---|---|---|
| PopMCQ | 73,260 MCQ items — 12,210 questions × 6 strategies | 🤗 Hub |
| PopDebias | training-free popularity-prior correction | popmcq/popdebias.py |
| Predictions | every model's output on every item, self-contained | results/predictions/ |
git clone https://github.com/DataScienceUIBK/PopMCQ.git
cd PopMCQ
pip install -e .Core install needs only numpy, scipy, scikit-learn, pandas, pyarrow. Add torch and transformers (pip install -e ".[eval]") only if you want to evaluate new models.
from popmcq import PopDebias, PopDebiasConfig
debias = PopDebias(PopDebiasConfig())
debias.fit(calibration_records) # small labelled split, once
new_probs, info = debias.transform(probs, popularities)
# info -> {'b': 0.71, 'gamma': 0.79, 'psi': 0.93, 'confidence': 0.33}probs is your model's probability vector over the K options; popularities are the matching popularity scores. That's the whole interface — one call, O(K) per item, no second forward pass.
Every prediction we report is in the repo, so you can go straight to the numbers:
from popmcq import load_records, PopDebias, PopDebiasConfig
from popmcq.metrics import aggregate_metrics
recs = load_records("results/predictions/musique_qa.parquet",
"results/predictions/items.parquet",
model="DeepSeek-V2-Lite", strategy="s2")
print(aggregate_metrics(recs)) # acc 31.1 · ρ −0.89 · HPSR 66.0
debias = PopDebias(PopDebiasConfig())
scored = debias.fit_from_pool(recs)
print(aggregate_metrics(debias.transform_records(scored))) # acc 85.2 · ρ −0.52 · HPSR 11.3python -m popmcq.evaluate \
--model meta-llama/Meta-Llama-3-8B-Instruct \
--data data/musique_qa_mcq_hypothesis_FIXED.json \
--strategies s1 s2 s3 s4 s5 s6 \
--out results/python scripts/reproduce_paper.py
# 27/27 checks passedNo arguments, no downloads — it runs off the predictions in this repo. Each headline value is asserted against the published one and the script exits non-zero on any drift. Add --results-dir <dir> to score your own evaluation dumps instead.
Four options each (one correct, three distractors), positions randomized over {A,B,C,D}. pop(a*) is the correct answer's popularity, from normalized Wikipedia page views.
| Strategy | Distractors | Tests | |
|---|---|---|---|
| S1 | 🎲 Baseline | 3 random, unconstrained | reference point |
| S2 | 🪤 Popular Trap | top-3 most popular, all > pop(a*) |
maximum popularity pressure |
| S3 | 📉 Gradient | strictly decreasing, all > pop(a*) |
fine-grained popularity sensitivity |
| S4 | ⚔️ Direct Contest | 1 highest-pop + 2 medium |
one famous competitor |
| S5 | 🔄 Reverse Control | all < pop(a*), correct answer most popular |
the control — does the effect flip? |
| S6 | 🚫 None of the Above | correct = "None of the above" (pop = 0) |
abstention under pressure |
S5 carries the argument. Same questions, same gold answers as S2 — only the distractor popularity changes. Accuracy that drops in S2 and recovers in S5 isolates popularity, not difficulty, as the cause.
Mean over the 22 models and 4 datasets:
| S1 | S2 | S3 | S4 | S5 | S6 | |
|---|---|---|---|---|---|---|
| Accuracy | 42.5 | 43.8 | 42.4 | 40.1 | 43.3 | 14.5 |
| + PopDebias | 49.6 | 75.7 | 64.5 | 55.9 | 45.8 | 25.9 |
| ρ | −0.3 | −0.9 | −0.8 | −0.6 | +0.2 | −0.5 |
| ρ + PopDebias | −0.1 | −0.7 | −0.6 | −0.4 | +0.3 | −0.6 |
| HPSR | 46.4 | 53.6 | 49.4 | 48.6 | 55.9 | 56.3 |
| HPSR + PopDebias | 32.5 | 21.6 | 23.9 | 28.7 | 48.8 | 32.6 |
Largest single gain: +54.1 pp (MuSiQue · S2 · DeepSeek-V2-Lite, 31.1 → 85.2). Under S5, where popularity already points at the truth, gains shrink to +2.5 pp — PopDebias corrects only when popularity and truth conflict.
📁 Per-model tables for all 22 models × 4 datasets × 6 strategies: results/README.md
📁 Machine-readable: results/popmcq_results.csv — 528 rows
📁 Raw predictions: results/predictions/ — 1.6 M rows, every model on every item, plus items.parquet with per-option popularity and gold index. 13 MB total, and enough to recompute every number in the paper offline.
Treat the observed distribution as a popularity prior times a popularity-free belief, then divide the prior out:
Per item, in one pass:
- Estimate bias strength
bfrom five inference-time features —Var(Pop),max Pop,Pop(top-1), popularity rank of top-1, confidence. Bias is high when the model is confident on a very popular option and the distractors cluster in popularity. - Down-weight by popularity:
wᵢ = 1 − b · Pop(oᵢ)/max Pop, then renormalize. - Temper adaptively:
γ = clip(1 − β·b·c·ψ, 0, 1), whereψ = 1 − H(r)/log Kmeasures how concentrated the popularity profile is. Flat popularity or low bias ⇒γ ≈ 1⇒ output ≈ input.
The regressor for b is fit once on a small labelled split (default 10%). At test time no gold labels are used. Cost is O(K) per item — compare permutation averaging, which needs K! forward passes.
python -m popmcq.build_dataset --input candidates.jsonl --output popmcq.jsonGiven a pool of ~20 scored candidate answers per question, strategy assembly is fully deterministic — which candidate lands in which strategy follows from the popularity scores alone. No LLM judgment enters, so the manipulation is independent of both the generating model and the evaluated model. See popmcq/build_dataset.py.
popmcq/
popdebias.py PopDebias — fit / transform / transform_records
metrics.py Accuracy · ρ · HPSR · PopGap · ECE
evaluate.py zero-shot MCQ evaluation
build_dataset.py assemble strategies S1–S6 from a candidate pool
predictions.py load released predictions, join back to the dataset
baselines.py question-blind baselines · PrideDebias
results/
README.md per-model tables, all datasets and strategies
popmcq_results.csv 528 cells, machine-readable
predictions/ raw model outputs + items.parquet (popularity, gold index)
scripts/
reproduce_paper.py asserts every headline number
MODELS.txt the 22 evaluated models
- Acc — proportion correct.
- ρ — Spearman correlation between correctness and selected-option popularity. Negative ⇒ picking popular options goes with being wrong.
- HPSR — High-Popularity Selection Rate: share of picks in the top half of the within-item popularity ranking. >50% ⇒ pull toward popular options.
- PopGap —
E[Pop(selected) − Pop(correct)]. Positive ⇒ selecting options more popular than the truth. - ECE — expected calibration error over the top-option probability.
Under S6, ρ is negative by construction — the correct option has
Pop = 0. Read S6 through accuracy and HPSR.
One forward pass over the prompt, then a softmax restricted to the next-token ids of A/ B/ C/ D — the standard max-probability MCQ protocol used by MMLU and ARC. No chat template is applied, including to instruction-tuned models, so all 22 models are scored identically.
@inproceedings{abdallah2026popmcq,
title = {Large Language Models Systematically Favor Popular Options:
Evidence and Mitigation Across {MCQ}s},
author = {Abdallah, Abdelrahman and Ali, Mohammed and Piryani, Bhawna and
Abdalla, Mahmoud and Jatowt, Adam},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
Natural Language Processing},
year = {2026}
}PopMCQ builds on four QA datasets — please cite them too: MuSiQue (Trivedi et al., 2022), Natural Questions (Kwiatkowski et al., 2019), EntityQuestions (Sciavolino et al., 2021), WebQuestions (Berant et al., 2013).
Code released under the MIT License. The dataset redistributes derivatives of the four source datasets above; see the dataset card for per-source attribution.