Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
142 lines (129 loc) · 21.4 KB
/
Copy pathindex.html
File metadata and controls
142 lines (129 loc) · 21.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Varest: semantics-guided image-token compression with variable-resolution tokenization and learned relevance pruning. Research, visualizations, and results.">
<meta name="theme-color" content="#ffffff">
<meta property="og:title" content="Varest — Just Respect the Semantics">
<meta property="og:description" content="Image-token compression via pre-LLM semantic variable-resolution tokenization and in-LLM relevance pruning.">
<meta property="og:type" content="website">
<meta property="og:image" content="https://varest.github.io/assets/video-poster-start.jpg">
<link rel="icon" href="assets/favicon.svg" type="image/svg+xml">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Noto+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="styles.css">
<script src="script.js" defer></script>
<script type="module" src="pdf-figures.js"></script>
<title>Varest: Just Respect the Semantics</title>
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<header class="institution-logos" aria-label="University affiliations">
<span class="institution-logo logo-toronto"><img src="assets/uoft.png" alt="University of Toronto" width="650" height="365"></span>
<span class="institution-logo logo-mcgill"><img src="assets/mcgill.png" alt="McGill University" width="2400" height="2400"></span>
</header>
<main id="main" class="page-shell">
<section class="hero" id="top" aria-labelledby="hero-title">
<div class="hero-copy">
<img class="project-mark" src="assets/logos/varest-mark.svg" width="96" height="88" alt="Varest logo">
<h1 id="hero-title">Varest: Just Respect the Semantics</h1>
<p class="paper-subtitle">Image-Token Compression via Pre-LLM Semantic Variable-Resolution Tokenization and In-LLM Relevance Pruning</p>
<p class="authors"><span><a href="https://orcid.org/0009-0007-2342-5350" target="_blank" rel="noopener" aria-label="Tianhao Fu (ORCID)" title="ORCID profile">Tianhao Fu</a><sup>†, 1, 2, </sup>*</span> <span><a href="https://orcid.org/0009-0008-3817-0589" target="_blank" rel="noopener" aria-label="Austin Wang (ORCID)" title="ORCID profile">Austin Wang</a><sup>†, 3</sup></span> <span><a href="https://orcid.org/0009-0005-7789-3876" target="_blank" rel="noopener" aria-label="Charles Chen (ORCID)" title="ORCID profile">Charles Chen</a><sup>†, 1, 4</sup></span> <span><a href="https://orcid.org/0009-0008-0149-2250" target="_blank" rel="noopener" aria-label="Roby Aldave-Garza (ORCID)" title="ORCID profile">Roby Aldave-Garza</a><sup>5</sup></span> <span><a href="https://orcid.org/0009-0007-8118-4049" target="_blank" rel="noopener" aria-label="Victor Xie (ORCID)" title="ORCID profile">Victor Xie</a><sup>3</sup></span> <span><a href="https://orcid.org/0009-0005-4432-8600" target="_blank" rel="noopener" aria-label="Xiyue Jiao (ORCID)" title="ORCID profile">Xiyue Jiao</a><sup>1</sup></span> <span><a href="https://orcid.org/0009-0000-9492-8958" target="_blank" rel="noopener" aria-label="Yucheng Chen (ORCID)" title="ORCID profile">Yucheng Chen</a><sup>1</sup></span> <span>Martin Tin<sup>1</sup></span></p>
<p class="contribution"><sup>†</sup> Equal contribution. <sup>†</sup>Project lead with correspondence.</p>
<p class="affiliations"><span><sup>1</sup> University of Toronto</span><span><sup>2</sup> Vector Institute</span><span><sup>3</sup> McGill University</span><span><sup>4</sup> University Health Network</span><span><sup>5</sup> University of Waterloo</span></p>
<!-- <p class="publication-status">CVPR 2027</p>-->
<div class="hero-actions">
<a class="button" href="https://arxiv.org/pdf/2610.18690"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/><path d="M14 2v6h6M8 13h8M8 17h6"/></svg>Paper</a>
<a class="button" href="https://arxiv.org/abs/2610.18690"><img class="resource-logo arxiv-logo" src="assets/logos/arxiv.svg" width="16" height="22" alt="">arXiv</a>
<a class="button" href="https://github.com/Varest/varest"><img class="resource-logo" src="assets/logos/github-white.svg" width="18" height="18" alt="">Code</a>
</div>
</div>
</section>
<div class="teaser">
<figure class="video-figure">
<video controls muted playsinline preload="none" data-scroll-play poster="assets/video-poster-start.jpg" aria-label="Varest method and visualizations overview"><source src="assets/varest-overview.mp4" type="video/mp4">Your browser does not support embedded video. <a href="assets/varest-overview.mp4">Download the overview video</a>.</video>
<figcaption><span class="figure-label">Video overview</span> From object boundaries and text localization to variable-resolution tokens and learned relevance.</figcaption>
</figure>
<p class="tldr"><strong>TL;DR</strong> Our method preserves nearly lossless performance at less than 25% of the tokens and >85% retention at only 5.56%, by maximizing the <em>semantic fungibility</em>.</p>
</div>
<section class="section" id="overview" aria-labelledby="overview-title">
<h2 id="overview-title">Overview</h2>
<p class="section-intro">An image is not a uniform grid of information. A coherent object can be summarized; a word or an object boundary needs finer detail. Varest uses this principle of <em>semantic fungibility</em> to decide where to merge, then learns which visual tokens the language model still needs.</p>
</section>
<section class="section" id="method" aria-labelledby="method-title">
<h2 id="method-title">Method</h2>
<p class="section-intro">Varest combines semantic merging before the LLM with relevance pruning inside it. Early decoder layers see the whole scene; later layers work with a smaller set of visual tokens.</p>
<figure class="paper-figure">
<div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 569"><div class="pdf-document" data-pdf="assets/method_simple.pdf" role="img" aria-label="Varest pipeline: input image, object boundary and textness fields, semantic partitioning, early LLM layers, relevance pruning, and optional adaptive halting."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/method_simple.pdf" target="_blank" rel="noopener" aria-label="Open PDF: Varest method overview">Open PDF <span aria-hidden="true">↗</span></a></div>
<figcaption><span class="figure-label">The Varest pipeline</span> Spatial coverage is preserved through semantic merging, before learned [REL] registers select tokens at the decoder midpoint.</figcaption>
</figure>
<div class="method-steps">
<article><span class="step-number">01 / Localize</span><h3>Find meaningful boundaries</h3><p>Lightweight object and text localizers identify semantic transitions and text regions, guiding where detail should be preserved.</p></article>
<article><span class="step-number">02 / Merge</span><h3>Use resolution where it counts</h3><p>Merge compatible regions into variable-resolution tokens. Preserve text at native resolution when the budget allows, while covering the full image.</p></article>
<article><span class="step-number">03 / Prune</span><h3>Let the model read first</h3><p>Four learned [REL] registers score visual tokens after early decoder processing, keeping a smaller subset for the remaining layers.</p></article>
</div>
<details class="figure-details"><summary>Explore the full architecture <span aria-hidden="true">+</span></summary><figure><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 1145"><div class="pdf-document" data-pdf="assets/method_cropped.pdf" role="img" aria-label="Detailed Varest training and inference architecture, including localizers, spatial position tags, relevance registers, and adaptive halting."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/method_cropped.pdf" target="_blank" rel="noopener" aria-label="Open PDF: full Varest architecture">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption>Training across token budgets allows one checkpoint per backbone. Optional adaptive halting removes the remaining image tokens when a prediction-stability and confidence criterion is met.</figcaption></figure></details>
</section>
<section class="section" id="visualizations" aria-labelledby="visualizations-title">
<h2 id="visualizations-title">Visualizations</h2>
<p class="section-intro">Explore how semantic structure, text, and scene context are represented as the token budget changes. Select a view below; use “Open PDF” to inspect a figure at full size.</p>
<div class="figure-browser" data-gallery>
<div class="tab-list" role="tablist" aria-label="Token visualization" hidden>
<button role="tab" id="tab-budgets" aria-selected="true" aria-controls="panel-budgets" tabindex="0">Across budgets</button>
<button role="tab" id="tab-baselines" aria-selected="false" aria-controls="panel-baselines" tabindex="-1">Baseline comparison</button>
<button role="tab" id="tab-prompts" aria-selected="false" aria-controls="panel-prompts" tabindex="-1">Different prompts</button>
</div>
<figure id="panel-budgets" role="tabpanel" aria-labelledby="tab-budgets" tabindex="0"><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 1088"><div class="pdf-document" data-pdf="assets/qualitative_analysis.pdf" role="img" aria-label="Tokenization and tokens retained by REL at budgets of 32, 64, 128, and 224, for chart, jersey, and building examples."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/qualitative_analysis.pdf" target="_blank" rel="noopener" aria-label="Open PDF: tokenization across budgets">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption><strong>Same scene, different granularity.</strong> Object regions become finer as the budget increases. Text tokens, highlighted in orange, remain identifiable across budgets. Examples use LLaVA-1.5-7B.</figcaption></figure>
<figure id="panel-baselines" role="tabpanel" aria-labelledby="tab-baselines" tabindex="0"><div class="pdf-figure figure-portrait"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 1965"><div class="pdf-document" data-pdf="assets/baseline_visualization.pdf" role="img" aria-label="Comparison of HiDrop, LearnPruner, VisionZip++, CORE, and Varest on reading the shirt of the man in white. Varest preserves the context and answers Bwin."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/baseline_visualization.pdf" target="_blank" rel="noopener" aria-label="Open PDF: baseline comparison">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption><strong>Keep the text and its context.</strong> This example contrasts Varest’s semantic regions with pruning and merging baselines on LLaVA-1.5-7B.</figcaption></figure>
<figure id="panel-prompts" role="tabpanel" aria-labelledby="tab-prompts" tabindex="0"><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 720"><div class="pdf-document" data-pdf="assets/prompt_sensitivity.pdf" role="img" aria-label="For two questions about a soccer ball and trees, Varest retains nearly identical tokens and correctly answers Left and Green."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/prompt_sensitivity.pdf" target="_blank" rel="noopener" aria-label="Open PDF: prompt comparison">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption><strong>Relevance after reading.</strong> Nearly the same tokens are retained for two unrelated questions. The example suggests that early layers can absorb information before the corresponding image tokens are removed.</figcaption></figure>
</div>
<details class="figure-details"><summary>Inside the tokenizer: object boundaries and text <span aria-hidden="true">+</span></summary>
<div class="localizer-grid"><figure><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 931"><div class="pdf-document" data-pdf="assets/object_localizer.pdf" role="img" aria-label="Input images and predicted object boundary fields across diagrams, products, charts, and street scenes."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/object_localizer.pdf" target="_blank" rel="noopener" aria-label="Open PDF: object localizer examples">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption><strong>Object localization.</strong> Boundary strength guides region merging.</figcaption></figure><figure><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 1139"><div class="pdf-document" data-pdf="assets/text_localizer.pdf" role="img" aria-label="Text localization predictions for documents, charts, diagrams, and natural scenes."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/text_localizer.pdf" target="_blank" rel="noopener" aria-label="Open PDF: text localizer examples">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption><strong>Text localization.</strong> Textness guides the allocation of fine detail.</figcaption></figure></div>
</details>
</section>
<section class="section" id="results" aria-labelledby="results-title">
<h2 id="results-title">Results</h2>
<div class="findings" aria-label="Research highlights">
<div><strong>5.56<span>%</span></strong><p>of the layer-averaged image tokens</p></div>
<div><strong>86.10–92.33<span>%</span></strong><p>of full-token performance retained</p></div>
<div><strong>3 <span>backbones ·</span> 13 <span>benchmarks</span></strong><p>highest mean retention against 15 baselines at every evaluated budget</p></div>
</div>
<p class="section-intro">Across three LLaVA backbones and 13 benchmarks, Varest achieves the highest mean performance retention among 15 baselines at every evaluated budget. At 5.56% of the layer-averaged image tokens, it retains 86.10–92.33% of full-token performance.</p>
<figure class="paper-figure"><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 841"><div class="pdf-document" data-pdf="assets/retention.pdf" role="img" aria-label="Performance retention versus token budget for three backbones. Varest leads all compared methods at every evaluated budget."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/retention.pdf" target="_blank" rel="noopener" aria-label="Open PDF: performance retention curves">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption><span class="figure-label">Performance vs. token budget</span> Retention is normalized to each uncompressed backbone; 100% means matching full-token performance.</figcaption></figure>
<figure class="radar-figure">
<div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2299 / 2400"><div class="pdf-document" data-pdf="assets/radar_magnified.pdf" role="img" aria-label="Radar chart comparing Varest across three backbones and four token budgets with the full-token reference."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/radar_magnified.pdf" target="_blank" rel="noopener" aria-label="Open PDF: per-benchmark performance figure">Open PDF <span aria-hidden="true">↗</span></a></div>
<figcaption>Performance across visual tasks, backbones, and token budgets.</figcaption>
</figure>
<div class="table-heading"><h3>Performance at a glance</h3><span class="mono">Mean performance retention ↑</span></div>
<div class="table-scroll" tabindex="0" role="region" aria-label="Performance retention table, scroll horizontally on small screens"><table><caption>Varest performance retention at four layer-averaged image-token ratios.</caption><thead><tr><th scope="col">Backbone</th><th scope="col">5.56% tokens</th><th scope="col">11.11% tokens</th><th scope="col">22.22% tokens</th><th scope="col">38.89% tokens</th></tr></thead><tbody><tr><th scope="row">LLaVA-1.5-7B</th><td>92.33%</td><td>97.08%</td><td>100.37%</td><td>101.89%</td></tr><tr><th scope="row">LLaVA-1.5-13B</th><td>91.31%</td><td>95.82%</td><td>100.03%</td><td>102.37%</td></tr><tr><th scope="row">LLaVA-NeXT-8B</th><td>86.10%</td><td>91.85%</td><td>96.04%</td><td>98.18%</td></tr></tbody></table></div>
<p class="table-note">Source: Table 1 of the <a href="https://arxiv.org/html/2610.18690v1#S4.T1">paper</a>. Mean retention averages 15 normalized metrics across 13 benchmarks; POPE contributes three metrics. Adaptive halting is disabled. For LLaVA-NeXT, ratios are applied to each image’s native sequence length.</p>
<div class="result-callout"><span class="callout-mark" aria-hidden="true">↗</span><p><strong>More performance at half the token budget.</strong> On all three backbones, Varest at 11.11% tokens outperforms every evaluated baseline at 22.22%. On LLaVA-1.5-7B, that is 97.08% retention versus the strongest baseline’s 94.50%.</p></div>
<details class="figure-details"><summary>Measured computation and optional adaptive halting <span aria-hidden="true">+</span></summary><figure><div class="pdf-figure"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 854"><div class="pdf-document" data-pdf="assets/retention_vs_complexity.pdf" role="img" aria-label="Mean performance retention plotted against measured TFLOPs. Varest and Varest with adaptive halting are compared with 15 baselines."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/retention_vs_complexity.pdf" target="_blank" rel="noopener" aria-label="Open PDF: performance versus computation">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption>Each point averages performance and computation independently over four budgets. Optional adaptive halting reduces FLOPs by a further 3.0–5.1%, with retention losses of 0.30–0.48 percentage points. These are computation savings, not measured latency improvements.</figcaption></figure></details>
</section>
<section class="section" id="discussion" aria-labelledby="discussion-title">
<h2 id="discussion-title">Discussion</h2>
<div class="discussion-grid"><div><h3>Semantics before selection</h3><p>Varest preserves full spatial coverage during merging so early decoder layers can integrate scene context. Its learned selection does not require decoder attention weights and remains compatible with FlashAttention.</p><p>General visual question answering is relatively robust to compression. Document reading, chart understanding, and referring-expression grounding benefit more from larger budgets.</p></div><div><h3>Scope and open questions</h3><p>The current evaluation covers three LLaVA backbones. Extending to other model families, including Qwen-VL and InternVL, remains future work.</p><p>Lower FLOPs do not automatically imply lower latency; efficient implementation is still needed. Applying semantic fungibility to other domains also requires domain-specific validation.</p></div></div>
<details class="figure-details"><summary>Where Varest fits among compression methods <span aria-hidden="true">+</span></summary><figure><div class="pdf-figure figure-portrait"><div class="pdf-viewport" style="--pdf-ratio: 2400 / 1600"><div class="pdf-document" data-pdf="assets/related_work.pdf" role="img" aria-label="Taxonomy of image-token compression methods by semantic, query, and saliency signals, and by compression stage."><p class="pdf-status">Use “Open PDF” to view this figure.</p></div></div><a class="pdf-link" href="assets/related_work.pdf" target="_blank" rel="noopener" aria-label="Open PDF: related work diagram">Open PDF <span aria-hidden="true">↗</span></a></div><figcaption>Varest combines explicit semantic guidance before the LLM with learned relevance selection inside it.</figcaption></figure></details>
</section>
<section class="section citation-section" id="citation" aria-labelledby="citation-title">
<div class="citation-heading"><h2 id="citation-title">BibTeX</h2><button class="button copy-button" id="copy-citation" hidden>Copy citation <span aria-hidden="true">⧉</span></button></div>
<pre><code id="bibtex">@misc{fu2026varest,
title={Just Respect the Semantics: Image-Token Compression via
Pre-LLM Semantic Variable-Resolution Tokenization and
In-LLM Relevance Pruning},
author={Tianhao Fu and Austin Wang and Charles Chen and
Roby Aldave-Garza and Victor Xie and Xiyue Jiao and
Yucheng Chen and Martin Tin},
year={2026},
eprint={2610.18690},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.18690}
}</code></pre><p id="copy-status" class="copy-status" role="status" aria-live="polite"></p>
</section>
</main>
<footer class="site-footer"><div class="page-shell"><a class="wordmark" href="#top">Varest <span aria-hidden="true">↑</span></a><p>This page was built with the inspiration from <a href="https://felixtaubner.github.io/mvp4d/">MVP4D</a>.</p></div></footer>
</body>
</html>