Seeing Beyond Redundancy: Task Complexity’s Role in Vision Token Specialization in VLLMsAuthors:Darryl Hannan, John Cooper, Dylan White, Yijing WatkinsDeep-Dive Summary:论文总结超越冗余——任务复杂度在 VLLM 视觉 Token 特化中的作用摘要视觉大语言模型VLLMs在视觉能力上始终滞后于其语言能力。研究表明VLLM 在处理细粒度视觉信息或空间推理任务时表现不佳。本文旨在探究视觉信息在模型中的处理方式及被丢弃的规律。研究引入了一个合成基准数据集和一套衡量视觉冗余的指标分析了不同复杂度的任务对视觉压缩的影响。研究发现任务复杂度与视觉压缩之间存在紧密联系高复杂度的视觉数据对于改变 VLLM 的视觉表示分布并提升其在复杂任务上的性能至关重要。2. 背景与相关工作视觉大语言模型 (VLLMs)与对比学习模型如 CLIP不同VLLM 通过自回归方式预测下一个 Token。模型通常将图像嵌入E ( i ) ∈ R N i × d \mathbf{E}^{(i)} \in \mathbb{R}^{N_{i} \times d}E(i)∈RNi×d与文本嵌入E ( t ) ∈ R N t × d \mathbf{E}^{(t)} \in \mathbb{R}^{N_{t} \times d}E(t)∈RNt×d连接后通过解码器处理。视觉 Token 压缩与冗余现有研究表明 VLLM 并不需要所有的视觉 Token。例如在 LlaVA 中随机删除 95% 的 Token 对性能影响极小。本文不仅利用这种冗余还通过研究 Molmo 和 Llama 3.2 等最新模型探索视觉信息如何在模型中传播。4. 实验4.1 零样本分析 (Zero-shot Analyses)研究选取了 Molmo 和 Llama 3.2。两者架构略有不同Molmo 采用全模态联合注意力而 Llama 3.2 采用交叉注意力。数据集合成数据集在白色背景上放置不同形状、颜色、大小的 2D 物体数量从 1 到 200 不等以便精确控制复杂度。MSCOCO 子集使用真实世界图像验证趋势。Token 压缩分析在 Molmo 中早期层将能量分散在大量视觉 Token 中中间层重新浓缩信息末尾层再次分散以丢弃冗余信息见图 1。物体数量与低压缩率强相关见图 2表明拥挤的场景需要更多 Token 来表示。在 COCO 数据集上见图 3物体类别的多样性和大小差异是降低视觉压缩的主要特征。图 3Molmo 在 COCO 数据集上压缩指标与视觉属性之间的 Spearman 相关性。探测实验 (Probes)信息转移实验发现文本 Token 能够高度预测视觉属性见图 4表明视觉信息在第一层之后就迅速整合到了文本空间中。冗余性几乎所有 Token 位置都能准确预测物体数量这表明模型中存在极高的视觉冗余。特征分化某些特征如“主色调”或 COCO 中的“主超级类别”在不同层表现出不同的预测能力说明模型在不同阶段侧重于不同的视觉特征。图 4Molmo 在合成数据集上对各种视觉属性的线性探测性能。图 5Molmo 在 COCO 数据集上对各种视觉属性的线性探测性能。Original Abstract:Vision capabilities in vision large language models (VLLMs) have consistently lagged behind their linguistic capabilities. In particular, numerous benchmark studies have demonstrated that VLLMs struggle when fine-grained visual information or spatial reasoning is required. However, we do not yet understand exactly why VLLMs struggle so much with these tasks relative to others. Some works have focused on visual redundancy as an explanation, where high-level visual information is uniformly spread across numerous tokens and specific, fine-grained visual information is discarded. In this work, we investigate this premise in greater detail, seeking to better understand exactly how various types of visual information are processed by the model and what types of visual information are discarded. To do so, we introduce a simple synthetic benchmark dataset that is specifically constructed to probe various visual features, along with a set of metrics for measuring visual redundancy, allowing us to better understand the nuances of their relationship. Then, we explore fine-tuning VLLMs on a number of complex visual tasks to better understand how redundancy and compression change based upon the complexity of the data that a model is trained on. We find that there is a connection between task complexity and visual compression, implying that having a sufficient ratio of high complexity visual data is crucial for altering the way that VLLMs distribute their visual representation and consequently improving their performance on complex visual tasks. We hope that this work will provide valuable insights for training the next generation of VLLMs.PDF Link:2602.06914v1部分平台可能图片显示异常请以我的博客内容为准