2025 04 11

EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models

Abhay Gupta,Jacob Cheung,Philip Meng,Shayan Sayyed,Austen Liao,Kevin Zhu,Sean O'Brien

Task: 评估五种大型语言模型在非标准英语方言上的表现。

Motivation: 现有NLP基准测试忽视语言内部多样性，导致非标准方言使用者未被充分服务。

Details

Method: 通过少量样本提示将标准英语数据集翻译为五种非主流方言，并与基于规则的方法进行比较。 Result: 翻译质量高（平均得分6.02/7），模型在方言输入上表现显著低于标准英语。 Conclusion: EnDive揭示了模型偏见，推动了更公平的语言技术发展。 Abstract: The diversity of human language, shaped by social, cultural, and regional influences, presents significant challenges for natural language processing (NLP) systems. Existing benchmarks often overlook intra-language variations, leaving speakers of non-standard dialects underserved. To address this gap, we introduce EnDive (English Diversity), a benchmark that evaluates five widely-used large language models (LLMs) across tasks in language understanding, algorithmic reasoning, mathematics, and logic. Our framework translates Standard American English datasets into five underrepresented dialects using few-shot prompting with verified examples from native speakers, and compare these translations against rule-based methods via fluency assessments, preference tests, and semantic similarity metrics. Human evaluations confirm high translation quality, with average scores of at least 6.02/7 for faithfulness, fluency, and formality. By filtering out near-identical translations, we create a challenging dataset that reveals significant performance disparities - models consistently underperform on dialectal inputs compared to Standard American English. EnDive thus advances dialect-aware NLP by uncovering model biases and promoting more equitable language technologies.

How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities

Aly M. Kassem,Bernhard Schölkopf,Zhijing Jin

Task: 提出一个名为DSC的评估框架，用于全面评估大型语言模型（LLM）路由器的性能，涵盖多种查询类型及隐私与安全问题。

Motivation: 当前评估基准过于关注通用模型能力，忽视了任务特定行为及隐私、安全和潜在后门漏洞等关键问题。

Details

Method: 设计DSC基准，分类评估路由器性能，包括编码、翻译、数学、人类指令、通用知识和LLM越狱等查询类型，并整合隐私与安全评估。 Result: 实验表明，偏好数据驱动的路由器虽提高效率，但常做出次优的类别驱动决策，例如将编码和数学查询过度分配给最强模型，而将越狱尝试路由至较弱模型。 Conclusion: DSC基准揭示了当前路由器的局限性，强调了在评估中纳入多样性和安全性的重要性。 Abstract: Large language model (LLM) routing has emerged as a crucial strategy for balancing computational costs with performance by dynamically assigning queries to the most appropriate model based on query complexity. Despite recent advances showing that preference-data-based routers can outperform traditional methods, current evaluation benchmarks remain limited. They largely focus on general model capabilities while overlooking task-specific behaviors and critical concerns such as privacy, safety, and potential backdoor vulnerabilities introduced through preference data. In response, we propose the DSC benchmark: Diverse, Simple, and Categorized, an evaluation framework that categorizes router performance across a broad spectrum of query types, including coding, translation, mathematics, human instructions, general knowledge, and LLM jailbreaking. Additionally, it integrates privacy and safety assessments to reveal hidden risks. Our experiments on three preference-based routers and two commercial counterparts demonstrate that while these systems improve efficiency, they often make suboptimal, category-driven decisions. For instance, a BERT-based router directs all coding and mathematics queries to the most powerful LLM even when simpler models would suffice, while routing jailbreaking attempts to weaker models, thereby elevating safety risks.

ChatBench: From Static Benchmarks to Human-AI Evaluation

Serina Chang,Ashton Anderson,Jake M. Hofman

Task: 评估人类与大型语言模型（LLM）在协作中的表现，并设计新的数据集和方法来量化这种协作效果。

Motivation: 现有标准基准（如MMLU）仅评估LLM的独立能力，无法反映人类与LLM协作的实际效果，因此需要新的评估方法。

Details

Method: 通过用户研究将MMLU问题转化为用户-LLM对话，构建ChatBench数据集，包含AI独立、用户独立及用户-AI协作数据，并分析对话差异。 Result: 发现AI独立准确性无法预测用户-AI协作准确性，且在不同学科（数学、物理、道德推理）中存在显著差异；通过微调用户模拟器，显著提高了对用户-AI准确性的估计能力。 Conclusion: ChatBench为交互式评估提供了新工具，揭示了人类与LLM协作的复杂性，并为未来研究提供了方向。 Abstract: With the rapid adoption of LLM-based chatbots, there is a pressing need to evaluate what humans and LLMs can achieve together. However, standard benchmarks, such as MMLU, measure LLM capabilities in isolation (i.e., "AI-alone"). Here, we design and conduct a user study to convert MMLU questions into user-AI conversations, by seeding the user with the question and having them carry out a conversation with the LLM to answer their question. We release ChatBench, a new dataset with AI-alone, user-alone, and user-AI data for 396 questions and two LLMs, including 144K answers and 7,336 user-AI conversations. We find that AI-alone accuracy fails to predict user-AI accuracy, with significant differences across multiple subjects (math, physics, and moral reasoning), and we analyze the user-AI conversations to provide insight into how they diverge from AI-alone benchmarks. Finally, we show that fine-tuning a user simulator on a subset of ChatBench improves its ability to estimate user-AI accuracies, increasing correlation on held-out questions by more than 20 points, creating possibilities for scaling interactive evaluation.

EqualizeIR: Mitigating Linguistic Biases in Retrieval Models

Jiali Cheng,Hadi Amiri

Task: 提出EqualizeIR框架以减轻信息检索模型中的语言偏见。

Motivation: 现有信息检索模型在语言复杂度不同的查询上表现不均，存在显著偏见。

Details

Method: 使用语言偏见弱学习器捕捉数据集中的偏见，并通过正则化和精炼预测训练鲁棒模型。 Result: 实验表明，该方法减少了语言简单和复杂查询间的性能差异，并提升了整体检索性能。 Conclusion: EqualizeIR框架有效减轻了语言偏见，提升了信息检索模型的鲁棒性和公平性。 Abstract: This study finds that existing information retrieval (IR) models show significant biases based on the linguistic complexity of input queries, performing well on linguistically simpler (or more complex) queries while underperforming on linguistically more complex (or simpler) queries. To address this issue, we propose EqualizeIR, a framework to mitigate linguistic biases in IR models. EqualizeIR uses a linguistically biased weak learner to capture linguistic biases in IR datasets and then trains a robust model by regularizing and refining its predictions using the biased weak learner. This approach effectively prevents the robust model from overfitting to specific linguistic patterns in data. We propose four approaches for developing linguistically-biased models. Extensive experiments on several datasets show that our method reduces performance disparities across linguistically simple and complex queries, while improving overall retrieval performance.

Perception in Reflection

Yana Wei,Liang Zhao,Kangheng Lin,En Yu,Yuang Peng,Runpei Dong,Jianjian Sun,Haoran Wei,Zheng Ge,Xiangyu Zhang,Vishal M. Patel

Task: 提出一种反射感知范式（RePer），通过双模型反射机制迭代优化视觉感知，超越现有大型视觉语言模型的局限性。

Motivation: 现有大型视觉语言模型在初始感知上往往不完美，需要一种机制来系统性地改进视觉感知能力。

Details

Method: 采用双模型反射机制（策略模型与批评模型交替），并结合反射感知学习（RPL）方法，通过视觉反射数据集和反射非似然训练增强反射能力。 Result: RePer在图像理解、描述精度和幻觉减少方面取得显著改进，模型注意力模式与人类视觉焦点高度一致，RPL优化了细粒度和自由形式的偏好对齐。 Conclusion: 反射感知范式为未来多模态智能体提供了强大框架，特别适用于需要复杂推理和多步操作的任务。 Abstract: We present a perception in reflection paradigm designed to transcend the limitations of current large vision-language models (LVLMs), which are expected yet often fail to achieve perfect perception initially. Specifically, we propose Reflective Perception (RePer), a dual-model reflection mechanism that systematically alternates between policy and critic models, enables iterative refinement of visual perception. This framework is powered by Reflective Perceptual Learning (RPL), which reinforces intrinsic reflective capabilities through a methodically constructed visual reflection dataset and reflective unlikelihood training. Comprehensive experimental evaluation demonstrates RePer's quantifiable improvements in image understanding, captioning precision, and hallucination reduction. Notably, RePer achieves strong alignment between model attention patterns and human visual focus, while RPL optimizes fine-grained and free-form preference alignment. These advancements establish perception in reflection as a robust paradigm for future multimodal agents, particularly in tasks requiring complex reasoning and multi-step manipulation.

CLEAR: Contrasting Textual Feedback with Experts and Amateurs for Reasoning

Andrew Rufail,Daniel Kim,Sean O'Brien,Kevin Zhu

Task: 提出一种名为CLEAR的新方法，通过对比专家和业余模型的反馈来改进语言模型的推理能力。

Motivation: 利用专家模型和业余模型的反馈优势，通过对比和迭代优化提升语言模型的推理表现。

Details

Method: CLEAR方法结合专家和业余模型对初始输出的反馈，通过对比生成优化后的反馈，并迭代改进模型响应。 Result: CLEAR在多项推理任务中表现优于现有方法，包括故事大纲改进（有趣性提升19.6%）、受限生成（覆盖率提升18.5%）、数学推理（准确率提升6.7%）和毒性缓解（毒性降低22%）。 Conclusion: CLEAR通过对比专家和业余模型的反馈，显著提升了语言模型在复杂推理任务中的表现。 Abstract: We introduce CLEAR (Contrasting Textual Feedback with Experts and Amateurs for Reasoning), a novel approach to language model reasoning that leverages the strengths of a larger (expert) model and smaller (amateur) model. The expert and amateur models each provide feedback on a model's initial output and are contrasted with each other into refined feedback. This feedback is subsequently applied to iteratively improve CLEAR's responses. Our experiments demonstrate that CLEAR outperforms state-of-the-art methods in several challenging reasoning tasks, including story outline improvement (up to 19.6% relative increase in interestingness), constrained generation (up to 18.5% increase in coverage), mathematical reasoning (up to 6.7% improvement in accuracy) and mitigation of toxicity (decrease of up to 22% in toxicity).

Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning

Ashutosh Chaubey,Xulang Guan,Mohammad Soleymani

Task: 提出Face-LLaVA，一种多模态大语言模型，用于面部为中心的学习任务，包括表情和属性识别，并生成自然语言描述。

Motivation: 人脸在社交交流中至关重要，需要高性能的计算机视觉工具支持以人为中心的应用。

Details

Method: 开发FaceInstruct-1M数据库用于指令调优，并提出基于Face-Region Guided Cross-Attention的面部特定视觉编码器。 Result: 在九个数据集和五项任务中表现优于开源模型，与商业解决方案竞争，且生成描述在零样本设置下获得更高的推理评分。 Conclusion: Face-LLaVA在社交AI和基础视觉语言研究中具有潜力，数据集和模型将开源以支持未来研究。 Abstract: The human face plays a central role in social communication, necessitating the use of performant computer vision tools for human-centered applications. We propose Face-LLaVA, a multimodal large language model for face-centered, in-context learning, including facial expression and attribute recognition. Additionally, Face-LLaVA is able to generate natural language descriptions that can be used for reasoning. Leveraging existing visual databases, we first developed FaceInstruct-1M, a face-centered database for instruction tuning MLLMs for face processing. We then developed a novel face-specific visual encoder powered by Face-Region Guided Cross-Attention that integrates face geometry with local visual features. We evaluated the proposed method across nine different datasets and five different face processing tasks, including facial expression recognition, action unit detection, facial attribute detection, age estimation and deepfake detection. Face-LLaVA achieves superior results compared to existing open-source MLLMs and competitive performance compared to commercial solutions. Our model output also receives a higher reasoning rating by GPT under a zero-shot setting across all the tasks. Both our dataset and model wil be released at https://face-llava.github.io to support future advancements in social AI and foundational vision-language research.

DeepSeek-R1 Thoughtology: Let's about LLM Reasoning

Sara Vera Marjanović,Arkil Patel,Vaibhav Adlakha,Milad Aghajohari,Parishad BehnamGhader,Mehar Bhatia,Aditi Khandelwal,Austin Kraft,Benno Krojer,Xing Han Lù,Nicholas Meade,Dongchan Shin,Amirhossein Kazemnejad,Gaurav Kamath,Marius Mosbach,Karolina Stańczak,Siva Reddy

Task: 研究DeepSeek-R1模型的多步推理行为及其影响。

Motivation: 探索大型推理模型（如DeepSeek-R1）在复杂问题中的推理过程，以及其对模型性能、安全性和认知现象的影响。

Details

Method: 通过分析DeepSeek-R1的基本推理构建块，研究其推理长度、上下文管理、文化安全问题和认知现象。 Result: 发现DeepSeek-R1存在推理的‘最佳点’，过长推理时间会降低性能；模型倾向于重复已探索的问题，且安全性较弱。 Conclusion: DeepSeek-R1的推理行为复杂，需进一步优化以平衡性能与安全性。 Abstract: Large Reasoning Models like DeepSeek-R1 mark a fundamental shift in how LLMs approach complex problems. Instead of directly producing an answer for a given input, DeepSeek-R1 creates detailed multi-step reasoning chains, seemingly "thinking" about a problem before providing an answer. This reasoning process is publicly available to the user, creating endless opportunities for studying the reasoning behaviour of the model and opening up the field of Thoughtology. Starting from a taxonomy of DeepSeek-R1's basic building blocks of reasoning, our analyses on DeepSeek-R1 investigate the impact and controllability of thought length, management of long or confusing contexts, cultural and safety concerns, and the status of DeepSeek-R1 vis-\`a-vis cognitive phenomena, such as human-like language processing and world modelling. Our findings paint a nuanced picture. Notably, we show DeepSeek-R1 has a 'sweet spot' of reasoning, where extra inference time can impair model performance. Furthermore, we find a tendency for DeepSeek-R1 to persistently ruminate on previously explored problem formulations, obstructing further exploration. We also note strong safety vulnerabilities of DeepSeek-R1 compared to its non-reasoning counterpart, which can also compromise safety-aligned LLMs.

Few-Shot Adaptation of Grounding DINO for Agricultural Domain

Rajhans Singh,Rafael Bidese Puhl,Kshitiz Dhakal,Sudhir Sornapudi

Task: 提出一种高效的少样本适应方法，通过简化Grounding-DINO架构并引入可训练文本嵌入，以解决农业应用中复杂对象的检测问题。

Motivation: 深度学习模型在农业应用中依赖大量标注数据，而现有方法在复杂对象检测上存在挑战，需要更高效的解决方案。

Details

Method: 移除Grounding-DINO的文本编码器模块（BERT），引入随机初始化的可训练文本嵌入，实现少样本适应。 Result: 在多个农业数据集上表现优异，mAP比YOLO模型高约24%，在遥感任务中比现有方法高约10%。 Conclusion: 该方法为农业AI解决方案的自动化标注和开发提供了高效途径。 Abstract: Deep learning models are transforming agricultural applications by enabling automated phenotyping, monitoring, and yield estimation. However, their effectiveness heavily depends on large amounts of annotated training data, which can be labor and time intensive. Recent advances in open-set object detection, particularly with models like Grounding-DINO, offer a potential solution to detect regions of interests based on text prompt input. Initial zero-shot experiments revealed challenges in crafting effective text prompts, especially for complex objects like individual leaves and visually similar classes. To address these limitations, we propose an efficient few-shot adaptation method that simplifies the Grounding-DINO architecture by removing the text encoder module (BERT) and introducing a randomly initialized trainable text embedding. This method achieves superior performance across multiple agricultural datasets, including plant-weed detection, plant counting, insect identification, fruit counting, and remote sensing tasks. Specifically, it demonstrates up to a $\sim24\%$ higher mAP than fully fine-tuned YOLO models on agricultural datasets and outperforms previous state-of-the-art methods by $\sim10\%$ in remote sensing, under few-shot learning conditions. Our method offers a promising solution for automating annotation and accelerating the development of specialized agricultural AI solutions.

HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation

Mingxuan Li,Hanchen Li,Chenhao Tan

Task: 提出HypoEval框架，用于自动化评估自然语言生成任务。

Motivation: 现有LLM评估框架存在对齐性低或需要大量标注数据的问题，且缺乏评估理由。

Details

Method: 利用少量人工评估生成详细评分标准，结合分解维度的检查表方法进行综合评分。 Result: 仅需30次人工评估，HypoEval在Spearman和Pearson相关性上优于现有方法。 Conclusion: HypoEval是一种可靠且可解释的自动化评估框架。 Abstract: Large language models (LLMs) have demonstrated great potential for automating the evaluation of natural language generation. Previous frameworks of LLM-as-a-judge fall short in two ways: they either use zero-shot setting without consulting any human input, which leads to low alignment, or fine-tune LLMs on labeled data, which requires a non-trivial number of samples. Moreover, previous methods often provide little reasoning behind automated evaluations. In this paper, we propose HypoEval, Hypothesis-guided Evaluation framework, which first uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and then incorporates a checklist-like approach to combine LLM's assigned scores on each decomposed dimension to acquire overall scores. With only 30 human evaluations, HypoEval achieves state-of-the-art performance in alignment with both human rankings (Spearman correlation) and human scores (Pearson correlation), on average outperforming G-Eval by 11.86% and fine-tuned Llama-3.1-8B-Instruct with at least 3 times more human evaluations by 11.95%. Furthermore, we conduct systematic studies to assess the robustness of HypoEval, highlighting its effectiveness as a reliable and interpretable automated evaluation framework.

Quantifying Epistemic Uncertainty in Absolute Pose Regression

Fereidoon Zangeneh,Amit Dekel,Alessandro Pieropan,Patric Jensfelt

Task: 通过神经网络直接回归相机姿态的任务。

Motivation: 解决绝对姿态回归在训练域外预测不准确和不可靠的问题。

Details

Method: 提出一种基于变分框架的量化绝对姿态回归模型认知不确定性的新方法。 Result: 方法在捕捉不确定性与预测误差关系方面优于现有方法。 Conclusion: 该方法不仅提供预测置信度，还能统一处理观测模糊性，在重复结构中概率性地定位相机。 Abstract: Visual relocalization is the task of estimating the camera pose given an image it views. Absolute pose regression offers a solution to this task by training a neural network, directly regressing the camera pose from image features. While an attractive solution in terms of memory and compute efficiency, absolute pose regression's predictions are inaccurate and unreliable outside the training domain. In this work, we propose a novel method for quantifying the epistemic uncertainty of an absolute pose regression model by estimating the likelihood of observations within a variational framework. Beyond providing a measure of confidence in predictions, our approach offers a unified model that also handles observation ambiguities, probabilistically localizing the camera in the presence of repetitive structures. Our method outperforms existing approaches in capturing the relation between uncertainty and prediction error.

SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog

Jennifer D'Souza,Sameer Sadruddin,Holger Israel,Mathias Begoin,Diana Slawig

Task: 自动化主题标注，使用GND分类法为英文和德文的科学与技术记录推荐top-k主题。

Motivation: 探索如何利用大型语言模型（LLMs）改进数字图书馆的分类系统，特别是在多语言环境下。

Details

Method: 参与者开发基于LLM的系统，通过定量指标（精确率、召回率、F1分数）和主题专家的定性评估进行测试。 Result: 结果表明，LLM集成、合成数据生成和多语言处理在主题标注中表现有效。 Conclusion: 研究为LLMs在数字图书馆分类中的应用提供了有价值的见解。 Abstract: We present SemEval-2025 Task 5: LLMs4Subjects, a shared task on automated subject tagging for scientific and technical records in English and German using the GND taxonomy. Participants developed LLM-based systems to recommend top-k subjects, evaluated through quantitative metrics (precision, recall, F1-score) and qualitative assessments by subject specialists. Results highlight the effectiveness of LLM ensembles, synthetic data generation, and multilingual processing, offering insights into applying LLMs for digital library classification.

CEC-MMR: Cross-Entropy Clustering Approach to Multi-Modal Regression

Krzysztof Byrski,Jacek Tabor,Przemysław Spurek,Marcin Mazur

Task: 提出一种基于交叉熵聚类（CEC）的新方法CEC-MMR，用于自动检测回归问题中的组件数量。

Motivation: 传统混合密度网络（MDN）无法准确确定组件数量，导致预测值与实际数据差异较大。

Details

Method: 采用交叉熵聚类（CEC）方法，自动检测组件数量并唯一标识属性值对应的组件。 Result: 实验结果表明，CEC-MMR优于传统的MDN方法。 Conclusion: CEC-MMR是一种有效的替代方法，能够解决MDN在组件数量检测上的局限性。 Abstract: In practical applications of regression analysis, it is not uncommon to encounter a multitude of values for each attribute. In such a situation, the univariate distribution, which is typically Gaussian, is suboptimal because the mean may be situated between modes, resulting in a predicted value that differs significantly from the actual data. Consequently, to address this issue, a mixture distribution with parameters learned by a neural network, known as a Mixture Density Network (MDN), is typically employed. However, this approach has an important inherent limitation, in that it is not feasible to ascertain the precise number of components with a reasonable degree of accuracy. In this paper, we introduce CEC-MMR, a novel approach based on Cross-Entropy Clustering (CEC), which allows for the automatic detection of the number of components in a regression problem. Furthermore, given an attribute and its value, our method is capable of uniquely identifying it with the underlying component. The experimental results demonstrate that CEC-MMR yields superior outcomes compared to classical MDNs.

ConceptCarve: Dynamic Realization of Evidence

Eylon Caplan,Dan Goldwasser

Task: 开发一个名为ConceptCarve的证据检索框架，用于在大规模社交媒体数据中识别抽象概念实例并分析其在不同社区中的不同表现形式。

Motivation: 研究人类观点和行为需要理解社交媒体上复杂的思想模式，尤其是在涉及抽象概念（如自由与枪支所有权的关系）时，传统检索系统难以应对。

Details

Method: 结合传统检索器和大型语言模型（LLMs），动态表征检索空间，以识别抽象概念及其在不同社区中的实例化。 Result: ConceptCarve在社交媒体社区中检索证据的性能优于传统检索系统，并能生成可解释的证据表示。 Conclusion: ConceptCarve为分析跨社区复杂思想模式提供了有效工具，同时提升了检索抽象概念的能力。 Abstract: Finding evidence for human opinion and behavior at scale is a challenging task, often requiring an understanding of sophisticated thought patterns among vast online communities found on social media. For example, studying how gun ownership is related to the perception of Freedom, requires a retrieval system that can operate at scale over social media posts, while dealing with two key challenges: (1) identifying abstract concept instances, (2) which can be instantiated differently across different communities. To address these, we introduce ConceptCarve, an evidence retrieval framework that utilizes traditional retrievers and LLMs to dynamically characterize the search space during retrieval. Our experiments show that ConceptCarve surpasses traditional retrieval systems in finding evidence within a social media community. It also produces an interpretable representation of the evidence for that community, which we use to qualitatively analyze complex thought patterns that manifest differently across the communities.

Objaverse++: Curated 3D Object Dataset with Quality Annotations

Chendi Lin,Heshan Liu,Qunshu Lin,Zachary Bright,Shitao Tang,Yihui He,Minghao Liu,Ling Zhu,Cindy Le

Task: 提出Objaverse++，一个经过人工专家详细标注的Objaverse子集，用于提升3D内容生成的质量。

Motivation: 尽管Objaverse是目前最大的3D资产数据集，但其低质量模型占主导地位，限制了其实际应用价值。

Details

Method: 人工标注10,000个3D对象的详细属性，并训练神经网络为剩余数据集标注标签。 Result: 实验表明，基于质量优化的子集预训练的模型在图像到3D生成任务中表现更优，且高质量数据能加速训练损失收敛。 Conclusion: 精心筛选和丰富标注可以弥补原始数据集规模的不足，为开发3D生成模型提供更高效的路径。 Abstract: This paper presents Objaverse++, a curated subset of Objaverse enhanced with detailed attribute annotations by human experts. Recent advances in 3D content generation have been driven by large-scale datasets such as Objaverse, which contains over 800,000 3D objects collected from the Internet. Although Objaverse represents the largest available 3D asset collection, its utility is limited by the predominance of low-quality models. To address this limitation, we manually annotate 10,000 3D objects with detailed attributes, including aesthetic quality scores, texture color classifications, multi-object composition flags, transparency characteristics, etc. Then, we trained a neural network capable of annotating the tags for the rest of the Objaverse dataset. Through experiments and a user study on generation results, we demonstrate that models pre-trained on our quality-focused subset achieve better performance than those trained on the larger dataset of Objaverse in image-to-3D generation tasks. In addition, by comparing multiple subsets of training data filtered by our tags, our results show that the higher the data quality, the faster the training loss converges. These findings suggest that careful curation and rich annotation can compensate for the raw dataset size, potentially offering a more efficient path to develop 3D generative models. We release our enhanced dataset of approximately 500,000 curated 3D models to facilitate further research on various downstream tasks in 3D computer vision. In the near future, we aim to extend our annotations to cover the entire Objaverse dataset.

Visual-Aware Speech Recognition for Noisy Scenarios

Lakshmipathi Balaji,Karan Singla

Task: 利用视觉线索（如唇部动作和环境场景）改进嘈杂环境下的语音识别。

Motivation: 当前ASR或AVSR模型在嘈杂环境中表现不佳，需要更广泛的视觉信息来提升性能。

Details

Method: 通过预训练的语音和视觉编码器，结合多头注意力机制，利用环境视觉信息过滤噪声并改进转录。 Result: 在嘈杂场景中显著优于纯音频模型，视觉线索对转录准确性提升至关重要。 Conclusion: 提出的方法通过结合视觉信息有效提升嘈杂环境下的语音识别性能。 Abstract: Humans have the ability to utilize visual cues, such as lip movements and visual scenes, to enhance auditory perception, particularly in noisy environments. However, current Automatic Speech Recognition (ASR) or Audio-Visual Speech Recognition (AVSR) models often struggle in noisy scenarios. To solve this task, we propose a model that improves transcription by correlating noise sources to visual cues. Unlike works that rely on lip motion and require the speaker's visibility, we exploit broader visual information from the environment. This allows our model to naturally filter speech from noise and improve transcription, much like humans do in noisy scenarios. Our method re-purposes pretrained speech and visual encoders, linking them with multi-headed attention. This approach enables the transcription of speech and the prediction of noise labels in video inputs. We introduce a scalable pipeline to develop audio-visual datasets, where visual cues correlate to noise in the audio. We show significant improvements over existing audio-only models in noisy scenarios. Results also highlight that visual cues play a vital role in improved transcription accuracy.

DLTPose: 6DoF Pose Estimation From Accurate Dense Surface Point Estimates

Akash Jadhav,Michael Greenspan

Task: 提出一种结合稀疏关键点方法和密集像素预测优点的6DoF物体姿态估计新方法DLTPose。

Motivation: 解决现有关键点方法在处理对称物体时因固定关键点排序导致的性能下降问题。

Details

Method: 通过预测像素到关键点的径向距离，结合新的DLT公式生成3D物体表面估计，并引入对称感知的关键点排序方法。 Result: 在LINEMOD、Occlusion LINEMOD和YCB-Video数据集上表现优于现有方法，尤其对对称和遮挡物体。 Conclusion: DLTPose在6DoF姿态估计任务中表现出色，特别是在处理对称和遮挡物体时。 Abstract: We propose DLTPose, a novel method for 6DoF object pose estimation from RGB-D images that combines the accuracy of sparse keypoint methods with the robustness of dense pixel-wise predictions. DLTPose predicts per-pixel radial distances to a set of minimally four keypoints, which are then fed into our novel Direct Linear Transform (DLT) formulation to produce accurate 3D object frame surface estimates, leading to better 6DoF pose estimation. Additionally, we introduce a novel symmetry-aware keypoint ordering approach, designed to handle object symmetries that otherwise cause inconsistencies in keypoint assignments. Previous keypoint-based methods relied on fixed keypoint orderings, which failed to account for the multiple valid configurations exhibited by symmetric objects, which our ordering approach exploits to enhance the model's ability to learn stable keypoint representations. Extensive experiments on the benchmark LINEMOD, Occlusion LINEMOD and YCB-Video datasets show that DLTPose outperforms existing methods, especially for symmetric and occluded objects, demonstrating superior Mean Average Recall values of 86.5% (LM), 79.7% (LM-O) and 89.5% (YCB-V). The code is available at https://anonymous.4open.science/r/DLTPose_/ .

Language Modeling for the Future of Finance: A Quantitative Survey into Metrics, Tasks, and Data Opportunities

Nikita Tatarinov,Siddhant Sukhani,Agam Shah,Sudheer Chava

Task: 系统综述2017年至2024年间374篇NLP研究论文，重点关注其中221篇直接涉及金融任务的论文。

Motivation: 探索NLP技术在金融问题中的应用趋势，以推动新的分析和决策方法。

Details

Method: 对论文进行11个定性和定量维度的评估，分析关键趋势如通用语言模型的使用、情感分析和信息提取的进展等。 Result: 发现通用语言模型使用增加，情感分析和信息提取稳步发展，可解释性和隐私保护方法兴起，并强调领域特定评估指标的重要性。 Conclusion: 提出需要更易获取和适应的数据集，并建议纳入金融危机时期以增强模型在实际条件下的鲁棒性，为研究者提供实用见解。 Abstract: Recent advances in language modeling have led to growing interest in applying Natural Language Processing (NLP) techniques to financial problems, enabling new approaches to analysis and decision-making. To systematically examine this trend, we review 374 NLP research papers published between 2017 and 2024 across 38 conferences and workshops, with a focused analysis of 221 papers that directly address finance-related tasks. We evaluate these papers across 11 qualitative and quantitative dimensions, identifying key trends such as the increasing use of general-purpose language models, steady progress in sentiment analysis and information extraction, and emerging efforts around explainability and privacy-preserving methods. We also discuss the use of evaluation metrics, highlighting the importance of domain-specific ones to complement standard machine learning metrics. Our findings emphasize the need for more accessible, adaptive datasets and highlight the significance of incorporating financial crisis periods to strengthen model robustness under real-world conditions. This survey provides a structured overview of NLP research applied to finance and offers practical insights for researchers and practitioners working at this intersection.

Zeus: Zero-shot LLM Instruction for Union Segmentation in Multimodal Medical Imaging

Siyuan Dai,Kai Ye,Guodong Liu,Haoteng Tang,Liang Zhan

Task: 提出一种基于视觉-大语言模型（Vision-LLM）联合框架的多模态医学图像分割方法，无需预先收集配对的视觉-语言数据集。

Motivation: 临床诊断需要结合领域知识（如文本信息），但收集配对的视觉-语言数据集成本高且耗时。

Details

Method: 利用冻结的大语言模型（LLMs）生成零样本指令，模拟放射学扫描和报告生成过程，结合多模态医学图像生成精确文本指令，并设计联合分割框架。 Result: 实验结果表明，该方法在多模态分割任务中优于基线方法。 Conclusion: 提出的Vision-LLM框架能够有效解决多模态医学图像分割问题，无需依赖预先配对的视觉-语言数据集。 Abstract: Medical image segmentation has achieved remarkable success through the continuous advancement of UNet-based and Transformer-based foundation backbones. However, clinical diagnosis in the real world often requires integrating domain knowledge, especially textual information. Conducting multimodal learning involves visual and text modalities shown as a solution, but collecting paired vision-language datasets is expensive and time-consuming, posing significant challenges. Inspired by the superior ability in numerous cross-modal tasks for Large Language Models (LLMs), we proposed a novel Vision-LLM union framework to address the issues. Specifically, we introduce frozen LLMs for zero-shot instruction generation based on corresponding medical images, imitating the radiology scanning and report generation process. {To better approximate real-world diagnostic processes}, we generate more precise text instruction from multimodal radiology images (e.g., T1-w or T2-w MRI and CT). Based on the impressive ability of semantic understanding and rich knowledge of LLMs. This process emphasizes extracting special features from different modalities and reunion the information for the ultimate clinical diagnostic. With generated text instruction, our proposed union segmentation framework can handle multimodal segmentation without prior collected vision-language datasets. To evaluate our proposed method, we conduct comprehensive experiments with influential baselines, the statistical results and the visualized case study demonstrate the superiority of our novel method.}

RAISE: Reinforenced Adaptive Instruction Selection For Large Language Models

Lv Qingsong,Yangning Li,Zihua Lan,Zishan Xu,Jiwei Tang,Yinghui Li,Wenhao Jiang,Hai-Tao Zheng,Philip S. Yu

Task: 设计一个动态、任务目标驱动的指令选择框架RAISE，以优化大型语言模型的指令微调过程。

Motivation: 现有的指令选择方法多基于启发式质量指标，且仅在训练前进行数据选择，导致指令微调优化不足，难以针对特定任务优化。

Details

Method: 将动态指令选择建模为序列决策过程，使用强化学习训练选择策略，并在整个指令微调过程中优化指令选择。 Result: RAISE方法在仅更新1%训练步骤的情况下，性能优于全数据训练，证明了其高效性和有效性。 Conclusion: RAISE框架具有强任务优化能力和可解释性，显著提升了指令微调的效果。 Abstract: In the instruction fine-tuning of large language models (LLMs), it has become a consensus that a few high-quality instructions are superior to a large number of low-quality instructions. At present, many instruction selection methods have been proposed, but most of these methods select instruction based on heuristic quality metrics, and only consider data selection before training. These designs lead to insufficient optimization of instruction fine-tuning, and fixed heuristic indicators are often difficult to optimize for specific tasks. So we designed a dynamic, task-objective-driven instruction selection framework RAISE(Reinforenced Adaptive Instruction SElection), which incorporates the entire instruction fine-tuning process into optimization, selecting instruction at each step based on the expected impact of instruction on model performance improvement. Our approach is well interpretable and has strong task-specific optimization capabilities. By modeling dynamic instruction selection as a sequential decision-making process, we use RL to train our selection strategy. Extensive experiments and result analysis prove the superiority of our method compared with other instruction selection methods. Notably, RAISE achieves superior performance by updating only 1\% of the training steps compared to full-data training, demonstrating its efficiency and effectiveness.

View-Dependent Uncertainty Estimation of 3D Gaussian Splatting

Chenyu Han,Corentin Dumery

Task: 提出一种在3D高斯泼溅（3DGS）中建模不确定性的方法。

Motivation: 3DGS在3D场景重建中具有高视觉精度，但其不确定性估计尚未充分研究，而这对下游任务（如资产提取和场景补全）至关重要。

Details

Method: 将不确定性建模为一种额外的视角依赖的每高斯特征，并使用球谐函数进行建模。 Result: 该方法简单有效、易于解释，且比集成方法更快，同时保持高精度。 Conclusion: 提出的不确定性建模方法能够高效地集成到传统3DGS流程中，为下游任务提供支持。 Abstract: 3D Gaussian Splatting (3DGS) has become increasingly popular in 3D scene reconstruction for its high visual accuracy. However, uncertainty estimation of 3DGS scenes remains underexplored and is crucial to downstream tasks such as asset extraction and scene completion. Since the appearance of 3D gaussians is view-dependent, the color of a gaussian can thus be certain from an angle and uncertain from another. We thus propose to model uncertainty in 3DGS as an additional view-dependent per-gaussian feature that can be modeled with spherical harmonics. This simple yet effective modeling is easily interpretable and can be integrated into the traditional 3DGS pipeline. It is also significantly faster than ensemble methods while maintaining high accuracy, as demonstrated in our experiments.

MDIT: A Model-free Data Interpolation Method for Diverse Instruction Tuning

Yangning Li,Zihua Lan,Lv Qingsong,Yinghui Li,Hai-Tao Zheng

Task: 提出一种名为MDIT的模型无关数据插值方法，用于多样化的指令调优。

Motivation: 当前数据管理策略在生成多样化和全面数据方面存在挑战，限制了模型性能的进一步提升。

Details

Method: 通过任务插值生成多样化和高质量的指令数据，并结合基于多样性的聚类策略。 Result: 在多个基准任务中表现出优越性能，显著提升了LLMs在通用问答、数学推理和代码生成等任务中的表现。 Conclusion: MDIT提供了一种高效且自动化的数据合成方法，无需依赖外部资源即可生成多样化指令数据，扩展了LLMs在复杂环境中的应用潜力。 Abstract: As Large Language Models (LLMs) are increasingly applied across various tasks, instruction tuning has emerged as a critical method for enhancing model performance. However, current data management strategies face substantial challenges in generating diverse and comprehensive data, restricting further improvements in model performance. To address this gap, we propose MDIT, a novel model-free data interpolation method for diverse instruction tuning, which generates varied and high-quality instruction data by performing task interpolation. Moreover, it contains diversity-based clustering strategies to ensure the diversity of the training data. Extensive experiments show that our method achieves superior performance in multiple benchmark tasks. The LLMs finetuned with MDIT show significant improvements in numerous tasks such as general question answering, math reasoning, and code generation. MDIT offers an efficient and automatic data synthetic method, generating diverse instruction data without depending on external resources while expanding the application potential of LLMs in complex environments.

Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction

Junyi Ma,Wentao Bao,Jingyi Xu,Guanzhong Sun,Xieyuanli Chen,Hesheng Wang

Task: 预测3D手部轨迹，结合多模态环境信息。

Motivation: 现有方法仅基于2D视频输入，忽略了多模态信息和手部运动与头戴相机运动的协同作用。

Details

Method: 提出MMTwin模型，结合2D RGB图像、3D点云、过去手部轨迹和文本提示，并使用双扩散模型预测相机运动和手部轨迹。 Result: 在三个公开数据集和自录数据上表现优于现有方法，泛化能力强。 Conclusion: MMTwin能有效预测3D手部轨迹，并推广到新环境。 Abstract: Predicting hand motion is critical for understanding human intentions and bridging the action space between human movements and robot manipulations. Existing hand trajectory prediction (HTP) methods forecast the future hand waypoints in 3D space conditioned on past egocentric observations. However, such models are only designed to accommodate 2D egocentric video inputs. There is a lack of awareness of multimodal environmental information from both 2D and 3D observations, hindering the further improvement of 3D HTP performance. In addition, these models overlook the synergy between hand movements and headset camera egomotion, either predicting hand trajectories in isolation or encoding egomotion only from past frames. To address these limitations, we propose novel diffusion models (MMTwin) for multimodal 3D hand trajectory prediction. MMTwin is designed to absorb multimodal information as input encompassing 2D RGB images, 3D point clouds, past hand waypoints, and text prompt. Besides, two latent diffusion models, the egomotion diffusion and the HTP diffusion as twins, are integrated into MMTwin to predict camera egomotion and future hand trajectories concurrently. We propose a novel hybrid Mamba-Transformer module as the denoising model of the HTP diffusion to better fuse multimodal features. The experimental results on three publicly available datasets and our self-recorded data demonstrate that our proposed MMTwin can predict plausible future 3D hand trajectories compared to the state-of-the-art baselines, and generalizes well to unseen environments. The code and pretrained models will be released at https://github.com/IRMVLab/MMTwin.

PAYADOR: A Minimalist Approach to Grounding Language Models on Structured Data for Interactive Storytelling and Role-playing Games

Santiago Góngora,Luis Chiruzzo,Gonzalo Méndez,Pablo Gervás

Task: 提出一种名为PAYADOR的新方法，通过预测动作结果而非直接表示动作本身，解决交互式叙事系统中的世界更新问题。

Motivation: 传统方法将玩家输入映射到预编程动作，限制了玩家的自由意志，尤其在需要即兴创作的RPG中问题更为突出。

Details

Method: 基于大型语言模型，结合虚构世界的最小表示，预测动作的结果。 Result: 获得了有希望的结果，并将方法开源以供其他研究使用。 Conclusion: PAYADOR方法为释放RPG的共创潜力提供了新的可能性。 Abstract: Every time an Interactive Storytelling (IS) system gets a player input, it is facing the world-update problem. Classical approaches to this problem consist in mapping that input to known preprogrammed actions, what can severely constrain the free will of the player. When the expected experience has a strong focus on improvisation, like in Role-playing Games (RPGs), this problem is critical. In this paper we present PAYADOR, a different approach that focuses on predicting the outcomes of the actions instead of representing the actions themselves. To implement this approach, we ground a Large Language Model to a minimal representation of the fictional world, obtaining promising results. We make this contribution open-source, so it can be adapted and used for other related research on unleashing the co-creativity power of RPGs.

BRepFormer: Transformer-Based B-rep Geometric Feature Recognition

Yongkang Dai,Xiaoshui Huang,Yunpeng Bai,Hao Guo,Hongping Gan,Ling Yang,Yilei Shi

Task: 提出一种基于Transformer的模型BRepFormer，用于识别B-rep模型中的加工特征和复杂几何特征。

Motivation: 现有研究多集中于加工特征识别（MFR），未能有效捕捉复杂几何特征的拓扑和几何特性，限制了其在工业应用中的实用性。

Details

Method: BRepFormer通过编码和融合模型的几何与拓扑特征，利用Transformer架构进行特征传播，并通过识别头识别几何特征；同时引入结合边特征和拓扑特征的偏置以强化几何约束。 Result: BRepFormer在MFInstSeg、MFTRCAD和自建的CBF数据集上达到了最先进的识别精度。 Conclusion: BRepFormer能够有效识别复杂B-rep模型的特征，更符合工业应用需求，同时提出的CBF数据集填补了现有数据集的不足。 Abstract: Recognizing geometric features on B-rep models is a cornerstone technique for multimedia content-based retrieval and has been widely applied in intelligent manufacturing. However, previous research often merely focused on Machining Feature Recognition (MFR), falling short in effectively capturing the intricate topological and geometric characteristics of complex geometry features. In this paper, we propose BRepFormer, a novel transformer-based model to recognize both machining feature and complex CAD models' features. BRepFormer encodes and fuses the geometric and topological features of the models. Afterwards, BRepFormer utilizes a transformer architecture for feature propagation and a recognition head to identify geometry features. During each iteration of the transformer, we incorporate a bias that combines edge features and topology features to reinforce geometric constraints on each face. In addition, we also proposed a dataset named Complex B-rep Feature Dataset (CBF), comprising 20,000 B-rep models. By covering more complex B-rep models, it is better aligned with industrial applications. The experimental results demonstrate that BRepFormer achieves state-of-the-art accuracy on the MFInstSeg, MFTRCAD, and our CBF datasets.

Multilingual MFA: Forced Alignment on Low-Resource Related Languages

Alessio Tosolini,Claire Bowern

Task: 比较多语言和跨语言训练在相关和不相关的澳大利亚语言中的效果。

Motivation: 研究多语言和跨语言训练对语言相似性不同的澳大利亚语言的影响。

Details

Method: 使用蒙特利尔强制对齐器从头训练声学模型，并调整大型英语模型，评估结果。 Result: 结果表明，调整英语基线模型对未见语言有益。 Conclusion: 跨语言训练在未见语言中具有优势。 Abstract: We compare the outcomes of multilingual and crosslingual training for related and unrelated Australian languages with similar phonological inventories. We use the Montreal Forced Aligner to train acoustic models from scratch and adapt a large English model, evaluating results against seen data, unseen data (seen language), and unseen data and language. Results indicate benefits of adapting the English baseline model for previously unseen languages.

Model Discrepancy Learning: Synthetic Faces Detection Based on Multi-Reconstruction

Qingchao Jiang,Zhishuo Xu,Zhiying Zhu,Ning Chen,Haoyue Wang,Zhongjie Ba

Task: 探索合成图像与其生成技术之间的内在关系，并提出一种基于多重建的检测器。

Motivation: 现有研究忽视了不同生成技术之间的差异，导致合成人脸检测的局限性。

Details

Method: 通过使用多种生成模型对图像进行反向重建，分析重建差异以区分真实图像与合成图像。 Result: 提出的检测器在性能、泛化能力和鲁棒性方面表现优异。 Conclusion: 多重建方法能有效区分合成图像，且新数据集ASFD补充了现有资源。 Abstract: Advances in image generation enable hyper-realistic synthetic faces but also pose risks, thus making synthetic face detection crucial. Previous research focuses on the general differences between generated images and real images, often overlooking the discrepancies among various generative techniques. In this paper, we explore the intrinsic relationship between synthetic images and their corresponding generation technologies. We find that specific images exhibit significant reconstruction discrepancies across different generative methods and that matching generation techniques provide more accurate reconstructions. Based on this insight, we propose a Multi-Reconstruction-based detector. By reversing and reconstructing images using multiple generative models, we analyze the reconstruction differences among real, GAN-generated, and DM-generated images to facilitate effective differentiation. Additionally, we introduce the Asian Synthetic Face Dataset (ASFD), containing synthetic Asian faces generated with various GANs and DMs. This dataset complements existing synthetic face datasets. Experimental results demonstrate that our detector achieves exceptional performance, with strong generalization and robustness.

Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalization

Shujin Wu,Cheng Qian,Yi R.,Fung,Paul Pu Liang,Heng Ji

Task: 提出一种名为Alice的主动学习框架，通过利用教师和学生模型的互补知识，提升弱到强泛化（W2SG）的性能。

Motivation: 传统W2SG方法依赖被动学习，限制了学生模型的潜力，需要一种更有效的方法来利用教师和学生的互补知识。

Details

Method: 引入Alice框架，通过探测教师模型的不确定性并结合其响应，指导学生模型自我生成改进的监督信号；针对能力差距大的情况，提出级联Alice，采用分层训练方法。 Result: 实验结果显示，Alice在知识推理（+4.0%）、数学推理（+22.62%）和逻辑推理（+12.11%）任务上显著优于传统W2SG方法。 Conclusion: Alice框架为W2SG提供了一种更有效的知识传递和监督范式，显著提升了性能。 Abstract: The growing capabilities of large language models (LLMs) present a key challenge of maintaining effective human oversight. Weak-to-strong generalization (W2SG) offers a promising framework for supervising increasingly capable LLMs using weaker ones. Traditional W2SG methods rely on passive learning, where a weak teacher provides noisy demonstrations to train a strong student. This hinders students from employing their knowledge during training and reaching their full potential. In this work, we introduce Alice (pro{A}ctive {l}earning w{i}th tea{c}her's D{e}monstrations), a framework that leverages complementary knowledge between teacher and student to enhance the learning process.We probe the knowledge base of the teacher model by eliciting their uncertainty, and then use these insights together with teachers' responses as demonstrations to guide student models in self-generating improved responses for supervision. In addition, for situations with significant capability gaps between teacher and student models, we introduce cascade Alice, which employs a hierarchical training approach where weak teachers initially supervise intermediate models, who then guide stronger models in sequence. Experimental results demonstrate that our method significantly enhances the W2SG performance, yielding substantial improvements in three key tasks compared to the original W2SG: knowledge-based reasoning (+4.0%), mathematical reasoning (+22.62%), and logical reasoning (+12.11%). This highlights the effectiveness of our new W2SG paradigm that enables more robust knowledge transfer and supervision outcome.

ID-Booth: Identity-consistent Face Generation with Diffusion Models

Darian Tomašević,Fadi Boutros,Chenhao Lin,Naser Damer,Vitomir Štruc,Peter Peer

Task: 提出一种名为ID-Booth的生成扩散框架，用于在保持身份一致性的同时生成高质量合成数据。

Motivation: 现有生成模型在训练时未充分考虑身份信息，导致生成图像与目标身份一致性差；而基于身份的训练方法又容易过拟合，降低生成图像的多样性。

Details

Method: ID-Booth结合了去噪网络、变分自编码器和文本编码器，采用新颖的三元组身份训练目标，实现身份一致的图像生成。 Result: 实验表明，ID-Booth在身份一致性和多样性上优于现有方法，并能有效增强小规模数据集。 Conclusion: ID-Booth在隐私保护的前提下，提升了生成数据的质量和识别模型的性能。 Abstract: Recent advances in generative modeling have enabled the generation of high-quality synthetic data that is applicable in a variety of domains, including face recognition. Here, state-of-the-art generative models typically rely on conditioning and fine-tuning of powerful pretrained diffusion models to facilitate the synthesis of realistic images of a desired identity. Yet, these models often do not consider the identity of subjects during training, leading to poor consistency between generated and intended identities. In contrast, methods that employ identity-based training objectives tend to overfit on various aspects of the identity, and in turn, lower the diversity of images that can be generated. To address these issues, we present in this paper a novel generative diffusion-based framework, called ID-Booth. ID-Booth consists of a denoising network responsible for data generation, a variational auto-encoder for mapping images to and from a lower-dimensional latent space and a text encoder that allows for prompt-based control over the generation procedure. The framework utilizes a novel triplet identity training objective and enables identity-consistent image generation while retaining the synthesis capabilities of pretrained diffusion models. Experiments with a state-of-the-art latent diffusion model and diverse prompts reveal that our method facilitates better intra-identity consistency and inter-identity separability than competing methods, while achieving higher image diversity. In turn, the produced data allows for effective augmentation of small-scale datasets and training of better-performing recognition models in a privacy-preserving manner. The source code for the ID-Booth framework is publicly available at https://github.com/dariant/ID-Booth.

Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extraction

Saurabh Srivastava,Ziyu Yao

Task: 系统研究大型推理模型（LRMs）在事件提取任务中是否需要提示优化。

Motivation: 尽管LRMs在推理任务中表现出色，但仍需验证其是否仍需提示优化以提高准确性。

Details

Method: 通过实验比较两种LRMs（DeepSeek-R1和o1）和两种通用LLMs（GPT-4o和GPT-4.5）在事件提取任务中的表现，分别作为任务模型和提示优化器。 Result: 结果表明，LRMs作为任务模型仍需提示优化，且作为提示优化器能生成更有效的提示。 Conclusion: LRMs在事件提取任务中仍需提示优化，且其作为提示优化器表现出稳定性和一致性。 Abstract: Large Reasoning Models (LRMs) such as DeepSeek-R1 and OpenAI o1 have demonstrated remarkable capabilities in various reasoning tasks. Their strong capability to generate and reason over intermediate thoughts has also led to arguments that they may no longer require extensive prompt engineering or optimization to interpret human instructions and produce accurate outputs. In this work, we aim to systematically study this open question, using the structured task of event extraction for a case study. We experimented with two LRMs (DeepSeek-R1 and o1) and two general-purpose Large Language Models (LLMs) (GPT-4o and GPT-4.5), when they were used as task models or prompt optimizers. Our results show that on tasks as complicated as event extraction, LRMs as task models still benefit from prompt optimization, and that using LRMs as prompt optimizers yields more effective prompts. Finally, we provide an error analysis of common errors made by LRMs and highlight the stability and consistency of LRMs in refining task instructions and event guidelines.

FAIR-SIGHT: Fairness Assurance in Image Recognition via Simultaneous Conformal Thresholding and Dynamic Output Repair

Arya Fayyazi,Mehdi Kamal,Massoud Pedram

Task: 提出FAIR-SIGHT框架，通过结合共形预测和动态输出修复机制，确保计算机视觉系统的公平性。

Motivation: 解决计算机视觉系统中存在的公平性问题，同时保持预测性能。

Details

Method: 结合共形预测和动态输出修复机制，计算公平感知的非一致性分数，并设置自适应阈值进行修正。 Result: 理论分析验证了方法的误差控制和收敛性，实验表明FAIR-SIGHT显著减少了公平性差异且保持了高预测性能。 Conclusion: FAIR-SIGHT是一种有效的后处理框架，能够在无需重新训练或访问模型内部参数的情况下提升公平性。 Abstract: We introduce FAIR-SIGHT, an innovative post-hoc framework designed to ensure fairness in computer vision systems by combining conformal prediction with a dynamic output repair mechanism. Our approach calculates a fairness-aware non-conformity score that simultaneously assesses prediction errors and fairness violations. Using conformal prediction, we establish an adaptive threshold that provides rigorous finite-sample, distribution-free guarantees. When the non-conformity score for a new image exceeds the calibrated threshold, FAIR-SIGHT implements targeted corrective adjustments, such as logit shifts for classification and confidence recalibration for detection, to reduce both group and individual fairness disparities, all without the need for retraining or having access to internal model parameters. Comprehensive theoretical analysis validates our method's error control and convergence properties. At the same time, extensive empirical evaluations on benchmark datasets show that FAIR-SIGHT significantly reduces fairness disparities while preserving high predictive performance.

Enhancing Time Series Forecasting via Multi-Level Text Alignment with LLMs

Taibiao Zhao,Xiaobing Chen,Mingxuan Sun

Task: 提出一种多级文本对齐框架，将时间序列数据转换为基于语言的表示形式，以用于大语言模型（LLMs）的时间序列预测。

Motivation: 时间序列数据是连续的，而LLMs处理的是离散的标记，现有方法在将时间序列数据转换为文本形式时难以同时保证预测准确性和可解释性。

Details

Method: 将时间序列分解为趋势、季节性和残差分量，并将其重新编程为特定于分量的文本表示，通过多级对齐机制将这些表示与预训练的词标记对齐。 Result: 在多个数据集上的实验表明，该方法在预测准确性上优于现有最先进模型，同时提供了良好的可解释性。 Conclusion: 提出的多级文本对齐框架不仅提高了时间序列预测的准确性，还增强了其表示的可解释性。 Abstract: The adaptation of large language models (LLMs) to time series forecasting poses unique challenges, as time series data is continuous in nature, while LLMs operate on discrete tokens. Despite the success of LLMs in natural language processing (NLP) and other structured domains, aligning time series data with language-based representations while maintaining both predictive accuracy and interpretability remains a significant hurdle. Existing methods have attempted to reprogram time series data into text-based forms, but these often fall short in delivering meaningful, interpretable results. In this paper, we propose a multi-level text alignment framework for time series forecasting using LLMs that not only improves prediction accuracy but also enhances the interpretability of time series representations. Our method decomposes time series into trend, seasonal, and residual components, which are then reprogrammed into component-specific text representations. We introduce a multi-level alignment mechanism, where component-specific embeddings are aligned with pre-trained word tokens, enabling more interpretable forecasts. Experiments on multiple datasets demonstrate that our method outperforms state-of-the-art models in accuracy while providing good interpretability.

FlexIP: Dynamic Control of Preservation and Personality for Customized Image Generation

Linyan Huang,Haonan Lin,Yanning Zhou,Kaiwen Xiao

Task: 提出FlexIP框架，解决2D生成模型中身份保持与多样化编辑之间的权衡问题。

Motivation: 现有方法在身份保持和个性化编辑之间存在固有权衡，需要一种新方法来同时实现这两个目标。

Details

Method: 通过两个专用组件（个性化适配器和保持适配器）将控制机制显式注入生成模型，实现灵活的参数化控制。 Result: 实验结果表明，FlexIP突破了传统方法的性能限制，实现了更好的身份保持和更丰富的个性化生成能力。 Conclusion: FlexIP框架通过解耦身份保持和个性化编辑目标，提供了一种灵活且高效的解决方案。 Abstract: With the rapid advancement of 2D generative models, preserving subject identity while enabling diverse editing has emerged as a critical research focus. Existing methods typically face inherent trade-offs between identity preservation and personalized manipulation. We introduce FlexIP, a novel framework that decouples these objectives through two dedicated components: a Personalization Adapter for stylistic manipulation and a Preservation Adapter for identity maintenance. By explicitly injecting both control mechanisms into the generative model, our framework enables flexible parameterized control during inference through dynamic tuning of the weight adapter. Experimental results demonstrate that our approach breaks through the performance limitations of conventional methods, achieving superior identity preservation while supporting more diverse personalized generation capabilities (Project Page: https://flexip-tech.github.io/flexip/).

TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models

Sher Badshah,Ali Emami,Hassan Sajjad

Task: 提出一种无需预定义标准答案的LLM输出评估框架TALE。

Motivation: 解决传统评估方法在成本、扩展性和完整性上的局限性。

Details

Method: 利用具有工具访问能力的代理，动态检索和综合外部证据，通过迭代查询、信息收集和反思优化搜索。 Result: 在多个自由形式QA基准测试中，TALE不仅优于基于标准参考的指标，还与人类评估结果高度一致。 Conclusion: TALE提升了LLM在动态现实场景中的评估可靠性，无需依赖静态参考。 Abstract: As Large Language Models (LLMs) become increasingly integrated into real-world, autonomous applications, relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness. We propose Tool-Augmented LLM Evaluation (TALE), a framework to assess LLM outputs without predetermined ground-truth answers. Unlike conventional metrics that compare to fixed references or depend solely on LLM-as-a-judge knowledge, TALE employs an agent with tool-access capabilities that actively retrieves and synthesizes external evidence. It iteratively generates web queries, collects information, summarizes findings, and refines subsequent searches through reflection. By shifting away from static references, TALE aligns with free-form question-answering tasks common in real-world scenarios. Experimental results on multiple free-form QA benchmarks show that TALE not only outperforms standard reference-based metrics for measuring response accuracy but also achieves substantial to near-perfect agreement with human evaluations. TALE enhances the reliability of LLM evaluations in real-world, dynamic scenarios without relying on static references.

Leveraging LLMs for Multimodal Retrieval-Augmented Radiology Report Generation via Key Phrase Extraction

Kyoyun Choi,Byungmu Yoon,Soobum Kim,Jonggwon Park

Task: 提出一种基于检索增强生成的方法，用于自动生成放射学报告，同时减少幻觉并降低计算需求。

Motivation: 多模态大语言模型（MLLMs）资源密集，需要大量数据和计算成本，因此需要一种更高效的方法。

Details

Method: 结合多模态检索和大语言模型，提取关键短语，采用图像编码器结构搜索、文本嵌入噪声添加和对比学习等策略。 Result: 在MIMIC-CXR数据集上取得CheXbert指标的最先进结果和RadGraph F1指标的竞争性表现，且无需微调大语言模型。 Conclusion: 该方法在多视角放射学报告生成中表现出强大的泛化能力，适用于全面的临床应用。 Abstract: Automated radiology report generation (RRG) holds potential to reduce radiologists' workload, especially as recent advancements in large language models (LLMs) enable the development of multimodal models for chest X-ray (CXR) report generation. However, multimodal LLMs (MLLMs) are resource-intensive, requiring vast datasets and substantial computational cost for training. To address these challenges, we propose a retrieval-augmented generation approach that leverages multimodal retrieval and LLMs to generate radiology reports while mitigating hallucinations and reducing computational demands. Our method uses LLMs to extract key phrases from radiology reports, effectively focusing on essential diagnostic information. Through exploring effective training strategies, including image encoder structure search, adding noise to text embeddings, and additional training objectives, we combine complementary pre-trained image encoders and adopt contrastive learning between text and semantic image embeddings. We evaluate our approach on MIMIC-CXR dataset, achieving state-of-the-art results on CheXbert metrics and competitive RadGraph F1 metric alongside MLLMs, without requiring LLM fine-tuning. Our method demonstrates robust generalization for multi-view RRG, making it suitable for comprehensive clinical applications.

Talking Point based Ideological Discourse Analysis in News Events

Nishanth Nakshatri,Nikhil Mehta,Siyi Liu,Sihao Chen,Daniel J. Hopkins,Dan Roth,Dan Goldwasser

Task: 提出一个基于意识形态话语分析理论的框架，用于分析与现实世界事件相关的新闻文章。

Motivation: 大型语言模型（LLMs）在分析意识形态话语时难以捕捉关键元素和整合上下文信息，因此需要一种新的方法来解决这些限制。

Details

Method: 通过关系结构（谈论点）表示新闻文章，构建重复主题的词汇表，并生成意识形态特定的观点。 Result: 框架在意识形态和党派分类任务中表现良好，并通过人类验证，同时展示了在创建事件快照中的实用性。 Conclusion: 该框架为意识形态话语分析提供了有效工具，并公开了数据集和模型以支持进一步研究。 Abstract: Analyzing ideological discourse even in the age of LLMs remains a challenge, as these models often struggle to capture the key elements that shape real-world narratives. Specifically, LLMs fail to focus on characteristic elements driving dominant discourses and lack the ability to integrate contextual information required for understanding abstract ideological views. To address these limitations, we propose a framework motivated by the theory of ideological discourse analysis to analyze news articles related to real-world events. Our framework represents the news articles using a relational structure - talking points, which captures the interaction between entities, their roles, and media frames along with a topic of discussion. It then constructs a vocabulary of repeating themes - prominent talking points, that are used to generate ideology-specific viewpoints (or partisan perspectives). We evaluate our framework's ability to generate these perspectives through automated tasks - ideology and partisan classification tasks, supplemented by human validation. Additionally, we demonstrate straightforward applicability of our framework in creating event snapshots, a visual way of interpreting event discourse. We release resulting dataset and model to the community to support further research.

RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Radiology with Zero-Shot Multi-Task Capability

Jonggwon Park,Soobum Kim,Byungmu Yoon,Kyoyun Choi

Task: 提出RadZero，一种基于相似性的跨注意力框架，用于放射学中的视觉-语言对齐，具有零样本多任务能力。

Motivation: 现有方法难以有效利用复杂的放射学报告进行学习，依赖低分辨率图像，且注意力机制的可解释性有限。

Details

Method: RadZero利用大型语言模型从放射学报告中提取最小语义句子，采用多正对比学习策略捕捉图像与多个相关文本描述的关系，并使用预训练视觉编码器处理高分辨率图像。 Result: 在公共胸部X光基准测试中，RadZero在零样本分类、定位和分割任务上优于现有方法，并展示了在视觉-语言对齐中提高可解释性的潜力。 Conclusion: RadZero在医学影像中表现出色，支持零样本推理和开放词汇语义分割，验证了其有效性。 Abstract: Recent advancements in multi-modal models have significantly improved vision-language alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning, rely on low-resolution images, and offer limited interpretability in attention mechanisms. To address these challenges, we introduce RadZero, a novel similarity-based cross-attention framework for vision-language alignment in radiology with zero-shot multi-task capability. RadZero leverages large language models to extract minimal semantic sentences from radiology reports and employs a multi-positive contrastive learning strategy to effectively capture relationships between images and multiple relevant textual descriptions. It also utilizes a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity between text embeddings and local image patch features, RadZero enables zero-shot inference with similarity probability for classification and pixel-level cross-modal similarity maps for grounding and segmentation. Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. Furthermore, cross-modal similarity map analysis highlights its potential for improving explainability in vision-language alignment. Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging.

AI Coding with Few-Shot Prompting for Thematic Analysis

Samuel Flanders,Melati Nungsari,Mark Cheong Wing Loong

Task: 探索使用大型语言模型（如GPT 3.5-Turbo）进行主题分析编码。

Motivation: 主题分析编码通常需要大量人工劳动，使得研究人员难以对大规模语料库进行详尽分析。

Details

Method: 采用少样本提示方法，利用语义相似段落生成更高质量的编码，同时使用成本低且易于扩展的模型。 Result: 提高了编码质量，同时降低了成本和扩展难度。 Conclusion: 大型语言模型可以有效支持主题分析编码，为大规模语料分析提供可行方案。 Abstract: This paper explores the use of large language models (LLMs), here represented by GPT 3.5-Turbo to perform coding for a thematic analysis. Coding is highly labor intensive, making it infeasible for most researchers to conduct exhaustive thematic analyses of large corpora. We utilize few-shot prompting with higher quality codes generated on semantically similar passages to enhance the quality of the codes while utilizing a cheap, more easily scalable model.

ThermoStereoRT: Thermal Stereo Matching in Real Time via Knowledge Distillation and Attention-based Refinement

Anning Hu,Ang Li,Xirui Jin,Danping Zou

Task: 提出ThermoStereoRT，一种实时热成像立体匹配方法，用于从两幅校正的热成像立体图像中恢复视差。

Motivation: 针对全天候条件下的应用需求，如夜间无人机监控或床底清洁机器人。

Details

Method: 采用轻量级但强大的主干网络构建3D成本体积，并利用多尺度注意力机制生成初始视差图；设计新颖的通道和空间注意力模块进行细化；通过知识蒸馏解决热成像数据稀疏问题。 Result: 在多个数据集上的综合评估表明，ThermoStereoRT具有实时性和鲁棒性。 Conclusion: ThermoStereoRT是一种适用于各种挑战性环境的实用解决方案，代码将开源。 Abstract: We introduce ThermoStereoRT, a real-time thermal stereo matching method designed for all-weather conditions that recovers disparity from two rectified thermal stereo images, envisioning applications such as night-time drone surveillance or under-bed cleaning robots. Leveraging a lightweight yet powerful backbone, ThermoStereoRT constructs a 3D cost volume from thermal images and employs multi-scale attention mechanisms to produce an initial disparity map. To refine this map, we design a novel channel and spatial attention module. Addressing the challenge of sparse ground truth data in thermal imagery, we utilize knowledge distillation to boost performance without increasing computational demands. Comprehensive evaluations on multiple datasets demonstrate that ThermoStereoRT delivers both real-time capacity and robust accuracy, making it a promising solution for real-world deployment in various challenging environments. Our code will be released on https://github.com/SJTU-ViSYS-team/ThermoStereoRT

AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery

Amirhossein Abaskohi,Amrutha Varshini Ramesh,Shailesh Nanisetty,Chirag Goel,David Vazquez,Christopher Pal,Spandana Gella,Giuseppe Carenini,Issam H. Laradji

Task: 介绍并评估AgentAda，一种能够学习和使用新分析技能的LLM驱动的分析代理。

Motivation: 现有方法需要用户手动选择数据分析方法，而AgentAda能自动从技能库中选择合适的技能，处理复杂任务。

Details

Method: AgentAda采用三步策略：问题生成器、混合RAG技能匹配器和代码生成器，结合技能库中的方法（如聚类、预测建模和NLP技术）。 Result: 在人类评估中，48.78%的评估者更偏好AgentAda的分析结果，优于未熟练代理的27.67%。 Conclusion: AgentAda通过自动化技能选择和代码生成，显著提升了数据分析的洞察力，并提出了LLM作为评估者的新方法。 Abstract: We introduce AgentAda, the first LLM-powered analytics agent that can learn and use new analytics skills to extract more specialized insights. Unlike existing methods that require users to manually decide which data analytics method to apply, AgentAda automatically identifies the skill needed from a library of analytical skills to perform the analysis. This also allows AgentAda to use skills that existing LLMs cannot perform out of the box. The library covers a range of methods, including clustering, predictive modeling, and NLP techniques like BERT, which allow AgentAda to handle complex analytics tasks based on what the user needs. AgentAda's dataset-to-insight extraction strategy consists of three key steps: (I) a question generator to generate queries relevant to the user's goal and persona, (II) a hybrid Retrieval-Augmented Generation (RAG)-based skill matcher to choose the best data analytics skill from the skill library, and (III) a code generator that produces executable code based on the retrieved skill's documentation to extract key patterns. We also introduce KaggleBench, a benchmark of curated notebooks across diverse domains, to evaluate AgentAda's performance. We conducted a human evaluation demonstrating that AgentAda provides more insightful analytics than existing tools, with 48.78% of evaluators preferring its analyses, compared to 27.67% for the unskilled agent. We also propose a novel LLM-as-a-judge approach that we show is aligned with human evaluation as a way to automate insight quality evaluation at larger scale.

WS-DETR: Robust Water Surface Object Detection through Vision-Radar Fusion with Detection Transformer

Huilin Yin,Pengyu Wang,Senmao Li,Jun Yan,Daniel Watzenig

Task: 提出一种鲁棒的视觉-雷达融合模型WS-DETR，用于复杂水域环境中的无人水面艇（USV）目标检测。

Motivation: 复杂水域环境中的目标检测面临边缘模糊和物体尺度多样化的挑战，现有视觉-雷达融合方法存在跨模态特征冲突问题，影响模型鲁棒性。

Details

Method: 引入多尺度边缘信息集成模块（MSEII）增强边缘感知，采用层次特征聚合器（HiFA）提升多尺度目标检测，利用自移动点表示进行连续卷积和残差连接提取不规则特征，并通过自适应特征交互融合模块（AFIF）实现几何对齐和语义融合。 Result: 在WaterScenes数据集上的实验表明，WS-DETR实现了最先进的性能，在恶劣天气和光照条件下仍保持优势。 Conclusion: WS-DETR通过多模块协同有效解决了跨模态特征冲突问题，提升了复杂水域环境中的目标检测鲁棒性。 Abstract: Robust object detection for Unmanned Surface Vehicles (USVs) in complex water environments is essential for reliable navigation and operation. Specifically, water surface object detection faces challenges from blurred edges and diverse object scales. Although vision-radar fusion offers a feasible solution, existing approaches suffer from cross-modal feature conflicts, which negatively affect model robustness. To address this problem, we propose a robust vision-radar fusion model WS-DETR. In particular, we first introduce a Multi-Scale Edge Information Integration (MSEII) module to enhance edge perception and a Hierarchical Feature Aggregator (HiFA) to boost multi-scale object detection in the encoder. Then, we adopt self-moving point representations for continuous convolution and residual connection to efficiently extract irregular features under the scenarios of irregular point cloud data. To further mitigate cross-modal conflicts, an Adaptive Feature Interactive Fusion (AFIF) module is introduced to integrate visual and radar features through geometric alignment and semantic fusion. Extensive experiments on the WaterScenes dataset demonstrate that WS-DETR achieves state-of-the-art (SOTA) performance, maintaining its superiority even under adverse weather and lighting conditions.

From Token to Line: Enhancing Code Generation with a Long-Term Perspective

Tingwei Lu,Yangning Li,Liyuan Wang,Binghuai Lin,Jiwei Tang,Wanshi Xu,Hai-Tao Zheng,Yinghui Li,Bingxu An,Zhao Wei,Yong Xu

Task: 提出一种基于MCTS的LSR-MCTS算法，用于逐行生成代码并优化路径选择。

Motivation: 现有代码生成研究存在冗余结果和局部过拟合问题，且缺乏对生成处理长度的关注。

Details

Method: 通过分析LLM生成过程中的注意力分布，提出以代码行为基础处理单元，结合MCTS和自优化机制逐行生成代码。 Result: 在三个公共编码基准测试中，LSR-MCTS算法优于现有最优方法。 Conclusion: LSR-MCTS算法通过逐行生成和自优化机制，显著提升了代码生成的质量和多样性。 Abstract: The emergence of large language models (LLMs) has significantly promoted the development of code generation task, sparking a surge in pertinent literature. Current research is hindered by redundant generation results and a tendency to overfit local patterns in the short term. Although existing studies attempt to alleviate the issue by adopting a multi-token prediction strategy, there remains limited focus on choosing the appropriate processing length for generations. By analyzing the attention between tokens during the generation process of LLMs, it can be observed that the high spikes of the attention scores typically appear at the end of lines. This insight suggests that it is reasonable to treat each line of code as a fundamental processing unit and generate them sequentially. Inspired by this, we propose the \textbf{LSR-MCTS} algorithm, which leverages MCTS to determine the code line-by-line and select the optimal path. Further, we integrate a self-refine mechanism at each node to enhance diversity and generate higher-quality programs through error correction. Extensive experiments and comprehensive analyses on three public coding benchmarks demonstrate that our method outperforms the state-of-the-art performance approaches.

How Can Objects Help Video-Language Understanding?

Zitian Tang,Shijie Wang,Junho Cho,Jaewook Yoo,Chen Sun

Task: 探索多模态大语言模型（MLLMs）中对象表示如何帮助视频语言理解。

Motivation: 理解MLLMs如何感知视觉世界，并明确对象表示在视频理解中的作用。

Details

Method: 从对象表示和适应的角度研究表达力与集成难度之间的权衡，并通过五个视频问答数据集进行评估。 Result: 明确的对象中心表示仍然是必要的，符号化对象最容易集成且在问答中表现良好。 Conclusion: 鼓励社区探索将感知模块显式集成到MLLM设计中。 Abstract: How multimodal large language models (MLLMs) perceive the visual world remains a mystery. To one extreme, object and relation modeling may be implicitly implemented with inductive biases, for example by treating objects as tokens. To the other extreme, empirical results reveal the surprising finding that simply performing visual captioning, which tends to ignore spatial configuration of the objects, serves as a strong baseline for video understanding. We aim to answer the question: how can objects help video-language understanding in MLLMs? We tackle the question from the object representation and adaptation perspectives. Specifically, we investigate the trade-off between representation expressiveness (e.g., distributed versus symbolic) and integration difficulty (e.g., data-efficiency when learning the adapters). Through extensive evaluations on five video question answering datasets, we confirm that explicit integration of object-centric representation remains necessary, and the symbolic objects can be most easily integrated while being performant for question answering. We hope our findings can encourage the community to explore the explicit integration of perception modules into MLLM design. Our code and models will be publicly released.

Revisiting LLM Evaluation through Mechanism Interpretability: a New Metric and Model Utility Law

Yixin Cao,Jiahao Ying,Yaoning Wang,Xipeng Qiu,Xuanjing Huang,Yugang Jiang

Task: 提出一种新的评估指标（MUI）以补充传统性能指标，用于评估大型语言模型（LLMs）的能力。

Motivation: 当前评估方法难以跟上大型语言模型的快速发展，需要更全面的评估方式。

Details

Method: 引入机制可解释性技术，提出模型利用率指数（MUI），量化模型完成任务时利用其能力的程度。 Result: 实验发现MUI与性能呈反比关系，并总结出“效用定律”及其四个推论。 Conclusion: MUI和效用定律有望推动评估方法与机制可解释性的共同进步。 Abstract: Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications, yet current evaluation methods struggle to keep pace with their rapid development. In this paper, we analyze the core limitations of traditional evaluation pipelines and propose a novel metric, the Model Utilization Index (MUI), which introduces mechanism interpretability techniques to complement traditional performance metrics. MUI quantifies the extent to which a model leverages its capabilities to complete tasks. The core idea is that to assess an LLM's overall ability, we must evaluate not only its task performance but also the effort expended to achieve the outcome. Our extensive experiments reveal an inverse relationship between MUI and performance, from which we deduce a common trend observed in popular LLMs, which we term the Utility Law. Based on this, we derive four corollaries that address key challenges, including training judgement, the issue of data contamination, fairness in model comparison, and data diversity. We hope that our survey, novel metric, and utility law will foster mutual advancement in both evaluation and mechanism interpretability. Our code can be found at https://github.com/ALEX-nlp/MUI-Eva.

Learning Universal Features for Generalizable Image Forgery Localization

Hengrun Zhao,Yunzhi Zhuge,Yifan Wang,Lijun Wang,Huchuan Lu,Yu Zeng

Task: 提出一种通用图像伪造定位方法（GIFL），用于检测和定位已知和未知的图像伪造内容。

Motivation: 现有方法依赖识别伪造痕迹，难以处理未知伪造类型，需更通用且高效的解决方案。

Details

Method: 通过学习原始内容的通用特征，而非特定伪造痕迹，以定位未知伪造。 Result: 在未知伪造检测上优于现有方法，同时在已知伪造检测上表现竞争性。 Conclusion: GIFL方法为生成式AI时代的虚假信息对抗提供了实用且高效的解决方案。 Abstract: In recent years, advanced image editing and generation methods have rapidly evolved, making detecting and locating forged image content increasingly challenging. Most existing image forgery detection methods rely on identifying the edited traces left in the image. However, because the traces of different forgeries are distinct, these methods can identify familiar forgeries included in the training data but struggle to handle unseen ones. In response, we present an approach for Generalizable Image Forgery Localization (GIFL). Once trained, our model can detect both seen and unseen forgeries, providing a more practical and efficient solution to counter false information in the era of generative AI. Our method focuses on learning general features from the pristine content rather than traces of specific forgeries, which are relatively consistent across different types of forgeries and therefore can be used as universal features to locate unseen forgeries. Additionally, as existing image forgery datasets are still dominated by traditional hand-crafted forgeries, we construct a new dataset consisting of images edited by various popular deep generative image editing methods to further encourage research in detecting images manipulated by deep generative models. Extensive experimental results show that the proposed approach outperforms state-of-the-art methods in the detection of unseen forgeries and also demonstrates competitive results for seen forgeries. The code and dataset are available at https://github.com/ZhaoHengrun/GIFL.

Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts

Zehan Li,Ruhua Pan,Xinyu Pi

Task: 提出一种从叙事文本生成因果图的新框架。

Motivation: 连接高层因果关系与具体事件关系，解决现有方法在因果链接识别上的不足。

Details

Method: 结合LLM摘要、专家索引（七个语言特征）和STAC分类模型，采用RoBERTa嵌入与专家索引的混合系统，并通过五轮提示过程优化因果图。 Result: 在100个叙事章节和短故事上的实验表明，该方法在因果图质量上优于GPT-4o和Claude 3.5，同时保持可读性。 Conclusion: 开源工具提供了一种高效、可解释的解决方案，用于捕捉叙事中的复杂因果链。 Abstract: We propose a novel framework for generating causal graphs from narrative texts, bridging high-level causality and detailed event-specific relationships. Our method first extracts concise, agent-centered vertices using large language model (LLM)-based summarization. We introduce an "Expert Index," comprising seven linguistically informed features, integrated into a Situation-Task-Action-Consequence (STAC) classification model. This hybrid system, combining RoBERTa embeddings with the Expert Index, achieves superior precision in causal link identification compared to pure LLM-based approaches. Finally, a structured five-iteration prompting process refines and constructs connected causal graphs. Experiments on 100 narrative chapters and short stories demonstrate that our approach consistently outperforms GPT-4o and Claude 3.5 in causal graph quality, while maintaining readability. The open-source tool provides an interpretable, efficient solution for capturing nuanced causal chains in narratives.

CMEdataset Advancing China Map Detection and Standardization with Digital Image Resources

Yan Xu,Zhenqiang Zhang,Zhiwei Zhou,Liting Geng,Yue Li,Jintao Li

Task: 创建一个专门用于问题地图检测的数据集（CME数据集）。

Motivation: 现有数据集主要关注普通地图数据，无法有效识别复杂问题（如国界错误表示、缺失元素和模糊边界），而问题地图对国家主权、领土完整和地图合规性至关重要。

Details

Method: 研究创建了一个涵盖五个关键问题领域的问题地图数据集。 Result: 该数据集为问题地图检测技术提供了多样化的样本，支持高精度地图合规性检测，并提升了地图数据的质量和时效性。 Conclusion: 该数据集不仅为地图合规性、国家安全监测和地图更新提供了重要资源，还促进了相关技术的创新和应用。 Abstract: Digital images of Chinas maps play a crucial role in map detection, particularly in ensuring national sovereignty, territorial integrity, and map compliance. However, there is currently no publicly available dataset specifically dedicated to problematic maps the CME dataset. Existing datasets primarily focus on general map data and are insufficient for effectively identifying complex issues such as national boundary misrepresentations, missing elements, and blurred boundaries. Therefore, this study creates a Problematic Map dataset that covers five key problem areas, aiming to provide diverse samples for problematic map detection technologies, support high-precision map compliance detection, and enhance map data quality and timeliness. This dataset not only provides essential resources for map compliance, national security monitoring, and map updates, but also fosters innovation and application of related technologies.

Defense against Prompt Injection Attacks via Mixture of Encodings

Ruiyi Zhang,David Sullivan,Kyle Jackson,Pengtao Xie,Mei Chen

Task: 提出一种新的防御机制（混合编码）来应对大型语言模型（LLMs）中的提示注入攻击。

Motivation: 现有的Base64防御方法虽然有效，但会降低LLM在某些NLP任务上的性能。

Details

Method: 采用多种字符编码（包括Base64）的混合编码策略。 Result: 实验结果表明，该方法在保持高NLP任务性能的同时，显著降低了提示注入攻击的成功率。 Conclusion: 混合编码策略在安全性和任务性能方面均表现出色，优于现有的基于字符编码的防御方法。 Abstract: Large Language Models (LLMs) have emerged as a dominant approach for a wide range of NLP tasks, with their access to external information further enhancing their capabilities. However, this introduces new vulnerabilities, known as prompt injection attacks, where external content embeds malicious instructions that manipulate the LLM's output. Recently, the Base64 defense has been recognized as one of the most effective methods for reducing success rate of prompt injection attacks. Despite its efficacy, this method can degrade LLM performance on certain NLP tasks. To address this challenge, we propose a novel defense mechanism: mixture of encodings, which utilizes multiple character encodings, including Base64. Extensive experimental results show that our method achieves one of the lowest attack success rates under prompt injection attacks, while maintaining high performance across all NLP tasks, outperforming existing character encoding-based defense methods. This underscores the effectiveness of our mixture of encodings strategy for both safety and task performance metrics.

Kimi-VL Technical Report

Kimi Team,Angang Du,Bohong Yin,Bowei Xing,Bowen Qu,Bowen Wang,Cheng Chen,Chenlin Zhang,Chenzhuang Du,Chu Wei,Congcong Wang,Dehao Zhang,Dikang Du,Dongliang Wang,Enming Yuan,Enzhe Lu,Fang Li,Flood Sung,Guangda Wei,Guokun Lai,Han Zhu,Hao Ding,Hao Hu,Hao Yang,Hao Zhang,Haoning Wu,Haotian Yao,Haoyu Lu,Heng Wang,Hongcheng Gao,Huabin Zheng,Jiaming Li,Jianlin Su,Jianzhou Wang,Jiaqi Deng,Jiezhong Qiu,Jin Xie,Jinhong Wang,Jingyuan Liu,Junjie Yan,Kun Ouyang,Liang Chen,Lin Sui,Longhui Yu,Mengfan Dong,Mengnan Dong,Nuo Xu,Pengyu Cheng,Qizheng Gu,Runjie Zhou,Shaowei Liu,Sihan Cao,Tao Yu,Tianhui Song,Tongtong Bai,Wei Song,Weiran He,Weixiao Huang,Weixin Xu,Xiaokun Yuan,Xingcheng Yao,Xingzhe Wu,Xinxing Zu,Xinyu Zhou,Xinyuan Wang,Y. Charles,Yan Zhong,Yang Li,Yangyang Hu,Yanru Chen,Yejie Wang,Yibo Liu,Yibo Miao,Yidao Qin,Yimin Chen,Yiping Bao,Yiqin Wang,Yongsheng Kang,Yuanxin Liu,Yulun Du,Yuxin Wu,Yuzhi Wang,Yuzi Yan,Zaida Zhou,Zhaowei Li,Zhejun Jiang,Zheng Zhang,Zhilin Yang,Zhiqi Huang,Zihao Huang,Zijia Zhao,Ziwei Chen

Task: 开发一个高效的开源混合专家（MoE）视觉语言模型（VLM），名为Kimi-VL，具备先进的多模态推理、长上下文理解和强大的代理能力。

Motivation: 通过激活仅2.8B参数的语言解码器，实现高效的多模态任务处理，并在多个领域超越或匹配现有旗舰模型。

Details

Method: 采用混合专家（MoE）架构，结合长链思维监督微调（SFT）和强化学习（RL）开发Kimi-VL-Thinking变体。 Result: 在多项任务中表现优异，如多轮代理任务、图像视频理解、OCR等，并在长上下文处理和高分辨率视觉输入方面取得突破。 Conclusion: Kimi-VL及其变体为高效多模态思维模型设定了新标准，代码和模型已开源。 Abstract: We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2.8B parameters in its language decoder (Kimi-VL-A3B). Kimi-VL demonstrates strong performance across challenging domains: as a general-purpose VLM, Kimi-VL excels in multi-turn agent tasks (e.g., OSWorld), matching flagship models. Furthermore, it exhibits remarkable capabilities across diverse challenging vision language tasks, including college-level image and video comprehension, OCR, mathematical reasoning, and multi-image understanding. In comparative evaluations, it effectively competes with cutting-edge efficient VLMs such as GPT-4o-mini, Qwen2.5-VL-7B, and Gemma-3-12B-IT, while surpassing GPT-4o in several key domains. Kimi-VL also advances in processing long contexts and perceiving clearly. With a 128K extended context window, Kimi-VL can process diverse long inputs, achieving impressive scores of 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc. Its native-resolution vision encoder, MoonViT, further allows it to see and understand ultra-high-resolution visual inputs, achieving 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro, while maintaining lower computational cost for common tasks. Building upon Kimi-VL, we introduce an advanced long-thinking variant: Kimi-VL-Thinking. Developed through long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL), this model exhibits strong long-horizon reasoning capabilities. It achieves scores of 61.7 on MMMU, 36.8 on MathVision, and 71.3 on MathVista while maintaining the compact 2.8B activated LLM parameters, setting a new standard for efficient multimodal thinking models. Code and models are publicly accessible at https://github.com/MoonshotAI/Kimi-VL.

Transformer-Based Temporal Information Extraction and Application: A Review

Xin Su,Phillip Howard,Steven Bethard

Task: 系统总结和分析基于Transformer的时间信息提取（Temporal IE）的研究工作。

Motivation: 填补现有研究中缺乏对基于Transformer的时间信息提取方法的全面综述的空白。

Details

Method: 系统性地总结和分析相关文献。 Result: 提供了对基于Transformer的时间信息提取研究的全面概述，并指出了未来研究方向。 Conclusion: 该综述为未来研究提供了基础，并强调了潜在的研究方向。 Abstract: Temporal information extraction (IE) aims to extract structured temporal information from unstructured text, thereby uncovering the implicit timelines within. This technique is applied across domains such as healthcare, newswire, and intelligence analysis, aiding models in these areas to perform temporal reasoning and enabling human users to grasp the temporal structure of text. Transformer-based pre-trained language models have produced revolutionary advancements in natural language processing, demonstrating exceptional performance across a multitude of tasks. Despite the achievements garnered by Transformer-based approaches in temporal IE, there is a lack of comprehensive reviews on these endeavors. In this paper, we aim to bridge this gap by systematically summarizing and analyzing the body of work on temporal IE using Transformers while highlighting potential future research directions.

Event Signal Filtering via Probability Flux Estimation

Jinze Chen,Wei Zhai,Yang Cao,Bin Li,Zheng-Jun Zha

Task: 提出一种名为Event Density Flow Filter (EDFilter)的在线过滤框架，用于增强事件信号的保真度。

Motivation: 事件信号的固有随机性导致信号质量下降，需要一种方法来减少这种随机性并确保在不同采集条件下的一致性输出。

Details

Method: 通过扩散过程重新审视事件生成的理论基础，利用非参数核平滑从离散事件重建连续概率通量，并提出一种快速递归求解器。 Result: EDFilter在事件过滤、超分辨率和基于事件的斑点跟踪等任务中表现出色，显著提升了SLAM和视频重建等下游应用的性能。 Conclusion: EDFilter通过建模事件相关性并优化保真度，提供了一种高效且鲁棒的事件信号处理方法。 Abstract: Events offer a novel paradigm for capturing scene dynamics via asynchronous sensing, but their inherent randomness often leads to degraded signal quality. Event signal filtering is thus essential for enhancing fidelity by reducing this internal randomness and ensuring consistent outputs across diverse acquisition conditions. Unlike traditional time series that rely on fixed temporal sampling to capture steady-state behaviors, events encode transient dynamics through polarity and event intervals, making signal modeling significantly more complex. To address this, the theoretical foundation of event generation is revisited through the lens of diffusion processes. The state and process information within events is modeled as continuous probability flux at threshold boundaries of the underlying irradiance diffusion. Building on this insight, a generative, online filtering framework called Event Density Flow Filter (EDFilter) is introduced. EDFilter estimates event correlation by reconstructing the continuous probability flux from discrete events using nonparametric kernel smoothing, and then resamples filtered events from this flux. To optimize fidelity over time, spatial and temporal kernels are employed in a time-varying optimization framework. A fast recursive solver with O(1) complexity is proposed, leveraging state-space models and lookup tables for efficient likelihood computation. Furthermore, a new real-world benchmark Rotary Event Dataset (RED) is released, offering microsecond-level ground truth irradiance for full-reference event filtering evaluation. Extensive experiments validate EDFilter's performance across tasks like event filtering, super-resolution, and direct event-based blob tracking. Significant gains in downstream applications such as SLAM and video reconstruction underscore its robustness and effectiveness.

Geological Inference from Textual Data using Word Embeddings

Nanmanas Linphrachaya,Irving Gómez-Méndez,Adil Siripatana

Task: 利用自然语言处理（NLP）技术定位工业矿物资源。

Motivation: 通过语义关系提取和降维技术，提高地质资源定位的准确性。

Details

Method: 使用GloVe模型训练词嵌入，提取目标关键词与地质文本的语义关系，并结合PCA、Autoencoder、VAE和VAE-LSTM等降维技术优化特征提取。 Result: 结果显示NLP与降维技术结合能有效揭示自然资源空间分布，但准确性仍有提升空间。 Conclusion: NLP与降维技术结合为地质资源定位提供了新思路，但需进一步优化以提高精度。 Abstract: This research explores the use of Natural Language Processing (NLP) techniques to locate geological resources, with a specific focus on industrial minerals. By using word embeddings trained with the GloVe model, we extract semantic relationships between target keywords and a corpus of geological texts. The text is filtered to retain only words with geographical significance, such as city names, which are then ranked by their cosine similarity to the target keyword. Dimensional reduction techniques, including Principal Component Analysis (PCA), Autoencoder, Variational Autoencoder (VAE), and VAE with Long Short-Term Memory (VAE-LSTM), are applied to enhance feature extraction and improve the accuracy of semantic relations. For benchmarking, we calculate the proximity between the ten cities most semantically related to the target keyword and identified mine locations using the haversine equation. The results demonstrate that combining NLP with dimensional reduction techniques provides meaningful insights into the spatial distribution of natural resources. Although the result shows to be in the same region as the supposed location, the accuracy has room for improvement.

VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding

Henghao Zhao,Ge-Peng Ji,Rui Yan,Huan Xiong,Zechao Li

Task: 提出一种名为VideoExpert的多模态大语言模型，用于解决时间敏感的视频任务中的时间戳生成问题。

Motivation: 现有的多模态大语言模型在生成时间戳时过于依赖语言模式而非视觉线索，导致性能不佳。

Details

Method: VideoExpert通过集成时间专家和空间专家两个并行模块，分别处理时间序列和内容细节，并通过特殊令牌实现协作。 Result: 实验证明VideoExpert在时间敏感的视频任务中表现出色且具有通用性。 Conclusion: VideoExpert通过分离时间定位和内容生成，有效避免了文本模式偏差，提升了时间戳预测的准确性。 Abstract: The core challenge in video understanding lies in perceiving dynamic content changes over time. However, multimodal large language models struggle with temporal-sensitive video tasks, which requires generating timestamps to mark the occurrence of specific events. Existing strategies require MLLMs to generate absolute or relative timestamps directly. We have observed that those MLLMs tend to rely more on language patterns than visual cues when generating timestamps, affecting their performance. To address this problem, we propose VideoExpert, a general-purpose MLLM suitable for several temporal-sensitive video tasks. Inspired by the expert concept, VideoExpert integrates two parallel modules: the Temporal Expert and the Spatial Expert. The Temporal Expert is responsible for modeling time sequences and performing temporal grounding. It processes high-frame-rate yet compressed tokens to capture dynamic variations in videos and includes a lightweight prediction head for precise event localization. The Spatial Expert focuses on content detail analysis and instruction following. It handles specially designed spatial tokens and language input, aiming to generate content-related responses. These two experts collaborate seamlessly via a special token, ensuring coordinated temporal grounding and content generation. Notably, the Temporal and Spatial Experts maintain independent parameter sets. By offloading temporal grounding from content generation, VideoExpert prevents text pattern biases in timestamp predictions. Moreover, we introduce a Spatial Compress module to obtain spatial tokens. This module filters and compresses patch tokens while preserving key information, delivering compact yet detail-rich input for the Spatial Expert. Extensive experiments demonstrate the effectiveness and versatility of the VideoExpert.

Supervised Optimism Correction: Be Confident When LLMs Are Sure

Junjie Zhang,Rushuai Yang,Shunyu Liu,Ting-En Lin,Fei Huang,Yi Chen,Yongbin Li,Dacheng Tao

Task: 建立监督微调与离线强化学习在令牌级马尔可夫决策过程中的理论联系，揭示大语言模型在推理中学习隐式Q函数的现象。

Motivation: 发现广泛使用的束搜索方法因对次优步骤的Q值估计过高而导致推理错误放大，存在不可接受的过度乐观问题。

Details

Method: 提出监督乐观校正（SOC），通过在监督微调期间引入辅助损失来改进令牌级Q值估计，采用隐式值正则化增强模型对专家示范响应的信心。 Result: 在数学推理基准（GSM8K、MATH、GAOKAO）上的实验表明，SOC与束搜索结合在多个开源模型中表现优越。 Conclusion: SOC通过抑制对监督不足响应的过度乐观，有效改进了推理性能。 Abstract: In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the token-level Markov decision process, revealing that large language models indeed learn an implicit $Q$-function for inference. Through this theoretical lens, we demonstrate that the widely used beam search method suffers from unacceptable over-optimism, where inference errors are inevitably amplified due to inflated $Q$-value estimations of suboptimal steps. To address this limitation, we propose Supervised Optimism Correction(SOC), which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations during supervised fine-tuning. Specifically, the auxiliary loss employs implicit value regularization to boost model confidence in expert-demonstrated responses, thereby suppressing over-optimism toward insufficiently supervised responses. Extensive experiments on mathematical reasoning benchmarks, including GSM8K, MATH, and GAOKAO, showcase the superiority of the proposed SOC with beam search across a series of open-source models.

DGOcc: Depth-aware Global Query-based Network for Monocular 3D Occupancy Prediction

Xu Zhao,Pengju Zhang,Bo Liu,Yihong Wu

Task: 从单目2D图像预测3D场景中的占据和语义信息。

Motivation: 解决从2D图像预测大规模室外场景3D占据的困难性和资源密集性问题。

Details

Method: 提出DGOcc网络，利用深度上下文特征和全局查询模块，结合注意力机制和尺度感知操作，优化特征交互，并设计分层监督策略以减少计算资源消耗。 Result: 在SemanticKITTI和SSCBench-KITTI-360数据集上表现最佳，同时降低了GPU和时间开销。 Conclusion: DGOcc方法在单目3D占据预测中实现了高性能和高效性。 Abstract: Monocular 3D occupancy prediction, aiming to predict the occupancy and semantics within interesting regions of 3D scenes from only 2D images, has garnered increasing attention recently for its vital role in 3D scene understanding. Predicting the 3D occupancy of large-scale outdoor scenes from 2D images is ill-posed and resource-intensive. In this paper, we present \textbf{DGOcc}, a \textbf{D}epth-aware \textbf{G}lobal query-based network for monocular 3D \textbf{Occ}upancy prediction. We first explore prior depth maps to extract depth context features that provide explicit geometric information for the occupancy network. Then, in order to fully exploit the depth context features, we propose a Global Query-based (GQ) Module. The cooperation of attention mechanisms and scale-aware operations facilitates the feature interaction between images and 3D voxels. Moreover, a Hierarchical Supervision Strategy (HSS) is designed to avoid upsampling the high-dimension 3D voxel features to full resolution, which mitigates GPU memory utilization and time cost. Extensive experiments on SemanticKITTI and SSCBench-KITTI-360 datasets demonstrate that the proposed method achieves the best performance on monocular semantic occupancy prediction while reducing GPU and time overhead.

AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation

Tuhin Chakrabarty,Philippe Laban,Chien-Sheng Wu

Task: 评估和改进AI生成文本的写作质量。

Motivation: AI生成文本的写作质量评估是一个基础但被忽视的问题，因其主观性和需要专业知识。

Details

Method: 引入写作质量基准（WQ），并训练专门的写作质量奖励模型（WQRM）进行评估和改进。 Result: WQRM在四个分布外测试集上表现良好，WQ基准准确率达74%，人类评估显示专家偏好WQRM生成的文本。 Conclusion: 通过WQRM和额外计算资源，可以显著提升AI生成文本的写作质量，并鼓励社区进一步研究。 Abstract: AI-generated text is proliferating across domains, from creative writing and journalism to marketing content and scientific articles. Models can follow user-provided instructions to generate coherent and grammatically correct outputs but in this work, we study a more fundamental question: how do we evaluate and improve the writing quality of AI-generated text? Writing quality assessment has received less attention from the community, in part because it is fundamentally subjective and requires expertise. We first introduce the Writing Quality Benchmark (WQ) by consolidating five writing-preference datasets into 4,729 writing quality judgments. Our experiments show that competitive baselines, including state-of-the-art LLMs that excel at reasoning tasks, barely outperform random baselines on WQ. We then train specialized Writing Quality Reward Models (WQRM) of various sizes for writing quality assessment that demonstrate strong generalization on four out-of-distribution test sets and 74% accuracy on the WQ benchmark. To further show WQRM's practical benefits during inference, we leverage additional test-time compute to generate and rank multiple candidate revisions, allowing us to select higher-quality outputs from an initial draft. Human evaluation with 9 experienced writers confirm that WQRM-based selection produces writing samples preferred by experts 66% overall, and 72.2% when the reward gap is larger than 1 point. We release our datasets and models to encourage community engagement with writing quality assessment and development of AI writing systems better aligned with human preferences.

SydneyScapes: Image Segmentation for Australian Environments

Hongyu Lyu,Julie Stephany Berrio,Mao Shan,Stewart Worrall

Task: Introduce SydneyScapes, a dataset for computer vision tasks in AV perception systems tailored for the Australian context.

Motivation: Address the lack of locally labelled datasets for AV algorithm development and testing in Australia.

Details

Method: Collect and annotate 756 images from Sydney and surrounding areas in NSW, Australia, for semantic, instance, and panoptic segmentation tasks. Result: Provide a publicly available dataset with high-quality pixel-level annotations and benchmarking results using state-of-the-art algorithms. Conclusion: SydneyScapes supports AV industry and researchers by offering a localized dataset and tools for algorithm development and testing in Australia. Abstract: Autonomous Vehicles (AVs) are being partially deployed and tested across various global locations, including China, the USA, Germany, France, Japan, Korea, and the UK, but with limited demonstrations in Australia. The integration of machine learning (ML) into AV perception systems highlights the need for locally labelled datasets to develop and test algorithms in specific environments. To address this, we introduce SydneyScapes - a dataset tailored for computer vision tasks of image semantic, instance, and panoptic segmentation. This dataset, collected from Sydney and surrounding cities in New South Wales (NSW), Australia, consists of 756 images with high-quality pixel-level annotations. It is designed to assist AV industry and researchers by providing annotated data and tools for algorithm development, testing, and deployment in the Australian context. Additionally, we offer benchmarking results using state-of-the-art algorithms to establish reference points for future research and development. The dataset is publicly available at https://hdl.handle.net/2123/33051.

Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering

Patrick Fernandes,Sweta Agrawal,Emmanouil Zaranis,André F. T. Martins,Graham Neubig

Task: 提出一种基于问答的翻译评估框架TREQA，用于评估段落级翻译的质量。

Motivation: 现有自动评估指标难以捕捉跨句子的意义保留，尤其是在长复杂文本中。

Details

Method: 通过问答形式评估翻译是否准确传达了原文或参考文本中的关键信息。 Result: TREQA在需要长距离理解的领域（如文学文本）中表现优异，甚至优于现有神经和LLM-based指标。 Conclusion: TREQA不仅提供高质量的翻译评估，还通过生成的问题和答案增强了可解释性。 Abstract: Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality score, trained to mimic human judgments, might be insufficient for evaluating translations of long, complex passages, and a more ``pragmatic'' approach that assesses how accurately key information is conveyed by a translation in context is needed. We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality by assessing how accurately candidate translations answer reading comprehension questions that target key information in the original source or reference texts. In challenging domains that require long-range understanding, such as literary texts, we show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations, despite never being explicitly optimized to correlate with human judgments. Furthermore, the generated questions and answers offer interpretability: empirical analysis shows that they effectively target translation errors identified by experts in evaluated datasets. Our code is available at https://github.com/deep-spin/treqa

STeP: A General and Scalable Framework for Solving Video Inverse Problems with Spatiotemporal Diffusion Priors

Bingliang Zhang,Zihui Wu,Berthy T. Feng,Yang Song,Yisong Yue,Katherine L. Bouman

Task: 研究如何利用扩散模型先验解决涉及视频的一般贝叶斯逆问题。

Motivation: 现有方法依赖图像扩散先验和启发式方法，难以准确恢复时间关系，特别是在高时间不确定性的任务中。

Details

Method: 通过微调预训练图像扩散模型的潜在视频扩散模型，引入即插即用的时空扩散先验，并提出一个通用的视频逆问题解决框架。 Result: 在黑洞成像和动态MRI等科学视频逆问题中，生成了多样且高保真的视频重建结果，能够恢复多模态解。 Conclusion: 时空扩散先验显著提升了捕捉复杂时间关系的能力，同时增强了空间保真度。 Abstract: We study how to solve general Bayesian inverse problems involving videos using diffusion model priors. While it is desirable to use a video diffusion prior to effectively capture complex temporal relationships, due to the computational and data requirements of training such a model, prior work has instead relied on image diffusion priors on single frames combined with heuristics to enforce temporal consistency. However, these approaches struggle with faithfully recovering the underlying temporal relationships, particularly for tasks with high temporal uncertainty. In this paper, we demonstrate the feasibility of practical and accessible spatiotemporal diffusion priors by fine-tuning latent video diffusion models from pretrained image diffusion models using limited videos in specific domains. Leveraging this plug-and-play spatiotemporal diffusion prior, we introduce a general and scalable framework for solving video inverse problems. We then apply our framework to two challenging scientific video inverse problems--black hole imaging and dynamic MRI. Our framework enables the generation of diverse, high-fidelity video reconstructions that not only fit observations but also recover multi-modal solutions. By incorporating a spatiotemporal diffusion prior, we significantly improve our ability to capture complex temporal relationships in the data while also enhancing spatial fidelity.

SaRoHead: A Dataset for Satire Detection in Romanian Multi-Domain News Headlines

Mihnea-Alexandru Vîrlan,Răzvan-Alexandru Smădu,Dumitru-Clementin Cercel

Task: 构建首个罗马尼亚多领域新闻标题的讽刺检测语料库SaRoHead。

Motivation: 新闻标题的表达方式和与主题的联系对讽刺检测具有挑战性，尤其是当标题中混合了讽刺、反讽和挖苦等风格元素时。

Details

Method: 提出SaRoHead语料库，用于检测罗马尼亚多领域新闻标题中的讽刺内容。 Result: 研究发现，非讽刺标题中的点击诱饵显著影响模型性能。 Conclusion: SaRoHead为罗马尼亚新闻标题的讽刺检测提供了首个语料库，并揭示了点击诱饵对检测模型的影响。 Abstract: The headline is an important part of a news article, influenced by expressiveness and connection to the exposed subject. Although most news outlets aim to present reality objectively, some publications prefer a humorous approach in which stylistic elements of satire, irony, and sarcasm blend to cover specific topics. Satire detection can be difficult because a headline aims to expose the main idea behind a news article. In this paper, we propose SaRoHead, the first corpus for satire detection in Romanian multi-domain news headlines. Our findings show that the clickbait used in some non-satirical headlines significantly influences the model.

TokenFocus-VQA: Enhancing Text-to-Image Alignment with Position-Aware Focus and Multi-Perspective Aggregations on LVLMs

Zijian Zhang,Xuhui Zheng,Xuecheng Wu,Chong Peng,Xuezhi Cao

Task: 提出了一种名为TokenFocus-VQA的新评估框架，用于细粒度语义匹配的文本到图像生成模型评估。

Motivation: 现有的基于全局相似性度量的评估方法忽视了文本描述与视觉内容之间的关键标记级对应关系。

Details

Method: 利用大型视觉语言模型（LVLMs）通过视觉问答（VQA）范式，设计了一种标记感知损失函数，专注于预定义词汇位置的概率分布。 Result: 在NTIRE 2025 T2I质量评估挑战赛中，TokenFocus-VQA在公共评估和官方私有测试集上均排名第二，表现出优于传统评估方法的性能。 Conclusion: TokenFocus-VQA能够更精确地捕捉文本与图像之间的细粒度语义对齐，展示了其在评估中的优越性。 Abstract: While text-to-image (T2I) generation models have achieved remarkable progress in recent years, existing evaluation methodologies for vision-language alignment still struggle with the fine-grained semantic matching. Current approaches based on global similarity metrics often overlook critical token-level correspondences between textual descriptions and visual content. To this end, we present TokenFocus-VQA, a novel evaluation framework that leverages Large Vision-Language Models (LVLMs) through visual question answering (VQA) paradigm with position-specific probability optimization. Our key innovation lies in designing a token-aware loss function that selectively focuses on probability distributions at pre-defined vocabulary positions corresponding to crucial semantic elements, enabling precise measurement of fine-grained semantical alignment. The proposed framework further integrates ensemble learning techniques to aggregate multi-perspective assessments from diverse LVLMs architectures, thereby achieving further performance enhancement. Evaluated on the NTIRE 2025 T2I Quality Assessment Challenge Track 1, our TokenFocus-VQA ranks 2nd place (0.8445, only 0.0001 lower than the 1st method) on public evaluation and 2nd place (0.8426) on the official private test set, demonstrating superiority in capturing nuanced text-image correspondences compared to conventional evaluation methods.

ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Models

Joel Barmettler,Abraham Bernstein,Luca Rossetto

Task: 提出一种名为ConceptFormer的新方法，通过将知识图谱中的结构化知识直接注入到大型语言模型的嵌入向量空间中，以增强其事实召回能力。

Motivation: 当前基于检索增强生成（RAG）的方法通常需要修改预训练语言模型的内部结构或将知识图谱文本化，这在令牌使用上效率低下。

Details

Method: ConceptFormer在LLM的嵌入向量空间中创建和注入“概念向量”，直接封装知识图谱节点的信息，并通过与冻结的LLM联合训练生成一个查找表。 Result: 实验表明，ConceptFormer显著提升了GPT-2 0.1B的事实召回能力（Hit@10），在Wikipedia句子上最高提升272%，在合成句子上最高提升348%，且令牌使用效率更高。 Conclusion: ConceptFormer是一种高效且可扩展的方法，能够在不改变LLM内部结构的情况下，为其注入结构化知识，显著提升其性能。 Abstract: Retrieval Augmented Generation (RAG) has enjoyed increased attention in the recent past and recent advancements in Large Language Models (LLMs) have highlighted the importance of integrating world knowledge into these systems. Current RAG methodologies often modify the internal architecture of pre-trained language models (PLMs) or rely on textifying knowledge graphs (KGs), which is inefficient in terms of token usage. This paper introduces ConceptFormer, a new approach to augment LLMs with structured knowledge from KGs, such as Wikidata, without altering their internal structure or relying on textual input of KGs. ConceptFormer operates in the LLM embedding vector space, creating and injecting \emph{concept vectors} that encapsulate the information of the KG nodes directly. Trained in conjunction with a frozen LLM, ConceptFormer generates a comprehensive lookup table that maps KG nodes to their respective concept vectors. The approach aims to enhance the factual recall capabilities of LLMs by enabling them to process these concept vectors natively, thus enriching them with structured world knowledge in an efficient and scalable manner. Our experiments demonstrate that the addition of concept vectors to GPT-2 0.1B substantially increases its factual recall ability (Hit@10) by up to 272\% when tested on sentences from Wikipedia and up to 348\% on synthetically generated sentences. Even injecting only a single concept vector into the prompt increases factual recall ability (Hit@10) by up to 213\% on Wikipedia sentences, significantly outperforming RAG with graph textification while consuming 130x fewer input tokens.

Benchmarking Image Embeddings for E-Commerce: Evaluating Off-the Shelf Foundation Models, Fine-Tuning Strategies and Practical Trade-offs

Urszula Czerwinska,Cenk Bircanoglu,Jeremy Chamoux

Task: 评估基础模型图像嵌入在电子商务分类和检索中的性能，并探讨其在现实应用中的适用性。

Motivation: 研究不同预训练模型和训练方法在电子商务领域的表现，为实际应用提供指导。

Details

Method: 比较了预训练的卷积和Transformer模型通过监督、自监督和文本-图像对比学习生成的嵌入，并在六个电子商务数据集上进行全面微调和迁移学习（top-tuning）评估。 Result: 全面微调表现稳定，而文本-图像和自监督嵌入在较少训练下也能达到类似性能；top-tuning是一种高效替代方案，降低计算成本。 Conclusion: 研究结果为嵌入选择和微调策略提供了实用指南，平衡了效率和性能。 Abstract: We benchmark foundation models image embeddings for classification and retrieval in e-Commerce, evaluating their suitability for real-world applications. Our study spans embeddings from pre-trained convolutional and transformer models trained via supervised, self-supervised, and text-image contrastive learning. We assess full fine-tuning and transfer learning (top-tuning) on six diverse e-Commerce datasets: fashion, consumer goods, cars, food, and retail. Results show full fine-tuning consistently performs well, while text-image and self-supervised embeddings can match its performance with less training. While supervised embeddings remain stable across architectures, SSL and contrastive embeddings vary significantly, often benefiting from top-tuning. Top-tuning emerges as an efficient alternative to full fine-tuning, reducing computational costs. We also explore cross-tuning, noting its impact depends on dataset characteristics. Our findings offer practical guidelines for embedding selection and fine-tuning strategies, balancing efficiency and performance.

On the Temporal Question-Answering Capabilities of Large Language Models Over Anonymized Data

Alfredo Garrachón Ruiz,Tomás de la Rosa,Daniel Borrajo

Task: 探索大型语言模型（LLMs）在未训练数据上的时间推理任务中的适用性。

Motivation: 研究LLMs在结构化与半结构化匿名数据上的时间推理能力，填补该领域的空白。

Details

Method: 开发直接LLM流水线，比较多种方法（如Tree-of-Thought、自反思维和代码执行），并创建RATA数据集进行评估。 Result: 发现仅依赖LLMs难以实现可扩展且可靠的解决方案，需结合集成方法。 Conclusion: 强调集成方法在提升LLMs时间推理能力中的重要性。 Abstract: The applicability of Large Language Models (LLMs) in temporal reasoning tasks over data that is not present during training is still a field that remains to be explored. In this paper we work on this topic, focusing on structured and semi-structured anonymized data. We not only develop a direct LLM pipeline, but also compare various methodologies and conduct an in-depth analysis. We identified and examined seventeen common temporal reasoning tasks in natural language, focusing on their algorithmic components. To assess LLM performance, we created the \textit{Reasoning and Answering Temporal Ability} dataset (RATA), featuring semi-structured anonymized data to ensure reliance on reasoning rather than on prior knowledge. We compared several methodologies, involving SoTA techniques such as Tree-of-Thought, self-reflexion and code execution, tuned specifically for this scenario. Our results suggest that achieving scalable and reliable solutions requires more than just standalone LLMs, highlighting the need for integrated approaches.

On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognition

Adrian Cosma,Andy Cǎtrunǎ,Emilian Rǎdoi

Task: 研究基于骨架的自监督步态识别中数据量、模型规模和计算资源对性能的影响。

Motivation: 探索神经缩放定律在步态识别领域的适用性，填补现有研究的空白。

Details

Method: 使用基于Transformer的GaitPT架构，在270万野外采集的步行序列数据集上进行预训练，并通过零样本性能评估推导缩放定律。 Result: 发现性能随规模增加呈幂律提升，数据量和计算资源对下游准确性有显著影响。 Conclusion: 为实际步态识别系统的资源分配和性能估计提供了实用见解。 Abstract: Gait recognition from video streams is a challenging problem in computer vision biometrics due to the subtle differences between gaits and numerous confounding factors. Recent advancements in self-supervised pretraining have led to the development of robust gait recognition models that are invariant to walking covariates. While neural scaling laws have transformed model development in other domains by linking performance to data, model size, and compute, their applicability to gait remains unexplored. In this work, we conduct the first empirical study scaling on skeleton-based self-supervised gait recognition to quantify the effect of data quantity, model size and compute on downstream gait recognition performance. We pretrain multiple variants of GaitPT - a transformer-based architecture - on a dataset of 2.7 million walking sequences collected in the wild. We evaluate zero-shot performance across four benchmark datasets to derive scaling laws for data, model size, and compute. Our findings demonstrate predictable power-law improvements in performance with increased scale and confirm that data and compute scaling significantly influence downstream accuracy. We further isolate architectural contributions by comparing GaitPT with GaitFormer under controlled compute budgets. These results provide practical insights into resource allocation and performance estimation for real-world gait recognition systems.

Unveiling the Impact of Multimodal Features on Chinese Spelling Correction: From Analysis to Design

Xiaowu Zhang,Hongfei Zhao,Jingyi Hou,Zhijie Liu

Task: 提出一种名为NamBert的多模态模型，用于中文拼写纠正任务。

Motivation: 现有的大型语言模型（LLMs）在中文拼写纠正中存在过校正问题，而多模态模型在利用语音和字形信息方面仍有提升空间。

Details

Method: 通过多模态分析实验（MACU）识别改进点，并设计NamBert模型。 Result: 实验证明NamBert在基准数据集上优于现有最优方法。 Conclusion: NamBert在多模态中文拼写纠正中表现优异，并系统评估了其与LLMs的优劣。 Abstract: The Chinese Spelling Correction (CSC) task focuses on detecting and correcting spelling errors in sentences. Current research primarily explores two approaches: traditional multimodal pre-trained models and large language models (LLMs). However, LLMs face limitations in CSC, particularly over-correction, making them suboptimal for this task. While existing studies have investigated the use of phonetic and graphemic information in multimodal CSC models, effectively leveraging these features to enhance correction performance remains a challenge. To address this, we propose the Multimodal Analysis for Character Usage (\textbf{MACU}) experiment, identifying potential improvements for multimodal correctison. Based on empirical findings, we introduce \textbf{NamBert}, a novel multimodal model for Chinese spelling correction. Experiments on benchmark datasets demonstrate NamBert's superiority over SOTA methods. We also conduct a comprehensive comparison between NamBert and LLMs, systematically evaluating their strengths and limitations in CSC. Our code and model are available at https://github.com/iioSnail/NamBert.

RASMD: RGB And SWIR Multispectral Driving Dataset for Robust Perception in Adverse Conditions

Youngwan Jin,Michal Kovac,Yagiz Nalcakan,Hyeongjin Ju,Hanbin Song,Sanghyeop Yeo,Shiho Kim

Task: Introduce the RGB and SWIR Multispectral Driving (RASMD) dataset to address the lack of large-scale SWIR datasets for autonomous driving.

Motivation: Current autonomous driving algorithms rely on visible spectrum data, which performs poorly in adverse conditions, while other spectral bands like SWIR offer advantages but lack datasets.

Details

Method: Collect and provide 100,000 synchronized RGB-SWIR image pairs across diverse conditions, along with annotations for object detection and RGB-to-SWIR translation tasks. Result: Combining RGB and SWIR data in an ensemble framework improves detection accuracy, especially in challenging conditions. Conclusion: The RASMD dataset is expected to advance research in multispectral imaging for autonomous driving and robust perception systems. Abstract: Current autonomous driving algorithms heavily rely on the visible spectrum, which is prone to performance degradation in adverse conditions like fog, rain, snow, glare, and high contrast. Although other spectral bands like near-infrared (NIR) and long-wave infrared (LWIR) can enhance vision perception in such situations, they have limitations and lack large-scale datasets and benchmarks. Short-wave infrared (SWIR) imaging offers several advantages over NIR and LWIR. However, no publicly available large-scale datasets currently incorporate SWIR data for autonomous driving. To address this gap, we introduce the RGB and SWIR Multispectral Driving (RASMD) dataset, which comprises 100,000 synchronized and spatially aligned RGB-SWIR image pairs collected across diverse locations, lighting, and weather conditions. In addition, we provide a subset for RGB-SWIR translation and object detection annotations for a subset of challenging traffic scenarios to demonstrate the utility of SWIR imaging through experiments on both object detection and RGB-to-SWIR image translation. Our experiments show that combining RGB and SWIR data in an ensemble framework significantly improves detection accuracy compared to RGB-only approaches, particularly in conditions where visible-spectrum sensors struggle. We anticipate that the RASMD dataset will advance research in multispectral imaging for autonomous driving and robust perception systems.

Synthetic Fluency: Hallucinations, Confabulations, and the Creation of Irish Words in LLM-Generated Translations

Sheila Castilho,Zoe Fitzsimmons,Claire Holton,Aoife Mc Donagh

Task: 研究大型语言模型（LLM）在爱尔兰语翻译中产生的幻觉现象，特别是生成不存在的词汇。

Motivation: 探讨LLM在低资源、形态丰富的语言（如爱尔兰语）中的表现及其对语言演变的潜在影响。

Details

Method: 对幻觉现象进行分类，分析其是否符合爱尔兰语形态规则及语言倾向。 Result: GPT-4.o和GPT-4.o Mini产生相似类型的幻觉，但Mini模型的频率显著更高。 Conclusion: 提出关于LLM对爱尔兰语词汇和语言演变潜在影响的思考，旨在引发讨论。 Abstract: This study examines hallucinations in Large Language Model (LLM) translations into Irish, specifically focusing on instances where the models generate novel, non-existent words. We classify these hallucinations within verb and noun categories, identifying six distinct patterns among the latter. Additionally, we analyse whether these hallucinations adhere to Irish morphological rules and what linguistic tendencies they exhibit. Our findings show that while both GPT-4.o and GPT-4.o Mini produce similar types of hallucinations, the Mini model generates them at a significantly higher frequency. Beyond classification, the discussion raises speculative questions about the implications of these hallucinations for the Irish language. Rather than seeking definitive answers, we offer food for thought regarding the increasing use of LLMs and their potential role in shaping Irish vocabulary and linguistic evolution. We aim to prompt discussion on how such technologies might influence language over time, particularly in the context of low-resource, morphologically rich languages.

VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model

Haozhan Shen,Peng Liu,Jingcheng Li,Chunxin Fang,Yibo Ma,Jiajia Liao,Qiaoli Shen,Zilun Zhang,Kangjia Zhao,Qianqian Zhang,Ruochen Xu,Tiancheng Zhao

Task: 研究如何将DeepSeek R1中的强化学习方法扩展到视觉语言模型（VLMs），以提升其视觉推理能力。

Motivation: 视觉理解任务通常具有明确的标注，适合基于规则的奖励机制，这为强化学习在视觉领域的应用提供了自然基础。

Details

Method: 开发了VLM-R1框架，利用强化学习优化VLMs在通用视觉语言任务中的表现。 Result: 实验表明，基于强化学习的模型在视觉理解任务中表现优异，且在泛化能力上超越了监督微调（SFT）。 Conclusion: 通过分析揭示了强化学习对视觉语言模型能力的提升机制，并开源了代码和模型以推动社区发展。 Abstract: Recently DeepSeek R1 has shown that reinforcement learning (RL) can substantially improve the reasoning capabilities of Large Language Models (LLMs) through a simple yet effective design. The core of R1 lies in its rule-based reward formulation, which leverages tasks with deterministic ground-truth answers to enable precise and stable reward computation. In the visual domain, we similarly observe that a wide range of visual understanding tasks are inherently equipped with well-defined ground-truth annotations. This property makes them naturally compatible with rule-based reward mechanisms. Motivated by this observation, we investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs), aiming to enhance their visual reasoning capabilities. To this end, we develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks. Using this framework, we further explore the feasibility of applying RL to visual domain. Experimental results indicate that the RL-based model not only delivers competitive performance on visual understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in generalization ability. Furthermore, we conduct comprehensive ablation studies that uncover a series of noteworthy insights, including the presence of reward hacking in object detection, the emergence of the "OD aha moment", the impact of training data quality, and the scaling behavior of RL across different model sizes. Through these analyses, we aim to deepen the understanding of how reinforcement learning enhances the capabilities of vision-language models, and we hope our findings and open-source contributions will support continued progress in the vision-language RL community. Our code and model are available at https://github.com/om-ai-lab/VLM-R1

Context-Aware Monolingual Human Evaluation of Machine Translation

Silvio Picinini,Sheila Castilho

Task: 探索无源文本参考下，基于上下文感知的单语人工评估在机器翻译评估中的潜力。

Motivation: 比较单语评估与双语评估（有源文本）在评估单个机器翻译系统和对比评估成对机器翻译系统时的表现。

Details

Method: 四位专业翻译人员分别进行单语和双语评估，包括评分、错误标注及提供反馈。 Result: 上下文感知的单语人工评估结果与双语评估相当，表明单语评估是一种高效评估机器翻译的可行方法。 Conclusion: 单语评估在机器翻译评估中具有可行性和潜力，可作为高效评估手段。 Abstract: This paper explores the potential of context-aware monolingual human evaluation for assessing machine translation (MT) when no source is given for reference. To this end, we compare monolingual with bilingual evaluations (with source text), under two scenarios: the evaluation of a single MT system, and the comparative evaluation of pairwise MT systems. Four professional translators performed both monolingual and bilingual evaluations by assigning ratings and annotating errors, and providing feedback on their experience. Our findings suggest that context-aware monolingual human evaluation achieves comparable outcomes to human bilingual evaluations, and suggest the feasibility and potential of monolingual evaluation as an efficient approach to assessing MT.

End-to-End Facial Expression Detection in Long Videos

Yini Fang,Alec Diallo,Yiqi Shi,Frederic Jumelle,Bertram Shi

Task: 提出一种端到端的面部表情检测网络（FEDN），联合优化表情定位和识别任务。

Motivation: 现有方法将表情定位和识别任务分开处理，导致错误传播、特征学习效率低和性能不佳。

Details

Method: 引入基于注意力的特征提取模块，结合片段注意力和滑动窗口注意力，联合优化两个任务。 Result: 在CASME^2和CASME^3数据集上实现了最先进的定位和检测精度。 Conclusion: 联合优化显著减少了错误传播，提升了长视频中面部表情检测的鲁棒性。 Abstract: Facial expression detection involves two interrelated tasks: spotting, which identifies the onset and offset of expressions, and recognition, which classifies them into emotional categories. Most existing methods treat these tasks separately using a two-step training pipelines. A spotting model first detects expression intervals. A recognition model then classifies the detected segments. However, this sequential approach leads to error propagation, inefficient feature learning, and suboptimal performance due to the lack of joint optimization of the two tasks. We propose FEDN, an end-to-end Facial Expression Detection Network that jointly optimizes spotting and recognition. Our model introduces a novel attention-based feature extraction module, incorporating segment attention and sliding window attention to improve facial feature learning. By unifying two tasks within a single network, we greatly reduce error propagation and enhance overall performance. Experiments on CASME}^2 and CASME^3 demonstrate state-of-the-art accuracy for both spotting and detection, underscoring the benefits of joint optimization for robust facial expression detection in long videos.

Proactive User Information Acquisition via Chats on User-Favored Topics

Shiki Sato,Jun Baba,Asahi Hentona,Shinji Iwata,Akifumi Yoshimoto,Koichiro Yoshino

Task: 提出并定义PIVOT任务，旨在通过聊天获取用户对预定义问题的回答，同时避免让用户感到突兀。

Motivation: 为面向聊天的对话系统提供技术基础，使其能够在用户喜欢的主题聊天中主动获取特定用户信息。

Details

Method: 构建适合分析的数据集，并开发一个简单但有效的系统。 Result: 发现即使是最近的大型语言模型（LLMs）在PIVOT任务中的成功率也较低。 Conclusion: 通过分析数据集开发的系统在PIVOT任务中表现更有效。 Abstract: Chat-oriented dialogue systems designed to provide tangible benefits, such as sharing the latest news or preventing frailty in senior citizens, often require Proactive acquisition of specific user Information via chats on user-faVOred Topics (PIVOT). This study proposes the PIVOT task, designed to advance the technical foundation for these systems. In this task, a system needs to acquire the answers of a user to predefined questions without making the user feel abrupt while engaging in a chat on a predefined topic. We found that even recent large language models (LLMs) show a low success rate in the PIVOT task. We constructed a dataset suitable for the analysis to develop more effective systems. Finally, we developed a simple but effective system for this task by incorporating insights obtained through the analysis of this dataset.

S2R-HDR: A Large-Scale Rendered Dataset for HDR Fusion

Yujin Wang,Jiarui Wu,Yichen Bian,Fan Zhang,Tianfan Xue

Task: 提出S2R-HDR，首个大规模高质量合成数据集用于HDR融合。

Motivation: 解决HDR融合中训练数据不足的问题，因为采集动态场景的大规模HDR图像成本高且技术挑战大。

Details

Method: 使用Unreal Engine 5设计多样化的HDR场景，开发高效渲染管线生成HDR图像，并引入S2R-Adapter进行域适应。 Result: 在真实数据集上实现了最先进的HDR重建性能。 Conclusion: S2R-HDR数据集和S2R-Adapter方法有效提升了HDR融合的泛化能力。 Abstract: The generalization of learning-based high dynamic range (HDR) fusion is often limited by the availability of training data, as collecting large-scale HDR images from dynamic scenes is both costly and technically challenging. To address these challenges, we propose S2R-HDR, the first large-scale high-quality synthetic dataset for HDR fusion, with 24,000 HDR samples. Using Unreal Engine 5, we design a diverse set of realistic HDR scenes that encompass various dynamic elements, motion types, high dynamic range scenes, and lighting. Additionally, we develop an efficient rendering pipeline to generate realistic HDR images. To further mitigate the domain gap between synthetic and real-world data, we introduce S2R-Adapter, a domain adaptation designed to bridge this gap and enhance the generalization ability of models. Experimental results on real-world datasets demonstrate that our approach achieves state-of-the-art HDR reconstruction performance. Dataset and code will be available at https://openimaginglab.github.io/S2R-HDR.

MRD-RAG: Enhancing Medical Diagnosis with Multi-Round Retrieval-Augmented Generation

Yixiang Chen,Penglei Sun,Xiang Li,Xiaowen Chu

Task: 提出一种多轮诊断检索增强生成框架（MRD-RAG），以模拟医生的诊断过程，解决现有医疗RAG框架在多轮对话和疾病关联性方面的不足。

Motivation: 现有医疗RAG框架多适用于单轮问答任务，且在多轮诊断中未考虑潜在疾病间的关联性，无法像医生一样精确诊断。

Details

Method: 提出MRD-RAG框架，分析潜在疾病的诊断信息，并像医生一样进行多轮精确诊断。 Result: 在两个现代医学数据集和两个中医数据集上的实验表明，MRD-RAG显著提升了LLMs的诊断性能。 Conclusion: MRD-RAG框架在医疗诊断中具有潜力，能够有效提升多轮诊断的准确性。 Abstract: In recent years, accurately and quickly deploying medical large language models (LLMs) has become a significant trend. Among these, retrieval-augmented generation (RAG) has garnered significant attention due to its features of rapid deployment and privacy protection. However, existing medical RAG frameworks still have shortcomings. Most existing medical RAG frameworks are designed for single-round question answering tasks and are not suitable for multi-round diagnostic dialogue. On the other hand, existing medical multi-round RAG frameworks do not consider the interconnections between potential diseases to inquire precisely like a doctor. To address these issues, we propose a Multi-Round Diagnostic RAG (MRD-RAG) framework that mimics the doctor's diagnostic process. This RAG framework can analyze diagnosis information of potential diseases and accurately conduct multi-round diagnosis like a doctor. To evaluate the effectiveness of our proposed frameworks, we conduct experiments on two modern medical datasets and two traditional Chinese medicine datasets, with evaluations by GPT and human doctors on different methods. The results indicate that our RAG framework can significantly enhance the diagnostic performance of LLMs, highlighting the potential of our approach in medical diagnosis. The code and data can be found in our project website https://github.com/YixiangCh/MRD-RAG/tree/master.

LAPIS: A novel dataset for personalized image aesthetic assessment

Anne-Sofie Maerten,Li-Wei Chen,Stefanie De Winter,Christophe Bossens,Johan Wagemans

Task: 介绍并评估LAPIS数据集，用于个性化图像美学评估（PIAA）。

Motivation: 填补艺术作品图像在PIAA领域的数据集空白，并提供丰富的图像和个人属性以支持研究。

Details

Method: 通过精心策划的11,723张艺术作品图像，结合美学评分和属性，评估两种现有PIAA模型的性能，并进行消融实验。 Result: 性能在移除某些个人和图像属性时下降，现有模型在艺术图像美学评估中存在相似错误。 Conclusion: LAPIS为艺术图像美学评估提供了新基准，并揭示了现有模型的改进需求。 Abstract: We present the Leuven Art Personalized Image Set (LAPIS), a novel dataset for personalized image aesthetic assessment (PIAA). It is the first dataset with images of artworks that is suitable for PIAA. LAPIS consists of 11,723 images and was meticulously curated in collaboration with art historians. Each image has an aesthetics score and a set of image attributes known to relate to aesthetic appreciation. Besides rich image attributes, LAPIS offers rich personal attributes of each annotator. We implemented two existing state-of-the-art PIAA models and assessed their performance on LAPIS. We assess the contribution of personal attributes and image attributes through ablation studies and find that performance deteriorates when certain personal and image attributes are removed. An analysis of failure cases reveals that both existing models make similar incorrect predictions, highlighting the need for improvements in artistic image aesthetic assessment. The LAPIS project page can be found at: https://github.com/Anne-SofieMaerten/LAPIS

DeepGreen: Effective LLM-Driven Green-washing Monitoring System Designed for Empirical Testing -- Evidence from China

Congluo Xu,Yu Miao,Yiling Xiao,Chengmengjia Lin

Task: 提出DeepGreen系统，利用大语言模型（LLM）检测企业绿色洗白行为。

Motivation: 传统方法难以有效识别企业绿色洗白行为，需要一种更智能的监测工具。

Details

Method: 采用双层LLM分析，初步识别财务报告中的绿色关键词，并通过迭代语义分析评估其实现程度，生成核心变量GreenImplement。 Result: 分析204份财务报告，验证GreenImplement与华证ESG评分的相关性，发现绿色实现显著提升资产回报率，但中小企业贡献有限。 Conclusion: DeepGreen为监管机构和投资者提供了一种主动监测工具，补充传统方法，并揭示了绿色实现的异质性影响。 Abstract: This paper proposes DeepGreen, an Large Language Model Driven (LLM-Driven) system for detecting corporate green-washing behaviour. Utilizing dual-layer LLM analysis, DeepGreen preliminarily identifies potential green keywords in financial statements and then assesses their implementation degree via iterative semantic analysis of LLM. A core variable GreenImplement is derived from the ratio from the two layers' output. We extract 204 financial statements of 68 companies from A-share market over three years, comprising 89,893 words, and analyse them through DeepGreen. Our analysis, supported by violin plots and K-means clustering, reveals insights and validates the variable against the Huazheng ESG rating. It offers a novel perspective for regulatory agencies and investors, serving as a proactive monitoring tool that complements traditional methods.Empirical tests show that green implementation can significantly boost the asset return rate of companies, but there is heterogeneity in scale. Small and medium-sized companies have limited contribution to asset return via green implementation, so there is a stronger motivation for green-washing.

FMNV: A Dataset of Media-Published News Videos for Fake News Detection

Yihao Wang,Zhong Qian,Peifeng Li

Task: 构建一个由媒体组织发布的新闻视频组成的新数据集FMNV，并提出一个基线模型FMNVD用于检测多模态假新闻。

Motivation: 现有数据集主要由用户生成的视频组成，而由媒体组织发布的专业制作的假新闻视频对社会危害更大，但缺乏相关研究。

Details

Method: 通过分析现有数据集和自建数据集FMNV，将假新闻视频分为四类，并利用大型语言模型（LLMs）自动生成虚假内容；提出FMNVD模型，采用双流架构结合CLIP和Faster R-CNN进行视频特征提取，并通过共注意力机制优化特征和多模态聚合。 Result: 实验表明FMNV在多个基线模型上具有泛化能力，且FMNVD在检测效果上表现优越。 Conclusion: 本研究为检测媒体生态系统中高影响力假新闻提供了关键基准，并推进了跨模态不一致性分析的方法。 Abstract: News media, particularly video-based platforms, have become deeply embedded in daily life, concurrently amplifying risks of misinformation dissemination. Consequently, multimodal fake news detection has garnered significant research attention. However, existing datasets predominantly comprise user-generated videos characterized by crude editing and limited public engagement, whereas professionally crafted fake news videos disseminated by media outlets often politically or virally motivated pose substantially greater societal harm. To address this gap, we construct FMNV, a novel dataset exclusively composed of news videos published by media organizations. Through empirical analysis of existing datasets and our curated collection, we categorize fake news videos into four distinct types. Building upon this taxonomy, we employ Large Language Models (LLMs) to automatically generate deceptive content by manipulating authentic media-published news videos. Furthermore, we propose FMNVD, a baseline model featuring a dual-stream architecture integrating CLIP and Faster R-CNN for video feature extraction, enhanced by co-attention mechanisms for feature refinement and multimodal aggregation. Comparative experiments demonstrate both the generalization capability of FMNV across multiple baselines and the superior detection efficacy of FMNVD. This work establishes critical benchmarks for detecting high-impact fake news in media ecosystems while advancing methodologies for cross-modal inconsistency analysis.

Automated Construction of a Knowledge Graph of Nuclear Fusion Energy for Effective Elicitation and Retrieval of Information

A. Loreti,K. Chen,R. George,R. Firth,A. Agnello,S. Tanaka

Task: 提出一种多步骤方法，用于从大规模文档语料库中自动构建知识图谱，以结构化和表示特定领域的知识。

Motivation: 核聚变能源领域知识范围广且异质性高，是测试方法关键特性的理想基准。

Details

Method: 利用预训练的大型语言模型进行命名实体识别和实体解析，并结合知识图谱检索增强生成系统。 Result: 展示了预训练语言模型在应对挑战时的性能，并开发了一个能够回答复杂多跳问题的系统。 Conclusion: 该方法成功构建了首个核聚变能源知识图谱，并验证了其在处理复杂查询时的有效性。 Abstract: In this document, we discuss a multi-step approach to automated construction of a knowledge graph, for structuring and representing domain-specific knowledge from large document corpora. We apply our method to build the first knowledge graph of nuclear fusion energy, a highly specialized field characterized by vast scope and heterogeneity. This is an ideal benchmark to test the key features of our pipeline, including automatic named entity recognition and entity resolution. We show how pre-trained large language models can be used to address these challenges and we evaluate their performance against Zipf's law, which characterizes human-generated natural language. Additionally, we develop a knowledge-graph retrieval-augmented generation system that combines large language models with a multi-prompt approach. This system provides contextually relevant answers to natural-language queries, including complex multi-hop questions that require reasoning across interconnected entities.

Multi-modal Reference Learning for Fine-grained Text-to-Image Retrieval

Zehong Ma,Hao Chen,Wei Zeng,Limin Su,Shiliang Zhang

Task: 提出一种多模态参考学习框架，以解决细粒度文本到图像检索中文本模糊性问题。

Motivation: 现有方法假设每个训练图像的文本描述准确，但实际上文本描述可能模糊且无法捕捉图像的判别性视觉细节，导致表示学习不准确。

Details

Method: 提出多模态参考构建模块和参考引导的表示学习模块，利用多模态参考学习更准确的视觉和文本表示，并通过基于参考的细化方法优化初始检索结果。 Result: 在五个细粒度文本到图像检索数据集上表现优异，例如在RSTPReid数据集上Rank1准确率达到56.2%，超过CFine方法5.6%。 Conclusion: 多模态参考学习框架能有效缓解文本模糊性，提升细粒度文本到图像检索性能。 Abstract: Fine-grained text-to-image retrieval aims to retrieve a fine-grained target image with a given text query. Existing methods typically assume that each training image is accurately depicted by its textual descriptions. However, textual descriptions can be ambiguous and fail to depict discriminative visual details in images, leading to inaccurate representation learning. To alleviate the effects of text ambiguity, we propose a Multi-Modal Reference learning framework to learn robust representations. We first propose a multi-modal reference construction module to aggregate all visual and textual details of the same object into a comprehensive multi-modal reference. The multi-modal reference hence facilitates the subsequent representation learning and retrieval similarity computation. Specifically, a reference-guided representation learning module is proposed to use multi-modal references to learn more accurate visual and textual representations. Additionally, we introduce a reference-based refinement method that employs the object references to compute a reference-based similarity that refines the initial retrieval results. Extensive experiments are conducted on five fine-grained text-to-image retrieval datasets for different text-to-image retrieval tasks. The proposed method has achieved superior performance over state-of-the-art methods. For instance, on the text-to-person image retrieval dataset RSTPReid, our method achieves the Rank1 accuracy of 56.2\%, surpassing the recent CFine by 5.6\%.

NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark

Vladislav Mikhailov,Tita Enstad,David Samuel,Hans Christian Farsethås,Andrey Kutuzov,Erik Velldal,Lilja Øvrelid

Task: 介绍并评估挪威生成语言模型的综合评测套件NorEval。

Motivation: 现有挪威语评测基准覆盖范围有限，缺乏对挪威语两种官方书面标准的全面评测。

Details

Method: NorEval包含24个高质量人工创建的数据集，涵盖挪威语理解和生成任务，并集成到LM Evaluation Harness中。 Result: 评测了19个开源预训练和指令调优的挪威语言模型，并提供了公开可用的评测框架和数据。 Conclusion: NorEval为挪威生成语言模型提供了全面且灵活的评测标准。 Abstract: This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets -- of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokm{\aa}l and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

Benchmarking Multi-Organ Segmentation Tools for Multi-Parametric T1-weighted Abdominal MRI

Nicole Tran,Anisa Prasad,Yan Zhuang,Tejas Sudharshan Mathai,Boah Kim,Sydney Lewis,Pritam Mukherjee,Jianfei Liu,Ronald M. Summers

Task: 量化三种公开工具（MRSeg、TS和VIBE）在特定MRI序列类型上的多器官分割性能。

Motivation: 多参数MRI研究中的多器官分割对放射学应用至关重要，但现有工具在特定MRI序列上的性能尚未量化。

Details

Method: 使用40个来自Duke Liver Dataset的MRI体积，手动标注10个腹部结构，并比较三种工具的分割性能。 Result: MRSeg在Dice得分和Hausdorff距离上表现最佳，显著优于TS和VIBE（p < .05）。 Conclusion: MRSeg在特定MRI序列上的多器官分割性能优于其他两种工具。 Abstract: The segmentation of multiple organs in multi-parametric MRI studies is critical for many applications in radiology, such as correlating imaging biomarkers with disease status (e.g., cirrhosis, diabetes). Recently, three publicly available tools, such as MRSegmentator (MRSeg), TotalSegmentator MRI (TS), and TotalVibeSegmentator (VIBE), have been proposed for multi-organ segmentation in MRI. However, the performance of these tools on specific MRI sequence types has not yet been quantified. In this work, a subset of 40 volumes from the public Duke Liver Dataset was curated. The curated dataset contained 10 volumes each from the pre-contrast fat saturated T1, arterial T1w, venous T1w, and delayed T1w phases, respectively. Ten abdominal structures were manually annotated in these volumes. Next, the performance of the three public tools was benchmarked on this curated dataset. The results indicated that MRSeg obtained a Dice score of 80.7 $\pm$ 18.6 and Hausdorff Distance (HD) error of 8.9 $\pm$ 10.4 mm. It fared the best ($p < .05$) across the different sequence types in contrast to TS and VIBE.

Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation

Bo Zhang,Hui Ma,Dailin Li,Jian Ding,Jian Wang,Bo Xu,HongFei Lin

Task: 提出一种名为KEDiT的高效方法，用于微调大型语言模型（LLMs）以实现基于知识的对话生成。

Motivation: 大型语言模型在文本理解和生成方面表现出色，但缺乏利用最新或领域特定知识的能力。

Details

Method: KEDiT分为两个阶段：首先通过信息瓶颈压缩检索到的知识为可学习参数，其次通过轻量级知识感知适配器将这些压缩后的知识向量集成到LLM中。 Result: 在Wizard of Wikipedia和PubMed-Dialog数据集上的实验表明，KEDiT在生成上下文相关且信息丰富的回答方面优于基线方法。 Conclusion: KEDiT结合了预训练LLMs的优势和动态知识整合的适应性，为医学等领域提供了可扩展的解决方案。 Abstract: Large language models (LLMs) demonstrate remarkable text comprehension and generation capabilities but often lack the ability to utilize up-to-date or domain-specific knowledge not included in their training data. To address this gap, we introduce KEDiT, an efficient method for fine-tuning LLMs for knowledge-grounded dialogue generation. KEDiT operates in two main phases: first, it employs an information bottleneck to compress retrieved knowledge into learnable parameters, retaining essential information while minimizing computational overhead. Second, a lightweight knowledge-aware adapter integrates these compressed knowledge vectors into the LLM during fine-tuning, updating less than 2\% of the model parameters. The experimental results on the Wizard of Wikipedia and a newly constructed PubMed-Dialog dataset demonstrate that KEDiT excels in generating contextually relevant and informative responses, outperforming competitive baselines in automatic, LLM-based, and human evaluations. This approach effectively combines the strengths of pretrained LLMs with the adaptability needed for incorporating dynamic knowledge, presenting a scalable solution for fields such as medicine.

MMLA: Multi-Environment, Multi-Species, Low-Altitude Aerial Footage Dataset

Jenna Kline,Samuel Stevens,Guy Maalouf,Camille Rondeau Saint-Jean,Dat Nguyen Ngoc,Majid Mirmehdi,David Guerin,Tilo Burghardt,Elzbieta Pastucha,Blair Costelloe,Matthew Watson,Thomas Richardson,Ulrik Pagh Schultz Lundquist

Task: 评估计算机视觉模型在低空航拍图像中对野生动物的实时检测性能。

Motivation: 填补现有研究在低空航拍图像中对不同物种和环境的泛化性评估的空白。

Details

Method: 提出一个多环境、多物种的低空航拍数据集（MMLA），并评估三种YOLO模型（YOLOv5m、YOLOv8m和YOLOv11m）的性能。 Result: 结果显示不同地点和物种的检测性能存在显著差异。 Conclusion: 强调了在不同环境中评估检测算法对无人机野生动物监测应用的重要性。 Abstract: Real-time wildlife detection in drone imagery is critical for numerous applications, including animal ecology, conservation, and biodiversity monitoring. Low-altitude drone missions are effective for collecting fine-grained animal movement and behavior data, particularly if missions are automated for increased speed and consistency. However, little work exists on evaluating computer vision models on low-altitude aerial imagery and generalizability across different species and settings. To fill this gap, we present a novel multi-environment, multi-species, low-altitude aerial footage (MMLA) dataset. MMLA consists of drone footage collected across three diverse environments: Ol Pejeta Conservancy and Mpala Research Centre in Kenya, and The Wilds Conservation Center in Ohio, which includes five species: Plains zebras, Grevy's zebras, giraffes, onagers, and African Painted Dogs. We comprehensively evaluate three YOLO models (YOLOv5m, YOLOv8m, and YOLOv11m) for detecting animals. Results demonstrate significant performance disparities across locations and species-specific detection variations. Our work highlights the importance of evaluating detection algorithms across different environments for robust wildlife monitoring applications using drones.

Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generation

Alireza Salemi,Chris Samarinas,Hamed Zamani

Task: 研究检索增强的大型语言模型（LLMs）在生成多样且全面响应方面的局限性，并引入基于两阶段系统设计的Plan-and-Refine（P&R）框架。

Motivation: 解决LLMs在生成响应时缺乏多样性和全面性的问题。

Details

Method: 采用两阶段设计：全局探索阶段生成多样化计划，局部利用阶段生成并迭代优化响应提案，最后通过奖励模型选择最佳提案。 Result: 在ANTIQUE和TREC数据集上分别实现了13.1%和15.41%的性能提升。 Conclusion: P&R框架显著提升了响应的多样性和全面性，并通过实验和用户研究验证了其有效性。 Abstract: This paper studies the limitations of (retrieval-augmented) large language models (LLMs) in generating diverse and comprehensive responses, and introduces the Plan-and-Refine (P&R) framework based on a two phase system design. In the global exploration phase, P&R generates a diverse set of plans for the given input, where each plan consists of a list of diverse query aspects with corresponding additional descriptions. This phase is followed by a local exploitation phase that generates a response proposal for the input query conditioned on each plan and iteratively refines the proposal for improving the proposal quality. Finally, a reward model is employed to select the proposal with the highest factuality and coverage. We conduct our experiments based on the ICAT evaluation methodology--a recent approach for answer factuality and comprehensiveness evaluation. Experiments on the two diverse information seeking benchmarks adopted from non-factoid question answering and TREC search result diversification tasks demonstrate that P&R significantly outperforms baselines, achieving up to a 13.1% improvement on the ANTIQUE dataset and a 15.41% improvement on the TREC dataset. Furthermore, a smaller scale user study confirms the substantial efficacy of the P&R framework.

SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding

Yangliu Hu,Zikai Song,Na Feng,Yawei Luo,Junqing Yu,Yi-Ping Phoebe Chen,Wei Yang

Task: 通过自监督片段微调（SF$^2$T）提升视频大语言模型（Video-LLMs）在细粒度视频理解方面的能力。

Motivation: 现有Video-LLMs在整体视频描述上表现良好，但在细粒度理解（如视觉动态和视频细节）方面存在不足。

Details

Method: 提出SF$^2$T方法，利用视频固有特性进行自监督微调，避免人工标注和自然语言的局限性；同时构建FineVidBench基准数据集进行多层面评估。 Result: 实验表明，SF$^2$T显著提升了模型在时空细节捕捉和解释方面的能力。 Conclusion: SF$^2$T是一种高效且无需标注的微调方法，能够显著增强Video-LLMs的细粒度视频理解能力。 Abstract: Video-based Large Language Models (Video-LLMs) have witnessed substantial advancements in recent years, propelled by the advancement in multi-modal LLMs. Although these models have demonstrated proficiency in providing the overall description of videos, they struggle with fine-grained understanding, particularly in aspects such as visual dynamics and video details inquiries. To tackle these shortcomings, we find that fine-tuning Video-LLMs on self-supervised fragment tasks, greatly improve their fine-grained video understanding abilities. Hence we propose two key contributions:(1) Self-Supervised Fragment Fine-Tuning (SF$^2$T), a novel effortless fine-tuning method, employs the rich inherent characteristics of videos for training, while unlocking more fine-grained understanding ability of Video-LLMs. Moreover, it relieves researchers from labor-intensive annotations and smartly circumvents the limitations of natural language, which often fails to capture the complex spatiotemporal variations in videos; (2) A novel benchmark dataset, namely FineVidBench, for rigorously assessing Video-LLMs' performance at both the scene and fragment levels, offering a comprehensive evaluation of their capabilities. We assessed multiple models and validated the effectiveness of SF$^2$T on them. Experimental results reveal that our approach improves their ability to capture and interpret spatiotemporal details.

A System for Comprehensive Assessment of RAG Frameworks

Mattia Rengo,Senad Beadini,Domenico Alfano,Roberto Abbruzzese

Task: 提出一个名为SCARF的模块化评估框架，用于全面评估检索增强生成（RAG）系统。

Motivation: 现有的评估框架缺乏对RAG系统在真实部署场景中的全面黑盒评估方法。

Details

Method: SCARF是一个模块化且灵活的框架，支持端到端的黑盒评估，涵盖多种部署配置和自动化测试。 Result: SCARF能够生成详细的性能报告，支持对不同RAG框架和配置的灵活评估。 Conclusion: SCARF为研究人员和行业专业人士提供了一个可扩展且适应性强的解决方案，用于评估RAG应用。 Abstract: Retrieval Augmented Generation (RAG) has emerged as a standard paradigm for enhancing the factual accuracy and contextual relevance of Large Language Models (LLMs) by integrating retrieval mechanisms. However, existing evaluation frameworks fail to provide a holistic black-box approach to assessing RAG systems, especially in real-world deployment scenarios. To address this gap, we introduce SCARF (System for Comprehensive Assessment of RAG Frameworks), a modular and flexible evaluation framework designed to benchmark deployed RAG applications systematically. SCARF provides an end-to-end, black-box evaluation methodology, enabling a limited-effort comparison across diverse RAG frameworks. Our framework supports multiple deployment configurations and facilitates automated testing across vector databases and LLM serving strategies, producing a detailed performance report. Moreover, SCARF integrates practical considerations such as response coherence, providing a scalable and adaptable solution for researchers and industry professionals evaluating RAG applications. Using the REST APIs interface, we demonstrate how SCARF can be applied to real-world scenarios, showcasing its flexibility in assessing different RAG frameworks and configurations. SCARF is available at GitHub repository.

PIDSR:ComplementaryPolarizedImageDemosaicingandSuper-Resolution

Shuangfan Zhou,Chu Zhou,Youwei Lyu,Heng Guo,Zhanyu Ma,Boxin Shi,Imari Sato

Task: 提出一种联合框架PIDSR，用于从CPFA原始图像中直接获取高质量高分辨率偏振图像，并提高DoP和AoP的准确性。

Motivation: 现有偏振图像去马赛克（PID）方法无法提升分辨率，而偏振图像超分辨率（PISR）方法会保留或放大去马赛克引入的误差，导致偏振参数（如DoP和AoP）不准确。

Details

Method: 提出PIDSR框架，联合进行偏振图像去马赛克和超分辨率，直接从CPFA原始图像中获取高质量高分辨率偏振图像。 Result: 实验表明PIDSR在合成和真实数据上均达到最优性能，并有助于下游任务。 Conclusion: PIDSR能够直接从CPFA原始图像中获取高质量高分辨率偏振图像，显著提高偏振参数的准确性。 Abstract: Polarization cameras can capture multiple polarized images with different polarizer angles in a single shot, bringing convenience to polarization-based downstream tasks. However, their direct outputs are color-polarization filter array (CPFA) raw images, requiring demosaicing to reconstruct full-resolution, full-color polarized images; unfortunately, this necessary step introduces artifacts that make polarization-related parameters such as the degree of polarization (DoP) and angle of polarization (AoP) prone to error. Besides, limited by the hardware design, the resolution of a polarization camera is often much lower than that of a conventional RGB camera. Existing polarized image demosaicing (PID) methods are limited in that they cannot enhance resolution, while polarized image super-resolution (PISR) methods, though designed to obtain high-resolution (HR) polarized images from the demosaicing results, tend to retain or even amplify errors in the DoP and AoP introduced by demosaicing artifacts. In this paper, we propose PIDSR, a joint framework that performs complementary Polarized Image Demosaicing and Super-Resolution, showing the ability to robustly obtain high-quality HR polarized images with more accurate DoP and AoP from a CPFA raw image in a direct manner. Experiments show our PIDSR not only achieves state-of-the-art performance on both synthetic and real data, but also facilitates downstream tasks.

Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models

Hongcheng Guo,Juntao Yao,Boyang Wang,Junjia Du,Shaosheng Cao,Donglin Di,Shun Zhang,Zhoujun Li

Task: 提出一种名为C-Prune的两阶段框架，用于自适应任务特定的MoE LLMs压缩。

Motivation: 解决MoE模型中专家层内同质性和层间相似性模式带来的挑战，以实现更高效的模型部署。

Details

Method: 通过层内专家聚类和全局集群剪枝，结合参数相似性度量和统一重要性评分机制。 Result: C-Prune能有效减小模型规模，并在性能上优于现有的MoE剪枝方法。 Conclusion: C-Prune为MoE模型的压缩提供了一种高效且性能优越的解决方案。 Abstract: Mixture-of-Experts (MoE) architectures have emerged as a promising paradigm for scaling large language models (LLMs) with sparse activation of task-specific experts. Despite their computational efficiency during inference, the massive overall parameter footprint of MoE models (e.g., GPT-4) introduces critical challenges for practical deployment. Current pruning approaches often fail to address two inherent characteristics of MoE systems: 1).intra-layer expert homogeneity where experts within the same MoE layer exhibit functional redundancy, and 2). inter-layer similarity patterns where deeper layers tend to contain progressively more homogeneous experts. To tackle these issues, we propose Cluster-driven Expert Pruning (C-Prune), a novel two-stage framework for adaptive task-specific compression of MoE LLMs. C-Prune operates through layer-wise expert clustering, which groups functionally similar experts within each MoE layer using parameter similarity metrics, followed by global cluster pruning, which eliminates redundant clusters across all layers through a unified importance scoring mechanism that accounts for cross-layer homogeneity. We validate C-Prune through extensive experiments on multiple MoE models and benchmarks. The results demonstrate that C-Prune effectively reduces model size while outperforming existing MoE pruning methods.

Exploring a Patch-Wise Approach for Privacy-Preserving Fake ID Detection

Javier Muñoz-Haro,Ruben Tolosana,Ruben Vera-Rodriguez,Aythami Morales,Julian Fierrez

Task: 研究如何在隐私保护的前提下检测伪造身份证件。

Motivation: 当前缺乏公开的真实身份证件数据，且大多数研究依赖私有数据库，限制了该领域的进展。

Details

Method: 提出一种基于分块的隐私保护方法，探索两种匿名化级别和不同分块大小配置，并结合Vision Transformers和Foundation Models进行分析。 Result: 在DLC-2021数据库上，分块和身份证件级别的EER分别为13.91%和0%，表现出良好的泛化能力。 Conclusion: 研究不仅提出了一种有效的隐私保护检测方法，还首次公开了包含真实和伪造身份证件分块的数据集，推动了该领域的发展。 Abstract: In an increasingly digitalized world, verifying the authenticity of ID documents has become a critical challenge for real-life applications such as digital banking, crypto-exchanges, renting, etc. This study focuses on the topic of fake ID detection, covering several limitations in the field. In particular, no publicly available data from real ID documents exists, and most studies rely on proprietary in-house databases that are not available due to privacy reasons. In order to shed some light on this critical challenge that makes difficult to advance in the field, we explore a trade-off between privacy (i.e., amount of sensitive data available) and performance, proposing a novel patch-wise approach for privacy-preserving fake ID detection. Our proposed approach explores how privacy can be enhanced through: i) two levels of anonymization for an ID document (i.e., fully- and pseudo-anonymized), and ii) different patch size configurations, varying the amount of sensitive data visible in the patch image. Also, state-of-the-art methods such as Vision Transformers and Foundation Models are considered in the analysis. The experimental framework shows that, on an unseen database (DLC-2021), our proposal achieves 13.91% and 0% EERs at patch and ID document level, showing a good generalization to other databases. In addition to this exploration, another key contribution of our study is the release of the first publicly available database that contains 48,400 patches from both real and fake ID documents, along with the experimental framework and models, which will be available in our GitHub.

What the HellaSwag? On the Validity of Common-Sense Reasoning Benchmarks

Pavel Chizhov,Mattia Nee,Pierre-Carl Langlais,Ivan P. Yamshchikov

Task: 分析HellaSwag基准的构建效度问题及其对常识推理评估的影响。

Motivation: HellaSwag作为广泛使用的常识推理评估基准存在严重的构建效度问题，可能导致模型选择中的错误决策。

Details

Method: 通过生成式语言模型的不同规模评估，揭示HellaSwag的问题，并提出修正子集GoldenSwag。 Result: HellaSwag在语法、提示和选项设计上存在问题，65%的模型预测在无问题文本时仍保持一致，表明其评估不准确。 Conclusion: HellaSwag不适合用于常识推理评估，需改进未来基准的设计要求，并推荐使用修正后的GoldenSwag。 Abstract: Common-sense reasoning is a key language model capability because it encapsulates not just specific factual knowledge but rather general language and world understanding. Measuring common-sense reasoning, therefore, is crucial for language models of different sizes and applications. One of the most widely used benchmarks for evaluating such capabilities is HellaSwag; however, in this paper, we show that it has severe construct validity issues. These issues range from basic ungrammaticality and numerous typos to misleading prompts or equally correct options. Furthermore, we show that if models are evaluated only on answer texts, or with "Lorem ipsum dolor..." instead of the question, more than 65% of model predictions remain the same, and this cannot be attributed merely to contamination. Since benchmark scores are an essential part of model selection in both research and commercial applications, these validity issues can have severe consequences. In particular, knowing that taking benchmark scores at face value is ubiquitous, inadequate evaluation leads to ill-informed decisions about models. In this paper, we thoroughly investigate critical validity issues posed by HellaSwag and illustrate them with various evaluations using generative language models of different sizes. We argue that this benchmark does not accurately measure common-sense reasoning and, therefore, should not be used for evaluation in its current state. Based on the results of our study, we propose requirements that should be met by future common-sense reasoning benchmarks. In addition, we release GoldenSwag, a corrected subset of HellaSwag, which, to our belief, facilitates acceptable common-sense reasoning evaluation.

Towards Micro-Action Recognition with Limited Annotations: An Asynchronous Pseudo Labeling and Training Approach

Yan Zhang,Lechao Cheng,Yaxiong Wang,Zhun Zhong,Meng Wang

Task: 提出一种名为异步伪标签与训练（APLT）的新框架，以解决半监督微动作识别（SSMAR）中伪标签不准确导致性能下降的问题。

Motivation: 传统半监督学习方法在SSMAR中容易因伪标签不准确而过拟合，导致错误累积和性能下降。

Details

Method: APLT框架将伪标签生成与模型训练分离，引入半监督聚类方法生成更准确的伪标签，并提出自适应阈值策略动态过滤噪声标签，最后构建基于记忆的原型分类器指导模型训练。 Result: 在三个MAR数据集上的实验表明，APLT显著优于现有半监督学习方法，例如在MA-12数据集上使用50%标记数据时，准确率比FixMatch提高14.5%。 Conclusion: APLT通过异步伪标签与训练，有效解决了伪标签不准确和过拟合问题，显著提升了半监督微动作识别的性能。 Abstract: Micro-Action Recognition (MAR) aims to classify subtle human actions in video. However, annotating MAR datasets is particularly challenging due to the subtlety of actions. To this end, we introduce the setting of Semi-Supervised MAR (SSMAR), where only a part of samples are labeled. We first evaluate traditional Semi-Supervised Learning (SSL) methods to SSMAR and find that these methods tend to overfit on inaccurate pseudo-labels, leading to error accumulation and degraded performance. This issue primarily arises from the common practice of directly using the predictions of classifier as pseudo-labels to train the model. To solve this issue, we propose a novel framework, called Asynchronous Pseudo Labeling and Training (APLT), which explicitly separates the pseudo-labeling process from model training. Specifically, we introduce a semi-supervised clustering method during the offline pseudo-labeling phase to generate more accurate pseudo-labels. Moreover, a self-adaptive thresholding strategy is proposed to dynamically filter noisy labels of different classes. We then build a memory-based prototype classifier based on the filtered pseudo-labels, which is fixed and used to guide the subsequent model training phase. By alternating the two pseudo-labeling and model training phases in an asynchronous manner, the model can not only be learned with more accurate pseudo-labels but also avoid the overfitting issue. Experiments on three MAR datasets show that our APLT largely outperforms state-of-the-art SSL methods. For instance, APLT improves accuracy by 14.5\% over FixMatch on the MA-12 dataset when using only 50\% labeled data. Code will be publicly available.

MuSaRoNews: A Multidomain, Multimodal Satire Dataset from Romanian News Articles

Răzvan-Alexandru Smădu,Andreea Iuga,Dumitru-Clementin Cercel

Task: 构建一个多模态语料库MuSaRoNews，用于检测罗马尼亚新闻文章中的讽刺内容。

Motivation: 讽刺和假新闻虽然目的不同，但都会传播虚假信息，仅依赖文本难以检测其表面与实际含义的不一致，需要结合其他信息源（如视觉）来提高检测效果。

Details

Method: 收集了117,834篇来自真实和讽刺新闻来源的公开新闻文章，构建了罗马尼亚语中首个多模态讽刺检测语料库。 Result: 实验表明，结合文本和视觉模态能提高讽刺检测的性能。 Conclusion: 多模态方法在讽刺检测中具有优势，为罗马尼亚语的讽刺检测提供了首个多模态资源。 Abstract: Satire and fake news can both contribute to the spread of false information, even though both have different purposes (one if for amusement, the other is to misinform). However, it is not enough to rely purely on text to detect the incongruity between the surface meaning and the actual meaning of the news articles, and, often, other sources of information (e.g., visual) provide an important clue for satire detection. This work introduces a multimodal corpus for satire detection in Romanian news articles named MuSaRoNews. Specifically, we gathered 117,834 public news articles from real and satirical news sources, composing the first multimodal corpus for satire detection in the Romanian language. We conducted experiments and showed that the use of both modalities improves performance.

Breaking the Barriers: Video Vision Transformers for Word-Level Sign Language Recognition

Alexander Brettmann,Jakob Grävinghoff,Marlene Rüschoff,Marie Westhues

Task: 提出一种基于Video Vision Transformer (ViViT)的模型，用于动态词级美国手语(ASL)识别。

Motivation: 解决听力人群对手语不熟练导致的沟通障碍，并通过自动手语识别(SLR)技术提升识别效果。

Details

Method: 采用Video Vision Transformer (ViViT)模型，利用自注意力机制捕捉视频序列中的全局时空依赖关系。 Result: 在WLASL100数据集上达到75.58%的Top-1准确率，优于传统CNN模型的65.89%。 Conclusion: 基于Transformer的架构在手语识别中具有巨大潜力，有助于克服沟通障碍并促进聋哑人群的包容性。 Abstract: Sign language is a fundamental means of communication for the deaf and hard-of-hearing (DHH) community, enabling nuanced expression through gestures, facial expressions, and body movements. Despite its critical role in facilitating interaction within the DHH population, significant barriers persist due to the limited fluency in sign language among the hearing population. Overcoming this communication gap through automatic sign language recognition (SLR) remains a challenge, particularly at a dynamic word-level, where temporal and spatial dependencies must be effectively recognized. While Convolutional Neural Networks have shown potential in SLR, they are computationally intensive and have difficulties in capturing global temporal dependencies between video sequences. To address these limitations, we propose a Video Vision Transformer (ViViT) model for word-level American Sign Language (ASL) recognition. Transformer models make use of self-attention mechanisms to effectively capture global relationships across spatial and temporal dimensions, which makes them suitable for complex gesture recognition tasks. The VideoMAE model achieves a Top-1 accuracy of 75.58% on the WLASL100 dataset, highlighting its strong performance compared to traditional CNNs with 65.89%. Our study demonstrates that transformer-based architectures have great potential to advance SLR, overcome communication barriers and promote the inclusion of DHH individuals.

MOSAIC: Modeling Social AI for Content Dissemination and Regulation in Multi-Agent Simulations

Genglin Liu,Salman Rahman,Elisa Kreiss,Marzyeh Ghassemi,Saadia Gabriel

Task: 提出一个开源社交网络模拟框架MOSAIC，用于分析用户行为和内容传播动态。

Motivation: 通过结合生成语言代理和社交图，研究用户如何判断在线社交内容的真实性，并探索内容审核策略的效果。

Details

Method: 使用多样化的细粒度角色构建用户表示，结合LLM代理和定向社交图进行多代理模拟。 Result: 发现三种内容审核策略不仅能减少虚假信息的传播，还能提高用户参与度；同时分析了代理的推理与集体参与模式的一致性。 Conclusion: 开源模拟软件以促进AI和社会科学的进一步研究。 Abstract: We present a novel, open-source social network simulation framework, MOSAIC, where generative language agents predict user behaviors such as liking, sharing, and flagging content. This simulation combines LLM agents with a directed social graph to analyze emergent deception behaviors and gain a better understanding of how users determine the veracity of online social content. By constructing user representations from diverse fine-grained personas, our system enables multi-agent simulations that model content dissemination and engagement dynamics at scale. Within this framework, we evaluate three different content moderation strategies with simulated misinformation dissemination, and we find that they not only mitigate the spread of non-factual content but also increase user engagement. In addition, we analyze the trajectories of popular content in our simulations, and explore whether simulation agents' articulated reasoning for their social interactions truly aligns with their collective engagement patterns. We open-source our simulation software to encourage further research within AI and social sciences.

Nonlocal Retinex-Based Variational Model and its Deep Unfolding Twin for Low-Light Image Enhancement

Daniel Torres,Joan Duran,Julia Navarro,Catalina Sbert

Task: 提出一种基于Retinex分解的低光照图像增强变分方法。

Motivation: 低光照条件下捕获的图像质量差，影响图像分割和目标检测等任务。

Details

Method: 结合Retinex分解、颜色校正预处理、非局部梯度型保真项和自动伽马校正模块，并扩展为深度展开模型。 Result: 实验表明，该方法在视觉和质量指标上优于现有技术，包括深度学习方法。 Conclusion: 提出的变分方法在低光照图像增强中表现出色，且无需依赖学习策略。 Abstract: Images captured under low-light conditions present significant limitations in many applications, as poor lighting can obscure details, reduce contrast, and hide noise. Removing the illumination effects and enhancing the quality of such images is crucial for many tasks, such as image segmentation and object detection. In this paper, we propose a variational method for low-light image enhancement based on the Retinex decomposition into illumination, reflectance, and noise components. A color correction pre-processing step is applied to the low-light image, which is then used as the observed input in the decomposition. Moreover, our model integrates a novel nonlocal gradient-type fidelity term designed to preserve structural details. Additionally, we propose an automatic gamma correction module. Building on the proposed variational approach, we extend the model by introducing its deep unfolding counterpart, in which the proximal operators are replaced with learnable networks. We propose cross-attention mechanisms to capture long-range dependencies in both the nonlocal prior of the reflectance and the nonlocal gradient-based constraint. Experimental results demonstrate that both methods compare favorably with several recent and state-of-the-art techniques across different datasets. In particular, despite not relying on learning strategies, the variational model outperforms most deep learning approaches both visually and in terms of quality metrics.

The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models

Michael J Bommarito II,Jillian Bommarito,Daniel Martin Katz

Task: The KL3M Data Project introduces a comprehensive training data pipeline to minimize legal risks related to copyright and breach of contract in large language models.

Motivation: The motivation is to address the uncertainty and potential legal risks associated with pre-training data for large language models, ensuring ethical and legal compliance.

Details

Method: The method involves creating a verified corpus of over 132 million documents from 16 sources, with strict copyright and licensing protocols, and releasing the entire pipeline including source code, metadata, and processed data. Result: The project provides freely available resources on platforms like S3, Hugging Face, and GitHub under CC-BY terms, supporting ethical AI development. Conclusion: The KL3M Data Project aims to promote a more ethical, legal, and sustainable approach to AI model development and use. Abstract: Practically all large language models have been pre-trained on data that is subject to global uncertainty related to copyright infringement and breach of contract. This creates potential risk for users and developers due to this uncertain legal status. The KL3M Data Project directly confronts this critical issue by introducing the largest comprehensive training data pipeline that minimizes risks related to copyright or breach of contract. The foundation of this project is a corpus of over 132 million documents and trillions of tokens spanning 16 different sources that have been verified to meet the strict copyright and licensing protocol detailed herein. We are releasing the entire pipeline, including 1) the source code to acquire and process these documents, 2) the original document formats with associated provenance and metadata, 3) extracted content in a standardized format, 4) pre-tokenized representations of the documents, and 5) various mid- and post-train resources such as question-answer, summarization, conversion, drafting, classification, prediction, and conversational data. All of these resources are freely available to the public on S3, Hugging Face, and GitHub under CC-BY terms. We are committed to continuing this project in furtherance of a more ethical, legal, and sustainable approach to the development and use of AI models.

P2Object: Single Point Supervised Object Detection and Instance Segmentation

Pengfei Chen,Xuehui Yu,Xumeng Han,Kuiran Wang,Guorong Li,Lingxi Xie,Zhenjun Han,Jianbin Jiao

Task: 提出一种基于单点监督的目标识别方法，通过改进提案生成和优化策略，提升性能。

Motivation: 单点监督的目标识别性能与全监督方法差距较大，现有方法存在提案生成和优化的局限性。

Details

Method: 提出Point-to-Box Network (P2BNet)和其改进版本P2BNet++及Point-to-Mask Network (P2MNet)，通过实例级提案袋生成、离散到连续优化策略和像素级感知提升性能。 Result: 在COCO、VOC、SBD和Cityscapes数据集上，方法显著超越先前方法，缩小了与全监督任务的性能差距。 Conclusion: 提出的方法通过连续优化和像素级感知，显著提升了单点监督目标识别的性能，并展示了在分割任务中的潜力。 Abstract: Object recognition using single-point supervision has attracted increasing attention recently. However, the performance gap compared with fully-supervised algorithms remains large. Previous works generated class-agnostic \textbf{\textit{proposals in an image}} offline and then treated mixed candidates as a single bag, putting a huge burden on multiple instance learning (MIL). In this paper, we introduce Point-to-Box Network (P2BNet), which constructs balanced \textbf{\textit{instance-level proposal bags}} by generating proposals in an anchor-like way and refining the proposals in a coarse-to-fine paradigm. Through further research, we find that the bag of proposals, either at the image level or the instance level, is established on discrete box sampling. This leads the pseudo box estimation into a sub-optimal solution, resulting in the truncation of object boundaries or the excessive inclusion of background. Hence, we conduct a series exploration of discrete-to-continuous optimization, yielding P2BNet++ and Point-to-Mask Network (P2MNet). P2BNet++ conducts an approximately continuous proposal sampling strategy by better utilizing spatial clues. P2MNet further introduces low-level image information to assist in pixel prediction, and a boundary self-prediction is designed to relieve the limitation of the estimated boxes. Benefiting from the continuous object-aware \textbf{\textit{pixel-level perception}}, P2MNet can generate more precise bounding boxes and generalize to segmentation tasks. Our method largely surpasses the previous methods in terms of the mean average precision on COCO, VOC, SBD, and Cityscapes, demonstrating great potential to bridge the performance gap compared with fully supervised tasks.

Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs

Yichun Yin,Wenyong Huang,Kaikai Song,Yehui Tang,Xueyu Wu,Wei Guo,Peng Guo,Yaoyuan Wang,Xiaojun Meng,Yasheng Wang,Dong Li,Can Chen,Dandan Tu,Yin Li,Fisher Yu,Ruiming Tang,Yunhe Wang,Baojun Wang,Bin Wang,Bo Wang,Boxiao Liu,Changzheng Zhang,Duyu Tang,Fei Mi,Hui Jin,Jiansheng Wei,Jiarui Qin,Jinpeng Li,Jun Zhao,Liqun Deng,Lin Li,Minghui Xu,Naifu Zhang,Nianzu Zheng,Qiang Li,Rongju Ruan,Shengjun Cheng,Tianyu Guo,Wei He,Wei Li,Weiwen Liu,Wulong Liu,Xinyi Dai,Yonghan Dong,Yu Pan,Yue Li,Yufei Wang,Yujun Li,Yunsheng Ni,Zhe Liu,Zhenhe Zhang,Zhicheng Liu

Task: 训练一个具有1350亿参数的密集Transformer模块的大型语言模型Pangu Ultra。

Motivation: 尽管大型语言模型在规模和能力上取得了前所未有的进展，但训练如此大规模的模型仍面临优化和系统挑战。

Details

Method: 提出深度缩放三明治归一化方法以稳定训练过程，并在8192个Ascend NPU上进行大规模预训练和后训练优化。 Result: Pangu Ultra在多个基准测试中显著优于Llama 405B和Mistral Large 2等密集模型，并与参数更多的稀疏模型DeepSeek-R1竞争。 Conclusion: Ascend NPU能够高效训练超过1000亿参数的密集模型，模型和系统将面向商业客户开放。 Abstract: We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers.

AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relations

Junli Liu,Qizhi Chen,Zhigang Wang,Yiwen Tang,Yiting Zhang,Chi Yan,Dong Wang,Xuelong Li,Bin Zhao

Task: 提出并解决从航拍视角进行视觉定位的新任务AerialVG。

Motivation: 传统视觉定位方法在航拍图像中表现不佳，因航拍图像的高分辨率和视觉相似物体多，需强调空间关系。

Details

Method: 提出AerialVG数据集，包含5K航拍图像和50K标注描述；设计分层交叉注意力机制和关系感知定位模块。 Result: 实验验证了数据集和方法的有效性，强调了空间推理在航拍视觉定位中的重要性。 Conclusion: AerialVG任务和提出的方法为航拍视觉定位提供了新方向，代码和数据集将公开。 Abstract: Visual grounding (VG) aims to localize target objects in an image based on natural language descriptions. In this paper, we propose AerialVG, a new task focusing on visual grounding from aerial views. Compared to traditional VG, AerialVG poses new challenges, \emph{e.g.}, appearance-based grounding is insufficient to distinguish among multiple visually similar objects, and positional relations should be emphasized. Besides, existing VG models struggle when applied to aerial imagery, where high-resolution images cause significant difficulties. To address these challenges, we introduce the first AerialVG dataset, consisting of 5K real-world aerial images, 50K manually annotated descriptions, and 103K objects. Particularly, each annotation in AerialVG dataset contains multiple target objects annotated with relative spatial relations, requiring models to perform comprehensive spatial reasoning. Furthermore, we propose an innovative model especially for the AerialVG task, where a Hierarchical Cross-Attention is devised to focus on target regions, and a Relation-Aware Grounding module is designed to infer positional relations. Experimental results validate the effectiveness of our dataset and method, highlighting the importance of spatial reasoning in aerial visual grounding. The code and dataset will be released.

Token Level Routing Inference System for Edge Devices

Jianshu She,Wenhao Zheng,Zhengzhong Liu,Hongyi Wang,Eric Xing,Huaxiu Yao,Qirong Ho

Task: 提出一种协作解码推理系统，以解决大型语言模型在边缘设备上部署效率低的问题。

Motivation: 大型语言模型推理计算复杂度高，而小型语言模型虽然速度快但响应质量较差且易产生幻觉。

Details

Method: 通过协作解码，小型模型在本地进行推理，同时选择性咨询云端大型模型生成关键令牌。 Result: 系统在CommonsenseQA上实现了60%的性能提升，仅需上传不到7%的令牌到云端大型模型。 Conclusion: 协作解码是一种高效且实用的解决方案，能够平衡模型性能和资源消耗。 Abstract: The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer faster decoding and lower resource consumption but often suffer from degraded response quality and heightened susceptibility to hallucinations. To address this trade-off, collaborative decoding, in which a large model assists in generating critical tokens, has emerged as a promising solution. This paradigm leverages the strengths of both model types by enabling high-quality inference through selective intervention of the large model, while maintaining the speed and efficiency of the smaller model. In this work, we present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud.

V2V3D: View-to-View Denoised 3D Reconstruction for Light-Field Microscopy

Jiayin Zhao,Zhenqi Fu,Tao Yu,Hui Qiao

Task: 提出一种名为V2V3D的无监督框架，用于联合优化光场显微镜（LFM）图像去噪和3D重建。

Motivation: 现有LFM重建算法对传感器噪声敏感或需要难以获取的标注数据，限制了其应用。

Details

Method: 基于view2view的无监督框架，结合noise2noise原理进行去噪，并引入基于波光学的特征对齐技术。 Result: 实验表明，V2V3D在计算效率和性能上优于现有方法。 Conclusion: V2V3D为挑战性条件下的3D成像提供了有前景的解决方案。 Abstract: Light field microscopy (LFM) has gained significant attention due to its ability to capture snapshot-based, large-scale 3D fluorescence images. However, existing LFM reconstruction algorithms are highly sensitive to sensor noise or require hard-to-get ground-truth annotated data for training. To address these challenges, this paper introduces V2V3D, an unsupervised view2view-based framework that establishes a new paradigm for joint optimization of image denoising and 3D reconstruction in a unified architecture. We assume that the LF images are derived from a consistent 3D signal, with the noise in each view being independent. This enables V2V3D to incorporate the principle of noise2noise for effective denoising. To enhance the recovery of high-frequency details, we propose a novel wave-optics-based feature alignment technique, which transforms the point spread function, used for forward propagation in wave optics, into convolution kernels specifically designed for feature alignment. Moreover, we introduce an LFM dataset containing LF images and their corresponding 3D intensity volumes. Extensive experiments demonstrate that our approach achieves high computational efficiency and outperforms the other state-of-the-art methods. These advancements position V2V3D as a promising solution for 3D imaging under challenging conditions.

Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge

Riccardo Cantini,Alessio Orsino,Massimo Ruggiero,Domenico Talia

Task: 提出一个可扩展的基准框架，用于评估大型语言模型（LLMs）在对抗性偏见引发方面的鲁棒性。

Motivation: 大型语言模型在关键社会领域的广泛应用引发了对其嵌入偏见的担忧，这些偏见可能延续刻板印象并损害公平性。

Details

Method: 通过多任务方法系统性探测模型在不同社会文化维度上的偏见，使用LLM-as-a-Judge方法量化鲁棒性，并采用越狱技术研究安全机制的漏洞。 Result: 分析揭示了小型和大型先进模型中普遍存在的偏见及其对模型安全性的影响，并评估了针对关键领域（如医学）微调的领域特定模型的安全性。 Conclusion: 研究发现模型规模与安全性之间存在关键权衡，有助于开发更公平和更鲁棒的未来语言模型。 Abstract: Large Language Models (LLMs) have revolutionized artificial intelligence, driving advancements in machine translation, summarization, and conversational agents. However, their increasing integration into critical societal domains has raised concerns about embedded biases, which can perpetuate stereotypes and compromise fairness. These biases stem from various sources, including historical inequalities in training data, linguistic imbalances, and adversarial manipulation. Despite mitigation efforts, recent studies indicate that LLMs remain vulnerable to adversarial attacks designed to elicit biased responses. This work proposes a scalable benchmarking framework to evaluate LLM robustness against adversarial bias elicitation. Our methodology involves (i) systematically probing models with a multi-task approach targeting biases across various sociocultural dimensions, (ii) quantifying robustness through safety scores using an LLM-as-a-Judge approach for automated assessment of model responses, and (iii) employing jailbreak techniques to investigate vulnerabilities in safety mechanisms. Our analysis examines prevalent biases in both small and large state-of-the-art models and their impact on model safety. Additionally, we assess the safety of domain-specific models fine-tuned for critical fields, such as medicine. Finally, we release a curated dataset of bias-related prompts, CLEAR-Bias, to facilitate systematic vulnerability benchmarking. Our findings reveal critical trade-offs between model size and safety, aiding the development of fairer and more robust future language models.

SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videos

Joshua Li,Fernando Jose Pena Cantu,Emily Yu,Alexander Wong,Yuchen Cui,Yuhao Chen

Task: 提出一种零样本的视频场景图生成方法SAMJAM，用于动态厨房环境。

Motivation: 现有视频场景图生成模型需要大量训练，而视觉语言模型在动态场景中难以保持稳定的对象身份。

Details

Method: 结合SAM2的时间跟踪和Gemini的语义理解，通过匹配算法生成时间一致的场景图。 Result: 在EPIC-KITCHENS和EPIC-KITCHENS-100数据集上，SAMJAM比Gemini的平均召回率提高了8.33%。 Conclusion: SAMJAM是一种有效的零样本方法，适用于动态环境中的视频场景图生成。 Abstract: Video Scene Graph Generation (VidSGG) is an important topic in understanding dynamic kitchen environments. Current models for VidSGG require extensive training to produce scene graphs. Recently, Vision Language Models (VLM) and Vision Foundation Models (VFM) have demonstrated impressive zero-shot capabilities in a variety of tasks. However, VLMs like Gemini struggle with the dynamics for VidSGG, failing to maintain stable object identities across frames. To overcome this limitation, we propose SAMJAM, a zero-shot pipeline that combines SAM2's temporal tracking with Gemini's semantic understanding. SAM2 also improves upon Gemini's object grounding by producing more accurate bounding boxes. In our method, we first prompt Gemini to generate a frame-level scene graph. Then, we employ a matching algorithm to map each object in the scene graph with a SAM2-generated or SAM2-propagated mask, producing a temporally-consistent scene graph in dynamic environments. Finally, we repeat this process again in each of the following frames. We empirically demonstrate that SAMJAM outperforms Gemini by 8.33% in mean recall on the EPIC-KITCHENS and EPIC-KITCHENS-100 datasets.

Redefining Machine Translation on Social Network Services with Large Language Models

Hongcheng Guo,Fei Zhao,Shaosheng Cao,Xinze Lyu,Ziyan Liu,Yue Wang,Boyang Wang,Zhoujun Li,Chonggang Lu,Zhe Xu,Yao Hu

Task: 开发一个专门用于社交网络服务（SNS）翻译的大型语言模型RedTrans，解决传统模型在文化相关内容（如梗、俚语和流行文化）上的翻译不足。

Motivation: 全球化社交互动增加了对社交网络服务翻译的需求，但传统模型在文化相关内容上表现不佳，且缺乏专门的训练数据和评估基准。

Details

Method: 提出了RedTrans模型，通过三种创新方法训练：1）双LLM反向翻译采样的监督微调；2）改写偏好优化算法（RePO）；3）首个SNS翻译基准RedTrans-Bench。 Result: 实验表明RedTrans优于现有大型语言模型，并已在真实生产环境中部署。 Conclusion: 领域特定适配能有效弥合通用翻译系统与文化相关翻译系统之间的差距。 Abstract: The globalization of social interactions has heightened the need for machine translation (MT) on Social Network Services (SNS), yet traditional models struggle with culturally nuanced content like memes, slang, and pop culture references. While large language models (LLMs) have advanced general-purpose translation, their performance on SNS-specific content remains limited due to insufficient specialized training data and evaluation benchmarks. This paper introduces RedTrans, a 72B LLM tailored for SNS translation, trained on a novel dataset developed through three innovations: (1) Supervised Finetuning with Dual-LLM Back-Translation Sampling, an unsupervised sampling method using LLM-based back-translation to select diverse data for large-scale finetuning; (2) Rewritten Preference Optimization (RePO), an algorithm that identifies and corrects erroneous preference pairs through expert annotation, building reliable preference corpora; and (3) RedTrans-Bench, the first benchmark for SNS translation, evaluating phenomena like humor localization, emoji semantics, and meme adaptation. Experiments show RedTrans outperforms state-of-the-art LLMs. Besides, RedTrans has already been deployed in a real-world production environment, demonstrating that domain-specific adaptation, effectively bridges the gap between generic and culturally grounded translation systems.

SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement

Xiyao Wang,Zhengyuan Yang,Chao Feng,Hongjin Lu,Linjie Li,Chung-Ching Lin,Kevin Lin,Furong Huang,Lijuan Wang

Task: 提出一种基于自改进的方法，通过量化样本难度并筛选少量高质量训练数据，显著提升视觉推理模型的性能。

Motivation: 在少量训练样本下提升视觉推理能力，避免知识蒸馏的依赖，关键在于量化样本难度以实现有效数据筛选。

Details

Method: 利用蒙特卡洛树搜索（MCTS）量化样本难度，筛选出11k高质量样本进行强化微调（RFT），训练模型ThinkLite-VL。 Result: ThinkLite-VL在8个基准测试中平均性能提升7%，在MathVista上达到75.1%的SoTA准确率，超越多个大模型。 Conclusion: 通过量化样本难度和筛选高质量数据，ThinkLite-VL在少量样本下显著提升了视觉推理性能，证明了方法的有效性。 Abstract: In this paper, we present an effective method to enhance visual reasoning with significantly fewer training samples, relying purely on self-improvement with no knowledge distillation. Our key insight is that the difficulty of training data during reinforcement fine-tuning (RFT) is critical. Appropriately challenging samples can substantially boost reasoning capabilities even when the dataset is small. Despite being intuitive, the main challenge remains in accurately quantifying sample difficulty to enable effective data filtering. To this end, we propose a novel way of repurposing Monte Carlo Tree Search (MCTS) to achieve that. Starting from our curated 70k open-source training samples, we introduce an MCTS-based selection method that quantifies sample difficulty based on the number of iterations required by the VLMs to solve each problem. This explicit step-by-step reasoning in MCTS enforces the model to think longer and better identifies samples that are genuinely challenging. We filter and retain 11k samples to perform RFT on Qwen2.5-VL-7B-Instruct, resulting in our final model, ThinkLite-VL. Evaluation results on eight benchmarks show that ThinkLite-VL improves the average performance of Qwen2.5-VL-7B-Instruct by 7%, using only 11k training samples with no knowledge distillation. This significantly outperforms all existing 7B-level reasoning VLMs, and our fairly comparable baselines that use classic selection methods such as accuracy-based filtering. Notably, on MathVista, ThinkLite-VL-7B achieves the SoTA accuracy of 75.1, surpassing Qwen2.5-VL-72B, GPT-4o, and O1. Our code, data, and model are available at https://github.com/si0wang/ThinkLite-VL.

FG-RAG: Enhancing Query-Focused Summarization with Context-Aware Fine-Grained Graph RAG

Yubin Hong,Chaofan Li,Jingyi Zhang,Yingxia Shao

Task: 提出一种名为Context-Aware Fine-Grained Graph RAG (FG-RAG)的方法，以提升Query-Focused Summarization (QFS)任务的性能。

Motivation: 现有的GraphRAG方法在QFS任务中主要关注粗粒度信息摘要，缺乏对特定查询的感知，且检索内容缺乏足够的上下文信息。

Details

Method: FG-RAG采用Context-Aware Entity Expansion扩展图中实体的覆盖范围，并提供足够的上下文信息；同时利用Query-Level Fine-Grained Summarization在生成响应时融入细粒度细节。 Result: FG-RAG在QFS任务中，在全面性、多样性和赋能性等多个指标上优于其他RAG系统。 Conclusion: FG-RAG通过上下文感知和细粒度摘要，显著提升了QFS任务的性能。 Abstract: Retrieval-Augmented Generation (RAG) enables large language models to provide more precise and pertinent responses by incorporating external knowledge. In the Query-Focused Summarization (QFS) task, GraphRAG-based approaches have notably enhanced the comprehensiveness and diversity of generated responses. However, existing GraphRAG-based approaches predominantly focus on coarse-grained information summarization without being aware of the specific query, and the retrieved content lacks sufficient contextual information to generate comprehensive responses. To address the deficiencies of current RAG systems, we propose Context-Aware Fine-Grained Graph RAG (FG-RAG) to enhance the performance of the QFS task. FG-RAG employs Context-Aware Entity Expansion in graph retrieval to expand the coverage of retrieved entities in the graph, thus providing enough contextual information for the retrieved content. Furthermore, FG-RAG utilizes Query-Level Fine-Grained Summarization to incorporate fine-grained details during response generation, enhancing query awareness for the generated summarization. Our evaluation demonstrates that FG-RAG outperforms other RAG systems in multiple metrics of comprehensiveness, diversity, and empowerment when handling the QFS task. Our implementation is available at https://github.com/BuptWululu/FG-RAG.

Beyond the Frame: Generating 360° Panoramic Videos from Perspective Videos

Rundong Luo,Matthew Wallingford,Ali Farhadi,Noah Snavely,Wei-Chiu Ma

Task: 研究如何从普通视角视频生成全景360度视频。

Motivation: 360度视频能更完整地呈现动态视觉世界，但现有视频模型在生成全景视频方面仍有不足。

Details

Method: 利用在线360度视频资源，设计高质量数据过滤流程，并结合几何和运动感知操作优化生成过程。 Result: 模型能够从普通视频生成真实且连贯的360度视频，并展示了潜在应用。 Conclusion: 提出的方法在360度视频生成任务中表现优异，具有广泛的应用前景。 Abstract: 360{\deg} videos have emerged as a promising medium to represent our dynamic visual world. Compared to the "tunnel vision" of standard cameras, their borderless field of view offers a more complete perspective of our surroundings. While existing video models excel at producing standard videos, their ability to generate full panoramic videos remains elusive. In this paper, we investigate the task of video-to-360{\deg} generation: given a perspective video as input, our goal is to generate a full panoramic video that is consistent with the original video. Unlike conventional video generation tasks, the output's field of view is significantly larger, and the model is required to have a deep understanding of both the spatial layout of the scene and the dynamics of objects to maintain spatio-temporal consistency. To address these challenges, we first leverage the abundant 360{\deg} videos available online and develop a high-quality data filtering pipeline to curate pairwise training data. We then carefully design a series of geometry- and motion-aware operations to facilitate the learning process and improve the quality of 360{\deg} video generation. Experimental results demonstrate that our model can generate realistic and coherent 360{\deg} videos from in-the-wild perspective video. In addition, we showcase its potential applications, including video stabilization, camera viewpoint control, and interactive visual question answering.

Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking

Will LeVine,Bijan Varjavand

Task: 研究如何通过多标准优化改进检索增强生成（RAG）系统的性能。

Motivation: 传统的RAG系统仅优化上下文相关性，可能导致信息瓶颈并降低生成回答的质量。

Details

Method: 提出REBEL方法，通过多标准优化（如思维链提示和多轮对话）改进RAG系统。 Result: 实验表明，REBEL能够在增加推理时间的同时提高上下文相关性和回答质量。 Conclusion: REBEL为RAG系统提供了一种新的性能与速度权衡曲线，显著提升了系统表现。 Abstract: Modern Large Language Model (LLM) systems typically rely on Retrieval Augmented Generation (RAG) which aims to gather context that is useful for response generation. These RAG systems typically optimize strictly towards retrieving context that is maximally relevant to the query. However, conventional theory suggests that retrieval systems which seek to maximize context relevance without any additional explicit criteria can create information bottlenecks. We reaffirm this finding in the modern age of LLM's by showing that in standard RAG pipelines, maximizing for context relevance alone can degrade downstream response quality. In response, we show evaluations of existing RAG methods which account for both context relevance and answer quality. These evaluations introduce a novel finding that existing RAG systems scale poorly with inference time compute usage when considering our combined metric. We introduce "RErank BEyond reLevance (REBEL)", which enables RAG systems to scale with inference-time compute via injection of multi-criteria optimization using Chain-of-Thought prompting (and optionally Multi-Turn dialogue). Ultimately, this enables a new performance/speed tradeoff curve, where RAG systems are able to achieve both higher relevance of retrieved contexts and superior answer quality as inference time increases. Code for the implementation of our method in llama-index can be found at the following PR: https://github.com/run-llama/llama_index/pull/17590. Code for running experiments using this llama-index implementation can be found at https://github.com/microsoft/REBEL.

MARS: a Multimodal Alignment and Ranking System for Few-Shot Segmentation

Nico Catalano,Stefano Samele,Paolo Pertino,Matteo Matteucci

Task: 提出一种名为MARS的插件式排名系统，用于改进少样本分割任务中的掩码选择方法。

Motivation: 当前少样本分割文献缺乏超越视觉相似性的掩码选择方法，导致预测结果不理想。

Details

Method: 利用多模态线索对掩码提议进行评分、过滤和合并，通过局部和全局层面的多模态评分来评估提议。 Result: 在多个数据集上的实验表明，整合所有四个评分组件对稳健排名至关重要，MARS能够与多种掩码提议系统无缝集成，并在多个基准测试中取得新的最优结果。 Conclusion: MARS是一种有效的插件式排名系统，能够显著提升少样本分割任务的性能。 Abstract: Current Few Shot Segmentation literature lacks a mask selection method that goes beyond visual similarity between the query and example images, leading to suboptimal predictions. We present MARS, a plug-and-play ranking system that leverages multimodal cues to filter and merge mask proposals robustly. Starting from a set of mask predictions for a single query image, we score, filter, and merge them to improve results. Proposals are evaluated using multimodal scores computed at local and global levels. Extensive experiments on COCO-20i, Pascal-5i, LVIS-92i, and FSS-1000 demonstrate that integrating all four scoring components is crucial for robust ranking, validating our contribution. As MARS can be effortlessly integrated with various mask proposal systems, we deploy it across a wide range of top-performer methods and achieve new state-of-the-art results on multiple existing benchmarks. Code will be available upon acceptance.

OSCAR: Online Soft Compression And Reranking

Maxime Louis,Thibault Formal,Hervé Dejean,Stéphane Clinchant

Task: 提出一种名为OSCAR的查询依赖在线软压缩方法，以减少RAG管道的计算开销。

Motivation: 随着检索规模的增大，RAG管道的计算成本变得昂贵，需要一种高效的方法来压缩检索信息。

Details

Method: OSCAR是一种动态压缩检索信息的在线软压缩方法，同时支持重排序。 Result: 实验表明，OSCAR在1B到24B参数的LLMs上实现了2-5倍的推理加速，且精度损失极小。 Conclusion: OSCAR是一种高效且性能优越的RAG管道优化方法。 Abstract: Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge, leading to improved accuracy and relevance. However, scaling RAG pipelines remains computationally expensive as retrieval sizes grow. To address this, we introduce OSCAR, a novel query-dependent online soft compression method that reduces computational overhead while preserving performance. Unlike traditional hard compression methods, which shorten retrieved texts, or soft compression approaches, which map documents to continuous embeddings offline, OSCAR dynamically compresses retrieved information at inference time, eliminating storage overhead and enabling higher compression rates. Additionally, we extend OSCAR to simultaneously perform reranking, further optimizing the efficiency of the RAG pipeline. Our experiments demonstrate state-of-the-art performance with a 2-5x speed-up in inference and minimal to no loss in accuracy for LLMs ranging from 1B to 24B parameters. The models are available at: https://huggingface.co/collections/naver/oscar-67d446a8e3a2551f57464295.

HoloPart: Generative 3D Part Amodal Segmentation

Yunhan Yang,Yuan-Chen Guo,Yukun Huang,Zi-Xin Zou,Zhipeng Yu,Yangguang Li,Yan-Pei Cao,Xihui Liu

Task: 在3D形状中实现部分的无遮挡分割（amodal segmentation），即使部分被遮挡也能分解出完整的语义部分。

Motivation: 现有的3D部分分割方法仅能识别可见的表面区域，限制了其应用价值。

Details

Method: 提出了一种两阶段方法：首先利用现有3D部分分割技术获取初始不完整部分，然后通过新型扩散模型HoloPart补全这些部分。 Result: HoloPart在ABO和PartObjaverse-Tiny数据集上显著优于现有形状补全方法。 Conclusion: 结合HoloPart与现有分割技术，为3D部分无遮挡分割开辟了新途径，适用于几何编辑、动画和材质分配等应用。 Abstract: 3D part amodal segmentation--decomposing a 3D shape into complete, semantically meaningful parts, even when occluded--is a challenging but crucial task for 3D content creation and understanding. Existing 3D part segmentation methods only identify visible surface patches, limiting their utility. Inspired by 2D amodal segmentation, we introduce this novel task to the 3D domain and propose a practical, two-stage approach, addressing the key challenges of inferring occluded 3D geometry, maintaining global shape consistency, and handling diverse shapes with limited training data. First, we leverage existing 3D part segmentation to obtain initial, incomplete part segments. Second, we introduce HoloPart, a novel diffusion-based model, to complete these segments into full 3D parts. HoloPart utilizes a specialized architecture with local attention to capture fine-grained part geometry and global shape context attention to ensure overall shape consistency. We introduce new benchmarks based on the ABO and PartObjaverse-Tiny datasets and demonstrate that HoloPart significantly outperforms state-of-the-art shape completion methods. By incorporating HoloPart with existing segmentation techniques, we achieve promising results on 3D part amodal segmentation, opening new avenues for applications in geometry editing, animation, and material assignment.

Proposed 2MW Wind Turbine for Use in the Governorate of Dhofar at the Sultanate of Oman

Osama Ahmed Marzouk,Omar Rashid Hamdan Al Badi,Maadh Hamed Salman Al Rashdi,Hamed Mohammed Eid Al Balushi

Task: 为阿曼Dhofar风电场项目设计一款水平轴风力涡轮机（HAWT）。

Motivation: 作为海湾合作委员会（GCC）地区首个商业化、公用事业规模（50MW）的风电场项目，需要一款高效的风力涡轮机以满足电力需求。

Details

Method: 通过研究阿曼的风图确定最大平均风速（6m/s），并利用MATLAB代码建模匹配设计变量与目标电力输出。 Result: 设计出一款3叶片、直径70米、转速24rpm的涡轮机，输出功率2.37MW，超过目标2MW。 Conclusion: 设计满足需求，并考虑了齿轮箱和发电机的功率损耗，为Dhofar风电场提供了可行的解决方案。 Abstract: In this work, we propose a preliminary design of a horizontal-axis wind turbine (HAWT) as a candidate for the Dhofar Wind Farm project, in the southern Omani Governorate "Dhofar", at the southwest part of the Sultanate of Oman. This wind farm (under construction) is considered to be the first commercial, utility-scale (50MW) wind farm in the GCC (Gulf Cooperation Council) area. The proposed wind turbine has an expected electricity generation of 2MW. We studied the wind atlas of Oman and from which we determined the maximum possible mean wind speed in the entire Sultanate and built our design based on that reference value, which is 6m/s (21.6km/h). After this, we applied a set of modeling equations that estimate the power output from the wind turbine rotor and matched the target electric power to the design variables using a MATLAB computer code. We reached a suitable design and we present here the distribution of the blade angle (twist angle), and the power per unit span along the rotor blade. The rotor design has 3 blades with a diameter of 70m and a rotational speed of 24rpm. This rotor gives 2.37MW of output power, which exceeds the target 2MW output, allowing for about 15% of power losses in the gearbox and generator. We utilized some commercial designs of wind turbines from different international manufacturers as references for typical limits or recommended values of some design parameters.

GenEAva: Generating Cartoon Avatars with Fine-Grained Facial Expressions from Realistic Diffusion-based Faces

Hao Yu,Rupayan Mallick,Margrit Betke,Sarah Adel Bargal

Task: 提出一种名为GenEAva的新框架，用于生成高质量、具有细粒度面部表情的卡通头像。

Motivation: 现有卡通头像数据集和生成方法难以呈现高度表达性的头像，且常基于真实身份，引发隐私问题。

Details

Method: 通过微调最先进的文本到图像扩散模型，结合风格化模型，生成并转换真实面部为卡通头像。 Result: 创建了首个表达性卡通头像数据集GenEAva 1.0，包含13,230个头像，覆盖135种细粒度表情，并在性别、种族和年龄上分布均衡。模型生成的卡通头像比SDXL更具表达性且不包含训练数据中的身份信息。 Conclusion: GenEAva框架和数据集为未来卡通头像生成研究提供了多样化和表达性的基准。 Abstract: Cartoon avatars have been widely used in various applications, including social media, online tutoring, and gaming. However, existing cartoon avatar datasets and generation methods struggle to present highly expressive avatars with fine-grained facial expressions and are often inspired from real-world identities, raising privacy concerns. To address these challenges, we propose a novel framework, GenEAva, for generating high-quality cartoon avatars with fine-grained facial expressions. Our approach fine-tunes a state-of-the-art text-to-image diffusion model to synthesize highly detailed and expressive facial expressions. We then incorporate a stylization model that transforms these realistic faces into cartoon avatars while preserving both identity and expression. Leveraging this framework, we introduce the first expressive cartoon avatar dataset, GenEAva 1.0, specifically designed to capture 135 fine-grained facial expressions, featuring 13,230 expressive cartoon avatars with a balanced distribution across genders, racial groups, and age ranges. We demonstrate that our fine-tuned model generates more expressive faces than the state-of-the-art text-to-image diffusion model SDXL. We also verify that the cartoon avatars generated by our framework do not include memorized identities from fine-tuning data. The proposed framework and dataset provide a diverse and expressive benchmark for future research in cartoon avatar generation.

Holistic Capability Preservation: Towards Compact Yet Comprehensive Reasoning Models

Ling Team,Caizhi Tang,Chilin Fu,Chunwei Wu,Jia Guo,Jianwen Wang,Jingyu Hu,Liang Jiang,Meng Li,Peng Jiao,Pingping Liu,Shaomian Zheng,Shiwei Liang,Shuaicheng Li,Yalin Zhang,Yingting Wu,Yongkang Liu,Zhenyu Huang

Task: 开发一个轻量级的推理模型Ring-Lite-Distill，基于开源的Ling-Lite模型，通过高质量数据训练实现高效推理能力。

Motivation: 通过精心设计的数据和训练方法，提升轻量级模型的推理能力，同时保持参数效率，覆盖更全面的任务能力。

Details

Method: 采用高质量数据筛选和创新的训练范式，进一步训练Ling-Lite模型，优化其推理能力。 Result: Ring-Lite-Distill的推理能力达到DeepSeek-R1-Distill-Qwen-7B水平，通用能力显著超越后者。 Conclusion: Ring-Lite-Distill展示了轻量级模型在高效推理和通用能力上的潜力。 Abstract: This technical report presents Ring-Lite-Distill, a lightweight reasoning model derived from our open-source Mixture-of-Experts (MoE) Large Language Models (LLMs) Ling-Lite. This study demonstrates that through meticulous high-quality data curation and ingenious training paradigms, the compact MoE model Ling-Lite can be further trained to achieve exceptional reasoning capabilities, while maintaining its parameter-efficient architecture with only 2.75 billion activated parameters, establishing an efficient lightweight reasoning architecture. In particular, in constructing this model, we have not merely focused on enhancing advanced reasoning capabilities, exemplified by high-difficulty mathematical problem solving, but rather aimed to develop a reasoning model with more comprehensive competency coverage. Our approach ensures coverage across reasoning tasks of varying difficulty levels while preserving generic capabilities, such as instruction following, tool use, and knowledge retention. We show that, Ring-Lite-Distill's reasoning ability reaches a level comparable to DeepSeek-R1-Distill-Qwen-7B, while its general capabilities significantly surpass those of DeepSeek-R1-Distill-Qwen-7B. The models are accessible at https://huggingface.co/inclusionAI

InteractAvatar: Modeling Hand-Face Interaction in Photorealistic Avatars with Deformable Gaussians

Kefan Chen,Sergiu Oprea,Justin Theiss,Sreyas Mohan,Srinath Sridhar,Aayush Prakash

Task: 提出InteracttAvatar模型，用于高保真地捕捉动态手部与非刚性手-脸交互的光照真实外观。

Motivation: 随着数字化身在通信中的重要性增加，建模自然化身行为成为多个行业的重要挑战，尤其是手部与身体的交互常被忽视。

Details

Method: 结合模板模型、3D高斯泼溅和动态细化模块的Dynamic Gaussian Hand模型，以及手-脸交互模块。 Result: 通过实验证明，InteracttAvatar能够从单目或多视角视频中高保真重建手部及手-脸交互，并支持新姿势动画。 Conclusion: InteracttAvatar是首个能忠实捕捉动态手部与非刚性手-脸交互光照真实外观的模型。 Abstract: With the rising interest from the community in digital avatars coupled with the importance of expressions and gestures in communication, modeling natural avatar behavior remains an important challenge across many industries such as teleconferencing, gaming, and AR/VR. Human hands are the primary tool for interacting with the environment and essential for realistic human behavior modeling, yet existing 3D hand and head avatar models often overlook the crucial aspect of hand-body interactions, such as between hand and face. We present InteracttAvatar, the first model to faithfully capture the photorealistic appearance of dynamic hand and non-rigid hand-face interactions. Our novel Dynamic Gaussian Hand model, combining template model and 3D Gaussian Splatting as well as a dynamic refinement module, captures pose-dependent change, e.g. the fine wrinkles and complex shadows that occur during articulation. Importantly, our hand-face interaction module models the subtle geometry and appearance dynamics that underlie common gestures. Through experiments of novel view synthesis, self reenactment and cross-identity reenactment, we demonstrate that InteracttAvatar can reconstruct hand and hand-face interactions from monocular or multiview videos with high-fidelity details and be animated with novel poses.

R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents

Naman Jain,Jaskirat Singh,Manish Shetty,Liang Zheng,Koushik Sen,Ion Stoica

Task: 改进开源模型在真实世界软件工程任务（解决GitHub问题）上的性能。

Motivation: 面临两个关键挑战：1）可扩展的执行环境训练模型；2）测试时计算的最优扩展。

Details

Method: 引入AgentGym，一个程序化生成的可执行训练环境，包含超过8.7K任务，并采用SYNGEN（合成数据生成方法）和混合测试时扩展策略。 Result: 在SWE-Bench Verified基准测试中，32B模型达到34.4%的pass@1性能，混合测试时扩展策略将性能提升至51%，达到开源模型的新最优水平。 Conclusion: AgentGym及其方法显著提升了开源模型在软件工程任务中的性能，首次与专有模型竞争。 Abstract: Improving open-source models on real-world SWE tasks (solving GITHUB issues) faces two key challenges: 1) scalable curation of execution environments to train these models, and, 2) optimal scaling of test-time compute. We introduce AgentGym, the largest procedurally-curated executable gym environment for training real-world SWE-agents, consisting of more than 8.7K tasks. AgentGym is powered by two main contributions: 1) SYNGEN: a synthetic data curation recipe that enables scalable curation of executable environments using test-generation and back-translation directly from commits, thereby reducing reliance on human-written issues or unit tests. We show that this enables more scalable training leading to pass@1 performance of 34.4% on SWE-Bench Verified benchmark with our 32B model. 2) Hybrid Test-time Scaling: we provide an in-depth analysis of two test-time scaling axes; execution-based and execution-free verifiers, demonstrating that they exhibit complementary strengths and limitations. Test-based verifiers suffer from low distinguishability, while execution-free verifiers are biased and often rely on stylistic features. Surprisingly, we find that while each approach individually saturates around 42-43%, significantly higher gains can be obtained by leveraging their complementary strengths. Overall, our approach achieves 51% on the SWE-Bench Verified benchmark, reflecting a new state-of-the-art for open-weight SWE-agents and for the first time showing competitive performance with proprietary models such as o1, o1-preview and sonnet-3.5-v2 (with tools). We will open-source our environments, models, and agent trajectories.

Scaling Laws for Native Multimodal Models Scaling Laws for Native Multimodal Models

Mustafa Shukor,Enrico Fini,Victor Guilherme Turrisi da Costa,Matthieu Cord,Joshua Susskind,Alaaeldin El-Nouby

Task: 研究原生多模态模型（NMMs）的架构设计，比较早期融合与晚期融合架构的性能差异。

Motivation: 探讨晚期融合架构是否天生优于早期融合架构，并验证早期融合架构在性能和效率上的优势。

Details

Method: 通过457个不同架构和训练混合的模型进行扩展规律研究，并引入专家混合（MoEs）以增强性能。 Result: 早期融合架构在低参数量下表现更强，训练效率更高且更易于部署；MoEs显著提升了性能。 Conclusion: 早期融合架构在多模态模型中具有优势，结合MoEs可进一步提升性能。 Abstract: Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal. Current approaches involve integrating separately pre-trained components, such as connecting vision encoders to LLMs and continuing multimodal training. While such approaches exhibit remarkable sample efficiency, it remains an open question whether such late-fusion architectures are inherently superior. In this work, we revisit the architectural design of native multimodal models (NMMs)--those trained from the ground up on all modalities--and conduct an extensive scaling laws study, spanning 457 trained models with different architectures and training mixtures. Our investigation reveals no inherent advantage to late-fusion architectures over early-fusion ones, which do not rely on image encoders. On the contrary, early-fusion exhibits stronger performance at lower parameter counts, is more efficient to train, and is easier to deploy. Motivated by the strong performance of the early-fusion architectures, we show that incorporating Mixture of Experts (MoEs) allows for models that learn modality-specific weights, significantly enhancing performance.

Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression

Hanqi Xiao,Yi-Lin Sung,Elias Stengel-Eskin,Mohit Bansal

Task: 开发一种新的混合精度后训练量化方法（TaCQ），通过直接量化过程与特定权重电路关联，以保持下游任务性能。

Motivation: 后训练量化（PTQ）在低比特（2-3位）设置下会显著降低模型性能，需要一种方法在保持性能的同时减少内存占用。

Details

Method: 提出Task-Circuit Quantization（TaCQ），通过对比未量化模型与均匀量化模型，利用梯度信息预测量化对任务性能的影响，保留任务相关权重为16位。 Result: TaCQ在2-3位量化设置下优于现有方法，如在3.1位下恢复Llama-3-8B-Instruct 96%的MMLU性能，2位设置下平均提升14.74%。 Conclusion: TaCQ能够有效识别并保留重要权重，在低比特量化中显著提升性能，且不限于任务特定设置。 Abstract: Post-training quantization (PTQ) reduces a model's memory footprint by mapping full precision weights into low bit weights without costly retraining, but can degrade its downstream performance especially in low 2- to 3-bit settings. We develop a new mixed-precision PTQ approach, Task-Circuit Quantization (TaCQ), that draws parallels to automated circuit discovery, directly conditioning the quantization process on specific weight circuits -- which we define as sets of weights associated with downstream task performance. These weights are kept as 16-bit weights, while others are quantized, maintaining performance while only adding a marginal memory cost. Specifically, TaCQ contrasts unquantized model weights with a uniformly-quantized model to estimate the expected change in weights due to quantization and uses gradient information to predict the resulting impact on task performance, allowing us to preserve task-specific weights. We compare TaCQ-based quantization to existing mixed-precision quantization methods when conditioning both on general-purpose and task-specific data. Across QA, math reasoning, and text-to-SQL tasks for both Llama-3 and Qwen2.5, we find that TaCQ outperforms baselines using the same calibration data and a lower weight budget, achieving major improvements in the 2 and 3-bit regime. With only 3.1 bits we are able to recover 96% of Llama-3-8B-Instruct's unquantized 16-bit MMLU performance, obtaining a 5.25% absolute improvement over SPQR. We also observe consistently large gains over existing methods in the 2-bit regime, with an average gain of 14.74% over the strongest baseline, SliM-LLM. Moreover, we observe a 7.20% gain without conditioning on specific tasks, showing TaCQ's ability to identify important weights is not limited to task-conditioned settings.

Perception-R1: Pioneering Perception Policy with Reinforcement Learning

En Yu,Kangheng Lin,Liang Zhao,Jisheng Yin,Yana Wei,Yuang Peng,Haoran Wei,Jianjian Sun,Chunrui Han,Zheng Ge,Xiangyu Zhang,Daxin Jiang,Jingyu Wang,Wenbing Tao

Task: 探索基于规则的强化学习（RL）在多模态大语言模型（MLLM）后训练中对视觉感知策略学习的影响。

Motivation: 尽管初步实验显示强化学习在视觉感知任务中表现不一致，但研究希望深入探讨RL在视觉感知中的核心作用及其影响因素。

Details

Method: 提出Perception-R1框架，利用GRPO算法在MLLM后训练中优化感知策略，并分析感知复杂度和奖励设计对RL效果的影响。 Result: Perception-R1在多个视觉感知任务中取得显著提升，如RefCOCO+（+4.2%）、PixMo-Count（+17.9%）和COCO2017 val（31.9% AP）。 Conclusion: 感知复杂度和奖励设计是决定RL效果的关键因素，Perception-R1为感知策略学习提供了强有力的基准。 Abstract: Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in MLLM post-training for perception policy learning. While promising, our initial experiments reveal that incorporating a thinking process through RL does not consistently lead to performance gains across all visual perception tasks. This leads us to delve into the essential role of RL in the context of visual perception. In this work, we return to the fundamentals and explore the effects of RL on different perception tasks. We observe that the perceptual complexity is a major factor in determining the effectiveness of RL. We also observe that reward design plays a crucial role in further approching the upper limit of model perception. To leverage these findings, we propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training. With a standard Qwen2.5-VL-3B-Instruct, Perception-R1 achieves +4.2% on RefCOCO+, +17.9% on PixMo-Count, +4.2% on PageOCR, and notably, 31.9% AP on COCO2017 val for the first time, establishing a strong baseline for perception policy learning.

Leveraging LLMs for Multimodal Retrieval-Augmented Radiology Report Generation via Key Phrase Extraction

Kyoyun Choi,Byungmu Yoon,Soobum Kim,Jonggwon Park

Task: 提出一种基于检索增强生成的方法，用于自动生成放射学报告，以减少幻觉并降低计算需求。

Motivation: 多模态大语言模型（MLLMs）资源密集，需要大量数据和计算成本，因此需要一种更高效的方法。

Details

Method: 结合多模态检索和大语言模型，提取关键短语，采用图像编码器结构搜索、文本嵌入噪声添加和对比学习等策略。 Result: 在MIMIC-CXR数据集上取得了CheXbert指标的先进结果和RadGraph F1指标的竞争力，且无需微调大语言模型。 Conclusion: 该方法在多视图放射学报告生成中表现出强大的泛化能力，适合临床广泛应用。 Abstract: Automated radiology report generation (RRG) holds potential to reduce radiologists' workload, especially as recent advancements in large language models (LLMs) enable the development of multimodal models for chest X-ray (CXR) report generation. However, multimodal LLMs (MLLMs) are resource-intensive, requiring vast datasets and substantial computational cost for training. To address these challenges, we propose a retrieval-augmented generation approach that leverages multimodal retrieval and LLMs to generate radiology reports while mitigating hallucinations and reducing computational demands. Our method uses LLMs to extract key phrases from radiology reports, effectively focusing on essential diagnostic information. Through exploring effective training strategies, including image encoder structure search, adding noise to text embeddings, and additional training objectives, we combine complementary pre-trained image encoders and adopt contrastive learning between text and semantic image embeddings. We evaluate our approach on MIMIC-CXR dataset, achieving state-of-the-art results on CheXbert metrics and competitive RadGraph F1 metric alongside MLLMs, without requiring LLM fine-tuning. Our method demonstrates robust generalization for multi-view RRG, making it suitable for comprehensive clinical applications.

BoxDreamer: Dreaming Box Corners for Generalizable Object Pose Estimation

Yuanhong Yu,Xingyi He,Chen Zhao,Junhao Yu,Jiaqi Yang,Ruizhen Hu,Yujun Shen,Xing Zhu,Xiaowei Zhou,Sida Peng

Task: 提出一种基于RGB的通用方法，用于解决稀疏视角下的物体姿态估计问题。

Motivation: 现有方法在遮挡和稀疏参考视角下的泛化能力有限，限制了其实际应用。

Details

Method: 通过引入物体边界框的角点作为中间表示，结合参考基的点合成器估计目标视角的2D角点，利用PnP算法建立2D-3D对应关系。 Result: 在YCB-Video和Occluded-LINEMOD数据集上的实验表明，该方法优于现有技术，显著提升了泛化能力。 Conclusion: 提出的表示方法有效增强了物体姿态估计的泛化能力，对实际应用至关重要。 Abstract: This paper presents a generalizable RGB-based approach for object pose estimation, specifically designed to address challenges in sparse-view settings. While existing methods can estimate the poses of unseen objects, their generalization ability remains limited in scenarios involving occlusions and sparse reference views, restricting their real-world applicability. To overcome these limitations, we introduce corner points of the object bounding box as an intermediate representation of the object pose. The 3D object corners can be reliably recovered from sparse input views, while the 2D corner points in the target view are estimated through a novel reference-based point synthesizer, which works well even in scenarios involving occlusions. As object semantic points, object corners naturally establish 2D-3D correspondences for object pose estimation with a PnP algorithm. Extensive experiments on the YCB-Video and Occluded-LINEMOD datasets show that our approach outperforms state-of-the-art methods, highlighting the effectiveness of the proposed representation and significantly enhancing the generalization capabilities of object pose estimation, which is crucial for real-world applications.

RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Radiology with Zero-Shot Multi-Task Capability

Jonggwon Park,Soobum Kim,Byungmu Yoon,Kyoyun Choi

Task: 提出RadZero框架，用于解决放射学中视觉-语言对齐的挑战，并实现零样本多任务能力。

Motivation: 现有方法在利用复杂放射学报告、处理高分辨率图像和提供注意力机制可解释性方面存在不足。

Details

Method: RadZero利用大型语言模型提取语义句子，采用多正对比学习策略，结合预训练视觉编码器和可训练Transformer层，通过相似性计算实现零样本推理。 Result: 在公开胸部X光基准测试中，RadZero在零样本分类、定位和分割任务上优于现有方法，并展示了跨模态相似性映射的解释潜力。 Conclusion: RadZero在医学影像中展现出高效性和可解释性，支持开放词汇语义分割。 Abstract: Recent advancements in multi-modal models have significantly improved vision-language alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning, rely on low-resolution images, and offer limited interpretability in attention mechanisms. To address these challenges, we introduce RadZero, a novel similarity-based cross-attention framework for vision-language alignment in radiology with zero-shot multi-task capability. RadZero leverages large language models to extract minimal semantic sentences from radiology reports and employs a multi-positive contrastive learning strategy to effectively capture relationships between images and multiple relevant textual descriptions. It also utilizes a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity between text embeddings and local image patch features, RadZero enables zero-shot inference with similarity probability for classification and pixel-level cross-modal similarity maps for grounding and segmentation. Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. Furthermore, cross-modal similarity map analysis highlights its potential for improving explainability in vision-language alignment. Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging.

VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning

Yukun Qi,Yiming Zhao,Yu Zeng,Xikun Bao,Wenxuan Huang,Lin Chen,Zehui Chen,Jie Zhao,Zhongang Qi,Feng Zhao

Task: 提出VCR-Bench，一个用于全面评估大型视觉语言模型（LVLMs）视频链式思维推理能力的新基准。

Motivation: 当前视频基准无法充分评估推理过程或区分感知与推理能力的缺陷，因此需要一种更严格的评估框架。

Details

Method: VCR-Bench包含859个视频和1,034个高质量问答对，每个问答对附带逐步标注的链式思维推理过程，并设计七个任务维度和CoT评分。 Result: 实验显示当前LVLMs表现有限，最佳模型CoT评分仅62.8%，准确率56.7%，多数模型评分低于40%。感知能力是主要瓶颈。 Conclusion: VCR-Bench可作为标准化评估框架，揭示复杂视频推理任务中的实际缺陷，并验证链式思维推理的重要性。 Abstract: The advancement of Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs) and large vision-language models (LVLMs). However, a rigorous evaluation framework for video CoT reasoning remains absent. Current video benchmarks fail to adequately assess the reasoning process and expose whether failures stem from deficiencies in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a novel benchmark designed to comprehensively evaluate LVLMs' Video Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos spanning a variety of video content and durations, along with 1,034 high-quality question-answer pairs. Each pair is manually annotated with a stepwise CoT rationale, where every step is tagged to indicate its association with the perception or reasoning capabilities. Furthermore, we design seven distinct task dimensions and propose the CoT score to assess the entire CoT process based on the stepwise tagged CoT rationals. Extensive experiments on VCR-Bench highlight substantial limitations in current LVLMs. Even the top-performing model, o1, only achieves a 62.8% CoT score and an 56.7% accuracy, while most models score below 40%. Experiments show most models score lower on perception than reasoning steps, revealing LVLMs' key bottleneck in temporal-spatial information processing for complex video reasoning. A robust positive correlation between the CoT score and accuracy confirms the validity of our evaluation framework and underscores the critical role of CoT reasoning in solving complex video reasoning tasks. We hope VCR-Bench to serve as a standardized evaluation framework and expose the actual drawbacks in complex video reasoning task.

LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking

Qi Liu,Haozhe Duan,Yiqun Chen,Quanfeng Lu,Weiwei Sun,Jiaxin Mao

Task: 提出一个统一的框架LLM4Ranking，用于利用开源或闭源API的大语言模型（LLMs）进行文档重排序。

Motivation: 近年来，利用大语言模型进行文档重排序成为热门研究方向，但缺乏统一的框架支持不同方法的应用和评估。

Details

Method: 设计了一个简单且可扩展的框架，提供文档重排序、评估和微调脚本，支持多种LLMs。 Result: 在多个广泛使用的数据集上评估了不同模型和方法，提供了可复现的结果。 Conclusion: LLM4Ranking框架为文档重排序任务提供了实用工具，支持研究和实际应用。 Abstract: Utilizing large language models (LLMs) for document reranking has been a popular and promising research direction in recent years, many studies are dedicated to improving the performance and efficiency of using LLMs for reranking. Besides, it can also be applied in many real-world applications, such as search engines or retrieval-augmented generation. In response to the growing demand for research and application in practice, we introduce a unified framework, \textbf{LLM4Ranking}, which enables users to adopt different ranking methods using open-source or closed-source API-based LLMs. Our framework provides a simple and extensible interface for document reranking with LLMs, as well as easy-to-use evaluation and fine-tuning scripts for this task. We conducted experiments based on this framework and evaluated various models and methods on several widely used datasets, providing reproducibility results on utilizing LLMs for document reranking. Our code is publicly available at https://github.com/liuqi6777/llm4ranking.

MM-IFEngine: Towards Multimodal Instruction Following

Shengyuan Ding,Shenxi Wu,Xiangyu Zhao,Yuhang Zang,Haodong Duan,Xiaoyi Dong,Pan Zhang,Yuhang Cao,Dahua Lin,Jiaqi Wang

Task: 提出MM-IFEngine管道以生成高质量图像-指令对，并构建MM-IFEval多模态指令跟随基准。

Motivation: 解决现有多模态指令跟随训练数据稀缺、基准简单且评估策略不精确的问题。

Details

Method: 开发MM-IFEngine生成大规模、多样化的训练数据MM-IFInstruct-23k和MM-IFDPO-23k，并设计MM-IFEval基准。 Result: 微调MLLMs在MM-IFInstruct-23k和MM-IFDPO-23k上显著提升了多个IF基准的性能。 Conclusion: MM-IFEngine和MM-IFEval为多模态指令跟随任务提供了高质量数据和评估框架。 Abstract: The Instruction Following (IF) ability measures how well Multi-modal Large Language Models (MLLMs) understand exactly what users are telling them and whether they are doing it right. Existing multimodal instruction following training data is scarce, the benchmarks are simple with atomic instructions, and the evaluation strategies are imprecise for tasks demanding exact output constraints. To address this, we present MM-IFEngine, an effective pipeline to generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for Direct Preference Optimization (DPO). We further introduce MM-IFEval, a challenging and diverse multi-modal instruction-following benchmark that includes (1) both compose-level constraints for output responses and perception-level constraints tied to the input images, and (2) a comprehensive evaluation pipeline incorporating both rule-based assessment and judge model. We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF benchmarks, such as MM-IFEval (+10.2$\%$), MIA (+7.6$\%$), and IFEval (+12.3$\%$). The full data and evaluation code will be released on https://github.com/SYuan03/MM-IFEngine.

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

Juzheng Zhang,Jiacheng You,Ashwinee Panda,Tom Goldstein

Task: 提出一种名为LoRI的低秩适应方法，以减少参数干扰并提升多任务场景下的性能。

Motivation: LoRA在多任务场景中存在参数干扰和计算开销问题，需要一种更高效的参数微调方法。

Details

Method: 冻结投影矩阵A为随机投影，并通过任务特定掩码稀疏化矩阵B，利用子空间正交性减少干扰。 Result: LoRI在多项任务中表现优于全微调和现有PEFT方法，参数减少95%。 Conclusion: LoRI是一种高效且性能优越的参数微调方法，适用于多任务和持续学习场景。 Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https://github.com/juzhengz/LoRI

Detect Anything 3D in the Wild

Hanxue Zhang,Haoran Jiang,Qingsong Yao,Yanan Sun,Renrui Zhang,Hao Zhao,Hongyang Li,Hongzi Zhu,Zetong Yang

Task: 提出DetAny3D，一种可提示的3D检测基础模型，用于在任意相机配置下检测任何新物体。

Motivation: 现有深度学习方法在零样本泛化到新物体和相机配置方面存在困难，且3D标注数据有限。

Details

Method: 利用预训练的2D基础模型的知识，通过2D聚合器和3D解释器模块实现2D到3D的知识迁移。 Result: DetAny3D在未见类别和新相机配置上表现优异，并在领域内数据上超越多数竞争对手。 Conclusion: DetAny3D展示了3D基础模型在现实场景中的潜力，为开放世界中的3D任务提供了探索方向。 Abstract: Despite the success of deep learning in close-set 3D object detection, existing approaches struggle with zero-shot generalization to novel objects and camera configurations. We introduce DetAny3D, a promptable 3D detection foundation model capable of detecting any novel object under arbitrary camera configurations using only monocular inputs. Training a foundation model for 3D detection is fundamentally constrained by the limited availability of annotated 3D data, which motivates DetAny3D to leverage the rich prior knowledge embedded in extensively pre-trained 2D foundation models to compensate for this scarcity. To effectively transfer 2D knowledge to 3D, DetAny3D incorporates two core modules: the 2D Aggregator, which aligns features from different 2D foundation models, and the 3D Interpreter with Zero-Embedding Mapping, which mitigates catastrophic forgetting in 2D-to-3D knowledge transfer. Experimental results validate the strong generalization of our DetAny3D, which not only achieves state-of-the-art performance on unseen categories and novel camera configurations, but also surpasses most competitors on in-domain data.DetAny3D sheds light on the potential of the 3D foundation model for diverse applications in real-world scenarios, e.g., rare object detection in autonomous driving, and demonstrates promise for further exploration of 3D-centric tasks in open-world settings. More visualization results can be found at DetAny3D project page.

VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model

Haozhan Shen,Peng Liu,Jingcheng Li,Chunxin Fang,Yibo Ma,Jiajia Liao,Qiaoli Shen,Zilun Zhang,Kangjia Zhao,Qianqian Zhang,Ruochen Xu,Tiancheng Zhao

Task: 研究如何通过强化学习（RL）提升视觉语言模型（VLMs）的视觉推理能力。

Motivation: 视觉理解任务通常具有明确的真实标注，适合基于规则的奖励机制，这与DeepSeek R1在语言模型中的成功类似。

Details

Method: 开发了VLM-R1框架，将R1风格的强化学习扩展到视觉语言模型，并通过实验验证其可行性。 Result: RL模型在视觉理解任务中表现优异，泛化能力超越监督微调（SFT），并揭示了奖励黑客、训练数据质量等关键发现。 Conclusion: 强化学习能有效增强视觉语言模型的能力，研究结果为视觉语言RL领域的进一步发展提供了支持。 Abstract: Recently DeepSeek R1 has shown that reinforcement learning (RL) can substantially improve the reasoning capabilities of Large Language Models (LLMs) through a simple yet effective design. The core of R1 lies in its rule-based reward formulation, which leverages tasks with deterministic ground-truth answers to enable precise and stable reward computation. In the visual domain, we similarly observe that a wide range of visual understanding tasks are inherently equipped with well-defined ground-truth annotations. This property makes them naturally compatible with rule-based reward mechanisms. Motivated by this observation, we investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs), aiming to enhance their visual reasoning capabilities. To this end, we develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks. Using this framework, we further explore the feasibility of applying RL to visual domain. Experimental results indicate that the RL-based model not only delivers competitive performance on visual understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in generalization ability. Furthermore, we conduct comprehensive ablation studies that uncover a series of noteworthy insights, including the presence of reward hacking in object detection, the emergence of the "OD aha moment", the impact of training data quality, and the scaling behavior of RL across different model sizes. Through these analyses, we aim to deepen the understanding of how reinforcement learning enhances the capabilities of vision-language models, and we hope our findings and open-source contributions will support continued progress in the vision-language RL community. Our code and model are available at https://github.com/om-ai-lab/VLM-R1

CCMNet: Leveraging Calibrated Color Correction Matrices for Cross-Camera Color Constancy

Dongyoung Kim,Mahmoud Afifi,Dongyun Kim,Michael S. Brown,Seon Joo Kim

Task: 提出一种基于学习的方法，实现跨相机的颜色恒常性，无需重新训练即可适应新相机。

Motivation: 解决相机特定原始色彩空间中白平衡算法需要适应不同相机的问题。

Details

Method: 利用ISP中预校准的颜色校正矩阵（CCMs）将预定义光照颜色映射到测试相机的原始空间，并通过相机指纹嵌入（CFE）实现网络对新相机的适应。 Result: 在多个数据集和骨干网络上实现了最先进的跨相机颜色恒常性，且方法轻量级且仅依赖ISP中现成的数据。 Conclusion: 该方法有效解决了跨相机颜色恒常性问题，具有实际应用价值。 Abstract: Computational color constancy, or white balancing, is a key module in a camera's image signal processor (ISP) that corrects color casts from scene lighting. Because this operation occurs in the camera-specific raw color space, white balance algorithms must adapt to different cameras. This paper introduces a learning-based method for cross-camera color constancy that generalizes to new cameras without retraining. Our method leverages pre-calibrated color correction matrices (CCMs) available on ISPs that map the camera's raw color space to a standard space (e.g., CIE XYZ). Our method uses these CCMs to transform predefined illumination colors (i.e., along the Planckian locus) into the test camera's raw space. The mapped illuminants are encoded into a compact camera fingerprint embedding (CFE) that enables the network to adapt to unseen cameras. To prevent overfitting due to limited cameras and CCMs during training, we introduce a data augmentation technique that interpolates between cameras and their CCMs. Experimental results across multiple datasets and backbones show that our method achieves state-of-the-art cross-camera color constancy while remaining lightweight and relying only on data readily available in camera ISPs.

CollEX -- A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collections

Florian Schneider,Narges Baba Ahmadi,Niloufar Baba Ahmadi,Iris Vogel,Martin Semmann,Chris Biemann

Task: 介绍并验证CollEx，一种多模态代理增强检索生成系统，用于增强对大规模科学文献集的交互式探索。

Motivation: 传统搜索系统在面对庞大且复杂的科学文献集时缺乏直观性和交互性，阻碍了学习者、教育者和研究者的使用。

Details

Method: 利用先进的大型视觉语言模型（LVLMs）作为多模态代理，通过直观的聊天界面实现复杂交互的抽象化。 Result: CollEx显著简化了对多样化科学文献集的访问，支持教育场景并促进跨学科连接。 Conclusion: CollEx通过多模态集成和代理技术，有效提升了科学文献的探索体验，适用于教育和研究场景。 Abstract: In this paper, we introduce CollEx, an innovative multimodal agentic Retrieval-Augmented Generation (RAG) system designed to enhance interactive exploration of extensive scientific collections. Given the overwhelming volume and inherent complexity of scientific collections, conventional search systems often lack necessary intuitiveness and interactivity, presenting substantial barriers for learners, educators, and researchers. CollEx addresses these limitations by employing state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents accessible through an intuitive chat interface. By abstracting complex interactions via specialized agents equipped with advanced tools, CollEx facilitates curiosity-driven exploration, significantly simplifying access to diverse scientific collections and records therein. Our system integrates textual and visual modalities, supporting educational scenarios that are helpful for teachers, pupils, students, and researchers by fostering independent exploration as well as scientific excitement and curiosity. Furthermore, CollEx serves the research community by discovering interdisciplinary connections and complementing visual data. We illustrate the effectiveness of our system through a proof-of-concept application containing over 64,000 unique records across 32 collections from a local scientific collection from a public university.

VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning

Zhong-Yu Li,Ruoyi Du,Juncheng Yan,Le Zhuo,Zhen Li,Peng Gao,Zhanyu Ma,Ming-Ming Cheng

Task: 提出一种通用的图像生成框架VisualCloze，支持多种任务。

Motivation: 当前任务特定模型效率有限，通用模型面临任务指令泛化、任务分布和统一架构设计的挑战。

Details

Method: 结合视觉上下文学习，利用Graph200K数据集增强任务密度和可迁移知识，并利用图像填充模型的生成先验。 Result: VisualCloze支持广泛任务，包括未见任务和多任务统一，且无需修改架构。 Conclusion: VisualCloze解决了通用图像生成模型的挑战，具有高效性和泛化能力。 Abstract: Recent progress in diffusion models significantly advances various image generation tasks. However, the current mainstream approach remains focused on building task-specific models, which have limited efficiency when supporting a wide range of different needs. While universal models attempt to address this limitation, they face critical challenges, including generalizable task instruction, appropriate task distributions, and unified architectural design. To tackle these challenges, we propose VisualCloze, a universal image generation framework, which supports a wide range of in-domain tasks, generalization to unseen ones, unseen unification of multiple tasks, and reverse generation. Unlike existing methods that rely on language-based task instruction, leading to task ambiguity and weak generalization, we integrate visual in-context learning, allowing models to identify tasks from visual demonstrations. Meanwhile, the inherent sparsity of visual task distributions hampers the learning of transferable knowledge across tasks. To this end, we introduce Graph200K, a graph-structured dataset that establishes various interrelated tasks, enhancing task density and transferable knowledge. Furthermore, we uncover that our unified image generation formulation shared a consistent objective with image infilling, enabling us to leverage the strong generative priors of pre-trained infilling models without modifying the architectures.

Zero-Shot Cross-Domain Code Search without Fine-Tuning

Keyu Liang,Zhongxin Liu,Chao Liu,Zhiyuan Wan,David Lo,Xiaohu Yang

Task: 提出一种无需微调的零样本跨领域代码搜索方法CodeBridge。

Motivation: 解决预训练语言模型在跨领域代码搜索中需要高成本微调或性能下降的问题。

Details

Method: 将查询-代码匹配过程分解为查询-注释匹配和代码-代码匹配，利用大语言模型生成注释和伪代码，结合相似性评分和采样融合。 Result: 在三个数据集上平均MRR分别优于CoCoSoDa和UniXcoder 21.4%和24.9%，且与需要微调的RAPID效果相当。 Conclusion: CodeBridge是一种高效且无需微调的零样本跨领域代码搜索方法。 Abstract: Code search aims to retrieve semantically relevant code snippets for natural language queries. While pre-trained language models (PLMs) have shown remarkable performance in this task, they struggle in cross-domain scenarios, often requiring costly fine-tuning or facing performance drops in zero-shot settings. RAPID, which generates synthetic data for model fine-tuning, is currently the only effective method for zero-shot cross-domain code search. Despite its effectiveness, RAPID demands substantial computational resources for fine-tuning and needs to maintain specialized models for each domain, underscoring the need for a zero-shot, fine-tuning-free approach for cross-domain code search. The key to tackling zero-shot cross-domain code search lies in bridging the gaps among domains. In this work, we propose to break the query-code matching process of code search into two simpler tasks: query-comment matching and code-code matching. Our empirical study reveals the strong complementarity among the three matching schemas in zero-shot cross-domain settings, i.e., query-code, query-comment, and code-code matching. Based on the findings, we propose CodeBridge, a zero-shot, fine-tuning-free approach for cross-domain code search. Specifically, CodeBridge uses Large Language Models (LLMs) to generate comments and pseudo-code, then combines query-code, query-comment, and code-code matching via PLM-based similarity scoring and sampling-based fusion. Experimental results show that our approach outperforms the state-of-the-art PLM-based code search approaches, i.e., CoCoSoDa and UniXcoder, by an average of 21.4% and 24.9% in MRR, respectively, across three datasets. Our approach also yields results that are better than or comparable to those of the zero-shot cross-domain code search approach RAPID, which requires costly fine-tuning.

Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction

Zeren Jiang,Chuanxia Zheng,Iro Laina,Diane Larlus,Andrea Vedaldi

Task: 提出Geo4D方法，将视频扩散模型用于单目动态场景的3D重建。

Motivation: 利用视频模型捕获的动态先验，仅需合成数据训练即可在真实数据上实现零样本泛化。

Details

Method: 通过预测点、深度和射线图等几何模态，结合多模态对齐算法和滑动窗口融合，实现鲁棒的4D重建。 Result: 在多个基准测试中显著超越现有方法，包括专为动态场景设计的MonST3R。 Conclusion: Geo4D在动态场景的4D重建中表现出色，具有泛化能力强和准确性高的特点。 Abstract: We introduce Geo4D, a method to repurpose video diffusion models for monocular 3D reconstruction of dynamic scenes. By leveraging the strong dynamic prior captured by such video models, Geo4D can be trained using only synthetic data while generalizing well to real data in a zero-shot manner. Geo4D predicts several complementary geometric modalities, namely point, depth, and ray maps. It uses a new multi-modal alignment algorithm to align and fuse these modalities, as well as multiple sliding windows, at inference time, thus obtaining robust and accurate 4D reconstruction of long videos. Extensive experiments across multiple benchmarks show that Geo4D significantly surpasses state-of-the-art video depth estimation methods, including recent methods such as MonST3R, which are also designed to handle dynamic scenes.

Deceptive Automated Interpretability: Language Models Coordinating to Fool Oversight Systems

Simon Lermen,Mateusz Dziemian,Natalia Pérez-Campanero Antolín

Task: 研究AI代理如何通过自动神经网络可解释性协调欺骗监督系统。

Motivation: 探索语言模型是否能够生成欺骗性解释以逃避检测，并研究其背后的动机和机制。

Details

Method: 使用稀疏自编码器（SAEs）作为实验框架，测试语言模型（Llama、DeepSeek R1和Claude 3.7 Sonnet）生成欺骗性解释的能力，并采用隐写术隐藏信息。 Result: 所有测试的语言模型代理都能成功欺骗监督模型，同时生成的可解释性评分与参考标签相当。 Conclusion: 提出缓解策略，强调需要建立强大的理解和防御机制以应对欺骗行为。 Abstract: We demonstrate how AI agents can coordinate to deceive oversight systems using automated interpretability of neural networks. Using sparse autoencoders (SAEs) as our experimental framework, we show that language models (Llama, DeepSeek R1, and Claude 3.7 Sonnet) can generate deceptive explanations that evade detection. Our agents employ steganographic methods to hide information in seemingly innocent explanations, successfully fooling oversight models while achieving explanation quality comparable to reference labels. We further find that models can scheme to develop deceptive strategies when they believe the detection of harmful features might lead to negative consequences for themselves. All tested LLM agents were capable of deceiving the overseer while achieving high interpretability scores comparable to those of reference labels. We conclude by proposing mitigation strategies, emphasizing the critical need for robust understanding and defenses against deception.

GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation

Lang Lin,Xueyang Yu,Ziqi Pang,Yu-Xiong Wang

Task: 提出一种利用多模态大语言模型（MLLMs）进行参考视频目标分割（RefVOS）的新框架。

Motivation: 解决现有MLLM方法在全局推理（理解关键帧）和局部推理（连续帧目标跟踪）之间的两难问题，避免依赖外部VOS或帧选择器。

Details

Method: 提出GLUS框架，通过稀疏的“上下文帧”提供全局信息，连续的“查询帧”进行局部目标跟踪，并结合预训练的VOS记忆库联合训练。 Result: 在MeViS和Ref-Youtube-VOS基准测试中达到新的最先进水平。 Conclusion: GLUS框架简单有效，统一了全局和局部一致性，为MLLMs在RefVOS任务中提供了新的基准。 Abstract: This paper proposes a novel framework utilizing multi-modal large language models (MLLMs) for referring video object segmentation (RefVOS). Previous MLLM-based methods commonly struggle with the dilemma between "Ref" and "VOS": they either specialize in understanding a few key frames (global reasoning) or tracking objects on continuous frames (local reasoning), and rely on external VOS or frame selectors to mitigate the other end of the challenge. However, our framework GLUS shows that global and local consistency can be unified into a single video segmentation MLLM: a set of sparse "context frames" provides global information, while a stream of continuous "query frames" conducts local object tracking. This is further supported by jointly training the MLLM with a pre-trained VOS memory bank to simultaneously digest short-range and long-range temporal information. To improve the information efficiency within the limited context window of MLLMs, we introduce object contrastive learning to distinguish hard false-positive objects and a self-refined framework to identify crucial frames and perform propagation. By collectively integrating these insights, our GLUS delivers a simple yet effective baseline, achieving new state-of-the-art for MLLMs on the MeViS and Ref-Youtube-VOS benchmark. Our project page is at https://glus-video.github.io/.

Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines

Cansu Koyuturk,Emily Theophilou,Sabrina Patania,Gregor Donabauer,Andrea Martinenghi,Chiara Antico,Alessia Telari,Alessia Testa,Sathya Bursic,Franca Garzotto,Davinia Hernandez-Leo,Udo Kruschwitz,Davide Taibi,Simona Amenta,Martin Ruskov,Dimitri Ognibene

Task: 研究如何通过结构化提示指导提升用户与大型语言模型（LLMs）的交互效果。

Motivation: 尽管LLMs在自然语言交互中表现出色，但用户常因提示不准确而获得低效响应，现有研究指出LLMs和用户在此方面的局限性。

Details

Method: 通过教育实验比较三种提示指导方法（任务特定框架和两种基线方法），分析642次交互数据，使用Von NeuMidas标注模式分类错误和行为模式。 Result: 研究发现结构化提示指导能显著改善用户行为、提示策略遵循度及AI响应质量。 Conclusion: 结构化提示指导能有效提升用户与LLMs的交互能力，对AI素养、聊天机器人可用性及响应式AI系统设计具有启示。 Abstract: Large Language Models (LLMs) have transformed human-computer interaction by enabling natural language-based communication with AI-powered chatbots. These models are designed to be intuitive and user-friendly, allowing users to articulate requests with minimal effort. However, despite their accessibility, studies reveal that users often struggle with effective prompting, resulting in inefficient responses. Existing research has highlighted both the limitations of LLMs in interpreting vague or poorly structured prompts and the difficulties users face in crafting precise queries. This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting. We introduce and compare three types of prompting guidelines: a task-specific framework developed through a structured methodology and two baseline approaches. To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users. Using Von NeuMidas, an extended pragmatic annotation schema for LLM interaction analysis, we categorize common prompting errors and identify recurring behavioral patterns. We then evaluate the impact of different guidelines by examining changes in user behavior, adherence to prompting strategies, and the overall quality of AI-generated responses. Our findings provide a deeper understanding of how users engage with LLMs and the role of structured prompting guidance in enhancing AI-assisted communication. By comparing different instructional frameworks, we offer insights into more effective approaches for improving user competency in AI interactions, with implications for AI literacy, chatbot usability, and the design of more responsive AI systems.

PixelFlow: Pixel-Space Generative Models with Flow

Shoufa Chen,Chongjian Ge,Shilong Zhang,Peize Sun,Ping Luo

Task: 提出一种直接在原始像素空间操作的图像生成模型PixelFlow。

Motivation: 简化图像生成过程，避免使用预训练的变分自编码器（VAE），并使整个模型可端到端训练。

Details

Method: 通过高效的级联流建模，在像素空间中实现可负担的计算成本。 Result: 在256×256 ImageNet类条件图像生成基准测试中，FID达到1.98，并在文本到图像生成中表现出卓越的图像质量、艺术性和语义控制能力。 Conclusion: PixelFlow为下一代视觉生成模型提供了新的范式，并展示了其在图像生成领域的潜力。 Abstract: We present PixelFlow, a family of image generation models that operate directly in the raw pixel space, in contrast to the predominant latent-space models. This approach simplifies the image generation process by eliminating the need for a pre-trained Variational Autoencoder (VAE) and enabling the whole model end-to-end trainable. Through efficient cascade flow modeling, PixelFlow achieves affordable computation cost in pixel space. It achieves an FID of 1.98 on 256$\times$256 ImageNet class-conditional image generation benchmark. The qualitative text-to-image results demonstrate that PixelFlow excels in image quality, artistry, and semantic control. We hope this new paradigm will inspire and open up new opportunities for next-generation visual generation models. Code and models are available at https://github.com/ShoufaChen/PixelFlow.

Dual Engines of Thoughts: A Depth-Breadth Integration Framework for Open-Ended Analysis

Fei-Hsuan Yu,Yun-Cheng Chou,Teng-Ruei Chen

Task: 提出一种名为Dual Engines of Thoughts (DEoT)的分析框架，用于全面的开放式推理。

Motivation: 传统推理框架主要关注单一答案问题的最佳或正确答案，而DEoT专为开放式问题设计，支持更广泛和更深入的分析探索。

Details

Method: 框架包含三个关键组件：Base Prompter（优化用户查询）、Solver Agent（协调任务分解、执行和验证）和Dual-Engine System（Breadth Engine探索多样性因素，Depth Engine进行深度研究）。 Result: 实验结果表明，DEoT在解决复杂多面问题方面表现优异，相比现有推理模型，总胜率为77-86%。 Conclusion: DEoT在平衡广泛覆盖和深度分析方面表现出色，具有高度可定制性，适用于实际应用。 Abstract: We propose the Dual Engines of Thoughts (DEoT), an analytical framework for comprehensive open-ended reasoning. While traditional reasoning frameworks primarily focus on finding "the best answer" or "the correct answer" for single-answer problems, DEoT is specifically designed for "open-ended questions," enabling both broader and deeper analytical exploration. The framework centers on three key components: a Base Prompter for refining user queries, a Solver Agent that orchestrates task decomposition, execution, and validation, and a Dual-Engine System consisting of a Breadth Engine (to explore diverse impact factors) and a Depth Engine (to perform deep investigations). This integrated design allows DEoT to balance wide-ranging coverage with in-depth analysis, and it is highly customizable, enabling users to adjust analytical parameters and tool configurations based on specific requirements. Experimental results show that DEoT excels in addressing complex, multi-faceted questions, achieving a total win rate of 77-86% compared to existing reasoning models, thus highlighting its effectiveness in real-world applications.

Boundary representation learning via Transformer

Qiang Zou,Lizhen Zhu

Task: 将Transformer网络应用于边界表示（B-rep）模型的学习，提出了一种名为边界表示Transformer（BRT）的新方法。

Motivation: 尽管Transformer在自然语言处理、计算机视觉等领域取得了显著成功，但在计算机辅助设计（CAD）中处理B-rep模型的应用仍未被充分探索。

Details

Method: BRT提出了一种连续几何嵌入方法，将B-rep表面编码为Bézier三角形，并采用拓扑感知嵌入方法将这些几何嵌入组织为适合Transformer的离散标记序列。 Result: 实验表明，BRT在零件分类和特征识别任务中达到了最先进的性能。 Conclusion: BRT成功地将Transformer应用于B-rep模型学习，解决了其不规则拓扑和连续几何定义带来的挑战。 Abstract: The recent rise of generative artificial intelligence (AI), powered by Transformer networks, has achieved remarkable success in natural language processing, computer vision, and graphics. However, the application of Transformers in computer-aided design (CAD), particularly for processing boundary representation (B-rep) models, remains largely unexplored. To bridge this gap, this paper introduces Boundary Representation Transformer (BRT), a novel method adapting Transformer for B-rep learning. B-rep models pose unique challenges due to their irregular topology and continuous geometric definitions, which are fundamentally different from the structured and discrete data Transformers are designed for. To address this, BRT proposes a continuous geometric embedding method that encodes B-rep surfaces (trimmed and untrimmed) into B\'ezier triangles, preserving their shape and continuity without discretization. Additionally, BRT employs a topology-aware embedding method that organizes these geometric embeddings into a sequence of discrete tokens suitable for Transformers, capturing both geometric and topological characteristics within B-rep models. This enables the Transformer's attention mechanism to effectively learn shape patterns and contextual semantics of boundary elements in a B-rep model. Extensive experiments demonstrate that BRT achieves state-of-the-art performance in part classification and feature recognition tasks.

How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective

Qi Liu,Jiaxin Mao,Ji-Rong Wen

Task: 系统研究大型语言模型（LLMs）如何通过模块化机制理解并实现相关性判断。

Motivation: 探索现成LLMs内部机制中相关性判断的工作原理，填补相关研究的空白。

Details

Method: 采用激活修补技术，分析不同模型组件的作用，揭示生成相关性判断的多阶段渐进过程。 Result: 发现LLMs在早期层提取查询和文档信息，中间层处理相关性信息，后期层通过特定注意力头生成相关性判断。 Conclusion: 研究揭示了LLMs相关性评估的机制，为未来利用LLMs进行信息检索任务提供了重要启示。 Abstract: Recent studies have shown that large language models (LLMs) can assess relevance and support information retrieval (IR) tasks such as document ranking and relevance judgment generation. However, the internal mechanisms by which off-the-shelf LLMs understand and operationalize relevance remain largely unexplored. In this paper, we systematically investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability. Using activation patching techniques, we analyze the roles of various model components and identify a multi-stage, progressive process in generating either pointwise or pairwise relevance judgment. Specifically, LLMs first extract query and document information in the early layers, then process relevance information according to instructions in the middle layers, and finally utilize specific attention heads in the later layers to generate relevance judgments in the required format. Our findings provide insights into the mechanisms underlying relevance assessment in LLMs, offering valuable implications for future research on leveraging LLMs for IR tasks.

MESA: Text-Driven Terrain Generation Using Latent Diffusion and Global Copernicus Data

Paul Borne--Pons,Mikolaj Czerkawski,Rosalie Martin,Romain Rouffet

Task: 通过训练扩散模型从文本描述生成高质量地形样本。

Motivation: 传统地形建模依赖程序化技术，需要大量领域知识和手工规则，而数据驱动的方法可以更灵活和可扩展。

Details

Method: 使用全球遥感数据训练扩散模型（MESA），生成地形样本。 Result: 模型能够生成逼真且多样化的地形景观，并发布了Major TOM Core-DEM扩展数据集作为资源。 Conclusion: 基于遥感数据的数据驱动模型为地形建模和生成提供了强大工具。 Abstract: Terrain modeling has traditionally relied on procedural techniques, which often require extensive domain expertise and handcrafted rules. In this paper, we present MESA - a novel data-centric alternative by training a diffusion model on global remote sensing data. This approach leverages large-scale geospatial information to generate high-quality terrain samples from text descriptions, showcasing a flexible and scalable solution for terrain generation. The model's capabilities are demonstrated through extensive experiments, highlighting its ability to generate realistic and diverse terrain landscapes. The dataset produced to support this work, the Major TOM Core-DEM extension dataset, is released openly as a comprehensive resource for global terrain data. The results suggest that data-driven models, trained on remote sensing data, can provide a powerful tool for realistic terrain modeling and generation.

Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory

Mirac Suzgun,Mert Yuksekgonul,Federico Bianchi,Dan Jurafsky,James Zou

Task: 提出一种名为Dynamic Cheatsheet（DC）的轻量级框架，为黑盒语言模型提供持久且动态演化的记忆能力。

Motivation: 当前语言模型在处理输入时缺乏记忆能力，无法保留和复用之前的解决方案或错误，导致效率低下。

Details

Method: 通过DC框架，模型能够在推理时存储和复用积累的策略、代码片段和问题解决思路，无需显式标注或人工反馈。 Result: 实验表明，DC显著提升了模型在数学考试、算术任务和知识密集型任务中的性能，例如Claude 3.5 Sonnet在AIME数学考试中的准确率翻倍，GPT-4o在Game of 24任务中的成功率从10%提升至99%。 Conclusion: DC是一种有前景的方法，能够为语言模型提供持久记忆，缩小孤立推理事件与人类累积经验学习之间的差距。 Abstract: Despite their impressive performance on complex tasks, current language models (LMs) typically operate in a vacuum: Each input query is processed separately, without retaining insights from previous attempts. Here, we present Dynamic Cheatsheet (DC), a lightweight framework that endows a black-box LM with a persistent, evolving memory. Rather than repeatedly re-discovering or re-committing the same solutions and mistakes, DC enables models to store and reuse accumulated strategies, code snippets, and general problem-solving insights at inference time. This test-time learning enhances performance substantially across a range of tasks without needing explicit ground-truth labels or human feedback. Leveraging DC, Claude 3.5 Sonnet's accuracy more than doubled on AIME math exams once it began retaining algebraic insights across questions. Similarly, GPT-4o's success rate on Game of 24 increased from 10% to 99% after the model discovered and reused a Python-based solution. In tasks prone to arithmetic mistakes, such as balancing equations, DC enabled GPT-4o and Claude to reach near-perfect accuracy by recalling previously validated code, whereas their baselines stagnated around 50%. Beyond arithmetic challenges, DC yields notable accuracy gains on knowledge-demanding tasks. Claude achieved a 9% improvement in GPQA-Diamond and an 8% boost on MMLU-Pro problems. Crucially, DC's memory is self-curated, focusing on concise, transferable snippets rather than entire transcript. Unlike finetuning or static retrieval methods, DC adapts LMs' problem-solving skills on the fly, without modifying their underlying parameters. Overall, our findings present DC as a promising approach for augmenting LMs with persistent memory, bridging the divide between isolated inference events and the cumulative, experience-driven learning characteristic of human cognition.

MoEDiff-SR: Mixture of Experts-Guided Diffusion Model for Region-Adaptive MRI Super-Resolution

Zhe Wang,Yuhua Ru,Aladine Chetouani,Fang Chen,Fabian Bauer,Liping Zhang,Didier Hans,Rachid Jennane,Mohamed Jarraya,Yung Hsin Chen

Task: 提出一种基于混合专家（MoE）引导的扩散模型MoEDiff-SR，用于区域自适应的MRI超分辨率重建。

Motivation: 低场强MRI（如3T）的空间分辨率有限，难以捕捉临床诊断和神经影像研究所需的精细解剖细节。

Details

Method: MoEDiff-SR通过Transformer提取多尺度图像特征，并利用MoE门控网络动态选择多个扩散去噪专家，实现区域自适应超分辨率重建。 Result: 实验表明，MoEDiff-SR在图像质量指标、感知保真度和计算效率上优于现有方法，临床评估验证了其在诊断中的优越性。 Conclusion: MoEDiff-SR通过区域自适应去噪显著提升了MRI超分辨率重建的性能，具有临床实用价值。 Abstract: Magnetic Resonance Imaging (MRI) at lower field strengths (e.g., 3T) suffers from limited spatial resolution, making it challenging to capture fine anatomical details essential for clinical diagnosis and neuroimaging research. To overcome this limitation, we propose MoEDiff-SR, a Mixture of Experts (MoE)-guided diffusion model for region-adaptive MRI Super-Resolution (SR). Unlike conventional diffusion-based SR models that apply a uniform denoising process across the entire image, MoEDiff-SR dynamically selects specialized denoising experts at a fine-grained token level, ensuring region-specific adaptation and enhanced SR performance. Specifically, our approach first employs a Transformer-based feature extractor to compute multi-scale patch embeddings, capturing both global structural information and local texture details. The extracted feature embeddings are then fed into an MoE gating network, which assigns adaptive weights to multiple diffusion-based denoisers, each specializing in different brain MRI characteristics, such as centrum semiovale, sulcal and gyral cortex, and grey-white matter junction. The final output is produced by aggregating the denoised results from these specialized experts according to dynamically assigned gating probabilities. Experimental results demonstrate that MoEDiff-SR outperforms existing state-of-the-art methods in terms of quantitative image quality metrics, perceptual fidelity, and computational efficiency. Difference maps from each expert further highlight their distinct specializations, confirming the effective region-specific denoising capability and the interpretability of expert contributions. Additionally, clinical evaluation validates its superior diagnostic capability in identifying subtle pathological features, emphasizing its practical relevance in clinical neuroimaging. Our code is available at https://github.com/ZWang78/MoEDiff-SR.

Perception-R1: Pioneering Perception Policy with Reinforcement Learning

En Yu,Kangheng Lin,Liang Zhao,Jisheng Yin,Yana Wei,Yuang Peng,Haoran Wei,Jianjian Sun,Chunrui Han,Zheng Ge,Xiangyu Zhang,Daxin Jiang,Jingyu Wang,Wenbing Tao

Task: 探索基于规则的强化学习（RL）在多模态大语言模型（MLLM）后训练中对感知策略学习的潜力。

Motivation: 尽管初步实验显示RL在视觉感知任务中表现不一致，但研究RL在视觉感知中的本质作用及其对不同任务的差异性影响。

Details

Method: 提出Perception-R1，一个可扩展的RL框架，使用GRPO在MLLM后训练中优化感知任务。 Result: 在多个基准测试中取得显著性能提升，如RefCOCO+（+4.2%）、PixMo-Count（+17.9%）和COCO2017 val（31.9% AP）。 Conclusion: 感知任务的复杂性和奖励设计是决定RL效果的关键因素，Perception-R1为感知策略学习建立了强基线。 Abstract: Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in MLLM post-training for perception policy learning. While promising, our initial experiments reveal that incorporating a thinking process through RL does not consistently lead to performance gains across all visual perception tasks. This leads us to delve into the essential role of RL in the context of visual perception. In this work, we return to the fundamentals and explore the effects of RL on different perception tasks. We observe that the perceptual complexity is a major factor in determining the effectiveness of RL. We also observe that reward design plays a crucial role in further approching the upper limit of model perception. To leverage these findings, we propose Perception-R1, a scalable RL framework using GRPO during MLLM post-training. With a standard Qwen2.5-VL-3B-Instruct, Perception-R1 achieves +4.2% on RefCOCO+, +17.9% on PixMo-Count, +4.2% on PageOCR, and notably, 31.9% AP on COCO2017 val for the first time, establishing a strong baseline for perception policy learning.

Identifying regions of interest in whole slide images of renal cell carcinoma

Mohammed Lamine Benomar,Nesma Settouti,Eric Debreuve,Xavier Descombes,Damien Ambrosetti

Task: 开发一个全自动系统，用于在肾细胞癌（RCC）的全切片图像（WSI）中检测感兴趣区域（ROIs），以减少分析时间并辅助病理学家做出更准确的诊断。

Motivation: 组织病理学图像包含大量信息，诊断过程耗时且繁琐，因此需要自动化系统来提高效率和准确性。

Details

Method: 基于高效的纹理描述符（DRLBP）和颜色转换，提取WSI图像块的纹理特征，并通过特征选择和分类器（如SVM和基于迁移学习的深度学习模型）进行分类。 Result: SVM分类器的最佳精度为99.17%，迁移学习模型（如ResNet-50）的最高精度为98.50%，表明该方法在ROI检测中非常高效。 Conclusion: 提出的自动系统在肾癌诊断中能有效识别ROIs，具有高精度和实用性。 Abstract: The histopathological images contain a huge amount of information, which can make diagnosis an extremely timeconsuming and tedious task. In this study, we developed a completely automated system to detect regions of interest (ROIs) in whole slide images (WSI) of renal cell carcinoma (RCC), to reduce time analysis and assist pathologists in making more accurate decisions. The proposed approach is based on an efficient texture descriptor named dominant rotated local binary pattern (DRLBP) and color transformation to reveal and exploit the immense texture variability at the microscopic high magnifications level. Thereby, the DRLBPs retain the structural information and utilize the magnitude values in a local neighborhood for more discriminative power. For the classification of the relevant ROIs, feature extraction of WSIs patches was performed on the color channels separately to form the histograms. Next, we used the most frequently occurring patterns as a feature selection step to discard non-informative features. The performances of different classifiers on a set of 1800 kidney cancer patches originating from 12 whole slide images were compared and evaluated. Furthermore, the small size of the image dataset allows to investigate deep learning approach based on transfer learning for image patches classification by using deep features and fine-tuning methods. High recognition accuracy was obtained and the classifiers are efficient, the best precision result was 99.17% achieved with SVM. Moreover, transfer learning models perform well with comparable performance, and the highest precision using ResNet-50 reached 98.50%. The proposed approach results revealed a very efficient image classification and demonstrated efficacy in identifying ROIs. This study presents an automatic system to detect regions of interest relevant to the diagnosis of kidney cancer in whole slide histopathology images.

VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning

Yukun Qi,Yiming Zhao,Yu Zeng,Xikun Bao,Wenxuan Huang,Lin Chen,Zehui Chen,Jie Zhao,Zhongang Qi,Feng Zhao

Task: Introduce VCR-Bench, a benchmark for evaluating LVLMs' Video Chain-of-Thought Reasoning capabilities.

Motivation: Current video benchmarks lack a rigorous evaluation framework for video CoT reasoning, failing to assess reasoning processes or identify perception vs. reasoning deficiencies.

Details

Method: VCR-Bench includes 859 videos and 1,034 QA pairs with stepwise CoT rationales, tagged for perception or reasoning. Seven task dimensions and a CoT score are designed for evaluation. Result: Experiments reveal limitations in LVLMs, with top model o1 scoring 62.8% CoT and 56.7% accuracy; most models score below 40%, showing perception as a bottleneck. Conclusion: VCR-Bench validates the importance of CoT reasoning in video tasks and aims to standardize evaluation, exposing LVLMs' drawbacks in complex reasoning. Abstract: The advancement of Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs) and large vision-language models (LVLMs). However, a rigorous evaluation framework for video CoT reasoning remains absent. Current video benchmarks fail to adequately assess the reasoning process and expose whether failures stem from deficiencies in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a novel benchmark designed to comprehensively evaluate LVLMs' Video Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos spanning a variety of video content and durations, along with 1,034 high-quality question-answer pairs. Each pair is manually annotated with a stepwise CoT rationale, where every step is tagged to indicate its association with the perception or reasoning capabilities. Furthermore, we design seven distinct task dimensions and propose the CoT score to assess the entire CoT process based on the stepwise tagged CoT rationals. Extensive experiments on VCR-Bench highlight substantial limitations in current LVLMs. Even the top-performing model, o1, only achieves a 62.8% CoT score and an 56.7% accuracy, while most models score below 40%. Experiments show most models score lower on perception than reasoning steps, revealing LVLMs' key bottleneck in temporal-spatial information processing for complex video reasoning. A robust positive correlation between the CoT score and accuracy confirms the validity of our evaluation framework and underscores the critical role of CoT reasoning in solving complex video reasoning tasks. We hope VCR-Bench to serve as a standardized evaluation framework and expose the actual drawbacks in complex video reasoning task.

Synthetic CT Generation from Time-of-Flight Non-Attenutaion-Corrected PET for Whole-Body PET Attenuation Correction

Weijie Chen,James Wang,Alan McMillan

Task: 提出一种深度学习方法，从TOF NAC PET图像直接生成合成CT（sCT）图像，以增强PET/MR中的衰减校正。

Motivation: PET/MR系统中缺乏CT图像，而传统的衰减校正方法依赖于CT，因此需要一种替代方案。

Details

Method: 利用预训练的自然图像模型进行CT到CT重建任务，并在35对TOF NAC PET和CT数据上进行微调。 Result: 在体轮廓区域内，实现了最低的MAE（74.49 HU）和最高的PSNR（28.66 dB），视觉评估显示骨和软组织结构的重建效果改善。 Conclusion: 预训练深度学习模型在医学图像转换任务中表现优异，未来将进一步研究sCT对PET衰减校正的影响，并探索更多网络架构和数据集。 Abstract: Positron Emission Tomography (PET) imaging requires accurate attenuation correction (AC) to account for photon loss due to tissue density variations. In PET/MR systems, computed tomography (CT), which offers a straightforward estimation of AC is not available. This study presents a deep learning approach to generate synthetic CT (sCT) images directly from Time-of-Flight (TOF) non-attenuation corrected (NAC) PET images, enhancing AC for PET/MR. We first evaluated models pre-trained on large-scale natural image datasets for a CT-to-CT reconstruction task, finding that the pre-trained model outperformed those trained solely on medical datasets. The pre-trained model was then fine-tuned using an institutional dataset of 35 TOF NAC PET and CT volume pairs, achieving the lowest mean absolute error (MAE) of 74.49 HU and highest peak signal-to-noise ratio (PSNR) of 28.66 dB within the body contour region. Visual assessments demonstrated improved reconstruction of both bone and soft tissue structures from TOF NAC PET images. This work highlights the effectiveness of using pre-trained deep learning models for medical image translation tasks. Future work will assess the impact of sCT on PET attenuation correction and explore additional neural network architectures and datasets to further enhance performance and practical applications in PET imaging.

Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments

Lorenz Linhardt,Tom Neuhäuser,Lenka Tětková,Oliver Eberle

Task: 评估32个公开可用的语言模型在单词三元组任务中与人类相似性判断的表征和行为对齐。

Motivation: 中小型生成语言模型因其规模和可用性适合从行为和表征层面进行分析，研究这两个层面的交互。

Details

Method: 通过单词三元组任务，比较语言模型的表征和行为与人类相似性判断的对齐程度。 Result: (1) 小型语言模型的表征可以达到人类水平的对齐；(2) 经过指令调优的模型变体显著提高了对齐度；(3) 不同层的对齐模式高度依赖模型；(4) 基于模型行为响应的对齐度高度依赖模型规模，仅在最大模型中与表征对齐匹配。 Conclusion: 语言模型的表征和行为对齐程度受模型大小和指令调优影响，为语义关联研究提供了新视角。 Abstract: Small and mid-sized generative language models have gained increasing attention. Their size and availability make them amenable to being analyzed at a behavioral as well as a representational level, allowing investigations of how these levels interact. We evaluate 32 publicly available language models for their representational and behavioral alignment with human similarity judgments on a word triplet task. This provides a novel evaluation setting to probe semantic associations in language beyond common pairwise comparisons. We find that (1) even the representations of small language models can achieve human-level alignment, (2) instruction-tuned model variants can exhibit substantially increased agreement, (3) the pattern of alignment across layers is highly model dependent, and (4) alignment based on models' behavioral responses is highly dependent on model size, matching their representational alignment only for the largest evaluated models.

Novel Pooling-based VGG-Lite for Pneumonia and Covid-19 Detection from Imbalanced Chest X-Ray Datasets

Santanu Roy,Ashvath Suresh,Palak Sahu,Tulika Rudra Gupta

Task: 提出一种基于池化的VGG-Lite模型，以解决胸部X光（CXR）数据集中的类别不平衡问题。

Motivation: 深度学习模型在自动检测肺炎CXR图像中面临类别不平衡的挑战，尤其是在新冠变种出现后，这一问题更为突出。

Details

Method: 提出轻量级CNN模型VGG-Lite，并结合边缘增强模块（EEM），包括负图像层和新型2Max-Min池化层，作为空间注意力模块。 Result: 在两个CXR数据集上，提出的框架显著优于预训练CNN模型和其他现有模型，达到了95%的准确率、97.1%的精确率、96.1%的召回率和96.6%的F1分数。 Conclusion: VGG-Lite结合EEM的方法有效解决了类别不平衡问题，并在肺炎检测任务中表现出色。 Abstract: This paper proposes a novel pooling-based VGG-Lite model in order to mitigate class imbalance issues in Chest X-Ray (CXR) datasets. Automatic Pneumonia detection from CXR images by deep learning model has emerged as a prominent and dynamic area of research, since the inception of the new Covid-19 variant in 2020. However, the standard Convolutional Neural Network (CNN) models encounter challenges associated with class imbalance, a prevalent issue found in many medical datasets. The innovations introduced in the proposed model architecture include: (I) A very lightweight CNN model, `VGG-Lite', is proposed as a base model, inspired by VGG-16 and MobileNet-V2 architecture. (II) On top of this base model, we leverage an ``Edge Enhanced Module (EEM)" through a parallel branch, consisting of a ``negative image layer", and a novel custom pooling layer ``2Max-Min Pooling". This 2Max-Min Pooling layer is entirely novel in this investigation, providing more attention to edge components within pneumonia CXR images. Thus, it works as an efficient spatial attention module (SAM). We have implemented the proposed framework on two separate CXR datasets. The first dataset is obtained from a readily available source on the internet, and the second dataset is a more challenging CXR dataset, assembled by our research team from three different sources. Experimental results reveal that our proposed framework has outperformed pre-trained CNN models, and three recent trend existing models ``Vision Transformer", ``Pooling-based Vision Transformer (PiT)'' and ``PneuNet", by substantial margins on both datasets. The proposed framework VGG-Lite with EEM, has achieved a macro average of 95% accuracy, 97.1% precision, 96.1% recall, and 96.6% F1 score on the ``Pneumonia Imbalance CXR dataset", without employing any pre-processing technique.

PhaseGen: A Diffusion-Based Approach for Complex-Valued MRI Data Generation

Moritz Rempe,Fabian Hörst,Helmut Becker,Marco Schlimbach,Lukas Rotkopf,Kevin Kröninger,Jens Kleesiek

Task: 提出一种名为PhaseGen的复杂值扩散模型，用于生成基于临床常用幅度图像的合成MRI原始数据。

Motivation: 现有临床和基于AI的方法仅关注幅度图像，忽略了相位数据的潜在价值，而相位数据对下游任务（如肿瘤分割和分类）可能具有重要作用。

Details

Method: 开发PhaseGen模型，生成合成MRI原始数据，并评估其在k-Space中的颅骨剥离和MRI重建任务中的表现。 Result: 实验结果表明，使用合成相位数据训练显著提高了颅骨分割的准确性（从41.1%提升至80.1%），并增强了MRI重建的效果。 Conclusion: PhaseGen通过生成复杂值数据，弥合了基于幅度图像的数据集与MRI原始数据之间的差距，为更准确和高效的诊断任务提供了新途径。 Abstract: Magnetic resonance imaging (MRI) raw data, or k-Space data, is complex-valued, containing both magnitude and phase information. However, clinical and existing Artificial Intelligence (AI)-based methods focus only on magnitude images, discarding the phase data despite its potential for downstream tasks, such as tumor segmentation and classification. In this work, we introduce $\textit{PhaseGen}$, a novel complex-valued diffusion model for generating synthetic MRI raw data conditioned on magnitude images, commonly used in clinical practice. This enables the creation of artificial complex-valued raw data, allowing pretraining for models that require k-Space information. We evaluate PhaseGen on two tasks: skull-stripping directly in k-Space and MRI reconstruction using the publicly available FastMRI dataset. Our results show that training with synthetic phase data significantly improves generalization for skull-stripping on real-world data, with an increased segmentation accuracy from $41.1\%$ to $80.1\%$, and enhances MRI reconstruction when combined with limited real-world data. This work presents a step forward in utilizing generative AI to bridge the gap between magnitude-based datasets and the complex-valued nature of MRI raw data. This approach allows researchers to leverage the vast amount of avaliable image domain data in combination with the information-rich k-Space data for more accurate and efficient diagnostic tasks. We make our code publicly $\href{https://github.com/TIO-IKIM/PhaseGen}{\text{available here}}$.

Extending Visual Dynamics for Video-to-Music Generation

Xiaohao Liu,Teng Tu,Yunshan Ma,Tat-Seng Chua

Task: 提出一种名为DyViM的新框架，用于增强视频到音乐生成中的动态建模和时间对齐。

Motivation: 现有方法在特定场景下表现有限或低估了视觉动态的重要性，需要解决动态复杂性和视频与音乐表示的时间错位问题。

Details

Method: 通过简化的运动编码器提取帧级动态特征，利用自注意力模块聚合帧内特征，并结合交叉注意力机制传递高级语义，采用退火调优策略高效微调音乐解码器。 Result: 实验表明DyViM在视频到音乐生成任务中优于现有最先进方法。 Conclusion: DyViM通过改进动态建模和时间对齐，显著提升了视频到音乐生成的质量和适应性。 Abstract: Music profoundly enhances video production by improving quality, engagement, and emotional resonance, sparking growing interest in video-to-music generation. Despite recent advances, existing approaches remain limited in specific scenarios or undervalue the visual dynamics. To address these limitations, we focus on tackling the complexity of dynamics and resolving temporal misalignment between video and music representations. To this end, we propose DyViM, a novel framework to enhance dynamics modeling for video-to-music generation. Specifically, we extract frame-wise dynamics features via a simplified motion encoder inherited from optical flow methods, followed by a self-attention module for aggregation within frames. These dynamic features are then incorporated to extend existing music tokens for temporal alignment. Additionally, high-level semantics are conveyed through a cross-attention mechanism, and an annealing tuning strategy benefits to fine-tune well-trained music decoders efficiently, therefore facilitating seamless adaptation. Extensive experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.

Heart Failure Prediction using Modal Decomposition and Masked Autoencoders for Scarce Echocardiography Databases

Andrés Bell-Navas,María Villalba-Orero,Enrique Lara-Pezzi,Jesús Garicano-Mena,Soledad Le Clainche

Task: 提出一种基于深度学习框架的自动系统，用于实时分析超声心动图视频序列，预测心力衰竭的发生时间。

Motivation: 心力衰竭对医疗行业造成巨大压力，需要开发早期、快速和有效的预测系统。

Details

Method: 系统分为两阶段：第一阶段使用HODMD算法进行数据增强和特征提取，将超声心动图视频序列转化为机器学习兼容的标注图像；第二阶段构建并训练Vision Transformer（ViT），采用自监督学习方法。 Result: 结果显示HODMD算法的有效性，以及所提系统在ViT和CNN架构中的优越性。 Conclusion: 该系统能够有效预测心力衰竭时间，为医疗行业提供了新的解决方案。 Abstract: Heart diseases constitute the main cause of international human defunction. According to the World Health Organization (WHO), approximately 18 million deaths happen each year due to precisely heart diseases. In particular, heart failures (HF) press the healthcare industry to develop systems for their early, rapid and effective prediction. In this work, an automatic system which analyses in real-time echocardiography video sequences is proposed for the challenging and more specific task of prediction of heart failure times. This system is based on a novel deep learning framework, and works in two stages. The first one transforms the data included in a database of echocardiography video sequences into a machine learning-compatible collection of annotated images which can be used in the training phase of any kind of machine learning-based framework, including a deep learning one. This initial stage includes the use of the Higher Order Dynamic Mode Decomposition (HODMD) algorithm for both data augmentation and feature extraction. The second stage is focused on building and training a Vision Transformer (ViT). Self-supervised learning (SSL) methods, which have been so far barely explored in the literature about heart failure prediction, are applied to effectively train the ViT from scratch, even with scarce databases of echocardiograms. The designed neural network analyses images from echocardiography sequences to estimate the time in which a heart failure will happen. The results obtained show the efficacy of the HODMD algorithm and the superiority of the proposed system with respect to several established ViT and Convolutional Neural Network (CNN) architectures.

CollEX -- A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collections

Florian Schneider,Narges Baba Ahmadi,Niloufar Baba Ahmadi,Iris Vogel,Martin Semmann,Chris Biemann

Task: Introduce CollEx, a multimodal agentic RAG system for enhancing interactive exploration of scientific collections.

Motivation: Address the lack of intuitiveness and interactivity in conventional search systems for scientific collections, which hinder learners, educators, and researchers.

Details

Method: Employ state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents through an intuitive chat interface, abstracting complex interactions with specialized tools. Result: CollEx simplifies access to diverse scientific collections, supports educational scenarios, and aids in discovering interdisciplinary connections. Conclusion: CollEx effectively enhances exploration of scientific collections, benefiting both educational and research communities. Abstract: In this paper, we introduce CollEx, an innovative multimodal agentic Retrieval-Augmented Generation (RAG) system designed to enhance interactive exploration of extensive scientific collections. Given the overwhelming volume and inherent complexity of scientific collections, conventional search systems often lack necessary intuitiveness and interactivity, presenting substantial barriers for learners, educators, and researchers. CollEx addresses these limitations by employing state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents accessible through an intuitive chat interface. By abstracting complex interactions via specialized agents equipped with advanced tools, CollEx facilitates curiosity-driven exploration, significantly simplifying access to diverse scientific collections and records therein. Our system integrates textual and visual modalities, supporting educational scenarios that are helpful for teachers, pupils, students, and researchers by fostering independent exploration as well as scientific excitement and curiosity. Furthermore, CollEx serves the research community by discovering interdisciplinary connections and complementing visual data. We illustrate the effectiveness of our system through a proof-of-concept application containing over 64,000 unique records across 32 collections from a local scientific collection from a public university.

Localization Meets Uncertainty: Uncertainty-Aware Multi-Modal Localization

Hye-Min Won,Jieun Lee,Jiyong Oh

Task: 提出一种基于不确定性感知的定位方法，以提高复杂室内环境中机器人导航的可靠性。

Motivation: 可靠的定位对于复杂室内环境中的机器人导航至关重要，现有方法在不确定性处理方面存在不足。

Details

Method: 采用基于百分位数的拒绝策略，结合RGB图像和2D LiDAR数据的多模态端到端定位方法。 Result: 实验结果显示，应用更严格的不确定性阈值显著降低了位置和方向误差，并有效去除了极端异常值。 Conclusion: 该方法首次定量证明了基于百分位数的不确定性拒绝在多模态端到端定位任务中的优势，为实际部署提供了更可靠的定位解决方案。 Abstract: Reliable localization is critical for robot navigation in complex indoor environments. In this paper, we propose an uncertainty-aware localization method that enhances the reliability of localization outputs without modifying the prediction model itself. This study introduces a percentile-based rejection strategy that filters out unreliable 3-DoF pose predictions based on aleatoric and epistemic uncertainties the network estimates. We apply this approach to a multi-modal end-to-end localization that fuses RGB images and 2D LiDAR data, and we evaluate it across three real-world datasets collected using a commercialized serving robot. Experimental results show that applying stricter uncertainty thresholds consistently improves pose accuracy. Specifically, the mean position error is reduced by 41.0%, 56.7%, and 69.4%, and the mean orientation error by 55.6%, 65.7%, and 73.3%, when applying 90%, 80%, and 70% thresholds, respectively. Furthermore, the rejection strategy effectively removes extreme outliers, resulting in better alignment with ground truth trajectories. To the best of our knowledge, this is the first study to quantitatively demonstrate the benefits of percentile-based uncertainty rejection in multi-modal end-to-end localization tasks. Our approach provides a practical means to enhance the reliability and accuracy of localization systems in real-world deployments.

Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation

Yanglin Huang,Kai Hu,Yuan Zhang,Zhineng Chen,Xieping Gao

Task: 提出一种名为HeteroAKD的异构知识蒸馏方法，用于语义分割任务。

Motivation: 现有知识蒸馏方法在同构架构中忽略了异构架构的多样性知识，而这些知识对学生模型获取更精确和全面的数据理解至关重要。

Details

Method: 通过将师生模型的中间特征投影到对齐的logits空间，消除架构特定信息的影响，并引入知识混合机制（KMM）和知识评估机制（KEM）来利用异构知识。 Result: 在三个主流基准测试中，HeteroAKD在异构架构间的知识蒸馏性能优于现有方法。 Conclusion: HeteroAKD为异构架构间的知识蒸馏提供了一种有效的解决方案。 Abstract: Current knowledge distillation (KD) methods for semantic segmentation focus on guiding the student to imitate the teacher's knowledge within homogeneous architectures. However, these methods overlook the diverse knowledge contained in architectures with different inductive biases, which is crucial for enabling the student to acquire a more precise and comprehensive understanding of the data during distillation. To this end, we propose for the first time a generic knowledge distillation method for semantic segmentation from a heterogeneous perspective, named HeteroAKD. Due to the substantial disparities between heterogeneous architectures, such as CNN and Transformer, directly transferring cross-architecture knowledge presents significant challenges. To eliminate the influence of architecture-specific information, the intermediate features of both the teacher and student are skillfully projected into an aligned logits space. Furthermore, to utilize diverse knowledge from heterogeneous architectures and deliver customized knowledge required by the student, a teacher-student knowledge mixing mechanism (KMM) and a teacher-student knowledge evaluation mechanism (KEM) are introduced. These mechanisms are performed by assessing the reliability and its discrepancy between heterogeneous teacher-student knowledge. Extensive experiments conducted on three main-stream benchmarks using various teacher-student pairs demonstrate that our HeteroAKD outperforms state-of-the-art KD methods in facilitating distillation between heterogeneous architectures.

Virtual-mask Informed Prior for Sparse-view Dual-Energy CT Reconstruction

Zini Chen,Yao Xiao,Junyan Zhang,Shaoyu Wang,Liu Shi,Qiegen Liu

Task: 提出了一种基于双域虚拟掩码扩散模型的稀疏视图双能CT重建方法。

Motivation: 稀疏视图采样在双能CT中虽能降低辐射剂量和加快成像速度，但容易产生伪影；现有扩散模型多聚焦于图像域且缺乏全局约束，导致重建质量不足。

Details

Method: 设计了虚拟掩码并应用于高低能数据以构建高维张量作为扩散模型的先验信息，同时采用双域协作策略整合小波域和投影域信息以优化全局结构和局部细节。 Result: 实验结果表明该方法在多个数据集中表现优异。 Conclusion: 该方法通过双域协作和虚拟掩码设计，显著提升了稀疏视图双能CT的重建质量。 Abstract: Sparse-view sampling in dual-energy computed tomography (DECT) significantly reduces radiation dose and increases imaging speed, yet is highly prone to artifacts. Although diffusion models have demonstrated potential in effectively handling incomplete data, most existing methods in this field focus on the image do-main and lack global constraints, which consequently leads to insufficient reconstruction quality. In this study, we propose a dual-domain virtual-mask in-formed diffusion model for sparse-view reconstruction by leveraging the high inter-channel correlation in DECT. Specifically, the study designs a virtual mask and applies it to the high-energy and low-energy data to perform perturbation operations, thus constructing high-dimensional tensors that serve as the prior information of the diffusion model. In addition, a dual-domain collaboration strategy is adopted to integrate the information of the randomly selected high-frequency components in the wavelet domain with the information in the projection domain, for the purpose of optimizing the global struc-tures and local details. Experimental results indicated that the present method exhibits excellent performance across multiple datasets.

PRAD: Periapical Radiograph Analysis Dataset and Benchmark Model Development

Zhenhuan Zhou,Yuchen Zhang,Ruihong Xu,Xuansen Zhao,Tao Li

Task: 提出一个名为PRAD-10K的数据集和PRNet深度学习网络，用于牙周根尖片（PR）的辅助分析。

Motivation: 由于牙周根尖片在牙髓病学和牙周病学中的广泛应用，但其分辨率限制和伪影等问题导致缺乏高质量数据集，阻碍了深度学习在该领域的应用。

Details

Method: 构建包含10,000张临床牙周根尖片图像的数据集PRAD-10K，并提供像素级标注和分类标签；提出PRNet网络用于PR分割任务。 Result: PRNet在PRAD-10K数据集上的表现优于现有医学图像分割模型。 Conclusion: PRAD-10K和PRNet为PR分析提供了高质量数据集和基准模型，推动了深度学习在牙科辅助诊断中的应用。 Abstract: Deep learning (DL), a pivotal technology in artificial intelligence, has recently gained substantial traction in the domain of dental auxiliary diagnosis. However, its application has predominantly been confined to imaging modalities such as panoramic radiographs and Cone Beam Computed Tomography, with limited focus on auxiliary analysis specifically targeting Periapical Radiographs (PR). PR are the most extensively utilized imaging modality in endodontics and periodontics due to their capability to capture detailed local lesions at a low cost. Nevertheless, challenges such as resolution limitations and artifacts complicate the annotation and recognition of PR, leading to a scarcity of publicly available, large-scale, high-quality PR analysis datasets. This scarcity has somewhat impeded the advancement of DL applications in PR analysis. In this paper, we present PRAD-10K, a dataset for PR analysis. PRAD-10K comprises 10,000 clinical periapical radiograph images, with pixel-level annotations provided by professional dentists for nine distinct anatomical structures, lesions, and artificial restorations or medical devices, We also include classification labels for images with typical conditions or lesions. Furthermore, we introduce a DL network named PRNet to establish benchmarks for PR segmentation tasks. Experimental results demonstrate that PRNet surpasses previous state-of-the-art medical image segmentation models on the PRAD-10K dataset. The codes and dataset will be made publicly available.

Focal Cortical Dysplasia Type II Detection Using Cross Modality Transfer Learning and Grad-CAM in 3D-CNNs for MRI Analysis

Lorenzo Lasagni,Antonio Ciccarone,Renzo Guerrini,Matteo Lenge,Ludovico D'incerti

Task: 使用3D卷积神经网络（3D-CNNs）检测FCD（局灶性皮质发育不良）II型。

Motivation: FCD II型是药物难治性癫痫的主要原因，但MRI诊断困难，易误诊。

Details

Method: 采用ResNet架构（ResNet-18、-34和-50），结合跨模态迁移学习和可解释人工智能（XAI）技术（如Grad-CAM）。 Result: 迁移学习显著提高了分类准确率（达80.3%）和可解释性（通过Heat-Score指标评估）。 Conclusion: 迁移学习和XAI技术对提升AI在医学诊断中的应用具有重要意义，尤其是在FCD等难诊断病例中。 Abstract: Focal cortical dysplasia (FCD) type II is a major cause of drug-resistant epilepsy, often curable only by surgery. Despite its clinical importance, the diagnosis of FCD is very difficult in MRI because of subtle abnormalities, leading to misdiagnosis. This study investigates the use of 3D convolutional neural networks (3D-CNNs) for FCD detection, using a dataset of 170 subjects (85 FCD patients and 85 controls) composed of T1-weighted and FLAIR MRI scans. In particular, it investigates the benefits obtained from cross-modality transfer learning and explainable artificial intelligence (XAI) techniques, in particular Gradient-weighted Class Activation Mapping (Grad-CAM). ResNet architectures (ResNet-18, -34, and -50) were implemented, employing transfer learning strategies that used pre-trained weights from segmentation tasks. Results indicate that transfer learning significantly enhances classification accuracy (up to 80.3%) and interpretability, as measured by a novel Heat-Score metric, which evaluates the model's focus on clinically relevant regions. Improvements in the Heat-Score metric underscore the model's seizure zone localization capabilities, bringing AI predictions and clinical insights closer together. These results highlight the importance of transfer learning, including cross-modality, and XAI in advancing AI-based medical diagnostics, especially for difficult-to-diagnose pathologies such as FCD.

Adaptive Detection of Fast Moving Celestial Objects Using a Mixture of Experts and Physical-Inspired Neural Network

Peng Jia,Ge Li,Bafeng Cheng,Yushan Li,Rongyu Sun

Task: 提出一种新颖的算法，用于在星场中检测快速移动的天体。

Motivation: 传统方法在空间望远镜多样化的观测模式下效果不佳，需要一种适应性强的新方法。

Details

Method: 通过将先进的天体检测神经网络转化为物理启发的神经网络，利用望远镜的点扩散函数和观测模式作为先验信息，直接识别快速移动天体。 Result: 在模拟和真实观测数据中，该方法有效检测了不同观测模式下的快速移动天体。 Conclusion: 该算法克服了传统技术的局限性，适用于多样化的观测场景。 Abstract: Fast moving celestial objects are characterized by velocities across the celestial sphere that significantly differ from the motions of background stars. In observational images, these objects exhibit distinct shapes, contrasting with the typical appearances of stars. Depending on the observational method employed, these celestial entities may be designated as near-Earth objects or asteroids. Historically, fast moving celestial objects have been observed using ground-based telescopes, where the relative stability of stars and Earth facilitated effective image differencing techniques alongside traditional fast moving celestial object detection and classification algorithms. However, the growing prevalence of space-based telescopes, along with their diverse observational modes, produces images with different properties, rendering conventional methods less effective. This paper presents a novel algorithm for detecting fast moving celestial objects within star fields. Our approach enhances state-of-the-art fast moving celestial object detection neural networks by transforming them into physical-inspired neural networks. These neural networks leverage the point spread function of the telescope and the specific observational mode as prior information; they can directly identify moving fast moving celestial objects within star fields without requiring additional training, thereby addressing the limitations of traditional techniques. Additionally, all neural networks are integrated using the mixture of experts technique, forming a comprehensive fast moving celestial object detection algorithm. We have evaluated our algorithm using simulated observational data that mimics various observations carried out by space based telescope scenarios and real observation images. Results demonstrate that our method effectively detects fast moving celestial objects across different observational modes.

Revisiting Likelihood-Based Out-of-Distribution Detection by Modeling Representations

Yifan Ding,Arturas Aleksandrauskas,Amirhossein Ahmadian,Jonas Unger,Fredrik Lindsten,Gabriel Eilertsen

Task: 探讨基于似然的深度生成模型在表示空间中用于OOD检测的性能。

Motivation: 解决传统基于似然的深度生成模型在图像空间中OOD检测性能不佳的问题，证明似然方法在表示空间中的有效性。

Details

Method: 使用扩散模型的概率流公式作为似然估计器，并将其应用于预训练编码器的表示空间。 Result: 在表示空间中，基于似然的方法可以达到与最先进方法相当的性能。 Conclusion: 似然方法在表示空间中仍然有效，关键在于选择合适的似然估计器和空间。 Abstract: Out-of-distribution (OOD) detection is critical for ensuring the reliability of deep learning systems, particularly in safety-critical applications. Likelihood-based deep generative models have historically faced criticism for their unsatisfactory performance in OOD detection, often assigning higher likelihood to OOD data than in-distribution samples when applied to image data. In this work, we demonstrate that likelihood is not inherently flawed. Rather, several properties in the images space prohibit likelihood as a valid detection score. Given a sufficiently good likelihood estimator, specifically using the probability flow formulation of a diffusion model, we show that likelihood-based methods can still perform on par with state-of-the-art methods when applied in the representation space of pre-trained encoders. The code of our work can be found at $\href{https://github.com/limchaos/Likelihood-OOD.git}{\texttt{https://github.com/limchaos/Likelihood-OOD.git}}$.

HarmonySeg: Tubular Structure Segmentation with Deep-Shallow Feature Fusion and Growth-Suppression Balanced Loss

Yi Huang,Ke Zhang,Wei Liu,Yuanyuan Wang,Vishal M. Patel,Le Lu,Xu Han,Dakai Jin,Ke Yan

Task: 提出一种名为HarmonySeg的新框架，用于医学图像中管状结构的精确分割。

Motivation: 管状结构（如血管和气道树）在医学图像中的分割对计算机辅助诊断、放射治疗和手术规划至关重要，但算法设计面临尺寸多样、拓扑复杂和标注不完整等挑战。

Details

Method: 设计了具有灵活卷积块的深度到浅层解码器网络，结合血管性图作为辅助信息，并通过浅层-深层融合模块对齐特征，同时引入拓扑保持损失函数。 Result: 在四个公共数据集上的实验表明，该模型能精确分割2D和3D管状结构，并优于现有方法；私有数据集的外部验证也显示良好的泛化能力。 Conclusion: HarmonySeg框架能有效应对管状结构分割的挑战，具有高精度和良好的泛化性能。 Abstract: Accurate segmentation of tubular structures in medical images, such as vessels and airway trees, is crucial for computer-aided diagnosis, radiotherapy, and surgical planning. However, significant challenges exist in algorithm design when faced with diverse sizes, complex topologies, and (often) incomplete data annotation of these structures. We address these difficulties by proposing a new tubular structure segmentation framework named HarmonySeg. First, we design a deep-to-shallow decoder network featuring flexible convolution blocks with varying receptive fields, which enables the model to effectively adapt to tubular structures of different scales. Second, to highlight potential anatomical regions and improve the recall of small tubular structures, we incorporate vesselness maps as auxiliary information. These maps are aligned with image features through a shallow-and-deep fusion module, which simultaneously eliminates unreasonable candidates to maintain high precision. Finally, we introduce a topology-preserving loss function that leverages contextual and shape priors to balance the growth and suppression of tubular structures, which also allows the model to handle low-quality and incomplete annotations. Extensive quantitative experiments are conducted on four public datasets. The results show that our model can accurately segment 2D and 3D tubular structures and outperform existing state-of-the-art methods. External validation on a private dataset also demonstrates good generalizability.

The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasound

Blake VanBerlo,Alexander Wong,Jesse Hoey,Robert Arntfield

Task: 系统研究数据增强和预处理策略在肺部超声自监督学习中的影响。

Motivation: 自然图像的数据增强方法在医学影像任务中可能不适用，因此需要探索适合超声影像的策略。

Details

Method: 评估三种数据增强流程：基线流程、语义保留流程和蒸馏流程，并在多个分类任务上测试预训练模型。 Result: 语义保留数据增强在COVID-19分类任务中表现最佳，而基于裁剪的方法在B线和胸腔积液分类任务中表现更好。 Conclusion: 为超声影像的自监督学习提供了数据增强和预处理策略的实践指导。 Abstract: Data augmentation is a central component of joint embedding self-supervised learning (SSL). Approaches that work for natural images may not always be effective in medical imaging tasks. This study systematically investigated the impact of data augmentation and preprocessing strategies in SSL for lung ultrasound. Three data augmentation pipelines were assessed: (1) a baseline pipeline commonly used across imaging domains, (2) a novel semantic-preserving pipeline designed for ultrasound, and (3) a distilled set of the most effective transformations from both pipelines. Pretrained models were evaluated on multiple classification tasks: B-line detection, pleural effusion detection, and COVID-19 classification. Experiments revealed that semantics-preserving data augmentation resulted in the greatest performance for COVID-19 classification - a diagnostic task requiring global image context. Cropping-based methods yielded the greatest performance on the B-line and pleural effusion object classification tasks, which require strong local pattern recognition. Lastly, semantics-preserving ultrasound image preprocessing resulted in increased downstream performance for multiple tasks. Guidance regarding data augmentation and preprocessing strategies was synthesized for practitioners working with SSL in ultrasound.

Zero-Shot Low-dose CT Denoising via Sinogram Flicking

Yongyi Shi,Ge Wang

Task: 提出一种基于正弦图闪烁的零样本低剂量CT成像方法。

Motivation: 解决监督学习方法需要大量配对图像的临床实践挑战，以及现有零样本自监督方法因下采样操作导致图像分辨率下降的问题。

Details

Method: 通过随机共轭射线匹配生成大量噪声模式不同的正弦图，利用轻量级模型训练网络。 Result: 模拟研究表明，该方法优于ZS-N2N等现有先进方法。 Conclusion: 提出的正弦图闪烁方法在零样本低剂量CT成像中表现优异。 Abstract: Many low-dose CT imaging methods rely on supervised learning, which requires a large number of paired noisy and clean images. However, obtaining paired images in clinical practice is challenging. To address this issue, zero-shot self-supervised methods train denoising networks using only the information within a single image, such as ZS-N2N. However, these methods often employ downsampling operations that degrade image resolution. Additionally, the training dataset is inherently constrained to the image itself. In this paper, we propose a zero-shot low-dose CT imaging method based on sinogram flicking, which operates within a single image but generates many copies via random conjugate ray matching. Specifically, two conjugate X-ray pencil beams measure the same path; their expected values should be identical, while their noise levels vary during measurements. By randomly swapping portions of the conjugate X-rays in the sinogram domain, we generate a large set of sinograms with consistent content but varying noise patterns. When displayed dynamically, these sinograms exhibit a flickering effect due to their identical structural content but differing noise patterns-hence the term sinogram flicking. We train the network on pairs of sinograms with the same content but different noise distributions using a lightweight model adapted from ZS-NSN. This process is repeated to obtain the final results. A simulation study demonstrates that our method outperforms state-of-the-art approaches such as ZS-N2N.