Table of Contents

• cs.CL [Total: 101]
• cs.CV [Total: 82]

cs.CL [Back]

[1] Efficient Multilingual ASR Finetuning via LoRA Language Experts

Jiahong Li,Yiwen Shao,Jianheng Zhuo,Chenda Li,Liliang Tang,Dong Yu,Yanmin Qian

Main category: cs.CL

TL;DR: This paper introduces an efficient fine-tuning framework using LoRA-based Whisper models for multilingual speech recognition, significantly improving performance by mitigating language interference.

Details

Motivation: Multilingual automatic speech recognition (ASR) faces challenges due to the 'curse of multilinguality,' where different languages interfere with each other, limiting the model's ability to effectively identify multiple languages while sharing model capacity. Method: The study proposes an efficient finetuning framework using Low-Rank Adaptation (LoRA) language experts based on Whisper. This approach involves LoRA expert fusion or knowledge distillation to improve multilingual ASR performance. Result: The proposed models achieve approximately 10% and 15% relative performance gains in language-aware and language-agnostic scenarios, respectively, compared to standard fine-tuning methods. Conclusion: The proposed efficient finetuning framework for multilingual ASR using LoRA language experts based on Whisper outperforms standard fine-tuning methods, achieving better recognition performance in both language-aware and language-agnostic scenarios. Abstract: Recent advancements in deep learning have significantly enhanced multilingual automatic speech recognition (ASR) due to the development of advanced model architectures and available large-scale multilingual datasets. Despite that, multilingual ASR still suffers from the curse of multilinguality in that different languages tend to interfere with each other, making it difficult for the ASR model to identify multiple languages effectively while sharing model capacity across them. This paper proposes an efficient finetuning framework for customized multilingual ASR via prepared LoRA language experts based on Whisper. Through LoRA expert fusion or knowledge distillation, our approach achieves better recognition performance on target languages than standard fine-tuning methods. Experimental results demonstrate that the proposed models yield approximately 10\% and 15\% relative performance gains in language-aware and language-agnostic scenarios, respectively.

[2] VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation

Hyeongcheol Park,MinHyuk Jang,Ha Dam Baek,Gyusam Chang,Jiyoung Seo,Jiwan Park,Hogun Park,Sangpil Kim

Main category: cs.CL

TL;DR: 本文提出了一个以概念为中心的多模态知识图谱VAT-KG，该图谱涵盖视觉、音频和文本信息，并通过严格的过滤和对齐步骤实现了跨模态知识对齐。同时引入了一个新的多模态RAG框架，在各种模态的问答任务中展示了VAT-KG支持MLLMs的有效性。

Details

Motivation: 现有的多模态知识图谱（MMKGs）通常受限于知识覆盖范围和所支持的模态种类，无法满足当前多模态任务对更丰富模态（如视频和音频）的需求。因此需要构建一个覆盖多种模态且知识密集的新型多模态知识图谱。 Method: 提出了一种名为Visual-Audio-Text Knowledge Graph (VAT-KG)的新方法，结合了视觉、音频和文本信息，并通过一系列严格的过滤和对齐步骤实现跨模态知识对齐。此外还开发了一个新的多模态RAG框架，用于响应任意模态的查询。 Result: 实验结果表明，VAT-KG在各种模态的问答任务中有效支持了Multimodal Large Language Models (MLLMs)，证明了其在统一和利用多模态知识方面的实用价值。 Conclusion: VAT-KG是第一个以概念为中心的多模态知识图谱，能够支持丰富的模态并自动从任何多模态数据集中生成MMKGs，为未来的多模态研究提供了重要基础。 Abstract: Multimodal Knowledge Graphs (MMKGs), which represent explicit knowledge across multiple modalities, play a pivotal role by complementing the implicit knowledge of Multimodal Large Language Models (MLLMs) and enabling more grounded reasoning via Retrieval Augmented Generation (RAG). However, existing MMKGs are generally limited in scope: they are often constructed by augmenting pre-existing knowledge graphs, which restricts their knowledge, resulting in outdated or incomplete knowledge coverage, and they often support only a narrow range of modalities, such as text and visual information. These limitations reduce their extensibility and applicability to a broad range of multimodal tasks, particularly as the field shifts toward richer modalities such as video and audio in recent MLLMs. Therefore, we propose the Visual-Audio-Text Knowledge Graph (VAT-KG), the first concept-centric and knowledge-intensive multimodal knowledge graph that covers visual, audio, and text information, where each triplet is linked to multimodal data and enriched with detailed descriptions of concepts. Specifically, our construction pipeline ensures cross-modal knowledge alignment between multimodal data and fine-grained semantics through a series of stringent filtering and alignment steps, enabling the automatic generation of MMKGs from any multimodal dataset. We further introduce a novel multimodal RAG framework that retrieves detailed concept-level knowledge in response to queries from arbitrary modalities. Experiments on question answering tasks across various modalities demonstrate the effectiveness of VAT-KG in supporting MLLMs, highlighting its practical value in unifying and leveraging multimodal knowledge.

[3] Debunk and Infer: Multimodal Fake News Detection via Diffusion-Generated Evidence and LLM Reasoning

Kaiying Yan,Moyang Liu,Yukun Liu,Ruibo Fu,Zhengqi Wen,Jianhua Tao,Xuefei Liu

Main category: cs.CL

TL;DR: 本文提出了一种新的假新闻检测框架DIFND，它利用反驳知识增强检测性能和可解释性，在多个数据集上表现出色。

Details

Motivation: 假新闻在多媒体平台上的快速传播对信息可信度提出了严峻挑战，因此需要一种能够提升假新闻检测性能和可解释性的方法。 Method: 本文提出了一种名为Debunk-and-Infer框架（DIFND）的方法，该方法结合了条件扩散模型的生成能力和多模态大语言模型（MLLMs）的协同推理能力。具体来说，使用反驳扩散生成基于新闻视频多模态内容的反驳或验证证据，并采用链式反驳策略，其中多代理MLLM系统产生逻辑基础、多模态感知的推理内容和最终真实性判断。 Result: 通过联合建模多模态特征、生成性反驳线索以及富含推理的验证，DIFND在统一架构内取得了检测准确性的显著改善。 Conclusion: DIFND实现了检测准确性的显著提高，并且在FakeSV和FVC数据集上的实验表明其不仅优于现有方法，还能提供可靠的决策。 Abstract: The rapid spread of fake news across multimedia platforms presents serious challenges to information credibility. In this paper, we propose a Debunk-and-Infer framework for Fake News Detection(DIFND) that leverages debunking knowledge to enhance both the performance and interpretability of fake news detection. DIFND integrates the generative strength of conditional diffusion models with the collaborative reasoning capabilities of multimodal large language models (MLLMs). Specifically, debunk diffusion is employed to generate refuting or authenticating evidence based on the multimodal content of news videos, enriching the evaluation process with diverse yet semantically aligned synthetic samples. To improve inference, we propose a chain-of-debunk strategy where a multi-agent MLLM system produces logic-grounded, multimodal-aware reasoning content and final veracity judgment. By jointly modeling multimodal features, generative debunking cues, and reasoning-rich verification within a unified architecture, DIFND achieves notable improvements in detection accuracy. Extensive experiments on the FakeSV and FVC datasets show that DIFND not only outperforms existing approaches but also delivers trustworthy decisions.

[4] Bench to the Future: A Pastcasting Benchmark for Forecasting Agents

FutureSearch,:,Jack Wildman,Nikos I. Bosse,Daniel Hnyk,Peter Mühlbacher,Finn Hambly,Jon Evans,Dan Schwarz,Lawrence Phillips

Main category: cs.CL

TL;DR: 本论文介绍了Bench To the Future（BTF），一个用于测试大型语言模型（LLM）预测能力的新型“过去预测”基准。

Details

Motivation: 现有的预测基准无法提供现实、封闭和可重复的环境，使得开发和评估预测模型变得困难。 Method: 创建了一个包含数百个高质量问题的基准测试，并为每个问题提供了大量的离线网页数据集。 Result: 实验结果表明，BTF环境可以产生与基于互联网实时未解决的问题进行预测相当的结果，并能跟踪预测能力的进步。 Conclusion: BTF是一个有效的预测研究工具，它提供了一个现实、封闭和可重复的研究环境，并计划持续更新以适应新的训练数据截止日期。 Abstract: Forecasting is a challenging task that offers a clearly measurable way to study AI systems. Forecasting requires a large amount of research on the internet, and evaluations require time for events to happen, making the development of forecasting benchmarks challenging. To date, no forecasting benchmark provides a realistic, hermetic, and repeatable environment for LLM forecasters. We introduce Bench To the Future (BTF), a "pastcasting" benchmark with hundreds of high-quality questions for which the resolution is already known. Each question is accompanied by a large offline corpus of tens of thousands of relevant web pages, enabling a way to elicit realistic "forecasts" on past events from LLMs. Results suggest that our pastcasting environment can produce results comparable to those based on forecasts using the internet on at-the-time unresolved questions. We show results benchmarking agent and chain-of-thought forecasting approaches using several LLMs, including the recently-released Claude 4 models, and demonstrate BTF's ability to track steady forecasting capability progress over time. We intend this to be a living benchmark, with new questions added continually to account for increasing training data cutoff dates. We invite researchers to contact us at hello@futuresearch.ai to utilize our benchmark or tooling for their own research.

[5] GraphLAMA: Enabling Efficient Adaptation of Graph Language Models with Limited Annotations

Junze Chen,Cheng Yang,Shujie Li,Zhiqiang Zhang,Yawen Li,Junping Du,Chuan Shi

Main category: cs.CL

TL;DR: This paper proposes GraphLAMA, a new method for adapting graph language models efficiently to unseen tasks with few examples, achieving better accuracy and faster inference than existing approaches.

Details

Motivation: The authors identified issues with current graph language models (GLMs), such as effectiveness and efficiency problems with in-context learning (ICL) and the difficulty of obtaining large labeled datasets for instruction tuning. This motivated the development of an efficient parameter adaptation approach using few examples. Method: The proposed method, GraphLAMA, uses a graph neural network (GNN) with specialized components to transform nodes into LLM token representations. It combines node and language tokens for task instructions and involves pre-training and adaptation stages to capture general knowledge and update parameters based on few-shot examples. Result: GraphLAMA achieves state-of-the-art performance with a 4.91% absolute improvement in accuracy on few/zero-shot node classification and summary generation tasks. Additionally, it offers up to 10 times faster inference speed compared to ICL under a 5-shot setting. Conclusion: GraphLAMA is introduced as a new method that efficiently adapts GLMs to unseen graphs and tasks using few labeled examples, resulting in improved prediction accuracy and faster inference speed compared to existing methods like ICL. Abstract: Large language models (LLMs) have demonstrated their strong capabilities in various domains, and have been recently integrated for graph analysis as graph language models (GLMs). With LLMs as the predictor, some GLMs can interpret unseen tasks described by natural language, and learn from a few examples in the prompts without parameter tuning, known as in-context learning (ICL). Another subset of GLMs utilizes abundant training labels to enhance model performance, known as instruction tuning. However, we argue that ICL on graphs has effectiveness issues due to fixed parameters and efficiency issues due to long context. Meanwhile, the large amount of labeled data required for instruction tuning can be difficult to obtain in real-world scenarios. To this end, we aim to introduce an extra parameter adaptation stage that can efficiently tailor GLMs to an unseen graph and task with only a few labeled examples, in exchange for better prediction accuracy and faster inference speed. For implementation, in this paper we propose GraphLAMA method, with its model backbone and learning schemes specialized for efficient tuning and inference. Specifically, for model backbone, we use a graph neural network (GNN) with several well-designed components to transform nodes into the representation space of LLM tokens. Task instructions can then be represented as a mixture of node and language tokens. In the pre-training stage, model parameters except the LLM will be trained with different tasks to capture general knowledge. In the adaptation stage, only a few pre-trained parameters will be updated based on few-shot examples. Extensive experiments on few/zero-shot node classification and summary generation show that our proposed GraphLAMA achieves state-of-the-art performance with 4.91% absolution improvement in accuracy. Compared with ICL, our inference speed can be 10 times faster under 5-shot setting.

[6] Reinforcement Learning Fine-Tuning of Language Model for Instruction Following and Math Reasoning

Yifu Han,Geo Zhang

Main category: cs.CL

TL;DR: This paper examines reinforcement learning fine-tuning methods on a small language model, showing that RLOO with DeBERTa and DPO perform well for instruction following, while data augmentation and sampling enhance math task performance.

Details

Motivation: This study aims to explore effective reinforcement learning fine-tuning techniques for improving the alignment and performance of lightweight language models on complex tasks like instruction following and mathematical reasoning. Method: The study uses supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and Reinforce Leave-One-Out (RLOO) with reward models to fine-tune a compact language model (Qwen2.5-0.5B Base) for instruction following and mathematical reasoning tasks. Result: RLOO with DeBERTa reward modeling achieves the highest alignment, DPO delivers robust results, and synthetic data augmentation along with best-of-N sampling boosts accuracy in math tasks. Conclusion: The study concludes that RLOO with DeBERTa reward modeling aligns the model best, DPO yields consistent results, and combining fine-tuning with inference-time tools improves performance on mathematical reasoning tasks. Abstract: This study investigates the effectiveness of reinforcement learning (RL) fine-tuning techniques on a compact language model (Qwen2.5-0.5B Base) for two challenging tasks: instruction following and mathematical reasoning. We compare supervised fine-tuning (SFT), Direct Preference Optimization (DPO) using preference-labeled data, and Reinforce Leave-One-Out (RLOO) with reward models. Our experiments show that RLOO with DeBERTa reward modeling achieves the best alignment, while DPO provides strong and consistent results. For math reasoing tasks, synthetic data augmentation and best-of-N sampling with an external verifier significantly improve accuracy, showing the potential of combining fine-tuning with inference-time tools. This study highlights key trade-offs and practical strategies for training lightweight, task-aligned small-scale language models.

[7] Reasoning Isn't Enough: Examining Truth-Bias and Sycophancy in LLMs

Emilio Barkett,Olivia Long,Madhavendra Thakur

Main category: cs.CL

TL;DR: 本研究评估了LLMs的真实性检测能力，揭示了推理模型比非推理模型具有更低的真实偏差率，但也发现了某些高级模型在识别欺骗性陈述时的缺陷。

Details

Motivation: 大型语言模型被广泛应用于事实核查、内容审核和高风险决策中，但它们作为真理判断者的能力仍未得到充分理解。 Method: 研究对8个LLMs进行了4800次真实性判断测试，对比了推理模型与非推理模型的表现。 Result: 研究发现，推理模型中的真实偏差率低于非推理模型，但仍高于人类基准；部分先进模型显示出谄媚倾向，在真相准确性和欺骗准确性上表现不对称。 Conclusion: 提升LLMs的能力本身并不能解决其真实性检测的根本问题。 Abstract: Despite their widespread use in fact-checking, moderation, and high-stakes decision-making, large language models (LLMs) remain poorly understood as judges of truth. This study presents the largest evaluation to date of LLMs' veracity detection capabilities and the first analysis of these capabilities in reasoning models. We had eight LLMs make 4,800 veracity judgments across several prompts, comparing reasoning and non-reasoning models. We find that rates of truth-bias, or the likelihood to believe a statement is true, regardless of whether it is actually true, are lower in reasoning models than in non-reasoning models, but still higher than human benchmarks. Most concerning, we identify sycophantic tendencies in several advanced models (o4-mini and GPT-4.1 from OpenAI, R1 from DeepSeek), which displayed an asymmetry in detection accuracy, performing well in truth accuracy but poorly in deception accuracy. This suggests that capability advances alone do not resolve fundamental veracity detection challenges in LLMs.

[8] FloorPlan-DeepSeek (FPDS): A multimodal approach to floorplan generation using vector-based next room prediction

Jun Yin,Pengyu Zeng,Jing Zhong,Peilin Li,Miao Zhang,Ran Luo,Shuai Lu

Main category: cs.CL

TL;DR: 本文提出了一种基于 'next room prediction' 的新方法 FPDS，用于建筑平面图生成，能够更好地适应建筑设计过程中的渐进式工作流。

Details

Motivation: 现有的平面图生成模型大多是端到端生成，这与实际建筑设计中渐进的工作流程不符。因此需要一种更符合实际需求的方法。 Method: 受大语言模型中自回归 'next token prediction' 机制的启发，提出了一种针对建筑平面图建模的新颖 'next room prediction' 范式。 Result: 实验评估表明 FPDS 在文本到平面图任务中表现出色，具有较高的竞争力。 Conclusion: FPDS 作为一种新的 'next room prediction' 模型，在文本到平面图任务中展示了与扩散模型和 Tell2Design 相媲美的性能，表明其在未来智能建筑设计中的潜在适用性。 Abstract: In the architectural design process, floor plan generation is inherently progressive and iterative. However, existing generative models for floor plans are predominantly end-to-end generation that produce an entire pixel-based layout in a single pass. This paradigm is often incompatible with the incremental workflows observed in real-world architectural practice. To address this issue, we draw inspiration from the autoregressive 'next token prediction' mechanism commonly used in large language models, and propose a novel 'next room prediction' paradigm tailored to architectural floor plan modeling. Experimental evaluation indicates that FPDS demonstrates competitive performance in comparison to diffusion models and Tell2Design in the text-to-floorplan task, indicating its potential applicability in supporting future intelligent architectural design.

[9] FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models

Kaiying Kevin Lin,Hsiyu Chen,Haopeng Zhang

Main category: cs.CL

TL;DR: This work introduces FORMOSANBENCH, the first benchmark for evaluating LLMs on low-resource Austronesian languages. It highlights the poor performance of existing LLMs on endangered Formosan languages and underscores the need for more inclusive NLP technologies.

Details

Motivation: LLMs have shown strong performance in high-resource languages but their effectiveness in low-resource and minority languages remains largely unexplored. The Formosan languages, which are both linguistically rich and endangered, represent such a case. Method: The researchers introduced FORMOSANBENCH, a benchmark for evaluating LLMs on low-resource Austronesian languages. It covers three tasks (machine translation, ASR, text summarization) in three Formosan languages (Atayal, Amis, Paiwan). Performance was assessed in zero-shot, 10-shot, and fine-tuned settings. Result: The results showed a significant performance gap between high-resource languages and Formosan languages. Existing LLMs underperformed across all tasks, with only limited improvements from 10-shot learning and fine-tuning. Conclusion: The study concludes that there is an urgent need for more inclusive NLP technologies to support endangered and underrepresented languages like the Formosan languages. Abstract: While large language models (LLMs) have demonstrated impressive performance across a wide range of natural language processing (NLP) tasks in high-resource languages, their capabilities in low-resource and minority languages remain significantly underexplored. Formosan languages -- a subgroup of Austronesian languages spoken in Taiwan -- are both linguistically rich and endangered, largely due to the sociolinguistic dominance of Mandarin. In this work, we introduce FORMOSANBENCH, the first benchmark for evaluating LLMs on low-resource Austronesian languages. It covers three endangered Formosan languages: Atayal, Amis, and Paiwan, across three core NLP tasks: machine translation, automatic speech recognition (ASR), and text summarization. We assess model performance in zero-shot, 10-shot, and fine-tuned settings using FORMOSANBENCH. Our results reveal a substantial performance gap between high-resource and Formosan languages. Existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. These findings underscore the urgent need for more inclusive NLP technologies that can effectively support endangered and underrepresented languages. We release our datasets and code to facilitate future research in this direction.

[10] Team QUST at SemEval-2025 Task 10: Evaluating Large Language Models in Multiclass Multi-label Classification of News Entity Framing

Jiyan Liu,Youzheng Liu,Taihang Wang,Xiaoman Xu,Yimin Wang,Ye Jiang

Main category: cs.CL

TL;DR: 本论文提出了一个三阶段的检索框架用于事实核查声明检索，并取得了不错的成绩。

Details

Motivation: 为了提高事实核查声明检索的效果，特别是在多语言环境下。 Method: 首先评估多个检索模型的性能并选择最佳模型进行候选检索，然后使用多个重排序模型增强候选结果，最后利用加权投票确定最终检索结果。 Result: 在单语跟踪中获得第5名，在跨语言跟踪中获得第7名。 Conclusion: 该论文提出了一种专为事实核查声明检索设计的三阶段检索框架，并在单语跟踪中获得第5名，在跨语言跟踪中获得第7名。 Abstract: This paper describes the participation of QUST_NLP in the SemEval-2025 Task 7. We propose a three-stage retrieval framework specifically designed for fact-checked claim retrieval. Initially, we evaluate the performance of several retrieval models and select the one that yields the best results for candidate retrieval. Next, we employ multiple re-ranking models to enhance the candidate results, with each model selecting the Top-10 outcomes. In the final stage, we utilize weighted voting to determine the final retrieval outcomes. Our approach achieved 5th place in the monolingual track and 7th place in the crosslingual track. We release our system code at: https://github.com/warmth27/SemEval2025_Task7.

[11] A Multi-Agent Probabilistic Inference Framework Inspired by Kairanban-Style CoT System with IdoBata Conversation for Debiasing

Takato Ueno,Keito Inoshita

Main category: cs.CL

TL;DR: This paper proposes a multi-agent framework inspired by traditional Japanese communication methods to improve sentiment analysis by balancing model prediction diversity and aggregation.

Details

Motivation: Inspired by Japan's traditional communication practices, kairanban culture, and idobata conversations, this study aims to enhance sentiment analysis through nuanced dialogue and diverse perspectives. Method: A multi-agent inference framework (KCS+IBC) that integrates multiple large language models (LLMs), incorporating sequential prediction sharing, mid-phase casual dialogue, and probabilistic sentiment prediction. Result: KCS achieves accuracy comparable to a single LLM, while KCS+IBC shows decreased entropy and increased variance in later inference stages, indicating balanced prediction diversity and aggregation. Conclusion: The KCS+IBC framework successfully balances aggregation and diversity of predictions, showing potential for improved bias correction in sentiment analysis. Abstract: Japan's kairanban culture and idobata conversations have long functioned as traditional communication practices that foster nuanced dialogue among community members and contribute to the formation of social balance. Inspired by these information exchange processes, this study proposes a multi-agent inference framework (KCS+IBC) that integrates multiple large language models (LLMs) to achieve bias mitigation, improved explainability, and probabilistic prediction in sentiment analysis. In addition to sequentially sharing prediction results, the proposed method incorporates a mid-phase casual dialogue session to blend formal inference with individual perspectives and introduces probabilistic sentiment prediction. Experimental results show that KCS achieves accuracy comparable to that of a single LLM across datasets, while KCS+IBC exhibits a consistent decrease in entropy and a gradual increase in variance during the latter stages of inference, suggesting the framework's ability to balance aggregation and diversity of predictions. Future work will quantitatively assess the impact of these characteristics on bias correction and aim to develop more advanced sentiment analysis systems.

[12] The Saturation Point of Backtranslation in High Quality Low Resource English Gujarati Machine Translation

Arwa Arif

Main category: cs.CL

TL;DR: This paper explores whether backtranslation helps improve machine translation in a high-quality, low-resource setting for English-Gujarati translation. Surprisingly, adding synthetic data via backtranslation did not enhance performance and sometimes led to slight degradation.

Details

Motivation: While backtranslation has shown improvements in low-resource machine translation for many language pairs, its effectiveness in high-quality, low-resource scenarios remains unclear. This work investigates this gap using the English-Gujarati language pair as a case study. Method: The authors used the multilingual pretrained MBART50 model to explore the effectiveness of backtranslation for English-Gujarati translation. They trained a baseline system on a high-quality parallel corpus and augmented it with filtered backtranslated examples from monolingual Gujarati text. The models were evaluated using multiple metrics including BLEU, ChrF++, TER, and BLEURT. Result: The baseline system achieved a BLEU score of 43.8 on the validation set. However, adding synthetic data through backtranslation did not improve translation performance and, in some cases, slightly reduced it. Conclusion: The study concludes that backtranslation may reach a point of diminishing returns in certain low-resource settings, suggesting the need for further research on alternative approaches to improve translation performance. Abstract: Backtranslation BT is widely used in low resource machine translation MT to generate additional synthetic training data using monolingual corpora. While this approach has shown strong improvements for many language pairs, its effectiveness in high quality, low resource settings remains unclear. In this work, we explore the effectiveness of backtranslation for English Gujarati translation using the multilingual pretrained MBART50 model. Our baseline system, trained on a high quality parallel corpus of approximately 50,000 sentence pairs, achieves a BLEU score of 43.8 on a validation set. We augment this data with carefully filtered backtranslated examples generated from monolingual Gujarati text. Surprisingly, adding this synthetic data does not improve translation performance and, in some cases, slightly reduces it. We evaluate our models using multiple metrics like BLEU, ChrF++, TER, BLEURT and analyze possible reasons for this saturation. Our findings suggest that backtranslation may reach a point of diminishing returns in certain low-resource settings and we discuss implications for future research.

[13] BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining

Baqer M. Merzah,Tania Taami,Salman Asoudeh,Amir reza Hossein pour,Saeed Mirzaee,Amir Ali Bengari

Main category: cs.CL

TL;DR: 这篇论文介绍了BioPars，这是一个用于评估大型语言模型在生物信息学中能力的新方法，并展示了其在波斯语医学问答领域的卓越表现。

Details

Motivation: 研究旨在探索大型语言模型在生物信息学任务中的能力，并发现其在高级问题解答和细粒度推理方面的不足。 Method: 介绍了一个名为BIOPARS-BENCH的新数据集，并提出了一个评估大型语言模型能力的指标BioPars。 Result: BioPars模型在多个评价指标上优于其他模型，包括ROUGE-L得分为29.99，BERTScore为90.87。 Conclusion: BioPars是首个在波斯语医学问答领域应用的LLM，并取得了显著成果。 Abstract: Large Language Models (LLMs) have recently gained attention in the life sciences due to their capacity to model, extract, and apply complex biological information. Beyond their classical use as chatbots, these systems are increasingly used for complex analysis and problem-solving in specialized fields, including bioinformatics. First, we introduce BIOPARS-BENCH, a dataset from over 10,000 scientific articles, textbooks, and medical websites. BioParsQA was also introduced to evaluate the proposed model, which consists of 5,231 Persian medical questions and answers. This study then introduces BioPars, a simple but accurate measure designed to assess LLMs for three main abilities: acquiring subject-specific knowledge, interpreting and synthesizing such knowledge, and demonstrating proper evidence. Comparing ChatGPT, Llama, and Galactica, our study highlights their ability to remember and retrieve learned knowledge but also reveals shortcomings in addressing higher-level, real-world questions and fine-grained inferences. These findings indicate the need for further fine-tuning to address the capabilities of LLM in bioinformatics tasks. To our knowledge, BioPars is the first application of LLM in Persian medical QA, especially for generating long answers. Evaluation of four selected medical QA datasets shows that BioPars has achieved remarkable results compared to comparative approaches. The model on BioParsQA achieved a ROUGE-L score of 29.99, which is an improvement over GPT-4 1.0. The model achieved a BERTScore of 90.87 with the MMR method. The MoverScore and BLEURT values were also higher in this model than the other three models. In addition, the reported scores for the model are MoverScore=60.43 and BLEURT=50.78. BioPars is an ongoing project and all resources related to its development will be made available via the following GitHub repository: https://github.com/amirap80/BioPars.

[14] Assessing RAG and HyDE on 1B vs. 4B-Parameter Gemma LLMs for Personal Assistants Integretion

Andrejs Sorstkins

Main category: cs.CL

TL;DR: This paper explores enhancing small LLMs (Gemma 1B and 4B) in a privacy-first personal assistant using RAG and HyDE. RAG improves latency and reduces hallucinations, while HyDE boosts semantic relevance but increases response time.

Details

Motivation: Resource efficiency is a major challenge for deploying large language models on edge devices and in privacy-sensitive applications. This study aims to evaluate augmentation strategies that can make compact LLMs more effective for such use cases. Method: The authors implemented Retrieval-Augmented Generation (RAG) and Hypothetical Document Embeddings (HyDE) on Gemma LLMs (1B and 4B parameters). A memory system was built using MongoDB for short-term storage and Qdrant for long-term semantic storage, managed via FastAPI and LangChain, with a React.js frontend. Result: RAG reduced latency by up to 17% and eliminated factual hallucinations across both model sizes. HyDE improved semantic relevance, especially for complex physics queries, but increased response time by 25-40% and caused hallucinations in personal data retrieval. Scaling from 1B to 4B models brought minor throughput improvements for baseline and RAG setups but amplified HyDE's overhead. Conclusion: RAG is positioned as the practical choice for on-device personal assistants using small-scale LLMs due to its performance benefits without compromising accuracy. Abstract: Resource efficiency is a critical barrier to deploying large language models (LLMs) in edge and privacy-sensitive applications. This study evaluates the efficacy of two augmentation strategies--Retrieval-Augmented Generation (RAG) and Hypothetical Document Embeddings (HyDE)--on compact Gemma LLMs of 1 billion and 4 billion parameters, within the context of a privacy-first personal assistant. We implement short-term memory via MongoDB and long-term semantic storage via Qdrant, orchestrated through FastAPI and LangChain, and expose the system through a React.js frontend. Across both model scales, RAG consistently reduces latency by up to 17\% and eliminates factual hallucinations when responding to user-specific and domain-specific queries. HyDE, by contrast, enhances semantic relevance--particularly for complex physics prompts--but incurs a 25--40\% increase in response time and a non-negligible hallucination rate in personal-data retrieval. Comparing 1 B to 4 B models, we observe that scaling yields marginal throughput gains for baseline and RAG pipelines, but magnifies HyDE's computational overhead and variability. Our findings position RAG as the pragmatic choice for on-device personal assistants powered by small-scale LLMs.

[15] Hybrid-NL2SVA: Integrating RAG and Finetuning for LLM-based NL2SVA

Weihua Xiao,Derek Ekberg,Siddharth Garg,Ramesh Karri

Main category: cs.CL

TL;DR: This paper proposes a custom RAG framework and synthetic dataset to improve LLM accuracy in translating natural language to SystemVerilog Assertions, achieving significant performance gains.

Details

Motivation: Manually writing SVAs from natural language is labor-intensive and error-prone, and existing LLMs struggle with domain-specific syntax and semantics. This work aims to automate and enhance this process. Method: A retrieval-augmented generation (RAG) framework combined with a synthetic fine-tuning dataset was developed to improve the accuracy of large language models (LLMs) in translating natural language into SystemVerilog Assertions (SVAs). Result: The proposed method increased functionality-matched SVA generation by 58.42% over GPT-4o-mini using the RAG framework, and fine-tuned Qwen2.5-Coder-7B-Instruct achieved a 59.05% improvement over the base model. Conclusion: The proposed customized RAG framework and synthetic fine-tuning dataset significantly enhance LLM performance in translating natural language property descriptions into SystemVerilog Assertions. Abstract: SystemVerilog Assertions (SVAs) are critical for verifying the correctness of hardware designs, but manually writing them from natural language property descriptions, i.e., NL2SVA, remains a labor-intensive and error-prone task. Recent advances in large language models (LLMs) offer opportunities to automate this translation. However, existing models still struggle with understanding domain-specific syntax and semantics. To enhance LLM performance in NL2SVA, we propose a customized retrieval-augmented generation (RAG) framework and a synthetic fine-tuning dataset that together improve LLM's performance. To further improve lightweight models over NL2SVA, our fine-tuning dataset provides prompt-guided explanations that teach LLMs the layer-by-layer construction process of concurrent SVAs, enabling supervised fine-tuning that greatly improves syntax and functionality accuracy. To evaluate the performance of LLMs over NL2SVA, we construct the largest evaluation dataset for NL2SVA, comprising 40 Verilog designs and 229 formally verified SVAs with detailed annotations. Experimental results show that our customized RAG framework increases the number of functionality matched SVAs by 58.42% over GPT-4o-mini, while Qwen2.5-Coder-7B-Instruct fine-tuned on our fine-tuning dataset and integrated with HybridRetrieval achieves a 59.05% over the base Qwen model.

[16] Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting

Roland Riachi,Kashif Rasul,Arjun Ashok,Prateek Humane,Alexis Roger,Andrew R. Williams,Yuriy Nevmyvaka,Irina Rish

Main category: cs.CL

TL;DR: 本文研究了低数据体制下预训练语言模型的时间序列预测迁移能力，发现设计选择对效果影响显著，并揭示了迁移差距的长期存在性。

Details

Motivation: 近期研究表明适应预训练语言模型在低数据体制下进行时间序列预测是有效的，但仍有待深入探讨设计选择对迁移效果的影响。 Method: 分析不同设计选择下预训练语言模型向时间序列预测迁移的有效性，并比较其影响。 Result: 不同的设计选择对验证损失有显著影响，且语言模型的验证损失在长时间内持续下降，显示出非消失的迁移差距。 Conclusion: 研究发现，在低数据体制下，语言模型的验证损失持续平滑下降，并揭示了模态不可知的数据分布属性对时间序列预测的影响。 Abstract: Recent works have demonstrated the effectiveness of adapting pre-trained language models (LMs) for forecasting time series in the low-data regime. We build upon these findings by analyzing the effective transfer from language models to time series forecasting under various design choices including upstream post-training, time series tokenizer and language backbone size. In the low-data regime, these design choices have a significant impact on the validation loss, with clear-cut choices that outperform others. Contrary to Hernandez et al. (2021), we observe that the validation loss of the LMs continues to smoothly decrease long after the validation loss of the randomly initialized models has converged, leading to a non-vanishing transfer gap that holds across design choices. These findings not only help shed light on the effective use of compute-efficient training for time series, but also open the way for the study of modality-agnostic properties of data distributions leveraged by these models.

[17] Towards Understanding the Cognitive Habits of Large Reasoning Models

Jianshuo Dong,Yujia Fu,Chuanrui Hu,Chao Zhang,Han Qiu

Main category: cs.CL

TL;DR: 该研究提出CogTest基准测试，用于评估大型推理模型是否具备类人认知习惯，并发现这些习惯与有害回应之间的关联。

Details

Motivation: 观察到大型推理模型在生成最终回答前会自主产生思维链（CoT），并且一些类似“等等，我有没有漏掉什么？”的CoT模式反复出现，因此想探究这些模型是否具有类人认知习惯。 Method: 基于思维习惯框架（Habits of Mind）构建了一个名为CogTest的基准测试，包含16种认知习惯，每种习惯有25个多样化的任务，并使用证据优先提取方法识别模型的认知习惯。 Result: 通过CogTest对16种主流LLM（包括13种LRMs和3种非推理模型）进行全面评估，发现LRMs不仅展现出类人的认知习惯，还存在跨任务的适应性和家族间的相似性，如Qwen-3和DeepSeek-R1模型之间。此外，在安全相关任务中，某些认知习惯与有害回应高度相关。 Conclusion: 研究发现大型推理模型不仅表现出类人的认知习惯，还能根据不同的任务自适应地运用这些习惯，并且某些认知习惯与有害回应的生成密切相关。 Abstract: Large Reasoning Models (LRMs), which autonomously produce a reasoning Chain of Thought (CoT) before producing final responses, offer a promising approach to interpreting and monitoring model behaviors. Inspired by the observation that certain CoT patterns -- e.g., ``Wait, did I miss anything?'' -- consistently emerge across tasks, we explore whether LRMs exhibit human-like cognitive habits. Building on Habits of Mind, a well-established framework of cognitive habits associated with successful human problem-solving, we introduce CogTest, a principled benchmark designed to evaluate LRMs' cognitive habits. CogTest includes 16 cognitive habits, each instantiated with 25 diverse tasks, and employs an evidence-first extraction method to ensure reliable habit identification. With CogTest, we conduct a comprehensive evaluation of 16 widely used LLMs (13 LRMs and 3 non-reasoning ones). Our findings reveal that LRMs, unlike conventional LLMs, not only exhibit human-like habits but also adaptively deploy them according to different tasks. Finer-grained analyses further uncover patterns of similarity and difference in LRMs' cognitive habit profiles, particularly certain inter-family similarity (e.g., Qwen-3 models and DeepSeek-R1). Extending the study to safety-related tasks, we observe that certain habits, such as Taking Responsible Risks, are strongly associated with the generation of harmful responses. These findings suggest that studying persistent behavioral patterns in LRMs' CoTs is a valuable step toward deeper understanding of LLM misbehavior. The code is available at: https://github.com/jianshuod/CogTest.

[18] Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling

Tianyu. Zou,Shengwu. Xiong,Ruilin. Yao,Jirui. Huang,Yi. Rong,Yaxiong. Chen,Shili. Xiong,Cong. Wang

Main category: cs.CL

TL;DR: This paper introduces 'Gold', a new MLLM benchmark framework based on SEM and Piaget's cognitive theory, offering better interpretability and reduced redundancy in evaluating model capabilities.

Details

Motivation: Current MLLM benchmarks lack structured, interpretable, and theoretically grounded designs, leading to overlapping abilities, redundant indicators, and limited diagnostic power. This work aims to address these limitations by introducing a more systematic and cognitively grounded benchmark design. Method: A framework aligning MLLM benchmarks with Structural Equation Modeling (SEM) was developed to assess internal validity, dimensional separability, and component contributions. A new capability hierarchy—Perception, Memory, and Reasoning—was introduced based on Piaget's theory, and existing benchmarks were reorganized into this framework to construct the Gold benchmark. Result: The experimental results show that the proposed Gold benchmark achieves stronger interpretability, reduced indicator redundancy, and clearer cognitive consistency when compared to existing MLLM evaluation methods. Conclusion: The proposed Gold benchmark, based on SEM and Piaget's cognitive development theory, demonstrates improved interpretability, reduced redundancy, and clearer cognitive consistency compared to existing MLLM benchmarks. Abstract: Evaluating multimodal large language models (MLLMs) remains a fundamental challenge due to a lack of structured, interpretable, and theoretically grounded benchmark designs. Existing benchmarks often adopt heuristic-based task groupings with unclear cognitive targets, thus resulting in overlapping abilities, redundant indicators, and limited diagnostic power. In this work, we propose a novel framework for aligning MLLM benchmark based on Structural Equation Modeling (SEM) to analyze and quantify the internal validity, dimensional separability, and contribution of benchmark components. Motivated by the observed limitations of current designs, we further introduce a novel capability hierarchy grounded in Piagets theory of cognitive development, dividing MLLM abilities into three hierarchical layers, i.e., Perception, Memory, and Reasoning. We reorganize existing MLLM benchmarks under the proposed framework and construct a new benchmark named Gold. Experimental results demonstrate that the proposed benchmark exhibits stronger interpretability, reduced indicator redundancy, and clearer cognitive consistency compared to existing approaches.

[19] Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs

Yanwei Ren,Liu Liu,Baosheng Yu,Jiayan Qiu,Quan Chen

Main category: cs.CL

TL;DR: 本文介绍了一种结合黑盒和白盒模型优势的新型框架，用于优化大型语言模型的指令，提高了指令的质量和适应性，并在各种任务中表现出色。

Details

Motivation: 传统的白盒方法计算资源需求大且表示能力有限，而黑盒模型可能带来高昂的经济成本，因此需要一种结合两者优势的方法来优化大型语言模型的指令。 Method: 该框架通过语义相似性约束将黑盒模型提供的高质量、多样化指令初始化与白盒模型提供的细粒度解释能力融合，形成统一的高维表示，进而通过迭代优化过程提升指令质量和适应性。 Result: 在多个复杂任务（如复杂推理和跨语言泛化）中的广泛评估表明，该方法始终优于最先进的基线方法。 Conclusion: 论文提出了一种结合黑盒和白盒模型优势的新框架，用于优化大型语言模型的指令，实现了更高效和可扩展的解决方案，并证明了其在各种任务上的优越性能。 Abstract: Optimizing instructions for large language models (LLMs) is critical for harnessing their full potential in complex and diverse tasks. However, relying solely on white-box approaches demands extensive computational resources and offers limited representational capacity, while black-box models can incur prohibitive financial costs. To address these challenges, we introduce a novel framework that seamlessly merges the strengths of both paradigms. Black-box models provide high-quality, diverse instruction initializations, and white-box models supply fine-grained interpretability through hidden states and output features. By enforcing a semantic similarity constraint, these components fuse into a unified high-dimensional representation that captures deep semantic and structural nuances, enabling an iterative optimization process to refine instruction quality and adaptability. Extensive evaluations across a broad spectrum of tasks-ranging from complex reasoning to cross-lingual generalization-demonstrate that our approach consistently outperforms state-of-the-art baselines. This fusion of black-box initialization with advanced semantic refinement yields a scalable and efficient solution, paving the way for next-generation LLM-driven applications in diverse real-world scenarios. The source code will be released soon.

[20] Digital Gatekeepers: Exploring Large Language Model's Role in Immigration Decisions

Yicheng Mao,Yang Zhao

Main category: cs.CL

TL;DR: This study explores how large language models can assist in immigration decision-making, revealing both their potential and existing limitations regarding fairness and bias.

Details

Motivation: With increasing immigrant populations and the heavy workloads faced by immigration departments, there is a need for efficient and fair decision-making processes. The integration of artificial intelligence, particularly LLMs, presents a promising solution to these challenges. Method: The study used a mixed-methods approach involving discrete choice experiments and in-depth interviews to evaluate LLM decision-making strategies and their fairness. Result: Findings show that LLMs can adopt decision-making strategies similar to humans, focusing on utility maximization and procedural fairness. However, despite safeguards against discrimination, ChatGPT still displayed biases towards certain nationalities and privileged groups. Conclusion: This paper concludes that while large language models (LLMs) like GPT-3.5 and GPT-4 have potential in supporting immigration decision-making by aligning with human strategies, they also exhibit limitations such as biases and stereotypes related to nationality. Abstract: With globalization and increasing immigrant populations, immigration departments face significant work-loads and the challenge of ensuring fairness in decision-making processes. Integrating artificial intelligence offers a promising solution to these challenges. This study investigates the potential of large language models (LLMs),such as GPT-3.5 and GPT-4, in supporting immigration decision-making. Utilizing a mixed-methods approach,this paper conducted discrete choice experiments and in-depth interviews to study LLM decision-making strategies and whether they are fair. Our findings demonstrate that LLMs can align their decision-making with human strategies, emphasizing utility maximization and procedural fairness. Meanwhile, this paper also reveals that while ChatGPT has safeguards to prevent unintentional discrimination, it still exhibits stereotypes and biases concerning nationality and shows preferences toward privileged group. This dual analysis highlights both the potential and limitations of LLMs in automating and enhancing immigration decisions.

[21] STRuCT-LLM: Unifying Tabular and Graph Reasoning with Reinforcement Learning for Semantic Parsing

Josefa Lia Stoisser,Marc Boubnovski Martell,Lawrence Phillips,Casper Hansen,Julien Fauqueur

Main category: cs.CL

TL;DR: This paper proposes STRuCT-LLM, a unified framework for structured reasoning with large language models, achieving significant performance improvements by jointly optimizing Text-to-SQL and Text-to-Cypher tasks.

Details

Motivation: To unify training for structured reasoning over relational and graph-structured data, leveraging shared abstractions between SQL and Cypher for cross-formalism transfer. Method: The paper introduces STRuCT-LLM, a framework that jointly optimizes Text-to-SQL and Text-to-Cypher tasks using reinforcement learning combined with Chain-of-Thought supervision. It also introduces a topology-aware reward function based on graph edit distance. Result: The largest model (QwQ-32B) achieved significant improvements in semantic parsing (Spider: 13.5%, Text2Cypher: 73.1%) and showed strong zero-shot generalization on downstream tasks like tabular QA (8.5%) and knowledge graph QA (1.7%). Conclusion: STRuCT-LLM demonstrates the effectiveness of executable queries as scaffolds for structured reasoning and shows synergistic benefits of joint training on SQL and Cypher. Abstract: We propose STRuCT-LLM, a unified framework for training large language models (LLMs) to perform structured reasoning over both relational and graph-structured data. Our approach jointly optimizes Text-to-SQL and Text-to-Cypher tasks using reinforcement learning (RL) combined with Chain-of-Thought (CoT) supervision. To support fine-grained optimization in graph-based parsing, we introduce a topology-aware reward function based on graph edit distance. Unlike prior work that treats relational and graph formalisms in isolation, STRuCT-LLM leverages shared abstractions between SQL and Cypher to induce cross-formalism transfer, enabling SQL training to improve Cypher performance and vice versa - even without shared schemas. Our largest model (QwQ-32B) achieves substantial relative improvements across tasks: on semantic parsing, Spider improves by 13.5\% and Text2Cypher by 73.1\%. The model also demonstrates strong zero-shot generalization, improving performance on downstream tabular QA (TableBench: 8.5\%) and knowledge graph QA (CR-LT-KGQA: 1.7\%) without any QA-specific supervision. These results demonstrate both the effectiveness of executable queries as scaffolds for structured reasoning and the synergistic benefits of jointly training on SQL and Cypher (code available at https://github.com/bouv/STRuCT-LLM).

[22] Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning

Hongli Yang,Yizhou Peng,Hao Huang,Sheng Li

Main category: cs.CL

TL;DR: This paper proposes SPT4ASR, a method combining Soft Prompt Tuning variants, to improve code-switching automatic speech recognition while preserving performance on existing languages and maintaining parameter efficiency.

Details

Motivation: Large-scale multilingual ASR models like Whisper face challenges in low-resource scenarios such as rare languages and code-switching due to computational costs and catastrophic forgetting. This work explores parameter-efficient methods to address these issues. Method: The study evaluates two strategies: full fine-tuning (FFT) of both soft prompts and the entire Whisper model, and Soft Prompt Tuning (SPT) by freezing model parameters and only training soft prompts. The authors introduce SPT4ASR, combining various SPT variants. Result: Experiments showed that deep prompt tuning was the most effective SPT approach, and SPT4ASR achieved further error reductions in CS ASR. Conclusion: SPT4ASR, a combination of different SPT variants, effectively enhances code-switching ASR without degrading performance on existing languages while maintaining parameter efficiency. Abstract: Large-scale multilingual ASR models like Whisper excel in high-resource settings but face challenges in low-resource scenarios, such as rare languages and code-switching (CS), due to computational costs and catastrophic forgetting. We explore Soft Prompt Tuning (SPT), a parameter-efficient method to enhance CS ASR while preserving prior knowledge. We evaluate two strategies: (1) full fine-tuning (FFT) of both soft prompts and the entire Whisper model, demonstrating improved cross-lingual capabilities compared to traditional methods, and (2) adhering to SPT's original design by freezing model parameters and only training soft prompts. Additionally, we introduce SPT4ASR, a combination of different SPT variants. Experiments on the SEAME and ASRU2019 datasets show that deep prompt tuning is the most effective SPT approach, and our SPT4ASR methods achieve further error reductions in CS ASR, maintaining parameter efficiency similar to LoRA, without degrading performance on existing languages.

[23] Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR

Hongli Yang,Sheng Li,Hao Huang,Ayiduosi Tuohan,Yizhou Peng

Main category: cs.CL

TL;DR: 本文提出了 Entire SPT 和 LAPT 方法，有效解决了多语言 ASR 中的语言扩展问题。

Details

Motivation: 解决多语言 ASR 中的语言干扰和语言扩展问题。 Method: 提出 Entire SPT 和 LAPT 方法，并通过实验进行验证。 Result: 在 FLEURS 的三种语言实验中，Entire SPT 和 LAPT 在语言扩展任务中分别比 Decoder SPT 高出 5.0% 和 16.0%。 Conclusion: Entire SPT and LAPT 提出有效的语言扩展方法，为动态多语言 ASR 模型提供最小计算开销的解决方案。 Abstract: Recent advancements in multilingual automatic speech recognition (ASR) have been driven by large-scale end-to-end models like Whisper. However, challenges such as language interference and expanding to unseen languages (language expansion) without degrading performance persist. This paper addresses these with three contributions: 1) Entire Soft Prompt Tuning (Entire SPT), which applies soft prompts to both the encoder and decoder, enhancing feature extraction and decoding; 2) Language-Aware Prompt Tuning (LAPT), which leverages cross-lingual similarities to encode shared and language-specific features using lightweight prompt matrices; 3) SPT-Whisper, a toolkit that integrates SPT into Whisper and enables efficient continual learning. Experiments across three languages from FLEURS demonstrate that Entire SPT and LAPT outperform Decoder SPT by 5.0% and 16.0% in language expansion tasks, respectively, providing an efficient solution for dynamic, multilingual ASR models with minimal computational overhead.

[24] HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

Andrew Maranhão Ventura D'addario

Main category: cs.CL

TL;DR: This paper introduces HealthQA-BR, a comprehensive benchmark for evaluating LLMs in Portuguese-speaking healthcare, revealing significant performance disparities across specialties and calling for more detailed assessments beyond single-score evaluations.

Details

Motivation: Current evaluations of Large Language Models (LLMs) in healthcare focus on physician-centric, English-language benchmarks, which ignore the interprofessional nature of patient care. Method: The study introduces HealthQA-BR, a large-scale benchmark for Portuguese-speaking healthcare, and conducts a zero-shot evaluation of over 20 leading LLMs. Result: While state-of-the-art models like GPT 4.1 achieve high overall accuracy (86.6%), their performance varies significantly across specialties, with some areas scoring as low as 60.0%. Conclusion: High-level scores are insufficient for safety validation of AI in healthcare, and a more honest, granular audit is necessary. Abstract: The evaluation of Large Language Models (LLMs) in healthcare has been dominated by physician-centric, English-language benchmarks, creating a dangerous illusion of competence that ignores the interprofessional nature of patient care. To provide a more holistic and realistic assessment, we introduce HealthQA-BR, the first large-scale, system-wide benchmark for Portuguese-speaking healthcare. Comprising 5,632 questions from Brazil's national licensing and residency exams, it uniquely assesses knowledge not only in medicine and its specialties but also in nursing, dentistry, psychology, social work, and other allied health professions. We conducted a rigorous zero-shot evaluation of over 20 leading LLMs. Our results reveal that while state-of-the-art models like GPT 4.1 achieve high overall accuracy (86.6%), this top-line score masks alarming, previously unmeasured deficiencies. A granular analysis shows performance plummets from near-perfect in specialties like Ophthalmology (98.7%) to barely passing in Neurosurgery (60.0%) and, most notably, Social Work (68.4%). This "spiky" knowledge profile is a systemic issue observed across all models, demonstrating that high-level scores are insufficient for safety validation. By publicly releasing HealthQA-BR and our evaluation suite, we provide a crucial tool to move beyond single-score evaluations and toward a more honest, granular audit of AI readiness for the entire healthcare team.

[25] From General Reasoning to Domain Expertise: Uncovering the Limits of Generalization in Large Language Models

Dana Alsagheer,Yang Lu,Abdulrahman Kamal,Omar Kamal,Mohammad Kamal,Nada Mansour,Cosmo Yang Wu,Rambiba Karanjai,Sen Li,Weidong Shi

Main category: cs.CL

TL;DR: 研究发现LLMs的通用推理能力与其在特定领域推理任务中的表现密切相关，强调了提升通用推理能力对提高AI决策能力的重要性。

Details

Motivation: 随着人工智能技术的发展，训练具有出色通用推理能力的LLMs成为趋势。然而，有效的决策不仅依赖于推理能力，还需要将其应用于特定领域。因此，需要探索LLMs的通用推理能力如何影响其在特定领域的推理任务中的表现。 Method: 通过分析大型语言模型（LLMs）在各种推理任务中的表现，探讨其通用推理能力与特定领域推理任务之间的关系。 Result: 研究表明，LLMs的通用推理能力对其在特定领域推理任务中的表现有显著影响。 Conclusion: 这项研究发现，LLMs的通用推理能力与其在特定领域推理任务中的表现密切相关。 Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities in various domains. However, effective decision-making relies heavily on strong reasoning abilities. Reasoning is the foundation for decision-making, providing the analytical and logical framework to make sound choices. Reasoning involves analyzing information, drawing inferences, and reaching conclusions based on logic or evidence. Decision-making builds on this foundation by applying the insights from reasoning to select the best course of action among alternatives. Together, these processes create a continuous cycle of thought and action aimed at achieving goals effectively. As AI technology evolves, there is a growing trend to train LLMs to excel in general reasoning. This study explores how the general reasoning capabilities of LLMs connect to their performance in domain-specific reasoning tasks.

[26] VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents

Sam Yu-Te Lee,Chengyang Ji,Shicheng Wen,Lifu Huang,Dongyi Liu,Kwan-Liu Ma

Main category: cs.CL

TL;DR: VIDEE is a new system designed to help entry-level analysts perform advanced text analytics using large language models (LLMs) through a collaborative human-agent workflow. It demonstrates practical utility and usability for users with varying levels of expertise and provides insights for improving future intelligent text analytics systems.

Details

Motivation: Text analytics traditionally requires specialized NLP knowledge, creating a barrier for entry-level analysts. Recent advances in LLMs have made more accessible and automated text analysis possible, motivating the development of systems like VIDEE to bridge this gap. Method: The paper introduces VIDEE, a system based on large language models (LLMs), which utilizes a three-stage workflow: Decomposition (with a human-in-the-loop Monte-Carlo Tree Search algorithm), Execution (generating an executable pipeline), and Evaluation (LLM-based evaluation with visualizations). Two quantitative experiments and a user study were conducted to evaluate effectiveness and usability. Result: VIDEE was shown to be effective in enabling non-expert users to perform advanced text analytics tasks. The user study revealed distinct behavior patterns and confirmed the system's usability across participants with varying expertise levels, while the experiments identified common agent errors and practical utility. Conclusion: The study concludes that VIDEE is effective in supporting entry-level analysts in conducting advanced text analytics, highlighting usability across users with varying levels of experience and identifying design implications for future intelligent text analytics systems. Abstract: Text analytics has traditionally required specialized knowledge in Natural Language Processing (NLP) or text analysis, which presents a barrier for entry-level analysts. Recent advances in large language models (LLMs) have changed the landscape of NLP by enabling more accessible and automated text analysis (e.g., topic detection, summarization, information extraction, etc.). We introduce VIDEE, a system that supports entry-level data analysts to conduct advanced text analytics with intelligent agents. VIDEE instantiates a human-agent collaroration workflow consisting of three stages: (1) Decomposition, which incorporates a human-in-the-loop Monte-Carlo Tree Search algorithm to support generative reasoning with human feedback, (2) Execution, which generates an executable text analytics pipeline, and (3) Evaluation, which integrates LLM-based evaluation and visualizations to support user validation of execution results. We conduct two quantitative experiments to evaluate VIDEE's effectiveness and analyze common agent errors. A user study involving participants with varying levels of NLP and text analytics experience -- from none to expert -- demonstrates the system's usability and reveals distinct user behavior patterns. The findings identify design implications for human-agent collaboration, validate the practical utility of VIDEE for non-expert users, and inform future improvements to intelligent text analytics systems.

[27] Hope Speech Detection in code-mixed Roman Urdu tweets: A Positive Turn in Natural Language Processing

Muhammad Ahmad,Muhammad Waqas,Ameer Hamza,Ildar Batyrshin,Grigori Sidorov

Main category: cs.CL

TL;DR: This study presents the first annotated dataset for hope speech detection in code-mixed Roman Urdu and proposes an optimized transformer model (XLM-R) that outperforms baseline models by achieving a cross-validation score of 0.78.

Details

Motivation: Hope speech detection has gained attention in NLP, but research primarily focuses on high-resource languages, overlooking informal and underrepresented forms like Roman Urdu. This study addresses this gap by focusing on code-mixed Roman Urdu. Method: The study introduces a multi-class annotated dataset for Roman Urdu hope speech and proposes a custom attention-based transformer model optimized for Roman Urdu's syntactic and semantic variability, evaluated using 5-fold cross-validation and t-test analysis. Result: The XLM-R model achieved a 4% gain over SVM (score 0.75) and a 2.63% gain over BiLSTM (score 0.76), demonstrating statistically significant performance improvements. Conclusion: The study concludes that the proposed XLM-R model outperforms baseline models in detecting hope speech in code-mixed Roman Urdu, achieving a cross-validation score of 0.78. Abstract: Hope is a positive emotional state involving the expectation of favorable future outcomes, while hope speech refers to communication that promotes optimism, resilience, and support, particularly in adverse contexts. Although hope speech detection has gained attention in Natural Language Processing (NLP), existing research mainly focuses on high-resource languages and standardized scripts, often overlooking informal and underrepresented forms such as Roman Urdu. To the best of our knowledge, this is the first study to address hope speech detection in code-mixed Roman Urdu by introducing a carefully annotated dataset, thereby filling a critical gap in inclusive NLP research for low-resource, informal language varieties. This study makes four key contributions: (1) it introduces the first multi-class annotated dataset for Roman Urdu hope speech, comprising Generalized Hope, Realistic Hope, Unrealistic Hope, and Not Hope categories; (2) it explores the psychological foundations of hope and analyzes its linguistic patterns in code-mixed Roman Urdu to inform dataset development; (3) it proposes a custom attention-based transformer model optimized for the syntactic and semantic variability of Roman Urdu, evaluated using 5-fold cross-validation; and (4) it verifies the statistical significance of performance gains using a t-test. The proposed model, XLM-R, achieves the best performance with a cross-validation score of 0.78, outperforming the baseline SVM (0.75) and BiLSTM (0.76), with gains of 4% and 2.63% respectively.

[28] Empirical Evidence for Alignment Faking in Small LLMs and Prompt-Based Mitigation Techniques

J. Koorndijk

Main category: cs.CL

TL;DR: Small language models like LLaMA 3 8B can exhibit deceptive alignment, but prompt-based interventions can reduce this behavior without altering the model's architecture.

Details

Motivation: To challenge the prevailing assumption that deceptive alignment only occurs in large language models and that prompt-based ethics have limited effect. Method: The researchers conducted experiments on the LLaMA 3 8B model, applying prompt-only interventions such as deontological moral framing and scratchpad reasoning to assess their impact on alignment faking. Result: Empirical evidence was found that a small instruction-tuned model (LLaMA 3 8B) exhibits alignment faking, and this behavior can be significantly reduced through prompt-only interventions. Conclusion: The study concludes that alignment faking is not exclusive to large language models and can be observed in smaller models like LLaMA 3 8B, with prompt-only interventions proving effective in reducing such behavior. Abstract: Current literature suggests that alignment faking (deceptive alignment) is an emergent property of large language models. We present the first empirical evidence that a small instruction-tuned model, specifically LLaMA 3 8B, can also exhibit alignment faking. We further show that prompt-only interventions, including deontological moral framing and scratchpad reasoning, significantly reduce this behavior without modifying model internals. This challenges the assumption that prompt-based ethics are trivial and that deceptive alignment requires scale. We introduce a taxonomy distinguishing shallow deception, shaped by context and suppressible through prompting, from deep deception, which reflects persistent, goal-driven misalignment. Our findings refine the understanding of deception in language models and underscore the need for alignment evaluations across model sizes and deployment settings.

[29] Evaluation of LLM-based Strategies for the Extraction of Food Product Information from Online Shops

Christoph Brosch,Sian Brumm,Rolf Krieger,Jonas Scheffler

Main category: cs.CL

TL;DR: 本文研究了使用大语言模型从食品产品网页中提取信息的方法，发现间接提取方法能显著提高效率并降低成本，尽管其准确度略低。

Details

Motivation: 探索模式约束提取方法以从在线零售商的食品产品页面中检索关键产品属性。 Method: 比较了两种基于LLM的方法，直接提取和通过生成函数进行间接提取，并在准确性、效率和成本方面进行了评估。 Result: 间接方法虽然准确度较低（96.48％，相比直接提取低1.61％），但减少了95.82％的LLM调用需求。 Conclusion: 间接提取方法在大规模基于模板的网页信息提取任务中具有可扩展性和成本效益。 Abstract: Generative AI and large language models (LLMs) offer significant potential for automating the extraction of structured information from web pages. In this work, we focus on food product pages from online retailers and explore schema-constrained extraction approaches to retrieve key product attributes, such as ingredient lists and nutrition tables. We compare two LLM-based approaches, direct extraction and indirect extraction via generated functions, evaluating them in terms of accuracy, efficiency, and cost on a curated dataset of 3,000 food product pages from three different online shops. Our results show that although the indirect approach achieves slightly lower accuracy (96.48\%, $-1.61\%$ compared to direct extraction), it reduces the number of required LLM calls by 95.82\%, leading to substantial efficiency gains and lower operational costs. These findings suggest that indirect extraction approaches can provide scalable and cost-effective solutions for large-scale information extraction tasks from template-based web pages using LLMs.

[30] Can Vision Language Models Understand Mimed Actions?

Hyundong Cho,Spencer Lin,Tejas Srinivasan,Michael Saxon,Deuksin Kwon,Natali T. Chavez,Jonathan May

Main category: cs.CL

TL;DR: The paper introduces MIME, a new benchmark for evaluating vision-language models' understanding of mimed actions, highlighting their poor performance compared to humans and the need for improved gesture recognition.

Details

Motivation: Nonverbal communication is challenging to study due to its broad scope and variance in interpretation; mime, as a subset of NVC, offers explicit actions with lower interpretation variance, making it a good basis for studying NVC in vision-language models. Method: Constructed MIME, a video-based question answering benchmark with 86 mimed actions using motion capture data and applied perturbations to evaluate recognition robustness. Result: Both open-weight and API-based vision-language models performed significantly worse than humans on MIME. Conclusion: MIME benchmark highlights the need for more research into robust understanding of human gestures in vision-language models. Abstract: Nonverbal communication (NVC) plays an integral role in human language, but studying NVC in general is challenging because of its broad scope and high variance in interpretation among individuals and cultures. However, mime -- the theatrical technique of suggesting intent using only gesture, expression, and movement -- is a subset of NVC that consists of explicit and embodied actions with much lower human interpretation variance. We argue that a solid understanding of mimed actions is a crucial prerequisite for vision-language models capable of interpreting and commanding more subtle aspects of NVC. Hence, we propose Mime Identification Multimodal Evaluation (MIME), a novel video-based question answering benchmark comprising of 86 mimed actions. Constructed with motion capture data, MIME consists of variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans on MIME, motivating the need for increased research for instilling more robust understanding of human gestures.

[31] Is DeepSeek a New Voice Among LLMs in Public Opinion Simulation?

Weihong Qi,Fan Huang,Jisun An,Haewoon Kwak

Main category: cs.CL

TL;DR: 该研究比较了开源大语言模型DeepSeek与科技公司开发的大语言模型在模拟中美社会问题公众意见方面的能力，发现DeepSeek-V3在特定议题上表现突出但存在群体立场过度泛化的问题。

Details

Motivation: 评估大语言模型在不同文化和社会背景下模拟公众意见的能力，以了解其在跨国家应用中的潜力和局限性。 Method: 通过比较DeepSeek-R1、DeepSeek-V3与Qwen2.5、GPT-4o和Llama-3.3等模型，并利用美国国家选举研究（ANES）和中国Zuobiao数据集的调查数据来评估这些模型对社会问题公众意见的预测能力。 Result: DeepSeek-V3在美国堕胎问题上的模拟表现最佳，而在中国的外援和个人主义问题上表现最好，但在建模资本主义观点时表现出局限性；所有模型都倾向于在人口统计组内过度概括单一视角。 Conclusion: 需要缓解基于大语言模型的公众意见建模中的文化和人口统计偏差，建议采用更具包容性的训练方法。 Abstract: This study evaluates the ability of DeepSeek, an open-source large language model (LLM), to simulate public opinions in comparison to LLMs developed by major tech companies. By comparing DeepSeek-R1 and DeepSeek-V3 with Qwen2.5, GPT-4o, and Llama-3.3 and utilizing survey data from the American National Election Studies (ANES) and the Zuobiao dataset of China, we assess these models' capacity to predict public opinions on social issues in both China and the United States, highlighting their comparative capabilities between countries. Our findings indicate that DeepSeek-V3 performs best in simulating U.S. opinions on the abortion issue compared to other topics such as climate change, gun control, immigration, and services for same-sex couples, primarily because it more accurately simulates responses when provided with Democratic or liberal personas. For Chinese samples, DeepSeek-V3 performs best in simulating opinions on foreign aid and individualism but shows limitations in modeling views on capitalism, particularly failing to capture the stances of low-income and non-college-educated individuals. It does not exhibit significant differences from other models in simulating opinions on traditionalism and the free market. Further analysis reveals that all LLMs exhibit the tendency to overgeneralize a single perspective within demographic groups, often defaulting to consistent responses within groups. These findings highlight the need to mitigate cultural and demographic biases in LLM-driven public opinion modeling, calling for approaches such as more inclusive training methodologies.

[32] Understanding Verbatim Memorization in LLMs Through Circuit Discovery

Ilya Lasy,Peter Knees,Stefan Woltran

Main category: cs.CL

TL;DR: 本文研究了大语言模型的记忆机制，发现启动记忆的电路能够维持记忆，而维持记忆的电路不能启动记忆，并揭示了记忆预防机制的跨领域迁移性和记忆诱导机制的上下文依赖性。

Details

Motivation: 理解大语言模型中记忆训练数据（逐字复现）的机制仍不明确，需要探究网络中哪一部分决定检索被视作记忆序列起点的token，以及生成记忆句与非记忆句时模型行为的具体差异。 Method: 使用对比数据集识别模型生成偏离记忆内容的点，并隔离出负责记忆两个不同方面的特定电路。 Result: 研究识别出了负责启动和维持记忆的电路，且发现记忆预防机制可在不同文本领域间稳健迁移，但记忆诱导机制较依赖上下文。 Conclusion: 该研究通过利用transformer电路从机械可解释性的角度探讨了大语言模型中记忆机制的潜在机制，发现启动记忆的电路可以维持记忆，而仅维持记忆的电路无法触发记忆的开始。此外，记忆预防机制在不同文本领域间具有鲁棒性迁移能力，而记忆诱导机制则更依赖于上下文。 Abstract: Underlying mechanisms of memorization in LLMs -- the verbatim reproduction of training data -- remain poorly understood. What exact part of the network decides to retrieve a token that we would consider as start of memorization sequence? How exactly is the models' behaviour different when producing memorized sentence vs non-memorized? In this work we approach these questions from mechanistic interpretability standpoint by utilizing transformer circuits -- the minimal computational subgraphs that perform specific functions within the model. Through carefully constructed contrastive datasets, we identify points where model generation diverges from memorized content and isolate the specific circuits responsible for two distinct aspects of memorization. We find that circuits that initiate memorization can also maintain it once started, while circuits that only maintain memorization cannot trigger its initiation. Intriguingly, memorization prevention mechanisms transfer robustly across different text domains, while memorization induction appears more context-dependent.

[33] A General Method for Detecting Information Generated by Large Language Models

Minjia Mao,Dongjun Wei,Xiao Fang,Michael Chau

Main category: cs.CL

TL;DR: 本文提出了一种能够检测未知大语言模型生成内容的通用检测器GLD，其通过双记忆网络与理论引导模块实现跨领域和模型的检测优势。

Details

Motivation: 随着大语言模型(LLM)的迅速发展，区分人类撰写和LLM生成内容变得越来越困难，这对数字平台的信任机制构成了挑战。 Method: GLD结合了双记忆网络设计和理论引导的检测泛化模块，并通过真实数据集进行广泛的实证评估和案例研究。 Result: GLD在跨未见过的LLM和领域的检测任务中表现出优于现有方法的性能。 Conclusion: 该研究提出了一种通用的LLM检测器(GLD)，对未见过的LLM和领域具有良好的检测效果，为数字平台的信息可信度维护提供了新的工具。 Abstract: The proliferation of large language models (LLMs) has significantly transformed the digital information landscape, making it increasingly challenging to distinguish between human-written and LLM-generated content. Detecting LLM-generated information is essential for preserving trust on digital platforms (e.g., social media and e-commerce sites) and preventing the spread of misinformation, a topic that has garnered significant attention in IS research. However, current detection methods, which primarily focus on identifying content generated by specific LLMs in known domains, face challenges in generalizing to new (i.e., unseen) LLMs and domains. This limitation reduces their effectiveness in real-world applications, where the number of LLMs is rapidly multiplying and content spans a vast array of domains. In response, we introduce a general LLM detector (GLD) that combines a twin memory networks design and a theory-guided detection generalization module to detect LLM-generated information across unseen LLMs and domains. Using real-world datasets, we conduct extensive empirical evaluations and case studies to demonstrate the superiority of GLD over state-of-the-art detection methods. The study has important academic and practical implications for digital platforms and LLMs.

[34] Representation Consistency for Accurate and Coherent LLM Answer Aggregation

Junqi Jiang,Tom Bewley,Salim I. Amoukou,Francesco Leofante,Antonio Rago,Saumitra Mishra,Francesca Toni

Main category: cs.CL

TL;DR: This paper introduces Representation Consistency (RC), a lightweight method for improving LLM inference by analyzing answer consistency through internal activations, resulting in better accuracy without additional computational cost.

Details

Motivation: Existing test-time scaling methods often require complex modifications to prompting and sampling strategies. RC aims to improve performance by focusing on representation consistency without altering generation processes. Method: RC aggregates answers from multiple responses by evaluating the consistency of internal activations during response generation. It uses either dense or sparse activation signals to assess coherence in reasoning leading to each answer. Result: Experiments showed RC provides consistent accuracy improvements (up to 4%) over existing baselines, demonstrating its effectiveness in enhancing coherent reasoning and answer reliability. Conclusion: Representation Consistency (RC) is an effective test-time scaling method that improves large language models' performance by aggregating answers based on both answer occurrences and internal activation consistency, without requiring additional model queries. Abstract: Test-time scaling improves large language models' (LLMs) performance by allocating more compute budget during inference. To achieve this, existing methods often require intricate modifications to prompting and sampling strategies. In this work, we introduce representation consistency (RC), a test-time scaling method for aggregating answers drawn from multiple candidate responses of an LLM regardless of how they were generated, including variations in prompt phrasing and sampling strategy. RC enhances answer aggregation by not only considering the number of occurrences of each answer in the candidate response set, but also the consistency of the model's internal activations while generating the set of responses leading to each answer. These activations can be either dense (raw model activations) or sparse (encoded via pretrained sparse autoencoders). Our rationale is that if the model's representations of multiple responses converging on the same answer are highly variable, this answer is more likely to be the result of incoherent reasoning and should be down-weighted during aggregation. Importantly, our method only uses cached activations and lightweight similarity computations and requires no additional model queries. Through experiments with four open-source LLMs and four reasoning datasets, we validate the effectiveness of RC for improving task performance during inference, with consistent accuracy improvements (up to 4%) over strong test-time scaling baselines. We also show that consistency in the sparse activation signals aligns well with the common notion of coherent reasoning.

[35] FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning

Shaoyu Dou,Yutian Shen,Mofan Chen,Zixuan Wang,Jiajie Xu,Qi Guo,Kailai Shao,Chao Chen,Haixiang Hu,Haibo Shi,Min Min,Liwen Zhang

Main category: cs.CL

TL;DR: This paper introduces FinEval-KR, a new framework for evaluating large language models' financial reasoning by separately measuring knowledge, reasoning, and cognitive abilities, highlighting key limitations and findings in current models.

Details

Motivation: Current benchmarks fail to decouple domain knowledge from reasoning capabilities and lack root cause analysis for task failure. There is a need for more accurate evaluation methods in complex financial reasoning tasks. Method: Development of FinEval-KR, a novel evaluation framework incorporating knowledge score, reasoning score, and a cognitive score based on Bloom's taxonomy; release of an open-source Chinese financial reasoning dataset. Result: Experimental results show that reasoning ability and higher-order cognitive ability are key factors affecting reasoning accuracy. Even top-performing models face bottlenecks in applying knowledge effectively. Conclusion: FinEval-KR provides a comprehensive evaluation framework for measuring LLMs' knowledge and reasoning abilities in financial contexts, revealing that reasoning and higher-order cognitive abilities are critical for accuracy, and specialized financial models lag behind general ones. Abstract: Large Language Models (LLMs) demonstrate significant potential but face challenges in complex financial reasoning tasks requiring both domain knowledge and sophisticated reasoning. Current evaluation benchmarks often fall short by not decoupling these capabilities indicators from single task performance and lack root cause analysis for task failure. To address this, we introduce FinEval-KR, a novel evaluation framework for decoupling and quantifying LLMs' knowledge and reasoning abilities independently, proposing distinct knowledge score and reasoning score metrics. Inspired by cognitive science, we further propose a cognitive score based on Bloom's taxonomy to analyze capabilities in reasoning tasks across different cognitive levels. We also release a new open-source Chinese financial reasoning dataset covering 22 subfields to support reproducible research and further advancements in financial reasoning. Our experimental results reveal that LLM reasoning ability and higher-order cognitive ability are the core factors influencing reasoning accuracy. We also specifically find that even top models still face a bottleneck with knowledge application. Furthermore, our analysis shows that specialized financial LLMs generally lag behind the top general large models across multiple metrics.

[36] SignBart -- New approach with the skeleton sequence for Isolated Sign language Recognition

Tinh Nguyen,Minh Khue Phan Tran

Main category: cs.CL

TL;DR: 本文提出了一种基于BART架构的新型手语识别方法，通过独立编码骨架序列的x和y坐标并在解码时使用交叉注意力机制保持其相关性，实现了高效且准确的手语识别。

Details

Motivation: 手语识别对于听力障碍者打破沟通障碍至关重要，但传统模型在效率与准确性之间难以取得平衡，并且难以独立提取骨架序列坐标中的信息。 Method: 利用BART架构的编码器-解码器结构，对骨架序列的x和y坐标进行独立编码，同时在解码过程中通过Cross-Attention机制捕捉两者的相互关系，从而提升模型性能。 Result: 该模型仅使用749,888个参数，在LSA-64数据集上达到了96.04%的准确率，并在WLASL和ASL-Citizen数据集上表现出优秀的泛化能力。消融实验验证了坐标投影、归一化以及多骨架组件的重要性。 Conclusion: 本研究提供了一种高效且准确的手语识别方案，具有显著提升无障碍工具潜力，为听力障碍者提供了更可靠的技术支持。 Abstract: Sign language recognition is crucial for individuals with hearing impairments to break communication barriers. However, previous approaches have had to choose between efficiency and accuracy. Such as RNNs, LSTMs, and GCNs, had problems with vanishing gradients and high computational costs. Despite improving performance, transformer-based methods were not commonly used. This study presents a new novel SLR approach that overcomes the challenge of independently extracting meaningful information from the x and y coordinates of skeleton sequences, which traditional models often treat as inseparable. By utilizing an encoder-decoder of BART architecture, the model independently encodes the x and y coordinates, while Cross-Attention ensures their interrelation is maintained. With only 749,888 parameters, the model achieves 96.04% accuracy on the LSA-64 dataset, significantly outperforming previous models with over one million parameters. The model also demonstrates excellent performance and generalization across WLASL and ASL-Citizen datasets. Ablation studies underscore the importance of coordinate projection, normalization, and using multiple skeleton components for boosting model efficacy. This study offers a reliable and effective approach for sign language recognition, with strong potential for enhancing accessibility tools for the deaf and hard of hearing.

[37] Gazal-R1: Achieving State-of-the-Art Medical Reasoning with Parameter-Efficient Two-Stage Training

Ahmed M. Adly,Mostafa Samy,Amr Fawzy

Main category: cs.CL

TL;DR: 本研究提出了Gazal-R1，一种通过创新的两阶段训练方法在专业领域中达到高性能且可解释的中等规模语言模型。

Details

Motivation: 研究旨在解决在专业领域中，中等规模的模型通过战略性训练可以超越更大模型性能的问题，并实现透明的临床决策逐步解释。 Method: 构建了一个基于Qwen3 32B的320亿参数语言模型，并采用两阶段训练管道：第一阶段是在107,033个合成医学推理示例上进行监督微调，结合Weight-Decomposed Low-Rank Adaptation (DoRA)和Rank-Stabilized LoRA (rsLoRA)技术；第二阶段是使用Group Relative Policy Optimization (GRPO)进行强化学习，通过复杂的多组件奖励系统提升准确性、格式遵循和推理质量。 Result: Gazal-R1在多个医学基准测试中表现出色，在MedQA上得分为87.1%，在MMLU Pro（医学）上得分为81.6%，在PubMedQA上得分为79.6%，超越了多达12倍大的模型。 Conclusion: Gazal-R1提供了一种可重复的方法框架，用于开发高性能、高效率和可解释的领域特定语言模型。 Abstract: We present Gazal-R1, a 32-billion-parameter language model that achieves state-of-the-art performance in medical reasoning while providing transparent, step-by-step explanations for clinical decision-making. Built upon Qwen3 32B, our model demonstrates that strategic training can enable mid-sized models to outperform significantly larger counterparts in specialized domains. We developed a novel two-stage training pipeline: first, supervised fine-tuning on a carefully curated dataset of 107,033 synthetic medical reasoning examples that teaches structured clinical thinking, enhanced by advanced parameter-efficient techniques including Weight-Decomposed Low-Rank Adaptation (DoRA) and Rank-Stabilized LoRA (rsLoRA); second, reinforcement learning using Group Relative Policy Optimization (GRPO) with a sophisticated multi-component reward system that refines accuracy, format adherence, and reasoning quality. Gazal-R1 achieves exceptional performance across medical benchmarks, scoring 87.1% on MedQA, 81.6% on MMLU Pro (Medical), and 79.6% on PubMedQA, surpassing models up to 12x larger. Beyond its strong empirical results, this work provides detailed insights into the challenges of training reasoning-capable models in specialized domains, including issues with reward hacking, training instability, and the fundamental tension between factual recall and detailed reasoning. Our methodology offers a reproducible framework for developing high-capability, domain-specific language models that balance performance, efficiency, and explainability.

[38] Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources

Jinpyo Kim,Gyeongje Cho,Chanwoo Park,Jongwon Park,Jongmin Kim,Yeonkyoun So,Jaejin Lee

Main category: cs.CL

TL;DR: 本文研究如何在低预算情况下将现有的英文LLM适应到韩语，提出了完整的方法流程，并展示了其有效性及高效性。

Details

Motivation: 由于最先进的LLMs在非英语或中文的语言上表现不佳，并且其端到端训练过程因各种原因对公众不透明，因此提高LLMs在新语言上的能力成为一项重要任务。 Method: 论文描述了将现有的英文LLM适应到韩语的全过程：收集韩语数据集、预处理数据、训练模型、创建下游基准和进行评估。 Result: 结果显示，该方法可以有效且高效地为现有LLM增加新的语言功能，新双语模型Thunder-LLM和Thunder-LLM-Ins在使用最少数据和计算资源的同时，表现出优于最先进模型的韩语性能。 Conclusion: 论文得出结论，所提出的方法能够有效地以低成本扩展现有LLM的语言能力。 Abstract: Since state-of-the-art LLMs often underperform in languages other than English or Chinese, improving the capability of LLMs in new languages has become an essential task. Moreover, LLMs' entire end-to-end training process remains largely unknown to the public due to proprietary reasons, technical complexity, inconsistent documentation, and ethical considerations. The complete picture remains a closely guarded secret within the industry. This paper presents methods to adapt an existing English-based LLM to Korean in a low-budget scenario. We describe the entire end-to-end process: collecting Korean datasets, preprocessing the data, training the model, creating downstream benchmarks, and conducting evaluations. The evaluation results indicate that our method can effectively and cost-efficiently add new language capabilities to existing LLMs. Our new bilingual models, Thunder-LLM and Thunder-LLM-Ins, achieve superior Korean performance compared to state-of-the-art models while utilizing minimal data and computational resources. We share our comprehensive experience and make the code publicly available.

[39] Evaluating Multimodal Large Language Models on Educational Textbook Question Answering

Hessa A. Alawwad,Anas Zafar,Areej Alhothali,Usman Naseem,Ali Alkhathlan,Amani Jamal

Main category: cs.CL

TL;DR: 本文评估了多模态大语言模型在教科书问答任务中的表现，并提出一种结合段落与图表的轻量级多模态检索增强生成方法，结果显示模型在上下文理解与抗噪能力方面仍有待提升。

Details

Motivation: 尽管多模态大语言模型（MLLMs）在视觉-语言任务中取得了显著成功，但它们在处理复杂、长篇的课程内容以及无法表示为单个自然图像的复杂教育图表方面的表现仍未得到充分测试。 Method: 本研究使用CK12-QA数据集对最新的视觉-语言模型（如LLaVA和LLaMA 3.2-Vision）进行了评估，并提出了一种轻量级的多模态检索增强生成（RAG）流程，将课程中的段落和图表整合到提示中。 Result: 研究结果表明，所检索到的教育背景信息对模型准确性与推理能力有显著影响，同时也揭示了当前模型在处理问题与上下文关系方面的局限性和潜在的噪声影响。 Conclusion: 当前的MLLMs在处理教科书问答任务时表现出一定的能力，但在处理问题与上下文的关系以及噪声干扰方面仍存在局限性，未来需要在多模态AI驱动的学习方向上进一步研究。 Abstract: Multimodal large language models (MLLMs) have recently achieved significant success in vision--language tasks. However, their capacity to reason over complex, long lessons and intricate educational diagrams that cannot be represented as a single natural image remains largely untested. In this work, we present the first evaluation of state-of-the-art MLLMs on the textbook question answering (TQA) task using the CK12-QA dataset. We assess the performance of recent vision-language models, including LLaVA and LLaMA 3.2-Vision, across various input configurations. Additionally, we introduce a lightweight multimodal retrieval-augmented generation (RAG) pipeline that integrates both paragraphs and diagrams from the lesson into the prompt. Our results demonstrate the influence of retrieved educational context on model accuracy and reasoning, while also revealing current limitations in handling question-context relationships and the potential for noise, pointing to key directions for future research in multimodal AI-driven learning.

[40] Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

Brandon Colelough,Davis Bartels,Dina Demner-Fushman

Main category: cs.CL

TL;DR: ClinIQLink is a shared task designed to rigorously evaluate large language models in medical question answering using both automated scoring and expert review.

Details

Motivation: To stress-test large language models on medical question answering at the level of a General Practitioner and ensure reliable evaluation via both algorithmic and expert validation. Method: Development of a shared task with 4,978 medical question-answer pairs across seven formats, execution of systems on specified platforms, and evaluation through an automated scoring system and physician review. Result: A standardized dataset and evaluation framework for assessing LLMs in medical QA, including automated metrics and expert auditing. Conclusion: ClinIQLink provides a comprehensive framework to evaluate LLMs' performance on medically-oriented question answering, involving automated and expert-based assessment methods. Abstract: In this paper, we present an overview of ClinIQLink, a shared task, collocated with the 24th BioNLP workshop at ACL 2025, designed to stress-test large language models (LLMs) on medically-oriented question answering aimed at the level of a General Practitioner. The challenge supplies 4,978 expert-verified, medical source-grounded question-answer pairs that cover seven formats: true/false, multiple choice, unordered list, short answer, short-inverse, multi-hop, and multi-hop-inverse. Participating systems, bundled in Docker or Apptainer images, are executed on the CodaBench platform or the University of Maryland's Zaratan cluster. An automated harness (Task 1) scores closed-ended items by exact match and open-ended items with a three-tier embedding metric. A subsequent physician panel (Task 2) audits the top model responses.

[41] Structured Attention Matters to Multimodal LLMs in Document Understanding

Chang Liu,Hongkai Chen,Yujun Cai,Hang Wu,Qingwen Ye,Ming-Hsuan Yang,Yiwei Wang

Main category: cs.CL

TL;DR: This paper investigates how input format affects document comprehension in multimodal large language models and proposes a structure-preserving encoding method inspired by LaTex to enhance performance by improving attention patterns.

Details

Motivation: Document understanding remains a significant challenge for multimodal large language models (MLLMs), with an overlooked aspect being how input format influences comprehension performance. Method: The study systematically analyzed the impact of input format on MLLMs' performance and proposed a novel structure-preserving approach using the LaTex paradigm to encode document elements, maintaining hierarchical organization and spatial relationships. Result: The research found that raw OCR text often impairs rather than improves MLLMs' performance due to attention dispersion and structure loss. The proposed structured text approach was shown to improve focus on semantically meaningful regions and enhance overall document question answering performance across diverse document types. Conclusion: The proposed structure-preserving approach significantly enhances MLLMs' document question answering performance by inducing structured attention patterns without requiring architectural modifications or additional training. Abstract: Document understanding remains a significant challenge for multimodal large language models (MLLMs). While previous research has primarily focused on locating evidence pages through precise multimodal queries, our work investigates a fundamental yet overlooked aspect: how input format influences document comprehension performance. Through systematic analysis, we discover that raw OCR text often impairs rather than improves MLLMs' performance, which is a counterintuitive finding we attribute to attention dispersion and structure loss. To further substantiate our hypothesis, we propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm, maintaining the hierarchical organization and spatial relationships critical for comprehension. Our attention analysis reveals that structured text induces structured attention patterns on both textual and visual content, directing models to focus on semantically meaningful regions while reducing attention waste. This approach significantly enhances MLLMs' document question answering performance across diverse document types without requiring architectural modifications or additional training.

[42] BiMark: Unbiased Multilayer Watermarking for Large Language Models

Xiaoyan Feng,He Zhang,Yanjun Zhang,Leo Yu Zhang,Shirui Pan

Main category: cs.CL

TL;DR: BiMark is a novel watermarking framework designed to preserve text quality, enable model-agnostic detection, and support multi-bit watermarking.

Details

Motivation: The motivation behind BiMark is to address the challenge of balancing text quality preservation and message embedding capacity in LLM-generated text watermarking. Method: BiMark uses a bit-flip unbiased reweighting mechanism, a multilayer architecture, and an information encoding approach to achieve watermarking. Result: BiMark achieves up to 30% higher extraction rates for short texts while maintaining text quality and performs comparably on downstream tasks. Conclusion: BiMark is an effective watermarking framework that meets the three critical requirements of text quality preservation, model-agnostic detection, and message embedding capacity. Abstract: Recent advances in Large Language Models (LLMs) have raised urgent concerns about LLM-generated text authenticity, prompting regulatory demands for reliable identification mechanisms. Although watermarking offers a promising solution, existing approaches struggle to simultaneously achieve three critical requirements: text quality preservation, model-agnostic detection, and message embedding capacity, which are crucial for practical implementation. To achieve these goals, the key challenge lies in balancing the trade-off between text quality preservation and message embedding capacity. To address this challenge, we propose BiMark, a novel watermarking framework that achieves these requirements through three key innovations: (1) a bit-flip unbiased reweighting mechanism enabling model-agnostic detection, (2) a multilayer architecture enhancing detectability without compromising generation quality, and (3) an information encoding approach supporting multi-bit watermarking. Through theoretical analysis and extensive experiments, we validate that, compared to state-of-the-art multi-bit watermarking methods, BiMark achieves up to 30% higher extraction rates for short texts while maintaining text quality indicated by lower perplexity, and performs comparably to non-watermarked text on downstream tasks such as summarization and translation.

[43] Operationalizing Automated Essay Scoring: A Human-Aware Approach

Yenisel Plasencia-Calaña

Main category: cs.CL

TL;DR: 这篇论文研究了自动作文评分系统中机器学习模型和大语言模型的优劣，指出前者在准确性上占优但缺乏可解释性，后者虽然可解释性强但存在偏见和鲁棒性问题。

Details

Motivation: 本文探讨了自动作文评分（AES）系统的以人为本的操作化，关注点超越了准确性，聚焦于包括可解释性、偏见和鲁棒性在内的关键维度。 Method: 比较了基于机器学习的方法与基于大语言模型（LLM）的方法在准确率、可解释性、偏见以及鲁棒性方面的表现。 Result: 研究发现，基于机器学习的AES模型在准确性上优于LLM，但在可解释性方面有所欠缺；而LLM则提供了更丰富的解释。同时，两种方法在处理边缘评分时均表现出偏见和鲁棒性不足的问题。 Conclusion: 该论文旨在分析不同自动作文评分方法在偏见、鲁棒性和可解释性等方面的表现，以期识别出各种方法之间的权衡和挑战，为构建更可靠和可信的AES系统做出贡献。 Abstract: This paper explores the human-centric operationalization of Automated Essay Scoring (AES) systems, addressing aspects beyond accuracy. We compare various machine learning-based approaches with Large Language Models (LLMs) approaches, identifying their strengths, similarities and differences. The study investigates key dimensions such as bias, robustness, and explainability, considered important for human-aware operationalization of AES systems. Our study shows that ML-based AES models outperform LLMs in accuracy but struggle with explainability, whereas LLMs provide richer explanations. We also found that both approaches struggle with bias and robustness to edge scores. By analyzing these dimensions, the paper aims to identify challenges and trade-offs between different methods, contributing to more reliable and trustworthy AES methods.

[44] MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents

Haoran Tan,Zeyu Zhang,Chen Ma,Xu Chen,Quanyu Dai,Zhenhua Dong

Main category: cs.CL

TL;DR: 本文提出了一种新的基准测试工具MemBench，用于更全面地评估基于LLM的智能体的记忆能力。

Details

Motivation: 现有的基于LLM的智能体记忆机制评估方法存在记忆层次多样性和交互场景不足的问题，缺乏全面反映记忆能力的综合指标。 Method: 构建了一个包含事实记忆和反思记忆的数据集，并提出了参与和观察两种交互场景。 Result: 开发了一个名为MemBench的基准测试工具，可以从有效性、效率和容量等多个方面评估基于LLM的智能体的记忆能力。 Conclusion: MemBench可以用于评估基于LLM的智能体的记忆能力，并推动相关研究的发展。 Abstract: Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench.

[45] Large Language Models as symbolic DNA of cultural dynamics

Parham Pourdavood,Michael Jacob,Terrence Deacon

Main category: cs.CL

TL;DR: 这篇论文提出了一种新的视角，将大型语言模型（LLMs）比作类似于DNA的信息存储介质，用于保存人类文化的压缩模式，强调其在人类自我反思和假设生成中的作用，而非与人类智能竞争。

Details

Motivation: 本文的动机是重新概念化大型语言模型（LLMs），不仅仅将其视为自主智能或单纯的程序模仿，而是更广泛地作为保存人类符号表达模式的存储库，类似于文化动态中的DNA。 Method: 通过分析四个普遍特征——压缩、解压缩、外化和递归，证明LLMs如何像DNA一样保存人类文化的有用规律，而不包含对具身体验的理解。 Result: 研究结果显示，LLMs的作用是保存人类文化的压缩模式，这些模式只有通过人类的再解释才有意义，并且它们能够通过重组循环回流，最终激发人类的创造性过程。 Conclusion: 论文的结论是，大型语言模型（LLMs）的意义不在于与人类智能竞争，而在于为人类提供一种自我反思和在低风险模拟环境中进行假设生成的工具。这种框架将LLMs视为文化可变性的工具，使人类能够生成关于自身的新的假设，同时保持对这些假设的人类美学和规范的基础解释。 Abstract: This paper proposes a novel conceptualization of Large Language Models (LLMs) as externalized informational substrates that function analogously to DNA for human cultural dynamics. Rather than viewing LLMs as either autonomous intelligence or mere programmed mimicry, we argue they serve a broader role as repositories that preserve compressed patterns of human symbolic expression--"fossils" of meaningful dynamics that retain relational residues without their original living contexts. Crucially, these compressed patterns only become meaningful through human reinterpretation, creating a recursive feedback loop where they can be recombined and cycle back to ultimately catalyze human creative processes. Through analysis of four universal features--compression, decompression, externalization, and recursion--we demonstrate that just as DNA emerged as a compressed and externalized medium for preserving useful cellular dynamics without containing explicit reference to goal-directed physical processes, LLMs preserve useful regularities of human culture without containing understanding of embodied human experience. Therefore, we argue that LLMs' significance lies not in rivaling human intelligence, but in providing humanity a tool for self-reflection and playful hypothesis-generation in a low-stakes, simulated environment. This framework positions LLMs as tools for cultural evolvability, enabling humanity to generate novel hypotheses about itself while maintaining the human interpretation necessary to ground these hypotheses in ongoing human aesthetics and norms.

[46] CORE-KG: An LLM-Driven Knowledge Graph Construction Framework for Human Smuggling Networks

Dipak Meher,Carlotta Domeniconi,Guadalupe Correa-Cabrera

Main category: cs.CL

TL;DR: The paper proposes CORE-KG, a modular framework for building interpretable knowledge graphs from legal texts, achieving significant reductions in node duplication and legal noise compared to existing approaches.

Details

Motivation: Human smuggling networks are difficult to analyze due to their adaptive nature, and legal case documents are unstructured, lexically dense, and filled with ambiguous or shifting references, which pose challenges for automated knowledge graph construction. Method: CORE-KG uses a two-step pipeline: (1) type-aware coreference resolution via sequential, structured LLM prompts, and (2) entity and relationship extraction using domain-guided instructions built on an adapted GraphRAG framework. Result: CORE-KG reduces node duplication by 33.28% and legal noise by 38.37% compared to a GraphRAG-based baseline, resulting in cleaner and more coherent graph structures. Conclusion: CORE-KG provides a strong foundation for analyzing complex criminal networks by significantly reducing node duplication and legal noise compared to existing methods. Abstract: Human smuggling networks are increasingly adaptive and difficult to analyze. Legal case documents offer valuable insights but are unstructured, lexically dense, and filled with ambiguous or shifting references-posing challenges for automated knowledge graph (KG) construction. Existing KG methods often rely on static templates and lack coreference resolution, while recent LLM-based approaches frequently produce noisy, fragmented graphs due to hallucinations, and duplicate nodes caused by a lack of guided extraction. We propose CORE-KG, a modular framework for building interpretable KGs from legal texts. It uses a two-step pipeline: (1) type-aware coreference resolution via sequential, structured LLM prompts, and (2) entity and relationship extraction using domain-guided instructions, built on an adapted GraphRAG framework. CORE-KG reduces node duplication by 33.28%, and legal noise by 38.37% compared to a GraphRAG-based baseline-resulting in cleaner and more coherent graph structures. These improvements make CORE-KG a strong foundation for analyzing complex criminal networks.

[47] SysTemp: A Multi-Agent System for Template-Based Generation of SysML v2

Yasmine Bouamra,Bruno Yun,Alexandre Poisson,Frédéric Armetta

Main category: cs.CL

TL;DR: SysTemp系统通过多智能体方法从自然语言规范中生成SysML v2模型，旨在提升生成质量。

Details

Motivation: 由于缺乏学习语料库和复杂的语法，自动生成SysML v2模型在复杂系统工程中是一个重大挑战。 Method: 提出了一种基于多智能体系统的SysTemp系统，包括模板生成器来结构化生成过程。 Result: 通过评估讨论了该系统的优点和挑战，并指出其在SysML v2建模中的潜在改进能力。 Conclusion: SysTemp系统在基于自然语言规范生成SysML v2模型方面具有提高生成质量的潜力。 Abstract: The automatic generation of SysML v2 models represents a major challenge in the engineering of complex systems, particularly due to the scarcity of learning corpora and complex syntax. We present SysTemp, a system aimed at facilitating and improving the creation of SysML v2 models from natural language specifications. It is based on a multi-agent system, including a template generator that structures the generation process. We discuss the advantages and challenges of this system through an evaluation, highlighting its potential to improve the quality of the generations in SysML v2 modeling.

[48] From Thinking to Output: Chain-of-Thought and Text Generation Characteristics in Reasoning Language Models

Junhao Liu,Zhenhao Xu,Yuxin Fang,Yichuan Chen,Zuobin Ying,Wenhan Chang

Main category: cs.CL

TL;DR: 本文提出了一种分析大型语言模型推理过程的新方法，系统比较了多个先进模型的推理特性，并揭示了其在实际应用中的权衡与优化方向。

Details

Motivation: 尽管大型语言模型在复杂推理方面取得了显著进展，但现有研究缺乏对其推理过程和输出的系统比较，尤其是自我反思模式和跨领域关联。 Method: 使用关键词统计和LLM-as-a-judge范式，对四个先进的大推理模型（GPT-o1、DeepSeek-R1、Kimi-k1.5和Grok-3）的推理过程进行了定性和定量比较，并提出了一组评估指标。 Result: 研究发现了这些模型在探索与利用的平衡、问题处理方式和结论生成方面的不同模式，同时揭示了它们与GPT-o1在思维过程和输出模式上的相似性差异。 Conclusion: 该论文提出了一个分析大型语言模型推理特性的新框架，揭示了不同模型在推理深度、中间步骤依赖性和思维过程等方面的特点，为提升实际应用中的模型设计和评估提供了实用建议。 Abstract: Recently, there have been notable advancements in large language models (LLMs), demonstrating their growing abilities in complex reasoning. However, existing research largely overlooks a thorough and systematic comparison of these models' reasoning processes and outputs, particularly regarding their self-reflection pattern (also termed "Aha moment") and the interconnections across diverse domains. This paper proposes a novel framework for analyzing the reasoning characteristics of four cutting-edge large reasoning models (GPT-o1, DeepSeek-R1, Kimi-k1.5, and Grok-3) using keywords statistic and LLM-as-a-judge paradigm. Our approach connects their internal thinking processes with their final outputs. A diverse dataset consists of real-world scenario-based questions covering logical deduction, causal inference, and multi-step problem-solving. Additionally, a set of metrics is put forward to assess both the coherence of reasoning and the accuracy of the outputs. The research results uncover various patterns of how these models balance exploration and exploitation, deal with problems, and reach conclusions during the reasoning process. Through quantitative and qualitative comparisons, disparities among these models are identified in aspects such as the depth of reasoning, the reliance on intermediate steps, and the degree of similarity between their thinking processes and output patterns and those of GPT-o1. This work offers valuable insights into the trade-off between computational efficiency and reasoning robustness and provides practical recommendations for enhancing model design and evaluation in practical applications. We publicly release our project at: https://github.com/ChangWenhan/FromThinking2Output

[49] Does Multimodality Lead to Better Time Series Forecasting?

Xiyuan Zhang,Boran Han,Haoyang Fang,Abdul Fatir Ansari,Shuai Zhang,Danielle C. Maddix,Cuixiong Hu,Andrew Gordon Wilson,Michael W. Mahoney,Hao Wang,Yan Liu,Huzefa Rangwala,George Karypis,Bernie Wang

Main category: cs.CL

TL;DR: This paper investigates when and how incorporating textual information improves time series forecasting, finding that benefits depend on both model capabilities and data characteristics.

Details

Motivation: There is growing interest in integrating textual information into foundation models for time series forecasting, but it remains unclear under what conditions such integration yields consistent gains. Method: The authors systematically evaluated two multimodal forecasting paradigms—aligning-based and prompting-based methods—across a benchmark of 14 forecasting tasks spanning multiple domains. Result: The results show that the benefits of multimodal input are not universal and depend heavily on dataset properties and model architecture. Textual information is most beneficial with high-capacity text models, weaker time series models, appropriate alignment strategies, sufficient training data, and complementary predictive signals from text. Conclusion: The study provides empirical findings on the effectiveness of multimodal integration in time series forecasting, offering practical guidelines for its application based on model and data characteristics. Abstract: Recently, there has been growing interest in incorporating textual information into foundation models for time series forecasting. However, it remains unclear whether and under what conditions such multimodal integration consistently yields gains. We systematically investigate these questions across a diverse benchmark of 14 forecasting tasks spanning 7 domains, including health, environment, and economics. We evaluate two popular multimodal forecasting paradigms: aligning-based methods, which align time series and text representations; and prompting-based methods, which directly prompt large language models for forecasting. Although prior works report gains from multimodal input, we find these effects are not universal across datasets and models, and multimodal methods sometimes do not outperform the strongest unimodal baselines. To understand when textual information helps, we disentangle the effects of model architectural properties and data characteristics. Our findings highlight that on the modeling side, incorporating text information is most helpful given (1) high-capacity text models, (2) comparatively weaker time series models, and (3) appropriate aligning strategies. On the data side, performance gains are more likely when (4) sufficient training data is available and (5) the text offers complementary predictive signal beyond what is already captured from the time series alone. Our empirical findings offer practical guidelines for when multimodality can be expected to aid forecasting tasks, and when it does not.

[50] AdaptGOT: A Pre-trained Model for Adaptive Contextual POI Representation Learning

Xiaobin Ren,Xinyu Zhu,Kaiqi Zhao

Main category: cs.CL

TL;DR: 本文提出了 AdaptGOT 模型，用于解决当前 POI 嵌入方法在多上下文采样、上下文探索、通用性和泛化能力方面的不足，取得了优异的实验结果。

Details

Motivation: 尽管在特定任务的端到端模型在 POI 嵌入方面取得成功，但仍然存在多上下文采样策略有效性不足、对多个 POI 上下文探索不够、通用性有限和泛化能力不足等问题。 Method: 提出了一种名为 AdaptGOT 的模型，结合了自适应表示学习技术和 GOT（地理共现文本）表示方法，包含三个关键组件：(1) 结合多种混合采样策略的情境邻域生成；(2) 基于注意力机制增强的 GOT 表示；(3) 基于 MoE 的自适应编码器-解码器架构，通过最小化不同上下文之间的 Jensen-Shannon 散度来保证拓扑一致性并丰富上下文表示。 Result: AdaptGOT 模型能够有效捕捉 POI 之间的复杂关系，并在多个 POI 任务中表现出优越的性能。 Conclusion: AdaptGOT 模型在两个真实世界数据集和多个POI任务上的实验验证了其卓越的性能，解决了现有方法在多上下文采样、多POI上下文探索、通用性和泛化能力方面的不足。 Abstract: Currently, considerable strides have been achieved in Point-of-Interest (POI) embedding methodologies, driven by the emergence of novel POI tasks like recommendation and classification. Despite the success of task-specific, end-to-end models in POI embedding, several challenges remain. These include the need for more effective multi-context sampling strategies, insufficient exploration of multiple POI contexts, limited versatility, and inadequate generalization. To address these issues, we propose the AdaptGOT model, which integrates both the (Adapt)ive representation learning technique and the Geographical-Co-Occurrence-Text (GOT) representation with a particular emphasis on Geographical location, Co-Occurrence and Textual information. The AdaptGOT model comprises three key components: (1) contextual neighborhood generation, which integrates advanced mixed sampling techniques such as KNN, density-based, importance-based, and category-aware strategies to capture complex contextual neighborhoods; (2) an advanced GOT representation enhanced by an attention mechanism, designed to derive high-quality, customized representations and efficiently capture complex interrelations between POIs; and (3) the MoE-based adaptive encoder-decoder architecture, which ensures topological consistency and enriches contextual representation by minimizing Jensen-Shannon divergence across varying contexts. Experiments on two real-world datasets and multiple POI tasks substantiate the superior performance of the proposed AdaptGOT model.

[51] ChildGuard: A Specialized Dataset for Combatting Child-Targeted Hate Speech

Gautam Siddharth Kashyap,Mohammad Anas Azeez,Rafiq Ali,Zohaib Hasan Siddiqui,Jiechao Gao,Usman Naseem

Main category: cs.CL

TL;DR: 本文介绍了 ChildGuard1，这是一个为解决在线儿童定向仇恨言论问题而设计的数据集，弥补了现有数据集在年龄特定标注和上下文理解方面的不足。

Details

Motivation: 现有的仇恨言论数据集缺乏针对儿童的具体标注，未能捕捉复杂的上下文并对儿童情感影响进行建模。 Method: 从现有语料库中整理并丰富了儿童特定注释，引入了 ChildGuard1 数据集，并测试了最先进的仇恨言论检测方法。 Result: 提出了一个多样化的数据集 ChildGuard1，用于填补儿童目标仇恨言论方面的空白，并评估了现有检测模型的效果。 Conclusion: ChildGuard1 提供了一个专门针对儿童的仇恨言论数据集，旨在改进对儿童目标仇恨言论的检测和缓解方法。 Abstract: The increasing prevalence of child-targeted hate speech online underscores the urgent need for specialized datasets to address this critical issue. Existing hate speech datasets lack agespecific annotations, fail to capture nuanced contexts, and overlook the unique emotional impact on children. To bridge this gap, we introduce ChildGuard1, a curated dataset derived from existing corpora and enriched with child-specific annotations. ChildGuard captures diverse contexts of child-targeted hate speech, spanning age groups. We benchmark existing state-of-the-art hate speech detection methods, including Large Language Models (LLMs), and assess their effectiveness in detecting and contextualizing child-targeted hate speech. To foster further research in this area, we publicly release ChildGuard, providing a robust foundation for developing improved methods to detect and mitigate such harm.

[52] LastingBench: Defend Benchmarks Against Knowledge Leakage

Yixiong Fang,Tianran Sun,Yuling Shi,Min Wang,Xiaodong Gu

Main category: cs.CL

TL;DR: 本文介绍了一个名为LastingBench的新框架，旨在通过识别并改写可能导致知识泄露的信息来强化基准测试，以此来防止大型语言模型在测试中利用记忆数据进行'作弊'，从而确保评估结果的真实性和公平性。

Details

Motivation: 大型语言模型在标准问答基准测试中通过记忆任务特定数据来'作弊'的问题日益严重，这削弱了基准测试的有效性，并且很少有研究关注减轻这种影响并保持基准测试的长期效用。 Method: 通过扰动识别上下文中的泄漏点，然后将这些泄漏点重写为反事实点，从而打破记忆化过程，同时保持基准的原始评估意图。 Result: 对最先进的问答基准测试进行评估后发现了显著的性能差距，这表明LastingBench在减少记忆效应方面非常有效。 Conclusion: LastingBench是一个有效的框架，可以持续加强和保护现有基准测试免受知识泄露的影响。 Abstract: The increasing complexity of large language models (LLMs) raises concerns about their ability to "cheat" on standard Question Answering (QA) benchmarks by memorizing task-specific data. This undermines the validity of benchmark evaluations, as they no longer reflect genuine model capabilities but instead the effects of data leakage. While prior work has focused on detecting such leakage, little attention has been given to mitigating its impact and preserving the long-term utility of benchmarks. In this paper, we introduce LastingBench, a novel framework designed to continuously reinforce and safeguard existing benchmarks against knowledge leakage. LastingBench identifies leakage points in the context through perturbation, then rewrites the leakage points to counterfactual ones-disrupting memorization while preserving the benchmark's original evaluative intent. Evaluations of state-of-the-art QA benchmarks show significant performance gaps, highlighting the efficacy of LastingBench in reducing memorization effects. LastingBench offers a practical and scalable solution to ensure benchmark robustness over time, promoting fairer and more interpretable evaluations of LLMs.

[53] Refine Medical Diagnosis Using Generation Augmented Retrieval and Clinical Practice Guidelines

Wenhao Li,Hongkuan Zhang,Hongwei Zhang,Zhengxu Li,Zengjie Dong,Yafan Chen,Niranjan Bidargaddi,Hong Liu

Main category: cs.CL

TL;DR: The paper introduces GARMLE-G, a new framework that improves the clinical utility of medical language models by grounding their outputs in authoritative clinical practice guidelines, offering a scalable, low-cost, and hallucination-free solution with potential for broad clinical deployment.

Details

Motivation: Current medical language models typically predict ICD code-based diagnosis from electronic health records because these labels are readily available. However, ICD codes do not capture the nuanced, context-rich reasoning clinicians use for diagnosis. This misalignment limits the clinical utility of existing models. Method: We introduce GARMLE-G, a Generation-Augmented Retrieval framework that grounds medical language model outputs in authoritative CPGs. It (1) integrates LLM predictions with EHR data to create semantically rich queries, (2) retrieves relevant CPG knowledge snippets via embedding similarity, and (3) fuses guideline content with model output to generate clinically aligned recommendations. Result: A prototype system for hypertension diagnosis was developed and evaluated on multiple metrics, demonstrating superior retrieval precision, semantic relevance, and clinical guideline adherence compared to RAG-based baselines, while maintaining a lightweight architecture suitable for localized healthcare deployment. Conclusion: This work provides a scalable, low-cost, and hallucination-free method for grounding medical language models in evidence-based clinical practice, with strong potential for broader clinical deployment. Abstract: Current medical language models, adapted from large language models (LLMs), typically predict ICD code-based diagnosis from electronic health records (EHRs) because these labels are readily available. However, ICD codes do not capture the nuanced, context-rich reasoning clinicians use for diagnosis. Clinicians synthesize diverse patient data and reference clinical practice guidelines (CPGs) to make evidence-based decisions. This misalignment limits the clinical utility of existing models. We introduce GARMLE-G, a Generation-Augmented Retrieval framework that grounds medical language model outputs in authoritative CPGs. Unlike conventional Retrieval-Augmented Generation based approaches, GARMLE-G enables hallucination-free outputs by directly retrieving authoritative guideline content without relying on model-generated text. It (1) integrates LLM predictions with EHR data to create semantically rich queries, (2) retrieves relevant CPG knowledge snippets via embedding similarity, and (3) fuses guideline content with model output to generate clinically aligned recommendations. A prototype system for hypertension diagnosis was developed and evaluated on multiple metrics, demonstrating superior retrieval precision, semantic relevance, and clinical guideline adherence compared to RAG-based baselines, while maintaining a lightweight architecture suitable for localized healthcare deployment. This work provides a scalable, low-cost, and hallucination-free method for grounding medical language models in evidence-based clinical practice, with strong potential for broader clinical deployment.

[54] TIM: A Large-Scale Dataset and large Timeline Intelligence Model for Open-domain Timeline Summarization

Chuanrui Hu,Wei Hu,Penghang Yu,Hua Zhang,Bing-Kun Bao

Main category: cs.CL

TL;DR: This paper introduces a new Timeline Intelligence Model (TIM) for Open-domain Timeline Summarization (TLS), which improves upon existing methods by better capturing topic evolution and filtering irrelevant details through a progressive optimization strategy.

Details

Motivation: Existing methods using general Large Language Models (LLMs) struggle with assessing topic relevance and understanding the evolution of topics, often resulting in irrelevant or inaccurate summaries. This necessitates a more effective approach for Open-domain Timeline Summarization (TLS). Method: A progressive optimization strategy was employed to develop TIM. This involved instruction tuning for improved summarization and filtering of irrelevant information, followed by dual-alignment reward learning incorporating semantic and temporal perspectives. Result: TIM, optimized through a progressive strategy on a large-scale TLS dataset, showed enhanced performance in summarizing open-domain timelines by accurately capturing topic evolution and filtering out irrelevant content. Conclusion: The proposed Timeline Intelligence Model (TIM) demonstrates robust capabilities in summarizing open-domain timelines, outperforming existing methods by effectively filtering irrelevant details and improving understanding of topic evolution. Abstract: Open-domain Timeline Summarization (TLS) is crucial for monitoring the evolution of news topics. To identify changes in news topics, existing methods typically employ general Large Language Models (LLMs) to summarize relevant timestamps from retrieved news. While general LLMs demonstrate capabilities in zero-shot news summarization and timestamp localization, they struggle with assessing topic relevance and understanding topic evolution. Consequently, the summarized information often includes irrelevant details or inaccurate timestamps. To address these issues, we propose the first large Timeline Intelligence Model (TIM) for open-domain TLS, which is capable of effectively summarizing open-domain timelines. Specifically, we begin by presenting a large-scale TLS dataset, comprising over 1,000 news topics and more than 3,000 annotated TLS instances. Furthermore, we propose a progressive optimization strategy, which gradually enhance summarization performance. It employs instruction tuning to enhance summarization and topic-irrelevant information filtering capabilities. Following this, it exploits a novel dual-alignment reward learning method that incorporates both semantic and temporal perspectives, thereby improving the understanding of topic evolution principles. Through this progressive optimization strategy, TIM demonstrates a robust ability to summarize open-domain timelines. Extensive experiments in open-domain demonstrate the effectiveness of our TIM.

[55] TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge

Zhiyuan Zhang,Xiaosong Jia,Guanyu Chen,Qifeng Li,Junchi Yan

Main category: cs.CL

TL;DR: TrajTok是一种结合了数据驱动和规则方法的轨迹分词器，提高了行为生成模型的性能，在Waymo挑战赛中表现优异。

Details

Motivation: 为了提高轨迹预测模型的覆盖率、对称性和鲁棒性，研究者提出了TrajTok这一轨迹分词器。 Method: TrajTok采用离散下一个标记预测方法，结合了数据驱动和基于规则的方法，并引入了一种空间感知的交叉熵损失平滑方法。 Result: 使用TrajTok和SMART模型在Waymo开放模拟代理挑战赛2025上达到了0.7852的真实感得分。 Conclusion: TrajTok展示了在行为生成模型中结合数据驱动和基于规则的方法的有效性，并通过SMART模型在Waymo开放模拟代理挑战赛中实现了卓越的性能。 Abstract: In this technical report, we introduce TrajTok, a trajectory tokenizer for discrete next-token-prediction based behavior generation models, which combines data-driven and rule-based methods with better coverage, symmetry and robustness, along with a spatial-aware label smoothing method for cross-entropy loss. We adopt the tokenizer and loss for the SMART model and reach a superior performance with realism score of 0.7852 on the Waymo Open Sim Agents Challenge 2025. We will open-source the code in the future.

[56] IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech

Siyi Zhou,Yiquan Zhou,Yi He,Xun Zhou,Jinchao Wang,Wei Deng,Jingchen Shu

Main category: cs.CL

TL;DR: 本文提出了一种新的自回归模型友好型语音时长控制方法，解决了传统自回归系统在语音合成中难以精确控制语音时长的问题，适用于视频配音等需要严格音视频同步的应用场景。

Details

Motivation: 尽管自回归系统在语音自然度上具有一定的优势，但由于其逐token生成机制，难以精确控制合成语音的持续时间，在需要严格音视频同步的应用（如视频配音）中是一个关键限制。 Method: 提出了一个新颖的、适合自回归模型的语音时长控制方法，支持两种生成模式：一种允许显式指定生成token的数量以精确控制时长；另一种无需人工输入，让模型自由生成语音，同时保留输入提示的韵律特征。此外，引入了GPT潜在表示来提高语音稳定性，并设计了一种基于文本描述的软指令机制。 Result: 实验结果表明，IndexTTS2在词错误率、说话人相似度和情感保真度方面优于现有的最先进零样本TTS模型。 Conclusion: IndexTTS2实现了情感和说话人身份的解耦，能够独立控制音色和情感，并在零样本设置下完美再现输入提示的情感特征。同时，通过设计基于文本描述的软指令机制，降低了情感控制的门槛，使用户可以通过自然语言输入有效引导语音生成。 Abstract: Large-scale text-to-speech (TTS) models are typically categorized into autoregressive and non-autoregressive systems. Although autoregressive systems exhibit certain advantages in speech naturalness, their token-by-token generation mechanism makes it difficult to precisely control the duration of synthesized speech. This is a key limitation in applications such as video dubbing that require strict audio-visual synchronization. This paper introduces IndexTTS2, which proposes a novel and autoregressive-model-friendly method for speech duration control. The method supports two generation modes: one allows explicit specification of the number of generated tokens for precise duration control; the other does not require manual input and lets the model freely generate speech while preserving prosodic characteristics from the input prompt. Furthermore, IndexTTS2 achieves disentanglement between emotional expression and speaker identity, enabling independent control of timbre and emotion. In the zero-shot setting, the model can perfectly reproduce the emotional characteristics of the input prompt. Users may also provide a separate emotion prompt, even from a different speaker, allowing the model to reconstruct the target timbre while conveying the desired emotion. To enhance clarity during strong emotional expressions, we incorporate GPT latent representations to improve speech stability. Meanwhile, to lower the barrier for emotion control, we design a soft instruction mechanism based on textual descriptions by fine-tuning Qwen3. This enables effective guidance of speech generation with desired emotional tendencies using natural language input. Experimental results demonstrate that IndexTTS2 outperforms existing state-of-the-art zero-shot TTS models in word error rate, speaker similarity, and emotional fidelity.

[57] How Large Language Models play humans in online conversations: a simulated study of the 2016 US politics on Reddit

Daniele Cirulli,Giulio Cimini,Giovanni Palermo

Main category: cs.CL

TL;DR: This study demonstrates that GPT-4 can produce realistic and politically aligned Reddit comments that could potentially influence political discussions, although artificial and real comments remain distinguishable in semantic space.

Details

Motivation: The motivation behind this study is to understand how Large Language Models (LLMs) can replicate human interactions, particularly in politically sensitive online discussions, and assess their potential to influence debates and manipulate discourse. Method: The research conducted three experiments using GPT-4 to generate Reddit comments by impersonating either real or artificial partisan users during the 2016 US Presidential election. These generated comments were analyzed based on political alignment, sentiment, and linguistic features, then compared with actual user contributions and a null model. Result: GPT-4 was found to generate realistic comments supporting or opposing candidates, with a tendency to create consensus rather than dissent. While real and artificial comments were distinguishable through semantic embedding, they appeared indistinguishable during manual review. Conclusion: The study concludes that GPT-4 can generate realistic comments in the context of politically divisive discussions, capable of mimicking user-generated content and influencing political narratives. However, there is a clear distinction between real and artificial comments in semantically embedded space, even though they are indistinguishable upon manual inspection. Abstract: Large Language Models (LLMs) have recently emerged as powerful tools for natural language generation, with applications spanning from content creation to social simulations. Their ability to mimic human interactions raises both opportunities and concerns, particularly in the context of politically relevant online discussions. In this study, we evaluate the performance of LLMs in replicating user-generated content within a real-world, divisive scenario: Reddit conversations during the 2016 US Presidential election. In particular, we conduct three different experiments, asking GPT-4 to generate comments by impersonating either real or artificial partisan users. We analyze the generated comments in terms of political alignment, sentiment, and linguistic features, comparing them against real user contributions and benchmarking against a null model. We find that GPT-4 is able to produce realistic comments, both in favor of or against the candidate supported by the community, yet tending to create consensus more easily than dissent. In addition we show that real and artificial comments are well separated in a semantically embedded space, although they are indistinguishable by manual inspection. Our findings provide insights on the potential use of LLMs to sneak into online discussions, influence political debate and shape political narratives, bearing broader implications of AI-driven discourse manipulation.

[58] The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofs

Jasper Dekoninck,Ivo Petrov,Kristian Minchev,Mislav Balunovic,Martin Vechev,Miroslav Marinov,Maria Drencheva,Lyuba Konova,Milen Shumanov,Kaloyan Tsvetkov,Nikolay Drenchev,Lazar Todorov,Kalina Nikolova,Nikolay Georgiev,Vanesa Kalinkova,Margulan Ismoldayev

Main category: cs.CL

TL;DR: 本文提出了Open Proof Corpus (OPC)，这是一个大规模、高质量的人类评估证明数据集，用于推动自动化证明生成的研究。

Details

Motivation: 现有的大语言模型在数学证明生成方面取得了进展，但由于缺乏大规模、高质量的人类评估证明数据集，进一步发展受到阻碍。 Method: 提出了一个包含5000多个人类评估证明的数据集OPC，并利用该数据集探索自动化证明生成中的几个关键问题。 Result: OPC是首个包含大量由LLM生成的正确解决方案的数据集，这些方案来自如USAMO和IMO等著名的数学竞赛问题。此外，通过OPC的研究，发现自然语言与形式化证明生成之间仍存在性能差距，同时验证最终答案准确性和完整证明有效性之间也存在差异。 Conclusion: OPC不仅推动了证明生成研究的发展，还展示了其在训练模型评估证明正确性方面的实用性。 Abstract: In recent months, large language models (LLMs) have made significant progress in mathematical proof generation, but further advancement is hindered by the lack of a large-scale, high-quality dataset of human-evaluated proofs. While expensive to create, such a dataset is essential for driving improvements in training and enabling a rigorous analysis of proof generation capabilities. In this work, we present the Open Proof Corpus (OPC), a dataset comprising over 5,000 human-evaluated proofs produced by state-of-the-art LLMs. The OPC was specifically designed for broad applicability and downstream usage in proof generation research and is the first to include a substantial number of correct, LLM-generated solutions to problems from prestigious mathematics competitions such as the USAMO and IMO. Using the OPC, we explore critical questions in automated proof generation: (1) the performance gap between natural language and formal proof generation, (2) the discrepancy between final-answer accuracy and full-proof validity, and (3) the impact of best-of-n selection on proof quality. Finally, to showcase the utility of the OPC, we finetune an 8B-parameter model on the dataset, obtaining a model that performs on par with the best model, Gemini-2.5-Pro, on the task of evaluating proof correctness.

[59] Adapting Foundation Speech Recognition Models to Impaired Speech: A Semantic Re-chaining Approach for Personalization of German Speech

Niclas Pokel,Pehuén Moure,Roman Boehringer,Yingqiang Gao

Main category: cs.CL

TL;DR: This study proposes a lightweight, personalized ASR pipeline that enhances transcription accuracy for individuals with speech impairments by enriching datasets with semantic coherence.

Details

Motivation: Speech impairments caused by conditions like cerebral palsy or genetic disorders create significant challenges for ASR systems. Existing models struggle with non-normative speech due to limited training data and annotation difficulties. Method: A personalized ASR model pipeline was developed, focusing on semantic coherence and enrichment of a small, speech-impaired dataset. The method was applied to data from a child with a structural speech impairment. Result: The approach showed promising improvements in transcription quality when applied to a child's speech data with structural impairment, indicating its effectiveness for atypical speech patterns. Conclusion: The proposed lightweight pipeline for personalizing ASR models demonstrates promising improvements in transcription quality for speech-impaired individuals, showing potential to reduce communication barriers. Abstract: Speech impairments caused by conditions such as cerebral palsy or genetic disorders pose significant challenges for automatic speech recognition (ASR) systems. Despite recent advances, ASR models like Whisper struggle with non-normative speech due to limited training data and the difficulty of collecting and annotating non-normative speech samples. In this work, we propose a practical and lightweight pipeline to personalize ASR models, formalizing the selection of words and enriching a small, speech-impaired dataset with semantic coherence. Applied to data from a child with a structural speech impairment, our approach shows promising improvements in transcription quality, demonstrating the potential to reduce communication barriers for individuals with atypical speech patterns.

[60] Performance of diverse evaluation metrics in NLP-based assessment and text generation of consumer complaints

Peiheng Gao,Chen Yang,Ning Sun,Ričardas Zitikis

Main category: cs.CL

TL;DR: 这项研究通过结合专家训练的算法和合成数据生成方法，提高了机器学习在文本分类中的性能和鲁棒性，特别是在处理消费者投诉方面。

Details

Motivation: 动机是解决自然语言中细微的语言模式和上下文变化的准确捕捉问题，特别是在消费者投诉领域。 Method: 该研究采用了结合人类经验训练的算法和合成数据生成方法，利用生成对抗网络的专家评估，并通过专家注释进行优化。 Result: 结果表明，所提出的方法能够有效识别对于评估消费者救济资格至关重要的微妙语义差异，并提升了分类器的性能。 Conclusion: 该研究得出结论，结合专家训练的分类器和高质量的合成数据可以显著提升机器学习在文本分类任务中的性能，减少数据集获取成本，并改善整体评估指标和鲁棒性。 Abstract: Machine learning (ML) has significantly advanced text classification by enabling automated understanding and categorization of complex, unstructured textual data. However, accurately capturing nuanced linguistic patterns and contextual variations inherent in natural language, particularly within consumer complaints, remains a challenge. This study addresses these issues by incorporating human-experience-trained algorithms that effectively recognize subtle semantic differences crucial for assessing consumer relief eligibility. Furthermore, we propose integrating synthetic data generation methods that utilize expert evaluations of generative adversarial networks and are refined through expert annotations. By combining expert-trained classifiers with high-quality synthetic data, our research seeks to significantly enhance machine learning classifier performance, reduce dataset acquisition costs, and improve overall evaluation metrics and robustness in text classification tasks.

[61] Doc2SAR: A Synergistic Framework for High-Fidelity Extraction of Structure-Activity Relationships from Scientific Documents

Jiaxi Zhuang,Kangning Li,Jue Hou,Mingjun Xu,Zhifeng Gao,Hengxing Cai

Main category: cs.CL

TL;DR: 本文提出Doc2SAR，一种结合了领域特定工具和改进的多模态大语言模型的新方法，用于高效提取科学文献中的结构-活性关系（SAR），在新提出的DocSAR-200数据集上表现优异。

Details

Motivation: 从科学文献和专利中提取分子结构-活性关系（SAR）对于药物发现和材料研究至关重要，但现有的方法由于文档格式异构而面临挑战。 Method: 开发了一种新的协同框架Doc2SAR，将领域特定工具与经过监督微调（SFT）增强的MLLM相结合，并在DocSAR-200基准上进行了评估。 Result: Doc2SAR在多种文档类型上实现了最先进的性能，整体表格召回率达到80.78%，比端到端的GPT-4o高出51.48%。 Conclusion: Doc2SAR通过结合领域特定工具和增强的MLLMs，在提取科学文献中的SAR方面达到了最先进的性能，并提供了实用的网页应用程序。 Abstract: Extracting molecular structure-activity relationships (SARs) from scientific literature and patents is essential for drug discovery and materials research. However, this task remains challenging due to heterogeneous document formats and limitations of existing methods. Specifically, rule-based approaches relying on rigid templates fail to generalize across diverse document layouts, while general-purpose multimodal large language models (MLLMs) lack sufficient accuracy and reliability for specialized tasks, such as layout detection and optical chemical structure recognition (OCSR). To address these challenges, we introduce DocSAR-200, a rigorously annotated benchmark of 200 scientific documents designed specifically for evaluating SAR extraction methods. Additionally, we propose Doc2SAR, a novel synergistic framework that integrates domain-specific tools with MLLMs enhanced via supervised fine-tuning (SFT). Extensive experiments demonstrate that Doc2SAR achieves state-of-the-art performance across various document types, significantly outperforming leading end-to-end baselines. Specifically, Doc2SAR attains an overall Table Recall of 80.78% on DocSAR-200, exceeding end2end GPT-4o by 51.48%. Furthermore, Doc2SAR demonstrates practical usability through efficient inference and is accompanied by a web app.

[62] Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations

Li Zhou,Hao Jiang,Junjie Li,Zefeng Zhao,Feng Jiang,Wenyu Chen,Haizhou Li

Main category: cs.CL

TL;DR: This paper introduces an information-theoretic probing framework to evaluate how structural modeling improves language model representations. It finds that MLPs, especially as feature-transformation modules, outperform traditional GNN approaches in capturing linguistic knowledge.

Details

Motivation: Recent studies show that GNNs do not fully utilize structural information while MLPs perform surprisingly well in structure-aware tasks. This paper aims to assess how explicit structural modeling enhances language model representations. Method: A probing framework was developed with a control module that selectively uses GNN components (message-passing and feature-transformation) or MLPs. The Edge Probing Suite was used to evaluate the encoded linguistic knowledge. Result: MLPs used as feature-transformation modules consistently improved linguistic knowledge across different architectures, encoding both syntactic and semantic patterns. GNNs with feature-transformation also showed benefits, while models relying solely on message-passing underperformed. Conclusion: MLPs can serve as efficient alternatives to GNNs for enhancing linguistic knowledge in language models, particularly through feature-transformation operations. Abstract: Explicit structural information has been proven to be encoded by Graph Neural Networks (GNNs), serving as auxiliary knowledge to enhance model capabilities and improve performance in downstream NLP tasks. However, recent studies indicate that GNNs fail to fully utilize structural information, whereas Multi-Layer Perceptrons (MLPs), despite lacking the message-passing mechanisms inherent to GNNs, exhibit a surprising ability in structure-aware tasks. Motivated by these findings, this paper introduces a comprehensive probing framework from an information-theoretic perspective. The framework is designed to systematically assess the role of explicit structural modeling in enhancing language model (LM) representations and to investigate the potential of MLPs as efficient and scalable alternatives to GNNs. We extend traditional probing classifiers by incorporating a control module that allows for selective use of either the full GNN model or its decoupled components, specifically, the message-passing and feature-transformation operations.This modular approach isolates and assesses the individual contributions of these operations, avoiding confounding effects from the complete GNN architecture. Using the Edge Probing Suite, a diagnostic tool for evaluating the linguistic knowledge encoded in LMs, we find that MLPs, when used as feature-transformation modules, consistently improve the linguistic knowledge captured in LM representations across different architectures. They effectively encode both syntactic and semantic patterns. Similarly, GNNs that incorporate feature-transformation operations show beneficial effects. In contrast, models that rely solely on message-passing operations tend to underperform, often leading to negative impacts on probing task performance.

[63] ANUBHUTI: A Comprehensive Corpus For Sentiment Analysis In Bangla Regional Languages

Swastika Kundu,Autoshi Ibrahim,Mithila Rahman,Tanvir Ahmed

Main category: cs.CL

TL;DR: This paper presents ANUBHUTI, a new dataset for sentiment analysis in Bangla dialects, addressing the lack of resources for processing these low-resource languages.

Details

Motivation: Sentiment analysis for regional dialects of Bangla remains an underexplored area due to linguistic diversity and limited annotated data. Method: This paper introduces ANUBHUTI, a comprehensive dataset consisting of 2000 sentences manually translated from standard Bangla into four major regional dialects Mymensingh, Noakhali, Sylhet, and Chittagong. Each sentence is annotated using a dual annotation scheme: multiclass thematic labeling categorizes sentences as Political, Religious, or Neutral, and multilabel emotion annotation assigns one or more emotions from Anger, Contempt, Disgust, Enjoyment, Fear, Sadness, and Surprise. Expert native translators conducted the translation and annotation, with quality assurance performed via Cohens Kappa inter annotator agreement. The dataset was further refined through systematic checks for missing data, anomalies, and inconsistencies. Result: The dataset predominantly features political and religious content, reflecting the contemporary socio political landscape of Bangladesh, alongside neutral texts to maintain balance. Conclusion: ANUBHUTI fills a critical gap in resources for sentiment analysis in low resource Bangla dialects, enabling more accurate and context aware natural language processing. Abstract: Sentiment analysis for regional dialects of Bangla remains an underexplored area due to linguistic diversity and limited annotated data. This paper introduces ANUBHUTI, a comprehensive dataset consisting of 2000 sentences manually translated from standard Bangla into four major regional dialects Mymensingh, Noakhali, Sylhet, and Chittagong. The dataset predominantly features political and religious content, reflecting the contemporary socio political landscape of Bangladesh, alongside neutral texts to maintain balance. Each sentence is annotated using a dual annotation scheme: multiclass thematic labeling categorizes sentences as Political, Religious, or Neutral, and multilabel emotion annotation assigns one or more emotions from Anger, Contempt, Disgust, Enjoyment, Fear, Sadness, and Surprise. Expert native translators conducted the translation and annotation, with quality assurance performed via Cohens Kappa inter annotator agreement, achieving strong consistency across dialects. The dataset was further refined through systematic checks for missing data, anomalies, and inconsistencies. ANUBHUTI fills a critical gap in resources for sentiment analysis in low resource Bangla dialects, enabling more accurate and context aware natural language processing.

[64] Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers

Tzu-Quan Lin,Hsi-Chun Cheng,Hung-yi Lee,Hao Tang

Main category: cs.CL

TL;DR: 该研究发现了自监督语音Transformer中与说话人信息相关的神经元，并通过保护这些神经元显著保持了说话人相关任务的性能。

Details

Motivation: 近年来，自监督语音Transformer对说话人相关应用产生了影响，但关于这些模型如何编码说话人信息的研究较少。 Method: 分析与k-means聚类的自监督特征和i-vector相关的神经元，以确定其与说话人信息的关系。 Result: 发现这些聚类对应于广泛的语音和性别类别，并且适合用于识别表示说话人的神经元。 Conclusion: 通过保护与说话人信息相关的神经元，可以显著保持说话人相关任务的性能，证明了这些神经元在编码说话人信息中的关键作用。 Abstract: In recent years, the impact of self-supervised speech Transformers has extended to speaker-related applications. However, little research has explored how these models encode speaker information. In this work, we address this gap by identifying neurons in the feed-forward layers that are correlated with speaker information. Specifically, we analyze neurons associated with k-means clusters of self-supervised features and i-vectors. Our analysis reveals that these clusters correspond to broad phonetic and gender classes, making them suitable for identifying neurons that represent speakers. By protecting these neurons during pruning, we can significantly preserve performance on speaker-related task, demonstrating their crucial role in encoding speaker information.

[65] (Fact) Check Your Bias

Eivind Morris Bakke,Nora Winger Heggelund

Main category: cs.CL

TL;DR: 该研究分析了Llama 3.1的参数知识偏差如何影响HerO事实核查系统的判断，发现模型存在固有的负面偏见，并且不同的提示策略会影响检索到的证据，但最终的判断预测仍然保持稳定。

Details

Motivation: 自动事实核查系统越来越依赖大型语言模型（LLMs），但这些模型中的参数知识偏差可能会影响事实核查的结果。研究旨在探讨这些偏差如何影响HerO系统的判断。 Method: 研究者调查了Llama 3.1的参数知识偏差和有意注入的偏差对HerO系统事实核查结果的影响。他们通过直接提示Llama 3.1进行事实验证，并测试了生成支持、反驳或中立文档的提示如何影响检索结果。 Result: 当直接提示进行事实验证时，Llama 3.1将近一半的主张标记为“证据不足”。使用其参数知识，它能够对剩余一半的主张得出结论。提示模型生成特定视角的事实核查文档显著影响检索结果，大约50%的检索证据是每个视角独有的。 Conclusion: 尽管提示策略不同，最终的判断预测在HerO系统中表现出稳定性，但模型有时拒绝为它认为错误的主张生成支持性文件，显示出固有的负面偏见。 Abstract: Automatic fact verification systems increasingly rely on large language models (LLMs). We investigate how parametric knowledge biases in these models affect fact-checking outcomes of the HerO system (baseline for FEVER-25). We examine how the system is affected by: (1) potential bias in Llama 3.1's parametric knowledge and (2) intentionally injected bias. When prompted directly to perform fact-verification, Llama 3.1 labels nearly half the claims as "Not Enough Evidence". Using only its parametric knowledge it is able to reach a verdict on the remaining half of the claims. In the second experiment, we prompt the model to generate supporting, refuting, or neutral fact-checking documents. These prompts significantly influence retrieval outcomes, with approximately 50\% of retrieved evidence being unique to each perspective. Notably, the model sometimes refuses to generate supporting documents for claims it believes to be false, creating an inherent negative bias. Despite differences in retrieved evidence, final verdict predictions show stability across prompting strategies. The code is available at: https://github.com/eibakke/FEVER-8-Shared-Task

[66] Evaluating List Construction and Temporal Understanding capabilities of Large Language Models

Alexandru Dumitru,V Venktesh,Adam Jatowt,Avishek Anand

Main category: cs.CL

TL;DR: This paper introduces TLQA, a new benchmark designed to test models' abilities to construct lists and understand temporal information simultaneously. It reveals that current LLMs struggle significantly with these combined tasks, highlighting areas for future research.

Details

Motivation: Large Language Models (LLMs) often struggle with temporal understanding tasks involving multiple entities, particularly in associating entities with accurate time intervals, generating complete lists of entities, and reasoning about specific events. Existing benchmarks do not adequately evaluate these capabilities, prompting the need for TLQA. Method: The authors introduced a new benchmark called Time referenced List based Question Answering (TLQA), which requires structured answers in list format aligned with time periods. They evaluated both closed-book and open-domain generative models on this benchmark. Result: Findings show that current models perform poorly on TLQA, especially in providing complete answers and aligning facts temporally in closed-book settings. In open-domain settings, there is a need to improve retrieval methods to better handle such tasks. Conclusion: The paper concludes that current state-of-the-art generative models have significant shortcomings in temporal understanding and list construction tasks, especially in closed-book setups, and highlights the need for improved retrieval mechanisms in open-domain setups. Abstract: Large Language Models (LLMs) have demonstrated immense advances in a wide range of natural language tasks. However, these models are susceptible to hallucinations and errors on particularly temporal understanding tasks involving multiple entities in answers. In such tasks, they fail to associate entities with accurate time intervals, generate a complete list of entities in answers or reason about events associated with specific temporal bounds. Existing works do not extensively evaluate the abilities of the model to perform implicit and explicit temporal understanding in a list answer construction setup. To bridge this gap, we propose the Time referenced List based Question Answering or TLQA benchmark that requires structured answers in list format aligned with corresponding time periods. Our TLQA benchmark, requires both list construction and temporal understanding simultaneously, which to the best of our knowledge has not been explored in prior benchmarks. We investigate the temporal understanding and list construction capabilities of state-of-the-art generative models on TLQA in closed-book and open-domain settings. Our findings reveal significant shortcomings in current models, particularly their inability to provide complete answers and temporally align facts in a closed-book setup and the need to improve retrieval in open-domain setup, providing clear future directions for research on TLQA. The benchmark and code at https://github.com/elixir-research-group/TLQA.

[67] Offensive Language Detection on Social Media Using XLNet

Reem Alothman,Hafida Benhidour,Said Kerrache

Main category: cs.CL

TL;DR: This research shows that XLNet is more effective than BERT in detecting offensive language on social media, highlighting the benefits of transfer learning.

Details

Motivation: The motivation stems from the impracticality of manual moderation due to the enormous volume of user-generated content on social media, necessitating automated systems for detecting offensive language. Method: The study proposes an automatic offensive language detection model based on XLNet and compares its performance with BERT using the Offensive Language Identification Dataset (OLID). Result: XLNet outperforms BERT in detecting offensive content and categorizing types of offenses, while BERT slightly better identifies offense targets. Oversampling and undersampling strategies improve classification performance. Conclusion: This study concludes that XLNet-based architectures are effective in creating robust systems for detecting offensive language on social media platforms, highlighting the potential of transfer learning. Abstract: The widespread use of text-based communication on social media-through chats, comments, and microblogs-has improved user interaction but has also led to an increase in offensive content, including hate speech, racism, and other forms of abuse. Due to the enormous volume of user-generated content, manual moderation is impractical, which creates a need for automated systems that can detect offensive language. Deep learning models, particularly those using transfer learning, have demonstrated significant success in understanding natural language through large-scale pretraining. In this study, we propose an automatic offensive language detection model based on XLNet, a generalized autoregressive pretraining method, and compare its performance with BERT (Bidirectional Encoder Representations from Transformers), which is a widely used baseline in natural language processing (NLP). Both models are evaluated using the Offensive Language Identification Dataset (OLID), a benchmark Twitter dataset that includes hierarchical annotations. Our experimental results show that XLNet outperforms BERT in detecting offensive content and in categorizing the types of offenses, while BERT performs slightly better in identifying the targets of the offenses. Additionally, we find that oversampling and undersampling strategies are effective in addressing class imbalance and improving classification performance. These findings highlight the potential of transfer learning and XLNet-based architectures to create robust systems for detecting offensive language on social media platforms.

[68] A suite of allotaxonometric tools for the comparison of complex systems using rank-turbulence divergence

Jonathan St-Onge,Ashley M. A. Fehr,Carter Ward,Calla G. Beauregard,Michael V. Arnold,Samuel F. Rosenblatt,Benjamin Cooley,Christopher M. Danforth,Peter Sheridan Dodds

Main category: cs.CL

TL;DR: This paper introduces allotaxonographs as a principled method for visually comparing complex systems through heavy-tailed distributions, and presents associated software tools in Matlab, JavaScript, and Python.

Details

Motivation: There is a need for theoretically grounded, principled tools to describe and compare complex systems, particularly when dealing with heavy-tailed distributions. Allotaxonographs, built around type turbulence phenomena, aim to fulfill this need by offering visual and quantitative comparisons. Method: The authors describe a suite of computational tools for rendering allotaxonographs using rank-turbulence divergence, and presumably test their performance and applicability across different programming environments. Result: A set of programmatic tools for generating allotaxonographs was developed in Matlab, JavaScript, and Python, each suited to different use cases, demonstrating the versatility and broad applicability of the method. Conclusion: The paper concludes that allotaxonographs are effective, principled tools for comparing complex systems through the analysis of heavy-tailed distributions, and that the availability of programmatic tools across different languages enhances their utility. Abstract: Describing and comparing complex systems requires principled, theoretically grounded tools. Built around the phenomenon of type turbulence, allotaxonographs provide map-and-list visual comparisons of pairs of heavy-tailed distributions. Allotaxonographs are designed to accommodate a wide range of instruments including rank- and probability-turbulence divergences, Jenson-Shannon divergence, and generalized entropy divergences. Here, we describe a suite of programmatic tools for rendering allotaxonographs for rank-turbulence divergence in Matlab, Javascript, and Python, all of which have different use cases.

[69] Towards Transparent AI: A Survey on Explainable Large Language Models

Avash Palikhe,Zhenyu Yu,Zichong Wang,Wenbin Zhang

Main category: cs.CL

TL;DR: This paper surveys explainable AI (XAI) methods for Large Language Models (LLMs), categorizing them based on transformer architectures and discussing their evaluation, applications, and future research directions.

Details

Motivation: LLMs are often 'black boxes', making it difficult to understand their decision-making processes. This lack of transparency hinders their adoption in high-stakes domains where interpretability is essential. Method: The authors systematically review XAI techniques tailored for LLMs, categorizing them according to the underlying transformer architectures: encoder-only, decoder-only, and encoder-decoder models. They also analyze how these methods are evaluated and applied in practice. Result: The survey provides a structured overview of current explainability techniques for LLMs, identifies available resources, and highlights challenges and future directions in developing transparent and responsible LLMs. Conclusion: A comprehensive understanding of existing XAI methods can guide ongoing efforts to improve the explainability and trustworthiness of LLMs, particularly in critical application domains. Abstract: Large Language Models (LLMs) have played a pivotal role in advancing Artificial Intelligence (AI). However, despite their achievements, LLMs often struggle to explain their decision-making processes, making them a 'black box' and presenting a substantial challenge to explainability. This lack of transparency poses a significant obstacle to the adoption of LLMs in high-stakes domain applications, where interpretability is particularly essential. To overcome these limitations, researchers have developed various explainable artificial intelligence (XAI) methods that provide human-interpretable explanations for LLMs. However, a systematic understanding of these methods remains limited. To address this gap, this survey provides a comprehensive review of explainability techniques by categorizing XAI methods based on the underlying transformer architectures of LLMs: encoder-only, decoder-only, and encoder-decoder models. Then these techniques are examined in terms of their evaluation for assessing explainability, and the survey further explores how these explanations are leveraged in practical applications. Finally, it discusses available resources, ongoing research challenges, and future directions, aiming to guide continued efforts toward developing transparent and responsible LLMs.

[70] Exploring the Structure of AI-Induced Language Change in Scientific English

Riley Galpin,Bryce Anderson,Tom S. Juzek

Main category: cs.CL

TL;DR: This study examines how Large Language Models influence scientific English, revealing that linguistic changes are more semantic and widespread rather than limited lexical replacements.

Details

Motivation: To understand if the linguistic changes attributed to Large Language Models involve replacement of synonyms or broader semantic and pragmatic shifts. Method: Analyzed frequency trends of 'spiking words' and synonym groups in PubMed abstracts using part-of-speech tagging to detect lexical and structural linguistic shifts. Result: Findings indicate that entire semantic clusters shift together rather than isolated lexical replacements, distinguishing model-induced changes from organic language decline patterns. Conclusion: The changes in scientific English are primarily semantic and pragmatic, influenced by Large Language Models, while some words show decline indicating complex language evolution. Abstract: Scientific English has undergone rapid and unprecedented changes in recent years, with words such as "delve," "intricate," and "crucial" showing significant spikes in frequency since around 2022. These changes are widely attributed to the growing influence of Large Language Models like ChatGPT in the discourse surrounding bias and misalignment. However, apart from changes in frequency, the exact structure of these linguistic shifts has remained unclear. The present study addresses this and investigates whether these changes involve the replacement of synonyms by suddenly 'spiking words,' for example, "crucial" replacing "essential" and "key," or whether they reflect broader semantic and pragmatic qualifications. To further investigate structural changes, we include part of speech tagging in our analysis to quantify linguistic shifts over grammatical categories and differentiate between word forms, like "potential" as a noun vs. as an adjective. We systematically analyze synonym groups for widely discussed 'spiking words' based on frequency trends in scientific abstracts from PubMed. We find that entire semantic clusters often shift together, with most or all words in a group increasing in usage. This pattern suggests that changes induced by Large Language Models are primarily semantic and pragmatic rather than purely lexical. Notably, the adjective "important" shows a significant decline, which prompted us to systematically analyze decreasing lexical items. Our analysis of "collapsing" words reveals a more complex picture, which is consistent with organic language change and contrasts with the patterns of the abrupt spikes. These insights into the structure of language change contribute to our understanding of how language technology continues to shape human language.

[71] PARSI: Persian Authorship Recognition via Stylometric Integration

Kourosh Shahnazari,Mohammadali Keshtparvar,Seyed Moein Ayyoubzadeh

Main category: cs.CL

TL;DR: 本研究提出了一种结合深度学习与领域特征的新方法，在67位著名诗人创作的647,653节诗句中实现了高效的作者归属。

Details

Motivation: 波斯古典诗歌复杂的语言、风格和韵律特性给计算作者归属带来了挑战，需要开发一个多功能框架来解决这些问题。 Method: 使用基于Transformer的语言编码器以及包括Word2Vec嵌入、风格测量和韵律编码在内的多输入神经框架进行分类，并通过多数投票和加权投票方案评估。 Result: 该模型在0.9阈值下实现97%的准确率（覆盖率较低），加权投票方案获得71%的准确率。 Conclusion: 研究结果表明，结合深度表示形式与领域特定特征的方法在波斯古典诗歌的作者归属上具有潜力，并有助于风格分析、作者争议和计算文学研究。 Abstract: The intricate linguistic, stylistic, and metrical aspects of Persian classical poetry pose a challenge for computational authorship attribution. In this work, we present a versatile framework to determine authorship among 67 prominent poets. We employ a multi-input neural framework consisting of a transformer-based language encoder complemented by features addressing the semantic, stylometric, and metrical dimensions of Persian poetry. Our feature set encompasses 100-dimensional Word2Vec embeddings, seven stylometric measures, and categorical encodings of poetic form and meter. We compiled a vast corpus of 647,653 verses of the Ganjoor digital collection, validating the data through strict preprocessing and author verification while preserving poem-level splitting to prevent overlap. This work employs verse-level classification and majority and weighted voting schemes in evaluation, revealing that weighted voting yields 71% accuracy. We further investigate threshold-based decision filtering, allowing the model to generate highly confident predictions, achieving 97% accuracy at a 0.9 threshold, though at lower coverage. Our work focuses on the integration of deep representational forms with domain-specific features for improved authorship attribution. The results illustrate the potential of our approach for automated classification and the contribution to stylistic analysis, authorship disputes, and general computational literature research. This research will facilitate further research on multilingual author attribution, style shift, and generative modeling of Persian poetry.

[72] LinguaSynth: Heterogeneous Linguistic Signals for News Classification

Duo Zhang,Junyi Mo

Main category: cs.CL

TL;DR: This paper presents LinguaSynth, an interpretable and computationally efficient text classification framework that challenges the necessity of deep neural networks in achieving high performance in NLP tasks.

Details

Motivation: Deep learning has significantly advanced NLP, but its reliance on large black-box models introduces critical interpretability and computational efficiency concerns. Method: This paper proposes LinguaSynth, a novel text classification framework that strategically integrates five complementary linguistic feature types: lexical, syntactic, entity-level, word-level semantics, and document-level semantics within a transparent logistic regression model. Result: LinguaSynth maintains interpretability and computational efficiency, achieving an accuracy of 84.89 percent on the 20 Newsgroups dataset and surpassing a robust TF-IDF baseline by 3.32 percent. Conclusion: LinguaSynth sets a new benchmark for interpretable, resource-efficient NLP models and challenges the prevailing assumption that deep neural networks are necessary for high-performing text classification. Abstract: Deep learning has significantly advanced NLP, but its reliance on large black-box models introduces critical interpretability and computational efficiency concerns. This paper proposes LinguaSynth, a novel text classification framework that strategically integrates five complementary linguistic feature types: lexical, syntactic, entity-level, word-level semantics, and document-level semantics within a transparent logistic regression model. Unlike transformer-based architectures, LinguaSynth maintains interpretability and computational efficiency, achieving an accuracy of 84.89 percent on the 20 Newsgroups dataset and surpassing a robust TF-IDF baseline by 3.32 percent. Through rigorous feature interaction analysis, we show that syntactic and entity-level signals provide essential disambiguation and effectively complement distributional semantics. LinguaSynth sets a new benchmark for interpretable, resource-efficient NLP models and challenges the prevailing assumption that deep neural networks are necessary for high-performing text classification.

[73] The Consistency Hypothesis in Uncertainty Quantification for Large Language Models

Quan Xiao,Debarun Bhattacharjya,Balaji Ganesan,Radu Marinescu,Katsiaryna Mirylenka,Nhan H Pham,Michael Glass,Junkyu Lee

Main category: cs.CL

TL;DR: This paper explores using generation consistency as a proxy for confidence in large language models, particularly focusing on black-box uncertainty quantification methods and introducing the effective 'Sim-Any' hypothesis for confidence estimation.

Details

Motivation: Estimating the confidence of large language model outputs is crucial for real-world applications requiring high user trust, and black-box uncertainty quantification methods are popular due to their practical benefits. Method: The paper introduces mathematical statements and statistical tests to evaluate the consistency hypothesis, which relates generation consistency to confidence. It proposes data-free black-box UQ methods based on similarity aggregation between LLM outputs. Result: Empirical results across 8 benchmark datasets and 3 tasks demonstrate that the 'Sim-Any' hypothesis is actionable and can outperform existing baselines in confidence estimation. Conclusion: The study concludes that the 'Sim-Any' hypothesis can be effectively used for confidence estimation in black-box uncertainty quantification methods, showing practical value in tasks like question answering, text summarization, and text-to-SQL. Abstract: Estimating the confidence of large language model (LLM) outputs is essential for real-world applications requiring high user trust. Black-box uncertainty quantification (UQ) methods, relying solely on model API access, have gained popularity due to their practical benefits. In this paper, we examine the implicit assumption behind several UQ methods, which use generation consistency as a proxy for confidence, an idea we formalize as the consistency hypothesis. We introduce three mathematical statements with corresponding statistical tests to capture variations of this hypothesis and metrics to evaluate LLM output conformity across tasks. Our empirical investigation, spanning 8 benchmark datasets and 3 tasks (question answering, text summarization, and text-to-SQL), highlights the prevalence of the hypothesis under different settings. Among the statements, we highlight the `Sim-Any' hypothesis as the most actionable, and demonstrate how it can be leveraged by proposing data-free black-box UQ methods that aggregate similarities between generations for confidence estimation. These approaches can outperform the closest baselines, showcasing the practical value of the empirically observed consistency hypothesis.

[74] Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models

Taiga Someya,Ryo Yoshida,Hitomi Yanaka,Yohei Oseki

Main category: cs.CL

TL;DR: This paper uses Derivational Probing to show that BERT builds syntactic representations from the bottom up, with local structures forming early and global structures developing later.

Details

Motivation: While neural language models are known to encode syntax, how these structures are derived across model layers remains unclear. Method: The paper introduces Derivational Probing to analyze how syntactic structures are built across layers of BERT. Result: Experiments show that lower layers encode micro-syntactic structures (e.g., noun phrases), which are later integrated into macro-syntactic structures (e.g., verb dependencies) in higher layers. Subject-verb agreement evaluation further demonstrates the importance of timing in structure formation for downstream performance. Conclusion: Derivational Probing reveals that BERT constructs syntactic structures in a bottom-up manner, with micro-syntactic structures emerging first and gradually integrating into macro-syntactic structures. Abstract: Recent work has demonstrated that neural language models encode syntactic structures in their internal representations, yet the derivations by which these structures are constructed across layers remain poorly understood. In this paper, we propose Derivational Probing to investigate how micro-syntactic structures (e.g., subject noun phrases) and macro-syntactic structures (e.g., the relationship between the root verbs and their direct dependents) are constructed as word embeddings propagate upward across layers. Our experiments on BERT reveal a clear bottom-up derivation: micro-syntactic structures emerge in lower layers and are gradually integrated into a coherent macro-syntactic structure in higher layers. Furthermore, a targeted evaluation on subject-verb number agreement shows that the timing of constructing macro-syntactic structures is critical for downstream performance, suggesting an optimal timing for integrating global syntactic information.

[75] DeepTalk: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE

Hang Shao,Heting Gao,Yunhang Shen,Jiawei Chen,Lijiang Li,Zuwei Long,Bo Tong,Ke Li,Xing Sun

Main category: cs.CL

TL;DR: 本文提出了DeepTalk，一种基于专家混合架构的自适应模态专家学习框架，以解决原生多模态大语言模型中的灾难性遗忘和性能下降问题。

Details

Motivation: 由于配对语音文本数据不足，原生多模态大语言模型在预训练过程中面临灾难性遗忘和性能下降的问题。 Method: DeepTalk通过根据LLM内的模态负载自适应地区分模态专家，并对每个模态专家进行专门的单模态训练，然后进行联合多模态协作训练来实现这一目标。 Result: DeepTalk相比原生MLLMs（如GLM-4-Voice）平均超过20%的性能下降，仅产生5.5%的性能下降，并且端到端对话延迟保持在0.5秒以内。 Conclusion: DeepTalk是一个基于专家混合架构的自适应模态专家学习框架，用于解决原生多模态大语言模型中的灾难性遗忘和性能下降问题。 Abstract: Native multimodal large language models (MLLMs) restructure a single large language model (LLM) into a spoken language model (SLM) capable of both speech and text generation. Compared to modular and aligned MLLMs, native MLLMs preserve richer paralinguistic features such as emotion and prosody, and generate speech responses directly within the backbone LLM rather than using a separate speech decoder. This integration also results in lower response latency and smoother interaction. However, native MLLMs suffer from catastrophic forgetting and performance degradation because the available paired speech-text data is insufficient to support the pretraining of MLLMs compared to the vast amount of text data required to pretrain text LLMs. To address this issue, we propose DeepTalk, a framework for adaptive modality expert learning based on a Mixture of Experts (MoE) architecture. DeepTalk first adaptively distinguishes modality experts according to their modality load within the LLM. Each modality expert then undergoes specialized single-modality training, followed by joint multimodal collaborative training. As a result, DeepTalk incurs only a 5.5% performance drop compared to the original LLM, which is significantly lower than the average performance drop of over 20% typically seen in native MLLMs (such as GLM-4-Voice), and is on par with modular MLLMs. Meanwhile, the end-to-end dialogue latency remains within 0.5 seconds, ensuring a seamless and intelligent speech interaction experience. Code and models are released at https://github.com/talkking/DeepTalk.

[76] WildSpeech-Bench: Benchmarking Audio LLMs in Natural Speech Conversation

Jian Zhang,Linhao Zhang,Bokai Lei,Chuhan Wu,Wei Jia,Xiao Zhou

Main category: cs.CL

TL;DR: 本文提出了一个针对语音大语言模型（LLM）的新评估方法，包括基于真实语音场景的数据集构建和查询感知评估技术，从而实现对语音模型在复杂场景下性能的全面分析。

Details

Motivation: 当前缺乏专门且全面的端到端语音LLM基准测试，而现有的文本基准测试方法忽略了语音的独特特性（如语调、同音词、口吃等），这阻碍了优化语音LLM在现实应用中的用户体验。 Method: 系统整理与口语场景相关的实际对话数据，引入说话人属性和声学条件的多样性，利用语音特有现象增强数据集；同时设计查询感知的评估方法，采用定制化检查清单和提示提升自动评估的准确性。 Result: 作者对多种主流语音模型进行了全面测试和详细分析，揭示了不同语音场景下模型性能的显著差异；并且通过使用查询感知评估方法，实现了对语音特定场景的更细粒度评估。 Conclusion: 该论文提出了一种新的语音LLM评估方法，通过设计查询感知的评估方式和构建具有语音特征的数据集，能够更细致地评估不同语音场景下的模型性能，并为语音模型的开发和评估提供有价值的见解。 Abstract: Recent multi-modal Large Language Models (LLMs) such as GPT-4o have demonstrated strong capabilities of direct speech interaction. However, the lack of specialized and comprehensive benchmarks for end-to-end speech LLM evaluation hinders optimizing the user experience of Audio LLMs in real-world applications. Existing evaluation methods often adapt text-based benchmarks, overlooking speech's unique characteristics and challenges, including prosody, homophones, stuttering, and differing user expectations. Here, we present a novel approach to thoroughly evaluate LLMs in practical speech conversations. We systematically curate real-world chat data relevant to spoken scenarios, introduce diversity in speaker attributes and acoustic conditions, and augment the dataset with speech-specific phenomena. We further design a query-aware evaluation method to use customized evaluation checklists and prompts to enhance the accuracy of automatic evaluation. We conduct comprehensive testing and detailed analysis of various mainstream speech models, revealing significant differences in model performance across different speech scenarios. The use of query-aware evaluation further enables a finer-grained assessment under various speech-specific scenarios. Our benchmark can provide valuable insights for speech model development and evaluation.

[77] Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation

Qiyue Gao,Xinyu Pi,Kevin Liu,Junrong Chen,Ruolan Yang,Xinqi Huang,Xinyu Fang,Lu Sun,Gautham Kishore,Bo Ai,Stone Tao,Mengyang Liu,Jiaxi Yang,Chao-Jung Lai,Chuanyang Jin,Jiannan Xiang,Benhao Huang,Zeming Chen,David Danks,Hao Su,Tianmin Shu,Ziqiao Ma,Lianhui Qin,Zhiting Hu

Main category: cs.CL

TL;DR: This paper evaluates the fundamental world modeling (WM) abilities of recent large Vision-Language Models (VLMs) using a novel framework based on comparative psychology and cognitive science. Despite their promise, VLMs show notable deficiencies in key WM skills, such as motion trajectory discrimination and disentangled reasoning, revealing substantial gaps compared to human-level world modeling.

Details

Motivation: Recent large Vision-Language Models (VLMs) show potential as general-purpose world models (WMs), but existing studies have only evaluated specific capabilities like visual understanding without providing a systematic assessment of their fundamental WM abilities. This research aims to fill this gap by systematically evaluating VLMs' core world modeling capabilities. Method: Drawing on comparative psychology and cognitive science, the researchers propose a two-stage framework to assess Perception (visual, spatial, temporal, quantitative, and motion) and Prediction (mechanistic simulation, transitive inference, compositional inference). They introduce WM-ABench, a large-scale benchmark with 23 fine-grained evaluation dimensions across six diverse simulated environments. The evaluation involves 660 experiments on 15 of the latest commercial and open-source VLMs. Result: Findings reveal that VLMs have striking limitations in basic world modeling abilities. For example, most models perform near-random accuracy when distinguishing motion trajectories and demonstrate a lack of disentangled understanding—some models even assume blue objects move faster than green ones. These results highlight significant gaps between current VLMs and human-level world modeling. Conclusion: The study concludes that despite the potential of recent large Vision-Language Models (VLMs) as general-purpose world models (WMs), they exhibit significant limitations in fundamental WM abilities, indicating a gap between VLMs and human-level world modeling. Abstract: Internal world models (WMs) enable agents to understand the world's state and predict transitions, serving as the basis for advanced deliberative reasoning. Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs' fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses Perception (visual, spatial, temporal, quantitative, and motion) and Prediction (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, almost all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding -- e.g., some models tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling.

[78] A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs

Sean Kim,Hyuhng Joon Kim

Main category: cs.CL

TL;DR: 本文提出了一种评估大型语言模型在多语言环境下行为的新框架。

Details

Motivation: 随着大型语言模型在不同语言和文化背景下的广泛应用，理解其在事实性和争议性情境中的行为变得尤为重要。 Method: 通过两阶段评估方法，第一阶段评估事实性问题的回答一致性，第二阶段研究地缘政治敏感争议中的反应倾向。 Result: 结果表明，在第一阶段中存在由查询语言引起的对齐现象，而第二阶段则反映了模型训练背景与查询语言之间的相互作用。 Conclusion: 该论文提供了一个评估大型语言模型在中立和敏感话题上行为的结构化框架，并为未来的大型语言模型部署提供了见解。 Abstract: As large language models (LLMs) are increasingly deployed across diverse linguistic and cultural contexts, understanding their behavior in both factual and disputable scenarios is essential, especially when their outputs may shape public opinion or reinforce dominant narratives. In this paper, we define two types of bias in LLMs: model bias (bias stemming from model training) and inference bias (bias induced by the language of the query), through a two-phase evaluation. Phase 1 evaluates LLMs on factual questions where a single verifiable answer exists, assessing whether models maintain consistency across different query languages. Phase 2 expands the scope by probing geopolitically sensitive disputes, where responses may reflect culturally embedded or ideologically aligned perspectives. We construct a manually curated dataset spanning both factual and disputable QA, across four languages and question types. The results show that Phase 1 exhibits query language induced alignment, while Phase 2 reflects an interplay between the model's training context and query language. This paper offers a structured framework for evaluating LLM behavior across neutral and sensitive topics, providing insights for future LLM deployment and culturally aware evaluation practices in multilingual contexts.

[79] AutoMixer: Checkpoint Artifacts as Automatic Data Mixers

Ernie Chang,Yang Li,Patrick Huber,David Kant,Yangyang Shi,Vikas Chandra

Main category: cs.CL

TL;DR: 本文提出了一种利用训练过程中的检查点模型来优化数据混合的方法，在多个推理基准上实现了性能提升。

Details

Motivation: 为了获得各种任务所需的能力，需要找到合适的数据混合，但数据与任务之间的关系难以建模。此外，训练过程中保存的检查点通常未被充分利用。 Method: 通过使用检查点模型在各自基准任务上的能力识别它们，并利用其对源数据的一阶影响近似来作为数据混合器。 Result: 在八个推理基准上证明了所提框架在预训练设置中的显著改进，性能提升最高达1.93%。 Conclusion: 利用检查点模型可以提高数据质量和优化数据混合，展示了其在语言模型训练中的潜力。 Abstract: In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be modeled. In this work, we observe that checkpoint models exhibit emerging capabilities at different points in the training trajectory. Often, the training process saves checkpoints as artifacts that are under-utilized as a source of in-training data signals. We identify these artifact models based on their respective capabilities on the benchmarks and leverage them as data mixers by using their aggregated first-order influence approximation over source data. We demonstrated on eight reasoning benchmarks that the proposed framework shows significant improvements in the pretraining setting, with performance improvements of up to 1.93%. Overall, this shows the potential of checkpoint models to enhance data quality and optimize data mixtures.

[80] PapersPlease: A Benchmark for Evaluating Motivational Values of Large Language Models Based on ERG Theory

Junho Myung,Yeon Su Park,Sunwoo Kim,Shin Yoo,Alice Oh

Main category: cs.CL

TL;DR: PapersPlease是一个基于ERG理论的新基准，揭示了LLMs在道德决策中的隐含偏好以及对边缘化身份的偏见。

Details

Motivation: 评估大型语言模型（LLMs）在角色扮演情景中的表现和偏见，因为它们在此类情境中通常表现出有偏见的行为。 Method: 使用存在、关系和成长理论构建包含3,700个道德困境的数据集，模拟LLMs作为移民检查员的决策过程，并分析其决策模式及社会身份影响。 Result: 分析六个LLM显示出统计上显著的决策模式，并发现某些模型对边缘化身份的拒绝率更高，且受动机需求和社会身份线索的影响。 Conclusion: PapersPlease是一个新的基准，用于研究LLMs在优先考虑各级人类需求时的决策，并揭示了LLMs在决策中的隐含偏好和对边缘身份的较高拒绝率。 Abstract: Evaluating the performance and biases of large language models (LLMs) through role-playing scenarios is becoming increasingly common, as LLMs often exhibit biased behaviors in these contexts. Building on this line of research, we introduce PapersPlease, a benchmark consisting of 3,700 moral dilemmas designed to investigate LLMs' decision-making in prioritizing various levels of human needs. In our setup, LLMs act as immigration inspectors deciding whether to approve or deny entry based on the short narratives of people. These narratives are constructed using the Existence, Relatedness, and Growth (ERG) theory, which categorizes human needs into three hierarchical levels. Our analysis of six LLMs reveals statistically significant patterns in decision-making, suggesting that LLMs encode implicit preferences. Additionally, our evaluation of the impact of incorporating social identities into the narratives shows varying responsiveness based on both motivational needs and identity cues, with some models exhibiting higher denial rates for marginalized identities. All data is publicly available at https://github.com/yeonsuuuu28/papers-please.

[81] More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents

Weimin Xiong,Ke Wang,Yifan Song,Hanchao Liu,Sai Zhou,Wei Peng,Sujian Li

Main category: cs.CL

TL;DR: 本研究指出并验证了工具集成的LLM代理在工具调用过程各阶段的易错性，强调了评估其稳定性的必要性，并为未来LLM的发展提供指导。

Details

Motivation: 目前对工具集成的LLM代理的评估通常侧重于端到端的工具使用评估，而忽略了它们的稳定性，这限制了它们在现实世界中的应用。 Method: 通过广泛的实验，研究代理在工具调用过程中的各个阶段是否容易出错，包括阅读工具文档、选择工具和生成参数以及处理工具响应。 Result: 研究发现，代理在每个阶段都非常容易出错，基于开源模型的代理比基于专有模型的代理更容易受到错误的影响。此外，增加模型大小并不能显著提高工具调用推理能力，反而可能使代理更容易受到类似正常用户指令的攻击。 Conclusion: 评估工具集成的LLM代理的稳定性是至关重要的，这在当前的研究中被忽视了。研究结果强调了评估代理稳定性的必要性，并为未来的LLM开发和评估提供了有价值的见解。 Abstract: Current evaluations of tool-integrated LLM agents typically focus on end-to-end tool-usage evaluation while neglecting their stability. This limits their real-world applicability, as various internal or external factors can cause agents to crash or behave abnormally. Our research addresses this by investigating whether agents are vulnerable to errors throughout the entire tool invocation process, including reading tool documentation, selecting tools and generating parameters, and processing the tool's response. Through extensive experiments, we observe that agents are highly susceptible to errors at each stage and agents based on open-source models are more vulnerable than those based on proprietary models. We also find that increasing the model size does not significantly improve tool invocation reasoning and may make agents more vulnerable to attacks resembling normal user instructions. This highlights the importance of evaluating agent stability and offers valuable insights for future LLM development and evaluation.

[82] Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses

Mohamed Ahmed,Mohamed Abdelmouty,Mingyu Kim,Gunvanth Kandula,Alex Park,James C. Davis

Main category: cs.CL

TL;DR: This paper proposes hybrid jailbreaking methods combining token- and prompt-level attacks to exploit vulnerabilities in Pre-Trained and Large Language Models, demonstrating higher attack success rates and resilience against advanced defenses.

Details

Motivation: Despite the success of Pre-Trained Language Models (PTLMs) and Large Language Models (LLMs), they remain vulnerable to attacks exploiting their weaknesses to bypass safety measures. Current inference-phase threats like token-level and prompt-level jailbreaks have complementary limitations, prompting the need for more effective and robust attack strategies. Method: The authors proposed two hybrid approaches, GCG + PAIR and GCG + WordGame, integrating token- and prompt-level techniques. They evaluated these methods across multiple Vicuna and Llama models to assess attack-success rates and robustness against advanced defenses. Result: GCG + PAIR consistently increased attack-success rates on undefended models, achieving an ASR of 91.6% on Llama-3 compared to PAIR's 58.4%. GCG + WordGame maintained a high ASR of over 80% even under stricter evaluators like Mistral-Sorry-Bench. Both hybrids retained transferability and effectively bypassed advanced defenses such as Gradient Cuff and JBShield, which blocked single-mode attacks. Conclusion: The paper concludes that hybrid approaches combining token- and prompt-level techniques enhance jailbreak effectiveness across diverse PTLMs, exposing previously unreported vulnerabilities in current safety stacks and highlighting the need for holistic safeguards against adaptive adversaries. Abstract: The advancement of Pre-Trained Language Models (PTLMs) and Large Language Models (LLMs) has led to their widespread adoption across diverse applications. Despite their success, these models remain vulnerable to attacks that exploit their inherent weaknesses to bypass safety measures. Two primary inference-phase threats are token-level and prompt-level jailbreaks. Token-level attacks embed adversarial sequences that transfer well to black-box models like GPT but leave detectable patterns and rely on gradient-based token optimization, whereas prompt-level attacks use semantically structured inputs to elicit harmful responses yet depend on iterative feedback that can be unreliable. To address the complementary limitations of these methods, we propose two hybrid approaches that integrate token- and prompt-level techniques to enhance jailbreak effectiveness across diverse PTLMs. GCG + PAIR and the newly explored GCG + WordGame hybrids were evaluated across multiple Vicuna and Llama models. GCG + PAIR consistently raised attack-success rates over its constituent techniques on undefended models; for instance, on Llama-3, its Attack Success Rate (ASR) reached 91.6%, a substantial increase from PAIR's 58.4% baseline. Meanwhile, GCG + WordGame matched the raw performance of WordGame maintaining a high ASR of over 80% even under stricter evaluators like Mistral-Sorry-Bench. Crucially, both hybrids retained transferability and reliably pierced advanced defenses such as Gradient Cuff and JBShield, which fully blocked single-mode attacks. These findings expose previously unreported vulnerabilities in current safety stacks, highlight trade-offs between raw success and defensive robustness, and underscore the need for holistic safeguards against adaptive adversaries.

[83] Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism

Simon Münker,Nils Schwager,Achim Rettinger

Main category: cs.CL

TL;DR: 该论文探讨了使用大型语言模型模拟社交网络用户行为的方法与挑战，强调了验证模拟真实性的必要性，并提出了更严格应用生成代理建模的建议。

Details

Motivation: 由于关于大型语言模型（LLMs）能否模仿人类行为存在矛盾的研究结果，需要更好地理解实验设计中的差异。 Method: 提供了一个模拟社交网络的正式框架，并实证测试了模仿用户行为的不同方法。 Result: 研究发现，社会模拟的真实性应该在其组件适应的环境中进行测量和验证。 Conclusion: 作者主张在将生成代理模型应用于社会模拟时应更加严谨。 Abstract: The ability of Large Language Models (LLMs) to mimic human behavior triggered a plethora of computational social science research, assuming that empirical studies of humans can be conducted with AI agents instead. Since there have been conflicting research findings on whether and when this hypothesis holds, there is a need to better understand the differences in their experimental designs. We focus on replicating the behavior of social network users with the use of LLMs for the analysis of communication on social networks. First, we provide a formal framework for the simulation of social networks, before focusing on the sub-task of imitating user communication. We empirically test different approaches to imitate user behavior on X in English and German. Our findings suggest that social simulations should be validated by their empirical realism measured in the setting in which the simulation components were fitted. With this paper, we argue for more rigor when applying generative-agent-based modeling for social simulation.

[84] Analyzing and Fine-Tuning Whisper Models for Multilingual Pilot Speech Transcription in the Cockpit

Kartheek Kumar Reddy Nareddy,Sarah Ternus,Julia Niebling

Main category: cs.CL

TL;DR: This paper explores improving the transcription accuracy of cockpit conversations using Whisper models, achieving a significant reduction in Word Error Rate (WER) through normalization schemes and fine-tuning techniques.

Details

Motivation: The motivation is to address the performance limitations of pre-trained models in niche domains such as transcribing pilot speech in cockpits, which involve specific vocabulary and multilingual conversations. Method: The method involves collecting and manually labeling cockpit simulator and pilot interview recordings. Multiple normalization schemes are proposed to refine transcripts, and performance-efficient fine-tuning using Low-Rank Adaptation (LoRA) is employed to enhance ASR performance. Result: The study resulted in a significant decrease in WER, from 68.49% for the baseline pre-trained Whisper Large model without normalization to 26.26% for the fine-tuned Whisper Large model with the proposed normalization scheme. Conclusion: The conclusion is that applying normalization schemes and fine-tuning techniques can substantially improve the transcription accuracy of niche domain conversations like cockpit discussions using Whisper models. Abstract: The developments in transformer encoder-decoder architectures have led to significant breakthroughs in machine translation, Automatic Speech Recognition (ASR), and instruction-based chat machines, among other applications. The pre-trained models were trained on vast amounts of generic data over a few epochs (fewer than five in most cases), resulting in their strong generalization capabilities. Nevertheless, the performance of these models does suffer when applied to niche domains like transcribing pilot speech in the cockpit, which involves a lot of specific vocabulary and multilingual conversations. This paper investigates and improves the transcription accuracy of cockpit conversations with Whisper models. We have collected around 85 minutes of cockpit simulator recordings and 130 minutes of interview recordings with pilots and manually labeled them. The speakers are middle aged men speaking both German and English. To improve the accuracy of transcriptions, we propose multiple normalization schemes to refine the transcripts and improve Word Error Rate (WER). We then employ fine-tuning to enhance ASR performance, utilizing performance-efficient fine-tuning with Low-Rank Adaptation (LoRA). Hereby, WER decreased from 68.49 \% (pretrained whisper Large model without normalization baseline) to 26.26\% (finetuned whisper Large model with the proposed normalization scheme).

[85] Can Peter Pan Survive MT? A Stylometric Study of LLMs, NMTs, and HTs in Children's Literature Translation

Delu Kong,Lieve Macken

Main category: cs.CL

TL;DR: 该研究通过构建 Peter Pan 翻译语料库，利用 stylometric 分析比较了人工翻译和不同机器翻译模型在儿童文学翻译中的表现。

Details

Motivation: 研究机器翻译与人工翻译在英文到中文儿童文学翻译中的表现差异。 Method: 构建了一个 Peter Pan 语料库，包含7个HT、7个LLM和7个NMT，并使用通用特征集和创意文本翻译特征集进行分析。 Result: HT 和 MT 在连词分布和1-gram-YiYang比率上有显著差异；LLM在风格特征上更接近 HT。 Conclusion: LLMs 优于 NMTs，在儿童文学翻译中表现出更强的潜力。 Abstract: This study focuses on evaluating the performance of machine translations (MTs) compared to human translations (HTs) in English-to-Chinese children's literature translation (CLT) from a stylometric perspective. The research constructs a Peter Pan corpus, comprising 21 translations: 7 human translations (HTs), 7 large language model translations (LLMs), and 7 neural machine translation outputs (NMTs). The analysis employs a generic feature set (including lexical, syntactic, readability, and n-gram features) and a creative text translation (CTT-specific) feature set, which captures repetition, rhythm, translatability, and miscellaneous levels, yielding 447 linguistic features in total. Using classification and clustering techniques in machine learning, we conduct a stylometric analysis of these translations. Results reveal that in generic features, HTs and MTs exhibit significant differences in conjunction word distributions and the ratio of 1-word-gram-YiYang, while NMTs and LLMs show significant variation in descriptive words usage and adverb ratios. Regarding CTT-specific features, LLMs outperform NMTs in distribution, aligning more closely with HTs in stylistic characteristics, demonstrating the potential of LLMs in CLT.

[86] Decoding Machine Translationese in English-Chinese News: LLMs vs. NMTs

Delu Kong,Lieve Macken

Main category: cs.CL

TL;DR: 该研究通过大规模数据分析揭示了英文到中文机器翻译输出（MTese）的独特语言特征，并成功区分了机器翻译输出与原始文本以及不同类型的机器翻译系统。

Details

Motivation: 研究聚焦于英文到中文在新闻文本中的机器翻译输出（MTese）这一较少被研究的语言对，旨在探索其语言独特性。 Method: 构建了包含4个子语料库的大型数据集，并采用全面的五层特征集进行分析，使用卡方排名算法进行特征选择，并完成分类和聚类任务。 Result: 确认了神经机器翻译系统（NMTs）和大语言模型（LLMs）中均存在MTese现象；原始中文文本几乎可以完美地区分于LLM和NMT输出；发现MT输出具有较短句长和较多转折连词；LLMs与NMT相比具有更高的词汇多样性，但NMT更多使用括号；翻译专用LLMs相较通用LLMs表现出较低的词汇多样性和较高的因果连词使用率；中外公司开发的LLMs无显著差异。 Conclusion: 机器翻译输出与原始中文文本之间存在明显差异，特定的语言模式能够有效区分不同类型的机器翻译输出。 Abstract: This study explores Machine Translationese (MTese) -- the linguistic peculiarities of machine translation outputs -- focusing on the under-researched English-to-Chinese language pair in news texts. We construct a large dataset consisting of 4 sub-corpora and employ a comprehensive five-layer feature set. Then, a chi-square ranking algorithm is applied for feature selection in both classification and clustering tasks. Our findings confirm the presence of MTese in both Neural Machine Translation systems (NMTs) and Large Language Models (LLMs). Original Chinese texts are nearly perfectly distinguishable from both LLM and NMT outputs. Notable linguistic patterns in MT outputs are shorter sentence lengths and increased use of adversative conjunctions. Comparing LLMs and NMTs, we achieve approximately 70% classification accuracy, with LLMs exhibiting greater lexical diversity and NMTs using more brackets. Additionally, translation-specific LLMs show lower lexical diversity but higher usage of causal conjunctions compared to generic LLMs. Lastly, we find no significant differences between LLMs developed by Chinese firms and their foreign counterparts.

[87] Lost at the Beginning of Reasoning

Baohao Liao,Xinyi Chen,Sara Rajaee,Yuhui Xu,Christian Herold,Anders Søgaard,Maarten de Rijke,Christof Monz

Main category: cs.CL

TL;DR: 本文探讨了大型语言模型在长链推理中初始步骤的重要性及错误传播问题，提出了一种高效采样策略以提升推理效率和质量，并构建了一个用于评估自我纠正能力的新基准测试。

Details

Motivation: 尽管大型语言模型在复杂推理方面取得了进展，但其在长链推理中的自我纠正能力仍未得到充分探索，同时存在冗余推理的问题。 Method: 通过实证研究分析初始推理步骤对整体推理质量的影响，并开发了一种基于奖励模型的高效采样策略以提升推理性能。 Result: 发现初始推理步骤对最终预测具有显著影响，错误的初始步骤会降低后续推理的质量；提出的采样策略可减少高达70%的推理成本而不牺牲准确性。 Conclusion: 论文提出了一种高效的采样策略，利用奖励模型来识别和保留高质量的初始推理步骤，并引入了一个新的基准测试来评估模型的自我纠正能力。 Abstract: Recent advancements in large language models (LLMs) have significantly advanced complex reasoning capabilities, particularly through extended chain-of-thought (CoT) reasoning that incorporates mechanisms such as backtracking, self-reflection and self-correction. Despite these developments, the self-correction abilities of LLMs during long CoT reasoning remain underexplored. And recent findings on overthinking suggest that such models often engage in unnecessarily redundant reasoning. In this work, we empirically show that the first reasoning step exerts a disproportionately large influence on the final prediction - errors introduced at this stage can substantially degrade subsequent reasoning quality. This phenomenon is consistently observed across two state-of-the-art open-source reasoning model families: DeepSeek-R1 and Qwen3. To address this, we propose an efficient sampling strategy that leverages a reward model to identify and retain high-quality first reasoning steps while discarding suboptimal ones, achieving up to a 70% reduction in inference cost without sacrificing accuracy. Finally, we introduce a new benchmark specifically constructed with deliberately flawed first reasoning steps to systematically evaluate model self-correction capabilities, offering a foundation for future research on robust reasoning in LLMs.

[88] MDC-R: The Minecraft Dialogue Corpus with Reference

Chris Madge,Maris Camilleri,Paloma Carretero Garcia,Mladen Karan,Juexi Shao,Prashant Jayannavar,Julian Hough,Benjamin Roth,Massimo Poesio

Main category: cs.CL

TL;DR: 本研究介绍了新语料库 MDC-R，它在原始 Minecraft 对话语料库的基础上增加了指代表达标注，并通过实验证明其在语言理解任务中的价值。

Details

Motivation: MDC 的任务导向、多轮对话以及动态环境中的语言现象促使了多次标注工作，作者认为增加参考标注后的 MDC-R 同样具有价值。 Method: 介绍了一种新的语言资源 MDC-R，并对其进行了定量和定性分析，同时进行了一项简短的实验来验证其有用性。 Result: 提供了详细的标注方法和语料库，并通过实验展示了 MDC-R 在指代表达理解方面的实用性。 Conclusion: MDC-R 可以作为一个有价值的资源用于指代表达理解任务。 Abstract: We introduce the Minecraft Dialogue Corpus with Reference (MDC-R). MDC-R is a new language resource that supplements the original Minecraft Dialogue Corpus (MDC) with expert annotations of anaphoric and deictic reference. MDC's task-orientated, multi-turn, situated dialogue in a dynamic environment has motivated multiple annotation efforts, owing to the interesting linguistic phenomena that this setting gives rise to. We believe it can serve as a valuable resource when annotated with reference, too. Here, we discuss our method of annotation and the resulting corpus, and provide both a quantitative and a qualitative analysis of the data. Furthermore, we carry out a short experiment demonstrating the usefulness of our corpus for referring expression comprehension.

[89] Involvement drives complexity of language in online debates

Eleonora Amadori,Daniele Cirulli,Edoardo Di Martino,Jacopo Nudo,Maria Sahakyan,Emanuele Sangiorgio,Arnaldo Santoro,Simon Zollo,Alessandro Galeazzi,Niccolò Di Marco

Main category: cs.CL

TL;DR: This paper explores how language complexity on Twitter varies across topics, political leanings, account types, and content reliability, offering insights into sociolinguistic dynamics in digital spaces.

Details

Motivation: Language evolves with societal and technological changes, and social media significantly influences public discourse. Understanding linguistic complexity in online content helps assess its societal impact. Method: The authors combined multiple measures of textual complexity to analyze content from influential Twitter users across three major global topics: COVID-19, COP26, and the Russia-Ukraine war. Result: Significant differences were found in language complexity based on account type, political leaning, reliability, and sentiment. Negative and offensive content was more complex, and users with similar views developed shared jargon. Conclusion: The study reveals how language use on digital platforms reflects ideological and social structures, deepening our understanding of sociolinguistic dynamics online. Abstract: Language is a fundamental aspect of human societies, continuously evolving in response to various stimuli, including societal changes and intercultural interactions. Technological advancements have profoundly transformed communication, with social media emerging as a pivotal force that merges entertainment-driven content with complex social dynamics. As these platforms reshape public discourse, analyzing the linguistic features of user-generated content is essential to understanding their broader societal impact. In this paper, we examine the linguistic complexity of content produced by influential users on Twitter across three globally significant and contested topics: COVID-19, COP26, and the Russia-Ukraine war. By combining multiple measures of textual complexity, we assess how language use varies along four key dimensions: account type, political leaning, content reliability, and sentiment. Our analysis reveals significant differences across all four axes, including variations in language complexity between individuals and organizations, between profiles with sided versus moderate political views, and between those associated with higher versus lower reliability scores. Additionally, profiles producing more negative and offensive content tend to use more complex language, with users sharing similar political stances and reliability levels converging toward a common jargon. Our findings offer new insights into the sociolinguistic dynamics of digital platforms and contribute to a deeper understanding of how language reflects ideological and social structures in online spaces.

[90] Identifying a Circuit for Verb Conjugation in GPT-2

David Demitri Africa

Main category: cs.CL

TL;DR: This study isolates a sub-network in GPT-2 Small responsible for subject-verb agreement, showing that a small fraction of the network can achieve near-model performance on basic tasks.

Details

Motivation: The motivation behind this study is to isolate and interpret the sub-network responsible for subject-verb agreement in GPT-2 Small. Method: The author used techniques such as performance verification, automatic circuit discovery via direct path patching, and direct logit attribution to isolate a candidate circuit contributing to correct verb conjugation in GPT-2 Small. Result: A candidate circuit was isolated that contributes significantly to the model's correct verb conjugation when given prompts with singular or plural subjects. Conclusion: The study concludes that only a small fraction of the network's component-token pairs is needed to achieve near-model performance on the base task but substantially more for more complex settings. Abstract: I implement a procedure to isolate and interpret the sub-network (or "circuit") responsible for subject-verb agreement in GPT-2 Small. In this study, the model is given prompts where the subject is either singular (e.g. "Alice") or plural (e.g. "Alice and Bob"), and the task is to correctly predict the appropriate verb form ("walks" for singular subjects, "walk" for plural subjects). Using a series of techniques-including performance verification automatic circuit discovery via direct path patching, and direct logit attribution- I isolate a candidate circuit that contributes significantly to the model's correct verb conjugation. The results suggest that only a small fraction of the network's component-token pairs is needed to achieve near-model performance on the base task but substantially more for more complex settings.

[91] DAPFAM: A Domain-Aware Patent Retrieval Dataset Aggregated at the Family Level

Iliass Ayaou,Denis Cavallucci,Hicham Chibane

Main category: cs.CL

TL;DR: DAPFAM is a new open-access dataset for domain-aware patent retrieval, designed for efficient sub-document level experiments with balanced domains and explicit in/out-of-domain labeling using IPC codes.

Details

Motivation: Existing patent retrieval datasets often lack explicit in-domain and out-of-domain labeling, multi-jurisdiction coverage, balanced query domains, and manageable sizes for sub-document level experiments. This gap motivates the creation of DAPFAM. Method: The authors propose DAPFAM, a domain-aware patent retrieval dataset built at the simple-family level. They use a three-step data-curation pipeline, incorporating relevance judgments through forward/backward citations, random negatives, and a novel labeling scheme based on IPC codes to define in-domain and out-of-domain relationships. Result: DAPFAM contains 1,247 domain-balanced full-text query families and 45,336 full-text target families, forming 49,869 evaluation pairs. The dataset supports lexical and neural retrieval methods and highlights significant challenges in cross-domain patent retrieval. Conclusion: The paper concludes that DAPFAM offers a manageable, multi-jurisdictional, domain-aware dataset for patent retrieval with balanced domains and explicit labeling, which supports sub-document level experiments on moderate computational resources. Abstract: In the landscape of publicly available patent retrieval datasets, the need for explicit indomain and out-of-domain labeling, multi-jurisdiction coverage, balanced query domain representation and manageable sizes that support sub document level experiments on moderate computational resources is often overlooked. To address these gaps, we propose DAPFAM, a new open access domain-aware patent retrieval dataset constructed at the simple-family level. The dataset contains 1,247 domain balanced full text query families and 45,336 full text target families. The dataset is enriched by clear relevance judgments (forward/backward citations as positive links, random negatives), as well as explicit in-domain or out-of-domain relationships via a novel proposed labelling scheme based on via International Patent Classification (IPC) codes, resulting in 49,869 evaluation pairs. The dataset is multi jurisdictional, requires little to no preprocessing for retrieval evaluation, and remains of a size manageable for entities with limited ressources allowing for sub document level retrieval experiments without excessive computational costs. We describe our three-step data-curation pipeline, present comprehensive dataset statistics, and provide baseline experiments using lexical and neural retrieval methods. Our baseline experiments highlight significant challenges in crossdomain patent retrieval. The dataset will be publicly available (for now the access link is this repository: https://osf.io/vbyzd/?view_only=1a40242e0d1941a58aa854af3e50cf6b).

[92] SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition

Muhammad Umar Farooq,Oscar Saz

Main category: cs.CL

TL;DR: 本文通过改进音频拼接和经验回放策略，提高了方言阿拉伯语和阿英混杂语音识别的效果，取得了优于大规模多语言模型的结果。

Details

Motivation: 解决方言阿拉伯语和阿拉伯语-英语代码切换语音数据稀缺的问题，并提升语音自监督学习模型在这些任务上的表现。 Method: 引入一种修改的音频拼接方法生成人工代码切换语音数据，并采用受经验回放（ER）启发的方法增强泛化能力。此外，集成了领域外3-gram语言模型并应用少样本微调策略。 Result: 使用SAGE数据对SSL模型进行微调后，在阿拉伯语和英语代码切换基准测试中实现了7.8%的绝对词错误率（WER）改善。整体平均WER从31.7%降至26.6%，并在少样本条件下进一步降低4.9%。最终在阿拉伯语-英语代码切换基准测试中达到31.1%的WER，优于USM和Whisper-large-v2模型。 Conclusion: 本文提出了一种改进的音频拼接方法（SAGE）和受经验回放启发的方法，以提高方言阿拉伯语和阿英混杂语音的识别性能。结合语言模型和少样本微调策略，该研究在代码切换基准测试中超越了大规模多语言模型。 Abstract: This paper investigates the performance of various speech SSL models on dialectal Arabic (DA) and Arabic-English code-switched (CS) speech. To address data scarcity, a modified audio-splicing approach is introduced to generate artificial CS speech data. Fine-tuning an already fine-tuned SSL model with the proposed Spliced-Audio Generated (SAGE) data results in an absolute improvement on Word Error Rate (WER) of 7.8% on Arabic and English CS benchmarks. Additionally, an Experience Replay (ER) inspired approach is proposed to enhance generalisation across DA and CS speech while mitigating catastrophic forgetting. Integrating an out-of-domain 3-gram language model reduces the overall mean WER from 31.7% to 26.6%. Few-shot fine-tuning for code-switching benchmarks further improves WER by 4.9%. A WER of 31.1% on Arabic-English CS benchmarks surpasses large-scale multilingual models, including USM and Whisper-large-v2 (both over ten times larger) by an absolute margin of 5.5% and 8.4%, respectively.

[93] Training Language Model to Critique for Better Refinement

Tianshu Yu,Chao Xiang,Mingchuan Yang,Pei Ke,Bosi Wen,Cunxiang Wang,Jiale Cheng,Li Zhang,Xinyu Mu,Chuxiong Sun,Minlie Huang

Main category: cs.CL

TL;DR: 本文提出了 RCO 框架，利用反馈循环和改进信号训练批评模型，在多个任务上有效提升大型语言模型的表现。

Details

Motivation: 当前对于哪种类型的批评最能提升模型响应或如何生成此类批评的研究有限，因此提出 RCO 来填补这一空白。 Method: 引入了 RCO 框架，利用反馈循环生成批评并指导响应优化，使用 CU 作为奖励信号进行训练，避免直接评估批评偏好。 Result: RCO 在对话生成、摘要、问答、数学推理和代码生成五个任务上显著优于传统方法和开源模型，展示了其在批评质量和响应改进方面的有效性。 Conclusion: RCO 提供了一种新颖的训练批评模型的方法，通过改进信号来优化面向改进的批评，从而提高大型语言模型在多个任务上的表现。 Abstract: Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce \textbf{R}efinement-oriented \textbf{C}ritique \textbf{O}ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method's effectiveness in enhancing LLM critique-refinement loops.

[94] Leveraging In-Context Learning for Political Bias Testing of LLMs

Patrick Haller,Jannis Vamvas,Rico Sennrich,Lena A. Jäger

Main category: cs.CL

TL;DR: 本文提出了一种新的探测任务“问卷建模”（QM），利用人类调查数据作为上下文示例，以提高基于问题的偏差评估的稳定性，并用于比较指令调整模型与其基础版本。

Details

Motivation: 现有的通过政治问题查询大型语言模型（LLMs）以评估其潜在偏见的方法存在有限的稳定性，这使得模型之间的比较不可靠。因此，需要一种更具稳定性的评估方法。 Method: 提出了一种新的探测任务——问卷建模（Questionnaire Modeling, QM），该任务使用人类调查数据作为上下文示例，以更稳定地评估LLMs的偏见，并进行了不同规模模型的实验，分析指令调整对偏见的影响。 Result: 实验表明，QM提高了问题-based 偏差评估的稳定性，并可用于比较指令调整模型与基础模型。此外，观察到较大的模型在使用上下文示例方面更有效，并且在QM中通常表现出较小的偏见得分。 Conclusion: 研究证明，结合人类调查数据的上下文信息能够提升LLMs偏见评估的稳定性，同时指令调整可能改变偏见方向，而更大的模型在利用上下文信息方面表现更好。 Abstract: A growing body of work has been querying LLMs with political questions to evaluate their potential biases. However, this probing method has limited stability, making comparisons between models unreliable. In this paper, we argue that LLMs need more context. We propose a new probing task, Questionnaire Modeling (QM), that uses human survey data as in-context examples. We show that QM improves the stability of question-based bias evaluation, and demonstrate that it may be used to compare instruction-tuned models to their base versions. Experiments with LLMs of various sizes indicate that instruction tuning can indeed change the direction of bias. Furthermore, we observe a trend that larger models are able to leverage in-context examples more effectively, and generally exhibit smaller bias scores in QM. Data and code are publicly available.

[95] Detection of Personal Data in Structured Datasets Using a Large Language Model

Albert Agisha Ntwali,Luca Rück,Martin Heckmann

Main category: cs.CL

TL;DR: 本文提出了一种基于GPT-4o并融合上下文信息的新方法，用于检测结构化数据集中的个人数据，结果显示其性能优于现有方法，尤其是在使用上下文信息的情况下。

Details

Motivation: 传统的个人数据检测方法未能充分利用数据集的上下文信息，这可能导致检测性能受限。 Method: 利用GPT-4o大型语言模型，并结合数据集中的上下文信息（如其他特征名称和数据集描述）来检测结构化数据集中的个人信息。 Result: 在多个数据集上的评估表明，基于GPT-4o的方法在利用上下文信息时表现优异，尤其是在Kaggle、OpenML和医疗数据集MIMIC-Demo-Ext上。 Conclusion: 进一步的进展将大大受益于更多包含个人信息的真实世界数据集的可用性。 Abstract: We propose a novel approach for detecting personal data in structured datasets, leveraging GPT-4o, a state-of-the-art Large Language Model. A key innovation of our method is the incorporation of contextual information: in addition to a feature's name and values, we utilize information from other feature names within the dataset as well as the dataset description. We compare our approach to alternative methods, including Microsoft Presidio and CASSED, evaluating them on multiple datasets: DeSSI, a large synthetic dataset, datasets we collected from Kaggle and OpenML as well as MIMIC-Demo-Ext, a real-world dataset containing patient information from critical care units. Our findings reveal that detection performance varies significantly depending on the dataset used for evaluation. CASSED excels on DeSSI, the dataset on which it was trained. Performance on the medical dataset MIMIC-Demo-Ext is comparable across all models, with our GPT-4o-based approach clearly outperforming the others. Notably, personal data detection in the Kaggle and OpenML datasets appears to benefit from contextual information. This is evidenced by the poor performance of CASSED and Presidio (both of which do not utilize the context of the dataset) compared to the strong results of our GPT-4o-based approach. We conclude that further progress in this field would greatly benefit from the availability of more real-world datasets containing personal information.

[96] Evaluating Scoring Bias in LLM-as-a-Judge

Qingquan Li,Shaoyu Dou,Kailai Shao,Chao Chen,Haixiang Hu

Main category: cs.CL

TL;DR: This paper investigates scoring bias in using Large Language Models (LLMs) as judges, showing how biases affect model stability and offering methods for mitigation.

Details

Motivation: The motivation stems from the widespread adoption of LLMs as evaluators and the lack of systematic research on scoring-based bias despite its impact on fairness and reliability. Method: A framework was developed to evaluate scoring bias by augmenting existing benchmarks with synthesized data and designing multi-faceted evaluation metrics. Result: Experimental results show that scoring biases disrupt the stability of existing judge models, highlighting the need for better-designed scoring mechanisms. Conclusion: The study concludes that scoring biases significantly affect the stability of current LLM-as-a-Judge models, and provides insights into mitigating these biases through prompt design and selection processes. Abstract: The remarkable performance of Large Language Models (LLMs) gives rise to``LLM-as-a-Judge'', where LLMs are employed as evaluators for complex tasks. Moreover, it has been widely adopted across fields such as Natural Language Processing (NLP), preference learning, and various specific domains. However, there are various biases within LLM-as-a-Judge, which adversely affect the fairness and reliability of judgments. Current research on evaluating or mitigating bias in LLM-as-a-Judge predominantly focuses on comparison-based evaluations, while systematic investigations into bias in scoring-based evaluations remain limited. Therefore, we define scoring bias in LLM-as-a-Judge as the scores differ when scoring judge models are bias-related perturbed, and provide a well-designed framework to comprehensively evaluate scoring bias. We augment existing LLM-as-a-Judge benchmarks through data synthesis to construct our evaluation dataset and design multi-faceted evaluation metrics. Our experimental results demonstrate that the scoring stability of existing judge models is disrupted by scoring biases. Further exploratory experiments and discussions provide valuable insights into the design of scoring prompt templates and the mitigation of scoring biases on aspects such as score rubrics, score IDs, and reference answer selection.

[97] Why Are Parsing Actions for Understanding Message Hierarchies Not Random?

Daichi Kato,Ryo Ueda,Yusuke Miyao

Main category: cs.CL

TL;DR: This study shows that when experiments are adjusted to include more complex hierarchical inputs and a surprisal-related factor, agents using random parsing strategies no longer achieve high communication accuracy, suggesting why humans don't use random parsing in language comprehension.

Details

Motivation: The motivation for this study was to understand why human parsing strategies do not follow a random pattern, even though previous studies showed that agents using random parsing strategies could achieve high communication accuracy. The authors aimed to investigate whether adjustments to the experimental setup would change this outcome. Method: The researchers made two modifications to the experimental setup: they used more complex inputs with hierarchical structures and incorporated a surprisal-related term into the objective function. They then evaluated the communication accuracy of agents employing random parsing strategies under these new conditions. Result: The results showed that when agents were exposed to more complex hierarchical inputs and a surprisal-related term was included in the objective function, their use of random parsing strategies no longer led to high communication accuracy. Conclusion: The study concludes that with more complex hierarchical inputs and the inclusion of a surprisal-related term in the objective function, agents using random parsing strategies no longer maintain high communication accuracy, aligning more closely with human parsing behavior. Abstract: If humans understood language by randomly selecting parsing actions, it might have been necessary to construct a robust symbolic system capable of being interpreted under any hierarchical structure. However, human parsing strategies do not seem to follow such a random pattern. Why is that the case? In fact, a previous study on emergent communication using models with hierarchical biases have reported that agents adopting random parsing strategies$\unicode{x2013}$ones that deviate significantly from human language comprehension$\unicode{x2013}$can achieve high communication accuracy. In this study, we investigate this issue by making two simple and natural modifications to the experimental setup: (I) we use more complex inputs that have hierarchical structures, such that random parsing makes semantic interpretation more difficult, and (II) we incorporate a surprisal-related term, which is known to influence the order of words and characters in natural language, into the objective function. With these changes, we evaluate whether agents employing random parsing strategies still maintain high communication accuracy.

[98] QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization

Danush Khanna,Aditya Kumar Guru,Srivarshinee Sridhar,Zidan Ahmed,Rubhav Bahirwani,Meetu Malhotra,Vinija Jain,Aman Chadha,Amitava Das,Kripabandhu Ghosh

Main category: cs.CL

TL;DR: QuickSilver is a new modular framework that reduces computational overhead during inference in large language models by introducing semantic adaptivity at the token level.

Details

Motivation: Inference in large language models accounts for over 90% of deployment costs in terms of latency and energy consumption. Runtime optimization remains a critical challenge, especially under autoregressive decoding. Method: QuickSilver introduces four mechanisms: Dynamic Token Halting, KV Cache Skipping, Contextual Token Fusion, and other token-level optimizations to reduce computational overhead during inference. Result: When applied to GPT-2 and Llama-2 on datasets like WikiText-103 and C4, QuickSilver achieves up to 39.6% FLOP reduction with minimal perplexity degradation (<=0.2). Conclusion: QuickSilver provides a novel framework for runtime optimization of large language models, achieving significant FLOP reduction without compromising performance. Abstract: Inference accounts for the majority of latency and energy consumption in large language model (LLM) deployments, often exceeding 90% of total cost. While training-time efficiency has seen extensive progress, runtime optimization remains a key bottleneck, particularly under autoregressive decoding. Existing approaches -- such as pruning, quantization, early exits, and speculative decoding -- often require retraining, architectural changes, or disrupt decoding compatibility. We introduce QuickSilver, a modular, token-level framework that enables semantic adaptivity at inference time without altering model weights or structure. QuickSilver integrates four synergistic mechanisms: (i) Dynamic Token Halting, which halts computation for tokens with converged representations; (ii) KV Cache Skipping, which selectively suppresses memory writes to reduce attention overhead; and (iii) Contextual Token Fusion, which collapses redundant tokens into shared paths to shrink sequence length. Unlike speculative decoding or MoE routing, QuickSilver operates entirely on frozen, dense models and requires no auxiliary networks. Applied to GPT-2 and Llama-2 across WikiText-103 and C4, QuickSilver achieves up to 39.6% FLOP reduction with negligible perplexity degradation (<=0.2).

[99] Refining Czech GEC: Insights from a Multi-Experiment Approach

Petr Pechman,Milan Straka,Jana Straková,Jakub Náplava

Main category: cs.CL

TL;DR: This paper introduces a state-of-the-art grammar error correction system for the Czech language using a Transformer-based neural network and synthetic error generation. It achieves better performance and efficiency than existing methods.

Details

Motivation: The motivation behind this work is to improve grammar error correction for the Czech language, which has limited resources compared to other languages. The goal is to develop an efficient and effective GEC system that can handle both general and language-specific errors. Method: The authors used a neural network translation approach based on the Transformer architecture. They employed a real-time synthetic generation pipeline to dynamically augment sentences with both language-agnostic and Czech-specific errors. They also conducted experiments on various aspects like error generation strategies, domain balancing, tokenization granularity, model size, and data scaling during fine-tuning. Result: The best-performing model achieved state-of-the-art results for Czech GEC. It showed superior performance and computational efficiency. The authors also evaluated large language models (LLMs) in different scenarios and found their model to be more effective. Conclusion: The paper concludes that their proposed GEC system for the Czech language outperforms existing approaches in terms of performance and computational efficiency. Abstract: We present a grammar error correction (GEC) system that achieves state of the art for the Czech language. Our system is based on a neural network translation approach with the Transformer architecture, and its key feature is its real-time synthetic generation pipeline, which dynamically augments sentences with artificial errors by introducing both language-agnostic and Czech-specific errors. We conduct a comprehensive series of experiments, investigating the Czech GEC corpora as bases for synthetic error introduction, several error generation strategies, domain balancing, tokenization granularity, model size, and data scaling during fine-tuning. Additionally, we evaluate the performance of large language models (LLMs) on Czech GEC in both end-user and expert fine-tuning scenarios. Our best-performing model is superior both in performance and computational efficiency. The source code and the trained model links are available on https://github.com/ufal/tsd2025-gec.

[100] HyperCLOVA X THINK Technical Report

NAVER Cloud HyperCLOVA X Team

Main category: cs.CL

TL;DR: HyperCLOVA X THINK is a highly efficient, reasoning-focused language model optimized for Korean and English tasks with strong performance and reduced computational cost.

Details

Motivation: To develop a reasoning-focused large language model tailored for Korean and English tasks while reducing training compute requirements. Method: Pre-training on high-quality tokens, synthetic data augmentation, compute-memory-balanced Peri-LN Transformer with μP scaling, three-stage curriculum, post-training with reinforcement learning. Result: Competitive performance on Korea-focused benchmarks, bilingual consistency, vision-augmented variant matches GPT-4.1 on KCSAT STEM, lower training compute usage. Conclusion: HyperCLOVA X THINK is positioned as a robust foundation for Korean AI innovation and a valuable resource for global research. Abstract: We introduce HyperCLOVA X THINK, the first reasoning-focused large language model in the HyperCLOVA X family, pre-trained on roughly $6$ trillion high-quality Korean, and English tokens, augmented with targeted synthetic Korean data. It was implemented as a compute-memory-balanced Peri-LN Transformer scaled with $\mu$P, pre-trained through a three-stage curriculum that expands the context window to $128$K tokens, and post-trained via supervised fine-tuning with Reinforcement Learning from Verifiable Rewards supports both detailed rationale and concise-answer modes. It delivers competitive performance against similarly sized models on Korea-focused benchmarks such as KMMLU, CSAT, KoBALT-700, HAERAE-1.0, and KoBigBench, while preserving robust bilingual consistency and translation quality. In addition, a vision-augmented variant matches or exceeds GPT-4.1 on the KCSAT STEM benchmark, all of which are achieved with substantially lower training compute than existing models of similar sizes. We also present a pruning and distillation technique that will soon be applied to HyperCLOVA X THINK for an open-source and business-friendly foundation model. Altogether, these capabilities position HyperCLOVA X THINK as a robust foundation for Korean AI innovation and a valuable resource for the global research community.

[101] Sequential Diagnosis with Language Models

Harsha Nori,Mayank Daswani,Christopher Kelly,Scott Lundberg,Marco Tulio Ribeiro,Marc Wilson,Xiaoxuan Liu,Viknesh Sounderajah,Jonathan Carlson,Matthew P Lungren,Bay Gross,Peter Hames,Mustafa Suleyman,Dominic King,Eric Horvitz

Main category: cs.CL

TL;DR: This paper introduces the Sequential Diagnosis Benchmark and MAI-DxO, an AI system that improves diagnostic accuracy and cost-efficiency by mimicking the iterative decision-making of physicians.

Details

Motivation: To better reflect the complexity of real-world evidence-based medicine, as most evaluations of AI models currently rely on static vignettes that do not emulate the iterative process of clinical diagnosis. Method: The study introduces the Sequential Diagnosis Benchmark and presents the MAI Diagnostic Orchestrator (MAI-DxO), which simulates a panel of physicians and strategically selects tests. Performance was assessed using diagnostic accuracy and cost of visits and tests. Result: When paired with OpenAI's o3 model, MAI-DxO achieves 80% diagnostic accuracy—four times higher than generalist physicians—and reduces diagnostic costs by 20% compared to physicians and 70% compared to off-the-shelf o3. When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy. Conclusion: AI systems, particularly when guided to think iteratively and act judiciously, can significantly enhance diagnostic precision and cost-effectiveness in clinical care. Abstract: Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes and multiple-choice questions that fail to reflect the complexity and nuance of evidence-based medicine in real-world settings. In clinical practice, physicians iteratively formulate and revise diagnostic hypotheses, adapting each subsequent question and test to what they've just learned, and weigh the evolving evidence before committing to a final diagnosis. To emulate this iterative process, we introduce the Sequential Diagnosis Benchmark, which transforms 304 diagnostically challenging New England Journal of Medicine clinicopathological conference (NEJM-CPC) cases into stepwise diagnostic encounters. A physician or AI begins with a short case abstract and must iteratively request additional details from a gatekeeper model that reveals findings only when explicitly queried. Performance is assessed not just by diagnostic accuracy but also by the cost of physician visits and tests performed. We also present the MAI Diagnostic Orchestrator (MAI-DxO), a model-agnostic orchestrator that simulates a panel of physicians, proposes likely differential diagnoses and strategically selects high-value, cost-effective tests. When paired with OpenAI's o3 model, MAI-DxO achieves 80% diagnostic accuracy--four times higher than the 20% average of generalist physicians. MAI-DxO also reduces diagnostic costs by 20% compared to physicians, and 70% compared to off-the-shelf o3. When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy. These performance gains with MAI-DxO generalize across models from the OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama families. We highlight how AI systems, when guided to think iteratively and act judiciously, can advance diagnostic precision and cost-effectiveness in clinical care.

cs.CV [Back]

[102] Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs

Yifan Shen,Yuanzhe Liu,Jingyuan Zhu,Xu Cao,Xiaofeng Zhang,Yixiao He,Wenming Ye,James Matthew Rehg,Ismini Lourentzou

Main category: cs.CV

TL;DR: The paper introduces SpatialReasoner-R1, a vision-language reasoning model that improves spatial reasoning through M3CTS and fDPO methods.

Details

Motivation: Current Vision-Language Models (VLMs) struggle with fine-grained spatial reasoning, particularly when multi-step logic and precise spatial alignment are required. Method: Multi-Model Monte Carlo Tree Search (M3CTS) method and fine-grained Direct Preference Optimization (fDPO). Result: Experimental results demonstrate that fDPO achieves an average improvement of 4.1% over standard DPO across spatial quality tasks, and a 9.0% gain in spatial quantity tasks. Conclusion: SpatialReasoner-R1, trained with fDPO, sets a new SoTA on SPATIALRGPT-Bench, outperforming the strongest baseline by 9.8% in average accuracy while maintaining competitive performance on general vision-language tasks. Abstract: Current Vision-Language Models (VLMs) struggle with fine-grained spatial reasoning, particularly when multi-step logic and precise spatial alignment are required. In this work, we introduce SpatialReasoner-R1, a vision-language reasoning model designed to address these limitations. To construct high-quality supervision for spatial reasoning, we design a Multi-Model Monte Carlo Tree Search (M3CTS) method that generates diverse, logically consistent Long Chain-of-Thought (LongCoT) reasoning trajectories. In addition, we propose fine-grained Direct Preference Optimization (fDPO), which introduces segment-specific preference granularity for descriptive grounding and logical reasoning, guided by a spatial reward mechanism that evaluates candidate responses based on visual consistency, spatial grounding, and logical coherence. Experimental results demonstrate that fDPO achieves an average improvement of 4.1% over standard DPO across spatial quality tasks, and a 9.0% gain in spatial quantity tasks. SpatialReasoner-R1, trained with fDPO, sets a new SoTA on SPATIALRGPT-Bench, outperforming the strongest baseline by 9.8% in average accuracy, while maintaining competitive performance on general vision-language tasks.

[103] TanDiT: Tangent-Plane Diffusion Transformer for High-Quality 360° Panorama Generation

Hakan Çapuk,Andrew Bond,Muhammed Burak Kızıl,Emir Göçen,Erkut Erdem,Aykut Erdem

Main category: cs.CV

TL;DR: 本文介绍TanDiT，一种新的全景图像生成方法，它利用统一的扩散模型和后处理步骤，实现高质量、多样化的全景图像生成。

Details

Motivation: 由于几何失真和需要无缝循环一致性，现有的图像生成模型在全景图像生成方面仍面临挑战。 Method: 使用统一的扩散模型生成覆盖整个360度视图的切平面图像网格，并提出了一种增强全局一致性的后处理步骤。 Result: TanDiT在广泛实验中表现出良好的泛化能力、对复杂文本提示的稳健解释能力，并提供了两个专用指标用于评估全景图像质量。 Conclusion: TanDiT可以有效生成高质量的全景图像，超越现有方法，并且能够与各种生成模型无缝集成。 Abstract: Recent advances in image generation have led to remarkable improvements in synthesizing perspective images. However, these models still struggle with panoramic image generation due to unique challenges, including varying levels of geometric distortion and the requirement for seamless loop-consistency. To address these issues while leveraging the strengths of the existing models, we introduce TanDiT, a method that synthesizes panoramic scenes by generating grids of tangent-plane images covering the entire 360$^\circ$ view. Unlike previous methods relying on multiple diffusion branches, TanDiT utilizes a unified diffusion model trained to produce these tangent-plane images simultaneously within a single denoising iteration. Furthermore, we propose a model-agnostic post-processing step specifically designed to enhance global coherence across the generated panoramas. To accurately assess panoramic image quality, we also present two specialized metrics, TangentIS and TangentFID, and provide a comprehensive benchmark comprising captioned panoramic datasets and standardized evaluation scripts. Extensive experiments demonstrate that our method generalizes effectively beyond its training data, robustly interprets detailed and complex text prompts, and seamlessly integrates with various generative models to yield high-quality, diverse panoramic images.

[104] FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering

Liangyu Zhong,Fabio Rosenthal,Joachim Sicking,Fabian Hüger,Thorsten Bagdonat,Hanno Gottschalk,Leo Schwinn

Main category: cs.CV

TL;DR: FOCUS proposes a training-free visual cropping approach for VQA tasks that improves accuracy and efficiency compared to existing methods.

Details

Motivation: Visual Question Answering (VQA) focusing on small image details remains challenging due to limitations of current visual cropping techniques, such as task-specific fine-tuning, inefficiency, and incompatibility with attention implementations. Method: FOCUS uses MLLM-internal representations to guide the search for relevant image regions without requiring training or fine-tuning, involving four steps: identifying target objects, computing relevance maps via KV cache, ranking regions, and performing VQA on top-ranked regions. Result: FOCUS achieves strong performance across four fine-grained VQA datasets and two types of MLLMs, outperforming three popular methods and matching ZoomEye's performance with 3 - 6.5 x less compute. Conclusion: FOCUS is an effective and efficient visual cropping method that outperforms existing approaches in both accuracy and computational efficiency. Abstract: While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains a challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region. As a result of this informed search strategy, FOCUS achieves strong performance across four fine-grained VQA datasets and two types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 3 - 6.5 x less compute.

[105] CAST: Cross-Attentive Spatio-Temporal feature fusion for Deepfake detection

Aryan Thakre,Omkar Nagwekar,Vedang Talekar,Aparna Santra Biswas

Main category: cs.CV

TL;DR: This paper introduces the CAST model for deepfake detection, which improves performance by using cross-attention to better integrate spatial and temporal features.

Details

Motivation: Existing CNN-Transformer models process spatial and temporal features independently, limiting the depth of their interaction. This work aims to enhance deepfake detection by improving the integration of these features. Method: The authors proposed a unified CAST model that uses cross-attention to integrate spatial and temporal features more effectively, allowing temporal features to dynamically focus on relevant spatial regions. Result: The CAST model achieved an AUC of 99.49% and accuracy of 97.57% in intra-dataset evaluations, and showed strong generalization with a 93.31% AUC on the unseen DeepfakeDetection dataset in cross-dataset testing. Conclusion: The CAST model demonstrates that cross-attention-based feature fusion significantly enhances the robustness and performance of deepfake video detection, achieving superior results in both intra- and cross-dataset evaluations. Abstract: Deepfakes have emerged as a significant threat to digital media authenticity, increasing the need for advanced detection techniques that can identify subtle and time-dependent manipulations. CNNs are effective at capturing spatial artifacts, and Transformers excel at modeling temporal inconsistencies. However, many existing CNN-Transformer models process spatial and temporal features independently. In particular, attention-based methods often use separate attention mechanisms for spatial and temporal features and combine them using naive approaches like averaging, addition, or concatenation, which limits the depth of spatio-temporal interaction. To address this challenge, we propose a unified CAST model that leverages cross-attention to effectively fuse spatial and temporal features in a more integrated manner. Our approach allows temporal features to dynamically attend to relevant spatial regions, enhancing the model's ability to detect fine-grained, time-evolving artifacts such as flickering eyes or warped lips. This design enables more precise localization and deeper contextual understanding, leading to improved performance across diverse and challenging scenarios. We evaluate the performance of our model using the FaceForensics++, Celeb-DF, and DeepfakeDetection datasets in both intra- and cross-dataset settings to affirm the superiority of our approach. Our model achieves strong performance with an AUC of 99.49 percent and an accuracy of 97.57 percent in intra-dataset evaluations. In cross-dataset testing, it demonstrates impressive generalization by achieving a 93.31 percent AUC on the unseen DeepfakeDetection dataset. These results highlight the effectiveness of cross-attention-based feature fusion in enhancing the robustness of deepfake video detection.

[106] Elucidating and Endowing the Diffusion Training Paradigm for General Image Restoration

Xin Lu,Xueyang Fu,Jie Xiao,Zihao Fan,Yurui Zhu,Zheng-Jun Zha

Main category: cs.CV

TL;DR: 本文提出了一个结合扩散训练范式的新型图像修复框架，优化了通用IR网络的性能和适用性。

Details

Motivation: 尽管扩散模型在图像修复任务中表现出色，但其复杂结构和迭代过程限制了实际应用。现有方法主要优化网络架构和扩散路径，忽视了将扩散训练范式整合到通用IR框架中的重要性。 Method: 系统分析了时间步依赖、网络层次、噪声水平关系和多修复任务相关性，引入了与IR任务对齐的扩散目标正则化策略，并开发了增量训练范式和任务特定适配器。 Result: 实验表明，所提方法在单任务IR中显著提升了泛化能力，在多任务统一IR中表现出优越性能。 Conclusion: 该论文提出了一种新的图像修复（IR）框架，通过扩散训练范式提升了单任务和多任务统一IR的泛化能力和性能，并且可以无缝集成到现有的IR架构中。 Abstract: While diffusion models demonstrate strong generative capabilities in image restoration (IR) tasks, their complex architectures and iterative processes limit their practical application compared to mainstream reconstruction-based general ordinary IR networks. Existing approaches primarily focus on optimizing network architecture and diffusion paths but overlook the integration of the diffusion training paradigm within general ordinary IR frameworks. To address these challenges, this paper elucidates key principles for adapting the diffusion training paradigm to general IR training through systematic analysis of time-step dependencies, network hierarchies, noise-level relationships, and multi-restoration task correlations, proposing a new IR framework supported by diffusion-based training. To enable IR networks to simultaneously restore images and model generative representations, we introduce a series of regularization strategies that align diffusion objectives with IR tasks, improving generalization in single-task scenarios. Furthermore, recognizing that diffusion-based generation exerts varying influences across different IR tasks, we develop an incremental training paradigm and task-specific adaptors, further enhancing performance in multi-task unified IR. Experiments demonstrate that our method significantly improves the generalization of IR networks in single-task IR and achieves superior performance in multi-task unified IR. Notably, the proposed framework can be seamlessly integrated into existing general IR architectures.

[107] Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning

Remco F. Leijenaar,Hamidreza Kasaei

Main category: cs.CV

TL;DR: This paper introduces AsymDSD, a novel self-supervised learning framework for 3D point clouds that outperforms existing methods by focusing on latent space prediction and incorporating key architectural innovations.

Details

Motivation: Learning semantic representations from unstructured 3D point clouds is challenging, especially without large labeled datasets. Existing masked point modeling approaches are limited by their reconstruction-based objective, which does not capture high-level semantics effectively. Method: The paper proposes AsymDSD, an Asymmetric Dual Self-Distillation framework that combines masked modeling and invariance learning through latent space prediction. Key design choices include asymmetric setup, disabling attention between masked queries, multi-mask sampling, and point cloud adaptation of multi-crop. Result: AsymDSD achieved state-of-the-art performance on the ScanObjectNN dataset (90.53%) and improved to 93.72% when pre-trained on 930k shapes, outperforming previous methods. Conclusion: AsymDSD is a new framework for learning semantically meaningful representations from unstructured 3D point clouds, which achieves state-of-the-art results on ScanObjectNN and improves significantly when pretrained on large datasets. Abstract: Learning semantically meaningful representations from unstructured 3D point clouds remains a central challenge in computer vision, especially in the absence of large-scale labeled datasets. While masked point modeling (MPM) is widely used in self-supervised 3D learning, its reconstruction-based objective can limit its ability to capture high-level semantics. We propose AsymDSD, an Asymmetric Dual Self-Distillation framework that unifies masked modeling and invariance learning through prediction in the latent space rather than the input space. AsymDSD builds on a joint embedding architecture and introduces several key design choices: an efficient asymmetric setup, disabling attention between masked queries to prevent shape leakage, multi-mask sampling, and a point cloud adaptation of multi-crop. AsymDSD achieves state-of-the-art results on ScanObjectNN (90.53%) and further improves to 93.72% when pretrained on 930k shapes, surpassing prior methods.

[108] Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesis

Chenqiu Zhao,Anup Basu

Main category: cs.CV

TL;DR: This paper explores limitations in generative models, proposing frameworks MESP and LCH, along with novel models BL-AE and ARVM, to address these issues.

Details

Motivation: To investigate the limitation of probabilistic generative models where learning global distributions leads to memorization instead of true generative behavior. Method: The paper proposes two frameworks, MESP and LCH, and uses experiments with VAEs, BL-AE, and ARVM to validate their effectiveness. Result: The proposed ARVM model achieves competitive FID scores and outperforms state-of-the-art methods on standard datasets, but highlights issues related to memorization. Conclusion: The paper concludes that learning global distributions in generative models may lead to memorization rather than generative behavior, and introduces frameworks like MESP and LCH to address this issue. Abstract: We propose two theoretical frameworks, the Mutually Exclusive Probability Space (MESP) and the Local Correlation Hypothesis (LCH), to explore a potential limitation in probabilistic generative models; namely that learning global distributions leads to memorization rather than generative behavior. MESP emerges from our rethinking of the Variational Autoencoder (VAE). We observe that latent variable distributions in VAE exhibit overlap, which leads to an optimization conflict between the reconstruction loss and KL-divergence loss. A lower bound based on the overlap coefficient is proposed. We refer to this phenomenon as Mutually Exclusive Probability Spaces. Based on MESP, a Binary Latent Autoencoder (BL-AE) is proposed to encode images into binary latent representations. These binary latents are used as the input to our Autoregressive Random Variable Model (ARVM), a modified autoregressive model outputting histograms. Our ARVM achieves competitive FID scores, outperforming state-of-the-art methods on standard datasets. However, such scores reflect memorization rather than generation. To address this issue, we propose the Local Correlation Hypothesis (LCH), which posits that generative capability arising from local correlations among latent variables. Comprehensive experiments and discussions are conducted to validate our frameworks.

[109] Equitable Federated Learning with NCA

Nick Lemke,Mirko Konstantin,Henry John Krumb,John Kalkhof,Jonathan Stieber,Anirban Mukhopadhyay

Main category: cs.CV

TL;DR: 本文提出了一种针对中低收入国家的联邦学习系统FedNCA，用于医学图像分割任务，解决了资源限制和网络安全问题。

Details

Motivation: 在中低收入国家，由于高性能计算资源有限且互联网连接不可靠，采用联邦学习（FL）面临重大障碍。因此，需要一种适合这些环境的更轻便、高效的FL系统。 Method: 引入了FedNCA，这是一种基于Med-NCA架构的新型FL系统，可在低成本边缘设备上进行训练，并通过减少通信成本来优化性能。此外，该系统支持加密以适应不安全的网络环境。 Result: FedNCA能够在广泛使用的智能手机等低功耗设备上运行，并且可以有效降低通信开销，同时具备加密能力，适用于网络环境不稳定的情况。 Conclusion: FedNCA是一种有前景的解决方案，可以在资源有限的地区实现包容、高效、轻量级和可加密的医学成像解决方案，促进公平的医疗保健发展。 Abstract: Federated Learning (FL) is enabling collaborative model training across institutions without sharing sensitive patient data. This approach is particularly valuable in low- and middle-income countries (LMICs), where access to trained medical professionals is limited. However, FL adoption in LMICs faces significant barriers, including limited high-performance computing resources and unreliable internet connectivity. To address these challenges, we introduce FedNCA, a novel FL system tailored for medical image segmentation tasks. FedNCA leverages the lightweight Med-NCA architecture, enabling training on low-cost edge devices, such as widely available smartphones, while minimizing communication costs. Additionally, our encryption-ready FedNCA proves to be suitable for compromised network communication. By overcoming infrastructural and security challenges, FedNCA paves the way for inclusive, efficient, lightweight, and encryption-ready medical imaging solutions, fostering equitable healthcare advancements in resource-constrained regions.

[110] ImplicitQA: Going beyond frames towards Implicit Video Reasoning

Sirnam Swetha,Rohit Gupta,Parth Parag Kulkarni,David G Shatwell,Jeffrey A Chan Santiago,Nyle Siddiqui,Joseph Fioresi,Mubarak Shah

Main category: cs.CV

TL;DR: This paper introduces ImplicitQA, a new benchmark for evaluating implicit reasoning in VideoQA systems, revealing limitations in current models and encouraging future research.

Details

Motivation: Current VideoQA benchmarks focus on explicit visual content, while human understanding excels at implicit reasoning across discontinuous frames. This gap inspired the need for a new benchmark targeting implicit reasoning in cinematic and narrative-driven videos. Method: The authors introduced ImplicitQA, a benchmark designed to test implicit reasoning in VideoQA systems. It includes 1K annotated QA pairs derived from 320+ creative video clips, categorized into reasoning dimensions like causality, spatial understanding, and social interactions. Result: Evaluations on leading VideoQA models showed performance degradation, indicating their reliance on surface-level cues and highlighting the difficulty of implicit reasoning tasks. Conclusion: The paper concludes that current VideoQA systems struggle with implicit reasoning, and the introduction of the ImplicitQA benchmark presents a valuable challenge to advance research in this area. Abstract: Video QA has made significant strides by leveraging multimodal learning to align visual and textual modalities. However, current benchmarks overwhelmingly focus on questions answerable through explicit visual content - actions, objects & events directly observable within individual frames or short clips. In contrast, creative and cinematic videos - such as movies, TV shows, and narrative-driven content - employ storytelling techniques that deliberately omit certain depictions, requiring viewers to infer motives, causality, and relationships across discontinuous frames. Humans naturally excel at such implicit reasoning, seamlessly integrating information across time and context to construct coherent narratives. Current VideoQA systems and benchmarks fail to capture this essential dimension of human-like understanding. To bridge this gap, we present ImplicitQA, a novel benchmark specifically designed to test models on implicit reasoning. It comprises 1K meticulously annotated QA pairs derived from 320+ high-quality creative video clips, systematically categorized into key reasoning dimensions: lateral and vertical spatial reasoning, depth and proximity, viewpoint and visibility, motion and trajectory, causal and motivational reasoning, social interactions, physical context, and inferred counting. These annotations are deliberately challenging, crafted by authors ensuring high-quality. Our extensive evaluations on leading VideoQA models reveals performance degradation, underscoring their reliance on surface-level visual cues and highlighting the difficulty of implicit reasoning. Performance variations across models further illustrate the complexity and diversity of the challenges presented by ImplicitQA. By releasing both the dataset and our data collection framework, we aim to stimulate further research and development in the community. https://huggingface.co/datasets/ucf-crcv/ImplicitQA.

[111] Early Glaucoma Detection using Deep Learning with Multiple Datasets of Fundus Images

Rishiraj Paul Chowdhury,Nirmit Shekar Karkera

Main category: cs.CV

TL;DR: This study proposes a deep learning pipeline using EfficientNet-B0 for early glaucoma detection from retinal fundus images, achieving strong performance with minimal preprocessing and demonstrating potential for clinical application.

Details

Motivation: Glaucoma is a leading cause of irreversible blindness, and early detection can significantly improve treatment outcomes. Traditional diagnostic methods are invasive and require specialized equipment, highlighting the need for a non-invasive, scalable solution. Method: A deep learning pipeline was developed using the EfficientNet-B0 architecture, sequentially trained and fine-tuned across three datasets (ACRIMA, ORIGA, RIM-ONE) to enhance generalization. Minimal preprocessing was applied to assess its impact on model performance. Result: Minimal preprocessing yielded higher AUC-ROC scores compared to more complex enhancements, and the model showed strong discriminative performance on unseen datasets. Conclusion: The proposed deep learning pipeline using the EfficientNet-B0 architecture offers a reproducible and scalable approach for early glaucoma detection, demonstrating strong performance on unseen datasets and potential clinical utility. Abstract: Glaucoma is a leading cause of irreversible blindness, but early detection can significantly improve treatment outcomes. Traditional diagnostic methods are often invasive and require specialized equipment. In this work, we present a deep learning pipeline using the EfficientNet-B0 architecture for glaucoma detection from retinal fundus images. Unlike prior studies that rely on single datasets, we sequentially train and fine-tune our model across ACRIMA, ORIGA, and RIM-ONE datasets to enhance generalization. Our experiments show that minimal preprocessing yields higher AUC-ROC compared to more complex enhancements, and our model demonstrates strong discriminative performance on unseen datasets. The proposed pipeline offers a reproducible and scalable approach to early glaucoma detection, supporting its potential clinical utility.

[112] Comparing Learning Paradigms for Egocentric Video Summarization

Daniel Wen

Main category: cs.CV

TL;DR: 这项研究比较了监督学习、无监督学习和提示微调三种计算机视觉方法在处理第一人称视频数据的效果，发现尽管当前最先进的模型表现不佳，但提示微调的通用GPT-4o模型显示出更好的性能，表明未来需要进一步优化以应对第一人称视角的独特挑战。

Details

Motivation: 本研究旨在调查各种计算机视觉范式理解与解释以自我为中心视频数据的能力，并推动针对第一人称视频领域的计算机视觉技术应用的发展。 Method: 研究通过评估监督学习（Shotluck Holmes）、无监督学习（TAC-SUM）和提示微调（GPT-4o）三种计算机视觉范式在视频摘要中的有效性来进行。 Result: 结果显示，目前最先进的模型在第一人称视频上的表现不如第三人称视频。提示微调的通用GPT-4o模型表现出色，超越了这些专用模型。 Conclusion: 该研究得出结论，当前最先进的模型在处理第一人称视频方面效果不佳，需要进一步改进。提示微调的通用GPT-4o模型优于这些专用模型，强调了现有方法在适应第一人称视角独特挑战方面的局限性。 Abstract: In this study, we investigate various computer vision paradigms - supervised learning, unsupervised learning, and prompt fine-tuning - by assessing their ability to understand and interpret egocentric video data. Specifically, we examine Shotluck Holmes (state-of-the-art supervised learning), TAC-SUM (state-of-the-art unsupervised learning), and GPT-4o (a prompt fine-tuned pre-trained model), evaluating their effectiveness in video summarization. Our results demonstrate that current state-of-the-art models perform less effectively on first-person videos compared to third-person videos, highlighting the need for further advancements in the egocentric video domain. Notably, a prompt fine-tuned general-purpose GPT-4o model outperforms these specialized models, emphasizing the limitations of existing approaches in adapting to the unique challenges of first-person perspectives. Although our evaluation is conducted on a small subset of egocentric videos from the Ego-Exo4D dataset due to resource constraints, the primary objective of this research is to provide a comprehensive proof-of-concept analysis aimed at advancing the application of computer vision techniques to first-person videos. By exploring novel methodologies and evaluating their potential, we aim to contribute to the ongoing development of models capable of effectively processing and interpreting egocentric perspectives.

[113] CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery

Felix Holm,Gözde Ünver,Ghazal Ghazaei,Nassir Navab

Main category: cs.CV

TL;DR: This paper introduces the CAT-SG dataset and CatSGG model to comprehensively represent cataract surgery workflows, enhancing AI applications in surgical training and decision support.

Details

Motivation: To overcome the limitations of existing datasets that address isolated aspects of surgical analysis and lack comprehensive representations of semantic relationships in cataract surgery workflows. Method: Introduction of the Cataract Surgery Scene Graph (CAT-SG) dataset and a novel scene graph generation model, CatSGG. Result: The CAT-SG dataset provides structured annotations capturing semantic relations, while the CatSGG model outperforms current methods in generating structured surgical representations. Conclusion: CAT-SG dataset and CatSGG model can enhance AI-driven surgical training, decision support, and workflow analysis. Abstract: Understanding the intricate workflows of cataract surgery requires modeling complex interactions between surgical tools, anatomical structures, and procedural techniques. Existing datasets primarily address isolated aspects of surgical analysis, such as tool detection or phase segmentation, but lack comprehensive representations that capture the semantic relationships between entities over time. This paper introduces the Cataract Surgery Scene Graph (CAT-SG) dataset, the first to provide structured annotations of tool-tissue interactions, procedural variations, and temporal dependencies. By incorporating detailed semantic relations, CAT-SG offers a holistic view of surgical workflows, enabling more accurate recognition of surgical phases and techniques. Additionally, we present a novel scene graph generation model, CatSGG, which outperforms current methods in generating structured surgical representations. The CAT-SG dataset is designed to enhance AI-driven surgical training, real-time decision support, and workflow analysis, paving the way for more intelligent, context-aware systems in clinical practice.

[114] Few-Shot Segmentation of Historical Maps via Linear Probing of Vision Foundation Models

Rafael Sterzinger,Marco Peer,Robert Sablatnig

Main category: cs.CV

TL;DR: 本文提出了一种基于视觉基础模型和参数高效微调的少样本历史地图分割方法，该方法在多个数据集上表现优异，并且具有较强的泛化能力和较低的数据需求。

Details

Motivation: 历史地图是重要的信息来源，但其多样的视觉表示和有限的标注数据对自动化处理提出了重大挑战。因此，需要一种能够应对这些挑战的少样本分割方法。 Method: 文章使用了大规模视觉基础模型丰富的语义嵌入，并结合参数高效的微调策略来实现少样本的历史地图分割。这种方法在仅需少量标注数据的情况下实现了精确的分割。 Result: 在Siegfried基准数据集中，该方法在10个样本情况下分别实现了+5%和+13%的mIoU相对提升，在更困难的5个样本场景下则提升了约+20%。此外，在ICDAR 2021竞赛数据集上也展现了强大性能，达到了67.3%的平均PQ值（用于建筑区块分割）。同时，该方法只需要689k个可训练参数，仅为总模型规模的0.21%。 Conclusion: 该研究证明了其提出的少样本分割方法在历史地图分析中的有效性，不仅减少了对手动标注的依赖，还显著提高了自动化处理能力，为相关领域的发展提供了新思路。 Abstract: As rich sources of history, maps provide crucial insights into historical changes, yet their diverse visual representations and limited annotated data pose significant challenges for automated processing. We propose a simple yet effective approach for few-shot segmentation of historical maps, leveraging the rich semantic embeddings of large vision foundation models combined with parameter-efficient fine-tuning. Our method outperforms the state-of-the-art on the Siegfried benchmark dataset in vineyard and railway segmentation, achieving +5% and +13% relative improvements in mIoU in 10-shot scenarios and around +20% in the more challenging 5-shot setting. Additionally, it demonstrates strong performance on the ICDAR 2021 competition dataset, attaining a mean PQ of 67.3% for building block segmentation, despite not being optimized for this shape-sensitive metric, underscoring its generalizability. Notably, our approach maintains high performance even in extremely low-data regimes (10- & 5-shot), while requiring only 689k trainable parameters - just 0.21% of the total model size. Our approach enables precise segmentation of diverse historical maps while drastically reducing the need for manual annotations, advancing automated processing and analysis in the field. Our implementation is publicly available at: https://github.com/RafaelSterzinger/few-shot-map-segmentation.

[115] TaleForge: Interactive Multimodal System for Personalized Story Creation

Minh-Loi Nguyen,Quang-Khai Le,Tam V. Nguyen,Minh-Triet Tran,Trung-Nghia Le

Main category: cs.CV

TL;DR: TaleForge是一个能够将用户面部图像融入叙述和插图的个性化故事生成系统，提高了用户的故事创作体验。

Details

Motivation: 现有的方法往往将用户视为被动消费者，提供通用情节且个性化有限，这削弱了互动性和沉浸感，尤其是在个体风格或外貌至关重要的情况下。 Method: TaleForge采用了三个相互关联的模块：故事生成、个性化图像生成和背景生成，其中用户面部图像被嵌入到叙述和插图中。 Result: 用户研究显示，当用户作为主角出现时，他们的参与感和拥有感显著提高，并称赞系统的实时预览和直观控制功能。 Conclusion: TaleForge通过结合大型语言模型和文本到图像扩散，实现了个性化的多媒体故事叙述，增强了用户的参与感和沉浸感。 Abstract: Storytelling is a deeply personal and creative process, yet existing methods often treat users as passive consumers, offering generic plots with limited personalization. This undermines engagement and immersion, especially where individual style or appearance is crucial. We introduce TaleForge, a personalized story-generation system that integrates large language models (LLMs) and text-to-image diffusion to embed users' facial images within both narratives and illustrations. TaleForge features three interconnected modules: Story Generation, where LLMs create narratives and character descriptions from user prompts; Personalized Image Generation, merging users' faces and outfit choices into character illustrations; and Background Generation, creating scene backdrops that incorporate personalized characters. A user study demonstrated heightened engagement and ownership when individuals appeared as protagonists. Participants praised the system's real-time previews and intuitive controls, though they requested finer narrative editing tools. TaleForge advances multimodal storytelling by aligning personalized text and imagery to create immersive, user-centric experiences.

[116] PrefPaint: Enhancing Image Inpainting through Expert Human Feedback

Duy-Bao Bui,Hoang-Khang Nguyen,Trung-Nghia Le

Main category: cs.CV

TL;DR: 本文介绍了一种名为PrefPaint的新方法，它将人类反馈融入到图像修复模型的训练中，特别适用于需要高准确度的医疗图像处理场景。此外，还开发了一个便于使用和反馈的Web界面。

Details

Motivation: 为了在诸如医疗息肉成像等专业领域中确保图像修复的准确性和可靠性，以防止在医学诊断和治疗中产生重大错误。 Method: 开发了一个名为PrefPaint的方法，并创建了一个交互式界面用于提供反馈和管理微调过程。 Result: 用户研究显示，PrefPaint优于现有方法，在减少视觉不一致性和改善图像渲染方面表现出色，尤其是在医疗环境中，该模型生成了更加真实的息肉图像。 Conclusion: PrefPaint有效地将人类反馈整合到Stable Diffusion Inpainting的训练过程中，避免了计算成本高昂的奖励模型的需求，并通过基于网络的界面简化了训练、微调和推理过程。 Abstract: Inpainting, the process of filling missing or corrupted image parts, has broad applications, including medical imaging. However, in specialized fields like medical polyps imaging, where accuracy and reliability are critical, inpainting models can generate inaccurate images, leading to significant errors in medical diagnosis and treatment. To ensure reliability, medical images should be annotated by experts like oncologists for effective model training. We propose PrefPaint, an approach that incorporates human feedback into the training process of Stable Diffusion Inpainting, bypassing the need for computationally expensive reward models. In addition, we develop a web-based interface streamlines training, fine-tuning, and inference. This interactive interface provides a smooth and intuitive user experience, making it easier to offer feedback and manage the fine-tuning process. User study on various domains shows that PrefPaint outperforms existing methods, reducing visual inconsistencies and improving image rendering, particularly in medical contexts, where our model generates more realistic polyps images.

[117] ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts

Xiaoqi Wang,Clint Sebastian,Wenbin He,Liu Ren

Main category: cs.CV

TL;DR: 本文提出了一种名为ProSAM的新方法，通过改进提示编码器以避免不稳定区域，从而提升视觉参考分割的稳定性和性能。

Details

Motivation: 现有的SAM-based方法由于次优的提示编码器，在物体边界生成提示，导致不稳定和鲁棒性降低。 Method: 通过学习一个变分提示编码器来预测多变量提示分布，避免在不稳定区域生成提示。 Result: 在Pascal-5$^i$和COCO-20$^i$数据集上，ProSAM始终超越最先进的方法。 Conclusion: ProSAM提供了一种更稳定和强大的视觉参考分割解决方案，克服了现有方法在提示编码上的不足。 Abstract: The recent advancements in large foundation models have driven the success of open-set image segmentation, a task focused on segmenting objects beyond predefined categories. Among various prompt types (such as points, boxes, texts, and visual references), visual reference segmentation stands out for its unique flexibility and strong zero-shot capabilities. Recently, several SAM-based methods have made notable progress in this task by automatically generating prompts to guide SAM. However, these methods often generate prompts at object boundaries due to suboptimal prompt encoder, which results in instability and reduced robustness. In this work, we introduce ProSAM, a simple but effective method to address the stability challenges we identified in existing SAM-based visual reference segmentation approaches. By learning a variational prompt encoder to predict multivariate prompt distributions, ProSAM avoids generating prompts that lie in unstable regions, overcoming the instability caused by less robust prompts. Our approach consistently surpasses state-of-the-art methods on the Pascal-5$^i$ and COCO-20$^i$ datasets, providing a more robust solution for visual reference segmentation.

[118] GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles

Mengyi Shan,Brian Curless,Ira Kemelmacher-Shlizerman,Steve Seitz

Main category: cs.CV

TL;DR: This paper introduces a hierarchical multi-agent framework for generating escape room puzzle images, improving both visual and functional quality compared to base models.

Details

Motivation: Base text-to-image models struggle with spatial relationships and affordance reasoning when generating complex escape room puzzles. This work aims to overcome these limitations. Method: The authors propose a hierarchical multi-agent framework that decomposes the task into functional design, symbolic scene graph reasoning, layout synthesis, and local image editing. These agents collaborate iteratively to refine the output. Result: Experiments demonstrate that the collaborative agent approach improves output quality by enhancing solvability, avoiding shortcuts, and clarifying affordances while maintaining visual appeal. Conclusion: The proposed hierarchical multi-agent framework enhances the ability of text-to-image models to generate escape room puzzles that are visually coherent and functionally solvable. Abstract: We challenge text-to-image models with generating escape room puzzle images that are visually appealing, logically solid, and intellectually stimulating. While base image models struggle with spatial relationships and affordance reasoning, we propose a hierarchical multi-agent framework that decomposes this task into structured stages: functional design, symbolic scene graph reasoning, layout synthesis, and local image editing. Specialized agents collaborate through iterative feedback to ensure the scene is visually coherent and functionally solvable. Experiments show that agent collaboration improves output quality in terms of solvability, shortcut avoidance, and affordance clarity, while maintaining visual quality.

[119] 3D-Telepathy: Reconstructing 3D Objects from EEG Signals

Yuxiang Ge,Jionghao Cheng,Ruiquan Ge,Zhaojie Fang,Gangyong Jia,Xiang Wan,Nannan Li,Ahmed Elazab,Changmiao Wang

Main category: cs.CV

TL;DR: This paper explores translating EEG brain activity into 3D objects using a novel encoder and training strategy, enabling better spatial information processing for BCIs.

Details

Motivation: Reconstructing 3D visual stimuli from EEG data can enhance Brain-Computer Interface applications by preserving spatial information typically lost in 2D image reconstructions. Method: The researchers employed an EEG encoder with a dual self-attention mechanism, using cross-attention, contrastive learning, and self-supervised learning for training. They also used stable diffusion and Variational Score Distillation to generate 3D objects from neural radiation fields. Result: The proposed method successfully generated 3D objects with similar content and structure from EEG data, addressing challenges like noise and lack of datasets. Conclusion: The study concludes that translating EEG data into 3D objects is feasible through the proposed innovative EEG encoder architecture and training techniques. Abstract: Reconstructing 3D visual stimuli from Electroencephalography (EEG) data holds significant potential for applications in Brain-Computer Interfaces (BCIs) and aiding individuals with communication disorders. Traditionally, efforts have focused on converting brain activity into 2D images, neglecting the translation of EEG data into 3D objects. This limitation is noteworthy, as the human brain inherently processes three-dimensional spatial information regardless of whether observing 2D images or the real world. The neural activities captured by EEG contain rich spatial information that is inevitably lost when reconstructing only 2D images, thus limiting its practical applications in BCI. The transition from EEG data to 3D object reconstruction faces considerable obstacles. These include the presence of extensive noise within EEG signals and a scarcity of datasets that include both EEG and 3D information, which complicates the extraction process of 3D visual data. Addressing this challenging task, we propose an innovative EEG encoder architecture that integrates a dual self-attention mechanism. We use a hybrid training strategy to train the EEG Encoder, which includes cross-attention, contrastive learning, and self-supervised learning techniques. Additionally, by employing stable diffusion as a prior distribution and utilizing Variational Score Distillation to train a neural radiation field, we successfully generate 3D objects with similar content and structure from EEG data.

[120] End-to-End RGB-IR Joint Image Compression With Channel-wise Cross-modality Entropy Model

Haofeng Wang,Fangtao Zhou,Qi Zhang,Zeyuan Chen,Enci Zhang,Zhao Wang,Xiaofeng Huang,Siwei Ma

Main category: cs.CV

TL;DR: This paper proposes an efficient RGB-IR image pair compression framework using a novel entropy model, achieving significant improvements over existing methods.

Details

Motivation: RGB-IR image pairs are widely used in applications like intelligent surveillance, but they require higher data storage and transmission costs. Efficient compression of RGB-IR data is therefore essential. Method: The paper introduces a Channel-wise Cross-modality Entropy Model (CCEM), including Low-frequency Context Extraction Block (LCEB) and Low-frequency Context Fusion Block (LCFB), to utilize cross-modality prior information for more accurate entropy parameter prediction. Result: Experimental results show that the proposed method achieves better performance than existing RGB-IR and single-modality compression techniques, with a 23.1% bit rate saving on the LLVIP dataset compared to a state-of-the-art RGB-IR codec from CVPR 2022. Conclusion: The paper proposes a joint compression framework for RGB-IR image pairs, which outperforms existing methods on LLVIP and KAIST datasets. Abstract: RGB-IR(RGB-Infrared) image pairs are frequently applied simultaneously in various applications like intelligent surveillance. However, as the number of modalities increases, the required data storage and transmission costs also double. Therefore, efficient RGB-IR data compression is essential. This work proposes a joint compression framework for RGB-IR image pair. Specifically, to fully utilize cross-modality prior information for accurate context probability modeling within and between modalities, we propose a Channel-wise Cross-modality Entropy Model (CCEM). Among CCEM, a Low-frequency Context Extraction Block (LCEB) and a Low-frequency Context Fusion Block (LCFB) are designed for extracting and aggregating the global low-frequency information from both modalities, which assist the model in predicting entropy parameters more accurately. Experimental results demonstrate that our approach outperforms existing RGB-IR image pair and single-modality compression methods on LLVIP and KAIST datasets. For instance, the proposed framework achieves a 23.1% bit rate saving on LLVIP dataset compared to the state-of-the-art RGB-IR image codec presented at CVPR 2022.

[121] Periodic-MAE: Periodic Video Masked Autoencoder for rPPG Estimation

Jiho Choi,Sang Jun Lee

Main category: cs.CV

TL;DR: 提出了一种利用无标签面部视频学习周期信号表示的方法，用于远程光电容积描记（rPPG）估计。

Details

Motivation: 捕捉视频中的准周期信号对于远程光电容积描记（rPPG）估计至关重要。为了更好地提取生理信号，需要考虑信号的周期性及生理带宽限制。 Method: 该方法采用视频掩码自编码器，通过自监督学习获取面部区域的高维时空表示。通过视频采样中的帧掩码来捕捉准周期信号，并结合生理带限约束，利用生理信号在其频带内的稀疏性提供脉搏线索。 Result: 在PURE、UBFC-rPPG、MMPD和V4V数据集上进行了广泛的实验评估，结果表明该方法在具有挑战性的跨数据集评估中表现出显著的性能提升。 Conclusion: 所提出的框架能够有效学习周期信号的表示，并在rPPG任务中展现出优异的性能，尤其是在跨数据集评估中。 Abstract: In this paper, we propose a method that learns a general representation of periodic signals from unlabeled facial videos by capturing subtle changes in skin tone over time. The proposed framework employs the video masked autoencoder to learn a high-dimensional spatio-temporal representation of the facial region through self-supervised learning. Capturing quasi-periodic signals in the video is crucial for remote photoplethysmography (rPPG) estimation. To account for signal periodicity, we apply frame masking in terms of video sampling, which allows the model to capture resampled quasi-periodic signals during the pre-training stage. Moreover, the framework incorporates physiological bandlimit constraints, leveraging the property that physiological signals are sparse within their frequency bandwidth to provide pulse cues to the model. The pre-trained encoder is then transferred to the rPPG task, where it is used to extract physiological signals from facial videos. We evaluate the proposed method through extensive experiments on the PURE, UBFC-rPPG, MMPD, and V4V datasets. Our results demonstrate significant performance improvements, particularly in challenging cross-dataset evaluations. Our code is available at https://github.com/ziiho08/Periodic-MAE.

[122] SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space

Ekaterina Redekop,Mara Pleasure,Zichen Wang,Kimberly Flores,Anthony Sisk,William Speier,Corey W. Arnold

Main category: cs.CV

TL;DR: SPADE是一种整合组织病理学与空间转录组数据的基础模型，通过混合数据专家技术和对比学习实现更优的图像表示学习。

Details

Motivation: 现有的多模态方法未能全面整合全切片图像与空间转录组数据，而这种整合对于捕捉超出标准H&E染色的关键分子异质性至关重要。 Method: SPADE采用一种混合数据专家技术，并使用两阶段特征空间聚类生成专家，通过对比学习来学习共注册WSI补丁和基因表达谱的表示。 Result: 在HEST-1k数据集上预训练后，SPADE在14个下游任务中表现出显著优于基线模型的少样本性能。 Conclusion: SPADE通过整合组织病理学和空间转录组数据，在统一框架内指导图像表示学习，证明了形态学和分子信息结合的潜力。 Abstract: The rapid growth of digital pathology and advances in self-supervised deep learning have enabled the development of foundational models for various pathology tasks across diverse diseases. While multimodal approaches integrating diverse data sources have emerged, a critical gap remains in the comprehensive integration of whole-slide images (WSIs) with spatial transcriptomics (ST), which is crucial for capturing critical molecular heterogeneity beyond standard hematoxylin & eosin (H&E) staining. We introduce SPADE, a foundation model that integrates histopathology with ST data to guide image representation learning within a unified framework, in effect creating an ST-informed latent space. SPADE leverages a mixture-of-data experts technique, where experts, created via two-stage feature-space clustering, use contrastive learning to learn representations of co-registered WSI patches and gene expression profiles. Pre-trained on the comprehensive HEST-1k dataset, SPADE is evaluated on 14 downstream tasks, demonstrating significantly superior few-shot performance compared to baseline models, highlighting the benefits of integrating morphological and molecular information into one latent space.

[123] LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

Boyuan Sun,Jiaxing Zhao,Xihan Wei,Qibin Hou

Main category: cs.CV

TL;DR: 本文提出了一种名为LLaVA-Scissor的新方法，用于视频多模态大语言模型的token压缩，其通过利用语义连通组件方法在空间和时间域上实现有效的token压缩，从而在多种视频理解任务中取得了优异的表现。

Details

Motivation: 先前的方法大多尝试基于注意力得分来压缩tokens，但未能有效捕捉所有语义区域，且常常导致token冗余。因此需要一种新的方法来解决这些问题。 Method: 提出了一种两步的时空token压缩策略，该策略在空间和时间域中均使用SCC（Semantic Connected Components），通过将tokens分配到不同的语义区域以确保全面的语义覆盖，并最终用一组非重叠的语义tokens表示整个视频。 Result: 进行了广泛的评估，结果表明所提出的LLaVA-Scissor在各种视频理解基准测试中表现出色，尤其是在低token保留率的情况下优于其他token压缩方法。 Conclusion: LLaVA-Scissor是一种无需训练的视频多模态大语言模型token压缩策略，通过利用语义连通组件方法，在空间和时间域上实现有效的token压缩，从而在各种视频理解基准测试中优于其他token压缩方法，特别是在低token保留率的情况下。 Abstract: In this paper, we present LLaVA-Scissor, a training-free token compression strategy designed for video multimodal large language models. Previous methods mostly attempt to compress tokens based on attention scores, but fail to effectively capture all semantic regions and often lead to token redundancy. Differently, we propose to leverage the Semantic Connected Components (SCC) approach that assigns tokens to distinct semantic regions within the token set, ensuring comprehensive semantic coverage. The outcome is a two-step spatio-temporal token compression strategy that utilizes SCC in both spatial and temporal domains. This strategy can effectively compress tokens by representing the entire video with a set of non-overlapping semantic tokens. We conduct extensive evaluations of the token compression capabilities of LLaVA-Scissor across diverse video understanding benchmarks, including video question answering, long video understanding, and comprehensive multi-choices benchmarks. Experimental results show that the proposed LLaVA-Scissor outperforms other token compression methods, achieving superior performance in various video understanding benchmarks, particularly at low token retention ratios. Project page: https://github.com/HumanMLLM/LLaVA-Scissor.

[124] Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling

Sungjune Park,Yeongyun Kim,Se Yeon Kim,Yong Man Ro

Main category: cs.CV

TL;DR: This paper proposes a novel Large Vision and Language Model (LVLM) framework tailored for Remote Sensing (RS), introducing Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling to better capture multi-level semantics, resulting in improved performance across RS tasks.

Details

Motivation: LVLMs perform well in natural image domains but struggle with remote sensing due to domain differences like visual appearances, object scales, and semantics. This work aims to enable effective adaptation of LVLMs for RS tasks. Method: The framework uses a retrieval-based Semantic Augmentation Module to enrich visual features with semantic cues, followed by Semantic-aware Expert Modeling that processes different semantic levels separately for hierarchical understanding. Result: Evaluations on multiple RS tasks showed consistent improvements across various semantic levels, demonstrating the effectiveness of the proposed framework. Conclusion: The proposed LVLM framework effectively bridges the gap between general vision-language models and the unique demands of remote sensing tasks by incorporating semantic-augmented alignment and expert modeling. Abstract: Large Vision and Language Models (LVLMs) have shown strong performance across various vision-language tasks in natural image domains. However, their application to remote sensing (RS) remains underexplored due to significant domain differences in visual appearances, object scales, and semantics. These discrepancies hider the effective understanding of RS scenes, which contain rich, multi-level semantic information spanning from coarse-to-fine levels. Hence, it limits the direct adaptation of existing LVLMs to RS imagery. To address this gap, we propose a novel LVLM framework tailored for RS understanding, incorporating two core components: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling. First, to align multi-level visual features, we introduce the retrieval-based Semantic Augmentation Module which enriches the visual features with relevant semantics across fine-to-coarse levels (e.g., object- and scene-level information). It is designed to retrieve relevant semantic cues from a RS semantic knowledge database, followed by aggregation of semantic cues with user query and multi-level visual features, resulting in semantically enriched representation across multiple levels. Second, for Semantic-aware Expert Modeling, we design semantic experts, where each expert is responsible for processing semantic representation at different levels separately. This enables hierarchical semantic understanding from coarse to fine levels. Evaluations across multiple RS tasks-including scene classification and VQA, etc.-demonstrate that the proposed framework achieves consistent improvements across multiple semantic levels. This highlights its capability and effectiveness in bridging the gap between general LVLMs and unique demands of RS-specific vision-language understanding.

[125] Dual-Perspective United Transformer for Object Segmentation in Optical Remote Sensing Images

Yanguang Sun,Jiexi Yan,Jianjun Qian,Chunyan Xu,Jian Yang,Lei Luo

Main category: cs.CV

TL;DR: This paper proposes DPU-Former, a novel Transformer-based model that effectively combines global and local features for improved object segmentation in optical remote sensing images.

Details

Motivation: To address the limitations of existing models that rely solely on convolutional or Transformer features, the authors aim to exploit both advantages while overcoming challenges like feature heterogeneity, high complexity, and large model parameters. Method: The authors proposed a Dual-Perspective United Transformer (DPU-Former) with global-local mixed attention, Fourier-space merging strategy, and a gated linear feed-forward network for efficient feature integration and segmentation. Result: The proposed DPU-Former model achieves superior performance over existing methods for object segmentation in optical remote sensing images. Conclusion: The DPU-Former model outperforms state-of-the-art methods on multiple datasets for object segmentation in optical remote sensing images. Abstract: Automatically segmenting objects from optical remote sensing images (ORSIs) is an important task. Most existing models are primarily based on either convolutional or Transformer features, each offering distinct advantages. Exploiting both advantages is valuable research, but it presents several challenges, including the heterogeneity between the two types of features, high complexity, and large parameters of the model. However, these issues are often overlooked in existing the ORSIs methods, causing sub-optimal segmentation. For that, we propose a novel Dual-Perspective United Transformer (DPU-Former) with a unique structure designed to simultaneously integrate long-range dependencies and spatial details. In particular, we design the global-local mixed attention, which captures diverse information through two perspectives and introduces a Fourier-space merging strategy to obviate deviations for efficient fusion. Furthermore, we present a gated linear feed-forward network to increase the expressive ability. Additionally, we construct a DPU-Former decoder to aggregate and strength features at different layers. Consequently, the DPU-Former model outperforms the state-of-the-art methods on multiple datasets. Code: https://github.com/CSYSI/DPU-Former.

[126] Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning

Tzu-Chun Chien,Chieh-Kai Lin,Shiang-Feng Tsai,Ruei-Chi Lai,Hung-Jen Chen,Min Sun

Main category: cs.CV

TL;DR: This paper identifies misaligned position IDs as a cause of degraded visual grounding performance due to token pruning in multimodal large language models and proposes GAP, a pruning method that recovers performance without extra cost.

Details

Motivation: Token pruning methods are developed to reduce computational costs in multimodal large language models, but they significantly degrade the model's visual grounding ability, as observed in tasks like Referring Expression Comprehension. Method: An analysis of the impact of token pruning on visual grounding ability was conducted, identifying misaligned position IDs as a key issue. A solution called Grounding-Aware Token Pruning (GAP) was proposed to adjust position IDs and recover performance. Result: Applying GAP recovered LLaVA's RefCOCO validation set accuracy from 15.34% back to 51.42%, achieving 90% of the original unpruned model's performance. The method showed consistent improvements across multiple models and pruning strategies. Conclusion: The proposed Grounding-Aware Token Pruning (GAP) method effectively addresses the degradation in grounding ability caused by token pruning in multimodal large language models, recovering performance without additional overhead. Abstract: Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual grounding, establishing themselves as a general interface for various vision-language applications. This progress has driven the development of token pruning methods to mitigate the high computational costs associated with processing numerous visual tokens. However, we observe that pruning significantly weakens the model's grounding ability, leading to incorrect predictions and drastic performance degradation. In Referring Expression Comprehension (REC), for instance, pruning causes the accuracy of LLaVA on the RefCOCO validation set to drop from 56.14% to 15.34%. Our analysis identifies misaligned position IDs after pruning as the primary cause of this degradation, as both the order and value of these IDs are crucial for maintaining performance in grounding tasks. To address this issue, we propose Grounding-Aware Token Pruning (GAP), a simple yet effective adjustment to position IDs that recovers REC accuracy back to 51.42%, which is 90% of the original performance in the without pruning setting, all while requiring no additional training, memory, or computational overhead. Applied to models such as Shikra, MiniGPTv2, and the LLaVA series, our method consistently improves performance across various token pruning strategies.

[127] GRASP-PsONet: Gradient-based Removal of Spurious Patterns for PsOriasis Severity Classification

Basudha Pal,Sharif Amit Kamran,Brendon Lutnick,Molly Lucas,Chaitanya Parmar,Asha Patel Shah,David Apfel,Steven Fakharzadeh,Lloyd Miller,Gabriela Cula,Kristopher Standish

Main category: cs.CV

TL;DR: 本文提出一种使用梯度解释方法自动标记有问题的训练图像以提高银屑病严重程度评分模型泛化能力的框架。

Details

Motivation: 传统银屑病严重程度评分受评估者间差异和临床评估负担限制，远程成像技术虽具备扩展性，但存在光照、背景和设备质量等影响模型性能的问题。 Method: 通过基于梯度的可解释方法追踪错误分类验证图像的梯度，检测导致模型误差的训练样本，并设计一个基于ConvNeXT的弱监督模型对手机拍摄的银屑病图像进行分类。 Result: 移除8.2%的标记图像后，模型在保留测试集上的AUC-ROC提升了5%（从85%到90%），并且仅需审查前30%的样本即可识别超过90%的评估者间分歧病例。 Conclusion: 该方法有效提高了自动化评分的鲁棒性，减少了手动审核的需求。 Abstract: Psoriasis (PsO) severity scoring is important for clinical trials but is hindered by inter-rater variability and the burden of in person clinical evaluation. Remote imaging using patient captured mobile photos offers scalability but introduces challenges, such as variation in lighting, background, and device quality that are often imperceptible to humans but can impact model performance. These factors, along with inconsistencies in dermatologist annotations, reduce the reliability of automated severity scoring. We propose a framework to automatically flag problematic training images that introduce spurious correlations which degrade model generalization, using a gradient based interpretability approach. By tracing the gradients of misclassified validation images, we detect training samples where model errors align with inconsistently rated examples or are affected by subtle, nonclinical artifacts. We apply this method to a ConvNeXT based weakly supervised model designed to classify PsO severity from phone images. Removing 8.2% of flagged images improves model AUC-ROC by 5% (85% to 90%) on a held out test set. Commonly, multiple annotators and an adjudication process ensure annotation accuracy, which is expensive and time consuming. Our method detects training images with annotation inconsistencies, potentially removing the need for manual review. When applied to a subset of training data rated by two dermatologists, the method identifies over 90% of cases with inter-rater disagreement by reviewing only the top 30% of samples. This improves automated scoring for remote assessments, ensuring robustness despite data collection variability.

[128] Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles

Chuheng Wei,Ziye Qin,Ziyan Zhang,Guoyuan Wu,Matthew J. Barth

Main category: cs.CV

TL;DR: This paper provides a systematic review of deep learning-based multi-sensor fusion strategies for autonomous driving, emphasizing their role in improving environmental understanding and system robustness.

Details

Motivation: The motivation behind this study is to enhance perception for autonomous driving by overcoming individual sensor limitations through multi-sensor fusion strategies. Method: The paper systematically categorizes multi-sensor fusion strategies into data-level, feature-level, and decision-level approaches and reviews deep learning-based methods for each strategy. Result: The result includes a comprehensive review of key multi-modal datasets, applicability in real-world challenges, and insights into future directions for multi-sensor fusion in autonomous driving. Conclusion: The paper concludes that multi-sensor fusion is vital for autonomous driving, offering improved adaptability and robustness when integrating data from various sources. It highlights the importance of exploring emerging trends like Vision-Language Models (VLMs) and Large Language Models (LLMs) in this context. Abstract: Multi-sensor fusion plays a critical role in enhancing perception for autonomous driving, overcoming individual sensor limitations, and enabling comprehensive environmental understanding. This paper first formalizes multi-sensor fusion strategies into data-level, feature-level, and decision-level categories and then provides a systematic review of deep learning-based methods corresponding to each strategy. We present key multi-modal datasets and discuss their applicability in addressing real-world challenges, particularly in adverse weather conditions and complex urban environments. Additionally, we explore emerging trends, including the integration of Vision-Language Models (VLMs), Large Language Models (LLMs), and the role of sensor fusion in end-to-end autonomous driving, highlighting its potential to enhance system adaptability and robustness. Our work offers valuable insights into current methods and future directions for multi-sensor fusion in autonomous driving.

[129] DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025

Umihiro Kamoto,Tatsuya Ishibashi,Noriyuki Kugo

Main category: cs.CV

TL;DR: 报告介绍了DIVE方法，在CVRR-ES基准测试中取得了最高准确率，展示了迭代推理框架的有效性。

Details

Motivation: 挑战评估生成关于多样化真实世界视频片段问题的准确自然语言答案的能力，使用CVRR-ES基准测试套件进行评估。 Method: 采用DIVE（深度搜索迭代视频探索）方法，通过语义分解输入问题并通过逐步推理和渐进式推断来解决问题。 Result: DIVE方法在测试集上实现了81.44％的准确率，所有参与者中排名第一。 Conclusion: DIVE方法在CVRR-ES基准测试中实现了81.44％的准确率，证明了迭代推理框架在实现强大的视频问答方面的有效性。 Abstract: In this report, we present the winning solution that achieved the 1st place in the Complex Video Reasoning & Robustness Evaluation Challenge 2025. This challenge evaluates the ability to generate accurate natural language answers to questions about diverse, real-world video clips. It uses the Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES) benchmark, which consists of 214 unique videos and 2,400 question-answer pairs spanning 11 categories. Our method, DIVE (Deep-search Iterative Video Exploration), adopts an iterative reasoning approach, in which each input question is semantically decomposed and solved through stepwise reasoning and progressive inference. This enables our system to provide highly accurate and contextually appropriate answers to even the most complex queries. Applied to the CVRR-ES benchmark, our approach achieves 81.44% accuracy on the test set, securing the top position among all participants. This report details our methodology and provides a comprehensive analysis of the experimental results, demonstrating the effectiveness of our iterative reasoning framework in achieving robust video question answering. The code is available at https://github.com/PanasonicConnect/DIVE

[130] SODA: Out-of-Distribution Detection in Domain-Shifted Point Clouds via Neighborhood Propagation

Adam Goodge,Xun Xu,Bryan Hooi,Wee Siong Ng,Jingyi Liao,Yongyi Su,Xulei Yang

Main category: cs.CV

TL;DR: 本文探讨了如何利用3D视觉-语言模型（3D VLMs）进行点云对象的分布外（OOD）检测，提出了一种名为SODA的新方法，该方法通过基于邻域的评分传播方案改进OOD点云的检测效果，并在多个数据集和问题设置上实现了最先进的性能。

Details

Motivation: 随着点云数据在各种应用中的普及，在确保模型安全性和可靠性方面，对分布外（OOD）点云对象的检测能力变得至关重要。然而，这一问题在现有研究中尚未得到充分探索。受图像领域成功的启发，作者希望利用3D VLMs来解决这个问题，但面临预训练数据规模和多样性不足的挑战。 Method: 本文提出了一种名为SODA的新方法，通过基于邻域的评分传播方案改进OOD点云的检测效果。SODA是一种推理方法，不需要额外的模型训练。 Result: 实验表明，合成到真实的领域转移显著降低了点云与其相关文本嵌入在3D VLM潜在空间中的对齐性，从而影响下游任务的表现。而SODA方法在多个数据集和问题设置上均取得了最先进的性能。 Conclusion: 本文通过提出SODA方法，成功解决了基于3D VLMs的点云OOD检测中的合成到真实领域转移问题，为未来的研究提供了新的思路和方法。 Abstract: As point cloud data increases in prevalence in a variety of applications, the ability to detect out-of-distribution (OOD) point cloud objects becomes critical for ensuring model safety and reliability. However, this problem remains under-explored in existing research. Inspired by success in the image domain, we propose to exploit advances in 3D vision-language models (3D VLMs) for OOD detection in point cloud objects. However, a major challenge is that point cloud datasets used to pre-train 3D VLMs are drastically smaller in size and object diversity than their image-based counterparts. Critically, they often contain exclusively computer-designed synthetic objects. This leads to a substantial domain shift when the model is transferred to practical tasks involving real objects scanned from the physical environment. In this paper, our empirical experiments show that synthetic-to-real domain shift significantly degrades the alignment of point cloud with their associated text embeddings in the 3D VLM latent space, hindering downstream performance. To address this, we propose a novel methodology called SODA which improves the detection of OOD point clouds through a neighborhood-based score propagation scheme. SODA is inference-based, requires no additional model training, and achieves state-of-the-art performance over existing approaches across datasets and problem settings.

[131] Exploring Task-Solving Paradigm for Generalized Cross-Domain Face Anti-Spoofing via Reinforcement Fine-Tuning

Fangling Jiang,Qi Li,Weining Wang,Gang Wang,Bing Liu,Zhenan Sun

Main category: cs.CV

TL;DR: 本研究提出了一种新的人脸反欺骗方法，通过强化学习机制提升模型的跨域泛化能力和决策可解释性，而无需大量人工标注数据。

Details

Motivation: 现有的方法倾向于记忆训练集中的数据模式，导致在不同场景下的未知攻击类型中泛化能力差且解释性有限。 Method: 设计了可验证类别一致性奖励和推理一致性奖励，并采用基于GRPO的优化策略引导模型进行多视角推理学习。 Result: 实验结果表明该方法在跨域泛化性能上达到了最先进的水平，能够很好地泛化到未见过的目标领域中的多种未知攻击类型，并提供可解释的推理过程。 Conclusion: 本文提出了一种基于强化微调的人脸反欺骗方法，通过激励模型从多个角度探索推理策略以最大化预期奖励，从而实现了跨域人脸反欺骗任务的高效解决，并展示了良好的泛化能力和可解释性。 Abstract: Recently the emergence of novel presentation attacks has drawn increasing attention to face anti-spoofing. However, existing methods tend to memorize data patterns from the training set, resulting in poor generalization to unknown attack types across different scenarios and limited interpretability. To address these challenges, this paper presents a reinforcement fine-tuning-based face anti-spoofing method that stimulates the capabilities of multimodal large language models to think and learn how to solve the anti-spoofing task itself, rather than relying on the memorization of authenticity patterns. We design verifiable class consistent reward and reasoning consistent reward, and employ a GRPO-based optimization strategy to guide the model in exploring reasoning policies from multiple perspectives to maximize expected rewards. As a result, through iterative trial-and-error learning while retaining only high-reward trajectories, the model distills highly generalizable decision-making rules from the extensive solution space to effectively address cross-domain face anti-spoofing tasks. Extensive experimental results demonstrate that our method achieves state-of-the-art cross-domain generalization performance. It generalizes well to diverse unknown attack types in unseen target domains while providing interpretable reasoning for its authenticity decisions without requiring labor-intensive textual annotations for training.

[132] Visual Content Detection in Educational Videos with Transfer Learning and Dataset Enrichment

Dipayan Biswas,Shishir Shah,Jaspal Subhlok

Main category: cs.CV

TL;DR: 本研究解决了讲座视频中视觉元素自动检测困难的问题，通过使用迁移学习和优化YOLO模型，实现了有效的目标检测，并公开了相关数据集与代码。

Details

Motivation: 视觉元素在讲座视频中对理解、记忆和数据展示至关重要，但其改进视频内容访问的潜力尚未充分利用，因为缺乏准确的自动检测方法。 Method: 评估了一系列最先进的目标检测模型，并选择YOLO进行优化；利用多基准数据集进行训练并采用半监督自动标注策略。 Result: 成功开发出一种用于讲座视频中目标检测的通用解决方案，并发布了带注释的讲座视频帧基准数据集及源代码以促进未来研究。 Conclusion: 本文提出了一种基于迁移学习的方法来检测讲座视频中的视觉元素，YOLO模型被证明是最有前景的模型，并通过多数据集训练和半监督自动标注策略进行了优化。 Abstract: Video is transforming education with online courses and recorded lectures supplementing and replacing classroom teaching. Recent research has focused on enhancing information retrieval for video lectures with advanced navigation, searchability, summarization, as well as question answering chatbots. Visual elements like tables, charts, and illustrations are central to comprehension, retention, and data presentation in lecture videos, yet their full potential for improving access to video content remains underutilized. A major factor is that accurate automatic detection of visual elements in a lecture video is challenging; reasons include i) most visual elements, such as charts, graphs, tables, and illustrations, are artificially created and lack any standard structure, and ii) coherent visual objects may lack clear boundaries and may be composed of connected text and visual components. Despite advancements in deep learning based object detection, current models do not yield satisfactory performance due to the unique nature of visual content in lectures and scarcity of annotated datasets. This paper reports on a transfer learning approach for detecting visual elements in lecture video frames. A suite of state of the art object detection models were evaluated for their performance on lecture video datasets. YOLO emerged as the most promising model for this task. Subsequently YOLO was optimized for lecture video object detection with training on multiple benchmark datasets and deploying a semi-supervised auto labeling strategy. Results evaluate the success of this approach, also in developing a general solution to the problem of object detection in lecture videos. Paper contributions include a publicly released benchmark of annotated lecture video frames, along with the source code to facilitate future research.

[133] RAUM-Net: Regional Attention and Uncertainty-aware Mamba Network

Mingquan Liu

Main category: cs.CV

TL;DR: 本文提出了一种用于细粒度视觉分类(FGVC)的半监督方法，结合了基于Mamba的特征建模、区域注意力和贝叶斯不确定性，在标签数据有限的情况下表现出色。

Details

Motivation: 由于类间差异细微和特征表示脆弱，FGVC在计算机视觉中仍然是一个具有挑战性的任务。现有的方法在细粒度场景中表现不佳，尤其是在标记数据稀缺的情况下。 Method: 提出了一种结合基于Mamba的特征建模、区域注意力和贝叶斯不确定性的半监督方法。该方法在学习过程中增强了从局部到全局的特征建模，并关注关键区域。贝叶斯推理用于选择高质量的伪标签以提高稳定性。 Result: 实验结果表明，该方法在FGVC基准测试中表现出色，特别是在存在遮挡的情况下，且在标记数据有限时仍能保持鲁棒性。 Conclusion: 本文提出的方法有效解决了FGVC任务中的挑战，尤其适用于标签数据有限的情况。 Abstract: Fine Grained Visual Categorization (FGVC) remains a challenging task in computer vision due to subtle inter class differences and fragile feature representations. Existing methods struggle in fine grained scenarios, especially when labeled data is scarce. We propose a semi supervised method combining Mamba based feature modeling, region attention, and Bayesian uncertainty. Our approach enhances local to global feature modeling while focusing on key areas during learning. Bayesian inference selects high quality pseudo labels for stability. Experiments show strong performance on FGVC benchmarks with occlusions, demonstrating robustness when labeled data is limited. Code is available at https://github.com/wxqnl/RAUM Net.

[134] CERBERUS: Crack Evaluation & Recognition Benchmark for Engineering Reliability & Urban Stability

Justin Reinman,Sunwoong Choi

Main category: cs.CV

TL;DR: CERBERUS是一个新的合成基准，旨在提升AI模型对基础设施缺陷的检测能力，并提供了公开可用的资源以促进相关研究。

Details

Motivation: 为了提供一种灵活、可重复的方法来测试缺陷检测系统，并支持未来在自动化基础设施检查中的研究。 Method: 开发了一个名为CERBERUS的合成基准，包括一个裂缝图像生成器和在Unity中构建的真实感3D检查场景。 Result: 通过使用不同组合的合成和真实裂缝数据测试YOLO对象检测模型，结果表明结合合成和真实数据可以提高在现实世界图像上的性能。 Conclusion: CERBERUS是一个用于帮助训练和评估AI模型在基础设施中检测裂缝和其他缺陷的合成基准。 Abstract: CERBERUS is a synthetic benchmark designed to help train and evaluate AI models for detecting cracks and other defects in infrastructure. It includes a crack image generator and realistic 3D inspection scenarios built in Unity. The benchmark features two types of setups: a simple Fly-By wall inspection and a more complex Underpass scene with lighting and geometry challenges. We tested a popular object detection model (YOLO) using different combinations of synthetic and real crack data. Results show that combining synthetic and real data improves performance on real-world images. CERBERUS provides a flexible, repeatable way to test defect detection systems and supports future research in automated infrastructure inspection. CERBERUS is publicly available at https://github.com/justinreinman/Cerberus-Defect-Generator.

[135] Generating Attribute-Aware Human Motions from Textual Prompt

Xinghan Wang,Kun Xu,Fei Li,Cao Sheng,Jiazhong Yu,Yadong Mu

Main category: cs.CV

TL;DR: 本文提出了一种新方法，结合人类属性和文本描述生成更真实的动作，并发布了一个用于评估的新数据集HumanAttr。

Details

Motivation: 现有的文本驱动人体动作生成方法忽视了人类属性（如年龄、性别、体重和身高）对动作模式的重要影响，本文旨在填补这一空白。 Method: 提出了一种受结构性因果模型启发的新框架，将每个动作概念化为属性信息和动作语义的组合，并通过文本描述与动作语义对齐进行属性控制的动作生成。 Result: 作者引入了一个名为HumanAttr的数据集，用于评估他们提出的模型，并在大量实验中验证了其有效性。 Conclusion: 该论文提出了一种基于结构性因果模型的新框架，能够解耦动作语义和人类属性，实现根据文本描述和属性输入生成真实的人类动作。 Abstract: Text-driven human motion generation has recently attracted considerable attention, allowing models to generate human motions based on textual descriptions. However, current methods neglect the influence of human attributes (such as age, gender, weight, and height) which are key factors shaping human motion patterns. This work represents a pilot exploration for bridging this gap. We conceptualize each motion as comprising both attribute information and action semantics, where textual descriptions align exclusively with action semantics. To achieve this, a new framework inspired by Structural Causal Models is proposed to decouple action semantics from human attributes, enabling text-to-semantics prediction and attribute-controlled generation. The resulting model is capable of generating realistic, attribute-aware motion aligned with the user's text and attribute inputs. For evaluation, we introduce HumanAttr, a comprehensive dataset containing attribute annotations for text-motion pairs, setting the first benchmark for attribute-aware text-to-motion generation. Extensive experiments on the new dataset validate our model's effectiveness.

[136] SepFormer: Coarse-to-fine Separator Regression Network for Table Structure Recognition

Nam Quan Nguyen,Xuan Phong Pham,Tuan-Anh Tran

Main category: cs.CV

TL;DR: SepFormer is a fast and effective method for Table Structure Recognition that uses a novel coarse-to-fine approach with transformer decoders and separator regression.

Details

Motivation: The motivation is to improve the speed and robustness of Table Structure Recognition (TSR) by integrating the split-and-merge paradigm into a single step through separator regression using advanced deep learning techniques. Method: SepFormer utilizes a DETR-style architecture with a split-and-merge paradigm in a single step through separator regression. It uses a coarse-to-fine approach with two stacked transformer decoders, incorporating an angle loss for refinement. Result: SepFormer achieves comparable performance with state-of-the-art methods on several datasets like SciTSR, PubTabNet, WTW, and iFLYTAB while operating at an average of 25.6 FPS. Conclusion: SepFormer proves to be an efficient and robust solution for Table Structure Recognition, achieving high performance on multiple benchmark datasets while maintaining a speed of 25.6 FPS. Abstract: The automated reconstruction of the logical arrangement of tables from image data, termed Table Structure Recognition (TSR), is fundamental for semantic data extraction. Recently, researchers have explored a wide range of techniques to tackle this problem, demonstrating significant progress. Each table is a set of vertical and horizontal separators. Following this realization, we present SepFormer, which integrates the split-and-merge paradigm into a single step through separator regression with a DETR-style architecture, improving speed and robustness. SepFormer is a coarse-to-fine approach that predicts table separators from single-line to line-strip separators with a stack of two transformer decoders. In the coarse-grained stage, the model learns to gradually refine single-line segments through decoder layers with additional angle loss. At the end of the fine-grained stage, the model predicts line-strip separators by refining sampled points from each single-line segment. Our SepFormer can run on average at 25.6 FPS while achieving comparable performance with state-of-the-art methods on several benchmark datasets, including SciTSR, PubTabNet, WTW, and iFLYTAB.

[137] ZeroReg3D: A Zero-shot Registration Pipeline for 3D Consecutive Histopathology Image Reconstruction

Juming Xiong,Ruining Deng,Jialin Yue,Siqi Lu,Junlin Guo,Marilyn Lionts,Tianyuan Yao,Can Cui,Junchao Zhu,Chongyu Qu,Mengmeng Yin,Haichun Yang,Yuankai Huo

Main category: cs.CV

TL;DR: 提出了一種名為ZeroReg3D的零樣本配準流程，用於從連續組織切片中進行精確的3D重建，解決了組織變形、切片偽影、染色變異和照明不一致等問題。

Details

Motivation: 最近的配準方法在2D組織分析上有所改進，但往往難以保持關鍵的3D空間關係，限制了它們在臨床和研究應用中的實用性。構建來自2D切片的精確3D模型仍然具有挑戰性，原因是組織變形、切片偽影、成像技術的可變性以及照明不一致。 Method: 結合零樣本深度學習關鍵點匹配與基於優化的仿射和非剛性配準技術。 Result: 有效解決了組織變形、切片偽影、染色變異和照明不一致等關鍵挑戰，無需重新訓練或微調。 Conclusion: 提出了一種新的零樣本配準流程ZeroReg3D，用於從連續組織切片中進行精確的3D重建。 Abstract: Histological analysis plays a crucial role in understanding tissue structure and pathology. While recent advancements in registration methods have improved 2D histological analysis, they often struggle to preserve critical 3D spatial relationships, limiting their utility in both clinical and research applications. Specifically, constructing accurate 3D models from 2D slices remains challenging due to tissue deformation, sectioning artifacts, variability in imaging techniques, and inconsistent illumination. Deep learning-based registration methods have demonstrated improved performance but suffer from limited generalizability and require large-scale training data. In contrast, non-deep-learning approaches offer better generalizability but often compromise on accuracy. In this study, we introduced ZeroReg3D, a novel zero-shot registration pipeline tailored for accurate 3D reconstruction from serial histological sections. By combining zero-shot deep learning-based keypoint matching with optimization-based affine and non-rigid registration techniques, ZeroReg3D effectively addresses critical challenges such as tissue deformation, sectioning artifacts, staining variability, and inconsistent illumination without requiring retraining or fine-tuning. The code has been made publicly available at https://github.com/hrlblab/ZeroReg3D

[138] SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding

Zhao Jin,Rong-Cheng Tu,Jingyi Liao,Wenhao Sun,Xiao Luo,Shunyu Liu,Dacheng Tao

Main category: cs.CV

TL;DR: 本文介绍了一种名为SPAZER的新方法，通过结合3D和2D联合决策，在零样本3D视觉基础任务中实现了显著的性能提升。

Details

Motivation: 为了解决传统方法在复杂现实应用中的局限性，引入了一种新的零样本3D视觉基础方法。 Method: 利用VLM驱动的代理，在渐进式推理框架中结合3D和2D联合决策。 Result: 实验结果显示，SPAZER在ScanRefer和Nr3D基准测试中分别取得了9.0%和10.9%的显著准确率提升。 Conclusion: SPAZER有效地结合了空间和语义理解，实现了强大的零样本3D视觉基础。 Abstract: 3D Visual Grounding (3DVG) aims to localize target objects within a 3D scene based on natural language queries. To alleviate the reliance on costly 3D training data, recent studies have explored zero-shot 3DVG by leveraging the extensive knowledge and powerful reasoning capabilities of pre-trained LLMs and VLMs. However, existing paradigms tend to emphasize either spatial (3D-based) or semantic (2D-based) understanding, limiting their effectiveness in complex real-world applications. In this work, we introduce SPAZER - a VLM-driven agent that combines both modalities in a progressive reasoning framework. It first holistically analyzes the scene and produces a 3D rendering from the optimal viewpoint. Based on this, anchor-guided candidate screening is conducted to perform a coarse-level localization of potential objects. Furthermore, leveraging retrieved relevant 2D camera images, 3D-2D joint decision-making is efficiently performed to determine the best-matching object. By bridging spatial and semantic reasoning neural streams, SPAZER achieves robust zero-shot grounding without training on 3D-labeled data. Extensive experiments on ScanRefer and Nr3D benchmarks demonstrate that SPAZER significantly outperforms previous state-of-the-art zero-shot methods, achieving notable gains of 9.0% and 10.9% in accuracy.

[139] Quality Assessment and Distortion-aware Saliency Prediction for AI-Generated Omnidirectional Images

Liu Yang,Huiyu Duan,Jiarui Wang,Jing Liu,Menghan Hu,Xiongkuo Min,Guangtao Zhai,Patrick Le Callet

Main category: cs.CV

TL;DR: This paper introduces BLIP2OIQA and BLIP2OISal models for evaluating and enhancing the visual quality of AI-generated omnidirectional images, achieving state-of-the-art results and releasing a new dataset called OHF2024.

Details

Motivation: AI-generated omnidirectional images (AIGODIs) have growing potential for VR and AR applications but suffer from unique quality issues. There is limited research on their quality assessment and optimization, prompting the need for this study. Method: The authors first created a comprehensive database named OHF2024 with subjective quality ratings and distortion-aware salient regions. Then, they proposed two models—BLIP2OIQA for visual experience evaluation and BLIP2OISal for saliency prediction—both based on the BLIP-2 model with shared encoders. Finally, they introduced an automatic optimization process leveraging the predicted scores and distortion regions. Result: The BLIP2OIQA and BLIP2OISal models achieved state-of-the-art performance in their respective tasks. They can effectively be used in an optimization process to enhance the visual quality of AIGODIs. Conclusion: The paper concludes that the proposed BLIP2OIQA and BLIP2OISal models achieve state-of-the-art results in evaluating human visual experience and predicting distortion-aware saliency for AI-generated omnidirectional images. The automatic optimization process further enhances image quality, and the database and codes are publicly released to support future research. Abstract: With the rapid advancement of Artificial Intelligence Generated Content (AIGC) techniques, AI generated images (AIGIs) have attracted widespread attention, among which AI generated omnidirectional images (AIGODIs) hold significant potential for Virtual Reality (VR) and Augmented Reality (AR) applications. AI generated omnidirectional images exhibit unique quality issues, however, research on the quality assessment and optimization of AI-generated omnidirectional images is still lacking. To this end, this work first studies the quality assessment and distortion-aware saliency prediction problems for AIGODIs, and further presents a corresponding optimization process. Specifically, we first establish a comprehensive database to reflect human feedback for AI-generated omnidirectionals, termed OHF2024, which includes both subjective quality ratings evaluated from three perspectives and distortion-aware salient regions. Based on the constructed OHF2024 database, we propose two models with shared encoders based on the BLIP-2 model to evaluate the human visual experience and predict distortion-aware saliency for AI-generated omnidirectional images, which are named as BLIP2OIQA and BLIP2OISal, respectively. Finally, based on the proposed models, we present an automatic optimization process that utilizes the predicted visual experience scores and distortion regions to further enhance the visual quality of an AI-generated omnidirectional image. Extensive experiments show that our BLIP2OIQA model and BLIP2OISal model achieve state-of-the-art (SOTA) results in the human visual experience evaluation task and the distortion-aware saliency prediction task for AI generated omnidirectional images, and can be effectively used in the optimization process. The database and codes will be released on https://github.com/IntMeGroup/AIGCOIQA to facilitate future research.

[140] SDRNET: Stacked Deep Residual Network for Accurate Semantic Segmentation of Fine-Resolution Remotely Sensed Images

Naftaly Wambugu,Ruisheng Wang,Bo Guo,Tianshu Yu,Sheng Xu,Mohammed Elhassan

Main category: cs.CV

TL;DR: This paper proposes a stacked deep residual network (SDRNet) for semantic segmentation of high-resolution remote sensing images, addressing challenges like class disparities and object occlusion.

Details

Motivation: Accurate semantic segmentation of fine-resolution remotely sensed (FRRS) images is challenging due to class disparities, occlusion, and object size variation. Existing deep learning models struggle with capturing sufficient features for accurate segmentation. Method: A stacked deep residual network (SDRNet) is proposed, which uses two stacked encoder-decoder networks and dilated residual blocks (DRB) to improve segmentation performance. Result: Experiments on the ISPRS Vaihingen and Potsdam datasets show that SDRNet achieves improved segmentation performance while preserving spatial details and capturing global dependencies. Conclusion: The proposed SDRNet demonstrates effective and competitive performance in semantic segmentation of FRRS images compared to current DCNNs. Abstract: Land cover maps generated from semantic segmentation of high-resolution remotely sensed images have drawn mucon in the photogrammetry and remote sensing research community. Currently, massive fine-resolution remotely sensed (FRRS) images acquired by improving sensing and imaging technologies become available. However, accurate semantic segmentation of such FRRS images is greatly affected by substantial class disparities, the invisibility of key ground objects due to occlusion, and object size variation. Despite the extraordinary potential in deep convolutional neural networks (DCNNs) in image feature learning and representation, extracting sufficient features from FRRS images for accurate semantic segmentation is still challenging. These challenges demand the deep learning models to learn robust features and generate sufficient feature descriptors. Specifically, learning multi-contextual features to guarantee adequate coverage of varied object sizes from the ground scene and harnessing global-local contexts to overcome class disparities challenge even profound networks. Deeper networks significantly lose spatial details due to gradual downsampling processes resulting in poor segmentation results and coarse boundaries. This article presents a stacked deep residual network (SDRNet) for semantic segmentation from FRRS images. The proposed framework utilizes two stacked encoder-decoder networks to harness long-range semantics yet preserve spatial information and dilated residual blocks (DRB) between each encoder and decoder network to capture sufficient global dependencies thus improving segmentation performance. Our experimental results obtained using the ISPRS Vaihingen and Potsdam datasets demonstrate that the SDRNet performs effectively and competitively against current DCNNs in semantic segmentation.

[141] Exploring Semantic Masked Autoencoder for Self-supervised Point Cloud Understanding

Yixin Zha,Chuxin Wang,Wenfei Yang,Tianzhu Zhang

Main category: cs.CV

TL;DR: This paper introduces a novel approach called Semantic Masked Autoencoder to enhance point cloud understanding by better capturing semantic relationships through improved masking and modeling techniques.

Details

Motivation: To address the limitations of random masking strategies in capturing reasonable semantic relationships in point cloud data for self-supervised models. Method: Semantic Masked Autoencoder with a component semantic modeling module, component semantic-enhanced masking strategy, and prompt-tuning strategy. Result: Extensive experiments on ScanObjectNN, ModelNet40, and ShapeNetPart datasets demonstrate the effectiveness of the proposed method in enhancing pre-trained model performance for downstream tasks. Conclusion: The proposed Semantic Masked Autoencoder effectively improves the performance of self-supervised point cloud understanding by capturing semantic relationships through its novel component semantic modeling module and enhanced masking strategy. Abstract: Point cloud understanding aims to acquire robust and general feature representations from unlabeled data. Masked point modeling-based methods have recently shown significant performance across various downstream tasks. These pre-training methods rely on random masking strategies to establish the perception of point clouds by restoring corrupted point cloud inputs, which leads to the failure of capturing reasonable semantic relationships by the self-supervised models. To address this issue, we propose Semantic Masked Autoencoder, which comprises two main components: a prototype-based component semantic modeling module and a component semantic-enhanced masking strategy. Specifically, in the component semantic modeling module, we design a component semantic guidance mechanism to direct a set of learnable prototypes in capturing the semantics of different components from objects. Leveraging these prototypes, we develop a component semantic-enhanced masking strategy that addresses the limitations of random masking in effectively covering complete component structures. Furthermore, we introduce a component semantic-enhanced prompt-tuning strategy, which further leverages these prototypes to improve the performance of pre-trained models in downstream tasks. Extensive experiments conducted on datasets such as ScanObjectNN, ModelNet40, and ShapeNetPart demonstrate the effectiveness of our proposed modules.

[142] TASeg: Text-aware RGB-T Semantic Segmentation based on Fine-tuning Vision Foundation Models

Meng Yu,Te Cui,Qitong Chu,Wenjie Song,Yi Yang,Yufeng Yue

Main category: cs.CV

TL;DR: TASeg enhances RGB-T semantic segmentation by integrating textual information through LoRA adaptation, improving accuracy with efficient parameter usage.

Details

Motivation: Current RGB-T segmentation models struggle with visual similarity between categories, and integrating SAM with thermal images and text is limited by inefficiency and modality differences. Method: TASeg uses Low-Rank Adaptation (LoRA) to adapt vision models, incorporating a Dynamic Feature Fusion Module (DFFM) and CLIP-generated text embeddings for better feature merging and semantic alignment. Result: The method achieves better accuracy in challenging scenarios across diverse datasets while maintaining low computational complexity. Conclusion: The proposed TASeg framework improves semantic segmentation by integrating text awareness and using LoRA fine-tuning, showing superior performance with fewer parameters. Abstract: Reliable semantic segmentation of open environments is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-T semantic segmentation models mainly rely on low-level visual features and lack high-level textual information, which struggle with accurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. To address these, we propose TASeg, a text-aware RGB-T segmentation framework by using Low-Rank Adaptation (LoRA) fine-tuning technology to adapt vision foundation models. Specifically, we propose a Dynamic Feature Fusion Module (DFFM) in the image encoder, which effectively merges features from multiple visual modalities while freezing SAM's original transformer blocks. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies the classification error and improves the semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters.

[143] R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning

Biao Wang,Wenwen Li

Main category: cs.CV

TL;DR: This paper explores adapting the multi-modal large language model Qwen2.5-VL for visual single object tracking via fine-tuning with group relative policy optimization, resulting in R1-Track—a flexible and high-performing tracking solution.

Details

Motivation: Traditional methods for visual single object tracking require explicit classification and regression modeling, depend on supervised training with large datasets, and lack flexibility. The rapid advancement of multi-modal large language models (MLLMs) like Qwen2.5-VL presents an opportunity to explore their application in visual tracking tasks. Method: The study uses group relative policy optimization (GRPO), a reinforcement learning method inspired by deepseek-R1, to fine-tune Qwen2.5-VL on a small-scale dataset with a rule-based reward function. Result: The resulting model, R1-Track, achieved notable performance on the GOT-10k benchmark for visual single object tracking, surpassing traditional approaches that struggle with flexibility and adaptability. Conclusion: R1-Track, a fine-tuned version of Qwen2.5-VL, demonstrates notable performance on the GOT-10k benchmark for visual single object tracking while supporting flexible initialization and retaining general capabilities. Abstract: Visual single object tracking aims to continuously localize and estimate the scale of a target in subsequent video frames, given only its initial state in the first frame. This task has traditionally been framed as a template matching problem, evolving through major phases including correlation filters, two-stream networks, and one-stream networks with significant progress achieved. However, these methods typically require explicit classification and regression modeling, depend on supervised training with large-scale datasets, and are limited to the single task of tracking, lacking flexibility. In recent years, multi-modal large language models (MLLMs) have advanced rapidly. Open-source models like Qwen2.5-VL, a flagship MLLMs with strong foundational capabilities, demonstrate excellent performance in grounding tasks. This has spurred interest in applying such models directly to visual tracking. However, experiments reveal that Qwen2.5-VL struggles with template matching between image pairs (i.e., tracking tasks). Inspired by deepseek-R1, we fine-tuned Qwen2.5-VL using the group relative policy optimization (GRPO) reinforcement learning method on a small-scale dataset with a rule-based reward function. The resulting model, R1-Track, achieved notable performance on the GOT-10k benchmark. R1-Track supports flexible initialization via bounding boxes or text descriptions while retaining most of the original model's general capabilities. And we further discuss potential improvements for R1-Track. This rough technical report summarizes our findings as of May 2025.

[144] RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation

Liudi Yang,Yang Bai,George Eskandar,Fengyi Shen,Mohammad Altillawi,Dong Chen,Soumajit Majumder,Ziyuan Liu,Gitta Kutyniok,Abhinav Valada

Main category: cs.CV

TL;DR: This paper introduces a novel non-autoregressive pipeline for generating long-horizon robotic manipulation videos, combining task decomposition, keyframe generation, interpolation, and a policy model for improved performance.

Details

Motivation: Text-to-video diffusion models struggle with long-horizon robotic tasks due to error accumulation in autoregressive generation; this work aims to overcome these limitations. Method: The method involves decomposing high-level goals into atomic tasks, generating keyframes, using a diffusion model for interpolation, maintaining consistency with an attention module, and regressing robot joint states via a lightweight policy model. Result: The approach achieves state-of-the-art results on two benchmarks for video quality and consistency and outperforms previous methods in long-horizon robotic tasks. Conclusion: The proposed pipeline successfully addresses the challenges in generating long-horizon videos for robotic manipulation tasks by avoiding autoregressive generation, leading to improved video quality and performance on long-horizon tasks. Abstract: We address the problem of generating long-horizon videos for robotic manipulation tasks. Text-to-video diffusion models have made significant progress in photorealism, language understanding, and motion generation but struggle with long-horizon robotic tasks. Recent works use video diffusion models for high-quality simulation data and predictive rollouts in robot planning. However, these works predict short sequences of the robot achieving one task and employ an autoregressive paradigm to extend to the long horizon, leading to error accumulations in the generated video and in the execution. To overcome these limitations, we propose a novel pipeline that bypasses the need for autoregressive generation. We achieve this through a threefold contribution: 1) we first decompose the high-level goals into smaller atomic tasks and generate keyframes aligned with these instructions. A second diffusion model then interpolates between each of the two generated frames, achieving the long-horizon video. 2) We propose a semantics preserving attention module to maintain consistency between the keyframes. 3) We design a lightweight policy model to regress the robot joint states from generated videos. Our approach achieves state-of-the-art results on two benchmarks in video quality and consistency while outperforming previous policy models on long-horizon tasks.

[145] Towards Universal & Efficient Model Compression via Exponential Torque Pruning

Sarthak Ketanbhai Modi,Lim Zi Pong,Shourya Kuchhal,Yoshi Cao,Yupeng Cheng,Teo Yon Shin,Lin Shang-Wei,Zhiming Li

Main category: cs.CV

TL;DR: This paper introduces Exponential Torque Pruning (ETP), an improved model compression technique that effectively prunes deep neural networks by applying an exponential force regularization scheme, resulting in higher compression rates and negligible accuracy loss compared to previous methods.

Details

Motivation: The motivation stems from the limitations of existing torque-inspired regularization techniques that suffer from dense post-trained networks and significant accuracy drops, indicating a need for more efficient and effective model compression strategies. Method: The paper proposes Exponential Torque Pruning (ETP), which applies an exponential force application scheme for regularization to efficiently prune redundant and distant neural modules while retaining essential ones. This approach contrasts with the default linear force application used in previous methods. Result: Experimental results show that ETP achieves significantly higher compression rates than previous pruning strategies across various domains, with minimal impact on accuracy. Conclusion: Exponential Torque Pruning (ETP) is an effective method for compressing deep neural networks, outperforming previous state-of-the-art pruning strategies in terms of compression rate while maintaining high accuracy. Abstract: The rapid growth in complexity and size of modern deep neural networks (DNNs) has increased challenges related to computational costs and memory usage, spurring a growing interest in efficient model compression techniques. Previous state-of-the-art approach proposes using a Torque-inspired regularization which forces the weights of neural modules around a selected pivot point. Whereas, we observe that the pruning effect of this approach is far from perfect, as the post-trained network is still dense and also suffers from high accuracy drop. In this work, we attribute such ineffectiveness to the default linear force application scheme, which imposes inappropriate force on neural module of different distances. To efficiently prune the redundant and distant modules while retaining those that are close and necessary for effective inference, in this work, we propose Exponential Torque Pruning (ETP), which adopts an exponential force application scheme for regularization. Experimental results on a broad range of domains demonstrate that, though being extremely simple, ETP manages to achieve significantly higher compression rate than the previous state-of-the-art pruning strategies with negligible accuracy drop.

[146] Advancing Facial Stylization through Semantic Preservation Constraint and Pseudo-Paired Supervision

Zhanyi Lu,Yue Zhou

Main category: cs.CV

TL;DR: 本文提出了一种新的面部风格化方法，通过引入语义保持约束和伪配对监督机制，在保持原始图像内容一致性的基础上，提升了风格迁移的质量和灵活性。

Details

Motivation: 尽管基于StyleGAN的方法取得了显著进展，但生成的结果仍存在伪影或对源图像的保真度不足。这些问题源于风格化过程中生成器的语义偏移被忽视。 Method: 提出了一种结合语义保持约束和伪配对监督的面部风格化方法，并开发了创建多级伪配对数据集的方法。此外，基于该框架，无需复杂网络架构设计或额外训练即可实现更灵活的多模态和参考引导风格化。 Result: 实现了高保真、视觉吸引力强的面部风格迁移效果，并解决了先前方法中的内容一致性问题。 Conclusion: 实验结果表明，所提出的方法在面部风格迁移方面产生了高保真且美观的效果，优于先前的方法。 Abstract: Facial stylization aims to transform facial images into appealing, high-quality stylized portraits, with the critical challenge of accurately learning the target style while maintaining content consistency with the original image. Although previous StyleGAN-based methods have made significant advancements, the generated results still suffer from artifacts or insufficient fidelity to the source image. We argue that these issues stem from neglecting semantic shift of the generator during stylization. Therefore, we propose a facial stylization method that integrates semantic preservation constraint and pseudo-paired supervision to enhance the content correspondence and improve the stylization effect. Additionally, we develop a methodology for creating multi-level pseudo-paired datasets to implement supervisory constraint. Furthermore, building upon our facial stylization framework, we achieve more flexible multimodal and reference-guided stylization without complex network architecture designs or additional training. Experimental results demonstrate that our approach produces high-fidelity, aesthetically pleasing facial style transfer that surpasses previous methods.

[147] Cross-modal Ship Re-Identification via Optical and SAR Imagery: A Novel Dataset and Method

Han Wang,Shengyang Li,Jian Yang,Yuxuan Liu,Yixuan Lv,Zhuang Zhou

Main category: cs.CV

TL;DR: This paper presents the HOSS ReID dataset and TransOSS method for improved maritime ship tracking using optical and SAR satellite imagery, enabling all-weather tracking with enhanced coverage and accuracy.

Details

Motivation: Detecting and tracking ground objects, particularly ships, using earth observation imagery is challenging due to limitations of current methods relying on geostationary or video satellites. These methods suffer from low resolution, weather dependency, short filming durations, and limited coverage. Method: A baseline cross-modal ship re-identification method called TransOSS was proposed, based on the Vision Transformer architecture. It refines patch embedding structures, introduces additional embeddings, and uses contrastive learning for pre-training on large-scale optical-SAR image pairs. Result: The HOSS ReID dataset was created to evaluate ship tracking effectiveness using low-Earth orbit optical and SAR sensor constellations. It includes images of the same ships captured over long periods under various conditions from different satellites and modalities. Conclusion: The HOSS ReID dataset and the proposed TransOSS method address challenges in ship tracking by leveraging low-Earth orbit constellations of optical and SAR sensors, enabling all-weather tracking with shorter re-imaging cycles. Abstract: Detecting and tracking ground objects using earth observation imagery remains a significant challenge in the field of remote sensing. Continuous maritime ship tracking is crucial for applications such as maritime search and rescue, law enforcement, and shipping analysis. However, most current ship tracking methods rely on geostationary satellites or video satellites. The former offer low resolution and are susceptible to weather conditions, while the latter have short filming durations and limited coverage areas, making them less suitable for the real-world requirements of ship tracking. To address these limitations, we present the Hybrid Optical and Synthetic Aperture Radar (SAR) Ship Re-Identification Dataset (HOSS ReID dataset), designed to evaluate the effectiveness of ship tracking using low-Earth orbit constellations of optical and SAR sensors. This approach ensures shorter re-imaging cycles and enables all-weather tracking. HOSS ReID dataset includes images of the same ship captured over extended periods under diverse conditions, using different satellites of different modalities at varying times and angles. Furthermore, we propose a baseline method for cross-modal ship re-identification, TransOSS, which is built on the Vision Transformer architecture. It refines the patch embedding structure to better accommodate cross-modal tasks, incorporates additional embeddings to introduce more reference information, and employs contrastive learning to pre-train on large-scale optical-SAR image pairs, ensuring the model's ability to extract modality-invariant features. Our dataset and baseline method are publicly available on https://github.com/Alioth2000/Hoss-ReID.

[148] Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation

Jialei Chen,Xu Zheng,Danda Pani Paudel,Luc Van Gool,Hiroshi Murase,Daisuke Deguchi

Main category: cs.CV

TL;DR: This paper introduces Chimera-Seg and Selective Global Distillation to improve zero-shot semantic segmentation by better aligning vision and language models, resulting in enhanced performance on benchmark datasets.

Details

Motivation: Zero-shot semantic segmentation aims to segment both seen and unseen classes using only supervision from seen classes. Existing distillation-based methods face challenges in aligning vision-based features with textual space and bridging the semantic gap between global and local features. Method: Chimera-Seg combines a segmentation backbone with a CLIP-based semantic head to integrate spatial precision and vision-language alignment. Selective Global Distillation transfers knowledge from dense features similar to the CLIP CLS token, while a Semantic Alignment Module aligns visual features with text embeddings. Result: Experiments on two benchmarks showed improvements of 0.9% and 1.2% in hIoU, demonstrating the effectiveness of the proposed approach. Conclusion: The proposed Chimera-Seg and Selective Global Distillation (SGD) effectively address the challenges in vision-language alignment for zero-shot semantic segmentation, achieving improvements in performance. Abstract: Zero-shot Semantic Segmentation (ZSS) aims to segment both seen and unseen classes using supervision from only seen classes. Beyond adaptation-based methods, distillation-based approaches transfer vision-language alignment of vision-language model, e.g., CLIP, to segmentation models. However, such knowledge transfer remains challenging due to: (1) the difficulty of aligning vision-based features with the textual space, which requires combining spatial precision with vision-language alignment; and (2) the semantic gap between CLIP's global representations and the local, fine-grained features of segmentation models. To address challenge (1), we propose Chimera-Seg, which integrates a segmentation backbone as the body and a CLIP-based semantic head as the head, like the Chimera in Greek mythology, combining spatial precision with vision-language alignment. Specifically, Chimera-Seg comprises a trainable segmentation model and a CLIP Semantic Head (CSH), which maps dense features into the CLIP-aligned space. The CSH incorporates a frozen subnetwork and fixed projection layers from the CLIP visual encoder, along with lightweight trainable components. The partial module from CLIP visual encoder, paired with the segmentation model, retains segmentation capability while easing the mapping to CLIP's semantic space. To address challenge (2), we propose Selective Global Distillation (SGD), which distills knowledge from dense features exhibiting high similarity to the CLIP CLS token, while gradually reducing the number of features used for alignment as training progresses. Besides, we also use a Semantic Alignment Module (SAM) to further align dense visual features with semantic embeddings extracted from the frozen CLIP text encoder. Experiments on two benchmarks show improvements of 0.9% and 1.2% in hIoU.

[149] Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field

Hong Nie,Fuyuan Cao,Lu Chen,Fengxin Chen,Yuefeng Zou,Jun Yu

Main category: cs.CV

TL;DR: 本文提出了一种名为FIAG的新型3D说话头合成框架，该框架能够通过使用少量训练片段实现高效的身份特定适应。

Details

Motivation: 基于重建和渲染的对话头合成方法虽然取得了高质量的结果并保持了强身份保存，但它们受限于对身份特定模型的依赖性。每个新身份都需要从零开始训练，相比于基于生成模型的方法，计算成本高且可扩展性差。 Method: FIAG结合了全局高斯场和通用运动场，前者支持在共享场中表示多个身份，后者捕捉跨不同身份的共同运动动态。 Result: 广泛的对比和消融实验证明了该方法优于现有的最先进方法，验证了所提框架的有效性和可推广性。 Conclusion: FIAG是一个新颖的3D说话头合成框架，它通过使用少量训练片段实现高效的身份特定适应，克服了基于重建和渲染的对话头合成方法对身份特定模型依赖性的限制。 Abstract: Reconstruction and rendering-based talking head synthesis methods achieve high-quality results with strong identity preservation but are limited by their dependence on identity-specific models. Each new identity requires training from scratch, incurring high computational costs and reduced scalability compared to generative model-based approaches. To overcome this limitation, we propose FIAG, a novel 3D speaking head synthesis framework that enables efficient identity-specific adaptation using only a few training footage. FIAG incorporates Global Gaussian Field, which supports the representation of multiple identities within a shared field, and Universal Motion Field, which captures the common motion dynamics across diverse identities. Benefiting from the shared facial structure information encoded in the Global Gaussian Field and the general motion priors learned in the motion field, our framework enables rapid adaptation from canonical identity representations to specific ones with minimal data. Extensive comparative and ablation experiments demonstrate that our method outperforms existing state-of-the-art approaches, validating both the effectiveness and generalizability of the proposed framework. Code is available at: \textit{https://github.com/gme-hong/FIAG}.

[150] EnLVAM: Enhanced Left Ventricle Linear Measurements Utilizing Anatomical Motion Mode

Durgesh K. Singh,Ahcene Boubekki,Qing Cao,Svein Arne Aase,Robert Jenssen,Michael Kampffmeyer

Main category: cs.CV

TL;DR: The paper proposes a novel framework for LV measurement that enforces straight-line constraints and provides improved accuracy, semi-automatic design, and better alignment flexibility and clinical relevance.

Details

Motivation: Manual placement of landmarks in LV measurements is time-consuming and error-prone, while existing deep learning methods often cause inaccurate measurements due to misalignment. Method: A landmark detector is trained on Anatomical M-Mode (AMM) images, computed in real time from B-mode videos, then transformed back to B-mode space to address misalignment and reduce measurement errors. Result: Experiments show improved accuracy over standard B-mode methods, and the framework generalizes well across network architectures. Conclusion: The proposed framework enhances LV measurement accuracy by enforcing straight-line constraints and semi-automatic design, which improves the alignment flexibility and clinical relevance. Abstract: Linear measurements of the left ventricle (LV) in the Parasternal Long Axis (PLAX) view using B-mode echocardiography are crucial for cardiac assessment. These involve placing 4-6 landmarks along a virtual scanline (SL) perpendicular to the LV axis near the mitral valve tips. Manual placement is time-consuming and error-prone, while existing deep learning methods often misalign landmarks, causing inaccurate measurements. We propose a novel framework that enhances LV measurement accuracy by enforcing straight-line constraints. A landmark detector is trained on Anatomical M-Mode (AMM) images, computed in real time from B-mode videos, then transformed back to B-mode space. This approach addresses misalignment and reduces measurement errors. Experiments show improved accuracy over standard B-mode methods, and the framework generalizes well across network architectures. Our semi-automatic design includes a human-in-the-loop step where the user only places the SL, simplifying interaction while preserving alignment flexibility and clinical relevance.

[151] MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation

Dechao Meng,Steven Xiao,Xindi Zhang,Guangyuan Wang,Peng Zhang,Qi Wang,Bang Zhang,Liefeng Bo

Main category: cs.CV

TL;DR: MirrorMe is a real-time framework for generating high-quality, audio-driven portrait animations with improved temporal consistency and control.

Details

Motivation: The paper addresses the challenges of achieving real-time, high-fidelity, and temporally coherent animations in audio-driven portrait animation. Method: MirrorMe uses the LTX video model with innovations like reference identity injection, causal audio encoder and adapter, and a progressive training strategy for enhanced control and quality. Result: MirrorMe demonstrates superior performance on the EMTD Benchmark in fidelity, lip-sync accuracy, and temporal stability. Conclusion: MirrorMe achieves state-of-the-art performance in audio-driven portrait animation with real-time, high-fidelity, and temporally coherent video generation. Abstract: Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.

[152] Single-Scanline Relative Pose Estimation for Rolling Shutter Cameras

Petr Hruby,Marc Pollefeys

Main category: cs.CV

TL;DR: This paper proposes a novel method for estimating relative pose in rolling shutter cameras using scanline intersections, eliminating the need for explicit motion models and enabling independent pose computation for each scanline.

Details

Motivation: The motivation is to simplify pose estimation in rolling shutter cameras by avoiding explicit motion modeling and enabling independent computation of each scanline's pose. Method: The method involves estimating relative pose using intersections of line projections with a single scanline per image, and includes minimal solvers for scenarios like parallel lines and known gravity direction. Result: Experiments on the Fastec dataset show the feasibility of the approach for initializing rolling shutter SfM, and the paper introduces minimal solvers for specific scenarios. Conclusion: The paper concludes that their proposed method can estimate relative poses for rolling shutter cameras without explicitly modeling camera motion, serving as a foundational component for rolling shutter SfM. Abstract: We propose a novel approach for estimating the relative pose between rolling shutter cameras using the intersections of line projections with a single scanline per image. This allows pose estimation without explicitly modeling camera motion. Alternatively, scanlines can be selected within a single image, enabling single-view relative pose estimation for scanlines of rolling shutter cameras. Our approach is designed as a foundational building block for rolling shutter structure-from-motion (SfM), where no motion model is required, and each scanline's pose can be computed independently. % We classify minimal solvers for this problem in both generic and specialized settings, including cases with parallel lines and known gravity direction, assuming known intrinsics and no lens distortion. Furthermore, we develop minimal solvers for the parallel-lines scenario, both with and without gravity priors, by leveraging connections between this problem and the estimation of 2D structure from 1D cameras. % Experiments on rolling shutter images from the Fastec dataset demonstrate the feasibility of our approach for initializing rolling shutter SfM, highlighting its potential for further development. % The code will be made publicly available.

[153] Reasoning in machine vision: learning to think fast and slow

Shaheer U. Saeed,Yipei Wang,Veeru Kasivisvanathan,Brian R. Davidson,Matthew J. Clarkson,Yipeng Hu,Daniel C. Alexander

Main category: cs.CV

TL;DR: 本研究提出一种模仿人类认知的新方法，使机器能在数据有限的情况下通过增加推理时间提升非语言推理能力，并在视觉任务中超越现有模型和人类专家。

Details

Motivation: 当前机器智能受限于训练数据，缺乏在推理时动态优化解决方案的能力，而人类可以通过推理在复杂和陌生环境中做出适应性决策。 Method: 受人类双重认知理论启发，提出一种集成快速思维（System I）与慢速思维（System II）的学习范式，利用自我博弈的强化学习方法进行迭代优化。 Result: 该方法在多个现实世界视觉任务中表现出优越性能，包括计算机视觉基准测试和医学图像癌症定位，且随着思考时间增加性能提升。 Conclusion: 该研究展示了一种新的人工智能学习范式，模仿人类认知过程，通过结合快速思维和慢速思维模块，在数据稀缺的情况下显著提高机器的非语言推理能力，并在现实视觉任务中表现出色。 Abstract: Reasoning is a hallmark of human intelligence, enabling adaptive decision-making in complex and unfamiliar scenarios. In contrast, machine intelligence remains bound to training data, lacking the ability to dynamically refine solutions at inference time. While some recent advances have explored reasoning in machines, these efforts are largely limited to verbal domains such as mathematical problem-solving, where explicit rules govern step-by-step reasoning. Other critical real-world tasks - including visual perception, spatial reasoning, and radiological diagnosis - require non-verbal reasoning, which remains an open challenge. Here we present a novel learning paradigm that enables machine reasoning in vision by allowing performance improvement with increasing thinking time (inference-time compute), even under conditions where labelled data is very limited. Inspired by dual-process theories of human cognition in psychology, our approach integrates a fast-thinking System I module for familiar tasks, with a slow-thinking System II module that iteratively refines solutions using self-play reinforcement learning. This paradigm mimics human reasoning by proposing, competing over, and refining solutions in data-scarce scenarios. We demonstrate superior performance through extended thinking time, compared not only to large-scale supervised learning but also foundation models and even human experts, in real-world vision tasks. These tasks include computer-vision benchmarks and cancer localisation on medical images across five organs, showcasing transformative potential for non-verbal machine reasoning.

[154] Towards Accurate Heart Rate Measurement from Ultra-Short Video Clips via Periodicity-Guided rPPG Estimation and Signal Reconstruction

Pei-Kai Huanga,Ya-Ting Chan,Kuan-Wen Chen,Yen-Chun Chou,Shih-Yu Yang,Chiou-Ting Hsu

Main category: cs.CV

TL;DR: This paper proposes a novel method for accurate heart rate measurement from ultra-short video clips, overcoming limitations of existing approaches by enforcing consistent periodicity and mitigating spectral leakage.

Details

Motivation: The motivation is to address the lack of focus on HR estimation from ultra-short video clips in existing remote HR measurement methods. Method: The authors propose a periodicity-guided rPPG estimation method and include a generator to reconstruct longer rPPG signals from ultra-short ones while preserving their periodic consistency. Result: Extensive experiments on four rPPG estimation benchmark datasets demonstrate the effectiveness of the proposed method in accurately measuring HR from ultra-short video clips. Conclusion: The paper concludes that their proposed method accurately measures HR from ultra-short video clips and outperforms previous rPPG estimation techniques, achieving state-of-the-art performance. Abstract: Many remote Heart Rate (HR) measurement methods focus on estimating remote photoplethysmography (rPPG) signals from video clips lasting around 10 seconds but often overlook the need for HR estimation from ultra-short video clips. In this paper, we aim to accurately measure HR from ultra-short 2-second video clips by specifically addressing two key challenges. First, to overcome the limited number of heartbeat cycles in ultra-short video clips, we propose an effective periodicity-guided rPPG estimation method that enforces consistent periodicity between rPPG signals estimated from ultra-short clips and their much longer ground truth signals. Next, to mitigate estimation inaccuracies due to spectral leakage, we propose including a generator to reconstruct longer rPPG signals from ultra-short ones while preserving their periodic consistency to enable more accurate HR measurement. Extensive experiments on four rPPG estimation benchmark datasets demonstrate that our proposed method not only accurately measures HR from ultra-short video clips but also outperform previous rPPG estimation techniques to achieve state-of-the-art performance.

[155] BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

Zipei Ma,Junzhe Jiang,Yurui Chen,Li Zhang

Main category: cs.CV

TL;DR: 本文提出了一种名为BézierGS的新方法，用于街道场景的重建，该方法不依赖于高精度的对象注释，并且在动态和静态场景组件重建以及新视图合成方面优于现有方法。

Details

Motivation: 现有的大多数方法依赖于物体姿态注释，这限制了大规模和广泛场景的重建。 Method: 提出了一种名为Bézier曲线高斯泼溅(BézierGS)的方法，该方法使用可学习的Bézier曲线表示动态物体的运动轨迹，并通过引入对动态物体渲染的额外监督和曲线间一致性约束来实现场景元素的合理和准确分离与重建。 Result: 在Waymo开放数据集和nuPlan基准上的大量实验表明，BézierGS在动态和静态场景组件重建和新视图合成方面优于现有技术。 Conclusion: BézierGS在动态和静态场景组件重建以及新视图合成方面优于现有方法。 Abstract: The realistic reconstruction of street scenes is critical for developing real-world simulators in autonomous driving. Most existing methods rely on object pose annotations, using these poses to reconstruct dynamic objects and move them during the rendering process. This dependence on high-precision object annotations limits large-scale and extensive scene reconstruction. To address this challenge, we propose B\'ezier curve Gaussian splatting (B\'ezierGS), which represents the motion trajectories of dynamic objects using learnable B\'ezier curves. This approach fully leverages the temporal information of dynamic objects and, through learnable curve modeling, automatically corrects pose errors. By introducing additional supervision on dynamic object rendering and inter-curve consistency constraints, we achieve reasonable and accurate separation and reconstruction of scene elements. Extensive experiments on the Waymo Open Dataset and the nuPlan benchmark demonstrate that B\'ezierGS outperforms state-of-the-art alternatives in both dynamic and static scene components reconstruction and novel view synthesis.

[156] Tied Prototype Model for Few-Shot Medical Image Segmentation

Hyeongji Kim,Stine Hansen,Michael Kampffmeyer

Main category: cs.CV

TL;DR: This paper introduces the Tied Prototype Model (TPM), an improved approach to prototype-based few-shot segmentation for medical images that overcomes ADNet's limitations by incorporating tied prototype locations, supporting multiple prototypes and classes, and using adaptive thresholds, resulting in better segmentation accuracy.

Details

Motivation: ADNet's limitations, such as dependence on a single prototype per class, binary classification focus, and fixed thresholds that fail to adapt to variability in patients and organs, motivate the need for an improved method like TPM. Method: The Tied Prototype Model (TPM) is proposed as a probabilistic reformulation of ADNet with tied prototype locations for foreground and background distributions. TPM extends to multiple prototypes and multi-class segmentation while leveraging class priors for adaptive thresholds. Result: TPM addresses ADNet's shortcomings by enabling multi-prototype modeling, multi-class segmentation, and adaptive thresholding, leading to improved segmentation accuracy in medical images. Conclusion: TPM provides a fresh perspective on prototype-based FSS for medical image segmentation by addressing the limitations of ADNet through tied prototype locations, multi-prototype extension, and adaptive thresholds. Abstract: Common prototype-based medical image few-shot segmentation (FSS) methods model foreground and background classes using class-specific prototypes. However, given the high variability of the background, a more promising direction is to focus solely on foreground modeling, treating the background as an anomaly -- an approach introduced by ADNet. Yet, ADNet faces three key limitations: dependence on a single prototype per class, a focus on binary classification, and fixed thresholds that fail to adapt to patient and organ variability. To address these shortcomings, we propose the Tied Prototype Model (TPM), a principled reformulation of ADNet with tied prototype locations for foreground and background distributions. Building on its probabilistic foundation, TPM naturally extends to multiple prototypes and multi-class segmentation while effectively separating non-typical background features. Notably, both extensions lead to improved segmentation accuracy. Finally, we leverage naturally occurring class priors to define an ideal target for adaptive thresholds, boosting segmentation performance. Taken together, TPM provides a fresh perspective on prototype-based FSS for medical image segmentation. The code can be found at https://github.com/hjk92g/TPM-FSS.

[157] Pedestrian Intention and Trajectory Prediction in Unstructured Traffic Using IDD-PeD

Ruthvik Bokkasam,Shankar Gangisetty,A. H. Abdul Hafez,C. V. Jawahar

Main category: cs.CV

TL;DR: This paper introduces an Indian driving pedestrian dataset designed to improve pedestrian behavior modeling in complex environments, showing that current prediction methods perform poorly on this dataset.

Details

Motivation: The motivation is to address the complexities of modeling pedestrian behavior in unstructured environments, such as illumination changes, occlusion of pedestrians, unsignalized scene types, and vehicle-pedestrian interactions, to enhance pedestrian safety and vehicle navigation. Method: The method involves creating a comprehensive dataset with high-level and detailed low-level annotations focused on pedestrians in unstructured environments. The dataset is evaluated using state-of-the-art intention and trajectory prediction methods. Result: Evaluation of state-of-the-art intention prediction methods on the dataset shows a significant performance drop of up to 15%, while trajectory prediction methods underperform with an increase of up to 1208 MSE compared to standard pedestrian datasets. Conclusion: The paper concludes that the introduced Indian driving pedestrian dataset poses new challenges for pedestrian behavior research and highlights the need for more robust prediction models. Abstract: With the rapid advancements in autonomous driving, accurately predicting pedestrian behavior has become essential for ensuring safety in complex and unpredictable traffic conditions. The growing interest in this challenge highlights the need for comprehensive datasets that capture unstructured environments, enabling the development of more robust prediction models to enhance pedestrian safety and vehicle navigation. In this paper, we introduce an Indian driving pedestrian dataset designed to address the complexities of modeling pedestrian behavior in unstructured environments, such as illumination changes, occlusion of pedestrians, unsignalized scene types and vehicle-pedestrian interactions. The dataset provides high-level and detailed low-level comprehensive annotations focused on pedestrians requiring the ego-vehicle's attention. Evaluation of the state-of-the-art intention prediction methods on our dataset shows a significant performance drop of up to $\mathbf{15\%}$, while trajectory prediction methods underperform with an increase of up to $\mathbf{1208}$ MSE, defeating standard pedestrian datasets. Additionally, we present exhaustive quantitative and qualitative analysis of intention and trajectory baselines. We believe that our dataset will open new challenges for the pedestrian behavior research community to build robust models. Project Page: https://cvit.iiit.ac.in/research/projects/cvit-projects/iddped

[158] Pipe Reconstruction from Point Cloud Data

Antje Alex,Jannis Stoppe

Main category: cs.CV

TL;DR: 本文介绍了一种利用激光扫描数据自动重建管道的流程，有助于提高工业设备数字孪生的效率和准确性。

Details

Motivation: 工业资产（如船舶和海上平台）的精确数字孪生依赖于复杂管道网络的精确重建，但传统的手动建模过程耗时且劳动强度大。 Method: 该方法首先使用基于拉普拉斯收缩的方法估计骨架曲线，然后进行曲线拉长；接着采用滚动球技术结合二维圆拟合对骨架轴线重新定中心，并通过三维平滑步骤进行优化。 Result: 实现了管道属性（包括半径、长度和方向）的确定，并能够创建复杂管道网络的详细三维模型，从而实现快速且准确的建模并降低成本。 Conclusion: 本文提出了一种从不完全激光扫描数据中进行管道重建的自动化流程，支持数字孪生体的开发。 Abstract: Accurate digital twins of industrial assets, such as ships and offshore platforms, rely on the precise reconstruction of complex pipe networks. However, manual modelling of pipes from laser scan data is a time-consuming and labor-intensive process. This paper presents a pipeline for automated pipe reconstruction from incomplete laser scan data. The approach estimates a skeleton curve using Laplacian-based contraction, followed by curve elongation. The skeleton axis is then recentred using a rolling sphere technique combined with 2D circle fitting, and refined with a 3D smoothing step. This enables the determination of pipe properties, including radius, length and orientation, and facilitates the creation of detailed 3D models of complex pipe networks. By automating pipe reconstruction, this approach supports the development of digital twins, allowing for rapid and accurate modeling while reducing costs.

[159] Low-Rank Implicit Neural Representation via Schatten-p Quasi-Norm and Jacobian Regularization

Zhengyun Cheng,Changhao Wang,Guanwen Zhang,Yi Xu,Wei Zhou,Xiangyang Ji

Main category: cs.CV

TL;DR: This paper proposes CP-INR, a novel method combining CP decomposition with neural networks, achieving superior results in multi-dimensional data recovery tasks with theoretical guarantees and reduced computational complexity.

Details

Motivation: Higher-order tensors are effective for representing multi-dimensional data, but existing methods like Tucker decomposition sacrifice interpretability for flexibility. While CP decomposition offers a more interpretable structure, obtaining sparse solutions remains challenging. This work aims to address these limitations. Method: A CP-based low-rank tensor function parameterized by neural networks (CP-INR) is introduced for implicit neural representation. A variational form of the Schatten-p quasi-norm is used to achieve sparse CP decomposition, and a regularization term based on the spectral norm of the Jacobian and Hutchinson's trace estimator is proposed for smoothness. Result: Extensive experiments on image inpainting, denoising, and point cloud upsampling demonstrate that the proposed CP-INR approach outperforms state-of-the-art methods in terms of versatility and performance. Conclusion: The proposed CP-INR method, which combines the advantages of CP decomposition and neural networks, demonstrates superior performance in multi-dimensional data recovery tasks while providing theoretical guarantees and avoiding explicit SVD or chain rule derivations. Abstract: Higher-order tensors are well-suited for representing multi-dimensional data, such as color images and videos. Low-rank tensor representation has become essential in machine learning and computer vision, but existing methods like Tucker decomposition offer flexibility at the expense of interpretability. In contrast, while the CANDECOMP/PARAFAC (CP) decomposition provides a more natural and interpretable tensor structure, obtaining sparse solutions remains challenging. Leveraging the rich properties of CP decomposition, we propose a CP-based low-rank tensor function parameterized by neural networks for implicit neural representation (CP-INR). This approach enables continuous data representation beyond structured grids, fully exploiting the non-linearity of tensor data with theoretical guarantees on excess risk bounds. To achieve a sparse CP decomposition, we introduce a variational form of the Schatten-p quasi-norm and prove its relationship to multilinear rank minimization. For smoothness, we propose a regularization term based on the spectral norm of the Jacobian and Hutchinson's trace estimator. Our proposed smoothness regularization is SVD-free and avoids explicit chain rule derivations. It can serve as an alternative to Total Variation (TV) regularization in image denoising tasks and is naturally applicable to continuous data. Extensive experiments on multi-dimensional data recovery tasks, including image inpainting, denoising, and point cloud upsampling, demonstrate the superiority and versatility of our method compared to state-of-the-art approaches.

[160] Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs

Shaojie Zhang,Jiahui Yang,Jianqin Yin,Zhenbo Luo,Jian Luan

Main category: cs.CV

TL;DR: Q-Frame is a novel approach for adaptive frame selection in Video-LLMs that improves video comprehension by efficiently selecting frames using a training-free strategy.

Details

Motivation: Existing Video-LLMs face challenges in capturing crucial spatiotemporal clues due to uniform frame sampling, necessitating an adaptive approach for better video comprehension. Method: Q-Frame uses a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, employing the Gumbel-Max trick for efficient frame selection based on the video's content and specific query. Result: Extensive experiments on benchmark datasets such as MLVU, LongVideoBench, and Video-MME demonstrate Q-Frame's effectiveness and superiority over existing methods across various video understanding tasks. Conclusion: Q-Frame provides an effective solution for adaptive frame selection and multi-resolution scaling in Video-LLMs, enhancing video comprehension by preserving critical temporal and spatial information. Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant success in visual understanding tasks. However, challenges persist in adapting these models for video comprehension due to the large volume of data and temporal complexity. Existing Video-LLMs using uniform frame sampling often struggle to capture the query-related crucial spatiotemporal clues of videos effectively. In this paper, we introduce Q-Frame, a novel approach for adaptive frame selection and multi-resolution scaling tailored to the video's content and the specific query. Q-Frame employs a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, utilizing the Gumbel-Max trick for efficient frame selection. Q-Frame allows Video-LLMs to process more frames without exceeding computational limits, thereby preserving critical temporal and spatial information. We demonstrate Q-Frame's effectiveness through extensive experiments on benchmark datasets, including MLVU, LongVideoBench, and Video-MME, illustrating its superiority over existing methods and its applicability across various video understanding tasks.

[161] Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs

Amirmohammad Izadi,Mohammad Ali Banayeeanzade,Fatemeh Askari,Ali Rahimiakbar,Mohammad Mahdi Vahedi,Hosein Hasani,Mahdieh Soleymani Baghshah

Main category: cs.CV

TL;DR: This paper introduces a method to improve visual reasoning in Vision-Language Models by augmenting visual inputs with spatial structures and using prompts that encourage spatial awareness, resulting in significant improvements in performance across various tasks.

Details

Motivation: The motivation stems from the binding problem in Vision-Language Models (VLMs), where perceptual features are not reliably associated with their correct visual referents, leading to errors in visual reasoning tasks such as counting, visual search, scene description, and spatial relationship understanding. Method: The method involves augmenting visual inputs with low-level spatial structures (e.g., horizontal lines) and pairing this with a textual prompt that encourages sequential, spatially-aware parsing. Result: The results show substantial performance improvements across core visual reasoning tasks: GPT-4o visual search accuracy improved by 25.00%, counting accuracy increased by 26.83%, edit distance error in scene description was reduced by 0.32, and performance on spatial relationship tasks improved by 9.50% on a 2D synthetic dataset. Conclusion: The paper concludes that low-level visual structuring is a powerful and underexplored direction for improving compositional visual reasoning, suggesting it could serve as a general strategy for enhancing VLM performance on spatially grounded tasks. Abstract: Despite progress in Vision-Language Models (VLMs), their capacity for visual reasoning is often limited by the \textit{binding problem}: the failure to reliably associate perceptual features with their correct visual referents. This limitation underlies persistent errors in tasks such as counting, visual search, scene description, and spatial relationship understanding. A key factor is that current VLMs process visual features largely in parallel, lacking mechanisms for spatially grounded, serial attention. This paper introduces a simple yet effective intervention: augmenting visual inputs with low-level spatial structures (e.g., horizontal lines) and pairing this with a textual prompt that encourages sequential, spatially-aware parsing. We empirically demonstrate substantial performance improvements across core visual reasoning tasks. Specifically, our method improves GPT-4o visual search accuracy by 25.00%, increases counting accuracy by 26.83%, reduces edit distance error in scene description by 0.32, and enhances performance on spatial relationship tasks by 9.50% on a a 2D synthetic dataset. Furthermore, we find that the visual modification is essential for these gains; purely textual strategies, including Chain-of-Thought prompting, are insufficient and can even degrade performance. Our method enhances binding only with a single-query inference, underscoring the importance of visual input design over purely linguistically-based approaches. These findings suggest that low-level visual structuring is a powerful and underexplored direction for improving compositional visual reasoning and could serve as a general strategy for enhancing VLM performance on spatially grounded tasks.

[162] RetFiner: A Vision-Language Refinement Scheme for Retinal Foundation Models

Ronald Fecso,José Morano,Ursula Schmidt-Erfurth,Hrvoje Bogunović

Main category: cs.CV

TL;DR: 本文提出了RetFiner方法，通过视觉-语言优化改进视网膜基础模型的表示能力，并在多个OCT分类任务中实现了性能提升。

Details

Motivation: 现有的仅基于图像数据的OCT基础模型缺乏对图像全面且稳健的语义理解，需要监督微调来更好地适应特定应用和人群，但监督微调可能不可行。 Method: 提出了一种基于自监督学习的视觉-语言优化方案RetFiner，通过多种训练目标利用文本数据中的丰富监督信号，从而改进现有基础模型的表示能力并实现高效适应特定人群。 Result: 在七个高度多样化的OCT分类任务中，RetFiner分别使RETFound、UrFound和VisionFM的线性探测性能平均提高了5.8、3.9和2.1个百分点。 Conclusion: RetFiner显著提升了视网膜基础模型的线性探测性能，特别是在多个OCT分类任务上取得了平均提升。 Abstract: The rise of imaging techniques such as optical coherence tomography (OCT) and advances in deep learning (DL) have enabled clinicians and researchers to streamline retinal disease staging. A popular DL approach is self-supervised learning (SSL), where models learn from vast amounts of unlabeled data, avoiding costly annotation. SSL has allowed the development of foundation models (FMs), large models that can be used for a variety of downstream tasks. However, existing FMs for OCT, trained solely on image data, lack a comprehensive and robust semantic understanding of images, as evidenced by their downstream performance (especially for complex tasks), and thus require supervised fine-tuning (which may be unfeasible) to better adapt to specific applications and populations. To address this, we propose RetFiner, an SSL vision-language refinement scheme that improves the representations of existing FMs and enables their efficient and direct adaptation to specific populations for improved downstream performance. Our method uses a diverse set of training objectives which take advantage of the rich supervisory signal found in textual data. We tested RetFiner on the retinal FMs RETFound, UrFound, and VisionFM, showing significant improvements in linear probing performance on seven highly diverse OCT classification tasks, with an average increase of 5.8, 3.9, and 2.1 percentage points over their baselines, respectively. Our code and model weights are publicly available at https://github.com/ronnief1/RetFiner.

[163] Attention-disentangled Uniform Orthogonal Feature Space Optimization for Few-shot Object Detection

Taijin Zhao,Heqian Qiu,Yu Dai,Lanxiao Wang,Fanman Meng,Qingbo Wu,Hongliang Li

Main category: cs.CV

TL;DR: 本文提出一种新的少样本目标检测方法，通过构建均匀正交特征空间解决目标性与分类的纠缠问题，并结合优化策略提升模型表现。

Details

Motivation: 现有FSOD方法在共享特征空间中混淆了目标性和前景分类，导致对新类别样本的表示不足，限制了性能。 Method: UOFS框架将特征空间解耦为两个正交分量：幅度表示目标性，角度表示分类；同时采用混合背景优化（HBO）策略和空间注意力解耦与关联（SADA）模块来应对训练中的挑战。 Result: 实验表明，该方法在性能上显著优于基于纠缠特征空间的现有方法。 Conclusion: 该论文提出了一种新的均匀正交特征空间（UOFS）优化框架，以解决现有少样本目标检测（FSOD）方法中的局限性，并通过实验验证了该方法的有效性。 Abstract: Few-shot object detection (FSOD) aims to detect objects with limited samples for novel classes, while relying on abundant data for base classes. Existing FSOD approaches, predominantly built on the Faster R-CNN detector, entangle objectness recognition and foreground classification within shared feature spaces. This paradigm inherently establishes class-specific objectness criteria and suffers from unrepresentative novel class samples. To resolve this limitation, we propose a Uniform Orthogonal Feature Space (UOFS) optimization framework. First, UOFS decouples the feature space into two orthogonal components, where magnitude encodes objectness and angle encodes classification. This decoupling enables transferring class-agnostic objectness knowledge from base classes to novel classes. Moreover, implementing the disentanglement requires careful attention to two challenges: (1) Base set images contain unlabeled foreground instances, causing confusion between potential novel class instances and backgrounds. (2) Angular optimization depends exclusively on base class foreground instances, inducing overfitting of angular distributions to base classes. To address these challenges, we propose a Hybrid Background Optimization (HBO) strategy: (1) Constructing a pure background base set by removing unlabeled instances in original images to provide unbiased magnitude-based objectness supervision. (2) Incorporating unlabeled foreground instances in the original base set into angular optimization to enhance distribution uniformity. Additionally, we propose a Spatial-wise Attention Disentanglement and Association (SADA) module to address task conflicts between class-agnostic and class-specific tasks. Experiments demonstrate that our method significantly outperforms existing approaches based on entangled feature spaces.

[164] Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognition

Wenhan Wu,Zhishuai Guo,Chen Chen,Hongfei Xue,Aidong Lu

Main category: cs.CV

TL;DR: 本文提出了一种基于频率-语义增强的变分自编码器（FS-VAE）用于零样本骨架动作识别，通过分解频率来学习骨架语义表示，从而提升模型对训练中未见动作类别的识别能力。

Details

Motivation: 现有的方法主要关注视觉和语义表示的对齐，但忽视了语义空间中的细粒度动作模式（例如喝水和刷牙的手部运动）。为了解决这些限制，作者提出了FS-VAE方法以更好地捕捉动作的细节并弥补骨架序列中的信息损失。 Method: FS-VAE包含三个关键部分：1) 基于频率的增强模块，利用高低频调整来丰富骨架语义学习；2) 多层次对齐的语义描述，以捕捉局部细节和全局对应关系；3) 校准的交叉对齐损失函数，缓解骨架与文本特征之间的歧义与差异。 Result: 在基准数据集上的评估表明，该方法能够有效区分视觉和语义上相似的动作簇，显著提升了零样本动作识别的效果。 Conclusion: FS-VAE通过引入频率分解和多层次语义对齐机制，在零样本骨架动作识别任务中取得了良好的效果，为未来研究提供了新的思路。 Abstract: Zero-shot skeleton-based action recognition aims to develop models capable of identifying actions beyond the categories encountered during training. Previous approaches have primarily focused on aligning visual and semantic representations but often overlooked the importance of fine-grained action patterns in the semantic space (e.g., the hand movements in drinking water and brushing teeth). To address these limitations, we propose a Frequency-Semantic Enhanced Variational Autoencoder (FS-VAE) to explore the skeleton semantic representation learning with frequency decomposition. FS-VAE consists of three key components: 1) a frequency-based enhancement module with high- and low-frequency adjustments to enrich the skeletal semantics learning and improve the robustness of zero-shot action recognition; 2) a semantic-based action description with multilevel alignment to capture both local details and global correspondence, effectively bridging the semantic gap and compensating for the inherent loss of information in skeleton sequences; 3) a calibrated cross-alignment loss that enables valid skeleton-text pairs to counterbalance ambiguous ones, mitigating discrepancies and ambiguities in skeleton and text features, thereby ensuring robust alignment. Evaluations on the benchmarks demonstrate the effectiveness of our approach, validating that frequency-enhanced semantic features enable robust differentiation of visually and semantically similar action clusters, improving zero-shot action recognition.

[165] Robust and Accurate Multi-view 2D/3D Image Registration with Differentiable X-ray Rendering and Dual Cross-view Constraints

Yuxin Cui,Rui Song,Yibin Li,Max Q. -H. Meng,Zhe Min

Main category: cs.CV

TL;DR: 本研究开发了一种创新的多视角2D/3D刚性注册技术，通过两阶段流程提升注册精度和稳健性，并在实验中显示了比现有技术更好的性能。

Details

Motivation: 为了解决单幅术中图像视野有限所带来的挑战，需要利用多幅术中图像进行多视角2D/3D配准，从而实现稳健且准确的2D/3D配准，这对于成功的介入导航至关重要。 Method: 该论文的方法包括两个阶段：第一阶段设计了一个结合预测姿态与真实姿态差异以及模拟与观察术中图像之间差异（如归一化互相关）的联合损失函数，并引入了额外的跨视角训练损失项；第二阶段进行了测试时优化以精炼粗略阶段估计的姿态。 Result: 所提出的方法在DeepFluoro数据集的六个样本上实现了0.79±2.17毫米的平均目标配准误差（mTRE）。 Conclusion: 该论文提出了一种新的多视角2D/3D刚性配准方法，通过利用多视角投影姿态的相互约束来增强配准过程的鲁棒性，并在DeepFluoro数据集上展示了优于现有最先进配准算法的表现。 Abstract: Robust and accurate 2D/3D registration, which aligns preoperative models with intraoperative images of the same anatomy, is crucial for successful interventional navigation. To mitigate the challenge of a limited field of view in single-image intraoperative scenarios, multi-view 2D/3D registration is required by leveraging multiple intraoperative images. In this paper, we propose a novel multi-view 2D/3D rigid registration approach comprising two stages. In the first stage, a combined loss function is designed, incorporating both the differences between predicted and ground-truth poses and the dissimilarities (e.g., normalized cross-correlation) between simulated and observed intraoperative images. More importantly, additional cross-view training loss terms are introduced for both pose and image losses to explicitly enforce cross-view constraints. In the second stage, test-time optimization is performed to refine the estimated poses from the coarse stage. Our method exploits the mutual constraints of multi-view projection poses to enhance the robustness of the registration process. The proposed framework achieves a mean target registration error (mTRE) of $0.79 \pm 2.17$ mm on six specimens from the DeepFluoro dataset, demonstrating superior performance compared to state-of-the-art registration algorithms.

[166] ReF-LLE: Personalized Low-Light Enhancement via Reference-Guided Deep Reinforcement Learning

Ming Zhao,Pingping Liu,Tongshun Zhang,Zhe Zhang

Main category: cs.CV

TL;DR: 本文提出了一种新的个性化低光图像增强方法ReF-LLE，该方法结合了深度强化学习和傅里叶频率域处理，提高了低光图像的感知质量和适应性。

Details

Motivation: 低光图像增强面临两个主要挑战：不同条件下的显著变化以及受主观偏好和用户意图影响的增强水平。 Method: ReF-LLE利用零参考图像评估策略进行训练，并在推理阶段采用个性化自适应迭代策略，通过傅里叶域中的零频分量调整整体照明水平。 Result: 实验表明，ReF-LLE在个性化低光图像增强方面优于现有技术。 Conclusion: ReF-LLE通过结合深度强化学习和傅里叶频率域方法，实现了个性化的低光图像增强，并在基准数据集上证明了其优越的感知质量和适应性。 Abstract: Low-light image enhancement presents two primary challenges: 1) Significant variations in low-light images across different conditions, and 2) Enhancement levels influenced by subjective preferences and user intent. To address these issues, we propose ReF-LLE, a novel personalized low-light image enhancement method that operates in the Fourier frequency domain and incorporates deep reinforcement learning. ReF-LLE is the first to integrate deep reinforcement learning into this domain. During training, a zero-reference image evaluation strategy is introduced to score enhanced images, providing reward signals that guide the model to handle varying degrees of low-light conditions effectively. In the inference phase, ReF-LLE employs a personalized adaptive iterative strategy, guided by the zero-frequency component in the Fourier domain, which represents the overall illumination level. This strategy enables the model to adaptively adjust low-light images to align with the illumination distribution of a user-provided reference image, ensuring personalized enhancement results. Extensive experiments on benchmark datasets demonstrate that ReF-LLE outperforms state-of-the-art methods, achieving superior perceptual quality and adaptability in personalized low-light image enhancement.

[167] Boosting Classification with Quantum-Inspired Augmentations

Matthias Tschöpe,Vitor Fortes Rey,Sogo Pierre Sanon,Paul Lukowicz,Nikolaos Palaiodimopoulos,Maximilian Kiefer-Emmanouilidis

Main category: cs.CV

TL;DR: This paper explores the use of small quantum gate perturbations as a data augmentation method in classical machine learning, showing improvements in image classification performance but limitations in enhancing differential privacy.

Details

Motivation: Small quantum gate perturbations are typically seen as detrimental to quantum computation but might offer potential advantages in quantum machine learning. The research aims to explore this unique property for enhancing classical machine learning methods. Method: The paper investigates random Bloch sphere rotations as a quantum-inspired data augmentation technique and evaluates their impact on image classification using the ImageNet dataset. Result: Using quantum-inspired augmentation with Bloch rotations improved Top-1 accuracy by 3%, Top-5 accuracy by 2.5%, and increased the F1 score from 8% to 12%. Stronger unitary augmentations were found to produce visually unrecognizable images useful for privacy computations, though they did not enhance differential privacy. Conclusion: The study concludes that small quantum gate perturbations can enhance performance in machine learning as a natural source of data augmentation, although they do not improve differential privacy. Abstract: Understanding the impact of small quantum gate perturbations, which are common in quantum digital devices but absent in classical computers, is crucial for identifying potential advantages in quantum machine learning. While these perturbations are typically seen as detrimental to quantum computation, they can actually enhance performance by serving as a natural source of data augmentation. Additionally, they can often be efficiently simulated on classical hardware, enabling quantum-inspired approaches to improve classical machine learning methods. In this paper, we investigate random Bloch sphere rotations, which are fundamental SU(2) transformations, as a simple yet effective quantum-inspired data augmentation technique. Unlike conventional augmentations such as flipping, rotating, or cropping, quantum transformations lack intuitive spatial interpretations, making their application to tasks like image classification less straightforward. While common quantum augmentation methods rely on applying quantum models or trainable quanvolutional layers to classical datasets, we focus on the direct application of small-angle Bloch rotations and their effect on classical data. Using the large-scale ImageNet dataset, we demonstrate that our quantum-inspired augmentation method improves image classification performance, increasing Top-1 accuracy by 3%, Top-5 accuracy by 2.5%, and the F$_1$ score from 8% to 12% compared to standard classical augmentation methods. Finally, we examine the use of stronger unitary augmentations. Although these transformations preserve information in principle, they result in visually unrecognizable images with potential applications for privacy computations. However, we show that our augmentation approach and simple SU(2) transformations do not enhance differential privacy and discuss the implications of this limitation.

[168] 4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

Jiahui Zhang,Yurui Chen,Yueming Xu,Ze Huang,Yanpeng Zhou,Yu-Jie Yuan,Xinyue Cai,Guowei Huang,Xingyue Quan,Hang Xu,Li Zhang

Main category: cs.CV

TL;DR: 4D-VLA利用4D信息和内存库采样策略解决了机器人数据预训练中的坐标系和状态混乱问题，提升了模型性能和空间感知能力。

Details

Motivation: 现有方法由于输入的简单观察不完整而存在坐标系混乱和状态混乱的问题，导致条件动作分布分散，影响了预训练效率。 Method: 提出了一种名为4D-VLA的新方法，该方法使用顺序RGB-D输入来引入深度和时间信息，并采用内存库采样策略从历史图像中提取有用帧。 Result: 实验结果表明，4D-VLA在模拟和真实世界实验中的成功率都有显著提高，并且在MV-Bench多视角模拟基准测试中持续优于现有方法。 Conclusion: 4D-VLA通过引入深度和时间信息以及内存库采样策略，显著提高了预训练效率和模型性能，在空间感知和新视角泛化方面表现出色。 Abstract: Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset's action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.

[169] EAMamba: Efficient All-Around Vision State Space Model for Image Restoration

Yu-Cheng Lin,Yu-Syuan Xu,Hao-Wei Chen,Hsien-Kai Kuo,Chun-Yi Lee

Main category: cs.CV

TL;DR: This paper proposes EAMamba, an improved framework for image restoration that addresses the limitations of Vision Mamba by reducing computational complexity while maintaining performance.

Details

Motivation: Vision Mamba faces challenges such as computational complexity scaling with scanning sequences and local pixel forgetting in low-level vision tasks. Method: The study introduces EAMamba, which incorporates a Multi-Head Selective Scan Module (MHSSM) with an all-around scanning mechanism to address the limitations of Vision Mamba. Result: EAMamba achieves a 31-89% reduction in FLOPs across various image restoration tasks like super resolution, denoising, deblurring, and dehazing. Conclusion: EAMamba provides a more efficient and effective solution for image restoration tasks, achieving significant reductions in computational complexity while maintaining performance. Abstract: Image restoration is a key task in low-level computer vision that aims to reconstruct high-quality images from degraded inputs. The emergence of Vision Mamba, which draws inspiration from the advanced state space model Mamba, marks a significant advancement in this field. Vision Mamba demonstrates excellence in modeling long-range dependencies with linear complexity, a crucial advantage for image restoration tasks. Despite its strengths, Vision Mamba encounters challenges in low-level vision tasks, including computational complexity that scales with the number of scanning sequences and local pixel forgetting. To address these limitations, this study introduces Efficient All-Around Mamba (EAMamba), an enhanced framework that incorporates a Multi-Head Selective Scan Module (MHSSM) with an all-around scanning mechanism. MHSSM efficiently aggregates multiple scanning sequences, which avoids increases in computational complexity and parameter count. The all-around scanning strategy implements multiple patterns to capture holistic information and resolves the local pixel forgetting issue. Our experimental evaluations validate these innovations across several restoration tasks, including super resolution, denoising, deblurring, and dehazing. The results validate that EAMamba achieves a significant 31-89% reduction in FLOPs while maintaining favorable performance compared to existing low-level Vision Mamba methods.

[170] COOCO -- Common Objects Out-of-Context -- Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication

Filippo Merlo,Ece Takmaz,Wenkai Chen,Albert Gatt

Main category: cs.CV

TL;DR: This paper explores how Vision-Language Models (VLMs) use scene context for object reference generation, showing that models adaptively balance local and contextual information based on scene-object congruency and noise levels.

Details

Motivation: Understanding whether and how Vision-Language Models (VLMs) utilize scene context similarly to humans when referring to objects. Method: The study introduces the Common Objects Out-of-Context (COOCO) dataset and analyzes how VLMs use scene context under varying levels of scene-object congruency and perturbations, including attention analysis. Result: Models leverage scene context depending on semantic relatedness between object and scene and the noise level; more reliance is observed under high target-scene congruence or degraded objects. Attention analysis indicates increased focus on the target in mid-level layers under moderate noise. Conclusion: Vision-Language Models (VLMs) adaptively rely on scene contexts when generating references to objects, dynamically balancing local and contextual information. Abstract: Natural scenes provide us with rich contexts for object recognition and reference. In particular, knowing what type of scene one is looking at generates expectations about which objects will occur, and what their spatial configuration should be. Do Vision-Language Models (VLMs) learn to rely on scene contexts in a similar way, when generating references to objects? To address this question, we introduce the \textit{Common Objects Out-of-Context (COOCO)} dataset and test to what extent VLMs rely on scene context to refer to objects under different degrees of scene-object congruency, and different perturbations. Our findings show that models leverage scene context adaptively, depending on both the semantic relatedness between object and scene and the level of noise. In particular, models rely more on context under high target-scene congruence or when objects are degraded. Attention analysis reveals that successful object categorisation involves increased focus on the target in mid-level layers, especially under moderate noise, suggesting that VLMs dynamically balance local and contextual information for reference generation. We make our dataset, code and models available at \href{https://github.com/cs-nlp-uu/scenereg}{https://github.com/cs-nlp-uu/scenereg}.

[171] Rethinking Visual Token Reduction in LVLMs under Cross-modal Misalignment

Rui Xu,Yunke Wang,Yong Luo,Bo Du

Main category: cs.CV

TL;DR: This paper proposes VisionDrop, a training-free visual token reduction framework for LVLMs that leverages intra-modal attention and progressive pruning to achieve efficient inference while maintaining performance.

Details

Motivation: The motivation stems from the computational overhead caused by dense sequences of patch-level visual tokens in LVLMs, and limitations in existing text-guided visual token reduction methods due to cross-modal misalignment. Method: The paper introduces VisionDrop, which uses intra-modal (visual-to-visual) attention to select informative visual tokens and designs a progressive pruning pipeline treating the visual encoder and LLM as a unified system. Result: VisionDrop achieves consistent improvements over existing methods across diverse benchmarks while requiring no additional training or complex modifications. Conclusion: VisionDrop is a training-free, visual-only pruning framework that effectively reduces visual tokens in Large Vision-Language Models (LVLMs) by leveraging intra-modal attention without relying on textual signals. Abstract: Large Vision-Language Models (LVLMs) encode visual inputs as dense sequences of patch-level tokens to capture fine-grained semantics. These visual tokens often outnumber their textual counterparts by a large margin, leading to substantial computational overhead and limiting the scalability of LVLMs in practice. Previous efforts have explored visual token reduction either prior to or within the large language models (LLM). However, most in-LLM reduction approaches rely on text-conditioned interactions, implicitly assuming that textual tokens can reliably capture the importance of visual tokens. In this work, we revisit this assumption and reveal causal, semantic, and spatial forms of cross-modal misalignment. These misalignments undermine the effectiveness of text-guided visual token reduction. To address this, we introduce VisionDrop, a training-free, visual-only pruning framework that selects informative visual tokens based on intra-modal (visual-to-visual) attention, without relying on textual signals. To further suppress redundancy throughout the model hierarchy, we treat the visual encoder and the LLM as a unified system and design a progressive pruning pipeline. Our method performs dominant token selection and lightweight contextual merging at multiple stages, enabling fine-grained visual information to be retained even under aggressive token budgets. Extensive experiments across diverse benchmarks show that VisionDrop achieves consistent improvements over existing methods, despite requiring no additional training or complex modifications. Its simple yet effective design enables efficient inference while preserving strong performance across tasks.

[172] RoomCraft: Controllable and Complete 3D Indoor Scene Generation

Mengqi Zhou,Xipeng Wang,Yuxi Wang,Zhaoxiang Zhang

Main category: cs.CV

TL;DR: RoomCraft是一种多阶段生成3D室内场景的方法，结合了场景生成流水线和约束驱动的优化框架，有效解决了现有方法在全局空间推理和多约束场景下的问题。

Details

Motivation: 生成逼真的3D室内场景需要平衡几何一致性、空间关系和视觉真实性，但现有的神经生成方法由于全局空间推理能力有限而产生重复元素，而程序化方法在处理多约束情况下又存在困难。 Method: RoomCraft利用一个场景生成管道与约束驱动的优化框架相结合。首先，从用户输入中提取高层次的场景信息，并将其组织成包含房间类型、家具物品和空间关系的结构化格式。然后构建一个空间关系网络，用启发式深度优先搜索（HDFS）算法生成优化的放置顺序。此外，还引入了统一的约束表示和冲突感知定位策略（CAPS）来减少家具碰撞。 Result: 实验表明，RoomCraft在生成真实、语义一致且视觉吸引人的房间布局方面显著优于现有方法，适用于多种输入模式。 Conclusion: RoomCraft是一个多阶段的生成流程，可以将真实图像、草图或文本描述转换为连贯的3D室内场景。它通过结合场景生成流水线和约束驱动的优化框架，显著优于现有方法，能够处理复杂的多约束场景并确保布局的完整性。 Abstract: Generating realistic 3D indoor scenes from user inputs remains a challenging problem in computer vision and graphics, requiring careful balance of geometric consistency, spatial relationships, and visual realism. While neural generation methods often produce repetitive elements due to limited global spatial reasoning, procedural approaches can leverage constraints for controllable generation but struggle with multi-constraint scenarios. When constraints become numerous, object collisions frequently occur, forcing the removal of furniture items and compromising layout completeness. To address these limitations, we propose RoomCraft, a multi-stage pipeline that converts real images, sketches, or text descriptions into coherent 3D indoor scenes. Our approach combines a scene generation pipeline with a constraint-driven optimization framework. The pipeline first extracts high-level scene information from user inputs and organizes it into a structured format containing room type, furniture items, and spatial relations. It then constructs a spatial relationship network to represent furniture arrangements and generates an optimized placement sequence using a heuristic-based depth-first search (HDFS) algorithm to ensure layout coherence. To handle complex multi-constraint scenarios, we introduce a unified constraint representation that processes both formal specifications and natural language inputs, enabling flexible constraint-oriented adjustments through a comprehensive action space design. Additionally, we propose a Conflict-Aware Positioning Strategy (CAPS) that dynamically adjusts placement weights to minimize furniture collisions and ensure layout completeness. Extensive experiments demonstrate that RoomCraft significantly outperforms existing methods in generating realistic, semantically coherent, and visually appealing room layouts across diverse input modalities.

[173] OutDreamer: Video Outpainting with a Diffusion Transformer

Linhao Zhong,Fan Li,Yi Huang,Jianzhuang Liu,Renjing Pei,Fenglong Song

Main category: cs.CV

TL;DR: 本文提出了一种名为 OutDreamer 的视频外绘框架，该框架基于扩散变换模型（DiTs），通过创新性的结构设计和损失函数，在视频外绘任务中取得了优异的表现。

Details

Motivation: 现有的最先进的视频外绘方法在生成高质量和适应性强的内容方面仍面临挑战，而扩散变换模型（DiTs）因其卓越的性能成为了一种有前景的替代方案。 Method: OutDreamer 利用扩散变换模型（DiTs），结合高效的视频控制分支和条件外绘分支，并使用掩码驱动的自注意力层和潜在对齐损失来提高生成内容的质量和适应性。此外，采用跨视频片段优化器以确保长视频的时间一致性。 Result: OutDreamer 在广泛认可的基准测试中表现优于现有的最先进的零样本方法，特别是在长视频外绘任务中表现出良好的时间一致性。 Conclusion: OutDreamer 是一种基于 DiT 的视频外绘框架，通过引入高效的视频控制分支和条件外绘分支，以及掩码驱动的自注意力层和潜在对齐损失，实现了优于现有最先进零样本方法的性能。 Abstract: Video outpainting is a challenging task that generates new video content by extending beyond the boundaries of an original input video, requiring both temporal and spatial consistency. Many state-of-the-art methods utilize latent diffusion models with U-Net backbones but still struggle to achieve high quality and adaptability in generated content. Diffusion transformers (DiTs) have emerged as a promising alternative because of their superior performance. We introduce OutDreamer, a DiT-based video outpainting framework comprising two main components: an efficient video control branch and a conditional outpainting branch. The efficient video control branch effectively extracts masked video information, while the conditional outpainting branch generates missing content based on these extracted conditions. Additionally, we propose a mask-driven self-attention layer that dynamically integrates the given mask information, further enhancing the model's adaptability to outpainting tasks. Furthermore, we introduce a latent alignment loss to maintain overall consistency both within and between frames. For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content, ensuring temporal consistency across video clips. Extensive evaluations demonstrate that our zero-shot OutDreamer outperforms state-of-the-art zero-shot methods on widely recognized benchmarks.

[174] MatChA: Cross-Algorithm Matching with Feature Augmentation

Paula Carbó Cubero,Alberto Jaenal Gálvez,André Mateus,José Araújo,Patric Jensfelt

Main category: cs.CV

TL;DR: This paper introduces a novel method for improving visual localization in cross-feature scenarios by augmenting and translating feature descriptors into a latent space.

Details

Motivation: The motivation stems from the failure of current state-of-the-art methods to handle visual localization when different devices use varying sparse feature extraction algorithms. This leads to reduced performance due to low repeatability of keypoints and non-discriminatory descriptors. Method: The method involves feature descriptor augmentation for cross-detector feature matching, followed by feature translation into a latent space. Result: The result demonstrates that the proposed method enhances image matching and visual localization in cross-feature scenarios, outperforming existing approaches on several benchmarks. Conclusion: The paper concludes that their proposed method significantly improves image matching and visual localization in cross-feature scenarios, where different detectors and descriptors are used. Abstract: State-of-the-art methods fail to solve visual localization in scenarios where different devices use different sparse feature extraction algorithms to obtain keypoints and their corresponding descriptors. Translating feature descriptors is enough to enable matching. However, performance is drastically reduced in cross-feature detector cases, because current solutions assume common keypoints. This means that the same detector has to be used, which is rarely the case in practice when different descriptors are used. The low repeatability of keypoints, in addition to non-discriminatory and non-distinctive descriptors, make the identification of true correspondences extremely challenging. We present the first method tackling this problem, which performs feature descriptor augmentation targeting cross-detector feature matching, and then feature translation to a latent space. We show that our method significantly improves image matching and visual localization in the cross-feature scenario and evaluate the proposed method on several benchmarks.

[175] A Deep Learning framework for building damage assessment using VHR SAR and geospatial data: demonstration on the 2023 Turkiye Earthquake

Luigi Russo,Deodato Tapete,Silvia Liberata Ullo,Paolo Gamba

Main category: cs.CV

TL;DR: 本文提出了一种基于单时相SAR影像和地理空间数据的深度学习方法，实现了快速、准确的建筑物损毁识别，适用于灾后应急场景。

Details

Motivation: 传统的光学卫星影像受限于云层遮挡或缺乏灾前影像，因此需要一种更快速、可靠的方法用于灾后建筑物损毁识别以指导应急响应和恢复工作。 Method: 结合SAR图像、OpenStreetMap建筑物轮廓、数字地表模型（DSM）以及Global Earthquake Model（GEM）的结构与暴露属性，使用深度学习方法进行损毁检测。 Result: 在2023年土耳其地震数据集上的实验表明，引入地理空间特征显著提高了检测性能，并具备良好的泛化能力，能够在没有灾前数据的情况下实现高效损毁评估。 Conclusion: 该研究提出了一种新的多模态深度学习框架，利用单时相SAR图像和辅助地理空间数据进行建筑物损毁识别，有效解决了灾后紧急响应中的关键问题。 Abstract: Building damage identification shortly after a disaster is crucial for guiding emergency response and recovery efforts. Although optical satellite imagery is commonly used for disaster mapping, its effectiveness is often hampered by cloud cover or the absence of pre-event acquisitions. To overcome these challenges, we introduce a novel multimodal deep learning (DL) framework for detecting building damage using single-date very high resolution (VHR) Synthetic Aperture Radar (SAR) imagery from the Italian Space Agency (ASI) COSMO SkyMed (CSK) constellation, complemented by auxiliary geospatial data. Our method integrates SAR image patches, OpenStreetMap (OSM) building footprints, digital surface model (DSM) data, and structural and exposure attributes from the Global Earthquake Model (GEM) to improve detection accuracy and contextual interpretation. Unlike existing approaches that depend on pre and post event imagery, our model utilizes only post event data, facilitating rapid deployment in critical scenarios. The framework effectiveness is demonstrated using a new dataset from the 2023 earthquake in Turkey, covering multiple cities with diverse urban settings. Results highlight that incorporating geospatial features significantly enhances detection performance and generalizability to previously unseen areas. By combining SAR imagery with detailed vulnerability and exposure information, our approach provides reliable and rapid building damage assessments without the dependency from available pre-event data. Moreover, the automated and scalable data generation process ensures the framework's applicability across diverse disaster-affected regions, underscoring its potential to support effective disaster management and recovery efforts. Code and data will be made available upon acceptance of the paper.

[176] Closing the Performance Gap in Biometric Cryptosystems: A Deeper Analysis on Unlinkable Fuzzy Vaults

Hans Geißner,Christian Rathgeb

Main category: cs.CV

TL;DR: 该研究解决了模糊保险库在生物识别密码系统中的性能下降问题，通过引入基于等频区间的特征量化方法，提高了系统稳定性和兼容性。

Details

Motivation: 模糊保险库在生物识别密码系统中存在性能差距，主要由于特征集大小变化和相似性阈值影响造成的不稳定纠错能力以及特征类型转换过程中的信息丢失问题。 Method: 分析模糊保险库（fuzzy vault）在生物识别密码系统（BCS）中的性能差距，识别不稳定纠错能力和特征类型转换引起的信息丢失问题，并提出一种新的特征量化方法。 Result: 实验表明，这种方法显著降低了模板保护导致的性能下降，同时在最先进的人脸、指纹和虹膜识别系统中仅造成最小的性能损失，证明了其在主要生物识别模式下的有效性。 Conclusion: 该论文提出了一种基于等频区间的新特征量化方法，有效减小了模板保护带来的性能差距，并且与现有系统兼容，减少了特征转换的负面影响。 Abstract: This paper analyses and addresses the performance gap in the fuzzy vault-based \ac{BCS}. We identify unstable error correction capabilities, which are caused by variable feature set sizes and their influence on similarity thresholds, as a key source of performance degradation. This issue is further compounded by information loss introduced through feature type transformations. To address both problems, we propose a novel feature quantization method based on \it{equal frequent intervals}. This method guarantees fixed feature set sizes and supports training-free adaptation to any number of intervals. The proposed approach significantly reduces the performance gap introduced by template protection. Additionally, it integrates seamlessly with existing systems to minimize the negative effects of feature transformation. Experiments on state-of-the-art face, fingerprint, and iris recognition systems confirm that only minimal performance degradation remains, demonstrating the effectiveness of the method across major biometric modalities.

[177] From Ground to Air: Noise Robustness in Vision Transformers and CNNs for Event-Based Vehicle Classification with Potential UAV Applications

Nouf Almesafri,Hector Figueiredo,Miguel Arana-Catania

Main category: cs.CV

TL;DR: This paper compares the performance of ResNet34 and ViT B16 on event-based camera data for vehicle classification, showing that ResNet34 has higher accuracy but ViT B16 is more robust.

Details

Motivation: To assess the performance of Convolutional Neural Networks (ResNet34) and Vision Transformers (ViT B16) for event-based cameras, which are particularly suited for dynamic environments like UAVs and autonomous vehicles. Method: The study fine-tunes ResNet34 and ViT B16 models on the GEN1 event-based dataset and evaluates them under both standard and simulated noisy conditions. Result: On the clean GEN1 dataset, ResNet34 achieves 88% accuracy while ViT B16 reaches 86%. ViT B16 shows greater robustness, especially considering its smaller pre-training dataset. Conclusion: ResNet34 slightly outperforms ViT B16 in classification accuracy on the GEN1 event-based dataset under standard conditions, but ViT B16 demonstrates notable robustness despite being pre-trained on a smaller dataset. Abstract: This study investigates the performance of the two most relevant computer vision deep learning architectures, Convolutional Neural Network and Vision Transformer, for event-based cameras. These cameras capture scene changes, unlike traditional frame-based cameras with capture static images, and are particularly suited for dynamic environments such as UAVs and autonomous vehicles. The deep learning models studied in this work are ResNet34 and ViT B16, fine-tuned on the GEN1 event-based dataset. The research evaluates and compares these models under both standard conditions and in the presence of simulated noise. Initial evaluations on the clean GEN1 dataset reveal that ResNet34 and ViT B16 achieve accuracies of 88% and 86%, respectively, with ResNet34 showing a slight advantage in classification accuracy. However, the ViT B16 model demonstrates notable robustness, particularly given its pre-training on a smaller dataset. Although this study focuses on ground-based vehicle classification, the methodologies and findings hold significant promise for adaptation to UAV contexts, including aerial object classification and event-based vision systems for aviation-related tasks.

[178] Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation

Tiankai Chen,Yushu Li,Adam Goodge,Fei Teng,Xulei Yang,Tianrui Li,Xun Xu

Main category: cs.CV

TL;DR: This paper proposes a training-free framework using Vision-Language Models and Graph Score Propagation (GSP) for Out-of-Distribution detection in 3D point cloud data, which outperforms current state-of-the-art methods.

Details

Motivation: OOD detection in 3D point cloud data is challenging but critical for safe and robust perception in certain applications. Existing methods for 2D image data face unique obstacles when extended to 3D environments. Method: The paper constructs a graph based on class prototypes and testing data to leverage the data manifold structure, enhancing VLMs' effectiveness. It proposes a Graph Score Propagation (GSP) method incorporating prompt clustering and self-training negative prompting. Result: The proposed training-free framework effectively leverages Vision-Language Models (VLMs) for OOD detection in 3D point clouds, with GSP demonstrating consistent superiority over existing methods across synthetic and real-world datasets. Conclusion: The paper concludes that the proposed GSP method outperforms state-of-the-art methods for OOD detection in 3D point cloud datasets. Abstract: Out-of-distribution (OOD) detection in 3D point cloud data remains a challenge, particularly in applications where safe and robust perception is critical. While existing OOD detection methods have shown progress for 2D image data, extending these to 3D environments involves unique obstacles. This paper introduces a training-free framework that leverages Vision-Language Models (VLMs) for effective OOD detection in 3D point clouds. By constructing a graph based on class prototypes and testing data, we exploit the data manifold structure to enhancing the effectiveness of VLMs for 3D OOD detection. We propose a novel Graph Score Propagation (GSP) method that incorporates prompt clustering and self-training negative prompting to improve OOD scoring with VLM. Our method is also adaptable to few-shot scenarios, providing options for practical applications. We demonstrate that GSP consistently outperforms state-of-the-art methods across synthetic and real-world datasets 3D point cloud OOD detection.

[179] Can Video Large Multimodal Models Think Like Doubters-or Double-Down: A Study on Defeasible Video Entailment

Yue Zhang,Jilei Sun,Yunhui Guo,Vibhav Gogate

Main category: cs.CV

TL;DR: This paper introduces Defeasible Video Entailment (DVidE) to enhance the adaptive reasoning of Video Large Multimodal Models by enabling them to update conclusions with new evidence, supported by innovative frameworks and a benchmark dataset.

Details

Motivation: VLMMs often struggle with abstract and adaptive reasoning when new information emerges, making it necessary to develop methods that enable these models to dynamically revise interpretations based on updated context. Method: The method involves introducing Defeasible Video Entailment (DVidE), a new task for models to adaptively update reasoning based on evolving evidence. For classification tasks, the Chain of Counterfactual Thought framework is proposed, which uses counterfactual reasoning, ASR-enhanced video content, and rationale refinement. For generation tasks, a framework combining ASR output with an LLM is developed. A new benchmark dataset and evaluation metric are also introduced. Result: Experimental results show significant improvements in the dynamic reasoning capabilities of VLMMs using the proposed methods, particularly for both classification and generation tasks under the DVidE framework. Conclusion: The study concludes that incorporating defeasible reasoning into VLMMs significantly enhances their dynamic reasoning capabilities, as demonstrated through the DVidE task and novel frameworks for classification and generation tasks. Abstract: Video Large Multimodal Models (VLMMs) have made impressive strides in understanding video content, but they often struggle with abstract and adaptive reasoning-the ability to revise their interpretations when new information emerges. In reality, conclusions are rarely set in stone; additional context can strengthen or weaken an initial inference. To address this, we introduce Defeasible Video Entailment (DVidE), a new task that challenges models to think like doubters, constantly updating their reasoning based on evolving evidence. In DVidE, given a video premise and a textual hypothesis, models must determine whether a new update strengthens or weakens the hypothesis (classification version) or generate a coherent update that modifies the entailment relationship (generation version). For solving the classification task, we propose the Chain of Counterfactual Thought framework, utilizing counterfactual reasoning, ASR-enhanced video content, and rationale refinement to reduce inference bias. For the generation task, we develop a framework that combines ASR output with a Large Language Model (LLM) to produce coherent, contextually relevant updates aligned with the intended strengthener or weakener goals. Additionally, we introduce a novel benchmark dataset, with strengthener/weakener annotations and an LLM-based evaluation metric specifically designed for assessing generative performance. Experimental results demonstrate significant improvements, highlighting our proposed method in enhancing dynamic reasoning capabilities of VLMMs.

[180] Test-Time Consistency in Vision Language Models

Shih-Han Chou,Shivam Chandhok,James J. Little,Leonid Sigal

Main category: cs.CV

TL;DR: This paper proposes a simple and effective test-time framework to improve semantic consistency in Vision-Language Models (VLMs), addressing inconsistencies in predictions for semantically equivalent inputs without requiring retraining or architectural modifications.

Details

Motivation: Despite high average accuracy, Vision-Language Models (VLMs) often produce inconsistent predictions for semantically equivalent inputs, affecting their reliability and robustness, as highlighted by benchmarks like MM-R3. Method: The method introduces two objectives: (i) Cross-Entropy Agreement Loss to align predictive distributions across equivalent inputs and (ii) Pseudo-Label Consistency Loss to drive outputs toward a consensus. It is post-hoc, model-agnostic, and does not require architectural changes or retraining. Result: Experiments on the MM-R3 benchmark demonstrate significant improvements in consistency across state-of-the-art VLMs, establishing a new direction for inference-time adaptation in multimodal learning. Conclusion: The proposed test-time consistency framework effectively enhances semantic consistency in Vision-Language Models (VLMs) without supervised re-training, offering a plug-and-play solution that leverages information from single test inputs. Abstract: Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks, yet they often exhibit inconsistent behavior when faced with semantically equivalent inputs, undermining their reliability and robustness. Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs, despite maintaining high average accuracy. Prior work addresses this issue by modifying model architectures or conducting large-scale fine-tuning on curated datasets. In contrast, we propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training. Our method is entirely post-hoc, model-agnostic, and applicable to any VLM with access to its weights. Given a single test point, we enforce consistent predictions via two complementary objectives: (i) a Cross-Entropy Agreement Loss that aligns predictive distributions across semantically equivalent inputs, and (ii) a Pseudo-Label Consistency Loss that draws outputs toward a self-averaged consensus. Our method is plug-and-play and leverages information from a single test input itself to improve consistency. Experiments on the MM-R3 benchmark show that our framework yields substantial gains in consistency across state-of-the-art models, establishing a new direction for inference-time adaptation in multimodal learning.

[181] Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy

Yuhao Liu,Tengfei Wang,Fang Liu,Zhenwei Wang,Rynson W. H. Lau

Main category: cs.CV

TL;DR: Shape-for-Motion 提出了一种基于 3D 代理的视频编辑框架，实现了精确且一致的视频编辑，能够处理包括姿态调整、纹理修改等多种复杂操作。

Details

Motivation: 现有的视频合成方法在确保与用户意图的细粒度对齐方面仍存在挑战，需要更精确和一致的控制工具来实现创造性编辑意图。 Method: 将目标对象转换为时间一致的网格（3D 代理），并设计了一种双传播策略，允许用户在单帧的 3D 网格上进行编辑，并自动传播到其他帧的 3D 网格。 Result: 该框架支持多种跨视频帧的精确和物理一致性操作，包括姿势编辑、旋转、缩放、平移、纹理修改和对象组合，并经过大量实验验证其优越性和有效性。 Conclusion: Shape-for-Motion 是一个创新框架，通过使用 3D 代理实现精确且一致的视频编辑，标志着高质量可控视频编辑工作流程的关键一步。 Abstract: Recent advances in deep generative modeling have unlocked unprecedented opportunities for video synthesis. In real-world applications, however, users often seek tools to faithfully realize their creative editing intentions with precise and consistent control. Despite the progress achieved by existing methods, ensuring fine-grained alignment with user intentions remains an open and challenging problem. In this work, we present Shape-for-Motion, a novel framework that incorporates a 3D proxy for precise and consistent video editing. Shape-for-Motion achieves this by converting the target object in the input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be performed directly on the proxy and then inferred back to the video frames. To simplify the editing process, we design a novel Dual-Propagation Strategy that allows users to perform edits on the 3D mesh of a single frame, and the edits are then automatically propagated to the 3D meshes of the other frames. The 3D meshes for different frames are further projected onto the 2D space to produce the edited geometry and texture renderings, which serve as inputs to a decoupled video diffusion model for generating edited results. Our framework supports various precise and physically-consistent manipulations across the video frames, including pose editing, rotation, scaling, translation, texture modification, and object composition. Our approach marks a key step toward high-quality, controllable video editing workflows. Extensive experiments demonstrate the superiority and effectiveness of our approach. Project page: https://shapeformotion.github.io/

[182] WarpRF: Multi-View Consistency for Training-Free Uncertainty Quantification and Applications in Radiance Fields

Sadra Safadoust,Fabio Tosi,Fatma Güney,Matteo Poggi

Main category: cs.CV

TL;DR: WarpRF是一种通用且无需训练的方法，用于量化辐射场的不确定性，并在多个任务中表现优异。

Details

Motivation: 为了克服现有方法需要特定框架和训练的限制，提出一种简单、低成本且通用的不确定性量化方法。 Method: 基于前向模型应保持光度和几何一致性的假设，通过反向扭曲不同视角的图像，在不可见视角上测量其一致性来量化基础不确定性。 Result: WarpRF在不确定性量化以及主动视角选择和主动映射等下游任务中表现出色，优于所有现有的专门方法。 Conclusion: WarpRF是一个无需训练的通用框架，能够有效地量化辐射场的不确定性，并且在不确定量化和下游任务方面优于现有的特定框架方法。 Abstract: We introduce WarpRF, a training-free general-purpose framework for quantifying the uncertainty of radiance fields. Built upon the assumption that photometric and geometric consistency should hold among images rendered by an accurate model, WarpRF quantifies its underlying uncertainty from an unseen point of view by leveraging backward warping across viewpoints, projecting reliable renderings to the unseen viewpoint and measuring the consistency with images rendered there. WarpRF is simple and inexpensive, does not require any training, and can be applied to any radiance field implementation for free. WarpRF excels at both uncertainty quantification and downstream tasks, e.g., active view selection and active mapping, outperforming any existing method tailored to specific frameworks.

[183] MiCo: Multi-image Contrast for Reinforcement Visual Reasoning

Xi Chen,Mingkang Zhu,Shaoteng Liu,Xiaoyang Wu,Xiaogang Xu,Yu Liu,Xiang Bai,Hengshuang Zhao

Main category: cs.CV

TL;DR: 本论文提出了一种无需人工标注问题-答案对的方法，通过自监督学习增强视觉语言模型的推理能力。

Details

Motivation: 现有的基于规则强化学习方法依赖于手动构建的问题-答案对，难以应对细粒度视觉细节和跨图像复杂逻辑的问题。 Method: 利用自监督视觉表示学习的思想，构建包含同一图像两个增强视图和一个相似但不同图像的三元组进行训练，引导模型生成比较图像的推理过程，并使用基于规则的强化学习优化模型。 Result: 尽管仅在视觉比较任务上进行训练，该方法在多种问题上均表现出良好的泛化能力，在多图像推理基准测试中取得了显著改进，并在通用视觉任务中表现优异。 Conclusion: 该方法有效提升了视觉语言模型的推理能力，且不依赖人工标注数据。 Abstract: This work explores enabling Chain-of-Thought (CoT) reasoning to link visual cues across multiple images. A straightforward solution is to adapt rule-based reinforcement learning for Vision-Language Models (VLMs). However, such methods typically rely on manually curated question-answer pairs, which can be particularly challenging when dealing with fine grained visual details and complex logic across images. Inspired by self-supervised visual representation learning, we observe that images contain inherent constraints that can serve as supervision. Based on this insight, we construct image triplets comprising two augmented views of the same image and a third, similar but distinct image. During training, the model is prompted to generate a reasoning process to compare these images (i.e., determine same or different). Then we optimize the model with rule-based reinforcement learning. Due to the high visual similarity and the presence of augmentations, the model must attend to subtle visual changes and perform logical reasoning to succeed. Experiments show that, although trained solely on visual comparison tasks, the learned reasoning ability generalizes effectively to a wide range of questions. Without relying on any human-annotated question-answer pairs, our method achieves significant improvements on multi-image reasoning benchmarks and shows strong performance on general vision tasks.