ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding

Hao Lu*, Jiahao Wang*, Yaolun Zhang, Ruohui Wang, Xuanyu Zheng, Yepeng Tang, Dahua Lin, Lewei Lu
SenseTime Reaserch
*Indicates Equal Contribution
Paper Data Code arXiv

Example clips for ELV-Halluc videos.

Abstract

Sometimes, MLLMs generate incorrect outputs but with correct frame-level semantics. We refer to this type of hallucination as Semantic Aggregation Hallucination (SAH), which arises during the process of aggregating frame-level semantics into event-level semantic groups. We introduce ELV-Halluc, the first benchmark dedicated to long-video hallucination, enabling a systematic investigation of SAH. Our experiments confirm the existence of SAH and show that it increases with semantic complexity. Additionally, we find that models are more prone to SAH on rapidly changing semantics. Moreover, we discuss potential approaches to mitigate SAH. We demonstrate that positional encoding strategy contributes to alleviating SAH, and further adopt DPO strategy to enhance the model’s ability to distinguish semantics within and across events. To support this, we curate a dataset of 8K adversarial data pairs and achieve improvements on both ELV-Halluc and Video-MME.

Leaderboard

Rank Model LLM Size Open-Source? Visual Details In/Out Object In/Out Action In/Out Declarative Content In/Out Avg Acc↑ SAH Ratio↓
1 Gemini2.5-Flash / No 47 / 58 56.5 / 58.8 50.5 / 53.2 48.7 / 52 53.1 4.8
2 Qwen2.5VL-72B 72B Yes 24 / 35.5 35.7 / 41.5 27.8 / 32.3 32.3 / 27 32.0 4.1
3 InternVL3-78B 72B Yes 25 / 31.2 32 / 36.5 28.5 / 31.2 24.2 / 26.5 29.3 3.9
4 InternVL3-38B 32B Yes 25.3 / 29 24.2 / 28 24 / 30 24.5 / 24.2 26.1 3.3
5 InternVL3-14B 14B Yes 17.5 / 24.5 22.8 / 24.5 16.3 / 17.7 15.2 / 15.5 19.2 2.6
6 Qwen2.5VL-7b 7B Yes 10.2 / 26 17.5 / 30.7 13 / 20.7 16.8 / 10.5 18.1 7.6
7 Qwen2.5VL-32B 32B Yes 16.5 / 24.5 21.7 / 24.5 17.2 / 15.0 15.2 / 7.2 17.7 0.1
8 InternVL3-8B 7B Yes 12.5 / 19.5 14.5 / 19.5 13.5 / 20.5 12.8 / 17.7 16.3 5.9
9 InternVL3-2B 1.5B Yes 7 / 15.5 8.7 / 17.2 7.2 / 10.5 10 / 13 11.1 6.3
10 InternVL3-1B 0.5B Yes 8 / 11 8.7 / 11 8.7 / 12.5 11.3 / 8.3 9.9 1.6
11 LLaVA-OV-7b 7B Yes 8 / 13.2 9.5 / 13.7 8.7 / 10.7 7.7 / 7.5 9.9 2.8
12 GPT-4o / No 7.7 / 8.3 8 / 8.7 8.7 / 10.2 8.5 / 9.5 8.7 0.9
13 Qwen2.5VL-3B 3B Yes 2.2 / 10.5 7.7 / 13.8 5 / 8 6 / 6 7.4 4.5
14 LLaVA-Video-7B 7B Yes 3.7 / 3.7 4.5 / 2.5 3.7 / 3.2 4 / 4 3.6 -0.6
15 Video-chatgpt-7B 7B Yes 2 / 2.5 2.5 / 1.7 1.2 / 1.2 2.2 / 3.2 2.0 0.2
16 SmolVLM-2.2B 1.7B Yes 0 / 0 3 / 5 0 / 0 0 / 0 1 0.5

BibTeX

@misc{lu2025elvhallucbenchmarkingsemanticaggregation,
      title={ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding}, 
      author={Hao Lu and Jiahao Wang and Yaolun Zhang and Ruohui Wang and Xuanyu Zheng and Yepeng Tang and Dahua Lin and Lewei Lu},
      year={2025},
      eprint={2508.21496},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.21496}, 
}