Video Abstractive and Extractive Reasoning Benchmark Leaderboard

Overview. VAEX-Bench is a benchmark for evaluating abstractive spatiotemporal reasoning in Large Multimodal Models (LMMs) using high-fidelity egocentric videos rendered from manually constructed 3D indoor environments. We design each episode as a controllable scenario in Trimble SketchUp and render photorealistic views with Chaos Enscape, enabling strict spatial consistency and unambiguous ground truth. VAEX-Bench decomposes beyond-the-visible reasoning into five tasks—Memory-Action, Map Direction, Map Scale, Simulation, and Global Counting—which jointly test long-horizon memory, allocentric map construction (direction and distance), viewpoint transformation, and global aggregation under partial observability. All QA pairs are human-authored and verified, and we provide task-specific error analyses to diagnose failure modes, revealing where current models break down when they must integrate dispersed cues across time and space rather than rely on frame-local evidence.

Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 Scenario 7 Scenario 8 Scenario 9 Scenario 10

Paper (ArXiv)

Code (Github) Coming soon!

VAEX-Bench Coming soon!

Leaderboard

Overview

The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extractive reasoning, where answers can be explicitly presented within spatiotemporal events. It remains unclear whether multimodal large language models can instead perform abstractive spatiotemporal reasoning, which requires integrating observations over time, combining dispersed cues, and inferring implicit spatial and contextual structure. To address this gap, we formalize abstractive spatiotemporal reasoning from videos by introducing a structured evaluation taxonomy that systematically targets its core dimensions and construct a controllable, scenario-driven synthetic egocentric video dataset tailored to evaluate abstractive spatiotemporal reasoning capabilities, spanning object-, room-, and floor-plan–level scenarios. Based on this framework, we present VAEX-BENCH, a benchmark comprising five abstractive reasoning tasks together with their extractive counterparts. Our extensive experiments compare the performance of state-of-the-art MLLMs under extractive and abstractive settings, exposing their limitations on abstractive tasks and providing a fine-grained analysis of the underlying bottlenecks. The dataset will be released soon.

01

Perceptual Bottleneck

We analyze failures on Global Counting , requiring models to detect and track object instances before aggregating them into global counts. To obtain a granular view of model failures, we decompose errors into three mutually exclusive categories:

01.

Omission Error An error in which an object that is actually present in the scenario is incorrectly judged as absent and therefore not counted.

02.

Hallucination Error An error in which an object that does not actually exist in the scenario is incorrectly judged to be present and is therefore included in the count.

03.

Miscounting Error An error in which an object is correclty recognized as being present in the scenario, but the number assigned to that object is incorrect.

---

02

Temporal Bottleneck

We analyze failures on Memory-Action , requiring models to recall the sequence of visited rooms along the trajectory and use that temporal memory to identify the correct action associated with a target location. To disentangle different sources of failure, we perform a fine-grained error analysis along two hierarchical components:

Global Temporal Memory (Hop 1)

● Pass: Recalls the full trajectory.

● Fail: Forgets the sequence of rooms.

Local Temporal Memory (Hop 2)

● Pass: Matches action to target.

● Fail: Fails to link memory to action.

---

03

Spatial Bottleneck

We analyze failures on Map Direction , Map Scale , Simulation , requiring a coherent spatial representation of the environment. Models must reason about relative room orientation, maintain consistent metric proportions, and simulate viewpoint transformations within a unified layout. To diagnose these spatial bottlenecks, we evaluate the MLLMs to explicitly reconstruct a global cognitive map via grid-based floor-plan prediciton.

We present VAEX-BENCH, a scenario-driven benchmark for evaluating abstractive and extractive spatiotemporal reasoning in egocentric videos, moving beyond evidence-present extraction toward long-horizon memory, allocentric map reasoning, and global evidence aggregation. Unlike prior benchmarks built on fixed captured footage, VAEX-BENCH is manually constructed with controlled scenario design and spatially consistent QA pairs, enabling precise diagnosis of failure modes. Our experiments reveal persistent bottlenecks in global spatial memory and structured evidence integration across state-of-the-art MLLMs. Future work will focus on automating the pipeline to scale benchmark generation while preserving its controllability.

@article{bang2026reasoning,
  title={Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence},
  author={Bang, Seunghwan and Song, Hwanjun},
  journal={arXiv preprint arXiv:2603.13091},
  year={2026}
}