DIVE-Bench tests video-language models when evidence is not confined to a few keyframes—from long-form educational content to precise high-motion trajectories.
1Northeastern University 2Princeton University 3University of Maryland
Most video benchmarks reward sparse glimpses. DIVE-Bench isolates the harder case: details accumulate over long timelines, or change faster than sparse sampling can preserve.
Educational High-FPS Videos · LPM
Questions target visible text and subtitles distributed across educational videos. The released, gated split contains 634 evaluation items, with two prompts per source video. This evaluation uses video frames, not an audio-input protocol.
High-Motion High-FPS Videos · v2 preview
Models predict a hand trajectory over a semantic 3×3 image grid. The corrected v2 reference uses the right-hand ring-finger metacarpal as a named palm proxy, with invalid reference positions masked. Published results cover the fixed first 1,000 source records, not the full 3,243-item test split. Source media remain separately authorized and are not bundled with the code.
Gated Residual Tokenization (GRT) treats video more like a codec: preserve a full visual reference, then selectively recompute residual content while consolidating repeated scene semantics.
High temporal coverage
Reuse static regions; recompute residual change
Consolidate semantically redundant tokens
Answer with temporally dense evidence
Motion-aware gating reuses selected patch projections for visual regions that remain stable from frame to frame.
Semantic-scene merging reduces redundant context after visual encoding while retaining dynamic evidence.
Recompute ratio measures patch-projection reuse; sampling density (fps) is sampled frames divided by source duration, not end-to-end throughput.
Switch tracks, search models, filter sources, and sort any metric. A dash means unreported or undefined under the valid-reference mask, never zero. High-Motion v2 shows metric-specific coverage; Educational methods without Open MOS sort after scored methods, then by Token F1.
29 Educational results, 19 High-Motion v2 preview results and 12 educational GRT comparison rows (60 CSV records), audited 15 September 2026. Full HTML tables (no JavaScript) · Download CSV · Comparison evidence. Educational scores are not a fresh GPU rerun. High-Motion compares one new GRT run with 18 CPU-rescored archived baselines; baseline inference was not repeated. Missing values are not zero; no score is imputed.
—
All three promoted educational profiles exceed their archived and matched controls on Open MOS and Token F1 while recomputing fewer patch projections. This is a point-estimate result, not a significance claim or a win on every metric. The larger gap to archived Qwen baselines includes protocol differences and must not be attributed entirely to gating.
| Family / control | Open MOS ↑ | Token F1 ↑ | Patch recompute ↓ | Mean throughput (fps) ↑ | Mean request time (s) ↓ |
|---|
Qwen 3B GRT is slower than its matched all-patch control on the reported mean throughput (1.673 vs 1.750 fps), despite better quality and lower patch compute. Timing is historical per-request telemetry, not repeated end-to-end speed benchmarking. Eight sampled frames do not establish high-FPS coverage; patch reuse is not end-to-end FLOPs. Inspect exact values and all control rows.
High-Motion v2: corrected references, mixed metric outcomes. On the fixed 1,000-record preview, GRT exceeds its corresponding HF 0.5B baseline on observed Grid Accuracy, with improved Grid ADE, Grid FDE and Token F1; Transition Accuracy regresses. All five exact differences remain visible. Both sides use the same corrected reference mask and unchanged questions. The right-ring metacarpal is an explicit palm proxy; valid projection is not proof of RGB visibility. This is not a full 3,243-record evaluation or a statistical-significance claim. Exact historical baseline weight revisions and consumed-tensor identity remain unproven.
The v2 reference/scoring contract, source-bound reproduction guide, pinned release manifest and numeric evidence bundle document the versioned correction. Legacy High-Motion scores remain withheld; the unchanged 27-run historical protocol audit is evidence, not approved rankings. Original construction and access limitations remain documented in the historical reference review.
How to read the table
There is no hidden composite score. Each track exposes its task metric and supporting quality or efficiency measures.
Result provenance. The immutable historical snapshot was generated from 21 evaluation artifacts on 20 August 2026. Educational Open MOS uses Qwen/Qwen3-VL-32B-Instruct as the open text judge. Public score audit · Educational reproduction protocol. Frozen score bytes remain available in the original snapshot; corrected High-Motion v2 uses its separate dated evidence bundle.
The minimal code and numerical evidence support an offline rebuild of 29 Educational results, 19 corrected High-Motion preview results and 12 educational GRT comparison rows. The original 32-row snapshot remains unchanged as historical evidence. Reaggregating saved numeric evidence does not rerun models. Fresh inference requires appropriate GPU resources and separately authorized source data; private source videos, HDF5 files and derived annotations are not bundled or licensed for redistribution by this code release.
git clone --branch main --single-branch \
https://github.com/Hai-chao-Zhang/DenseVideoUnderstand.git DIVE-Bench
cd DIVE-Bench
python -m pip install 'PyYAML>=6'
# 29 Educational + 19 High-Motion preview + 12 comparison rows (60 CSV records).
# Pinned v2 manifest: 1f64ff54ec8eb09d72c37d6ef3a944e8ccae0a58fe5b4d45fabdfb0a7449d0dc
python -m tools.densevideo.build_complete_leaderboard --verify-only
python -m tools.densevideo.build_complete_leaderboard --output outputs/leaderboard-complete
# Open outputs/leaderboard-complete/leaderboard.html; output must be new.Protocol notes
Framework submissions
VLMEvalKit #1686 and lmms-eval #1521 are submitted Draft PRs, not upstream acceptance. Data permissions and manuscript/protocol differences remain documented in the public audit.
If the benchmark or GRT is useful in your research, please cite the project paper. The citation below identifies the earlier educational-scope arXiv version; the bundled revised manuscript covers both tasks.
@article{zhang2025dive,
title = {Dense Video Understanding with
Gated Residual Tokenization},
author = {Zhang, Haichao and Chai, Wenhao and
He, Shwai and Li, Ang and Fu, Yun},
journal = {arXiv preprint arXiv:2509.14199},
year = {2025}
}