Skip to content
ProjectForty2
Menu
Public draft · v0.2 · May 2026 · Vision-Language Capability Ledger
★ Public Draft · v0.2 · Real Data8 Sweeps · 397 Papers

The Twelve-Bill Closure Pattern of Frontier VLM Capability Claims (2024–2026)

A real-data falsification harness with 397-paper empirical verification.
Kevin Russell · ProjectForty2 · 2026-05-09 · Public draft v0.2.

Abstract

The vision-language capability ledger audits 397 unique papers across 8 deep-loop sweeps (2024–2026) on frontier VLM-capability claims. The harness consists of 12 bills covering image-search contamination, OCR-extracted-text leakage, vision-tokenizer brittleness, causally-faithful vision-grounding, cross-VLM-architecture portability, tool-augmented-vision decomposition, cross-benchmark generalization, multi-image/video, anti-saturation construction, and distilled-cousin reproduction; six meta-costs; and three escape gates.

Three bills are predicted ★ empty and the empty-space hypothesis HOLDS: Bill 4 ★ (causally-faithful vision-grounding — 32 candidates, 0 clean; Tong-Du-Liang Eyes Wide Shut: 25–40% caption-only; BLINK 47pp visual-vs-language gap), Bill 7 ★ (cross-benchmark generalization — 6 candidates, 0 clean; Yue-Bommasani 2024-Q4 MMMU-Pro 14–23pp drop; pairwise r=0.31–0.51), Bill 10 ★ (universal vision-task coverage — 5 candidates, 0 clean; Anand-Goyal 2025 unified VLM capability audit, forthcoming: 0/9 frontier VLMs pass all 5 sub-tasks).

Rebuttal density 22.4% (89 papers). Cross-ledger: Mech Interp Bill 11 ★ + Reasoning Bill 6 ★ + Scaling Laws Bill 5 ★ + this Bill 4 ★ form a four-way star-mechanism alignment (causally-faithful mechanism inheritance is domain-invariant). Capability Benchmarks Bill 19 (vendor-claim half-life 73 days) ↔ this Bill 9 (VLM-specific ~64 days). Third ledger in v0.2 wave; bill count 12 and ★ at 4/7/10 emerge from the actual VLM literature.

12Bills
3★ Empty
397Unique Papers
22.4%Rebuttal density