01The gap
The promise of VLMs runs into the messy last mile.
Picture a tour guide spotting a distant mountain and telling you its history.
For humans this is automatic — effortlessly linking subtle visual cues with
structured facts via broad knowledge. For modern Vision-Language Models,
it remains nearly impossible. The challenge is not visual grounding or knowledge
search alone, but their tight coupling.
— From the Pix2Fact paper, Section 1
01
Pixel-level grounding
Locate a small visual detail — a phone number on a sign, a logo on a tent, a book title spine-on — inside a 4K+ scene full of distractors.
02
Open-world retrieval
Using only that detail, formulate a query, search the open web, navigate to unstructured sources, and extract the right fact.
03
Their conjunction
Existing benchmarks test either one. Pix2Fact tests both at once — what real human assistance requires, and where today's models break.
Figure 6 · Traffic & Infrastructure
Highway aerial · mall behind scaffolding
(replace with 4K original)
(replace with 4K original)
A real Pix2Fact question
A tourist might naturally ask:
"The building behind the scaffolding is a mall. Which number should we call for leasing inquiries to open a clothing store there?"
Correct answer
(86 10) 6505-6688
What a model has to do: get past the foreground roads, through the scaffolding, read the sign, identify the specific mall, then navigate its website to the leasing page — a chain general search engines rarely surface directly.
Why this is hard · the 5 things a model must do, in order
1
Locate the right building
Past foreground roads & distractor buildings.
SPATIAL · GROUNDING
2
Read tiny signage
OCR the mall name from behind scaffolding.
FINE OCR
3
Identify which mall
Disambiguate among real-world entities.
ENTITY LINKING
4
Find the official site
Construct a query that surfaces the right page.
QUERY · RETRIEVAL
5
Extract the contact
Navigate to leasing page, isolate the number.
NAVIGATION · EXTRACT
