A vision-and-language navigation benchmark
Within Three Meters.
Is That Success?
NavProbe evaluates how well navigation agents follow instructions, from the route they take to where they stop.
- 500
- Whole instructions
- 1,702
- Whole Judgements
- 885
- Eligible Micro tasks
- 11
- Unseen scenes
Benchmark
How NavProbe scores a run
An agent can stop within three meters of the goal and still miss a doorway the instruction asked it to cross. NavProbe reports conventional navigation scores alongside the results for each requirement.
- Whole
- The original instruction and starting pose.
- Judgement
- An executable check for a grounded requirement, such as crossing the specified opening or stopping by the correct object.
- Micro
- A self-contained fragment of the instruction, evaluated from a validated reset pose using the matching Judgements.
What did the agent actually do?
Four real runs, shown on schematic floor plans. Each diagram highlights the specific requirement that explains the score.
- Endpoint success (SR)
- Path similarity (nDTW)
- NavProbe outcome
Model replays
Browse 325 scored runs across 13 models. Each replay shows the agent’s camera view, trajectory, and Judgement map.
Replays rendered in Habitat using Matterport3D scenes for non-commercial academic research.