A dataset & benchmark for human–robot interaction
HEIR — Harness Egocentric Intent for Human–Robot Interactions
Connecting what people see, look at, and say with how robots navigate, manipulate, and collaborate in everyday spaces.
Recorded interaction
Continuous sessions
Request-level episodes
Execution subtasks
Participants
01 / The missing perspective
A request is more than a sentence. Its meaning depends on a person's attention, activity, and the interactions that came before it.
HEIR preserves both sides of the interaction: human egocentric video, measured gaze, and speech alongside robot observations and teleoperated navigation–manipulation trajectories.
02 / In everyday spaces
Two recorded interactions bring human intent and mobile robot execution into the same view.
“Please bring that object over for me.”
A situated request leads into object retrieval, movement across the room, and handover. The multi-view recording shows the interaction from both human and robot perspectives.
Edited clip · 1:37 · Speed changes are marked in the video.
“Let's smooth out the bedsheet together.”
A person and a dual-arm robot grasp and adjust the same bedsheet, illustrating the physical context surrounding a collaborative household request.
Edited clip · 0:44 · Speed changes are marked in the video.
Qualitative interaction examples. These clips illustrate the tasks; quantitative success rates are reported separately below.
03 / The HEIR dataset
Requests unfold within sessions, so later episodes can refer to objects and states established earlier.
A continuous household interaction
One natural human request
Navigation or manipulation
Pupil Labs Neon eye-tracking glasses
Galaxea R1 Lite mobile dual-arm platform
Human annotations with VLM-assisted refinement
Training / testing sessions
Split at the session level
Average session duration
Episodes per session, on average
Subtasks per episode, on average
04 / A hierarchical benchmark
Evaluate each component, then examine what happens when they work together in the real world.
WHAT DOES THE HUMAN MEAN?
Use speech, measured gaze, human and robot views, and interaction history to infer a grounded task description, target objects, destinations, and a nominal execution plan.
Metrics: Slot F1 · Location accuracy · Plan exact match · Entity–visibility F1
STRUCTURED INTENT · ILLUSTRATIVE
The planner chooses Continue, Switch, or Done from current observations and execution history.
05 / What the benchmark reveals
Results from the manuscript. Offline component scores and real-robot success measure different parts of the problem.
Slot F1 rises from 60.38 with text only to 77.80 with human and robot views plus measured gaze.
Gemma-4-31B · no interaction history · Table 2Fine-tuned InternVLA-N1 reaches 90.69% offline action accuracy, compared with 49.08% for its original checkpoint.
Offline navigation action prediction · Table 5The integrated system succeeds on 5 of 27 episodes, revealing error accumulation across long-horizon interactions.
Real-robot episode success · Section 5.5EXPLORE THE RESULTS
| Input condition | Slot F1 ↑ | Location ↑ | Plan EM ↑ | RV F1 ↑ | BERTScore ↑ |
|---|---|---|---|---|---|
| Text only | 60.38 | 19.42 | 33.24 | N/A | 92.96 |
| Robot view | 71.39 | 55.38 | 64.32 | 61.63 | 93.19 |
| Human + robot | 75.74 | 72.18 | 73.51 | 60.94 | 93.27 |
| Human + robot + center prior | 75.24 | 70.08 | 78.92 | 64.10 | 93.23 |
| Human + robot + gaze | 77.80 | 77.95 | 79.19 | 63.95 | 93.23 |
Table 2. Measured gaze improves grounding beyond a center prior; the center-prior condition is slightly higher on entity–visibility F1.
THE REAL-ROBOT GAP
End-to-end success requires intent, planning, navigation, and manipulation to stay aligned throughout the episode.
06 / Research resources
Explore the benchmark data and task definitions behind HEIR. The paper and code are coming soon.
The manuscript describes planned releases of annotations, processed evaluation instances, prompts, and evaluation code. Sensitive raw recordings are intended for controlled access after privacy review.