Inferring Goals with Gaze during Teleoperated Manipulation

Reuben M. Aronson, Nadia Almutlak, Henny Admoni

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations


Assistive robot manipulators help people with upper motor impairments perform tasks by themselves. However, teleoperating a robot to perform complex tasks is difficult. Shared control algorithms make this easier: these algorithms predict the user's goal, autonomously generate a plan to accomplish the goal, and fuse that plan with the user's input. To accurately predict the user's goal, these algorithms typically use the user's input command (e.g., joystick input) directly. We use another sensing modality: the user's natural eye gaze behavior, which is highly task-relevant and informative early in the task. We develop an algorithm using hidden Markov models to infer goals from natural eye gaze behavior that appears while users are teleoperating a robot. We show that gaze-based predictions outperform goal prediction based on the control input and that our sequence model improves the prediction quality relative to gaze-based aggregate models.
Original languageEnglish (US)
Title of host publication2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Number of pages8
ISBN (Print)9781665417143
StatePublished - Dec 16 2021
Externally publishedYes

Bibliographical note

KAUST Repository Item: Exported on 2022-06-22
Acknowledgements: This work was supported by the Paralyzed Veterans of America, the National Science Foundation (IIS-1755823 and IIS-1943072), and the Tang Family Foundation Innovation Fund. Nadia AlMutlak was supported by the King Abdullah University of Science and Technology.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.


Dive into the research topics of 'Inferring Goals with Gaze during Teleoperated Manipulation'. Together they form a unique fingerprint.

Cite this