Task Learning through Visual Demonstration and Situated Dialogue

Changsong Liu¹, Joyce Y. Chai¹, Nishant Shukla², Song-Chun Zhu²

Language And Interaction Research, Michigan State University¹

Center for Vision, Cognition, Learning and Autonomy, UCLA²

Introduction

To enable effective collaborations between humans and cognitive robots, it is important for robots to continuously acquire and learn task knowledge from human partners. To address this issue, we are currently developing a framework that supports task learning through visual demonstration and natural language dialogue. One core component of this framework is the integration of language and vision that is driven by dialogue for task representations. This paper describes our ongoing effort, particularly, grounded task acquisition through joint processing of discourse and video using And-Or-Graphs (AOG).