SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Date
2023
Authors
Rana, K.
Haviland, J.
Garg, S.
Abou-Chakra, J.
Reid, I.
Sünderhauf, N.
Editors
Tan, J.
Toussaint, M.
Darvish, K.
Toussaint, M.
Darvish, K.
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
Proceedings of The 7th Conference on Robot Learning, as published in Proceedings of Machine Learning Research, 2023 / Tan, J., Toussaint, M., Darvish, K. (ed./s), vol.229, pp.23-72
Statement of Responsibility
Krishan Ranay, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, Niko Sünderhauf
Conference Name
Conference on Robot Learning (CoRL) (6 Nov 2023 - 9 Nov 2023 : Atlanta, Georgia, USA)
Abstract
Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic search for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an iterative replanning pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page sayplan.github.io.
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
Authors retain copyright.