Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments

Simple item page
dc.contributor.authorAnderson, P.
dc.contributor.authorWu, Q.
dc.contributor.authorTeney, D.
dc.contributor.authorBruce, J.
dc.contributor.authorJohnson, M.
dc.contributor.authorSünderhauf, N.
dc.contributor.authorReid, I.D.
dc.contributor.authorGould, S.
dc.contributor.authorHengel, A.V.D.
dc.contributor.conferenceIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (18 Jun 2018 - 23 Jun 2018 : Salt Lake City, UT)
dc.date.issued2018
dc.description.abstractA robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matter-port3D Simulator - a large-scale reinforcement learning environment based on real imagery [11]. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings - the Room-to-Room (R2R) dataset1.
dc.description.statementofresponsibilityPeter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S, underhauf, Ian Reid, Stephen Gould, Anton van den Hengel
dc.identifier.citationProceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, vol.abs/1711.07280, pp.3674-3683
dc.identifier.doi10.1109/CVPR.2018.00387
dc.identifier.isbn9781538664209
dc.identifier.issn2575-7075
dc.identifier.orcidWu, Q. [0000-0003-3631-256X]
dc.identifier.orcidTeney, D. [0000-0003-2130-6650]
dc.identifier.orcidReid, I.D. [0000-0001-7790-6423]
dc.identifier.orcidHengel, A.V.D. [0000-0003-3027-8364]
dc.identifier.urihttp://hdl.handle.net/2440/120112
dc.language.isoen
dc.publisherIEEE
dc.relation.granthttp://purl.org/au-research/grants/arc/CE140100016
dc.relation.granthttp://purl.org/au-research/grants/arc/DP160102156
dc.relation.ispartofseriesIEEE Conference on Computer Vision and Pattern Recognition
dc.rights© 2018 IEEE
dc.source.urihttps://doi.org/10.1109/cvpr.2018.00387
dc.titleVision-and-language navigation: interpreting visually-grounded navigation instructions in real environments
dc.typeConference paper
pubs.publication-statusPublished

Files

Collections

Computer Science publications