Published December 20, 2022
| public
Report
Embodied Question Answering
Abstract
We present a new AI task -- Embodied Question Answering (EmbodiedQA) -- where an agent is spawned at a random location in a 3D environment and asked a question ("What color is the car?"). In order to answer, the agent must first intelligently navigate to explore the environment, gather information through first-person (egocentric) vision, and then answer the question ("orange"). This challenging task requires a range of AI skills -- active perception, language understanding, goal-driven navigation, commonsense reasoning, and grounding of language into actions. In this work, we develop the environments, end-to-end-trained reinforcement learning agents, and evaluation protocols for EmbodiedQA.
Additional Information
We are grateful to the developers of PyTorch [38] for building an excellent framework. We thank Yuxin Wu for help with the House3D environment. This work was funded in part by NSF CAREER awards to DB and DP, ONR YIP awards to DP and DB, ONR Grant N00014-14-1-0679 to DB, ONR Grant N00014-16-1-2713 to DP, an Allen Distinguished Investigator award to DP from the Paul G. Allen Family Foundation, Google Faculty Research Awards to DP and DB, Amazon Academic Research Awards to DP and DB, AWS in Education Research grant to DB, and NVIDIA GPU donations to DB. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.Additional details
- Eprint ID
- 118436
- Resolver ID
- CaltechAUTHORS:20221219-204833051
- NSF
- N00014-14-1-0679
- Office of Naval Research (ONR)
- N00014-16-1-2713
- Office of Naval Research (ONR)
- Paul G. Allen Family Foundation
- Google Faculty Research Award
- Amazon Web Services
- NVIDIA Corporation
- Created
-
2022-12-20Created from EPrint's datestamp field
- Updated
-
2023-06-02Created from EPrint's last_modified field