Explicit knowledge-based reasoning for visual question answering

Abstract

We describe a method for visual question answering which is capable of reasoning about an image on the basis of information extracted from a largescale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can explain the reasoning by which it developed its answer. It is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in testing. We also provide a dataset and a protocol by which to evaluate general visual question answering methods.