This work proposes Recurrent Neural Network (RNN) models to predict structured image situations - actions and noun entities fulfilling semantic roles related to the action. We transform the task of noun entity prediction to that of sequential prediction and use RNNs in contrast to prior work that uses Conditional Random Fields (CRFs). By using an action prediction network and a separate network with an RNN for noun prediction, we obtain state-of-the-art accuracy on the challenging recent imSitu dataset, beating CRF-based models, including ones trained with additional data. Further, we show that specialized features learned from situation prediction can be transferred to the task of image captioning to more accurately describe human-object interactions.
Submitted 18 Mar 2017 to Computer Vision and Pattern Recognition
Published 21 Mar 2017