Home | Publications | MSG+21

Scene Graph Generation for Better Image Captioning?

MCML Authors

Vladimir Golkov

Dr.

→ Group Daniel Cremers
Computer Vision & Artificial Intelligence

Hinrich Schütze

Prof. Dr.

Principal Investigator

Computational Linguistics

Daniel Cremers

Prof. Dr.

Director

Computer Vision & Artificial Intelligence

Abstract

We investigate the incorporation of visual relationships into the task of supervised image caption generation by proposing a model that leverages detected objects and auto-generated visual relationships to describe images in natural language. To do so, we first generate a scene graph from raw image pixels by identifying individual objects and visual relationships between them. This scene graph then serves as input to our graph-to-text model, which generates the final caption. In contrast to previous approaches, our model thus explicitly models the detection of objects and visual relationships in the image. For our experiments we construct a new dataset from the intersection of Visual Genome and MS COCO, consisting of images with both a corresponding gold scene graph and human-authored caption. Our results show that our methods outperform existing state-of-the-art end-to-end models that generate image descriptions directly from raw input pixels when compared in terms of the BLEU and METEOR evaluation metrics.

misc