Detect-and-describe: Joint learning framework for detection and description of objects

¹ School of Software, Shanghai Jiao Tong University, Shanghai, China
² School of Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China

^* Corresponding author: adeelz92@gmail.com

Abstract

Traditional object detection answers two questions; “what” (what the object is?) and “where” (where the object is?). “what” part of the object detection can be fine grained further i-e. “what type”, “what shape” and “what material” etc. This results in shifting of object detection task to object description paradigm. Describing object provides additional detail that enables us to understand the characteristics and attributes of the object (“plastic boat” not just boat, “glass bottle” not just bottle). This additional information can implicitly be used to gain insight about unseen objects (e.g. unknown object is “metallic”, “has wheels”), which is not possible in traditional object detection. In this paper, we present a new approach to simultaneously detect objects and infer their attributes, we call it Detectand- Describe (DaD) framework. DaD is a deep learning-based approach that extends object detection to object attribute prediction as well. We train our model on aPascal train set and evaluate our approach on aPascal test set. We achieve 97.0% in Area Under the Receiver Operating Characteristic Curve (AUC) for object attributes prediction on aPascal test set. We also show qualitative results for object attribute prediction on unseen objects, which demonstrate the effectiveness of our approach for describing unknown objects.