A deep learning pipeline for product recognition on store shelves

Abstract

Recognition of grocery products in store shelves poses peculiar challenges. Firstly, the task mandates the recognition of an extremely high number of different items, in the order of several thousands for medium-small shops, with many of them featuring small inter and intra class variability. Then, available product databases usually include just one or a few studio-quality images per product (referred to herein as reference images), whilst at test time recognition is performed on pictures displaying a portion of a shelf containing several products and taken in the store by cheap cameras (referred to herein as query images). Moreover, as the items on sale in a store as well as their appearance change frequently overtime, a practical recognition system should handle seamlessly new products/packages. We developed a deep learning based pipeline to solve this task. First we deploy state of the art object detectors to obtain an initial product-agnostic item detection, then, we pursue product recognition through a similarity search between global descriptors computed on reference and cropped query images. To maximize performance, we learn an ad-hoc global descriptor by a CNN trained on reference images based on an image embedding loss. We have tested our pipeline on the standard grocery product [1] dataset and improved the currents state of the art. While computationally expensive at training time our system turn out not only accurate but also quite fast at test time.