Watch The Video
Zeyad Deeb

By: Zeyad Deeb on September 13th, 2018

Print/Save as PDF

In the (AI)sle™ - Part I

(AI)sle  |  Artificial Intelligence  |  Technology

How Artificial Intelligence can make a difference at the grocery store

With the advent of veganism, the keto diet, and the rise of severe food allergies in kids, it's no wonder that retailers and brands alike are looking for ways to stand out in the chaos that is grocery store aisles. Consumers have more choices than ever, but the complexities of modern packaging and labeling can confuse the decision-making process. That's where we come in.

1 in 7 consumers say they have special diet [Source: Food Insight] and about 4% of adults [Source: FARE] in the US have some form of food allergy. The problem is there's no hard and fast rules governing the organization of product labels. Put a box of crackers next to a bottle of soda and you'll see some similarities- the number of calories, an ingredients list, and a large title. Other than that, you have to really look for the details that may mean most to you: fiber, saturated fats, or allergens. Labels are becoming increasingly complicated and the problem is there are no hard and fast rules yet governing these new and ever-changing product labels. 

Machine learning is shining a light on the previously dim-lit world of food labels. By using image recognition, object localization, and natural language processing to find patterns in food labels, we can take the grunt work out of how to best organize a particular product or set of products at the store.

Part I - A product package is worth a thousand words: Utilizing Image Classification @Label Insight

You've heard it before: color has an impact on our mood and our appetite. Consumers have adjusted to years of evolving marketing tactics designed to draw shopper attention, build credibility, and communicate value. One common problem for consumers is that product names and designs can change even though the product ingredients remain the same. This is commonly referred to as a "rebrand." Our data enables retailers and consumers to effectively sift through what is similar/different between products. We train our algorithms to provide a high degree of accuracy and consistency.

In the last few years, the field of machine learning has made tremendous progress in addressing these difficult problems. In particular, deep Convolutional Neural Networks can achieve reasonable performance on hard visual recognition tasks. Sometimes these algorithms are so sophisticated that they beat humans, who have traditionally maintained a natural advantage in image recognition.

Deep Learning & Convolutional Neural Networks

Recently many Neural Network architectures have emerged that enable numerous use cases, the most widely known among them is Convolutional Neural Network (CNN). CNNs were inspired by research done on the visual cortex of mammals to understand how they perceive the world using a layered architecture of neurons in the brain. Think of this model of the visual cortex as groups of neurons designed specifically to recognize different shapes. Each group of neurons fires at the sight of a particular shape or outline and communicates with each other to develop a holistic understanding of the perceived object.


[Source: Wikipedia]

At Label Insight we strive to utilize different Neural Network architectures to help us provide insight into traditional data. There are so many different ways to classify a product, from aisle location to storage instructions (refrigerated or non-perishable), to diet-friendly orientation. Our unique dataset allows us to train Deep Learning algorithms so that we can enrich the standard information captured on a package with a high degree of accuracy in a short amount of time. Essentially, our wealth of knowledge helps cut the time needed to recognize even new images and accurately classify them, just as if you were to expose a person to similar types of information multiple times - you learn, and so do our AIs.

The next anecdote exemplifies a classic retailer problem. All grocery stores are organized differently, so what can we as a brand assume as to whether or not a prepared healthy food can be placed next to prepared less-healthy food? While there is no "right" answer to this problem, we can utilize Big Data to understand relationships between products and get a good feel for if the answer is "Yes" or "No," given what we already know about that particular retailer's product offerings. At Label Insight we put accuracy and transparency above all else, whether you're a local grocer or a multi-national wholesaler.

How we use CNN: Product Classification, Hierarchy, and Taxonomy

Recently, we started doing research into finding the best taxonomy structure for a given set of products. For instance, a typical neural network can provide you with one or more prediction labels (Soup, Chunky Soup, Canned Soup), but the product hierarchy (ex: canned soup is a subset of prepared foods) is still key to grouping similar products. The FDA provides a general taxonomy for regulatory and guideline purposes, but savvy retailers will want to curate unique shopping experiences different than the standard option.

A CNN model trained on our data can accurately predict the top five classes for a specific product:


Seeing that this is just a good first step, we also trained additional models to explain the taxonomy, classification and hierarchy as well. We invested resources into our dietitian and food specialists to structure our product hierarchy in a way that can meet retailers' needs. We are able to build semantic taxonomy by merely utilizing a product image. Simply put, we feed the algorithm images of related products, and the algorithm tells us the given product hierarchy (food> prepared food> canned foods> soup). We have approximately ~2000 unique product categories that can be mapped to any classification retailers might need to better understand their products. The process is customizable based on a brand or store's specific needs.

We build our data products on cutting-edge research while adjusting coefficients and loss functions to be unique for our use case. Most recently we have been exploring the embedding loss function utilized to build the product hierarchy, one of them being:

$$\sum_{(u, v)\in D}{d(u, v) + max(0, m − d(u', v'))}$$

Where $D$ is the Kullback-Leibler (KL) Divergence [Source]. Our trained neural networks understand the deep underlying structures of almost all product sets, and therefore we classify everything from marketing statements on the packages all the way to ingredients and nutrition information. The neural network works for both descriptive statements ("chunky") and more factual product data ("12g") alike.


It is now more important than ever for brands and retailers to understand representations of product data from both text and images so they can build relationships between the items they sell, which leads to the development of logical and effective store aisles. Curious to learn more about using Label Insight’s data? We’re helping CPG brands and retailers tackle their toughest product data challenges. Let’s connect!

About Zeyad Deeb

Data Science at Label Insight