The robots are coming for this job at least
One of the main task of the Criteo CAMLET team(Catalog & Applied Machine Learning — Enrichment & Text) is to enrich each day over 2 billion products to create one of the largest e-commerce catalogs worldwide: 25+ billion products. These products are provided by our e-commerce partners and initially possess some information that we standardize, complete and enhance using machine learning. One of those enrichment steps consists in extracting product attributes based on its title and description — For example, finding the color of a dress, the genre of a video game, or the size of a TV.
Generating training data for those models has historically been a costly and lengthy task, but the latest advances in open source zero-shot models are enabling new approaches to create this data at a fraction of the cost.
Criteo has several tens of thousands of e-commerce partners, providing our custom recommendation engine with a catalog of more than 25 billions products to choose from. Unfortunately, the data in our partners’ catalogs has been created using a myriad of custom rules and taxonomy, each defined by our partners. To ensure the relevance of the online ads we propose to internet users and of the ad campaigns of our e-commerce partners, we need both to standardize and enhance this set of heterogeneous catalogs through a succession of enrichments. Each enrichment is performed by taking as input some subset of the data provided by our e-commerce partners as well as previous enrichments outputs and feeding them into a dedicated machine learning model.
Given the large variety in data that our models need to handle — tens of thousands of e-commerce partners providing tens of billions of products with thousands of categories and over a dozen languages — each of those models requires a vast quantity of training data that need to be annotated. Like most companies faced with this challenge, Criteo relies on third parties offering annotation services. Once a group of annotators has been assigned to a task, they need to be trained on what is expected for this specific enrichment and set of data before manually adding the expected output to tens of thousands of sample products.This process is very costly and lengthy but, worse of all, provide very noisy data with both false positives and false negatives aplenty. Indeed, the annotations process is often ambiguous, and it is both hard to ensure all annotators provide consistent outputs and that they avoid making mistakes due to the repetitive nature of their task.
While it is possible to somewhat mitigate these issues by i.e. relying on double-annotations and reviewing ourselves part of the data, this only increase the cost, both in time and money, required to generate the training data of our models. Moreover, the value of this data tends to deprecate over time, both because consumer preferences evolve over time and because the specifications and needs of our models are modified to better fit our clients strategies. Training data thus need to be refreshed on a regular basis to prevent the performance of our models to deteriorate over time.
The difficult and costly nature of training data generation has long been accepted as a cost of doing business in Machine Learning. However, recent advances in zero-shot models open the door to the generation of high-quality synthetic data — training data generated without any costly human annotations.
The most impressive models in Natural Language Processing over the past few years have mainly been based on the transformers architecture. Transformers models rely on a clever attention mechanism that allows them to properly represent how much each word in a sentence relates to any other. Given enough parameters, such transformers models are able to learn the structure of human writing and much of the world relationships embedded in the written corpus of the Internet. Training such models from scratch is a very costly endeavor, and out of the reach of most groups interested in Natural Language Processing (NLP). However, they can be trained in a self-supervised manner on vast amount of unstructured data downloaded from the Internet by masking some words in a given sentence and asking the model to predict those words. Transformers pretrained by AI research groups with vast resources are often open-sourced and made available to the public. These models can then be used to solve many unrelated problems by using transfer learning: First, remove the top layer of the model — its head — and replace it with a layer dedicated to a new NLP task, such as the classification of a product description into a product category. Then, by fine-tuning the model on a few thousands annotated examples for the new task, it is possible to achieve state of the art results on a wide range of downstream tasks.
In the past couple of years, transformers models with significantly more parameters have been trained and started exhibiting remarkable zero-shot capabilities: instead of requiring the fine-tuning of the model on annotated examples of the downstream task, the task could be described in natural language as part of the input fed to the model, and the model would be able to interpret the task and directly provide the desired output. The first model to demonstrate such capabilities to a significant degree was GPT-3 from OpenAI [1]. However, GPT-3 is not open-sourced, and even if it was, its 175 billions parameters make it almost impossible to use for most organizations.
At the end of 2021, Hugging Face released a significantly lighter model, T0PP [2], claiming performance close to GPT-3 on a wide range of downstream tasks despite using only 11 billions parameters. This was achieved by converting most open source NLP datasets into natural language prompts and using those as training data for their model. The resulting model is much less generic than GPT-3, and trying to use the model for free form text generation, for example, produces much less impressive results. However, T0PP performs very well on most standard NLP tasks and is able to generalize well beyond it’s training dataset without any fine-tuning.
By carefully designing prompts for the model, it is then possible to enrich thousands of examples in a matter of hours, and then to use this data as a training set to fine-tune other algorithms.
But wait: if T0PP can perform such tasks directly, why couldn’t we use it to replace all other models? Unfortunately, T0PP and its 11 billions parameters remain far larger than standard transformer models such as Bert and its 250 millions parameters. The model can only be loaded into the memory of the latest industry-grade GPUs such as the A100 from NVIDIA. Since we lacked readily available GPUs of such calibers, we had to settle for CPU inferences, which took about 5 seconds apiece. While this is good enough to generate a few thousands to tens of thousand of examples over a few hours or days, it is at least 4 orders of magnitude slower than we would need to enrich all our daily products. We will just have to settle for generating high-quality training data at a fraction of the cost we’re used to for now.
Given the gerenic nature of T0PP, such an approach is likely to be suitable for a wide range of NLP tasks and enrichments. We chose to focus initially on generating data for the Product ATtributes Extraction (PATE) enrichment, for which generating high quality training data had been a pain point at Criteo during the past year. Given a category of products, the goal of PATE is to find attribute modalities in the title and description of the product. For example, we may want to find occurrences of colors in dress descriptions, or of resolution in TV descriptions. For each category of products, a list of attributes is defined beforehand (i.e. color, dress length, and material for a dress), and the goal of our models is to classify the product to the corresponding modalities.
Starting from a raw dataset of products titles and descriptions, we created a notebook that generates a prompt for each attribute based on a template and pass it into a T0PP model for inference. The resulting raw predictions are then saved to disk for future reference.
Once all predictions have been executed, a second notebook loads back the results to perform the following post-processing steps to clean up the data:
- Discard all T0PP predictions that are not found in the corresponding product text. T0PP is trying to predict the most likely answer to our questions, which can allow it to find useful data that is not present in the text. For example, asked to find the manufacturer of a Focus car, T0PP can infer that it is Ford. However, asked to find the color of a dress that possess no such information, T0PP will provide a likely answer, such as “Blue”, regardless of its accuracy. Discarding all creative answers allow us to significanly reduce the amount of false positives in our dataset.
- Group all T0PP predictions for a given attribute by frequency and discard all predictions below a threshold that is manually defined by quickly looking at the sorted list. T0PP can have catastrophic failures with nonsense answers, and predictions that occur very rarely are often irrelevant. On top of that, keeping hundreds of modalities often provide very little added value, and it is often best to limit possible values to a few most relevant dozen modalities.
- Manually go over the resulting list to remove unnecessary modalities, which are usually either plain wrong answers, or subcategories of another modality (for example, given a console “PlayStation 4 Pro”, it is usually better to only keep the “PlayStation 4” part, which will be a more frequent modality anyway)
The resulting enriched dataset is then saved to disk in a separate file and ready to be used as training data.
So far, we only looked at a few examples while refining our initial prompts. To validate the usefulness of the data, we sent the raw initial dataset of video game products for annotations using our regular pipeline leveraging human annotators and compared it with the post-processed dataset created using T0PP. We analyzed ourselves each difference in prediction over 600 examples, classifying them in either T0PP errors or human errors. We found that not only did T0PP predictions maintain over 95% accuracy for each attribute, the overall accuracy of T0PP predictions was even higher than the accuracy of human annotations. T0PP made some errors due to missing some implied context, such as predicting that the franchise of a game was “SimCity” when the product description contained the sentence “Create your own SimCity style city”. However, such errors were much less frequent than errors made by humans due to inconsistency, missed clicks, or simply lack of domain knowledge.
Given the impressive performance of the approach, we have already started scaling the process to other categories, offering us for the first time a realistic path to scaling the PATE project to a large number of new categories of products. We are also looking at many other tasks that could similarly benefit from it, such as generating data for the model responsible to predict which category each product belongs to.
While the results of this new approach are extremely promising and will allow us to scale numerous products far beyond what we thought possible, it is critical when working with large language models to be mindful of potential biases stemming from the data corpus used for training them. We can here salute HuggingFace for providing an analysis on bias and fairness for each of their model. T0PP is no exception, and biases embedded either in the internet data used to train the original T5 [3] or in the public datasets Hugging Face used to train T5 into T0PP impact the resulting model. Human supervision of the data extracted remains a necessary step to monitor and reduce such biases in our own models.
Thanks for reading!