Multitask Neural Network Entity Representation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-dimensional and sparse TF-IDF bag-of-words representations for entities, such as customers or products, are difficult to use in machine learning due to slow training times, overfitting, and the need for manual data scientist efforts, making it inefficient to develop effective representations for ML models.
Innovation Solution
A multitask neural network (MNN) is trained to generate low-dimensional, dense entity representations by combining an encoder with decoders, using recurrent neural networks and Long Short-Term Memory units, to capture attribute signals relevant for downstream ML tasks, reducing the need for manual intervention and improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If high-dimensional TF-IDF bag-of-words representations are used to represent entities, then the representation captures detailed textual information, but the training time increases and the model becomes computationally expensive
Solution Approach 1:
The patent segments the high-dimensional TF-IDF representation into two parts: (1) a fixed-length contextual embedding vector that captures semantic meaning, and (2) a variable-length sequence of token embeddings. This segmentation allows the model to process information more efficiently while preserving both semantic context and detailed textual information.
Solution Approach 2:
The patent transforms the high-dimensional sparse TF-IDF representation into a lower-dimensional dense representation by introducing a new dimensional space through neural network embeddings. The contextual embedding projects the original high-dimensional space into a compact fixed-length vector that preserves semantic relationships, effectively changing the dimensionality from thousands of sparse features to a manageable dense vector.
2Loss of information
If high-dimensional TF-IDF bag-of-words representations are used, then comprehensive entity features are captured, but the model is prone to overfitting
Solution Approach 1:
The patent changes the parameters of the representation by transforming the discrete, high-dimensional TF-IDF vectors into continuous, lower-dimensional embedding vectors. This parameter transformation reduces the number of parameters the model needs to learn, thereby reducing overfitting risk while maintaining the ability to capture comprehensive entity features through the learned embedding space.
Solution Approach 2:
The contextual embedding serves multiple functions simultaneously: it captures semantic meaning, reduces dimensionality, prevents overfitting, and provides a unified representation that works across different tasks. This multi-functionality allows the same representation mechanism to address both feature capture and overfitting prevention.
3Adaptability or versatility
If manual tasks are performed by data scientists to develop entity representations, then customized representations can be created, but the development process becomes inefficient and resource-intensive
Solution Approach 1:
The patent implements self-service by enabling the system to automatically learn and generate entity representations through the neural network model. Instead of requiring data scientists to manually engineer features, the model automatically learns optimal representations from the input data, thereby maintaining adaptability while dramatically improving development efficiency and reducing resource requirements.
Solution Approach 2:
The patent replaces the mechanical process of manual feature engineering by data scientists with an automated neural network-based learning system. This substitution eliminates the need for manual tasks while preserving the ability to create customized representations tailored to specific tasks through the learned embeddings.
4Loss of information
If high-dimensional representations are used, then detailed entity information is preserved, but computing semantically meaningful nearest neighbors becomes difficult
Solution Approach 1:
The patent resolves the nearest neighbor computation difficulty by transforming the high-dimensional sparse TF-IDF space into a lower-dimensional dense embedding space. In this new dimensional space, semantic relationships are preserved and can be efficiently measured using standard distance metrics, making nearest neighbor computation tractable while maintaining entity information through the learned embeddings.
Data Source
AI summary
Systems and methods are disclosed to implement a neural network training system to train a multitask neural network (MNN) to generate a low-dimensional entity representation based on a sequence of events associated with the entity. In embodiments, an encoder is combined with a group of decoders to form a MNN to perform different machine learning tasks on entities. During training, the encoder takes a sequence of events in and generates a low-dimensional representation of the entity. The decoders then take the representation and perform different tasks to predict various attributes of the entity. As the MNN is trained to perform the different tasks, the encoder is also trained to generate entity representations that capture different attribute signals of the entities. The trained encoder may then be used to generate semantically meaningful entity representations for use with other machine learning systems.