Word Embedding Layer for Categorical Data Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for transforming categorical data, such as one-hot encoding and numerical conversion, fail to effectively capture correlations between feature values and often result in increased dimensionality or false associations, lacking interpretability in machine learning models.
Innovation Solution
A word embedding layer that converts categorical input features into a two-dimensional space, using trainable coefficients to determine feature importance and relationships, with a graphical user interface for heatmap visualization to illustrate these relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If one-hot encoding is used to transform categorical data, then the data can be used in machine learning models, but the dimensionality increases significantly
Solution Approach 1:
The patent applies dimensionality reduction by transforming one-hot encoded categorical data into a lower-dimensional continuous space using learned embedding vectors. Instead of keeping categorical data in high-dimensional sparse one-hot form, the invention maps these vectors to a compact continuous representation that preserves semantic relationships while reducing dimensionality significantly.
2Quantity of substance
If numerical conversion is used to transform categorical data, then dimensionality is reduced, but false associations and correlations are created
Solution Approach 1:
The patent employs dynamic learned embeddings instead of static numerical conversions. The embedding vectors are trained alongside the machine learning model, allowing the representation to adapt and capture true correlations from the data rather than imposing artificial ordinal relationships. This dynamic adjustment enables the system to learn accurate semantic relationships without creating false associations.
3Ease of manufacture
If traditional encoding methods are used, then transformation is simple, but interpretability of feature relationships is lost
Solution Approach 1:
The patent introduces continuous embedding vectors as an intermediary representation between discrete categorical values and the machine learning model. These embedding vectors serve as a bridge that preserves interpretability by maintaining semantic relationships in a continuous space, allowing visualization and analysis of feature relationships through techniques like t-SNE or PCA while still providing the computational benefits of continuous representations.
Data Source
AI summary
A machine learning model, including: a categorical input feature, having a defined set of values; a plurality of non-categorical input features; a word embedding layer configured to convert the categorical input feature into an output in a word space having two dimensions; and a machine learning network configured to receive the output of the word embedding layer and the plurality of non-categorical input features and to produce a machine learning model output.