Pyramid Embedding Architecture for Machine Learning Feature Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional embedding techniques using deep learning architectures lack information from recursive analysis of outputs across layers, leading to reduced predictive capabilities and inefficient resource utilization due to fixed embedding lengths for all features, failing to encode complexity of higher-cardinality attributes.
Innovation Solution
A pyramid architecture for generating embeddings, where each embedding layer's input is the concatenation of outputs from all previous layers, and input embeddings are scaled by feature cardinality, enabling wide and deep learning across embedded representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional deep learning architectures use fixed embedding lengths for all features, then the model structure is simple and easy to implement, but the predictive capability is reduced and resource utilization is inefficient
Solution Approach 1:
The patent applies local quality by making embedding layer outputs adaptable to different features based on their cardinality. Each feature receives a customized embedding dimension proportional to its cardinality, allowing the model to allocate computational resources efficiently while maintaining high predictive capability for complex features without unnecessarily complicating the overall structure.
Solution Approach 2:
The patent implements dynamics by transitioning from fixed embedding lengths to dynamic, feature-specific embedding dimensions. The embedding size for each feature is determined by its cardinality, allowing the model structure to adapt flexibly to different input requirements while maintaining computational efficiency and avoiding unnecessary complexity.
2Loss of information
If conventional embedding techniques use fixed embedding lengths, then the implementation is straightforward, but higher-cardinality attributes cannot be properly encoded
Solution Approach 1:
The patent applies parameter changes by making embedding dimensions variable based on feature cardinality. Instead of using a fixed embedding length for all features, the system dynamically adjusts the embedding dimension parameter to be proportional to each feature's cardinality, enabling proper encoding of high-cardinality attributes while maintaining reasonable implementation complexity through a systematic approach.
3Reliability
If pyramid architecture with concatenated outputs from all previous layers is used, then recursive analysis information is preserved improving predictive accuracy, but computational resources increase
Solution Approach 1:
The patent applies segmentation by dividing the embedding process into multiple layers where each layer processes and transforms the concatenated outputs from previous layers. This segmentation allows recursive analysis information to be preserved and progressively refined across layers, improving predictive accuracy while distributing computational load efficiently across the network architecture.
Solution Approach 2:
The patent implements dimensionality change by organizing embedding layers in a hierarchical pyramid structure where each layer operates in a higher-dimensional space formed by concatenating previous layer outputs. This dimensional progression enables comprehensive recursive analysis while managing computational resources through structured dimensionality increases rather than uniform expansion across all features.
Data Source
AI summary
The disclosed embodiments provide a system for processing data. During operation, the system performs processing related to a first set of features for a first entity using a first series of embedding layers, wherein the processing includes applying each embedding layer in the first series of embedding layers to a concatenation of all outputs of one or more layers preceding the embedding layer. Next, the system obtains a first embedding as an output of a first final layer in the first series of embedding layers. The system then outputs the first embedding for use by a machine learning model.