Lazy One-Hot Encoding for Large-Scale Categorical Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning algorithms struggle with efficiently encoding categorical features, particularly nominal features, in large-scale data processing due to computational bottlenecks caused by immediate evaluation of unique categories during training.
Innovation Solution
Implementing a lazy one-hot encoding approach using Apache Spark's relational API and DataFrame API, which delays the evaluation of unique categories until needed, allowing for optimizations and reducing the load on the driver side and space requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If immediate evaluation of unique categories is performed during training, then encoding accuracy is maintained, but computational performance deteriorates due to bottlenecks in large-scale data processing
Solution Approach 1:
The patent applies dynamic evaluation by delaying the materialization of unique categories until the encoding phase. During training, only the structure for identifying unique categories is established, while actual evaluation occurs dynamically during encoding based on the data being processed. This dynamic approach allows the system to adapt to the actual data distribution and avoid premature computation bottlenecks.
Solution Approach 2:
The patent performs preliminary setup of the encoding structure during training without completing the full evaluation of unique categories. The system prepares the framework for identifying and encoding categories, but defers the actual computation to the encoding phase. This preliminary action reduces training time while ensuring accurate encoding when needed.
2Reliability
If all unique categories are evaluated during training, then complete encoding coverage is achieved, but space requirements increase due to storing encoder state
Solution Approach 1:
The patent extracts only the essential information needed for encoding from the training data, storing minimal encoder state rather than complete category evaluations. By taking out only the necessary identifiers and mappings required for encoding, the system reduces space requirements while maintaining encoding coverage through lazy evaluation of actual category values during the encoding phase.
3Ease of manufacture
If traditional one-hot encoding is implemented, then categorical features are properly transformed for ML algorithms, but network communication increases during distributed processing
Solution Approach 1:
The patent segments the encoding process into distinct training and encoding phases with different computational requirements. By segmenting when category evaluation occurs (during encoding rather than training), the system reduces unnecessary network communication during distributed training while maintaining the ability to properly transform categorical features for machine learning algorithms.
Data Source
AI summary
A method and one or more non-transitory storage media are provided to train and implement a one-hot encoder. During a training phase, computation of an encoder state is performed by executing a set of relational statements to extract unique categories in a first training data set, associate each unique category with a unique index, and generate a one-hot encoding for each unique category. The set of relational statements are executed by a query optimization engine. Execution of the set of relational statements is postponed until a result of each relational statement is needed, and the query optimization engine implements one or more optimizations when executing the set of relational statements. During an encoding phase, a set of categorical features in a second training data set are encoded based on the encoder state to form a set of encoded categorical features.