Dynamic Vocabulary Selection for Categorical Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models face inefficiencies in processing categorical features with large numbers of possible values, leading to high computational resource consumption and poor performance, especially in tasks like recommendation systems.
Innovation Solution
A system that trains machine learning models to select optimal vocabularies and embedding dimensionalities for categorical features, using a controller neural network to generate output sequences that adjust parameters based on performance metrics, optimizing prediction accuracy and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional machine learning models process all possible categorical feature values, then comprehensive coverage is achieved, but computational resource consumption increases significantly
Solution Approach 1:
The patent applies dynamics by making the vocabulary selection adaptive and dynamic rather than static. The system dynamically determines which categorical feature values to include in the vocabulary based on the specific input being processed, allowing the model to adapt its computational resources to the actual needs of each prediction task, thereby reducing unnecessary computational overhead while maintaining accuracy.
Solution Approach 2:
The patent applies local quality by allowing different parts of the categorical feature space to have different levels of detail. Instead of uniformly processing all categorical values, the system selectively activates only those vocabulary items that are relevant to the current input, creating a non-uniform, localized processing approach that reduces overall computational resource consumption while preserving necessary prediction accuracy.
2Reliability
If machine learning models use large vocabularies for categorical features, then prediction accuracy is maintained, but model complexity increases
Solution Approach 1:
The system dynamically adjusts the vocabulary size and composition based on the input data characteristics, making the model complexity adaptive rather than fixed. This allows the model to use larger vocabularies only when necessary for accurate prediction while using smaller vocabularies for simpler cases, effectively managing model complexity.
Solution Approach 2:
The patent implements local quality by applying different vocabulary sizes to different categorical features based on their specific needs. Each categorical feature can have its own optimized vocabulary subset, allowing the model to maintain high prediction accuracy for critical features while reducing complexity for less important features.
3Loss of information
If machine learning models process all categorical feature values, then complete information is utilized, but processing time increases
Solution Approach 1:
The patent applies the extraction principle by selectively removing irrelevant categorical feature values from the processing pipeline. The system extracts and activates only those vocabulary items that contain useful information for the current prediction task, discarding unnecessary values that would only increase processing time without contributing to prediction accuracy.
Solution Approach 2:
The system dynamically determines the appropriate level of information extraction based on input characteristics, allowing it to process complete information when necessary while reducing processing scope when full information is not needed, thereby optimizing the trade-off between information completeness and processing time.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining, for each of one or more categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active during processing of inputs by a machine learning model. In one aspect, a method comprises: generating a batch of output sequences, each output sequence in the batch specifying, for each of the categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active; for each output sequence in the batch, determining a performance metric of the machine learning model on a machine learning task after the machine learning model has been trained to perform the machine learning task with only the respective vocabulary of categorical feature values of each categorical feature specified by the output sequence being active.