Machine Learning Feature Evaluation via Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The cumbersome and time-consuming process of modifying machine learned models in online systems due to the large number of features used, which makes testing and inclusion of additional features inefficient.
Innovation Solution
The online system generates partitions of historical data based on the dimensions of new features, allowing for efficient evaluation of modified machine learned models by applying them to training sets, and determines whether to apply the modified model to data based on improved performance results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the online system uses one or more machine learned models to select content items for presentation to users, then the content selection accuracy and user engagement are improved, but the complexity of modifying and testing additional features in the models becomes cumbersome and time-consuming
Solution Approach 1:
The patent segments the model evaluation process into multiple parallel evaluations, where different machine learned models with different feature sets are tested simultaneously on historical data. This allows the system to evaluate multiple feature configurations without sequentially modifying and testing each one, thereby reducing the complexity and time required for model modification while maintaining content selection accuracy.
Solution Approach 2:
The patent applies machine learned models to historical data in advance to generate predicted values before actual content selection occurs. This preliminary evaluation on historical data allows the system to assess model performance and feature effectiveness without impacting real-time content selection, enabling efficient feature testing and model modification.
2Measurement precision
If the online system tests inclusion of additional features in machine learned models using conventional methods, then the model accuracy may be improved, but the testing process becomes time-consuming and inefficient
Solution Approach 1:
The patent divides the testing process into parallel evaluations of multiple models with different feature sets applied to historical data simultaneously. This segmentation allows comprehensive feature testing to occur in what would traditionally be sequential time periods, significantly reducing the time required to test additional features while maintaining model accuracy through thorough evaluation.
Solution Approach 2:
The patent creates copies of machine learned models with different feature configurations and applies these copied models to historical data for evaluation. This allows multiple feature sets to be tested without modifying the original production model, enabling efficient parallel testing that reduces time requirements while ensuring model accuracy through comparison of multiple variants.
3Productivity
If the online system modifies machine learned models to include additional features, then the model performance may be improved, but the process of modification and evaluation becomes cumbersome
Solution Approach 1:
The patent segments model modification into creation of separate model copies, each with different feature sets, rather than sequentially modifying a single model. This allows multiple performance improvements to be evaluated in parallel, making the modification process more manageable and easier to operate while achieving enhanced model performance through systematic comparison.
Solution Approach 2:
The patent introduces historical data as an intermediary medium for evaluating model modifications. Instead of directly modifying and immediately deploying changes to production models, the system uses historical data as a safe testing ground where modified models can be evaluated for performance improvements before being applied to real-time content selection, thereby simplifying the modification process.
Data Source
AI summary
An online system identifies an additional feature to evaluate for inclusion in a machine learned model. The additional feature is based on characteristics of one or more dimensions of information maintained by the online system. To generate data for evaluating the additional feature, the online system generates various partitions of stored data, where each partition includes characteristics associated with one or more dimensions on which the additional feature is based. Using values of characteristics in a partition, the online system generates values for the additional feature and includes the values of the additional feature in the partition. Values for the additional feature are generated for various partitions based on the values of characteristics in each partition. The online system combines multiple partitions that include values for the additional feature to generate a training set for evaluating a machine learned model including the additional feature.