AI Learning Data Preprocessing Using Generalization Indices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for preprocessing learning data for artificial intelligence models require large amounts of high-quality data, which is resource-intensive and difficult to secure, especially for efficient learning with a small amount of data.
Innovation Solution
The method involves calculating generalization indices based on the popularity or reliability of the learning data, preprocessing the data to prioritize high generalization index data, and classifying it into groups for efficient learning, even with a small dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of learning data are used to improve model performance, then the performance of artificial intelligence models is enhanced, but the resource consumption and difficulty to secure high-quality data increases
Solution Approach 1:
The patent changes the parameter of data selection criteria from quantity-based to quality-based by introducing generalization indices. Instead of using large amounts of data, the system selects data with high generalization indices (indicating broad applicability and reliability) to train models, thereby achieving good performance with smaller datasets. This transforms the data utilization approach from volume-driven to quality-driven.
Solution Approach 2:
The patent extracts only the most relevant and generalizable data from the entire learning dataset. By calculating generalization indices for each data item and selecting only those with indices above a threshold, the system extracts a subset of high-quality data that is sufficient for model training, eliminating the need to process and store the entire large dataset.
2Reliability
If high-quality learning data is selected to improve model performance, then the reliability of learned models is enhanced, but the complexity of data preprocessing and selection increases
Solution Approach 1:
The patent performs preliminary action by pre-calculating generalization indices for data items before model training. The generalization index calculation is performed in advance using features like data source reliability, recency, and popularity metrics. This preliminary processing organizes data in a way that simplifies subsequent selection and training processes, as the system can directly use pre-computed indices rather than performing complex analysis during training.
Data Source
AI summary
Proposed is a method for preprocessing learning data for an artificial intelligence model using generalization indices. The method may include collecting learning data for learning an artificial intelligence model, calculating generalization indices of the collected learning data, and preprocessing the collected learning data based on the calculated generalization indices. The method may include generating a plurality of groups, the plurality of groups respectively corresponding to generalization ranges determined based on the calculated generalization indices, and assigning the collected learning data to one of the plurality of groups based on the calculated generalization indices and generating a learning data set used for learning the artificial intelligence model by selecting at least one data from each group. A number of collected learning data selected from each group may be determined based on generalization indices of all of collected learning data assigned to the each group.