Unbalanced Sample Data Preprocessing via Characteristic Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional oversampling methods, such as the SMOTE algorithm, fail to effectively balance unbalanced data sets, leading to inadequate new data generation that does not meet actual data standards, resulting in persistent sample imbalance issues, especially in datasets with small sample sizes and significant type imbalances.
Innovation Solution
An unbalanced sample data preprocessing method that classifies initial data into first-class and second-class sample sets based on a preset threshold, extracts K sample points, analyzes their characteristics, generates new data labels, and adjusts the amount of new data to balance the ratio between first-class and second-class sample sets, thereby increasing the number of first-class samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional oversampling algorithms (e.g., SMOTE) are used to generate new data, then the sample size of minority classes is increased, but the generated data does not meet actual data standards and sample imbalance persists
Solution Approach 1:
The patent applies preliminary action by performing characteristic analysis on K first sample points before generating new data. The system analyzes characteristics of selected sample points to determine new data characteristics, ensuring that generated data meets actual data standards. This preliminary analysis step prevents generation of low-quality data while maintaining the desired sample size increase.
Solution Approach 2:
The patent changes parameters by adjusting the amount of new data generated based on the ratio between first-class and second-class sample sets. The system dynamically controls the number of new samples to generate, modifying this parameter until the ratio between minority and majority classes reaches a balanced state, thereby solving both quantity and quality issues simultaneously.
2Quantity of substance
If the amount of new data is increased to balance sample sets, then the ratio between first-class and second-class sample sets improves, but data preprocessing complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the data processing into distinct stages: extracting K first sample points, analyzing their characteristics to determine new data characteristics, generating new data labels, and adjusting the amount of new data. This segmented approach simplifies the overall preprocessing complexity by breaking down the complex task of balanced data generation into manageable, sequential steps.
3Productivity
If new data is generated without analyzing sample point characteristics, then the processing speed is faster, but the generated data does not meet actual data standards
Solution Approach 1:
The patent applies partial action by analyzing characteristics of only K first sample points rather than all sample points in the dataset. This selective analysis of a subset of representative samples maintains processing speed while ensuring that the generated new data meets actual data standards. The system extracts sufficient information from a partial set to guide the generation of high-quality new data.
Data Source
AI summary
Provided is an unbalanced sample data preprocessing method, which includes: a data acquisition request is received and initial data is acquired according to the data acquisition request, and the initial data is classified according to a preset classification rule to obtain first-class sample sets and second-class sample sets; characteristics of K first sample points extracted are analyzed to obtain a new data characteristic of the first-class sample sets; a new data label of the first-class sample sets is generated according to a first label corresponding to the first-class sample sets; a ratio between the number of first-class sample sets and the number of second-class sample sets is calculated; and new data of the first-class sample sets is generated according to the new data characteristic and the new data label, and the amount of new data is adjusted according to the ratio to increase the number of first-class sample sets.