Diversified Dataset Sampling for Imbalanced AI Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining and machine learning algorithms struggle with imbalanced data distribution, particularly in high-dimensional datasets, leading to misclassification issues and inefficiencies in feature selection, which is critical in applications like credit card fraud detection and medical diagnosis.
Innovation Solution
A method and system for selecting diversified data from a dataset by determining numerical and categorical data, forming correlations, allocating into risk groups based on predefined factors, and iteratively choosing samples using angular and euclidean distances to generate a final dataset for training AI models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional feature selection algorithms are used on high-dimensional imbalanced data, then the process is simple, but classification accuracy deteriorates due to inability to handle imbalanced distribution
Solution Approach 1:
The patent segments the feature selection process into multiple stages: initial feature selection, iterative refinement, and diversity-based sampling. Each stage handles specific aspects of the imbalanced data problem, breaking down the complex task into manageable components that collectively improve classification accuracy.
Solution Approach 2:
The patent implements a dynamic, iterative feature selection process that adapts to the data distribution. The algorithm continuously refines feature subsets across multiple iterations, adjusting selection criteria based on detected patterns in imbalanced data, rather than using a static single-pass approach.
2Measurement precision
If a large sample size is used to capture diverse data variations, then classification accuracy improves, but computational load and processing time increase
Solution Approach 1:
The patent extracts only the most relevant and diverse samples from the large dataset using angular and Euclidean distance calculations. By identifying and extracting key representative samples rather than processing the entire dataset, the method achieves high classification accuracy with reduced computational load.
Solution Approach 2:
The patent changes the sampling parameters dynamically during the iterative process, adjusting selection criteria based on the diversity metrics and class distribution. This allows the system to optimize the sample size for each iteration, achieving accuracy without excessive processing requirements.
3Reliability
If diverse data sampling is implemented to handle imbalanced distribution, then classification performance improves, but the algorithm complexity increases
Solution Approach 1:
The patent segments the diverse sampling process into distinct computational stages: distance calculation, diversity assessment, and sample selection. Each stage performs a specific function with clear inputs and outputs, making the overall complex algorithm more manageable and implementable.
Solution Approach 2:
The patent introduces intermediary computational steps that bridge the gap between simple random sampling and complex diversity-based sampling. These intermediary calculations (angular distance, Euclidean distance) serve as mediators that systematically evaluate sample diversity without requiring overly complex algorithms.
4Measurement precision
If manual tagging by compliance experts is performed on all transactions, then data quality for risk models improves, but time consumption and resource requirements increase significantly
Solution Approach 1:
The patent performs preliminary automated preprocessing and initial tagging of transaction data using the diversified sampling algorithm. This preliminary action prepares the data in advance with reasonable quality, allowing compliance experts to focus only on reviewing and validating the pre-tagged data rather than creating tags from scratch for all transactions.
Solution Approach 2:
The patent implements a self-service automated tagging system that uses the diversified sampling algorithm to generate initial tags and risk assessments. This self-service capability handles the bulk of tagging work automatically, reducing the time and resources required for manual expert intervention while maintaining acceptable data quality.
Data Source
AI summary
The present disclosure relates to a method for selecting diversified data from a dataset. The method comprises determining numerical data and categorical data from the dataset and formulating a correlation between the numerical data and categorical data. A subset of data comprising uncorrelated numerical data and the categorical data is then prepared and allocated into one or more risk groups based on predefined risk factors, including at least one of geographical location, user identity, and number of identical transactions. Samples of data from a risk group of the one or more risk groups are chosen by first selecting an initial data point of the risk group and then iteratively selecting a subsequent data point of the risk group based on angular and euclidean distances of the initial data point from the subsequent data point of the risk group. A final dataset is generated and stored for training an AI model.