Bagging Ensemble Classifiers for Imbalanced Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large imbalanced datasets pose challenges for machine learning classification models, making it difficult to identify minority classes and requiring significant computational resources, which is impractical for personal computing devices due to memory constraints.
Innovation Solution
The system generates an ensemble of machine learning base models by separating the dataset into majority and minority cases, training each model with a chunk containing all minority cases and a subset of majority cases, and validating them using a testing dataset, allowing for effective classification on local devices without the need for expensive cloud computing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all data from a large imbalanced dataset is used for training, then classification accuracy is improved, but computational resources and memory requirements increase significantly
Solution Approach 1:
The patent divides the large imbalanced dataset into multiple smaller chunks, where each chunk contains a representative subset of minority cases and majority cases. Multiple base models are trained on different chunks, and their predictions are aggregated through voting. This segmentation allows training on personal devices with limited memory while maintaining classification accuracy through ensemble aggregation.
2Reliability
If a large imbalanced dataset is used for training, then the model can recognize all classes, but minority classes are still difficult to identify due to data imbalance
Solution Approach 1:
The patent ensures that each training chunk has a different composition of minority and majority cases, creating local variations in data quality across chunks. Some chunks may have higher concentrations of minority cases while others have more majority cases. This local quality variation allows the ensemble to capture different aspects of the imbalanced data distribution, improving minority class recognition without requiring the entire large dataset to be loaded into memory simultaneously.
3Ease of manufacture
If training is performed on a personal computing device, then cost is reduced, but processing power and memory are insufficient for large datasets
Solution Approach 1:
The patent trains multiple base models on partial subsets (chunks) of the complete dataset rather than requiring all data to be processed simultaneously. Each base model is trained on a manageable chunk that fits within personal device memory constraints. The ensemble of multiple partially-trained models achieves performance comparable to a single model trained on the complete dataset, enabling cost-effective training on personal computing devices.
Data Source
AI summary
Disclosed embodiments may include a method for bagging ensemble classifiers for imbalanced big data. The system may receive user input comprising a number of machine learning base models to generate. The system may generate the machine learning base models based on the user input. Iteratively for each machine learning base model of the machine learning base models until all machine learning base models are trained, the system may: determine a chunk for a machine learning base model of the machine learning base models, wherein the chunk comprises all minority cases from training data and a plurality of majority cases from the training data and train the machine learning base model with the chunk.


