Mini-Classifier Ensemble With Dropout Regularization For Small Sample Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In bio-life sciences, the challenge lies in developing classifiers for biological samples with limited sample numbers and a large number of attributes, where existing methods struggle due to the imbalance between the number of samples and attributes, particularly in deep data problems where p>>n.
Innovation Solution
A method involving the construction of mini-classifiers using sets of feature values, filtering based on performance, and combining them using a regularized combination method like logistic training with dropout regularization to generate a master classifier, allowing for the classification of biological samples effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard machine learning methods are used to develop classifiers, then classification performance can be achieved when large training data sets are available, but the methods fail when the number of samples is limited and the number of attributes exceeds the number of samples
Solution Approach 1:
The patent segments the classifier development process into multiple stages: constructing numerous individual mini-classifiers from different feature subsets, filtering them by performance, and combining the filtered mini-classifiers into a master classifier. This segmentation allows the system to handle limited sample data by distributing the classification task across many specialized mini-classifiers rather than relying on a single classifier trained on all features simultaneously.
Solution Approach 2:
The patent transforms the classification problem from a direct high-dimensional approach to a multi-layered approach. Instead of training one classifier on p features with n samples, it creates multiple mini-classifiers on different feature subsets, then combines them in a second dimension (the combination layer). This dimensional transformation effectively increases the information extraction capability from limited samples.
2Loss of information
If the number of attributes (measurements) is increased to provide deeper description of individual instances, then information gain from individual samples improves, but the imbalance between attributes and samples worsens
Solution Approach 1:
The patent segments the large set of p attributes into multiple smaller feature subsets, each used to train individual mini-classifiers. This segmentation prevents any single mini-classifier from being overwhelmed by the full dimensionality, allowing each to effectively utilize the limited sample data while collectively covering the entire feature space through their combination.
Solution Approach 2:
The patent employs excessive action by constructing far more mini-classifiers than strictly necessary (using multiple feature subsets of size s where s<<p). This excessive number of mini-classifiers ensures comprehensive coverage of the feature space and provides redundancy, allowing the filtering and combination process to select only the most informative subsets while discarding redundant or noisy features.
3Measurement precision
If mini-classifiers are combined without regularization, then classification accuracy may improve, but overfitting to the limited training data increases
Solution Approach 1:
The patent performs preliminary filtering of mini-classifiers before combination, retaining only those that meet performance thresholds on the training data. This preliminary action prevents poorly performing mini-classifiers from contributing to overfitting, while the subsequent regularized combination further ensures generalizability by weighting mini-classifiers according to their performance and incorporating dropout regularization.
Solution Approach 2:
The patent implements feedback mechanisms through performance threshold filtering and regularized combination weighting. Mini-classifiers are evaluated on their classification performance, and this feedback determines whether they are retained for combination and what weight they receive in the master classifier. This feedback loop prevents overfitting by systematically eliminating or down-weighting mini-classifiers that do not generalize well.
Data Source
AI summary
A method for classifier generation includes a step of obtaining data for classification of a multitude of samples, the data for each of the samples consisting of a multitude of physical measurement feature values and a class label. Individual mini-classifiers are generated using sets of features from the samples. The performance of the mini-classifiers is tested, and those that meet a performance threshold are retained. A master classifier is generated by conducting a regularized ensemble training of the retained/filtered set of mini-classifiers to the classification labels for the samples, e.g., by randomly selecting a small fraction of the filtered mini-classifiers (drop out regularization) and conducting logistical training on such selected mini-classifiers. The set of samples are randomly separated into a test set and a training set. The steps of generating the mini-classifiers, filtering and generating a master classifier are repeated for different realizations of the separation of the set of samples into test and training sets, thereby generating a plurality of master classifiers. A final classifier is defined from one or a combination of more than one of the master classifiers.