Neural Classifier Ensembles With Orthogonal Decision Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning technologies are vulnerable to perturbation-bounded evasion attacks, where adversarial examples can fool classifiers despite being trivially classified by humans, and principled analysis based on convolutional neural networks (CNNs) is challenging due to the complexity of kernel mappings.
Innovation Solution
A system constructs an ensemble of neural network-based classifiers that optimize a diversity metric by defining a diversity metric based on pairwise angles between decision boundaries of affine classifiers, incorporating this metric as a regularization term in a loss function optimization, and training neural networks to ensure parameters are consistent with affine classifiers, thereby enhancing robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional ensemble methods are used to defend against adversarial examples, then robustness against attacks is improved, but explicit control over the generated classifiers and their diversity is lost
Solution Approach 1:
The patent changes the parameter space by formulating classifier diversity in terms of geometric parameters (angles between decision boundaries) rather than traditional ensemble parameters. By optimizing the angle parameter to be 90 degrees, the method achieves explicit control over classifier relationships while maintaining robustness against adversarial examples.
Solution Approach 2:
The patent applies preliminary action by pre-specifying the desired geometric relationship (orthogonality) between classifiers before training. The diversity metric is built into the loss function in advance, guiding the training process to produce classifiers with the desired properties rather than relying on post-hoc analysis.
2Reliability
If classifiers are made more diverse to improve robustness, then adversarial robustness is enhanced, but the complexity of optimizing multiple diverse classifiers increases
Solution Approach 1:
The patent simplifies the optimization complexity by changing the diversity parameter from general distributional differences to a specific geometric parameter (angle between decision boundaries). This parameter change transforms a complex multi-dimensional optimization problem into a more tractable problem with a clear target (90-degree angles).
Solution Approach 2:
The patent substitutes the traditional mechanical approach of training multiple independent classifiers with a geometric approach based on orthogonal decision boundaries. This substitution leverages the mathematical properties of orthogonality to achieve diversity without the computational burden of training and coordinating multiple complex models.
3Manufacturing precision
If affine classifiers are designed with orthogonal decision boundaries, then diversity metric is optimized, but the difficulty of training neural networks to match affine classifier parameters increases
Solution Approach 1:
The patent applies feedback by using the affine classifiers as teachers to guide the training of neural networks. The neural networks are trained to match the parameters of the affine classifiers, with the affine classifiers providing target parameter values. This feedback mechanism ensures that the neural networks achieve the desired orthogonal decision boundaries while maintaining parameter consistency.
Data Source
AI summary
One embodiment provides a system which facilitates construction of an ensemble of neural network-based classifiers that optimize a diversity metric. During operation, the system defines a diversity metric based on pairwise angles between decision boundaries of three or more affine classifiers. The system includes the diversity metric as a regularization term in a loss function optimization for designing a pair of mutually orthogonal affine classifiers of the three or more affine classifiers. The system trains one or more neural networks such that parameters of the one or more neural networks are consistent with parameters of the affine classifiers to obtain an ensemble of neural network-based classifiers which optimize the diversity metric. The system predicts an outcome for a testing data object based on the obtained ensemble of neural-network based classifiers which optimize the diversity metric.