Robust Neural Network Learning via Subset Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing method of generating multiple machine learning models with different gradient vectors to enhance robustness against adversarial samples requires extensive calculation, leading to inefficient learning time as the number of models increases, with a computational complexity of O(n^2) when updating parameters for n models.
Innovation Solution
A robust learning device and method that selects a subset of neural networks among n, calculates a limited objective function focusing on the selected networks, and updates parameters to minimize the prediction loss and similarity between gradient vectors, reducing the computational burden by selecting p models for updates instead of n.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple neural networks are trained to improve robustness against adversarial samples, then the robustness is improved, but the learning time and computational complexity increase significantly
Solution Approach 1:
The patent segments the training process by dividing n neural networks into multiple batches. Instead of updating all n models simultaneously (which would require O(n^2) computation), the training iterates through batches of models, updating parameters for each batch separately. This segmentation reduces the computational burden per iteration while maintaining the robustness benefits of multiple diverse models.
Solution Approach 2:
The patent applies partial action by updating parameters for only a subset of models in each iteration rather than all models. In each training iteration, a batch of p models (where p < n) is selected for parameter updates based on their gradient vector similarities. This partial updating approach significantly reduces computational complexity while still achieving robustness through the diversity of gradient directions across the full set of n models.
2Reliability
If parameters for all n models are updated to reduce gradient similarity, then the robustness is improved, but the computational complexity becomes O(n^2)
Solution Approach 1:
The patent segments the model set into batches for processing. Instead of computing gradient similarities for all n models simultaneously (O(n^2) complexity), the training process divides models into batches of size p and processes each batch independently. This segmentation reduces the per-iteration computational complexity to O(p^2) where p << n, making the system scalable.
Solution Approach 2:
The patent performs parameter updates for only a partial subset of models in each iteration. By selecting and updating parameters for p models out of n total models based on gradient similarity metrics, the system achieves robustness improvements without the full computational burden of updating all models simultaneously. This partial action approach maintains the essential diversity benefit while reducing complexity.
Data Source
AI summary
A robust learning device is a learning device that, with a parameter of n neural networks, training data, and a correct label serving as inputs, outputs the updated parameter, including: a model selection unit that selects neural networks, which are less than n and equal to or more than two, among the n neural networks; a limited objective function calculation unit that calculates, in a calculation process of an objective function including a process in which a value of the objective function becomes smaller as an output of the neural networks to the training data is closer to the correct label and a degree of similarity between the neural networks is smaller, a limited objective function including only the process relating to the neural networks selected by the model selection unit; and an update unit that updates the parameter such that a value of the limited objective function is decreased.