Stochastic Ensemble DNN Hardening Against Adversarial Attacks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) deployed in uncontrolled environments remain vulnerable to adversarial attacks, and existing methods for enhancing robustness are either computationally expensive or require generating multiple DNNs with different weights, which is inefficient.
Innovation Solution
Training a single DNN and applying stochastic quantization to its layers to generate diverse ensembles, where the diversity is optimized using information theory, allowing for robustness against adversarial attacks without the need for multiple DNNs with different weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple DNNs with different weights are trained in parallel to create ensembles, then robustness to adversarial attacks is improved, but computational cost and device complexity increase significantly
Solution Approach 1:
The patent segments the ensemble creation process by applying stochastic quantization to different layers of a single DNN, creating multiple quantized versions of the same network architecture. This allows ensemble diversity to be achieved through layer-specific quantization rather than training multiple complete DNNs from scratch, significantly reducing computational complexity while maintaining robustness.
Solution Approach 2:
The patent changes the precision parameter of DNN weights through stochastic quantization, creating ensembles with varying levels of quantization (e.g., 1-bit, 2-bit, 4-bit weights). This parameter transformation allows a single DNN to generate multiple diverse ensemble members without requiring separate training processes, thereby reducing computational cost while improving adversarial robustness.
2Adaptability or versatility
If multiple DNNs with different weights are generated through random initialization, then ensemble diversity is improved, but training time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by training a single DNN to convergence first, then applies stochastic quantization to generate multiple ensemble members. This reverses the conventional approach of training multiple DNNs in parallel from random initialization. The single DNN is trained once with full precision, and diversity is introduced post-training through layer-specific quantization, significantly reducing total training time while maintaining ensemble diversity.
Solution Approach 2:
The patent creates ensemble members by copying the trained DNN architecture and applying different quantization levels to specific layers. Instead of training multiple separate DNNs, the system copies the same trained weights and applies stochastic quantization to generate diverse ensemble members, reducing computational resources and training time while preserving diversity through variation in quantization parameters.
3Productivity
If stochastic quantization is applied to generate quantized DNN ensembles, then computational efficiency is improved, but the complexity of managing different bit precision levels increases
Solution Approach 1:
The patent applies local quality by applying different quantization levels to different layers of the DNN based on their importance and sensitivity. Critical layers may use higher precision (e.g., 4-bit) while less critical layers use lower precision (e.g., 1-bit or 2-bit). This localized approach optimizes computational efficiency by quantizing only where necessary, reducing overall computational cost while managing complexity through hierarchical precision levels.
Data Source
AI summary
In general, the disclosure describes techniques for implementing an MI-based attack detector. In an example, a method includes training a neural network using training data, applying stochastic quantization to one or more layers of the neural network, generating, using the trained neural network, an ensemble of neural networks having a plurality of quantized members, wherein at least one of weights or activations of each of the plurality of quantized members have different bit precision, and combining predictions of the plurality of quantized members of the ensemble to detect one or more adversarial attacks and/or determine performance of the ensemble of neural networks.