Robust Neural Network Training via RSIP-SSIP Abstraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training neural networks (NNs) are limited in providing robustness against various types of perturbations, particularly beyond white noise, as they often fail to consider all adversarial examples and do not offer formal guarantees for robustness, especially in safety-critical applications like autonomous vehicles and medical imaging.
Innovation Solution
The introduction of the Reversed Symbolic Interval Propagation-Standard Symbolic Interval Propagation (RSIP-SSIP) abstraction method, which combines Standard Symbolic Interval Propagation (SSIP) and Reversed Symbolic Interval Propagation (RSIP) to calculate bounds on reachable values in NNs, allowing for the definition of loss functions that promote robustness to white noise perturbations, and the use of transformation layers to encode perturbations such as brightness and contrast, effectively reducing other perturbation types to white noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If abstraction-based methods are used to calculate overestimating reachable sets for robustness, then formal guarantees for robustness are provided, but the methods are limited to white noise perturbations only
Solution Approach 1:
The patent extends the RSIP-SSIP abstraction method to handle multiple perturbation types (white noise, contrast, brightness, and other smooth intensity perturbations) within a single unified framework. The method calculates overestimating reachable sets that are valid across different perturbation domains, making the abstraction technique universally applicable to various robustness problems rather than limited to white noise only.
Solution Approach 2:
The patent modifies the abstraction method by introducing perturbation-specific parameters (such as contrast ratios and brightness values) into the reachable set calculation framework. This allows the same RSIP-SSIP algorithm to accommodate different perturbation types by adjusting input parameters, thereby achieving both formal guarantees and versatility without requiring separate methods for each perturbation type.
2Reliability
If existing robust training methods are used, then networks show good empirical robustness, but they do not provide formal guarantees and mostly target white noise perturbations
Solution Approach 1:
The patent replaces empirical, data-driven robustness evaluation with a formal verification approach using RSIP-SSIP abstractions. Instead of relying on statistical performance metrics and empirical testing, the method provides mathematically provable guarantees by calculating exact reachable sets through symbolic interval propagation, substituting the mechanical/euristic approach with a formal mathematical framework.
Solution Approach 2:
The patent introduces RSIP-SSIP abstractions as an intermediary layer between the neural network and the perturbation analysis. This abstraction mechanism serves as a mediator that translates network computations into bounded interval representations, enabling formal verification of robustness properties without requiring direct manipulation of the complex network dynamics.
3Measurement precision
If RSIP-SSIP abstraction method is applied to calculate bounds on reachable values, then precise bounds are obtained, but computational complexity increases
Solution Approach 1:
The patent divides the neural network into discrete computational layers and applies RSIP-SSIP abstractions to each layer sequentially. By segmenting the network structure and processing bounds layer-by-layer, the method manages computational complexity through systematic progression rather than attempting to compute all bounds simultaneously, maintaining precision while controlling resource requirements.
Solution Approach 2:
The patent applies RSIP-SSIP abstractions selectively to critical network components and intermediate layers rather than uniformly across all layers. This partial application strategy obtains precise bounds where most needed (at critical decision points) while reducing overall computational effort by skipping less critical regions, achieving a balance between precision and computational efficiency.
Data Source
AI summary
The present disclosure relates to systems, methods, and computerized techniques for robust training of neural networks (NNs) against various perturbations of their inputs. More specifically, the disclosed techniques introduce abstraction-based robust training methods with one or more features as identified herein. One such NN training feature can include a Reversed Symbolic Interval Propagation-Standard Symbolic Interval Propagation (RSIP-SSIP) abstraction method. RSIP-SSIP can use a combination of SSIP and RSIP to calculate an abstraction of the NN for a given type of perturbation of the NN's inputs. The RSIP-SSIP method can further be used to calculate bounds on reachable values of nodes in the underlying NN, which, in turn, may be used to define loss functions that promote robustness to the given type of perturbation, or any other type of perturbation, including bias field perturbations.