Bounded Nonlinear Function for Deep Neural Network Robustness

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional deep learning models are vulnerable to noise and do not adequately improve robustness against adversarial attacks, despite methods like logit squeezing.

Innovation Solution

An inference method that includes a first conversion step using a bounded nonlinear function in the final layer of a deep neural network, followed by a second conversion step with an activation function, to maintain the logit at a bounded value and enhance model robustness.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a conventional deep learning model is used, then the model can process images and output identification results, but the model is vulnerable to noise and adversarial attacks

Engineering Contradiction:

Improverobustness against noiseVSAvoidvulnerability to adversarial attacks

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The patent applies parameter changes by introducing a bounded nonlinear function that transforms the logit output parameters to be constrained within a specific range. This transformation changes the parameter space of the model output, making it resistant to adversarial attacks that rely on unbounded logit values. The bounded function effectively modifies the output parameters without changing the underlying model architecture or training data.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If logit squeezing is applied to constrain the norm of logit, then some robustness is improved, but robustness is not sufficiently improved

Engineering Contradiction:

ImproverobustnessVSAvoidinsufficient robustness improvement

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

Instead of constraining the logit norm from the inside (logit squeezing), the patent applies a bounded nonlinear function from the outside that transforms unbounded logits into bounded outputs. This inverted approach - transforming after generation rather than constraining during generation - provides more effective robustness against adversarial attacks while maintaining model simplicity.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS20230267316A1Inference method, training method, inference device, training device, and program

Publication Date: 2023.08.24 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

An inference device executes a first conversion step of converting an output from an intermediate layer using a bounded nonlinear function in a final layer of a deep neural network having the intermediate layer and the final layer. Moreover, the inference device executes a second conversion step of converting a value obtained by conversion in the first conversion step using an activation function.