Neural Network Normalization Layer Scaling Preservation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing neural network implementations lose information about absolute scaling during normalization, which is crucial for accurate classification, especially in data where scale is indicative of the file's maliciousness.

Innovation Solution

Incorporating parameter nodes in the normalization layer to calculate and provide mean and standard deviation as additional inputs to the activation layer, thereby preserving scaling information and preventing its loss during normalization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Stability of the object's composition

If normalization is applied to stabilize neural network behavior, then stability is improved, but information about absolute scaling is lost

Engineering Contradiction:

Improveneural network stabilityVSAvoidabsolute scaling information

Core Design Contradiction:

Stability of the object's compositionVSLoss of information

Solution Approach 1:

The patent extracts the normalization parameters (mean and standard deviation) from the normalization process and outputs them separately. This allows the normalization to stabilize the network while the extracted parameters preserve the absolute scaling information that would otherwise be lost.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces parameter nodes as intermediaries that carry the normalization parameters between layers. These parameter nodes act as mediators that preserve the scaling information through the normalization process, enabling subsequent layers to recover or utilize the absolute scaling data.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of information

If normalization parameters are provided as additional inputs to preserve scaling information, then information preservation is improved, but computational complexity increases

Engineering Contradiction:

Improvescaling information preservationVSAvoidneural network structure

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent segments the neural network into distinct components: normalization nodes that perform the normalization operation and separate parameter nodes that handle the normalization parameters. This segmentation allows the system to manage complexity by dividing functions into specialized, modular units.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The parameter nodes serve multiple functions: they store normalization parameters, transmit scaling information across layers, and enable subsequent layers to utilize this information for their computations. This multi-functionality reduces the need for separate dedicated structures for each purpose.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11947668B2Methods and apparatus for preserving information between layers within a neural network

Publication Date: 2024.04.02 SOPHOS LTD

AI summary

In some embodiments, an apparatus includes a memory and a processor. The processor is configured to extract a set of features from a potentially malicious file and provide the set of features as an input to a normalization layer of a neural network. The processor is configured to implement the normalization layer by calculating a set of parameters associated with the set of features and normalizing the set of features based on the set of parameters to define a set of normalized features. The processor is further configured to provide the set of normalized features and the set of parameters as inputs to an activation layer of the neural network such that the activation layer produces an output based on the set of normalized features and the set of parameters. The output can be used to produce a maliciousness classification of the potentially malicious file.