Multi-Layer Neural Network Channel Expansion for Model Simplification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing multi-layer neural network models face challenges in simplifying their structure while maintaining performance, as current methods like network pruning, parameter quantization, low-rank approximation, and ternarized weights either reduce accuracy or increase model size, failing to achieve a balance between simplification and performance.

Innovation Solution

Expanding the number of channels in convolutional layer filters during both forward propagation and training, allowing for enriched weight connections and maintaining model performance through channel replication and gradient value updating.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If the network model is made deeper and wider to improve service accuracy, then the accuracy is improved, but the computation amount increases and processing speed decreases

Engineering Contradiction:

Improveservice accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent uses channel replication to create multiple copies of filter channels. Instead of using a single complex filter, multiple simplified filter copies are used in parallel, achieving the same computational effect with simpler individual units that can be more efficiently processed

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent segments the convolutional layer into multiple groups where each group contains replicated filter channels. This segmentation allows independent processing of each group, enabling parallel computation and improving processing speed while maintaining the overall model depth and accuracy

Inventive Principle:

Principle #1Segmentation

2Device complexity

If network pruning is applied to simplify the network model, then the model complexity is reduced, but it is difficult to achieve accurate hyper-parameter setting to determine which filters can be removed

Engineering Contradiction:

Improvenetwork model complexityVSAvoidhyper-parameter setting difficulty

Core Design Contradiction:

Device complexityVSEase of manufacture

Solution Approach 1:

The patent changes the parameter organization by replicating channels instead of removing filters. The number of channels is expanded through replication (e.g., 3 channels replicated to 6 channels), and this replication factor becomes the controllable parameter, making it easier to adjust model complexity without the difficulty of selecting which specific filters to remove

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The replicated filter channels serve multiple functions: they maintain the original filter's computational capability while enabling parallel processing, reducing computation amount, and allowing flexible adjustment of model complexity. The same replicated structure can be applied to different convolutional layers universally

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Volume of stationary object

If parameter quantization is used to reduce storage space, then the storage space is reduced, but the reduced representation precision makes the performance of the network model worse

Engineering Contradiction:

Improvestorage spaceVSAvoidperformance

Core Design Contradiction:

Volume of stationary objectVSMeasurement precision

Solution Approach 1:

The patent uses channel replication to achieve compression without quantization. By replicating channels and adjusting the replication factor, the model can be made more compact while maintaining full precision parameters, avoiding the performance degradation associated with quantization

Inventive Principle:

Principle #26Copying

4Volume of stationary object

If low-rank approximation is applied to decompose filter parameters, then the network model size is reduced, but the compression ratio is limited and does not bring about a significant reduction in the computation amount

Engineering Contradiction:

Improvenetwork model sizeVSAvoidcomputation amount reduction

Core Design Contradiction:

Volume of stationary objectVSProductivity

Solution Approach 1:

The patent uses channel replication to achieve significant compression ratios. By replicating channels with a replication factor greater than 1, the model can be compressed more effectively than low-rank approximation while simultaneously reducing computation amount through the simplified replicated structure

Inventive Principle:

Principle #26Copying

5Measurement precision

If ternarized weights with Residual Expansion technology are used to improve accuracy, then the accuracy is improved, but more filters of ternarized weights are introduced, resulting in a significant increase in the size of the network model

Engineering Contradiction:

ImproveaccuracyVSAvoidnetwork model size

Core Design Contradiction:

Measurement precisionVSVolume of stationary object

Solution Approach 1:

The patent applies channel replication selectively to specific convolutional layers based on their importance and computational cost. Not all layers are replicated with the same factor, allowing the model to maintain accuracy in critical layers while reducing overall model size in less critical layers

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11847569B2Training and application method of a multi-layer neural network model, apparatus and storage medium

Publication Date: 2023.12.19 CANON KK

AI summary

The present disclosure provides a training and application method of a multi-layer neural network model, apparatus and storage medium. A number of channels of a filter in at least one convolutional layer in the multi-layer neural network model is expanded, and a convolution computation is performed by using the filter after expanding the number of channels, so that the performance of the network model does not degrade while simplifying the network model.