Neural Network Layer Compression via Segmentation and Nonlinearity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning neural networks face challenges in reducing computational complexity and memory footprint, which is essential for deploying models on resource-constrained devices like smartphones and data centers, while maintaining classification accuracy.
Innovation Solution
The method involves compressing neural network layers by replacing them with multiple compressed layers and inserting nonlinearity, followed by fine-tuning the network to update weight values, thereby reducing the number of parameters and operations while preserving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning neural networks are used to achieve excellent classification results, then classification accuracy is improved, but computational complexity becomes prohibitively high
Solution Approach 1:
The patent segments a single complex neural network layer into multiple compressed layers with fewer parameters each. By dividing the computational task across multiple simpler layers, the overall computational complexity is reduced while maintaining the representational capacity needed for accurate classification.
Solution Approach 2:
The patent introduces nonlinearity between compressed layers as an additional dimensional element that enables the simplified architecture to capture complex patterns. This dimensional addition allows the compressed network to achieve comparable classification accuracy despite having fewer parameters.
2Adaptability or versatility
If deep learning neural networks are deployed on resource-constrained devices, then model accessibility is improved, but memory footprint becomes prohibitively large
Solution Approach 1:
The patent segments the neural network into compressed layers that collectively use less memory. By distributing the representation across multiple layers with fewer parameters each, the total memory footprint is reduced, enabling deployment on resource-constrained devices like smartphones and embedded systems.
3Productivity
If neural network layers are compressed to reduce parameters, then computational efficiency is improved, but classification accuracy drops substantially
Solution Approach 1:
The patent applies fine-tuning to dynamically adjust the parameters of compressed layers after initial compression. This dynamic adjustment process optimizes the compressed network's parameters for the specific classification task, recovering accuracy that would otherwise be lost due to compression.
Solution Approach 2:
The patent introduces nonlinearity between compressed layers, adding a new dimensional element that enables the compressed architecture to capture complex decision boundaries. This allows the network to maintain high classification accuracy despite having fewer parameters, thus preserving computational efficiency.
Data Source
AI summary
Compressing a machine learning network, such as a neural network, includes replacing one layer in the neural network with compressed layers to produce the compressed network. The compressed network may be fine-tuned by updating weight values in the compressed layer(s).