Neural Network Weight Tensor Compression via Unified Tensor Basis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network compression techniques, such as Tensor Ring decompositions, face challenges in achieving a favorable 'compression ratio - performance degradation' trade-off due to complex and costly implementations, especially when deploying deep networks on resource-constrained devices like mobile robots and IoT systems.
Innovation Solution
A method that parametrizes each core of weight tensors in a vector space by embedding and reshaping them with hyperparameters, using a single tensor basis to represent all weight tensors, allowing for efficient compression and computation without converting to full tensor representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If Tensor Ring decomposition is used for neural network compression, then the network size is reduced, but the implementation complexity and computing resource cost increase
Solution Approach 1:
The patent merges multiple independent Tensor Ring factorizations into a single unified decomposition framework. Instead of applying separate decompositions to each layer (which increases complexity), the invention combines them into one coordinated system that shares computational structures, thereby reducing overall implementation complexity while maintaining compression effectiveness.
Solution Approach 2:
The patent creates a universal decomposition framework that can be applied across different neural network architectures and layers. The single Tensor Ring decomposition approach serves multiple functions: it compresses weights, reduces storage requirements, and provides a unified computational interface, thereby simplifying implementation compared to layer-specific approaches.
2Volume of stationary object
If independent Tensor Ring factorizations are applied to each layer, then compression is achieved, but computing resource cost increases
Solution Approach 1:
The patent combines multiple independent factorizations into a single unified decomposition that processes multiple layers together. This merging eliminates redundant computations that would occur if each layer were factorized independently, thereby reducing computing resource cost while achieving the same compression ratio.
Solution Approach 2:
The patent identifies and eliminates redundant computational operations across layers. By discarding duplicate factorization steps and recovering only the essential computations through the unified framework, the invention reduces overall computing resource consumption while preserving the compression benefits.
3Volume of stationary object
If high rank decompositions are implemented, then compression ratio is improved, but implementation complexity increases
Solution Approach 1:
The patent optimizes the rank parameter of the Tensor Ring decomposition to achieve an optimal balance between compression ratio and implementation complexity. By carefully selecting and tuning the rank parameter, the invention achieves high compression ratios without requiring excessively complex high-rank decompositions, thereby reducing implementation complexity while maintaining effective compression.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A method for compressing a neural network comprising a plurality of layers, each layer comprising a weight tensor. Said method comprises a step of selecting (E10) hyperparameters to define at least one basis of tensors referred to as "basis tensors", including a rank hyperparameter R, a dimension hyperparameter N and a size hyperparameter representative of the number of basis tensors. Said method also comprises, for each weight tensor, a set of steps of: - embedding (E20) the weight tensor into an envelope tensor, - reshaping (E30) said envelope tensor by tensorization, - determining (E40) a representation of the reshaped envelope tensor in the form of a sequence of elements comprising tensors referred to as "cores", - parameterizing (E50) each core into a combination of the basis tensors.