Tiled Outer Product Layer for Neural Network Parameter Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Fully connected layers in neural networks require a large number of parameters, leading to significant memory usage, which is particularly expensive in embedded device implementations.
Innovation Solution
The method involves partitioning input and output vectors into subvectors and computing submatrices as outer products of vectors uij and vij, reducing the number of parameters needed by representing each submatrix Wi,j as vij⊗uij, thereby forming a tiled outer product layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional fully connected layers are used with n inputs and m outputs, then the network can process complex patterns, but the number of parameters becomes n x m which requires excessive memory bandwidth
Solution Approach 1:
The patent segments the large weight matrix W into multiple smaller submatrices W_ij, where each submatrix corresponds to a specific output subset and input subset. This segmentation allows the network to process complex patterns through multiple localized transformations rather than a single large transformation, reducing the memory bandwidth requirement from O(nm) to O((n/t)(m/s)t) where t is the tile dimension.
Solution Approach 2:
The patent introduces a spatial dimension by organizing the weight matrix into a tiled structure W_ij where i and j are two-dimensional indices. This transforms the flat n x m parameter space into a hierarchical structure with (n/t) x (m/s) tiles, each of size t x s, enabling more efficient memory access patterns and reducing peak memory bandwidth requirements.
2Quantity of substance
If the number of parameters is reduced using outer product decomposition, then memory usage decreases, but the computational structure becomes more complex
Solution Approach 1:
The patent changes the parameter representation from dense n x m weights to a factorized form using outer products of smaller vectors. Each submatrix W_ij is decomposed into u_i ⊗ v_j where u_i is of size t and v_j is of size s, reducing the parameter count from nm to (n/t)(m/s)(t+s). This parameter transformation maintains computational equivalence while reducing memory footprint.
3Measurement precision
If larger fully connected layers are used with n=9216 and m=4096, then the network achieves higher accuracy, but 150 MB memory is required per layer which is expensive in embedded devices
Solution Approach 1:
The patent segments the large 9216 x 4096 weight matrix into smaller tiles of size t x s, where typical values might be t=256 and s=256, creating (9216/256) x (4096/256) = 36 x 16 = 576 tiles. Each tile stores only t+s = 512 parameters instead of t x s = 65536 parameters, achieving a compression ratio of 128x per tile while maintaining the same computational functionality through outer product reconstruction.
Data Source
AI summary
The present disclosure is drawn to the reduction of parameters in fully connected layers of neural networks. For a layer whose output is defined by y=Wx, where y is the output vector, x is the input vector, and W is a matrix of connection parameters, vectors uij and vij are defined and submatrices Wi,j are computed as the outer product of uij and vij, so that Wi,j=vij⊗uij, and W is obtained by appending submatrices Wi,j.