Sparse DNN Weight Structure via Walsh-Hadamard Transform
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep convolutional neural networks (CNNs) face challenges in deploying large models on energy-constrained IoT devices due to their high computational and memory requirements, as existing efficient strategies like Winograd's minimal filtering and model compression are not compatible when applied together, especially due to the loss of sparsity in convolution kernels.
Innovation Solution
A non-convolution based framework that trains deep neural networks in heterogeneous transform domains using element-wise multiplications and binary-valued fast linear transforms like the discrete Walsh-Hadamard transform, allowing for sparse-orthogonal kernels and canonical signed-digit representation for bit-sparse non-uniform quantization to reduce computation and memory needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If Winograd's minimal filtering algorithm or discrete Fourier transforms are used to replace convolution with element-wise multiplications, then computation complexity is reduced, but sparsity of convolution kernels is lost
Solution Approach 1:
The patent changes the domain parameters by training deep neural networks in heterogeneous transform domains (such as Walsh-Hadamard transform domain) rather than spatial domain. This parameter change allows the system to maintain sparsity in the transform domain while enabling efficient element-wise multiplication operations, thus resolving the contradiction between computation complexity reduction and sparsity preservation
Solution Approach 2:
The patent introduces transform domains as an intermediary between the input spatial domain and the output spatial domain. By transforming inputs to heterogeneous transform domains, performing element-wise multiplications with sparse kernels in these intermediate domains, and then transforming back, the system achieves both computational efficiency and sparsity utilization
2Device complexity
If model compression is applied through weight pruning and quantization, then memory requirements and computation are reduced, but accuracy is degraded
Solution Approach 1:
The patent applies non-uniform quantization with canonical signed-digit (CSD) representation in the transform domain, which preserves more information than uniform quantization. This parameter change in the quantization scheme allows for reduced precision representation while maintaining better accuracy compared to conventional uniform quantization methods
Solution Approach 2:
The patent moves the network operation from spatial domain to transform domain, effectively adding a dimensional transformation. This dimensional change allows the network to achieve compression and efficiency gains while maintaining accuracy, as the transform domain representation captures the essential features more efficiently
3Productivity
If deep neural networks are deployed on energy-constrained IoT devices, then real-time application capability is achieved, but energy consumption becomes excessive
Solution Approach 1:
The patent replaces the conventional spatial domain convolution mechanism with transform domain element-wise multiplication. This substitution fundamentally changes the computational mechanism from intensive convolution operations to simpler element-wise operations, significantly reducing energy consumption while maintaining real-time processing capability on constrained devices
Data Source
AI summary
A computer-implemented method is presented for performing a computation with a neural network. The method includes: receiving a first input patch of data; applying a Walsh-Hadamard transform to the input patch to yield a transformed input patch in a transformed domain; computing an element-wise product of the transformed input patch and a kernel of the neural network; applying an inverse Walsh-Hadamard transform to the element-wise product to yield an intermediate matrix; and creating a first output patch from the intermediate matrix, where the size of the first output patch is smaller than the intermediate matrix.