Neural Network Acceleration via Scatter Accumulate Truth Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network hardware accelerators are inefficient in executing matrix multiplication kernels with sub-byte weights and byte activations, leading to high storage requirements and performance issues, particularly in mobile and power-constrained devices.
Innovation Solution
A matrix decomposition/restructuring method that leverages a scatter accumulate operation, utilizing a SIMD instruction or ANN accelerator crossbar array, to reduce operation count and memory overhead, enabling efficient computation of aggressively quantized neural network layers with parsimonious memoization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If matrix multiplication kernels are executed on existing hardware ANN accelerators and CPUs with sub-byte weights and byte activations, then neural network computation can be performed, but execution efficiency is extremely low
Solution Approach 1:
The patent changes the parameter representation by encoding sub-byte weights (4-bit values) into index values that select from pre-computed truth tables. This transformation allows the hardware to work with standard byte-sized indices and truth table values, making efficient use of existing CPU/accelerator instruction sets and memory structures without requiring specialized sub-byte arithmetic units.
Solution Approach 2:
The patent pre-computes and stores truth tables containing all possible multiplication results for sub-byte weights and byte activations during an offline phase. This preliminary action eliminates the need for complex runtime computation of sub-byte multiplications, allowing the hardware to perform simple table lookups and additions using standard instructions, thereby dramatically improving execution efficiency on existing hardware.
2Quantity of substance
If quantized neural network parameters are used, then storage requirements are reduced, but accuracy is sacrificed compared to full-precision models
Solution Approach 1:
The patent introduces a new dimension of computation by using truth tables that pre-compute results across the precision boundary. Instead of directly multiplying sub-byte weights with byte activations (which loses precision), the truth tables store results as if full-precision multiplication were performed, then the hardware retrieves and accumulates these pre-computed high-precision results, maintaining accuracy while using compact quantized parameters.
3Productivity
If native convolution operations are performed, then the actual convolution computation can be executed, but complicated dataflow and expensive datapaths are required
Solution Approach 1:
The patent extracts the computationally expensive and complex native convolution operation into an equivalent matrix multiplication formulation. By converting convolution to GEMM (General Matrix Multiplication), the patent leverages the simpler, more standardized matrix multiplication hardware and software infrastructure, eliminating the need for specialized convolution datapaths and complex dataflow management while maintaining computational equivalence.
Data Source
AI summary
A neural network system, method and apparatus are provided. A truth table matrix, an index vector and an input data tensor are read from a memory. At least a portion of the input data tensor is flattened into an input data vector. A scatter accumulate instruction is executed on the index vector and the input data vector to generate an intermediate vector. The truth table matrix and the intermediate vector are then multiplied to generate an output data vector.