Subband Decomposition for Low Precision Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face challenges in efficiency and performance due to their large size and computational intensity, particularly as dataset sizes increase, leading to increased non-linearity and power/memory requirements that exceed typical mobile device capabilities.
Innovation Solution
The approach involves decomposing input data into subbands with differing information content, allocating varying precision bits for weights and activations at the subband or layer level, and fusing subbands during training to reduce the number of parameters and resources needed, while maintaining robustness and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks use more parameters and layers to capture complex patterns, then classification performance is improved, but computational and memory resources increase exponentially
Solution Approach 1:
The input data is decomposed into multiple subbands (e.g., frequency subbands or spatial regions) with different information content. Each subband is processed separately by the neural network, allowing the system to handle complex patterns in a distributed manner. This segmentation reduces the computational burden on any single processing path while maintaining overall classification performance through aggregation of subband results.
2Reliability
If dataset size increases to improve model training, then classification accuracy is improved, but the degree of non-linearity increases exponentially making training more difficult
Solution Approach 1:
By decomposing the training dataset into subbands with different characteristics, the system can train multiple specialized neural network paths on different subsets of the data. Each subband-trained path learns specific patterns independently, reducing the overall training complexity while maintaining the ability to handle large datasets through parallel processing of subbands.
3Reliability
If neural network uses higher precision weights and activations, then learning stability is improved, but memory and computational power requirements increase
Solution Approach 1:
Different subbands are allocated different precision levels based on their information content and importance. Subbands containing critical features maintain higher precision to ensure learning stability, while subbands with less critical information use lower precision. This local quality differentiation reduces overall memory and computational requirements while preserving learning stability for the most important features.
Data Source
AI summary
Artificial neural network systems involve the receipt by a computing device of input data that defines a pattern to be recognized (such as faces, handwriting, and voices). The computing device may then decompose the input data into a first subband and a second subband, wherein the first and second subbands include different characterizing features of the pattern in the input data. The first and second subbands may then be fed into first and second neural networks being trained to recognize the pattern. Reductions in power expenditure, memory usage, and time taken, for example, allow resource-limited computing devices to perform functions they otherwise could not.