Neural Network Inference Using Orthogonal Binary Basis Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State-of-the-art neural networks are large and complex, requiring significant computational resources, making them unsuitable for deployment on resource-constrained devices like embedded systems and IoT devices, and existing compression techniques like model binarization often result in significant performance drops.
Innovation Solution
A neural network model is generated and optimized for resource-constrained devices using orthogonal binary basis vectors and associated coefficients, which reduces memory usage and computational requirements while maintaining high inference accuracy by leveraging binary operations and selective control over resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If state-of-the-art neural network models are used, then prediction accuracy is improved, but computational resource requirements increase making them unsuitable for resource-constrained devices
Solution Approach 1:
The neural network model is segmented into binary components (weights and activations quantized to binary values), separating the computational burden into simpler operations that can be executed efficiently on resource-constrained devices while preserving the network's predictive capabilities
Solution Approach 2:
The parameter representation is changed from standard floating-point or integer formats to binary values, fundamentally altering how weights and activations are stored and processed. This parameter transformation reduces memory requirements and enables the use of efficient binary neural network operations on devices with limited computational resources
2Quantity of substance
If model binarization is applied, then memory use is reduced by 32 times, but inference performance drops significantly due to quantisation
Solution Approach 1:
The system performs preliminary training of the neural network with binary weights and activations before deployment. This pre-training phase allows the model to adapt to binary operations and learn representations that are optimized for binary computation, thereby maintaining inference accuracy despite the quantization to binary values
Solution Approach 2:
The patent introduces binary dummy activations as an intermediary mechanism during inference. These dummy binary activations facilitate efficient binary convolution operations while the system learns to compensate for the quantization effects, thereby maintaining accuracy while benefiting from reduced memory usage
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
This specification describes methods for performing inferencing based on input data, the methods comprising: initialising a neural network based on a set of stored model information, which defines a plurality of orthogonal binary basis vectors which are to be used to implement kernels in one or more hidden layers of the neural network, and plural sets of plural coefficients, each set of plural coefficients corresponding to a respective one of the kernels, wherein each of the coefficients in a given set of coefficients is associated with a respective one of the one or more orthogonal binary basis vectors; passing input data through the neural network such that convolution operations between the kernels and data arriving at the kernels are performed, wherein each of the kernels is implemented using a respective set of coefficients and the orthogonal binary basis vectors with which the coefficients in the set are associated; and outputting data from the neural network, the output data representing an inference corresponding to the input data. The specification also describes methods for generating model information based on which neural networks may be initialised.