Neural Network Matrix Segmentation for Resource-Constrained Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep machine learning models consume substantial resources such as storage, memory bandwidth, and energy during inference, making it difficult to deploy them on mobile and embedded devices.
Innovation Solution
A method involving dividing an input matrix into vectors, grouping them into regular and irregular groups, and pruning these groups to reduce the computational load, allowing for efficient processing on neural network accelerators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the dimensions of machine learning models are increased to improve model accuracy, then model accuracy is improved, but storage, memory bandwidth, energy consumption, and computational resources are substantially consumed
Solution Approach 1:
The patent divides the weight matrix into multiple sub-matrices and further segments them into groups that can be processed independently. This segmentation allows the model to be broken down into manageable chunks that can be executed on resource-constrained devices, reducing the energy consumption associated with processing large monolithic models while preserving the overall model accuracy through systematic reconstruction of the full model functionality.
Solution Approach 2:
The patent transforms the weight matrix parameters by dividing and grouping operations, changing the computational parameters from processing the entire matrix at once to processing segmented portions. This parameter transformation reduces memory bandwidth requirements and energy consumption by optimizing how data is loaded, processed, and stored during inference, while maintaining model accuracy through systematic computation reconstruction.
2Measurement precision
If the dimensions of machine learning models are increased to improve model accuracy, then model accuracy is improved, but storage and memory bandwidth are substantially consumed
Solution Approach 1:
The weight matrix is divided into multiple sub-matrices that can be stored and processed separately. This segmentation reduces the storage burden on mobile and embedded devices by breaking down large model parameters into smaller, more manageable units that can be loaded incrementally, while the systematic grouping and processing ensure that model accuracy is preserved through complete utilization of all segments during inference.
3Adaptability or versatility
If deep machine learning models are deployed on mobile and embedded devices, then model functionality is achieved, but resource constraints make deployment difficult
Solution Approach 1:
The patent segments the weight matrix into sub-matrices and groups them in a way that enables distributed processing across multiple compute units. This segmentation approach reduces the complexity burden on individual processing units in mobile and embedded devices, allowing the deep model to be deployed by distributing computational tasks across available hardware resources while maintaining overall model functionality.
Solution Approach 2:
The patent introduces dynamic grouping and processing of matrix segments that can adapt to the available computational resources. The method dynamically adjusts how segments are grouped and processed based on device capabilities, enabling flexible deployment on various mobile and embedded platforms with different resource constraints while preserving the deep model's functional capabilities.
Data Source
AI summary
Embodiments of the disclosure provide methods and systems for processing a neural network associated with an input matrix having a first number of elements. The method can include: dividing the input matrix into a plurality of vectors, each vector having a second number of elements; grouping the plurality of vectors into a first group of vectors and a second group of vectors; and pruning the first group of vectors and the second group of vectors.