Embedded Neural Network Compression for Low-Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large machine learning models for autonomous vehicles and robots are inefficient due to high latency and computational requirements, making them unsuitable for embedded applications with limited processing power.
Innovation Solution
A deep learning model is over-parameterized during training and then compressed into a compact architecture using element-wise addition of component tensors, allowing for faster inference while maintaining performance benefits of larger models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large machine learning models are used to improve prediction performance, then model expressivity and accuracy are improved, but inference time and latency increase
Solution Approach 1:
The parameter tensor is segmented into multiple component tensors during training, allowing the model to learn with over-parameterization benefits. During inference, these segmented tensors are combined through element-wise addition to reconstruct the original parameter space, achieving both accuracy and speed
Solution Approach 2:
The invention changes the parameter representation by introducing component tensors that sum to the original parameter tensor. This parameter transformation allows the model to maintain the expressivity of large models during training while enabling efficient inference by operating with smaller component tensors
2Measurement precision
If large machine learning models are used to improve prediction performance, then model expressivity is improved, but computational requirements and processing power increase
Solution Approach 1:
By segmenting the parameter tensor into component tensors, the computational burden is divided and distributed. Each component tensor requires less computational resources individually, while their combination through element-wise addition preserves the full model capability
Solution Approach 2:
The invention creates component tensors that are computational copies or representations of the original parameter tensor. These component tensors can be processed more efficiently on embedded hardware while still capturing the essential model behavior through their sum
3Speed
If model size is reduced for embedded applications, then inference speed and latency are improved, but model expressivity and performance decrease
Solution Approach 1:
The invention transforms the parameter representation to allow the model to operate with smaller component tensors during inference while maintaining the ability to access the full parameter space through element-wise addition of these components, thus preserving accuracy without sacrificing speed
Data Source
AI summary
The present disclosure relates to computer-implemented methods for automatically controlling a machine includes: receiving data generated using at least one sensor of a machine; performing one or more prediction tasks on the data using a neural network, wherein the neural network includes at least one parameter tensor having at least one element, and the at least one parameter tensor was over-parameterized during training into a plurality of component tensors; and controlling the machine based on results of the one or more prediction tasks. The present disclosure further relates to a computing system for carrying out the method, a method for generating a machine learned neural network for the method, a data structure, a machine, a mobile agent, a data processing system, and a computer program, machine-readable storage medium, or a data carrier signal.