Edge Machine Learning Accelerator Compiler for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning networks require significant computational resources and data handling, often necessitating centralized cloud-based compute facilities, which can lead to latency and inefficiencies due to the remote location of input sources like cameras and sensors.
Innovation Solution
The integration of a machine learning accelerator (MLA) and compiler within edge devices, allowing for the embedding of machine learning networks directly on these devices, such as cameras, to perform computations in real-time with reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If machine learning networks are implemented on centralized cloud-based compute facilities, then significant computational resources can be accessed, but latency increases and processing speed decreases due to remote location of input sources
Solution Approach 1:
The patent segments the computing system by embedding machine learning networks directly into edge devices (cameras, sensors) rather than relying on centralized cloud facilities. This distribution of computational tasks to multiple edge locations enables local real-time processing while maintaining access to cloud resources for model training and updates, thereby resolving the latency-speed tradeoff.
Solution Approach 2:
The patent introduces edge devices as intermediary components between input sources (cameras, sensors) and centralized cloud compute facilities. These edge devices perform local machine learning inference to reduce latency, while still communicating with the cloud for resource-intensive tasks, thus mediating between the need for fast local processing and access to powerful centralized computing.
2Speed
If machine learning networks are embedded on edge devices, then real-time processing and reduced latency are achieved, but device complexity increases
Solution Approach 1:
The patent makes edge devices universal by integrating machine learning networks into diverse devices such as cameras, sensors, and other edge computing platforms. This multi-functionality allows various edge devices to perform both their primary functions and machine learning inference, distributing the computational burden across many devices rather than requiring complex centralized systems.
Solution Approach 2:
The patent applies preliminary action by pre-training machine learning models on centralized cloud facilities and deploying them to edge devices in a pre-processed state. This allows edge devices to execute optimized, ready-to-run models without requiring complex model training capabilities, thereby reducing the complexity burden on edge devices while maintaining fast local inference.
3Power
If data is continuously transferred to the cloud for processing, then centralized computational resources can be utilized, but power consumption increases and costs rise
Solution Approach 1:
The patent applies partial action by transferring only the necessary data and computational tasks to the cloud rather than continuously transferring all data. Edge devices perform local inference for most tasks, reserving cloud connectivity for model updates, training, and only when centralized computational power is specifically needed, thereby reducing unnecessary energy consumption and data transfer costs.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
A compiler receives a description of a machine learning network and generates a computer program that implements the machine learning network. The computer program includes statically scheduled instructions that are executed by a mesh of processing elements (Tiles). The instructions executed by the Tiles are statically scheduled because the compiler can determine which instructions are executed by which Tiles at what times. For example, for the statically scheduled instructions, there are no conditions, branching or data dependencies that can be resolved only at run-time, and which would affect the timing and order of the execution of the instructions.