MLA Compiler Tile Allocation for Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning networks require significant computational resources and data handling, making them inefficient when deployed on edge devices due to the need for continuous data transfer to cloud-based compute facilities, which can lead to latency and high power consumption.
Innovation Solution
A machine learning accelerator (MLA) with a compiler that allocates instructions to groups of processing elements (Tiles) for static scheduling, allowing for deterministic phases of computation and reducing the need for run-time arbitration, thereby simplifying the hardware and improving performance, power efficiency, and cost-effectiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If machine learning networks are deployed on cloud-based compute facilities, then computational power and data handling capacity are improved, but latency and power consumption increase due to continuous data transfer between edge devices and cloud facilities
Solution Approach 1:
The patent segments the machine learning network into multiple layers and allocates different layers to different groups of processing elements (Tiles). This segmentation allows the system to distribute computational tasks across multiple specialized units, enabling parallel processing while maintaining data locality at edge devices, thus reducing latency while preserving computational power.
Solution Approach 2:
The patent introduces a new dimension of computation allocation by organizing processing elements into a two-dimensional mesh topology with multiple groups and layers. This spatial organization enables efficient data flow and computation parallelism, allowing the system to achieve high computational power with reduced latency through optimized data paths and localized processing.
2Power
If machine learning networks are deployed on cloud-based compute facilities, then computational resources are improved, but power consumption increases due to continuous data transfer
Solution Approach 1:
The patent extracts the machine learning computation workload from remote cloud facilities and embeds it directly into edge devices through dedicated processing element groups. This extraction eliminates the need for continuous data transfer between edge devices and cloud facilities, reducing power consumption while maintaining access to significant computational resources locally.
Solution Approach 2:
The patent enables edge devices to perform machine learning computations autonomously through embedded processing groups, making them self-sufficient. The device can process data locally without continuously communicating with external cloud facilities, thereby reducing power consumption associated with data transmission while maintaining computational capabilities.
3Adaptability or versatility
If dynamic scheduling is used for machine learning network execution, then adaptability is improved, but hardware complexity and run-time arbitration requirements increase
Solution Approach 1:
The patent performs computation allocation and scheduling decisions in advance during the compilation phase, before runtime execution. The compiler statically assigns different layers of the machine learning network to specific groups of processing elements, eliminating the need for complex runtime arbitration logic. This preliminary action maintains adaptability for different workloads while significantly reducing hardware complexity.
Solution Approach 2:
The patent replaces complex mechanical runtime arbitration mechanisms with a software-based compilation approach. Instead of using hardware logic to dynamically schedule tasks at runtime, the system uses a compiler to generate predetermined execution schedules, substituting mechanical hardware complexity with software preprocessing that achieves the same adaptability with simpler hardware.
4Productivity
If more processing elements are allocated to machine learning network layers, then throughput is improved, but device complexity and cost increase
Solution Approach 1:
The patent segments the processing elements into multiple independent groups, each dedicated to specific layers of the machine learning network. This segmentation allows the system to scale throughput by adding or configuring groups as needed, rather than requiring a monolithic increase in processing elements. Each group can be independently optimized, maintaining manageable device complexity while improving overall throughput.
Data Source
AI summary
A compiler receives a description of a machine learning network and generates a computer program that implements the machine learning network. The compiler allocates instructions of the computer program to different groups of processing elements (Tiles) for execution such that different groups of Tiles implement different layers of the machine learning network. The compiler may determine the size of the different groups based on a partial computation metric associated with the computations performed to implement the corresponding layer. Furthermore, the compiler may assign specific Tiles to each group based on a set of predefined layout constraints. The compiler may statically schedule at least a portion of the instructions into one or more deterministic phases for execution by the groups of Tiles.