Unified ML Acceleration Framework for Heterogeneous AI Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning (ML) and artificial intelligence (AI) frameworks, such as TensorFlow and PyTorch, primarily support graphics processing units (GPUs) and lack optimized performance for various AI hardware accelerators, making it difficult for users to utilize diverse AI hardware offerings due to private application programming interfaces and lack of transparent support for accelerators other than GPUs.
Innovation Solution
A unified ML acceleration framework is developed, combining an end-to-end machine learning compiler framework with an interposer block and a resolver block to modify and recompile ML models for specific hardware accelerators, allowing transparent deployment on low-level runtimes and returning results as if generated by the upstream framework, thereby supporting a wide range of accelerators including CPUs and specialized hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a common ML framework is used for all hardware accelerators, then framework simplicity is maintained, but performance optimization for specific accelerators is lost
Solution Approach 1:
The system segments the ML execution pipeline into three distinct parts: high-level framework (user code), translation layer (compiler), and low-level runtime (hardware-specific). This segmentation allows the high-level framework to remain simple and hardware-agnostic while the translation layer handles hardware-specific optimizations during the compilation phase.
Solution Approach 2:
The translation layer acts as an intermediary between the high-level framework and low-level runtime. It translates hardware-agnostic operations into hardware-specific instructions, enabling optimized execution on target accelerators without requiring users to modify their high-level code for different hardware platforms.
2Productivity
If users create models for specific accelerators with private APIs, then hardware-specific performance is optimized, but user ease of operation is reduced
Solution Approach 1:
The high-level framework provides universal support for multiple hardware accelerators through a single unified interface. Users can write and train models once using the high-level framework, and the system automatically translates them to execute on different accelerators (GPUs, FPGAs, CPUs) without requiring users to learn multiple private APIs.
Solution Approach 2:
The translation layer serves as an intermediary that hides hardware-specific complexity from users. It automatically handles the translation from high-level framework calls to hardware-specific operations, eliminating the need for users to directly interact with private APIs of different accelerators.
3Device complexity
If upstream frameworks support only GPUs and FPGAs, then framework simplicity is maintained, but adaptability to diverse hardware accelerators is reduced
Solution Approach 1:
The system achieves universality by implementing a hardware-agnostic high-level framework that can target multiple accelerator types (GPUs, FPGAs, CPUs) through a single unified interface. The translation layer enables this framework to adapt to diverse hardware architectures without requiring separate framework implementations for each device type.
Solution Approach 2:
The system changes the abstraction parameters of the framework by introducing a translation layer that maps high-level framework operations to hardware-specific implementations. This parameter transformation allows the same framework code to operate on different hardware accelerators by changing the translation rules rather than modifying the framework itself.
Data Source
AI summary
Disclosed is a system for converting a high-level runtime model to a low-level runtime model where the high-level runtime model runs on a client computer system, and the low-level runtime model runs on a server computer system. The server system has installed thereon a pool of hardware accelerators, and the low-level runtime model is targeted to the pool of accelerators. Outputs of the low-level runtime model are returned to the high-level runtime model as if the high-level runtime model computed the outputs.