Latency Optimized Heterogeneous CNN Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for heterogeneous deployment of convolutional neural networks (CNNs) face challenges in accurately predicting latency across different hardware platforms, especially for unseen model structures, due to their reliance on proxy metrics like FLOPs or layer-wise predictors, which are not universally applicable and limited to homogeneous implementations.
Innovation Solution
A method and system for latency-optimized heterogeneous deployment of CNNs that transforms the model by identifying performance-equivalent layers, partitions the layers, determines execution times on various hardware platforms, and calculates optimal heterogeneous and homogeneous latencies, using a performance database to optimize deployment across multiple hardware accelerators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If FLOPs or layer-wise predictors are used for latency estimation, then the prediction process is simple, but the accuracy is insufficient and not universally applicable to unseen model structures
Solution Approach 1:
The system pre-collects latency measurements for various CNN layers across multiple hardware platforms and stores them in a performance database before deployment. This preliminary data collection enables accurate latency estimation for unseen models by retrieving pre-measured performance characteristics, resolving the contradiction between simple prediction processes and accurate estimates.
Solution Approach 2:
Instead of directly measuring latency for each new model deployment, the system creates a performance database that copies and stores latency characteristics from previously measured models and layers. This copying approach allows the system to reuse measured performance data across different models and hardware platforms, maintaining both simplicity and accuracy.
2Productivity
If heterogeneous deployment is implemented to improve computational efficiency, then processing speed improves, but system complexity increases due to workload division and hardware configuration
Solution Approach 1:
The system segments the CNN model into individual layers and evaluates each layer's latency characteristics separately on different hardware platforms. This segmentation allows the system to identify optimal hardware assignments for each layer based on pre-measured performance data, simplifying the complex task of heterogeneous deployment by breaking it down into manageable layer-level decisions.
Solution Approach 2:
The performance database acts as an intermediary between the CNN model and the heterogeneous hardware system. It stores pre-measured latency data and provides guidance for optimal hardware assignment, mediating the complex interaction between multiple hardware platforms and reducing the overall system complexity by centralizing performance information.
3Measurement precision
If multiple experiments and hardware access are conducted to optimize heterogeneous deployment, then deployment accuracy improves, but engineering effort and time increase significantly
Solution Approach 1:
The system performs latency measurements for all CNN layers across multiple hardware platforms in advance and stores this data in a performance database. This preliminary action eliminates the need for time-consuming experiments during actual deployment, as the system can directly query pre-measured performance data to determine optimal heterogeneous configurations.
Solution Approach 2:
The system copies performance characteristics from previously measured layer-hardware combinations to predict and optimize deployment for new models. By reusing measured data across different models and scenarios, the system maintains high deployment accuracy without repeating expensive and time-consuming hardware experiments.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
This disclosure relates generally to a method and system for latency optimized heterogeneous deployment of convolutional neural network (CNN). State-of-the-art methods for optimal deployment of convolutional neural network provide a reasonable accuracy. However, for unseen networks the same level of accuracy is not attained. The disclosed method provides an automated and unified framework for the convolutional neural network (CNN) that optimally partitions the CNN and maps these partitions to hardware accelerators yielding a latency optimized deployment configuration. The method provides an optimal partitioning of the CNN for deployment on heterogeneous hardware platforms by searching network partition and hardware pair optimized for latency while including communication cost between hardware. The method employs performance model-based optimization algorithm to optimally deploy components of a deep learning pipeline across right heterogeneous hardware for high performance.