Hardware-Aware Model Optimization for Accelerator Memory and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network optimization methods fail to efficiently optimize operation schemes without changing the network structure, leading to increased operation time and latency due to memory usage and data attribute changes.
Innovation Solution
A dynamic model optimization technique that utilizes a graph intermediate representation to optimize operation schemes based on input data attributes and predicted performance, incorporating tiling schemes and data storage strategies to minimize memory usage and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing neural network optimization methods are used, then the network structure remains unchanged, but operation time and latency increase due to memory usage and data attribute changes
Solution Approach 1:
The patent implements dynamic optimization by adjusting operation schemes based on runtime data attributes and hardware characteristics. The system dynamically selects optimization strategies such as tiling schemes and data storage locations based on actual input data characteristics, rather than using static optimization methods. This dynamic adaptation resolves the contradiction by enabling effective optimization while maintaining low operation time and latency through context-aware decision making.
Solution Approach 2:
The patent changes operational parameters such as tiling sizes, data storage locations, and computation strategies based on data attributes and hardware characteristics. By adjusting these parameters dynamically according to runtime conditions, the system achieves effective optimization without incurring excessive operation time or latency penalties.
2Productivity
If dynamic optimization based on data attributes and hardware characteristics is implemented, then computational efficiency improves, but system complexity increases
Solution Approach 1:
The patent introduces an intermediate representation layer that serves as a mediator between the neural network model and the hardware execution layer. This intermediate layer captures data attributes and hardware characteristics, enabling dynamic optimization without requiring complex direct interactions between the model and hardware. The intermediary simplifies the optimization process by providing a standardized interface for analyzing data characteristics and selecting appropriate optimization strategies.
Solution Approach 2:
The patent segments the optimization process into distinct modules: data attribute analysis, hardware characteristic evaluation, optimization strategy selection, and execution. This segmentation allows each module to handle specific aspects of the optimization problem independently, reducing overall system complexity while maintaining high computational efficiency through coordinated module operation.
3Quantity of substance
If tiling schemes and data storage strategies are optimized, then memory usage decreases, but optimization time increases
Solution Approach 1:
The patent performs preliminary analysis of data attributes and hardware characteristics to pre-determine optimal tiling schemes and data storage strategies before execution. By analyzing data characteristics in advance and selecting appropriate optimization strategies beforehand, the system minimizes runtime optimization overhead while achieving efficient memory usage during actual computation.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A method with model optimization includes: determining a graph representing operations performed in a target model; determining an attribute of input data of the target model; determining a predicted performance of the target model based on a behavior pattern of hardware that executes the target model; and optimizing the operations performed in the target model based on the graph, the attribute of the input data, and the predicted performance of the target model.