Neural Network Conversion Tool for Mobile Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, such as NN models, face challenges in direct execution on mobile devices due to limited computation resources, leading to degraded performance and efficiency during inferencing, as they require frequent off-chip memory access and lack optimal kernel operations for combining layers.
Innovation Solution
A conversion tool that optimizes NN models by generating device-specific net-list and weight objects, utilizing a model parser, graph optimizer, and net-list generator to simplify computing graphs and combine layers into single kernel operations, reducing off-chip memory access and eliminating the need for human labeling of combinable sub-structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If NN models are executed directly on mobile devices with limited computation resources, then device autonomy and reduced data transmission are improved, but performance and efficiency deteriorate due to frequent off-chip memory access and lack of optimized kernel operations
Solution Approach 1:
The computing graph is segmented into multiple layers that can be individually optimized and combined. The system parses the NN model into discrete layers, allowing selective combination of adjacent layers into single kernel operations, thereby reducing memory access frequency while maintaining device autonomy.
Solution Approach 2:
Adjacent layers in the computing graph are merged into single kernel operations when their combination is supported by the device kernel library. This merging reduces the number of separate operations and memory access events, improving inferencing efficiency without sacrificing device autonomy.
2Productivity
If multiple layers of NN model are combined into single kernel operations, then memory access operations are reduced and efficiency is improved, but device complexity and conversion process complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-processing the NN model through parsing, optimization, and net-list generation before execution. The conversion tool prepares the model in advance by identifying combinable layers and generating device-specific net-lists, so that the actual inferencing requires minimal complex operations.
Solution Approach 2:
A conversion tool acts as an intermediary between the standard NN model format and the device-specific execution format. This intermediary automatically performs graph parsing, optimization, and net-list generation, shielding users from the complexity of manual layer combination while enabling efficient execution.
3Ease of operation
If automatic layer mapping and optimization is performed, then human intervention is reduced and ease of operation is improved, but conversion tool complexity increases
Solution Approach 1:
The conversion tool performs self-service by automatically parsing the computing graph, identifying combinable layers, optimizing the graph structure, and generating device-specific net-lists without human intervention. The system serves itself by having built-in capabilities to handle the entire conversion process autonomously.
Solution Approach 2:
The conversion tool serves as an automatic intermediary that handles all complex conversion tasks. It includes components like the computing graph parser, optimizer, and net-list generator that work together to automatically map high-level NN operations to device-specific kernel operations, eliminating the need for manual optimization while containing complexity within the tool itself.
Data Source
AI summary
In various examples, a machine learning model is converted for execution by a computing device. For example, a computing graph is generated based on the machine learning model and sub-graphs within the computing graph that match sub-structures that are detected and combined into a vertex to generate an optimized computing graph. A net-list object and weight object are then generated based on the optimized computing graph and provided to the computing device to enable inferencing operations.