Image Inference Resource Allocation via Weight Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image inference methods are unable to perform at high speeds, failing to meet the requirements of high-speed production scenarios, especially when using trained deep learning models in conjunction with hardware accelerators.
Innovation Solution
A computer device method that pre-trains multiple neural network models and uses a preset weight table to determine the most efficient hardware accelerator and machine learning framework combination for image inference, optimizing the usage state and estimated time duration to achieve the shortest inference time, allowing for parallel processing when possible.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a trained deep learning model is used with a single hardware accelerator and machine learning framework, then the system structure is simple, but the inference speed is slow and cannot meet high-speed production requirements
Solution Approach 1:
The patent segments the inference system into multiple independent components: multiple hardware accelerators (GPU, CPU, NPU), multiple machine learning frameworks (TensorFlow, PyTorch, Caffe), and multiple pre-trained neural network models. Each component can operate independently, and the system selects the optimal combination for each inference task based on the preset weight table, thereby achieving high-speed inference without requiring a completely complex unified architecture
Solution Approach 2:
The patent creates a universal inference system that supports multiple hardware accelerators and multiple machine learning frameworks simultaneously. The preset weight table stores collocation relationships between different hardware accelerators, frameworks, and models, enabling the system to adapt to various inference scenarios using the same overall architecture, thus improving inference speed while maintaining reasonable system complexity
2Productivity
If multiple hardware accelerators and machine learning frameworks are used to improve inference speed, then the productivity increases, but the device complexity and resource management difficulty increase
Solution Approach 1:
The patent performs preliminary actions by pre-training multiple neural network models in advance and storing their collocation relationships with different hardware accelerators and frameworks in a preset weight table. When an inference request arrives, the system directly queries the weight table to obtain the optimal configuration, eliminating the need for complex real-time decision-making and resource allocation, thus improving both inference throughput and ease of operation
Solution Approach 2:
The patent introduces a preset weight table as an intermediary between the inference request and the actual inference execution. The weight table stores pre-calculated collocation relationships between hardware accelerators, machine learning frameworks, and neural network models, acting as a mediator that simplifies resource management by providing direct lookup of optimal configurations rather than requiring complex real-time optimization algorithms
3Productivity
If a single machine learning framework is used, then the system is easy to manage, but it cannot fully utilize different hardware accelerators and limits inference efficiency
Solution Approach 1:
The patent changes the parameter of framework compatibility by supporting multiple machine learning frameworks (TensorFlow, PyTorch, Caffe) simultaneously. The preset weight table stores collocation relationships between different frameworks and hardware accelerators, allowing the system to select the optimal framework-hardware combination for each inference task, thereby fully utilizing different hardware accelerators while maintaining manageable system complexity through structured parameter storage
Data Source
AI summary
An image inference method is provided by the present disclosure. The method includes determining a target collocation scheme for an image according to an inference request and a preset weight table, the target collocation scheme including a hardware accelerator in an idle state and an estimated time duration of inferring the image. A usage state of the hardware accelerator in the target collocation scheme is updated to be an in use state, and the image is inferred according to the target collocation scheme. When the inferring of the image is completed, the usage state of the hardware accelerator is updated from the in use state to be the idle state. Once an actual time duration of inferring the image is obtained, the estimated time duration is updated to be the actual time duration.