Graph-Based AI Model Deployment for Low-Latency Inference Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI model deployment is complicated by data formatting issues and runtime conflicts, leading to inefficient data flow and high latency, especially in client/server pipelines where pre- and post-processing requirements are not aligned with the AI model's predefined data formats.
Innovation Solution
Convert pre-process, inference, and post-process codes into graph-based pipelines with extensible operators, allowing for dynamic placement and communication optimization to simplify deployment and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional AI model deployment is used with pre-process and post-process code, then the AI model can be deployed with basic functionality, but data formatting issues and runtime conflicts occur leading to complicated deployment and high latency
Solution Approach 1:
The patent merges pre-process code, inference model code, and post-process code into a single unified graph-based pipeline. This integration eliminates the need for separate deployment of multiple code components, resolving runtime conflicts between them and simplifying the deployment process while maintaining all necessary functionality.
Solution Approach 2:
The patent introduces a graph-based intermediate representation as a mediator between the original code and the deployed model. This graph structure serves as a universal intermediary that standardizes data formats and communication protocols, eliminating formatting issues and runtime conflicts that arise from direct code integration.
2Productivity
If conventional AI model deployment is used, then basic inference functionality is provided, but data flow efficiency is poor and latency is high
Solution Approach 1:
The patent implements dynamic optimization of the inference pipeline by representing the model as a graph structure that enables flexible data flow routing. This dynamic representation allows the system to optimize execution paths and reduce unnecessary data transformations, thereby improving inference speed and reducing latency.
3Adaptability or versatility
If code-based pre-process and post-process operations are used, then data formatting is handled, but runtime conflicts and dependencies create deployment complications
Solution Approach 1:
The patent transforms the operational state of pre-process and post-process code into a static graph-based parameter representation. By converting executable code into graph nodes with defined input/output parameters, the system maintains data format compatibility while eliminating runtime conflicts, as the graph structure pre-defines all data flow relationships before deployment.
Data Source
AI summary
Aspects of the invention include systems and methods configured to provide simplified and efficient artificial intelligence (AI) model deployment. A non-limiting example computer-implemented method includes receiving an AI model deployment input having pre-process code, inference model code, and post-process code. The pre-process code is converted to a pre-process graph. The inference model and the post-process model are similarly converted to an inference graph and a post-process graph, respectively. A pipeline path is generated by connecting nodes in the pre-process graph, the inference graph, and the post-process graph. The pipeline path is deployed as a service for inference.