Unified Inference Engine for Multi-Model Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Configuring, deploying, and managing multiple machine learning models (MLMs) with different architectures and functionalities for processing complex data, including streaming data, is a challenging task that requires significant expertise and effort from code developers.
Innovation Solution
The use of an inference engine that enables concurrent inference execution of multiple MLMs through parallel, sequential, or batch execution, utilizing various computational platforms such as CPU, GPU, or combined platforms, and providing a user API for high-level command execution to simplify deployment and data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple machine learning models with different architectures are deployed for processing complex data, then the processing capability and functionality are improved, but the deployment complexity and management difficulty increase
Solution Approach 1:
The patent implements a unified inference engine that can execute multiple machine learning models with different architectures (decision trees, neural networks, etc.) through a common interface. The engine provides universal support for various model types, allowing diverse models to be deployed and managed through the same platform, thereby improving versatility while controlling deployment complexity through standardization
2Productivity
If concurrent inference execution of multiple models is implemented, then the productivity and processing efficiency are improved, but the computational resource requirements and system complexity increase
Solution Approach 1:
The patent merges multiple model execution capabilities into a single inference engine that can concurrently execute multiple models. By combining model loading, execution, and management functions into one unified system, the patent achieves high processing efficiency through concurrent execution while avoiding the complexity that would arise from managing separate execution environments for each model
Solution Approach 2:
The inference engine acts as an intermediary layer between the user API and the actual model executions. This mediator handles the complexity of concurrent execution, resource allocation, and coordination internally, while presenting a simplified interface to users. The engine manages multiple model executions simultaneously through this intermediary layer, improving productivity without exposing system complexity to users
3Ease of operation
If a user API for high-level command execution is provided, then the ease of operation and deployment are improved, but the abstraction layer complexity increases
Solution Approach 1:
The user API serves as an intermediary that abstracts complex inference engine operations into simple high-level commands. The API layer handles model deployment, execution control, and result retrieval through intuitive interfaces, improving ease of operation. The complexity of the abstraction layer is managed by implementing the API as a thin wrapper that translates high-level commands into engine-specific operations, minimizing the complexity burden
Data Source
AI summary
Apparatuses, systems, and frameworks for provisioning of efficient pipelines capable of multi-model inference and data processing, including streaming data applications. The disclosed techniques allow efficient deployment and execution of multiple machine learning using pluggable inference and data processing backends by users without specialized developer experience.