Inference Server Batching Architecture for Multi-Framework Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning frameworks are tightly integrated with specific platforms, making concurrent use of different frameworks and hardware accelerators difficult, and often lack optimization for efficient inference processing.
Innovation Solution
A multi-layered inference server architecture with ingestion, batching, and execution layers, utilizing high-performance queues and multi-threading to concurrently handle requests from multiple machine learning platforms and hardware accelerators, allowing flexible interaction and improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning frameworks are tightly integrated with specific platforms, then the framework can be optimized for that platform, but concurrent use of different frameworks and hardware accelerators becomes difficult
Solution Approach 1:
The patent introduces an intermediary layer (the inference server architecture with batching layer and execution layer) that sits between the machine learning frameworks and hardware accelerators. This intermediary enables multiple frameworks to communicate with multiple hardware accelerators without direct tight integration, resolving the contradiction by allowing platform optimization through the execution layer while maintaining versatility through the batching layer's ability to route requests to different endpoints.
Solution Approach 2:
The inference server is segmented into distinct layers: the batching layer that handles request aggregation and routing, and the execution layer that performs actual inference. This segmentation allows each layer to be optimized independently - the batching layer for versatility in handling multiple frameworks, and the execution layer for platform-specific optimization - thereby resolving the contradiction between reliability and adaptability.
2Productivity
If machine learning frameworks are not implemented with focus on highly optimized inference, then hardware acceleration is necessary, but the framework defines only a single type of hardware accelerator
Solution Approach 1:
The execution layer is designed with multi-functionality to support multiple types of hardware accelerators (GPUs, FPGAs, ASICs, CPUs) through a unified interface. The batching layer routes inference requests to appropriate endpoints that correspond to different hardware accelerator types, enabling the system to achieve high productivity through optimized inference while maintaining versatility in supporting multiple hardware platforms simultaneously.
3Ease of operation
If inference requests are processed individually, then request handling is simple, but throughput and efficiency are reduced
Solution Approach 1:
The batching layer merges multiple individual inference requests into batches that are then processed together by the execution layer. This combining approach maintains ease of operation at the request interface while dramatically improving throughput by enabling efficient batch processing on hardware accelerators, which are optimized for processing multiple inference requests simultaneously rather than individually.
4Device complexity
If a single machine learning framework is used, then the system is simpler to manage, but concurrent use of different frameworks is difficult
Solution Approach 1:
The inference server acts as an intermediary that abstracts the complexity of managing multiple machine learning frameworks. The batching layer provides a unified interface for submitting inference requests to different frameworks, while the execution layer handles the actual framework-specific processing. This architecture maintains system management simplicity by presenting a consistent interface to users while enabling versatile support for multiple frameworks concurrently in the background.
Data Source
AI summary
An inference server is capable of receiving a plurality of inference requests from one or more client systems. Each inference request specifies one of a plurality of different endpoints. The inference server can generate a plurality of batches each including one or more of the plurality of inference requests directed to a same endpoint. The inference server also can process the plurality of batches using a plurality of workers executing in an execution layer therein. Each batch is processed by a worker of the plurality of workers indicated by the endpoint of the batch.