Endpoint-Based Inference Server Batching for Multi-Framework Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning frameworks are tightly integrated with specific platforms, making concurrent use of different frameworks and hardware accelerators difficult, and often lack optimization for efficient inference processing.
Innovation Solution
A multi-layered software architecture for an inference server that utilizes a multi-threaded approach with high-performance queues, allowing concurrent interaction with multiple machine learning platforms and hardware accelerators, including GPUs, FPGAs, and ASICs, through a flexible ingestion, batching, and execution layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning frameworks are tightly integrated with specific platforms, then the framework can be optimized for that platform, but concurrent use of different frameworks and hardware accelerators becomes difficult
Solution Approach 1:
The system segments the machine learning platform into distinct worker processes, each dedicated to a specific framework (TensorFlow, PyTorch, etc.). This segmentation allows each worker to be optimized for its specific framework while the overall system can concurrently handle multiple frameworks through the job queue architecture.
Solution Approach 2:
The job queue and worker management system provides universal functionality that can handle multiple different machine learning frameworks. The system defines a common job interface that can accommodate various frameworks, making the infrastructure multi-functional while maintaining framework-specific optimizations in individual workers.
2Productivity
If machine learning frameworks are not implemented with focus on highly optimized inference, then hardware acceleration becomes necessary, but the framework often defines only a single type of hardware accelerator
Solution Approach 1:
The system segments hardware accelerator support into separate worker types, with each worker configured for a specific hardware type (GPU, FPGA, ASIC). This allows the system to maintain highly optimized inference paths for each hardware type while providing versatility to choose from multiple hardware accelerators through the job queue system.
Solution Approach 2:
The system changes the parameter of hardware accelerator type by configuring different workers with different hardware targets. The job queue accepts requests specifying desired hardware types, and the worker selection dynamically adjusts based on hardware availability and job requirements, enabling flexible hardware acceleration without framework limitations.
3Device complexity
If a single-threaded approach is used for inference processing, then the system is simpler to implement, but throughput and efficiency are limited
Solution Approach 1:
The system segments the inference processing into multiple independent worker threads that can execute concurrently. Each worker processes jobs independently, allowing the system to achieve high throughput through parallel processing while maintaining relatively simple individual worker implementations that can be easily understood and maintained.
Solution Approach 2:
The job queue system ensures continuous useful action by maintaining a pool of ready jobs that can be immediately assigned to available workers. This eliminates idle time between job completions and maintains continuous processing throughput, while the asynchronous job submission and processing model keeps the system architecture manageable.
4Ease of operation
If inference requests are processed individually, then request handling is straightforward, but batch processing efficiency is lost
Solution Approach 1:
The system segments request processing into two distinct phases: job creation (individual request handling) and job execution (batch processing). The job queue accepts individual requests and maintains them as discrete job objects, preserving ease of operation. When workers are available, jobs are executed in batches, achieving processing efficiency without complicating the request submission interface.
Solution Approach 2:
The system performs preliminary action by pre-processing and queuing individual requests before execution. Jobs are validated, configured, and placed in the queue individually, maintaining straightforward request handling. The batch processing occurs during the execution phase, where multiple queued jobs are efficiently processed together by workers, achieving both individual request simplicity and batch processing efficiency.
Data Source
AI summary
An inference server is capable of receiving a plurality of inference requests from one or more client systems. Each inference request specifies one of a plurality of different endpoints. The inference server can generate a plurality of batches each including one or more of the plurality of inference requests directed to a same endpoint. The inference server also can process the plurality of batches using a plurality of workers executing in an execution layer therein. Each batch is processed by a worker of the plurality of workers indicated by the endpoint of the batch.