Machine-Learning Service Using Local Channels for Flexible Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud-based machine-learning models process requests synchronously, leading to latency issues due to the need for copying data through intervening services, which limits throughput in high-volume request scenarios.
Innovation Solution
Implementing a machine-learning model service that executes an interface component and a stream manager application within the same cloud-computing container, utilizing a local communication channel to bypass network interface hardware, allowing for both synchronous and asynchronous data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous processing is used with conventional services, then data processing can be performed sequentially, but latency increases due to copying data through processing services
Solution Approach 1:
The patent extracts the stream manager application from the conventional service architecture and places it directly within the cloud-computing container alongside the machine-learning model. This eliminates the intermediate processing service layer that causes copying latency, while maintaining reliable data processing through direct local communication channels between the stream manager and model.
Solution Approach 2:
The stream manager application acts as an intermediary component that receives input data and directly transmits it to the machine-learning model through local communication channels. This intermediary eliminates the need for external processing services to copy and forward data, reducing latency while ensuring reliable data flow.
2Ease of operation
If conventional processing services are used to receive and transmit data, then data can be routed through services, but processing resources are consumed by copying operations
Solution Approach 1:
The patent merges the stream manager application and the machine-learning model into the same cloud-computing container, enabling direct local communication between them. This consolidation eliminates redundant data copying operations that would otherwise be performed by external processing services, reducing processing resource consumption while maintaining ease of data routing through the stream manager.
3Reliability
If synchronous processing is used, then requests are processed sequentially, but throughput is limited in high-volume request scenarios
Solution Approach 1:
The patent implements dynamic processing modes that allow the system to switch between synchronous and asynchronous operations. The stream manager can handle multiple input data streams concurrently and process them through the machine-learning model in parallel where possible, increasing throughput while maintaining processing accuracy through the reliable local communication channel.
Solution Approach 2:
The stream manager application continuously receives input data from multiple sources and maintains continuous data flow to the machine-learning model through local communication channels. This eliminates idle waiting periods associated with synchronous request-response cycles, enabling sustained high throughput while preserving processing reliability through the direct communication path.
Data Source
AI summary
Techniques discussed herein include dynamically providing synchronous and/or asynchronous data processing by a machine-learning model service. The machine-learning model service (“the service”) executes a stream manager application, a web interface, and a machine-learning model via a common container. The stream manager application can obtain input data (e.g., from an input data stream, a partition of an input data stream, etc.) and provide the data to the machine-learning model through the web interface using a local communication channel (e.g., a loopback interface that bypasses local network interface hardware of the computing device on which the model executes). Prediction results from the model may be provided as output data (e.g., to an output data stream, to a partition of an output data stream, etc.).