Machine-Learning Model Services with Local Loopback Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud-based machine-learning models process requests synchronously, leading to latency issues due to the need for input and output data copying through intervening services, which limits processing efficiency, especially in high-volume tasks.
Innovation Solution
Implementing a machine-learning model service that executes an interface component and stream manager application within the same cloud-computing container, utilizing a loopback network interface to bypass network interface hardware, allowing for synchronous and asynchronous data processing without network copying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional synchronous processing with intervening service is used, then request processing is handled sequentially, but latency increases due to data copying through the service
Solution Approach 1:
The patent merges the stream manager application and machine-learning model into the same cloud-computing container, eliminating the intervening service layer. The stream manager directly provides input data to the model via a local communication channel, removing the copying step through external services and reducing latency while maintaining processing throughput.
Solution Approach 2:
The patent introduces a local communication channel as an intermediary mechanism that bypasses the traditional network interface hardware. This local channel enables direct data transfer between the stream manager and model within the same container, eliminating the need for external network copying and reducing processing latency.
2Device complexity
If data copying through intervening service is implemented, then service architecture is simplified, but processing resources are consumed and latency increases
Solution Approach 1:
The patent combines multiple functions (stream management and machine-learning model execution) within a single cloud-computing container, eliminating the need for separate intervening services. This consolidation reduces the number of data copying operations and decreases processing resource consumption while maintaining architectural simplicity.
Solution Approach 2:
The patent extracts the data copying operation from the external network interface and relocates it to a local communication channel within the same container. This extraction eliminates the resource-intensive network copying step while preserving the necessary data transfer function.
3Reliability
If synchronous processing is used, then sequential processing is ensured, but high-volume task efficiency is reduced
Solution Approach 1:
The patent implements a dynamic processing system that can adapt between synchronous and asynchronous modes based on the workload. The stream manager selectively processes instances of input data using synchronous or asynchronous processes, allowing the system to maintain sequence consistency when needed while improving efficiency for high-volume tasks through asynchronous processing.
Solution Approach 2:
The patent segments the processing workflow into distinct synchronous and asynchronous processing paths within the same container. This segmentation allows the system to handle different types of requests appropriately - maintaining reliability for critical sequential operations while optimizing productivity for high-volume tasks that can be processed asynchronously.
Data Source
AI summary
Techniques discussed herein include dynamically providing synchronous and/or asynchronous data processing by a machine-learning model service. The machine-learning model service (“the service”) executes a stream manager application, a web interface, and a machine-learning model via a common container. The stream manager application can obtain input data (e.g., from an input data stream, a partition of an input data stream, etc.) and provide the data to the machine-learning model through the web interface using a local communication channel (e.g., a loopback interface that bypasses local network interface hardware of the computing device on which the model executes). Prediction results from the model may be provided as output data (e.g., to an output data stream, to a partition of an output data stream, etc.).