Neural Network Subgraph Partitioning for Asynchronous Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, particularly neural networks, face inefficiencies when input features are available at different times, leading to increased latency in inference processing, especially in natural language processing systems where data resolution is asynchronous.
Innovation Solution
Implementing machine learning model architectures that allow asynchronous, incremental inference by partitioning the network into subgraphs that can process early-arriving features independently, persisting intermediate states, and resuming inference when all features are available, thereby reducing latency and enabling parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the machine learning model waits for all input features to be available before performing inference, then the model can process complete data and achieve accurate predictions, but the critical path latency increases and processing efficiency decreases
Solution Approach 1:
The patent partitions the machine learning model into multiple subgraphs that can be independently evaluated. Each subgraph processes a subset of input features asynchronously, allowing inference to proceed in parallel segments rather than waiting for all features to be available sequentially. This segmentation enables early-arriving features to be processed immediately while later features are handled by their respective subgraphs, reducing overall latency without sacrificing prediction accuracy.
2Device complexity
If the model processes features synchronously in sequence, then the architecture remains simple and easy to implement, but processing efficiency and productivity are reduced
Solution Approach 1:
The patent implements dynamic feature processing where subgraphs are evaluated asynchronously based on feature availability rather than following a fixed synchronous sequence. The system adapts to the arrival patterns of different input features, allowing early-arriving features to trigger immediate subgraph evaluation while later features are processed when they become available. This dynamic approach significantly improves processing efficiency without introducing substantial architectural complexity.
3Adaptability or versatility
If the model uses sophisticated architectures with multiple layers and subgraphs, then processing capability and pattern recognition improve, but critical path latency increases due to sequential processing requirements
Solution Approach 1:
The patent divides sophisticated multi-layer architectures into multiple independent subgraphs that can be evaluated in parallel. Each subgraph handles specific feature subsets and can be processed independently, allowing complex pattern recognition capabilities to be maintained while eliminating sequential processing bottlenecks. This segmentation enables sophisticated architectures to process features asynchronously, reducing critical path latency even as model complexity increases.
Data Source
AI summary
Devices and techniques are generally described for incremental, asynchronous machine learning inference. In various examples, first feature data generated from a first process during a first time period may be input into a first subgraph of a neural network. Second feature data generated from the first process during a second time period may be input into a second subgraph of the neural network. First intermediate data may be generated by the first subgraph based on the first feature data. Second intermediate data may be generated by the second subgraph based on the second feature data. First output data may be generated by the neural network based on the first intermediate data and the second intermediate data.