Stream Computing DAG Reuse for Duplicate Processing Elimination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Optimizing multiple data processing pipelines in stream computing is challenging due to non-obvious relationships between pipelines, leading to non-performant, non-scalable, and uneconomic operations, particularly in detecting and avoiding duplicate computing.
Innovation Solution
A scheduling system that compares directed acyclic graphs (DAGs) of data processing flows to identify common portions between stream requests and running streams, allowing for selective reuse of common portions and deployment to the same computing machine, thereby eliminating duplicate processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple data processing pipelines are provided in parallel for stream computing, then processing capacity and throughput are improved, but duplicate computing occurs and resource efficiency deteriorates
Solution Approach 1:
The patent merges multiple data processing pipelines by identifying common subgraphs in their DAG representations. When duplicate processing is detected through graph comparison algorithms, the system consolidates redundant computation nodes and edges, allowing multiple pipelines to share common processing logic and data transformation steps, thereby eliminating duplicate computing while preserving parallel processing capabilities.
Solution Approach 2:
The patent creates universal computation nodes that can serve multiple pipelines simultaneously. By abstracting common processing operations into shared graph substructures, a single computation node can fulfill the processing requirements of multiple different data processing pipelines, improving resource utilization and eliminating redundancy.
2Loss of energy
If duplicate computing is avoided through comprehensive pipeline comparison, then resource efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the pipeline comparison task into hierarchical levels: first comparing high-level graph structures (DAG topologies), then progressively examining subgraphs and individual computation nodes. This segmented approach allows the system to quickly identify and eliminate obvious duplicates through coarse-grained comparison, while applying finer-grained analysis only when necessary, thereby reducing overall computational complexity.
Solution Approach 2:
The patent performs preliminary normalization and canonical representation of DAG structures before comparison. By pre-processing the graph data into standardized forms and pre-computing invariant properties, the system prepares the comparison task in advance, enabling faster and more efficient duplicate detection without requiring complex real-time analysis.
3Speed
If stream computing is made continuous and endless for real-time analytics, then responsiveness is improved, but difficulty in detecting and avoiding duplicate computing increases
Solution Approach 1:
The patent creates immutable copies of DAG representations for each stream request and stores them in a registry. These copied graph structures serve as reference templates that can be efficiently compared against new incoming stream requests. The copying approach enables rapid duplicate detection through graph isomorphism checks while maintaining the continuous, real-time nature of stream processing.
Solution Approach 2:
The patent implements a feedback mechanism where the results of graph comparisons are fed back into the stream processing system. When duplicate processing is detected through DAG comparison, the system provides feedback to eliminate redundant stream requests or merge them with existing processing pipelines, thereby maintaining real-time responsiveness while avoiding duplicate computing in continuous operation.
Data Source
AI summary
Methods, systems, and computer-readable storage media for receiving a first stream request, providing a first DAG representative of a data processing flow of the first stream request, comparing one or more DAGs representative of respective data processing flows of one or more running streams in a set of running streams to provide a set of comparison results, each running stream being processed by a computing machine of the stream computing system, and determining that the first stream request has a common portion with respect to a first running stream based on a first comparison result of the comparison results and, in response: providing the first stream request and the first comparison result to a computing machine that is processing the first running stream, the computing machine executing a stream merger that merges a non-common portion of the first stream request with the common portion within the first running stream.