Streaming Graph Optimization via Subgraph Disassembly
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream computing platforms face issues with imbalanced computing logic and redundant flow transmission in user-compiled streaming graphs, leading to poor resource utilization and execution performance, particularly for users lacking stream application development experience.
Innovation Solution
A streaming graph optimization method that disassembles user-provided streaming graphs into subgraphs based on the maximum atom division principle and performs adjacency operator combination to balance processing logic complexity, reducing redundant flow transmission and enhancing resource utilization and execution performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users compile streaming graphs without professional stream application development experience, then the system is easy to operate, but the computing logic becomes imbalanced and execution performance deteriorates
Solution Approach 1:
The system performs preliminary optimization actions by automatically analyzing the streaming graph structure before execution, identifying imbalanced computing logic and redundant flow transmissions, and applying optimization transformations to balance the computing logic across operators, thereby improving execution performance without requiring user expertise
Solution Approach 2:
The streaming graph optimization apparatus enables the system to self-optimize by automatically detecting imbalanced computing patterns and applying corrective transformations without human intervention, allowing the system to serve itself in optimizing its own performance while maintaining ease of operation for users
2Adaptability or versatility
If users manually design streaming graphs, then flexibility and adaptability are improved, but device complexity and time consumption increase
Solution Approach 1:
The system automatically analyzes the streaming graph structure and identifies optimization opportunities without requiring manual user intervention, enabling the system to self-diagnose and self-optimize its own configuration, thereby reducing the complexity burden on users while preserving design flexibility
3Productivity
If streaming graphs are optimized through automatic analysis and transformation, then execution performance is improved, but the complexity of the optimization process increases
Solution Approach 1:
The optimization process is segmented into distinct automated stages: parsing the streaming graph into operators and data streams, analyzing the graph structure to identify imbalanced computing logic, transforming the graph to balance operator complexity, and deploying the optimized graph. This segmentation automates each step, improving performance while containing the complexity within the system rather than transferring it to the user
Solution Approach 2:
The streaming graph optimization apparatus acts as an intermediary between the user's original streaming graph and the execution engine, automatically performing analysis and transformation operations. This intermediary handles the complexity of optimization internally, allowing users to maintain simple graph designs while achieving optimized execution performance
4Ease of manufacture
If streaming graphs contain redundant flow transmission, then ease of compilation is maintained, but resource utilization and execution efficiency deteriorate
Solution Approach 1:
The system performs preliminary analysis of the streaming graph structure before deployment, identifying redundant flow transmissions between operators and applying transformations to eliminate these redundancies, thereby optimizing resource utilization before the application begins execution without affecting the ease of compilation
Data Source
Figure 1~2A
Figure 2B
Figure 3
AI summary
A streaming graph optimization method and apparatus are disclosed, relating to the stream processing field. A stream application streaming graph provided by a user is received (S601); the streaming graph is parsed and a streaming graph described by an operator node and a data stream side is constructed (S602); the streaming graph is disassembled according to a maximum atom division principle, so as to obtain at least one streaming subgraph (S603); adjacency operator combination is performed on the at least one streaming subgraph according to a combination algorithm, so as to obtain an optimized streaming graph (S604). By disassembling and combining the streaming graph provided by the user, balanced streaming graph computing logic is implemented, redundant flow transmission is reduced, and stream system resource utilization and stream application execution performance are improved.