Graph-Based Program Subset Visualization and Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph-based computation systems face inefficiencies in execution on computer servers, including difficulty in reconfiguring data parallelism, balancing load, and slow startup times due to unnecessary process initiation, and they struggle to adapt to varying computation resources and data characteristics.
Innovation Solution
A method for processing graph-based programs that involves rendering a graph-based program specification to identify execution sets, generating prepared code for subsets, and dynamically distributing processing tasks among servers, allowing for efficient parallel execution and adaptation to changing resources and load variations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If processes are initiated at the start-up of graph execution, then computation tasks can be performed, but startup time increases and memory is wasted
Solution Approach 1:
The system performs preliminary actions by pre-starting worker processes before graph execution begins. These worker processes are initialized and ready to receive tasks in advance, eliminating the need to start processes during execution startup. This preliminary preparation reduces startup time and allows immediate processing when data becomes available.
Solution Approach 2:
The computation system is segmented into independent worker processes that can be managed separately. Each worker process handles specific graph components, allowing selective initialization and execution. This segmentation enables the system to start only the necessary worker processes rather than all processes at once, reducing both startup time and memory waste.
2Productivity
If multiple processes are initiated to handle graph components, then parallelism is achieved, but memory consumption increases
Solution Approach 1:
The system dynamically adjusts the number and state of worker processes based on the actual execution needs. Worker processes can be started, stopped, or put into idle states as graph execution progresses. This dynamic management allows the system to maintain parallelism when needed while minimizing memory consumption when full parallelism is not required, optimizing the balance between productivity and resource usage.
3Productivity
If graph configuration is fixed at compilation time, then execution is efficient, but adaptability to resource changes is reduced
Solution Approach 1:
The graph execution system is designed to be dynamic and adaptable to changing resources. The coordinator can modify graph execution parameters, start or stop worker processes, and reconfigure data flow paths during execution based on available resources. This dynamic capability allows the system to maintain execution efficiency while adapting to resource changes, resolving the contradiction between fixed configuration efficiency and adaptability.
Data Source
Figure 1
Figure 2A~2B
Figure 2C
AI summary
User input is received specifying components of a graph-based program specification (628). User input is received specifying links, at least some connecting an output port of an upstream component to an input port of a downstream component. The graph-based program specification is processed to identify one or more subsets (630, 631) of the components, including: identifying one or more subset entry points and one or more subset exit points that occur between components in different subsets based at least in part on data processing characteristics of linked components, and forming the subsets based on the identified subset entry points and exit points. A visual representation of the formed subsets is rendered within a user interface. Prepared code is generated for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.