Dynamic Dataflow Graph Loading for Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies in processing data flows due to high startup times for dataflow graphs, especially when handling frequent batches of data, as they require loading and initializing multiple graphs concurrently, which wastes resources and reduces overall efficiency.
Innovation Solution
The system dynamically loads and unloads compiled dataflow graphs into a running container graph, allowing only the necessary graphs to be active, and uses micrographs as subgraphs within a containing dataflow graph to process different units of work efficiently, reducing startup times and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple dataflow graphs are loaded and initialized concurrently to handle frequent data batches, then the system can process multiple data types, but resource usage increases and startup time is wasted
Solution Approach 1:
The system dynamically loads and unloads dataflow graphs based on the actual data type being processed. Instead of having all graphs loaded simultaneously, the system activates only the specific graph needed for the current batch of data, making the system configuration dynamic rather than static. This resolves the contradiction by maintaining versatility through on-demand graph selection while reducing resource usage through selective activation.
Solution Approach 2:
The system performs preliminary compilation of dataflow graphs into executable form and stores them in a library for quick retrieval. When a data batch arrives, the pre-compiled graph matching the data type is rapidly loaded into memory, avoiding the need for lengthy initialization at runtime. This preliminary preparation enables fast graph switching without requiring concurrent loading of multiple graphs, thus reducing resource consumption while maintaining the ability to handle multiple data types.
2Adaptability or versatility
If multiple dataflow graphs are loaded and initialized concurrently to handle frequent data batches, then the system can process multiple data types, but startup time increases
Solution Approach 1:
Dataflow graphs are pre-compiled into executable form and stored in a library during system setup. When a data batch arrives, the matching pre-compiled graph is rapidly loaded from storage into memory, bypassing the lengthy compilation and initialization process at runtime. This preliminary preparation dramatically reduces startup time while maintaining the capability to handle multiple data types through selective graph loading.
Solution Approach 2:
The system implements dynamic graph loading that activates only the specific dataflow graph required for the current data type being processed. This dynamic approach eliminates the need to load multiple graphs concurrently, reducing startup time by focusing resources on loading a single necessary graph while maintaining versatility through on-demand graph selection based on data type analysis.
3Productivity
If dataflow graphs are loaded into memory for processing, then processing speed improves, but memory usage increases
Solution Approach 1:
The system dynamically manages memory allocation by loading dataflow graphs into memory only when needed for processing specific data batches and unloading them when no longer required. This dynamic memory management allows the system to maintain high processing speed by having graphs in memory during active use while minimizing overall memory usage by removing graphs when they are not currently needed, thus resolving the contradiction between speed and memory consumption.
4Adaptability or versatility
If compiled dataflow graphs are stored in data storage system for dynamic loading, then system flexibility improves, but access time may increase
Solution Approach 1:
Dataflow graphs are pre-compiled into executable form and stored in an organized library structure during system setup. This preliminary compilation creates optimized executable representations that can be rapidly retrieved and loaded into memory when needed. The pre-compilation step, though performed beforehand, enables very fast loading times during operation, thus maintaining system flexibility through stored graph variants while minimizing the actual loading time when graphs are accessed.
Data Source
Figure 1~2
Figure 3A~3B
Figure 4
AI summary
Processing data includes: receiving units of work that each include one or more work elements, and processing a first unit of work using a first compiled dataflow graph (160) loaded into a data processing system (100) in response to receiving the first unit of work. The processing includes: analysis to determine a characteristic of the first unit of work; identifying one or more compiled dataflow graphs from graphs stored in a data storage system (107) that include at least some that were compiled for processing a unit of work having the determined characteristic; loading one of the identified compiled dataflow graphs into the data processing system (100) as the first compiled dataflow graph (160); and generating one or more output work elements from at least one work element in the first unit of work.