Flowlet-Based Distributed Data Processing for MapReduce Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current high-performance computing systems lag behind the increasing size of data sets in terms of processing efficiency, particularly due to limitations in MapReduce techniques that lead to load imbalances and excessive disk usage.
Innovation Solution
The implementation of 'flowlets' in a distributed computing system, which utilize user-defined functions and flow control mechanisms to manage data processing, including throttling and state preservation, to improve data flow and reduce overheads through key/value pair aggregation and intra-node load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If MapReduce techniques are used to process large data sets, then data processing capability is improved, but load imbalance and excessive disk usage occur
Solution Approach 1:
The patent segments the MapReduce workflow into flowlets (fine-grained units of work) that can be executed independently and asynchronously. This segmentation allows partial results to be computed and stored in memory before being aggregated, reducing the need for intermediate disk storage and improving overall processing efficiency.
Solution Approach 2:
The patent implements preliminary action by computing partial results in memory during the map phase before the reduce phase begins. Flowlets perform preliminary computations and store intermediate results in memory buffers, avoiding the need to write all intermediate data to disk as required by traditional MapReduce.
2Productivity
If MapReduce techniques are used to process large data sets, then data processing capability is improved, but load imbalance occurs
Solution Approach 1:
The patent introduces dynamic load balancing through flowlet execution, where the system can dynamically allocate and execute flowlets across available compute nodes based on current system state. This dynamic approach allows the system to adapt to varying workloads and maintain balance, unlike the static task allocation in traditional MapReduce.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors flowlet execution progress and system state, then uses this information to dynamically adjust resource allocation and flowlet scheduling. This feedback loop enables automatic load balancing without manual intervention, resolving the load imbalance problem in traditional MapReduce systems.
3Reliability
If traditional MapReduce is used, then data processing is achieved, but synchronization overhead and throughput are limited
Solution Approach 1:
The patent employs periodic action through checkpointing mechanisms where flowlets periodically save their state to persistent storage. This allows the system to recover from failures without losing significant progress, maintaining reliability while enabling more aggressive parallelization and asynchronous execution that improves throughput.
Solution Approach 2:
The patent achieves continuity of useful action by allowing multiple flowlets to execute concurrently and asynchronously rather than waiting for sequential completion. The system can continue processing new data while previous flowlets are still executing, eliminating idle time and improving overall throughput while maintaining correctness through proper state management.
Data Source
AI summary
A data processing task may be implemented in a distributed computing system by the use of a workflow broken into flowlets that are arranged in a directed acyclic graph between data sources and data sinks. Such an arrangement may include various flow control and/or fault tolerance schemes, among other features.