Workload-Aware Partitioning for FPGA Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the amount of data to be processed increases, existing distributed parallel processing systems face challenges in reducing data analysis time to improve performance, particularly due to variations in processing performance across nodes in a cluster.
Innovation Solution
A workload-aware distributed data processing apparatus and method that utilizes a partition proportion scheme to distribute mapping results across partitions based on processing performance, with hardware acceleration on a field programmable gate array (FPGA), allowing for efficient inter-node parallelization and reduction operations, including sorting and memory buffer management to prevent bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed parallel processing is used to handle large data, then processing capacity increases, but data analysis time increases due to performance variations across nodes
Solution Approach 1:
The patent applies local quality by creating workload-aware partitioning that assigns different proportions of data partitions to different reduce nodes based on their specific processing performance characteristics. Fast nodes receive larger partition proportions while slow nodes receive smaller proportions, optimizing the overall processing time by matching data distribution to node capabilities
Solution Approach 2:
The patent implements dynamics by dynamically determining partition proportions based on measured processing performance of reduce nodes. The system measures actual processing speeds and adjusts the partition distribution strategy accordingly, rather than using static equal partitioning, allowing the system to adapt to varying node performances
2Device complexity
If equal partition distribution is used across reduce nodes, then system simplicity is maintained, but processing efficiency decreases due to performance variations
Solution Approach 1:
The patent applies feedback by measuring the processing performance of reduce nodes and using this information to determine optimal partition proportions. The system continuously monitors processing speeds and adjusts data distribution accordingly, creating a closed-loop control system that optimizes efficiency while maintaining manageable complexity through automated performance-based allocation
Data Source
AI summary
A workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration are provided. The data processing apparatus includes a memory buffer including partitions. The data processing apparatus further includes a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme. The data processing apparatus further includes a reduce node configured to receive content of a corresponding one of the partitions, and perform a reduction operation on the content to generate a reduce result.