Parallel Data Processing Optimization via Hardware-Aware Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Optimizing parallel data processing on hardware platforms with multiple processing units is challenging due to the difficulty in designing programming code that efficiently executes tasks in parallel, as execution time varies with hardware architecture, especially influenced by memory access and data transfer speeds.
Innovation Solution
A method that breaks down data into subsets and determines optimal processing divisions by executing programming code on the hardware platform multiple times to measure and calculate performance metrics, allowing for automatic adaptation and generation of optimized code for parallel data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If programming code is manually optimized for parallel execution, then execution efficiency can be improved, but the complexity of code design increases significantly
Solution Approach 1:
The system performs self-optimization by automatically analyzing hardware platform characteristics and generating optimized parallel execution code without requiring manual intervention. The compilation system itself conducts performance measurements and generates optimized code configurations, making the optimization process self-service rather than relying on programmer expertise.
Solution Approach 2:
The system automatically adjusts execution parameters such as the number of processing threads, data partitioning strategies, and memory access patterns based on measured hardware performance characteristics. By dynamically changing these parameters through automated measurement and analysis, the system achieves optimized execution efficiency without manual parameter tuning.
2Speed
If the number of parallel tasks is increased to utilize more processing units, then processing speed improves, but memory access and data transfer bottlenecks worsen
Solution Approach 1:
The system performs excessive measurements by executing test programs multiple times with different configurations to gather comprehensive performance data. This excessive action ensures that the optimal configuration is identified despite the time cost of multiple measurements, ultimately reducing total execution time by avoiding suboptimal configurations.
Solution Approach 2:
The system implements feedback loops where performance measurements from executed programs are used to adjust and optimize subsequent execution configurations. The compilation system measures actual hardware performance, uses this feedback to refine its understanding of the platform characteristics, and generates improved code configurations based on this learned information.
3Reliability
If performance measurements are taken multiple times to ensure accuracy, then measurement reliability improves, but total execution time increases
Solution Approach 1:
The system performs preliminary measurements during the compilation phase rather than during actual program execution. By conducting performance measurements and optimizations in advance, the system ensures measurement reliability without adding time overhead to the critical execution path. The measurements are performed once during compilation, and the results are reused for multiple executions.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The invention relates to a method for optimising the parallel processing of data on a hardware platform comprising at least one calculation unit comprising a plurality of processing units capable of executing a plurality of executable tasks in parallel, wherein all the data to be processed is broken down into subsets of data, a same sequence of operations being carried out on each subset of data. The method of the invention comprises obtaining (50, 52) the maximum number of subsets of data to be processed by a same sequence of operations, and a maximum number of tasks that can be executed in parallel by a calculation unit of the hardware platform, determining (54) at least two processing partitions, each of said processing partitions corresponding to the partition of all the data into a number of data groups, and to the assignment of at least one executable task, capable of executing said sequence of operations, to each subset of data from said data group, and selecting (60, 62) the processing partition that makes it possible to obtain an optimal measurement value depending on a predetermined criterion. Programming code instructions implementing said selected processing partition are then obtained. One use of the method of the invention is the selection of an optimal hardware platform according to a measurement of execution performance.