Configurable Data Partitioning for Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data parallelism approaches lack flexibility and efficiency in partitioning data structures, particularly for non-linear data structures and those without random access support, which limits scalability and performance.
Innovation Solution
A system and method for configuring data partitioning that supports pluggable partitioning algorithms, allowing for customizable partitioning techniques through a data partitioning interface and implementation, enabling dynamic partitioning and load balancing by providing operations for creating and managing data partitions, and tracking execution status.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If fixed partitioning schemes are used for data parallelism, then implementation simplicity is improved, but adaptability to different data structures and algorithms deteriorates
Solution Approach 1:
The patent segments the data parallelism system into distinct components: a data partitioning interface, multiple interchangeable data partitioning implementations, and algorithm-specific partitioning strategies. This segmentation allows each component to be developed and optimized independently, maintaining implementation simplicity while enabling adaptability through the modular architecture.
Solution Approach 2:
The patent introduces dynamic partitioning capabilities where the partitioning strategy can be adjusted at runtime based on the specific algorithm and data structure being processed. The system transitions from static, fixed partitioning schemes to dynamic, adaptive partitioning that responds to actual processing needs, resolving the contradiction between simplicity and adaptability.
2Ease of operation
If generic data partitioning approaches are used, then ease of operation is improved, but processing efficiency for specific algorithms deteriorates
Solution Approach 1:
The patent applies local quality by allowing different partitioning strategies to be used for different algorithms and data structures. Each algorithm can specify its optimal partitioning approach (e.g., range partitioning for arrays, stripe partitioning for distributed data), ensuring that the partitioning method is locally optimized for the specific processing task while maintaining a unified interface for ease of operation.
Solution Approach 2:
The system enables parameter changes by allowing the partitioning strategy, granularity, and organization to be adjusted based on the specific algorithm requirements. The data partitioning interface accepts parameters that can be modified to optimize processing efficiency for different scenarios while preserving the same operational interface.
3Device complexity
If static partitioning is used, then system complexity is reduced, but load balancing capability deteriorates
Solution Approach 1:
The patent introduces dynamic partitioning that can adjust data distribution during runtime based on processing progress and load conditions. The system monitors execution status and redistributes data partitions dynamically to achieve better load balancing without requiring complex manual configuration, thus maintaining relatively simple system architecture while improving adaptability.
Solution Approach 2:
The system implements feedback mechanisms where execution status is tracked and used to inform subsequent partitioning decisions. The data partitioning interface receives feedback about processing progress and uses this information to dynamically adjust partition assignments, enabling effective load balancing through a relatively simple feedback-driven approach.
Data Source
Figure 1~2
Figure 3~5
Figure 4
AI summary
A data partitioning interface provides procedure headings to create data partitions for processing data elements in parallel, and for obtaining data elements to process, without specifying the organizational structure of a data partitioning. A data partitioning implementation associated with the data partitioning interface provides operations to implement the interface procedures, and may also provide dynamic partitioning to facilitate load balancing.