User Defined Data Partitioning for Parallel Computing Co-location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current parallel computing and data management systems face performance bottlenecks due to improper data partitioning across computer cluster nodes, leading to a mismatch between computation and data, which is not addressed by traditional methods that focus on general-purpose parallel data access without considering application-level semantics.
Innovation Solution
The implementation of User Defined Data Partitioning (UDP) keys that generate partition keys based on application-level semantics and computation models, enabling data partitioning consistent with the computation model and accommodating data dependency graphs for efficient co-location of computation and data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional data partitioning methods are used, then general-purpose parallel data access is supported, but application-level semantics and computation models are not considered, leading to mismatch between computation and data
Solution Approach 1:
The patent applies local quality by customizing data partitioning strategies to match specific application-level semantics and computation models. Instead of using a one-size-fits-all partitioning approach, the system identifies and applies partitioning keys that are locally relevant to each application's computational requirements, ensuring that data is partitioned in a way that aligns with the specific computation model being executed.
Solution Approach 2:
The system changes the partitioning parameters dynamically based on the computation model and application semantics. By adjusting partitioning keys, data distribution strategies, and allocation parameters according to the specific computational workload, the system achieves optimal data-computation co-location while maintaining versatility across different applications.
2Productivity
If data is partitioned without considering computation model, then data management is simplified, but data flow increases and performance deteriorates
Solution Approach 1:
The system performs preliminary data partitioning and co-location actions before computation begins. By pre-partitioning data according to the computation model and allocating data to appropriate computational nodes in advance, the system eliminates the need for extensive data movement during execution, thereby reducing data flow overhead and improving access performance.
Solution Approach 2:
The patent introduces an intermediary data partitioning layer that mediates between data storage and computation execution. This intermediary layer intelligently routes and partitions data based on computation requirements, acting as a buffer that reduces direct data flow overhead while maintaining high productivity through optimized data access patterns.
3Speed
If computation and data are not co-located, then system flexibility is maintained, but data access speed decreases
Solution Approach 1:
The system segments data into partitions that can be independently managed and allocated to different computational nodes. This segmentation enables fine-grained control over data-computation co-location, allowing the system to achieve fast data access by placing data near the computations that need it most, while managing complexity through modular partitioning strategies.
Data Source
AI summary
Methods, systems, and computer program products are provided for generating application-aware data partitioning to support parallel computing. A label for a user defined data partitioning (UDP) key is generated by a labeling process to configure data partitions of original data. The UDP is labeled by the labeling process to include at least one key property excluded from the original data. The data partitions are evenly distributed to co-locate and balance the data partitions and corresponding computations performed by computational servers. A data record of the data partitions is retrieved by performing an all-node parallel search of the computational servers using the UDP key.