Data Partition Splitting via Precomputed Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scale out techniques for distributing workload across multiple nodes in processing environments face inefficiencies due to high latency and increased workload when splitting processing responsibility for data sets, as existing methods require nodes to scan and identify data partitions, exacerbating the overburdened node condition.
Innovation Solution
Implementing a logical partitioning scheme where data partitions are defined beforehand, allowing new nodes to access storage units without physically moving data, thereby reducing the burden of split operations by using partition keys to isolate and manage separate storage units for each partition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a node scans and identifies data partitions to split processing responsibility, then the split operation can be performed, but the latency increases and the node becomes overburdened
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing partition key information in a partition map during data ingestion or storage operations. When a split operation is needed, the partition map is already prepared with all necessary partition key to storage location mappings, eliminating the need for time-consuming scanning and identification during the actual split operation.
Solution Approach 2:
The system creates and maintains a partition map that is a copy or representation of the actual data partitioning structure. This partition map contains simplified metadata about partition keys and their locations, allowing the system to query partition information without accessing or scanning the actual data partitions, thus reducing latency and node burden.
2Productivity
If a node scans and identifies data partitions to split processing responsibility, then the split operation can be performed, but the workload on the node increases
Solution Approach 1:
The partition map is pre-computed and maintained during normal data operations, so when a split is required, the node can immediately query the pre-prepared partition map without performing additional scanning or identification workloads on the actual data partitions.
Solution Approach 2:
The partition map acts as an intermediary layer between the node and the actual data partitions. Instead of the node directly scanning and identifying partitions in the data storage, it queries the partition map which has already organized the partition information, thereby reducing the harmful workload on the processing node.
3Adaptability or versatility
If data partitions are not pre-defined and stored separately, then storage flexibility is maintained, but split operations require extensive data scanning
Solution Approach 1:
The system segments the data storage into distinct partitions based on partition keys, with each partition stored in separate storage units. The partition map maintains metadata about these segmented partitions, allowing the system to maintain both storage flexibility through separate partition storage and reduced scanning complexity through the organized partition map structure.
Data Source
AI summary
Processing responsibility for a data set may be split according to separately stored partitions of the data set. A data set may be stored in different storage units that separately store different partitions of the data set. Mapping information may be maintained indicating the assignment of a node to process access requests to multiple partitions of a data set. A split event may be detected for the node, and one or more partitions assigned to the node may be reassigned to a different node. The storage units for the reassigned partitions may be made accessible to the different node. Then mapping information can be updated to indicate change in assignment from the original node to the different node so that the different node can begin processing access requests for the reassigned partitions.