Multi-dimensional Data Partitioning for Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing and storage methods for large data objects are inefficient, as they often ignore structural features and introduce dependencies between data portions, leading to increased network traffic and inefficiencies, especially when handling voluminous multi-dimensional data.
Innovation Solution
A technique that splits data objects into portions at boundaries within the data object, transforming them into segments for independent processing on multiple computing nodes, reducing network traffic and enabling true parallel processing by eliminating the need for communication among nodes to resolve dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data objects are divided into equal-sized portions for distributed storage, then storage capacity is improved, but processing efficiency deteriorates due to dependencies between portions and increased network traffic
Solution Approach 1:
The patent divides data objects into segments based on structural boundaries (such as complete rows in tabular data) rather than equal byte divisions. This segmentation ensures that each segment is independently processable, eliminating dependencies between segments while maintaining distributed storage capacity.
Solution Approach 2:
The patent applies different segmentation strategies to different data types and access patterns. By analyzing the local structure and access requirements of specific data portions, the system creates optimally-sized segments that balance storage distribution with processing independence, rather than applying uniform division rules.
2Ease of manufacture
If data portions are divided at arbitrary byte boundaries, then distribution simplicity is improved, but data integrity deteriorates as rows are cut in the middle requiring access to multiple portions
Solution Approach 1:
The patent performs preliminary analysis of data structure boundaries before division. By identifying structural boundaries (such as row endings in tabular data) in advance, the system ensures that segments are created at meaningful boundaries rather than arbitrary byte positions, maintaining data integrity while enabling distribution.
3Loss of information
If all data portions are gathered to perform processing tasks, then processing completeness is improved, but network traffic increases due to transferring large copies of data
Solution Approach 1:
The patent extracts only the necessary segments needed to complete a processing task rather than gathering all data portions. By determining which segments contain the required data and processing those locally distributed segments, the system maintains processing completeness while minimizing network traffic to only essential data transfers.
4Productivity
If segments are created as individually processable units, then processing parallelism is improved, but segmentation complexity increases requiring boundary identification and transformation
Solution Approach 1:
The patent changes the parameters used for segmentation from simple byte-based divisions to structure-aware divisions based on data type characteristics. By adjusting segmentation parameters according to the specific data format and access patterns, the system achieves independent processable units without excessive complexity in the segmentation logic.
Data Source
AI summary
A technique for managing multi-dimensional data includes providing an original dataset containing data arranged along multiple dimensions, each dimension covering a respective original range of dimensional units. The technique further includes extracting multiple portions of data from the original dataset, each portion extending over a reduced range of dimensional units, smaller than the original range, in at least one dimension, and all extracted portions together covering the original ranges of the original dataset in all dimensions.