Data Partitioning for Low-Locality to High-Locality Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Systems processing unstructured data face bottlenecks due to random access patterns when accessing low-locality data stored in external storage devices, leading to high latency and inefficient bandwidth utilization.
Innovation Solution
A method and system that partition unstructured data into high-locality data by recursively loading and partitioning it between on-chip and off-chip storage devices, using a partitioning scheme that maximizes storage space and reduces the number of sequential passes, allowing for sequential access patterns that improve performance and reduce random access costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If unstructured data is stored in external storage device, then storage capacity is improved, but access latency increases due to random access patterns
Solution Approach 1:
The patent divides unstructured data into multiple partitions and reorganizes them into structured formats with improved locality. By segmenting the data storage layout, the system enables sequential access patterns while maintaining external storage capacity, thereby reducing access latency without sacrificing storage capability.
2Adaptability or versatility
If data is accessed randomly from external storage, then data retrieval flexibility is improved, but bandwidth utilization deteriorates
Solution Approach 1:
The patent performs preliminary data reorganization and partitioning operations before actual data processing. By pre-structuring the data with improved locality properties, the system maintains retrieval flexibility for different queries while optimizing bandwidth utilization during access operations, as sequential reads are more efficient than random reads.
3Stability of the object's composition
If complete sorting is applied to unstructured data, then data locality is improved, but processing time increases
Solution Approach 1:
Instead of performing complete sorting on the entire unstructured dataset, the patent applies partial reorganization through partitioning and restructuring operations that achieve sufficient locality improvement. This partial action approach enhances data locality to the extent needed for efficient processing without incurring the excessive time cost of complete sorting.
Data Source
AI summary
A system and method for processing unstructured source data is described. Input data having a range of V is loaded from off-chip storage to on-chip storage. The input data is partitioned into P temporary parent partitions via the on-chip storage, where a particular one of the P temporary parent partitions has a range of V/P. The P temporary parent partitions are stored from the on-chip storage to the off-chip storage. The P temporary parent partitions are partitioned for generating P temporary child partitions until the target number of T partitions is generated, where data from of the P temporary parent partitions is source data for recursively loading, partitioning, and storing the source data. An application is configured to access partitioned data from the T partitions for generating an output. The accesses of the partitioned data are sequential read accesses of the off-chip storage.