Multi-level Spatial Data Colocation on MapReduce
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional MapReduce frameworks fragment logically colocated spatial data, leading to significant data movement costs during spatial analytics operations due to physical fragmentation across disks.
Innovation Solution
Implement multi-level colocation and processing of spatial data by correlating and colocating spatial and attribute data within a file system based on a data block placement policy, ensuring replicas are clustered across multiple levels of spatial granularity, thereby minimizing shuffle costs and optimizing parallelism across different analytics scales.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is divided into large blocks of fixed sizes and randomly distributed across nodes, then data storage capacity is improved, but spatial data colocation is lost and data movement costs increase
Solution Approach 1:
The patent applies local quality by implementing a data block placement policy that assigns specific spatial blocks to specific nodes based on their spatial characteristics. Instead of random distribution, the system colocation spatially-related data blocks together on the same node, giving each node a specialized local quality that matches its data storage role. This resolves the contradiction by maintaining high storage capacity while eliminating unnecessary data movement for spatial analytics operations.
Solution Approach 2:
The patent implements preliminary action by pre-colocating spatial data blocks and their corresponding attribute data blocks on the same nodes before analytics operations are executed. The system performs data organization and colocation in advance, creating an optimized data layout that anticipates future analytics needs. This preliminary organization eliminates the need for costly data shuffling during query execution, resolving the contradiction between storage efficiency and data movement costs.
2Ease of operation
If spatial data blocks are physically fragmented across disks, then data distribution is improved, but aggregation costs increase for spatial analytics operations
Solution Approach 1:
The patent applies merging by combining spatial data blocks and their corresponding attribute data blocks into unified collocated units on the same nodes. Instead of keeping spatial and attribute data separate and fragmented, the system merges them together in a coordinated manner. This merging maintains easy data distribution across the cluster while enabling efficient local aggregation during spatial analytics operations, thus resolving the contradiction between distribution ease and processing efficiency.
3Adaptability or versatility
If data blocks are randomly distributed across nodes, then system scalability is improved, but network bottlenecks increase during analytics operations
Solution Approach 1:
The patent implements dynamics by creating a flexible data block placement system that can adapt to different spatial analytics workloads while maintaining optimized colocation patterns. The placement policy dynamically determines which spatial and attribute data blocks should be collocated based on their spatial relationships and analytics access patterns. This dynamic approach maintains system scalability by allowing new nodes to be added while preserving the colocation benefits, thereby reducing network bandwidth consumption without sacrificing adaptability.
Data Source
AI summary
Methods, systems, and computer program products for multi-level colocation and analytical processing of spatial data on MapReduce are provided herein. A method includes correlating multiple items of spatial data and multiple items of attribute data within a file system to generate multiple blocks of correlated data; colocating each of the multiple blocks of correlated data on a given node within the file system based on a data block placement policy; and clustering multiple replicas generated for each of the multiple data blocks at multiple levels of spatial granularity within the file system.