Parallel Data Access for Remote-Sensing Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data storage systems for remote-sensing images face challenges in handling binary entity data and text-type attribute data, particularly in large-scale parallel access and processing, due to limitations in handling spatial data and single-point failures, and are inefficient in managing varying data sizes and resolutions.
Innovation Solution
A parallel data access method and system that segments remote-sensing images into grid data blocks, uses a distributed object storage system like Ceph, and balances load by selecting pools with minimum load for data storage, supporting real-time re-projection and resampling, and elastic extension of storage capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed file system with master-slave mode is used for data storage, then storage capacity and parallel access are improved, but single point of failure and inability to handle binary entity data with text attributes occur
Solution Approach 1:
The patent segments the centralized master node into multiple worker nodes that can independently handle metadata operations. Each worker node manages a portion of the metadata, eliminating the single point of failure while maintaining parallel access capabilities. The segmentation allows the system to distribute both data and metadata management across multiple nodes.
2Ease of manufacture
If fixed-size data blocks are used for storage, then storage simplicity is improved, but inability to consider spatial range information and efficient access to remote-sensing image data occurs
Solution Approach 1:
The patent applies local quality by associating spatial metadata with each data block, allowing the storage system to differentiate and optimize access based on spatial information. Each data block carries its own spatial coordinates and range information, enabling efficient queries and access patterns specific to remote-sensing image data while maintaining the simplicity of fixed-size block storage.
3Speed
If same grid system is used for all remote-sensing images, then data retrieval speed is improved, but inability to directly merge and calculate data of different resolutions occurs
Solution Approach 1:
The patent introduces dynamic grid sizing that adapts to the resolution of each remote-sensing image. Instead of using a fixed grid size for all images, the system dynamically determines appropriate grid dimensions based on the specific image resolution and characteristics. This allows fast retrieval within each grid while enabling proper merging and calculation across different resolutions by adjusting the grid parameters accordingly.
Data Source
AI summary
A parallel data access method for massive remote-sensing images includes: 1) segmenting a remote-sensing image to be processed using a set grid system, data in each grid corresponding to a data block; 2) collecting a data access log of an underlying distributed object storage system Ceph in a past period of time, and measuring a load index of each Ceph cluster and a load index of each pool; 3) selecting a pool with a minimum load in a Ceph cluster with a minimum current load to serve as a storage position of a current data block, and writing each data block into a corresponding pool; 4) returning a data identifier dataid and a data access path of the remote-sensing image; and 5) storing metadata of each data block in a metadata database. The method can support rapid and high-concurrency read and write of large-area data of a grid data block.