Distributed Data Storage Compression via Metadata Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently managing I/O access to distributed data objects across multiple node apparatuses, particularly in utilizing storage space effectively and rebalancing data, while maintaining performance and scalability.
Innovation Solution
A data storage system that employs a metadata structure with direct and indirect metadata nodes to manage I/O access, divides data objects into compression units, and modifies block pointers to facilitate efficient compression and decompression, allowing for efficient I/O access and data redistribution across node apparatuses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data objects are distributed across multiple node apparatuses in a cluster system, then system scalability and reliability are improved, but managing I/O access efficiently and maintaining performance becomes more complex
Solution Approach 1:
The data object is divided into multiple compression units, each independently compressed and stored on different node apparatuses. The metadata structure is segmented into root metadata nodes and direct metadata nodes that track compression units separately, enabling independent management of each segment across the distributed cluster.
Solution Approach 2:
A metadata structure acts as an intermediary between I/O access requests and the distributed compression units. The metadata nodes store location information and manage the mapping between logical data blocks and physical compression units across nodes, simplifying access management without requiring direct knowledge of data distribution.
2Quantity of substance
If data compression is applied to increase storage efficiency, then storage capacity utilization is improved, but I/O access performance and processing time may deteriorate
Solution Approach 1:
The data object is divided into multiple compression units that can be independently compressed and accessed. This segmentation allows selective decompression of only the required portions rather than entire data objects, reducing I/O access time while maintaining high storage efficiency through compression.
Solution Approach 2:
Compression units are pre-compressed and stored before I/O access occurs. The metadata structure is prepared in advance with location information for each compression unit, enabling rapid retrieval and selective decompression only when needed, thus minimizing access time overhead.
3Quantity of substance
If block pointers are modified to reference compressed units, then storage efficiency is improved, but the complexity of managing metadata and data location mapping increases
Solution Approach 1:
The metadata structure is segmented into hierarchical levels with root metadata nodes and direct metadata nodes. Each node manages a specific subset of compression units, distributing the mapping complexity across multiple manageable units rather than requiring a single complex mapping structure.
Solution Approach 2:
Direct metadata nodes store location information for only the compression units they are responsible for, rather than maintaining complete system-wide mappings. This partial action approach reduces the complexity burden on individual metadata nodes while collectively providing complete data location tracking.
4Productivity
If data is rebalanced and redistributed across node apparatuses, then system performance and load distribution are improved, but maintaining data location tracking and I/O access efficiency becomes more difficult
Solution Approach 1:
Before data redistribution, the metadata structure records the current locations of all compression units. During rebalancing, only the metadata location information needs to be updated rather than reorganizing the entire data structure, enabling efficient load distribution while maintaining tracking efficiency.
Solution Approach 2:
The metadata structure serves as an intermediary layer that abstracts the physical location of compression units from I/O access operations. When data is rebalanced across nodes, the metadata nodes are updated to reflect new locations, while I/O access continues to operate efficiently through the metadata abstraction without needing to know physical data locations.
Data Source
AI summary
The present disclosure relates to storing a data object to one or more storage devices of the data storage system in units of data blocks; storing a metadata structure for the data object including one or more direct metadata nodes, and optionally including a root metadata node and optionally further including one or more indirect metadata nodes, each direct metadata node including block pointers referencing respective data blocks of the respective data object; dividing the data object into plural compression units; compressing each compression unit of the plural compression units to a respective compressed unit associated with the respective compression unit; modifying, for each compression unit, block pointers of the direct metadata node referencing respective data blocks of the respective compression unit on the basis of the associated compressed unit; and managing I/O access to the data object based on the metadata structure of the data object.