Selective Metadata Compression for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in metadata storage due to indiscriminate compression techniques, which lead to suboptimal resource utilization and operational inefficiencies, especially when dealing with varying compressibility of metadata, such as marker data and cryptographic hash functions like SHA-1.
Innovation Solution
Implementing an intelligent compression method that selectively compresses metadata based on predetermined conditions, such as metadata segment size, marker density, and available processing resources, using a deduplication engine and segment identifier to determine suitable candidates for compression and apply appropriate algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If indiscriminate compression techniques are applied to all metadata, then storage space is reduced, but resource utilization becomes suboptimal and operational efficiency decreases
Solution Approach 1:
The system changes the parameter of compression application from universal to selective based on metadata characteristics. It evaluates compressibility parameters and applies compression only when parameters indicate suitability, transforming the compression process from a fixed operation to a dynamic, condition-based operation that optimizes both storage space and operational efficiency.
Solution Approach 2:
The system applies different compression treatments to different portions of metadata based on their individual compressibility characteristics. Instead of uniform compression, it identifies specific metadata segments that benefit from compression and applies compression selectively, while leaving other metadata uncompressed, thereby optimizing resource utilization.
2Quantity of substance
If compression is applied to all metadata segments, then storage utilization improves, but resource usage increases unnecessarily
Solution Approach 1:
The system performs partial compression action by evaluating each metadata segment and applying compression only to those that meet compressibility criteria. This partial action approach avoids the excessive resource consumption of compressing all metadata while still achieving significant storage utilization improvement through selective compression of suitable segments.
Solution Approach 2:
The system uses compressibility parameters to dynamically determine whether compression should be applied to each metadata segment. By changing the parameter from binary (compress or not) to conditional (compress if parameters indicate suitability), the system optimizes the balance between storage utilization and processing resource usage.
3Quantity of substance
If metadata size increases due to inline markers, then storage space requirement increases, but deduplication efficiency should be maintained
Solution Approach 1:
The system applies compression to metadata segments containing inline markers when compressibility parameters indicate suitability. This parameter-based selective compression reduces the storage space requirement for marker-containing metadata while maintaining deduplication efficiency by preserving the structural integrity and deduplication-friendly characteristics of the compressed metadata.
Data Source
AI summary
Data segments and metadata segments to be stored in a storage system, where the data segments are deduplicated segments and each of the metadata segments includes a fingerprint for the corresponding data segment, for each of the metadata segments. It is determined that the metadata segment contains one or markers inserted by a client device of the storage system. The metadata segment is examined to determine whether the metadata segment satisfies a predetermined condition. In response to determining that the metadata satisfies the predetermined condition, the metadata segment is compressed using a predetermined compression algorithm. The compressed metadata segment is stored in the storage system, otherwise the metadata segment is stored in the storage system without compression. Thereafter, the data segments are stored in the storage system.