Popularity-Sorted Data Decompression for Lower Read Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data reduction technologies suffer from read amplification issues during decompression, leading to increased resource consumption and high read bandwidth overheads due to the need to decompress data sequentially, which is exacerbated by constraints like Input/Output Operations Per Second (IOPS).
Innovation Solution
The method involves sorting compressed data based on popularity values, where frequently accessed data is decompressed first, followed by less frequently accessed data, and storing popular data in high-performance memory while less popular data is stored in secondary memory, thereby reducing the average read amplification ratio.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed using combination compression or delta compression, then storage space is reduced, but read amplification occurs during decompression requiring sequential reading of multiple objects
Solution Approach 1:
The patent segments compressed data into multiple independent compressed objects, each containing its own decompression context. This allows selective decompression of individual objects without requiring sequential reading of all preceding objects, thereby reducing read amplification while maintaining storage efficiency.
Solution Approach 2:
The patent performs preliminary organization of compressed data by creating self-contained compressed objects with embedded metadata and context information. This preliminary structuring enables the decompression system to access and decompress specific objects directly without sequential processing, reducing I/O operations during decompression.
2Device complexity
If data is decompressed as a whole to access a single object, then decompression overhead is reduced, but read bandwidth consumption increases due to reading all preceding objects
Solution Approach 1:
The patent extracts and isolates the necessary decompression context and metadata within each compressed object itself, rather than requiring external context from preceding objects. This extraction enables independent decompression of individual objects, reducing read bandwidth overhead while maintaining decompression efficiency.
3Speed
If frequently accessed data is stored in high-performance memory, then access speed is improved, but storage resource utilization decreases due to memory hierarchy constraints
Solution Approach 1:
The patent applies local quality by organizing compressed data into uniformly structured objects that can be selectively loaded into high-performance memory based on access patterns. Each compressed object's self-contained nature allows the system to load only frequently accessed objects into fast memory, improving access speed for hot data while maintaining overall storage efficiency through selective caching.
Data Source
AI summary
In one example method, a server obtains compressed data, where the compressed data includes at least a first part and a second part, and where the compressed data is sorted based on popularity values of parts of the compressed data, and a popularity value of a part of the compressed data is greater than a popularity value of a subsequent part of the compressed data. The server decompresses the first part, and decompresses the second part after decompressing the first part, where a popularity value of the first part is greater than a popularity value of the second part.