Similarity-Hash Garbage Collection for Live Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in garbage collection and data compression, particularly in scaling up storage capacity and managing large data segments, where similar data blocks are not effectively grouped and compressed, leading to suboptimal storage efficiency and increased computational costs.
Innovation Solution
The implementation of a similarity hash-based approach that identifies and groups similar data blocks, using techniques like Jaccard distance and rolling hashes, to compress and manage data segments more efficiently, allowing for concurrent processing and reduced interaction with logical elements during garbage collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional garbage collection is used without content similarity analysis, then the process is simpler and faster, but storage efficiency is suboptimal because similar data blocks are not grouped and compressed together
Solution Approach 1:
The patent applies preliminary action by computing content similarity hashes and organizing data blocks into similarity groups before the garbage collection process begins. This pre-organization enables efficient identification and compression of similar data blocks during garbage collection, improving storage efficiency without significantly increasing the complexity of the main garbage collection operation.
2Productivity
If data blocks are processed sequentially during garbage collection, then processing is simpler to manage, but productivity is reduced due to inability to perform concurrent compression operations
Solution Approach 1:
The patent segments the garbage collection process by dividing data blocks into independent similarity groups based on content hashing. Each similarity group can be processed concurrently for compression operations, enabling parallel processing that improves productivity. The segmentation is achieved through content-based grouping rather than sequential processing, allowing multiple compression operations to proceed simultaneously while maintaining manageable complexity through modular group processing.
Data Source
AI summary
A storage system performs garbage collection, with data compression, in storage memory. The system obtains hash results from data segments. The system determines similarity of content of data segments, based on the hash results. The system performs data compression of live data of two or more data segments that have similarity of content meeting a similarity threshold. The system writes the compressed live data of the two or more data segments into the storage memory.