Untying Compression Links to Stale Reference Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face inefficiencies in managing links between reference and referring chunks, particularly when reference chunks become stale, leading to increased storage size due to retained data from dependent chunks, which cannot be deleted until all references are resolved.
Innovation Solution
A method and system for selectively untieing compression-related links to stale reference chunks by using similarity hashing and metadata to identify and reassess compression effectiveness based on the number of referring chunks, allowing for the reconstruction of data blocks and updating reference counts to enable deletion of stale reference chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If compression links are maintained to preserve data redundancy and improve compression ratios, then storage efficiency improves, but storage size increases due to retained stale reference chunk data
Solution Approach 1:
The system performs preliminary identification of stale reference chunks using metadata and similarity hashing before attempting deletion. By pre-assessing which chunks are stale and which referring chunks depend on them, the system can proactively untie compression links before deletion, preventing storage size increase while maintaining efficiency.
Solution Approach 2:
The system uses reference counts and similarity hashing to continuously monitor and feedback on the status of reference-chunk relationships. When a reference chunk's reference count drops to zero or below a threshold, the system feedbacks this information to trigger the untying of compression links, dynamically adjusting storage allocation based on actual usage patterns.
2Quantity of substance
If all compression links are untied to enable deletion of reference chunks, then storage size is reduced, but compression effectiveness deteriorates
Solution Approach 1:
The system applies different treatments to different compression links based on local conditions. Instead of universally untieing all links, it selectively unties only those links where the reference chunk is stale and no longer needed, while preserving links where the reference chunk remains useful. This localized approach maintains compression effectiveness for active chunks while reducing storage size for stale chunks.
3Reliability
If reference counts are strictly enforced to prevent deletion of referenced chunks, then data integrity is maintained, but storage utilization decreases
Solution Approach 1:
The system dynamically adjusts the reference count threshold for deletion decisions. Instead of using a fixed threshold of zero, it allows deletion when the reference count falls below a dynamically determined threshold based on chunk staleness, similarity hashing results, and storage policies. This dynamic approach enables deletion of stale reference chunks even with remaining references, improving storage utilization while maintaining data integrity for active chunks.
Data Source
AI summary
A method for selectively untying at least one of compression related links to a stale reference chunk, the method may include determining whether a compression effectiveness condition is fulfilled in relation to a set of referring chunks that are compressed using the stale reference chunk and are linked to the to the stale reference chunk by the compression related links; wherein the compression effectiveness condition is responsive to, at least, a number of the referring chunks of the set; and untying the at least one compression related links to the stale reference chunks when the compression effectiveness condition is not fulfilled.