Compressed Data Layout for Optimizing Random Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data compression in file systems often results in internal fragmentation due to varying compression rates, leading to wasted unused space as some compression groups are not fully utilized, causing inefficiencies in storage and access.
Innovation Solution
A system and method that manages compressed data by partitioning data blocks, maintaining an in-memory table to track compression, and reallocating space as needed to minimize internal fragmentation, ensuring that compression entities are fully utilized and optimizing random file access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is performed on fixed-size compression groups, then storage space is reduced, but internal fragmentation occurs resulting in wasted unused space
Solution Approach 1:
The patent divides compression groups into multiple partitions, where each partition can be independently managed and tracked. This segmentation allows the system to identify and eliminate internal fragmentation by tracking which partitions are actually used versus which contain unused space, thereby improving storage efficiency while maintaining the benefits of compression.
Solution Approach 2:
The patent implements a tracking mechanism that monitors the usage status of each partition within compression groups. This feedback system identifies partitions with internal fragmentation (unused space) and enables the system to reallocate or consolidate these spaces, resolving the contradiction between maintaining fixed-size compression groups and eliminating wasted storage space.
2Productivity
If different compression ratios are applied to data blocks, then compression efficiency is improved, but internal fragmentation increases causing wasted space in each compression group
Solution Approach 1:
By segmenting compression groups into tracked partitions, the system can apply different compression ratios to different partitions while maintaining visibility into the actual space used. This allows the system to optimize compression efficiency for each partition based on its characteristics without creating unmanageable internal fragmentation across the entire compression group.
Solution Approach 2:
The patent enables dynamic adjustment of compression parameters at the partition level rather than applying uniform compression ratios across entire compression groups. This parameter change approach allows optimization of compression efficiency for different data types while the tracking mechanism ensures that resulting internal fragmentation is identified and managed, resolving the contradiction between compression efficiency and space utilization.
3Quantity of substance
If compression groups are fully utilized to minimize fragmentation, then storage efficiency is improved, but random file access performance deteriorates
Solution Approach 1:
The patent segments compression groups into tracked partitions that can be independently managed. This segmentation allows the system to maintain high storage efficiency by eliminating internal fragmentation across partitions while enabling selective access to specific partitions during random file operations, thereby preserving access performance without sacrificing storage efficiency.
Solution Approach 2:
The tracking mechanism acts as an intermediary between the compressed data structure and the file access operations. It maintains the efficient packed structure of compression groups while providing the metadata needed for fast random access, allowing the system to achieve both high storage efficiency and good access performance by mediating between these two requirements.
Data Source
AI summary
The embodiments described herein relate to managing compressed data to optimize file compression for efficient random access to the data. A first partition of a first data block of a compression group is compressed. The first compressed partition is stored in a first compression entity. An in-memory table is maintained, which includes updating the in-memory table with data associated with an address of the stored compressed first partition. At such time as it is determined that the first compression entity is full, the in-memory table is compressed and written to the first compression entity. Accordingly, the in-memory table, which stores partition compression data, is store with the compression entity.