Container-Level File Compression for Storage and Read Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems lack flexibility in data compression and management of storage resources, particularly in balancing storage efficiency and read performance due to inefficient per-file and per-volume compression methods.
Innovation Solution
Implementing container-level data compression management, allowing a mix of compressed and uncompressed containers within a file system volume, with each container having its own set of compression attributes, and using a heat map to intelligently manage compression based on data access patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If per-file compression is used, then storage space is reduced for individual files, but compression efficiency deteriorates due to insufficient data size in smaller files
Solution Approach 1:
The patent merges multiple files into a container that aggregates their data blocks. This container-level compression combines data from multiple sources to achieve sufficient data size for efficient compression, while still reducing overall storage space. The container acts as an intermediate structure that collects data from multiple files before applying compression.
2Quantity of substance
If per-volume compression is used, then storage efficiency is improved through bulk compression, but flexibility deteriorates as all data must be compressed with the same parameters
Solution Approach 1:
The patent segments the volume into multiple containers, each of which can be compressed independently with different parameters. This segmentation allows selective compression of specific data portions while maintaining the ability to use different compression algorithms and settings for different containers, thus achieving both efficiency and flexibility.
Solution Approach 2:
The patent applies different compression attributes to different containers based on their specific characteristics and access patterns. Hot data containers may use faster compression algorithms while cold data containers use more aggressive compression, allowing local optimization rather than uniform treatment across the entire volume.
3Adaptability or versatility
If compression parameters are changed after volume compression is enabled, then adaptability is improved, but system complexity increases due to volume destruction and recreation
Solution Approach 1:
The patent implements dynamic container-level compression where each container can have its compression parameters independently adjusted without affecting other containers. This dynamic approach allows parameter changes to be applied selectively to individual containers rather than requiring complete volume recreation, reducing system complexity while maintaining adaptability.
4Quantity of substance
If all data on a volume is compressed, then storage footprint is reduced, but read performance deteriorates due to decompression overhead
Solution Approach 1:
The patent applies compression selectively to specific containers based on their access patterns. Cold data containers that are rarely accessed are compressed to reduce storage footprint, while hot data containers that are frequently accessed remain uncompressed or use lighter compression to maintain read performance. This local quality approach optimizes both storage and performance for different data portions.
Data Source
AI summary
A file system volume's space is internally allocated as a set of containers, with each container corresponding to a different subset of storage clusters. The set of containers includes a first container corresponding to a first subset of storage clusters storing first data that is compressed according to a first set of compression attributes, a second container corresponding to a second subset of storage clusters storing second data that is uncompressed, and a third container corresponding to a third subset of storage clusters that are free. Based on determining that the second data of the second container is to be compressed, compressed second data is created based on a second set of compression attributes. The second set of compression attributes is different than the first set of compression attributes. The compressed second data is written to the third subset of storage clusters of the third container.