Sub-track Compression Pattern Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face inefficiencies in managing and accessing large data sets due to the overhead of processing and managing fixed-size allocation units, which can lead to increased metadata complexity and resource utilization inefficiencies, especially when only part of the data is required.
Innovation Solution
Implementing pattern-aware sub-allocation units, where data matching known patterns is represented by pattern identifiers, allowing for selective storage and retrieval of only necessary sub-allocation units, reducing the need for compression/decompression and accessing backend storage, and de-allocating space when all units match patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If fixed-size allocation units are used for data storage, then data management is simplified, but resource utilization efficiency deteriorates due to overhead of processing and managing entire allocation units even when only part of the data is required
Solution Approach 1:
The allocation unit is divided into multiple sub-allocation units, allowing the system to manage and access data at a finer granularity. This segmentation enables processing of only the necessary portions of data rather than entire allocation units, improving resource utilization while maintaining the simplicity of fixed-size unit management through the sub-unit structure.
2Reliability
If all sub-allocation units are stored on backend storage, then data durability is ensured, but access time increases due to the need to access backend storage for every read operation
Solution Approach 1:
Frequently accessed sub-allocation units are pre-loaded into the cache before they are actually needed by the host application. This preliminary action ensures that when data is accessed, it is already available in the faster cache memory rather than requiring a slower backend storage access, thereby reducing access time while maintaining data durability through the cache coherence protocol.
3Quantity of substance
If compression is applied to reduce data storage space, then storage capacity utilization improves, but processing overhead increases due to compression and decompression operations
Solution Approach 1:
Compression is applied selectively to specific sub-allocation units based on their compressibility characteristics and access patterns, rather than uniformly to all data. This local quality approach allows the system to compress only the portions of data that benefit most from compression, reducing storage capacity requirements while minimizing the processing overhead associated with compression and decompression operations.
4Quantity of substance
If pattern identifiers are used to represent repeated data, then storage space is reduced, but metadata complexity increases
Solution Approach 1:
Instead of storing actual repeated data patterns, the system creates compact pattern identifiers that represent these patterns. When data matches a known pattern, only the pattern identifier is stored rather than the full data, significantly reducing storage space requirements. The metadata complexity is managed through efficient pattern matching algorithms that can quickly identify and replace repeated patterns with their corresponding identifiers.
Data Source
AI summary
Allocation of physical storage capacity within a storage array may be managed in standard size allocation units, e.g. 128 kb tracks, and smaller sub-allocation units, e.g. 32 kb quarter tracks. Data associated with sub-allocation units that match a known pattern may be represented by a hint rather than data in storage. The hint may be used to mitigate overhead associated with accessing backend storage. The hint may also mitigate the need for performing compression/decompression of the corresponding sub-allocation unit if the data is compressed. In response to a read or write request associated with an allocation unit organized into sub-allocation units, only those sub-allocation units that are required to service the read or write request and fail to match a pattern are accessed, decompressed or compressed.