Machine Learning Data Reduction in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face inefficiencies in managing data reduction due to resource constraints, as they require extensive CPU resources and cache space for deduplication and compression processes, often retaining non-compressible data blocks until they meet a certain threshold, leading to unnecessary resource usage.
Innovation Solution
A machine learning system is trained to assess the compressibility and dedupability of data blocks using entropy values, allowing for quick determination and optimization of cache space by storing entropy values instead of hashes and deleting them once the assessment is made, thereby reducing CPU resources and storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data storage systems perform data reduction using software compression and deduplication, then storage efficiency is improved, but CPU resources and cache space are excessively consumed
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing entropy values for data blocks before actual compression or deduplication operations. The system calculates entropy values in advance and stores them in a cache, allowing rapid determination of data reducibility without performing full compression or deduplication processes, thereby significantly reducing CPU resource consumption during actual data operations.
Solution Approach 2:
The patent introduces entropy values as an intermediary between raw data and the compression/deduplication processes. Instead of directly performing resource-intensive compression or deduplication operations, the system uses entropy values as a mediator to quickly assess data reducibility, enabling efficient decision-making about which data blocks to compress or deduplicate, thus reducing overall CPU resource usage.
2Manufacturing precision
If data blocks are retained in cache until they meet compression threshold, then compression quality is improved, but cache space is unnecessarily consumed
Solution Approach 1:
The patent extracts the essential characteristic of data reducibility by calculating and storing only entropy values in the cache, rather than retaining entire data blocks. This extraction approach allows the system to determine compression potential without consuming cache space proportional to the actual data block size, significantly reducing cache space requirements while maintaining the ability to assess compression quality.
Solution Approach 2:
The patent uses entropy values as temporary, disposable objects that serve their purpose quickly. Entropy values are calculated, used to determine data reducibility, and then discarded after the assessment is complete. These lightweight entropy values consume minimal cache space compared to retaining actual data blocks, enabling the system to maintain compression quality assessment capability with much smaller cache requirements.
3Measurement precision
If full data reduction processing is performed on all data blocks, then data reduction accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by performing only the necessary entropy calculation and assessment steps rather than executing complete compression or deduplication processes on all data blocks. The system calculates entropy values to assess data reducibility and applies data reduction only to blocks that meet certain criteria, avoiding unnecessary processing of blocks that would not benefit from compression or deduplication, thus reducing overall processing time while maintaining adequate reduction accuracy.
Data Source
AI summary
A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.