ML Compression Algorithm Selection for Binary Data Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face inefficiencies in compression due to unpredictable compressibility of individual blocks or files with specific compression algorithms, leading to wasted processor cycles on poorly compressible data.
Innovation Solution
A compute node with a trained data model predicts the compression efficiency of binary data structures using multiple algorithms based on component size, recommending the most efficient compression algorithm for selected binary data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression is implemented to reduce storage space requirements, then storage efficiency is improved, but processor cycles are wasted on compression of data that is poorly compressible
Solution Approach 1:
The system performs preliminary analysis of data compressibility using a trained data model before actually applying compression algorithms. This preliminary action identifies which data blocks are likely to be poorly compressible, allowing the system to avoid wasting processor cycles on such data while still compressing highly compressible data blocks efficiently.
Solution Approach 2:
A recommendation engine acts as an intermediary between the data and compression algorithms. The recommendation engine uses a trained data model to predict compressibility and recommends whether to compress specific data blocks and which algorithm to use, preventing direct application of compression to all data and thereby avoiding wasted processor cycles.
2Productivity
If multiple compression algorithms are tested to find the most efficient one, then compression efficiency is improved, but the complexity of the compression system increases
Solution Approach 1:
The trained data model performs preliminary prediction of which compression algorithm will be most efficient for each data block based on its characteristics. This preliminary action eliminates the need to actually test multiple algorithms on every data block, reducing system complexity while maintaining the ability to select the most efficient algorithm when needed.
Solution Approach 2:
The system changes the approach from directly comparing multiple compression algorithms to using a trained data model that predicts algorithm performance based on data parameters. This transforms the problem from algorithm comparison to parameter-based prediction, simplifying the system architecture while preserving compression efficiency.
Data Source
AI summary
A data model is trained to predict compressibility of binary data structures based on component entropy and predict relative compression efficiency for various compression algorithms based on component size. A recommendation engine in a storage system uses the data model to predict compressibility of binary data and determines whether to compress the binary data based on predicted compressibility. If the recommendation engine determines that compression of the binary data is justified, then a compression algorithm is recommended based on predicted relative compression efficiency. For example, the compression algorithm predicted to yield the greatest compression ratio or shortest compression/decompression time may be recommended.