Neural Data Characterization for Compression-Aware Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems like SANs and NAS face increased storage space requirements due to the lack of indication on data compressibility, making it difficult to efficiently manage and reduce storage space without external input on compression suitability.
Innovation Solution
A compute node with a trained data model and recommendation engine identifies the compression state of host application data, grouping uncompressed and unencrypted data structures by similarity and applying an optimal compression algorithm to reduce storage space requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication and backup functions are implemented to avoid data loss, then data availability is improved, but storage space requirements increase
Solution Approach 1:
The patent applies parameter changes by implementing data compression techniques that transform the physical state of stored data. By changing the representation parameters of data (compressing it), the system reduces storage space requirements while maintaining data availability through replication and backup functions.
2Quantity of substance
If compression is applied to reduce storage space requirements, then storage efficiency is improved, but the system cannot determine whether data is suitable for compression without external indication
Solution Approach 1:
The storage system performs self-service by automatically determining compression suitability through analysis of data characteristics. The system independently evaluates whether stored data is suitable for compression without requiring external input or host server indication, enabling autonomous compression decisions that reduce storage space while avoiding complexity of external coordination.
3Measurement precision
If the storage system waits for external input on compression suitability, then compression accuracy is improved, but automation is reduced and additional communication overhead is introduced
Solution Approach 1:
The storage system implements feedback mechanisms by analyzing data characteristics and using this information to automatically determine compression suitability. The system continuously monitors data properties, provides feedback on compression potential, and adjusts compression operations accordingly, achieving both accurate identification and full automation without external input requirements.
Data Source
AI summary
A data model is trained to determine whether data is raw, compressed, and/or encrypted. The data model may also be trained to recognize which compression algorithm was used to compress data and predict compression ratios for the data using different compression algorithms. A storage system uses the data model to independently identify raw data. The raw data is grouped based on similarity of statistical features and group members are compressed with the same compression algorithm and may be encrypted after compression with the same encryption algorithm. The data model may also be used to identify sub-optimally compressed data, which may be uncompressed and grouped for compression using a different compression algorithm.