Classified Storage Data Paths for Selective Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies due to noise and data redundancy in data processing, particularly for machine learning and artificial intelligence applications, leading to increased processing time and reduced efficiency.
Innovation Solution
Implementing a storage system with multiple data paths that classify data based on type, where one path processes data without compression and the other path processes data through a pre-processor to remove noise and redundancy before storage, thereby forming compressed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored without compression, then storage speed is improved, but storage space efficiency deteriorates
Solution Approach 1:
The storage system divides data into different segments based on classification tags, routing different types of data through different data paths. Compressible data goes through the compression path while non-compressible data takes the direct path, optimizing both speed and space efficiency for different data types simultaneously
Solution Approach 2:
The system dynamically selects data paths based on real-time data classification. The controller evaluates data characteristics and dynamically routes data through appropriate paths (compression or direct), allowing the system to adapt its storage approach based on data type rather than using a static single path
2Quantity of substance
If data is compressed before storage, then storage space efficiency is improved, but processing time increases
Solution Approach 1:
The system extracts only the compressible portion of data processing by separating compressible data from non-compressible data. By identifying and extracting data that benefits from compression, the system applies compression only where it provides value, avoiding unnecessary processing time for data that cannot be compressed
Solution Approach 2:
Instead of compressing all data uniformly, the system applies partial compression action only to data types that are suitable for compression. This selective approach avoids the excessive processing time that would result from attempting to compress all data types, including those that are already in optimal format
3Reliability
If noise and redundancy are removed from data, then data quality for ML/AI is improved, but processing complexity increases
Solution Approach 1:
The system performs preliminary data processing actions by removing noise and redundancy before data is stored. This preliminary cleaning occurs during the write operation, so that when data is later retrieved for ML/AI processing, it is already in optimized form, reducing the need for additional processing complexity during read operations
Data Source
AI summary
In some implementations, a storage system may receive, via a system controller of the storage system, a write command and data associated with the write command. The storage system may classify, via the system controller, the data. The storage system may associate, via the system controller, the data with a queue based on classifying the data. The storage system may retrieve, via a processor of the storage system, the data associated with the queue. The storage system may compress, via the processor, the data to form compressed data for storage in a memory device of the storage system based on the write command.