Compressed Data Interface for Direct Storage Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems are unable to efficiently utilize compressed data, leading to increased bandwidth and storage consumption, as they lack interfaces that can interact with compressed data without decompressing it, which is essential for applications like backups and replication.
Innovation Solution
The implementation of new interfaces and operations that allow applications to read and write compressed data directly, using a system with an interface processor, controller logic, and media processor to manage compressed data storage, including metadata for integrity verification and deduplication, enabling applications to interact with compressed data without decompression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed in storage device view, then storage capacity and performance are improved, but application view becomes unaware of compression leading to inability to efficiently use compressed data
Solution Approach 1:
The patent introduces a compressed data interface as an intermediary layer between the application and the compressed data in storage. This interface includes commands for reading, writing, and verifying compressed data without requiring the application to decompress it first. The interface acts as a mediator that translates application requests into operations on compressed data, enabling applications to efficiently utilize compressed data while maintaining storage device compression benefits.
2Reliability
If uncompressed data is used for validity check, then data integrity verification is possible, but bandwidth and storage consumption increase
Solution Approach 1:
The patent changes the parameter of validity check from operating on uncompressed data to operating on compressed data directly. The interface includes commands that compute and verify validity check information (such as checksums or hashes) on the compressed data form, eliminating the need to decompress data for integrity verification. This parameter change maintains reliability while reducing bandwidth and storage consumption associated with handling uncompressed data for verification purposes.
3Ease of operation
If data is decompressed for application access, then application can process data, but backup and replication time increase
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing validity check information and metadata associated with compressed data. The interface is prepared in advance to handle compressed data operations, with commands and data structures ready to work with compressed formats. This preliminary preparation eliminates the need for time-consuming decompression operations during backup and replication, allowing applications to efficiently access and process compressed data directly.
Data Source
AI summary
A system and method for efficiently managing data through compression interfaces may include receiving, by a controller, data, generating, by the controller, a compressed payload based on the data, generating, by the controller, metadata describing the compressed payload, the metadata including fixed size metadata and variable size metadata, generating, by the controller, a data container comprising the uncompressed payload and the metadata, and transmitting, by the controller, the data container to an application. Some implementations of the system may include a storage media, and a storage controller executable by a processor that may include an interface processor, a controller logic, and a media processor configured to communicate with an application and the storage media to perform aspects of the method.