Host-Side Data Encryption with Deduplication for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data encryption methods in distributed storage systems face challenges such as increased storage space requirements due to ineffective compression and deduplication of encrypted data, as well as security risks during data transit between application servers and storage systems, particularly with self-encrypting drives and controller-managed encryption.
Innovation Solution
The implementation of a Distributed Virtual Array (DVA) system that encrypts data at the host-side before storage, using a host-side module to partition, fingerprint, and encrypt data blocks, and manages encryption keys and algorithms centrally, allowing for secure data transfer and storage while supporting deduplication and compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If application level encryption is used to encrypt data before writing to storage, then data security during network transit is improved, but storage space requirements increase due to inability to apply compression and deduplication
Solution Approach 1:
The patent applies compression and deduplication techniques before encryption (pre-processing) rather than after. This preliminary action allows the data to be compressed and deduplicated in its original form, then encrypted for secure storage, thereby maintaining both storage efficiency and security.
Solution Approach 2:
The patent segments the data processing workflow into distinct stages: first compression, then deduplication, then encryption. Each stage operates on the output of the previous stage, allowing optimal processing at each step without the limitations of applying all operations to already-encrypted data.
2Quantity of substance
If self-encrypting storage drives are used to encrypt data at storage, then storage space efficiency is improved through compression and deduplication, but data security during network transit deteriorates
Solution Approach 1:
The patent performs compression and deduplication as preliminary actions before data leaves the application server, then encrypts the data before network transmission. This ensures both storage efficiency and network security without relying on self-encrypting drives.
Solution Approach 2:
The patent introduces an intermediary encryption layer between the application server and the storage system. This intermediary encrypts data before it leaves the application server, ensuring network security, while still allowing the storage system to perform compression and deduplication on the encrypted data.
3Adaptability or versatility
If controller-managed encryption is used to encrypt data at storage controller, then flexibility to change encryption strategy is improved, but data security during network transit from application server deteriorates
Solution Approach 1:
The patent applies encryption as a preliminary action at the application server before data is transmitted over the network. This ensures data security during transit, and the encrypted data is then stored without requiring controller-managed encryption, thus maintaining both security and flexibility.
Solution Approach 2:
The patent introduces an intermediary encryption mechanism at the application server that acts as a mediator between the application and the storage system. This intermediary encrypts data before transmission, ensuring network security, while the storage system remains flexible to change encryption strategies independently.
4Reliability
If encryption is applied to data before storage, then data security is improved, but compression and deduplication effectiveness deteriorates
Solution Approach 1:
The patent performs compression and deduplication as preliminary actions before encryption is applied. This sequence ensures that the data is optimized for storage efficiency before security measures are applied, maintaining both productivity and reliability.
Solution Approach 2:
The patent segments the data processing pipeline into distinct stages: compression stage, deduplication stage, and encryption stage. Each stage operates independently on the output of the previous stage, allowing optimal performance of each function without interference from encryption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.