Metadata Encryption in Distributed Storage via Background Defragmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in encrypting large amounts of data, managing encryption keys, and efficiently handling metadata in a cloud computing environment, leading to increased latency and complexity, especially when dealing with data from multiple users.
Innovation Solution
A system that compresses and encrypts metadata in a distributed storage system, using a processor and machine-readable instructions to manage encryption keys and data units, allowing for transparent encryption of metadata and data across multiple user accounts, with the ability to dynamically turn encryption on and off, and utilizing age-based garbage collection for background encryption during defragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted for security reasons, then security is improved, but latency increases due to the time required for encrypting and storing the data
Solution Approach 1:
The system performs encryption in advance during background operations such as defragmentation and garbage collection, rather than encrypting data when it is written or accessed. This preliminary encryption action eliminates the need for real-time encryption operations that would cause latency, while still providing security when the data is stored.
Solution Approach 2:
The encryption process is integrated into existing background maintenance operations (defragmentation, garbage collection) that the storage system performs anyway. This allows the system to serve dual purposes: maintaining storage efficiency and encrypting data simultaneously, without requiring separate encryption operations that would increase latency.
2Reliability
If encryption is performed on all data, then security is improved, but compute resources are consumed and processing overhead increases
Solution Approach 1:
The system applies encryption selectively based on local conditions - specifically, it encrypts data during background operations on a per-operation basis rather than universally encrypting all data at all times. This allows the system to provide security where and when it is most beneficial while minimizing unnecessary compute resource consumption.
Solution Approach 2:
Encryption is performed periodically during scheduled background maintenance operations rather than continuously or at every data access point. This periodic approach to encryption reduces the overall compute resource burden while still ensuring that data is encrypted when stored, balancing security requirements with resource constraints.
3Reliability
If metadata is encrypted along with data, then security is improved, but the complexity of managing encryption keys increases
Solution Approach 1:
The system merges the encryption of metadata with the encryption of data by using the same encryption key for both. This is achieved by deriving a single key from the customer-provided key that can encrypt both data and metadata, thereby simplifying key management while maintaining security for both data types.
4Reliability
If encryption is performed on large amounts of data, then security is improved, but the time required for encrypting and storing the data increases
Solution Approach 1:
The system performs encryption in advance during background operations rather than during data write operations. This preliminary encryption action allows data to be written and stored without the overhead of real-time encryption, maintaining high storage throughput while still providing security through pre-performed encryption during maintenance windows.
Solution Approach 2:
The system maintains continuous storage operations without interruption for encryption by performing encryption during scheduled background maintenance periods. This allows the storage system to continuously accept and write data at full throughput while encryption occurs continuously in the background during defragmentation and garbage collection operations, ensuring both productivity and security.
Data Source
AI summary
A system receives data of one or more types from one or more sources having distinct identities for storing the data in a distributed storage system. The system stores metadata associated with storing the data in data structures in the distributed storage system. The system selects a portion of the data and a portion of the metadata associated with storing the data in the distributed storage system. The system compresses the selected portions of the data and the metadata, encrypts the compressed data and metadata using a single key or using a separate key for each data type, and stores the encrypted compressed data and metadata in the distributed storage system. The system also encrypts unencrypted metadata and associated data during defragmentation procedure.