Blob Data Encryption at the Front-End Layer to Cut Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage environments, encrypting large amounts of data increases latency, and managing encryption keys and determining which data to encrypt can be complex, especially in cloud computing systems where data is stored across multiple users and layers.
Innovation Solution
A system that compresses and encrypts data upfront at the front-end layer of a cloud storage system, reducing latency by encrypting data at the first layer of servers, allowing dynamic encryption enablement, and tracking encryption status, using unique keys for each account and rotating keys for enhanced security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted in a distributed storage environment, then data security is improved, but latency increases and management complexity increases
Solution Approach 1:
The system performs encryption at the first layer of servers immediately when data is received, before data is distributed to subsequent layers. This preliminary encryption action ensures that data is secured early in the storage pipeline, preventing latency accumulation during later encryption operations and reducing the time data remains in plaintext during transit across the distributed system.
Solution Approach 2:
The encryption process is segmented and distributed across different layers of the storage system. The first layer of servers handles initial encryption, while subsequent layers handle encrypted data storage and management. This segmentation allows encryption operations to be parallelized and distributed, reducing overall latency while maintaining security across the entire data pipeline.
2Reliability
If encryption is applied to all data in a distributed storage environment, then data security is improved, but device complexity increases
Solution Approach 1:
The system segments key management responsibilities across different layers of the distributed storage system. The first layer of servers manages encryption keys for data encryption, while subsequent layers manage metadata and storage operations. This segmentation distributes the complexity of key management across multiple components rather than concentrating it in a single system, making the overall system more manageable and scalable.
Solution Approach 2:
The patent introduces an intermediary key management mechanism where keys are generated and managed at the first layer of servers, which then distributes encrypted data to subsequent layers. This intermediary layer acts as a mediator between data sources and storage systems, handling the complexity of key generation, rotation, and management while presenting a simplified interface to both data producers and storage consumers.
3Loss of time
If data is compressed and encrypted at the first layer of servers, then latency is reduced, but processing requirements increase
Solution Approach 1:
The system merges compression and encryption operations at the first layer of servers into a unified processing pipeline. By combining these operations at a single point in the data flow, the system eliminates the need for separate compression and encryption passes, reducing overall processing time and latency while efficiently utilizing computational resources at the entry point of the storage system.
Data Source
AI summary
A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.