Account-Level Blob Encryption to Cut Distributed Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage environments like cloud computing systems, encrypting large amounts of data increases latency, and managing encryption keys and determining which data to encrypt can be complex, especially when dealing with multiple users and dynamic encryption needs.
Innovation Solution
A system that compresses and encrypts data upfront at a front-end layer, reducing latency by encrypting at the first layer of servers, allowing dynamic encryption enablement/disabling, and tracking encryption status, using unique keys for each account block and storing seeds in headers for decryption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted in a distributed storage environment, then data security is improved, but latency increases and key management becomes complex
Solution Approach 1:
The system performs compression and encryption of data at the first layer of servers (front-end layer) before the data is transferred to subsequent layers. This preliminary action ensures that encryption occurs once at the source, rather than repeatedly during data transfer between layers, thereby maintaining security while reducing the latency associated with multiple encryption operations.
Solution Approach 2:
The patent divides the storage system into multiple layers with distinct functions: the first layer handles authentication, compression, and encryption; the second layer manages data structures; and the third layer performs actual data storage. This segmentation allows each layer to specialize in specific tasks, improving overall efficiency and reducing the latency impact of encryption by confining it to the first layer only.
2Reliability
If encryption is applied to all data, then security is improved, but system complexity increases
Solution Approach 1:
The system applies encryption selectively based on account-level settings rather than uniformly to all data. Each account can independently enable or disable encryption, allowing the system to maintain security where needed while avoiding the complexity of managing encryption keys for all data universally. This local quality approach tailors the encryption application to specific accounts and their requirements.
Solution Approach 2:
The system automatically manages encryption key generation, storage, and rotation without requiring manual intervention. The first layer servers handle key management autonomously, including generating unique keys for each account and block, storing them securely, and rotating them according to policies. This self-service capability reduces the operational complexity of key management while maintaining robust security.
3Quantity of substance
If compression is performed before encryption, then storage efficiency is improved, but processing time increases
Solution Approach 1:
The system performs compression as a preliminary action before encryption at the first layer of servers. By compressing data first, the system reduces the volume of data that subsequently requires encryption processing. This approach improves storage efficiency while minimizing the overall processing time, as compressing smaller amounts of data (before encryption) is more efficient than encrypting larger amounts and then compressing.
Data Source
AI summary
A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.