Key-Durable Storage Service Object Loss Reporting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems prioritize performance characteristics like latency, throughput, and availability over cost, making them unsuitable for archival storage where data durability is essential but access is infrequent, and they require complex key mapping for redundancy, increasing costs.
Innovation Solution
The implementation of a key-durable storage service that generates system keys for stored objects, providing low-cost storage with granular failure detection by focusing on key redundancy rather than object redundancy, allowing clients to manage object redundancy and key mapping externally.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If object redundancy is used to ensure data durability, then reliability is improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts the key mapping functionality from the storage system itself and places it externally with the client. The storage system only maintains simple object-to-locator mappings without complex key management, while clients handle key generation, encryption, and key-locator mapping externally. This separation reduces the storage system's complexity while maintaining data durability through external key management.
Solution Approach 2:
Instead of the storage system managing key mappings for objects, the patent inverts the approach by having clients manage their own key mappings externally. The storage system receives encrypted objects with embedded locators and stores them without maintaining key maps, reversing the conventional model where the storage system would be responsible for key management.
2Productivity
If conventional data storage systems prioritize performance characteristics, then productivity is improved, but reliability for archival storage deteriorates
Solution Approach 1:
The patent segments the storage system into two independent components: (1) the storage system optimized for performance that handles object storage and retrieval with fast throughput and low latency, and (2) the external key management system that handles durability and recovery. This segmentation allows each component to be optimized independently - the storage system for productivity and the key management system for reliability.
3Reliability
If object redundancy is implemented in storage systems, then reliability is improved, but cost increases
Solution Approach 1:
The patent uses cryptographic copying where the entire object is encrypted with a key, and the encrypted copy is stored with an embedded locator. Instead of storing multiple physical replicas of the object data, the system stores one encrypted copy where the key enables recovery. This cryptographic approach provides durability without requiring multiple copies of the actual data, reducing storage capacity requirements.
Data Source
AI summary
In response to receiving a request from a client to store an object, a key-durable storage system may assign the object to a volume in its data store, generate a key for the object (e.g., an opaque identifier that encodes information for locating the object in the data store), store the object on one disk in the assigned volume, store the key redundantly in the assigned volume (e.g., using a replication or erasure coding technique), and may return the key to the client. To retrieve the object, the client may send a request including the key, and the system may return the object to the client. If a disk fails, the system may determine which objects were lost, and may return the corresponding keys to the appropriate clients in a notification. The system may be used to back up a more expensive object-redundant storage system.