Redundancy-Encoded Data Storage for Fast Access and Eventual Durability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network computing and data storage systems face challenges in balancing data availability and integrity, often degrading customer experience due to delays in retrieving data from archival systems that use error correcting and error tolerance schemes.
Innovation Solution
The proposed solution involves storing data in preliminary storage initially for rapid access and then migrating it to durable storage using redundancy encoding techniques after it becomes less volatile, ensuring durability without compromising access speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If error correcting and error tolerance schemes are used to ensure data integrity, then data durability is improved, but data retrieval speed deteriorates
Solution Approach 1:
The system performs preliminary actions by storing data in both preliminary storage and durable storage before it is actually needed. The durable storage with error correcting codes is prepared in advance, so when data retrieval is needed, the system can quickly determine whether to fetch from preliminary or durable storage, avoiding real-time computation overhead.
Solution Approach 2:
The system dynamically adjusts the retrieval strategy based on data volatility and access patterns. Frequently accessed or volatile data remains in preliminary storage for fast access, while less volatile data is migrated to durable storage. The system adapts its error handling approach based on the specific data being accessed.
2Reliability
If data is stored in durable storage with redundancy encoding, then data durability is improved, but storage overhead increases
Solution Approach 1:
The system applies different storage qualities to different data based on their volatility characteristics. Frequently changing or volatile data is stored in preliminary storage without redundancy encoding, while stable, less volatile data is stored in durable storage with redundancy encoding. This localized application of quality ensures durability only where needed, reducing overall storage overhead.
3Quantity of substance
If data is migrated from preliminary storage to durable storage, then storage efficiency is improved, but access latency increases
Solution Approach 1:
The system performs preliminary migration of data to durable storage based on volatility analysis before access patterns change. By proactively identifying and migrating stable data, the system prepares the durable storage in advance, minimizing the impact on access latency when the data is eventually retrieved.
Solution Approach 2:
The system uses an intermediary mechanism (the data storage service) that manages the complexity of dual-storage management, migration decisions, and retrieval coordination. This intermediary handles the overhead of determining which data to migrate and from where to retrieve data, making the process transparent to users while optimizing the balance between efficiency and latency.
Data Source
AI summary
Techniques for making storage of data objects eventually durable using redundancy encoding are described herein. Data objects are stored in a first set of data storage devices with a first durability. After a predetermined length of time, the data objects are converted to data shards and distributed to a second set of data storage devices with a second durability that is distinct from the first durability.