Erasure Code Replication in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in balancing performance speed, storage capacity, and data retrieval speed while also needing to protect against hardware failures, which can result in data loss, especially when using traditional persistent storage devices that are slower than byte addressable storage media.
Innovation Solution
The implementation of a controller in a distributed storage system that utilizes erasure coding replication and non-volatile memory (NVM) devices to store and retrieve data chunks, prioritizing NVM devices for faster operations and storing coding chunks in other devices for data protection, allowing for efficient data reconstruction even in case of failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional persistent storage devices are used for data storage, then storage capacity is increased, but data retrieval speed and write speed deteriorate
Solution Approach 1:
The patent segments data into K data chunks and M coding chunks, storing them separately in different storage devices. This segmentation allows the system to retrieve data faster by only needing to access K chunks rather than all storage devices, while still maintaining robust storage capacity through the distributed chunk architecture.
Solution Approach 2:
The patent performs preliminary erasure coding to generate both data chunks and coding chunks before storage. The coding chunks are pre-calculated and stored alongside data chunks, enabling faster data retrieval and recovery without requiring real-time computation during read operations.
2Reliability
If erasure coding replication is implemented, then data protection against hardware failures is improved, but storage operation speed deteriorates
Solution Approach 1:
The patent applies local quality by storing coding chunks in specific storage devices with different characteristics than those storing data chunks. This allows the system to optimize for both reliability (through distributed coding chunks) and performance (by placing coding chunks in devices that don't bottleneck data retrieval operations).
3Reliability
If all data chunks and coding chunks are stored simultaneously, then data protection is improved, but write operation speed deteriorates
Solution Approach 1:
The patent performs preliminary action by first storing the K data chunks in storage devices, allowing the system to become operational with basic data protection. The M coding chunks are then stored in subsequent operations, enhancing data protection without requiring all chunks to be written simultaneously, thus reducing total write operation time.
Data Source
AI summary
The present disclosure involves systems and methods for managing data in a distributed storage system. The distributed storage system may include non-volatile memory (NVM) storage devices and utilize erasure code replication for storage of data. A controller may first store at least some of the K data chunks in NVM devices before storing the coding chunks in other storage devices. In addition, the controller may transmit read requests to the NVM devices of the system first to begin receiving data chunks or coding chunks related to the data object. By writing to and reading from NVM devices first, storage and reading of the data object may occur faster than conventional storage systems.