Erasure-Coded Data Writes With Threshold Node Completion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems using erasure coding face issues with traffic amplification and write delays due to node failures, leading to unstable performance in direct-write EC scenarios.
Innovation Solution
A data storage method that encodes data blocks using erasure coding, writes fragments to allocated storage nodes, and determines successful writing based on the number of nodes successfully written within a defined time duration, allowing for asynchronous rewriting and seal operations to manage failures and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If direct-write EC is used to avoid traffic amplification, then storage efficiency is improved, but write delay increases and performance stability deteriorates when storage nodes fail
Solution Approach 1:
The patent applies partial action by requiring only a threshold number of storage nodes (e.g., 2 out of 3) to successfully write data fragments, rather than requiring all nodes to complete the write operation. This allows the system to tolerate node failures while avoiding full write delays, resolving the contradiction between storage efficiency and write performance stability.
Solution Approach 2:
The patent dynamically adjusts the write success threshold parameter based on system conditions and node availability. By changing this parameter, the system can adapt to different failure scenarios and optimize the balance between avoiding traffic amplification and maintaining acceptable write delays, thereby improving performance stability.
2Reliability
If write attempts continue until timeout or failure in direct-write EC, then data reliability is maintained, but write delay and failure rate increase
Solution Approach 1:
The patent performs preliminary assessment of storage node status before initiating write operations and establishes timeout thresholds in advance. This allows the system to proactively identify failed nodes and switch to alternative strategies (such as rewriting to different nodes) without waiting for complete timeouts, thereby maintaining data reliability while reducing write delays and improving performance stability.
Solution Approach 2:
The patent implements feedback mechanisms that monitor write operation status in real-time and provide information about node failures. Based on this feedback, the system dynamically adjusts write strategies, such as selecting alternative storage nodes or adjusting the success threshold, which maintains data reliability while preventing prolonged write delays and improving overall performance stability.
3Reliability
If all storage nodes must be successfully written for data block write success, then data integrity is ensured, but write delay increases and performance becomes unstable under node failures
Solution Approach 1:
The patent requires only a threshold number of storage nodes (partial completion) rather than all nodes to successfully write data fragments. This partial action approach maintains data integrity through erasure coding while significantly reducing write delays when some nodes fail, thereby resolving the contradiction between data integrity and write performance stability.
Solution Approach 2:
The patent dynamically adjusts the write success threshold based on real-time node availability and system conditions. This dynamic approach allows the system to maintain data integrity when nodes are available while automatically reducing requirements when failures occur, preventing performance instability and reducing unnecessary write delays.
Data Source
AI summary
Data storage method, apparatus, and system are disclosed. The data storage method includes: encoding a data block using an erasure coding to obtain corresponding multiple data fragments; separately writing the multiple data fragments to corresponding allocated storage nodes; and determining that writing of the data block is successful if a number of storage nodes to which data fragments is successfully written within a first defined time duration is greater than or equal to a defined value, wherein the defined value is greater than or equal to a number of data block fragments in the multiple data fragments and is less than or equal to a number of the multiple data fragments. As such, the latency of a data write can be reduced, and the performance stability can be improved when a storage node encounters a failure.