Erasure-Coded Parity Updates With Threshold Consensus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coded data storage systems face inefficiencies during the update process, as small changes to the original data result in the regeneration of a large amount of parity data, leading to time-consuming and resource-intensive updates.
Innovation Solution
Implementing efficient consensus protocols in a data storage system where a leader data storage device generates and transmits updated parity data, allowing only the affected sections of parity data to be updated, and only committing the update once a threshold number of parity devices have acknowledged, thereby minimizing the data that needs to be rewritten.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coded parity data is used to protect against data loss, then data reliability is improved, but the amount of parity data that needs to be regenerated during updates increases
Solution Approach 1:
The system segments parity data into multiple parity data subsections, where each subsection corresponds to a specific section of the data set. This segmentation allows only the affected parity data subsection to be updated when a particular data section changes, rather than regenerating all parity data. The leader data storage device identifies which specific parity subsection needs updating and transmits only that portion to the relevant parity storage device.
Solution Approach 2:
The invention extracts and updates only the necessary portion of parity data (the affected parity data subsection) rather than processing the entire parity data set. When update data is received for a specific data section, the system extracts only the corresponding parity subsection that needs to be regenerated, leaving the rest of the parity data unchanged and in place.
2Stability of the object's composition
If all parity data is regenerated for small data updates, then data consistency is maintained, but update time increases
Solution Approach 1:
By segmenting parity data into subsections tied to specific data sections, the system enables selective updates. When a small portion of data changes, only the corresponding parity subsection is regenerated and updated, maintaining data consistency for the changed portion while avoiding unnecessary regeneration of unchanged parity data, thus significantly reducing update time.
Solution Approach 2:
The system performs partial action by updating only the necessary parity data subsection rather than the complete parity data set. This partial update approach maintains sufficient consistency for the updated data section while avoiding the excessive time cost of regenerating all parity data, achieving an optimal balance between consistency and update efficiency.
3Reliability
If a consensus protocol is implemented to ensure update reliability, then data integrity is improved, but the complexity of the update process increases
Solution Approach 1:
The leader data storage device serves as an intermediary that coordinates the update process. It receives update data, generates the updated parity data subsection, transmits it to the appropriate parity storage device, and manages the consensus protocol. This intermediary approach centralizes coordination logic, simplifying the overall system architecture while maintaining update reliability through structured communication and acknowledgment protocols.
Solution Approach 2:
The system implements feedback mechanisms where parity storage devices send update notifications back to the leader data storage device to confirm successful updates. This feedback loop ensures data integrity by verifying that parity data subsections were correctly updated, while the structured feedback process manages complexity through clear, defined communication protocols rather than distributed complex coordination.
Data Source
AI summary
Technology is provided for updating a data set at a data storage system. In an example storage system, the system stores a data set in a plurality of data storage devices. The system stores parity data at a plurality of parity devices. The system receives update data from a client system for a first section of the data set. The system generates updated parity data based on an original version of the first section of the data set and the update data. The system transmits update parity data to the plurality of parity devices. The system receives update notifications from a plurality of parity devices. The system determines that update notifications have been received from at least a threshold number of parity devices in the plurality of parity devices. In response, the system updates the first section of the data set at the leader data storage device.