Distributed Storage Parity Update Method for Network Traffic Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing distributed storage systems face performance issues with update writes in erasure-coded data, as they require recalculation of parity data, leading to increased network traffic and reduced performance, especially for small update sizes.
Innovation Solution
A distributed storage system that switches between two parity update methods based on network traffic: full-stripe parity calculation and read-modify-write (RMW) parity calculation, optimizing network usage by selecting the method that minimizes traffic during updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If recalculation of parity data is performed for update writes in erasure-coded distributed storage systems, then data protection reliability is maintained, but write performance deteriorates and network traffic increases
Solution Approach 1:
The system dynamically switches between two parity update methods (full-stripe recalculation and RMW-based update) based on real-time network traffic conditions. When network traffic is low, full-stripe recalculation is used to ensure data reliability. When network traffic is high, RMW-based updates are used to maintain write performance, thus adaptively resolving the contradiction between reliability and productivity
Solution Approach 2:
The invention changes the operational parameters of parity update by selecting different calculation methods based on network traffic thresholds. The system monitors network traffic volume and adjusts the parity update strategy accordingly, transforming the fixed reliability mechanism into a flexible parameter-driven approach that can optimize write performance under varying network conditions
2Reliability
If full-stripe parity recalculation is performed for all data units, then data protection reliability is improved, but network traffic volume increases significantly
Solution Approach 1:
Instead of always performing full-stripe parity recalculation on all data units, the system applies partial action by using RMW-based updates that only recalculate parity for affected data units when network traffic is high. This partial approach reduces network traffic volume while maintaining adequate data protection, resolving the contradiction between reliability and traffic volume
3Productivity
If RMW operations are performed on original data and parity data for small update sizes, then write performance is maintained, but network traffic between nodes increases
Solution Approach 1:
The system dynamically adjusts the parity update strategy based on network traffic conditions. When network traffic is already high, the system switches from RMW operations to full-stripe recalculation, which reduces network traffic between nodes even if it temporarily impacts write performance. This dynamic adjustment resolves the contradiction by adapting to current network conditions
Data Source
AI summary
A distributed storage places data units and parity units constituting a stripe formed by divided data into storage nodes in a distributed manner. In reference to determination formulas, either a full-stripe parity calculation method or an RPM parity calculation method is selected so as to minimize an amount of network traffic.