Distributed Storage Upgrade via Client Write Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for upgrading distributed storage systems either result in prolonged unavailability of high-level services, increased response times, or risk user data loss due to rapid rolling restarts of data servers.
Innovation Solution
A method where clients send write requests to multiple data servers simultaneously, determine successful writes, and send feedback to data servers to control their upgrade states, allowing the upgrade control server to manage rolling upgrades without stopping high-level services, ensuring data redundancy and reducing response times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire distributed storage system is stopped for upgrading, then all data servers can be upgraded completely, but high-level services become unavailable for a significant period
Solution Approach 1:
The system divides the upgrade process into segments by allowing individual data servers to be upgraded independently while others remain operational. The upgrade control server manages multiple data servers in a coordinated manner, enabling some servers to be upgraded without stopping the entire system.
Solution Approach 2:
The system performs preliminary actions by having clients send write requests to multiple data servers before the upgrade begins. This ensures that data is replicated across multiple servers before any server is taken offline for upgrading, maintaining service continuity.
2Loss of time
If data servers are restarted in a rolling manner rapidly, then service availability is maintained, but data reliability decreases due to risk of data loss
Solution Approach 1:
The system implements feedback mechanisms where clients monitor the number of successful writes to data servers. Based on this feedback, clients can determine whether to wait for recovery or retry with other servers, allowing dynamic adjustment of the upgrade pace to maintain data reliability.
Solution Approach 2:
The upgrade process is made dynamic by allowing the system to adjust the rolling restart pace based on real-time conditions. The upgrade control server coordinates upgrades dynamically, pausing or resuming based on data redundancy status and system conditions, rather than following a fixed rapid schedule.
3Reliability
If clients retry failed write requests after data server restart, then access availability is ensured, but response time increases significantly
Solution Approach 1:
The system performs preliminary actions by pre-replicating data to multiple data servers before the upgrade begins. This way, when a server is restarted, clients can immediately access data from other servers without waiting for recovery, eliminating the need for time-consuming retries.
Solution Approach 2:
The system ensures that different data servers have different roles at different times during the upgrade process. Some servers are in upgrade state while others are in service state, allowing clients to access data from servers that are currently operational rather than waiting for failed servers to recover.
4Productivity
If only one data server successfully writes data during rolling upgrade, then service continues, but user data is lost if that server fails
Solution Approach 1:
The system ensures that different data servers have different roles at different times during the upgrade process. By coordinating upgrades across multiple servers with different states (upgradeable vs. non-upgradeable), the system ensures data is written to multiple servers simultaneously, creating local redundancy at each server level.
Solution Approach 2:
The system prepares compensatory measures in advance by requiring data to be successfully written to a minimum number of data servers before allowing the upgrade to proceed. This creates a safety cushion that prevents data loss even if one server fails during the upgrade process.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the disclosure provide a method and apparatus for upgrading a distributed storage system, and the disclosure relates to the field of distributed computer technology. In the disclosure, a client sends write requests regarding the same to-be-written data to a plurality of data servers at the same time; then how many data servers the to-be-written data is successfully written into is analyzed; it is determined whether the number of the successful writes is greater than a pre-defined number; and first feedback information or second feedback information is sent to each data server having a successful write according to the determining result. The data server then determines whether it is in an upgradeable state or a non-upgradeable state according to the received first feedback information or second feedback information. Based on the state of the data server, an upgrade control server may select to notify, in a rolling manner, the data server to perform an upgrade operation. Therefore, a shorter response time of the system to the client is ensured without having to stop the high-level services, and data reliability is improved, significantly reducing the risk of user data loss.