Distributed Data Management System with Segmented Consensus Modules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data management systems, existing consensus algorithms like Paxos increase latency due to increased communication times during failures, and either require a large number of processes or reduce the number of allowable failures, making them inefficient for low-latency and high-availability scenarios.
Innovation Solution
A data management system with a combination of process-saving 1-step and 2-step consensus modules, along with a PAXOS consensus module, determines data consistency with a reduced number of processes and communication times, maintaining high availability by allowing more allowable failures, thus minimizing latency and server scale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Paxos consensus algorithm is used to guarantee data consistency, then data consistency is ensured, but communication latency increases due to multiple communication rounds and increased number of processes
Solution Approach 1:
The patent segments the consensus determination into two independent modules: a first determination module that requires fewer communication rounds (lower latency) and a second determination module that provides stronger consistency guarantees (higher reliability). This segmentation allows the system to balance between speed and reliability by selecting appropriate modules based on failure scenarios.
Solution Approach 2:
The system dynamically selects which determination module to use based on the failure scenario. When failures occur, the system can switch between the first and second determination modules, adjusting the consistency verification strategy adaptively rather than using a fixed approach, thereby optimizing latency while maintaining reliability.
2Loss of time
If the number of processes is increased to reduce communication rounds, then communication latency decreases, but the system complexity and resource requirements increase
Solution Approach 1:
The patent divides the consensus verification into two segmented modules with different process requirements. The first determination module uses fewer processes for quick verification, while the second module uses more processes for thorough verification. This segmentation allows the system to achieve low latency without permanently increasing the total number of processes, as each module can be configured independently.
Solution Approach 2:
The system changes the parameter of process quantity dynamically based on which determination module is active. Instead of maintaining a large fixed number of processes, the system adjusts the effective process count based on the verification needs, thereby reducing overall system complexity while enabling low-latency operation when using the first determination module.
3Reliability
If the number of allowable failures is reduced to maintain consistency, then data consistency is guaranteed, but system availability decreases
Solution Approach 1:
The system dynamically adjusts the allowable failure threshold based on which determination module is used. The first determination module has a higher failure threshold for better availability, while the second module has a lower threshold for stronger consistency. This dynamic adjustment allows the system to maintain high availability under normal conditions while ensuring consistency when failures occur.
Solution Approach 2:
The patent segments the consistency guarantee mechanism into two modules with different failure tolerances. This segmentation allows the system to handle different failure scenarios appropriately, maintaining high availability for common cases while providing strong consistency guarantees for critical failure scenarios, thereby resolving the contradiction between availability and consistency.
4Reliability
If communication rounds are increased to ensure consistency, then data consistency is guaranteed, but the minimum number of communication times increases
Solution Approach 1:
The patent segments the communication verification into two stages: the first determination module uses fewer communication rounds for initial verification, and the second determination module uses more communication rounds for final verification. This segmentation allows the system to complete consistency checks with a lower minimum number of communication times by utilizing the first module when possible.
Solution Approach 2:
The system applies partial verification action by using the first determination module which performs a lighter verification with fewer communication rounds. This partial action is sufficient for many scenarios and reduces the minimum communication times required, while the second module provides excessive verification when absolutely necessary for strong consistency guarantees.
Data Source
AI summary
A data management system including a plurality of servers each having a processor, a memory, and a storage device, the system receiving and storing data using the plurality of servers and multiplexing the data, wherein the servers are provided with: a first determination unit that determines the consistency of the multiplexed data; a second determination unit that has a greater number of allowable server failures than the first determination unit for determining the consistency of the multiplexed data but a greater minimum number of times of server-to-sever communications for determining the consistency of the data; a combination unit that receives a data consistency determination result from the first determination unit or the second determination unit, and that, if the determination result includes consistency guaranteeing data, outputs the consistency guaranteed data; and a data storage unit that stores the data output by the combination unit.