Log Processing Method for Distributed Database Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed databases, log storage collisions occur due to inconsistent log sequence numbers (LSNs) among compute and storage nodes, leading to data inconsistencies and collisions when a new master compute node is selected.
Innovation Solution
A log processing method where the first node sends a write operation request to the master storage node, ensuring agreement with the slave storage node, and uses iteration identifiers to determine the latest master compute node, preventing log collisions by maintaining consistency in LSNs through asynchronous message sending and configuration manager management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the master storage node assigns LSNs sequentially without verifying consistency with compute nodes, then log storage efficiency is improved, but log collision occurs when a new master compute node is selected
Solution Approach 1:
The master storage node sends feedback messages to compute nodes to verify LSN consistency before finalizing log storage. This feedback mechanism ensures that the compute node's LSN record matches the storage node's LSN record, preventing log collisions when master compute nodes are replaced.
Solution Approach 2:
The system performs preliminary LSN consistency verification between compute nodes and storage nodes before the master compute node fails or is replaced. This preliminary action ensures that LSN records are synchronized in advance, preventing log collisions during master node transitions.
2Speed
If the master compute node sends logs to the master storage node without LSN verification, then data write speed is improved, but log sequence number inconsistency occurs between compute and storage nodes
Solution Approach 1:
The master storage node sends feedback messages containing LSN information to compute nodes after receiving logs. This feedback ensures that compute nodes have accurate LSN records, maintaining LSN consistency while enabling fast data write operations.
Solution Approach 2:
Compute nodes automatically update their LSN records based on feedback from the master storage node. This self-service mechanism ensures LSN consistency without requiring complex verification protocols, maintaining high data write speed.
3Reliability
If the system implements comprehensive LSN verification before log storage, then log consistency is improved, but system complexity increases
Solution Approach 1:
The system uses simple feedback messages between the master storage node and compute nodes to verify LSN consistency. This feedback approach maintains log consistency without requiring complex verification protocols or additional system components.
Solution Approach 2:
The master storage node acts as an intermediary that coordinates LSN verification between compute nodes and the storage system. This intermediary approach simplifies the verification process by centralizing coordination, reducing overall system complexity.
4Speed
If the master storage node processes logs asynchronously without coordination, then processing speed is improved, but log collision occurs during master compute node replacement
Solution Approach 1:
The master storage node sends feedback messages to compute nodes to confirm LSN consistency before processing logs asynchronously. This feedback mechanism ensures log uniqueness even during asynchronous processing and master compute node replacement.
Solution Approach 2:
The system performs preliminary LSN consistency verification to prevent log collisions before they can occur during asynchronous processing or master node replacement. This preliminary anti-action ensures log uniqueness while maintaining high processing speed.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A log processing method, and a related device and system are disclosed. The method includes: obtaining, by a first node during initialization, a status message of a storage node cluster (S301), where the status message marks a master storage node (241) of the storage node cluster; and the first node is a master compute node (221) that has a write operation capability and a read operation capability in a compute node cluster; sending, by the first node, a first write operation request to the master storage node (241) based on the status message (S302), so that the master storage node (241) agrees on a first write operation after agreeing on a received log with a slave storage node (242) in the storage node cluster (S304); and receiving, by the first node, a feedback message that is sent after the master storage node (241) agrees on the first write operation with the slave storage node (242) in the storage node cluster (S306), where the feedback message includes identification information and a response to the first write operation request, and the identification information is used to indicate the