Distributed Database Export Management Node Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data export methods for distributed databases face challenges in ensuring data consistency across nodes, leading to inconsistencies and performance issues, particularly during online data exports, which affects business operations and maintainability.
Innovation Solution
A data processing method and device that utilizes a database import and export management node to acquire active transaction lists and export table distribution information, notify corresponding databases to perform data exports, generate data consistency reverse compensation statements, and execute these statements to ensure data consistency, thereby addressing the inconsistency issues and improving usability and maintainability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the database agent is prohibited to provide external service to ensure data consistency during offline data export, then data consistency is improved, but online business is significantly affected
Solution Approach 1:
The patent segments the data export process into multiple independent steps: obtaining consistent read-only snapshots from multiple database nodes, exporting data locally without blocking external services, and then performing data merging. This allows the database agent to continue providing external services while data export operations are performed on consistent snapshots, resolving the contradiction between data consistency and online business availability.
Solution Approach 2:
The patent performs preliminary actions by obtaining consistent read-only snapshots of database data before the actual data export operation. These snapshots are created in advance with consistent timestamps, ensuring data consistency is established before export begins. This preliminary snapshot creation allows the export process to proceed without requiring the database agent to stop external services.
2Productivity
If the database agent redirects and outputs data to a data file, then data export functionality is achieved, but network I/O and system resources are significantly occupied
Solution Approach 1:
The patent extracts the data export operation from the centralized database agent and performs it locally at each database node. Each node exports its own data to local files independently, eliminating the need for the database agent to redirect and transfer data across the network. This extraction of the export function to the node level significantly reduces network I/O and system resource occupation while maintaining data export capability.
3Reliability
If the distributed database waits for active transactions to complete before exporting data, then data consistency is improved, but the data export process becomes too long
Solution Approach 1:
The patent performs preliminary snapshot creation with consistent timestamps across all database nodes before the export process begins. These pre-created snapshots capture the database state at specific points in time, allowing the export to proceed without waiting for active transactions to complete. This preliminary snapshot approach ensures data consistency while significantly reducing the overall export process duration.
Solution Approach 2:
The patent introduces dynamic timestamp-based snapshot creation that adapts to the current transaction state. Instead of statically waiting for all transactions to complete, the system dynamically creates snapshots at appropriate points, using timestamp information to ensure consistency. This dynamic approach allows the export process to proceed efficiently while maintaining data consistency across distributed nodes.
4Productivity
If a selection command is executed by a single node database and results are returned to the database agent, then data export is achieved, but performance of the distributed database is severely affected
Solution Approach 1:
The patent segments the data export operation so that each database node independently executes its own data export to local files without involving the database agent in the actual data retrieval process. This segmentation eliminates the performance bottleneck where the database agent would otherwise coordinate and transfer data from multiple nodes, allowing each node to operate independently and maintain high performance while the export function is still achieved.
Data Source
AI summary
A data processing method includes: after receiving an export request for exporting data from the distributed database, acquiring, by a database import and export management node an active transaction list and export table distribution information at a current moment; notifying, by the database import and export management node, a corresponding database that the corresponding database performs a data export operation according to the export table distribution information; after determining that the corresponding database completes the data export operation, sending, by the database import and export management node, the active transaction list to a database agent node of the corresponding database; and after receiving a data consistency reverse compensation statement returned by the database agent node, importing, by the database import and export management node, the data exported by the corresponding database into a predetermined database and instructing the predetermined database to execute the data consistency reverse compensation statement.