Shard Schema Snapshot Alignment for Continuous Distributed DML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed databases with sharding, schema changes (DDL) lead to inconsistent metadata versions across shards, causing data replication abnormalities and service interruptions due to disabled DML operations to maintain consistency.
Innovation Solution
Implementing shard node mode snapshots and global logic mode snapshots to adjust log data consistency, ensuring atomicity of DDL operations, allowing continuous DML without interruptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If DML write is disabled during DDL change to avoid multiple versions of data, then data consistency is improved, but data service continuity deteriorates
Solution Approach 1:
The patent segments the DDL execution process into multiple phases: schema evolution phase where different shards can have different schema versions, and data output phase where log data is adjusted to match the target schema version. This allows DML to continue during schema transitions while ensuring data consistency in the output stream.
Solution Approach 2:
The patent introduces a schema version management mechanism as an intermediary between the physical DDL execution and the global log output. This intermediary tracks schema versions across shards and adjusts log data accordingly, allowing DML operations to proceed while maintaining data consistency in the replicated stream.
2Productivity
If physical DDL is executed immediately on shard nodes to enable continuous DML, then service continuity is improved, but schema metadata consistency deteriorates
Solution Approach 1:
The patent performs preliminary schema version tracking and log data adjustment before data is output downstream. By preparing the schema version information and adjusting log data in advance, the system allows immediate DDL execution on shards while ensuring that the output data maintains consistency with the target schema version.
Solution Approach 2:
The patent implements a feedback mechanism where the system continuously monitors schema version status across shards and dynamically adjusts log data output. This feedback loop ensures that even though shards execute DDL at different times, the downstream data stream maintains consistent schema versions.
3Reliability
If schema changes are synchronized across all shards before allowing DML, then schema consistency is improved, but data service efficiency deteriorates
Solution Approach 1:
The patent introduces dynamic schema version management where the system adapts to different schema versions across shards during the transition period. Instead of requiring all shards to synchronize before allowing DML, the system dynamically adjusts log data output based on the target schema version, allowing continuous service while maintaining output consistency.
4Manufacturing precision
If log data is adjusted to match global logic mode snapshot, then data replication accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent introduces a schema version adjustment mechanism as an intermediary layer between log data generation and data output. This intermediary automatically adjusts log data to match the target schema version based on schema version tracking, improving replication accuracy while abstracting the complexity from the core data processing path.
Data Source
AI summary
Embodiments of the present application provide a data processing method, a distributed database system, an electronic device and a computer storage medium, wherein a data processing method is applied to a shard node of a distributed database system. The method comprises: receiving a physical DDL instruction converted from a logic DDL instruction for change of data in the distributed database system; executing the physical DDL instruction and generating a shard node mode snapshot for the shard node after the physical DDL instruction has been successfully executed; and if it is determined that the shard node mode snapshot is inconsistent with a global logic mode snapshot of the distributed database system, adjusting log data of the shard node aggregated into a global log to be consistent with the global logic mode snapshot.