Metadata Cache Consistency Routing for Distributed Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing cloud-native data warehouse systems face high network requirements and high costs due to the need to carry metadata of all data partitions in the execution plan, which limits the ability to meet concurrency demands and higher transaction isolation levels.
Innovation Solution
Implement a data analysis method where metadata is cached in a computing cluster, and a coordinator node determines the consistency of metadata caches to schedule analysis requests to appropriate clusters, reducing the need for network metadata synchronization and query performance requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata of all data partitions is carried in the execution plan, then data consistency is ensured, but network bandwidth requirement increases and cost increases
Solution Approach 1:
The patent extracts only the necessary metadata information (version numbers, modification times) from the complete metadata set and transmits it separately from the execution plan. This allows the execution plan to be lightweight while still enabling data consistency verification through the extracted metadata parameters.
Solution Approach 2:
The patent introduces an intermediary metadata synchronization mechanism where metadata is transmitted independently from the execution plan. This intermediary channel allows consistency information to be conveyed without burdening the main execution plan transmission, reducing network bandwidth requirements while maintaining reliability.
2Measurement precision
If metadata synchronization is performed frequently to ensure consistency, then data accuracy is improved, but network requirement increases and query performance requirement increases
Solution Approach 1:
The patent performs preliminary metadata synchronization by attaching version information and modification timestamps to the execution plan beforehand. This preliminary action enables the computing cluster to verify data consistency locally without requiring frequent subsequent synchronization operations, thereby maintaining data accuracy while reducing ongoing network requirements.
Solution Approach 2:
The patent implements a feedback mechanism where the computing cluster uses the synchronized metadata (version numbers, modification times) to verify data consistency locally. This feedback loop allows the system to maintain high data accuracy without requiring continuous metadata synchronization, thus improving query performance while preserving data integrity.
3Reliability
If all metadata is synchronized to meet concurrency demands, then transaction isolation level is improved, but system cost increases
Solution Approach 1:
The patent extracts only the critical metadata elements (version information, modification timestamps) needed for transaction isolation verification. By transmitting only these extracted metadata components rather than complete metadata sets, the system achieves high transaction isolation levels while significantly reducing the quantity of data transmitted and stored, thereby lowering system costs.
4Reliability
If metadata is carried in execution plan to ensure consistency, then data reliability is improved, but device complexity increases
Solution Approach 1:
The patent segments the metadata transmission process into two independent parts: (1) execution plan transmission with embedded version information, and (2) separate metadata synchronization. This segmentation reduces the complexity of each individual component while maintaining overall data reliability through the coordinated operation of both segments.
Data Source
AI summary
A data analysis method, performed by a data analysis system that includes a coordinator node, a metadata management apparatus, and a plurality of computing clusters. The computing cluster includes a metadata cache. The metadata management apparatus records a status of the metadata cache. The data analysis method includes the coordinator node that receives a query statement, and delivers a consistency determining request to the metadata management apparatus. The metadata management apparatus obtains a consistency determining result of a metadata cache of at least one computing cluster based on the status of the metadata cache, then returns the consistency determining result to the coordinator node, and next determines a target computing cluster from the plurality of computing clusters based on the consistency determining result. The target computing cluster performs data analysis according to an analysis request delivered by the coordinator node.