Metadata Access Server for Distributed Data Warehouse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data warehouses face challenges in efficiently accessing and sharing metadata due to the difficulty in realizing distributed storage, leading to prolonged access times and decreased performance as the amount of metadata increases.
Innovation Solution
Storing authentication information in a global database server and table information in segment database servers, allowing the metadata accessing server to obtain authentication information from the global database server and table information from segment database servers, thereby reducing access time and enhancing sharing and exchange speed in distributed data warehouses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is stored in a centralized data warehouse, then data consistency is maintained, but access time increases and performance decreases as metadata volume grows
Solution Approach 1:
The patent segments metadata into two distinct types: authentication information (stored in global database servers) and table information (stored in segment database servers). This segmentation allows different access patterns for different metadata types, reducing overall access time while maintaining consistency through the metadata accessing server's coordination mechanism.
2Productivity
If distributed storage is implemented for metadata, then access speed and expandability improve, but system complexity increases
Solution Approach 1:
The metadata accessing server acts as an intermediary between query engines and the distributed storage system (global and segment database servers). It manages the complexity of distributed storage by providing a unified access interface, handling authentication verification, and coordinating queries across multiple servers, thereby improving information sharing speed without exposing system complexity to users.
3Device complexity
If all metadata is stored in a single database, then system structure is simple, but access time prolongs and performance decreases with increasing data volume
Solution Approach 1:
The patent introduces a new dimensional organization to metadata storage by creating a hierarchical structure with global database servers (for authentication information) and segment database servers (for table information). This dimensional change from a single flat database to a multi-level distributed structure enables parallel access paths, significantly improving metadata access efficiency while maintaining manageable system structure through clear separation of concerns.
Data Source
AI summary
A method for accessing metadata includes: receiving an authentication information query request from a query engine, and sending the authentication information query request to a global database server; receiving authentication information from the global database server; sending the authentication information to the query engine so that the query engine determines, according to privilege authentication result of the authentication information, whether to access table information. A corresponding server and system are also disclosed. By storing authentication information in global database server and storing table information in one or more segment database servers, distributed storage of metadata is achieved; the metadata accessing server obtains from the global server the authentication information queried by the query engine and obtains from the segment database server the table information queried by the query engine, which reduces metadata access time and increases the speed of sharing and exchange of information in a distributed data warehouse.