Metadata Access Service for Distributed Data Lakes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data lake users face difficulties in accessing Hive metadata, which is crucial for their operations, due to its storage in external relational databases and lack of implicit storage in traditional systems, leading to restricted access and inconvenience.
Innovation Solution
A computing platform loads metadata from an external metadata database into a staging database, transforms it by decoupling and denormalizing data models, and then moves it to a master database, allowing authorized access to distributed data lakes through role-based access control, thereby providing seamless integration and automated metadata refresh.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If metadata is stored in external relational databases (Hive metastore), then metadata can be maintained with structured organization, but distributed data lake users face restricted access and inconvenience
Solution Approach 1:
The patent introduces a metadata service as an intermediary layer between users and the Hive metastore. This service exposes simplified RESTful APIs that allow users to access metadata without directly interacting with the complex Hive metastore infrastructure, thereby improving ease of operation while managing access architecture complexity
Solution Approach 2:
The patent segments the metadata access functionality into distinct service components with specific responsibilities (e.g., table metadata access, partition metadata access, schema validation). This segmentation allows users to access only the specific metadata they need through targeted API endpoints, improving operational ease while maintaining organized complexity in the backend
2Productivity
If metadata is accessed directly from Hive metastore, then data accuracy is maintained, but manual intervention and operational overhead increase
Solution Approach 1:
The metadata service implements self-service functionality by automatically querying the Hive metastore and caching metadata results. Users can access metadata through automated API calls without requiring manual intervention from administrators, thereby improving productivity and eliminating time loss associated with manual metadata provisioning
Solution Approach 2:
The service performs preliminary actions by pre-fetching and caching metadata from the Hive metastore before users request it. This preliminary metadata retrieval and caching mechanism reduces the time users wait for metadata responses while maintaining data accuracy, thereby improving operational efficiency
3Reliability
If role-based access control is implemented for metadata, then security is improved, but access control complexity increases
Solution Approach 1:
The metadata service implements a universal authentication mechanism that handles multiple authorization scenarios through a single role-based access control framework. The same authentication infrastructure serves both simple table access and complex partition access control, thereby improving security while managing access control system complexity through consolidation
Data Source
AI summary
Aspects of the disclosure relate to providing metadata access to distributed data lake users. In some embodiments, a computing platform may load metadata from an external metadata database into a staging database. Then, the computing platform may transform the metadata in the staging database and move the transformed metadata to a master database. The master database may comprise information indicating a relationship between the transformed metadata and one or more distributed data lakes. The computing platform may receive a request to access one or more metadata items. Then, the computing platform may authorize a distributed computing cluster user to access at least one metadata item based on the information. Based on the authorization, the computing platform may provide, to the distributed computing cluster user, access to the at least one metadata item of the one or more metadata items.