Virtual File Metadata Access for Real-Time Data Lake Governance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data lakes face challenges in maintaining up-to-date metadata, which hinders data governance and security features like ACID transactions, row-level security, column-level security, data versioning, and auditing, due to the asynchronous nature of metadata snapshot exports.
Innovation Solution
A virtual file system that pulls real-time metadata from a metadata catalog, providing up-to-date metadata to users without waiting for scheduled exports, and supports formats compatible with data analysis tools.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata snapshots are exported to remote storage services, then data governance and management can be maintained, but the metadata may not accurately reflect the current data state due to asynchronous exports
Solution Approach 1:
A virtual file system is introduced as an intermediary layer between the data lake and data analysis tools. This virtual file system maintains a live view of metadata by pulling updates from the data lake in real-time, eliminating the need to wait for asynchronous snapshots while ensuring metadata accuracy reflects the current data state.
Solution Approach 2:
The virtual file system continuously synchronizes metadata with the data lake by pulling updates as they occur, rather than relying on periodic snapshots. This continuous synchronization ensures that metadata is always up-to-date without interruption, eliminating time delays associated with scheduled exports.
2Adaptability or versatility
If metadata is exported at scheduled intervals or on user request, then data governance can be maintained, but ACID transactions and row-level security cannot be provided
Solution Approach 1:
The virtual file system acts as a mediator that provides transactional capabilities by maintaining a live, synchronized view of metadata. This enables ACID transactions, row-level security, and column-level security to be implemented without adding complex governance infrastructure, as the virtual file system handles these features through its real-time synchronization mechanism.
3Adaptability or versatility
If metadata snapshots are used, then data governance can be maintained, but data versioning and auditing features cannot be provided
Solution Approach 1:
The virtual file system continuously pulls metadata updates from the data lake, maintaining an ever-updated view that enables data versioning and auditing. This continuous synchronization ensures that version history and audit trails reflect the most current data state, eliminating the time delays inherent in scheduled snapshot exports.
Data Source
AI summary
A data storage management method includes detecting a data change to data in a data repository, identifying metadata of the data change, and storing the metadata in a virtual file, the virtual file being in a data storage format that is compatible with one or more data analysis tools. In response to a subsequent user request to access metadata of the data in the data repository, the method may transmit one or more virtual files containing metadata identified in the user request.