Virtual File System for Real-Time Data Lake Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing data in data lakes is challenging due to difficulties in identifying correct data sources, ensuring metadata security, and maintaining data governance, which is exacerbated by the delay in metadata snapshot exports, leading to inconsistencies and the lack of features like ACID transactions, row-level security, and data versioning.
Innovation Solution
Implementing a virtual file system that pulls up-to-date metadata from a metadata catalog associated with the data lake, providing real-time metadata access compatible with data analysis tools, and optionally generating snapshots at user request or scheduled intervals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata snapshots are exported to remote storage services, then data governance and management are improved, but metadata consistency and up-to-date information are lost due to export delays
Solution Approach 1:
The patent introduces a virtual file system as an intermediary layer between the data lake and users. This virtual file system provides a data warehouse interface that serves as a mediator, delivering consistent and up-to-date metadata information without requiring direct access to the raw data lake storage. The intermediary resolves the contradiction by maintaining data governance through structured metadata access while ensuring consistency through real-time virtual file system queries.
2Reliability
If metadata snapshots are exported at scheduled intervals or on user request, then data governance is maintained, but response time and data freshness are worsened
Solution Approach 1:
The virtual file system performs preliminary actions by pre-loading and maintaining up-to-date metadata in a readily accessible format. When users make requests, the system can immediately serve fresh metadata without waiting for scheduled exports. This preliminary preparation of metadata in the virtual file system eliminates the time delay between data changes and user access.
3Quantity of substance
If data is stored in raw format in data lakes, then storage efficiency is improved, but data management and governance difficulty increase
Solution Approach 1:
The patent segments the data access system into two distinct layers: the raw data lake for efficient storage and the virtual file system for structured access. The virtual file system creates segmented views of the data that are organized and managed according to governance requirements, while the underlying data lake maintains raw storage efficiency. This segmentation allows each layer to optimize for its specific function.
4Reliability
If metadata is accessed from data lake storage services, then data governance is achieved, but ACID transactions, row-level security, and data versioning features are lost
Solution Approach 1:
The virtual file system provides universal access to the data lake, serving multiple functions simultaneously. It enables data governance through structured metadata access while also supporting ACID transactions, row-level security, column-level security, data versioning, and auditing. The virtual file system acts as a universal interface that delivers all these features through a single consistent access mechanism.
Data Source
AI summary
A data storage management method includes detecting a data change to data in a data repository, identifying metadata of the data change, and storing the metadata in a virtual file, the virtual file being in a data storage format that is compatible with one or more data analysis tools. In response to a subsequent user request to access metadata of the data in the data repository, the method may transmit one or more virtual files containing metadata identified in the user request.