Unified Runtime Catalog for Cross-Format Cloud Table Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query engines for data lakes and data warehouses have separate runtime catalogs that cannot be synchronized, leading to data silos, inconsistencies, and difficulties in enforcing security policies due to incompatible formats and access controls, resulting in issues like missing, duplicated, or outdated table attributes and exposure of sensitive data.
Innovation Solution
A unifying runtime catalog with a bi-directional translation layer that converts data warehouse and data lake table formats, supports various table types, and enforces uniform governance policies, allowing query engines to access and manage data across both systems while maintaining security and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate runtime catalogs are maintained for data lakes and data warehouses, then each system can operate independently with its own query engines, but data silos are created and synchronization becomes difficult
Solution Approach 1:
The patent merges separate runtime catalogs for data lakes and data warehouses into a single unified runtime catalog. This unified catalog serves both data lake query engines and data warehouse query engines, eliminating data silos while maintaining independent operation capabilities. The catalog contains translation layers that enable both types of query engines to access and interpret table metadata from either data source.
Solution Approach 2:
The patent introduces translation layers as intermediary components within the unified runtime catalog. These translation layers convert between data lake table formats and data warehouse table formats, enabling seamless communication between different query engine types without requiring separate catalogs. The intermediary translation mechanisms preserve all table attributes while making them accessible to both data lake and data warehouse query engines.
2Ease of operation
If independent runtime catalogs are updated separately, then each system maintains its own update schedule, but synchronization and consistency across systems becomes difficult
Solution Approach 1:
The patent combines multiple independently-updatable catalogs into a single unified runtime catalog that maintains a consistent state. The unified catalog allows update operations from either data lake or data warehouse sides while ensuring that all changes are reflected consistently across both data sources. This eliminates synchronization issues while preserving the ability to perform updates independently.
Solution Approach 2:
The patent implements feedback mechanisms within the unified runtime catalog that track and propagate changes between data lake and data warehouse components. When table metadata is updated through either interface, the translation layers ensure that corresponding updates are reflected in the unified catalog, providing automatic consistency feedback without requiring separate synchronization processes.
3Reliability
If data warehouse and data lake table formats are kept separate, then each format can be optimized for its specific use case, but cross-system querying becomes difficult
Solution Approach 1:
The patent introduces bidirectional translation layers as intermediary components within the unified runtime catalog. These translation layers enable data warehouse query engines to interpret data lake table formats and data lake query engines to interpret data warehouse table formats. The translation mechanisms preserve the optimized format-specific characteristics while enabling cross-system querying capability.
Solution Approach 2:
The unified runtime catalog implements universal table metadata structures that can represent both data lake and data warehouse table formats. The catalog is designed to handle multiple table types and formats simultaneously, providing a universal interface that maintains the reliability of format-specific optimizations while enabling versatile cross-system query operations.
4Ease of operation
If governance policies are enforced separately in each system, then each system can have its own security rules, but uniform security enforcement across systems becomes difficult
Solution Approach 1:
The patent merges separate governance policy enforcement mechanisms into a unified policy management system within the unified runtime catalog. This unified approach allows security policies to be defined and enforced consistently across both data lake and data warehouse operations. The system maintains separate policy management interfaces while ensuring uniform enforcement through the shared catalog and translation layers.
Data Source
AI summary
Aspects of the disclosure are directed to a runtime catalog for a cloud storage engine that unifies data lakes and data warehouses. The runtime catalog can expose a single universe of cloud storage tables through an endpoint for query engines for data lakes and another endpoint for query engines for data warehouses. The runtime catalog can allow the query engines for data lakes and the query engines for data warehouses to query any cloud storage table by representing data warehouse native tables in a format compatible with data lakes and representing data lake native tables in a format compatible with data warehouses.