Storage-Agnostic Semantic Artifact for Tiered Cloud Data Warehousing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in a data warehousing environment is challenging due to the high cost and difficulty of maintaining data in an in-memory database, while hyperscalers offer better scalability and cost-efficiency for long-term storage, necessitating a seamless integration of both storage types.
Innovation Solution
A storage-agnostic approach is implemented in the data warehousing environment, allowing data to be stored in either an in-memory database or a Lakehouse architecture, leveraging hyperscalers, with mechanisms for schema enforcement and evolution, and utilizing open data formats like Apache Parquet and Delta Lake for efficient data management and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in an in-memory database, then data access speed is improved, but storage cost and maintenance difficulty increase
Solution Approach 1:
The patent segments data storage into multiple tiers: hot data in in-memory database for fast access, warm data in columnar storage for balanced performance, and cold data in object storage for cost efficiency. This segmentation resolves the contradiction by allowing different storage media to serve different access speed requirements while managing overall system complexity.
Solution Approach 2:
The patent introduces a temporal dimension to data storage by implementing data lifecycle management that automatically transitions data between storage tiers based on age and access patterns. This dimensional approach allows the system to optimize for speed when needed while reducing maintenance complexity through automated policies.
2Adaptability or versatility
If data is stored in hyperscaler object storage, then scalability and cost-efficiency are improved, but data access performance deteriorates
Solution Approach 1:
The patent applies local quality by optimizing storage characteristics for specific data types and access patterns. Frequently accessed data is placed in in-memory or columnar storage with high-speed access characteristics, while less frequently accessed data resides in object storage. This localized optimization resolves the contradiction by ensuring high scalability through object storage while maintaining high access performance for critical data.
Solution Approach 2:
The patent implements preliminary action through data pre-loading and caching mechanisms. Data that is predicted to be needed is pre-loaded from object storage into faster storage tiers before actual access occurs. This anticipatory approach maintains scalability while improving access performance by reducing the frequency of slow object storage accesses.
3Ease of operation
If a unified storage interface is implemented, then ease of operation is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary layer called a unified storage interface that abstracts the complexity of multiple storage systems. This interface provides consistent APIs and management operations across in-memory, columnar, and object storage tiers, allowing users to manage diverse storage resources through a single simplified interface while the underlying complexity is handled by the intermediary layer.
Solution Approach 2:
The unified storage interface implements universality by providing a single management plane that can operate across multiple storage types and providers. This multi-functional interface handles data movement, policy enforcement, and access operations uniformly regardless of the underlying storage technology, improving ease of operation while distributing system complexity across standardized components.
4Manufacturing precision
If data is kept in native format in data lake, then data quality is improved, but compatibility with data warehousing operations deteriorates
Solution Approach 1:
The patent applies asymmetry by maintaining different data representations for different operational needs. Raw data in native format is preserved in the data lake for quality and fidelity, while transformed and curated data in standardized formats is maintained in the data warehouse for operational compatibility. This asymmetric approach allows both high data quality and broad compatibility to coexist by serving different purposes with different data forms.
Solution Approach 2:
The patent implements dynamics through dynamic data format conversion and enrichment processes. Data transitions from static native format in the data lake to dynamically transformed formats in the data warehouse based on operational requirements. This dynamic transformation pipeline maintains data quality by preserving the original native format while creating compatible versions as needed, resolving the contradiction between quality and compatibility.
Data Source
AI summary
In an example embodiment, a data warehousing environment, or a similar architecture, is extended to permit data storage in either an in-memory database or a Lakehouse architecture, which leverages one or more hyperscalers for the underlying storage. More specifically, a single artifact is defined in Datasphere that stores data in either the in-memory database or the Lakehouse architecture, and does so in a storage-agnostic manner.