Storage-Agnostic Semantic Artifact for Hybrid Warehouse Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data warehousing environments face challenges in managing large volumes of data efficiently, particularly due to the high cost and complexity of maintaining data in in-memory databases, while hyperscalers offer better scalability and cost-effectiveness, necessitating a storage-agnostic solution that integrates in-memory and object storage seamlessly.
Innovation Solution
A storage-agnostic semantic artifact is introduced in the data warehousing environment, allowing data to be stored in either an in-memory database or a Lakehouse architecture, leveraging hyperscalers, with mechanisms for schema enforcement and evolution, supporting both business intelligence and machine learning workloads, and utilizing open data formats like Apache Parquet and Delta Lake for efficient data management and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in an in-memory database, then data access speed is improved, but storage cost and complexity increase
Solution Approach 1:
The patent segments data storage into two distinct layers: hot data stored in in-memory databases for fast access, and cold data stored in object storage for cost-effective long-term retention. This segmentation allows the system to optimize for both speed and cost by placing different data types in appropriate storage media.
Solution Approach 2:
The patent introduces an intermediary layer consisting of a unified namespace and metadata management system that abstracts the underlying storage complexity. This intermediary enables seamless access to data regardless of whether it resides in memory or object storage, hiding the complexity from users while maintaining fast access paths.
2Adaptability or versatility
If data is stored in hyperscaler object storage, then scalability and cost-effectiveness are improved, but data access performance deteriorates
Solution Approach 1:
The patent applies local quality by optimizing storage characteristics for specific data access patterns. Frequently accessed data receives the quality of fast in-memory access, while infrequently accessed data utilizes the scalability and cost benefits of object storage. This localized optimization ensures each data element is stored with the appropriate performance characteristics.
Solution Approach 2:
The system performs preliminary actions by pre-loading frequently accessed data into in-memory databases before it is needed, and by establishing optimized access paths in advance. This allows the system to maintain fast access performance for critical data while leveraging object storage scalability for the broader dataset.
3Ease of operation
If a unified storage interface is implemented, then ease of operation is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal storage interface that provides a single, consistent API for accessing data regardless of the underlying storage medium. This unified namespace allows users to interact with both in-memory and object storage data through the same interface, eliminating the need for separate access mechanisms and simplifying operations.
Solution Approach 2:
The unified namespace acts as an intermediary that translates simple user requests into complex storage operations. It handles the complexity of data location, retrieval, and management internally while presenting a simplified interface to users, effectively hiding the storage architecture complexity from end users.
4Productivity
If in-memory storage is used, then data processing speed is improved, but cost increases
Solution Approach 1:
The patent changes the storage parameter from exclusively in-memory to a hybrid configuration, adjusting the proportion of data stored in memory versus object storage based on access patterns and cost considerations. This parameter optimization allows the system to maintain high processing speeds for critical operations while reducing overall storage costs by placing less critical data in cheaper storage media.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In an example embodiment, a data warehousing environment, or a similar architecture, is extended to permit data storage in either an in-memory database or a Lakehouse architecture, which leverages one or more hyperscalers for the underlying storage. More specifically, a single artifact is defined in Datasphere that stores data in either the in-memory database or the Lakehouse architecture, and does so in a storage-agnostic manner.