Cloud Data Ingestion Buffering for Fast and Scalable Warehousing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in an in-memory database is challenging due to scalability and cost issues, while hyperscalers offer better scalability and lower costs, but integrating data storage across these systems poses challenges related to performance, cost, and data latency.
Innovation Solution
A data warehousing environment that extends to permit data storage in either an in-memory database or a Lakehouse architecture, leveraging hyperscalers, with mechanisms for schema enforcement and evolution, and using a storage-agnostic approach to manage data across both systems, incorporating tools like Apache Parquet and Delta Lake for efficient data management and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in an in-memory database, then data access speed is improved, but scalability and cost efficiency deteriorate
Solution Approach 1:
The patent segments data storage into two distinct layers: an in-memory database layer for frequently accessed data requiring fast retrieval, and a hyperscaler object storage layer for bulk data requiring scalability and cost efficiency. This segmentation allows each layer to optimize for its specific purpose without compromising the other.
Solution Approach 2:
The patent introduces a data lakehouse architecture as an intermediary layer between the in-memory database and hyperscaler storage. This mediator handles data transformation, format conversion (to Apache Parquet and Delta Lake formats), and coordination between the two storage systems, enabling seamless operation across the contradiction.
2Speed
If data is stored in an in-memory database, then data access speed is improved, but cost efficiency deteriorates
Solution Approach 1:
The patent applies local quality by storing only the most frequently accessed data in the expensive in-memory database, while bulk data resides in cost-effective hyperscaler storage. This selective placement optimizes the cost-performance ratio by matching data location with access patterns.
Solution Approach 2:
The patent segments the data storage architecture into hot data (in-memory) and cold data (hyperscaler), allowing the system to pay premium prices only for data that requires fast access, while storing the majority of data in economical storage solutions.
3Quantity of substance
If data storage is extended to hyperscalers, then scalability and cost efficiency are improved, but data latency and integration complexity increase
Solution Approach 1:
The patent implements preliminary action by pre-processing and formatting data in the data lakehouse layer before it needs to be accessed by the in-memory database. Data is transformed into optimized formats (Apache Parquet, Delta Lake) in advance, reducing the time required for data retrieval and processing.
Solution Approach 2:
The data lakehouse acts as a mediator that buffers and pre-processes data between hyperscaler storage and the in-memory database, reducing the latency impact of accessing data from distributed cloud storage by preparing data in advance.
4Adaptability or versatility
If a storage-agnostic approach is used, then adaptability and versatility are improved, but system complexity increases
Solution Approach 1:
The patent implements universality by designing the data lakehouse architecture to work with multiple storage backends (hyperscalers, on-premise storage) and multiple data formats (Apache Parquet, Delta Lake). This multi-functional design allows the same architecture to adapt to different storage environments without requiring complete redesign.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A landing zone is established in an open table format object storage. More specifically, the landing zone is implemented as a local ingestion table file. This represents a special subtype of the local data file that is used in the inbound layer when ingesting data from an external system. This local ingestion table file is further subdivided into an active data portion and an inbound buffer. When data is written from the external system, it is first placed in the inbound buffer. This data is then merged into the active data, after processing and postprocessing. The result of these postprocessing operations is then stored in the active data portion of the local ingestion table file.