Cloud Data Ingestion Buffering for Scalable Warehouse Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in in-memory databases is challenging due to scalability and cost issues, while hyperscalers offer better scalability and lower costs, but integrating them with in-memory databases poses challenges in maintaining data integrity and performance during data ingestion.
Innovation Solution
A data warehousing environment that supports storage in either an in-memory database or a Lakehouse architecture, leveraging hyperscalers, with mechanisms for schema enforcement and evolution, and using open data formats like Apache Parquet and Delta Lake for efficient data management and processing, integrating with Spark for advanced analytics and machine learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in an in-memory database, then data access speed is improved, but scalability and cost are worsened
Solution Approach 1:
The system segments data storage into two distinct layers: hot data stored in in-memory databases for fast access, and cold data stored in hyperscaler object storage for scalable, cost-effective retention. This segmentation allows the system to optimize for both speed and capacity by placing different data types in appropriate storage mediums.
Solution Approach 2:
An intermediary data ingestion service is introduced that acts as a mediator between external data sources and the dual storage system. This service automatically determines whether to write data to in-memory or object storage based on data characteristics, enabling seamless operation of the segmented storage architecture without requiring application-level decisions.
2Quantity of substance
If hyperscalers are used for data storage, then scalability and cost are improved, but data integrity and performance during ingestion are worsened
Solution Approach 1:
The system performs preliminary actions by establishing robust validation, schema enforcement, and error handling mechanisms within the data ingestion service before data is written to hyperscaler storage. This ensures data integrity is maintained from the point of ingestion, preventing corruption or inconsistent states in the scalable storage layer.
Solution Approach 2:
The data ingestion service implements feedback mechanisms that monitor data quality, validate schemas, and provide error reporting during the ingestion process. This feedback loop ensures that only valid, high-quality data is written to hyperscaler storage, maintaining reliability while leveraging scalable capacity.
3Productivity
If data is written directly to active data, then write speed is improved, but data consistency and quality are worsened
Solution Approach 1:
The data ingestion service serves as an intermediary layer between data sources and the active data tables. It performs schema validation, data quality checks, and transformation operations before data is written, ensuring both speed and quality are maintained without requiring direct application-to-storage writes.
Solution Approach 2:
Validation and transformation operations are performed as preliminary actions within the ingestion service before data is committed to storage. This approach maintains write speed by batching operations while ensuring data quality through pre-write validation, avoiding the need for slow post-write correction processes.
Data Source
AI summary
A landing zone is established in an open table format object storage. More specifically, the landing zone is implemented as a local ingestion table file. This represents a special subtype of the local data file that is used in the inbound layer when ingesting data from an external system. This local ingestion table file is further subdivided into an active data portion and an inbound buffer. When data is written from the external system, it is first placed in the inbound buffer. This data is then merged into the active data, after processing and postprocessing. The result of these postprocessing operations is then stored in the active data portion of the local ingestion table file.