Cloud Data Transformation Flow for Lakehouse Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in an in-memory database is challenging due to scalability and cost issues, while hyperscalers offer better scalability and cost-efficiency, but integrating data storage across these systems is complex and requires a unified, seamless solution.
Innovation Solution
A data warehousing environment that extends to permit data storage in either an in-memory database or a Lakehouse architecture, leveraging hyperscalers for underlying storage, with mechanisms for schema enforcement and evolution, and supports both business intelligence and machine learning workloads, using open data formats like Apache Parquet and Delta Lake, and integrating with Spark for advanced data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in an in-memory database, then data access speed is improved, but scalability and cost efficiency deteriorate when managing large volumes of data
Solution Approach 1:
The system segments data storage into two distinct layers: hot data stored in an in-memory database for fast access, and cold data stored in cloud-based object storage for scalability. This segmentation allows the system to maintain high-speed access for frequently accessed data while achieving scalable storage for large volumes of data across different temperature zones.
Solution Approach 2:
The patent introduces a data lakehouse architecture as an intermediary layer between the in-memory database and cloud object storage. This mediator manages data movement, format conversion, and access coordination, enabling seamless integration of volatile and non-volatile storage while maintaining scalability and access performance.
2Speed
If data is stored in an in-memory database, then data access speed is improved, but cost efficiency deteriorates when managing large volumes of data
Solution Approach 1:
The system segments data storage into hot data in in-memory databases and cold data in cloud object storage, ensuring that only frequently accessed data consumes expensive in-memory resources. This segmentation optimizes cost efficiency by aligning storage resource usage with actual access patterns while maintaining fast access speeds for critical data.
Solution Approach 2:
The patent applies local quality by providing different storage characteristics to different data based on access requirements. Frequently accessed data receives high-speed in-memory storage, while less frequently accessed data is stored in cost-effective cloud object storage, optimizing the overall cost-performance ratio of the data storage system.
3Adaptability or versatility
If data storage is extended to include Lakehouse architecture with hyperscalers, then scalability and cost efficiency are improved, but system integration complexity increases
Solution Approach 1:
The data lakehouse architecture serves multiple functions: it acts as a data lake for unstructured data, a data warehouse for structured data, and a processing layer for transformations. This multi-functionality reduces the need for separate systems while maintaining scalability, though it does increase integration complexity that must be managed through standardized interfaces.
Solution Approach 2:
The patent introduces standardized data formats (Apache Parquet, Delta Lake) and processing frameworks (Spark) as intermediaries that simplify integration between the in-memory database and cloud object storage. These standardization layers provide common protocols and tools that reduce the complexity of managing heterogeneous storage systems.
4Adaptability or versatility
If multiple storage types are integrated, then data management flexibility is improved, but data consistency and quality maintenance becomes more difficult
Solution Approach 1:
The patent enforces schema consistency across different storage types by implementing centralized schema validation and evolution mechanisms. This homogenization approach ensures that data maintains consistent structure and quality standards whether stored in in-memory databases or cloud object storage, despite the heterogeneity of the underlying storage systems.
Data Source
AI summary
A solution for transforming and managing data in a data lake environment is disclosed. A definition of a transformation flow is received via a graphical user interface. The transformation flow specifies operations to transform source data into target data stored in a first format object storage. A virtual procedure corresponding to the transformation flow is generated and stored in an in-memory cloud database. Then the virtual procedure is executed, causing retrieval of metadata from the first format object storage and triggering a transformation job on a computing cluster. The transformation job reads the source data from the first format object storage, applies one or more transformations specified in the definition of the transformation flow, and writes results to target data in the first format object storage.