Lakehouse Data Import With Rewrite-Free Schema Evolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data loading methods for upgrading data lake tables to data lakehouse tables are computationally expensive and error-prone, particularly due to full-copy loading and the limitations of zero-copy loading, which fail to support schema evolution and partitioning schemes.
Innovation Solution
Implement rewrite-free loading that tracks and stores missing information from data files in table metadata, headers, or footers, creating composite files or copying data files without decoding, to maintain data integrity and support schema evolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full-copy loading is used to load data files to the data lakehouse, then data integrity and completeness are improved, but computational costs and storage costs increase significantly
Solution Approach 1:
The patent uses a selective copying approach where only the data file content is copied to the data lakehouse, while metadata (headers and footers) are preserved from the source location. This partial copying strategy maintains data integrity without requiring full-copy loading of all file contents, thereby reducing computational and storage costs.
Solution Approach 2:
The patent extracts and stores only the essential metadata information (headers and footers) from the source data files in the data lakehouse table metadata, rather than copying the entire file content. This extraction approach maintains the necessary data integrity information while significantly reducing the computational and storage overhead associated with full-copy loading.
2Use of energy by moving object
If zero-copy loading is used to load data files to the data lakehouse, then computational costs and storage costs are reduced, but data mapping errors and garbage collection issues occur
Solution Approach 1:
The patent employs a hybrid approach that copies only the necessary data file content to the data lakehouse while preserving metadata references to the original source location. This selective copying strategy avoids the data mapping errors and garbage collection issues associated with zero-copy loading, while still reducing computational and storage costs compared to full-copy loading.
Solution Approach 2:
The patent introduces table metadata as an intermediary layer that stores references to the original data file locations (headers and footers) while maintaining copies of the actual data content. This intermediary metadata structure enables accurate data mapping and prevents garbage collection issues while reducing the need for full-copy loading operations.
3Quantity of substance
If zero-copy loading is used to load data files to the data lakehouse, then storage costs are reduced, but schema evolution and partitioning scheme support are compromised
Solution Approach 1:
The patent copies data file content to the data lakehouse while preserving metadata that includes schema information and partitioning details from the source location. This selective copying approach maintains storage cost efficiency while ensuring full support for schema evolution and partitioning schemes, as the metadata preserves all necessary information about the data structure.
Solution Approach 2:
The patent performs preliminary extraction and storage of schema information and partitioning metadata before the actual data loading process. This preliminary action ensures that all necessary information for schema evolution and partitioning support is available in the data lakehouse table metadata, enabling full adaptability while maintaining storage cost efficiency.
4Reliability
If full-copy loading is used to load data files to the data lakehouse, then data completeness is improved, but query processing time increases
Solution Approach 1:
The patent uses selective copying to load only the necessary data file content to the data lakehouse, reducing the amount of data that needs to be processed during query operations. This approach maintains data completeness for the loaded files while significantly improving query processing speed by reducing the volume of data the query engine must read and process.
Solution Approach 2:
The patent extracts and stores only the essential data content in the data lakehouse while maintaining references to the original source location in metadata. This extraction approach ensures data completeness for the copied files while reducing the overall data volume that query processing must handle, thereby improving query processing speed.
Data Source
AI summary
Aspects of the disclosure are directed to managing files in data lakehouses using rewrite-free loading. Rewrite-free loading includes keeping track of information from data files that could be missing when imported to the data lakehouses without having to perform full-copy loading. Rewrite-free loading can store this information in table metadata or augment headers and/or footers of the data files with this information when importing to a data lakehouse. Rewrite-free loading allows for more accurate management of data lakehouses with lower computational costs.