Database Row Data Duplication via Data Lakehouse Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently processing and storing large volumes of data due to limitations in hardware, data storage mechanisms, and restricted co-processing options, leading to bottlenecks in data processing and query execution.
Innovation Solution
A database system architecture that utilizes a parallelized data input, storage, and query processing system, employing multiple field-based storage mechanisms and distributed query execution across multiple nodes, to enhance data processing efficiency and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a traditional database system with centralized storage, then data consistency and reliability are maintained, but data processing speed and scalability are limited due to hardware constraints
Solution Approach 1:
The patent segments data storage into two distinct layers: a database system storing only metadata and transaction logs, and a data lake storing actual row data. This segmentation allows the database to remain lightweight and fast for transactions while the data lake handles bulk storage and analytics, thereby improving data processing speed without significantly increasing overall system complexity.
Solution Approach 2:
The patent introduces a new dimensional aspect to data storage by separating operational data storage (database) from analytical data storage (data lake). This dimensional separation enables simultaneous optimization for both transactional speed and analytical scalability, resolving the contradiction between processing speed and system complexity.
2Adaptability or versatility
If distributed storage is implemented across multiple nodes, then scalability and processing parallelism are improved, but data retrieval complexity and latency increase
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing columnar data representations in the data lake alongside row-based storage. When queries are executed, the system can directly retrieve pre-processed columnar data without performing complex real-time aggregations, thereby reducing data retrieval latency despite distributed storage architecture.
Solution Approach 2:
The patent introduces an intermediary layer (the data lake with columnar storage) between the distributed database nodes and analytical query workloads. This intermediary pre-aggregates and organizes data in columnar format, mediating between the distributed storage complexity and the need for fast analytical queries, thus reducing retrieval latency.
3Productivity
If columnar storage is used for analytical queries, then query performance is improved, but storage efficiency and write operations are compromised
Solution Approach 1:
The patent merges two storage formats within the same data lake infrastructure: row-based storage for efficient data ingestion and writing, and columnar storage for efficient analytical querying. This merging allows the system to maintain both storage efficiency for incoming data and query performance for analytical workloads without sacrificing one for the other.
Solution Approach 2:
The patent implements dynamic storage format selection where data is initially ingested in row-based format for efficient writes, then transformed and stored in columnar format for analytical queries. This dynamic approach allows the system to optimize storage format based on the operational phase (ingestion vs. querying), thereby maintaining both storage efficiency and query performance.
4Adaptability or versatility
If data is duplicated across database and data lake, then data accessibility and processing flexibility are enhanced, but storage redundancy increases
Solution Approach 1:
The patent applies local quality by storing data in different formats and locations optimized for specific purposes: row-based storage in the database for transactional operations and columnar storage in the data lake for analytical queries. This localized optimization allows efficient data processing for each use case without requiring full duplication of data across both systems, thereby reducing total data volume while maintaining processing flexibility.
Data Source
AI summary
A database system is operable to store a plurality of segment row data via a first storage mechanism corresponding to a first durability level and facilitate storage of the plurality of segment row data via at least one file stored in a second storage mechanism corresponding to a second durability level, where table metadata corresponding to the at least one file is further stored via the second storage mechanism. A failure of storage of one of the plurality of segment row data via the first storage mechanism is detected. The one of the plurality of segment row data is recovered for storage via the first storage mechanism based on accessing of the at least one file based on accessing the table metadata corresponding to the at least one file based on communications between at least one storage system interface and a metadata processing system of the second storage mechanism.