Managed Table Metadata for ACID Data Lake Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data lakes built using open file formats and cloud object stores face challenges with atomicity, consistency, isolation, and durability (ACID) transactions, snapshot consistency for reads, and strong read-after-write consistency, leading to lower write throughput, query performance, and operational overhead.
Innovation Solution
A managed table system that separates table data from physical metadata, using a scalable metadata storage system to enable ACID transactions, performant data manipulation, and high throughput stream ingestion, with features like columnar formatting and automatic storage optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If open file formats and cloud object stores are used for data lakes, then storage cost is reduced and durability is improved, but ACID transaction support deteriorates and operational overhead increases
Solution Approach 1:
The patent introduces a managed table service as an intermediary layer between the query engine and cloud object storage. This service provides ACID transaction support, metadata management, and operational simplification while leveraging the durability and cost benefits of cloud object stores. The intermediary handles complex operations like transactions, snapshots, and consistency guarantees that would otherwise require significant operational overhead when using raw open file formats on object storage.
2Reliability
If open-source table formats are used to achieve ACID transactions, then transaction support is improved, but write throughput decreases and query performance deteriorates
Solution Approach 1:
The patent segments the data lake system into distinct components: cloud object storage for durable data retention, a managed table service for transaction management, and a query engine for data access. This segmentation allows each component to be optimized independently - object storage for durability and cost, the table service for ACID transactions and metadata management, and the query engine for high-performance data access. This resolves the contradiction by avoiding the performance penalties of open-source table formats while maintaining ACID guarantees through the managed service layer.
3Ease of manufacture
If cloud storage interfaces with single object mutations are used, then storage simplicity is maintained, but multi-object transaction atomicity deteriorates
Solution Approach 1:
The patent merges multiple cloud storage objects (data files and metadata files) into a unified managed table structure. This consolidation enables atomic multi-object operations through the managed table service, which can simultaneously update data files and their corresponding metadata files while maintaining ACID guarantees. The merge allows the system to maintain storage simplicity by using standard cloud object storage while achieving transaction atomicity through the unified table abstraction.
Data Source
AI summary
Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.