Managed Data Lake Tables With Scalable Metadata for ACID Writes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data lakes built using open file formats and cloud object stores face challenges with atomicity, consistency, isolation, and durability (ACID) transactions, snapshot consistency, and read-after-write consistency, leading to lower write throughput, query performance, and operational overhead.

Innovation Solution

A managed table system that separates table data from physical metadata, using a scalable metadata storage system to enable ACID transactions, performant DML, and high throughput stream ingestion, with features like columnar format conversion, automatic storage optimization, and fine-grained security.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If open-source table formats are used to achieve ACID transactions on data lakes, then transaction support is improved, but write throughput decreases

Engineering Contradiction:

ImproveACID transaction supportVSAvoidwrite throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent segments the data lake system into multiple components: cloud object store for data storage, distributed file system for metadata management, and transaction log for ACID guarantees. This segmentation allows each component to be optimized independently, maintaining high write throughput while ensuring transactional integrity through the transaction log.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a transaction log as an intermediary layer between the cloud object store and the query engine. This transaction log acts as a mediator that records all data modifications, enabling ACID transactions without requiring the cloud object store itself to support complex transaction protocols, thus preserving high write throughput.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If open-source table formats are used to enable ACID transactions, then consistency is improved, but query performance decreases

Engineering Contradiction:

Improvedata consistencyVSAvoidquery performance

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The patent performs preliminary actions by pre-computing and storing metadata in the distributed file system, including data location information, schema definitions, and transaction state. This preliminary organization of data and metadata enables query engines to quickly locate and process data without scanning entire tables, maintaining high query performance while ensuring consistency through pre-established metadata structures.

Inventive Principle:

Principle #10Preliminary action

3Quantity of substance

If cloud object stores are used for data lake storage, then storage cost is reduced, but support for multi-object transactions is lost

Engineering Contradiction:

Improvestorage capacityVSAvoidmulti-object transaction support

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent merges the capabilities of cloud object stores with a distributed file system and transaction log. The cloud object store provides cheap, durable storage, while the distributed file system layer adds metadata management and the transaction log provides ACID guarantees. This combination achieves multi-object transaction support without sacrificing the cost benefits of cloud object storage.

Inventive Principle:

Principle #5Merging (Combining)

4Adaptability or versatility

If open-source table formats are used, then data lake openness is maintained, but operational overhead increases

Engineering Contradiction:

Improvedata lake opennessVSAvoidinfrastructure management overhead

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent creates a universal platform that works with multiple open-source table formats (Parquet, Avro, ORC) through a common architecture. The distributed file system and transaction log provide universal functionality for metadata management and transaction coordination across different formats, reducing operational overhead while maintaining data lake openness and format flexibility.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12511266B2Managed tables for data lakes

Publication Date: 2025.12.30 GOOGLE LLC

AI summary

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.