Distributed Table Commits With Ordered Committables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed table formats like Apache Iceberg™ face conflicts and concurrency issues when rewriting real-time data due to out-of-order writes or changes to committed files, leading to retries and inefficiencies.
Innovation Solution
A distributed streaming platform that prioritizes and reorders committables based on rules and heuristics, splitting the write path into a writer and a committer to process incoming records and generate committables, ensuring conflict-free commits by using a committer to order and prioritize maintainer and materializer committables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Apache Iceberg uses Compare-And-Set commits with conflict checks to ensure data consistency, then reliability is improved, but device complexity and processing time increase due to retries and conflict resolution
Solution Approach 1:
The patent applies preliminary action by performing maintenance operations (compaction, deletion) before the commit phase. Writers generate committables and stage them in a queue, while maintenance operations are scheduled and executed in advance. This separation ensures that when commits occur, the data is already in a consistent state, eliminating conflicts and the need for Compare-And-Set retry logic.
Solution Approach 2:
The patent segments the write path into distinct components: writers that generate committables, a committable queue that stages them, and a committer that processes them. This segmentation allows independent optimization of each component and eliminates the need for complex conflict resolution mechanisms by ensuring proper ordering and timing of operations.
2Reliability
If Apache Iceberg performs conflict checks and retries to handle concurrent operations, then reliability is improved, but productivity decreases due to operation rejections and retries
Solution Approach 1:
Maintenance operations are scheduled and executed in advance before commits occur. The committer scheduler prioritizes maintenance committables over writer committables, ensuring that compaction and deletion operations complete before new data is committed. This eliminates concurrent conflicts and allows continuous high-throughput writing without retries.
Solution Approach 2:
The committable queue acts as an intermediary between writers and the committer. Writers push committables to the queue without immediate commitment, and the committer processes them in order. This intermediary buffer decouples writers from commits, allowing concurrent writers to operate at full speed while commits are processed sequentially without conflicts.
3Reliability
If Apache Iceberg uses a linear history of snapshots to maintain data versioning, then reliability is improved, but device complexity increases due to snapshot management overhead
Solution Approach 1:
The patent segments snapshot management into automated committable generation and processing. Each maintenance operation (compaction, deletion) generates a committable that represents a discrete change to the snapshot history. The committer automatically processes these committables in order, maintaining a clean linear history without requiring complex manual snapshot management.
Solution Approach 2:
The system performs self-service by automatically generating committables for maintenance operations and processing them through the committer queue. The linear snapshot history is maintained automatically as committables are processed in order, eliminating the need for complex snapshot management logic and reducing operational complexity.
Data Source
AI summary
Systems and methods are directed to transforming data into a distributed table format. The system accesses new data for a topic. One or more materializers generate data files from the new data, whereby each of the materializers can process a specific range of offsets within a partition of the topic. The materializers also generate materializer committables corresponding to the data files. A maintainer performs one or more maintenance operations on previously committed files and generates a maintainer committable for each maintenance operation performed. The maintenance operations can include merging previously committed smaller files into a larger file and deleting obsolete files. Subsequently, a committer scheduler schedules the maintainer and materializer committables for a commit process. Scheduling includes prioritizing which committables are applied first and ordering materializer committables based on their offsets. The commit process is performed based on the scheduling resulting in a single snapshot of the table.