Iceberg Table Auto-Refresh with Split Metadata Pipelines

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database systems face inefficiencies in workload distribution, high latency, and inefficient allocation of compute resources during data processing, particularly in managing and refreshing database tables like Iceberg tables.

Innovation Solution

The implementation of a file processing manager (FPM) that configures a source monitor pipeline and a refresh pipeline to auto-refresh Iceberg tables, utilizing a processing pipeline definition (PPD) to manage metadata retrieval, generate query plans, and allocate compute resources efficiently.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional database systems are used for storing and processing data, then data can be organized and accessed, but workload distribution is inefficient and latency is high

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidlatency

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent segments data into structured tables and unstructured files, processed by different pipelines (refresh pipeline for tables, file processing pipeline for files). This segmentation allows parallel processing and optimizes workload distribution, directly addressing the inefficiency in traditional monolithic database systems.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts processing based on data type and pipeline status. The file processing manager monitors the refresh pipeline's workload and dynamically routes files to either the refresh pipeline (when idle) or a separate file processing pipeline (when busy), optimizing resource utilization and reducing latency.

Inventive Principle:

Principle #15Dynamics

2Power

If more compute resources are allocated to data processing, then processing capacity increases, but resource allocation becomes inefficient

Engineering Contradiction:

Improvecompute resource capacityVSAvoidcompute resource waste

Core Design Contradiction:

PowerVSLoss of energy

Solution Approach 1:

The file processing manager implements self-service by automatically monitoring pipeline status and making routing decisions without manual intervention. It dynamically allocates files to appropriate pipelines based on current workload, ensuring compute resources are utilized efficiently without waste.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system incorporates feedback mechanisms where the file processing manager continuously monitors the refresh pipeline's execution status and uses this feedback to dynamically adjust file routing. This closed-loop control optimizes resource allocation by matching compute capacity to actual processing needs.

Inventive Principle:

Principle #23Feedback

3Productivity

If manual refresh processes are used for database tables, then data can be updated, but the process is time-consuming and labor-intensive

Engineering Contradiction:

Improvetable refresh speedVSAvoidrefresh time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-configuring processing pipelines and establishing file-to-table mappings before data refresh is needed. The refresh pipeline is pre-built with knowledge of table schemas and dependencies, enabling immediate execution when triggered, thus reducing refresh time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces manual mechanical refresh operations with an automated electronic pipeline system. The file processing manager and refresh pipeline automatically execute data loading and transformation tasks that would otherwise require manual intervention, significantly reducing both time and human effort.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Ease of manufacture

If a simple file processing approach is used, then implementation is straightforward, but workload distribution is inefficient

Engineering Contradiction:

Improvesystem implementation simplicityVSAvoidworkload distribution efficiency

Core Design Contradiction:

Ease of manufactureVSProductivity

Solution Approach 1:

The file processing manager serves multiple functions: it monitors pipeline status, routes files to appropriate pipelines, manages workload distribution, and coordinates between the refresh pipeline and file processing pipeline. This multi-functional design maintains implementation simplicity while achieving efficient workload distribution through a single coordinating component.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20250362952A1Iceberg table auto-refresh

Publication Date: 2025.11.27 SNOWFLAKE INC

AI summary

Provided herein are systems and methods for data table auto-refresh. An example method includes configuring a first processing pipeline definition comprising a first plurality of configurations associated with a corresponding plurality of notification fetching jobs for metadata of a database table. A second processing pipeline definition is configured to include a second plurality of configurations associated with the metadata. A source monitor pipeline is instantiated based on the first processing pipeline definition to fetch a manifest file based on the first plurality of configurations. A refresh pipeline is instantiated based on the second processing pipeline definition to perform a refresh operation of the metadata and generate refreshed metadata based on the second plurality of configurations.