Dynamic Table Dependency Tracking for Incremental Refresh

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database systems face inefficiencies in data processing, including inefficient workload distribution, high latency, and challenges in query configuration and refresh, particularly in managing dynamic tables, which can lead to costly and time-consuming processes.

Innovation Solution

The implementation of dynamic tables (DTs) with a DT manager for automatic evolution, allowing incremental refreshes and automatic determination of base object dependencies, reducing the need for costly reinitialization and improving query flexibility.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If dynamic tables are manually configured and refreshed, then query flexibility and data freshness are improved, but configuration complexity and time consumption increase

Engineering Contradiction:

Improvequery flexibilityVSAvoidconfiguration complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system automatically tracks dependencies between dynamic tables and base objects, and autonomously performs refresh operations without manual intervention. The database manager monitors changes in base objects and automatically updates dependent dynamic tables, eliminating the need for users to manually configure refresh schedules or dependency relationships.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements a feedback mechanism where the database manager continuously monitors changes in base objects and automatically triggers refresh operations in dependent dynamic tables. This closed-loop system ensures that dynamic tables are always updated based on the current state of their dependencies, maintaining data freshness automatically.

Inventive Principle:

Principle #23Feedback

2Reliability

If dynamic tables are manually refreshed, then data freshness is improved, but processing time and computational resources increase

Engineering Contradiction:

Improvedata freshnessVSAvoidrefresh time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary tracking of dependencies between dynamic tables and base objects, so that when changes occur, the system already knows which tables need refreshing and can execute updates immediately without delay. This advance preparation eliminates the need for manual discovery of dependencies during refresh operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces manual mechanical refresh operations with an automated database manager system that uses change tracking and dependency monitoring. Instead of users manually executing refresh commands, the system automatically detects changes and triggers appropriate refresh operations based on tracked dependencies, reducing both time and computational overhead.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Adaptability or versatility

If dynamic tables are reinitialized, then schema changes are accommodated, but computational costs and downtime increase

Engineering Contradiction:

Improveschema adaptabilityVSAvoidcomputational cost

Core Design Contradiction:

Adaptability or versatilityVSLoss of energy

Solution Approach 1:

The system segments the refresh operation into targeted updates based on tracked dependencies rather than performing complete reinitializations. When schema changes occur in base objects, only the specific dynamic tables that depend on those objects are updated, leaving other tables unchanged. This segmented approach reduces computational costs significantly.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts refresh strategies based on actual dependency relationships tracked in the database manager. Instead of using static reinitialization approaches, the system adapts its behavior by monitoring which dynamic tables depend on which base objects, and automatically determines the minimal set of tables that need updating when schema changes occur.

Inventive Principle:

Principle #15Dynamics

4Productivity

If base object dependencies are manually tracked, then query optimization is improved, but system complexity and maintenance effort increase

Engineering Contradiction:

Improvequery performanceVSAvoidtracking complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The database manager automatically tracks and maintains dependency relationships between base objects and dynamic tables without user intervention. The system self-monitors schema changes and maintains an updated dependency graph, eliminating the need for users to manually track dependencies while still enabling optimized query execution.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS20250370979A1Automatic evolution of dynamic tables based on object tracking

Publication Date: 2025.12.04 SNOWFLAKE INC

AI summary

Provided herein are systems and methods for configuring automatic evolution of dynamic tables. An example method includes parsing, by at least one hardware processor, a query associated with a dynamic table to determine a current base object dependency of the dynamic table on at least a first base object. A prior base object dependency of the dynamic table on at least a second base object is retrieved. A delta between data stored by the at least first base object and data stored by the at least second base object is determined. The dynamic table is updated based on the delta.