Managed Dynamic Iceberg Tables for Incremental Refresh and External Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in data processing, including inefficient workload distribution, high latency, and challenges in configuring queries and refreshing database tables, which can be time-consuming and resource-intensive.
Innovation Solution
The implementation of managed dynamic Iceberg tables, which are configured using a network-based database system (NBDS) to manage and store data in customer-supplied storage, allowing for incremental refreshes and automated orchestration of data pipelines through a declarative SQL interface, leveraging change-tracking columns for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database systems are used for data storage and processing, then data can be organized and accessed in tables, but workload distribution is inefficient and latency is high
Solution Approach 1:
The patent segments data storage into dynamic iceberg tables that separate frequently accessed data from historical data, allowing efficient querying of recent data while maintaining comprehensive data retention. This segmentation enables parallel processing and improves workload distribution across storage layers.
Solution Approach 2:
The patent introduces a temporal dimension to data storage by implementing time-travel queries and versioned tables, allowing users to query data as it existed at any point in time. This adds a new dimension to traditional table-based storage, enabling efficient historical analysis without impacting current data access performance.
2Ease of operation
If manual configuration and refresh of database tables is performed, then queries can be consumed and tables updated, but the process is challenging and time-consuming
Solution Approach 1:
The patent implements self-service dynamic iceberg tables that automatically refresh and update themselves based on incoming data streams. The system autonomously manages table configurations, performs incremental updates, and maintains data synchronization without requiring manual intervention, thereby reducing both operational complexity and refresh time.
Solution Approach 2:
The patent pre-configures table schemas, data types, and refresh policies in advance through declarative definitions. This preliminary setup allows the system to automatically execute optimized refresh operations without requiring real-time configuration decisions, reducing both the complexity of operation and the time required for updates.
3Adaptability or versatility
If data is copied and moved between storage systems, then data can be accessed from different locations, but costs increase due to data copying requirements
Solution Approach 1:
The patent introduces a virtualization layer that acts as an intermediary between users and physical storage locations. This layer provides unified access to data regardless of its physical location, eliminating the need for data copying while maintaining flexibility in storage placement. The virtualization layer manages data routing and access optimization without requiring data movement.
4Productivity
If comprehensive data management features are implemented, then data can be organized, queried, and refreshed, but device complexity increases
Solution Approach 1:
The patent implements a universal dynamic iceberg table framework that handles multiple data management functions through a single unified interface. The same table structure supports querying, refreshing, versioning, and time-travel operations, reducing the need for separate complex systems for each function while maintaining comprehensive data management capabilities.
Data Source
AI summary
Provided herein are systems and methods for configuring managed dynamic Iceberg tables. An example method includes parsing, by at least one hardware processor, a table definition to determine a lag duration value, an external volume indicator, and a location indicator. A dynamic table (DT) manager generates a dynamic Iceberg table based on the table definition. The generating is based on selecting an external storage volume of a network-based database system based on the external volume indicator and the location indicator. The DT manager stores a base Iceberg table at a storage location associated with the external storage volume. The DT manager configures the base Iceberg table as the dynamic Iceberg table based on the lag duration value. The lag duration value indicates a maximum time period that a result of a prior refresh of the dynamic Iceberg table lags behind a current time instance.