Virtual Data Warehouse Table Links for Multi-Tenancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-tenancy data warehouses face challenges in managing data replication and capacity as they grow, with traditional solutions either leading to inefficient data duplication or high operational costs due to the need for multiple physical warehouses.
Innovation Solution
The implementation of table links within a data warehouse allows tenants to access and import data from other virtual data warehouses without replication, using parameters like data location and time ranges, and dynamically updates links based on usage patterns to optimize data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is copied to each smaller warehouse, then data access is improved, but data replication increases
Solution Approach 1:
The patent uses symbolic copying through table links rather than physical data duplication. When a table is linked from one virtual warehouse to another, only the table definition and metadata are copied, not the actual data. This allows multiple virtual warehouses to reference the same physical data without replication, resolving the contradiction between data access and data replication.
Solution Approach 2:
The patent introduces a new dimension of virtualization by separating the logical view (virtual warehouses) from the physical storage (data lakes). This dimensional separation allows multiple virtual warehouses to access the same physical data through parameterized views, eliminating the need for data replication while maintaining ease of access.
2Volume of stationary object
If data is deleted from the data warehouse, then storage capacity is improved, but data availability for future analysis is reduced
Solution Approach 1:
The patent makes the data lake serve multiple functions simultaneously: it acts as both the primary storage repository and the source for multiple virtual warehouses. By making the data lake universal and accessible to all virtual warehouses through table links, the system maintains data availability for future analysis while preventing storage capacity issues through efficient parameterized access.
3Quantity of substance
If multiple physical warehouses are created, then data capacity is improved, but operational costs increase
Solution Approach 1:
The patent merges multiple virtual warehouses into a single physical data lake infrastructure. By combining the storage resources into one physical system while maintaining multiple logical views through table links, the system achieves high data capacity without the operational costs and complexity of managing multiple separate physical warehouses.
4Speed
If data is replicated across virtual warehouses, then data access speed is improved, but storage efficiency is reduced
Solution Approach 1:
The patent implements symbolic copying where only table definitions and metadata are replicated across virtual warehouses, not the actual data. This allows fast local access to table schemas and query optimization while the actual data remains stored once in the data lake, maintaining storage efficiency. The system achieves data access speed through parameterized views and query optimization rather than physical data replication.
Data Source
AI summary
Some embodiments include a plurality of virtual data warehouses having table link capabilities that are built on top of a data center (e.g., running Apache Hive). Each virtual data warehouse can be modeled as a database and manage data in forms of database tables. The virtual data warehouse can include links which import tables from other virtual data warehouses by reference. Each link may contain partition metadata for the table partitions by dates of the source table and retention metadata to declare the needed retention time period for the partitions of the source table. The links can be dynamic and update when the corresponding source table receives new partitions or drops partitions. When a virtual data warehouse is migrated to another data center, the system can retain necessary table partitions on the current data center based on the partition and retention metadata of the links.