Zero-Copy Table Duplication Using Shared Data Block References
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Duplicating large tables in cloud databases is resource-intensive and time-consuming due to the need for reading, decoding, and re-encoding data blocks, leading to excessive disk usage and computational overhead.
Innovation Solution
A query optimizer that increments reference counts and appends metadata references for data blocks during copying, allowing a 'shallow copy' operation without duplicating data, thus optimizing the process to O(1) complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional copying methods are used to duplicate data blocks from a first table to a second table, then the operation is straightforward and reliable, but system resource consumption and storage costs increase significantly
Solution Approach 1:
The patent applies the copying principle by creating metadata references (pointers) that point to the original data blocks instead of physically duplicating the data blocks themselves. This allows the second table to reference the same data blocks as the first table, achieving the effect of copying table structure and data access patterns without the resource cost of actual data duplication.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the first table and the second table. This metadata contains references to the original data blocks, acting as a mediator that allows the second table to access data blocks from the first table without direct physical copying. The metadata serves as a lightweight proxy that enables table duplication operations with minimal resource consumption.
2Reliability
If traditional copying methods are used to duplicate data blocks, then complete data independence is achieved, but storage costs and disk write operations increase dramatically
Solution Approach 1:
The patent creates a logical copy of the table structure through metadata references rather than physical data duplication. The second table receives a copy of the metadata structure that references the original data blocks, achieving table-level independence while maintaining data block sharing, thus reducing storage capacity requirements.
Solution Approach 2:
The patent merges the storage of data blocks across multiple tables by allowing the second table to reference the same data blocks as the first table. This consolidation reduces redundant storage and eliminates duplicate data blocks, while the metadata layer maintains the logical separation and independence of each table.
3Reliability
If data blocks are physically copied from the first table to the second table, then data access is independent, but query latency increases due to decoding and duplication overhead
Solution Approach 1:
The patent performs a lightweight copy operation by duplicating only the metadata references rather than the actual data blocks. This metadata copying operation is significantly faster than physical data block copying, reducing query latency while maintaining the ability of the second table to access data independently through the reference structure.
Solution Approach 2:
The patent extracts only the essential metadata component from the full data block structure, separating the reference management function from the actual data storage. By taking out and duplicating only the metadata layer, the system achieves table independence without the performance penalty of copying the entire data blocks, thus improving query latency.
4Reliability
If traditional table duplication is performed, then complete table copying is achieved, but computational resources and time are excessively consumed
Solution Approach 1:
The patent implements a selective copying strategy that duplicates only the metadata structure and references rather than the entire table data. This approach achieves complete table duplication in terms of logical structure and data access patterns, while the operation speed is dramatically improved by avoiding the computational overhead of copying actual data blocks.
Solution Approach 2:
The patent applies partial action by performing only the necessary metadata copying operation rather than the complete physical data block copying. This partial action (copying only metadata) is sufficient to achieve the goal of table duplication and independence, while significantly reducing the computational resources and time required compared to excessive full data copying.
Data Source
AI summary
A computer-implemented method includes receiving a query specifying an operation to perform on a first table of a plurality of data blocks stored. Each data block in the first table includes a respective reference count indicating a number of tables referencing the data block. The method also includes determining that the operation specified by the query includes copying the plurality of data blocks in the first table into a second table and, in response, for each data block of the plurality of data blocks in the first table copied into the second table, incrementing, the respective reference count associated with the data block in the first table, appending, by the data processing hardware, into metadata of the second table, a reference of the corresponding data block copied into the second table.