Directory Table Metadata Deduplication for Internal Stage Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
During data replication across different deployments of a data sharing platform, discrepancies in file-level metadata can lead to duplicate files being loaded, as the content key for the same file may differ between source and destination deployments, and files may not be correctly registered in the directory table, resulting in inefficient deduplication and potential duplication of files.
Innovation Solution
The directory table on the internal stage is modified to store and retrieve original file-level metadata, with additional metadata columns to ensure accurate deduplication by refining the initial list of candidate files and inspecting user-defined metadata to correct inconsistencies, thereby preventing duplicate files from being loaded.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If file replication is performed across deployments using content key for deduplication, then replication speed is improved, but duplicate files may be loaded due to content key discrepancies between source and destination deployments
Solution Approach 1:
The patent introduces an intermediary deduplication mechanism that compares files against both the destination deployment's existing files and the source deployment's load history. This intermediary step resolves the contradiction by maintaining fast replication through content key matching while preventing duplicates through additional verification layers, ensuring reliability without sacrificing productivity
Solution Approach 2:
The system performs preliminary actions by pre-loading the source deployment's load history into the directory table before replication begins. This preliminary action enables the replication process to quickly identify and skip duplicate files during the actual replication, maintaining high speed while ensuring accurate deduplication through advance preparation of reference data
2Reliability
If directory table is modified to store original file-level metadata, then deduplication accuracy is improved, but device complexity increases
Solution Approach 1:
The patent segments the metadata storage by introducing a separate load history table that stores original file-level metadata independently from the main directory table. This segmentation allows the directory table to maintain its simple structure for fast access while the separate load history table handles the complex deduplication metadata, thus improving deduplication accuracy without significantly increasing the complexity of the primary directory table structure
Solution Approach 2:
The system creates a copy of the load history from the source deployment and stores it in the destination deployment's directory table. This copying approach enables accurate deduplication by having reference data available locally, while avoiding the need to modify the fundamental directory table structure, thereby maintaining simplicity while improving reliability
3Productivity
If files are replicated without checking load history, then replication speed is improved, but resource efficiency decreases due to duplicate file loading
Solution Approach 1:
The patent implements self-service deduplication where the replication process automatically checks the load history and skips duplicate files without requiring external intervention. This self-service mechanism maintains high replication speed by using efficient in-memory comparisons while preventing wasted resources on duplicate file transfers, thus achieving both productivity and resource efficiency
Solution Approach 2:
The system incorporates feedback by continuously comparing incoming files against the load history during replication and adjusting the replication process in real-time. When duplicates are detected, the system immediately skips those files, providing feedback-driven optimization that maintains high speed while eliminating resource waste on duplicate operations
Data Source
AI summary
Embodiments of the present disclosure provide techniques for deduplicating files during internal stage replication using a directory table of the replicated internal stage that is modified as a cache for storing and retrieving original file-level metadata for the replicated files. An initial list of candidate files for loading from the internal stage to a table of the target deployment is prepared based on the files listed in the internal stage, and refined using a directory table lookup. If there is any inconsistency between the files registered in the directory table and the files listed in the internal stage, the target deployment will inspect the user-defined file-level metadata to obtain original file-level metadata for each file that is present in the internal stage but not in the directory table. This information may be used during deduplication to ensure that no duplicate files are loaded.