Unstructured Staged Data Replication via Directory Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network-based database systems lack efficient methods for replicating unstructured staged data across different deployments, requiring manual and time-consuming processes that are impractical for large data sets.
Innovation Solution
A staged data replication service that replicates unstructured data by using a directory table and stage metadata to facilitate replication between database deployments, allowing access to data without copying it, or copying and storing it at the remote deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual replication processes are used for unstructured staged data, then data can be replicated between deployments, but the process becomes time-consuming and impractical for large data sets
Solution Approach 1:
The patent uses metadata copying instead of actual data copying. The system replicates only the directory database table and stage metadata between deployments, while the actual unstructured data remains in place at the source. This allows the destination deployment to access and use the data without physical duplication, dramatically reducing replication time and resource consumption while maintaining full data accessibility and integrity
Solution Approach 2:
The patent extracts and replicates only the essential metadata components (directory database table and stage metadata) that are needed to access and manage the unstructured data, separating these from the actual data files. This extraction approach allows rapid replication of the data access framework without the burden of copying terabytes or petabytes of actual data, solving the time-consuming replication problem
2Reliability
If manual replication processes are used for unstructured staged data, then data can be replicated between deployments, but the process requires significant manual effort
Solution Approach 1:
The system implements self-service replication by automatically identifying staged data in the source deployment and replicating the necessary metadata to the destination deployment without manual intervention. The staged data replication service autonomously manages the entire replication process, including identifying the directory database table and stage metadata, copying them to the destination, and making the data accessible there, thereby eliminating manual effort while ensuring reliable data replication
3Ease of operation
If data is copied and stored at remote deployment, then data accessibility is improved, but storage requirements and data transfer time increase significantly
Solution Approach 1:
The patent implements a metadata-only copying strategy where only the directory database table and stage metadata are replicated to the destination deployment, while the actual unstructured data files remain at the source. This approach maintains full data accessibility at the destination through metadata references without duplicating the actual data storage, thereby eliminating the storage overhead and data transfer time associated with traditional data copying methods
Data Source
AI summary
The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.