Data Replication Partitioning for RDBMS to Hadoop Coherency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication methods, especially when merging changes from a relational database management system (RDBMS) into a write-once file system like Hadoop, are inefficient and slow, particularly when dealing with large data sets, leading to delays in maintaining database coherency.
Innovation Solution
A system and method that separates data into priority sets, storing higher priority data on faster storage devices and lower priority data on slower devices, optimizing data organization and storage based on access likelihood, thereby improving replication and update efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is copied from RDBMS to Hadoop using conventional replication methods, then data coherency is maintained, but replication speed and update efficiency deteriorate due to read-merge-rewrite operations in write-once file systems
Solution Approach 1:
The patent segments data into two distinct sets: frequently updated data and infrequently updated data. This segmentation allows different storage strategies to be applied to each data set, thereby optimizing both replication speed and update efficiency while maintaining data coherency.
Solution Approach 2:
The patent applies local quality by storing frequently updated data on fast storage devices (such as RDBMS or SSDs) where rapid read-merge-rewrite operations can occur, while storing infrequently updated data on slow storage devices (such as HDFS). This localized optimization resolves the contradiction by matching storage characteristics to data access patterns.
2Reliability
If entire database is re-copied to maintain coherency, then data consistency is ensured, but time and resources required for replication increase significantly
Solution Approach 1:
The patent extracts only the frequently updated data subset from the entire database and replicates this smaller set to the fast storage device. This extraction approach maintains data consistency for the critical subset while dramatically reducing replication time compared to copying the entire database.
Solution Approach 2:
The patent applies partial action by replicating only the necessary subset of data (frequently updated records) rather than the complete database. This partial replication strategy achieves sufficient data consistency for operational needs while minimizing replication time and resource consumption.
3Reliability
If changes are merged into second database using read-merge-rewrite process, then data coherency is maintained, but processing efficiency deteriorates due to sequential operations on large data sets
Solution Approach 1:
The patent segments data based on update frequency, separating frequently updated data from infrequently updated data. This segmentation enables the read-merge-rewrite process to operate only on the smaller frequently updated subset, thereby maintaining data coherency while significantly improving update efficiency.
Solution Approach 2:
The patent applies local quality by confining the resource-intensive read-merge-rewrite operations to only the frequently updated data subset stored on fast storage devices. Infrequently updated data on slow storage devices bypasses this process, localizing the heavy processing to where it is most needed and improving overall update efficiency.
Data Source
AI summary
The presently claimed invention relates to a system and method for organizing data replicated in a target data repository. The method of the presently claimed invention may receive data from a primary data store for replication in the target data repository. The method may then determine that the received data should be organized and stored according to one or more priority metrics. The method may then organize the received data according to the one or more priority metrics, and store the received data based on the priority metrics. Higher priority data may be stored faster data storage devices or be stored in smaller files where lower priority data may be stored in slower data storage devices or be stored in larger files.