Distributed Database Sync Using Block Hash Differencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronizing heterogeneous databases over the Internet is inefficient and poses security risks, particularly when dealing with large data volumes and limited access to transaction logs, leading to potential breaches of protected health information.
Innovation Solution
A system that computes a differential result set between source and sink databases, segments data into blocks based on configurable sizes, generates hashes, and securely transmits these hashes over the Internet to update the sink database, optimizing data transfer and reducing CPU load on the source server.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is transmitted over the Internet between heterogeneous databases, then connectivity and accessibility are improved, but security risks and data breach potential increase
Solution Approach 1:
The patent segments large data sets into smaller data blocks, each processed and transmitted independently. This segmentation allows for selective transmission of only necessary data portions, reducing the attack surface and potential impact of security breaches while maintaining Internet connectivity for database synchronization.
Solution Approach 2:
The patent introduces a data synchronization system with coordinated servers that act as intermediaries between source and sink databases. These intermediaries manage the synchronization process, control data flow, and implement security measures, reducing direct exposure and security risks associated with heterogeneous database connections over the Internet.
2Quantity of substance
If large data volumes are synchronized between databases, then data completeness is improved, but network usage and transmission time increase
Solution Approach 1:
The patent extracts and transmits only the differential portions of data that need synchronization, rather than transmitting entire large data sets. By identifying and extracting only the changed or missing data blocks, the system maintains data completeness while significantly reducing transmission time and network usage.
Solution Approach 2:
The patent implements partial action by synchronizing only the necessary portions of data rather than complete data sets. The system determines which data blocks require synchronization and transmits only those, avoiding the excessive transmission of already-synchronized data and optimizing network resource utilization.
3Productivity
If data is segmented into blocks for transmission, then transmission efficiency is improved, but processing complexity increases
Solution Approach 1:
The patent divides large data sets into manageable data blocks with defined boundaries and identifiers. This segmentation enables parallel processing, selective transmission, and efficient reassembly, improving transmission efficiency while the standardized block structure actually reduces processing complexity compared to handling large monolithic data sets.
Solution Approach 2:
The patent performs preliminary actions by pre-segmenting data into blocks, pre-calculating hashes for integrity verification, and pre-establishing block boundaries before transmission. This preliminary preparation simplifies the actual transmission and processing steps, reducing overall processing complexity despite the additional initial segmentation step.
4Reliability
If schema information is limited, then source server access control is improved, but synchronization accuracy may worsen
Solution Approach 1:
The patent implements self-service mechanisms where the synchronization system autonomously discovers and adapts to the source database schema through limited access queries. The system can determine necessary schema information through controlled read operations, maintaining strict access control while achieving sufficient schema understanding for accurate synchronization through self-directed information gathering.
Data Source
AI summary
Methods and systems for synchronizing distributed databases are provided. In one aspect, a method includes transmitting a request to determine a schema of a first database and updating a metadata store with the schema. The method includes generating at least one database query. The method includes, responsive to the database query, selectively determining a data block size for a second database; segmenting a data source stored in the second database into sink data blocks; processing the data source into sink hashes; and transmitting, to an agent, at least one selected sink hash, wherein the agent transmits instructions to segment a table associated with the first database into source data blocks; process the table into source hashes; and generate, in response to a determination that the at least one selected sink hash differs from a corresponding source hash, query results. The method includes updating the data source with the query results.