Bitwise Data Comparison Using Dynamic Temporal Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data reconciliation methods in enterprise-level software applications, such as healthcare software, face inefficiencies due to high network and processing resource usage when synchronizing large datasets between databases, particularly in processes like ETL, as they often require extensive communication and record-by-record comparisons.
Innovation Solution
The method employs bitwise fingerprints to compare and synchronize data by dividing records into blocks, generating fingerprints for each block, and using bitwise operations to identify and update records across databases, thereby reducing network and processing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional record-by-record comparison methods are used to synchronize databases, then data synchronization accuracy is maintained, but network resource usage and processing time increase significantly
Solution Approach 1:
The patent divides large datasets into smaller blocks and generates bitwise fingerprints for each block. Instead of comparing individual records, the system compares block-level fingerprints to identify which blocks contain changes. This segmentation approach maintains synchronization accuracy while dramatically reducing the time and network resources required, as only blocks with changes need to be transferred and reconciled at the record level.
Solution Approach 2:
The patent introduces bitwise fingerprints as an intermediary mechanism between the source and target databases. These fingerprints serve as a mediator that enables rapid comparison of data blocks without requiring direct record-by-record communication. By using this intermediary, the system achieves both high synchronization accuracy and efficient resource utilization.
2Reliability
If complete dataset synchronization is performed nightly, then data consistency between databases is ensured, but network bandwidth and processing resources are excessively consumed
Solution Approach 1:
The patent extracts and compares only the essential characteristic of data blocks through bitwise fingerprints, rather than transferring and processing complete datasets. This extraction approach ensures data consistency by identifying which blocks contain changes, while minimizing network bandwidth consumption and processing resource usage by transferring only the necessary block identifiers and fingerprints.
Solution Approach 2:
The patent performs partial synchronization by identifying and processing only the blocks that contain changes, rather than synchronizing the entire dataset. This partial action approach maintains data consistency for changed records while significantly reducing the energy and resources consumed compared to complete dataset synchronization.
3Loss of energy
If block-based bitwise fingerprinting is implemented, then network resource usage is reduced, but system complexity increases due to additional processing steps
Solution Approach 1:
The patent replaces complex record-by-record comparison mechanics with simpler bitwise operation mechanics. By substituting traditional comparison algorithms with efficient bitwise fingerprinting and XOR operations, the system reduces network resource usage while the added complexity is offset by the computational efficiency of bitwise operations compared to full record comparisons.
Data Source
AI summary
A method for synchronizing records stored at a first data storage location with records stored at a second, remote data storage location utilizes bitwise fingerprints and bitwise operations to determine records that exist in a first data set but not a second data set. A method for synchronizing records stored at a first data storage location with records stored at a second, remote data storage location utilizes bitwise fingerprints, and bitwise operations, and dynamic temporal segmentation to compare the timeliness of records in the first and second data sets.