Heterogeneous Data Sync via Block Pipelining and Parallel Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for large-scale heterogeneous data synchronization across the Internet face limitations in data throughput and efficiency, being non-lightweight and product-specific, which hinders effective synchronization of diverse data sources.
Innovation Solution
A generic multi-source heterogeneous large-scale data synchronization system is developed, utilizing a tree structure topology with sync nodes, block & pipelining units, unilateral and bilateral sync units, and correctness guarantee mechanisms to manage data synchronization through SyncML protocols, ensuring efficient and reliable data replication across heterogeneous systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current generic synchronization methods are used for massive heterogeneous data, then data synchronization can be achieved, but data throughput is limited and synchronization efficiency is low
Solution Approach 1:
The patent segments large-scale heterogeneous data into fine-grained sync blocks at the client end, allowing parallel processing and transmission of multiple blocks simultaneously. This segmentation enables the system to handle larger data volumes with improved throughput while maintaining synchronization efficiency through modular block-level operations.
Solution Approach 2:
The patent implements pipelining mechanisms where sync blocks are transmitted continuously through multiple processing stages (encoding, transmission, decoding, application) without idle gaps. The server end processes blocks as they arrive, maintaining continuous data flow and eliminating waiting time between synchronization operations, thereby improving both throughput and efficiency.
2Adaptability or versatility
If current generic synchronization methods are used, then data replication can be performed, but the system is non-lightweight and involves implementation details of concrete database products
Solution Approach 1:
The patent creates a universal synchronization framework that can handle multiple heterogeneous data sources (relational databases, NoSQL databases, file systems, cloud storage) through a common block-based interface. The system uses standardized protocols and abstracted data representations that are independent of specific database products, enabling broad adaptability without increasing system complexity or creating product-specific dependencies.
Solution Approach 2:
The patent introduces an intermediary layer consisting of sync blocks and encoding/decoding mechanisms that mediate between diverse data sources and the synchronization target. This intermediary abstraction hides the complexity of heterogeneous data formats and protocols, allowing the system to maintain lightweight architecture while achieving universal compatibility across different data sources.
Data Source
AI summary
The present disclosure provides a generic multi-source heterogeneous large-scale data system, including a sync node config unit, an install & deployment unit, a block & pipelining unit, a unilateral sync unit, a bilateral sync unit and a correctness guarantee unit. The system operates on a middle layer which is above a node database layer and beneath an application logic layer. In a data synchronization process, a client end transmits captured local change information to a server end in accordance with a synchronization task plan; the server end receives and transfers the change information to an asynchronous parallel message processing mechanism to be stored in a corresponding message queue; the server end polls a local message queue for reading to-be-processed change information, and then performs subsequent data change in accordance with rules for heterogeneous data mapping, so as to maintain consistence of synchronized data objects between a source end and a target end. The system of the present disclosure operates independently in a manner parallel to local applications of a synchronous node, and provides a guarantee mechanism of relaxed transaction for Internet distributed multi-source heterogeneous data synchronization through collaboration of loose coupling.