Deltalog-Based Connector for External List Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in reliably transferring data to external systems due to varying operations supported by these systems, leading to inconsistencies and inefficiencies in data synchronization.
Innovation Solution
A connector pattern is implemented that uses a deltalog to track changes in the dataset, computes a changeset, and issues API calls to external systems to update records, ensuring consistent data transfer across different external systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is transferred to multiple external systems with varying operations supported, then data accessibility is improved, but system complexity and reliability deteriorate
Solution Approach 1:
The patent introduces a mediator component that sits between the centralized data repository and multiple external systems. This mediator translates and adapts data transfer operations, allowing the system to access multiple external systems with varying capabilities while maintaining consistent and reliable data synchronization through a unified interface layer.
Solution Approach 2:
The system implements a universal data transfer mechanism that can operate across multiple external systems with different supported operations. By creating a multi-functional transfer layer that handles various operation types (full refresh, incremental updates, deltalog-based sync), the system achieves broad adaptability while maintaining reliable synchronization through standardized protocols.
2Measurement precision
If frequent data synchronization is performed to external systems, then data accuracy is improved, but network congestion and performance deterioration
Solution Approach 1:
Instead of performing complete data transfers for every synchronization cycle, the system applies partial action by transferring only the necessary portions of data.通过使用deltalog记录的数据变更日志，系统可以精确识别并仅传输发生变化的数据记录，而不是整个数据集，从而在保证数据准确性的同时显著减少网络流量和性能开销。
Solution Approach 2:
The system implements periodic synchronization based on data changes rather than fixed time intervals. The deltalog mechanism enables the system to trigger synchronization operations periodically when specific change thresholds are met, maintaining data accuracy while avoiding excessive network traffic during periods of no or minimal data changes.
3Stability of the object's composition
If complete dataset transfers are performed to external systems, then data consistency is improved, but transfer time and productivity deterioration
Solution Approach 1:
The system extracts only the changed portions of the dataset using the deltalog mechanism. Instead of transferring complete datasets, it identifies and extracts specifically the records that have been added, modified, or deleted since the last synchronization, maintaining data consistency while dramatically improving transfer efficiency by reducing the volume of data that needs to be transmitted.
Solution Approach 2:
The data synchronization process is segmented into different operation types (full refresh, incremental updates, deltalog-based synchronization). This segmentation allows the system to choose the appropriate transfer strategy based on the specific scenario, using complete transfers only when necessary and incremental transfers for routine synchronizations, thereby maintaining consistency while optimizing transfer efficiency.
Data Source
AI summary
In some implementations, the techniques described herein relate to a system including: a database storing a dataset, the database including a persistent storage device; and a connector, the connector including a computing device configured to access data stored in the database and transfer data and commands to an external system, the connector configured to: update a deltalog responsive to a change in the dataset, the deltalog storing one or more of additions or deletions to data in the dataset, determine that a list refresh is needed for the external system, compute a changeset based on the deltalog and the dataset, the changeset representing one or more of additions or deletions to the dataset, and issue at least one application programming interface call to the external system based on the changeset to update a list of records stored by the external system.