Dynamic Filter Tracking for Data Subset Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional synchronization systems face challenges in efficiently synchronizing subsets of data among multiple devices with varying set membership, leading to issues like non-convergence, metadata growth, and storage limitations, especially in loosely coupled multi-master environments.
Innovation Solution
The system employs replica tracking filters, filter forgotten knowledge, and ghost representations to efficiently track and exchange synchronization metadata, allowing devices to maintain subsets of data without pre-defined filters and supporting dynamic filter set membership.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional synchronization systems synchronize all data items between devices, then data completeness is improved, but metadata growth becomes proportional to the number of items in the subset leading to unscalable systems
Solution Approach 1:
The patent extracts only the necessary metadata for synchronized items rather than tracking all items. By using set membership information to identify which items are actually synchronized between devices, the system sends and stores metadata only for those specific items, making metadata growth proportional to the number of nodes rather than the total number of items in the subset.
Solution Approach 2:
The patent segments the data synchronization problem into two parts: (1) identifying which items belong to the synchronized subset using set membership tracking, and (2) exchanging metadata only for those identified items. This segmentation allows the system to handle large data sets efficiently by focusing computational and storage resources only on relevant items.
2Quantity of substance
If devices store only subsets of data to accommodate limited storage, then storage efficiency is improved, but synchronization convergence becomes difficult due to varying set membership
Solution Approach 1:
The patent implements dynamic set membership tracking that adapts as items move in and out of synchronized subsets. The system continuously updates which items are members of the synchronized set based on current device states and synchronization needs, allowing flexible storage optimization while maintaining convergence through proper tracking of membership changes over time.
Solution Approach 2:
The patent uses feedback mechanisms where devices exchange information about their current set membership status during synchronization. This feedback allows the system to detect and resolve conflicts arising from varying set membership, ensuring that all devices converge to a consistent state even when storing only subsets of data.
3Loss of information
If synchronization metadata tracks all items including those never in the subset, then item tracking completeness is improved, but storage requirements and processing capabilities become excessive for limited devices
Solution Approach 1:
The patent applies local quality by maintaining different levels of tracking information for different items based on their relevance. Synchronized items receive full metadata tracking with version information and change details, while non-synchronized items receive minimal or no tracking. This selective approach ensures complete tracking for relevant items while minimizing storage requirements overall.
4Adaptability or versatility
If full set of synchronization metadata is synchronized to all devices, then data recognition capability is improved, but storage efficiency deteriorates as each device stores redundant data it does not recognize
Solution Approach 1:
The patent changes the parameters of metadata exchange by including set membership information and filtering criteria in the synchronization process. Instead of exchanging complete metadata sets, devices exchange filtered metadata parameters that are relevant to their specific synchronized subsets, improving storage efficiency while maintaining the ability to recognize and process synchronized data.
Data Source
AI summary
An efficient way and general mechanism is provided to represent and exchange knowledge and/or partial knowledge across nodes when synchronizing between any two nodes including custom or move filters where set membership can change over time at a device as data items come in and out of filter scope. A first node sends a second node its knowledge and/or partial knowledge, including objects and versions of those objects. The second node compares its knowledge and/or partial knowledge with the knowledge and/or partial knowledge of the first node, and then sends the first node any latest versions of objects of which the first node is unaware. In addition, the second node sends its knowledge and/or partial knowledge to the first node. The first node then performs a similar object-by-object version comparison to determine any conflicts due to independent evolution of objects and any changes that can be sent to the second node in order to bring the objects of the second node up to date with the knowledge and/or partial knowledge of the first node. Replicas can track filters to efficiently support synchronizing with filtered replicas. Filter forgotten knowledge is also used to represent how recently a replica can guarantee to know which items have been in the filter. Ghosts can be used to represent items that have been in the filter recently but have currently moved out. Filter forgotten knowledge concept allows both new filters to be tracked when they are introduced into the community, as well as periodically cleaning up of tombstones and ghosts.