Ontology-Driven Data Synchronization in Distributed Meshes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing data at scale in distributed systems is complex due to the variety and volume of data sources and the need for efficient data synchronization across multiple nodes, which existing technologies struggle to address effectively.
Innovation Solution
An ontology-driven data synchronization mechanism that automatically determines which data elements to synchronize and how, using a 'need to know' mechanism to optimize data transfer between data nodes in a distributed data mesh, employing methods like pull, push, replicate, or remote access, and defining data access and synchronization strategies through a knowledge graph.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is synchronized across all data nodes in a distributed data mesh, then data availability and accessibility are improved, but data transfer bandwidth consumption and network load increase
Solution Approach 1:
The system implements differentiated data synchronization strategies where different data elements are synchronized to different degrees based on their specific characteristics. Not all data nodes receive all data, but rather each node receives only the data relevant to its function, achieving local optimization of data availability while minimizing unnecessary bandwidth consumption.
Solution Approach 2:
The patent segments the data mesh into autonomous data nodes with specific responsibilities. Each node independently determines what data it needs to synchronize based on its functional requirements, breaking the monolithic data synchronization approach into smaller, optimized segments that reduce overall network load.
2Reliability
If comprehensive data synchronization is implemented across all data nodes, then data consistency is improved, but system complexity and computational overhead increase
Solution Approach 1:
Each data node autonomously determines its own data synchronization requirements based on its functional characteristics and data consumption patterns. Nodes self-manage their data imports and synchronizations without requiring complex centralized orchestration, thereby maintaining data consistency while reducing system-level complexity.
Solution Approach 2:
The system pre-defines data synchronization strategies based on data element characteristics and node requirements before actual data transfer occurs. This preliminary determination of synchronization needs eliminates the need for complex real-time decision-making mechanisms during data transfer operations.
3Speed
If data is imported and cached at all data nodes, then data access speed is improved, but memory usage and storage requirements increase
Solution Approach 1:
Data nodes cache and store only the specific data elements that are relevant to their functional requirements, rather than maintaining copies of all data in the system. This localized data caching approach optimizes access speed for each node's specific workload while minimizing overall memory consumption across the distributed system.
Solution Approach 2:
The system implements partial data import where nodes receive only the subset of data they actually need for their operations, rather than importing complete datasets. This partial action approach achieves sufficient data access speed for local operations without the excessive memory burden of full data replication.
Data Source
Figure 1
Figure 2
Figure 3A~3D
AI summary
Implementations include obtaining a knowledge graph comprising a computer-readable data structure and including nodes and connections between the nodes, the nodes including: data nodes each representing a computational resource, analysis nodes each representing an analysis, and source nodes each representing a source of a data element; and determining, using the knowledge graph, an access strategy and a synchronization strategy for performing an analysis, by, automatically: identifying a first source node representing a source of a data element on which the analysis is to be performed, identifying a first data node representing a computational resource on which the analysis is to run, identifying a second data node representing a computational resource on which the data element is to reside, determining the access strategy between the first source node and the second data node, and determining the synchronization strategy between the first data node and the second data node.