Dynamic Cluster Compaction for Distributed Knowledge Graph Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed knowledge graphs experience inefficiencies in query processing due to latency and delay caused by information exchange across a distributed environment, making them less appealing for enterprise organizations that require fast query response times.
Innovation Solution
Implementing dynamic cluster compaction and expansion techniques to manage and process queries by identifying interconnected nodes, representing them as C-Nodes, and storing each cluster in a distributed database, which minimizes network traffic and latency through hash tables for efficient information retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If knowledge graphs are stored in a distributed manner, then data security and risk reduction are improved, but query processing latency and response time deteriorate
Solution Approach 1:
The knowledge graph is segmented into multiple clusters that are distributed across different computing devices. Each cluster represents a subset of interconnected nodes stored locally, allowing queries to be processed in parallel across multiple devices rather than requiring centralized access, thus reducing latency while maintaining distributed storage benefits
Solution Approach 2:
Node information is pre-computed and cached in hash tables before queries are executed. The system performs preliminary clustering and stores adjacency information in optimized data structures, so that when queries arrive, the system can quickly retrieve pre-processed information without performing complex computations in real-time, significantly reducing query response time
2Volume of stationary object
If information is exchanged across a distributed environment, then data storage capacity and security are improved, but network latency and processing efficiency worsen
Solution Approach 1:
Each computing device in the distributed system maintains local hash tables containing node information and adjacency data for its assigned cluster. This local storage approach allows the system to scale storage capacity across multiple devices while enabling fast local access to frequently queried information, eliminating the need for repeated network exchanges for the same data
Solution Approach 2:
The system creates copies of node information and adjacency data in hash tables at each relevant computing device. Instead of accessing the original distributed storage for every query, the system uses these local copies to serve queries rapidly, reducing network traffic and improving processing efficiency while maintaining data availability across the distributed environment
Data Source
AI summary
Aspects of the disclosure relate to dynamic cluster compaction and expansion for query processing. A computing platform may receive information associated with an abstracted information cluster and may store the information as a first table containing the information and corresponding adjacency information. Subsequently, the computing platform may receive a query requesting a subset of the information and additional information not associated with the abstracted information cluster. Using the first table, the computing platform may identify the subset of the information and corresponding adjacency information. Then, the computing platform may identify a location of the additional information, and may access, from the identified location, a second table containing the additional information and corresponding adjacency information. Based on the subset of the information and its corresponding adjacency information, and the additional information and its corresponding adjacency information, the computing platform may send a response to the query.