Distributed Graph Query Processing with Dynamic Message Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph databases and processing systems face challenges in scalability and efficiency, particularly in handling complex computational tasks and maintaining historical data states.
Innovation Solution
A node-centric architecture with asynchronous message-passing and half-edge representation, combined with scalable storage and query mechanisms, allows for efficient processing and querying of graph data across multiple computing devices, supporting historical state retrieval and flexible consistency modes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If graph databases use traditional centralized architecture, then data consistency is maintained, but scalability is limited
Solution Approach 1:
The patent segments the graph database into multiple independent partitions distributed across different computing devices. Each partition maintains its own data and can be processed independently, enabling horizontal scaling while preserving data consistency within each partition through local transaction management.
Solution Approach 2:
The patent introduces a new dimensional approach by implementing multi-version concurrency control across distributed partitions. This allows the system to scale horizontally across multiple computing devices while maintaining consistency through versioned data access, effectively adding a scalability dimension without sacrificing reliability.
2Productivity
If graph processing systems allocate message processors to all nodes, then query processing speed is improved, but resource consumption increases
Solution Approach 1:
The patent implements dynamic message processor allocation where processors are assigned to nodes based on current workload and query requirements rather than being permanently allocated. This dynamic allocation allows the system to optimize query processing speed by activating processors only when needed, thereby reducing overall resource consumption while maintaining high productivity during peak operations.
Solution Approach 2:
The system changes the operational parameters of message processors by adjusting their activation state based on system load and query complexity. Processors can be scaled up or down dynamically, changing their resource consumption parameters while maintaining the capability for high-speed processing when required.
3Measurement precision
If graph databases store complete historical states, then historical query accuracy is improved, but storage requirements increase
Solution Approach 1:
The patent extracts only the essential state information required for historical queries rather than storing complete historical states. By identifying and retaining only the critical data elements needed for accurate historical reconstruction, the system maintains query accuracy while significantly reducing storage requirements through selective data retention.
Solution Approach 2:
The system applies partial action by storing a representative subset of historical states rather than complete states. This partial storage approach uses sampling and selective retention of key state changes to maintain sufficient historical query accuracy without the exponential storage growth that would result from storing every complete historical state.
Data Source
AI summary
A graph comprising nodes and edges is stored by a distributed system as a collection of nodes and half-edges stored with their respective nodes. A message processor is associated with a node as needed to process messages passed between nodes, such that a given node has zero or one message processor assigned to it at a given time. Queries of the graph are resolved by processing a first portion of the query at a first node, and forwarding the results with the remaining portions of the query to a node linked by an edge to the present node.