Distributed Graph Processing with Active Vertex Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current single-machine data processing methods are inadequate for handling large-scale data due to resource limitations, failing to meet the demands of big data processing needs.
Innovation Solution
A distributed data processing system is employed, where graph data are pre-generated based on event information, and active vertex sets are determined to facilitate update processing across distributed nodes, utilizing various data processing modes to enhance efficiency and memory utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If single-machine data processing is used, then data processing is simple and convenient with high performance, but it cannot satisfy big data processing needs due to limited resources
Solution Approach 1:
The patent divides the data processing system into multiple distributed nodes, each handling a portion of the graph data. The graph data is partitioned across different machines, with each node maintaining local data in its external memory. This segmentation allows the system to process large-scale data that exceeds the capacity of a single machine while maintaining processing efficiency through localized operations.
2Quantity of substance
If distributed data processing is used, then big data processing needs are satisfied with expanded memory capacity, but system complexity increases due to network communication and data interaction
Solution Approach 1:
The patent pre-generates graph data based on event information before the actual processing begins. This preliminary action organizes the data into a structured graph format with vertices and edges, storing it in external memory of distributed nodes. By preparing the data structure in advance, the system reduces the complexity of real-time processing operations and enables efficient query execution without complex runtime data organization.
Solution Approach 2:
The patent introduces an active vertex set as an intermediary structure to manage data flow between distributed nodes. This active vertex set tracks which vertices need to be processed or updated, serving as a mediator that coordinates communication between nodes. By using this intermediary, the system simplifies the complexity of direct node-to-node communication and data synchronization in the distributed environment.
3Reliability
If graph data are pre-generated based on event information, then data association is improved, but data processing time increases due to pre-processing requirements
Solution Approach 1:
The patent implements periodic batch processing of the active vertex set rather than continuous processing. The system processes vertices in discrete batches, updating the active vertex set periodically based on the previous batch's results. This periodic action reduces the time complexity compared to continuous processing while maintaining accurate data associations through the structured graph data that was pre-generated from event information.
Data Source
AI summary
Embodiments of this specification provide distributed data processing methods, apparatuses, and devices. One method includes: determining an active vertex set that currently participates in data processing in target graph data, in response to determining that an external memory of a first distributed node stores an active vertex in the active vertex set, determining, from a plurality of predetermined data processing modes, a target data processing mode that matches the active vertex set, determining, based on the target data processing mode, a to-be-updated vertex according to the association relationship with the active vertex, and sending, based on first data of the active vertex in the external memory, a first update message to a target distributed node in which the to-be-updated vertex is located.