Distributed Graph Embedding for Virtual Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing graph embeddings for network traffic data in a centralized manner is bandwidth-intensive and can lead to storage and compute resource bottlenecks, as all graph connectivity information needs to be transmitted and processed on a single machine.
Innovation Solution
A distributed approach where each host system computes node embeddings for localized VMs independently and transmits only necessary data for non-localized VMs to a management server, using a biased random walk strategy and distributed iterative SGD algorithm to reduce bandwidth and balance compute/storage loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If graph embedding computation is performed in a centralized manner, then the computation can be simplified and coordinated, but bandwidth consumption increases and resource bottlenecks occur
Solution Approach 1:
The patent segments the graph embedding computation by partitioning the graph into multiple subgraphs, each assigned to different computing nodes. This allows distributed computation of node embeddings across multiple machines, reducing the bandwidth required to transmit entire graph data to a single centralized processor while maintaining computation coordination through the structured partitioning approach.
2Measurement precision
If all graph connectivity information is transmitted to a centralized machine, then complete graph embedding computation can be performed, but storage and compute resource bottlenecks occur
Solution Approach 1:
The patent divides the large graph into multiple subgraphs that can be stored and processed distributedly across multiple computing nodes. Each node stores only the subgraph data it needs to process, significantly reducing the storage requirement on any single machine while still enabling comprehensive graph embedding computation through aggregation of results from all nodes.
3Ease of operation
If a centralized machine processes all graph data, then computation results can be standardized, but compute resource bottlenecks occur
Solution Approach 1:
The patent segments the computation task across multiple processing nodes, each handling specific subgraphs. This distributed approach increases overall computation throughput by parallelizing the embedding computation process, while standardized algorithms are applied at each node to ensure consistency and ease of operation across the distributed system.
Solution Approach 2:
The patent combines the computation results from multiple distributed nodes to produce the final graph embedding. By merging the partial results through a coordinated aggregation process, the system achieves both high throughput (through parallel computation) and result consistency (through standardized merging operations).
Data Source
AI summary
In one set of embodiments, a host system can perform a random walk along a graph representing network traffic in a virtual network, where the virtual network comprises a plurality of virtual machines (VMs) running on a plurality of host systems including the host system, and where the random walk starts from a node of the graph corresponding to a VM running on the host system. The host system can further construct, based on the random walk, a local neighborhood of VMs associated with the VM and determine, based on the local neighborhood, whether the VM is a localized VM. Upon determining that the VM is not a localized VM, the host system can transmit a random walk data entry identifying the VM and the local neighborhood to a server communicatively coupled with the plurality of host systems.