Map-Reduce Graph Clustering via Edge-Oriented Data Inversion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing graph processing technologies, such as Map-Reduce, face challenges in efficiently handling edge-oriented algorithms due to high network traffic and data sharing issues, making it difficult to process large-scale graphs in distributed computing environments, particularly in social network and cloud computing applications.

Innovation Solution

The implementation of the Markov Clustering Algorithm (MCL) using a Map-Reduce model, which organizes graph data by columns, calculates probability matrices, and computes powers of matrices in a distributed manner to find clusters, optimizing memory usage and reducing data sharing overhead through sub-matrix decomposition and strategic data processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If edge-oriented algorithms are executed on vertex-oriented infrastructure in distributed computing environments, then graph processing can be performed, but network traffic becomes excessive and processing efficiency deteriorates

Engineering Contradiction:

Improvegraph processing efficiencyVSAvoidnetwork traffic volume

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent inverts the conventional vertex-oriented storage approach by organizing graph data in an edge-oriented manner, where edges are stored with their incident vertices. This inversion allows edge-oriented algorithms to execute efficiently without generating excessive network traffic, as the data structure naturally supports edge-centric operations.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent changes the data organization parameter from vertex-centric to edge-centric storage structure. By modifying how graph data is structured and stored, the system enables efficient execution of edge-oriented algorithms while reducing network communication overhead in distributed environments.

Inventive Principle:

Principle #35Parameter changes

2Ease of manufacture

If graph data is stored in vertex manner, then data storage is simplified, but network traffic increases when edge state modifications require vertex notifications

Engineering Contradiction:

Improvedata storage simplicityVSAvoidnetwork traffic

Core Design Contradiction:

Ease of manufactureVSObject-generated harmful factors

Solution Approach 1:

Inventive Principle:

Principle #13The other way round (Inversion)

3Productivity

If Map-Reduce model is used for large-scale graph processing, then processing capability is enhanced, but implementation complexity increases for edge-oriented algorithms

Engineering Contradiction:

Improvelarge-scale graph processing capabilityVSAvoidalgorithm implementation complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the graph processing tasks into map and reduce phases, where the map phase processes edges locally and the reduce phase aggregates results. This segmentation enables edge-oriented algorithms to be implemented in the Map-Reduce model by dividing the complex processing into manageable, independent tasks.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary data structure that bridges edge-oriented algorithms and the Map-Reduce framework. This intermediary structure enables efficient translation and execution of edge-oriented operations within the distributed Map-Reduce environment, reducing implementation complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS8943011B2Methods and systems for using map-reduce for large-scale analysis of graph-based data

Publication Date: 2015.01.27 SALESFORCE INC

AI summary

Embodiments are described for a method for processing graph data by executing a Markov Clustering algorithm (MCL) to find clusters of vertices of the graph data, organizing the graph data by column by calculating a probability percentage for each column of a similarity matrix of the graph data to produce column data, generating a probability matrix of states of the column data, performing an expansion of the probability matrix by computing a power of the matrix using a Map-Reduce model executed in a processor-based computing device; and organizing the probability matrix into a set of sub-matrices to find the least amount of data needed for the Map-Reduce model given that two lines of data in the matrix are required to compute a single value for the power of the matrix. One of at least two strategies may be used to computing the power of the matrix (matrix square, M2) based on simplicity of execution or improved memory usage.