DFS Cycle Detection on Pregel Model

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing cycle detection algorithms on the Pregel model, which employ a breadth-first search (BFS) approach, require excessive messages and suffer from high memory consumption due to the scale of large graphs, leading to inefficiencies in processing and increased chances of machine failure during computation.

Innovation Solution

The implementation of a depth-first search (DFS) approach for cycle detection in the Pregel model, which reduces memory consumption by iteratively sending and receiving IDs through edges, allowing for more efficient detection of cycles with fewer messages and improved locality of memory access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a breadth-first search (BFS) approach is used for cycle detection on the Pregel model, then the algorithm can detect cycles in large graphs, but it requires excessive messages and suffers from high memory consumption

Engineering Contradiction:
Improvecycle detection capabilityVSAvoidmemory consumption
Core Design Contradiction:
ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the cycle detection process into multiple iterations, where each iteration detects cycles of a specific length. Instead of attempting to detect all cycles simultaneously (which would require storing all paths in memory), the algorithm divides the problem into n-1 iterations, each handling cycles of length k. This segmentation allows the system to process large graphs with limited memory by focusing on one cycle length at a time.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent employs preliminary action by pre-assigning unique identifiers (IDs) to vertices and preparing the graph structure before cycle detection begins. Each vertex is pre-configured with its ID, and the system pre-establishes the iteration framework. This preliminary setup enables the iterative DFS process to proceed efficiently without requiring excessive memory during the actual detection phase, as the basic structure is already in place.

Inventive Principle:
Principle #10Preliminary action

2Reliability

If a breadth-first search (BFS) approach is used for cycle detection, then cycles can be detected, but the number of messages required is excessive

Engineering Contradiction:
Improvecycle detection capabilityVSAvoidmessage transmission time
Core Design Contradiction:
ReliabilityVSLoss of time

Solution Approach 1:

The patent inverts the traditional BFS approach by using depth-first search (DFS) instead. In BFS, messages are propagated level-by-level across the graph, requiring many messages to reach distant vertices. In contrast, DFS follows paths deeply before backtracking, allowing the algorithm to detect cycles with fewer message transmissions. This inversion of the search strategy fundamentally reduces the message complexity from O(V+E) in BFS to a more efficient pattern suited for cycle detection.

Inventive Principle:
Principle #13The other way round (Inversion)

Solution Approach 2:

The patent introduces dynamics by making the search strategy adaptive through iterations. Each iteration k focuses on detecting cycles of length k, dynamically adjusting the detection scope. The algorithm dynamically terminates when no more cycles are found or when k exceeds the graph diameter. This dynamic approach allows the system to adapt message transmission to the actual graph structure and cycle distribution, reducing unnecessary messages compared to a static BFS approach.

Inventive Principle:
Principle #15Dynamics

3Productivity

If graph processing is distributed over many machines, then the system can handle large graphs, but it exacerbates the locality issue and increases the probability of machine failure

Engineering Contradiction:
Improvegraph processing capacityVSAvoidmachine failure probability
Core Design Contradiction:
ProductivityVSReliability

Solution Approach 1:

The patent segments the graph processing workload across multiple machines in a distributed environment, with each machine handling a partition of the graph. The iterative DFS algorithm is designed to work efficiently in this distributed setting by maintaining local state at each vertex and communicating only necessary information (cycle detections and path extensions) between machines. This segmentation allows the system to scale to large graphs while minimizing the impact of individual machine failures, as the iterative nature allows for checkpointing and recovery.

Inventive Principle:
Principle #1Segmentation

Data Source

PatentUS12032632B2DFS-based cycle detection on Pregel model
Publication Date: 2024.07.09 INTERNATIONAL BUSINESS MACHINE CORPORATION
  • US12032632B2 patent drawing
  • US12032632B2 patent drawing
  • US12032632B2 patent drawing

AI summary

A method, system and computer program product are presented for reducing memory consumption in depth first search (DFS)-based cycle detection processes to detect an n-length cycle on a Pregel model. The method includes generating a graph including a plurality of vertices and edges connecting the plurality of vertices, commencing a cycle from a first vertex of the plurality of vertices, setting an ID of the first vertex to a path and the first vertex as a target vertex, and iterating the following n−1 times: send the path to out-edges starting from the first vertex and transferring the path, via out-edges, n−1 times by subsequent received vertices, find one vertex before returning to the target vertex, add an ID of a found vertex to the path, and set the found vertex as the target vertex such that n=n−1, where is a number of iterations.