Vectorized Graph Processing Engine for Large Dataset Performance

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing graph processing technologies face challenges in handling large datasets due to increased complexity and cache misses caused by random memory access patterns, leading to performance degradation and high computational costs.

Innovation Solution

The method involves decomposing input graphs into topology and properties components, generating matrix representations, partitioning them into sub-matrices, and performing vectorized operations to preserve locality of access, using a vectorized property graph engine that supports bit-wise and vectorized processing for efficient graph traversal and aggregation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If graph processing is performed on large datasets, then the number of nodes and edges increases, but the computation complexity and memory access patterns become more complex, leading to performance degradation

Engineering Contradiction:

Improvenumber of nodes and edgesVSAvoidprocessing speed

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The graph is decomposed into a graph topology component and a graph properties component, with the topology represented as an adjacency matrix partitioned into sub-matrices. This segmentation allows for efficient processing by dividing the large graph into manageable blocks that can be processed in parallel, reducing the complexity of handling large datasets while maintaining processing speed

Inventive Principle:

Principle #1Segmentation

2Reliability

If conventional graph processing is used, then existing technologies can handle graphs, but random memory access patterns cause cache misses and performance degradation

Engineering Contradiction:

Improvegraph processing capabilityVSAvoidcache misses

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The adjacency matrix is partitioned into sub-matrices before processing, and the graph properties are organized in a structured format. This preliminary organization ensures that memory access patterns during graph traversal are sequential and cache-friendly, preventing random memory accesses and cache misses while maintaining reliable graph processing

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If graph operations are performed on large datasets, then comprehensive analysis is achieved, but computational costs increase significantly

Engineering Contradiction:

Improvegraph analysis capabilityVSAvoidcomputational cost

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent changes the representation parameters by using an adjacency matrix format with partitioned sub-matrices and organizing graph properties in a structured manner. This parameter change enables vectorized operations that process multiple data points simultaneously, reducing computational cost while maintaining comprehensive graph analysis capability

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS9916394B2Vectorized graph processing

Publication Date: 2018.03.13 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

An input graph is decomposed into a graph topology component and a graph properties component. A matrix representation is generated for each of the graph topology component and the graph properties component. Each of the graph topology matrix representation and graph properties matrix representation are partitioned into one or more sub-matrices. A forward pass comprising one or more vectorized operations is performed over the one or more sub-matrices. An output matrix is generated in response to the performing step.