In-Memory Graph Analytics: Navigation Data Structures for Traversal and Mutation Trade-offs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional graph processing systems are optimized for either transactional or analytic workloads but not both, leading to inefficiencies in updating and analyzing large graphs, particularly in real-time data analysis scenarios where fast analysis and quick updates are required.
Innovation Solution
The implementation of navigation data structures such as source vertex arrays, neighbor arrays, and bidirectional maps to accelerate traversal, mutation, and versioning of directed graphs, allowing for efficient traversal, update, and version management by decoupling change specification from execution and maintaining multiple graph versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If an in-memory graph analytics system is used to enable fast analysis, then analysis speed is improved, but memory consumption increases and graph mutation becomes inefficient
Solution Approach 1:
The patent segments the graph data structure into multiple versions stored in memory, allowing the system to maintain fast in-memory access while managing memory consumption through version pruning and selective retention. Each graph version is independently manageable, enabling efficient memory utilization.
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing graph snapshots at different versions before mutations occur. This allows rapid analysis at any point in time without requiring real-time computation during queries, improving analysis speed while managing memory through planned snapshot retention.
2Productivity
If graph mutations are performed frequently to support real-time updates, then update capability is improved, but system performance and stability deteriorate
Solution Approach 1:
The system creates graph snapshots before mutations occur, preserving the original state. This preliminary action allows mutations to proceed without disrupting ongoing analyses, maintaining system performance and stability while enabling frequent updates.
Solution Approach 2:
The patent creates copies of graph data at different versions through snapshotting. These copies allow mutations to be applied to one version without affecting other versions, enabling frequent updates while maintaining system reliability through isolation of mutation operations.
3Adaptability or versatility
If multiple graph versions are maintained to support versioning, then version management capability is improved, but memory consumption increases
Solution Approach 1:
The patent segments graph versions into discrete, independently managed snapshots. Each version is a separate entity that can be retained or discarded independently, allowing versatile version management while controlling memory consumption through selective retention policies.
4Manufacturing precision
If graph data is materialized from original graph and change set to create snapshots, then snapshot accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary materialization of graph snapshots before they are needed for analysis. By pre-computing and storing accurate graph states at different versions, the system ensures snapshot accuracy while avoiding time-consuming processing during actual analysis operations.
Data Source
AI summary
Techniques herein are for navigation data structures for graph traversal. In an embodiment, navigation data structures that a computer stores include: a source vertex array of vertices; a neighbor array of dense identifiers of target vertices terminating edges; a bidirectional map associating, for each vertex, a sparse identifier of the vertex with a dense identifier of the vertex; and a vertex array containing, when a dense identifier of a source vertex is used as an offset, a pair of offsets defining an offset range, for use with the neighbor array. The source vertex array, using the dense identifier of a particular vertex as an offset, contains an offset, into a neighbor array, of a target vertex terminating an edge originating at the particular vertex. The neighbor array contiguously stores dense identifiers of target vertices terminating edges originating from a same source vertex.