Delta Compressed Sparse Rows Graph Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph processing systems face challenges in updating in-memory graphs while maintaining snapshot isolation and analytical performance, as current methods either require significant resources for snapshot creation, degrade performance, or compromise on snapshot isolation.
Innovation Solution
The technique employs delta compressed sparse rows (CSR) with forward edge offsets to store graph data, using delta logs to record changes, allowing for fast graph updates while preserving analytical performance by minimizing memory consumption and maintaining snapshot isolation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a new version of the graph is created by incorporating changes as a new set of data structures independent from the previous version, then snapshot isolation is ensured and analytical performance is maintained, but memory consumption and time required for snapshot creation increase significantly
Solution Approach 1:
The patent implements versioning by nesting delta logs within the graph data structure. Each graph version contains embedded delta logs that record changes relative to the base graph, allowing multiple versions to share the same underlying data structure while maintaining isolation. This nested approach enables snapshot creation without duplicating the entire graph, significantly reducing memory consumption compared to creating independent copies.
Solution Approach 2:
The patent segments the graph data structure into a base graph component and separate delta log components. The base graph contains the immutable foundational data, while delta logs contain version-specific changes. This segmentation allows the system to maintain snapshot isolation by referencing different combinations of base graph and delta logs for different versions, avoiding the need to copy the entire graph for each version.
2Quantity of substance
If a single graph index and storage units are used with in-place updates, then memory consumption is reduced, but snapshot isolation is lost and analytical performance degrades
Solution Approach 1:
The patent performs preliminary actions by pre-defining schema templates and data structure layouts before updates occur. The schema template specifies the structure of vertices, edges, and properties in advance. When updates occur, the system applies changes within this pre-defined framework using delta logs, ensuring that snapshot isolation is maintained without requiring complete data structure recreation. This preliminary structuring enables efficient in-place updates while preserving versioning capabilities.
3Reliability
If versioning information is added to data-segments to support concurrent manipulation, then snapshot isolation is achieved, but memory consumption increases and analytical performance degrades due to additional traversal requirements
Solution Approach 1:
The patent extracts versioning information from the core graph data structure and places it into separate delta log structures. Instead of embedding version tags within every vertex and edge data segment, the system extracts versioning to a higher level where delta logs reference changes at the graph or subgraph level. This extraction reduces the overhead within individual data segments and minimizes the impact on analytical traversals, as version checking occurs at the delta log level rather than during edge-by-edge processing.
4Adaptability or versatility
If non-contiguous memory allocations are used for neighbor lists to enable edge additions and removals, then update flexibility is improved, but analytical performance decreases and memory consumption increases
Solution Approach 1:
The patent implements dynamic neighbor lists that can adapt their structure based on update requirements. The neighbor list structure uses a hybrid approach: it maintains contiguous memory allocation for the base graph to optimize analytical performance, but incorporates dynamic linking capabilities through delta logs that allow efficient insertion and deletion of edges. This dynamic structure enables the neighbor lists to transition between contiguous and non-contiguous states as needed, balancing update flexibility with analytical performance.
Data Source
AI summary
Techniques are provided for updating in-memory property graphs in a fast manner, while minimizing memory consumption. A graph is represented as delta compressed sparse rows (CSR), in which its data structure stores forward edge offsets that map reverse edges to forward edges, enabling fast traversals of graph edges in forward and reverse directions. To support fast graph updates, delta logs are used to store changes to the graph. In an embodiment, a base version of the graph data structure is initially loaded or created, and subsequent versions of the graph are created from the reference to the initial graph and a delta log data structure that records the changes compared to the base version of the graph.