Graph Query Processing Across Heterogeneous Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph query processing systems face challenges in efficiently handling large graphs due to memory limitations, requiring manual query composition, and posing security risks through ad hoc queries, especially when dealing with heterogeneous data storage systems.
Innovation Solution
The system optimally distributes graph query processing across heterogeneous tiers by decomposing queries into filter expressions and assigning them to appropriate filtration tiers, utilizing a pipeline concurrency approach and applying De Morgan's Laws for optimal execution, allowing for efficient execution of filter expressions across database management systems, stream evaluators, and in-memory evaluators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire graph is loaded into main memory for analysis, then analysis speed is improved, but memory capacity constraints prevent processing of large graphs
Solution Approach 1:
The patent divides the graph processing into multiple segments distributed across different memory tiers. Hot data that fits in main memory is processed quickly, while cold data residing in secondary storage is accessed only when needed, enabling processing of graphs larger than available memory while maintaining high speed for frequently accessed portions.
Solution Approach 2:
The patent introduces a hierarchical memory dimension with multiple tiers (main memory, secondary storage, distributed file systems). This multi-dimensional memory architecture allows the system to handle graphs of any size by distributing data across different capacity and speed tiers, resolving the contradiction between speed and capacity.
2Adaptability or versatility
If manual query composition is required for each graph storage system, then query flexibility is improved, but system complexity and security risks increase
Solution Approach 1:
The patent implements a universal graph query interface that works across multiple graph storage systems and memory tiers. The unified API abstracts away the complexities of different underlying systems, allowing users to write queries once and execute them anywhere in the heterogeneous architecture without manually composing different queries for each system.
Solution Approach 2:
The patent introduces an intermediary layer (the unified graph API and query processing framework) that mediates between user queries and the diverse underlying storage systems. This intermediary handles the complexity of interfacing with different graph databases and memory tiers, shielding users from system complexity while maintaining query flexibility.
3Adaptability or versatility
If ad hoc queries are accepted from users, then query capability is improved, but security risks through injection attacks increase
Solution Approach 1:
The patent applies preliminary anti-action by implementing query validation and sanitization mechanisms that proactively prevent injection attacks before they can execute. The unified query interface enforces strict syntax rules and parameter validation, blocking malicious queries while allowing legitimate ad hoc queries to proceed, thus maintaining query capability while eliminating security risks.
Data Source
AI summary
Techniques herein optimally distribute graph query processing across heterogeneous tiers. In an embodiment, a computer receives a graph query to extract a query result (QR) from a graph in a database operated by a database management system (DBMS). The graph has vertices interconnected by edges. Each vertex has vertex properties, and each edge has edge properties. The computer decomposes the graph query into filter expressions (FE's). Each FE is processed as follows. A filtration tier to execute the FE is selected from: the DBMS which sends at least the QR to a stream, a stream evaluator that processes the stream as it arrives without waiting for the entire QR to arrive and that stores at least the QR into memory, and an in-memory evaluator that identifies the QR in memory. A translation of the FE executes on the filtration tier to obtain vertices and/or edges that satisfy the FE.