In-Memory Database Search Using Graph Community Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory database systems face challenges in efficiently searching large volumes of data due to high processing and memory resource consumption, which can be exacerbated by the need to compare all records in response to queries, especially when dealing with millions or billions of records.
Innovation Solution
The implementation of a graph community structure is used to optimize database searching by creating a graph from transaction data, identifying communities of similar value-ids, and executing queries on a per-community basis, thereby reducing memory usage and processing intensity by skipping records outside the relevant community.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all records are searched in response to queries, then query completeness is improved, but processing time and memory resource consumption increase
Solution Approach 1:
The patent segments the database records into distinct communities based on graph community detection. Each community represents a cluster of related records with similar value-ids. During query processing, the system identifies which communities are relevant to the query and searches only those communities, rather than scanning all records. This segmentation enables the system to maintain query completeness for relevant data while dramatically reducing processing time by excluding irrelevant records from the search.
2Reliability
If all records are searched in response to queries, then query completeness is improved, but memory resource consumption increases
Solution Approach 1:
The patent segments the database records into distinct communities based on graph community detection. Each community represents a cluster of related records with similar value-ids. During query processing, the system identifies which communities are relevant to the query and searches only those communities, rather than scanning all records. This segmentation enables the system to maintain query completeness for relevant data while dramatically reducing processing time by excluding irrelevant records from the search.
3Productivity
If graph community structure is created and queries are executed on per-community basis, then search efficiency is improved, but device complexity increases
Solution Approach 1:
The patent applies graph community detection algorithms during the data loading or preprocessing phase to pre-identify and organize records into communities. The community structure is stored as metadata alongside the record data. When queries are executed, the system uses this pre-computed community information to quickly determine which communities are relevant, avoiding the need to perform complex community detection during query processing. This preliminary action shifts the computational complexity from query time to data loading time, significantly improving search efficiency while maintaining manageable system complexity during operations.
Data Source
AI summary
Database searching is optimized utilizing a graph community structure. A graph is created from transaction data based upon adjacent value occurrences. This may be done by scanning transaction data from top to bottom, and creating an edge between a current index value and a previous index value. Next, algorithms identify communities in the graph to create a graph community structure. These communities comprise blocks of patterns of similar value-ids distributed in the transaction data. Scanning and transition indices may be created with an eye toward reducing memory usage and enhancing performance. Query searching is then executed in an efficient manner on a per-community basis. For example, exact queries, range queries, and/or “AND” queries may be executed more efficiently upon communities of records, skipping those not belonging to the same community. Embodiments are suited to search an in-memory database having large volumes of column-oriented data stored in RAM.