Hybrid Dataset Storage for Faster Path-Based Database Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems, both relational and graph-based, face inefficiencies in handling path-oriented searches, particularly in large or complex datasets, as traditional indexes and graph databases often require excessive time and resources to produce accurate result sets.
Innovation Solution
A hybrid approach that concurrently stores data in both a non-graph and graph database, where data objects are transformed into nodes and edges in the graph database for metadata, enabling simultaneous searches in both systems, with query translation to optimize path-based queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If path-based queries are executed against traditional relational or columnar databases, then the query can be submitted to the database, but the server computer system must check each row to determine if the row satisfies the query, requiring unacceptable amounts of time to produce result sets for path-oriented searches in large or complex datasets
Solution Approach 1:
The patent segments the data storage and query processing into two distinct systems: a non-graph database for storing datasets and a graph database for storing metadata and relationships. This segmentation allows each system to optimize for its specific function, with the graph database handling path-oriented searches efficiently while the non-graph database maintains the primary data repository.
Solution Approach 2:
The graph database acts as an intermediary system that stores metadata about relationships between data items. When path-based queries are executed, the graph database serves as a mediator that can rapidly traverse relationships and return results without requiring the primary database to perform expensive row-by-row checks.
2Productivity
If graph database systems are used to organize metadata and conduct searches using logical graphs, then graph traversal algorithms can be superior to traditional relational database indexes, but the data object schema implemented by a graph database is considerably different than the schema or ontology of a non-graph database system
Solution Approach 1:
The patent divides the system into two separate database systems with different schemas: a non-graph database that maintains the original data schema and ontology, and a graph database that uses a graph-based schema for metadata. This segmentation allows each system to use its native schema without compromise.
Solution Approach 2:
The system creates a copy of the metadata and relationship information in the graph database, separate from the primary data storage. This copying approach allows the graph database to optimize for graph traversal while the original database maintains its schema integrity.
3Loss of time
If concurrent storage in both non-graph and graph databases is implemented, then path-oriented searches can leverage graph traversal capabilities for faster results, but the system must maintain data in two separate repository systems
Solution Approach 1:
The system performs preliminary action by automatically transforming and storing metadata in the graph database concurrently with data storage in the non-graph database. This preliminary organization of relationship data enables fast path-based queries without requiring complex real-time processing during query execution.
Solution Approach 2:
The system implements self-service through automatic transformation and concurrent storage mechanisms that handle the dual repository maintenance without requiring manual intervention. The transformation process automatically adapts data between schemas and maintains consistency across both database systems.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for improving database searches are described herein. In an embodiment, a computer-implemented data processing method comprises receiving a request to store a digital data object in a non-graph data repository, the data object comprising one or more first properties, the data object conforming to a first schema of the non-graph data repository; in response to the initiating, concurrently: digitally storing the data object in the non-graph data repository; transforming the data object into one or more nodes and edges of a graph; storing the one or more nodes and edges of the graph in a graph data repository that is separate from the non-graph data repository; electronically receiving a search query that specifies a path-based search; in response to receiving the search query: transforming the search query into a graph search query; submitting the graph search query to the graph data repository; receiving a result set of nodes from the graph data repository; transforming the result set of nodes into a set of corresponding data objects in the non-graph data repository; providing the set of corresponding data objects as output in a computer display.