RDF Database System Scalable Distributed Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current RDF database systems face scalability issues and performance bottlenecks when handling large-scale, heterogeneous, and semi-structured data, particularly in distributed environments, due to inefficient indexing and query processing strategies that lead to excessive join operations and inter-host communication costs.
Innovation Solution
The proposed RDF database system employs a novel architecture that stores RDF data in larger chunks, indexed with multiple domain-specific indexes, and uses the Mnemosyne Query Language (MQL) to retrieve and process data in memory, applying SPARQL queries to in-memory RDF graphs, with a 'gather-sift-clear' workflow orchestrated by MQL scripts to efficiently manage data chunks and reduce the need for extensive index consultations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If RDF data is stored as individually indexed triples with lexicographic ordering in B+ trees, then query lookup for single triples is efficient, but complex pattern queries require excessive join operations that degrade performance at scale
Solution Approach 1:
The patent segments the RDF graph into multiple partitions distributed across different hosts. Each partition maintains local indexes for efficient local queries, while the system manages distributed joins by coordinating between partitions. This segmentation allows complex pattern queries to be processed in parallel across multiple hosts rather than requiring sequential joins on a centralized index, thereby improving throughput for complex queries while maintaining efficient local lookup performance.
2Quantity of substance
If the index is divided into parts assigned to different hosts for distributed processing, then storage scalability is improved, but inter-host communication costs increase significantly for nested loop joins
Solution Approach 1:
The patent applies preliminary action by pre-computing and caching join results for frequently accessed data patterns. When data is accessed locally without requiring remote joins, the system serves requests from local cache, avoiding inter-host communication entirely. This preliminary caching strategy reduces the frequency of remote communications while maintaining distributed storage scalability, thereby reducing communication overhead without sacrificing storage capacity.
3Reliability
If conventional SPARQL engines perform automatic RDFS and OWL inference by behaving as if inferred triples are present, then query completeness is improved, but processing time and memory consumption increase
Solution Approach 1:
The patent implements partial inference by selectively applying RDFS and OWL inference only to the portions of the graph relevant to the current query, rather than pre-computing all possible inferred triples. The system performs inference on-demand during query execution, focusing computational resources only on the necessary subgraphs. This approach maintains query result completeness by ensuring all necessary inferences are made, while reducing overall processing time by avoiding unnecessary inference computations on unrelated data.
Data Source
AI summary
In general, techniques are described for an RDF (Resource Description Framework) database system which can scale to huge size for realistic data sets of practical interest. In some examples, a database system includes a Resource Description Framework (RDF) database that stores a plurality of data chunks to one or more storage drives, wherein each of the plurality of data chunks includes a plurality of triples of the RDF database. The database system also includes a working memory, a query interface that receives a query for the RDF database, a SPARQL engine that identifies a subset of the data chunks relevant to the query, and an index interface that includes one or more bulk loaders that load the subset of the data chunks to the working memory. The SPARQL engine executes the query only against triples included within the loaded subset of the data chunks to obtain a query result.