Hybrid RDF Graph Database with ACID Streaming and Batch Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing RDF databases lack a scalable and efficient method for distributed storage that supports ACID guarantees while allowing for real-time updates and queries, particularly in heterogeneous environments where compute and storage have different cost models.
Innovation Solution
A method for updating a virtual RDF graph database using a combination of a first RDF graph database for streaming updates and a second read-only RDF graph database for batch updates, with a catalog for metadata management, ensuring ACID properties through consistent write operations and concurrency control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed storage is used to scale RDF databases to billions of triples, then storage capacity and scalability are improved, but data consistency and ACID guarantee are worsened
Solution Approach 1:
The system segments the RDF database into multiple partitions distributed across different storage nodes. Each partition can be independently managed while maintaining global consistency through the catalog metadata system. This segmentation enables scaling to billions of triples while preserving ACID properties through coordinated metadata management.
Solution Approach 2:
The catalog serves as an intermediary metadata management system that coordinates access to distributed RDF partitions. It maintains metadata about data location, versioning, and transaction state, enabling ACID-compliant operations across distributed storage by mediating between query processing and physical data storage.
2Speed
If real-time streaming updates are implemented, then update speed and responsiveness are improved, but data integrity and ACID compliance are worsened
Solution Approach 1:
The system performs preliminary actions by maintaining versioned snapshots of the RDF database state before applying updates. Streaming updates are applied to new versions while previous versions remain available for ongoing transactions, ensuring ACID compliance by preparing backup states before modifications occur.
Solution Approach 2:
The system dynamically manages multiple versions of the RDF database simultaneously, with the catalog tracking which version each transaction should see. This dynamic version management enables real-time streaming updates while maintaining ACID properties by allowing concurrent reads from stable versions and writes to new versions.
3Productivity
If compute and storage are distributed separately with different cost models, then resource optimization and cost efficiency are improved, but system complexity and coordination overhead are worsened
Solution Approach 1:
The catalog serves multiple functions: it stores metadata about distributed RDF partitions, manages transaction coordination, tracks data versioning, and provides query optimization information. This multi-functionality reduces the need for separate specialized systems, simplifying the overall architecture while enabling distributed compute and storage with different cost models.
Data Source
AI summary
A computer-implemented method for updating a virtual RDF graph database, the virtual database comprising tuples including a file storage having a durability property of ACID property and guarantying consistent write, providing a virtual RDF graph database including a first RDF graph database updatable by streams and a second read-only RDF graph database stored on the file storage and updatable by batches, a catalog for storing metadata describing the second read-only RDF graph database on the file storage, the catalog being compliant with ACID and obtaining a stream of tuples and a batch of tuples, applying the stream of tuples on the first RDF graph database, applying the batch of tuples on the second RDF graph database by computing a snapshot of the second RDF graph database, including the batch of tuples, the computed snapshot and registering the computed snapshot in the catalog.