Log-Based Database Catalog Updates for Multi-Tenant Concurrency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database catalog management systems in multi-tenant environments using Open Table Formats (OTFs) face performance bottlenecks due to the reliance on relational database management systems, which hinder concurrency and scalability when handling thousands of tables and clients.
Innovation Solution
Representing the database catalog as a set of log entries, where each entry describes incremental updates, and using a log-based data structure for fast, append-only updates and scans, along with RDF triples to manage transactions and signatures to identify conflicts, allowing concurrent updates by tenants.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a relational database management system is used to manage the database catalog, then the catalog can be represented in a structured format, but the system becomes a performance bottleneck and cannot scale well in environments with thousands of tables and clients
Solution Approach 1:
The patent extracts the catalog management functionality from the relational database management system and implements it as a separate log-based catalog. This removes the RDBMS as a bottleneck while maintaining catalog management capabilities through a dedicated log structure that handles metadata operations independently.
Solution Approach 2:
The patent introduces a log-based catalog as an intermediary layer between the RDBMS and the OTF drivers. This mediator handles catalog operations efficiently using append-only logs and version vectors, reducing the burden on the RDBMS while maintaining structured catalog representation.
2Reliability
If the catalog is managed by a separate relational database management system, then catalog metadata can be stored systematically, but concurrency benefits of OTF are compromised due to the RDBMS becoming a bottleneck
Solution Approach 1:
The patent changes the fundamental parameters of catalog storage from traditional relational tables to an append-only log structure with version vectors. This transformation enables concurrent reads and writes by multiple tenants simultaneously, as each operation appends to the log rather than modifying existing records, thereby maintaining metadata integrity while dramatically improving concurrency.
Solution Approach 2:
The log-based catalog enables continuous catalog updates without requiring locks or transactions that would block other operations. Multiple tenants can continuously read and write catalog metadata concurrently, with the log structure naturally handling conflicts through version vectors and append-only semantics, ensuring uninterrupted service.
3Adaptability or versatility
If thousands of clients query the catalog concurrently, then the system can support large-scale multitenancy, but the relational database management system performance degrades
Solution Approach 1:
The patent segments the catalog management workload from the RDBMS by implementing a dedicated log-based catalog system. This segmentation allows the RDBMS to focus on data storage while the log-based catalog handles metadata operations, enabling the system to scale to thousands of concurrent clients without degrading query response time.
Solution Approach 2:
The patent creates a separate copy of the catalog in log-based format, independent of the RDBMS. This copy enables high-speed concurrent access for thousands of clients, while the original RDBMS catalog remains for persistence. The log-based copy can be read concurrently without affecting the RDBMS, dramatically improving query speed at scale.
Data Source
AI summary
A system includes reception of a first transaction including updates to a database catalog comprising transactions stored on a log and a timestamp of a snapshot of the database catalog, determination of a signature of a transaction stored on the log after the timestamp of the snapshot, determination, based on the signature, that a conflict exists between the updates to the database catalog and updates of the stored transaction, and, in response to the determination that a conflict exists, reject the first transaction.