Log-Structured Database Catalog with Signature-Based Conflict Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database catalog management systems in multitenant environments face performance bottlenecks due to inefficient concurrency handling, particularly when thousands of clients frequently read and update the catalog, compromising the benefits of cloud-based object stores.
Innovation Solution
Implementing a log-structured database catalog that represents the catalog as a set of log entries, allowing concurrent updates by computing incremental changes and using signatures to detect conflicts, thereby enabling fast, append-only operations and minimizing lock duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a relational database management system is used to manage the database catalog, then the catalog can be represented in a structured format, but the system becomes a performance bottleneck in multitenant environments with thousands of tables and clients
Solution Approach 1:
The patent segments the monolithic catalog management system into distributed, independent catalog instances that can be queried and updated concurrently. Each catalog instance operates autonomously without requiring centralized coordination, enabling parallel processing of catalog operations across multiple tenants and clients simultaneously.
Solution Approach 2:
The patent introduces a catalog service as an intermediary layer between clients and the relational database management system. This service handles catalog operations through a standardized interface, buffering and managing the complexity of concurrent access, and allowing clients to interact with catalogs without directly burdening the underlying RDBMS with concurrency control complexity.
2Adaptability or versatility
If the catalog is frequently read and updated in a multitenant environment, then the system supports dynamic operations, but lock duration increases and performance degrades
Solution Approach 1:
The patent implements self-service catalogs where each catalog instance manages its own state and operations independently. Catalog operations are performed directly on individual catalog instances without requiring locks or coordination with other catalogs, eliminating lock wait times and enabling concurrent read and write operations across multiple tenants simultaneously.
Solution Approach 2:
The patent performs preliminary actions by pre-configuring catalog instances with default schemas and structures before they are needed. This allows catalogs to be ready for immediate use without requiring time-consuming initialization operations during runtime, reducing the duration of any necessary locks or synchronization operations.
3Productivity
If cloud-based object stores are used for shared storage, then elastic scale-out and high availability are achieved, but catalog management becomes more complex in distributed environments
Solution Approach 1:
The patent creates universal catalog instances that can serve multiple functions and tenants through a standardized interface. Each catalog instance is designed to be multi-functional, supporting various operations (create, read, update, delete) and serving different tenants with the same structural blueprint, thereby simplifying distributed management while maintaining storage scalability.
Solution Approach 2:
The patent enables parameter changes in catalog instances through configurable schemas and metadata. Catalog instances can be dynamically adjusted by modifying their parameters and configurations without changing their fundamental structure, allowing flexibility in storage management while maintaining consistency across the distributed system.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system includes reception of a first transaction including updates to a database catalog comprising transactions stored on a log and a timestamp of a snapshot of the database catalog, determination of a signature of a transaction stored on the log after the timestamp of the snapshot, determination, based on the signature, that a conflict exists between the updates to the database catalog and updates of the stored transaction, and, in response to the determination that a conflict exists, reject the first transaction.