Database System Catalog Overrides and Lock Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems face inefficiencies in performing index and tenant validity checks and lock management, leading to increased communication overhead and potential inconsistencies, especially when handling multi-step metadata operations across a database and its storage catalog server.
Innovation Solution
Implementing a shared memory system that stores current system catalog overrides, allowing database systems to perform validity checks and manage locks without continuous communication with the storage catalog server, and using workflow daemons to manage lock ownership and prevent invalid interleaving of operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous communication with storage catalog server is performed for validity checks and lock management, then data consistency is maintained, but computational and communication overhead increases
Solution Approach 1:
The system pre-loads system catalog data into shared memory before it is needed for validity checks and lock management operations. This preliminary action allows the database system to perform operations using cached data rather than continuously communicating with the storage catalog server, thereby reducing communication overhead while maintaining data consistency through selective updates from the catalog server.
2Ease of operation
If locks are acquired and dropped frequently during database operations, then operation granularity is improved, but system performance deteriorates due to continuous lock management overhead
Solution Approach 1:
The patent implements dynamic lock management where lock duration and scope are adjusted based on workflow requirements. Instead of uniformly acquiring and dropping locks at fixed points, the system maintains locks for the duration of workflow operations that require them, and uses workflow daemons to coordinate lock ownership transfers. This dynamic approach reduces the frequency of lock acquisition and release operations while maintaining appropriate operation granularity.
3Loss of information
If system catalog changes are immediately visible to all operations, then data freshness is improved, but consistency during multi-step operations deteriorates due to potential invalid interleaving
Solution Approach 1:
The system introduces workflow daemons as intermediary components that mediate between system catalog changes and database operations. These daemons manage lock ownership and coordinate access to shared resources during multi-step operations. When system catalog changes occur, the workflow daemons ensure that operations are properly synchronized and that locks are held for the duration of critical sections, preventing invalid interleaving while allowing catalog updates to propagate through the system.
4Adaptability or versatility
If catalog server is separated from database system, then system modularity is improved, but communication overhead increases for metadata operations
Solution Approach 1:
The patent segments the catalog data into two parts: frequently accessed system catalog data is cached in shared memory accessible by the database system, while the storage catalog server retains the authoritative copy for updates and persistence. This segmentation allows the database system to perform most operations using local cached data without communication overhead, while the catalog server remains available for metadata updates and consistency operations.
Solution Approach 2:
The system creates copies of system catalog data in shared memory that can be accessed by the database system without communicating with the storage catalog server. These copies are updated from the catalog server at appropriate intervals and triggered by specific events, allowing the database to operate with low-latency access to metadata while the catalog server maintains the authoritative source of truth.
Data Source
AI summary
Systems and methods are provided for a shared memory between a database system and a storage catalog server, which stores system catalog overrides and locks for at least a first workflow. The current system catalog overrides include at least a portion of the system catalog overrides and locks that are retrieved by the shared memory from the storage catalog server at a predetermined point in time. The database system performs a check of at least one of index validity and tenant validity for a second workflow or other database operation by checking a state of the corresponding tenant or index in the current system catalog overrides. The system catalog overrides suppress visibility of the respective system catalog changes when the database system determines that at least one of the index and the tenant is invalid based on the current system catalog overrides for the first workflow.