Local Storage Engine Metadata Parallel Processing via Database Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The performance of local storage engines is severely affected when handling a large number of small data blocks, leading to performance degradation due to serial data writing and increased metadata submission, which bottlenecks the system's processing capacity.
Innovation Solution
The method involves dividing the database into sub-databases and allocating IDs and storage space, creating metadata submitting threads that correspond to each sub-database, and processing transactions in parallel by converting upper-layer transactions into local transactions and submitting metadata to the appropriate sub-database, thereby improving metadata submission efficiency and system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is serially written to storage in the internal structure of the database, then data consistency is maintained, but the performance of the database is seriously affected when the submission of metadata increases
Solution Approach 1:
The patent divides the database into multiple sub-databases (first sub-database, second sub-database, etc.) and assigns different metadata submission tasks to different sub-databases. This segmentation allows parallel processing of metadata submissions while maintaining data consistency within each sub-database, thereby resolving the contradiction between reliability and productivity.
2Productivity
If the local storage engine is optimized for SSDs by eliminating logging functions, then write amplification is reduced and storage efficiency is improved, but the system suffers from serious performance degradation when processing a large number of small blocks of data
Solution Approach 1:
The patent segments the storage system into multiple sub-databases that can process different data blocks in parallel. This segmentation enables the system to handle a large number of small data blocks efficiently by distributing the processing load across multiple sub-databases, thereby maintaining performance stability while preserving storage efficiency.
3Productivity
If the database is divided into multiple sub-databases and metadata submission is parallelized, then metadata submission efficiency is improved, but the system complexity increases
Solution Approach 1:
The patent divides the database into multiple sub-databases with clearly defined responsibilities for different metadata submissions. Each sub-database handles specific metadata tasks independently, which simplifies the management of complexity by creating modular, manageable units rather than a monolithic complex system.
Solution Approach 2:
The patent introduces a coordination mechanism that acts as an intermediary to manage the parallel metadata submission processes across multiple sub-databases. This intermediary coordinates the parallel operations and ensures consistency, thereby managing system complexity while maintaining high metadata submission efficiency.
Data Source
AI summary
A data processing method, apparatus, and device for a local storage engine system, and a computer-readable storage medium. The method includes: dividing a database and building a plurality of sub-databases; building metadata submitting threads corresponding to the sub-databases; when an upper-layer transaction to be processed is received, converting the upper-layer transaction to a local transaction; determining an ID of a target sub-database used by the local transaction; adding the local transaction to a corresponding database queue, giving a notification for a corresponding target metadata submitting thread to extract the local transaction from the database queue, and sequentially processing metadata, needing to be submitted, in the local transaction; and calling a corresponding database interface to submit the metadata. By means of the method, apparatus, and device, and the computer-readable storage medium, parallel processing of metadata is implemented, thereby improving the performance of a local storage engine.