Database Analysis Generation via LSM Tree Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems consume excessive resources in generating database analysis information, as they require traversing or sampling all data in the database, which is inefficient and resource-intensive.
Innovation Solution
A method and system for generating database analysis information by merging data levels using a log-structured merge-tree (LSM tree), where data is read and merged only when necessary, reducing the need to record additional modification amounts and alleviating resource consumption by generating analysis information during the merging process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all data in the database is traversed or sampled to generate database analysis information, then the completeness of analysis information is improved, but resource consumption increases
Solution Approach 1:
The database is segmented into multiple data levels (L0, L1, L2, etc.) with different degrees of merging. Analysis information is generated selectively based on the data level, traversing only necessary portions rather than all data, thus reducing resource consumption while maintaining information completeness.
Solution Approach 2:
Instead of traversing all data in the database, the system performs partial traversal only on the specific data levels required for generating analysis information. This partial action approach reduces computational resources while still producing complete analysis information by leveraging the hierarchical structure.
2Measurement precision
If database analysis information is generated independently by SQL commands or timing tasks, then the accuracy of analysis information is improved, but system overhead increases
Solution Approach 1:
The system merges the function of generating analysis information with the existing data level merging process. When data levels are merged, analysis information is generated as a byproduct, eliminating the need for separate independent generation processes and reducing system overhead while maintaining accuracy.
Solution Approach 2:
The data level merging process is given multiple functions: it both merges data between levels and generates analysis information simultaneously. This multi-functionality reduces the need for separate dedicated processes, thereby reducing system overhead while maintaining the accuracy of analysis information generation.
3Reliability
If data levels are merged frequently to maintain data consistency, then the reliability of database operations is improved, but productivity decreases
Solution Approach 1:
Data is pre-processed and organized into different data levels before merging is needed. This preliminary organization allows merging operations to be performed less frequently while still maintaining data consistency, as the hierarchical structure ensures data is ready for efficient merging when required.
Solution Approach 2:
The system dynamically adjusts the merging frequency and triggers merging operations based on actual data changes and query patterns. This dynamic approach ensures data consistency is maintained only when necessary, avoiding excessive merging operations that would reduce productivity, while still ensuring reliability when merging does occur.
Data Source
AI summary
One or more implementations of the present specification disclose a database analysis information generation method, including: reading data in a to-be-merged data level and data in a merging data level; generating a new merging data level by merging the data in the to-be-merged data level and the data in the merging data level in a merging process; and performing an operation of generating new database analysis information based on data in the new merging data level and data in existing database analysis information in the process of generating the new merging data level.