Metadata Management for Transactional Storage Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance degradation due to the need to scan millions of data blocks during queries, especially when many queries are simultaneous, as they often include both active and converted data blocks, leading to increased scan times without improving query efficiency.
Innovation Solution
A metadata management method and system that set a read epoch timestamp based on the earliest active write epoch timestamp and selectively scan data blocks from this timestamp onwards, reducing the number of blocks to be scanned by excluding converted blocks, and optionally updating timestamps to optimize scanning efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system scans all data blocks including converted blocks to ensure complete query coverage, then query completeness is improved, but query response time deteriorates
Solution Approach 1:
The patent extracts and removes converted data blocks from the scan set by using conversion indicators stored in metadata. The query processing system identifies data blocks with conversion indicators set, excludes them from scanning, and only processes active data blocks, thereby reducing scan time while maintaining query completeness for relevant data.
Solution Approach 2:
The patent performs preliminary actions by pre-calculating and storing conversion indicators in metadata before queries are executed. The system maintains epoch timestamps and conversion status information in advance, allowing query processors to quickly determine which data blocks are active without real-time analysis, thus reducing query response time.
2Measurement precision
If the system scans all data blocks to handle concurrent queries, then query accuracy is improved, but system throughput deteriorates
Solution Approach 1:
The patent extracts converted data blocks from the processing pipeline using metadata conversion indicators. By identifying and excluding data blocks marked as converted through stored indicators, the system reduces the number of blocks requiring scanning for each query, thereby improving system throughput while maintaining accuracy for active data.
Solution Approach 2:
The system performs preliminary classification of data blocks into active and converted categories using epoch timestamps and conversion indicators stored in metadata. This pre-processing allows concurrent queries to quickly identify and scan only active data blocks, improving both query accuracy and system throughput under concurrent workloads.
3Productivity
If the system maintains detailed metadata for all data blocks to enable selective scanning, then query efficiency is improved, but storage overhead increases
Solution Approach 1:
The patent uses lightweight metadata structures with simple conversion indicators (boolean flags) and epoch timestamps for each data block. These minimal metadata elements provide sufficient information to distinguish active from converted blocks without requiring complex or extensive metadata storage, thus improving query efficiency while minimizing storage overhead.
Data Source
AI summary
A method for managing metadata for a transactional storage system include receiving a query request at a snapshot timestamp. The query request requests return of at least one data block from a plurality of data blocks. Each data block includes a corresponding write epoch timestamp and a corresponding conversion indicator indicating whether the data block is active or has been converted at a respective conversion timestamp. The method also includes setting a read epoch timestamp equal to the earliest one of the write epoch and determining whether any of the respective conversion timestamps occurring at or before the snapshot timestamp occur after the read epoch timestamp. The method also includes determining the at least one data block requested by the query request by scanning each of the data blocks including corresponding write epoch timestamps occurring at or after the read epoch timestamp.