LSM Tree Aggregate Statistics for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face inefficiencies in collecting and querying aggregate statistics, leading to high processing burdens and resource consumption, particularly during billing operations, and imprecise load balancing due to the need to scan entire data sets for each query.
Innovation Solution
Implementing an aggregated statistics storage system based on a Log Structured Merge (LSM) tree that maintains aggregate values during index merge and garbage collection processes, allowing for efficient querying and minimal I/O and computation effort by storing aggregated statistics in index pages, which can be used for improved load balancing and billing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire index is scanned for each query to collect statistics, then complete and accurate statistical information can be obtained, but the processing burden and resource consumption increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing aggregate statistics in the LSM tree index structure during data ingestion and index maintenance operations. Aggregate values are calculated in advance and stored at multiple levels of the tree, enabling fast query responses without scanning the entire index. This resolves the contradiction by preparing statistical data beforehand, so when queries arrive, the system can retrieve pre-computed aggregates rather than performing expensive full-index scans.
Solution Approach 2:
The patent segments the index into multiple levels (base LSM tree and delta LSM trees) with aggregate statistics stored at different hierarchical levels. Each level contains aggregated data for specific key ranges, allowing the system to retrieve only the relevant segments needed for a query rather than scanning the entire index. This segmentation enables efficient query processing while maintaining statistical accuracy for the requested key ranges.
2Measurement precision
If frequent queries are performed to collect statistics on customer data, then up-to-date statistical information is obtained, but the processing burden on the storage system increases significantly
Solution Approach 1:
The system performs preliminary computation of aggregate statistics during index maintenance operations (merge and garbage collection processes). Aggregate values are updated in advance as data is ingested and index structures are maintained, so that when queries arrive, the statistical information is already ready. This eliminates the need for expensive full-index scans during frequent queries, reducing processing burden while maintaining data freshness.
Solution Approach 2:
The LSM tree index structure serves itself by automatically maintaining and updating aggregate statistics during its normal operation (merge and garbage collection). The index structure incorporates aggregate computation into its inherent maintenance routines, so that statistical data is kept up-to-date without requiring separate processing tasks. This self-service approach reduces the processing burden during queries while ensuring statistical freshness.
3Productivity
If aggregate statistics are maintained during index merge and garbage collection processes, then efficient querying is enabled, but the complexity of index maintenance operations increases
Solution Approach 1:
The patent merges the aggregate statistics maintenance functionality with the existing index merge and garbage collection processes. Instead of adding separate maintenance operations, the system combines aggregate computation with the inherent merge and GC processes of the LSM tree. This merging approach enables efficient querying through maintained aggregates while avoiding additional complexity by leveraging existing maintenance infrastructure.
Solution Approach 2:
The index maintenance processes (merge and garbage collection) are designed to serve multiple functions: they maintain the index structure, update aggregate statistics, and prepare data for queries. This multi-functionality reduces the need for separate aggregate maintenance operations, making the system more efficient while keeping the complexity manageable by consolidating multiple responsibilities into single maintenance routines.
4Measurement precision
If the system scans the entire stored data for each query, then complete statistical information is obtained, but billing operations become expensive and time-consuming
Solution Approach 1:
The system pre-computes and stores aggregate statistics during index maintenance operations, so that billing queries can retrieve pre-computed aggregate values rather than scanning the entire stored data. This preliminary action reduces billing operation duration from exponential to polynomial time complexity while maintaining statistical accuracy for billing calculations.
Solution Approach 2:
The patent extracts aggregate statistics from the full index structure and stores them separately at multiple levels of the LSM tree. This extraction allows billing operations to query only the relevant aggregate data rather than scanning the entire stored data set. The extracted aggregates are maintained during index operations, ensuring billing statistics remain accurate while significantly reducing processing time.
Data Source
AI summary
Embodiments provide a method to collect aggregate information or usage data quickly and efficiently with minimal lag. Additionally, the system can use this aggregate information internally for improved load balancing, better data placement, optimization, and enhanced debugging. The system can quickly look at aggregate information across a huge amount of data and drill down cheaply because the aggregate information is generated using existing processes. Aggregated statistics storage and collection may be built on top of an LSM tree used to store a persistent index for a cloud storage system. The statistics may also represent the result of an operation (e.g., max, min, sum, average) on selected parameter(s) or attribute(s) of stored data. Aggregate statistics values may be efficiently maintained during index merge and garbage collection processes or any other index management. As delta LSM trees are merged into a base LSM tree, the aggregates are updated in delta fashion.