Structured File System Journals for Low-Latency Metric Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large-scale data storage systems with numerous files becomes costly and interferes with performance due to the need for traversing large numbers of distributively stored files and continuous updating of file system metrics.
Innovation Solution
A method and system for managing file systems using a network computer that provides update information, generates a journal with ordered base and level bins, and employs these bins to reduce latency in generating query results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If file system metrics are continuously updated by traversing large numbers of distributively stored files, then metric accuracy is improved, but file system performance deteriorates
Solution Approach 1:
The patent segments the file system into hierarchical levels (root level, intermediate levels, and leaf level) and maintains separate data structures for each level. This segmentation allows metrics to be updated and queried at appropriate granularity without traversing the entire file system, thus maintaining accuracy while improving performance.
Solution Approach 2:
The patent pre-computes and stores aggregate metrics at each hierarchical level in advance. When a metric query is issued, the system retrieves pre-computed values from the hierarchical structure rather than computing them on-demand by traversing all files, thereby eliminating performance degradation while maintaining accuracy.
2Loss of information
If large-scale file systems are traversed to aggregate file metrics, then comprehensive metric information is obtained, but time consumption increases
Solution Approach 1:
The patent introduces a hierarchical dimension to the metric storage structure, organizing metrics by file system levels rather than flat enumeration. This dimensional transformation allows the system to retrieve comprehensive metrics by aggregating data from multiple hierarchical levels simultaneously, dramatically reducing time consumption while maintaining information completeness.
Solution Approach 2:
The patent implements a nested hierarchical structure where metrics at each level contain or reference metrics from subordinate levels. This nesting allows comprehensive metric information to be obtained by examining metrics at a single hierarchical level, as each level's metrics inherently incorporate data from lower levels, thus eliminating the need for time-consuming traversals.
3Measurement precision
If individual file metrics are aggregated across distributed storage, then accurate aggregate metrics are achieved, but computing resources are excessively consumed
Solution Approach 1:
The patent merges metric computation and storage operations by maintaining a hierarchical metric structure that combines aggregate metrics from multiple levels. This merging eliminates redundant computation of the same metrics across different file system levels, significantly reducing computing resource consumption while preserving the accuracy of aggregate metrics through the hierarchical aggregation relationship.
Data Source
AI summary
Embodiments are directed to managing file systems. Update information associated with a change of a metric associated with a target object may be provided. A journal that includes a base bin that includes base records that associate the metric with each object in the file system. Records that include the change of the metric associated with the ancestors of target object may be generated. Another record that includes the change of the metric associated with the target object may be generated. A level bin associated with the base bin of the journal may be provided based on the update information. The records may be stored in the level bin using a sort order based on the ordering of the base bin records. In response a query, the journal may be employed to reduce latency in generating query results.