Filesystem Hierarchical Metric Aggregation via Alternate Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional filesystems face significant latency and performance issues when aggregating metadata attributes to provide hierarchical aggregate metric values, leading to inefficient system performance and inaccurate data reflections due to the need for exhaustive traversal of filesystem trees and high Input/Output operations.
Innovation Solution
The solution involves maintaining persistent hierarchical aggregates or metric values for file attributes at different levels within a filesystem tree, allowing for rapid access and updates without exhaustive traversal, using deterministic functions to aggregate attributes and store them in directories, and employing alternate data streams (ADS) features to manage extended metadata alongside file data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional filesystems aggregate metadata attributes by exhaustive traversal of filesystem trees, then complete and accurate hierarchical aggregate metric values are obtained, but significant latency and performance degradation occur
Solution Approach 1:
The patent pre-calculates and stores hierarchical aggregate metric values in persistent storage locations within the filesystem tree structure. When metadata attributes change, the system performs selective updates to pre-computed aggregates rather than traversing the entire tree, thereby maintaining accuracy while reducing latency significantly
Solution Approach 2:
The filesystem tree is divided into hierarchical segments or subtrees, each with its own aggregate metric values stored in directory structures. This segmentation allows the system to query and update only specific portions of the filesystem rather than performing exhaustive traversal of the entire tree, reducing time loss while maintaining complete accuracy through proper aggregation of segmented data
2Measurement precision
If conventional filesystems perform exhaustive traversal to aggregate metadata, then complete data coverage is achieved, but high Input/Output operations and processing overhead occur
Solution Approach 1:
Aggregate metric values are pre-computed and cached in persistent storage within the filesystem hierarchy. When files are created, modified, or deleted, the system performs targeted updates to affected aggregate values rather than re-traversing entire subtrees, dramatically reducing I/O operations while maintaining complete data coverage through incremental update mechanisms
Solution Approach 2:
The patent extracts and stores aggregate metric values separately from the actual filesystem data structures in dedicated metadata locations. This extraction allows the system to retrieve and update aggregates without accessing or traversing the complete filesystem tree, reducing processing overhead and I/O operations while maintaining complete data coverage through the separated aggregate storage mechanism
3Speed
If filesystems store persistent hierarchical aggregates at different levels, then rapid access to metric values is enabled, but additional storage space and data structure complexity are required
Solution Approach 1:
The patent implements nested hierarchical aggregates where directory structures contain both their own aggregate metric values and references to aggregates of their child directories. This nesting allows rapid access to metrics at any hierarchical level by simply querying the corresponding directory structure, while the nested organization naturally manages data structure complexity through the existing filesystem hierarchy rather than requiring separate complex indexing mechanisms
Data Source
AI summary
Embodiments are directed to managing data in a file system. A query may be provided to determine characteristics associated with a file system and a sampling model. The sampling model may be employed to determine objects in the file system based on a traversal of the file system. The sampling model may be employed to determine metrics associated with the alternate data streams (ADSs) included in the objects such that each of the objects that includes a separate main data stream and the ADSs may be associated with data blocks and the ADSs may each be associated with other data blocks. The sampling model and the query may be employed to sample the objects to provide partial results such that the sampling may be distributed between the objects based on the sampling model and the metrics. A result to the query may be provided based on the partial results.