File System Aggregate Metrics via Alternate Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional filesystems face significant latency and performance issues when aggregating metadata attributes to provide hierarchical aggregate metric values, leading to inefficient system performance and inaccurate data reflections due to the need for exhaustive traversal of filesystem trees and high Input/Output operations.
Innovation Solution
A software and hardware facility that maintains persistent hierarchical aggregates or metric values for file attributes in each directory within a filesystem tree, allowing for rapid access and reducing the need for extensive tree traversal by storing aggregated data at various levels, including the use of alternate data streams (ADS) for enhanced metadata management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional filesystems aggregate metadata attributes by exhaustive traversal of filesystem trees, then complete and accurate aggregate metric values are obtained, but significant latency and performance degradation occur
Solution Approach 1:
The patent pre-calculates and stores aggregate metric values (such as total file count, total size, maximum file size) in metadata structures before they are needed. When a query is made, the pre-computed aggregates are retrieved directly rather than traversing the entire filesystem tree, thus eliminating the time penalty while maintaining accuracy.
Solution Approach 2:
The filesystem tree is divided into hierarchical segments (directories and subdirectories), with aggregate metrics maintained at each level. This segmentation allows the system to compute and store partial aggregates locally at each directory level, which can then be combined or retrieved without traversing the entire tree, reducing both time and computational overhead.
2Measurement precision
If exhaustive traversal of filesystem trees is performed to aggregate metadata, then accurate hierarchical aggregate metric values are produced, but high Input/Output operations and processing overhead occur
Solution Approach 1:
Aggregate metrics are pre-computed and cached in metadata structures during filesystem operations. When files are created, modified, or deleted, the corresponding aggregate values are updated incrementally and stored ahead of time, so that query operations can retrieve these pre-computed values without performing expensive traversals, thus maintaining both accuracy and high productivity.
Solution Approach 2:
Instead of traversing the original filesystem tree structure to compute aggregates, the patent creates and maintains separate copy structures (metadata aggregates) that mirror the hierarchical organization but store pre-computed summary information. These copies enable rapid retrieval of accurate aggregate metrics without the I/O overhead of traversing the actual file system contents.
3Loss of information
If filesystems store detailed metadata for all files and directories, then complete information is available for analysis, but storage overhead and access complexity increase
Solution Approach 1:
Metadata is segmented into multiple hierarchical levels (individual file metadata, directory-level aggregates, root-level aggregates). Each level stores only the information necessary for its level, with aggregate values computed from child levels. This segmentation reduces the complexity of managing complete metadata while preserving all necessary information through the hierarchical structure.
Solution Approach 2:
The patent extracts and separates aggregate metric information from the detailed file metadata. Instead of storing and processing all detailed metadata whenever aggregate information is needed, the system extracts only the necessary aggregate values (file count, total size, max size) into dedicated metadata fields, reducing access complexity while maintaining information completeness when needed.
Data Source
AI summary
Embodiments are direct to managing file systems. Alternate data streams (ADSs) of an object in a file system may be modified based on a request from a client and metrics associated with the ADSs may be modified. Values for reconcile fields associated with the object may be set based on the modified metrics such that the object with the modified ADSs may be included in a reconcile list. And, such that an acknowledgment of the request may be provided to the client. A reconcile engine may determine a parent object of the object. The object may be reconciled and the parent object may be placed on the reconcile list for reconciling in turn.