Intelligent File System Scanning Using USN Checkpoints to Reduce Polling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing filesystem scanning methods are computationally expensive and time-consuming, particularly when dealing with large amounts of data, leading to degraded system performance.
Innovation Solution
Implement a file stream scanning module that utilizes a sparse file stream to identify file events based on Update Sequence Numbers (USNs) and a logfile scanning module that uses Log Sequence Numbers (LSNs) to minimize unnecessary scans, combined with a velocity-based scanning module to adjust scan frequency based on filesystem activity density.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional filesystem scanning is performed to identify all file events, then complete file event detection is achieved, but computational cost and scan time increase significantly
Solution Approach 1:
The patent extracts only the necessary information (USN changes) from the filesystem metadata rather than scanning entire files. By reading only the USN journal and comparing sequence numbers, the system identifies file events without processing file contents, dramatically reducing scan time while maintaining detection completeness.
Solution Approach 2:
The system performs preliminary scanning of the USN journal to identify changed files before initiating full file scans. By pre-identifying which files have USN changes using minimal I/O operations, the system avoids unnecessary full scans of unchanged files, reducing overall scan time while ensuring no events are missed.
2Speed
If frequent filesystem polling is performed to detect file events, then file event detection及时性 is improved, but system performance degradation increases
Solution Approach 1:
The system implements periodic scanning based on USN journal changes rather than continuous polling. By checking the USN sequence numbers at intervals and only scanning when changes are detected, the system maintains fast event detection while avoiding the performance degradation of constant filesystem access.
Solution Approach 2:
The USN journal automatically records file system changes, allowing the scanning system to query change information without actively polluting the filesystem. The journaling mechanism itself provides the change detection service, eliminating the need for aggressive polling and reducing system performance impact.
3Measurement precision
If full filesystem scans are performed to ensure accurate file event identification, then detection accuracy is maintained, but computational load increases
Solution Approach 1:
The patent segments the scanning process into two phases: first scanning only the USN journal to identify changed files, then scanning only those specific files. This segmentation avoids the computational load of scanning entire filesystems while maintaining accuracy by ensuring all changed files are captured through the USN change detection.
Solution Approach 2:
The USN journal acts as an intermediary data structure that records file change information. By querying the USN journal rather than directly scanning files, the system obtains accurate file event information with minimal computational load, as the journal already contains the change metadata without requiring file content analysis.
Data Source
AI summary
A system may parse, during a current filesystem check, a sparse file stream comprising a plurality of Update Sequence Numbers (USNs). A system may access a checkpoint USN that is locally stored apart from the sparse file stream and serves as a checkpoint of a prior USN that was accessed during a prior filesystem check. A system may identify a next byte offset for a next USN of a file in the sparse file stream after the checkpoint USN based on the byte offset of the checkpoint USN. A system may update the checkpoint USN with the next USN. A system may identify a file event from the sparse file stream based on the next USN without polling files or directories on the filesystem. A system may pass the file event to a downstream system.