File Access Counter System for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file analytics solutions face challenges in accurately tracking file access patterns, particularly in managing lesser-accessed files and maintaining data integrity during system crashes or power outages, leading to inefficiencies and high write amplification.
Innovation Solution
Implementing an access counter system that increments file access events in memory and persists data only when specific thresholds are met, allowing for intelligent storage optimization between fast and long-term storage, reducing write amplification and improving data reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If access counter information is persisted on storage media after every file access, then data reliability is improved, but write amplification increases and performance deteriorates
Solution Approach 1:
The patent implements periodic persistence of access counter information by using a generation counter that increments with each access. Instead of persisting after every access, the system periodically flushes access counter data to storage media when the generation counter reaches a threshold value, thereby reducing write operations while maintaining data reliability.
Solution Approach 2:
The patent uses a buffer in memory to preliminarily store access counter information before persisting it to storage media. This buffer accumulates access counter data from multiple file accesses, and only when the buffer reaches a certain capacity or time threshold does the system perform the actual write operation to storage, reducing the frequency of write operations.
2Productivity
If access counter information is stored in memory only, then write amplification is reduced and performance is improved, but data reliability deteriorates due to potential loss during crashes or power outages
Solution Approach 1:
The patent replaces the traditional mechanical approach of immediately writing to storage media with a software-based buffering mechanism in memory. The buffer temporarily holds access counter information and uses intelligent flushing strategies (based on generation counters and thresholds) to determine when to persist data, thereby improving performance while maintaining reliability through selective persistence.
3Measurement precision
If identification of top set of most heavily-accessed files is retained, then file analytics capability is improved, but information about lesser-accessed files is lost
Solution Approach 1:
The patent segments the file access tracking system into two parts: a buffer in memory that tracks access counters for all files (providing comprehensive information), and a persistent storage structure that periodically saves this information. This segmentation allows the system to maintain detailed analytics for all files in memory while only persisting data periodically, thus preserving information about lesser-accessed files without the overhead of continuous writing.
Data Source
AI summary
Optimizing file access includes a process for identifying a file access event for a first accessed file, and incrementing a first access counter in an access list in a memory, which also includes access counters for other accessed files. The process further includes exporting the first access counter to a performance monitoring dashboard, or exporting to a storage allocator and, based on the value, moving the first accessed file between a first storage and a second storage. The process also includes determining whether the value of the first access counter meets a first threshold, or a sum of values of the access counters for the other accessed files meets a second threshold. Based on meeting the first threshold or meeting the second threshold, the process includes persisting the access counters on a storage media. The access counters also provide security monitoring (e.g., identifying excessive file access).