Atomic Cache Management for File Collections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data platforms manage cache eviction on a file-by-file basis, leading to inefficiencies and resource waste, as files are evicted individually even if related files are still in use, causing data corruption and requiring re-transfer of entire packages.
Innovation Solution
Implementing a disk cache with lock files to manage collections of files atomically, ensuring that all or none of related files are evicted from the cache, preventing corruption and optimizing resource utilization by maintaining files in the cache until no longer needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If files are evicted from cache on a file-by-file basis, then cache space is freed incrementally, but data corruption occurs and resource waste increases due to re-transfer of entire packages
Solution Approach 1:
The patent merges multiple individual file eviction operations into a single atomic package-level operation. By treating a collection of related files as an indivisible unit managed through a common lock file, the system ensures that either all files in the package are evicted together or none are, preventing partial evictions that cause data corruption while still freeing cache space when the entire package is no longer needed.
Solution Approach 2:
The patent introduces a lock file as an intermediary mechanism between the cache management system and the collection of data files. This lock file serves as a coordination point that enables atomic operations on the entire package of files, allowing the system to manage evictions at the package level rather than individually managing each file, thus preventing corruption while optimizing resource usage.
2Reliability
If atomic cache management is implemented using lock files, then data integrity is maintained and resource waste is reduced, but system complexity increases due to lock file management overhead
Solution Approach 1:
The lock file mechanism is designed to be self-managing within the cache system. When a package of files is evicted, the lock file automatically tracks the eviction status and manages the atomic operation without requiring complex external coordination. The lock file itself serves as the state tracker and coordination mechanism, simplifying the overall system architecture while maintaining data integrity.
3Productivity
If collections of files are managed atomically, then processing efficiency is enhanced and resource waste is reduced, but cache management complexity increases due to atomic operation requirements
Solution Approach 1:
The patent combines multiple file operations into a single atomic package-level operation managed by a lock file. This merging approach enhances processing efficiency by preventing partial evictions that would require re-transfers, while the lock file provides a simple mechanism to coordinate these atomic operations without introducing excessive complexity into the cache management system.
Data Source
AI summary
Systems and methods for caching file system collections atomically. The systems and methods perform operations comprising: receiving a request to access a collection of data comprising a plurality of files stored on a storage device; in response to receiving the request, transferring the collection of data from the storage device to a first subfolder in a cache associated with the storage device; generating a lock file comprising a reference to the first subfolder in the cache; and atomically controlling access to the collection of data in the first subfolder and removal of the collection of data in the first subfolder from the cache via the lock file.