Cache Pre-Reading for De-duplicated File Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing file reading methods for de-duplication files are inefficient due to high hardware costs and slow reading speeds, primarily because they require high-performance storage devices like SSDs to compensate for the non-contiguous storage of chunks on disks, leading to increased costs and latency.
Innovation Solution
A file reading method that utilizes a cache with faster access speeds than the primary storage medium, where data is pre-read from the storage medium and stored in the cache for subsequent read requests, focusing on contiguous and frequently accessed file segments, thereby reducing the need for high-capacity SSDs and minimizing storage overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If an additional storage device (SSD) is used as a cache to store all backed up duplication chunks, then the reading speed is improved, but the hardware cost increases
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different data regions. Instead of uniformly caching all duplication chunks in expensive SSD storage, the system identifies and caches only the most frequently accessed chunks (hot data) in the SSD cache, while less frequently accessed chunks remain in the primary storage medium. This selective caching strategy optimizes reading speed for critical data while minimizing hardware costs.
Solution Approach 2:
The patent employs volatile memory (DRAM) as a cache layer that is cheaper than SSD but loses data on power loss. This volatile cache is used to temporarily store frequently accessed duplication chunks during active reading operations, providing fast access without the high cost of persistent SSD storage. The cache is replenished from primary storage as needed, creating a cost-effective fast access layer.
2Productivity
If all backed up duplication chunks are stored in an additional storage device, then the reading performance is improved, but the capacity requirement of the storage device becomes huge
Solution Approach 1:
The patent extracts only the necessary portion of duplication chunks from the primary storage medium into the cache. Instead of pre-loading all possible chunks, the system extracts and caches only those chunks that are likely to be needed based on access patterns, file being read, and predicted future requests. This extraction strategy improves reading performance for active files while keeping the cache capacity requirement manageable.
Solution Approach 2:
The patent implements preliminary action by pre-loading duplication chunks into the cache before they are actually needed for reading operations. The system predicts which chunks will be needed next based on current read operations and pre-fetches them into the cache, thereby reducing latency when those chunks are actually required. This proactive caching approach improves performance without requiring the cache to hold all possible chunks simultaneously.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Embodiments of the present invention provide a file reading method, storage device, and reading system, which relates to the field of file reading. The method includes: receiving, by a storage device, a first read request sent by a client, where to-be-read data requested by the first read request is a part of the file; reading, from a cache, data that is of the to-be-read data and located in the cache, and reading, from a first storage medium, data that is of the to-be-read data and not located in the cache; and pre-reading, from the first storage medium, data in at least one of the containers, and storing the pre-read data into the cache, where the pre-read container includes at least one unread file segment of the file. The storage device includes: a receive module, a reading module, and a pre-reading module. The reading system includes: a user equipment and the foregoing storage device. According to the present invention, a time overhead in reading data is reduced, and no additional hardware expenditure is required, thereby reducing hardware costs.