File Merging Controller for SSD-HDD Hybrid Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current key-value storage engines based on log-structured merge-trees face inefficiencies in data reading, as random I/O is converted into sequential I/O, but the method to improve read performance is not effectively addressed.
Innovation Solution
A file merging method is introduced, where a controller reads files from a solid state disk, determines if data keys are the same, and merges them into a new file on a mechanical hard disk, optimizing data storage by placing new data on the solid state disk for faster reading and old data on the mechanical hard disk for reduced costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a B+ tree structure, then random I/O performance is maintained, but write performance deteriorates due to frequent random writes
Solution Approach 1:
The patent segments data storage into two distinct structures: an LSM-tree for bulk sequential writes and a B+ tree for random read operations. This segmentation allows each data structure to optimize for its specific function, resolving the contradiction between write speed and random I/O performance by preventing the B+ tree from handling sequential write operations
Solution Approach 2:
The patent introduces an intermediary mechanism (data mapping table and coordinate conversion module) that translates random read requests from the B+ tree into sequential read requests for the LSM-tree. This intermediary enables the system to maintain random I/O performance while utilizing the high write-speed characteristics of the LSM-tree structure
2Speed
If files are merged on the solid state disk, then data reading efficiency is improved, but storage costs increase due to duplicate data retention
Solution Approach 1:
The patent extracts duplicate data identification and removal functionality from the traditional file merging process. By implementing a data mapping table that tracks data locations across files, the system can identify and eliminate duplicates during the merge process, thereby improving reading efficiency without proportionally increasing storage consumption
Solution Approach 2:
The patent changes the parameter of file organization by implementing a hierarchical file structure with different retention policies. Merged files are stored with deduplication enabled, while original files are retained temporarily for validation purposes, allowing the system to optimize storage utilization while maintaining reading efficiency
3Speed
If all data is stored on the solid state disk, then data reading efficiency is maximized, but storage costs increase
Solution Approach 1:
The patent applies local quality by differentiating storage locations based on data characteristics and access patterns. Frequently accessed and recently written data are prioritized for SSD storage, while less frequently accessed data are stored on HDD. This localized optimization achieves high reading efficiency for critical data while controlling overall storage costs
Solution Approach 2:
The patent introduces a temporal dimension to storage management by implementing a data lifecycle policy. Data transitions between SSD and HDD storage based on age and access frequency, creating a multi-dimensional storage hierarchy that balances reading efficiency requirements with cost constraints
Data Source
AI summary
A file merging method performed by a controller in a storage system includes reading a first file and a second file on a solid state disk, determining whether a key of the first data is the same as a key of the second data, creating a third file on a mechanical hard disk when the key of the first data is the same as the key of the second data, merging the first data and the second data, and writing the merged data into the third file.