Persistent Memory Data Storage Method for Reducing Write Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems using volatile memories are prone to data loss during power failures or system crashes, leading to reduced storage efficiency and increased latency due to the need for data recovery from log files, which occupies system resources and affects IO performance.
Innovation Solution
Implementing a data storage method that uses persistent memory to cache to-be-written data in a log file and then write it to a memory table when predetermined conditions are met, allowing the memory table to be merged into a mass storage apparatus, thereby improving storage efficiency by reducing data loss and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If volatile memory is used for data processing and caching, then data access speed is improved, but data loss occurs during power failure or system crash
Solution Approach 1:
The patent introduces persistent memory as an intermediary layer between volatile memory and non-volatile storage devices. This persistent memory acts as a mediator that provides both the fast access characteristics of volatile memory and the data persistence of non-volatile storage, thereby resolving the contradiction between speed and reliability.
Solution Approach 2:
The patent changes the physical state parameter of the memory medium from volatile to persistent, fundamentally altering the data retention characteristics while maintaining fast access speeds. This parameter change enables the system to achieve both high speed and data persistence simultaneously.
2Reliability
If data is frequently dumped from persistent memory to secondary storage, then data persistence is improved, but write amplification increases and system performance decreases
Solution Approach 1:
The patent implements periodic flushing of data from persistent memory to secondary storage based on thresholds (such as memory table size or time intervals) rather than continuous dumping. This periodic action reduces write amplification while maintaining data persistence, thereby improving system performance.
Solution Approach 2:
The patent maintains a continuous data flow from persistent memory to secondary storage through background flushing operations, ensuring data persistence without interrupting foreground operations. This continuous action balances reliability and performance by performing dumps asynchronously.
3Reliability
If data recovery from log files is performed during system failure, then data loss is prevented, but system resources are occupied and IO performance is affected
Solution Approach 1:
The patent performs preliminary data persistence operations by writing data to persistent memory before processing, and implements periodic flushing to secondary storage before system failures can occur. This preliminary action eliminates the need for intensive recovery operations from log files, thereby maintaining high IO performance while ensuring data recovery capability.
Data Source
AI summary
Embodiments of the present disclosure provide a data storage method, an electronic device, and a computer program product. The method includes: determining the number of at least one first storage apparatus of a storage system, each first storage apparatus being persistent and storing a log that includes record entries of to-be-stored data; and writing, if it is determined that the number is greater than a threshold number, the to-be-stored data into a corresponding memory table from the log where the to-be-stored data is located, wherein data in the memory table will be written to a second storage apparatus of the storage system. In this way, the storage efficiency of the storage system can be improved.