In-Memory Database Index Persistence for Fast Restart Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory databases face significant challenges in index recovery time due to restarts, which restrict their expansion capability and cause resource wastage, especially when large data volumes are involved.
Innovation Solution
A data processing method that involves determining a memory allocation area, obtaining and writing index data and allocation state to a magnetic disk file, and mapping this data back to memory upon restart, ensuring persistence and efficient recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If index data is backed up at a set time node in the in-memory database, then persistence is achieved, but restart time increases significantly when data volume is large
Solution Approach 1:
The patent applies preliminary action by pre-writing index data to the magnetic disk before database restart occurs. The system continuously maintains index data on the magnetic disk during normal operation, so when restart is needed, the data is already prepared and available, eliminating the need for time-consuming scanning and recovery operations during restart.
Solution Approach 2:
The patent uses copying by creating and maintaining a copy of the index data on the magnetic disk. Instead of storing data only in volatile memory, the system creates and updates a persistent copy on magnetic storage, allowing rapid restoration by simply loading the existing copy rather than reconstructing indexes from scratch.
2Speed
If the in-memory database stores large amounts of data in memory, then read and write speed improves, but expansion capability is restricted due to long index recovery time
Solution Approach 1:
By pre-maintaining index data on magnetic disk during normal operation, the system enables rapid expansion capability. When the database needs to scale or restart, the indexes are already prepared on disk, allowing the system to handle larger data volumes without proportionally increasing restart time, thus improving expansion capability while maintaining high speed performance.
3Reliability
If index recovery is performed by scanning data upon restart, then completeness is ensured, but time consumption increases with large data volumes
Solution Approach 1:
The system maintains a persistent copy of index data on magnetic disk that is updated during normal operation. Upon restart, instead of scanning all data to reconstruct indexes, the system directly loads the pre-existing copy from disk, ensuring recovery completeness while dramatically reducing recovery time regardless of data volume.
Solution Approach 2:
The index data is prepared and written to magnetic disk in advance during normal database operation. This preliminary preparation ensures that when restart occurs, the complete and accurate index data is already available, eliminating the need for time-consuming scanning while maintaining recovery completeness.
Data Source
AI summary
Embodiments of the specification provide a data processing method and apparatus, where the data processing method includes: determining a memory allocation area corresponding to a regional address and a memory allocator associated with the memory allocation area; obtaining index data of the memory allocation area and an allocation state of the memory allocator, and writing the index data and the allocation state to a magnetic disk file; reading the index data and the allocation state from the magnetic disk file upon restarting an in-memory database containing the memory allocation area; and mapping the index data to the memory allocation area according to the regional address, and performing an update on a state of the memory allocator according to the allocation state.