In-Memory Database Archiving with Compressed Partition Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data production increases, in-memory databases face capacity limitations, leading to a need for efficient data archiving techniques, particularly to prevent data loss by rapidly transferring data from volatile to non-volatile storage.
Innovation Solution
A method and apparatus for compressing data records in a volatile in-memory database, generating compression partitions, and storing them in non-volatile storage with associated storage keys, while maintaining sorting information in index tables for efficient retrieval and archiving.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is archived in non-volatile storage using compression, then storage capacity efficiency is improved, but data retrieval speed may deteriorate
Solution Approach 1:
The patent pre-compresses data into partitions and stores them in non-volatile storage before actual retrieval is needed. Index tables are pre-built with storage keys and sorting information, enabling rapid location of compressed data without decompression overhead during search operations. This preliminary preparation resolves the contradiction by making compression costs paid in advance rather than during retrieval.
Solution Approach 2:
The patent introduces index tables as an intermediary layer between the compressed data in non-volatile storage and the retrieval operations. The index tables store storage keys and sorting information that enable direct navigation to specific compressed partitions without scanning or decompressing unrelated data. This intermediary structure maintains fast retrieval speeds while allowing comprehensive compression of archived data.
2Reliability
If data is transferred from volatile to non-volatile storage, then data safety is improved, but access speed deteriorates
Solution Approach 1:
The patent segments archived data into distinct compression partitions, each with unique storage keys. This segmentation allows the system to transfer only specific partitions to non-volatile storage based on archiving criteria, rather than moving entire datasets. The segmented structure enables selective access to compressed partitions, maintaining fast access speeds for frequently needed data while ensuring safety for archived portions in non-volatile storage.
Solution Approach 2:
The patent applies different storage qualities to different data segments based on their access patterns and importance. Frequently accessed data remains in volatile memory with fast access, while archived data with lower access frequency is stored in non-volatile storage with enhanced safety. The index table maintains local quality information (sorting keys, storage keys) that enables rapid navigation to specific data locations regardless of their physical storage location, resolving the speed-safety contradiction.
Data Source
AI summary
Provided are a data processing method and apparatus capable of rapidly searching for data compressed and stored in non-volatile storage means by compressing a data record, stored in a volatile in-memory database, in a partition unit, storing the compressed data record in the non-volatile storage means, and storing, in a table of the volatile in-memory database, a storage key to uniquely identify each of partitions stored in the non-volatile storage means and sorting information used to determine a partition for a data record.