Vector Database Recovery Using Journal and Backup Merge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data recovery methods for vector databases are inefficient and irreversible, taking a long time to apply write events to backup databases and potentially leading to irrecoverable states.
Innovation Solution
A recovery query that merges data from a journal and backup vector database without applying journal entries to the backup first, using a recovery agent to retrieve and merge data points that satisfy the query, allowing for rapid recovery of a target database version.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If write events are applied to backup database before recovery, then data consistency is improved, but recovery time increases and reversibility is lost
Solution Approach 1:
The system performs preliminary actions by maintaining a journal of write events separately from the backup database. The journal pre-stores all write operations in chronological order, allowing the backup to remain in a consistent state without applying events during recovery. This preliminary organization of data enables fast, reversible recovery by simply replaying journal events without modifying the backup first.
Solution Approach 2:
The recovery system segments the backup process into two independent components: the backup database (containing base data) and the journal (containing write events). This segmentation allows the backup to be prepared independently without applying write events, enabling parallel processing and reducing recovery time while maintaining the ability to reverse by discarding the journal.
2Reliability
If write events are applied to backup database before recovery, then data completeness is improved, but reversibility is worsened
Solution Approach 1:
The journal serves as an intermediary between the source database and backup database. It captures write events without immediately applying them to the backup, allowing the system to maintain both completeness (through journal recording) and reversibility (by being able to discard or rollback journal events). The journal mediates the conflict by storing data in a transient, easily reversible format.
Solution Approach 2:
The system creates a copy of write events in the journal rather than directly modifying the backup. This copying approach preserves the original backup state while maintaining a complete record of changes. The journal copy can be discarded or replayed as needed, providing full reversibility while ensuring data completeness through the recorded event log.
3Manufacturing precision
If journal entries are applied to backup database, then data accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The system uses periodic action by batching journal event application. Instead of applying each write event immediately to the backup (which would slow down writes), the system periodically applies journal events to the backup in batches during low-activity periods. This maintains data accuracy while preserving high write speeds and enabling fast recovery by simply continuing the periodic application process.
Data Source
AI summary
A a system may replicate a write event to a journal database, the write event involving a change of a source database including data points arranged in a multidimensional space. The replication of the write event may add a journal data point to the journal database. The journal data point may include write data of the write event and metadata indicating a write operation associated with the write data. Based on receipt of a recovery query associated with recovering data of the source database, the system may obtain, according to a filter in the recovery query, a first result from the journal database and a second result from the backup database. The system may merge the first result and the second result to produce a recovery result in response to the recovery query.