Search Index Backup via Staging Copy Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In multi-tenant database environments, existing methods for backing up and restoring search indexes are inefficient, leading to prolonged downtime during re-indexing and potential data loss due to the high availability and constant data churn, making it challenging to maintain consistent and reliable index backups.

Innovation Solution

Implement a system that temporarily suspends indexing to create a staging copy of the search index, splits it into partitions, compresses, and stores them asynchronously, allowing for quick restoration by swapping the backup index into the active position and using a replay log to update it with recent changes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If search index backup is performed in multi-tenant database environments with high availability and constant data churn, then data consistency and reliability of index backups are improved, but backup time and system downtime increase

Engineering Contradiction:

Improvedata consistencyVSAvoidbackup time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by creating a staging copy of the search index before actual backup operations. This staging copy is created in advance and prepared for backup, allowing the actual backup process to start from a ready state rather than from scratch, thus reducing backup time while maintaining consistency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The search index is divided into multiple partitions that can be processed independently and asynchronously. This segmentation allows parallel processing of different index segments, reducing the overall backup time while ensuring that each partition maintains data consistency through controlled access and transaction management.

Inventive Principle:

Principle #1Segmentation

2Reliability

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system introduces a staging area as an intermediary between the active search index and the backup storage. This staging area acts as a buffer that simplifies the backup process by providing a controlled environment where index data can be prepared, validated, and staged before being transferred to final backup storage, reducing the complexity of direct backup operations.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system creates a staging copy of the search index that serves as an intermediate representation. This copy can be manipulated and prepared for backup without affecting the original active index, simplifying the backup process by working with a static copy rather than the dynamic original, while maintaining data consistency through controlled copying mechanisms.

Inventive Principle:

Principle #26Copying

3Productivity

If search index is restored quickly by swapping backup index into active position, then system availability and downtime are improved, but data freshness and completeness may be compromised

Engineering Contradiction:

Improverestoration speedVSAvoiddata freshness

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The system performs preliminary actions by pre-processing and validating the staging copy of the index before restoration. This includes applying any necessary updates or transformations in advance, so that when the index is swapped into the active position, it is already in its final, complete state, eliminating the need for post-restoration synchronization and ensuring data freshness.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces the traditional mechanical approach of incremental synchronization after restoration with a comprehensive pre-processing approach. Instead of gradually updating the restored index with changes that occurred during backup, the system substitutes this with a complete pre-computation of the final index state in the staging area, ensuring data completeness while enabling fast restoration through simple swapping.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS9922061B2Methods and systems for backing up a search index

Publication Date: 2018.03.20 SALESFORCE INC

AI summary

An index to be backed up in a computing environment is identified. A staged copy of the index to be backed up is created. The staged copy of the index is segmented. The segments are uploaded to a pre-selected location.