Live Index Migration in Document Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document stores face challenges in live data migration, requiring downtime and failing to ensure data integrity and security during schema updates, which is not feasible for production environments.
Innovation Solution
A method and system for live index migration in document stores, where a new index is created with requested changes, documents are copied to the new index while the old index remains active, and upon completion, the old index is deleted, ensuring continuous access and data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data migration is performed in traditional document stores, then schema updates can be completed, but the index must be shut down during migration causing downtime
Solution Approach 1:
A new index is created in advance with the desired schema changes before the old index is deleted. This preliminary creation of the target index allows the migration to proceed without forcing shutdown, as the new index is ready to receive data and handle queries immediately upon completion of the copy process
Solution Approach 2:
Data is copied from the old index to the new index while both indexes coexist. This copying mechanism enables the migration to occur without destroying the old index prematurely, thus maintaining service availability throughout the migration process and eliminating downtime
2Adaptability or versatility
If data is copied to a new index during migration, then schema changes can be applied, but no access is allowed to the index during copying
Solution Approach 1:
The system creates a copy of data from the old index to the new index rather than modifying the existing index in place. This copying approach allows the old index to remain accessible for reads and writes during the migration, while the new index is being populated in the background
Solution Approach 2:
The new index acts as an intermediary during the migration process. It receives data copies from the old index while the old index continues to serve application requests. This intermediary mechanism decouples the schema update process from service disruption
3Adaptability or versatility
If traditional data migration is performed, then schema updates are achieved, but data integrity and stateful requirements cannot be ensured
Solution Approach 1:
The system implements a feedback mechanism where the migration process monitors and ensures data consistency between the old and new indexes. Write operations are tracked and replicated to maintain stateful requirements, and the system verifies data integrity before completing the migration by switching to the new index
Solution Approach 2:
The new index is created with the correct schema and data in advance before the old index is deleted. This preliminary population ensures that when the migration completes, the new index is ready to immediately handle all operations, guaranteeing data integrity and stateful requirements are met
Data Source
AI summary
A method and system for live migration of an index in a document store are provided. The method includes creating a new index based on a change request, wherein the change request designates at least one change relative to a current index, wherein the new index includes all mappings and settings of the current index together with the at least one requested change, wherein the new index and the current index are properties of the document store; initiating a reindexing process of the new index by copying documents from the current index to the new index, wherein the current index remains active during the reindexing of the new index; resharding the current index; and upon successful completion of the reindexing process, deleting the current index and directing all write and read requests to the new index.