Data Store Partition Replication for Scalable Entity Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face challenges in scaling to handle increasing volumes of customer information, leading to resource limitations such as storage and processing capacity issues, which can be costly and logistically difficult to address by adding new hardware, and may disrupt network-based services during data migration.
Innovation Solution
A scalable data store framework that migrates data across partitions by replicating entities from a target partition to new partitions, allowing for load balancing and maintaining data availability during the process, using a data store system with a global entity-partition table and partition management components to manage and redirect requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If additional database hardware is added to increase storage capacity, then the database can handle more customer information, but the cost and logistical difficulty increase
Solution Approach 1:
The patent divides the database into multiple partitions that can be independently managed and replicated. Each partition holds a subset of customer information, allowing the system to scale by creating new partitions rather than adding monolithic hardware. This segmentation enables parallel processing and distributes the load across multiple smaller units.
Solution Approach 2:
The patent implements data replication where partitions are copied to multiple locations. When a new partition is needed, existing partitions are replicated to create the new one, allowing the system to increase capacity through copying rather than requiring complex hardware addition and data migration processes.
2Quantity of substance
If data migration is performed to add new database hardware, then storage capacity increases, but data access is locked and network services are disturbed
Solution Approach 1:
The patent creates new partitions and replicates data to them before actually migrating the data. This preliminary creation of destination partitions allows the system to prepare the infrastructure in advance, then perform migration during maintenance windows with minimal disruption to ongoing network services.
Solution Approach 2:
The patent implements dynamic partition management where data can be redirected between partitions based on load and availability. The system can dynamically route data access requests to different partitions, enabling continuous operation during migration by switching traffic between active and inactive partitions.
Data Source
AI summary
A scalable data store framework is provided to migrate data across partitions in a data store without compromising data availability for customers. The scalable data store framework includes one or more partitions configured to manage a set of “entities” for a network-based service. When a new-partition event is detected, a target partition is identified for bifurcation and various additional partitions are added to the data store. After the target partition is replicated into the additional partitions, some entities associated with the target partition are migrated to the additional partitions in order to maintain good load balance in the data store.