Data Repartitioning for Server Federation Load Balancing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stateful data management systems face challenges in achieving high scale and high availability while minimizing capital and operational costs, as existing methods for distributing data across multiple servers are resource-intensive and economically infeasible, especially when dealing with frequent data changes and high transaction rates.

Innovation Solution

The solution involves extending a federation of servers by adding a new server, moving backup data structures, creating new data structures, and balancing the data load by suspending writes, copying logical partitions, redirecting reads and writes, and deleting original partitions, or using temporary replication streams to ensure data consistency and availability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the distributed copy method is used in stateful data systems, then high availability is improved, but system performance deteriorates due to resource consumption from full propagation of changes across all copies

Engineering Contradiction:

Improvehigh availabilityVSAvoidsystem performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent divides data into logical partitions that can be independently distributed across different physical partitions. This allows selective replication of only necessary data portions rather than replicating entire data sets, reducing the resource consumption associated with full propagation while maintaining availability through distributed logical partitions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts data distribution by creating physical partitions and assigning logical partitions to them based on workload requirements. This dynamic reconfiguration allows the system to optimize between availability and performance by adjusting which data is replicated and where, rather than using static full-copy replication.

Inventive Principle:

Principle #15Dynamics

2Reliability

If numerous identical copies of all data are maintained in various locations, then high availability is improved, but capital and operational costs increase

Engineering Contradiction:

Improvehigh availabilityVSAvoiddata storage resources

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

Instead of maintaining numerous identical copies of all data, the patent segments data into logical partitions and selectively replicates only those partitions that are needed for high availability. This reduces the quantity of data stored across multiple locations while still providing backup and availability where required.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements different replication strategies for different logical partitions based on their importance and access patterns. Critical partitions are replicated more extensively while less critical partitions have minimal or no replication, optimizing the balance between availability and storage resource consumption.

Inventive Principle:

Principle #3Local quality

3Stability of the object's composition

If changes are fully propagated across all copies before further processing, then data consistency is improved, but system throughput deteriorates

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem throughput

Core Design Contradiction:

Stability of the object's compositionVSProductivity

Solution Approach 1:

The patent segments data into logical partitions that can be processed independently. This allows the system to maintain consistency within each partition without requiring synchronization across all partitions, enabling parallel processing and improving throughput while maintaining local data consistency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system can prepare and buffer changes locally at each physical partition before full propagation is complete. This preliminary action allows processing to continue without waiting for complete propagation, maintaining throughput while ensuring eventual consistency through mechanisms like replication streams or transaction logs.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS7921424B2Systems and methods for the repartitioning of data

Publication Date: 2011.04.05 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Various embodiments of the present invention are directed toward extending a federation of servers and balancing the data load of said federation of servers, in order to improve accessibility and scope of the system by moving the first backup data structure on the second server to the new server, creating a second data structure on the new server, and creating a second backup data structure for the second data structure on the second server. To balance the load, certain embodiments employ a method comprising suspending writes to the selected logical partitions, copying selected logical partitions to the new physical partition, redirecting reads and writes to the logical partitions in the new physical partition, and removing the original selected logical partitions. Alternative embodiments for balancing data load comprise the substitute steps of creating a temporary replication stream for the selected logical partitions to the new physical partition, copying selected logical partitions to the new physical partition, deconflicting inconsistencies in the logical partitions on the new physical partition, redirecting reads and writes for the selected logical partitions to the logical partitions in the new physical partition, and removing the original selected logical partitions that were copied to the new physical partition.