Scalable Partitioning Framework for Data Service Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network purchasing services face challenges with database availability due to hard failures (blackouts) and high CPU loads (brownouts), and struggle with scalability as user demand increases, making it difficult to manage customer data efficiently.
Innovation Solution
A partitioning framework is implemented to divide customer data into multiple partitions using partitionable keys, allowing for flexible data redistribution and migration, reducing direct dependencies between partitions and buckets, thereby enhancing system availability and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a single database is used to manage customer data, then data management is simple and straightforward, but the system becomes a single point of failure and cannot handle high CPU loads or failures without complete downtime
Solution Approach 1:
The patent divides the single database into multiple independent partitions, each storing a subset of customer data. This segmentation allows the system to maintain simplicity in managing individual partitions while improving overall reliability through distribution. When one partition fails or experiences high CPU load, other partitions remain operational, eliminating the single point of failure problem.
2Reliability
If a backup database is maintained to solve blackout or brownout problems, then system availability improves, but the backup database itself becomes vulnerable to the same CPU load issues and may also be taken down
Solution Approach 1:
Instead of maintaining a separate backup database, the patent segments customer data across multiple partitions. This distribution inherently provides redundancy and fault tolerance without requiring a complete backup system. The segmentation approach reduces complexity by eliminating the need for separate backup infrastructure while maintaining availability through distributed data storage.
3Productivity
If new database hardware is added to increase capacity, then the system can handle increased user demand, but data redistribution and migration are required which disturbs customer services
Solution Approach 1:
The patent implements dynamic partitioning where data can be redistributed across partitions without service interruption. New partitions can be added to the system, and data migration occurs in a controlled manner that maintains service continuity. This dynamic approach allows capacity expansion while keeping customer services undisturbed through incremental data redistribution.
4Reliability
If data is distributed across multiple partitions, then system availability and scalability improve, but direct dependencies between partitions and partitionable keys create complexity in data migration
Solution Approach 1:
The patent introduces buckets as an intermediary layer between partitions and partitionable keys. Instead of direct mapping, buckets group partitionable keys and associate them with partitions. This intermediary structure simplifies data migration because entire buckets can be moved between partitions as units, reducing the complexity of managing individual key-partition dependencies while maintaining system availability.
Data Source
AI summary
A method and system is provided for maintaining customer data in a data store system utilizing a scalable partitioning framework. More specifically, the data store of a customer service system is partitioned (divided) into multiple partitions by a partitionable key of customer data so that each partition owns a subset of the customer data. By maintaining several partitions that store a subset of the customer data exclusive to each other, blackout or brownout problems can be local to one partition and, thus, the availability for the entire system will be increased. Moreover, a set of partitionable keys, a minimal unit to be moved between partitions, is grouped and associated with a partition. By eliminating direct dependencies between the partitions and the partitionable keys, the system can have great flexibility with regard to migrating customer data between partitions and adding a new partition.