Policy-Based Storage Distribution via Consistent Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems face inefficiencies in managing large numbers of queues across multiple servers, particularly in banking contexts where multiple transactions per account lead to errors and complex manual rebalancing, especially when servers are added or removed.
Innovation Solution
Implementing a policy-based storage structure with consistent hashing to automate the distribution of queues across servers, allowing multiple entries per key and enabling seamless addition or removal of nodes without manual recalculations, thereby achieving efficient and dynamic load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual distribution of queues over servers is used, then data can be stored across multiple servers, but the complexity of managing and rebalancing queues increases significantly when servers are added or removed
Solution Approach 1:
The system uses consistent hashing to enable automatic self-organization of queues across servers. When servers are added or removed, the hashing algorithm automatically redistributes queues without requiring manual intervention, allowing the system to self-adjust and maintain balance dynamically
Solution Approach 2:
The invention changes the distribution parameter from manual assignment to hash-based automatic assignment. By using consistent hashing with a configurable number of virtual nodes, the system dynamically adjusts queue distribution based on server availability, transforming the management approach from static manual control to dynamic automated parameter adjustment
2Productivity
If multiple transactions per account are allowed, then banking operations can be processed in parallel, but errors occur when multiple entries share the same key in traditional databases
Solution Approach 1:
The invention segments the queue management by creating separate queue structures for each key while maintaining them within the same distributed system. Each key gets its own queue entry that can hold multiple transactions, allowing parallel processing while maintaining data integrity through structured separation of transaction streams
Solution Approach 2:
The system adds a dimensional layer to traditional key-value storage by introducing a queue structure as an intermediate layer. Instead of flat key-value pairs, the system organizes data as key-queue-value tuples, where the queue dimension allows multiple transactions to be stacked for the same key without conflict, enabling parallel processing while preserving reliability
3Ease of operation
If hundreds of thousands of queues are manually balanced, then load distribution can be achieved, but the time and effort required for manual scheduling becomes prohibitive
Solution Approach 1:
The invention replaces the mechanical manual process of queue balancing with an automated computational system based on consistent hashing. The hashing algorithm computationally determines queue distribution across servers, eliminating the need for human operators to manually calculate and redistribute queues, thereby saving significant time and effort
Data Source
AI summary
Policy-based storage and retrieval combined with a distribution algorithm results in automatic and even distribution of policy-based storage structures across a set of nodes and dynamic, automated homing or ownership of policy-based storage structures. Large numbers of policy-based storage structures may be distributed without manual administration, allowing for rapid creation and destruction of storage structures. The overall load may be distributed and balanced across the server pool. Multiple entries having the same key value in a database- or table-like structure allow for distribution of policy-based storage and retrieval by key value and for queue semantics to be utilized for microqueues in the large database- or table-like structure.