Client-Side Sharding Without Exclusive Locks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database shard systems, the use of exclusive locks for data management leads to inefficiencies and performance degradation, particularly during rebalancing events, as it restricts client operations and introduces significant overhead, and can result in cascading failures due to lock retention by non-functional processes.
Innovation Solution
Implementing a system that allows multiple clients to perform operations asynchronously without exclusive locks, enabling rebalancing events to occur automatically without locks, and maintaining data consistency through versioning and tombstone attributes, with programmatic code on clients rather than shard servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exclusive locks are used to prevent concurrent client operations on the same data item, then data consistency is maintained, but system efficiency and performance degrade due to lock overhead and bottlenecks
Solution Approach 1:
The patent extracts the locking mechanism from the client-side operations and replaces it with a versioning system. Instead of using exclusive locks to prevent concurrent access, the system allows multiple clients to access data items simultaneously and uses version numbers to track changes. This removes the performance bottleneck caused by lock acquisition and release while maintaining data consistency through version comparison.
Solution Approach 2:
The patent replaces the mechanical locking system with a software-based versioning mechanism. Rather than physically blocking access through locks, the system uses version metadata and conflict detection algorithms to manage concurrent access. This substitution eliminates the inherent overhead of lock management while preserving data integrity.
2Reliability
If exclusive locks are held during rebalancing operations to prevent data item movement conflicts, then data consistency is maintained, but client operations are restricted and system performance degrades
Solution Approach 1:
The patent implements preliminary version tagging before rebalancing operations. Data items are marked with version information and rebalancing metadata in advance, allowing the system to track and manage movements without requiring exclusive locks. Clients can continue operations by checking version consistency, eliminating the need to wait for rebalancing to complete.
Solution Approach 2:
The patent introduces version metadata and tombstone attributes as intermediaries between data items and clients during rebalancing. These intermediaries carry information about data item states and movements, allowing clients to make informed decisions without direct lock management. The intermediary layer decouples the rebalancing process from client operations, enabling both to proceed concurrently.
3Device complexity
If non-functional processes retain exclusive locks until timer expiration, then lock management is simplified, but system performance degrades due to cascading failures and forced waiting
Solution Approach 1:
The patent extracts the lock retention problem entirely by eliminating the locking mechanism. Instead of managing lock release timing and handling stuck locks, the system uses version-based conflict detection. Non-functional processes cannot hold locks, so there is no risk of cascading failures from retained locks. The versioning system naturally handles conflicts without requiring forced waiting or complex lock management.
Solution Approach 2:
The patent implements a self-service conflict detection mechanism where clients independently verify data item versions before and after operations. Rather than relying on centralized lock management that can fail, each client autonomously checks version consistency and handles conflicts. This distributed approach eliminates single points of failure and removes the need for timer-based lock release mechanisms.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Multiple clients can be enabled to perform operations relative to data items in a shard system asynchronously to each other without the use by those clients of exclusive locks. A rebalancing event, in which data items are redistributed automatically among a set of shards due to a modification of the quantity of shards in the system, can be performed without the use of exclusive locks by clients. Clients can continue to perform operations relative to at least some of the data items in the shard system even while rebalancing processes are redistributing at least some of the data items asynchronously during a system-wide rebalancing event. All of these benefits can be obtained without sacrificing data consistency within the shard system.