Lock-Free 2-Phase Commit for Distributed Key Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in managing encryption keys due to the risk of split brain scenarios, where multiple controllers may attempt to update or retire keys simultaneously, leading to conflicts over which keys are current or obsolete.
Innovation Solution
A method involving atomic operations to set and commit version identifiers for encryption keys across nodes, ensuring that only when no changes are detected in the new value of the version identifier are the shards of the key encryption key written to node-specific memory, thereby coordinating key updates and rotations while tolerating failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple controllers are used in a distributed storage system to improve reliability and tolerance to failures, then the system can continue operating during failures, but split brain problems occur where multiple controllers attempt to update or retire keys simultaneously, leading to conflicts
Solution Approach 1:
The system performs preliminary actions by establishing a total ordering of key management operations across all controllers before execution. Each controller is assigned a unique identifier and operations are ordered based on these identifiers, ensuring that key updates and retirements occur in a predetermined sequence that prevents conflicts. This preliminary ordering is implemented through the version identifier mechanism that all controllers agree upon before performing key operations.
Solution Approach 2:
The patent introduces version identifiers as an intermediary mechanism that mediates between multiple controllers attempting to manage encryption keys. The version identifier acts as a mediator that all controllers must agree upon before performing key operations, ensuring that only one controller can successfully update or retire a key at any given time. This intermediary prevents split brain scenarios by providing a centralized coordination point without requiring actual centralized control.
2Object-generated harmful factors
If traditional locking mechanisms are used to coordinate key management operations, then conflicts can be prevented, but the system cannot tolerate controller failures and requires centralized control
Solution Approach 1:
The system implements self-service by enabling each controller to independently determine the correct order of key management operations using the version identifier mechanism. Controllers do not need to acquire locks or communicate extensively with a central authority; instead, they autonomously check the current version identifier, compare it with their intended operation's version, and proceed if they match. This self-service approach eliminates the need for traditional locking mechanisms while maintaining conflict prevention.
Solution Approach 2:
The patent replaces the mechanical locking system with a software-based version identifier comparison mechanism. Instead of using physical or software locks that require centralized coordination and can fail when controllers go down, the system uses immutable version identifiers that are compared through simple equality checks. This substitution eliminates the single point of failure inherent in centralized locking while maintaining the ability to prevent conflicts through logical ordering.
3Reliability
If key rotation is performed frequently to improve security, then security posture is enhanced, but the complexity of coordinating key updates across distributed controllers increases
Solution Approach 1:
The patent segments the key management coordination problem into independent, versioned operations. Each key rotation or update operation is assigned a unique version identifier, allowing the system to handle multiple key rotations simultaneously across different controllers without coordination complexity. The segmentation of key versions into discrete, ordered units simplifies the coordination of frequent key rotations by reducing the problem to simple version number comparisons rather than complex inter-dependent operations.
Data Source
AI summary
A method of encryption key management in a storage system having a plurality of nodes and more than one key manager, performed by the storage system, is provided. The method includes setting, in a first atomic operation to a distributed store of the plurality of nodes, a version identifier to a new value, and writing shards of a key encryption key, to node-specific memory of the plurality of nodes. The method includes committing the shards of the key encryption key by updating, in a second atomic operation, a set of version identifiers in the distributed store including a current version identifier, responsive to finding no change to the new value of the version identifier.