Worker-Node Storage Connections With Credential-Aware Self-Healing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing iSCSI connections in Kubernetes environments fail to automatically reestablish when authentication credentials are rotated, leading to pod downtime and manual intervention is required, which disrupts storage access for worker nodes.
Innovation Solution
A method for worker nodes to establish TCP-based connections with storage systems, monitor connection status, and upon failure, request new credentials from a controller node to reestablish connections using the new credentials, thereby automating the reconnection process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If CHAP authentication is used to secure Kubernetes clusters with certificate-based authentication, then security and privacy are enhanced, but connection reliability deteriorates when credentials are rotated
Solution Approach 1:
The system performs preliminary actions by detecting connection failures before they propagate to pod downtime. The node server proactively monitors iSCSI session states and identifies failed connections, then automatically initiates reconnection procedures with updated credentials before the failure impacts workload execution.
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring the state of iSCSI sessions and comparing actual connection status against expected operational state. When discrepancies are detected (failed connections), the system feeds this information back into the reconnection logic, which automatically requests new credentials from the controller and reestablishes connections using updated authentication information.
2Productivity
If manual intervention is required to reestablish failed iSCSI connections, then connection security is maintained through credential validation, but productivity deteriorates due to downtime and manual operations
Solution Approach 1:
The system enables self-service by empowering the node server to autonomously detect connection failures, request new credentials from the controller, and reestablish iSCSI sessions without human intervention. The node server independently monitors its own connection state, identifies failures, and executes the complete reconnection workflow including credential renewal and session reestablishment.
Solution Approach 2:
The system applies dynamics by making the reconnection process adaptive and state-driven. Rather than following a static manual procedure, the system dynamically responds to actual connection states, automatically adjusting its behavior based on whether connections are operational or failed. The automation level scales with the severity and persistence of connection issues.
3Loss of time
If iSCSI connections are not automatically reestablished after credential rotation, then credential security is preserved, but loss of time increases due to connection downtime
Solution Approach 1:
The system performs preliminary detection of connection failures and proactively initiates reconnection before the failure impacts workload performance. By monitoring session states continuously and detecting failures early, the system can request new credentials and reestablish connections during brief intervals, minimizing the time connection is lost while maintaining security through validated credential updates.
Data Source
AI summary
The technology disclosed herein enables worker nodes to reestablish connections when the connection protocols fail to successfully reconnect on their own. In a particular example, a method includes establishing a plurality of TCP-based connections (e.g., iSCSI and/or NVMe) between a plurality of worker nodes and a storage system and determining status of the plurality of TCP-based connections. When the status indicates a first connection of the plurality of TCP-based connections has failed, the method includes determining credentials used to establish the first connection are invalid. In response to determining the credentials are invalid, the method includes requesting new credentials from a controller node and reestablishing the first connection using the new credentials.