Node Controller Flighting via Secondary Instance

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In clustered computing systems, updating node controllers can lead to risks of deficiencies and downtime, as new components may not perform as expected in production environments, and existing methods require downtime for decommissioning and activating new controllers.

Innovation Solution

Deploying a secondary node controller within a flight container, which appears as resources to the primary node controller, allowing for delegation of jobs and monitoring of performance without disrupting the production environment, enabling real-time deployment or zero-downtime upgrades.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a new node controller is installed in a production environment, then the system can benefit from new features and improvements, but there is a risk of deficiencies and potential system failure

Engineering Contradiction:

Improvenode controller versionVSAvoidsystem stability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system segments the node controller functionality by running multiple instances (primary and secondary) simultaneously. The primary node controller handles production workloads while the secondary instance runs the new version in parallel, allowing independent testing without affecting system stability. This segmentation enables safe adoption of new versions by isolating experimental instances from critical production operations.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary action by deploying and testing the new node controller version in a secondary instance before full production deployment. The workload is gradually shifted from primary to secondary controller through a multi-phase process, allowing the new version to be validated in a controlled manner before becoming the primary production controller, thus preventing premature deployment of potentially defective versions.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If the node controller is updated, then the system can access new capabilities, but downtime is required for decommissioning and activation

Engineering Contradiction:

Improvenode controller versionVSAvoiddowntime

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system maintains continuity of useful action by running both primary and secondary node controllers simultaneously during the transition period. The secondary controller is pre-warmed and gradually receives workload while the primary controller continues handling production jobs. This overlapping operation eliminates traditional downtime by ensuring continuous workload processing throughout the controller version transition.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system applies dynamics by making the workload distribution between primary and secondary controllers flexible and adjustable. The workload mixing ratio can be dynamically changed during the transition, allowing gradual shift from primary to secondary controller. This dynamic adjustment enables smooth migration without fixed downtime periods, adapting the transition pace to system conditions and performance monitoring results.

Inventive Principle:

Principle #15Dynamics

3Reliability

If extensive testing is performed on the new node controller, then reliability improves, but deployment time increases

Engineering Contradiction:

Improvecontroller performanceVSAvoiddeployment time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system implements periodic action through phased workload migration and iterative performance monitoring. Instead of completing all testing before deployment, the system periodically evaluates the secondary controller's performance and gradually increases its workload share. This periodic assessment allows continuous validation while maintaining production operations, reducing total deployment time by interleaving testing with gradual production integration.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS10523581B2Flighting of node controller in pilot mode in resource management system

Publication Date: 2019.12.31 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Implementing a new node controller in a cluster environment. A method includes, at a primary node controller in the cluster environment, deploying a secondary node controller to be updated, associated with a flight container. The secondary node controller appears logically to the primary node controller as resources available to the primary node controller to which the primary node controller can assign cluster jobs. The primary node controller delegates cluster jobs to the secondary node controller. Sub-containers are started in the flight container for the secondary node controller to start delegated cluster jobs. Performance of the secondary node controller is monitored with respect to the started jobs.