Distributed Learning Master-Slave Architecture for Asynchronous Sub-Problem Solving

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Multi-machine distributed learning systems face challenges in efficiently solving complex machine learning problems, such as non-convex optimization and Bayesian learning, due to limitations in network communication speed and error tolerance, particularly when dealing with malfunctioning nodes in master-slave architectures.

Innovation Solution

The system employs a master-slave architecture with parallel computing, allowing slave computers to work independently and asynchronously on sub-problems without mutex locks, utilizing stochastic gradient descent and quantum processors for faster and more precise solutions, and incorporating a submitter to manage information access and error tolerance by storing updates on a master computer.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If master-slave architecture with parallel computing is used, then productivity is improved, but device complexity increases

Engineering Contradiction:

Improvespeed of machine learning processesVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The machine learning problem is divided into multiple sub-problems that are distributed across different slave computers. Each slave computer independently handles a portion of the computational workload, enabling parallel processing and significantly improving the overall speed of solving machine learning problems while maintaining manageable complexity through modular decomposition

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The master computer acts as an intermediary that coordinates communication between slave computers, manages parameter distribution, and aggregates results. This intermediary structure enables complex parallel computations to be orchestrated systematically, improving productivity while containing device complexity through centralized coordination

Inventive Principle:

Principle #24Intermediary (Mediator)

2Manufacturing precision

If quantum processors are used, then manufacturing precision is improved, but device complexity increases

Engineering Contradiction:

Improveprecision of solutionsVSAvoidquantum processor integration

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

Quantum processors are selectively deployed only for specific sub-problems that benefit from quantum computational advantages, such as certain optimization tasks or linear algebra operations. Classical computers handle other portions of the workflow, creating a hybrid architecture that achieves high precision where needed while avoiding the complexity of full quantum system integration

Inventive Principle:

Principle #3Local quality

3Productivity

If asynchronous independent computation is used, then productivity is improved, but reliability decreases

Engineering Contradiction:

Improvecomputational speedVSAvoiderror tolerance

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

Slave computers continuously report their computational status and results to the master computer, which monitors progress and coordinates parameter updates. This feedback mechanism ensures that asynchronous computations remain synchronized with the overall problem-solving process, maintaining reliability while preserving the productivity benefits of parallel independent computation

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system is designed with error detection and correction capabilities built in advance. Checksum verification, redundant computations, and rollback mechanisms are implemented to cushion against potential computational errors, ensuring that asynchronous parallel processing maintains high reliability even when individual slave computers encounter issues

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

4Productivity

If network communication is reduced, then productivity is improved, but loss of information increases

Engineering Contradiction:

Improvespeed of machine learning processesVSAvoidparameter synchronization

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

Parameter information is pre-distributed to slave computers at the beginning of each computation phase, and slave computers perform multiple local computation iterations using this pre-received information before results are aggregated. This preliminary action reduces the frequency of network communication rounds while ensuring that sufficient information is available locally to complete substantial computational work, thereby improving productivity without significant information loss

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11861466B1Multi-machine distributed learning systems

Publication Date: 2024.01.02 GOOGLE LLC

AI summary

A system comprises a network of computers comprising a master computer and slave computers. For a machine learning problem that is partitioned into a number of correlated sub-problems, each master computer is configured to store tasks associated with the machine learning problem, and each of the slave computers is assigned one of the correlated sub-problems. Each slave computer is configured to store variables or parameters or both associated with the assigned one of the correlated sub-problems; obtain information about one or more tasks stored by the master computer without causing conflict with other slave computers with regard to the information; perform computations to update the obtained information and the variables or parameters or both of the assigned sub-problem; send the updated information to the master computer to update the information stored at the master computer; and store the updated variables or parameters or both of the assigned sub-problem.