Distributed Machine Learning via Asynchronous Algebraic Model Sharing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current machine learning systems, particularly neural networks, face challenges in scaling up due to the need for frequent and synchronous communication among numerous processing units, which is costly and difficult to engineer, and struggle to incorporate formal knowledge and adapt to complex tasks without clear rules or large datasets.

Innovation Solution

A distributed machine learning method using discrete algebraic models with idempotent operators, where each computing device calculates an algebraic model independently and shares indecomposable components asynchronously, allowing for cooperative learning without synchronization and reducing communication bandwidth constraints.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If neural networks are scaled up as a single large system, then learning capability is improved, but communication cost and system complexity increase significantly

Engineering Contradiction:

Improvelearning capabilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent divides the machine learning system into multiple independent computing devices, each capable of autonomously calculating algebraic models. This segmentation allows the system to scale without requiring proportional increases in communication infrastructure, as each device operates independently while contributing to the overall learning capability through shared indecomposable components.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces indecomposable components as intermediaries that enable cooperation between independent computing devices. These components serve as a standardized interface through which devices can share knowledge and coordinate without requiring complex direct communication protocols, thus reducing system complexity while maintaining learning capability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If frequent synchronous communication is implemented among processing units, then learning performance is improved, but communication cost and engineering difficulty increase

Engineering Contradiction:

Improvelearning performanceVSAvoidengineering difficulty

Core Design Contradiction:

ProductivityVSEase of manufacture

Solution Approach 1:

The patent implements periodic action by allowing computing devices to share indecomposable components at irregular intervals rather than requiring synchronous communication at fixed frequencies. This approach maintains learning performance by ensuring knowledge is shared when beneficial, while avoiding the engineering complexity of coordinating frequent synchronous interactions among all processing units.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

Each computing device independently determines when to share its indecomposable components with others, eliminating the need for centralized coordination. This self-service mechanism allows devices to autonomously optimize their contribution timing based on their own learning state, improving learning performance without requiring complex communication scheduling infrastructure.

Inventive Principle:

Principle #25Self-service

3Speed

If high-performance communication busses are used, then data transfer speed is improved, but system cost increases

Engineering Contradiction:

Improvedata transfer speedVSAvoidsystem cost

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent extracts the critical data transfer requirements from the communication infrastructure by identifying and sharing only indecomposable components between devices. This extraction allows the system to achieve efficient knowledge transfer using standard communication channels rather than requiring expensive high-performance communication busses, as the shared components represent the essential distilled knowledge rather than raw data streams.

Inventive Principle:

Principle #2Taking out (Extraction)

4Loss of time

If processing units are placed closer together, then communication latency is reduced, but hardware cost and density requirements increase

Engineering Contradiction:

Improvecommunication latencyVSAvoidhardware cost

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The patent applies preliminary action by pre-processing and distilling knowledge into indecomposable components before sharing. This preliminary preparation allows the components to be transmitted and utilized efficiently over standard communication distances without requiring ultra-low latency connections, as the condensed format enables faster processing and reduces the critical impact of transmission delays.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11521133B2Method for large-scale distributed machine learning using formal knowledge and training data

Publication Date: 2022.12.06 MARTIN MAROTO FERNANDO

AI summary

A method for large-scale distributed machine learning using input data comprising formal knowledge and/or training data. The method consisting of independently calculating discrete algebraic models of the input data in one or many computing devices, and in sharing indecomposable components of the algebraic models among the computing devices without constraints on when or on how many times the sharing needs to happen. The method uses an asynchronous communication among machines or computing threads, each working in the same or related learning tasks. Each computing device improves its algebraic model every time it receives new input data or the sharing from other computing devices, thereby providing a solution to the scaling-up problem of machine learning systems.