Distributed Machine Learning Model Training in Low Power Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Low Power and Lossy Networks (LLNs), such as IoT networks, face challenges in routing, Quality of Service (QoS), security, network management, and traffic engineering due to their complex nature, where traditional approaches are inefficient and machine learning algorithms have not been effectively utilized due to the high number of parameters and resource constraints.
Innovation Solution
A distributed approach for training a machine learning model, specifically an Artificial Neural Network (ANN), is implemented across multiple devices in the network, where each device receives and generates new model parameters using local data, and shares them with others to create a globally trained model, leveraging Stochastic Gradient Descent for optimization and reducing computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional machine learning algorithms are used to train models in LLNs, then model training can be performed, but the high number of parameters and resource constraints make it inefficient and impractical
Solution Approach 1:
The patent segments the model training process into distributed micro-batches that are processed across multiple devices in the network. Each device handles a portion of the training data and parameters, breaking down the complex global training task into manageable local tasks that fit within resource constraints of LLN devices.
Solution Approach 2:
The patent introduces a new dimension of distributed spatial processing across the network topology. Instead of concentrating all computation in one location, training is distributed across multiple nodes in the network, utilizing the network's spatial structure to parallelize the training process and reduce individual device burden.
2Measurement precision
If centralized model training is performed using all network data, then model accuracy improves, but network bandwidth consumption and processing time increase significantly
Solution Approach 1:
The patent performs preliminary local training at each device before global aggregation. Each device pre-processes its local data and computes local model updates in advance, so that when results are aggregated globally, the overall training converges faster with fewer rounds of communication, reducing total training time while maintaining accuracy.
Solution Approach 2:
The patent enables continuous model training by overlapping local training computations with communication phases. Devices continuously perform local gradient computations while simultaneously exchanging and aggregating results with other devices, maximizing resource utilization and reducing idle time throughout the training process.
3Adaptability or versatility
If more training data is collected from the network, then model generalizability improves, but network overhead and energy consumption increase
Solution Approach 1:
The patent applies local quality by allowing each device to contribute training data and compute local model updates based on its specific local conditions and data characteristics. This ensures the global model adapts to local variations across the network, improving generalizability without requiring uniform data collection from all nodes, thus reducing overall energy consumption.
Solution Approach 2:
The patent uses partial action by selecting a representative subset of training data and devices for each training round rather than processing all available data. This partial processing achieves sufficient model generalizability while significantly reducing the energy and bandwidth overhead associated with processing the complete dataset at every iteration.
Data Source
AI summary
In one embodiment, a device determines that a machine learning model is to be trained by a plurality of devices in a network. A set of training devices are identified from among the plurality of devices to train the model, with each of the training devices having a local set of training data. An instruction is then sent to each of the training devices that is configured to cause a training device to receive model parameters from a first training device in the set, use the parameters with at least a portion of the local set of training data to generate new model parameters, and forward the new model parameters to a second training device in the set. Model parameters from the training devices are also received that have been trained using a global set of training data that includes the local sets of training data on the training devices.