Key-Value Neural Network Architecture for Scalable Distributed Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed neural networks face scalability limitations due to bottlenecks in data distribution within datacenters, limited network bandwidth, resource availability, and static deployment requirements, leading to inefficiencies in resource utilization and computation.
Innovation Solution
A distributed machine learning architecture that combines model and data parallelism through key-value stores (KVS), utilizing a serverless cloud native compute framework with microfunctions to enable computation colocation with data, allowing dynamic resource allocation and automatic distribution across servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data parallelism is used in distributed neural networks, then ease of implementation and fault tolerance are improved, but scalability for large models deteriorates
Solution Approach 1:
The patent combines model parallelism and data parallelism into a unified distributed architecture. Model parallelism divides the neural network model across different servers, while data parallelism distributes training data across servers. This hybrid approach resolves the contradiction by enabling both ease of implementation (through data parallelism) and scalability for large models (through model parallelism), allowing the system to handle both small and large-scale neural networks effectively.
2Adaptability or versatility
If model parallelism is used in distributed neural networks, then scalability for large models is improved, but ease of implementation and fault tolerance deteriorate
Solution Approach 1:
The patent merges model parallelism and data parallelism to create a flexible distributed architecture. Model parallelism is applied to divide large models across servers for scalability, while data parallelism is used to simplify implementation and improve fault tolerance. This combination allows the system to achieve scalability for large models without sacrificing ease of implementation, as the architecture can automatically adapt to different model sizes and complexity levels.
3Power
If neural network training is performed on a single node with GPU, then computing power is improved, but training time deteriorates
Solution Approach 1:
The patent segments the neural network training process across multiple servers in a distributed architecture. Each server handles a portion of the training workload, either through model parallelism (different parts of the model) or data parallelism (different batches of data). This segmentation distributes the computational burden, maintaining high computing power through GPU utilization on each node while reducing overall training time through parallel processing across the distributed system.
4Quantity of substance
If distributed KVS cluster is used for data storage, then data storage capacity and access speed are improved, but system complexity deteriorates
Solution Approach 1:
The patent implements a universal key-value store architecture that serves multiple functions within the distributed neural network system. The KVS cluster provides data storage, parameter management, and coordination services across all servers. This multi-functional design increases data storage capacity and access speed while managing system complexity through a standardized, unified interface that simplifies data access patterns and reduces the need for separate specialized systems.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and system for implementing a neural node in a neural network in a key value store (KVS) system. The method and system monitor a first KVS key of the neural node for an update of an input value. The method and system execute a microfunction for the neural node on the input value to generate an output value, in response to detecting a change in the input value and write the output value to a second KVS key for an output neural node.