Distributed Machine Learning for Non-Convex Consensus Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing consensus algorithms in edge computing, such as Distributed ADMM and PDMM, face challenges including low convergence rates, inability to handle non-convex functions, confidentiality issues, and high transmission requirements, especially when dealing with heterogeneous data sets.
Innovation Solution
A modified consensus algorithm that optimizes both primal and dual variables asynchronously, using operator splitting methods like Peaceman-Rachford splitting, to enhance convergence speed and reduce transmission needs while maintaining data confidentiality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional consensus algorithms (Distributed ADMM, PDMM) are used for distributed machine learning, then data confidentiality is maintained through distributed processing, but convergence rate is slow
Solution Approach 1:
The patent implements dynamic variable update mechanisms where primal and dual variables are updated asynchronously rather than in fixed sequences. Node portions can update variables based on local conditions and receive updates from neighbors at different times, creating a dynamic convergence process that accelerates learning while maintaining distributed confidentiality
Solution Approach 2:
The patent performs preliminary actions by pre-calculating gradient information and using momentum terms based on historical update directions. This allows node portions to make more informed updates without requiring synchronous communication with all neighbors, thereby accelerating convergence while reducing communication overhead
2Adaptability or versatility
If conventional consensus algorithms are used, then distributed processing is achieved, but ability to handle non-convex functions is lost
Solution Approach 1:
The patent transforms the optimization approach by changing parameters from direct variable updates to gradient-based updates with momentum. By using gradient descent with momentum terms and adaptive learning rates, the algorithm can navigate non-convex landscapes effectively while maintaining the distributed consensus framework, thus handling non-convex functions with improved reliability
Solution Approach 2:
The patent introduces gradient information and momentum terms as intermediaries between the local loss functions and the consensus variables. These intermediaries allow node portions to exchange meaningful information about the optimization direction without revealing confidential data, enabling handling of non-convex functions while maintaining distributed processing
3Stability of the object's composition
If synchronous updates of primal and dual variables are performed, then convergence stability is maintained, but transmission requirements are high
Solution Approach 1:
The patent segments the update process into independent primal variable updates and dual variable updates that can occur asynchronously. Node portions update primal variables using local gradient information and then update dual variables using segmented consensus steps with neighbors, reducing the need for simultaneous full-duplex communication while maintaining convergence stability through coordinated update rules
Data Source
AI summary
Machine learning techniques which allow machine learning to be performed even when a cost function is not a convex function are provided. A machine learning system includes a plurality of node portions which learn mapping that uses one common primal variable by machine learning based on their respective input data while sending and receiving information to and from each other. The machine learning is performed so as to minimize, instead of a cost function of a non-convex function originally corresponding to the machine learning, a proxy convex function serving as an upper bound on the cost function. The proxy convex function is represented by a formula of a first-order gradient of the cost function with respect to the primal variable or by a formula of a first-order gradient and a formula of a second-order gradient of the cost function with respect to the primal variable.