Flat minima searching-based federated learning device and method

Abstract

The present invention relates to a flat minima searching-based federated learning device and method by which flat minima are searched for between a global model and local models so as to converge to flat minima in the global model, and by which the characteristics of the plurality of local models are reflected such that the accuracy of the global model can be increased. The flat minima searching-based federated learning device of the present invention comprises: a server for training the global model for federated learning; and a client for training the local models for federated learning.

Description

Minimum Uniform Searching-based Federated Learning Device and Method

[0001] The present invention relates to a federated learning apparatus and method based on flat minima searching, and more specifically, to searching for flat minima between a global model and a local model. Furthermore, the present invention relates to a federated learning apparatus and method based on flat minima searching capable of performing flat minima searching on a global model.

[0002] Federated Learning (FL) is receiving significant attention as a core framework that enables decentralized learning across a vast number of distributed clients while preserving data privacy.

[0003] The core of Federated Learning (FL) is to store local data on the client side and communicate gradients or model parameters between the server and the client, while direct access to the server's local data is prohibited.

[0004] Nevertheless, as diversity or heterogeneity in data distribution among clients remains an unresolved challenge that hinders the successful aggregation of global model parameters, leading to performance degradation and inhibiting model convergence, efforts to overcome this have continued.

[0005] For example, Korean Patent Publication No. 10-2024-0119732 discloses a federated learning and apparatus that receives weights from multiple clients, uses a dequantizer to change the intrinsic precision corresponding to each of the weights to a reference precision, determines masks corresponding to each of the weights based on the intrinsic precision, merges the weights changed to the reference precision based on the masks to determine an integrated weight, quantizes the weight to the intrinsic precision, and transmits it to multiple clients.

[0006] However, even in this case, there is a disadvantage in that the results of flat minima searching converge heterogeneously between the local and global.

[0007] The objective of the present invention is to provide a federated learning apparatus and method based on minimum uniformity searching that can converge to the minimum uniformity in the global model by searching for the minimum uniformity between the global model and the local model.

[0008] Another objective of the present invention is to provide a federated learning device and method based on minimum uniform searching that can improve the accuracy of a global model by reflecting the characteristics of multiple local models.

[0009] A federated learning device based on minimum uniform searching according to the present invention may include a server that learns a global model of federated learning and a client that learns a local model of federated learning.

[0010] Here, the server may include a global disturbance model calculation unit that calculates a global disturbance model based on a global model.

[0011] Additionally, the server may include a local model training request transmission unit that transmits a global model and a global disturbance model to a client to request local model training.

[0012] Here, the server may include a local model training result receiving unit that receives local model training results from a client.

[0013] In addition, the server may include a global parameter update unit that updates global parameters based on the local model training results.

[0014] Here, the client may include a local model training request receiving unit that receives a global model and a global disturbance model from the server and requests the training of a local model.

[0015] Additionally, the client may include a local model initializer that initializes the local model based on the global model.

[0016] Here, the client may include a local parameter update unit that updates local parameters.

[0017] Additionally, the client may include a model difference calculation unit that calculates the difference between the global model and the local model.

[0018] Here, the client may include a local model training result transmission unit that transmits the training results of the local model to the server.

[0019] In addition, the local parameter update unit may include a local model gradient calculation module that derives the gradient of the local model.

[0020] Here, the local parameter update unit may include a local disturbance model calculation module that derives a local disturbance model based on the local model.

[0021] In addition, the local parameter update unit may include an interpolation model generation module that generates an interpolated model by interpolating the global disturbance model and the local disturbance model.

[0022] Here, the local parameter update unit may include a local model update module that updates the local model based on the local model and the interpolation model.

[0023] Additionally, the global parameter update unit may include an inter-model difference averaging module that performs the averaging of the differences between the global model and multiple local models.

[0024] Here, the global parameter update unit may include an inter-model distance averaging module that performs the average of the distances between the global model and local models based on the difference between the global model and multiple local models.

[0025] In addition, the global parameter update unit may include a global model update module that updates the global model based on the average of the differences between the global model and multiple local models.

[0026] Here, the global parameter update unit may include an inter-model distance average binarization module that binarizes the average of the distances between the global model and the local model.

[0027] Additionally, the global parameter update unit may include an inter-model distance window average module that performs a window average on the result binarized by the inter-model distance average binarization module.

[0028] A federated learning method based on minimum uniform searching according to another embodiment of the present invention may include a server learning step for learning a global model of federated learning on a server and a client learning step for learning a local model of federated learning on a client.

[0029] Here, the server learning phase may include a global disturbance model calculation phase in which the global disturbance model calculation unit calculates the global disturbance model based on the global model.

[0030] In addition, the server learning phase may include a local model learning request transmission phase in which the local model learning request transmission unit transmits the global model and the global disturbance model to the client to request local model learning.

[0031] Here, the server training phase may include a local model training result receiving phase in which the local model training result receiving unit receives the local model training result from the client.

[0032] In addition, the server training phase may include a global parameter update phase in which global parameters are updated based on the local model training results in the global parameter update section.

[0033] Here, the client learning stage may include a local model learning request receiving stage in which a global model and a global disturbance model are received from a server, and a local model learning request receiving unit receives a request for the learning of a local model.

[0034] In addition, the client training phase may include a local model initialization step in which the local model initialization unit initializes the local model based on the global model.

[0035] Here, the client learning phase may include a local parameter update step in which local parameters are updated in the local parameter update section.

[0036] In addition, the client learning phase may include a model difference calculation step that calculates the difference between the global model and the local model in the model difference calculation unit.

[0037] Here, the client learning phase may include a local model learning result transmission phase in which the local model learning result transmission unit transmits the local model learning result to the server.

[0038] In addition, the local parameter update step may include a local model gradient calculation step that derives the gradient of the local model from the local model gradient calculation module.

[0039] Here, the local parameter update step may include a local disturbance model calculation step in which a local disturbance model is derived based on the local model in the local disturbance model calculation module.

[0040] Additionally, the local parameter update step may include an interpolation model generation step in which an interpolation model generation module generates an interpolation model through the interpolation of the global disturbance model and the local disturbance model.

[0041] Here, the local parameter update step may include a local model update step in which the local model update module updates the local model based on the local model and the interpolation model.

[0042] Additionally, the global parameter update step may include a model difference averaging step in which the model difference averaging module performs the averaging of the differences between the global model and multiple local models.

[0043] Here, the global parameter update step may include an inter-model distance averaging step that performs the average of the distances between the global model and local models based on the difference between the global model and multiple local models in the inter-model distance averaging module.

[0044] In addition, the global parameter update step may include a global model update step in which the global model is updated based on the average of the differences between the global model and multiple local models in the global model update module.

[0045] Here, the global parameter update step may include a model distance average binarization step in which the model distance average binarization module binarizes the average of the distances between the global model and the local model.

[0046] Additionally, the global parameter update step may include a model distance window average step that performs a window average on the result binarized by the model distance average binarization module in the model distance window average module.

[0047] The flat minima searching-based federated learning device and method according to the present invention has the advantage of being able to converge to the flat minima in the global model by searching for the flat minima between the global model and the local model.

[0048] In addition, the minimum uniform searching-based federated learning device and method according to the present invention has the advantage of being able to improve the accuracy of a global model by reflecting the characteristics of multiple local models.

[0049] FIG. 1 is a schematic diagram showing a federated learning device based on flat minima searching according to one embodiment of the present invention.

[0050] Figure 2 is a schematic diagram showing the server of Figure 1 in detail.

[0051] Figure 3 is a schematic diagram showing the client of Figure 1 in detail.

[0052] Figure 4 is a schematic diagram showing the local parameter update section of Figure 3 in detail.

[0053] Figure 5 is a schematic diagram showing the global parameter update section of Figure 2 in detail.

[0054] Figure 6 is a diagram showing the operation of the client of Figure 1.

[0055] FIG. 7 is a flowchart illustrating a federated learning method based on minimum uniform searching according to an embodiment of the present invention.

[0056] Figure 8 is a flowchart showing the server learning step of Figure 7 in detail.

[0057] Figure 9 is a flowchart showing the client learning steps of Figure 7 in detail.

[0058] Figure 10 is a flowchart showing the local parameter update step of Figure 9 in detail.

[0059] Figure 11 is a flowchart showing the global parameter update step of Figure 8 in detail.

[0060] Hereinafter, specific embodiments for carrying out the present invention will be described with reference to the attached drawings.

[0061] In describing the present invention, terms such as first, second, etc. may be used to describe various components, but the components may not be limited by the terms. The terms are intended solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0062] When it is described that a component is connected to or coupled with another component, it may be directly connected to or coupled with that other component, but it can also be understood that there may be other components in between.

[0063] The terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions may include plural expressions unless the context clearly indicates otherwise.

[0064] In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0065] In addition, the shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

[0066] Hereinafter, a minimum uniform searching-based federated learning device and method according to the present invention will be described in detail with reference to the attached drawings.

[0067] FIG. 1 is a schematic diagram showing a flat minima searching-based federated learning device according to one embodiment of the present invention, and FIGS. 2 to 6 are detailed schematic diagrams and drawings for explaining FIG. 1 in detail.

[0068] Hereinafter, a federated learning device based on minimum uniform searching according to an embodiment of the present invention will be described with reference to FIGS. 1 to 6.

[0069] First, referring to FIG. 1, a minimum uniform searching-based federated learning device according to one embodiment of the present invention comprises a server (100) that learns a global model of federated learning and a client (200) that learns a local model of federated learning.

[0070] Here, the server (100) transmits a global learning model to the client (200) to learn the global model of federated learning, and the client (200) initializes and updates a local model based on the global learning model and retransmits the difference to the server (100), and the server (100) performs an update of the global model based on this.

[0071] In the prior art, the server (100) exhibits a flatness discrepancy that indicates heterogeneity and loss error among multiple clients (200). In this case, strong heterogeneity causes serious reconsideration of the flatness minimum search in federated learning discrepancies, which ultimately leads to a degradation of the performance of the global model.

[0072] The minimum uniform searching-based federated learning device according to the present invention has the advantage of mitigating flatness inconsistency to lower the minimum flatness value of the global model, and the operation of the server (100) is explained in FIGS. 2 and 5, the operation of the client (200) is explained in FIGS. 3 and 4, and FIG. 6 is explained comprehensively by example.

[0073]

[0074] FIG. 2 is a schematic diagram showing the server (100) of FIG. 1 in detail.

[0075] As can be seen in FIG. 2, the server (100) is composed of a global disturbance model calculation unit (110) that calculates a global disturbance model based on a global model, a local model learning request transmission unit (120) that transmits the global model and the global disturbance model to a client (200) to request local model learning, a local model learning result reception unit (130) that receives local model learning results from the client (200), and a global parameter update unit (140) that updates global parameters based on the local model learning results.

[0076] For federated learning, a server (100) requests local learning from a local model learning request transmission unit (120) for multiple clients (200), receives the results from the local model learning result reception unit (130), and reflects the received results in the global parameter update unit (140), which can be defined as one round.

[0077] At this time, the global disturbance model calculation unit (110) specifies the global model and the global disturbance model as identical in the initial round, but from the subsequent rounds, it can calculate the global disturbance model based on the global model as in Equation 1 by reflecting the results from the global parameter update unit (140).

[0078]

[0079] Here, is the global perturbed model in the r-th round,

[0080] is the global model at the r-th round,

[0081] is the average error between the global model and the local model in the r-th round.

[0082] Here, the global disturbance model is a value shifted by a certain amount in the update direction of the difference between the global model and the local model (the direction in which loss increases), regardless of the magnitude of the difference between the global model and the local model.

[0083] Afterwards, the local model learning request transmission unit (120) transmits the global model and the global disturbance model to the client (200), and the results learned by the client (200) are received by the local model learning result receiving unit (130), and the local model learning result received by the local model learning result receiving unit (130) is the difference between the global model and the local model.

[0084] The global parameter update unit (140) updates global parameters based on the difference between the global model and the local model, as explained in detail in FIG. 5.

[0085]

[0086] FIG. 3 is a schematic diagram showing the client (200) of FIG. 1 in detail.

[0087] As can be seen in FIG. 3, the client (200) is composed of a local model learning request receiving unit (210) that receives a global model and a global disturbance model from the server (100) and requests the learning of a local model, a local model initialization unit (220) that initializes a local model based on the global model, a local parameter update unit (230) that updates local parameters, a model difference calculation unit (240) that calculates the difference between the global model and the local model, and a local model learning result transmission unit (250) that transmits the learning result of the local model to the server (100).

[0088] For federated learning, the client (200) receives the global model and the global disturbance model from the server (100) at the local model learning request receiving unit (210) and initializes the local model based on the global model at the local model initialization unit (220), at which time the local model can be specified to be the same as the global model.

[0089] Meanwhile, the client (200) can update local parameters based on local data in the local parameter update unit (230), as described in detail in FIG. 4.

[0090] At this time, the difference between models calculation unit (240) calculates the difference between the global model and the local model based on the local model updated in the local parameter update unit (230), and the result can be transmitted to the server (100) from the local model learning result transmission unit (250).

[0091]

[0092] FIG. 4 is a schematic diagram showing in detail the local parameter update section (230) of FIG. 3.

[0093] As can be seen in FIG. 4, the local parameter update unit (230) consists of a local model gradient calculation module (231) that calculates the gradient of the local model, a local disturbance model calculation module (232) that calculates the local disturbance model based on the local model, an interpolation model generation module (233) that generates an interpolated model by interpolating the global disturbance model and the local disturbance model, and a local model update module (234) that updates the local model based on the local model and the interpolation model.

[0094] Here, the local model gradient calculation module (231) can calculate the gradient of the local model using the local model and local data as shown in Equation 2.

[0095]

[0096] Here, is the gradient of the local model in the r-th round, the i-th client, and the k-th training,

[0097] is the local model in the r-th round, the i-th client, and the k-th training,

[0098] is the i-th client, and the local model in the k-th training,

[0099] is a model and input data Slope for.

[0100] At this time, the local disturbance model calculation module (232) can calculate the local disturbance model as in Equation 3, and the local disturbance model is a value that has moved by a certain amount in the update direction of the local model's gradient (the direction in which the loss increases) regardless of the magnitude of the local model's gradient.

[0101]

[0102] Here, is the local perturbed model at the r-th round, i-th client, and k-th training.

[0103] Meanwhile, the interpolation model generation module (233) generates an interpolated model by interpolating the global disturbance model and the local disturbance model as shown in Equation 4. At this time, if the difference between the global model and the local model is large, the proportion of the global disturbance model reflected can be increased, and if the difference between the global model and the local model is small, the proportion of the local disturbance model reflected can be increased.

[0104]

[0105] Here, is an interpolated model for the r-th round, i-th client, k-th training, and constant c,

[0106] c is a constant from 0 to 1.

[0107] Afterwards, the local model update module (234) can calculate the gradient based on local data for the interpolation model generated by the interpolation model generation module (233) and update the local model according to the gradient as in Equation 5.

[0108]

[0109] Here, is the i-th client, k-th training local data,

[0110] is a model and input data Slope for.

[0111] Therefore, the federated learning device based on minimum uniformity searching according to the present invention has the advantage of finding the minimum uniformity of the global model by reducing heterogeneity between the global model and the local model, by increasing the reflection weight of the global disturbance model when the difference between the global model and the local model is large and increasing the reflection weight of the local disturbance model when the difference between the global model and the local model is small, and performing the process until the difference between the global model and the local model is small.

[0112]

[0113] FIG. 5 is a schematic diagram showing in detail the global parameter update section (140) of FIG. 2.

[0114] As can be seen in FIG. 5, the global parameter update unit (140) is composed of a model difference average module (141) that performs an average of the differences between a global model and a number of local models, a model distance average module (142) that performs an average of the distance between a global model and a number of local models based on the differences between a global model and a number of local models, a global model update module (143) that updates a global model based on the average of the differences between a global model and a number of local models, a model distance average binarization module (144) that binarizes the average of the distance between a global model and a number of local models, and a model distance window average module (145) that performs a window average of the result binarized by the model distance average binarization module (144).

[0115] Here, the difference between models average module (141) can average the difference between the global model and the multiple local models received from multiple clients (200) as in Equation 6, which can be seen as increasing the loss of the client (200).

[0116]

[0117] Here, It is defined as,

[0118] The difference between the global model and multiple local models for the r-th round and the i-th client.

[0119] Additionally, the model-to-model distance average module (142) can calculate the average of the distances between the global model and the local model based on the difference between the global model and the local model received from multiple clients (200), as shown in Equation 7.

[0120]

[0121] Afterwards, the global model update module (143) can update the global model based on the average of the distances between the global model and the local model as in Equation 8.

[0122]

[0123] Meanwhile, in the inter-model distance average binarization module (144), the average of the distance between the global model and the local model, which is the output of the inter-model distance average module (142), is defined as 1 for values ​​greater than or equal to a certain value and 0 for values ​​less than or equal to a certain value as in Equation 9, and the inter-model distance window average module (145) can calculate the window average as in Equation 10.

[0124]

[0125] Here, I is a value of 1 if the internal condition of [] is satisfied, and 0 if it is not.

[0126]

[0127] Here, W is the window value.

[0128] Here, the window is for a certain range of the previous round, and can be represented as 1 if the difference between the global model and the local model is large, and as 0 if the difference between the global model and the local model is small, and can be reflected in the interpolation model generation module (233) of FIG. 4.

[0129] Therefore, the federated learning device based on minimum uniformity searching according to the present invention has the advantage of finding the minimum uniformity of the global model by reducing heterogeneity between the global model and the local model, by increasing the reflection weight of the global disturbance model when the difference between the global model and the local model is large and increasing the reflection weight of the local disturbance model when the difference between the global model and the local model is small, and performing the process until the difference between the global model and the local model is small.

[0130] FIG. 6 is a diagram illustrating the operation of the client (200) of FIG. 1. As can be seen in FIG. 6, FIG. 6 is a diagram illustrating the operation of the client (200) of FIG. 1. As can be seen in FIG. 6, the global model and the global disturbance model located in the second model (MD2) are received from the server (100), and the local model can be initialized to the location of the third model (MD3) based on the global model. Subsequently, the local disturbance model calculation module (232) calculates the local disturbance model and places it in the fifth model (MD5), and the local disturbance model calculation module (232) generates an interpolation model to be located in the first model (MD1) through the interpolation of the global disturbance model and the local disturbance model, and the local model update module (234) updates the local model located in the third model (MD3) to the location of the fourth model (MD4) based on the slope of the interpolation model located in the first model (MD1).

[0131] Therefore, the federated learning device based on minimum uniformity searching according to the present invention has the advantage of finding the minimum uniformity of the global model by reducing heterogeneity between the global model and the local model, by increasing the reflection weight of the global disturbance model when the difference between the global model and the local model is large and increasing the reflection weight of the local disturbance model when the difference between the global model and the local model is small, and performing the process until the difference between the global model and the local model is small.

[0132]

[0133] FIG. 7 is a flowchart illustrating a federated learning method based on flat minima searching according to an embodiment of the present invention, and FIGS. 8 to 11 are detailed flowcharts for explaining FIG. 7 in detail.

[0134] Hereinafter, a federated learning method based on minimum uniform searching according to another embodiment of the present invention will be described with reference to FIGS. 7 to 11.

[0135] First, referring to FIG. 7, a federated learning method based on minimum uniform searching according to another embodiment of the present invention comprises a server learning step (S100) for learning a global model of federated learning at a server (100) and a client learning step (S200) for learning a local model of federated learning at a client (200).

[0136] Here, in the server learning step (S100), the server (100) transmits the global learning model to the client (200) to learn the global model of federated learning, and in the client learning step (S200), the local model is initialized and updated based on the global learning model and the difference is retransmitted from the client (200) to the server (100), and in the server learning step (S100), the server (100) performs an update of the global model based on this.

[0137] In the prior art, the server (100) exhibits a flatness discrepancy that indicates heterogeneity and loss error among multiple clients (200). In this case, strong heterogeneity causes serious reconsideration of the flatness minimum search in federated learning discrepancies, which ultimately leads to a degradation of the performance of the global model.

[0138] The federated learning method based on minimum uniform searching according to the present invention has the advantage of lowering the minimum flatness value of the global model by mitigating flatness discrepancies, and the operation of the server learning step (S100) is explained in FIGS. 8 and 11, and the operation of the client learning step (S200) is explained in FIGS. 9 and 10.

[0139]

[0140] FIG. 8 is a flowchart showing the server learning step (S100) of FIG. 7 in detail.

[0141] As can be seen in FIG. 8, the server learning stage (S100) consists of a global disturbance model calculation stage (S110) in which a global disturbance model is calculated based on a global model by a global disturbance model calculation unit (110), a local model learning request transmission stage (S120) in which a local model learning request transmission unit (120) transmits the global model and the global disturbance model to a client (200) to request local model learning, a local model learning result reception stage (S130) in which a local model learning result reception unit (130) receives the local model learning result from the client (200), and a global parameter update stage (S140) in which a global parameter update unit (140) updates the global parameter based on the local model learning result.

[0142] For federated learning, a server (100) requests local learning from a local model learning request transmission unit (120) for multiple clients (200), receives the results from the local model learning result reception unit (130), and reflects the received results in the global parameter update unit (140), which can be defined as one round.

[0143]

[0144] At this time, in the global disturbance model calculation step (S110), the global model and the global disturbance model are designated as identical in the initial round, but from the subsequent rounds, the global disturbance model can be calculated based on the global model as in Equation 1 by reflecting the results from the global parameter update step (S140).

[0145] Here, the global disturbance model is a value shifted by a certain amount in the update direction of the difference between the global model and the local model (the direction in which loss increases), regardless of the magnitude of the difference between the global model and the local model.

[0146] Afterwards, the global model and the global disturbance model are transmitted to the client (200) in the local model learning request transmission step (S120), and the results learned by the client (200) are received in the local model learning result reception step (S130), and the local model learning result received in the local model learning result reception step (S130) is the difference between the global model and the local model.

[0147] In the global parameter update step (S140), global parameters are updated based on the difference between the global model and the local model, as explained in detail in Fig. 11.

[0148]

[0149] FIG. 9 is a flowchart showing the client learning step (S200) of FIG. 7 in detail.

[0150] As can be seen in FIG. 9, the client learning stage (S200) consists of a local model learning request receiving stage (S210) in which a global model and a global disturbance model are received from the server (100) and a local model learning request receiving unit (210) receives a request for learning of the local model, a local model initialization stage (S220) in which the local model is initialized based on the global model, a local parameter update stage (S230) in which local parameters are updated in the local parameter update unit (230), a model difference calculation stage (S240) in which the difference between the global model and the local model is calculated in the model difference calculation unit (240), and a local model learning result transmission stage (S250) in which the local model learning result is transmitted to the server (100) in the local model learning result transmission unit (250).

[0151] For federated learning, in the client learning stage (S200), the local model learning request receiving unit (210) receives the global model and the global disturbance model from the server (100), and the local model initialization unit (220) initializes the local model based on the global model. At this time, the local model can be designated to be the same as the global model.

[0152] Meanwhile, in the client learning stage (S200), local parameters can be updated based on local data in the local parameter update stage (S230), as described in detail in FIG. 10.

[0153] At this time, in the model difference calculation step (S240), the difference between the global model and the local model is calculated based on the local model updated in the local parameter update step (S230), and the result can be transmitted to the server (100) in the local model learning result transmission step (S250).

[0154]

[0155] FIG. 10 is a flowchart showing the local parameter update step (S230) of FIG. 9 in detail.

[0156] As can be seen in FIG. 10, the local parameter update step (S230) consists of a local model gradient calculation step (S231) in which the gradient of the local model is derived in the local model gradient calculation module (231), a local disturbance model calculation step (S232) in which the local disturbance model is derived based on the local model in the local disturbance model calculation module (232), an interpolation model generation step (S233) in which the interpolation model is generated by the interpolation of the global disturbance model and the local disturbance model in the interpolation model generation module (233), and a local model update step (S234) in which the local model is updated based on the local model and the interpolation model in the local model update module (234).

[0157] Here, in the local model gradient calculation step (S231), the gradient of the local model can be calculated using the local model and local data as shown in Equation 2.

[0158] At this time, in the local disturbance model calculation step (S232), the local disturbance model can be calculated as shown in Equation 3, and the local disturbance model is a value in which the local model has moved by a certain amount in the update direction of the local model's gradient (the direction in which the loss increases) regardless of the magnitude of the local model's gradient.

[0159] Meanwhile, in the interpolation model generation step (S233), an interpolation model is generated as shown in Equation 4 by interpolating the global disturbance model and the local disturbance model. At this time, if the difference between the global model and the local model is large, the weight of the global disturbance model can be increased, and if the difference between the global model and the local model is small, the weight of the local disturbance model can be increased.

[0160] Subsequently, in the local model update step (S234), the gradient is calculated based on local data for the interpolation model created in the interpolation model creation step (S233), and the local model can be updated according to the gradient as in Equation 5.

[0161] Therefore, the federated learning method based on minimum uniformity searching according to the present invention has the advantage of finding the minimum uniformity of the global model by reducing heterogeneity between the global model and the local model, by increasing the weight of the global disturbance model when the difference between the global model and the local model is large and increasing the weight of the local disturbance model when the difference between the global model and the local model is small, and performing the process until the difference between the global model and the local model is small.

[0162] Meanwhile, the operation of the server learning step (S100) described in FIGS. 8 and 11, and the operation of the client learning step (S200) described in FIGS. 9 and 10 have been comprehensively explained as an example in FIG. 6, so a description thereof is omitted.

[0163]

[0164] FIG. 11 is a flowchart showing the global parameter update step (S140) of FIG. 8 in detail.

[0165] As can be seen in FIG. 11, the global parameter update step (S140) consists of: a model difference average step (S141) in which the average of the differences between the global model and a number of local models is performed in the model difference average module (141); a model distance average step (S142) in which the average of the distance between the global model and the number of local models is performed in the model distance average module (142) in which the average of the distance between the global model and the number of local models is performed; a global model update step (S143) in which the global model is updated in the global model update module (143) in which the global model is updated in which the average of the differences between the global model and the number of local models is updated in the model distance average binarization module (144) in which the average of the distance between the global model and the local models is binarized in the model distance average binarization module (144); and a model distance window average step (S145) in which the window average of the result binarized in the model distance average binarization module (144) is performed in the model distance window average module (145).

[0166] Here, in the average difference step between models (S141), the difference between the global model and the multiple local models received from multiple clients (200) can be averaged as in Equation 6, which can be seen as increasing the loss of the client (200).

[0167] Additionally, the model distance averaging step (S142) can calculate the average distance between the global model and the local model based on the difference between the global model and the local model received from multiple clients (200), as shown in Equation 7.

[0168] Afterwards, in the global model update step (S143), the global model can be updated as shown in Equation 8 based on the average of the distances between the global model and the local model.

[0169] Meanwhile, in the inter-model distance average binarization step (S144), the average of the distance between the global model and the local model, which is the output of the inter-model distance average step (S142), is defined as 1 for values ​​greater than or equal to a certain value and 0 for values ​​less than or equal to a certain value as in Equation 9, and the inter-model distance window average module (145) can calculate the window average as in Equation 10.

[0170] Here, the window is for a certain range of the previous round, and can be represented as 1 if the difference between the global model and the local model is large, and as 0 if the difference between the global model and the local model is small, and can be reflected in the interpolation model generation step (S233) of Fig. 10.

[0171] The federated learning method based on minimum uniformity searching according to the present invention has the advantage of finding the minimum uniformity of the global model by reducing heterogeneity between the global model and the local model, by increasing the weight of the global disturbance model when the difference between the global model and the local model is large and increasing the weight of the local disturbance model when the difference between the global model and the local model is small, and performing the process until the difference between the global model and the local model is small.

[0172]

[0173] As described above, the federated learning device and method based on minimum uniformity searching according to the present invention have the advantage of being able to converge to the minimum uniformity in the global model by searching for minimum uniformity between the global model and the local model, and have the advantage of being able to improve the accuracy of the global model by reflecting the characteristics of multiple local models.

[0174]

[0175] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the invention.

[0176] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0177] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that, based on design priorities, the specific order or hierarchy of steps in the processes may be rearranged within the scope of the invention. The appended method claims provide various step elements in a sample order, but do not imply limitation to the specific order or hierarchy presented.

[0178] The description of the presented embodiments is provided so that any person skilled in the art may use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Thus, the present invention is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

Claims

1. A server that learns a global model of federated learning; and A minimum uniform searching-based federated learning device comprising a client that learns a local model of the federated learning above.

2. In Paragraph 1, A minimum uniform searching-based federated learning device characterized by including a global disturbance model calculation unit that calculates a global disturbance model based on the global model in the above server.

3. In Paragraph 2, A minimum uniform searching-based federated learning device characterized by including: a server that transmits the global model and the global disturbance model to the client to request local model learning; and a local model learning request transmission unit.

4. In Paragraph 3, A minimum uniform searching-based federated learning device characterized by including a local model learning result receiving unit that receives local model learning results from the client, wherein the server above includes:

5. In Paragraph 4, A minimum uniform searching-based federated learning device characterized by including a global parameter update unit that updates global parameters based on the local model training results of the server.

6. In Paragraph 5, A minimum uniform searching-based federated learning device characterized by including: a local model learning request receiving unit in which the client receives the global model and the global disturbance model from the server and receives a request for the learning of the local model.

7. In Paragraph 6, A minimum uniform searching-based federated learning device characterized by including a local model initialization unit that initializes the local model based on the global model in the above client.

8. In Paragraph 7, A minimum uniform searching-based federated learning device characterized by including a local parameter update unit that updates local parameters in the above client.

9. In Paragraph 8, A minimum uniform searching-based federated learning device characterized by including a model difference calculation unit that calculates the difference between the global model and the local model in the above client.

10. In Paragraph 9, A minimum uniform searching-based federated learning device characterized by including a local model learning result transmission unit that transmits the learning result of the local model to the server, wherein the above client is a local model learning result transmission unit.

11. In Paragraph 8, A minimum uniform searching-based federated learning device characterized by including a local parameter update unit that includes a local model gradient calculation module for deriving the gradient of the local model.

12. In Paragraph 11, A minimum uniform searching-based federated learning device characterized by including a local parameter update unit that includes a local disturbance model calculation module that derives a local disturbance model based on the local model.

13. In Paragraph 12, A minimum uniform searching-based federated learning device characterized by including: an interpolation model generation module that generates an interpolated model by interpolating the global disturbance model and the local disturbance model, wherein the local parameter update unit described above comprises the above-mentioned global disturbance model and the local disturbance model.

14. In Paragraph 13, A minimum uniform searching-based federated learning device characterized by including a local parameter update unit that updates the local model based on the local model and the interpolation model.

15. In Paragraph 10, A minimum uniform searching-based federated learning device characterized by including a global parameter update unit that performs an average of the differences between the global model and a plurality of local models.

16. In Paragraph 15, A minimum uniform searching-based federated learning device characterized by including a global parameter update unit that performs an average of the distances between the global model and the local models based on the difference between the global model and a plurality of local models.

17. In Paragraph 16, A minimum uniform searching-based federated learning device characterized by including a global parameter update unit that updates the global model based on the average of the differences between the global model and a plurality of local models.

18. In Paragraph 17, A minimum uniform searching-based federated learning device characterized by including a global parameter update unit comprising a model-to-model distance average binarization module that binarizes the average of the distances between the global model and the local model.

19. In Paragraph 18, A minimum uniform searching-based federated learning device characterized by including: a global parameter update unit comprising a model distance window average module that performs a window average on the result binarized by the model distance average binarization module.

20. A server training step for training a global model of federated learning on a server; and A minimum uniform searching-based federated learning method comprising: a client learning step for learning a local model of the federated learning at the client.

21. In Paragraph 20, A federated learning method based on minimum uniform searching, characterized by including a global disturbance model calculation step in which a global disturbance model is calculated based on the global model in a global disturbance model calculation unit during the server learning step.

22. In Paragraph 21, A minimum uniform searching-based federated learning method characterized by including: a local model learning request transmission step in which, in the server learning step, the local model learning request transmission unit transmits the global model and the global disturbance model to the client to request local model learning.

23. In Paragraph 22, A minimum uniform searching-based federated learning method characterized by including, in the server learning step above, a local model learning result receiving step in which a local model learning result receiving unit receives a local model learning result from the client.

24. In Paragraph 23, A minimum uniform searching-based federated learning method characterized by including, in the server learning step above, a global parameter update step in which global parameters are updated based on the local model learning results in a global parameter update unit.

25. In Paragraph 24, A federated learning method based on minimum uniform searching, characterized by including: a local model learning request receiving step in which the client learning step receives the global model and the global disturbance model from the server and receives a request for learning the local model from a local model learning request receiving unit.

26. In Paragraph 25, A minimum uniform searching-based federated learning method characterized by including, in the above client learning step, a local model initialization step in which the local model is initialized based on the global model in the local model initialization unit.

27. In Paragraph 26, A minimum uniform searching-based federated learning method characterized by including a local parameter update step in which local parameters are updated in a local parameter update unit during the above client learning step.

28. In Paragraph 27, A minimum uniform searching-based federated learning method characterized by including, in the above client learning step, a model difference calculation step in which the difference between the global model and the local model is calculated in a model difference calculation unit.

29. In Paragraph 28, A minimum uniform searching-based federated learning method characterized by including, in the above client learning step, a local model learning result transmission step in which a local model learning result transmission unit transmits the learning result of the local model to the server.

30. In Paragraph 29, A federated learning method based on minimum uniform searching, characterized by including a local model gradient calculation step in which the gradient of the local model is derived from a local model gradient calculation module in the local parameter update step.

31. In Paragraph 30, A federated learning method based on minimum uniform searching, characterized by including a local disturbance model calculation step in which a local disturbance model is derived based on the local model in a local disturbance model calculation module during the local parameter update step.

32. In Paragraph 31, A minimum uniform searching-based federated learning method characterized by including, in the above-mentioned local parameter update step, an interpolation model generation step in which an interpolated model is generated by the interpolation of the global disturbance model and the local disturbance model in the interpolation model generation module.

33. In Paragraph 32, A minimum uniform searching-based federated learning method characterized by including a local model update step in which the local model is updated based on the local model and the interpolation model in the local parameter update step.

34. In Paragraph 29, A federated learning method based on minimum uniform searching, characterized by including, in the global parameter update step, a model difference averaging step in which the model difference averaging module performs the averaging of the differences between the global model and a plurality of local models.

35. In Paragraph 34, A minimum uniform searching-based federated learning method characterized by including, in the global parameter update step, a model distance averaging step in which the average of the distances between the global model and the local models is performed based on the difference between the global model and a plurality of local models in the model distance averaging module.

36. In Paragraph 35, A federated learning method based on minimum uniform searching, characterized by including a global model update step in which the global model is updated based on the average of the differences between the global model and a plurality of local models in the global parameter update step.

37. In Paragraph 36, A federated learning method based on minimum uniform searching, characterized in that the global parameter update step includes a model distance average binarization step in which the average of the distances between the global model and the local model is binarized in a model distance average binarization module.

38. In Paragraph 37, A federated learning method based on minimum uniform searching, characterized by including, in the global parameter update step above, a model distance window averaging step in which a model distance window averaging module performs a window averaging on the result binarized in the model distance average binarization module above.

Images