Vertical federated learning method and vertical federated learning server

Abstract

The purpose of the present invention is to make it possible for data associated with IDs for which there is no data at some clients to be used for vertical federated learning. In this vertical federated learning system, all clients use records for common IDs to calculate bottom model outputs, and a server finds the correlations between the bottom model outputs. Clients that have data for partially common IDs use records for the partially common IDs to find bottom model outputs (calculated activations), and the server estimates bottom model outputs (estimated activations) for clients that do not have data for the partially common IDs from the calculated activations and the correlations. The server uses the calculated activations and the estimated activations to train a top model.

Description

Vertical association learning method and vertical association learning server

[0001] The disclosure technology relates to vertical association learning.

[0002] [Federated Learning] Federated learning is a technique that allows for the generation of machine learning models that reflect the characteristics of each dataset by sharing only the information necessary for training machine learning models, without sharing the datasets (collections of data) managed by each organization (i.e., while protecting the confidentiality / privacy of the data).

[0003] [Vertical Associative Learning] Figure 1 is a diagram of the configuration of an associative learning system of the form called "vertically associative learning" as described in Non-Patent Document 1. In the associative learning system 1, a server 101 and multiple clients 102-1 to 102-N cooperate, for example, via a communication network 103. The neural network learned by each client is called the bottom model, and the neural network learned by the server is called the top model. Multiple bottom models and one top model together form a single neural network. The bottom models have an input layer, and the top models have an output layer.

[0004] Each client shall have a data table as shown in Figure 2(a). Each record in each data table shall consist of a data ID and one or more features. The features may differ for each client. The ID may be generated using a feature (such as a phone number) that can uniquely identify the record from the features held by each client. The operation of the federated learning system 1 will be explained below.

[0005] <STEP 1> The server and clients identify the data IDs that are common to all clients and determine the order of the data IDs to be used for training. In Figure 2(a), ID=(4,5) is the data ID common to all clients.

[0006] <STEP 2> Each client inputs the data into the bottom model in the order of the determined data IDs and performs forward propagation (FP) calculations on the bottom model.

[0007] <STEP3> Each client sends the result (activation value) obtained by FP using the bottom model to the server.

[0008] <STEP4> The server aggregates the activation values received from each client and performs FP on the top model.

[0009] <STEP5> The server performs error backpropagation calculation (Backward Propagation: BP) using the top model output and the correct data (for example, when the model is used for predicting the sales of a certain product, the actual sales data). Hereinafter, the correct data is referred to as "label".

[0010] <STEP6> The server divides the result (derivative value of the error function) obtained by BP using the top model and sends it to each client. Hereinafter, the derivative value of the error function is referred to as "gradient". Also, the above-mentioned "division" means the operation of extracting the gradient required for BP of the bottom model from the result obtained by BP using the top model.

[0011] <STEP VII> Each client performs BP on the bottom model using the received gradient.

[0012] The client and the server repeat STEP2 to STEP7 until a predetermined end condition is satisfied, and finally obtain the top model and the bottom model.

[0013] Kang Wei et al., "Vertical Federated Learning: Challenges, Methodologies and Experiments", https: / / arxiv.org / abs / 2202.04309, 2022.

[0014] As mentioned above, in vertical associative learning, before training, the clients participating in the associative learning coordinate with each other to see if they have data with the same ID. As a result, as seen in Figure 2(a), only data with IDs common to all guest devices can be used for training, and the entire dataset is not effectively utilized. For example, as shown in Figure 2(b), if we interpolate ID=(6,7,8) using the median or mean of the data with ID=(1,2,3,4,5) for the first client, and interpolate ID=(1,2,3) using the median or mean of the data with ID=(4,5,6,7,8) for the Nth client, then the data with ID=(1,2,3) for the first client and the data with ID=(6,7,8) for the Nth client can also be used for training. However, with such interpolation, it is not possible to capture the correlations between data distributed among each client for each ID, making it difficult to improve the accuracy of the neural network.

[0015] The vertically federated learning system that solves the above problem comprises a server and multiple clients. The neural network to be trained is divided into multiple bottom models including input layers and one top model including an output layer. Multiple clients, each holding a data table consisting of IDs and feature records, train their respective bottom models, while the server, which holds the correct data (labels), trains the top model. The IDs consist of IDs for which all clients have data (common IDs) and IDs for which only some clients have data (partially common IDs). All clients calculate the bottom model output using the records of the common IDs, and the server finds the correlation between the bottom model outputs. Clients that hold data for some of the common IDs find the bottom model output (computational activation value) using the records of the partially common IDs, and the server estimates the bottom model output of clients that do not hold data for some of the common IDs from the computational activation value and correlation, and finds the estimated activation value. The server trains the top model using the computational activation value and the estimated activation value.

[0016] This allows for the use of data associated with IDs that some clients do not have data for, enabling effective utilization of the diverse range of data that clients possess.

[0017] A diagram illustrating a vertically federated learning system according to the prior art. A diagram illustrating an example of client data according to the prior art. A functional block diagram of the vertically federated learning system according to the first embodiment. A diagram illustrating the data held by the server and client according to the first embodiment. A flowchart showing the overall operation of the vertically federated learning system according to the first embodiment. Detailed flowcharts of steps S503 and S505. Detailed flowchart of step S504. Detailed flowchart of step S506. A functional block diagram of the vertically federated learning system according to the second embodiment. A flowchart showing the overall operation of the vertically federated learning system according to the second embodiment. Detailed flowchart of step S1004. A diagram showing an example of the functional configuration of a computer.

[0018] The embodiments of the disclosed technology will be described in detail below. Components with the same function will be numbered identically, and redundant explanations will be omitted.

[0019] [First Embodiment] In the first embodiment, the case where there are two clients will be described as an example. Figure 3 is a functional block diagram showing an example of the configuration of a vertically federated learning system according to the first embodiment. In the vertically federated learning system 3, the server 300 and clients 310 and 320 are connected via a communication network 330.

[0020] Server 300 includes an ID linking unit 301, a top model learning unit 302, a correlation calculation unit 303, an estimation unit 304, and a recording unit 305. First client 310 includes an ID linking unit 311, a bottom model learning unit 312, and a recording unit 313. Second client 320 includes an ID linking unit 321, a bottom model learning unit 322, and a recording unit 323.

[0021] The first client, the second client, and the server are assumed to have the data tables shown in Figure 4. The first client holds data for ID=(1,2,3,4,5) but not for ID=(6,7,8). The second client holds data for ID=(4,5,6,7,8) but not for ID=(1,2,3). The server holds labels (ground truth data) for ID=(4,5,6,7,8) but not for ID=(1,2,3).

[0022] Figure 5 is a flowchart showing the overall operation of the vertically associated learning system 3. Figure 6 is a detailed flowchart of steps S503 and S505 in Figure 5. Figure 7 is a detailed flowchart of step S504 in Figure 5. Figure 8 is a detailed flowchart of step S506 in Figure 5. The following explanation will use Figures 3 to 8.

[0023] The ID linking units 301, 311, and 321 of the server 300, the first client 310, and the second client 320 determine the order of the data IDs to be used for learning (step S501). In the following explanation, it is assumed that the IDs are learned in the order of (4, 5, 6, 7, 8, 1, 2, 3).

[0024] <Learning Based on Common ID> The vertical federated learning system 3 first performs learning using the record of the ID (common ID) for which all clients have data. In this case, ID = (4,5) is the common ID. The bottom model learned by the first client 310 will be called BM1, the bottom model learned by the second client 320 will be called BM2, and the top model learned by the server 300 will be called TM. It is assumed that predetermined initial parameters are set for BM1, BM2, and TM. The first client 310, the second client 320, and the server 300 perform FP and BP with the data of ID = (4,5) and update BM1, BM2, and TM (step S502).

[0025] <Calculation of Correlation Relationship> The vertical federated learning system 3 calculates the correlation relationship between the activation values of BM1 and BM2 (step S503). This will be described using FIG. 6. The first client 310 performs FP of BM1 updated with the data of ID=(4,5) (step S601). The first client 310 transmits the output of BM1 to the server 300 (step S602). In the first client, the activation value obtained from the data of ID=4 is A1 4 , and the activation value A1 obtained from the data of ID=5 5 . The second client 320 performs FP of BM2 updated with the data of ID=(4,5) (step S603). The second client 320 transmits the output of BM2 to the server 300 (step S604). In the second client, the activation value obtained from the data of ID=4 is A2 4 , and the activation value A2 obtained from the data of ID=5 5 .

[0026] The server 300 uses (A1 4 , A1 5 ) and (A2 4 , A2 5 ) to calculate the correlation relationship S 12 between the activation values of BM1 and BM2 (step S605). For the calculation of the correlation relationship, for example, Scaled Dot-Product Attention (Reference 1: A. Vaswani et ai., "Attention Is All You Need", arXiv:1706.03762, 2017) can be used.

[0027] <Learning Part 1 Based on Some Common IDs> Next, the vertical federated learning system 3 performs learning using the records of IDs (some common IDs) that only some clients have data for (step S504). First, the learning using ID=(6,7,8) will be described using FIG. 7. ID=(6,7,8) is an ID that only the second client holds data for. The bottom model learning unit 322 of the second client performs FP of BM2 with the data of ID=k (k = 6,7,8) and obtains the activation value A2 k . The second client 320 has the activation value A2k Send this to server 300 (step S702).

[0028] The estimation unit 304 of server 300 determines the activity value A2 k and correlation S 12 Using this, the activity value A2 k The output activity value of BM1 corresponding to this is estimated (step S703). The estimated value is a1 k The activity value can be estimated using a known method, for example, Scaled Dot-Product Attention (Reference 1). The top model learning unit 302 of the server 300 is (a1 k A2 k Perform TM's FP using (step S704).

[0029] The top model learning unit 302 checks whether the data table of the recording unit 305 holds label data for ID=k (step S705). In this case, the recording unit 305 holds label data for ID=(6,7,8) (Yes in step S705), so the top model learning unit 302 performs BP on the TM using the recorded labels (step S707). The top model learning unit 302 divides the gradient obtained from the TM BP and sends it to the second client (step S708). The top model learning unit 302 also updates the TM using the obtained gradient (step S709).

[0030] The second client 320 receives the gradient and performs a gradient branching (BP) on BM2 in the bottom model learning unit 322 (step S710). The bottom model learning unit 322 also updates BM2 using the gradient obtained as a result of the BP (step S711). The meanings of "gradient" and "gradient branching" are as explained in [Vertical Associative Learning].

[0031] The second client 320 and server 300 perform steps S701 to S711 above for k=6, 7, and 8. Note that the estimated value a1 kUsing mixup (Reference 2: H. Zhang et al., "mixup: Beyond Empirical Risk Minimization", arXiv: 1710.09412, 2018) to train a top model is expected to improve training stability.

[0032] <Correlation Update> Return to Figure 5. Since BM2 has been updated due to learning of some common IDs (6,7,8), the vertical association learning system 3 updates the correlation (step S505). The detailed flow of step S505 is the same as in step S503.

[0033] <Learning based on a partially common ID, part 2> The vertically associated learning system 3 then performs learning using ID = (1,2,3) (step S506). This will be explained using Figure 8. ID = (1,2,3) is an ID whose data is held only by the first client. The bottom model learning unit 312 of the first client performs FP of BM1 with the data of ID = k (k=1,2,3) and the activation value A1 k Obtain (step S801). The first client 310 has an activation value A1 k Send this to server 300 (step S802).

[0034] The estimation unit 304 of server 300 determines the activity value A1 k And the correlation S updated in step S505 12 Using the activity value A1 k The output activity value of BM2 corresponding to this is estimated (step S803). The estimated value is a2 k The top model learning unit 302 of server 300 is (A1 k a2 k Perform TM FP using (step S804). The output of TM is C k Let's assume that.

[0035] The top model learning unit 302 checks whether the data table of the recording unit 305 holds label data for ID=k (step S805). In this case, the recording unit 305 does not hold label data for ID=(1,2,3) (No in step S805), so the top model learning unit 302 checks the TM output C k The data is recorded in the recording unit 305 as a pseudo-label with ID=k (step S806). The top model learning unit 302 performs BP on the TM using the recorded label (step S807). The top model learning unit 302 divides the gradient obtained from the TM's BP and transmits it to the first client (step S808). Note that C k If this is set as the pseudo-label, the error between the ground truth data and the TM output is zero, so all gradients obtained from the BP result are also zero. Therefore, in this case, even if the TM update process (step S809) is performed, the TM will not change (will not be updated).

[0036] The first client 310 receives the gradient (zero gradient), and the bottom model learning unit 312 performs a BP on BM1 (step S810). In this case, the gradient obtained from the BP on BM1 is also zero, so even if the BM1 update process (step S811) is performed, BM1 does not change (is not updated).

[0037] The first client 310 and the server 300 perform steps S801 to S811 above for k=1,2,3.

[0038] The vertically combined learning system 9 then determines whether the learning termination conditions are met. If they are not met (No in step S510), it returns to step S502 and performs learning again from the common ID. Note that in the retraining, the results of FP by TM for some common IDs = (1,2,3) will generally not match the pseudo-labels mentioned above. If the pseudo-labels and TM output do not match, TM and BM will be updated. If the learning termination conditions are met (Yes in step S510), the vertically combined learning system 9 terminates learning.

[0039] The above describes the first embodiment. As mentioned above, neither TM nor BM1 are updated immediately after setting the pseudo-label, so if step S806 is performed, steps S807 to S811 may be skipped. Also, although the above <updating the correlation> was performed after ID=(6,7,8), the correlation may be updated each time a record with some common IDs is processed (each time the bottom model is used). Furthermore, in the above description, BM and TM were updated by performing FP and BP of BM and TM once for each ID, but for each ID, FP and BP of BM and TM may be repeated until predetermined conditions (number of FP / BP operations or magnitude of error in the TM output) are satisfied. Alternatively, the processing of common IDs (FP and BP) may be performed until predetermined convergence conditions are met, and the processing of some common IDs (FP and BP) may be performed a predetermined number of times.

[0040] [Second Embodiment] In the first embodiment, the case where the number of clients is 2 was described as an example. In the second embodiment, the case where the number of clients is N (where N is a natural number greater than or equal to 2) will be described. Figure 9 shows the vertically federated learning system 9 according to the second embodiment. The vertically federated learning system 9 further includes the third to the Nth client as shown in Figure 3, and the configuration of the server and each client is the same as in the first embodiment. Figure 10 is a flowchart explaining the operation of the vertically federated learning system 9 according to the second embodiment. Hereinafter, the first to the Nth clients will be described, and clients C1 to client C N I will explain it by referring to it as such.

[0041] Let {Call} be the set of identifiers for all clients, {comID} be the set of common IDs, n be the number of some common IDs, {Cp} be the set of identifiers for clients that have data for some common IDs, and {Co} be the set of identifiers for clients that do not have data for some common IDs. {Cp}∩{Co} = empty set, and {Cp}∪{Co} = {Call}. Note that there are generally multiple some common IDs. Client C i(i∈{Call}) and server 300 determine the IDs and order to be used for learning (step S1001). Note that learning is performed starting with the common ID.

[0042] <Learning based on a common ID> Client C i (i∈{Call}) and server 300 perform vertical federative learning on the data of the common ID to create the bottom model BM i Then learn the top model™ (step S1002).

[0043] <Calculation of Correlation> Client C i This uses data with a common ID of m (m∈{comID}) to perform BM i Activity value A i m The following is calculated and sent to server 300. Server 300 then processes A i m and A j m Correlation S based on (j∈{Call}∧j≠i) ij Calculate (step S1003).

[0044] <Learning based on a partially common ID> The vertical federative learning system 9 uses data where a partially common ID is n to learn from client C k The bottom model BM for (k∈{Cp}) k Update the TM (Step S1004). Figure 11 shows the detailed flow of Step S1004. Client C k This is BM with data ID=n k Perform FP and the activity value A k n Obtain (step S1101). Client C k The activity value is A k n Send this to server 300 (step S1102).

[0045] Server 300 has an activity value of A k n and correlation S kl Using (l∈{Co}), the activity value A k n BM corresponding to lThe output activity value is estimated (step S1103). The estimated value is a l n Server 300 is A k n (k∈{Cp}) and a l n Perform FP of TM using (l∈{Co}) (step S1104). The output of TM is C n The server 300 checks if the data table of the recording unit 305 holds label data for ID=n. If there is no label for ID=n (No in step S1105), the server 300 checks C n This is set in the data table as a pseudo-label with ID=n (step S1106). If a label with ID=n exists (Yes in step S1105), the server 300 reads the label from the data table and performs TM's BP (step S1107).

[0046] Server 300 divides the gradient obtained from TM's BP and Client C k Send to (k∈{Cp}) (step S1108). Server 300 also updates TM using the obtained gradient (step S1109). Client C k It receives the gradient, BM k Perform the BP (step S1110). Client C k Also, BM k Using the gradient obtained by BP k Update (step S1111).

[0047] Return to Figure 10. The vertical federative learning system 9 determines whether it has completed learning for all partially common IDs. If it has not completed (No in step S1005), it determines whether pseudo-labels were generated in step S1106. If pseudo-labels have not been generated (No in step S1007), the BM has been updated, so it returns to step S1003 to recalculate the correlations and then processes the next partially common ID. If pseudo-labels have been generated (Yes in step S1007), the BM has not been updated, so it returns to step S1004 and processes the next partially common ID using the previously calculated correlations.

[0048] If learning has been completed for all of the partial common IDs (Yes in step S1005), the vertical federative learning system 9 determines whether the learning termination condition has been met. If it has not been met (No in step S1006), it returns to step S1002 and starts learning again from the common IDs. If the termination condition is met (Yes in step S1006), the learning is terminated.

[0049] The above describes the second embodiment. As with the first embodiment, neither TM nor BM are updated immediately after setting the pseudo-label, so if step S1106 is performed, steps S1107 to S1111 may be skipped. In addition, in the description of the second embodiment, the <updating of correlation> was performed each time a record of a partially common ID was processed (each time the bottom model was used), but for example, as in the first embodiment, partially common IDs with common {Cp} may be processed consecutively, and then the correlation may be updated. Also, in the above description, BM and TM were updated by performing FP and BP of BM and TM once for a single ID, but for a single ID, FP and BP of BM and TM may be repeated until predetermined conditions (number of FP / BP or magnitude of error in TM output) are satisfied. Alternatively, the processing of common IDs (FP and BP) may be performed until predetermined convergence conditions are met, and the processing of partially common IDs (FP and BP) may be performed a predetermined number of times.

[0050] [Programs, Recording Media] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), a CPU (a Central Processing Unit), conventional circuits, and / or a combination thereof, programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory.

[0051] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0052] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0053] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 12, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0054] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0055] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0056] A computer executing such a program may, for example, first store the program, either recorded on a portable storage medium or transferred from a server computer, in its own memory. Then, when processing is to be executed, the computer reads the program stored in its memory and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the aforementioned programs include information used for processing by electronic computers that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0057] Furthermore, although the above explanation assumes that the device is configured by executing a predetermined program on a computer, at least a part of these processes may be implemented in hardware.

Claims

1. A vertically federated learning method comprising: dividing a neural network into multiple bottom models including an input layer and one top model including an output layer; multiple clients holding a data table consisting of IDs and feature records each learning the bottom models; and a server holding labels which are the correct data learning the top model, wherein the IDs consist of IDs for which all clients have data (common IDs) and IDs for which only some clients have data (partially common IDs); all clients calculate the output of the bottom models using the records of the common IDs; the server finds the correlation between the bottom model outputs; clients holding data for the partially common IDs find the output of the bottom models (computed activation values) using the records of the partially common IDs; the server finds the bottom model output (estimated activation values) of clients that do not hold data for the partially common IDs from the computed activation values ​​and the correlation; and the server learns the top model using the computed activation values ​​and the estimated activation values.

2. A server comprising a vertically federated learning system in which a neural network is divided into multiple bottom models including an input layer and one top model including an output layer, multiple clients holding a data table consisting of IDs and feature records each learn the bottom models, and a server holding labels which are the correct data learns the top model, wherein the IDs consist of IDs for which all clients have data (common IDs) and IDs for which only some clients have data (partially common IDs), and a correlation calculation unit that obtains bottom model outputs calculated from the records of the common IDs from all clients and calculates the correlation between the bottom model outputs, an estimation unit that obtains bottom model outputs (computed activation values) from clients that hold data for the partially common IDs and calculates the bottom model outputs (estimated activation values) of clients that do not hold data for the partially common IDs from the calculated activation values ​​and the correlation, and a top model learning unit that learns the top model using the calculated activation values ​​and the estimated activation values.

3. A vertical federated learning server according to claim 2, wherein if the server does not retain the label for the partially common ID, the top model learning unit uses the output of the top model based on the calculated activation value and the estimated activation value as the label for the partially common ID.

4. A vertical federative learning server according to claim 2, wherein when at least one of the bottom models is updated, the server obtains the bottom model output calculated from records of common IDs from all the clients and updates the correlation relationship in the correlation calculation unit.

Images