Training system, training method, and program

Abstract

This training system for training a machine learning model includes a database, a search unit, an acquisition unit, and a training unit. The database stores information about machine learning data, at least a portion of which resides outside the training system. The search unit searches the machine learning data for training data that can match evaluation data for the machine learning model, on the basis of the information stored in the database. The acquisition unit acquires the training data searched for by the search unit. The training unit trains the machine learning model using the training data.

Description

Learning System, Learning Method, and Program

[0001] The present disclosure relates to a learning system, a learning method, and a program.

[0002] In a machine learning model, a process of learning with teacher data including learning data and its correct answer data in advance is necessary. If the quality of the teacher data is low, the quality of the machine learning model also deteriorates. On the other hand, it takes a great deal of effort to collect a large amount of good learning data and generate correct answer data. Patent Document 1 discloses a technique for improving the accuracy of a machine learning model while reducing running costs by selectively adding data that is not similar to existing data to learning data and updating the learning model.

[0003] Japanese Patent Application Laid-Open No. 2021-157654

[0004] However, in the prior art, there is still a problem that it is necessary to collect and selectively hold data by oneself, and the storage capacity increases and it takes effort.

[0005] An object of the present disclosure is to provide a learning system, a learning method, and a program capable of performing highly accurate learning on a machine learning model with less labor and storage capacity.

[0006] To achieve the above object, one aspect of the present disclosure is a learning system for learning a machine learning model, including: a database that stores information on machine learning data, at least a part of which exists outside the learning system; a search unit that searches for learning data that can match the evaluation data of the machine learning model among the machine learning data based on the information stored in the database; an acquisition unit that acquires the learning data searched by the search unit; and a learning unit that learns the machine learning model using the learning data.

[0007] According to the present disclosure, there is an effect that highly accurate learning can be performed on a machine learning model with less labor and storage capacity.

[0008] This is a block diagram showing the functional configuration of the learning device of the first embodiment. This is a diagram illustrating the flow of learning a machine learning model. This is a diagram explaining the contents of the database. This is a flowchart showing the procedure for the learning control process. This is a block diagram showing the functional configuration of the learning device of the second embodiment. This is a flowchart showing the control procedure for the learning control process. This is a flowchart showing the control procedure for the learning control process of the third embodiment.

[0009] The embodiments will be described below with reference to the drawings. Figure 1 is a block diagram showing the functional configuration of the learning device 1 of the first embodiment. The learning device 1, which is the learning system of this embodiment, may be a normal information processing device, i.e., a computer. The learning device 1 includes a CPU 11 (Central Processing Unit), RAM 12 (Random Access Memory), storage unit 13, database 14, display unit 15, operation reception unit 16, and communication unit 17.

[0010] The CPU 11 is a hardware processor that performs arithmetic processing and provides overall control over the operation of the learning device 1. The CPU 11 may have a single core or it may have multiple cores. Furthermore, there may be one CPU 11 or multiple CPUs 11. Multiple CPUs 11 may perform parallel processing or each may perform arithmetic processing independently depending on the application.

[0011] RAM 12 provides the CPU 11 with a working memory space and stores temporary data. This temporary data includes evaluation data 121. The evaluation data 121 is used to evaluate the learning progress during the training of the machine learning model 1311. The evaluation data 121 is also used as information for searching for training data to be used in training the machine learning model 1311 and obtaining it from an external device 20.

[0012] The storage unit 13 has non-volatile memory. The non-volatile memory may be, for example, flash memory. The non-volatile memory may include an HDD (Hard Disk Drive). The storage unit 13 stores the program 131 and setting data, etc. The program 131 includes various control programs. The program 131 includes a learning control program for the machine learning model 1311. The machine learning model 1311 is the target for learning, as will be described later.

[0013] Database 14 stores information about machine learning data available for machine learning in an external device 20, which stores such data. Specifically, database 14 stores information about the location and contents of the machine learning data. Database 14 may be various types of non-volatile memory. Non-volatile memory may include an HDD. Database 14 will be described later.

[0014] The display unit 15 displays information to be communicated to the user or others based on the control of the CPU 11. The display unit 15 has a display screen. The display screen may be, for example, a liquid crystal display screen (LCD) or an organic EL (Electro-Luminescent) screen. The display unit 15 may further have LEDs (Light Emitting Diodes) or the like.

[0015] The operation reception unit 16 receives input operations from an external source and outputs an operation signal to the CPU 11 according to the received content. The operation reception unit 16 may, for example, have a touch panel positioned overlapping the display screen. The operation reception unit 16 may also have various switching elements such as push-button switches, slide switches, rocker switches, toggle switches, or a numeric keypad.

[0016] The communication unit 17 controls communication operations performed with external devices 20 and the like. The communication unit 17, for example, is equipped with a network card and acquires necessary data from the outside, in this case machine learning data, via TCP / IP or HTTP connections to the outside. As shown in Figure 1, the external device 20 may include a plurality of external data servers 21 to 23. The location of each of the external data servers 21 to 23 is stored in the database 14 described above.

[0017] The external device 20 stores machine learning data as described above. The machine learning data includes training data, which includes data to be trained and data that shows the correct answers. Such an external device 20 may include publicly available training data websites. In academic fields, such as medicine and biochemistry, the external device 20 may include a database of training data that is only made available to registered users within that field. The external device 20 may be a regular computer or a database device configured for database use.

[0018] Next, the training of the machine learning model 1311 will be described. The learning device 1 trains the machine learning model 1311. The machine learning model 1311 is trained using supervised learning so that it can recognize the desired detection target. The data used to detect the target is, for example, an image, but it may also be video, sound, text, graph waveforms, etc. The algorithm of the machine learning model 1311 may be, for example, deep learning, or may be appropriately selected and set depending on the target. The following description will focus on the case where the target is an image.

[0019] Figure 2 illustrates the flow of training the machine learning model 1311. First, evaluation data for the machine learning model 1311 is acquired (P1). As described above, the evaluation data includes information about the target that the machine learning model 1311 is to detect. For example, a search key is set as second-order ground truth data to match the format information of the evaluation data and the target to be detected.

[0020] In image recognition, ground truth data has classification labels that indicate the identification information of the correct answer, and, if necessary, information indicating the range of the correct answer within the image. Classification labels are also called annotations. The identification information of the correct answer may differ depending on the purpose, even if the target of detection is the same. For example, an image of a Shiba Inu puppy may be labeled as animal, mammal, pet, companion animal, dog, puppy, Shiba Inu, etc. In machine learning data, a single target of detection may have multiple labels.

[0021] Training data exists in multiple formats. If the input data to the machine learning model 1311 is limited to data in a predetermined format, machine learning data in a format different from that predetermined format may be excluded from detection. Alternatively, machine learning data that is in a different format but has the same content may be included in the detection target. In this case, the learning device 1 may have a format conversion application and convert the format of the machine learning data using this format conversion application. That is, the learning device 1 may include data that is not suitable for evaluation data as is, but which can be suitable for evaluation data, as a target for detection from the database 14.

[0022] Furthermore, the information indicating the detection target in the image, i.e., the range of the correct answer, may be the contour shape of the target or a mask along the contour, or it may be an outer frame that includes the target. Alternatively, the information indicating the range may not be included in the correct answer data.

[0023] There are several ways to set the correct data for image recognition. One is to simply assign a label, or simple classification information, to the entire image. In this case, the image may be a close-up of the subject, may have multiple labels, or may contain elements other than the labeled subject. The second is to assign a bounding box to the image that represents the area containing the labeled subject. The frame is usually rectangular, but may be of other shapes, and may enclose a wider area than the subject indicated by the label. The third is to assign a segmentation mask (outline or shape) of the target area corresponding to the label or annotation. In other words, the outline is, in principle, the same shape and size as the subject indicated by the label.

[0024] If the evaluation data includes this format information or labels, these may be used directly to set the search key. If the evaluation data does not contain this information, the content of the search key may be received separately from an external source, such as a user, by the operation reception unit 16, which acts as a reception unit. Alternatively, the content of the search key set by a terminal device (not shown) may be received by input via the communication unit 17, which acts as a reception unit. The operations received by the operation reception unit 16, etc., may include setting operations for contours or outer frames of images in the evaluation data.

[0025] Figure 3 is a diagram illustrating the contents of database 14. Database 14 stores a list of location information for various machine learning data and information on the type and label of images associated with that location information. The location information may be access location information for external data servers 21-23, for example, a URL related to HTTP access, or location information for a shared folder on another computer shared within the LAN. In addition, some of the machine learning data is held by the storage unit 13 of the learning device 1, and the location information may be folder information of the storage unit 13, etc. That is, at least a portion of the location information for the machine learning data is located outside the learning device 1, and the less machine learning data the learning device 1 holds, the smaller the storage capacity required by the learning device 1 becomes.

[0026] As described above, the database 14 may also include image format information, various format information for training data, size, resolution, number of channels, number of layers, and type information for black and white / color images. Image formats include, for example, JPG and PNG. Known training data formats include, for example, YOLO, COCO, PascalVOC, and ImageFolder. Other more common formats such as Json and CSV may also be included. Any format or format that is convertible may be included in a predetermined range that matches the search key. In other words, machine learning data from which desired training data can be obtained by format conversion may be included in a predetermined range that matches the search key. Each piece of machine learning data in the database 14 may include additional text such as README or supplementary information. Such a database 14 is generated before machine learning. The generation of the database 14 may also be immediately before machine learning. This allows training data to be obtained by referring to the latest machine learning data. The external device 20 referenced in the database 14 may be specified in advance. The external device 20 may be changed or added when the database 14 is updated.

[0027] In this way, the database 14 is searched for the location of machine learning data containing the first correct answer data that matches the search key set (P2; search unit). The match with the search key here may simply mean that the correct answer data for each machine learning data contains a word that matches the search key, or it may be an exact match with the correct answer data. If the same machine learning data contains both matching and non-matching words, machine learning data with a mix of matching and non-matching data may be detected for the time being. In other words, the range of matching may be determined by an appropriate predetermined range.

[0028] When the location of the machine learning data where the target training data is stored is searched, data is retrieved from the searched location (P3; retrieval unit). The learning device 1 requests the detected data from the external device 20 corresponding to the location. The external device 20 outputs data to the learning device 1 according to the request. If data with different labels is mixed within the location, the external device 20 may output only the data that matches the requested search key, or it may output all the data for the requested location. If all the data for the requested location has been output from the external device 20, the learning device 1 extracts data that matches the search key from the data retrieved according to the search key and acquires it as training data.

[0029] Furthermore, a single image may contain a mix of labels that match the search key and labels that do not. In this case, labels that do not match the search key and information indicating the range corresponding to those labels may be excluded during data acquisition. This exclusion process may be performed by the external device 20, or it may be performed by the learning device 1 after the data has been acquired by the learning device 1.

[0030] Furthermore, if format conversion is required from the machine learning data of the external device 20, the format of the acquired training data is converted by the format conversion application described above and used as training data. A training dataset is generated by selecting some or all of the multiple training data obtained through or by omitting such conversion operations (P4).

[0031] Image data from the generated training dataset is input to the machine learning model 1311, and the machine learning model 1311 is trained by comparing its output with the ground truth data (P5; training unit). Conventional methods may be used for training, for example, backpropagation may be used. Training does not have to be performed with all the data in the set training dataset. For example, in backpropagation, if the error worsens due to training, or if there is a stagnation in the improvement of the error, the training may be terminated as appropriate. Once training is complete, a trained model is obtained (P6).

[0032] Evaluation data 121 is input to the trained model, and the accuracy of the trained model is evaluated according to the accuracy rate or error tendency (P7). If a predetermined standard accuracy is achieved, the generated trained model is output as a finished product.

[0033] Figure 4 is a flowchart showing the procedure for the learning control process in the learning device 1 of this embodiment. The CPU 11, acting as a reception unit, acquires evaluation data from an external source (S1). The CPU 11 sets a search key from the evaluation data (S2). Alternatively, the CPU 11 may accept input operations for the search key via a direct operation reception unit 16 or the like, separately from acquiring evaluation data, and set the search key.

[0034] The CPU 11, acting as a search unit, searches the database 14 for matching data information based on the search key (S3; search means). Based on the information obtained through the search, the CPU 11 acquires location information that matches the search key (S4). Multiple pieces of location information may be acquired.

[0035] The CPU 11, acting as the acquisition unit, acquires suitable training data from the training data at the location specified by the acquired location information (S5; acquisition means). The CPU 11 only needs to selectively acquire data that matches the search key from the machine learning data at the above location. If the acquired training data contains frames or contours corresponding to multiple correct labels, information on frames or contours that do not match the search key may be deleted. Furthermore, if the number of frames or contours that match the search key is fewer than the number of frames or contours that do not match the search key, the acquisition of this machine learning data as training data may be canceled.

[0036] The CPU 11 sets a training dataset using some or all of the acquired training data (S6). If there is too much acquired data, the training dataset does not need to include all of it. Alternatively, the training dataset may include all of the acquired data, but the CPU 11 may evaluate the training status as needed during training (described later) and terminate training when overfitting occurs.

[0037] The CPU 11, acting as the learning unit, uses the training data included in the training dataset to train an untrained machine learning model 1311 (S7; learning means). The CPU 11 evaluates the learning progress during training and terminates the training when the accuracy stops improving (S8).

[0038] The CPU 11 outputs the trained model (S9). This output means setting the trained machine learning model 1311 to be used, but it may also be output to another computer that will use the machine learning model 1311. Then, the CPU 11 terminates the learning control process.

[0039] [Second Embodiment] Next, a learning device of the second embodiment will be described. Figure 5 is a block diagram showing the functional configuration of the learning device 1a of the second embodiment.

[0040] In learning device 1a, program 131 of learning device 1 has been replaced with program 131a. Program 131a includes a machine learning model 1311 as well as an inclusion determination unit 1312 as a determination unit. The other configurations are the same for learning device 1 and learning device 1a, so a detailed explanation is omitted.

[0041] The inclusion determination unit 1312 determines the labels of the first ground truth data, i.e., the elements of the second ground truth data, that are included in the set search key. For example, if the search key is "animal," then labels such as "dog," "cat," and "monkey" in the machine learning data are concepts encompassed by "animal," i.e., subordinate concepts. Also, if the search key is "dog," then labels such as "puppy," "adult dog," and each dog breed are all subordinate concepts. The program 131a in this embodiment acquires machine learning data having labels of subordinate concepts encompassed by the search key as training data. The inclusion determination unit 1312 may determine the inclusion relationship based, for example, on natural language processing and language vectors representing terms as multidimensional vectors. In this case, the program 131a may include processing instructions to convert labels into search keys, i.e., higher-level concepts. Note that instead of acquiring all machine learning data of subordinate concepts that can be converted into higher-level concepts, a priority order may be set for acquiring data as training data among the subordinate concepts.

[0042] Furthermore, if each machine learning data in the database 14 contains additional text, this additional text may be used to determine the inclusion relationship. Even if the labels do not correspond to complete superordinate or subordinate concepts, data that actually have an appropriate inclusion relationship can be determined based on the additional text.

[0043] Furthermore, if the data type in the search key is a tag, program 131a may also acquire data whose data type is a frame or contour, remove the frame or contour setting, and convert the data type back to a tag. Alternatively, if the data type in the search key is a frame, program 131a may also acquire data whose data type is a contour, and convert the contour setting back to a frame that circumscribes the contour.

[0044] FIG. 6 is a flowchart showing the control procedure of the learning control process in the learning device 1a of the second embodiment. In this learning control process, steps S3 and S4 are changed to steps S3a and S4a, respectively, with respect to the above learning control process, and steps S10 to S12 are added. Other processes are the same, and the same reference numerals are given to the same process contents, and detailed descriptions thereof are omitted.

[0045] After step S2, the CPU 11 determines the inclusion relationship between the search key and each label registered in the database 14 by the inclusion determination unit 1312 (S3a). The CPU 11 acquires the locations of the external data servers 21 to 23 that hold data matching the search key or data of a lower concept of the search key (S4a). Then, the process of the CPU 11 proceeds to step S5.

[0046] After step S5, the CPU 11, as a conversion unit, performs a process of converting the labels and data types of the lower concepts of the search key into the upper concept to make them the same as the search key (S10). Then, the process of the CPU 11 proceeds to step S6.

[0047] After step S8, the CPU 11, as an evaluation unit, evaluates the performance of the machine learning model using the evaluation data (S11). The CPU 11 determines whether the evaluation result satisfies a predetermined criterion and is OK (S12). If it is determined that the evaluation result is not OK (S12; N), the process of the CPU 11 returns to step S6. In step S6, if the learning data set was set with a part of the learning data by the previous process, the CUP 11 may replace the learning data included in the learning data set. If the learning data set was set with all of the learning data, the CPU 11 may change the order of the learning data in the learning data set.

[0048] If it is determined that the evaluation result is OK (S12; Y), the process of the CPU 11 proceeds to step S9.

[0049] [Third Embodiment] Figure 7 is a flowchart showing the control procedure of the learning control process in the third embodiment of the learning device 1a. This learning control process has steps S13 to S15 added to the learning control process of the second embodiment. Also, step S12 has been changed to step S12a. Other processes are the same between the learning control processes, and the same reference numerals are used for processes with the same processing content, and detailed explanations are omitted.

[0050] After step S11, the CPU 11 determines whether a predetermined number of evaluations have been performed on each set dataset (S12a). If it is determined that no evaluations have been performed (S12a; N), the CPU 11 returns to step S6. If it is determined that a predetermined number of evaluations have been performed (S12a; Y), the CPU 11, as a calculation unit, calculates the contribution of each training data to improving accuracy based on the correspondence between the dataset and the evaluation. The contribution is, in other words, the learning contribution. The CPU 11 identifies training data that do not meet the criteria for improving the accuracy of machine learning and have a low contribution (S13). The CPU 11 excludes the identified data so that it is not added to the training dataset (S14).

[0051] The CPU 11 determines whether or not a predetermined condition has been met (S15). The predetermined condition may be the evaluation level, or it may be the removal of low-contributing training data a predetermined number of times. If it is determined that the predetermined condition has not been met (S15: N), the CPU 11 returns to process S6. If it is determined that the predetermined condition has been met (S15: Y), the CPU 11 proceeds to process S10.

[0052] As described above, the learning device 1 of this embodiment is a learning system for training a machine learning model 1311, and comprises a database 14 and a CPU 11. The database 14 stores information about machine learning data, at least a portion of which exists outside the learning system. The CPU 11, as a search unit, searches for learning data from the machine learning data that can be matched to the evaluation data of the machine learning model 1311, based on the information stored in the database 14. The CPU 11, as an acquisition unit, acquires the searched learning data. The CPU 11, as a learning unit, trains the machine learning model 1311 using the acquired learning data. This learning device 1 does not hold all the machine learning data itself, but instead holds information about the machine learning data held by the external device 20, so it only needs to quickly and temporarily acquire the necessary data when needed. Except for a minimum number of evaluation images, the user does not need to generate correct answer data on their own, so the effort and labor are greatly reduced. On the other hand, since the necessary data is searched within the range of the database that holds appropriate data, processing is easy and the possibility of low-quality training data being mixed in is reduced. Therefore, the learning device 1 can enable the machine learning model 1311 to perform highly accurate training with less effort and memory capacity.

[0053] The learning device 1 may include an operation reception unit 16 and / or a communication unit 17 as reception units for acquiring evaluation data from an external source. The user can easily set the image to be evaluated and easily add necessary information to the image.

[0054] The database 14 may store information about the first correct answer data in the machine learning data. The CPU 11, acting as a search unit, may search for training data in which the content of the first correct answer data matches the content of the second correct answer data based on evaluation data within a predetermined range. In this way, the necessary training data is obtained by comparing correct answer information, making it easy to search for training data.

[0055] Database 14 may store classification label information of the first correct answer data as information about the first correct answer data. In particular, since the fit is determined by the similarity or difference of the correct answer labels, it is possible to quickly obtain clearly correct training data.

[0056] The CPU 11 of the learning device 1 may, as a conversion unit, convert machine learning data according to evaluation data. The CPU 11 as a search unit may search for training data such that the content before conversion falls within a predetermined range. That is, the CPU 11 may acquire machine learning data as training data in which the first correct answer data and the second correct answer data do not perfectly match. In this case, processing may be performed to convert the parts that do not match, such as the format of the training data, so that they match. This makes it possible to efficiently use data that has essentially the same content and easily collect a large amount of training data.

[0057] The CPU 11, acting as a conversion unit, may convert the first correct answer data, which includes elements contained within the elements of the second correct answer data, to match the elements of the second correct answer data. In addition to or instead of the above format conversion, the learning device 1 may acquire machine learning data having correct answer labels that correspond to lower-level concepts of the correct answer of the evaluation data, and convert the correct answer labels to higher-level concepts that match the correct answer of the evaluation data. Since changing the correct answer of a lower-level concept to a higher-level concept does not change the correct answer, the learning device 1, which can easily obtain even more learning data, may include an inclusion determination unit 1312 that determines the inclusion relationship between the first correct answer data and the second correct answer data. The inclusion determination unit 1312 is a function executed by the CPU 11 and may be implemented by program 131a. This makes it easy to determine the inclusion relationship between the first correct answer data and the second correct answer data.

[0058] The CPU 11, acting as a conversion unit, may convert at least one of the classification labels or the format of the first correct answer data to match the second correct answer data. Since the content that can be adjusted between the correct answer data is limited, these can be quickly determined and adjusted to match the second correct answer data. Therefore, the learning device 1 can easily acquire a large amount of learning data.

[0059] The CPU 11 of the learning device 1 may operate as a calculation unit that calculates the learning contribution of the training data. The CPU 11, acting as a learning unit, may exclude data from the training data if the calculated learning contribution does not meet the criteria. By quantitatively evaluating the learning performed while replacing part or all of the training dataset, it is possible to determine which training data contributes to the learning. This allows the learning device 1 to further improve its learning accuracy. In particular, if the training data can be easily acquired, such replacement of the training dataset becomes possible.

[0060] The CPU 11 of the learning device 1 may, as an evaluation unit, evaluate the machine learning model 1311 that has been trained using the evaluation data 121. If the evaluation of the machine learning model 1311 meets the criteria, the training of the machine learning model 1311 may be terminated. By performing accuracy evaluations as the machine learning progresses, it is possible to reduce the possibility of continuing training indefinitely without improving accuracy or overfitting occurring.

[0061] The learning method of this embodiment includes the following steps: (1) Based on the information in a database that stores information about machine learning data, at least a portion of which exists outside the learning system, the system searches for learning data from the machine learning data that can be matched to the evaluation data of the machine learning model 1311. (2) The searched learning data is acquired. (3) The machine learning model 1311 is trained using the acquired learning data. This learning method makes it easy to collect and train learning data from outside the learning system based on the evaluation data 121, thus greatly reducing the effort required. On the other hand, since the necessary data is searched within the scope of a database that holds appropriate data, processing is easy and the possibility of low-quality training data being mixed in is reduced. Therefore, the learning method can perform highly accurate machine learning with less effort and memory capacity.

[0062] By installing and running the program 131 of this embodiment on a computer, the effort required to prepare and generate training data is reduced, and it becomes possible to easily and accurately train the machine learning model 1311 through software control.

[0063] This disclosure is not limited to the embodiments described above, and various modifications are possible. For example, although the learning device 1 is described above as having a format conversion application, it is not limited to this. The acquired data requiring format conversion may be output to another device capable of format conversion, and the format-converted learning data may be obtained.

[0064] Furthermore, regarding the determination of inclusion relationships, the CPU 11 may request the inclusion determination process from an external source outside the learning device 1 and obtain only the result.

[0065] Furthermore, once the inclusion relationship between the labels of the first ground truth data and the labels of the second ground truth data being searched has been determined, the labels of the first ground truth data may be listed and used or referenced thereafter. Also, the main subordinate concepts for the labels of the second ground truth data may be pre-listed and used for inclusion determination.

[0066] Furthermore, while the above example shows how to calculate the contribution to learning for each training data point, this is not the only way. The contribution may be calculated for each data set containing multiple training data points, and a decision may be made as to whether or not to exclude them from the training data.

[0067] Furthermore, database 14 only needs to be capable of performing search operations based on search keys, and does not need to have the appearance of a database in the strict sense.

[0068] Furthermore, while the above explanation assumes that all training data is acquired initially, this is not the only option. Only the data necessary for the initial training dataset may be acquired, and additional training data may be acquired based on the search results when generating subsequent training datasets.

[0069] Furthermore, although the above described a learning system consisting of a single learning device 1, 1a, the learning system may be a combination of multiple computers. Also, some components, such as the database 14, may be externally connected to the learning devices 1, 1a, etc., or located on a network for easy access.

[0070] Furthermore, while the above description uses a storage unit 13 consisting of an HDD, flash memory, or other non-volatile memory as an example of a computer-readable medium for storing the machine learning control program 131 of this disclosure, the invention is not limited to these. Other computer-readable media include other non-volatile memory such as MRAM, or portable recording media such as CD-ROMs and DVD discs. Carrier waves can also be used as a medium for providing the program data of this disclosure via a communication line. In addition, the specific configurations, processing operations, and procedures shown in the above embodiments can be modified as appropriate without departing from the spirit of this disclosure. The scope of the present invention includes the scope of the invention as described in the claims and its equivalents.

[0071] This disclosure can be used in learning systems, learning methods, and programs.

[0072] 1, 1a Learning device 11 CPU 12 RAM 121 Evaluation data 13 Storage unit 131, 131a Program 1311 Machine learning model 1312 Inclusion determination unit 14 Database 15 Display unit 16 Operation reception unit 17 Communication unit 20 External devices 21-23 External data server

Claims

1. A learning system for training a machine learning model, comprising: a database that stores information about machine learning data, at least a portion of which exists outside the learning system; a search unit that searches for training data from the machine learning data that can be adapted to evaluation data for the machine learning model based on the information stored in the database; an acquisition unit that acquires the training data found by the search unit; and a learning unit that trains the machine learning model using the training data.

2. The learning system according to claim 1, further comprising a receiving unit for acquiring the evaluation data from an external source.

3. The learning system according to claim 1, wherein the database stores information on first correct answer data in the machine learning data, and the search unit searches for learning data in which the content of the first correct answer data matches the content of second correct answer data based on the evaluation data within a predetermined range.

4. The learning system according to claim 3, wherein the database stores classification label information of the first correct answer data as information of the first correct answer data.

5. The learning system according to claim 3, comprising a conversion unit that converts the machine learning data according to the evaluation data, wherein the search unit searches the learning data such that the content before conversion by the conversion unit falls within the predetermined range.

6. The learning system according to claim 5, wherein the conversion unit converts the first correct answer data, which includes elements included in the elements of the second correct answer data, to match the elements of the second correct answer data.

7. The learning system according to claim 6, further comprising a determination unit that determines the inclusion relationship between each element of the first correct answer data and each element of the second correct answer data.

8. The learning system according to claim 5, wherein the conversion unit converts at least one of the classification labels of the first correct answer data and the format of the first correct answer data to match the second correct answer data.

9. The learning system according to claim 1, comprising a calculation unit for calculating the learning contribution of the learning data, wherein the learning unit excludes data from the learning data for which the learning contribution calculated by the calculation unit does not meet a standard.

10. The learning system according to claim 1, comprising an evaluation unit that evaluates the learned machine learning model using the evaluation data, and terminating the learning of the machine learning model when the evaluation of the machine learning model meets the criteria.

11. A learning method for a learning system that trains a machine learning model, comprising: searching for training data from the machine learning data that can be adapted to the evaluation data of the machine learning model, based on information stored in a database that stores information about machine learning data that is at least partly located outside the learning system; acquiring the searched training data; and training the machine learning model using the training data.

12. A program that causes a computer in a learning system for training a machine learning model to function as: a storage means for storing information about machine learning data that exists at least in part outside the learning system; a search means for searching for training data from the machine learning data that can be adapted to evaluation data for the machine learning model based on the information stored in the storage means; an acquisition means for acquiring the training data found by the search means; and a learning means for training the machine learning model using the training data.

Images