Label correction system, label correction method, and program

Abstract

Provided are a label correction system, a label correction method, and a program that make it possible to edit labels attached to images individually for each image. This label correction system comprises a processor and a memory in which commands executed by the processor are stored. The processor: executes a command to display candidate images that are a plurality of images which are classified into any of two or more classes and to which labels corresponding to the classes are attached, the candidate images being candidates for which the labels are to be corrected; receives selection of a label correction target image for which the label is to be corrected from among the candidate images; and, for each label correction target image, receives correction of the label attached to the label correction target image.

Description

Label correction system, label correction method, and program

[0001] The present invention relates to a label correction system, a label correction method, and a program.

[0002] To realize a high-precision AI, a large amount of learning data with various variations is required. The learning data needs to be correctly labeled, but the annotation of a large amount of learning data is a burden for users. Note that AI is an abbreviation for Artificial Intelligence.

[0003] Patent Document 1 describes a method of modeling an unlabeled or partially labeled target data set using a machine learning model for classification or regression.

[0004] Japanese Patent Application Laid-Open No. 2018-537798

[0005] However, to optimize a classification model that classifies an image into two or more classes and assigns a label for each class, it is necessary to confirm whether the label for each image is correct. If the label is incorrect, the label is edited, and the label editing is fed back to the classification model. For label editing, a GUI that presents the label for each image to the user and accepts the label correction from the user is required.

[0006] Patent Document 1 discloses a GUI that has a field for a user to input a label for each of a plurality of clustered images, but it is difficult to edit the label for each image with such a GUI. Note that GUI is an abbreviation for graphical user interface.

[0007] The present invention has been made in view of such circumstances, and an object thereof is to provide a label correction system, a label correction method, and a program that enable editing for each image with respect to the label assigned to an image.

[0008] A label correction system according to a first aspect of this disclosure comprises a processor and a memory in which instructions to be executed by the processor are stored, wherein the processor executes instructions to display candidate images of candidates for which labels to be corrected, which are a plurality of images classified into one of two or more classes and assigned labels corresponding to the classes, and accepts the selection of a label correction target image from among the candidate images, and accepts the correction of the label assigned to the label correction target image for each label correction target image.

[0009] According to the label correction system of the first aspect of this disclosure, for each candidate image that has been labeled with a label corresponding to one of two or more classes, the selection of an image to be corrected is accepted, and the label correction is accepted for each image to be corrected. This enables label correction on an image-by-image basis.

[0010] In the label correction system according to the second embodiment, the processor may accept the selection of a displayed label correction target image as the selection of a label correction target image in the label correction system according to the first embodiment.

[0011] In the label correction system according to the third embodiment, the processor may display the labels attached to the candidate images and accept the selection of the labels attached to the candidate images as the selection of the images to be labeled.

[0012] In the label correction system according to the fourth embodiment, the processor may display a checkbox for each candidate image and accept the selection of a checkbox as the selection of an image to be labeled.

[0013] In the label correction system according to the fifth embodiment, the processor may display a sample image for each label and a pair of the sample image with the corresponding label, in any one of the label correction systems according to the first to fourth embodiments.

[0014] In the label correction system according to the sixth embodiment, the processor may display all class labels for each candidate image, in the label correction system according to the first embodiment.

[0015] In the label correction system according to the seventh embodiment, the processor may display the labels by applying different display modes to each label, as in the label correction system according to the third embodiment.

[0016] The label correction system according to the eighth embodiment is a label correction system according to any one embodiment from the first to the seventh embodiment, in which the processor may accept user input of additional information for each image to be labeled and display the additional information.

[0017] In the label correction system according to the ninth embodiment, the processor may display user operation information representing an operation performed by the user, in any one of the label correction systems according to the first to eighth embodiments.

[0018] In the label correction system according to the tenth embodiment, in the label correction system according to any one embodiment from the first to the ninth embodiment, the processor may display multiple candidate images in the order of the evaluation values ​​of the labels for each of the multiple candidate images.

[0019] In the label correction system according to the 11th embodiment, the processor may change the display mode of the image to be labeled in any one embodiment from the 1st to the 10th embodiment.

[0020] A label modification method according to a twelfth aspect of the present disclosure is a label modification method comprising a computer having a processor and a memory that stores instructions executed by the processor, which performs the steps of: displaying candidate images of a plurality of images that are classified into one of two or more classes and to which labels corresponding to the classes are assigned, and which are candidates for which labels should be modified; accepting the selection of a label modification target image from among the candidate images; and accepting the modification of the label assigned to the label modification target image for each label modification target image.

[0021] According to the label modification method of the twelfth aspect of this disclosure, it is possible to obtain the same effects and advantages as the label modification system of the first aspect of this disclosure.

[0022] In the label modification method according to the twelfth embodiment, the same matters as those specified in the second to eleventh embodiments can be appropriately combined. In that case, the components responsible for the processing and functions specified in the label modification system can be understood as components of the label modification method that are responsible for the corresponding processing and functions.

[0023] A thirteenth aspect of this disclosure is a program that enables a computer comprising a processor and a memory in which instructions executed by the processor are stored, to have a function to display candidate images of candidates for which labels should be modified, a function to accept the selection of a label to be modified from among the candidate images, and a function to accept the modification of the label attached to the label to be modified for each label to be modified.

[0024] According to the program relating to the 13th aspect of this disclosure, it is possible to obtain the same effects and advantages as the label correction system relating to the first aspect of this disclosure.

[0025] In the program relating to the 13th embodiment, the same matters as those specified in the 2nd to 11th embodiments can be appropriately combined. In that case, the components responsible for the processing and functions specified in the label correction system can be understood as components of the program responsible for the corresponding processing and functions.

[0026] According to the present invention, for each candidate image labeled with a label corresponding to one of two or more classes, the selection of an image to be labeled is accepted, and the label modification is accepted for each image to be labeled. This enables label modification on an image-by-image basis.

[0027] Figure 1 is a block diagram showing an example configuration of a disaster certificate issuance system to which the classification system according to the embodiment is applied. Figure 2 is a conceptual diagram of the house collapse judgment AI shown in Figure 1. Figure 3 is a schematic diagram of the label assignment method according to the embodiment. Figure 4 is a flowchart showing the procedure of the label assignment method according to the embodiment. Figure 5 is a schematic diagram of the learning step applied to the classification algorithm. Figure 6 is a schematic diagram of the inference step applied to the classification algorithm. Figure 7 is an explanatory diagram of label correction. Figure 8 is a flowchart showing the procedure of the label correction method according to the embodiment. Figure 9 is a schematic diagram of an annotation GUI related to a specific example. Figure 10 is an explanatory diagram of a specific example of the house image display unit. Figure 11 is a schematic diagram of an annotation GUI showing an example of displaying a house image for each label. Figure 12 is an explanatory diagram of the label correction function according to the first example. Figure 13 is an explanatory diagram of the label correction function according to the second example. Figure 14 is an explanatory diagram of the label correction function according to the third example. Figure 15 is an explanatory diagram of the label correction function according to the fourth example. Figure 16 is an explanatory diagram of the label display unit according to the first example. Figure 17 is an explanatory diagram of the label display unit according to the second example. Figure 18 is an explanatory diagram of the label display unit according to the third example. Figure 19 is an explanatory diagram of the information assignment unit according to the first example. Figure 20 is an explanatory diagram of the information assignment unit according to the second example. Figure 21 is an explanatory diagram of the information assignment unit according to the third example. Figure 22 is a schematic diagram of updating the classification algorithm according to the first example. Figure 23 is a flowchart showing the procedure for updating the classification algorithm according to the first example. Figure 24 is a schematic diagram of updating the classification algorithm according to the second example. Figure 25 is a flowchart showing the procedure for updating the classification algorithm according to the second example. Figure 26 is an explanatory diagram of multi-class classification with three or more classes. Figure 27 is a block diagram showing an example configuration of the classification system according to the embodiment. Figure 28 is a block diagram showing an example configuration of the classification system according to the embodiment.

[0028] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In this specification, identical components are denoted by the same reference numerals, and redundant descriptions are omitted as appropriate. Furthermore, when multiple components are listed as examples in the following embodiments, it can be interpreted that at least one of the multiple components is included.

[0029] [Overall Configuration of the Disaster Victim Certificate Issuance System] Figure 1 is a block diagram showing an example configuration of a disaster victim certificate issuance system to which the classification system according to the embodiment is applied. Figure 1 illustrates a disaster victim certificate issuance system that includes an information gathering step, an analysis and judgment step, a planning step, an on-site survey step, and an issuance step.

[0030] In the information gathering step, resident information is acquired using the municipal business systems of local governments, etc. Furthermore, wide-area photography using drones is conducted during the information gathering step, and aerial images are acquired. Here, the term "image" can include image data, which is the electrical signal of an image.

[0031] In the analysis and decision step, the extent of the damage is visualized using resident information and aerial images obtained in the information gathering step. In addition, in the analysis and decision step, georeferencing and geocoding are applied to the aerial images to extract individual houses from the images and identify the location of each house.

[0032] In the analysis and judgment step, a house collapse judgment AI is applied, and for each house extracted from aerial images, an automatic judgment is performed to determine, for example, whether it has collapsed or is normal. The automatic judgment may determine whether it is completely destroyed or not, and may determine whether it is completely destroyed, partially destroyed, or normal. The conditions of the houses to which the automatic judgment is applied are not limited to the above examples and may be determined according to the purpose, etc.

[0033] In the analysis and judgment step, the judgment results for each individual house output from the house collapse judgment AI are mapped onto map information, generating geospatial information, which is then visualized. Through the information gathering step and the analysis and judgment step, the extent of the damage is accurately visualized, allowing planners to grasp the damage situation.

[0034] In the planning step, the comprehensive damage assessment management system creates and optimizes the assessment plan. Specifically, in the planning step, the priority of the assessment area is determined, the number of support personnel required for the assessment is determined, and an assessment plan that takes past experience into account is created. Furthermore, in the planning step, the assessment plan is automatically optimized by considering the building structure of the area to be assessed, the skill sets of the assessors, and the progress of the assessment.

[0035] In the field survey step, surveyors conduct the survey using a surveyor app. The surveyor app is a tablet application installed and running on a tablet device. The tablet application has functions to guide surveyors through the survey procedure and to automatically calculate the degree of damage. Using the tablet application eliminates the need for paper survey forms, survey manuals, and cameras to photograph houses, which can improve the accuracy and efficiency of the field survey.

[0036] In the issuance step, a simplified form output is performed using the local government's business system. Specifically, the issuance step involves receiving the application for the issuance of the disaster certificate, outputting the disaster certificate form, and outputting the recipient's address on the disaster certificate.

[0037] [Overview of the House Collapse Judgment AI] Figure 2 is a conceptual diagram of the house collapse judgment AI shown in Figure 1. The house collapse judgment AI 10 takes house image BI for each house extracted from aerial image AIm of the survey area SuA taken using a camera mounted on a drone Dr as input, and outputs damage estimation information JI representing the degree of collapse for each house. Map information MI is referenced to extract the house image BI.

[0038] Figure 2 illustrates an example in which three house images BI11, BI12, and BI13 are extracted from an aerial image AIm. The figure also illustrates an example in which the three house images BI11, BI12, and BI13 are input to the house collapse determination AI10, and damage estimation information JI1, JI2, and JI3 are output for each of the three house images BI11, etc.

[0039] The damage estimation information JI includes the house's address, assessment result, and the house's coordinates in aerial images or maps. The assessment result may be one of the following: completely destroyed, partially destroyed, normal (undamaged), or unknown. The introduction of the house collapse assessment AI 10 can help determine which areas should be surveyed first and assist in the creation of survey plans. Furthermore, it eliminates the need to travel to the site for surveys, improving the efficiency of work related to creating survey plans and enabling the prompt issuance of disaster certificates to victims.

[0040] [Challenges of AI for House Collapse Prediction] Developing a highly accurate AI for house collapse prediction 10 requires a large amount of training data with various variations. However, there is a limit to the amount of training data that can be acquired, and the variation in training data is extremely small. While an AI for house collapse prediction 10 trained using training data with little variation and a small amount of data shows some results, further improvement in accuracy is desired.

[0041] Furthermore, it was found that the distribution of feature labels differs from region to region in the feature space representing the distribution of features extracted from the house image BI. The labels represent house collapses, etc., as illustrated in Figure 2. In other words, by optimizing the house collapse detection AI 10 for each region, further improvement in the accuracy of the house collapse detection AI 10 can be expected. In order to optimize the house collapse detection AI 10 for each region, labeled training data from the region to which the house collapse detection AI 10 is applied is required.

[0042] Acquiring labeled training data can be achieved by having users label a portion of the house image BI collected by region. User-performed labeling can be included in annotation. User-performed annotation includes the following tasks 1, 2, and 3.

[0043] [Problem 1] Annotating unknown house image BIs from scratch is burdensome for users. Unknown house image BIs are synonymous with unclassified house image BIs.

[0044] [Problem 2] Without a GUI used for annotation, it is difficult to implement annotation, and it becomes difficult to create an optimal house collapse determination AI 10 for each region. Note that GUI is an abbreviation for graphical user interface.

[0045] [Problem 3] Without a method for optimizing the classification algorithm applied to annotation, it is difficult to achieve the desired accuracy for the labels obtained as a result of annotation.

[0046] Regarding the labeling method for the above-mentioned Problem 1, Problem 2, and Problem 3, the GUI applied to label correction, and the update method of the classification algorithm to which label correction is fed back, will be described in detail below.

[0047] [Configuration Example of Labeling Method] Figure 3 is a schematic diagram of the labeling method according to the embodiment. As a countermeasure against the above-mentioned Problem 1, the labeling method illustrated in the figure can be cited. In the labeling method illustrated in Figure 3, a label of a binary or multi-valued class of three or more values is assigned to an unknown house image BI.

[0048] In Figure 3, as an unknown house image BI, an example is illustrated in which any one of label LaA or label LaB is assigned as label La to each of house image BI21, house image BI22, and house image BI23. The unknown house image BI to which label La is assigned is applied to the learning for generating a learning model specialized for the region where, for example, the unknown house image B1 is collected, regarding the house collapse determination AI 10 illustrated in Figure 2.

[0049] That is, label La functions as a temporary label of the unknown house image BI used for learning data. Note that label LaA, label LaB, and label LaC described later represent any different labels La.

[0050] The labeling method illustrated in Figure 3 is implemented using one or more computers equipped with a processor and memory, with the processor executing a program stored in memory. The labeling method may be implemented using a computer functioning as a trained image recognition model 12 and a computer functioning as a classification algorithm 14. The computer functioning as the classification algorithm 14 outputs a classification result 16. The classification result 16 may be displayed using a display device.

[0051] The computer that functions as the trained image recognition model 12 and the computer that functions as the classification algorithm 14 may be implemented using one computer, or each may be implemented using one or more computers.

[0052] The trained image recognition model 12 extracts features from each of the multiple unknown house image BIs. That is, the trained image recognition model 12 performs dimensionality reduction on each of the multiple unknown house image BIs and extracts the features from each of the multiple unknown house image BIs. The features from each of the multiple unknown house image BIs are understood as feature points that constitute the feature distribution in the feature space.

[0053] The pre-trained image recognition model 12 is a pre-trained model that has been trained using general images. Examples of pre-trained image recognition models 12 include MoCo, BYOL, and DINO.

[0054] The classification algorithm 14 classifies each feature into two or more classes using a classification plane CP defined in the feature space. The distance Di from the classification plane to each feature may be the shortest distance between each feature and the classification plane CP.

[0055] Examples of two or more classes include three classes: total destruction, partial destruction, and normal. Two or more classes may also include two classes: total destruction and partial destruction. Two or more classes may include unknown, undamaged, and not total destruction. The unknown house image BI, which is represented as a feature, is assigned a label La corresponding to each class.

[0056] The classification plane CP may be an (n-1)-dimensional hyperplane applied to the n-dimensional feature space when n is an integer greater than or equal to 2. That is, the classification algorithm 14 may be a linear classification model. Examples of linear classification models include linear regression models and support vector machines. The classification plane CP may be any plane defined in the nonlinear model.

[0057] The classification plane CP applied to the classification algorithm 14 may be a classification plane CP trained using pairs of house image BIs with labels La and labels La as training data, or a trained classification plane CP whose validity has been evaluated using evaluation data may be applied. In other words, the classification algorithm 14 may be to which a trained classification model is applied.

[0058] The classification result 16 includes a label La for each unknown house image BI, and distance information including the distance Di from the classification plane CP for each feature extracted from the house image BI. Figure 3 shows an example of the distance Di from the classification plane CP, which is represented using a numerical value between 0 and 1.0.

[0059] The distance Di from the classification plane CP for each feature represents the confidence level of the label La for each feature. A relatively large distance Di indicates a relatively high confidence level, while a relatively small distance Di indicates a relatively low confidence level.

[0060] [Procedure for Labeling Method] Figure 4 is a flowchart showing the procedure for labeling according to the embodiment. Step S10 is when an unknown house image BI to which label La is to be assigned is obtained. In step S10, for example, the unknown house image BI may be obtained from a house image storage device in which the unknown house image BI is stored.

[0061] In step S12, the trained image recognition model 12 shown in Figure 3 performs dimensionality reduction on the unknown house image BI and extracts the features of the unknown house image BI.

[0062] In step S14, the classification algorithm 14 shown in Figure 3 assigns a label La to each feature based on the classification plane CP in the feature space where the features of the unknown house image BI are represented. In the example shown in Figure 3, the features on the upper side of the classification plane CP are assigned the label LaA, and the features on the lower side of the classification plane CP are assigned the label LaB.

[0063] In step S16, the classification algorithm 14 outputs a classification result 16 for each unknown house image BI. The classification result 16 for each unknown house image BI is stored as regional training data for the house collapse judgment AI 10 shown in Figure 2, for example, to create an AI specialized for the region where the unknown house image BI was collected. Note that the labeling method illustrated in Figure 4 is an example of the classification method of this disclosure.

[0064] [Example of the structure of the learning steps of a classification algorithm] Figure 5 is a schematic diagram of the learning steps applied to a classification algorithm. In the learning steps applied to classification algorithm 14, the classification plane CP applied to the feature space is learned. That is, classification algorithm 14 applies the classification plane CP learned using the training data TD.

[0065] The training data TD for classification algorithm 14 is a combination of labeled house images BI and labels La. Labeled house images BI are generated by manually assigning either label LaA or label LaB to each house image BI.

[0066] In the learning step, the trained image recognition model 12 is used to extract features from the labeled house image BI. In the learning step, the classification plane CP is defined based on the features extracted from the labeled house image BI and the label La assigned to the labeled house image BI. In the example shown in Figure 5, the classification plane CP is defined as the classification boundary that classifies the features to which label LaA is assigned and the features to which label LaB is assigned.

[0067] In training classification algorithm 14, it is difficult to define accurate classification boundaries when the amount of training data is insufficient or insufficient. When training is performed using a small amount of training data, overfitting is likely to occur, and classification boundaries specific to a particular data set may be defined. In that case, while high-accuracy classification is achieved for a specific data set, there is a concern that the classification accuracy will decrease for other data sets that fall outside of that specific data set.

[0068] The classification algorithm 14 according to the embodiment preferably has high generalizability, achieving a certain classification accuracy for various types of data sets. Therefore, the classification algorithm 14 preferably uses linear classification, which is less prone to overfitting even when a small amount of training data is used. Note that classification with an arbitrary classification boundary and linear classification with a classification plane CP defined as the classification boundary are examples of the classification rules of this disclosure.

[0069] [Example of the configuration of the inference step of the classification algorithm] Figure 6 is a schematic diagram of the inference step applied to the classification algorithm. In the inference step, the classification algorithm 14 may be updated to fit the classification algorithm 14 to the unknown house image BI. In the inference step, the label La may be assigned to the unknown house image BI using the updated classification algorithm 14. Here, the unknown house image BI is the house image BI extracted from the aerial image AIm of a new region before the label La is assigned.

[0070] In other words, in the inference step, the pre-trained image recognition model 12 is used to extract features for each unknown house image BI. In the training step, the pre-trained classification algorithm 14 is used to assign a label La corresponding to the features to each unknown house image BI.

[0071] Figure 6 illustrates an example in which a binary label, either label LaA or label LaB, is assigned to the feature quantities of each unknown house image BI.

[0072] In the inference step, a portion of the house image BI to which label La has been assigned is selected using the classification algorithm 14, and the selected house image BI and label may be displayed using the annotation GUI 18. The annotation GUI 18 shown in Figure 6 displays the house image BI to which label LaA indicates normal, and the house image BI to which label LaB indicates collapse.

[0073] The user modifies the label La for the house image BIs displayed in the annotation GUI18 that need to have their label La modified. Specifically, the user uses the annotation GUI18 to select the house image BIs whose label La needs to be modified and enters the correct label La.

[0074] In the inference step, the results of the label La modifications made by the user may be fed back to the classification algorithm 14. The classification algorithm 14 may be modified by modifying the classification plane CP shown in Figure 6, or it may be modified into an algorithm that does not use the classification plane CP.

[0075] In the inference step, the updated classification algorithm 14 may be used to assign labels La to unknown house image BIs. The combination of the thus derived labels LaA or LaB and the house image BIs to which labels LaA or LaB are assigned is used as training data for the house collapse detection AI 10 shown in Figure 2, for example, to create an AI specific to the area where unknown house image BIs were collected.

[0076] [Effects of the labeling method] The labeling method according to the embodiment can obtain the following effects.

[0077] [1] For each house image BI extracted from aerial images of the area targeted for disaster damage survey, features are extracted using a trained image recognition model 12, and labels La based on the features are assigned using a classification algorithm 14. This allows the user to be presented with house image BIs that have been labeled La, reducing the effort required of the user in annotating unknown house image BIs.

[0078] [2] The classification algorithm 14 assigns a label La based on the classification plane CP defined in the feature space to the unknown house image BI. As a result, a label La corresponding to the feature distribution of the house image BI in the feature space is assigned to the unknown house image BI.

[0079] [3] The classification plane CP is an integer greater than or equal to 2, and when the number of dimensions of the feature space is n, an n-1 dimensional hyperplane is applied. That is, the classification algorithm 14 is a linear classification algorithm. The classification plane CP is learned and defined using labeled training data. This ensures that even when the amount of training data used to learn the classification plane CP is insufficient, the learning of the classification plane CP is carried out in a way that prevents overfitting and realizes a classification algorithm 14 with generalizability.

[0080] [4] In the training of the classification plane CP, the validity of the classification plane CP is evaluated using evaluation data. This improves the accuracy of the classification algorithm 14.

[0081] [5] The pre-trained image recognition model 12 is a pre-trained model that has been trained using general images. This enables the extraction of highly versatile features.

[0082] [6] The classification algorithm 14 outputs a house image BI, a label La, and a distance Di from the classification plane CP. This allows the user to recognize the label for each house image BI and the confidence level of the label La, which is expressed as the distance Di from the classification plane CP.

[0083] [Specific Example of Annotation GUI] Figure 7 is an explanatory diagram of label modification. Note that label modification is synonymous with class modification. In response to the above problem 2, an annotation GUI 18 is proposed to support the modification of label La performed by the user. Figure 7 schematically illustrates the modification of label La performed when annotation GUI 18 is applied.

[0084] The annotation GUI 18 is a combination of a house image BI and a label La assigned using the classification algorithm 14 shown in Figure 3. Selected combinations of house image BIs and label Las are presented to the user as candidate images for which label Las should be modified. The annotation GUI 18 shown in Figure 7 displays house image BIs that have either a label La indicating normal or a label La indicating collapse as candidate images.

[0085] The house image BI to be displayed in annotation GUI18 may be selected if it has few similarities to other house image BIs. Similarities include the shape of the house, the color of the house, and the degree of damage to the house.

[0086] The user can input the result of determining whether label La is correct or not using the annotation GUI18. In other words, the user can select the image to be corrected for label modification, and the user can modify the label La assigned to each image.

[0087] Furthermore, the user can use the annotation GUI 18 to add new information that contributes to labeling the house image BI. This allows the classification algorithm 14 to utilize the information that contributes to labeling, and an improvement in the accuracy of the classification algorithm 14 is expected. An example of information that contributes to labeling is information indicating the likelihood of collapse of wooden houses and non-wooden houses. The classification algorithm 14 is modified based on the results of the modification of label La.

[0088] Figure 7 illustrates the case where, as part of the correction of label La, the label La of feature P13 is corrected from normal to abnormal, and the label La of feature P23 is corrected from abnormal to normal. The correction of label La is fed back to the classification algorithm 14.

[0089] [Procedure for Label Correction Method] Figure 8 is a flowchart showing the procedure for a label correction method according to the embodiment. The label correction method illustrated in the figure is implemented by a label correction system to which one or more computers equipped with a processor and memory are applied, where the processor executes a program stored in memory.

[0090] In step S100, the annotation GUI 18 shown in Figure 7 is displayed using a display device. The annotation GUI 18 displays a house image BI to which multiple selected labels La have been assigned.

[0091] In step S102, modifications to label La are accepted. That is, modifications to label La entered by the user using the annotation GUI 18 are accepted. In step S102, input of information that contributes to label assignment may be accepted.

[0092] In step S104, the label La received in step S102 is modified. In step S106, the modification of label La is fed back to the classification algorithm 14, and the classification algorithm 14 is updated.

[0093] [Specific Example of Annotation GUI] Figure 9 is a schematic diagram of an annotation GUI according to a specific example. The annotation GUI 18 shown in Figure 9 includes a house image display unit 30 that displays the house image BI to be modified for each label La. The house image display unit 30 may display only one house image BI, or it may display a predetermined number of house image BIs. Figure 9 shows multiple house image display units 30 in which multiple house image BIs are displayed individually.

[0094] The annotation GUI 18 has a function to modify the label La. When selecting a house image BI to modify the label La, the user may select the icon 33 that displays the label La, or they may select the house image BI itself. When selecting a house image BI to modify the label La, the user may select a checkbox. Note that the checkbox is not shown in Figure 9.

[0095] The annotation GUI 18 may include a label display unit 32 that displays the label La. Figure 9 shows examples of label display units 32 corresponding to three types of label La. Examples of the three types of label La include a label La corresponding to normal, a label La corresponding to partial damage, and a label La corresponding to total damage.

[0096] The label display unit 32 shown in Figure 9 includes a plurality of icons 33. For example, icon 33A corresponds to label La representing normal, icon 33B corresponds to label La representing partial damage, and icon 33C corresponds to label La representing total damage. The label display unit 32 may also include an icon 33 corresponding to label La representing an unknown extent of damage to the house.

[0097] The annotation GUI 18 may include a sample display unit 34. The sample display unit 34 displays a combination of a sample house image BI for each label La and the sample label La assigned to the sample house image BI.

[0098] For each label La, one or more sample house image BIs are defined as sample house image BIs. The sample display unit 34 displays the sample house image BIs for all labels La. The sample house image BIs may be the labeled house image BIs used to train the classification algorithm 14.

[0099] The sample display unit 34 may display the degree of damage based on the Guidelines for the Application of Damage Assessment Standards for Houses Related to Disasters, formulated by the Cabinet Office. The annotation GUI 18 may be configured in a way that does not include the sample display unit 34. The sample house image BI is an example of a sample image in this disclosure.

[0100] The annotation GUI 18 may include an information assignment unit 36. The information assignment unit 36 ​​is used when the user inputs various information related to the annotation of label La. For example, the information assignment unit 36 ​​may display character information that contributes to the assignment of label La. The characters here may include numbers, symbols, and figures.

[0101] Figure 9 illustrates a configuration in which multiple information assignment units 36 are displayed corresponding to each of the multiple house image BIs. The annotation GUI 18 may be configured in a way that does not include the information assignment units 36. The various types of information input by the user to the information assignment units 36 are examples of the additional information described herein.

[0102] The annotation GUI 18 may include an operation instruction section 38. The operation instruction section 38 displays instructions for operating the annotation GUI 18. The operation instruction section 38 shown in Figure 9 displays text information indicating that the house image BI to which label La is to be changed is selected. The annotation GUI 18 may also be configured without an operation instruction section 38. The instructions for operation displayed in the operation instruction section 38 are an example of user operation information representing an operation performed by the user of this disclosure.

[0103] The annotation GUI 18 includes an exit button 40. The exit button 40 is operated when the user wants to exit the display of the annotation GUI 18. The arrangement of the exit button 40 shown in Figure 9 is an example, and the exit button 40 may be placed in any position.

[0104] [Specific Example of House Image Display Unit] Figure 10 is an explanatory diagram of a specific example of a house image display unit. The figure shows a house image display unit 30A that displays only one house image BI. The house image display unit 30A may have a function to enlarge the house image BI. For example, the house image display unit 30A may include a combo box that allows the user to select whether or not to enlarge the house image BI when the user clicks on the house image BI.

[0105] The house image display unit 30A may include a remaining number display unit 42 that displays the remaining number of house image BIs. The remaining number display unit 42 shown in Figure 10 displays 3, which is the remaining number out of a total of 100 house image BIs.

[0106] The house image display unit 30A may be equipped with screen transition buttons 44. The screen transition buttons 44 shown in Figure 10 include a back button 44A, a next button 44B, and an exit button 44C. The back button 44A is operated to proceed to the previous house image BI. The next button 44B is operated to proceed to the next house image BI. The exit button 44C is operated to end the display of the house image BI.

[0107] When displaying multiple house image BIs, the number of house image BIs to be displayed on the house image display unit 30 shown in Figure 9 may be a predetermined number. The number of house image BIs to be displayed on the house image display unit 30 may vary depending on the region. As an example of applying different numbers depending on the region, the number of house image BIs to be displayed on the house image display unit 30 may be determined according to the number of newly acquired house image BIs.

[0108] Multiple house image BIs may be arranged randomly regardless of the label La for each house image BI, and may be sorted using the distance Di from the classification plane CP for each feature in the classification algorithm 14. As shown in Figure 9, multiple house image BIs may be partitioned according to label La. Note that the distance Di from the classification plane CP for each feature is an example of an evaluation value in this disclosure.

[0109] The house image display unit 30, which displays multiple house image BIs, may display all types of house image BIs for label La. If M1 is the total number of types for label La, and M2 is an integer of 2 or more satisfying M1 > M2, the house image display unit 30 may display M2 types of house image BIs for label La. The house image display unit 30 that displays M2 types of house image BIs for label La may be equipped with a screen transition button 44 as shown in Figure 10.

[0110] Figure 11 is a schematic diagram of the annotation GUI showing an example of displaying house images for each label. Figure 11 schematically illustrates the transition from the house image display unit 30B, which displays multiple house image BIs determined to be normal, to the house image display unit 30C, which displays multiple house image BIs determined to be completely destroyed.

[0111] The house image display unit 30B may be equipped with a next button 46. When the next button 46 is pressed, the display unit may transition from the house image display unit 30B, which displays multiple house image BIs determined to be normal, to the house image display unit 30C, which displays multiple house image BIs determined to be completely destroyed.

[0112] The house image display unit 30C may be equipped with an exit button 48. When the exit button 48 is operated, the display of the house image display unit 30C may be terminated. The house image display units 30B and 30C may be equipped with screen transition buttons 44 as shown in Figure 9.

[0113] [First Example of Label Correction Function] Figure 12 is an explanatory diagram of the label correction function according to the first example. As the operation to select the house image BI to which label La is to be corrected, the operation to select the label display unit 32 which indicates normal may be applied. The operation to select the label display unit 32 may be the operation to select the icon 33 included in the label display unit 32.

[0114] Figure 12 illustrates a scenario in which, among the house images BI with the label La indicating normal status displayed in the house image display unit 30D, the icon 33 to the left of the leftmost house image BI in the figure is selected using the cursor 50, and the house image display unit 30E assigns a different color to the selected icon 33 from other icons 33 corresponding to the same label La. In other words, the display mode of the selected icon 33 may be different from the display mode of the unselected icons 33 corresponding to the same type of label La.

[0115] [Second Example of Label Correction Function] Figure 13 is an explanatory diagram of the label correction function related to the second example. As the operation to select the house image BI to which label La is to be corrected, the operation to select the house image BI itself may be applied.

[0116] Figure 13 illustrates a configuration in which, among the house image BIs with the label La indicating normality displayed on the house image display unit 30F, the leftmost house image BI in the figure is selected, and a frame 54 surrounding the selected house image BI is added to the house image BI in the house image display unit 30G. The house image BI with the frame 54 added may be colored, or both color and frame may be used in combination.

[0117] [Third Example of Label Correction Function] Figure 14 is an explanatory diagram of the label correction function according to the third example. As the operation to select the house image BI to which label La is to be corrected, the operation of selecting the checkbox 56 corresponding to the house image BI may be applied.

[0118] Figure 14 shows a house image display unit 30H that shows a state in which a checkbox 56 corresponding to the central house image BI, which is labeled La to indicate normal, is selected using a cursor 50. Also shown is a house image display unit 30I that displays the selected house image BI with a frame 54 surrounding it. The house image BI with the frame 54 may be colored, or both color and frame may be used. A check mark 57 may be placed on the checkbox 56 of the house image BI with the frame 54.

[0119] The house image display units 30H and 30I shown in Figure 14 display an update button 52. The update button 52 shown in the same figure is operated when the label La is updated for house image BI other than the selected house image BI. The house image display units 30D shown in Figure 12 and others may also display the update button 52 shown in Figure 14.

[0120] [Fourth Example of Label Correction Function] Figure 15 is an explanatory diagram of the label correction function according to the fourth example. The figure shows the house image display unit 30J in a state where the house image BI selected as the target for correction of label La is highlighted.

[0121] The house image display unit 30J is equipped with an update button 58. When the update button 58 is operated after the user has selected all the house image BIs to be modified for label La, the modification of label La is fed back to the classification algorithm 14, and the classification algorithm 14 is updated.

[0122] [First Example of Label Display Unit] Figure 16 is an explanatory diagram of the label display unit according to the first example. In this figure, an arbitrary house image BI and an icon 33 corresponding to the house image BI are shown in an enlarged view.

[0123] Figure 16 shows the state of the label display unit 32 before the label La is modified. The figure shows the label display unit 32 which includes icons 33A corresponding to label La representing normal, icon 33B corresponding to label La representing partial damage, and icon 33C corresponding to label La representing total damage.

[0124] The label display unit 32 shown in Figure 16 assigns the same color to icons 33 corresponding to the same type of label La, regardless of the label La of the house image BI, and assigns different colors to icons 33 corresponding to different types of labels La. Figure 16 illustrates an embodiment in which icons 33A corresponding to label La representing normal are assigned the same color, icons 33B corresponding to label La representing partial destruction are assigned the same color, and icons 33C corresponding to label La representing total destruction are assigned the same color.

[0125] [Second Example of Label Display Unit] Figure 17 is an explanatory diagram of the label display unit according to the second example. In the label display unit 32 shown in the figure, only the icon 33 representing the label that matches the label La assigned to the house image BI is colored.

[0126] In other words, for a house image BI assigned the label La indicating normal, the icon 33A corresponding to the label La indicating normal will be colored, while icons 33B and 33C will not be colored. Similarly, for a house image BI assigned the label La indicating partial damage, the icon 33B corresponding to the label La indicating partial damage will be colored, while icons 33A and 33C will not be colored.

[0127] [Third Example of Label Display Unit] Figure 18 is an explanatory diagram of the label display unit according to the third example. The figure schematically illustrates the screen display transition when, as an operation to select a house image BI to be corrected for label La, an icon 33B corresponding to label La representing partial collapse is selected for a house image BI that has been assigned label La representing normal.

[0128] The label display unit 32 corresponding to the house image BI of label La that is to be modified is colored for icon 33B, which corresponds to label La representing partial destruction, while icons 33A and 33C are not colored.

[0129] [First Example of Information Assignment Unit] Figure 19 is an explanatory diagram of the information assignment unit according to the first example. The figure shows an embodiment in which the information assignment unit 36 ​​is equipped with a form 60 in which the user can input text information, etc. The form 60 may be enlarged or reduced according to the number of characters to be entered, etc.

[0130] The information assignment unit 36 ​​may be placed at any position. For example, the information assignment unit 36 ​​may be placed above the house image BI. The same applies to the information assignment unit 36 ​​shown in Figures 20 and 21.

[0131] [Second Example of Information Assignment Unit] Figure 20 is an explanatory diagram of the information assignment unit according to the second example. The figure illustrates an embodiment in which the information assignment unit 36 ​​is equipped with radio buttons 62, which include radio buttons 62A corresponding to wooden structures and radio buttons 62B corresponding to non-wooden structures. The items to be assigned to the radio buttons 62 are predetermined.

[0132] Figure 20 illustrates an information-providing unit 36 ​​equipped with two radio buttons 62, but the number of radio buttons 62 equipped in the information-providing unit 36 ​​is not limited to two; it may be one, three or more, or any other number.

[0133] [Third Example of Information Assignment Unit] Figure 21 is an explanatory diagram of the information assignment unit according to the third example. The figure shows an embodiment in which the information assignment unit 36 ​​is equipped with checkboxes 64. The items to be assigned to the checkboxes 64 are predetermined.

[0134] If there are many items that can be assigned to checkbox 64, the information assignment unit 36 ​​may include a combo box 66 that is displayed when checkbox 64 is operated. Figure 21 shows a configuration in which the combo box 66 includes a checkbox 66A corresponding to wooden structures and a checkbox 66B corresponding to non-wooden structures, and these are configured to be selectable. Note that checkbox 64 is an example of a checkbox for each candidate image in this disclosure.

[0135] [Effects of the Annotation GUI] The annotation GUI according to this embodiment can achieve the following effects.

[0136] [1] An annotation GUI18 is provided that is applied to annotations performed by the user. This provides support for annotations performed by the user.

[0137] [2] The annotation GUI 18 includes a house image display unit 30 on which house image BIs are displayed. This allows the user to recognize house image BIs that are candidates for modification.

[0138] [3] The house image BIs that are candidates for correction are sorted and displayed using the distance Di from the classification plane CP for each feature. This displays the house image BIs that are candidates for correction in order of highest confidence level.

[0139] [4] The annotation GUI 18 includes a label display unit 32 that displays a label La for each house image BI. This allows the user to recognize the label La for each house image BI.

[0140] [5] The annotation GUI 18 accepts modifications to the label La for each house image BI. This enables modifications to the label La for each house image BI. Furthermore, the modifications to the label La are fed back to the classification algorithm 14, which may enable an update of the classification algorithm 14.

[0141] [6] When selecting a house image BI to modify the label La, the annotation GUI 18 accepts operations such as clicking on the house image BI, selecting a checkbox for each house image BI, or selecting a label display unit 32 for each house image BI. This enables the selection of the house image BI to be modified.

[0142] [7] The display mode of the house image BI selected as the target for modifying label La is made different from that of the unselected house image BI. This highlights the selected house image BI.

[0143] [8] The annotation GUI 18 includes an update button 58 that instructs the user to provide feedback to the classification algorithm 14 regarding the modification of label La. This allows the user to provide feedback to the classification algorithm 14 regarding the modification of label La.

[0144] [9] The annotation GUI 18 includes a sample display unit 34 that displays sample house image BIs for each type of label La. The user can recognize typical house image BIs for each type of label La.

[0145]

[10] The annotation GUI 18 is used when the user inputs various information regarding the annotation of label La, and includes an information assignment unit 36 ​​that displays various information. This allows the user to recognize the various information they have entered.

[0146]

[11] The annotation GUI 18 includes an operation instruction unit 38 that displays instructions on how to operate the annotation GUI 18. This assists the user in performing annotations.

[0147]

[12] The annotation GUI 18 includes an exit button 40. This allows the user to signal the termination of the annotation GUI 18.

[0148] [Updating the Classification Algorithm] In response to the above issue 3, the modification of label La is fed back into the classification algorithm 14, and the classification algorithm 14 is updated. The update of the classification algorithm 14 is explained in detail below.

[0149] Figure 22 is a schematic diagram of the update of the classification algorithm in the first example. In the first example of updating the classification algorithm 14, the modification of label La is fed back to the classification algorithm 14, and an update algorithm is applied that updates the classification algorithm 14 so as to modify the classification result output from the classification algorithm 14.

[0150] The update algorithm consists of selecting the data to be modified and updating the classification algorithm 14 in accordance with the data modification. The updating of the classification algorithm 14 in accordance with the data modification may be performed repeatedly. The data referred to here represents the house image BI or the features of the house image BI. The same applies to the second example of updating the classification algorithm shown in Figure 25.

[0151] In data selection, a pre-trained image recognition model 12 is used to extract features from unknown house image BIs, which are house image BIs of a new region. The features extracted from the unknown house image BIs are then assigned labels La using a classification algorithm 14 to which a classification plane CP is applied.

[0152] In data selection, for each feature assigned a label La, k first representative points are selected. k is an integer greater than or equal to 2. Figure 22 shows an example where k = 3. A feature assigned a label La is synonymous with an unknown house image BI corresponding to that feature.

[0153] The first representative point represents the feature of each house image BI, which is represented as a location in the feature space. Figure 22 shows the state in which the first representative points P111, P112, and P113 are selected for label LaA, and the first representative points P211, P212, and P213 are selected for label LaB.

[0154] The first representative point P111, etc., is selected as a feature in the feature space that is located at a distance greater than or equal to a specified distance from the classification plane CP, and whose distance from other representative points with the same label La is greater than or equal to a specified distance. The specified distance can be appropriately defined according to the feature distribution in the feature space. The same applies to distances related to features hereafter.

[0155] When a first representative point P111, etc., is selected, it is determined whether the label La corresponding to each of the first representative points P111, etc., is correct or not. The result of the determination of the label La for the first representative points P111, etc., is fed back, and the classification algorithm 14 is updated.

[0156] In other words, in the first example of updating the classification algorithm 14, the label La of the first representative point P111 or the closest point for all feature points is changed to a new label La, the correspondence between the feature quantities and labels La is corrected, and the classification algorithm 14 is updated so that classification is performed based on the corrected correspondence between the feature quantities and labels La.

[0157] The first example of updating the classification algorithm is applied, and the updated classification algorithm 14 is applied to the classification of features extracted from the unknown house image BI, which is the house image BI of the new region. This enables the assignment of a suitable label La to the unknown house image BI, which is the house image BI of the new region.

[0158] Figure 23 is a flowchart showing the procedure for updating the classification algorithm according to the first example. The classification algorithm updating method illustrated in Figure 23 is implemented using one or more computers equipped with a processor and memory, and is executed by the processor executing a program stored in memory.

[0159] In step S200, new house image BIs for a new region, known house image BIs, are acquired. In step S202, the trained image recognition model 12 shown in Figure 22 is used to extract feature quantities for each house image BI acquired in step S200.

[0160] In step S204, when k is an integer greater than or equal to 2, k first representative points are selected for each label La in the feature distribution representing the distribution of features for each house image BI. In step S206, the determination result for the label La for the first representative points entered by the user is obtained.

[0161] In step S208, the label La of the first representative point is modified based on the determination result of the label La for the first representative point. However, in step S208, there may be cases where the label La of the first representative point is not modified.

[0162] In step S210, the determination result of the label La of the first representative point is fed back, and the classification algorithm 14 is updated based on the determination result of the label La of the first representative point. Through the following procedure, the classification algorithm 14 that is already used to assign label La to the house image BI is updated to a classification algorithm 14 that is suitable for the house image BI of the new region.

[0163] Figure 24 is a schematic diagram of the update of the classification algorithm in the second example. The update algorithm applied to the second example of updating the classification algorithm 14 consists of selecting the data to be modified and updating the classification algorithm 14 in accordance with the data modification.

[0164] In data selection, for each type of label La, m representative points are selected from the feature distribution to which the label La was assigned in the first example of updating the classification algorithm. These second representative points are farther from the first representative point P111, etc., where m is an integer greater than or equal to 2. A second representative point farther from the first representative point P111, etc., means a second representative point whose distance from the first representative point is greater than a specified distance. The specified distance is determined according to the feature distribution.

[0165] Figure 24 illustrates the state in which three second representative points P114, P115, and P116 are selected for label LaA, and second representative points P214, P215, and P216 are selected for label LaB.

[0166] In the second example of updating the classification algorithm 14, it is determined whether the label La assigned to each of the house image BIs corresponding to the second representative point P114 is correct. Figure 24 illustrates the case where the label LaA for the second representative point P116 is corrected to label LaB, and the label LaB for the second representative point P214 is corrected to label LaA.

[0167] In the second example of updating the classification algorithm 14, the result of determining the label La of the second representative point P116, etc., is fed back, and the classification algorithm 14 is updated. That is, in the second example of updating the classification algorithm 14, the correspondence between the feature quantity and label La is modified so that for all feature points, the label La of the first representative point P111, etc., or the second representative point P114, etc., which is closest in distance, is set as the new label La. The classification algorithm 14 is then updated to perform classification based on the modified correspondence between the feature quantity and label La. The second example of updating the classification algorithm may be repeated multiple times.

[0168] Figure 25 is a flowchart illustrating the procedure for updating the classification algorithm in the second example. The classification algorithm updating method illustrated in Figure 25 is implemented using one or more computers equipped with a processor and memory, and is executed by the processor executing a program stored in memory.

[0169] Steps S300, S302, S304, and S306 shown in Figure 25 are identical to steps S200, S202, S204, and S206 shown in Figure 23.

[0170] In step S308, as illustrated in Figure 25, the label La of the first representative point P111 is modified based on the determination result of the label La of the first representative point P111. In step S308, there may be cases where the label La of the first representative point P111 is not modified.

[0171] In step S310, if p is an integer greater than or equal to 2, p second representative points P114, etc., are selected for each label La. In step S312, the determination result for label La for the second representative points entered by the user is obtained.

[0172] In step S314, the label La of the second representative point is modified based on the determination result of the label La for the second representative point. However, in step S314, there may be cases where the label La of the second representative point is not modified.

[0173] Each step from step S310 to step S314 is repeated a predetermined number of times. Once each step from step S310 to step S314 has been repeated a predetermined number of times, the process proceeds to step S316.

[0174] The predetermined number of iterations is set in advance. The predetermined number of iterations may be determined based on the agreement rate between the output of the classification algorithm 14 and the determination result of label La. For example, if the agreement rate is relatively high, a relatively small number of iterations may be set, and if the agreement rate is relatively low, a relatively large number of iterations may be set.

[0175] In step S316, the determination result of the label La of the second representative point is fed back, and the classification algorithm 14 is updated based on the determination result of the label La of the second representative point. Through the following procedure, the classification algorithm 14 that is already used to assign label La to the house image BI is updated to a classification algorithm 14 that is suitable for the house image BI of the new region.

[0176] The classification algorithm update method illustrated in Figure 25 is an example of an information processing method in this disclosure, and the computer that implements the classification algorithm update method is an example of an information processing system in this disclosure.

[0177] [Specific Examples of Data Selection] The selection of representative points carried out in step S204 illustrated in Figure 23 and step S304 illustrated in Figure 25 may include the following embodiments. Hereinafter, the first representative point P111 etc. and the second representative point P114 etc. may be referred to as representative points without distinction. The term representative point means at least one of the first representative point P111 etc. and the second representative point P114 etc.

[0178] Representative points may be selected from feature points in regions where the distance Di from the classification plane CP, as shown in Figure 24, is relatively large. For example, if the distance Di from the classification plane CP to each feature is expressed using a numerical value between 0 and 1.0, then a feature with a distance Di of 0.6 or more may be selected as a representative point.

[0179] Representative points are selected for each distance Di from the classification plane CP, and for each round, features with a relatively small distance from the classification plane CP may be selected as representative points. In other words, as the number of rounds increases, features with a relatively small distance from the classification plane CP may be selected as representative points.

[0180] A round refers to a series of processes that include the selection of a representative point, the determination of the label La of the representative point, and feedback of the determination result. For example, in step S304 shown in Figure 25, if the distance Di from the classification plane CP to each feature point is expressed using a numerical value between 0 and 1.0, a feature point with a distance Di from a classification plane CP of 0.6 or more is selected as the first representative point, and in step S310, a feature point with a distance Di from a classification plane CP of 0.4 or more and 0.8 is selected as the second representative point.

[0181] Representative points may be selected from features in the neighborhood of the classification plane CP. The neighborhood of the classification plane CP is a region with a relatively small distance Di from the classification plane CP. For example, representative points may be selected from features in a region with a distance Di of 0.5 or less from the classification plane CP, or below a specified distance.

[0182] Representative points may be selected evenly across the feature space, regardless of the type of label La for each feature. It is preferable that the representative points are separated by a specified distance or more.

[0183] The second representative point P114, etc., may be selected if the label La has been modified due to an update to the classification algorithm 14. The second representative point P114, etc., may be selected if the distance from the already selected first representative point P111, etc., and the second representative point P114, etc., is greater than a specified distance.

[0184] [Specific Examples of Updating the Classification Algorithm] Updating the classification algorithm may involve modifying the classification plane CP shown in Figure 22, etc., to an arbitrary classification boundary, and updating the classification rules to which the classification plane CP is applied to classification rules to which the classification plane CP is not applied. For example, the classification algorithm 14 before the update may be updated to a classification algorithm in which the correspondence between features and labels is modified, such that the label La of the nearest representative point is the label La for each feature, and classification is realized based on the modified correspondence between features and labels.

[0185] The label La for each feature is determined according to the distance of each feature point from each representative point. That is, when multiple representative points are considered in assigning labels La to each feature point, the label La for each feature is determined using the distance of each feature point from each representative point as a weight. As described above, the correspondence between features and labels La is modified, and the classification algorithm 14 before the update may be updated to a classification algorithm that realizes classification based on the modified correspondence between features and labels La.

[0186] The following is an example of updating the classification algorithm 14 to modify the classification plane CP. When the classification algorithm 14 performs classification into two classes, normal or abnormal, the classification algorithm 14 may be updated so that the new classification plane CP is a plane that passes through and is perpendicular to the line segment connecting the average of the features labeled La representing normal and the average of the features labeled La representing abnormal. The plane perpendicular to the line segment may be a plane that passes through the center of the line segment.

[0187] Furthermore, when the classification algorithm 14 performs classification into two classes, normal or abnormal, the classification algorithm 14 may be updated so that the plane passing through the line segment connecting the centroid of the feature labeled La representing normal and the centroid of the feature labeled La representing abnormal, and which is perpendicular to the line segment, is defined as the new classification plane CP. The plane perpendicular to the line segment may be a plane passing through the center of the line segment.

[0188] [Example of application to multi-class classification with three or more values] Figure 26 is an explanatory diagram of multi-class classification with three or more values. Figure 26 illustrates an example in which one of the labels LaA, LaB, or LaC is assigned to a feature in a two-dimensional feature space.

[0189] Multiple binary classification algorithms may be created as classification algorithms for multi-class classification with three or more classes. For example, in the case of three-class classification, two binary classification algorithms may be used. In the example shown in Figure 26, the first classification plane CPA is derived between the feature to which label LaA is assigned and the feature to which label LaB or label LaC is assigned.

[0190] Next, the second classification plane CPB is derived, consisting of features assigned the label LaC and features assigned the labels LaA or LaB. Using the first classification plane CPA, the features assigned the label LaA are removed, and using the second classification plane CPB, the features assigned the label LaC are removed, resulting in features assigned the label LaB.

[0191] The calculation of the distance from the classification plane for each feature in ternary classification is performed in the same way as the calculation of the distance Di from the classification plane for each feature in binary classification.

[0192] [Effects of updating the classification algorithm] Updating the classification algorithm according to this embodiment can yield the following effects.

[0193] [1] From the features extracted from the unknown house image BI, a first representative point to be determined for label La is selected for each label La. Based on the determination result of label La of the first representative point, label La of the first representative point is modified, the correspondence between the features and label La is modified based on the modification of the first representative point, and the classification algorithm 14 is updated based on the correspondence between the modified feature point and label La. This enables the updating of the classification algorithm 14 to be suitable for the unknown house image BI.

[0194] [2] Multiple first representative points are selected. Each of the multiple first representative points is selected for its features that are far from the classification plane CP, and each of the multiple first representative points is selected for its features that are far from each other. This enables the updating of the classification algorithm 14 based on features with relatively high confidence for each label La.

[0195] [3] The label La of the nearest first representative point for each feature is set as the label La for that feature. The correspondence between the feature and the label La is modified, and the classification algorithm 14 is updated based on the modification of the correspondence between the feature and the label La. This results in an updated classification algorithm 14 that reflects the modification of the label La of the first representative point.

[0196] [4] A second representative point is selected for each label La, located away from each of the multiple first representative points. Based on the determination result of the label La of the second representative point, the label La of the second representative point is modified, the correspondence between the feature and label La is modified based on the modification of the second representative point, and the classification algorithm 14 is updated based on the correspondence between the modified feature point and label La. This enables the classification algorithm 14 to be updated to be more suitable for classifying unknown house image BI.

[0197] [5] The label La of the nearest first or second representative point for each feature is set as the label La for that feature, and the correspondence between the feature and label La is modified. Based on the modification of the correspondence between the feature and label La, the classification algorithm 14 is updated. This results in an updated classification algorithm 14 that reflects the modification of the labels La of the first and second representative points.

[0198] [6] The classification algorithm 14 is modified a predetermined number of times based on the determination result of the label La of the second representative point. This improves the classification accuracy of the updated classification algorithm 14 for unknown house image BIs.

[0199] [Example of Classification System Configuration] Figure 27 is a block diagram showing an example of the configuration of a classification system according to the embodiment. The classification system 100 includes a house image acquisition unit 102, a feature extraction unit 104, a classification unit 106, a judgment information receiving unit 114, a display control unit 116, and a label correction information receiving unit 118. The classification system 100 also includes an input device 222 and a display device 224.

[0200] The house image acquisition unit 102 acquires house image BIs such as the house image BI 11 shown in Figure 2. The feature extraction unit 104 extracts features from the house image BIs acquired using the house image acquisition unit 102. The feature extraction unit 104 includes a trained image recognition model 12 and uses the trained image recognition model 12 to extract features from the house image BIs.

[0201] The classification unit 106 comprises a classification algorithm 14, an update unit 110, and a determination information acquisition unit 112. The classification algorithm 14 assigns a binary or multi-valued label La to each house image BI based on the feature quantities of each house image BI. The classification algorithm 14 may be configured to classify feature quantities based on a classification plane CP defined in the feature space.

[0202] The judgment information acquisition unit 112 acquires label judgment information that indicates whether the label La assigned to the house image BI is correct or not. The judgment information acquisition unit 112 acquires the label judgment information entered by the user using the input device 222 via the judgment information reception unit 114. The judgment information acquisition unit 112 is an example of a component of the label correction system of this disclosure.

[0203] The update unit 110 receives feedback on whether the label La assigned to the house image BI is correct or not, and updates the classification algorithm 14. The update unit 110 may also obtain label determination information via the determination information acquisition unit 112 and update the classification algorithm 14 by feeding back the label determination information.

[0204] The judgment information receiving unit 114 receives label judgment information entered by the user using the input device 222. The judgment information receiving unit 114 transmits the label judgment information to the update unit 110. The judgment information receiving unit 114 is an example of a component of the label correction system of this disclosure.

[0205] The display control unit 116 transmits display control signals to the display device 224 that represent various types of information presented to the user from the classification system 100. The display device 224 displays the various types of information presented to the user. For example, the display device 224 displays the annotation GUI 18 shown in Figure 9, etc. Note that the display control unit 116 is an example of a component of the label correction system of this disclosure.

[0206] The label correction information receiving unit 118 receives label correction information entered by the user. The label correction information receiving unit 118 receives selection information such as the house image BI selected by the user in the annotation GUI 18 shown in Figure 9, etc., as label correction information.

[0207] The label correction information receiving unit 118 is an example of a component of the label correction system of this disclosure. The update unit 110, the determination information acquisition unit 112, and the determination information receiving unit 114 are also examples of components of the information processing system of this disclosure.

[0208] The input device 222 is operated when the user inputs various types of information. The display device 224 displays various types of information. The display device 224 displays the annotation GUI 18 as shown in Figure 9, etc. The input device 222 is operated by the user when inputting information to the annotation GUI 18.

[0209] [Example of Birdware Configuration of Classification System] Figure 28 is a block diagram showing an example of Birdware configuration of a classification system according to an embodiment. Various processing units of the classification system 100, such as the feature extraction unit 104 and the classification unit 106 shown in Figure 27, can be implemented using any computer.

[0210] Any computer may have a processor that executes programs to perform various processes of the classification system 100. Any computer may be a general-purpose computer such as a personal computer, or a computer designed for a specific purpose such as a server computer. Any computer may be a system such as a workstation, or other hardware element capable of executing programs such as a virtual machine.

[0211] At least some of the functions of the various processing units of the classification system 100 may be implemented using cloud computing. At least some of the functions of the classification system 100 may be provided as SaaS. SaaS is an abbreviation for Software as a Service.

[0212] The various processing units of the classification system 100 include a processor 202, a computer-readable medium 204, a communication interface 206, an input / output interface 208, and a bus 210. The computer-readable medium 204 is a non-temporary physical object and includes a memory 212 which is the main memory and a storage 214 which is the auxiliary memory.

[0213] The processor 202 is connected to the memory 212, storage 214, input / output interface 208, input device 222, and display device 224 via the bus 210.

[0214] Memory 212 includes RAM. Memory 212 may also include ROM. Storage 214 may be, for example, a hard disk drive, a solid-state drive, or a combination of these. Storage 214 may also include an external storage device such as removable media.

[0215] RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory. Hard disk drives can be referred to as HDDs, using the abbreviation for Hard Disk Drive. Solid state drives can be referred to as SSDs, using the abbreviation for Solid State Drive.

[0216] The computer-readable medium 204 stores programs and data that realize various functions of the classification system 100. The processor 202 executes the programs stored in the memory 212 to realize various functions of the classification system 100. The processor 202 comprehensively controls the various processing units of the classification system 100, as well as the various devices and units connected to each of the processing units, and performs various processes.

[0217] The input / output interface 208 includes a communication interface that can connect to a telecommunications line such as a local area network, and a connection interface that can connect to external devices. Examples of connection interfaces that can connect to external devices include the Universal Serial Bus and HDMI (HDMI is a registered trademark). HDMI is an abbreviation for High-Definition Multimedia Interface.

[0218] The processor 202 communicates with various processing units and various devices of the classification system 100 via the input / output interface 208, and transmits and receives various types of information.

[0219] Examples of input devices 222 include keyboards and pointing devices such as mice. Input devices 222 may include numeric keypads and various switch buttons. Input devices 222 may include voice input devices. Input devices 222 may be touch panel type input devices that are integrated with the display screen of the display device 224.

[0220] The display device 224 may be a liquid crystal display, an organic EL display, or a projector. The display device 224 may be an appropriate combination of liquid crystal displays, etc. Various types of information are displayed on the display device 224. The display device 224 is used as part of the UI when receiving input from the input device 222. The display device 224 is not limited to one; a multi-display configuration with multiple display devices is also possible.

[0221] Note that OLED is an abbreviation for organic electro-luminescence. UI is an abbreviation for User Interface.

[0222] In this embodiment, each process is performed on any computer. Furthermore, any computer may be fitted with a processor, a program, or a combination thereof to perform these processes. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other hardware element capable of running a program.

[0223] The processor 202 may be configured with one or more hardware components, and the type of hardware is not limited. The hardware of the processor 202 may include a CPU, MPU, and programmable logic devices such as FPGAs. The processor 202 may also include dedicated circuits such as ASICs that perform specific processing. The hardware of the processor 202 may include a GPU specialized for image processing, and an NPU specialized for AI processing.

[0224] The processor 202 functions as various processing units, which are various processing units that execute various processes, and various processing means, which are various processing means that execute various processes. Note that CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro-Processing Unit, FPGA is an abbreviation for Field-Programmable Gate Array, GPU is an abbreviation for Graphics Processing Unit, AI is an abbreviation for Artificial Intelligence, and NPU is an abbreviation for Neural Network Processing Unit.

[0225] The processor 202 may be configured by combining different types of hardware. The hardware of the processor 202 may include electrical circuits, which are combinations of electrical circuit elements such as semiconductor elements. Electrical circuits can be referred to as circuits.

[0226] When multiple hardware components perform any one or more processes of the processor 202, each of the multiple hardware components may be located in a physically separate device or in the same device. The order in which the processor 202 performs the processes is not limited to the order disclosed herein and may be changed as appropriate. The hardware components are configured using electrical circuits and the like, which are combinations of circuit elements such as semiconductor elements.

[0227] Furthermore, this embodiment may be implemented using hardware, software, firmware, microcode, or a combination thereof. The software, firmware, and microcode are configured by applying a program. For example, the program may be a group of program modules, and the functions of the software, etc., may be implemented by applying a processor that performs each function.

[0228] The program may consist of program code and multiple code segments stored on one or more non-temporary computer-readable media, such as memory media and storage devices. The program may be divided and stored on multiple non-temporary computer-readable media located on devices that are physically separated from each other.

[0229] Program code, or code segment, may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, instructions, data structures, or program statements. Program code, or code segment, may be connected to other code segments or hardware circuits by sending or receiving information, data, arguments, parameters, or memory contents.

[0230] In this embodiment, an example is the investigation of the damage to buildings in the event of a disaster such as an earthquake. However, the same configuration as in this embodiment may also be applied to inspections of changes in buildings over time, investigations of the asset value of buildings, and completion inspections of new construction or renovations.

[0231] The technical scope of the present invention is not limited to the scope described in the embodiments above. The configurations and other elements in each embodiment can be appropriately combined with those in each embodiment without departing from the spirit of the present invention.

[0232] 10 House Collapse Judgment AI 12 Trained Image Recognition Model 14 Classification Algorithm 16 Classification Result 18 Annotation GUI 30 House Image Display Unit 30A House Image Display Unit 30B House Image Display Unit 30C House Image Display Unit 30D House Image Display Unit 30E House Image Display Unit 30F House Image Display Unit 30G House Image Display Unit 30H House Image Display Unit 30I House Image Display Unit 30J House Image Display Unit 32 Label Display Unit 33 Icon 33A Icon 33B Icon 33C Icon 34 Sample Display Unit 36 ​​Information Assignment Unit 38 Operation Instruction Unit 40 Exit Button 42 Remaining Count Display Unit 44 Screen Transition Button 44A Back Button 44B Next Button 44C Exit Button 46 Next Button 48 Exit button 50 Cursor 52 Update button 54 Frame 56 Checkbox 57 Checkmark 58 Update button 60 Form 62 Radio button 62A Radio button 62B Radio button 64 Checkbox 66 Combo box 66A Checkbox 66B Checkbox 100 Classification system 102 House image acquisition unit 104 Feature extraction unit 106 Classification unit 110 Update unit 112 Judgment information acquisition unit 114 Judgment information reception unit 116 Display control unit 118 Label correction information reception unit 202 Processor 204 Computer-readable medium 206 Communication interface 208 Input / output interface 210 Bus 212 Memory 214 Storage 222 Input device 224 Display device AIm Aerial image BI House image BI11 House image BI12 House image BI13 House image BI21 House image BI22 House image BI23 House image CP Classification plane CPA First classification plane CPB Second classification plane Di Distance Dr Drone JI1 Damage estimation information JI2 Damage estimation information JI3 Damage estimation information MI Map information La Label LaA Label LaB Label LaC Label P11 Feature P12 Feature P13 Feature P21 Feature P22 Feature P23 Feature P111 First representative point P112 First representative point P113 First representative point P114 Second representative point P115 Second representative point P116 Second representative pointP211 First representative point P212 First representative point P213 First representative point P214 Second representative point P215 Second representative point P216 Second representative point SuA Survey area TD Training data S10 to S14 Each step of the classification method S100 to S106 Each step of the label correction method S200 to S210 Classification algorithm update method S300 to S316 Classification algorithm update method

Claims

1. A label correction system comprising: a processor; a memory storing instructions to be executed by the processor, wherein the processor executes the instructions to display candidate images of a plurality of images, each classified into one of two or more classes and assigned a label corresponding to the class, and which are candidates for which the label should be corrected; accepts the selection of a label correction target image from among the candidate images; and accepts the correction of the label assigned to the label correction target image for each label correction target image.

2. The label correction system according to claim 1, wherein the processor accepts the selection of the displayed label correction target image as the selection of the label correction target image.

3. The label correction system according to claim 1, wherein the processor displays the labels attached to the candidate images and accepts the selection of the labels attached to the candidate images as the selection of the images to be corrected.

4. The label correction system according to claim 1, wherein the processor displays a checkbox for each of the candidate images and accepts the selection of the checkbox as the selection of the image to be label corrected.

5. The label correction system according to claim 1, wherein the processor displays a pair of a sample image for each label and the label of the sample image.

6. The label correction system according to claim 1, wherein the processor displays all class labels for each candidate image.

7. The label correction system according to claim 3, wherein the processor displays the label by applying a different display mode to each label.

8. The label correction system according to claim 1, wherein the processor receives user input of additional information for each of the label correction target images and displays the additional information.

9. The label modification system according to claim 1, wherein the processor displays user operation information representing an operation performed by the user.

10. The label correction system according to claim 1, wherein the processor displays the evaluation values ​​of the labels for each of the candidate images in order when displaying a plurality of candidate images.

11. The label correction system according to claim 1, wherein the processor changes the display mode of the label to be corrected image.

12. A label modification method comprising a computer having a processor and a memory in which instructions executed by the processor are stored, the computer performs the steps of: displaying candidate images of a plurality of images classified into one of two or more classes and assigned labels corresponding to the class, wherein the candidate images are candidates for which the labels will be modified; accepting the selection of a label modification target image from among the candidate images; and accepting the modification of the label assigned to the label modification target image for each label modification target image.

13. A program for a computer comprising a processor and memory for storing instructions executed by the processor, the program having the following functions: a function to display candidate images of a plurality of images classified into one of two or more classes and assigned labels corresponding to the classes, wherein the candidate images are candidates for which the labels should be modified; a function to accept the selection of a label-modification target image from among the candidate images; and a function to accept the modification of the label assigned to each label-modification target image.

14. A non-temporary and computer-readable recording medium on which the program described in claim 13 is recorded.

Images