Information processing method, information processing device, and information processing program

Abstract

An information processing method according to the present disclosure identifies an abnormal state of a device, wherein, on the basis of a captured image indicating the operating state of the device, a computer inputs into a generative AI model a first command statement indicating a first generation instruction for generating first identification information identifying the abnormal state of the device, and outputs the first identification information generated by the generative AI model.

Description

Information Processing Method, Information Processing Apparatus, and Information Processing Program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and an information processing program.

[0002] Conventionally, when monitoring and notifying the lighting state of the lamps on the operation panel of a device such as a laboratory in an unmanned state using a camera, a machine learning model is used to classify (annotate) the lighting state of the lamps in advance, and the abnormal state of the device is identified based on whether it matches the annotated pattern. A technique has been disclosed.

[0003] Japanese Unexamined Patent Application Publication No. 2010-266424, Japanese Unexamined Patent Application Publication No. 2006-84496

[0004] However, when the number of lamps increases or the lighting state becomes complicated / diversified, it is necessary to perform annotation for machine learning on the lighting state of each lamp, so there is room for further improvement.

[0005] An object of the present disclosure is to provide an information processing method, an information processing apparatus, and an information processing program that can more easily identify an abnormal state of a device as compared with the prior art.

[0006] To achieve the above object, the information processing method of the present disclosure is an information processing method for identifying an abnormal state of a device, wherein a computer inputs a first command sentence indicating a first generation instruction for generating first identification information for identifying the abnormal state of the device based on an imaging image indicating the operating state of the device into a generation AI model, and outputs the first identification information generated by the generation AI model.

[0007] Figure 1 is a schematic diagram showing an example of an information processing system according to the embodiment. Figure 2 is a functional block diagram showing an example of the functional configuration of a user terminal according to the embodiment. Figure 3 is a functional block diagram showing an example of the functional configuration of a server according to the embodiment. Figure 4 is a schematic diagram for explaining device information according to the embodiment. Figure 5 is a schematic diagram for explaining a sample image according to the embodiment. Figure 6 is a schematic diagram for explaining the explanatory text corresponding to the sample image according to the embodiment. Figure 7 is a schematic diagram for explaining the first command statement according to the embodiment. Figure 8 is a schematic diagram for explaining the first specific information according to the embodiment. Figure 9 is a flowchart showing an example of the processing flow executed by the server according to the embodiment. Figure 10 is a flowchart showing an example of the processing flow executed by a user terminal according to the embodiment. Figure 11 is a functional block diagram showing an example of the functional configuration of a server according to the first modified example. Figure 12 is a schematic diagram for explaining the second command statement according to the first modified example. Figure 13 is a schematic diagram for explaining the second specific information according to the first modified example. Figure 14 is a schematic diagram for explaining the third command statement according to the first modified example. Figure 15 is a schematic diagram for explaining the third specific information according to the first modified example. Figure 16 is a schematic diagram illustrating the fourth instruction statement related to the first modified example. Figure 17 is a schematic diagram illustrating the fourth specific information related to the first modified example. Figure 18 is a flowchart showing an example of the processing flow executed by the server related to the first modified example. Figure 19 is a functional block diagram showing an example of the functional configuration of the server related to the third modified example. Figure 20 is a flowchart showing an example of the processing flow executed by the server related to the third modified example. Figure 21 is a schematic diagram illustrating a sample image related to the fourth modified example. Figure 22 is a block diagram showing an example of the hardware configuration of the information processing system related to the embodiment and the modified example.

[0008] The information processing method, information processing apparatus, and information processing program according to the embodiments of this disclosure will be described in detail below with reference to the attached drawings.

[0009] (Embodiment) Figure 1 is a schematic diagram showing an example of an information processing system according to an embodiment. As shown in Figure 1, the information processing system 100 includes a camera 1, a server 2, and a user terminal 3. The camera 1, server 2, and user terminal 3 are connected to each other so as to be able to communicate via, for example, a network.

[0010] Camera 1 captures images of the device. For example, Camera 1 captures images of the operating status of a device managed by the user. Camera 1 transmits the captured images to Server 2 via the network.

[0011] Server 2 identifies the abnormal state of the device based on the captured image transmitted by camera 1 and outputs the identified information to user terminal 3. Server 2 is an example of an information processing device. Details of these processes will be described later.

[0012] User terminal 3 is a terminal used by users such as administrators who manage the device captured by camera 1 or administrators who manage server 2. User terminal 3 can be various mobile devices such as notebook PCs, tablet devices, or smartphones as appropriate. Alternatively, a stationary terminal such as a regular personal computer may be used as user terminal 3. User terminal 3 receives abnormal status information about the device identified by server 2.

[0013] Figure 2 is a functional block diagram showing an example of the functional configuration of a user terminal 3 according to an embodiment. The user terminal 3 comprises a communication unit 31, an input unit 32, a display unit 33, a storage unit 34, and a control unit 35. The communication unit 31, input unit 32, display unit 33, storage unit 34, and control unit 35 are connected to each other via a bus or the like for communication.

[0014] The communication unit 31 is a communication interface that communicates with the server 2. For example, the communication unit 31 transmits device information input by the input unit 32 to the server 2. For example, the communication unit 31 receives specific information transmitted by the server 2. For example, the communication unit 31 transmits feedback information to the server 2 indicating feedback on the specific information input by the input unit 32. The communication unit 31 also communicates various information with external devices. Details of the device information, specific information, and feedback information will be described later.

[0015] The input unit 32 accepts various operations from the user. The input unit 32 may be, for example, a keyboard and a pointing device such as a mouse, a microphone, etc. The display unit 33 displays or outputs various information. The display unit 33 may be a display that shows various information, a speaker that outputs various sounds, etc. The input unit 32 and the display unit 33 may be integrated into a touch panel. Alternatively, the input unit 32 and the display unit 33 may be configured separately from the user terminal 3. In this case, the input unit 32 and the display unit 33 and the user terminal 3 may be connected in a way that allows them to communicate with each other.

[0016] The storage unit 34 stores various types of data. The storage unit 34 may be, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disc. The storage unit 34 may also be a storage device located outside the user terminal 3. Furthermore, the storage unit 34 may be a storage medium on which programs and various types of information are downloaded and stored or temporarily stored via a LAN (Local Area Network) or the Internet.

[0017] The control unit 35 is a computer that performs information processing on the user terminal 3. The control unit 35 may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. It may also be implemented by a dedicated IC or other processor, i.e., by hardware. The storage unit 34 may be implemented using a combination of software and hardware. At least one of the one or more functional units included in the control unit 35 may be mounted on an external information processing device that is connected to the user terminal 3 via a network or the like in a way that allows communication.

[0018] Figure 3 is a functional block diagram showing an example of the functional configuration of Server 2 according to an embodiment. Server 2 is an example of an information processing device. In Server 2, the communication unit 21, storage unit 22, first acquisition unit 23, receiving unit 24, first generation unit 25, second acquisition unit 26, and output unit 27 are connected to each other via a bus or the like so as to be able to communicate.

[0019] The communication unit 21 is a communication interface that communicates with the camera 1 and the user terminal 3. The communication unit 21 also communicates various information with external devices.

[0020] The storage unit 22 stores various types of information. The storage unit 22 may be, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disk. The storage unit 22 may also be a storage device located outside the server 2. Alternatively, the storage unit 22 may be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN or the internet.

[0021] The first acquisition unit 23, the receiving unit 24, the first generation unit 25, the second acquisition unit 26, and the output unit 27 are computers that perform information processing on the server 2. The first acquisition unit 23, the receiving unit 24, the first generation unit 25, the second acquisition unit 26, and the output unit 27 may be implemented by having a processor such as a CPU execute a program, i.e., by software. They may also be implemented by a dedicated processor such as an IC, i.e., by hardware. The storage unit 22 may be implemented using a combination of software and hardware. At least one of the first acquisition unit 23, the receiving unit 24, the first generation unit 25, the second acquisition unit 26, and the output unit 27 may be mounted on an external information processing device that is connected to the server 2 via a network or the like in a way that allows communication.

[0022] The first acquisition unit 23 acquires device information. Specifically, the first acquisition unit 23 acquires device information transmitted by the user terminal 3. Here, the device information will be explained using Figure 4.

[0023] Figure 4 is a schematic diagram illustrating the device information according to the embodiment. Figure 4 shows a display screen 40 on which the user terminal 3 inputs device information. The display screen 40 is output to, for example, the display unit 33 of the user terminal 3. The display screen 40 outputs a first reception unit 41, a second reception unit 42, and a third reception unit 43. The first reception unit 41 is, for example, a button with the text "Sample Image Upload Field". The second reception unit 42 is, for example, a button with the text "Sample Image Description Input Field". The third reception unit 43 is, for example, a button with the text "Operation Manual Upload Field".

[0024] Device information is reference information that Server 2 uses to identify abnormal conditions in the device. In order for Server 2 to identify abnormal conditions in the device, information on the normal and abnormal operating states of the device is necessary. Therefore, the user inputs device information using the display screen 40 and sends it to Server 2. Server 2 obtains device information corresponding to various states indicating the operating status of the device from the device's operation manual or user input and inputs it into the generating AI model. For example, device information includes a sample image of the device, a description corresponding to the sample image, and the device's operation manual. The sample image of the device is also called the reference image that serves as the basis for the device. Server 2 may also input prompts (command statements) containing information about the device into the generating AI model, or it may input the device information into the generating AI model as training data.

[0025] For example, the user inputs a sample image of the device into the first reception unit 41. The user also inputs a descriptive text corresponding to the sample image into the second reception unit 42. Here, the sample image and the descriptive text corresponding to the sample image will be explained using Figures 5 and 6.

[0026] Figure 5 is a schematic diagram illustrating a sample image according to the embodiment. The sample image 50 shown in Figure 5 is an image showing the operation panel of a device managed by a user. In this embodiment, the sample image 50 is an image showing the operating status of the operation panel in order to identify an abnormal state of the device, but it is not limited to this. The sample image 50 may be any image showing the operating status of the device, for example, an image showing the operating status of the device's meter.

[0027] Figure 6 is a schematic diagram illustrating the explanatory text corresponding to the sample image 50 according to the embodiment. The explanatory text 60 shown in Figure 6 contains the following text: "• At the top of the image, there are 10 rows of square buttons arranged neatly in a grid of 2 columns. • At the bottom of the image, there are 4 rows of square panels arranged neatly in a grid of 2 columns. • Of the buttons at the top of the image, the first and second from the left in the first row are lit yellow. • Of the buttons at the top of the image, the third from the left in the first row is lit red."

[0028] Returning to Figure 4, the explanation continues. After inputting information into the first reception unit 41 and the second reception unit 42, the user inputs the device operation manual into the third reception unit 43. The communication unit 31 of the user terminal 3 also transmits the input device information to the server 2. The first acquisition unit 23 then acquires the device information transmitted by the user terminal 3. This allows the server 2 to recognize the operating status of the device managed by the user.

[0029] Returning to Figure 3, let's continue the explanation. The receiving unit 24 receives the captured image. Specifically, the receiving unit 24 receives the captured image transmitted by the camera 1. The captured image is an image that shows the operating status of the device. The captured image includes, for example, the state of the operation panel which can identify the operating status of the device, and the date and time the image was taken. However, the captured image is not limited to this.

[0030] The first generation unit 25 generates a first command statement indicating a first generation instruction to cause a generation AI model to generate first specific information that identifies an abnormal state of the device, based on the captured image. Specifically, the first generation unit 25 generates a first command statement indicating a first generation instruction to cause a generation AI model to generate first specific information that identifies an abnormal state of the device, based on the captured image received by the receiving unit 24. For example, the first generation unit 25 generates a first command statement indicating a first generation instruction to cause a language model capable of providing first specific information that identifies an abnormal state of the device to generate it, based on the captured image received by the receiving unit 24.

[0031] Generative AI models are language models that enable computer programs and systems to perform tasks such as question answering and text generation, for example, in the field of natural language processing (NLP). Examples of language models include large language models (LLMs), generative language models (such as GPT (Generative Pretrained Transformer)), representational language models (such as Bidirectional Encoder Representations from Transformers (BERT)), and / or any other type of language model.

[0032] Here, the first instruction statement will be explained using Figure 7. Figure 7 is a schematic diagram for explaining the first instruction statement according to the embodiment. The first instruction statement 70 shown in Figure 7 contains the following text: "• This image shows the control panel of the device with operation buttons lined up. • Of the 10 columns x 2 rows of buttons at the top, the first and second buttons from the left in the first row should be lit yellow, and the third button should be lit red. • Is this image normal or abnormal?"

[0033] Returning to Figure 3, the explanation continues. The second acquisition unit 26 inputs a first command statement indicating a first generation instruction to generate first specific information that identifies an abnormal state of the device based on the captured image to the generation AI model, and acquires the first specific information generated by the generation AI. Specifically, the second acquisition unit 26 inputs a first command statement, generated by the first generation unit 25, indicating a first generation instruction to generate first specific information that identifies an abnormal state of the device based on the captured image to the generation AI, and acquires the first specific information generated by the generation AI model. For example, the second acquisition unit 26 inputs a first command statement, generated by the first generation unit 25, indicating a first generation instruction to generate first specific information that identifies an abnormal state of the device based on the captured image to the language model, and acquires the first specific information generated by the language model.

[0034] The output unit 27 outputs the first specific information. Specifically, the output unit 27 outputs the first specific information acquired by the second acquisition unit 26 to the user terminal 3. The communication unit 31 of the user terminal 3 receives the first specific information output by the server 2. The control unit 35 of the user terminal 3 then outputs the first specific information received by the communication unit 31 to the display unit 33. The first specific information will now be explained using Figure 8.

[0035] Figure 8 is a schematic diagram illustrating the first identification information according to the embodiment. The first identification information 80 shown in Figure 8 displays text 81 with the content "• This image is abnormal." when the operating state of the device is identified as abnormal. The first identification information 80 also displays text 82 with the content "• This image is normal." when the operating state of the device is identified as normal. The first identification information 80 may also output the identification result of the first identification information 80 and the reason for the identification of the first identification information 80 in association with each other. Here, the identification result of the first identification information 80 is, for example, text 81 or text 82 shown in Figure 8. The reason for the identification of the first identification information 80 is the reason for the identification corresponding to the identification result.

[0036] Furthermore, the user sends first feedback information to the server 2 indicating whether the first specific information 80 is correct or incorrect. The first feedback information indicates the user's feedback corresponding to the first specific information 80. For example, the input unit 32 of the user terminal 3 receives first feedback information from the user indicating whether the first specific information 80 is correct or incorrect. The communication unit 31 of the user terminal 3 also sends the first feedback information received by the input unit 32 to the server 2.

[0037] Then, the first acquisition unit 23 of the server 2 associates the first feedback information transmitted by the user terminal 3 with the first specific information 80 and stores it in the storage unit 22. Here, for example, if the first specific information 80 is correct, the first feedback information is text that says "The first specific information is correct." On the other hand, if the first specific information 80 is incorrect, the first feedback information is text that says "The first specific information is incorrect."

[0038] Figure 9 is a flowchart showing an example of the processing flow executed by Server 2 according to this embodiment.

[0039] The first acquisition unit 23 acquires device information transmitted by the user terminal 3 (step S91). Subsequently, the receiving unit 24 receives the captured image transmitted by the camera 1 (step S92). Then, the first generation unit 25 generates a first command statement indicating a first generation instruction that causes the generation AI model to generate first specific information for identifying the abnormal state of the device, based on the captured image received by the receiving unit 24 (step S93).

[0040] Next, the second acquisition unit 26 inputs a first command statement to the generation AI model indicating a first generation instruction to generate first specific information that identifies an abnormal state of the device based on the captured image generated by the first generation unit 25, and acquires the first specific information generated by the generation AI model (step S94). Subsequently, the output unit 27 outputs the first specific information acquired by the second acquisition unit 26 to the user terminal 3 (step S95). When the processing in step S95 is completed, this process executed by the server 2 is completed.

[0041] FIG. 10 is a flowchart showing an example of the processing flow executed by the user terminal 3 according to the embodiment. The flowchart shown in FIG. 10 explains the content of the first feedback information transmitted by the user terminal 3 to the server 2.

[0042] The control unit 35 outputs the first specific information received by the communication unit 31 to the display unit 33 (step S101). Subsequently, the input unit 32 inputs first feedback information indicating the correct / incorrect result of the first specific information 80 by the user (step S102). Subsequently, the communication unit 31 transmits the first feedback information input by the input unit 32 to the server 2 (step S103). When the processing of step S103 ends, this processing executed by the user terminal 3 ends.

[0043] As described above, the information processing method of the embodiment is an information processing method for specifying an abnormal state of a device, and a computer inputs a first command sentence indicating a first generation instruction for generating first specific information for specifying the abnormal state of the device based on a captured image indicating the operating state of the device into a generation AI model, and outputs the first specific information generated by the generation AI model. Also, in the information processing method of the embodiment, the computer acquires device information corresponding to various states indicating the operating state of the device based on the device operation manual or user input, and inputs it into the generation AI model.

[0044] For example, the conventional technology for specifying an abnormal state of a device uses a machine learning model to classify (annotate) the lighting state of a lamp in advance, and specifies the abnormal state of the device based on whether it matches the annotated pattern. Therefore, when the number of lamps increases or the lighting state becomes complicated / diversified, it is necessary to perform annotation for machine learning on the lighting state of each lamp, so there is room for further improvement.

[0045] In contrast, the information processing method of the embodiment does not require performing annotation for machine learning by causing the generation AI model to generate first specific information for specifying the abnormal state of the device based on a captured image indicating the operating state of the device. Thereby, the information processing method of the embodiment specifies the abnormal state of the device from the captured image indicating the operating state of the device.

[0046] Incidentally, the above-described embodiments can be appropriately modified and implemented by changing part of the configuration or function of each of the above-described devices. Therefore, some modification examples according to the above-described embodiments will be described below as other embodiments. In the following, the points different from the above-described embodiments will be mainly described, and detailed descriptions of the points common to the already described content will be omitted.

[0047] (First Modification Example) In the above-described embodiment, the device information has been described in a form including a sample image of the device, an explanatory text corresponding to the sample image, and an operation manual of the device. In the first modification example, the case where the device information includes a sample image of the device and an operation manual of the device will be described hereinafter with respect to the content processed by the server 2. That is, the operation information in the first modification example does not include the explanatory text corresponding to the sample image.

[0048] FIG. 11 is a functional block diagram showing an example of the functional configuration of the server 2 according to the first modification example. The server 2, the communication unit 21, the storage unit 22, the first acquisition unit 23, the reception unit 24, the first generation unit 25, the second acquisition unit 26, the output unit 27, the second generation unit 28, and the third acquisition unit 29 are communicably connected via a bus or the like.

[0049] The second generation unit 28 generates a second command sentence indicating a second generation instruction for causing the AI model to generate second specific information for specifying the operating state of the device based on the device information. Specifically, the second generation unit 28 generates a second command sentence indicating a second generation instruction for causing the AI model to generate second specific information for specifying the operating state of the device based on the device information acquired by the first acquisition unit 23.

[0050] Here, the second instruction will be explained using Figure 12. Figure 12 is a schematic diagram illustrating the second instruction related to the first modification. The second instruction 120 shown in Figure 12 contains the text: "This photograph is of the control panel of a device. You are a monitor who monitors the illumination status of the lights on this panel and reports any abnormalities. In order to correctly determine the illumination status, you need to correctly understand the configuration of this panel. How many buttons are there in total in this photograph, and how many buttons are lit?" Further wording may be included to indicate that the instruction is a preliminary question preceding this question, so that the answer generated by the generation AI model is more effective.

[0051] Returning to Figure 11, the explanation continues. The third acquisition unit 29 inputs a second command statement 120, which indicates a second generation instruction to generate second specific information that identifies the operating status of the device based on the device information, into the generation AI model, and acquires the second specific information generated by the generation AI model. Specifically, the third acquisition unit 29 inputs a second command statement 120, which indicates a second generation instruction to generate second specific information that identifies the operating status of the device based on the device information generated by the second generation unit 28, into the generation AI model, and acquires the second specific information generated by the generation AI model.

[0052] The output unit 27 then outputs the second specific information. Specifically, the output unit 27 outputs the second specific information acquired by the third acquisition unit 29 to the user terminal 3. The communication unit 31 of the user terminal 3 receives the second specific information output by the server 2. The control unit 35 of the user terminal 3 then outputs the second specific information received by the communication unit 31 to the display unit 33. The second specific information will now be explained using Figure 17.

[0053] Figure 13 is a schematic diagram illustrating the second specific information relating to the first modified example. The second specific information 130 shown in Figure 13 represents the case where the state of the device corresponding to the second instruction statement 120 is specified. For example, if the second specific information 130 is correct, text 131 is shown that says, "There are a total of 20 buttons, and 3 of them are lit up." Also, for example, if the second specific information 130 is incorrect, text 132 is shown that says, "There are a total of 6 buttons in this photograph, and 3 of them are lit up," or text 132 that says, "There are a total of 15 buttons in this photograph, and 2 of them are lit up."

[0054] Furthermore, the user sends second feedback information to the server 2 indicating whether the second specific information 130 is correct or incorrect. For example, the input unit 32 of the user terminal 3 receives second feedback information from the user indicating whether the second specific information 130 is correct or incorrect. The communication unit 31 of the user terminal 3 also sends the second feedback information entered by the input unit 32 to the server 2. The first acquisition unit 23 of the server 2 then stores the second feedback information sent by the user terminal 3 and the second specific information 130 in the storage unit 22, associating them.

[0055] For example, if the second specific information 130 is correct, the second feedback information will be the text "The second specific information is correct." On the other hand, if the second specific information 130 is incorrect, the second feedback information will be the text "The second specific information is incorrect. The correct information is that there are a total of 20 buttons, and 3 of them are lit up."

[0056] Server 2 continues the following process in order to further learn the operating status of the device information. Specifically, the second generation unit 28 generates a third instruction statement that indicates a third generation instruction to cause the generating AI model to generate third specific information that identifies the operating status of the device, based on the device information acquired by the first acquisition unit 23.

[0057] Here, the third instruction will be explained using Figure 14. Figure 14 is a schematic diagram for explaining the third instruction related to the first modified example. The third instruction 140 shown in Figure 14 contains the following text: "• Of the buttons arranged in two rows, which lamp is lit? • Please tell us in the format of the second from the left in the first row. (Please also tell us the color if you know it.)"

[0058] The third acquisition unit 29 inputs a third command statement 140 to the generation AI model, which indicates a third generation instruction to generate third specific information that identifies the operating status of the device based on the device information generated by the second generation unit 28, and acquires the third specific information generated by the generation AI model. The output unit 27 then outputs the third specific information acquired by the third acquisition unit 29 to the user terminal 3. The communication unit 31 of the user terminal 3 receives the third specific information output by the server 2. The control unit 35 of the user terminal 3 then outputs the third specific information received by the communication unit 31 to the display unit 33. The third specific information will now be explained using Figure 15.

[0059] Figure 15 is a schematic diagram illustrating the third specific information relating to the first modified example. The third specific information 150 shown in Figure 15 indicates the state of the device corresponding to the third instruction statement 140. For example, if the third specific information 150 is correct, text 151 is shown which contains the following: "• There are 20 buttons in total, and 3 of them are lit. • The 1st, 2nd, and 3rd lamps from the left in the first row are lit. The colors from left to right are orange, orange, and red." Also, for example, if the third specific information 150 is incorrect, text 152 is shown which contains the following: "• The 1st, 4th, and 6th lamps from the left in the first row are lit. • The 1st and 2nd lamps from the left in the first row are lit, and the color is orange."

[0060] Furthermore, the user sends third feedback information to the server 2 indicating whether the third specific information 150 is correct or incorrect. For example, the input unit 32 of the user terminal 3 receives second feedback information from the user indicating whether the third specific information 150 is correct or incorrect. The communication unit 31 of the user terminal 3 also sends the third feedback information entered by the input unit 32 to the server 2. The first acquisition unit 23 of the server 2 then associates the third feedback information sent by the user terminal 3 with the third specific information 150 and stores it in the storage unit 22.

[0061] For example, if the third specific information 150 is correct, the third feedback information will be the text "The third specific information is correct." On the other hand, if the third specific information 150 is incorrect, the third feedback information will be the text "The third specific information is incorrect. The correct information is: There are a total of 20 buttons, and 3 of them are lit. The first, second, and third lights from the left in the first row are lit. The colors from left to right are orange, orange, and red."

[0062] Server 2 continues the following process in order to further learn the operating status of the device information. Specifically, the second generation unit 28 generates a fourth instruction statement that indicates a fourth generation instruction to cause the generation AI model to generate fourth specific information that identifies the operating status of the device, based on the device information acquired by the first acquisition unit 23.

[0063] Here, the fourth instruction will be explained using Figure 16. Figure 16 is a schematic diagram for explaining the fourth instruction related to the first modified example. The fourth instruction 160 shown in Figure 16 contains the following text: "• Of the buttons arranged in two rows, the leftmost button in the first row and the second and third buttons from the left in the first row should be lit. If those lights are off or any other lights are lit, it is abnormal. • Is the state of the panel in this photograph abnormal?"

[0064] The third acquisition unit 29 inputs a fourth command statement 160 to the generation AI model, which indicates a fourth generation instruction to generate fourth specific information that identifies the operating status of the device based on the device information generated by the second generation unit 28, and acquires the fourth specific information generated by the generation AI model. The output unit 27 then outputs the fourth specific information acquired by the third acquisition unit 29 to the user terminal 3. The communication unit 31 of the user terminal 3 receives the fourth specific information output by the server 2. The control unit 35 of the user terminal 3 then outputs the fourth specific information received by the communication unit 31 to the display unit 33. The fourth specific information will now be explained using Figure 17.

[0065] Figure 17 is a schematic diagram illustrating the fourth specific information relating to the first modified example. The fourth specific information 170 shown in Figure 17 represents the case where the state of the device corresponding to the fourth instruction statement 160 is specified. For example, if the fourth specific information 170 is correct, text 171 is shown that reads, "• The state of the panel in this photograph is normal. • The leftmost button in the first row and the second and third buttons from the left are lit." Also, for example, if the fourth specific information 170 is incorrect, text 172 is shown that reads, "• Looking at the panel in the photograph, the leftmost button in the first row and the second button from the left are lit, but the third button is not lit. • Also, the fourth button from the left is lit. • Therefore, this is not a normal state but an abnormal state."

[0066] Furthermore, the user sends fourth feedback information to the server 2 indicating whether the fourth identification information 170 is correct or incorrect. For example, the input unit 32 of the user terminal 3 receives fourth feedback information from the user indicating whether the fourth identification information 170 is correct or incorrect. The communication unit 31 of the user terminal 3 also sends the fourth feedback information entered by the input unit 32 to the server 2. The first acquisition unit 23 of the server 2 associates the fourth feedback information sent by the user terminal 3 with the fourth identification information 170 and stores it in the storage unit 22. The server 2 then receives the captured image transmitted by the camera 1 and performs processing to identify the abnormal state of the device based on the captured image.

[0067] For example, if the fourth specific information 170 is correct, the fourth feedback information will be the text "The fourth specific information is correct." On the other hand, if the fourth specific information 170 is incorrect, the fourth feedback information will be the text "The fourth specific information is incorrect. The panel in this photo is in normal condition. The leftmost button in the first row and the second and third buttons from the left are lit."

[0068] In the first modified example, the server 2 was described as learning the state of the sample image using three commands, but it is not limited to this. The number of commands used to train the server 2 can be set as appropriate. By doing so, the server 2 can improve its accuracy in identifying abnormal states of the device by learning from the sample images. The second command 120 may also include the third command 140 and the fourth command 160.

[0069] Figure 18 is a flowchart showing an example of the processing flow executed by Server 2 according to the first modified example. Note that the content of steps S91 and S92 shown in Figure 18 is the same as the content of steps S91 and S92 shown in Figure 9, so the explanation is omitted.

[0070] In step S181, the second generation unit 28 generates a second command statement 120 indicating a second generation instruction to cause the generation AI model to generate second specific information that identifies the operating status of the device, based on the device information acquired by the first acquisition unit 23 (step S181). Subsequently, the third acquisition unit 29 inputs the second command statement generated by the second generation unit 28, indicating a second generation instruction to cause the generation AI model to generate second specific information that identifies the operating status of the device, based on the device information, to the generation AI model and acquires the second specific information 130 generated by the generation AI model (step S182). Subsequently, the output unit 27 outputs the second specific information 130 acquired by the third acquisition unit 29 to the user terminal 3 (step S183). Subsequently, the first acquisition unit 23 associates the second feedback information transmitted by the user terminal 3 with the second specific information 130 and stores it in the storage unit 22 (step S184).

[0071] Next, the second generation unit 28 generates a third command statement indicating a third generation instruction to generate a generation AI model for third specific information that identifies the operating status of the device, based on the device information acquired by the first acquisition unit 23 (step S185). Next, the third acquisition unit 29 inputs the third command statement 140 generated by the second generation unit 28, which indicates a third generation instruction to generate third specific information that identifies the operating status of the device, based on the device information, into the generation AI model and acquires the third specific information 150 generated by the generation AI model (step S186). Next, the output unit 27 outputs the third specific information 150 acquired by the third acquisition unit 29 to the user terminal 3 (step S187). Next, the first acquisition unit 23 associates the third feedback information transmitted by the user terminal 3 with the third specific information 150 and stores it in the storage unit 22 (step S188).

[0072] Next, the second generation unit 28 generates a fourth command statement indicating a fourth generation instruction to cause the generation AI model to generate fourth specific information that identifies the operating status of the device, based on the device information acquired by the first acquisition unit 23 (step S189). Next, the third acquisition unit 29 inputs the fourth command statement 160 generated by the second generation unit 28, which indicates a fourth generation instruction to cause the generation AI model to generate fourth specific information that identifies the operating status of the device, based on the device information, to the generation AI model and acquires the fourth specific information 170 generated by the generation AI model (step S190). Next, the output unit 27 outputs the fourth specific information 170 acquired by the third acquisition unit 29 to the user terminal 3 (step S191). Next, the first acquisition unit 23 associates the fourth feedback information transmitted by the user terminal 3 with the fourth specific information 170 and stores it in the storage unit 22 (step S192).

[0073] As described above, the first modified information processing method involves the computer inputting a second instruction statement into the generation AI model, which indicates a second generation instruction to generate second specific information that identifies the operating status of the device based on the device information. Furthermore, the first modified information processing method involves the computer associating the second specific information with the feedback information corresponding to the second specific information and storing it in the storage unit 22.

[0074] As a result, the information processing method of the first modified example can learn a generating AI model so that it can identify the operating status of the device based on the device information. For example, if the second identification information output by the generating AI model is incorrect, the server 2 can learn correct answers one after another by associating the second identification information with user feedback information and storing it, thereby improving the accuracy rate of the second identification information.

[0075] (Second Modification) In the first modification, the feedback information was described in a form in which the server 2 associates the second specific information with the corresponding feedback information and stores it in the storage unit 22 when the user confirms the second specific information and the user inputs the feedback information into the user terminal 3. However, the system is not limited to this. For example, feedback information may be output using a different generating AI model than the generating AI model that outputs the second specific information.

[0076] For example, Server 2 inputs the second specific information to a generating AI model that determines the accuracy of the second specific information. The generating AI model that determines the accuracy of the second specific information determines the accuracy of the second specific information by comparing it with a predetermined answer, for example. Here, the predetermined answer is, for example, the explanatory text 60 shown in Figure 6.

[0077] The generating AI model that determines the accuracy of the second specific information compares the explanatory text 60 shown in Figure 6 with the second specific information. If the content described in the second specific information is included in the explanatory text 60, it determines that the second specific information is correct and outputs feedback information to the generating AI model that outputs the second specific information based on the result of this determination. On the other hand, the generating AI model that verifies the accuracy of the second specific information compares the explanatory text 60 shown in Figure 6 with the second specific information. If the content described in the second specific information differs from the explanatory text 60, it determines that the second specific information is incorrect and outputs feedback information to the generating AI model that outputs the second specific information, combining the result of this determination with the result of a correct answer.

[0078] As a result, for example, if the second specific information output by the generating AI model is incorrect, server 2 can learn correct answers one after another by associating and storing the feedback information from the generating AI model that determines the accuracy of the second specific information with the second specific information, thereby improving the accuracy rate of the second specific information.

[0079] (Third Modification) For example, the server 2 may receive an image that is not clear. If the image is not clear, the server 2 may not be able to identify the abnormal condition of the device. Therefore, the server 2 of the second modification will be described in which it adjusts the received image.

[0080] Figure 19 is a functional block diagram showing an example of the functional configuration of server 2 according to the third modified example. Server 2 consists of a communication unit 21, a storage unit 22, a first acquisition unit 23, a receiving unit 24, a first generation unit 25, a second acquisition unit 26, an output unit 27, a second generation unit 28, a third acquisition unit 29, and an image adjustment unit 30, all of which are connected to each other via a bus or the like.

[0081] The image adjustment unit 30 adjusts the captured image based on a reference image. Specifically, the image adjustment unit 30 adjusts the parameters of the captured image received by the receiving unit 24 based on a sample image included in the device information acquired by the first acquisition unit 23.

[0082] For example, the image adjustment unit 30 compares the sample image with the captured image and, if the field of view of the captured image is smaller than that of the sample image, adjusts the field of view of the captured image to be enlarged so that it becomes equivalent to that of the sample image. Also, for example, the image adjustment unit 30 compares the sample image with the captured image and, if the field of view of the captured image is larger than that of the sample image, adjusts the field of view of the captured image to be reduced so that it becomes equivalent to that of the sample image.

[0083] For example, the image adjustment unit 30 compares the sample image with the captured image and, if the brightness of the captured image is lower than that of the sample image, adjusts the brightness of the captured image to be higher so that it is equal to that of the sample image. Also, for example, the image adjustment unit 30 compares the sample image with the captured image and, if the brightness of the captured image is higher than that of the sample image, adjusts the brightness of the captured image to be lower so that it is equal to that of the sample image. However, the content of the processing that the image adjustment unit 30 performs to adjust the captured image is not limited to these examples. For example, the image adjustment unit 30 may adjust parameters such as contrast, highlights, and saturation of the captured image.

[0084] Figure 20 is a flowchart showing an example of the processing flow executed by Server 2 according to the third modified example. Note that the content of the processing from steps S91 to S95 shown in Figure 20 is the same as the content of the processing from steps S91 to S95 shown in Figure 9, so the explanation is omitted.

[0085] In step S201, the image adjustment unit 30 adjusts the captured image received by the receiving unit 24 based on the sample image included in the device information acquired by the first acquisition unit 23 (step S201).

[0086] As explained above, the information processing method according to the third modified example adjusts the parameters of the captured image based on a reference image. As a result, the information processing method according to the third modified example can reduce false detections when the generating AI model identifies abnormal conditions of the device from the captured image, thereby improving the accuracy of the first identified information.

[0087] (Fourth Modification) In the embodiments described above, the operating state of the device was determined from the lamp status on the device's control panel, but the invention is not limited to this. For example, the operating state of the device may be determined from the data status of the device's analog meter.

[0088] Figure 21 is a schematic diagram illustrating a sample image relating to the fourth modified example. The sample image 121 shown in Figure 21 is an image showing the analog meter of a device managed by the user. The sample image 121 shows the analog meter 122 and the numerical value 123 corresponding to the analog meter 122. The numerical value 123 shown in Figure 21 represents, for example, the value "0111", and is also called the data state of the analog meter 122.

[0089] (Hardware Configuration) Figure 22 is a block diagram showing an example of the hardware configuration of the information processing system 100 according to the embodiment and modified example. The camera 1, server 2, and user terminal 3 of the above embodiment and modified example are interconnected by a bus 8, with a processor 4, main memory 5, auxiliary storage 6, and device I / F 7, etc., and have a hardware configuration that uses a normal computer.

[0090] The processor 4 is, for example, a CPU, and is a computing device that controls the camera 1, server 2, and user terminal 3 in the above embodiment and its modified version. The main memory 5 is, for example, RAM, and stores programs and the like that realize information processing by the processor 4. The auxiliary storage device 6 is, for example, ROM, and stores data necessary for various processes by the processor 4. The device I / F 7 is an interface connected to the communication unit, storage unit, etc., for sending and receiving data. In the camera 1, server 2, and user terminal 3 in the above embodiment and its modified version, the processor 4 reads programs from the main memory device 5 onto the auxiliary storage device 6 and executes them, thereby realizing each of the above-mentioned functional units on the computer.

[0091] Furthermore, the programs for executing the above-mentioned processes performed by the camera 1, server 2, and user terminal 3 in the above-described embodiment and modified version may be stored in an HDD (hard disk drive). Alternatively, the programs for executing the above-mentioned processes performed by the camera 1, server 2, and user terminal 3 in the above-described embodiment and modified version may be pre-installed and provided in the main memory 5.

[0092] Furthermore, the program for executing the above-described process performed by the camera 1, server 2, and user terminal 3 in the above-described embodiment and modified version may be provided as a computer program product by being stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD).

[0093] Furthermore, the program for executing the information processing performed by the camera 1, server 2, and user terminal 3 in the above embodiment and modified version may be stored on a computer connected to a network such as the Internet and provided by allowing users to download it via the network. Alternatively, the program for executing the information processing performed by the camera 1, server 2, and user terminal 3 in the above embodiment and modified version may be provided or distributed via a network such as the Internet.

[0094] According to at least one embodiment described above, it is possible to identify abnormal conditions in the device more easily than in the conventional method.

[0095] Although embodiments have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0096] (Note) The various aspects of this disclosure are described below as a summary. (1) An information processing method for identifying an abnormal state of a device, wherein a computer inputs a first command statement indicating a first generation instruction to a generation AI model to generate first specific information for identifying an abnormal state of the device based on an image captured showing the operating state of the device, and outputs the first specific information generated by the generation AI model. (2) The information processing method according to (1), wherein the computer acquires device information corresponding to various states showing the operating state of the device from the device's operation manual or user input and inputs it to the generation AI model. (3) The information processing method according to (1), wherein the computer outputs the result of identifying the first specific information and the reason for identifying the first specific information in association. (4) The information processing method according to (3), wherein the computer stores feedback information indicating user feedback corresponding to the first specific information and the first specific information in association. (5) The information processing method according to (2), wherein the computer inputs a second command statement indicating a second generation instruction to generate a second specific information that identifies the operating state of the device based on the device information to the generation AI model. (6) The information processing method according to (2), wherein the computer adjusts the parameters of the captured image based on a reference image that is included in the device information. (7) The information processing method according to (1), wherein the operating state of the device is the lamp status of the operation panel of the device or the data status of the analog meter of the device. (8) An information processing device comprising at least a processor, wherein the processor inputs a first command statement indicating a first generation instruction to generate a first specific information that identifies an abnormal state of the device based on a captured image that indicates the operating state of the device to the generation AI model, and outputs the first specific information generated by the generation AI model. (9) An information processing program for causing a computer to input a first command statement indicating a first generation instruction to generate first specific information for identifying an abnormal state of the device based on an image captured showing the operating state of the device into a generating AI model, and output the first specific information generated by the generating AI model.

[0097] 1 Camera 2 Server 3 User terminal 21 Communication unit 22 Storage unit 23 First acquisition unit 24 Receiving unit 25 First generation unit 26 Second acquisition unit 27 Output unit 28 Second generation unit 29 Third acquisition unit 30 Image adjustment unit 100 Information processing system

Claims

1. An information processing method for identifying an abnormal state of a device, comprising: a computer inputting a first command statement indicating a first generation instruction to a generation AI model that causes the computer to generate first specific information for identifying an abnormal state of the device based on an image captured showing the operating state of the device; and outputting the first specific information generated by the generation AI model.

2. The information processing method according to claim 1, wherein the computer acquires device information corresponding to various states indicating the operating status of the device from the device's operation manual or user input, and inputs this information into the generated AI model.

3. The information processing method according to claim 1, wherein the computer outputs the result of identifying the first specific information and the reason for identifying the first specific information in association with each other.

4. The information processing method according to claim 3, wherein the computer stores feedback information indicating user feedback corresponding to the first specific information in association with the first specific information.

5. The information processing method according to claim 2, wherein the computer inputs a second instruction statement indicating a second generation instruction to generate a second specific information that identifies the operating status of the device based on the device information into the generation AI model.

6. The information processing method according to claim 2, wherein the computer adjusts the parameters of the captured image based on a reference image included in the device information.

7. The information processing method according to claim 1, wherein the operating status of the device is the status of the lamps on the control panel of the device, or the data status of the analog meter of the device.

8. An information processing device comprising at least a processor, wherein the processor inputs a first instruction statement indicating a first generation instruction to a generation AI model to generate first specific information for identifying an abnormal state of the device based on an image captured showing the operating state of the device, and outputs the first specific information generated by the generation AI model.

9. An information processing program that causes a computer to input a first command statement indicating a first generation instruction to a generation AI model to generate first specific information that identifies an abnormal state of the device based on an image captured showing the operating status of the device, and to output the first specific information generated by the generation AI model.

Images