Processing system, first processing device, second processing device, program, and processing method

Abstract

This processing system comprises a first processing device and a second processing device. The first processing device acquires a first feature amount of a first image and transmits the first feature amount. The second processing device receives the first feature amount from the first processing device, acquires a first restored image obtained by restoring the first image on the basis of the first feature amount, and performs processing using the first restored image.

Description

Processing System, First Processing Device, Second Processing Device, Program, and Processing Method

[0001] This disclosure relates to communication.

[0002] Patent Document 1 discloses a technique in which an imaging device transmits an image of a crop to a control device.

[0003] Japanese Patent Application Laid-Open No. 2019-165655

[0004] There is room for improvement in communication.

[0005] Therefore, this disclosure has been made in view of the above points, and an object thereof is to provide a technique capable of reducing a communication load.

[0006] One aspect of a processing system includes a first processing device and a second processing device. The first processing device acquires a first feature amount of a first image and transmits the first feature amount. The second processing device receives the first feature amount from the first processing device, acquires a first restored image obtained by restoring the first image based on the first feature amount, and performs processing using the first restored image.

[0007] Also, one aspect of the second processing device is the second processing device included in the above-described processing system.

[0008] Also, one aspect of the first processing device is the first processing device included in the above-described processing system.

[0009] Also, one aspect of a program is a program for causing a computer device to function as the above-described second processing device.

[0010] Also, one aspect of a program is a program for causing a computer device to function as the above-described first processing device.

[0011] Also, one aspect of a processing method is to receive a first feature amount of a first image, acquire a first restored image obtained by restoring the first image based on the first feature amount, and perform processing using the first restored image.

[0012] The communication load can be reduced.

[0013] Figure 1 is a schematic diagram showing an example of the configuration of the processing system. Figure 2 is a schematic diagram showing an example of the configuration of the terminal device. Figure 3 is a schematic diagram showing an example of the configuration of the server. Figure 4 is a schematic diagram showing an example of the configuration of the processing system. Figure 5 is a schematic diagram showing an example of a captured image. Figure 6 is a schematic diagram showing an example of a target image. Figure 7 is a flowchart showing an example of the operation of the processing system. Figure 8 is a schematic diagram showing an example of the configuration of the generative model. Figure 9 is a schematic diagram showing an example of the feature space. Figure 10 is a schematic diagram showing an example of the server display. Figure 11 is a schematic diagram showing an example of the server display. Figure 12 is a schematic diagram showing an example of a crop map. Figure 13 is a schematic diagram showing an example of a crop map. Figure 14 is a schematic diagram showing an example of the server display. Figure 15 is a flowchart showing an example of the operation of the processing system. Figure 16 is a schematic diagram showing an example of the server display. Figure 17 is a schematic diagram showing an example of the server display. Figure 18 is a schematic diagram showing an example of the server display. Figure 19 is a flowchart showing an example of the operation of the processing system. Figure 20 is a schematic diagram showing an example of the server display. Figure 21 is a schematic diagram showing an example of the configuration of the processing system. Figure 22 is a flowchart showing an example of the operation of the processing system. Figure 23 is a schematic diagram showing an example of the server display. Figure 24 is a schematic diagram showing an example of a crop area. Figure 25 is a schematic diagram showing an example of a server display. Figure 26 is a schematic diagram showing an example of a server display. Figure 27 is a schematic diagram showing an example of a server configuration. Figure 28 is a flowchart showing an example of the operation of the processing system. Figure 29 is a schematic diagram showing an example of a server display. Figure 30 is a schematic diagram showing an example of a server display. Figure 31 is a schematic diagram showing an example of a server display. Figure 32 is a schematic diagram showing an example of a server display. Figure 33 is a schematic diagram showing an example of a server display. Figure 34 is a schematic diagram to explain an example of server operation. Figure 35 is a schematic diagram showing an example of a server display. Figure 36 is a schematic diagram showing an example of a server display. Figure 37 is a schematic diagram showing an example of the configuration of a processing system. Figure 38 is a flowchart showing an example of the operation of a processing system. Figure 39 is a schematic diagram showing an example of a server display. Figure 40 is a schematic diagram showing an example of a server display. Figure 41 is a schematic diagram showing an example of a server display. Figure 42 is a schematic diagram showing an example of the configuration of a processing system.Figure 43 is a flowchart showing an example of the operation of the processing system. Figure 44 is a schematic diagram showing an example of the server display. Figure 45 is a schematic diagram showing an example of the server display. Figure 46 is a schematic diagram showing an example of the server display. Figure 47 is a schematic diagram showing an example of the server display. Figure 48 is a flowchart showing an example of the operation of the processing system. Figure 49 is a schematic diagram showing an example of the server display. Figure 50 is a schematic diagram showing an example of the server display. Figure 51 is a schematic diagram showing an example of the server display. Figure 52 is a schematic diagram showing an example of the server display. Figure 53 is a schematic diagram showing an example of the server display.

[0014] <Example of Processing System Configuration> Figure 1 is a schematic diagram showing an example of a processing system 1. As shown in Figure 1, the processing system 1 includes, for example, processing units 3 and 4 that can communicate with each other. Processing units 3 and 4 are connected to, for example, a network 2. The network 2 may include, for example, the Internet or a LAN. LAN is an abbreviation for Local Area Network. Processing units 3 and 4 can communicate with each other through the network 2.

[0015] The processing device 4 acquires, for example, an image of the object that the processing system 1 is processing. The processing device 3 performs, for example, processing on the object. As part of the processing on the object, the processing device 3 determines, for example, the state of the object. Some kind of action is performed on the object. The object can also be called, for example, the object to be treated.

[0016] Processing unit 3 is, for example, a server, and processing unit 4 is, for example, a terminal device. Processing unit 4 is also called, for example, an edge device. Processing unit 4 is located, for example, remotely from processing unit 3. Processing system 1 may include multiple processing units 4. Hereafter, processing unit 3, which is a server, may be referred to as server 3, and processing unit 4, which is a terminal device, may be referred to as terminal device 4. Note that processing unit 3 does not have to be a server, and processing unit 4 does not have to be a terminal device.

[0017] <Overview of Processing System Operation Example> Terminal device 4 is used, for example, in a farm where many crops (also called agricultural products) of the same variety are grown. The target object is, for example, a crop. Terminal device 4 may be used in a farm where fruits of the same variety (for example, grapes or strawberries of the same variety) are grown, or in a farm where vegetables of the same variety (for example, Chinese cabbage or cabbage of the same variety) are grown, or in a farm where other crops of the same variety are grown. Terminal device 4 is capable of acquiring target images in the farm that show the crops as the target object (in other words, the object to be processed). The crops shown in the target images acquired by terminal device 4 may be fruits such as grapes or strawberries, vegetables such as Chinese cabbage or cabbage, or other agricultural products. Terminal device 4 is capable of acquiring multiple target images, each showing multiple crops of the same variety grown in the farm. Hereafter, a farm where terminal device 4 is used may be referred to as a target farm.

[0018] The terminal device 4 is connected to, for example, a camera 5 and a location information acquisition unit 6, which are configured separately from the processing system 1. The terminal device 4, camera 5, and location information acquisition unit 6 are mounted, for example, on a vehicle 10 that travels within the target farm. Therefore, the terminal device 4, camera 5, and location information acquisition unit 6 can move together with the vehicle 10. The vehicle 10 may be capable of autonomous driving, remote operation, or direct human operation. An autonomous vehicle is also called a rover, for example. The camera 5 and location information acquisition unit 6 may constitute part of the processing system 1.

[0019] Camera 5 is a type of sensor capable of photographing crops within the target farm. Camera 5 can, for example, continuously take photographs while vehicle 10 is moving, and photograph all crops within the target farm. Vehicle 10 moves within the target farm so that camera 5 can photograph all crops within the target farm. Each image 50 generated by camera 5 is input to terminal device 4. Multiple images 50 input to terminal device 4 show all crops within the target farm. The images 50 may be, for example, color images (for example, RGB images). Terminal device 4 extracts the portion of the image 50 acquired from camera 5 that shows the crops, and obtains a target image in which the crops are shown prominently. Terminal device 4 obtains a target image in which each crop in the target farm is shown prominently.

[0020] The location information acquisition unit 6 can acquire location information 60 (also called vehicle location information 60) that indicates the absolute position of the vehicle 10. The location information acquisition unit 6 repeatedly acquires vehicle location information 60 while the vehicle 10 is in motion. The location information acquisition unit 6 can acquire vehicle location information 60 based on received signals received from, for example, GNSS satellites (e.g., GPS satellites). GNSS is an abbreviation for Global Navigation Satellite System. GPS is an abbreviation for Global Positioning System. Each vehicle location information 60 acquired by the location information acquisition unit 6 is input to the terminal device 4. Based on each vehicle location information 60 from the location information acquisition unit 6 and each captured image 50 from the camera 5, the terminal device 4 acquires location information (also called crop location information) that indicates the absolute position of each crop in the target farm.

[0021] Terminal device 4 acquires the feature quantities of each target image it has obtained. In other words, terminal device 4 acquires the feature quantities of each of the multiple target images, each of which shows multiple crops within the target farm. Then, terminal device 4 transmits the feature quantities of each target image to server 3.

[0022] Server 3 processes the features of multiple target images, each depicting different crops within the target farm. For example, Server 3 obtains a reconstructed image based on the features of the target image. Then, Server 3 processes the obtained reconstructed image. For example, Server 3 displays the reconstructed image.

[0023] Furthermore, Server 3 performs a determination regarding objects depicted in the target image, for example, based on the feature quantities of the target image. In this way, Server 3 can perform a determination regarding objects depicted in the target image based on the feature quantities of the target image, thereby enabling it to perform a proper determination regarding objects depicted in the target image. When Server 3 determines the state of an object as part of its determination regarding the object, it can perform a proper determination of the state of the object.

[0024] Server 3 determines the state of the crops in the target image based on the features of the target image. Server 3 determines the harvest time of the crops as part of the crop's state. Server 3 displays the harvest time determination result (in other words, the crop state determination result) together with the reconstructed image of the crop. This allows users of Server 3 (also called server users) to easily check the reconstructed image and the harvest time determination result (also called the harvest time determination result) of the crops shown in the reconstructed image. Server 3 can acquire and display the reconstructed image of each crop in the target farm and the harvest time determination result for that crop. The harvest time determination can also be called a harvest time prediction, and the harvest time determination result can also be called a harvest time prediction result.

[0025] The server user may be, for example, an expert in crop cultivation. The server user may, for example, refer to the restored image displayed on server 3 and determine whether the harvest time determination result displayed on server 3 is valid.

[0026] If the server user determines that the harvest time determination result is valid, they notify server 3 of this by, for example, making a predetermined input. If the server user determines that the harvest time determination result is invalid, they instruct server 3 to correct the harvest time determination result by, for example, making a predetermined input. Server 3 corrects the harvest time determination result according to the correction instruction from the server user. In other words, server 3 corrects the determined harvest time (or, in other words, the estimated harvest time) according to the correction instruction from the server user.

[0027] When server 3 receives notification from the server user that the harvest timing determination result is valid, it transmits the acquired harvest timing determination result as is to terminal device 4. On the other hand, when server 3 corrects the harvest timing determination result according to the correction instructions from the server user, it transmits the corrected harvest timing determination result to terminal device 4. For each crop in the target farm, server 3 transmits the harvest timing determination result for that crop to terminal device 4.

[0028] Terminal device 4 displays the harvest time determination results (including the corrected harvest time determination results) received from server 3. This allows users of terminal device 4 (also called terminal users) to know the harvest time of the crops. Terminal device 4 displays the harvest time determination results for each crop in the target farm. For example, terminal device 4 displays the harvest time determination result of a crop together with the crop's location information.

[0029] A terminal user is, for example, a worker who performs treatments on crops within the target farm. Treatments performed by terminal users on crops include, for example, fertilizing, thinning (such as fruit thinning), harvesting, and discarding crops. For example, a terminal user harvests a crop according to the harvest time determination result displayed by terminal device 4. A crop or subject to treatment can also be called, for example, a sample.

[0030] Note that the terminal device 4, camera 5, and location information acquisition unit 6 do not necessarily have to be mounted on the vehicle 10. In this case, the camera 5 may be able to photograph all the crops in the target farm by having a person with the terminal device 4, camera 5, and location information acquisition unit 6 move around within the target farm.

[0031] <Example of Terminal Device Configuration> Figure 2 is a schematic diagram showing an example of the configuration of terminal device 4. As shown in Figure 2, terminal device 4 comprises a control unit 40, a storage unit 41, interfaces 42, 45, and 46, a display unit 47, and an input unit 48. Terminal device 4 is, for example, a computer device.

[0032] Interface 42 is capable of communicating with network 2. Interface 42 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. Interface 42 is capable of communicating with server 3 through network 2. Interface 42 may communicate with network 2 via wired connection or via wireless connection. Interface 42 may be capable of performing wireless communication compliant with Wi-Fi, for example, or communication compliant with other standards. Interface 42 inputs information received from network 2 to control unit 40. Interface 42 also transmits information from control unit 40 to network 2. Terminal device 4 receives information from server 3 and transmits information to server 3 through interface 42.

[0033] Interface 45 can communicate with camera 5. Interface 45 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. Interface 45 may communicate with camera 5 via wired communication or wireless communication. Interface 45 receives captured images 50 from camera 5 and outputs the received captured images 50 to control unit 40.

[0034] Interface 46 can communicate with the location information acquisition unit 6. Interface 46 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. Interface 46 may communicate with the location information acquisition unit 6 via wired communication or wireless communication. Interface 46 receives vehicle location information 60 from the location information acquisition unit 6 and outputs the received vehicle location information 60 to the control unit 40.

[0035] The control unit 40 can comprehensively manage the operation of the terminal device 4 by controlling other components of the terminal device 4. The control unit 40 can also be called a processing unit that performs various processes. The control unit 40 can also be called a control circuit or a processing circuit. The control unit 40 includes, for example, at least one processor 40a. The at least one processor 40a in the control unit 40 may include a CPU (Central Processing Unit). The control unit 40 stores the captured image 50 from the camera 5 and the vehicle position information 60 from the position information acquisition unit 6 in the storage unit 41.

[0036] The storage unit 41 includes, for example, non-volatile memory and volatile memory. The storage unit 41 can also be described as, for example, a memory circuit. The non-volatile memory and volatile memory can also be described as non-temporary recording media that can be read by a processor such as a CPU.

[0037] The non-volatile memory of the storage unit 41 may include, for example, flash memory. The non-volatile memory of the storage unit 41 stores, for example, a program 41a for controlling the terminal device 4. Various functions of the control unit 40 are realized, for example, by the CPU of the control unit 40 executing the program 41a.

[0038] The volatile memory of the storage unit 41 functions as work memory for data processing by the control unit 40. The volatile memory of the storage unit 41 may include, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). RAM is an abbreviation for Random Access Memory.

[0039] The configuration of the control unit 40 is not limited to the above example. For example, at least one processor 40a in the control unit 40 may include multiple CPUs, or at least one DSP (Digital Signal Processor). Furthermore, all or some of the functions of the control unit 40 may be implemented by hardware circuits that do not require software to realize those functions. Also, the storage unit 41 may include, for example, a small hard disk drive and an SSD (Solid State Drive).

[0040] The input unit 48 is capable of receiving various inputs from the user. The input unit 48 may include, for example, a mouse and a keyboard. The input unit 48 may also include a touch sensor that receives touch operations from the user. The input unit 48 may also include an audio input unit that receives voice input from the user. The audio input unit may include, for example, a microphone. The control unit 40 can identify the content of the input received by the input unit 48 based on the output signal from the input unit 48.

[0041] The display unit 47 can display various types of information under the control of the control unit 40. The display unit 47 has a display surface for displaying various types of information. The display unit 47 may be, for example, a liquid crystal display, an organic electroluminescent (EL) display, or a plasma display. Furthermore, if the input unit 48 is equipped with a touch sensor, the touch sensor and the display surface of the display unit 47 may constitute a touch panel display having both display and touch detection functions. In this case, the input unit 48 can detect touch operations on the display surface of the display unit 47. The input unit 48 and the display unit 47 constitute a user interface.

[0042] The terminal device 4 may be a portable device that can be carried by a user, or a stationary device installed in the vehicle 10. The terminal device 4 may be a tablet device, a mobile phone such as a smartphone, a notebook or desktop personal computer, or a wearable device such as smart glasses. Note that at least one of the camera 5 and the position information acquisition unit 6 may be included in the terminal device 4.

[0043] <Server Configuration Example>FIG. 3 is a schematic diagram showing an example of the configuration of the server 3. As shown in FIG. 3, the server 3 includes a control unit 30, a storage unit 31, an interface 32, a display unit 37, and an input unit 38. The server 3 is, for example, a computer device. The server 3 may be, for example, a cloud server.

[0044] The interface 32 can communicate with the network 2. The interface 32 can also be referred to as, for example, an interface circuit, a communication unit, or a communication circuit. The interface 32 can communicate with the interface 42 of the terminal device 4 through the network 2. The interface 32 may perform wired communication or wireless communication with the network 2. The interface 32 may be able to perform wireless communication compliant with, for example, Wi-Fi, or communication compliant with other standards. The interface 32 inputs the information received from the network 2 to the control unit 30. Further, the interface 32 transmits the information from the control unit 30 to the network 2. The server 3 receives information from the terminal device 4 or transmits information to the terminal device 4 through the interface 32.

[0045] The control unit 30 can comprehensively manage the operation of the server 3 by controlling other components of the server 3. The control unit 30 can also be referred to as, for example, a processing unit that performs various processes. The control unit 30 can also be referred to as, for example, a control circuit or a processing circuit. The control unit 30 includes, for example, at least one processor 30a. The at least one processor 30a included in the control unit 30 may include a CPU.

[0046] The storage unit 31 includes, for example, a non-volatile memory and a volatile memory. The storage unit 31 can also be said to be a storage circuit, for example. The non-volatile memory of the storage unit 31 may include, for example, a flash memory. In the non-volatile memory of the storage unit 31, for example, a program 31a for controlling the server 3 and the like are stored. Various functions of the control unit 30 are realized, for example, when the CPU of the control unit 30 executes the program 31a. The volatile memory of the storage unit 31 functions as a work memory when the control unit 30 processes data. The volatile memory of the storage unit 31 may include, for example, SRAM or DRAM.

[0047] The configuration of the control unit 30 is not limited to the above example. For example, at least one processor 30a included in the control unit 30 may include a plurality of CPUs or at least one DSP. Also, all or some of the functions of the control unit 30 may be realized by a hardware circuit that does not require software for realizing the function. Further, the storage unit 31 may include, for example, a small hard disk drive and an SSD (Solid State Drive).

[0048] The input unit 38 can receive various inputs from the user. The input unit 38 may include, for example, a mouse and a keyboard. Also, the input unit 38 may include a touch sensor that receives a touch operation of the user. Further, the input unit 38 may include a voice input unit that receives a voice input from the user. The voice input unit may include, for example, a microphone. The control unit 30 can recognize the content of the input received by the input unit 38 based on the output signal from the input unit 38.

[0049] The display unit 37 can display various types of information under the control of the control unit 30. The display unit 37 has a display surface for displaying various types of information. The display unit 37 may be, for example, a liquid crystal display, an organic EL display, or a plasma display. Furthermore, if the input unit 38 is equipped with a touch sensor, the touch sensor and the display surface of the display unit 37 may constitute a touch panel display having both a display function and a touch detection function. In this case, the input unit 38 can detect touch operations on the display surface of the display unit 37.

[0050] <Details of the Processing System Operation Example> Figure 4 is a schematic diagram showing an example of the multiple functional blocks provided by the processing system 1. As shown in Figure 4, the server 3 includes, for example, a restoration unit 130 and a determination unit 131 as functional blocks. The restoration unit 130 and the determination unit 131 are functional blocks formed by the CPU of the control unit 30 executing a program 31a in the storage unit 31. The terminal device 4 includes, for example, an image processing unit 140 and a feature acquisition unit 141 as functional blocks. The image processing unit 140 and the feature acquisition unit 141 are functional blocks formed by the CPU of the control unit 40 executing a program 41a in the storage unit 41.

[0051] The restoration unit 130 generates a restored image based on the feature quantities of the target image received from the terminal device 4. The determination unit 131 determines the harvest time of the crops shown in the target image based on the feature quantities of the target image.

[0052] The image processing unit 140 acquires the location information of each of the multiple crops in the target farm. The image processing unit 140 functions as a location information acquisition unit that acquires the location information of the crops. The image processing unit 140 also acquires multiple target images, each showing one of the multiple crops in the target farm. The image processing unit 140 functions as an image acquisition unit that acquires the target images. The feature quantity acquisition unit 141 acquires the feature quantities of each target image acquired by the image processing unit 140.

[0053] Furthermore, all or some of the functions of the restoration unit 130 may be implemented by hardware circuits that do not require software to realize those functions. The same applies to the determination unit 131, the image processing unit 140, and the feature acquisition unit 141, as well as the functional blocks provided by the server 3 and terminal device 4, which will be described later.

[0054] The processing system 1 performs a farm harvest determination process, for example, to determine the harvest time for each crop in the target farm based on multiple captured images 50 obtained by the camera 5. The farm harvest determination process is performed, for example, at the harvest forecast stage when the crops in the target farm are likely to be ready for harvest.

[0055] During the harvest forecast stage, vehicle 10 travels within the target farm. Camera 5, during the harvest forecast stage, photographs all crops in the target farm while vehicle 10 is moving, acquiring multiple images 50 that show all crops in the target farm. In addition, location information acquisition unit 6 repeatedly acquires vehicle location information 60 of vehicle 10 while it is moving during the harvest forecast stage. The farm harvest determination process starts, for example, while vehicle 10 is moving.

[0056] In the farm harvest determination process, the image processing unit 140 of the terminal device 4 acquires a target image 150 (see Figure 6 described later) in which the crops are pictured, and crop location information indicating the location of the crops. Since crop location information can also be said to be location information that indicates the absolute position of the object, crop location information can also be said to be object location information.

[0057] Information about crops, such as crop location information, can be considered information about the target object or information about the target of treatment. In this disclosure, terms containing "crop," such as crop location information, can be replaced with "target object" or "target of treatment." For example, as mentioned above, crop location information can be considered information about the target object or information about the target of treatment.

[0058] In the farm harvest determination process, the image processing unit 140 acquires target images 150 and crop location information based on the captured images 50 and vehicle location information 60 stored in the storage unit 41. In the farm harvest determination process, the image processing unit 140 acquires multiple target images 150, each showing multiple crops in the target farm, and multiple crop location information, each indicating the location of multiple crops in the target farm, based on the multiple captured images 50 obtained by the camera 5 and the multiple vehicle location information 60 obtained by the location information acquisition unit 6. The target images 150 and crop location information acquired by the image processing unit 140 are stored in the storage unit 41.

[0059] Figures 5 and 6 are schematic diagrams illustrating examples of captured images 50 and target images 150 obtained at the expected harvest stage, respectively, when the crop is grapes.

[0060] The image processing unit 140 performs object recognition processing (also called object detection processing) on ​​the captured image 50 using machine learning or template matching to identify partial images 51 in the captured image 50 in which crops are visible. In the example in Figure 5, two partial images 51 are identified from the captured image 50. A bunch of grapes is clearly visible in each of the two partial images 51.

[0061] Next, the image processing unit 140 enlarges each acquired partial image 51 to generate a target image 150 of a predetermined size. This results in the acquisition of a target image 150 in which the crop is shown in detail. In the example in Figure 6, the target image 150 shows a bunch of grapes. The target image 150 is a real image of the actual crop, not a CG image. CG is an abbreviation for Computer Graphics. The target image 150 can also be called a captured image or camera image.

[0062] The range of crops depicted in the target image 150 and the range of crops depicted in the partial image 51 from which the target image 150 is derived are determined by object recognition processing performed by the image processing unit 140. If the crop is a strawberry, for example, one strawberry will be depicted in the partial image 51 and the target image 150 acquired at the harvest stage. Similarly, if the crop is a watermelon, for example, one watermelon will be depicted in the partial image 51 and the target image 150 acquired at the harvest stage.

[0063] Furthermore, in the farm harvest determination process, the image processing unit 140 generates crop position information indicating the absolute position of the crops captured in the partial image 51, based on vehicle position information 60 indicating the absolute position of the vehicle 10 when the captured image 50 was acquired by the camera 5, the relative position of the partial image 51 in the captured image 50, and various parameters related to the camera 5 (for example, the relative position of the camera 5 to the vehicle 10 and the field of view of the camera 5). When the captured image 50 shown in Figure 5 is obtained, the image processing unit 140 generates crop position information indicating the absolute position of the crops captured in each of the two partial images 51 identified from the captured image 50.

[0064] In the farm harvest determination process, the image processing unit 140 acquires a target image containing a crop and crop location information indicating the location of the crop. It then stores the target image, the crop location information, and the crop's identification information (also called a crop ID) in the storage unit 41, associating them with each other. ID is an abbreviation for Identifier. In the processing system 1, each crop in the target farm is managed by its crop ID. The crop ID can also be called an object ID or a processing target ID.

[0065] When the image processing unit 140 acquires a target image and crop location information based on the captured image 50 and vehicle location information 60 stored in the storage unit 41, it may delete the captured image 50 and vehicle location information 60 from the storage unit 41.

[0066] Processing system 1, in the farm harvest determination process, performs a crop-unit determination process for each of the multiple crops in the target farm, determining the harvest time of that crop based on the target image in which the crop is depicted. In the farm harvest determination process, the crop-unit determination process is executed for each crop in the target farm. At least a portion of the multiple crop-unit determination processes that are performed for each of the multiple crops in the target farm in the farm harvest determination process may be executed in parallel or sequentially.

[0067] Figure 7 is a flowchart showing an example of a single crop unit determination process. In the farm harvest determination process, the crop unit determination process shown in Figure 7 is executed for each crop in the target farm. Hereafter, the crop unit determination process of interest, or in other words, the crop unit determination process being explained, will be referred to as the "focused crop unit determination process." Furthermore, the crop whose harvest time is determined in the focused crop unit determination process will be called the "focused crop." Also, the target image in which the focused crop is shown, used in the focused crop unit determination process, will be called the "focused image."

[0068] In the crop unit determination process, first, in step s1, the feature acquisition unit 141 of the terminal device 4 acquires the feature quantities (also called focus features) of the target image 150.

[0069] In this process, the processing system 1 uses, for example, a generative model 100 to acquire the features of the target image and to acquire a reconstructed image based on the features of the target image. The generative model 100 is a type of machine learning model. The generative model 100 is composed of, for example, a neural network such as a convolutional neural network.

[0070] Figure 8 is a schematic diagram showing an example of the configuration of the generative model 100. The generative model 100 may be, for example, a stochastic generative model in which the latent variables follow a probability distribution. The generative model 100 may also be, for example, a variational autoencoder (VAE).

[0071] The generation model 100 includes, for example, an encoding model (also called an encoder) 101 to which the target image 150 is input, and a decoding model (also called a decoder) 102. In this example, the encoding model 101 constitutes the feature acquisition unit 141, and the decoding model 102 constitutes the reconstruction unit 130. Both the encoding model 101 and the decoding model 102 are types of machine learning models.

[0072] The encoding model 101 converts the target image 150 into its feature quantities. In other words, the encoding model 101 obtains the feature quantities of the target image 150. Feature quantities are also called latent variables. The feature quantities of the target image 150 can be said to be values ​​that represent the characteristics of the appearance of the crop (in other words, the object) depicted in the target image 150.

[0073] The features obtained by the encoding model 101 are represented, for example, by vectors. If the target image 150 has, for example, 128 pixels vertically and 128 pixels horizontally, the encoding model 101 outputs features represented by, for example, a 400-dimensional vector. Hereafter, the vector representing the features may be called a feature vector. Also, the features of an image containing crops, such as the target image acquired by the terminal device 4, may be called the features corresponding to the crop or the features of the crop.

[0074] The decoding model 102 reconstructs the target image 150 input to the encoding model 101 based on the features (in other words, feature vectors) output from the encoding model 101. In other words, the decoding model 102 obtains a reconstructed image 250, which is the reconstructed version of the target image 150, based on the features of the target image 150 obtained by the encoding model 101. The decoding model 102 can also be described as an acquisition model that obtains the reconstructed image 250. For example, the decoding model 102 converts a 400-dimensional feature vector into an image corresponding to the target image 150, which has 128 pixels vertically and 128 pixels horizontally, and outputs it. Ideally, the reconstructed image 250 obtained by the decoding model 102 is identical to the target image 150 input to the encoding model 101.

[0075] Note that the generative model 100 may be a probabilistic generative model other than a variational autoencoder. For example, the generative model 100 may be a generative adversarial network (GAN) or a diffusion model. Also, the generative model 100 may be a regular autoencoder without the "variational" designation.

[0076] The generative model 100 is trained, for example, on server 3. The control unit 30 of server 3 has the generative model 100 before training and trains the generative model 100. The encoded model 101 of the generative model 100 trained by the control unit 30 is transferred to the terminal device 4 as a feature acquisition unit 141. In addition, the decoded model 102 of the generative model 100 trained by the control unit 30 is used as a reconstruction unit 130. The generative model 100 is trained, for example, through unsupervised learning.

[0077] The memory unit 31 of server 3 stores multiple training images, each depicting a different crop. The control unit 30 learns the generative model 100 based on the multiple training images in the memory unit 31.

[0078] Multiple training images may include images of actual crops at past harvest stages in the target farm. Multiple training images may also include images of actual crops at past harvest stages in other farms where the same type of crop as the target farm is grown. Furthermore, multiple training images may include free stock images from websites. (Web is an abbreviation for World Wide Web.) Additionally, multiple training images may include images purchased by the server user from a company that sells training data.

[0079] In step s1, when the feature quantities of the image of interest are obtained, in step s2, the control unit 40 of the terminal device 4 generates first crop-related information that includes the feature quantities of the image of interest and the crop ID of the crop of interest shown in the image of interest. The terminal device 4 then transmits the first crop-related information to the server 3. The first crop-related information can also be called object-related information or treatment target-related information.

[0080] The terminal device 4 may execute steps s1 and s2 each time it acquires one target image, or it may execute steps s1 and s2 for each target image each time it acquires multiple target images. In the latter case, the terminal device 4 may execute steps s1 and s2 for each target image each time it acquires several target images, or it may execute steps s1 and s2 for each target image each time it acquires more than ten target images, or it may execute steps s1 and s2 for each target image each time it acquires several dozen target images. Alternatively, the terminal device 4 may execute steps s1 and s2 for each target image after all the crops in the target farm have been photographed by the camera 5 and multiple target images showing all the crops have been acquired.

[0081] In the crop unit determination process, in step s11, server 3 receives first crop-related information from terminal device 4. Next, in step s12, the restoration unit 130 (in other words, the decoding model 102) of server 3 obtains a restored image (also called a restored image of interest) by restoring the image of interest based on the features of interest contained in the first crop-related information from terminal device 4.

[0082] After step s12, in step s13, the determination unit 131 determines the harvest time of the crop of interest shown in the image of interest based on the features of interest.

[0083] In this example, the memory unit 31 of server 3 stores multiple determination features used to determine the harvest time of crops. Determination features are, for example, the features of an image in which the crop is clearly visible. Determination features are acquired, for example, by server 3. For example, the encoding model 101 of the trained generative model 100 provided by server 3 acquires the features of an image in which the crop is clearly visible as determination features. Hereafter, images from which determination features are acquired by server 3 will be called determination images. The determination features of a determination image in which a crop is visible can also be called the determination features corresponding to that crop or the determination features of that crop.

[0084] Multiple classification images, each possessing multiple classification features, may include at least a portion of the multiple training images used to train the generative model 100. In this case, at least a portion of the multiple training images are used interchangeably as classification images. Furthermore, the multiple classification images may include images of crops that are separate from the training images.

[0085] The multiple classification features stored by server 3 are divided into L groups (where L is an integer greater than or equal to 2) according to the harvest time of the crops corresponding to the classification features (in other words, the crops shown in the classification images that possess the classification features). Each of the multiple classification features belongs to one of the L groups.

[0086] For example, with L=7, the multiple classification features are divided into groups: 10 days or more later group, 5 to 9 days later group, 2 days later group, 1 day later group, harvest group, 1 day earlier group, and 2 days earlier group.

[0087] The "More than 10 days later" group includes the determinative features corresponding to crops whose harvest date (in other words, harvest time) is 10 days or more away. It can also be said that the "More than 10 days later" group includes the determinative features corresponding to crops that are 10 days or more away from their harvest date. The harvest date for crops corresponding to the determinative features in the "More than 10 days later" group is 10 days or more after the current state of the crop.

[0088] The 5-9 day group is the group to which the determinative features corresponding to crops whose harvest date is 5 days or more but 9 days or less later belong. It can also be said that the 5-9 day group is the group to which the determinative features corresponding to crops that are 5 days or more but 9 days or less before their harvest date belong. The harvest date of crops corresponding to the determinative features belonging to the 5-9 day group is 5 days or more but 9 days or less later than the state of that crop.

[0089] The "2-Day Later" group is the group to which the determinative features corresponding to crops whose harvest date is two days away belong. It can also be said that the "2-Day Later" group is the group to which the determinative features corresponding to crops that are two days away from their harvest date belong. The harvest date for crops corresponding to the determinative features in the "2-Day Later" group is two days after the current state of the crop.

[0090] The "1-Day Later" group is the group to which the determinative features corresponding to crops whose harvest date is one day away belong. It can also be said that the "1-Day Later" group is the group to which the determinative features corresponding to crops that are one day before their harvest date belong. The harvest date for crops corresponding to the determinative features in the "1-Day Later" group is one day after the current state of the crop.

[0091] The harvest group is the group to which the determination features corresponding to crops that are ready for harvest belong. Crops corresponding to the determination features belonging to the harvest group are ready for harvest.

[0092] The "1 day ago" group is the group to which the determinative features corresponding to crops whose harvest date was one day ago belong. It can also be said that the "1 day ago" group is the group to which the determinative features corresponding to crops that are in a state one day after their harvest date belong. The harvest date of the crops corresponding to the determinative features belonging to the "1 day ago" group is one day prior to the current state of that crop.

[0093] The "two days ago" group is the group to which the determinative features corresponding to crops whose harvest date was two days ago belong. It can also be said that the "two days ago" group is the group to which the determinative features corresponding to crops that are in a state two days after their harvest date belong. The harvest date of the crops corresponding to the determinative features belonging to the "two days ago" group is two days prior to the state of that crop.

[0094] In the storage unit 31 of server 3, for each determination feature quantity, the determination feature quantity and group information that identifies the group to which the determination feature quantity belongs are stored in correspondence with each other.

[0095] It can be said that each of the grouped classification features is associated with the harvest time of the crop corresponding to that classification feature. For example, a classification feature belonging to the "1 day later" group is associated with the harvest date of the corresponding crop being one day later from the current state of the crop. Similarly, a classification feature belonging to the "harvest" group is associated with the state in which the corresponding crop should be harvested.

[0096] The group to which the judgment feature belongs is determined, for example, by a skilled server user. In this case, for example, multiple judgment images are stored in the storage unit 31 of server 3. For each judgment image, the storage unit 31 stores the judgment image, the judgment feature of the judgment image, and group information that identifies the group to which the judgment feature belongs, in association with each other.

[0097] Here, the image of interest for determination is called the "target determination image." The control unit 30 displays the target determination image from the storage unit 31 on the display unit 37. The server user checks the state of the crops shown in the target determination image displayed on the display unit 37 and estimates (in other words, determines) the harvest time of the crops. Next, based on the estimated harvest time of the crops shown in the target determination image (in other words, the determination result), the server user determines which of the L groups the determination feature quantity of the target determination image belongs to. That is, the server user determines the group to which the determination feature quantity of the target determination image belongs. Then, the server user makes a predetermined input to the input unit 38 and notifies the server 3 of the group to which the determination feature quantity of the target determination image belongs. The control unit 30 stores the group information indicating the group to which the determination feature quantity of the target determination image belongs, which has been notified to the server 3, in the storage unit 31, associating it with the determination feature quantity and the target determination image.

[0098] For example, if the server user estimates that the harvest date of the crops shown in the attention-grabbing image displayed on the display unit 37 is 10 days or more away, the server user determines that the judgment features of the attention-grabbing image belong to the 10 days or more away group. Also, if the server user estimates that the crops shown in the attention-grabbing image displayed on the display unit 37 are ready for harvest, the server user determines that the judgment features of the attention-grabbing image belong to the harvest group.

[0099] As described above, the multiple determination feature quantities in the memory unit 31 are grouped. After the grouping of the multiple determination feature quantities is complete, the multiple determination images may be deleted from the memory unit 31.

[0100] In step s13, the determination unit 131 determines which of the L groups the feature of interest belongs to. The determination unit 131 then sets the harvest time corresponding to the group to which the feature of interest belongs as the harvest time for the crop of interest corresponding to the feature of interest.

[0101] If the determination unit 131 determines that the feature of interest belongs to the "more than 10 days later" group, it determines that the harvest date of the crop of interest is more than 10 days later from today. If the determination unit 131 determines that the feature of interest belongs to the "5 to 9 days later" group, it determines that the harvest date of the crop of interest is 5 days or more and 9 days or less later from today. If the determination unit 131 determines that the feature of interest belongs to the "2 days later" group, it determines that the harvest date of the crop of interest is 2 days later from today. If the determination unit 131 determines that the feature of interest belongs to the "1 day later" group, it determines that the harvest date of the crop of interest is 1 day later from today. If the determination unit 131 determines that the feature of interest belongs to the "harvest" group, it determines that the harvest date of the crop of interest is today. If the determination unit 131 determines that the feature of interest belongs to the "1 day earlier" group, it determines that the harvest date of the crop of interest was 1 day earlier from today. If the determination unit 131 determines that the feature of interest belongs to the 2-days-ago group, it determines that the harvest date of the crop of interest was two days ago from today.

[0102] When the determination unit 131 determines which of the L groups a particular feature belongs to, it places multiple determination features from the memory unit 31 and the particular feature in the feature space. The feature space is also called the latent space. The degree of the feature space is the same as the degree of the features. For example, if the degree of the features is 400 dimensions, the degree of the feature space will be 400 dimensions. Hereafter, the feature space in which the multiple determination features from the memory unit 31 and the particular feature are placed will be called the specific feature space.

[0103] The determination unit 131 identifies the group among the L groups in the specific feature space whose distribution of multiple determination features is closest to the feature of interest. The determination unit 131 then determines that the feature of interest belongs to the identified group. For example, for each of the L groups, the determination unit 131 calculates the Mahalanobis distance, which represents the distance between the distribution of multiple determination features belonging to that group and the feature of interest in the specific feature space. The determination unit 131 then identifies the group with the smallest calculated Mahalanobis distance as the group whose distribution of multiple determination features is closest to the feature of interest.

[0104] Figure 9 is a schematic diagram showing an example of a specific feature space 300. For the sake of explanation, in Figure 9, among the multiple determination features 305 belonging to L groups, the determination feature 305a for the 2-day group 310a, the determination feature 305b for the 1-day group 310b, and the determination feature 305c for the harvest group 310c are shown. In Figure 9, the approximate ranges 315a for the 2-day group 310a, 315b for the 1-day group 310b, and 315c for the harvest group 310c are shown. In the example in Figure 9, the determination unit 131 determines that the feature of interest 301 belongs to the 1-day group 310b. The determination unit 131 then determines that the harvest date of the crop of interest is 1 day later.

[0105] If the harvest time of the crop of interest is determined in step s13, step s14 is executed. In step s14, the control unit 30 displays the restored image of the crop of interest acquired by the restoration unit 130 and the determination result of the harvest time of the crop of interest determined by the determination unit 131 on the display unit 37.

[0106] Figure 10 is a schematic diagram showing an example of the display of the display unit 37 in step s14. As shown in Figure 10, in step s14, the display unit 37 displays a display screen 330 that includes, for example, the restored image of interest 250, the harvest time determination result 350 of the crop of interest, an input area 360, and an input area 365. On the display screen 330, for example, the string "Restored Image" is displayed around (for example, below) the restored image of interest 250.

[0107] In the example shown in Figure 10, the crop of interest is grapes, and the display unit 37 displays the restored image 250 of the crop of interest, which shows grapes. Also in the example shown in Figure 10, the display unit 37 displays the harvest time determination result 350 when the determination unit 131 determines that the harvest date for the crop of interest is two days from now.

[0108] For example, the server user checks the condition of the crops shown in the restored image 250 displayed on the display unit 37, and based on that condition, determines whether the harvest time determination result 350 for the crop of interest is appropriate.

[0109] The input area 360 is an area for notifying the server 3 that the server user has determined that the harvest time determination result 350 for the crop of interest is valid. When the server user selects the input area 360, that is, when the input unit 38 receives a selection input to select the input area 360, the control unit 30 recognizes that the server user has determined that the harvest time determination result 350 for the crop of interest is valid ("valid" in step s15). In other words, the control unit 30 recognizes that the server user does not wish to modify the harvest time determination result 350 for the crop of interest. The selection input for selecting an object to be selected, such as the input area 360, which is displayed on the display unit 37, may be a predetermined touch operation on the object to be selected, or a predetermined mouse operation while the pointer is on the object to be selected.

[0110] The input area 365 is an area for the server user to notify the server 3 of their desire to correct the harvest time determination result 350 of the crop of interest. The server user selects the input area 365 if they determine that the harvest time determination result 350 of the crop of interest is not valid. In other words, the server user makes a selection input to the input unit 38 to select the input area 365. When the input unit 38 receives the selection input to select the input area 365 (step s15), the control unit 30 recognizes that the server user desires to correct the harvest time determination result 350 of the crop of interest ("request for correction" in step s15). Then, in step s16, the control unit 30 displays a correction screen 380 for correcting the harvest time determination result 350 of the crop of interest on the display unit 37.

[0111] Figure 11 is a schematic diagram showing an example of the correction screen 380. As shown in Figure 11, the correction screen 380 includes the restored image of interest 250, the harvest time determination result 350 for the crop of interest, and the instruction areas 381 to 386. The correction screen 380 also includes the string "Restored Image" around the restored image of interest 250 (for example, below it). Note that the restored image of interest 250 and the string "Restored Image" do not have to be included in the correction screen 380, nor do the harvest time determination result 350 have to be included.

[0112] The instruction area 381 is an area for the server user to instruct the server 3 to correct the harvest time determination result to "10 days or more later". The server user can instruct the server 3 to change the harvest time determination result to "10 days or more later" by selecting the instruction area 381, that is, by performing a selection operation to select the instruction area 381 at the input unit 38. When the input unit 38 receives the selection operation to select the instruction area 381 (step s17), the determination unit 131 corrects the harvest time determination result to "10 days or more later".

[0113] The instruction area 382 is an area for the server user to instruct the server 3 to correct the harvest time determination result to "a number of days later of 5 days or more and 9 days or less". The server user can, for example, instruct the server 3 to change the harvest time determination result to "a number of days later of 5 days or more and 9 days or less" by performing a selection operation to select the instruction area 382 on the input unit 38. When the input unit 38 receives the selection operation to select the instruction area 382 (step s17), the determination unit 131 corrects the harvest time determination result to "a number of days later of 5 days or more and 9 days or less" according to the correction instruction from the server user.

[0114] The instruction area 383 is an area for instructing the server to correct the harvest time determination result to "two days later". The server user can, for example, instruct the server 3 to change the harvest time determination result to "two days later" by performing a selection operation on the input unit 38 to select the instruction area 383. When the input unit 38 receives the selection operation to select the instruction area 383 (step s17), the determination unit 131 corrects the harvest time determination result to "two days later" in accordance with the correction instruction from the server user.

[0115] The instruction area 384 is an area for instructing the server to correct the harvest time determination result to "one day later". The server user can, for example, instruct the server 3 to change the harvest time determination result to "one day later" by performing a selection operation on the input unit 38 to select the instruction area 384. When the input unit 38 receives the selection operation to select the instruction area 384 (step s17), the determination unit 131 corrects the harvest time determination result to "one day later" in accordance with the correction instruction from the server user.

[0116] The instruction area 385 is an area for instructing the server to correct the harvest time determination result to "today". The server user can, for example, instruct the server 3 to change the harvest time determination result to "today" by performing a selection operation on the input unit 38 to select the instruction area 385. When the input unit 38 receives the selection operation to select the instruction area 385 (step s17), the determination unit 131 corrects the harvest time determination result to "today" in accordance with the correction instruction from the server user.

[0117] The instruction area 386 is an area for instructing the server to correct the harvest time determination result to "one day earlier". The server user can, for example, instruct the server 3 to change the harvest time determination result to "one day earlier" by performing a selection operation on the input unit 38 to select the instruction area 386. When the input unit 38 receives the selection operation to select the instruction area 386 (step s17), the determination unit 131 corrects the harvest time determination result to "one day earlier" in accordance with the correction instruction from the server user.

[0118] The instruction area 387 is an area for instructing the server to correct the harvest time determination result to "two days ago". The server user can, for example, instruct the server 3 to change the harvest time determination result to "two days ago" by performing a selection operation on the input unit 38 to select the instruction area 387. When the input unit 38 receives the selection operation to select the instruction area 387 (step s17), the determination unit 131 corrects the harvest time determination result to "two days ago" in accordance with the correction instruction from the server user.

[0119] Furthermore, the modification screen 380 does not necessarily have to include the instruction area corresponding to the harvest time determined by the determination unit 131 among the multiple instruction areas 381 to 387. As shown in the example in Figure 10, if the determination unit 131 determines the harvest time to be "two days later", the modification screen 380 does not necessarily have to include the instruction area 383 corresponding to "two days later".

[0120] When the input area 360 shown in Figure 10 is selected and the control unit 30 recognizes that the server user has determined that the harvest time determination result 350 for the crop of interest is valid ("valid" in step s15), in step s18, the harvest time determination result 350 for the crop of interest is transmitted from the server 3 to the terminal device 4 along with the crop ID of the crop of interest.

[0121] Furthermore, when step s17 is executed and the determination unit 131 corrects the harvest time determination result of the crop of interest in accordance with the correction instruction from the server user, in step s18, the corrected harvest time determination result is transmitted from the server 3 to the terminal device 4 along with the crop ID of the crop of interest. Thereafter, if the harvest time determination result is corrected, the term "harvest time determination result" will refer to the corrected harvest time determination result unless otherwise specified.

[0122] In the crop unit determination process, in step s3, the terminal device 4 receives the harvest time determination result and crop ID of the crop of interest from the server 3. The terminal device 4 then stores the harvest time determination result and crop ID from the server 3 in the storage unit 41, associating them with each other. The types of harvest time determination results that the terminal device 4 receives from the server 3 include the uncorrected harvest time determination result and the corrected harvest time determination result.

[0123] After step s3, in step s4, the display unit 47 displays the harvest time determination result from the server 3 under the control of the control unit 40. The display unit 47 may, for example, display the harvest time determination result when the input unit 48 receives a predetermined input. The display unit 47 may display the harvest time determination result of the crop of interest together with the crop location information and crop ID of the crop of interest.

[0124] By performing the above-described crop-specific determination process for each crop in the target farm, the terminal device 4 can obtain the harvest time determination results for each crop in the target farm from the server 3. The display unit 47 may display the harvest time determination results for each crop one by one, or it may display a list of harvest time determination results for multiple crops in the target farm.

[0125] The display unit 47 may display a crop map 400 that shows the distribution of multiple crops within the target farm. The crop map 400 shows how multiple crops are distributed in the target farm. In other words, the crop map 400 shows the distribution of multiple crops in the target farm. In the crop map 400, for example, the location of each crop is shown in a different manner depending on the harvest time determination result from the server 3. The crop map 400 is generated, for example, by the control unit 40.

[0126] Figure 12 is a schematic diagram showing an example of a crop map 400. In the crop map 400, for example, the location of each crop is represented by a figure 401. Figure 401 is, for example, a figure that mimics the outline of the crop. Figure 12 shows the figure 401 when the crop is a strawberry. Figure 401 mimics the outline of a strawberry fruit.

[0127] In the crop map 400 shown in Figure 12, the shape 401a representing the location of crops whose harvest time determination result is "today" is shown with upward-sloping hatching. The shape 401b representing the location of crops whose harvest time determination result is "1 day later" is shown with sandy hatching. The shape 401c representing the location of crops whose harvest time determination result is "2 days later" is shown with downward-sloping hatching.

[0128] Furthermore, the crop map 400 may show a figure 401 representing the location of crops whose harvest time determination result is "10 days or more from now," or a figure 401 representing the location of crops whose harvest time determination result is "5 days or more and 9 days or less from now." In addition, the crop map 400 may show a figure 401 representing the location of crops whose harvest time determination result is "1 day from now," or a figure 401 representing the location of crops whose harvest time determination result is "2 days from now."

[0129] In this way, the crop map 400 shows, for example, the location of each crop in a different manner depending on the harvest time determination result. When such a crop map 400 is displayed on the display unit 47 of the terminal device 4, the operator, who is the terminal user, can easily identify the location of each crop and the harvest time for each crop.

[0130] Furthermore, the method of showing the location of each crop in the crop map 400 in different ways for each harvest time determination result is not limited to the example in Figure 12. For example, in the crop map 400, the locations of crops whose harvest time determination result is "today," the locations of crops whose harvest time determination result is "1 day later," and the locations of crops whose harvest time determination result is "2 days later" may be shown by shapes 401 of different colors.

[0131] Information about a crop (also called second crop-related information) may be associated with the location of a crop on the crop map 400. In this case, the second crop-related information is individually associated with the location of each crop on the crop map 400.

[0132] The second crop-related information for a particular crop may include, for example, a target image of the crop, the crop ID of the crop, the crop location information of the crop, and the harvest time determination result of the crop. Note that the second crop-related information does not necessarily have to include a target image.

[0133] For example, when the input unit 48 receives a selection input to select a crop location 405 on the crop map 400 displayed on the display unit 47, the control unit 40 displays a crop information screen 410 showing second crop-related information associated with the selected location 405, as shown in Figure 13. The selection of a crop location 405 on the crop map 400 can also be seen as the selection of that crop. Hereafter, the crop corresponding to the selected location 405, that is, the selected crop, will be referred to as the selected crop.

[0134] The crop information screen 410 includes a target image 150 showing the selected crop, the crop ID 411 of the selected crop, the crop location information 412 of the selected crop, and the harvest time determination result 413 of the selected crop. The crop information screen 410 also includes an instruction area 415 for instructing the display unit 47 to hide the crop information screen 410. When the input unit 48 receives a selection input to select the instruction area 415, the display unit 47 hides the crop information screen 410.

[0135] In this example, when the input unit 48 receives a selection input for selecting the location of a crop on the crop map 400, the display unit 47 displays the harvest time determination result for that crop. This allows the operator to check the harvest time determination result for a crop by selecting its location on the crop map 400.

[0136] If the terminal device 4 does not display the target image, it may delete the target image from the storage unit 41 after acquiring the feature quantities of the target image in step s2.

[0137] As described above, in this example, since the server 3 obtains a reconstructed image by recovering the target image based on the feature quantities received from the terminal device 4, it can perform processing using the reconstructed image as a substitute for the target image, even without receiving the target image, which has a large amount of data, from the terminal device 4. Therefore, the communication load between the terminal device 4 and the server 3 can be reduced.

[0138] Furthermore, in this example, since server 3 displays a restored image of the target object, even if the server does not receive the target image, the server user can still see the appearance of the object through the restored image displayed on server 3.

[0139] Furthermore, in this example, server 3 can appropriately determine the harvest time of crops depicted in the target image by determining the harvest time based on the feature quantities of the target image.

[0140] Furthermore, in this example, since server 3 displays the harvest time determination result for the crops shown in the target image, server users can easily check the harvest time determination result for the crops shown in the target image.

[0141] In this example, server 3 displays both the restored image of the target image and the harvest time determination result for the crops depicted in that image. This allows, for example, a server user to check the condition of the crops from the restored image displayed by server 3 and determine the validity of the harvest time determination result for those crops.

[0142] In this example, the features used for classification in multiple classification images, each depicting a different crop, are divided into multiple groups according to the harvest time of the crop. Server 3 then determines the harvest time of the crop depicted in the image by selecting the harvest time corresponding to the group to which the features of the target image belong. In this way, Server 3 can easily determine the harvest time of a crop by selecting the harvest time corresponding to the group to which the features of the target image belong.

[0143] Server 3 may use the feature quantities received from terminal device 4 (also called received features) as determination features. In this case, the control unit 30 of Server 3 determines which of the L groups the received feature belongs to, based on the harvest time determination result of the crop corresponding to the received feature. Specifically, if the harvest time determination result of the crop corresponding to the received feature has not been modified, the control unit 30 includes the received feature in the group corresponding to the harvest time indicated by the unmodified harvest time determination result. For example, if the harvest time determination result of the crop corresponding to the received feature is "two days later" and has not been modified, the received feature is included in the "two days later" group. Also, if the harvest time determination result of the crop corresponding to the received feature has been modified in accordance with instructions from the server user, the control unit 30 includes the received feature in the group corresponding to the harvest time indicated by the modified harvest time determination result. For example, if the modified harvest time determination result of the crop corresponding to the received feature is "two days ago," the received feature is included in the "two days ago" group.

[0144] If a server user determines that the harvest time determination result for a crop of interest is not valid, they may instruct server 3 to correct the harvest time determination result for the crop of interest by specifying the corrected harvest time numerically. For example, a server user may, after confirming the restored image 250 displayed on server 3, select the input area 365 shown in Figure 10, and then make a predetermined input to the input unit 38 to specify the corrected harvest time numerically. In this case, the determination unit 131 corrects the harvest time determination result so that it becomes the numerical value specified by the user.

[0145] The determination unit 131 may determine the harvest time of the crop of interest corresponding to the feature of interest based on the harvest times of multiple determination features in the memory unit 31 that are similar to the feature of interest.

[0146] For example, if the determination unit 131 determines in step s13 that the feature of interest belongs to the 2-day later group, it may determine the harvest time of the crop of interest corresponding to the feature of interest based on the harvest times of multiple determination features similar to the feature of interest. This makes it possible to determine the harvest time of the crop of interest in detail.

[0147] Here, the feature quantities used for determination that belong to the 2-day group are called the 2-day corresponding features, the feature quantities used for determination that belong to the 1-day group are called the 1-day corresponding features, and the feature quantities used for determination that belong to the harvest group are called the harvest time corresponding features. Also, the feature quantities used for determination that belong to the 1-day group are called the 1-day previous corresponding features, and the feature quantities used for determination that belong to the 2-day group are called the 2-day previous corresponding features. Furthermore, among the multiple feature quantities used for determination in the memory unit 31, the feature quantities used for determination that belong to groups other than the 10-day or later group and the 5-9 day later group are called specific determination features. The multiple specific determination features in the memory unit 31 include the 1-day corresponding features, the 2-day corresponding features, the harvest time corresponding features, the 1-day before and after corresponding features, and the 2-day corresponding features.

[0148] The harvest time (in other words, the harvest day) corresponding to the feature corresponding to 2 days later is "2 days later". The harvest time corresponding to the feature corresponding to 1 day later is "1 day later". The harvest time corresponding to the feature corresponding to the time of harvest is "today". The harvest time corresponding to the feature corresponding to 1 day ago is "1 day ago". The harvest time corresponding to the feature corresponding to 2 days ago is "2 days ago".

[0149] In step s13, the determination unit 131 may, for example, determine the harvest time of the crop of interest corresponding to the feature of interest based on the harvest times of a plurality of specific determination features similar to the feature of interest, if it determines that the feature of interest belongs to the group one day later, if it determines that the feature of interest belongs to the harvest group, or if it determines that the feature of interest belongs to the group one day earlier.

[0150] If the determination unit 131 determines that the feature of interest belongs to the 1-day group, it sets a plurality of specific determination features whose Euclidean distance from the feature of interest is less than or equal to a first predetermined value in the feature space (also called the second specific feature space) where the feature of interest and a plurality of specific determination features in the memory unit 31 are arranged, and sets them as a plurality of specific determination features similar to the feature of interest. Then, the determination unit 131 determines the harvest time of the crop of interest corresponding to the feature of interest based on the harvest times corresponding to the plurality of specific determination features similar to the feature of interest. For example, the determination unit 131 calculates the average of the harvest times corresponding to the plurality of specific determination features similar to the feature of interest, and sets the calculated average as the harvest time of the crop of interest.

[0151] Here, the determination unit 131 expresses the harvest time corresponding to a specific determination feature as a ± value relative to "today". Specifically, the determination unit 131 expresses the harvest time corresponding to a feature corresponding to harvest time as "0 days", the harvest time corresponding to a feature corresponding to one day later as "+1 day", the harvest time corresponding to a feature corresponding to two days later as "+2 days", the harvest time corresponding to a feature corresponding to one day ago as "-1 day", and the harvest time corresponding to a feature corresponding to two days ago as "-2 days". Then, the determination unit 131 uses these values ​​to calculate the average of the harvest times corresponding to multiple specific determination features that are similar to the feature of interest.

[0152] For example, suppose the harvest times corresponding to multiple specific determination features similar to the feature of interest belonging to the "1 day later" group are "+1 day", "+1 day", "+1 day", "0 days", and "0 days". In this case, the average of the harvest times corresponding to multiple specific determination features similar to the feature of interest is "+0.6 days". The determination unit 131 determines that the harvest time of the crop of interest is "0.6 days later".

[0153] As another example, suppose the harvest times corresponding to multiple specific determination features similar to the feature of interest belonging to the harvest group are "0 days", "0 days", "0 days", "+1 day", and "+1 day". In this case, the average of the harvest times corresponding to the multiple specific determination features similar to the feature of interest is "+0.4 days". The determination unit 131 determines that the harvest time for the crop of interest is "0.4 days later".

[0154] As another example, suppose the harvest times corresponding to multiple specific determination features similar to the feature of interest belonging to the "1 day ago" group are "-1 day", "-1 day", "-1 day", "-2 days", and "-2 days". In this case, the average of the harvest times corresponding to multiple specific determination features similar to the feature of interest is "-1.4 days". The determination unit 131 determines that the harvest time of the crop of interest is "1.4 days ago".

[0155] In step s14, the server 3 may display, together with the reconstructed image 250, a determination image from among the multiple clear determination images in the storage unit 31 that has determination features similar to the feature of interest. A determination image that has determination features similar to the feature of interest can be said to be an image similar to the target image that is the basis for the reconstructed image 250.

[0156] Hereafter, images with features similar to those of the target image may be referred to as similar images. Additionally, similar images with features similar to those of the feature of interest may be referred to as feature-similar images. In this example, similar images are judgment images with judgment features similar to those of the target image, and feature-similar images are judgment images with judgment features similar to those of the feature of interest.

[0157] Figure 14 is a schematic diagram showing an example of a display screen 330 when the focus-restored image 250 and a focus-similar image 450 that is similar to the focus-target image from which the focus-restored image 250 is derived are displayed together.

[0158] In the example shown in Figure 14, the display unit 37 displays a display screen 330 that includes the restored image of interest 250, the harvest time determination result 350 for the crop of interest, the similar image of interest 450, an input area 360, and an input area 365. In the display screen 330 of the example in Figure 14, the string "Restored Image" is shown around the restored image of interest 250 (for example, below it), and the string "Similar Image" is shown around the similar image of interest 450 (for example, below it).

[0159] The control unit 30, in a specific feature space where the feature of interest and a plurality of determination feature quantities in the storage unit 31 are arranged, selects a determination feature quantity from among the plurality of determination feature quantities that has the smallest Euclidean distance from the feature of interest and that has a Euclidean distance of less than or equal to a first predetermined value as a determination feature quantity similar to the feature of interest. The control unit 30 then displays the feature of interest similarity image 450, which is a determination image having a determination feature quantity similar to the feature of interest, and the feature of interest restoration image 250 together on the display unit 37, as shown in Figure 14.

[0160] Furthermore, if there are multiple determination features in the memory unit 31 that are similar to the feature of interest, multiple determination images (i.e., multiple similar images of interest 450) each having multiple determination features similar to the feature of interest may be displayed together with the restored image of interest 250. Also, if there are no determination features in the memory unit 31 whose Euclidean distance from the feature of interest is less than or equal to a first predetermined value, the display in the display unit 37 will be the same as in Figure 10.

[0161] Thus, when a similar image of interest, which is the basis for the restored image of interest, is displayed, the server user can indirectly confirm the state of the crop of interest from the state of the crop shown in the similar image of interest displayed on server 3. Therefore, if the restored image of interest displayed on server 3 is unclear, the server user can indirectly confirm the state of the crop of interest from the clear similar image of interest displayed on server 3. Consequently, if the restored image of interest is unclear, the server user can confirm the state of the crop of interest from the similar image of interest and then determine the validity of the harvest time determination result for the crop of interest displayed on server 3. The similar image of interest can also be said to be a reference image for the restored image of interest, and therefore can be called a reference image of interest or a similar reference image of interest.

[0162] If no similar image of interest exists among the multiple judgment images in the storage unit 31, the server 3 may acquire the image of interest from the terminal device 4 and display the acquired image of interest. Figure 15 is a flowchart showing an example of the operation of the server 3 in this case.

[0163] After the server 3 executes step s13 described above, it executes step s21 as shown in Figure 15. In step s21, the control unit 30 determines whether or not a similar image of interest exists among the multiple determination images in the storage unit 31. For example, if there are no determination features among the multiple determination features in the storage unit 31 whose Euclidean distance from the feature of interest is less than or equal to a first predetermined value, the control unit 30 determines that a similar image of interest does not exist among the multiple determination images in the storage unit 31. If the determination in step s21 is YES, the above-described step s14 is executed, and the display unit 37 displays a display screen 330 including the restored image of interest 250 and the similar image of interest 450, as shown in Figure 14.

[0164] On the other hand, if NO is determined in step s21, in step s22 the control unit 40 generates transmission instruction information instructing the transmission of the target image, and the server 3 transmits the transmission instruction information to the terminal device 4.

[0165] In step s31, the terminal device 4 receives the transmission instruction information, and in step s32, the control unit 40 acquires the target image from the storage unit 41. The terminal device 4 then transmits the target image acquired from the storage unit 41 to the server 3.

[0166] In step s23, the server 3 acquired the image of interest, and in step s24, the display unit 37 displays the image of interest from the terminal device 4.

[0167] Figure 16 is a schematic diagram showing an example of the display of the target image 150 on the display unit 37. As shown in Figure 16, in step s24, the display unit 37 displays a display screen 340 that includes, for example, the target image 150, the harvest time determination result 350 of the crop of interest, an input area 360, and an input area 365. On the display screen 340, for example, the string "Original Image" is displayed around (for example, below) the target image 150. The operation of the server 3 after step s24 is the same as the operation after step s14.

[0168] In this way, if the server 3 finds no similar image of interest among the multiple judgment images in the storage unit 31, it acquires the image of interest from the terminal device 4 and displays the acquired image of interest, allowing the server user to check the status of the crop of interest from the image of interest 150 displayed by the server 3.

[0169] As shown in Figure 17, the display unit 37 may display the restored image 250 together with the image of interest 150. Alternatively, after displaying the display screen 330 including the restored image 250 in step s14, the server 3 may, in response to instructions from the server user, acquire the image of interest from the terminal device 4 and display the display screen 340 including the acquired image of interest. Figure 18 is a schematic diagram showing an example of the display screen 330 displayed in step s14 in this case.

[0170] In the example shown in Figure 18, the display screen 330 includes an instruction area 368 in addition to the restored image of interest 250, the harvest time determination result 350, the input area 360, and the input area 365. The instruction area 368 is an area for instructing the display of the target image, which is the original image of the restored image of interest 250. When the input unit 38 receives a selection operation to select the instruction area 368, the control unit 30 generates transmission instruction information instructing the transmission of the target image of interest, and the server 3 transmits this transmission instruction information to the terminal device 4. The terminal device 4, having received the transmission instruction information, transmits the target image of interest 150 from the storage unit 41 to the server 3. The server 3, having received the target image of interest 150, displays a display screen 340 including the acquired target image of interest 150, as shown in the example in Figure 16 or Figure 17.

[0171] The terminal device 4 may delete the image of interest from the storage unit 41 after transmitting the image of interest to the server 3. Alternatively, if the terminal device 4 receives the harvest time determination result for the crop of interest from the server 3 without transmitting the image of interest to the server 3, it may delete the image of interest from the storage unit 41.

[0172] The feature acquisition unit 141 of the terminal device 4 may include multiple types of encoding models 101 capable of acquiring feature quantities of a target image. These multiple types of encoding models 101 may include, for example, a general-purpose model 101, a light-color model 101 corresponding to a target image containing light-colored crops, and a dark-color model 101 corresponding to a target image containing dark-colored crops. Furthermore, these multiple types of encoding models 101 may include, for example, a large-size model 101 corresponding to a target image containing large-sized crops, a small-size model 101 corresponding to a target image containing small-sized crops, and a damaged crop model 101 corresponding to a target image containing damaged crops.

[0173] The light-colored model 101 can also be described as a model suited to light-colored crops. The dark-colored model 101 can also be described as a model suited to dark-colored crops. The large-size model 101 can also be described as a model suited to large-size crops. The small-size model 101 can also be described as a model suited to small-size crops. The damaged model 101 can also be described as a model suited to crops with damage. The light color of the crop, the dark color of the crop, the large size of the crop, the small size of the crop, and the presence of damage to the crop can all be described as conditions of the crop. The light-colored model 101, the dark-colored model 101, the large-size model 101, the small-size model 101, and the damaged model can each be described as a model suited to a condition that the crop may be in.

[0174] The recovery unit 130 of server 3 has a decode model 102 corresponding to the general-purpose model 101 (also called the general-purpose model 102), a decode model 102 corresponding to the light-color model 101 (also called the light-color model 102), and a decode model 102 corresponding to the dark-color model 101 (also called the dark-color model 102). Furthermore, the recovery unit 130 has a decode model 102 corresponding to the large-size model 101 (also called the large-size model 102), a decode model 102 corresponding to the small-size model 101 (also called the small-size model 102), and a decode model 102 corresponding to the scratched model 101 (also called the scratched model 102).

[0175] The generative model 100 (also referred to as the general-purpose generative model 100), which has general-purpose models 101 and 102, is trained based on multiple training images showing crops in various states at the harvest stage. The multiple training images used to train the general-purpose generative model 100 include, for example, training images showing light-colored crops, training images showing dark-colored crops, training images showing large crops, training images showing small crops, training images showing crops with blemishes, and training images showing crops without blemishes.

[0176] The generative model 100 (generative model 100 for light-colored crops), which has light-colored models 101 and 102, is trained based only on multiple training images showing light-colored crops at the harvest-expected stage. In the trained generative model 100 for light-colored crops, when a target image showing light-colored crops is input to the light-colored model 101, the light-colored model 102 becomes more likely to obtain a clear reconstructed image. Alternatively, a general-purpose generative model 100, which has undergone additional training such as fine-tuning based on multiple training images showing light-colored crops at the harvest-expected stage, may be used as the trained generative model 100 for light-colored crops.

[0177] The generative model 100 (generative model 100 for dark colors), which has models 101 and 102 for dark colors, is trained based only on multiple training images showing dark-colored crops at the harvest-expected stage. In the trained generative model 100 for dark colors, when a target image showing dark-colored crops is input to model 101 for dark colors, model 102 for dark colors becomes more likely to obtain a clear reconstructed image. Alternatively, a general-purpose generative model 100, which has undergone additional training such as fine-tuning based on multiple training images showing dark-colored crops at the harvest-expected stage, may be used as the trained generative model 100 for dark colors.

[0178] The generative model 100 (large-size generative model 100), which has large-size models 101 and 102, is trained based only on multiple training images showing large crops at the expected harvest stage. In the trained large-size generative model 100, when a target image showing a large crop is input to the large-size model 101, the large-size model 102 becomes more likely to obtain a clear reconstructed image. Alternatively, a general-purpose generative model 100, which has undergone additional training such as fine-tuning based on multiple training images showing large crops at the expected harvest stage, may be used as the trained large-size generative model 100.

[0179] The generative model 100 (also called the small-size generative model 100), which has small-size models 101 and 102, is trained based only on multiple training images showing small crops at the expected harvest stage. In the trained small-size generative model 100, when a target image showing small crops is input to the small-size model 101, the small-size model 102 becomes more likely to obtain a clear reconstructed image. Alternatively, a general-purpose generative model 100, which has undergone additional training such as fine-tuning based on multiple training images showing small crops at the expected harvest stage, may be used as the trained small-size generative model 100.

[0180] The generative model 100 (also called the damaged generative model 100), which has damaged models 101 and 102, is trained based only on multiple training images showing damaged crops at the expected harvest stage. In the trained damaged generative model 100, when a target image showing damaged crops is input to the damaged model 101, the damaged model 102 becomes more likely to obtain a clear reconstructed image. Alternatively, a general-purpose generative model 100, which has undergone additional training such as fine-tuning based on multiple training images showing damaged crops at the expected harvest stage, may be used as the trained damaged generative model 100.

[0181] When the terminal device 4 obtains a reconstructed image based on features acquired using a certain type of encoding model 101, the server 3 obtains the reconstructed image using a decoding model 102 corresponding to that encoding model 101. For example, when the terminal device 4 obtains a reconstructed image based on features acquired using a general-purpose model 101, the server 3 obtains the reconstructed image using a general-purpose model 102 corresponding to the general-purpose model 101. Also, when the terminal device 4 obtains a reconstructed image based on features acquired using a dark color model 101, the server 3 obtains the reconstructed image using a dark color model 102 corresponding to the dark color model 101.

[0182] In this example, during the crop unit determination process, Server 3 instructs Terminal Device 4 to change the encoding model 101 in response to instructions from the server user. Terminal Device 4 changes the encoding model 101 to be used in response to the instruction from Server 3 to change the encoding model 101. Then, Terminal Device 4 acquires new focus features of the target image using the changed encoding model 101 and sends the newly acquired focus features to Server 3. Server 3 acquires a newly restored focus image by reconstructing the target image based on the new focus features from Terminal Device 4. At this time, Server 3 acquires a newly restored focus image using a decoding model 102 corresponding to the changed encoding model 101 used to acquire the new focus features. Then, Server 3 displays the newly acquired restored focus image.

[0183] Figure 19 is a flowchart showing an example of the crop unit determination process in this example. As shown in Figure 19, the server 3 executes step s40 after step s13, or after determining YES in step s21 (see Figure 15). In step s40, the display unit 37 displays a display screen 331 which includes the target restoration image 250, an input area 360, an input area 365, and instruction areas 501 to 506 for instructing a change in the encoding model 101 used by the terminal device 4. Figure 20 is a schematic diagram showing an example of the display screen 331. The display screen 331 may or may not include the target similar image 450, as in the example in Figure 20.

[0184] Instruction area 501 is an area for instructing the terminal device 4 to change the encoding model 101 (also called the model used 101) to the light color model 101. Instruction area 502 is an area for instructing the model used 101 to change to the dark color model 101. Instruction area 503 is an area for instructing the model used 101 to change to the large size model 101. Instruction area 504 is an area for instructing the model used 101 to change to the small size model 101. Instruction area 505 is an area for instructing the model used 101 to change to the scratched model 101.

[0185] In this example, in step s1 of the crop unit determination process, the general-purpose model 101 is used to acquire feature quantities. The input unit 38 receives a selection operation to select the instruction area 501, and when the server user instructs the server 3 to change the model 101 to the light-color model 101 (step s41), in step s42, the control unit 30 generates first change instruction information instructing the server to change the model 101 to the light-color model 101. The server 3 then transmits the first change instruction information to the terminal device 4.

[0186] When the input unit 38 receives a selection operation to select the instruction area 502, and the server user instructs the server 3 to change the model used 101 to the dark color model 101 (step s41), in step s42, the control unit 30 generates second change instruction information instructing the server to change the model used 101 to the dark color model 101. The server 3 then transmits the second change instruction information to the terminal device 4.

[0187] When the input unit 38 receives a selection operation to select the instruction area 503, and the server user instructs the server 3 to change the model used 101 to the size-specific model 101 (step s41), in step s42, the control unit 30 generates a third change instruction information instructing to change the model used 101 to the larger size-specific model 101. The server 3 then transmits the third change instruction information to the terminal device 4.

[0188] When the input unit 38 receives a selection operation to select the instruction area 504, and the server user instructs the server 3 to change the model used 101 to the smaller size model 101 (step s41), in step s42, the control unit 30 generates a fourth change instruction information instructing the server to change the model used 101 to the smaller size model 101. The server 3 then transmits the fourth change instruction information to the terminal device 4.

[0189] When the input unit 38 receives a selection operation to select the instruction area 505, and the server user instructs the server 3 to change the model used 101 to the model used for damaged items (step s41), in step s42, the control unit 30 generates a fifth change instruction information instructing the server to change the model used 101 to the model used for damaged items. The server 3 then transmits the fifth change instruction information to the terminal device 4.

[0190] For example, the server user compares the restored image 250 of interest included in the display screen 331 with a clearer similar image 450 of interest, or looks only at the restored image 250 of interest, and if they determine that the restored image 250 of interest is unclear and it is difficult to determine the state of the crop of interest shown in the restored image 250, they select one of the instruction areas 501 to 505. Also, for example, the server user determines that the restored image 250 of interest is significantly different from the similar image 450 of interest, they select one of the instruction areas 501 to 505.

[0191] For example, if the server user determines that the target reconstructed image 250 is unclear and that the target crop is faintly colored in the target reconstructed image 250, they select the indication area 501. Alternatively, if the server user determines that the target reconstructed image 250 is significantly different from the target similar image 450 and that the target crop is faintly colored in the target reconstructed image 250, they also select the indication area 501.

[0192] For example, if the server user determines that the reconstructed image 250 of interest is unclear and contains a dark-colored crop of interest, they select the indicated area 502. Alternatively, if the server user determines that the reconstructed image 250 of interest is significantly different from the similar image 450 of interest and contains a dark-colored crop of interest, they also select the indicated area 502.

[0193] For example, if the server user determines that the focus image 250 is unclear and contains a large focus crop, they select the indicated area 503. Alternatively, if the server user determines that the focus image 250 is significantly different from the focus similar image 450 and contains a large focus crop, they also select the indicated area 503.

[0194] For example, if the server user determines that the reconstructed image 250 of interest is unclear and contains a small crop of interest, they select the indicated area 504. Alternatively, if the server user determines that the reconstructed image 250 of interest is significantly different from the similar image 450 of interest and contains a small crop of interest, they also select the indicated area 504.

[0195] For example, if the server user determines that the restored image 250 is unclear and that the restored image 250 contains a crop of interest with damage, they select the indicated area 505. Alternatively, if the server user determines that the restored image 250 is significantly different from the similar image 450 and that the restored image 250 contains a crop of interest with damage, they also select the indicated area 505.

[0196] In step s51, when the terminal device 4 receives the first change instruction information, in step s52, the feature acquisition unit 141 changes the model 101 used to the light color model 101. Then, the feature acquisition unit 141 uses the light color model 101 to acquire new feature quantities of interest from the image of interest. In step s53, the newly acquired feature quantities of interest are transmitted from the terminal device 4 to the server 3.

[0197] In step s51, when the terminal device 4 receives the second change instruction information, in step s52, the feature acquisition unit 141 changes the model 101 used to the dark color model 101. Then, the feature acquisition unit 141 uses the dark color model 101 to acquire new feature quantities of interest from the image of interest. In step s53, the newly acquired feature quantities of interest are transmitted from the terminal device 4 to the server 3.

[0198] In step s51, when the terminal device 4 receives the third change instruction information, in step s52, the feature acquisition unit 141 changes the model 101 used to the larger size model 101. Then, the feature acquisition unit 141 uses the larger size model 101 to acquire new feature quantities of interest from the image of interest. In step s53, the newly acquired feature quantities of interest are transmitted from the terminal device 4 to the server 3.

[0199] In step s51, when the terminal device 4 receives the fourth change instruction information, in step s52, the feature acquisition unit 141 changes the model 101 used to the smaller size model 101. Then, the feature acquisition unit 141 uses the smaller size model 101 to acquire new feature quantities of interest from the image of interest. In step s53, the newly acquired feature quantities of interest are transmitted from the terminal device 4 to the server 3.

[0200] In step s51, when the terminal device 4 receives the fifth change instruction information, in step s52, the feature acquisition unit 141 changes the model 101 used to the model 101 for images with scratches. The feature acquisition unit 141 then uses the model 101 for images with scratches to acquire new feature quantities of the image of interest. In step s53, the newly acquired feature quantities of interest are transmitted from the terminal device 4 to the server 3.

[0201] In step s43, the server 3 receives a new feature of interest from the terminal device 4. In step s44, the restoration unit 130 uses a decoding model 102 corresponding to the model 101 used to acquire the new feature of interest to restore the target image based on the new feature of interest. Next, in step s45, the display unit 37 displays the new restored image of interest acquired by the restoration unit 130. For example, the display unit 37 displays the new restored image of interest 250 on the display screen 331, replacing the previously displayed restored image of interest 250 (in other words, the old restored image of interest 250).

[0202] After step s45, if one of the instruction areas 501 to 505 on the display screen 331 is selected (step s41), the server 3 executes step s42, and the processing system 1 operates similarly thereafter. Also, after step s45, if either input area 360 or 365 on the display screen 331 is selected, step s15 is executed, and the processing system 1 operates similarly thereafter. Furthermore, if, after step s40, step s41 is not executed and either input area 360 or input area 365 on the display screen 331 is selected, step s15 is executed, and the processing system 1 operates similarly thereafter.

[0203] In this example, the crop unit determination process involves changing the model 101 used in response to instructions from the server user, and then using the modified model 101 to acquire new features of interest. Based on these newly acquired features of interest, a reconstructed image of the target image is then acquired and displayed. This makes it easier to display a clear reconstructed image of the target, allowing the server user to more easily confirm the status of the crop of interest from the reconstructed image.

[0204] In the crop unit determination process, the processing system 1 may modify the restored image in response to instructions from the server user. Figure 21 is a schematic diagram showing an example of the functional blocks provided by the terminal device 4 and the server 3 in this case. In the example in Figure 21, the control unit 40 of the terminal device 4 includes a difference acquisition unit 142 as a functional block. The control unit 30 of the server 3 also includes an extraction unit 132 as a functional block.

[0205] Figure 22 is a flowchart showing an example of the crop unit determination process performed by the processing system 1 shown in Figure 21. In the example in Figure 22, the server 3 executes step s60 after step s13, or after determining YES in step s21 (see Figure 15). In step s60, the display unit 37 displays a display screen 332 which includes the focus restoration image 250, an input area 360, an input area 365, and an instruction area 510 for instructing the correction of the focus restoration image 250. Figure 23 is a schematic diagram showing an example of the display screen 332. The display screen 332 may or may not include the focus similar image 450, as in the example in Figure 23.

[0206] When the instruction area 510 is selected, in other words, when the input unit 38 receives a correction instruction for the focus image 250 from the server user (step s61), step s62 is executed. For example, if the server user checks the display on the display unit 37 and determines that the focus image 250 is unclear, they select the instruction area 510 and give a correction instruction to the server 3 for the focus image 250. Alternatively, if the server user determines that the focus image 250 is significantly different from the focus similar image 450, they select the instruction area 510.

[0207] In step s62, the extraction unit 132 of the control unit 30 extracts the crop of interest region 251, in which the crop of interest is visible, from the reconstructed image of interest 250. The crop of interest region 251 can also be described as the region obtained by removing the background region from the reconstructed image of interest 250. Figure 24 is a schematic diagram showing an example of the crop of interest region 251. The crop of interest region 251 can also be described as the object region in which the object is visible, or as the treatment target region in which the treatment target is visible. Furthermore, the crop of interest region 251 can also be described as the image in which the object is visible.

[0208] The extraction unit 132 may extract the crop region 251 of interest from the reconstructed image 250 of interest using, for example, semantic segmentation or instance segmentation. Semantic segmentation and instance segmentation are implemented using, for example, a neural network. The extraction unit 132 extracts the crop region 251 of interest from the reconstructed image 250 of interest using, for example, a machine learning model that performs semantic segmentation or instance segmentation.

[0209] In step s62, when the crop region 251 of interest is extracted from the restored image 250 of interest, the control unit 30 generates third crop-related information that includes the crop region 251 of interest, which is a type of image, and the respective pixel positions of the multiple pixels that constitute the crop region 251 of interest. The pixel positions can also be called, for example, the coordinates of the pixels. Then, in step s62, the server 3 transmits the third crop-related information to the terminal device 4.

[0210] In step s71, the terminal device 4 acquires third crop-related information. In step s72, the difference acquisition unit 142 of the control unit 40 acquires the image of interest from the storage unit 41. The difference acquisition unit 142 then acquires difference information showing the difference between the image of interest and the third crop-related information for the crop region 251.

[0211] Here, pixels in the image of interest are called first pixels, and pixels in the restored image of interest are called second pixels. Based on the third crop-related information, the difference acquisition unit 142 identifies second pixels (also called differing second pixels) in a plurality of second pixels constituting the crop region of interest 251 whose difference from the corresponding first pixel is greater than or equal to a threshold. The first pixel corresponding to the second pixel is the first pixel located at the same pixel position as the second pixel. The difference between the second pixel and the first pixel corresponding to the second pixel can also be said to be the difference between a pixel value in the crop region of interest and a pixel value in the image of interest at the same pixel position as that pixel value. The difference acquisition unit 142 then uses information including at least one first pixel corresponding to at least one differing second pixel included in the crop region of interest 251, and the pixel position of that at least one first pixel, as difference information indicating the difference between the crop region of interest 251 and the image of interest.

[0212] When the differential information is obtained in step s72, in step s73, the terminal device 4 transmits the differential information to the server 3.

[0213] In step s63, the server 3 receives the difference information, and in step s64, the restoration unit 130 modifies the focus image 250 based on the difference information. Specifically, the restoration unit 130 modifies the focus image 250 by replacing each first pixel included in the difference information with a second pixel located at the same pixel position as the first pixel in the focus image 250. The modified focus image 250 (also called the modified focus image 250a) is displayed in step s65 by the display unit 37. After step s65, step s15 is executed, and thereafter the processing system 1 operates in the same manner.

[0214] Figure 25 is a schematic diagram showing an example of the display of the display unit 37 in step s65. As shown in Figure 25, the display unit 37 displays a display screen 345 that includes, for example, the corrected focus restoration image 250a, the harvest time determination result 350, an input area 360, and an input area 365. On the display screen 345, for example, the string "Corrected Restoration Image" is displayed around (for example, below) the corrected focus restoration image 250a. The display screen 345 may or may not include the focus similar image 450, as in the example in Figure 25.

[0215] If, after step s60, step s61 is not executed and input area 360 or input area 365 in the display screen 332 is selected, step s15 is executed, and thereafter the processing system 1 operates in the same manner.

[0216] In step s62, the server 3 may transmit third crop-related information, including the reconstructed image of interest, to the terminal device 4 instead of the crop region of interest 251. In this case, in step s72, the terminal device 4's difference acquisition unit 142 acquires difference information (also called overall difference information) indicating the difference between the reconstructed image of interest 250 and the target image of interest. The difference acquisition unit 142 identifies a second pixel, i.e., a difference second pixel, in which the difference with the corresponding first pixel is greater than or equal to a threshold value among a plurality of second pixels constituting the reconstructed image of interest 250 from the server 3. The difference acquisition unit 142 then uses information including at least one first pixel corresponding to at least one difference second pixel included in the reconstructed image of interest 250, and the pixel position of said at least one first pixel, as overall difference information indicating the difference between the reconstructed image of interest 250 and the target image of interest. Once the overall difference information is acquired in step s72, in step s73, the terminal device 4 transmits the overall difference information to the server 3.

[0217] In step s63, the server 3 receives the overall difference information, and in step s64, the restoration unit 130 modifies the focus restoration image 250 based on the overall difference information. Specifically, the restoration unit 130 modifies the focus restoration image 250 by replacing each first pixel included in the overall difference information with a second pixel located at the same pixel position as the first pixel in the focus restoration image 250. The modified focus restoration image 250, that is, the modified focus restoration image 250a, is displayed on the display unit 37 in step s65, as described above.

[0218] In step s62, the server 3 may transmit third crop-related information to the terminal device 4, which includes the area of ​​interest crop region 251 extracted from the restored image of interest 250 and specified by the server user (also called the user-specified area). In this case, the display unit 37 displays, for example, the area of ​​interest crop region 251 extracted by the extraction unit 132. When the area of ​​interest crop region 251 is displayed on the display unit 37, the server user makes a predetermined input to the input unit 38 to specify the area of ​​interest crop region 251 that needs to be corrected (for example, an unclear area or an area that is significantly different from the similar image of interest). The area specified by the server user can also be said to be the area of ​​interest for the server user. The server 3 transmits third crop-related information to the terminal device 4, which includes the user-specified area (in other words, the user-focused area) from the area of ​​interest crop region 251.

[0219] In the terminal device 4, in step s72, the difference acquisition unit 142 acquires difference information (also called specified area difference information) that shows the difference between the user-specified area and the target image of interest. The difference acquisition unit 142 identifies a second pixel, i.e., a difference second pixel, in which the difference from the corresponding first pixel is greater than or equal to a threshold value among a plurality of second pixels constituting the user-specified area from the server 3. Then, the difference acquisition unit 142 uses information including at least one first pixel corresponding to at least one difference second pixel included in the user-specified area, and the pixel position of said at least one first pixel, as specified area difference information that shows the difference between the user-specified area and the target image of interest. Once the specified area difference information is acquired in step s72, in step s73, the terminal device 4 transmits the specified area difference information to the server 3.

[0220] In step s63, the server 3 receives the specified area difference information, and in step s64, the restoration unit 130 modifies the focus restoration image 250 based on the specified area difference information. Specifically, the restoration unit 130 modifies the focus restoration image 250 by replacing each first pixel included in the specified area difference information with a second pixel located at the same pixel position as the first pixel in the focus restoration image 250. The modified focus restoration image 250, that is, the modified focus restoration image 250a, is displayed on the display unit 37 in step s65, as described above.

[0221] In this example, server 3 transmits at least a portion of the restored image of interest to terminal device 4 in response to instructions from the server user. Terminal device 4 then acquires difference information indicating the difference between at least a portion of the restored image of interest and the target image, and transmits the acquired difference information to server 3. Server 3 modifies the restored image of interest based on the difference information and displays the modified restored image of interest. As a result, for example, if the restored image of interest was unclear, it will become clearer when modified based on the difference information. In this case, the modification of the restored image of interest can also be said to be a sharpening of the restored image of interest. The server user can more easily check the condition of the crop of interest from the modified, clearer restored image of interest displayed on server 3. Also, if the restoration accuracy of the restored image of interest is poor and it differs significantly from the similar image of interest, the restored image of interest will become closer to the original target image when modified based on the difference information. Therefore, the server user can more easily grasp the condition of the crop of interest from the modified restored image of interest displayed on server 3.

[0222] In the crop unit determination process, when Server 3 displays a similar image of interest, it may display either the image as shown in Figure 20 or the image as shown in Figure 23, depending on the similarity (also called feature similarity) between the feature quantities of the similar image of interest and the feature quantity of interest. Specifically, Server 3 may display the image as shown in Figure 20 when the feature similarity is low, and display the image as shown in Figure 23 when the feature similarity is high. An example of the operation of the processing system 1 in this case will be described below.

[0223] As described above, in the specific feature space, the determination image is the determination image that has the determination feature with the smallest Euclidean distance from the feature of interest, and whose said Euclidean distance is less than or equal to a first predetermined value. The control unit 30 determines that the feature similarity is high if, in the specific feature space, the Euclidean distance between the features of the feature similar image and the feature of interest is less than a second predetermined value which is smaller than the first predetermined value. When the feature similarity is high, the feature similar image can be said to be very similar to the image of interest. On the other hand, the control unit 30 determines that the feature similarity is low if, in the specific feature space, the Euclidean distance between the features of the feature similar image and the feature of interest is greater than or equal to a second predetermined value and less than or equal to a first predetermined value. When the feature similarity is low, the feature similar image can be said to be reasonably similar to the image of interest.

[0224] If the control unit 30 determines that the feature similarity is high, the display unit 37 displays the information as shown in Figure 23. After that, the operation of the processing system 1 is the same as in the example shown in Figure 22. On the other hand, if the control unit 30 determines that the feature similarity is low, the display unit 37 displays the information as shown in Figure 20. After that, the operation of the processing system 1 is the same as in the example shown in Figure 19.

[0225] Furthermore, after step s13, or after determining YES in step s21 (see Figure 15), the server 3 may display a display screen 333 that includes instruction areas 501 to 505 for instructing a change in the model used 101 and an instruction area 510 for instructing a modification of the image of interest. Figure 26 is a schematic diagram showing an example of the display screen 333. As shown in Figure 26, the display screen 333 is, for example, the display screen 331 shown in Figure 20 with the instruction area 510 added. The operation of the processing system 1 when any one of the input areas 360, 365, instruction areas 501 to 505, and instruction area 510 is selected is the same as described above.

[0226] In the crop unit determination process, the server 3 may determine, based on the restored image, whether or not there are any defects on the crop (in other words, the object) shown in the restored image. Figure 27 is a schematic diagram showing an example of a functional block provided by the server 3 in this case. In the example of Figure 27, the control unit 30 of the server 3 includes a defect determination unit 135 as a functional block that determines, based on the restored image, whether or not there are any defects on the crop (in other words, the object) shown in the restored image.

[0227] The scratch detection unit 135 may be composed of, for example, a machine learning model. Hereafter, the machine learning model that constitutes the scratch detection unit 135 may be referred to as the scratch detection model. The scratch detection model is composed of a neural network.

[0228] The damage detection model receives a reconstructed image as input. Based on the input reconstructed image, the damage detection model determines whether or not there are damages on the crop of interest shown in the reconstructed image. The damage detection model outputs the result of whether or not damages are present on the crop shown in the input reconstructed image.

[0229] A blemish detection model is, for example, trained using supervised learning. Multiple training datasets are used in training the blemish detection model. Each training dataset includes a training image of a crop and training data indicating whether or not the crop has blemishes. Each of the multiple training datasets contains multiple training images, including training images of crops with blemishes and training images of crops without blemishes.

[0230] Figure 28 is a flowchart showing an example of the crop unit determination process. As shown in Figure 28, the server 3 executes step s80 after step s13, or after a YES determination in step s21 (see Figure 15). In step s80, the damage determination unit 135 determines, based on the restoration image of the subject, whether or not there are damages to the subject crop shown in the restoration image. Determining whether or not there are damages to the subject crop can also be called detecting damage to the subject crop. If it is determined in step s80 that there are no damages to the subject crop (in other words, if no damage to the subject crop is detected), step s40 is executed. Thereafter, the processing system 1 operates in the same manner as in the example in Figure 19.

[0231] On the other hand, if it is determined in step s80 that a defect exists in the crop of interest (in other words, if a defect is detected in the crop of interest), the display unit 37 displays the display screen 335 in step s81. Figure 29 is a schematic diagram showing an example of the display screen 335.

[0232] As shown in Figure 29, the display screen 335 is, for example, the same as the display screen 331 shown in Figure 20, with the addition of notification information 550. The notification information 550 is information that notifies the server user that there is a defect in the crop of interest (grapes in the example of Figure 28) shown in the restored image of interest 250. By displaying the notification information 550, the server user can understand that there is a defect in the crop of interest from the notification information 550, even if the restored image of interest 250 is unclear and it is difficult to see from the restored image of interest 250 that there is a defect in the crop of interest. The display screen 335 may or may not include a similar image of interest 450, as in the example of Figure 29.

[0233] After step s81, when the instruction area 505 included in the display screen 335 is selected (step s41), the processing system 1 operates in the same manner as in the example in Figure 19. As a result, the restored image 250, which has been reconstructed based on the feature quantities of the target image acquired by the scratched image model 101, is displayed on the display screen 335 in place of the old restored image 250.

[0234] Even when notification information 550 is displayed, the server user may still select an instruction area other than instruction area 505 from among the multiple instruction areas 501 to 505 based on the state of the crop of interest shown in the restored image 250. In this case as well, the processing system 1 operates in the same manner as in the example shown in Figure 19.

[0235] In this example, Server 3 determines whether or not there are defects in the crop shown in the reconstructed image based on the reconstructed image of interest. If Server 3 determines that there are defects in the crop of interest, it instructs Terminal Device 4 to change the model 101 in response to instructions from the server user. Terminal Device 4 changes the model 101 to the model 101 for crops with defects in response to the instruction from Server 3 to change the model 101 in response to the instruction from Server 3. Terminal Device 4 then acquires new features of interest using the model 101 for crops with defects and transmits the newly acquired features of interest to Server 3. Server 3 acquires a new reconstructed image of interest based on the new features of interest from Terminal Device 4 and displays the newly acquired reconstructed image of interest. This makes it easier to obtain a clear reconstructed image of the crop of interest when there are defects in the crop of interest.

[0236] If it is determined in step s80 that a defect exists in the crop of interest, a selection screen 560 may be displayed on the display unit 37 to select whether or not to switch the model 101 to the model 101 for crops with defects.

[0237] Figure 30 is a schematic diagram showing an example of the selection screen 560. The selection screen 560 includes notification information 561 that notifies the user that there is damage to the crop of interest. The selection screen 560 also includes instruction areas 565 and 566. Instruction area 565 is an area for instructing the user to switch the model 101 to the model 101 for crops with damage. Instruction area 566 is an area for instructing the user not to switch the model 101. The selection screen 560 may include the restored image 250 and the similar image 450, as in the example in Figure 30, or it may not include the restored image 250 and the similar image 450. Alternatively, the selection screen 560 may include only the restored image 250 among the restored image 250 and the similar image 450.

[0238] When instruction area 565 is selected, the processing system 1 executes steps s42, s51, s52, s53, s43, and s44 in Figures 19 and 28, similar to when instruction area 505 in Figures 26 and 29 is selected. As a result, a restored image 250 is obtained, which is reconstructed based on the feature quantities of the target image acquired by the scratched image model 101. Subsequently, the display unit 37 displays a display screen 330 including the newly acquired restored image 250 as shown in Figure 10 or Figure 14. On the other hand, when instruction area 566 is selected, the display on the display unit 37 becomes as shown in Figure 10 or Figure 14.

[0239] In the example shown in Figure 28, if, after step s40, step s41 is not executed and input area 360 or input area 365 in the display screen 331 is executed, step s15 is executed, and thereafter the processing system 1 operates similarly. Also, if, after step s81, step s41 is not executed and input area 360 or input area 365 in the display screen 335 is executed, step s15 is executed, and thereafter the processing system 1 operates similarly.

[0240] In the crop unit determination process shown in the example of Figure 15, when the server 3 acquires the image of interest from the terminal device 4, the control unit 30 may store the image of interest and the feature quantity of interest in the storage unit 31 as a determination image and its determination feature quantity, respectively. In this case, the control unit 30 includes the feature quantity of interest in the group corresponding to the harvest time indicated by the harvest time determination result of the crop of interest, out of the L groups. The control unit 30 then stores the image of interest as a determination image, the feature quantity of interest as a determination feature quantity, and group information identifying the group to which the feature quantity of interest belongs in the storage unit 31, associating them with each other. When the image of interest and the feature quantity of interest are used as a determination image and its determination feature quantity in the crop unit determination process, the display unit 37 may display the image of interest as a similar image of interest 450, as shown in the example of Figure 14.

[0241] In the crop unit determination process shown in the example of Figure 22, if the control unit 30 of the server 3 modifies the restored image of interest, it may store the modified restored image of interest and the feature quantity of interest in the storage unit 31 as the determination image and its determination feature quantity, respectively. In other words, the control unit 30 may store the modified restored image and the feature quantity of the original target image of the restored image in the storage unit 31 as the determination image and its determination feature quantity, respectively. In this case, the server 3 includes the feature quantity of interest in the group corresponding to the harvest time indicated by the harvest time determination result of the crop of interest, out of the L groups. The control unit 30 then stores the modified restored image of interest as the determination image, the feature quantity of interest as the determination feature quantity, and group information identifying the group to which the feature quantity of interest belongs in the storage unit 31, associating them with each other.

[0242] When the corrected restored image (also called the corrected restored image) and the feature quantities of the original target image of the restored image are used as the judgment image and its judgment feature quantities, respectively, the display unit 37 may display the corrected restored image as the attention similar image 450 in steps s14, s40, s45, s60, s65, s81, etc. In this case, the display unit 37 may also display notification information 600 indicating that the attention similar image 450 is a restored image together with the attention similar image 450. When the notification information 600 is displayed, the control unit 30 stores the judgment image, which is the corrected restored image, in the storage unit 31 in association with information indicating that the judgment image is a restored image. This allows the control unit 30 to determine whether each judgment image in the storage unit 31 is a restored image (more specifically, a corrected restored image).

[0243] Figure 31 is a schematic diagram showing an example of how notification information 600 is displayed on the display screen 330 shown in step s14. In the example in Figure 31, notification information 600 is shown next to the text "Similar Image" which is shown below the featured similar image 450, which is the corrected and restored image.

[0244] In step s14, etc., when the server 3 displays the modified restored image, which is the similar image of interest 450, and the notification information 600, it may display a non-restored similar image from among the multiple judgment images stored in the storage unit 31, in response to instructions from the server user. A non-restored similar image is a judgment image that has judgment features similar to the features of interest, but is not a restored image obtained by the restoration unit 130. Non-restored similar images may include judgment images originally stored in the storage unit 31. Also, when the target image is stored in the storage unit 31 as a judgment image as described above, the judgment image that is the target image may be considered a non-restored similar image. Multiple judgment images included in the storage unit 31 include multiple non-restored similar images.

[0245] Figure 32 is a schematic diagram showing an example of the display screen 330. The display screen 330 shown in Figure 32 is the same as the display screen 330 shown in Figure 31, but with the addition of an instruction area 610. The instruction area 610 is an area for instructing the display of a non-restored similar image.

[0246] When the instruction area 610 is selected, the control unit 30 identifies a determination feature in the specific feature space that is closest to the determination feature of the target similar image 450, among the determination feature quantities of multiple unrestored similar images in the storage unit 31, and whose Euclidean distance from the target feature quantity is less than or equal to a first predetermined value. The control unit 30 then displays the unrestored similar image 660 having the identified determination feature quantity on the display unit 37.

[0247] Figure 33 is a schematic diagram showing an example of the display of the unrestored similar image 660. When the instruction area 610 is selected, the display unit 37 displays a display screen 650 that includes, for example, the unrestored similar image 660, the focus restoration image 250, the input area 360, and the input area 365. When either the input area 360 or 365 is selected, the processing system 1 operates in the same manner as described above.

[0248] For example, if the focus-on similar image 450 is unclear, the server user selects the instruction area 610. This displays a non-restored similar image 660, which is not a restored image but has similar features to the focus-on similar image of the target image, on the display unit 37. This allows the server user to indirectly confirm the state of the target crop from a clear non-restored similar image 660 that is similar to the target image, even if the focus-on restored image 250 is unclear.

[0249] Furthermore, the display screen 330 shown in Figure 32 may include instruction areas 501 to 505 for instructing a change in the model 101 being used, or it may include an instruction area 510 for instructing a modification of the restored image of interest.

[0250] Furthermore, the display screen 650 shown in Figure 33 may include instruction areas 501 to 505 for instructing a change in the model used 101, or an instruction area 510 for instructing a modification of the image of interest to be restored. In this case, the server user can more easily determine whether the image of interest to be restored 250 is unclear by, for example, comparing the image of interest to be restored 250 with the non-restored similar image 660. If the server user determines that the image of interest to be restored 250 is unclear, they can instruct the processing system 1 to change the model used 101 or to modify the image of interest to be restored 25.

[0251] Server 3 may, in the crop unit determination process, predict future features of the crop of interest based on the crop of interest's features and past features of the crop of interest. For example, as shown in Figure 34, the control unit 30 places the features 302 of the crop of interest from X days ago (where X is an integer of 1 or more) in the specific feature space 300. The features 302 of the crop of interest from X days ago are features of the crop of interest obtained in a crop unit determination process executed X days before the crop unit determination process. The features 302 of the crop of interest from X days ago can be said to be features of the target image in which the crop of interest was photographed X days ago.

[0252] Next, the control unit 30 determines a first vector V1 in the specific feature space 300, which points from the feature 302 from X days ago to the feature of interest 301. The first vector V1 can also be said to be a vector that represents the time change of the feature of the crop of interest.

[0253] Next, the control unit 30 places a second vector V2 in the specific feature space 300, which has the same magnitude and direction as the first vector V1 and starts from the feature of interest 301. Next, the control unit 30 uses the feature located at the endpoint of the second vector V2 in the specific feature space 300 as the predicted value of the feature X days later. The feature X days later can also be said to be the predicted value of the feature in the target image showing the crop of interest X days later. X may be 1, 2, or any other value.

[0254] Next, the control unit 30 inputs the predicted feature values ​​for X days later into the decoding model 102 of the reconstruction unit 130. The image output by the decoding model 102 based on the predicted feature values ​​for X days later is a predicted image of the target image showing the crop of interest after X days. The image output by the decoding model 102 based on the predicted feature values ​​for X days later can be said to be a predicted image of the reconstructed image showing the crop of interest after X days later, or it can be said to be a predicted image of the reconstructed image in which the target image showing the crop after X days has been reconstructed.

[0255] The display unit 37 displays the predicted image 710 (also called the predicted image for X days later 710) of the target image showing the crop of interest X days later, obtained by the decoding model 102, together with the restored image 250 of the crop of interest today. For example, as shown in Figure 35, the display unit 37 may display the predicted image for X days later 710 on the display screen 330 that includes the restored image of interest 250. When the display unit 37 displays the predicted image for X days later 710, it may also display the string "Predicted image for X days later" around the predicted image for X days later 710 (for example, below it), as in the example in Figure 35.

[0256] The predicted image 710 for X days later may be shown on display screen 340, display screen 331, display screen 332, or display screen 345. Alternatively, the predicted image 710 for X days later may be shown on display screen 333, display screen 335, selection screen 560, or display screen 650.

[0257] In this way, by displaying the predicted image 710 for X days later, the server user can predict the state of the crop of interest X days later from the predicted image 710.

[0258] Server 3 may retrain the generative model 100 using the reconstructed images acquired by the reconstruction unit 130 as training images. Alternatively, Server 3 may retrain the generative model 100 using the target images acquired from the terminal device 4 as training images. The encoded model 101 of the retrained generative model 100 is transferred to the terminal device 4.

[0259] The determination unit 131 of server 3 may use a machine learning model to determine the harvest time of the crop of interest. In this case, the machine learning model provided by the determination unit 131 is input with the features of interest, and the machine learning model outputs a determination result for the harvest time of the crop of interest.

[0260] If the terminal device 4 is equipped with multiple types of encoding models 101 capable of acquiring feature quantities of a target image, in step s2, the first crop-related information including the feature quantities of interest acquired by each of the multiple types of encoding models 101 for the image of interest may be transmitted to the server 3. In this case, the server 3, for example, acquires a reconstructed image based on each of the multiple feature quantities of interest of the image of interest included in the first crop-related information. The server 3 may then display the multiple reconstructed images of interest acquired based on the multiple feature quantities of interest of the image of interest on screens 330, 332, 340, 380, and 650, etc. In this case, the server user can easily identify the state of the crop because they can check the reconstructed images acquired from multiple perspectives of the same crop.

[0261] Figure 36 is a schematic diagram showing an example of a display screen 330 that includes multiple reconstructed images based on multiple feature quantities of interest for the image of interest. The display screen 330 shown in Figure 36 shows a reconstructed image 250c based on feature quantities of interest acquired by the general-purpose model 101, a reconstructed image 250d based on feature quantities of interest acquired by the light-color model 101, and a reconstructed image 250e based on feature quantities of interest acquired by the dark-color model 101. The display screen 330 also shows a reconstructed image 250f based on feature quantities of interest acquired by the large-size model 101, a reconstructed image 250g based on feature quantities of interest acquired by the small-size model 101, and a reconstructed image 250h based on feature quantities of interest acquired by the scratched image model 101.

[0262] When server 3 obtains multiple feature quantities for the image of interest from terminal device 4, it may make a provisional determination of the harvest time of the crop of interest based on each of the obtained feature quantities. In this case, server 3 may ultimately adopt the average value of the multiple harvest times obtained by the provisional determination as the harvest time of the crop of interest.

[0263] Furthermore, if multiple images of interest 250 are displayed on the display screen 332 (see Figure 23), which includes an instruction area 510 for instructing the modification of one of the image of interest 250, as shown in Figure 36, then when the instruction area 510 is selected, each of the multiple images of interest 250 may be modified.

[0264] Server 3 may determine whether a crop of interest should be discarded based on its features, similar to how it determines the harvest time of a crop of interest based on its features. For example, a discard group is prepared that includes multiple features from images of crops that should be discarded. Server 3 places the features of interest and the multiple features belonging to the discard group in the feature space. Server 3 calculates the Mahalanobis distance, which represents the distance between the distribution of the multiple features belonging to the discard group and the features of interest in the feature space. Server 3 determines that the features of interest belong to the discard group if the calculated Mahalanobis distance is less than or equal to a threshold. If Server 3 determines that the features of interest belong to the discard group, it determines that the crop of interest should be discarded and notifies the terminal device 4 of the result. If Server 3 does not determine that the crop of interest should be discarded, it may determine the harvest time of the crop of interest based on its features in the same manner as described above.

[0265] Furthermore, Server 3 may determine whether a crop of interest should be thinned based on a particular feature, similar to how the harvest time of a crop of interest is determined based on that feature. When the crop is grapes, thinning is also called grape thinning. For example, a thinning group is prepared that includes multiple feature quantities from images of crops that should be thinned. Server 3 places the particular feature and the multiple feature quantities belonging to the thinning group in the feature space. Server 3 calculates the Mahalanobis distance, which represents the distance between the distribution of the multiple feature quantities belonging to the thinning group and the particular feature in the feature space. Server 3 determines that the particular feature belongs to the thinning group when the calculated Mahalanobis distance is less than or equal to a threshold. When Server 3 determines that the particular feature belongs to the thinning group, it determines that the crop of interest should be thinned and notifies the terminal device 4 of the result.

[0266] <Other Examples of Processing System Operation> In the above example, processing system 1 determines the state of the object, such as the harvest time of the crop. However, the state of the object that processing system 1 determines is not limited to this. For example, processing system 1 may determine that there is an abnormality in the object. The first and second examples of processing system 1's operation in determining an abnormality in an object are described below.

[0267] <First example of abnormality detection> The processing system 1 (also called the second processing system 1) in this example can detect abnormalities in structures such as bridges, highways, railway lines, tunnels, or buildings. In this example, as shown in Figure 37, a drone 12 having a terminal device 4, a camera 5, and a position information acquisition unit 6 can communicate with a server 3. The drone 12 comprises an airframe 11 and a terminal device 4, a camera 5, and a position information acquisition unit 6 mounted on the airframe 11. The terminal device 4 can control the flight of the airframe 11. A drone is also called, for example, an unmanned aerial vehicle. A drone can also be called a flying object.

[0268] The second processing system 1 determines abnormalities in a structure (also called the target structure), and multiple determination points are set for which abnormalities are to be determined. These determination points can also be called the target determination points. The multiple determination points are located in different places from each other. The second processing system 1 determines abnormalities in each of the multiple determination points. Each determination point is composed of a part of the target structure. A determination point can also be called, for example, an inspection point, an inspection determination point, a confirmation point, or a confirmation determination point.

[0269] The drone 12 flies to the vicinity of each determination point on the target structure and photographs the determination point with the camera 5. The drone 12 flies to the vicinity of the determination point by controlling the flight of the aircraft 11 by the terminal device 4. The terminal device 4 may know the location of each determination point in advance, or it may be notified of the location of each determination point from the server 3. The terminal device 4 controls the flight of the aircraft 11 based on the location information indicating the absolute position of the drone 12, which is acquired by the location information acquisition unit 6, and flies the drone 12 to the vicinity of the determination point.

[0270] The second processing system 1 determines an abnormality in the area to be judged based on the captured image of the area taken by the camera 5. The determination of an abnormality in the area to be judged can be seen as a determination of damage to the area to be judged, or as a determination of deterioration of the area to be judged.

[0271] In this example, each determination point becomes an object, and a captured image showing one determination point becomes a target image showing the object. The image processing unit 140 of the terminal device 4 acquires multiple target images from the camera 20, each showing multiple determination points of the target structure.

[0272] Similar to the example above, terminal device 4 acquires the feature quantities of each target image it has obtained. Terminal device 4 then transmits the feature quantities of each target image to server 3. Server 3 obtains a reconstructed image by restoring the target image based on the feature quantities of the target image. Server 3 then displays the obtained reconstructed image. Server 3 also determines whether there is an anomaly in the area to be judged in the target image based on the feature quantities of the target image. Server 3 displays, for example, the result of the anomaly determination in the area to be judged (also called the first anomaly determination result) together with the reconstructed image in which the area to be judged is shown.

[0273] The server user may be, for example, an administrator managing the target structure. The server user can make an appropriate judgment regarding the area of ​​determination by referring to, for example, the restored image displayed on server 3 and the first abnormality determination result displayed on server 3. For example, the server user may determine that the area of ​​determination is normal, that progress checks of the area of ​​determination are necessary, that on-site inspections of the area of ​​determination are necessary, or that repairs of the area of ​​determination are necessary.

[0274] Note that the terminal device 4, camera 5, and location information acquisition unit 6 do not necessarily have to be mounted on the drone 12. In this case, for example, a person carrying the terminal device 4, camera 5, and location information acquisition unit 6 may move to the vicinity of the judgment point, allowing the camera 5 to photograph the judgment point.

[0275] <Details of the operation example of the second processing system> The second processing system 1 performs a structure determination process to determine abnormalities at each determination point of the target structure based on multiple target images obtained by the camera 5. The structure determination process is performed repeatedly, for example, at intervals of time. For example, the structure determination process may be performed once a month, once every few months, once a year, or once every few years.

[0276] The second processing system 1, in the structure determination process, performs a unit determination process for each of the multiple determination points of the target structure, based on the target image in which the determination point is captured, to determine if there is an abnormality in that determination point. In the structure determination process, the unit determination process is executed for each determination point of the target structure. At least a portion of the multiple unit determination processes performed for each of the multiple determination points of the target structure in the structure determination process may be executed in parallel or sequentially.

[0277] Figure 38 is a flowchart showing an example of a unit determination process. In the structure determination process, the unit determination process shown in Figure 38 is executed for each determination point of the target structure. Hereafter, the determination point of interest will be referred to as the "target determination point." The unit determination process in which an abnormality is determined at the target determination point will be referred to as the "target unit determination process." In this example, the "target image of interest" refers to the target image in which the target determination point is captured.

[0278] When the unit of interest determination process begins, the drone 12 flies and remains stationary near the area of ​​interest. While the drone 12 is stationary, the camera 20 takes a photograph of the area of ​​interest. In the unit of interest determination process, first, in step s100, the image processing unit 140 of the terminal device 4 acquires the image of the object of interest from the camera 5. The image of the object of interest acquired by the image processing unit 140 is stored in the storage unit 41.

[0279] Next, in step s101, the feature acquisition unit 141 acquires the features of the image of interest, i.e., the feature of interest. Then, in step s102, the terminal device 4 transmits the feature of interest acquired in step s100 to the server 3. The feature acquisition unit 141 has, for example, an encoding model 101 of the generation model 100, as described above. The restoration unit 130 of the server 3 also has, as described above, a decoding model 102 of the generation model 100.

[0280] In the unit of interest determination process, in step s111, server 3 receives and acquires the feature of interest from terminal device 4. The feature of interest acquired by server 3 is stored in storage unit 31.

[0281] Next, in step s112, the restoration unit 130 of the server 3 acquires a restored image, i.e., a restored image of interest, which is the image of interest, based on the features of interest acquired in step s111.

[0282] Next, in step s113, the determination unit 131 determines an abnormality in the area of ​​interest captured in the image of interest based on the feature of interest acquired in step s111. For example, the determination unit 131 determines an abnormality in the area of ​​interest based on the feature of interest acquired in step s111 and the feature of interest acquired in the past.

[0283] Here, the feature quantity obtained in step s111 of the unit of interest determination process in the structure determination process that was first executed when the operation of the second processing system 1 began is called the initial feature quantity of interest. Also, the feature quantity obtained in step s111 of the unit of interest determination process currently being executed in the second processing system 1 is called the current feature quantity of interest.

[0284] The determination unit 131 determines an abnormality in the area of ​​interest based, for example, on the current area of ​​interest and the initial area of ​​interest. For example, the determination unit 131 places the current area of ​​interest and the initial area of ​​interest in the feature space. Then, the determination unit 131 calculates the Euclidean distance between the current area of ​​interest and the initial area of ​​interest in the feature space. The determination unit 131 determines that there is an abnormality in the area of ​​interest if the calculated Euclidean distance is greater than or equal to a threshold. On the other hand, the determination unit 131 determines that the area of ​​interest is normal if the calculated Euclidean distance is less than the threshold. When the Euclidean distance between the current area of ​​interest and the initial area of ​​interest is greater than or equal to a threshold, it means that the state of the area of ​​interest has changed significantly from the state when the operation of the second processing system 1 started. Therefore, if the Euclidean distance between the current area of ​​interest and the initial area of ​​interest is greater than or equal to a threshold, it can be seen that the area of ​​interest, which was normal when the operation of the second processing system 1 started, has become abnormal due to aging or other reasons.

[0285] Furthermore, when the determination unit 131 determines that there is an abnormality in the area of ​​interest, it may use the features of an image showing a normal area of ​​interest (also called a reference feature) instead of the initial area of ​​interest feature. In this case, if the Euclidean distance between the current area of ​​interest feature and the reference feature in the feature space where the current area of ​​interest feature and the reference feature are located is greater than or equal to a threshold, it may be determined that there is an abnormality in the area of ​​interest. As an image showing a normal area of ​​interest, for example, an image showing the area of ​​interest when the target structure was installed may be used.

[0286] If an abnormality is detected in the area of ​​interest in step s113, step s114 is executed. In step s114, the control unit 30 displays the restored image of interest acquired by the restoration unit 130 and the result of the determination of the abnormality of the area of ​​interest by the determination unit 131 on the display unit 37.

[0287] Figures 39 and 40 are schematic diagrams showing examples of the display of the display unit 37 in step s114. Figure 39 shows an example of the display in step s114 when an abnormality is determined in step s113 at the area of ​​interest. Figure 40 shows an example of the display in step s114 when the area of ​​interest is determined to be normal in step s113.

[0288] As shown in Figures 39 and 40, in step s114, the display unit 37 displays a display screen 750 that includes, for example, a restored image 760 showing the area of ​​interest and a first abnormality determination result 770 of the area of ​​interest. The first abnormality determination result 770 shown in Figure 39 indicates that there is an abnormality in the area of ​​interest. The first abnormality determination result 770 shown in Figure 40 indicates that the area of ​​interest is normal. By displaying the first abnormality determination result 770 indicating an abnormality, the server 3 can notify the server user of an alert if it determines that there is an abnormality in the area of ​​interest.

[0289] In this way, by displaying the focus restoration image 760 and the first anomaly detection result 770 on server 3, the server user can refer to the focus restoration image 760 and the first anomaly detection result 770 and make an appropriate judgment regarding the area of ​​focus. For example, if the first anomaly detection result 770 on server 3 indicates an anomaly, and the server user checks the state of the area of ​​focus shown in the focus restoration image 760 and determines that there is an anomaly in the area of ​​focus, the server user may decide that repair of the area of ​​focus is necessary. Alternatively, even if the first anomaly detection result 770 on server 3 indicates a normal state, if the server user checks the state of the area of ​​focus shown in the focus restoration image 760 and determines that there is something wrong with the area of ​​focus, the server user may decide that on-site inspection of the area of ​​focus is necessary.

[0290] <Regarding the generation model in the second processing system> The second processing system 1 has a generation model 100 for each of the multiple determination points of the target structure. The second processing system 1 has multiple generation models 100 corresponding to each of the multiple determination points of the target structure. The feature acquisition unit 141 of the terminal device 4 has an encoding model 101 of the multiple generation models 100 corresponding to each of the multiple determination points. The restoration unit 130 of the server 3 has a decoding model 102 of the multiple generation models 100 corresponding to each of the multiple determination points.

[0291] When the feature acquisition unit 141 acquires a feature of interest, it uses the encoding model 101 of the generation model 100 corresponding to the area of ​​interest determination to acquire the feature of interest. The encoding model 101 of the generation model 100 corresponding to the area of ​​interest determination can be said to be the encoding model 101 corresponding to the area of ​​interest determination. When the reconstruction unit 130 acquires a reconstructed image of interest based on the feature of interest, it uses the decoding model 102 of the generation model 100 corresponding to the area of ​​interest determination to acquire the reconstructed image of interest. The decoding model 102 of the generation model 100 corresponding to the area of ​​interest determination can be said to be the decoding model 102 corresponding to the area of ​​interest determination.

[0292] The multiple training images (also called multiple focus training images) used to train the generative model 100 according to the focus area may include, for example, training images showing structures made of the same material as the focus area. For example, if the material of the focus area is concrete, the multiple focus training images may include training images showing structures made of concrete. Also, if the material of the focus area is metal, the multiple focus training images may include training images showing structures made of metal.

[0293] Furthermore, multiple training images for focus may include training images depicting structures with shapes similar to the shape of the area of ​​focus. Additionally, multiple training images for focus may include training images depicting structures made of the same material as the area of ​​focus and having shapes similar to the area of ​​focus. For example, consider a case where the target structure is a bridge and there are other bridges of the same type. In this case, a part of the same type of bridge located at the same position as the area of ​​focus would be an example of a structure made of the same material as the area of ​​focus and having shapes similar to the area of ​​focus.

[0294] Furthermore, the multiple training images of interest may include training images showing the areas of interest that were captured before the operation of the second processing system 1 began.

[0295] If the generative model 100 corresponding to the area of ​​interest is trained based on at least one of the training images showing the structure of the same material as the area of ​​interest and the training image showing the area of ​​interest, then the generative model 100 corresponding to the area of ​​interest can be said to be a generative model 100 corresponding to the material of the area of ​​interest. The generative model 100 corresponding to the material of the area of ​​interest comprises an encoding model 101 corresponding to the material of the area of ​​interest and a decoding model 102 corresponding to the material of the area of ​​interest.

[0296] In the second processing system 1, by using a generation model 100 that corresponds to the material of the area of ​​interest, it becomes easier to obtain a reconstructed image in which the area of ​​interest is clearly visible.

[0297] Server 3 may generate a basic generative model 100, and then perform additional training such as fine-tuning on the basic generative model 100 based on multiple training images prepared individually for each judgment location, thereby generating multiple generative models 100 corresponding to each judgment location.

[0298] For example, suppose the material of the area to be judged is either concrete or metal. In this case, Server 3 learns a generative model 100 based on multiple training images showing structures made of concrete, and designates the learned generative model 100 as the first basic generative model 100. Server 3 also learns a generative model 100 based on multiple training images showing structures made of metal, and designates the learned generative model 100 as the second basic generative model 100. If the material of the area to be judged is concrete, Server 3 further trains the first basic generative model 100 based on multiple training images showing the area to be judged, and designates the further trained first basic generative model 100 as the trained generative model 100 corresponding to the area to be judged. On the other hand, if the material of the area to be judged is metal, Server 3 further trains the second basic generative model 100 based on multiple training images showing the area to be judged, and designates the further trained second basic generative model 100 as the trained generative model 100 corresponding to the area to be judged. In this way, the server 3 may generate multiple trained generative models 100, each corresponding to a multiple decision point.

[0299] Furthermore, multiple training images of interest may include training images that depict structures in the same state as the area of ​​interest that can be judged. For example, multiple training images of interest may include training images that depict structures in the same degraded state as the area of ​​interest that can be judged. In other words, multiple training images of interest may include training images that depict structures in which the same degradation as the degradation that can occur in the area of ​​interest has occurred.

[0300] For example, suppose a crack is likely to occur at the first of several judgment points, and the possible deterioration state of the first judgment point is a state in which it is cracked. In this case, the multiple training images used to train the generative model 100 corresponding to the first judgment point may include training images that show a structure in a cracked state.

[0301] Furthermore, cracks and rust are likely to occur at the first judgment point, and the possible deterioration states of the first judgment point are a state with cracks and a state with rust. In this case, the multiple training images used to train the generative model 100 corresponding to the first judgment point may include training images showing the structure in a cracked state, training images showing the structure in a rusted state, or training images showing the structure in a state with both cracks and rust.

[0302] Furthermore, bolt loosening is likely to occur at the second of the multiple judgment points, and a state of bolt loosening can be considered as a possible deterioration state for the second judgment point. In this case, the multiple training images used to train the generative model 100 corresponding to the second judgment point may include training images showing a structure with loose bolts.

[0303] Furthermore, bolt loosening and rust are likely to occur at the second judgment point, and the possible deterioration states of the second judgment point are considered to be a state with loose bolts and a state with rust. In this case, the multiple training images used to train the generative model 100 corresponding to the second judgment point may include training images showing the structure in a state with loose bolts, training images showing the structure in a rusted state, or training images showing the structure in a state with both loose bolts and rust.

[0304] If the generative model 100 corresponding to the area of ​​interest is trained based only on training images that depict a structure in the same state as a certain state that the area of ​​interest may be in, then the generative model 100 corresponding to the area of ​​interest can be said to be the generative model 100 corresponding to that certain state. Furthermore, if the generative model 100 corresponding to the area of ​​interest is trained based only on training images that depict a structure in the same deteriorated state as a certain deteriorated state that the area of ​​interest may be in (for example, a cracked state), then the generative model 100 corresponding to the area of ​​interest can be said to be the generative model 100 corresponding to that deteriorated state (for example, a cracked state).

[0305] In the second processing system 1, a generation model 100 corresponding to the possible states of the area of ​​interest is used, making it easier to obtain a restored image of the area of ​​interest in which that state is clearly visible when the area of ​​interest is in that state. As a result, server users can more easily identify the state of the area of ​​interest from the restored image of the area of ​​interest displayed on server 3.

[0306] Furthermore, in the second processing system 1, a generation model 100 corresponding to the possible degradation state of the area of ​​interest is used. This makes it easier to obtain a restored image of the area of ​​interest in which the area of ​​interest is clearly visible when it reaches that degradation state. As a result, server users can more easily identify the degradation state of the area of ​​interest from the restored image of the area of ​​interest displayed on server 3.

[0307] Furthermore, if the generative model 100 corresponding to the area of ​​interest is trained based only on training images that depict a structure made of the same material as the area of ​​interest and in the same state as the area of ​​interest that it may be in, then the generative model 100 corresponding to the area of ​​interest can be said to be a generative model 100 corresponding to the material of the area of ​​interest and the state that the area of ​​interest may be in.

[0308] The second processing system 1 may have multiple generative models 100 for a single determination point. In other words, the second processing system 1 may have multiple generative models 100 corresponding to the determination point of interest. For example, if there are multiple states that the determination point of interest can be in, the second processing system 1 may have multiple generative models 100 corresponding to each of the multiple states that the determination point of interest can be in. For example, if there are multiple degradation states that the determination point of interest can be in, the second processing system 1 may have multiple generative models 100 corresponding to each of the multiple degradation states that the determination point of interest can be in.

[0309] For example, if cracks and rust are likely to occur at the first determination point, consider the possible deterioration states of the first determination point as a state with cracks and a state with rust. In this case, the second processing system 1 may have a generation model 100 (also called the crack model 100) corresponding to the cracked state that the first determination point may be in, and a generation model 100 (also called the rust model 100) corresponding to the rusted state that the first determination point may be in.

[0310] Furthermore, bolt loosening and rust are likely to occur at the second judgment point, and the possible deterioration states of the second judgment point are a state in which the bolts are loose and a state in which rust is present. In this case, the second processing system 1 may have a generation model 100 corresponding to the state in which the bolts are loose (also called the bolt loosening model 100) and a generation model 100 corresponding to the state in which the second judgment point is rusted (i.e., the rust model 100).

[0311] The crack model 100 is trained based on training images that show a structure where only cracks are present as deterioration, and there is no rust or loose bolts. The rust model 100 is trained based on training images that show a structure where only rust is present as deterioration, and there is no crack or loose bolts. The bolt loosening model 101 is trained based on training images that show a structure where only loose bolts are present as deterioration, and there is no crack or rust.

[0312] If the second processing system 1 has multiple generation models 100 corresponding to the areas of interest, in step s101 of the unit determination process, the feature acquisition unit 141 acquires the features of interest of the target image using, for example, each of the encoding models 101 of the multiple generation models 100 corresponding to the areas of interest (in other words, multiple encoding models 101 corresponding to the areas of interest). Then, in step s102, the terminal device 4 transmits the multiple features of interest acquired in step s101 to the server 3.

[0313] In step s111, the server 3 receives multiple attention features from the terminal device 4. Then, in step s112, the reconstruction unit 130 obtains a attention-reconstructed image based on each of the multiple attention features obtained in step s111. When the reconstruction unit 130 obtains a attention-reconstructed image based on attention features obtained by the encoding model 101 of a certain generative model 100 corresponding to the attention determination location, it uses the decoding model 102 of that generative model 100.

[0314] After step s112, in step s113, the determination unit 131 makes a provisional determination of whether the area of ​​interest is abnormal based on each of the multiple features of interest acquired in step s111. If the determination unit 131 determines that the area of ​​interest is abnormal in the provisional determination for at least one of the multiple features of interest, it makes a final determination that the area of ​​interest is abnormal. On the other hand, if the determination unit 131 determines that the area of ​​interest is normal in the provisional determination for all of the multiple features of interest, it makes a final determination that the area of ​​interest is normal. Then, in step s114, the display unit 37 displays a display screen 750 including the first abnormality determination result 770 as the final abnormality determination result and the multiple restored images of interest acquired in step s112.

[0315] Figure 41 is a schematic diagram showing an example of a display screen 750 that includes multiple focus restoration images displayed in step s114. Figure 41 shows an example of a display screen 750 when the second processing system 1 has multiple generation models 100 corresponding to the focus determination area, including a crack model 100 and a rust model 100. The display screen 750 includes a focus restoration image 760a acquired by the decoding model 102 of the crack model 100 and a focus restoration image 760b acquired by the decoding model 102 of the rust model 100. For example, if a crack occurs at the focus determination area, the crack is more likely to be clearly visible in the focus restoration image 760a acquired by the crack model 100. On the other hand, if rust occurs at the focus determination area, the rust is more likely to be clearly visible in the focus restoration image 760b acquired by the rust model 100.

[0316] For example, if the crack model 100 corresponding to the area of ​​interest is trained based only on training images that show a structure made of the same material as the area of ​​interest, and where only cracks are present as a form of deterioration, then the crack model 100 corresponding to the area of ​​interest can be said to be a generation model 100 that corresponds to the cracked state that the area of ​​interest can be in, and to the material of the area of ​​interest.

[0317] In this way, when the reconstructed images of the area of ​​interest, acquired by each of the multiple generation models 100 corresponding to the area of ​​interest, are displayed on the server 3, the server user can view the reconstructed images acquired from multiple perspectives of the same area of ​​interest, making it easier to identify the state of the area of ​​interest. Therefore, the server user can more easily identify abnormalities in the area of ​​interest from the reconstructed images.

[0318] The second processing system 1 may use a generation model 100 in the unit of focus determination process that corresponds to the surrounding environment of the area of ​​focus determination in the target image. In this case, the drone 12 has, for example, an environmental sensor 8 that detects the surrounding environment of the drone 12, as shown in Figure 42.

[0319] The second processing system 1 may include, as generation models 100 corresponding to the area of ​​interest, for example, a sunny generation model 100 corresponding to when the surrounding environment of the area of ​​interest is sunny, a cloudy generation model 100 corresponding to when the surrounding environment of the area of ​​interest is cloudy, and a rainy generation model 100 corresponding to when the surrounding environment of the area of ​​interest is rainy. In this case, the environment sensor 8 may include a rain sensor that detects rain around the drone 12 and an illuminance sensor that detects the illuminance around the drone 12. The rain sensor may be a sensor that detects rain using visible light or infrared light, or a sensor that detects rain by measuring resistance.

[0320] The sunny-day generative model 100 is trained based only on training images of structures taken on sunny days. The multiple training images used to train the sunny-day generative model 100 may include, for example, training images taken on sunny days that show structures made of the same material as the area of ​​interest, or training images taken on sunny days that show structures with a shape similar to the area of ​​interest. In addition, the multiple training images used to train the sunny-day generative model 100 may include training images taken on sunny days that show the area of ​​interest, or training images taken on sunny days that show structures in the same state as the area of ​​interest that it may be in.

[0321] The cloudy-type generation model 100 is trained based only on training images of structures taken on cloudy days. The multiple training images used to train the cloudy-type generation model 100 may include, for example, training images taken on cloudy days that show structures made of the same material as the area of ​​interest, or training images taken on cloudy days that show structures with a shape similar to the area of ​​interest. In addition, the multiple training images used to train the cloudy-type generation model 100 may include training images taken on cloudy days that show the area of ​​interest, or training images taken on cloudy days that show structures in the same state as the area of ​​interest that may be.

[0322] The rain-resistant generative model 100 is trained based only on training images of structures taken on rainy days. The multiple training images used to train the rain-resistant generative model 100 may include, for example, training images taken on rainy days that show structures made of the same material as the area of ​​interest, or training images taken on rainy days that show structures with a shape similar to the area of ​​interest. In addition, the multiple training images used to train the rain-resistant generative model 100 may include training images taken on rainy days that show the area of ​​interest, or training images taken on rainy days that show structures in the same state as the area of ​​interest that may be.

[0323] In step s101 of the unit of interest determination process, the feature acquisition unit 141 acquires the features of the image of interest using, for example, the encoding model 101 of the generation model 100 that corresponds to the surrounding environment of the area of ​​interest determination. For example, if the rain sensor does not detect rain and the illuminance detected by the illuminance sensor is above a threshold (i.e., the surrounding environment of the area of ​​interest determination is sunny), the feature acquisition unit 141 acquires the features of the image of interest using the encoding model 101 of the sunny generation model 100. In this example, the structure determination process is performed during the daytime. Also, if the rain sensor does not detect rain and the illuminance detected by the illuminance sensor is below a threshold (i.e., the surrounding environment of the area of ​​interest determination is cloudy), the feature acquisition unit 141 acquires the features of the image of interest using the encoding model 101 of the cloudy generation model 100. Then, when the rain sensor detects rain (i.e., when the surrounding environment of the area to be judged is rainy), the feature acquisition unit 141 acquires the features of the target image using the encoding model 101 of the rain generation model 100. The features of interest acquired in step s101 are transmitted to the server 3 in step s102.

[0324] In server 3, in step s112, the reconstruction unit 130 acquires a focus image based on the focus feature received in step s111. At this time, the reconstruction unit 130 acquires the focus image using a decoding model 102 corresponding to the encoding model 101 used to acquire the focus feature. For example, if the focus feature was acquired using the encoding model 101 of the sunny generation model 100, the reconstruction unit 130 acquires the focus image using the decoding model 102 of the sunny generation model 100. Also, if the focus feature was acquired using the encoding model 101 of the cloudy generation model 100, the reconstruction unit 130 acquires the focus image using the decoding model 102 of the cloudy generation model 100. And if the focus feature was acquired using the encoding model 101 of the rainy generation model 100, the reconstruction unit 130 acquires the focus image using the decoding model 102 of the rainy generation model 100. Once the desired restored image is obtained in step s112, steps s113 and s114 are executed.

[0325] Thus, in the second processing system 1, by using a generation model 100 that corresponds to the surrounding environment of the area of ​​interest in the target image, it becomes easier to obtain a reconstructed image in which the area of ​​interest is clearly visible. As a result, server users can more easily identify the state of the area of ​​interest from the reconstructed image displayed on server 3.

[0326] Furthermore, if there are multiple possible states for the area of ​​interest to be determined, the second processing system 1 may provide multiple generation models 100 corresponding to each of the multiple possible states for the area of ​​interest to be determined, separately for sunny, cloudy, and rainy conditions around the area of ​​interest to be determined.

[0327] For example, consider two possible states for the first determination point: a state with cracks and a state with rust. In this case, the second processing system 1 may have a generation model 100 corresponding to the state with cracks and sunny weather (also called the sunny / cracked model 100), a generation model 100 corresponding to the state with cracks and cloudy weather (also called the cloudy / cracked model 100), a generation model 100 corresponding to the state with cracks and rain (also called the rain / cracked model 100), a generation model 100 corresponding to the state with rust and sunny weather (also called the sunny / rusted model 100), a generation model 100 corresponding to the state with rust and cloudy weather (also called the cloudy / rusted model 100), and a generation model 100 corresponding to the state with rust and rain (also called the rain / rusted model 100). In this case, in step s101, the feature acquisition unit 141 acquires the feature of interest using the encoding model 101 of the sunny / cracked model 100 and the encoding model 101 of the sunny / rusty model 100 if the surrounding environment of the area of ​​interest is sunny. Furthermore, if the surrounding environment of the area of ​​interest is cloudy, the feature acquisition unit 141 acquires the feature of interest using the encoding model 101 of the cloudy / cracked model 100 and the encoding model 101 of the cloudy / rusty model 100. Then, if the surrounding environment of the area of ​​interest is rainy, the feature acquisition unit 141 acquires the feature of interest using the encoding model 101 of the rain / cracked model 100 and the encoding model 101 of the rain / rusty model 100.

[0328] The display screen 750 shown in step s114 of this example may include similar images of interest that have features similar to the feature of interest, similar to the display screen 330 shown in Figure 14 above. In this case, the server 3 may use multiple training images used to train the generative model 100 as multiple extraction target images from which similar images of interest can be extracted, and among the multiple extraction target images, the extraction target image that has features similar to the feature of interest can be used as the similar image of interest.

[0329] Furthermore, as shown in the examples in Figures 15-17 above, if no similar images of interest exist among multiple images to be extracted, the server 3 may acquire the image of interest from the terminal device 4 and display the acquired image of interest.

[0330] Furthermore, as shown in the example in Figure 18 above, after displaying the restored image of interest, the server 3 may, in response to instructions from the server user, acquire the target image of interest from the terminal device 4 and display the acquired target image of interest.

[0331] Furthermore, when server 3 acquires a target image from terminal device 4, it may include the acquired target image among multiple extraction target images.

[0332] Furthermore, as shown in the examples in Figures 21 to 26 above, the second processing system 1 may, in the unit of focus determination process, modify the restored image of focus in response to instructions from the server user and display the modified restored image of focus.

[0333] Furthermore, the server 3 may include the corrected restored image (i.e., the corrected restored image) among multiple images to be extracted. In this case, when the display unit 37 displays the corrected restored image as a similar image of interest, it may display notification information 600 indicating that the similar image of interest is a restored image, together with the similar image of interest, as shown in the examples in Figures 31 and 32.

[0334] Furthermore, when the server 3 displays the reconstructed image of interest (a similar image) and the notification information 600, it may display a non-reconstructed similar image from among the multiple images to be extracted, in response to instructions from the server user, as shown in the examples in Figures 32 and 33. In this example, a non-reconstructed similar image is an extracted image that has similar features to the features of interest, but is not a reconstructed image obtained by the reconstruction unit 130.

[0335] Furthermore, in the unit determination process, server 3 may predict the future features of a unit of interest based on the current features of the unit of interest and the past features of the unit of interest, similar to how it predicts the future features of a crop of interest.

[0336] For example, as shown in the example in Figure 34 above, the control unit 30 places the current feature vector of the area of ​​interest and the feature vector of the area of ​​interest Y months ago (where Y is an integer greater than or equal to 1) into the feature vector space. Next, the control unit 30 finds a third vector in the feature vector space that points from the feature vector of interest Y months ago to the current feature vector of interest. Next, the control unit 30 places a fourth vector in the feature vector space that starts from the current feature vector of interest and has the same magnitude and direction as the third vector. Next, the control unit 30 takes the feature vector located at the endpoint of the fourth vector in the feature vector space as the predicted value of the feature vector of the area of ​​interest Y months later.

[0337] The control unit 30 may input the predicted feature values ​​for Y months later to the decoding model 102 of the reconstruction unit 130. The image output by the decoding model 102 based on the predicted feature values ​​for Y months later will be a predicted image of the target image showing the area of ​​interest determined Y months later. The display unit 37 may display the predicted image of the target image showing the area of ​​interest determined Y months later, obtained by the decoding model 102, together with the restored image of the area of ​​interest determined today (i.e., the restored image of the area of ​​interest obtained based on the current area of ​​interest), as shown in the example in Figure 35 above.

[0338] As described above, in this example, camera 5 is mounted on drone 12. The position and orientation of drone 12 are susceptible to wind. Therefore, the position and orientation of drone 12 may be unstable when camera 5 is photographing the area of ​​interest. This may prevent camera 5 from photographing the area of ​​interest from the desired position and angle. In other words, the position and angle at which the area of ​​interest is captured in the target image may differ from the desired position and angle. Consequently, even in the reconstructed image of the target image, the position and angle at which the area of ​​interest is captured may differ from the desired position and angle. As a result, it may be difficult for the server user to make an appropriate judgment regarding the area of ​​interest by referring to the reconstructed image of the target displayed on server 3.

[0339] Furthermore, because the position and attitude of the drone 12 are unstable, variations in the position and angle at which the camera 5 photographs the area of ​​interest cause variations in the feature quantities (i.e., the feature quantities of interest) of the image of the target object in which the area of ​​interest is captured. As a result, the accuracy of the server 3's determination of anomalies in the area of ​​interest based on the feature quantities of interest may decrease.

[0340] Therefore, after the drone 12 has stopped near the area of ​​interest, camera 5 may take multiple consecutive photographs of the area of ​​interest and acquire multiple images of the area of ​​interest. The terminal device 4 may then acquire feature quantities of the area of ​​interest images that are similar to the reference image from among the multiple images of the area of ​​interest acquired by camera 5, and transmit the acquired feature quantities to server 3. As a result, server 3 can acquire a reconstructed image of the area of ​​interest or determine if there is an anomaly in the area of ​​interest based on the feature quantities of the area of ​​interest similar to the reference image. By using an image of the area of ​​interest taken at a desired position and angle as the reference image, the position and angle in which the area of ​​interest is captured in the reconstructed image of the area of ​​interest will be closer to the desired position and angle, making it easier for the server user to make an appropriate judgment regarding the area of ​​interest by referring to the reconstructed image of the area of ​​interest displayed on server 3. In addition, since feature quantities of the area of ​​interest images similar to a fixed reference image are acquired, the variability of the feature quantities of the area of ​​interest is reduced. Therefore, the accuracy of the judgment when server 3 determines an anomaly in the area of ​​interest based on the feature quantities of the area of ​​interest is improved. Figure 43 is a flowchart of an example of the area of ​​interest determination process in this case.

[0341] In this example, when the focus unit determination process starts, the camera 5 takes N consecutive photos (where N is an integer of 2 or more) of the focus area after the drone 12 has come to rest near the focus area, thereby acquiring N images of the focus area. In other words, the camera 5 acquires N images of the focus area. In the focus unit determination process in this example, first, in step s120, the image processing unit 140 acquires the N images of the focus area acquired by the camera 5 from the camera 5. Due to the instability of the position and attitude of the drone 12, at least one of the positions and angles of the focus area captured in the focus area may differ between at least two of the N images of the focus area.

[0342] Next, in step s121, the image processing unit 140 extracts the edges of each of the N target images acquired in step s120 and obtains an edge image representing those edges. The edge image is, for example, a binary image. In step s121, N edge images corresponding to each of the N target images are acquired. The edge images representing the edges of the target images are less susceptible to the effects of degradation in the area of ​​focus.

[0343] Next, in step s122, the feature acquisition unit 141 acquires the feature quantities of each of the N edge images acquired in step s121. Hereafter, the feature quantities of edge images may be referred to as edge features.

[0344] After step s122, in step s123, the terminal device 4 transmits the N edge features acquired in step s122 to the server 3.

[0345] To obtain edge features, for example, a second encoding model is used, which is part of a second generative model having a similar configuration to generative model 100. The second encoding model obtains edge features from the edge image. The second generative model may be, for example, a variational autoencoder (VAE), a generative adversarial network, a diffusion model, or a regular autoencoder without the "variational" designation.

[0346] The second processing system 1, like the generation model 100, has a second generation model for each of the multiple determination points of the target structure. The feature acquisition unit 141 of the terminal device 4 has a second encoding model for a plurality of second generation models corresponding to each of the multiple determination points. The feature acquisition unit 141 acquires edge features of the edge image corresponding to the target image in which the determination point of interest is captured, using the second encoding model of the second generation model corresponding to the determination point of interest.

[0347] The training of the second generative model is performed, for example, on server 3, similar to the training of the generative model 100. For training the second generative model corresponding to the area of ​​focus, multiple training edge images corresponding to the multiple training images used for training the generative model 100 corresponding to the area of ​​focus may be used. A training edge image corresponding to a training image is an edge image that represents the edge of the training image. Server 3 acquires multiple training edge images corresponding to the multiple training images used for training the generative model 100 corresponding to the area of ​​focus, and trains the second generative model corresponding to the area of ​​focus based on the acquired training edge images. The second encoded model of the trained second generative model is then transferred to terminal device 4.

[0348] In step s131, server 3 receives N edge features from terminal device 4. Next, in step s132, the control unit 30 of server 3 identifies edge features from among the N edge features that are similar to the features of the edge image corresponding to the reference image corresponding to the area of ​​interest for determination. The edge image corresponding to the reference image is an edge image that represents the edges of the reference image. Hereafter, the edge image corresponding to the reference image may be referred to as the reference edge image. Also, the features of the reference edge image may be referred to as the reference edge features.

[0349] In this example, there are multiple reference images corresponding to multiple determination points on the target structure. The reference image corresponding to the determination point of interest is an image obtained when camera 5 photographs the determination point of interest at a desired position and angle. The reference image can be obtained, for example, by having camera 5 of drone 12 photograph the determination point of interest when there is no wind.

[0350] The memory unit 31 of server 3 stores reference edge feature quantities of multiple reference edge images corresponding to multiple reference images corresponding to multiple determination points of the target structure. In step s132, the control unit 30 identifies edge feature quantities from among the N edge feature quantities that are similar to the reference edge feature quantity corresponding to the determination point of interest (i.e., the reference edge feature quantity of the reference edge image corresponding to the reference image corresponding to the determination point of interest). For example, the control unit 30 places the N edge feature quantities and the reference edge feature quantity corresponding to the determination point of interest (also called the target reference edge feature quantity) in the feature quantity space. Then, in the feature quantity space, the control unit 30 identifies the edge feature quantity with the smallest Euclidean distance from the N edge feature quantities to the target reference edge feature quantity as the edge feature quantity similar to the target reference edge feature quantity. Hereafter, the edge feature quantity similar to the target reference edge feature quantity will be called the similar edge feature quantity.

[0351] After step s132, in step s133, the server 3 transmits specific information to the terminal device 4 for identifying similar edge features identified by the control unit 30.

[0352] In step s125, terminal device 4 receives and acquires specific information from server 3. Next, in step s126, feature acquisition unit 141 acquires the feature quantities of the target image corresponding to the similar edge feature quantities identified from the specific information from server 3, from among the N target images acquired in step s120. The target image corresponding to the similar edge feature quantities is the target image corresponding to the edge image that has similar edge feature quantities among the N edge images.

[0353] Here, since similar edge features are similar to the features of the reference edge image, edge images with similar edge features are similar to the reference edge image. And, it can be said that the target image corresponding to an edge image similar to the reference edge image is similar to the reference image corresponding to the reference edge image. Therefore, it can be said that the target image corresponding to an edge image with similar edge features is similar to the reference image. Thus, in step s126, it can be said that the feature acquisition unit 141 acquires the target features of the target images that are similar to the reference image among the N target images.

[0354] After step s126, step s102 is executed, and terminal device 4 transmits the acquired feature quantities of the target image similar to the reference image to server 3. Server 3 executes steps s111 to s114. In step s112, the restoration unit 130 acquires a restored image based on the feature quantities of the target image similar to the reference image corresponding to the area of ​​focus determination. In step s113, the determination unit 131 determines an anomaly at the area of ​​focus determination based on the feature quantities of the target image similar to the reference image corresponding to the area of ​​focus determination.

[0355] Furthermore, the terminal device 4 may change the number of times the camera 5 acquires the target image (i.e., the value of N). In this case, the environmental sensor 8 mounted on the drone 12 (see Figure 42) may include an airflow sensor that detects the airflow around the drone 12. The control unit 40 of the terminal device 4 may increase the number of times the camera 5 acquires the target image (in other words, the number of times the camera 5 takes a picture of the target image) as the airflow detected by the airflow sensor increases (i.e., the wind is stronger around the target area) when the camera 5 photographs the area of ​​interest.

[0356] For example, suppose the initial value of N is 5. When the airflow detected by the airflow sensor is less than the threshold, the control unit 40 keeps the value of N at its initial value and causes the camera 5 to acquire the target image 5 times. In this case, in step s120, the image processing unit 140 acquires 5 target images from the camera 5. On the other hand, when the airflow detected by the airflow sensor is greater than or equal to the threshold, the control unit 40 sets the value of N to, for example, 20 and causes the camera 5 to acquire the target image 20 times. In this case, in step s120, the image processing unit 140 acquires 20 target images from the camera 5.

[0357] In this way, by increasing the number of times the camera 5 acquires images of the target area according to the wind strength around the target area, the number of times the camera 5 acquires images of the target area can be increased when the position and attitude of the drone 12 are highly unstable when the camera 5 is photographing the target area. As a result, even when the position and attitude of the drone 12 are highly unstable, the server 3 can transmit feature quantities of the target area image that are similar to the reference image and have less variation. Therefore, even when the position and attitude of the drone 12 are highly unstable, the server user can more easily identify the state of the target area from the reconstructed target image, and the accuracy of the server 3's detection of anomalies in the target area is improved.

[0358] Furthermore, a 6-axis motion sensor may be mounted on the drone 12. In this case, the control unit 40 may increase the value of N from its initial value if at least one of the 6-axis values ​​of the motion sensor is greater than a threshold when the camera 5 photographs the area of ​​interest. This allows the camera 5 to increase the number of times it acquires images of the area of ​​interest when the camera 5 is photographing the area of ​​interest, similar to when an airflow sensor is used, if the position and attitude of the drone 12 are unstable.

[0359] In the example shown in Figure 43, the identification of similar edge features is performed by Server 3, but it may also be performed by Terminal Device 4. In this case, Server 3 sends reference edge features to Terminal Device 4. Terminal Device 4 identifies edge features from among the N edge images that are similar to the reference edge features from Server 3 (i.e., similar edge features). Then, Terminal Device 4 acquires the feature of interest of the target image corresponding to the edge image having the identified similar edge features (step s126), and sends the acquired feature of interest to Server 3 (step s102).

[0360] Furthermore, even if camera 5 is not mounted on a flying object such as a drone 12, if the position and orientation of camera 5 are unstable when photographing the area of ​​interest, camera 5 may photograph the area of ​​interest multiple times, similar to the example in Figure 43. For example, if a person holding camera 5 is instructing camera 5 to photograph the area of ​​interest, camera 5 may photograph the area of ​​interest multiple times if there is significant camera shake.

[0361] In the example above, server 3 notifies the server user of an alert when it determines that there is an abnormality in the area of ​​interest, but it may also notify the server user of multiple levels of alerts. In this case, the convenience of server 3 will be improved.

[0362] For example, in step s113, the determination unit 131 of server 3 calculates the Euclidean distance (referred to here as the first Euclidean distance) between the current feature of interest and the initial feature of interest in the feature space where the current feature of interest and the initial feature of interest are located. The determination unit 131 then determines that the area of ​​interest is normal if the first Euclidean distance is less than the first threshold. In other words, the determination unit 131 determines that the area of ​​interest is normal when the current feature of interest has not changed much from the initial feature of interest. The determination unit 131 also determines that there is no abnormality in the area of ​​interest, but attention is needed, if the first Euclidean distance is greater than or equal to the first threshold and less than the second threshold. In other words, the determination unit 131 determines that there is no abnormality in the area of ​​interest, but attention is needed, if the current feature of interest has changed to some extent from the initial feature of interest. The determination unit 131 then determines that there is an abnormality in the area of ​​interest if the first Euclidean distance is greater than or equal to the second threshold. The second threshold is set to a value greater than the first threshold. In other words, the determination unit 131 determines that there is an abnormality in the area of ​​interest when the current feature of interest has changed significantly from the initial feature of interest.

[0363] In step s113, if the area of ​​interest is determined to be normal, in step s114, a display screen 750 including a first abnormality determination result 770 indicating normality is displayed on the display unit 37, as shown in the example in Figure 40 above.

[0364] In step s113, if it is determined that there is no abnormality in the area of ​​focus but attention is required for that area, a display screen 750 including caution information 780 indicating that attention is required for the area of ​​focus is displayed on the display unit 37, as shown in Figure 44. This notifies the server user of an alert (which can also be called a first-stage alert) that attention is required for the area of ​​focus.

[0365] In step s113, if an abnormality is detected in the area of ​​interest, a display screen 750 including a first abnormality detection result 770 indicating the abnormality is displayed on the display unit 37, as shown in the examples in Figures 39 and 41 above. This notifies the server user of an alert (which can also be called a second-stage alert) that there is an abnormality in the area of ​​interest.

[0366] The method for determining whether attention is needed for a particular unit of interest is not limited to the examples above. For example, in the second processing system 1, the feature of interest obtained in step s111 of the unit of interest determination process executed one step prior to the currently executing unit of interest determination process is called the previous feature of interest. For example, if the structure determination process is executed every month, the feature of interest obtained in step s111 of the unit of interest determination process executed one year prior to the currently executing unit of interest determination process is the previous feature of interest. Also, the feature of interest obtained in step s111 of the unit of interest determination process executed two steps prior to the currently executing unit of interest determination process is called the feature of interest two steps prior.

[0367] In step s113, the determination unit 131 of server 3 places the current feature of interest, the initial feature of interest, the previous feature of interest, and the feature of interest two steps prior into the feature space. Then, the determination unit 131 calculates the first Euclidean distance between the current feature of interest and the initial feature of interest in the feature space.

[0368] The determination unit 131 determines that there is an abnormality in the area of ​​interest if the first Euclidean distance is greater than or equal to a threshold. On the other hand, if the first Euclidean distance is less than a threshold, the determination unit 131 calculates the second Euclidean distance between the current area of ​​interest and the previous area of ​​interest, and the third Euclidean distance between the previous area of ​​interest and the area of ​​interest two levels prior, in the custom quantity space. Next, the determination unit 131 calculates the absolute value of the difference between the second Euclidean distance and the third Euclidean distance. Then, the determination unit 131 determines that the area of ​​interest is normal if the absolute value of the calculated difference is less than a threshold. In other words, the determination unit 131 determines that there is no abnormality in the area of ​​interest if the current area of ​​interest has not changed much from the initial area of ​​interest, and the amount of change in the area of ​​interest since the last time has not changed significantly. On the other hand, if the absolute value of the calculated difference is greater than or equal to a threshold, the determination unit 131 determines that there is no abnormality in the area of ​​interest, but attention should be paid to the area of ​​interest. In other words, the determination unit 131 determines that even if the current feature of interest has not changed much from the initial feature of interest, if the amount of change in the feature of interest since the last time has changed significantly, there is no abnormality in the area of ​​interest being determined, but attention should be paid to the area of ​​interest being determined.

[0369] Furthermore, if the first Euclidean distance is less than a threshold, the determination unit 131 may determine a first vector in the feature space from the previous feature of interest to the current feature of interest, and a second vector from the feature of interest two steps prior to the previous feature of interest to the feature of interest one step prior. The determination unit 131 may then determine that the area of ​​interest is normal if the angle between the first vector and the second vector is less than a threshold. In other words, if the current feature of interest has not changed much from the initial feature of interest, and the direction of change in the feature of interest since the last time has not changed significantly, the determination unit 131 may determine that there is no abnormality in the area of ​​interest. On the other hand, if the angle between the first vector and the second vector is greater than or equal to a threshold, the determination unit 131 may determine that there is no abnormality in the area of ​​interest, but attention should be paid to the area of ​​interest. In other words, if the current feature of interest has not changed much from the initial feature of interest, but the direction of change in the feature of interest since the last time has changed significantly, the determination unit 131 may determine that there is no abnormality in the area of ​​interest, but attention should be paid to the area of ​​interest.

[0370] The method by which the determination unit 131 determines an anomaly in the area of ​​interest is not limited to the above example. For example, the determination unit 131 may use a machine learning model to determine an anomaly in the area of ​​interest. In this case, the machine learning model provided by the determination unit 131 receives the feature of interest as input, and the machine learning model outputs a result indicating whether the area of ​​interest has an anomaly.

[0371] Furthermore, the determination unit 131 may determine abnormalities in the area of ​​interest in a manner similar to the method for determining the harvest time of the crop of interest as shown in the example in Figure 9. In this case, the storage unit 31 of the server 3 stores multiple determination features for determining abnormalities in the area of ​​interest. These multiple determination features are then grouped into an abnormality group consisting of multiple determination features corresponding to the features of the target image in which the abnormal area of ​​interest is captured, and a normal group consisting of multiple determination features corresponding to the features of the target image in which a normal area of ​​interest is captured. The determination unit 131 may determine abnormalities in the area of ​​interest by, for example, using the Mahalanobis distance in the feature space to determine whether the feature belongs to the abnormal group or the normal group.

[0372] Furthermore, if the target structure is a structure with a continuous surface structure, such as the inner wall of a tunnel, the determination unit 131 may determine an anomaly in the area of ​​interest based on the feature quantities of the target image showing the area of ​​interest and the feature quantities of multiple captured images showing each of the multiple locations surrounding the area of ​​interest. In this case, the camera 5 photographs not only the area of ​​interest but also each of the multiple locations surrounding the area of ​​interest. By appropriately changing the position of the drone 12 when the camera 5 is taking pictures, the camera 5 can photograph both the area of ​​interest and each of the multiple locations surrounding the area of ​​interest. The terminal device 4 acquires not only the feature quantities of interest but also the feature quantities (also called surrounding feature quantities) of the multiple captured images taken by the camera 5 showing each of the multiple locations surrounding the area of ​​interest. The terminal device 4 then transmits the feature quantities of interest and the multiple surrounding feature quantities to the server 3. Hereafter, the captured images showing the locations surrounding the area of ​​interest may be referred to as surrounding images. The feature quantities of the surrounding images become the surrounding feature quantities.

[0373] Upon receiving the feature of interest and multiple surrounding features, the server 3 determines that the feature of interest and the multiple surrounding features are arranged in the feature space. Next, the determination unit 131 calculates the Euclidean distance between each surrounding feature and the feature of interest in the feature space. The determination unit 131 then determines that the area of ​​interest is normal if each of the calculated Euclidean distances is below a threshold. In other words, the determination unit 131 determines that the area of ​​interest is normal if its appearance is not significantly different from its surroundings. On the other hand, the determination unit 131 determines that the area of ​​interest is abnormal if at least one of the calculated Euclidean distances is above a threshold. In other words, the determination unit 131 determines that the area of ​​interest is abnormal if its appearance is significantly different from its surroundings.

[0374] Server 3 may obtain a reconstructed image (also called a reconstructed surrounding image) for each of the multiple surrounding feature quantities, based on the said surrounding feature quantity. Server 3 may then display a display screen 750 that includes not only a reconstructed image of the area of ​​interest, but also multiple reconstructed surrounding images, each of which shows multiple areas surrounding the area of ​​interest.

[0375] Figure 45 is a schematic diagram showing an example of a display screen 750 that includes a focus image 760 and multiple surrounding image reconstructions 761. In the example of Figure 45, the focus image 760 and the multiple surrounding image reconstructions 761 are displayed on the display screen 750 such that the positional relationship between the focus image 760 and the multiple surrounding image reconstructions 761 matches the positional relationship between the focus determination point and multiple surrounding points around the focus determination point.

[0376] In the example above, server 3 displays the display screen 750, which includes the restored image of the area of ​​interest, regardless of whether it has determined that there is an abnormality in the area of ​​interest. However, it is not necessary to display the display screen 750 when it has determined that there is an abnormality in the area of ​​interest, and not to display the display screen 750 when it has determined that the area of ​​interest is normal.

[0377] Furthermore, when the server 3 determines that there is an abnormality at the area of ​​focus during the currently executing area of ​​focus determination process, it may display a display screen 750 that includes the restored area of ​​focus image obtained in the currently executing area of ​​focus determination process and the restored area of ​​focus images obtained in past area of ​​focus determination processes (i.e., area of ​​focus determination processes executed before the currently executing area of ​​focus determination process). In this case, the display screen 750 may include multiple restored area of ​​focus images obtained in multiple area of ​​focus determination processes executed immediately before the currently executing area of ​​focus determination process, as the restored area of ​​focus images obtained in past area of ​​focus determination processes. This allows the server user to check not only the area of ​​focus of focus at the time the abnormality occurred, but also the area of ​​focus of focus immediately before the abnormality occurred, from the restored image displayed on the server 3.

[0378] Figure 46 is a schematic diagram showing an example of a display screen 750 that includes the restored focus image 760c acquired in the currently running focus unit determination process, the restored focus image 760d acquired in the focus unit determination process executed immediately before the currently running focus unit determination process, and the restored focus image 760e acquired in the focus unit determination process executed two processes before the currently running focus unit determination process.

[0379] Furthermore, when server 3 determines that there is an abnormality in the area of ​​focus, it may automatically acquire the image of the area of ​​focus from terminal device 4 and display a display screen 750 in which the acquired image of the area of ​​focus is included in place of the restored image of the area of ​​focus. In this case, as shown in Figure 47, server 3 may display a display screen 750 that includes the image of the area of ​​focus 900 and the restored images of the area of ​​focus 760d and 760e acquired in past area of ​​focus determination processing. When server 3 displays the image of the area of ​​focus 900 when it determines that there is an abnormality in the area of ​​focus, the server user can check the state of the area of ​​focus from the original clear image of the area of ​​focus.

[0380] <Second example of abnormality detection> The processing system 1 in this example (also called the third processing system 1) can detect abnormalities in liquids. For example, the third processing system 1 can detect abnormalities in wastewater treated by the wastewater treatment system. The detection of abnormalities in wastewater can be seen as a detection of abnormalities in the treatment by the wastewater treatment system, or as a detection of abnormalities in the wastewater treatment system. It can also be said that the third processing system 1 determines whether the wastewater is being treated normally.

[0381] In this example, camera 5 is fixedly installed near the waste liquid (also called the target waste liquid) that the third processing system 1 is determining to be abnormal. In this example, the target waste liquid is the target object, and the image of the target waste liquid captured by camera 5 becomes the target image of the target object. The third processing system 1 determines the abnormality of the target waste liquid based on the target image of the target waste liquid acquired by camera 5. Note that the third processing system 1 does not necessarily have to include a location information acquisition unit 6.

[0382] In the third processing system 1, as described above, the terminal device 4 acquires feature quantities of the target image from the camera 5. The terminal device 4 then transmits the feature quantities of the target image to the server 3. The server 3 acquires a reconstructed image by restoring the target image based on the feature quantities of the target image. The server 3 then displays the acquired reconstructed image. The server 3 also determines whether there is an abnormality in the target waste liquid captured in the target image based on the feature quantities of the target image. The server 3 then displays the result of the abnormality determination of the target waste liquid (also called the second abnormality determination result) together with the reconstructed image in which the target waste liquid is captured.

[0383] A server user may be, for example, an administrator managing a wastewater treatment system. The server user can make appropriate judgments regarding the target wastewater by referring to, for example, the restored image displayed on server 3 and the second abnormality judgment result displayed on server 3. For example, the server user may determine that the target wastewater is normal, that progress monitoring of the target wastewater is necessary, or that on-site inspection of the target wastewater is necessary. The fact that the target wastewater is normal can also be said to mean that the target wastewater is being treated normally, or that the treatment of the target wastewater in the wastewater treatment system is normal.

[0384] Furthermore, camera 5 does not necessarily have to be installed near the target waste liquid. In this case, for example, the person holding terminal device 4 and camera 5 may move to the vicinity of the target waste liquid, allowing camera 5 to photograph the target waste liquid.

[0385] <Details of the operation example of the third processing system> The third processing system 1 performs a waste liquid determination process to determine abnormalities in the target waste liquid based on the target image obtained by the camera 5. The waste liquid determination process is performed repeatedly, for example, at intervals of time. For example, the waste liquid determination process may be performed every 30 minutes or every hour.

[0386] In this example, the "target image" refers to the image showing the target waste liquid. Hereafter, the target waste liquid may be referred to as the "target waste liquid." In the waste liquid detection process, abnormalities in the target waste liquid are determined based on the target image obtained by camera 5.

[0387] Figure 48 is a flowchart showing an example of waste liquid determination processing. In waste liquid determination processing, first, in step s150, the image processing unit 140 of the terminal device 4 acquires a target image containing the waste liquid of interest from the camera 5. The target image acquired by the image processing unit 140 is stored in the storage unit 41.

[0388] Next, in step s151, the feature acquisition unit 141 acquires the feature quantities of the image of interest, i.e., the feature quantity of interest. Then, in step s152, the terminal device 4 transmits the feature quantity of interest acquired in step s151 to the server 3.

[0389] The feature acquisition unit 141 has, for example, an encoding model 101 of the generation model 100, similar to the above. The restoration unit 130 of the server 3 also has, similar to the above, a decoding model 102 of the generation model 100. The encoding model 101 of the feature acquisition unit 141 acquires the features of the target image based on the target image. The decoding model 102 of the restoration unit 130 acquires a restored target image by restoring the target image based on the features acquired by the encoding model 101. Multiple training images used to train the generation model 100 include images showing abnormal waste liquid and images showing normal waste liquid. Abnormal waste liquid includes waste liquid containing foreign matter that should be removed by processing in the waste liquid treatment system.

[0390] In the waste liquid determination process, in step s161, server 3 receives and acquires the feature quantity of interest from terminal device 4. The feature quantity of interest acquired by server 3 is stored in storage unit 31.

[0391] Next, in step s162, the restoration unit 130 (specifically the decoding model 102) of the server 3 obtains a restored image of interest, which is the restored image of interest, based on the features of interest acquired in step s161.

[0392] Next, in step s163, the determination unit 131 determines an abnormality in the waste liquid of interest captured in the image of interest based on the feature quantities of interest acquired in step s161. Similar to the determination of abnormalities in the area of ​​interest of the target structure, the determination unit 131 determines an abnormality in the waste liquid of interest based, for example, on the feature quantities of interest acquired in step s161 and feature quantities of interest acquired in the past.

[0393] In this example, the feature quantity of interest obtained in step s161 of the waste liquid determination process that was first executed after the operation of the third processing system 1 began is called the initial feature quantity of interest. Furthermore, the feature quantity of interest obtained in step s161 of the waste liquid determination process currently being executed in the third processing system 1 is called the current feature quantity of interest.

[0394] The determination unit 131 determines whether the waste liquid of interest is abnormal, for example, based on the current feature of interest and the initial feature of interest. For example, the determination unit 131 places the current feature of interest and the initial feature of interest in the feature space. Then, the determination unit 131 calculates the Euclidean distance between the current feature of interest and the initial feature of interest in the feature space. The determination unit 131 determines that there is an abnormality in the waste liquid of interest if the calculated Euclidean distance is greater than or equal to a threshold. On the other hand, the determination unit 131 determines that the waste liquid of interest is normal if the calculated Euclidean distance is less than the threshold. When the Euclidean distance between the current feature of interest and the initial feature of interest is greater than or equal to a threshold, it means that the state of the waste liquid of interest has changed significantly from the state when the operation of the third treatment system 1 started. Therefore, if the Euclidean distance between the current feature of interest and the initial feature of interest is greater than or equal to a threshold, it can be seen that the waste liquid of interest, which was normal when the operation of the third treatment system 1 started, has become abnormal due to a malfunction of the waste liquid treatment system or the like.

[0395] Furthermore, when the determination unit 131 determines that there is an abnormality in the waste liquid of interest, it may use the feature quantities of an image showing a normal waste liquid of interest (also called a reference feature quantity) instead of the initial feature quantities of interest. In this case, if the Euclidean distance between the current feature quantity of interest and the reference feature quantity in the feature space where the current feature quantity of interest and the reference feature quantity are located is greater than or equal to a threshold, it may be determined that there is an abnormality in the waste liquid of interest.

[0396] If an abnormality in the waste liquid of interest is detected in step s163, step s164 is executed. In step s164, the control unit 30 displays the restored image of interest acquired by the restoration unit 130 and the result of the determination of the abnormality of the waste liquid of interest detected by the determination unit 131 on the display unit 37.

[0397] Figures 49 and 50 are schematic diagrams showing examples of the display of the display unit 37 in step s164. Figure 49 shows an example of the display in step s164 when an abnormality is determined in step s163 at the area of ​​interest. Figure 50 shows an example of the display in step s164 when the area of ​​interest is determined to be normal in step s163.

[0398] As shown in Figures 49 and 50, in step s164, the display unit 37 displays a display screen 800 that includes, for example, a restored image 810 of the waste liquid of interest and a second abnormality determination result 820 of the waste liquid of interest. The second abnormality determination result 820 shown in Figure 49 indicates that there is an abnormality in the waste liquid of interest. The second abnormality determination result 820 shown in Figure 50 indicates that the waste liquid of interest is normal. By displaying the second abnormality determination result 820 indicating an abnormality, the server 3 can notify the server user of an alert if it determines that there is an abnormality in the waste liquid of interest.

[0399] In this way, by displaying the reconstructed image of interest and the second anomaly judgment result, the server user can refer to the reconstructed image of interest and the second anomaly judgment result and make an appropriate judgment regarding the waste liquid of interest. For example, if the second anomaly judgment result indicates an anomaly, and the server user checks the state of the waste liquid of interest as seen in the reconstructed image of interest and determines that there is an anomaly in the waste liquid of interest, the server user may decide that it is necessary to check the operation of the waste liquid treatment system. Also, even if the second anomaly judgment result indicates a normal state, if the server user checks the state of the waste liquid of interest as seen in the reconstructed image of interest and determines that there is something wrong with the waste liquid of interest, the server user may decide that it is necessary to check the operation of the waste liquid treatment system.

[0400] Furthermore, when server 3 determines that there is an abnormality in the waste liquid of interest, it will display screen 750, but it does not need to display screen 750 when it determines that the waste liquid of interest is normal.

[0401] The method by which the determination unit 131 determines abnormalities in the waste liquid of interest is not limited to the above example. For example, the determination unit 131 may use a machine learning model to determine abnormalities in the waste liquid of interest. In this case, the machine learning model provided by the determination unit 131 is input with the features of interest, and the machine learning model outputs a determination result of whether the waste liquid of interest is abnormal.

[0402] Alternatively, the determination unit 131 may determine abnormalities in the wastewater of interest in a manner similar to the method for determining the harvest time of the crop of interest as shown in the example in Figure 9. In this case, the storage unit 31 of the server 3 stores multiple determination features for determining abnormalities in the wastewater of interest. The determination features are the features of the image in which the wastewater is captured. The multiple determination features are then grouped into an abnormal group consisting of the features of the image in which the wastewater of interest is captured, and a normal group consisting of the features of the image in which the wastewater of interest is captured. The determination unit 131 may determine abnormalities in the wastewater of interest by, for example, using the Mahalanobis distance in the feature space to determine whether the feature of interest belongs to the abnormal group or the normal group.

[0403] If the determination unit 131 determines that there is an abnormality in the waste liquid of interest, it may determine the type of abnormality in the waste liquid of interest based on the feature quantity of interest. For example, suppose there are three types of abnormalities in the waste liquid: a first abnormality in which a first foreign substance is mixed in the waste liquid, a second abnormality in which a second foreign substance is mixed in the waste liquid, and a third abnormality in which a third foreign substance is mixed in the waste liquid. The first, second, and third foreign substances are all different types of foreign substances. The determination unit 131 may determine whether the abnormality in the waste liquid of interest is a first, second, or third abnormality. In this example, it is assumed that the waste liquid does not contain multiple types of foreign substances simultaneously.

[0404] The determination unit 131 may, for example, include a machine learning classification model. The classification model may be EfficientNet, ResNet, or MobileNet. The classification model may, for example, calculate and output a first confidence score indicating the probability that the anomaly of the waste liquid of interest is a first anomaly, a second confidence score indicating the probability that the anomaly of the waste liquid of interest is a second anomaly, and a third confidence score indicating the probability that the anomaly of the waste liquid of interest is a third anomaly, based on the features of interest. The confidence scores are also called confidence scores or confidence scores. The classification model may be trained using supervised learning. Multiple training features used to train the classification model include features of an image showing waste liquid contaminated with a first foreign substance, features of an image showing waste liquid contaminated with a second foreign substance, and features of an image showing waste liquid contaminated with a third foreign substance.

[0405] The determination unit 131 identifies the maximum score among the first confidence score, second confidence score, and third confidence score output by the classification model. Then, if the maximum score is greater than or equal to a threshold, the determination unit 131 determines that the type of abnormality in the waste liquid of interest corresponds to the type of abnormality of the maximum score. For example, if the maximum score is the first confidence score and the first confidence score is greater than or equal to a threshold, the determination unit 131 determines that the type of abnormality in the waste liquid of interest is the first abnormality. In other words, the determination unit 131 determines that the waste liquid of interest contains the first foreign substance. Also, if the maximum score is the second confidence score and the second confidence score is greater than or equal to a threshold, the determination unit 131 determines that the type of abnormality in the waste liquid of interest is the second abnormality. In other words, the determination unit 131 determines that the waste liquid of interest contains the second foreign substance. And if the maximum score is the third confidence score and the third confidence score is greater than or equal to a threshold, the determination unit 131 determines that the type of abnormality in the waste liquid of interest is the third abnormality. In other words, the determination unit 131 determines that the waste liquid of interest contains a third foreign substance. On the other hand, if the maximum score is below the threshold, the determination unit 131 determines that it could not determine the type of abnormality in the waste liquid of interest.

[0406] When the determination unit 131 determines the type of abnormality of the waste liquid of interest, the display unit 37 may display a display screen 800 that includes the determination result of the determination unit 131 (also called the abnormality type determination result).

[0407] Figures 51 and 52 are schematic diagrams showing an example of a display screen 800 including an abnormality type determination result 830. The abnormality type determination result 830 shown in Figure 51 indicates that the abnormality type of the waste liquid of interest is a first abnormality. In other words, the abnormality type determination result 830 indicates that a first foreign substance is mixed in the waste liquid of interest. The abnormality type determination result 830 shown in Figure 52 indicates that the abnormality type of the waste liquid of interest could not be determined.

[0408] As shown in the example in Figure 52, when server 3 notifies the server user that it was unable to determine the type of abnormality of the waste liquid of interest, the server user can, for example, determine the type of abnormality of the waste liquid of interest from the restored image 810 included in the display screen 800, or determine the type of abnormality of the waste liquid of interest by actually inspecting the waste liquid of interest on-site.

[0409] The method by which the determination unit 131 determines the type of abnormality in the waste liquid of interest is not limited to the above example. For example, the determination unit 131 may determine the type of abnormality in the waste liquid of interest in the same manner as the method for determining the harvest time of the crop of interest as shown in the example in Figure 9. In this case, the storage unit 31 of the server 3 stores a plurality of determination feature quantities for determining the type of abnormality in the waste liquid of interest. The determination feature quantities are feature quantities of the image in which the abnormal waste liquid is captured. The plurality of determination feature quantities are then grouped into a first abnormality group consisting of feature quantities of the target image in which the first abnormal waste liquid of interest of interest is captured, a second abnormality group consisting of feature quantities of the target image in which the second abnormal waste liquid of interest is captured, and a third abnormality group consisting of feature quantities of the target image in which the third abnormal waste liquid of interest is captured. The determination unit 131 may, for example, determine the type of abnormality at the location of interest by determining which of the first, second, or third abnormality group the feature quantity belongs to, using the Mahalanobis distance in the feature space. Furthermore, the determination unit 131 may determine that it could not determine the type of abnormality of the waste liquid of interest if the feature of interest is far removed from any of the first abnormality group, the second abnormality group, and the third abnormality group in the feature space.

[0410] The third treatment system 1 may have multiple generation models 100 corresponding to multiple states that the waste liquid of interest may be in. For example, the third treatment system 1 may have multiple generation models 100 corresponding to multiple types of abnormalities in the waste liquid of interest. For example, the third treatment system 1 may include a generation model 100 corresponding to a first abnormality (also called the first abnormality model 100), a generation model 100 corresponding to a second abnormality (also called the second abnormality model 100), and a generation model 100 corresponding to a third abnormality (also called the third abnormality model 100). The first abnormality, the second abnormality, and the third abnormality can each be said to be states that the waste liquid of interest may be in, and can be said to be abnormal states that the waste liquid of interest may be in. The first abnormality model 100, the second abnormality model 100, and the third abnormality model 100 are models corresponding to multiple states that the waste liquid of interest may be in.

[0411] The first anomaly model 100 is trained, for example, based only on training images showing waste liquid containing a first foreign substance. The second anomaly model 100 is trained, for example, based only on training images showing waste liquid containing a second foreign substance. The third foreign substance model 100 is trained, for example, based only on training images showing waste liquid containing only a third foreign substance.

[0412] Hereafter, the encoding model 101 and decoding model 102 of the first anomaly model 100 will be referred to as the first anomaly model 101 and the first anomaly model 102, respectively. Similarly, the encoding model 101 and decoding model 102 of the second anomaly model 100 will be referred to as the second anomaly model 101 and the second anomaly model 102, respectively. And the encoding model 101 and decoding model 102 of the third anomaly model 100 will be referred to as the third anomaly model 101 and the third anomaly model 102, respectively.

[0413] In step s150 of the waste liquid determination process, the first anomaly model 101, the second anomaly model 101, and the third anomaly model 101 of the feature acquisition unit 141 each acquire feature quantities of the image of interest. Then, in step s152, the terminal device 4 transmits the multiple feature quantities of interest acquired in step s151 to the server 3.

[0414] In step s161, the server 3 receives multiple feature quantities of interest from the terminal device 4. In step s162, the first anomaly model 102, the second anomaly model 102, and the third anomaly model 103 of the reconstruction unit 130 each acquire a reconstructed image of interest based on the feature quantities of interest. The first anomaly model 102 acquires a reconstructed image of interest based on the feature quantities of interest acquired by the first anomaly model 101. The third anomaly model 102 acquires a reconstructed image of interest based on the feature quantities of interest acquired by the third anomaly model 101.

[0415] In step s163, the determination unit 131 makes a preliminary determination of whether the waste liquid of interest is abnormal based on each of the multiple feature quantities of interest acquired in step s161. If the determination unit 131 determines that the waste liquid of interest is abnormal in the preliminary determination for at least one of the multiple feature quantities of interest, it makes a final determination that the waste liquid of interest is abnormal. On the other hand, if the determination unit 131 determines that the waste liquid of interest is normal in the preliminary determination for all of the multiple feature quantities of interest, it makes a final determination that the waste liquid of interest is normal. Subsequently, in step s164, the display unit 37 displays a display screen 800 including the final second abnormality determination result 820 and the multiple restored images of interest acquired in step s162.

[0416] Figure 53 is a schematic diagram showing an example of a display screen 800 that is displayed in step s164 and includes multiple focus reconstructed images. The display screen 800 includes a focus reconstructed image 810a acquired by the first anomaly model 102, a focus reconstructed image 810b acquired by the second anomaly model 102, and a focus reconstructed image 810c acquired by the third anomaly model 102.

[0417] If a first anomaly occurs in the waste liquid of interest, the reconstructed image 810a acquired with the first anomaly model 102 will show the waste liquid of interest containing the first foreign substance more clearly. If a second anomaly occurs in the waste liquid of interest, the reconstructed image 810b acquired with the second anomaly model 102 will show the waste liquid of interest containing the second foreign substance more clearly. If a third anomaly occurs in the waste liquid of interest, the reconstructed image 810c acquired with the third anomaly model 102 will show the waste liquid of interest containing the third foreign substance more clearly.

[0418] In this way, when the reconstructed images of the target waste liquid, each acquired by one of the multiple generation models 100 corresponding to different types of abnormalities in the target waste liquid, are displayed on the server 3, the server user can more easily identify the type of abnormality in the target waste liquid from the reconstructed image.

[0419] The third processing system 1 may, in the waste liquid determination process, use a generation model 100 corresponding to the surrounding environment of the waste liquid of interest in the image of interest, in the same manner as described in the second processing system 1. In this case, similar to the example described in the second processing system 1, the third processing system 1 may include, for example, a sunny generation model 100 corresponding to when the surrounding environment of the waste liquid of interest is sunny, a cloudy generation model 100 corresponding to when the surrounding environment of the waste liquid of interest is cloudy, and a rainy generation model 100 corresponding to when it is raining in the waste liquid surrounding the area of ​​interest determination. The terminal device 4 may then acquire the feature quantities of the image of interest using the encoding model 101 of the generation model 100 corresponding to the weather surrounding the waste liquid of interest, among the sunny generation model 100, cloudy generation model 100, and rainy generation model 100, based on the detection result from the rain sensor that detects rain around the waste liquid of interest and the detection result from the illuminance sensor that detects the illuminance around the waste liquid of interest. At this time, the restoration unit 130 of server 3 uses the decoding model 102 of the generation model 100 that has the encoding model 101 used by terminal device 4 to acquire the features of interest, from among the generation model 100 for sunny weather 100, the generation model 100 for cloudy weather 100, and the generation model 100 for rainy weather 100, to acquire the restored image of interest based on the features of interest.

[0420] The third treatment system 1 may individually determine abnormalities for each of the multiple locations in the target waste liquid. In this case, each of the multiple locations in the target waste liquid becomes a target object. Similar to the second treatment system 1, the third treatment system 1 may have a separate generation model 100 for each of the multiple locations in the target waste liquid. Furthermore, the third treatment system 1 may have multiple separate generation models 100 for each of the multiple locations in the target waste liquid. In addition, when the third treatment system 1 individually determines the type of abnormality for each of the multiple locations in the target waste liquid, it may have a separate classification model for each of the multiple locations in the target waste liquid.

[0421] The display screen 800 shown in step s164 may include similar images having features similar to the feature of interest, similar to the display screen 330 shown in Figure 14 above. In this case, the server 3 may use multiple training images used to train the generative model 100 as multiple extraction target images from which similar images of interest can be extracted, and among the multiple extraction target images, the extraction target image having features similar to the feature of interest can be used as the similar image of interest.

[0422] Furthermore, as shown in the examples in Figures 15-17 above, if no similar images of interest exist among multiple images to be extracted, the server 3 may acquire the image of interest from the terminal device 4 and display the acquired image of interest.

[0423] Furthermore, as shown in the example in Figure 18 above, after displaying the restored image of interest, the server 3 may, in response to instructions from the server user, acquire the target image of interest from the terminal device 4 and display the acquired target image of interest.

[0424] Furthermore, when server 3 acquires a target image from terminal device 4, it may include the acquired target image among multiple extracted target images.

[0425] Furthermore, as shown in the examples in Figures 21 to 26 above, the third processing system 1 may, in the waste liquid determination process, modify the focus restoration image in response to instructions from the server user and display the modified focus restoration image.

[0426] Furthermore, the server 3 may include the corrected restored image (i.e., the corrected restored image) among multiple images to be extracted. In this case, when the display unit 37 displays the corrected restored image as a similar image of interest, it may display notification information 600 indicating that the similar image of interest is a restored image, together with the similar image of interest, as shown in the examples in Figures 31 and 32.

[0427] Furthermore, when the server 3 displays the modified and restored image, which is the similar image of interest, and the notification information 600, it may display a non-restored similar image from among the multiple images to be extracted stored in the storage unit 31, in response to instructions from the server user, as shown in the examples in Figures 32 and 33. In this example, a non-restored similar image is an extracted image that has similar features to the features of interest, but is not a restored image obtained by the restoration unit 130.

[0428] Furthermore, in the wastewater determination process, the server 3 may predict the future features of the wastewater of interest based on the wastewater of interest's past features, similar to when predicting the future features of the crop of interest and when predicting the future features of the location of the target structure. In this case, the control unit 30 may input the predicted values ​​of the future features to the decode model 102 of the restoration unit 130. The image output by the decode model 102 based on the predicted values ​​of the future features will be a predicted image of the target image in which the wastewater of interest will be captured in the future. The display unit 37 may display the predicted image of the target image in which the wastewater of interest will be captured in the future, obtained by the decode model 102, together with the restored image of the location of interest captured today, as shown in the example in Figure 35 above.

[0429] Furthermore, Server 3 may notify the server user of multiple levels of alerts, similar to the case when determining abnormalities in a structure. In this case, Server 3 may notify the server user of a first-level alert when it determines that attention is needed regarding the waste liquid of interest, even though it cannot be said that there is an abnormality in the waste liquid of interest, and notify the server user of a second-level alert when it determines that there is an abnormality in the waste liquid of interest.

[0430] The method by which Server 3 determines whether attention is needed regarding the waste liquid of interest is the same as the method by which Server 3 determines whether attention is needed regarding the location of interest in a structure. For example, if the first Euclidean distance is greater than or equal to the first threshold but less than the second threshold, Server 3 may determine that there is no abnormality in the waste liquid of interest, but attention is needed regarding the location of interest. Also, if the first Euclidean distance is less than the threshold, and the absolute value of the difference between the second and third Euclidean distances is greater than or equal to the threshold, Server 3 may determine that there is no abnormality in the location of interest, but attention is needed regarding the location of interest.

[0431] When the server 3 determines that there is an abnormality in the waste liquid of interest during the currently running waste liquid determination process, it may display a display screen 800 that includes the restored image of interest acquired in the currently running waste liquid determination process and the restored image of interest acquired in past waste liquid determination processes, as shown in the example in Figure 46 above. In this case, the display screen 800 may include multiple restored images of interest acquired in multiple waste liquid determination processes that were executed immediately before the currently running waste liquid determination process, as the restored images of interest acquired in past waste liquid determination processes.

[0432] Furthermore, when server 3 determines that there is an abnormality in the waste liquid of interest, it may automatically acquire the image of the object of interest from terminal device 4, as shown in the example in Figure 47 above, and display a display screen 800 that includes the acquired image of the object of interest in place of the restored image of interest. In this case, when server 3 determines that there is an abnormality in the waste liquid of interest, the server user can check the condition of the waste liquid of interest from the original clear image of the object of interest.

[0433] Furthermore, if the server 3 cannot determine the type of abnormality in the waste liquid of interest, it may automatically acquire the image of the object of interest from the terminal device 4 and display a display screen 800 that includes the acquired image of the object of interest in place of the restored image of interest. In this case, if the server 3 cannot determine the type of abnormality in the waste liquid of interest, the server user can identify the type of abnormality from the original clear image of the object of interest.

[0434] In the example above, the third treatment system 1 detects abnormalities in the wastewater, but it may also detect abnormalities in other liquids. For example, the third treatment system 1 may detect abnormalities in the water flowing into a river.

[0435] In the examples above, the target image in which the object is captured was a color image, which is a type of visible light image, but it may be any other type of image. For example, the target image may be a grayscale image. Alternatively, the target image may be an infrared image obtained with an infrared camera. Alternatively, the target image may be a millimeter-wave image obtained with an imaging radar that uses millimeter waves.

[0436] The functions of the elements disclosed herein may be implemented using general-purpose processors, dedicated processors, integrated circuits, ASICs ("Application-Specific Integrated Circuits"), conventional circuit configurations and / or combinations thereof, or processing circuit configurations, which are configured to perform the disclosed elements or programmed to perform the disclosed functions. A processor is considered a processing circuit configuration or circuit configuration if it includes transistors and other circuit configurations within it. In this disclosure, a circuit configuration, unit or means is hardware that performs the listed functions or hardware programmed to perform such functions. Hardware may be any hardware disclosed herein or other known hardware programmed to perform the listed functions or configured to perform such functions. When hardware is a processor that may be considered a type of circuit configuration, a circuit configuration, means or unit is a combination of hardware and software, software used to configure the hardware and / or processor.

[0437] As described above, the processing system has been explained in detail, but the above explanation is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied insofar as they do not contradict each other. And it is understood that countless variations not illustrated can be envisioned without falling outside the scope of this disclosure.

[0438] This disclosure includes the following aspects:

[0439] The processing system according to the first embodiment includes a first processing unit that acquires a first feature quantity of a first image and transmits the first feature quantity, and a second processing unit that receives the first feature quantity from the first processing unit, acquires a first reconstructed image obtained by reconstructing the first image based on the first feature quantity, and performs processing using the first reconstructed image.

[0440] The processing system according to the second embodiment is the processing system according to the first embodiment, wherein the second processing device displays the first restored image.

[0441] The processing system according to the third embodiment is the processing system according to the second embodiment, wherein the second processing device stores a plurality of second images and displays together the first reconstructed image and similar images from the plurality of second images that have a feature quantity similar to the first feature quantity.

[0442] The processing system according to the fourth embodiment is the processing system according to the first embodiment, wherein the second processing device stores a plurality of second images, and when it determines that there are similar images having similar features to the first feature in the plurality of second images, it displays the similar images and the first reconstructed image together, and when it determines that there are no similar images in the plurality of second images, it obtains the first image from the first processing device and displays the obtained first image.

[0443] The processing system according to the fifth embodiment is a processing system according to the third or fourth embodiment, wherein if the similar image is a reconstructed image obtained by the second processing device, the second processing device displays notification information indicating that the similar image is a reconstructed image together with the similar image.

[0444] The processing system according to the sixth embodiment is the processing system according to the fifth embodiment, wherein if the similar image is a reconstructed image obtained by the second processing device, the second processing device displays, in response to instructions from the user, a second image from among the plurality of second images that has a feature quantity similar to the first feature quantity, and is not a reconstructed image obtained by the second processing device.

[0445] The processing system according to the seventh embodiment is a processing system according to any one of the second to sixth embodiments, wherein the second processing unit transmits at least a portion of the first reconstructed image to the first processing unit in response to instructions from the user, the first processing unit acquires difference information indicating the difference between at least a portion of the first reconstructed image and the first image, transmits the acquired difference information to the second processing unit, the second processing unit modifies the first reconstructed image based on the difference information, and displays the modified first reconstructed image.

[0446] The processing system according to the eighth embodiment is a processing system according to any one of the second to seventh embodiments, wherein the second processing unit instructs the first processing unit to change the model for acquiring the first feature in response to instructions from the user, the first processing unit changes the model for acquiring the first feature in response to the instruction from the second processing unit to change the model, acquires the first feature anew using the changed model, transmits the acquired new first feature to the second processing unit, the second processing unit acquires the first reconstructed image anew based on the new first feature from the first processing unit, and displays the acquired new first reconstructed image.

[0447] The processing system according to the ninth aspect is a processing system according to the eighth aspect, wherein the second processing unit determines, based on the first reconstructed image, whether or not there is a scratch on the first object shown in the first reconstructed image, and if it determines that there is a scratch on the first object, it instructs the first processing unit to change the model for acquiring the first feature quantity in response to instructions from the user, the first processing unit changes the model for acquiring the first feature quantity to a model corresponding to the first object having a scratch in response to the instruction from the second processing unit to change the model, acquires the first feature quantity anew using the changed model, transmits the newly acquired first feature quantity to the second processing unit, and the second processing unit acquires the first reconstructed image anew based on the new first feature quantity from the first processing unit and displays the newly acquired first reconstructed image.

[0448] The processing system according to the 10th aspect is a processing system according to any one of the 1st to 7th aspects, wherein the first processing device acquires the first feature amount of the first image in which the first object appears and the second feature amount of the second image in which a second object located at a location different from the first object appears, transmits the first feature amount and the second feature amount to the second processing device, the second processing device acquires the first restored image obtained by restoring the first image based on the first feature amount, acquires the second restored image obtained by restoring the second image based on the second feature amount, performs processing using the first restored image and processing using the second restored image, and the first processing device has a first model for acquiring the first feature amount and a second model for acquiring the second feature amount.

[0449] The processing system according to the 11th aspect is a processing system according to the 10th aspect, wherein the first model is a model according to the material of the first object, and the second model is a model according to the material of the second object.

[0450] The processing system according to the 12th aspect is a processing system according to the 10th or 11th aspect, wherein the first model is a model according to the state that the first object can take, and the second model is a model according to the state that the second object can take.

[0451] The processing system according to the 13th aspect is a processing system according to any one of the 1st to 7th aspects, wherein the first processing device has a first model for acquiring the first feature amount and a second model for acquiring the second feature amount of the first image, transmits the first feature amount and the second feature amount to the second processing device, the second processing device acquires the first restored image obtained by restoring the first image based on the first feature amount, acquires the second restored image obtained by restoring the first image based on the second feature amount, and performs processing using the first restored image and processing using the second restored image.

[0452] The processing system according to the 14th aspect is the processing system according to the 13th aspect, wherein a first object is depicted in the first image, the first model is a model corresponding to a first state that the first object can assume, and the second model is a model corresponding to a second state that the first object can assume.

[0453] The processing system according to the 15th aspect is the processing system according to any one of the 1st to 7th aspects, wherein the first processing device has a plurality of models capable of acquiring feature amounts of the first image in which a first object is depicted, and among the plurality of models, the first feature amount is acquired using a model corresponding to the surrounding environment of the first object in the first image, and the first feature amount is transmitted to the second processing device.

[0454] The processing system according to the 16th aspect is the processing system according to any one of the 1st to 15th aspects, wherein the first processing device acquires the first feature amount of the first image that is similar to the reference image among the first images in which the first object is depicted and acquired by the sensor a plurality of times for the first object at the same location, and transmits the acquired first feature amount to the second processing device.

[0455] The processing system according to the 17th aspect is the processing system according to the 16th aspect, wherein the first processing device is capable of changing the number of times the first image is acquired by the sensor.

[0456] The processing system according to the 18th aspect is the processing system according to any one of the 1st to 17th aspects, wherein the second processing device determines the state of the object depicted in the first image based on the first feature amount.

[0457] The processing system according to the 19th aspect is the processing system according to the 18th aspect, wherein the second processing device displays the determination result of the state of the object.

[0458] The processing system according to the 20th aspect is the processing system according to the 19th aspect, wherein the second processing device displays the determination result together with the first restored image.

[0459] The processing system according to the 21st embodiment is a processing system according to the 19th or 20th embodiment, wherein the second processing device divides the second feature quantities of a plurality of second images, each in which an object is depicted, into a plurality of groups according to the state of the object, the second processing device determines which of the plurality of groups the first feature quantity belongs to, and the state corresponding to the group to which the first feature quantity belongs is defined as the state of the object.

[0460] The processing system according to the 22nd embodiment is a processing system according to the 21st embodiment, wherein the second processing device displays the determination result of the state of the object together with the first restored image and a second image from among the plurality of second images that has a second feature similar to the first feature.

[0461] The processing system according to the 23rd embodiment is a processing system according to any one of the 18th to 22nd embodiments, wherein the object is a crop, and the second processing device determines the harvest time of the object based on the first feature quantity.

[0462] The processing system according to the 24th embodiment is a processing system according to the 23rd embodiment, wherein the second processing device associates a second feature quantity of each of a plurality of second images in which crops are depicted with the harvest time of the crops depicted in the second image, and the second processing device determines the harvest time of the object based on the harvest times corresponding to a plurality of similar feature quantities among the second feature quantities of the plurality of second images that are similar to the first feature quantity.

[0463] The processing system according to the 25th embodiment is a processing system according to any one of the 18th to 22nd embodiments, wherein the second processing device determines an abnormality of the object based on the first feature quantity.

[0464] The processing system according to the 26th embodiment is the processing system according to the 25th embodiment, wherein the second processing device determines the type of abnormality of the object based on the first feature quantity.

[0465] The second processing device according to the 27th embodiment is a second processing device provided in a processing system according to any one of the first to 26th embodiments.

[0466] The first processing device according to the 28th embodiment is a first processing device provided in a processing system according to any one of the seventh to seventeenth embodiments.

[0467] The program according to the 29th aspect is a program for causing a computer device to function as a second processing unit according to the 27th aspect.

[0468] The program according to the 30th aspect is a program for causing a computer device to function as a first processing unit according to the 28th aspect.

[0469] The processing method according to the 31st aspect involves receiving a first feature quantity of a first image, obtaining a first reconstructed image by reconstructing the first image based on the first feature quantity, and performing processing using the first reconstructed image.

[0470] 1 Processing System 3 Processing Unit (Server) 4 Processing Unit (Terminal Device) 5 Camera (Sensor) 101 Encoding Model 150, 900 Target Images 250, 250c, 250d, 250e, 250f, 250g, 250h, 760, 760a, 760b, 760c, 760d, 760e, 810, 810a, 810b, 810c Restored Image 250a Modified Restored Image 251 Crop Area (Target Area) 350 Harvest Time Determination Result 450 Similar Images 600 Notification Information 770 First Anomaly Determination Result 820 Second Anomaly Determination Result

Claims

1. A processing system comprising: a first processing unit that acquires a first feature quantity of a first image and transmits the first feature quantity; and a second processing unit that receives the first feature quantity from the first processing unit, acquires a first reconstructed image by reconstructing the first image based on the first feature quantity, and performs processing using the first reconstructed image.

2. The processing system according to claim 1, wherein the second processing device displays the first restored image.

3. A processing system according to claim 2, wherein the second processing device stores a plurality of second images and displays together the first reconstructed image and similar images from the plurality of second images that have a feature quantity similar to the first feature quantity.

4. A processing system according to claim 1, wherein the second processing device stores a plurality of second images, and when it determines that there are similar images having similar features to the first feature in the plurality of second images, it displays the similar images and the first reconstructed image together, and when it determines that there are no similar images in the plurality of second images, it obtains the first image from the first processing device and displays the obtained first image.

5. A processing system according to claim 3, wherein, if the similar image is a restored image obtained by the second processing device, the second processing device displays notification information indicating that the similar image is a restored image together with the similar image.

6. A processing system according to claim 5, wherein, if the similar image is a reconstructed image obtained by the second processing device, the second processing device displays, in response to instructions from the user, a second image from among the plurality of second images that has a feature quantity similar to the first feature quantity, and is not a reconstructed image obtained by the second processing device.

7. A processing system according to claim 2, wherein the second processing device transmits at least a portion of the first reconstructed image to the first processing device in response to instructions from a user; the first processing device acquires difference information indicating the difference between the at least portion and the first image, transmits the acquired difference information to the second processing device; and the second processing device modifies the first reconstructed image based on the difference information and displays the modified first reconstructed image.

8. A processing system according to claim 2, wherein the second processing unit instructs the first processing unit to change the model for acquiring the first feature quantity in response to instructions from a user; the first processing unit changes the model for acquiring the first feature quantity in response to instructions from the second processing unit to change the model, acquires the first feature quantity anew using the changed model, transmits the acquired new first feature quantity to the second processing unit; and the second processing unit acquires the first reconstructed image anew based on the new first feature quantity from the first processing unit, and displays the acquired new first reconstructed image.

9. A processing system according to claim 8, wherein the second processing device determines, based on the first reconstructed image, whether or not there is a scratch on the first object shown in the first reconstructed image, and if it determines that there is a scratch on the first object, instructs the first processing device to change the model for acquiring the first feature quantity in response to instructions from the user; the first processing device changes the model for acquiring the first feature quantity to a model corresponding to the first object having a scratch in response to the instruction from the second processing device to change the model; acquires the first feature quantity anew using the changed model; transmits the newly acquired first feature quantity to the second processing device; and the second processing device acquires the first reconstructed image anew based on the new first feature quantity from the first processing device and displays the newly acquired first reconstructed image.

10. A processing system according to claim 1, wherein the first processing device acquires a first feature quantity from a first image in which a first object is depicted and a second feature quantity from a second image in which a second object located at a different location from the first object is depicted, transmits the first feature quantity and the second feature quantity to the second processing device, the second processing device acquires a first restored image by restoring the first image based on the first feature quantity, acquires a second restored image by restoring the second image based on the second feature quantity, performs processing using the first restored image and processing using the second restored image, and the first processing device comprises a first model for acquiring the first feature quantity and a second model for acquiring the second feature quantity.

11. The processing system according to claim 10, wherein the first model is a model corresponding to the material of the first object, and the second model is a model corresponding to the material of the second object.

12. The processing system according to claim 10, wherein the first model is a model corresponding to a state that the first object can be in, and the second model is a model corresponding to a state that the second object can be in.

13. A processing system according to claim 1, wherein the first processing device comprises a first model for acquiring the first feature quantity and a second model for acquiring the second feature quantity of the first image, transmits the first feature quantity and the second feature quantity to the second processing device, and the second processing device acquires a first restored image obtained by restoring the first image based on the first feature quantity, acquires a second restored image obtained by restoring the first image based on the second feature quantity, and performs processing using the first restored image and processing using the second restored image.

14. The processing system according to claim 13, wherein the first image shows a first object, the first model is a model corresponding to a first state that the first object can be in, and the second model is a model corresponding to a second state that the first object can be in.

15. A processing system according to claim 1, wherein the first processing device has a plurality of models capable of acquiring feature quantities of the first image in which the first object is depicted, and the processing system acquires the first feature quantities using a model from the plurality of models that corresponds to the surrounding environment of the first object in the first image, and transmits the first feature quantities to the second processing device.

16. A processing system according to claim 1, wherein the first processing device acquires a first feature quantity of the first image that is similar to a reference image from among the first images of the first object captured by a sensor multiple times for the first object at the same location, and transmits the acquired first feature quantity to the second processing device.

17. A processing system according to claim 16, wherein the first processing device is capable of changing the number of times the first image is acquired by the sensor.

18. A processing system according to claim 1, wherein the second processing device determines the state of an object captured in the first image based on the first feature quantity.

19. A processing system according to claim 18, wherein the second processing device displays the result of determining the state of the object.

20. A processing system according to claim 19, wherein the second processing device displays the determination result together with the first restored image.

21. A processing system according to claim 19, wherein the second processing device divides the second feature quantities of a plurality of second images, each in which an object is depicted, into a plurality of groups according to the state of the object, and the second processing device determines which of the plurality of groups the first feature quantity belongs to, and sets the state corresponding to the group to which the first feature quantity belongs as the state of the object.

22. A processing system according to claim 21, wherein the second processing device displays the determination result of the state of the object together with the first reconstructed image and a second image from among the plurality of second images having a second feature similar to the first feature.

23. A processing system according to claim 18, wherein the object is a crop, and the second processing device determines the harvest time of the object based on the first characteristic quantity.

24. A processing system according to claim 23, wherein the second processing device associates a second feature quantity of a second image with the harvest time of the crops depicted in the second image, and the second processing device determines the harvest time of the object based on the harvest times corresponding to a plurality of similar feature quantities among the second feature quantities of the plurality of second images that are similar to the first feature quantity.

25. A processing system according to claim 18, wherein the second processing device determines an abnormality of the object based on the first feature quantity.

26. A processing system according to claim 25, wherein the second processing device determines the type of abnormality of the object based on the first feature quantity.

27. A second processing device comprising the processing system according to any one of claims 1 to 26.

28. A first processing apparatus comprising the processing system according to any one of claims 7 to 17.

29. A program for causing a computer device to function as the second processing unit described in claim 27.

30. A program for causing a computer device to function as the first processing unit described in claim 28.

31. A processing method comprising receiving a first feature quantity of a first image, obtaining a first reconstructed image by reconstructing the first image based on the first feature quantity, and performing processing using the first reconstructed image.

Images