Training device, inference device, training program, and inference program

Abstract

A training device (100) has: an acquisition unit (120) that acquires a plurality of images including a target object, and acquires vibration information corresponding to each of the plurality of images; and a training unit (140) that generates a trained model using the plurality of images and the plurality of items of vibration information corresponding to the plurality of images.

Description

Learning device, estimation device, learning program, and estimation program

[0001] This disclosure relates to a learning device, an estimation device, a learning program, and an estimation program.

[0002] Structures may vibrate. Techniques for identifying structural vibrations have been proposed (see Patent Document 1). The analysis method in Patent Document 1 involves capturing a series of images of the object under inspection, generating a dominant frequency image based on the series of images, analyzing the dominant frequency image, and identifying the natural vibration and amplitude of the object under inspection.

[0003] Patent No. 6675929

[0004] When vibrations are identified, it is necessary to estimate the resources required. However, various types of structures exist. Therefore, the amount of computation in image processing algorithms is not constant due to branching and looping. For this reason, estimating resources is difficult.

[0005] The purpose of this disclosure is to facilitate resource estimation.

[0006] A learning device according to one aspect of the present disclosure is provided. The learning device includes an acquisition unit that acquires a plurality of images including an object and acquires vibration information corresponding to each of the plurality of images, and a learning unit that generates a trained model using the plurality of images and the plurality of vibration information corresponding to the plurality of images.

[0007] This disclosure makes it possible to easily estimate resources.

[0008] This diagram shows the learning device and estimation device of Embodiment 1. This diagram shows the hardware of the learning device of Embodiment 1. This is a block diagram showing the functions of the learning device of Embodiment 1. This diagram shows a specific example of generating learning data of Embodiment 1. This diagram shows the hardware of the estimation device of Embodiment 1. This is a block diagram showing the functions of the estimation device of Embodiment 1. This is a block diagram showing the functions of the learning device of Embodiment 2. This diagram shows a specific example of generating learning data of Embodiment 2. This is a block diagram showing the functions of the estimation device of Embodiment 2.

[0009] The embodiments will be described below with reference to the drawings.

[0010] Embodiment 1. Figure 1 shows the learning device and estimation device of Embodiment 1. The learning device 100 and the estimation device 200 may be connected via a wired network or a wireless network. The learning device 100 generates a trained model. The learning device 100 may transmit the trained model to the estimation device 200. The estimation device 200 performs estimation using the trained model. Here, the learning device 100 and the estimation device 200 may be the same device.

[0011] As described above, the learning device 100 generates a trained model. First, we will explain how the trained model is generated. After explaining how the trained model is generated, we will explain the estimation device 200.

[0012] <Learning Phase> The hardware of the learning device 100 will be described. Figure 2 is a diagram showing the hardware of the learning device of Embodiment 1. The learning device 100 is also called a computer. The learning device 100 has a processor 101, a volatile memory device 102, and a non-volatile memory device 103.

[0013] The processor 101 controls the entire learning device 100. For example, the processor 101 may be a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), etc. The processor 101 may also be a multiprocessor. A multiprocessor may be a combination of a CPU and an FPGA, a CPU and a GPU (Graphics Processing Unit), a CPU and a specially designed processing circuit, or a combination of these. The learning device 100 may also have a processing circuit. Both the processor and the processing circuit may be composed of electrical circuits.

[0014] The volatile memory device 102 is the main memory of the learning device 100. For example, the volatile memory device 102 is RAM (Random Access Memory). The non-volatile memory device 103 is the auxiliary memory of the learning device 100. For example, the non-volatile memory device 103 is an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0015] Next, the functions of the learning device 100 will be described. Figure 3 is a block diagram showing the functions of the learning device according to Embodiment 1. The learning device 100 has a storage unit 110, an acquisition unit 120, a clipping unit 130, a learning unit 140, and an output unit 150.

[0016] The storage unit 110 may be implemented as a storage area reserved in a volatile storage device 102 or a non-volatile storage device 103. The storage unit 110 may also be called a memory. Some or all of the acquisition unit 120, the clipping unit 130, the learning unit 140, and the output unit 150 may be implemented by processing circuits. Also, some or all of the acquisition unit 120, the clipping unit 130, the learning unit 140, and the output unit 150 may be implemented as modules of a program executed by the processor 101. For example, the program executed by the processor 101 is also called a learning program or a learning program product. For example, the learning program is recorded on a recording medium.

[0017] The memory unit 110 stores various types of information.

[0018] Here, the generation of training data will be explained using a specific example. Figure 4 is a diagram showing a specific example of the generation of training data in Embodiment 1. The acquisition unit 120 acquires multiple images including the target object. For example, the acquisition unit 120 acquires the multiple images from the storage unit 110. Alternatively, for example, the acquisition unit 120 acquires the multiple images from an external device. The external device is a device located outside the learning device 100. For example, the external device may be a cloud server or external memory. The diagram of the external device is omitted. Alternatively, for example, the acquisition unit 120 acquires the multiple images from the camera that generated the multiple images. The camera is also an example of an external device.

[0019] The object in question is described below. Preferably, the object is one whose vibration cannot be easily measured. In other words, the object can be described as follows: When acceleration sensors or vibration sensors are installed on the object to measure its vibration, the installation of numerous sensors is required. Therefore, the object is one in which the vibration of the entire structure cannot be easily measured unless numerous sensors are installed. For example, the object may be a bridge, a transmission tower, or a high-rise building. Furthermore, the objects included in the image may also include figures, symbols, and characters prepared as a test chart.

[0020] The following describes the multiple images. These multiple images may be obtained by photographing the object in succession. These multiple images may also be obtained without photographing the object in succession. These multiple images may also be images using a test chart that shows the object. The test chart includes white and black areas. The test chart may also be a white, black, gray, and color image. These multiple images may be generated as follows: An exciter is attached to an image containing the object. Multiple images are generated by photographing the vibrating image.

[0021] Figure 4 shows images 10_1, ..., 10_n. Images 10_1, ..., 10_n are examples of multiple images. Images 10_1, ..., 10_n are images obtained by taking consecutive photographs of the bridge. That is, images 10_1, ..., 10_n are taken at time t １ ~t ｎ This is an image generated during the specified time period. Note that n is a positive integer.

[0022] The cropping unit 130 crops out a region representing a part of the object from each of the multiple images. The cropping unit 130 also crops out the same region from each of the multiple images. The cropped images are called cropped images. The part of the object shown in each cropped image is the same part. For example, if the bridge is a suspension bridge, the part of the object is the tip of the suspension bridge tower. It is desirable that the part of the object represents the swaying caused by vibration. Figure 4 shows cropped images 11_1, ..., 11_n. The acquisition unit 120 acquires the multiple cropped images from the cropping unit 130.

[0023] The acquisition unit 120 acquires vibration information corresponding to each of the multiple cropped images from the storage unit 110 or an external device. The vibration information is information related to vibration. For example, the vibration information includes frequency, amplitude, phase, and information indicating whether or not the camera that photographed the object was shaking. For example, the frequency and amplitude are values ​​that have been measured in advance. Figure 4 shows the vibration information ft １ , ..., ft ｎ This is shown. Furthermore, if the vibration information has a temporal structure (e.g., periodicity, etc.), then the ft in Figure 4... １ , ..., ft ｎ Instead of providing multiple values ​​as shown above, you can use a single value to represent the vibration information. For example, the vibration information could be a single frequency value.

[0024] Furthermore, if multiple images acquired by the acquisition unit 120 are generated by simulation, the vibration information is the vibration information set in the simulation. Also, if each of the multiple images is obtained by taking an image with the vibrator attached, the vibration information is the vibration information set for the vibrator.

[0025] The learning unit 140 generates a trained model using multiple cropped images and vibration information corresponding to the cropped images. The trained model generated here is a model for estimating the vibration information of an object from cropped images showing a part of the object. The generation process will be explained in detail. The learning unit 140 uses cropped image 11_1 and vibration information ft １are used as learning data. The learning unit 140 uses the learning data to train a model. This training may be referred to as supervised learning. Similarly, the learning unit 140 uses the clipped images 11_2,..., 11_n and the vibration information ft ２ ,..., ft ｎ as learning data. Then, the learning unit 140 uses the learning data to train a model. When the training is completed, a trained model is generated. When a clipped image showing a part of the object is input to the trained model, the vibration information of the object is output.

[0026] The output unit 150 outputs the trained model. For example, the output unit 150 outputs the trained model to the estimation device 200 or an external device. Also, for example, the output unit 150 outputs the trained model to the storage unit 110. The trained model stored in the storage unit 110 may be stored in a recording medium. The user may store the trained model stored in the recording medium in the estimation device 200.

[0027] According to the training phase, a trained model is prepared. By preparing the trained model, it becomes easy to estimate the resources for calculating the vibration information of the object. This is because, unlike the case of using an image processing algorithm, by using the trained model, the occurrence of unspecified branching processes and iterative processes can be suppressed, and the amount of calculation in the trained model does not vary significantly.

[0028] In the above, the case where the clipping unit 130 clips a region showing a part of the object from an image has been described. The learning device 100 may not have the clipping unit 130. When the learning device 100 does not have the clipping unit 130, an external device acquires a plurality of images including the object. The external device clips a region showing a part of the object from each of the plurality of images. Then, the acquisition unit 120 acquires the plurality of clipped images from the external device.

[0029] Furthermore, the above describes the case where the training data consists of cropped images and vibration information. The training data may also consist of images containing the object and vibration information. When the images and vibration information are the training data, the acquisition unit 120 acquires vibration information corresponding to each of the multiple images containing the object from the storage unit 110 or an external device. The learning unit 140 generates a trained model using the multiple images and the vibration information corresponding to the images. When an image containing the object is input to the trained model, it outputs the vibration information of the object. When the images containing the object and vibration information are the training data, the learning device 100 does not have a cropping unit 130.

[0030] When the learning device 100 uses cropped images and vibration information as training data, the cropped images can have a smaller data size than the original images, thus reducing the computational cost of training when generating a trained model. On the other hand, when the learning device 100 uses images containing the object that is the source of the cropped images and vibration information as training data, the images containing the object have a larger amount of information, allowing it to generate trained models for estimating vibration information from a variety of images, thus increasing versatility.

[0031] The learning unit 140 may include image interval information in the learning data when multiple images acquired by the acquisition unit 120 are generated. The image interval information can be any information that shows the time interval between consecutive images. For example, the image interval information may be information about the time between images, or information indicating how many images were acquired per second. For example, the frame rate can be used as the image interval information. A specific example will be given when the frame rate is used as the image interval information. The frame rate when images 10_1, ..., 10_n are generated is called the first frame rate. The first frame rate is 60 fps. The learning unit 140 uses 60 fps, the cropped image 11_2, and vibration information ft ２The learning unit 140 generates learning data including the following. The learning unit 140 then performs learning using this learning data. The learning unit 140 may also generate learning data that includes a second frame rate which is a different frame rate from the first frame rate. The second frame rate is 30 fps. The learning unit 140 uses 30 fps, the cropped image 11_2, and vibration information ft １ The learning unit 140 generates learning data including the frame rate, cropped image, and vibration information, and performs learning using this learning data. Furthermore, the learning unit 140 may generate learning data including the frame rate, image, and vibration information. A specific example will be explained. The learning unit 140 generates learning data including 60 fps, image 10_2, and vibration information ft ２ The learning unit 140 generates learning data including the following. The learning unit 140 then performs learning using the learning data. The learning unit 140 uses 30fps, image 10_2, and vibration information ft １ The learning unit 140 generates training data that includes the frame rate, images, and vibration information, and then performs training using this training data. In other words, the learning unit 140 can generate a trained model for each image interval information. Furthermore, the learning unit 140 can also generate a trained model that supports both 30fps and 60fps (i.e., multiple image interval information) by training with both 30fps and 60fps image interval information and the images corresponding to each image interval information during training. The learning unit 140 further generates a trained model using the image interval information. As a result, when the input image and image have different image interval information than the image interval information used to generate the trained model, the learning unit 140 can generate a trained model that can estimate vibration information more accurately by converting the vibration information output based on the image interval information.

[0032] Next, the utilization phase will be explained. <Utilization Phase> The hardware of the estimation device 200 will be explained. Figure 5 is a diagram showing the hardware of the estimation device of Embodiment 1. The estimation device 200 is also called a computer. The estimation device 200 has a processor 201, a volatile memory device 202, and a non-volatile memory device 203.

[0033] The processor 201 controls the entire estimation device 200. For example, the processor 201 may be a CPU, FPGA, etc. The processor 201 may also be a multiprocessor. The multiprocessor may be a combination of a CPU and FPGA, a CPU and a GPU, a CPU and a specially designed processing circuit, or a combination of these. The estimation device 200 may also have a processing circuit. Both the processor and the processing circuit may be composed of electrical circuits.

[0034] The volatile memory device 202 is the main memory of the estimation device 200. For example, the volatile memory device 202 is RAM. The non-volatile memory device 203 is the auxiliary memory of the estimation device 200. For example, the non-volatile memory device 203 is an HDD or SSD.

[0035] Next, the functions of the estimation device 200 will be described. Figure 6 is a block diagram showing the functions of the estimation device of Embodiment 1. The estimation device 200 has a storage unit 210, an acquisition unit 220, a clipping unit 230, an estimation unit 240, and an output unit 250.

[0036] The storage unit 210 may be implemented as a storage area reserved in a volatile storage device 202 or a non-volatile storage device 203. Some or all of the acquisition unit 220, the clipping unit 230, the estimation unit 240, and the output unit 250 may be implemented by processing circuits. Also, some or all of the acquisition unit 220, the clipping unit 230, the estimation unit 240, and the output unit 250 may be implemented as modules of a program executed by the processor 201. For example, the program executed by the processor 201 is also called an estimation program or estimation program product. For example, the estimation program is recorded on a recording medium.

[0037] The storage unit 210 stores various information. For example, the storage unit 210 stores the learned model.

[0038] The acquisition unit 220 acquires an image including an object. For example, the acquisition unit 220 acquires the image from the storage unit 210. Also, for example, the acquisition unit 220 acquires the image from an external device. Here, the external device is a device existing outside the estimation device 200. For example, the external device is a cloud server, an external memory, etc. The figure of the external device is omitted. The acquisition unit 220 may acquire the image from the camera that generated the image. Note that the camera is also an example of an external device.

[0039] The object included in the image may be the same object as the object learned in the learning phase. For example, when the object learned in the learning phase is a bridge, the object included in the image is a bridge. Note that it is particularly preferable that the object included in the image used in the utilization phase is an object for which vibration cannot be easily measured by installing an acceleration sensor or a vibration sensor on the object. For example, it is particularly preferable that the object is a structure such as a bridge, a铁塔 (iron tower), or a high-rise building.

[0040] The acquisition unit 220 acquires the learned model generated by the learning device 100 from the storage unit 210 or an external device.

[0041] The acquisition unit 220 acquires information indicating the cutout area from the storage unit 210 or an external device. Also, the acquisition unit 220 may acquire information indicating the cutout area from the learning device 100. The information indicating the cutout area is information indicating the location where the learning device 100 performed the cutout process. For example, when the bridge is a suspension bridge and the location where the learning device 100 performed the cutout process is the tip part of the tower, the information indicating the cutout area is information indicating the tip part of the tower.

[0042] The cutout unit 230 cuts out an area indicating a part of the object from the image based on the information indicating the cutout area. The acquisition unit 220 acquires the cutout image from the cutout unit 230.

[0043] The estimation unit 240 uses the cropped image and the trained model to estimate the vibration information of the object represented by the cropped image. Specifically, when the estimation unit 240 inputs the cropped image to the trained model, the trained model outputs the vibration information.

[0044] The output unit 250 outputs vibration information. For example, the output unit 250 outputs vibration information to the display of the estimation device 200. In this way, the estimation device 200 can estimate the vibration information of the object.

[0045] In the application phase, the estimation device 200 can perform estimations using resources that have been easily estimated by using a pre-trained model.

[0046] The above describes the case where the cropping unit 230 crops an area from the image that represents a part of the object. The estimation device 200 does not necessarily have a cropping unit 230. If the estimation device 200 does not have a cropping unit 230, the external device acquires an image that includes the object. The external device crops an area from the image that represents a part of the object. Then, the acquisition unit 220 acquires the cropped image from the external device.

[0047] Furthermore, the above describes the case in which vibration information is estimated using a cropped image. The estimation device 200 may also estimate vibration information using an image that includes the object. Specifically, the estimation unit 240 estimates the vibration information of the object using the image and a trained model. Specifically, when the estimation unit 240 inputs the image to the trained model, the trained model outputs vibration information. When estimation is performed using the image, the estimation device 200 does not have a cropping unit 230.

[0048] The acquisition unit 220 may acquire image interval information from the storage unit 210 or an external device when an image including the object is generated. The image interval information is the same as the image interval information used in the learning phase described above. A specific example will be given when the frame rate is used as the image interval information. The estimation unit 240 estimates the vibration information of the object using the cropped image or the image including the object, the trained model, and the frame rate. In detail, when the estimation unit 240 inputs the cropped image or the image including the object and the frame rate to the trained model, the trained model outputs the vibration information. When the frame rate as image interval information is input to the trained model, the estimation unit 240 compares the image interval information when the trained model was generated with the input image interval information when estimating the vibration information, and outputs the vibration information as is if they match. On the other hand, the estimation unit 240 compares the image interval information when the trained model was generated with the input image interval information, and if they are different, converts the vibration information based on the image interval information and outputs it. For example, if the frame rate at which the trained model was generated was 60 fps and the input frame rate is 30 fps, the estimation unit 240 will convert the vibration information to twice or half its original value based on the respective frame rates because the frame rates are different, and then output it. In this way, the estimation device 200 can estimate vibration information according to the frame rate, enabling more accurate estimation of vibration information. Of course, if there are multiple trained models for each of the first and second frame rates mentioned above, the estimation unit 240 compares the image interval information at the time the trained model was generated with the input image interval information for each trained model. If there is a trained model with matching image interval information, the estimation unit 240 can output the vibration information as is using that trained model. In other words, the estimation unit 240 can convert and output vibration information in any trained model if the image interval information at the time the trained model was generated does not match the input image interval information.Furthermore, the estimation unit 240 compares the image interval information generated when the trained model was created with the input image interval information. If they differ, and the image interval information is linear, such as the frame rate, the estimation unit 240 can use this linearity to estimate the vibration information. For example, if the frame rate generated when the trained model was created was 30 fps and the input frame rate is 60 fps, the estimation unit 240 can also estimate the vibration information using images captured at a frame rate of 60 fps. It should be noted that this type of linearity-based estimation requires interpolation of the estimation results while also considering linearity, resulting in lower estimation resolution, but it is still capable of sufficiently estimating vibration information. Needless to say, the inference unit 240 can arbitrarily change the vibration information estimation operation based on user requests, predefined commands, etc.

[0049] Embodiment 2. Next, Embodiment 2 will be described. Embodiment 2 will mainly describe the differences from Embodiment 1. In Embodiment 2, the explanation of matters common to Embodiment 1 will be omitted.

[0050] <Learning Phase> Figure 7 is a block diagram showing the functions of the learning device of Embodiment 2. The learning device 100 further includes a generation unit 160. Part or all of the generation unit 160 may be implemented by a processing circuit. Alternatively, part or all of the generation unit 160 may be implemented as a module of a program executed by the processor 101. The functions of the generation unit 160 will be described later.

[0051] The generation of training data will be explained using a specific example. Figure 8 shows a specific example of the generation of training data in Embodiment 2. Figure 8 shows images 10_1, ..., 10_n acquired by the acquisition unit 120.

[0052] The generation unit 160 generates multiple difference images using multiple images. The generation process will be explained in detail. A reference image is set. For example, the reference image is image 10_1. The generation unit 160 compares image 10_1 with images 10_2, ..., 10_n and generates difference images. For example, the generation unit 160 compares image 10_1 and image 10_2 to generate difference image 12_1. For example, the generation unit 160 compares image 10_1 and image 10_3 to generate difference image 12_2. For example, the generation unit 160 compares image 10_1 and image 10_n to generate difference image 12_n. The generated difference images can be said to be images that show the amount of movement of the object shown in the first image 10_1 which was used as the reference image. Note that the reference image does not have to be the first image. For example, the generation unit 160 may use image 10_n as a reference image and generate a difference image between image 10_n and other images (i.e., 10_1, ..., image 10_n).

[0053] The cropping unit 130 crops out a region representing a part of the object from each of the multiple difference images. The cropped image is called a cropped difference image. Figure 8 shows the cropped difference images 13_1, ..., 13_n. The acquisition unit 120 acquires the multiple cropped difference images from the cropping unit 130. Note that the multiple cropped difference images are expressed as multiple images obtained by performing a cropping process on multiple difference images generated using multiple images.

[0054] The acquisition unit 120 acquires vibration information corresponding to each of the multiple cropped difference images from the storage unit 110 or an external device.

[0055] The learning unit 140 generates a trained model using multiple cropped difference images and vibration information corresponding to the cropped difference images. The generation process is the same as in Embodiment 1, so the explanation is omitted. When a cropped difference image is input to the trained model, it outputs vibration information. In this way, the learning device 100 can generate a trained model using cropped images based on difference images as training data.

[0056] The above describes the case where the generation unit 160 generates difference images. The learning device 100 does not necessarily have a generation unit 160. If the learning device 100 does not have a generation unit 160, the external device acquires multiple images including the target object. The external device generates multiple difference images using the multiple images. The acquisition unit 120 then acquires the multiple difference images from the external device. Also, if the learning device 100 does not have a cropping unit 130, the external device crops out a region showing a part of the target object from each of the multiple difference images. The acquisition unit 120 then acquires the multiple cropped difference images from the external device.

[0057] Furthermore, the above describes the case where the training data consists of cropped difference images and vibration information. The training data may also consist of difference images and vibration information. When difference images and vibration information are the training data, the acquisition unit 120 acquires vibration information corresponding to each of the multiple difference images from the storage unit 110 or an external device. The learning unit 140 generates a trained model using the multiple difference images and the vibration information corresponding to the difference images. When a difference image is input to the trained model, it outputs vibration information. When difference images and vibration information are the training data, the learning device 100 does not have a cropping unit 130.

[0058] The learning device 100 uses difference images as training data, so that it uses images in which the parts that have changed due to the vibration of the object remain. This allows for a larger identification of the difference due to vibration, making it easier to determine the vibration period of the object. In other words, the image with the largest change can be identified as the maximum amplitude of the object, and the image where the difference has disappeared can be identified as half a period, one period, etc. As a result, the learning device 100 can generate a trained model that enables more accurate estimation of vibration.

[0059] The above describes the case where the cropping process is performed after the difference image is generated. However, the generation unit 160 may also generate multiple cropped difference images using multiple cropped images after the cropping process has been performed. When difference images are generated from cropped images, the amount of computation is reduced. Note that multiple cropped difference images are expressed as multiple images obtained by performing a comparison process on multiple cropped images.

[0060] <Utilization Phase> Figure 9 is a block diagram showing the functions of the estimation device of Embodiment 2. The estimation device 200 further includes a generation unit 260. Part or all of the generation unit 260 may be implemented by a processing circuit. Alternatively, part or all of the generation unit 260 may be implemented as a module of a program executed by the processor 201. The functions of the generation unit 260 will be described later.

[0061] The acquisition unit 220 acquires multiple images, including the target object, from the storage unit 210 or an external device.

[0062] The generation unit 260 selects two images from multiple images. If there are only two images, the generation unit 260 does not perform a selection. The generation unit 260 generates a difference image using the two selected images. When generating the difference image, the generation unit 260 generates the difference image in the same way that the generation unit 160 generated the difference image. For example, if the generation unit 160 generates the difference image based on a reference image, the generation unit 260 generates the difference image based on the reference image. For example, the acquisition unit 220 acquires information from the learning device 100 indicating how the generation unit 160 generated the difference image. The generation unit 260 then generates the difference image based on this information. This allows the generation unit 260 to generate the difference image in the same way that the generation unit 160 generated the difference image. The acquisition unit 220 acquires the difference image from the generation unit 260.

[0063] The acquisition unit 220 acquires information indicating the cut-out area from the storage unit 210 or an external device. Alternatively, the acquisition unit 220 may acquire information indicating the cut-out area from the learning device 100. The information indicating the cut-out area is information indicating the location where the learning device 100 has performed the cut-out process.

[0064] The cropping unit 230 crops out a region representing a part of the object from the difference image based on information indicating the cropping area. The acquisition unit 220 acquires the cropped difference image from the cropping unit 230. The cropped difference image is described as an image obtained by performing a cropping process on a difference image generated using multiple images including the object.

[0065] The estimation unit 240 estimates the vibration information of the object using the cropped difference image and the trained model. Specifically, when the estimation unit 240 inputs the cropped difference image to the trained model, the trained model outputs the vibration information. In this way, the estimation device 200 can estimate the vibration information of the object using the cropped difference image.

[0066] The above describes the case where the generation unit 260 generates a difference image. The estimation device 200 does not necessarily have a generation unit 260. If the estimation device 200 does not have a generation unit 260, the external device acquires multiple images including the target object. The external device also acquires information from the learning device 100 indicating how the generation unit 160 generated the difference image. The external device selects two images from the multiple images and generates a difference image using this information and the two images. The acquisition unit 220 then acquires the difference image from the external device. If the estimation device 200 does not have a cropping unit 230, the external device crops a region showing a part of the target object from the difference image. The acquisition unit 220 then acquires the cropped difference image from the external device.

[0067] Furthermore, the above describes the case in which vibration information is estimated using a cropped difference image. The estimation device 200 may also estimate vibration information using a difference image. In detail, the estimation unit 240 estimates the vibration information of the object using the difference image and a trained model. Specifically, when the estimation unit 240 inputs the difference image to the trained model, the trained model outputs vibration information. When estimation is performed using a difference image, the estimation device 200 does not have a cropping unit 230.

[0068] The estimation device 200 uses difference images, which means that the image retains the portion that has changed due to the vibration of the object. This allows for a larger identification of the difference due to vibration, making it easier to determine the vibration period of the object. In other words, the image with the largest change can be identified as the maximum amplitude of the object, and the image where the difference from the reference image has disappeared can be identified as half a period, one period, etc. This enables the estimation device 200 to estimate vibrations with greater accuracy.

[0069] The above describes a case where the cropping process is performed after the difference image is generated. However, the generation unit 260 may also generate a cropped difference image using multiple cropped images after the cropping process has been performed. The cropped difference image is described as an image obtained by performing a comparison process on multiple cropped images.

[0070] Each embodiment is merely illustrative, and various modifications are possible within the scope of this disclosure. Furthermore, the features of each embodiment can be combined with each other as appropriate.

[0071] 10_1, ..., 10_n Images, 11_1, ..., 11_n Cropped images, 12_1, ..., 12_n Difference images, 13_1, ..., 13_n Cropped difference images, 100 Learning device, 101 Processor, 102 Volatile memory device, 103 Non-volatile memory device, 110 Storage unit, 120 Acquisition unit, 130 Cropping unit, 140 Learning unit, 150 Output unit, 160 Generation unit, 200 Estimation device, 201 Processor, 202 Volatile memory device, 203 Non-volatile memory device, 210 Storage unit, 220 Acquisition unit, 230 Cropping unit, 240 Estimation unit, 250 Output unit, 260 Generation unit.

Claims

1. A learning device comprising: an acquisition unit that acquires multiple images including an object and acquires vibration information corresponding to each of the multiple images; and a learning unit that generates a trained model using the multiple images and the vibration information corresponding to the multiple images.

2. The learning device according to claim 1, wherein the acquisition unit acquires a plurality of difference images generated using the plurality of images, and acquires vibration information corresponding to each of the plurality of difference images, and the learning unit generates a trained model using the plurality of difference images and the vibration information corresponding to the plurality of difference images.

3. The learning device according to claim 1, wherein the acquisition unit acquires a plurality of cropped images showing a part of the object, acquires vibration information corresponding to each of the plurality of cropped images, and the learning unit generates a trained model using the plurality of cropped images and the vibration information corresponding to the plurality of cropped images.

4. The learning device according to claim 3, wherein the acquisition unit acquires a plurality of cropped difference images obtained by performing a comparison process on the plurality of cropped images, or a plurality of cropped difference images obtained by performing a cropping process on a plurality of difference images generated using the plurality of images, and acquires vibration information corresponding to each of the plurality of cropped difference images, and the learning unit generates a trained model using the plurality of cropped difference images and the vibration information corresponding to the plurality of cropped difference images.

5. The learning device according to any one of claims 1 to 4, wherein the acquisition unit acquires image interval information when the plurality of images are generated, and the learning unit further uses the image interval information to generate a trained model.

6. An estimation device comprising: an acquisition unit that acquires an image including an object and a trained model; an estimation unit that estimates vibration information of the object using the image and the trained model; and an output unit that outputs the vibration information.

7. The estimation device according to claim 6, wherein the acquisition unit acquires image interval information when the image is generated, and the estimation unit estimates the vibration information of the object using the image, the trained model, and the image interval information.

8. The estimation device according to claim 6, wherein the acquisition unit acquires a difference image generated using a plurality of images including the object, and the estimation unit estimates the vibration information of the object using the difference image and the trained model.

9. The estimation device according to claim 6, wherein the acquisition unit acquires a cropped image showing a part of the object, and the estimation unit estimates vibration information of the object using the cropped image and the trained model.

10. The estimation device according to claim 9, wherein the acquisition unit acquires image interval information when generating the image including the object, and the estimation unit estimates vibration information of the object using the cropped image, the trained model, and the image interval information.

11. The estimation apparatus according to claim 9, wherein the acquisition unit acquires a cropped difference image obtained by performing a comparison process on a plurality of cropped images, or a cropped difference image obtained by performing a cropping process on a difference image generated using a plurality of images including the object, and the estimation unit estimates the vibration information of the object using the cropped difference image and the trained model.

12. A learning program that causes a learning device to execute a process that acquires multiple images including an object, acquires vibration information corresponding to each of the multiple images, and generates a trained model using the multiple images and the vibration information corresponding to the multiple images.

13. An estimation program that causes an estimation device to perform the following processes: acquire an image containing an object and a trained model; estimate the vibration information of the object using the image and the trained model; and output the vibration information.

Images