Information processing device

Abstract

An information processing device 10 comprises a control unit 11 that: acquires a captured image 31 obtained by capturing a structure that is a target facility; detects a region of corrosion present in the structure from the acquired captured image 31; inputs the captured image 31, the detected region of corrosion, environmental data related to environmental factors that may affect progress of corrosion in the structure, and a desired prediction time to a trained model, to acquire a prediction image 33 indicating a region of corrosion predicted to be present in the structure at the prediction time; and outputs the acquired prediction image 33.

Description

Information Processing Apparatus

[0001] The present disclosure relates to an information processing apparatus.

[0002] Numerous steel-structured infrastructure facilities such as road bridges, steel towers, and guardrails have been installed, and the aging of these infrastructure facilities has become a major social problem. The main deterioration factor that promotes the aging of these facilities is corrosion of steel materials. Therefore, facility managers check whether corrosion occurs and its progress status through periodic facility inspections.

[0003] However, the number of facilities is enormous, and corrosion progresses year by year, so inspections require a large amount of time and cost. In addition, it is desirable to inspect facilities with rapidly progressing corrosion at shorter intervals, but at present, it is difficult to accurately predict the future state of the facilities. For this reason, it is common to inspect all facilities at a uniform interval.

[0004] As described above, in order to reduce the work cost of facility inspection and realize safe maintenance and management in accordance with the progress of deterioration, it is necessary to set an inspection interval for each facility in accordance with the progress of corrosion instead of a uniform inspection interval.

[0005] Non-Patent Document 1 describes a model that applies supervised machine learning to corrosion data and environmental data of structures to predict a corrosive environment from environmental data.

[0006] Hideki Katayama, 2 other authors, "Corrosion Prediction from Environmental Data by Machine Learning", [online], Surface Technology, Vol. 71, No. 2, p. 193, [retrieved March 1, 2025], Internet <URL: https: / / doi.org / 10.4139 / sfj.71.193>

[0007] However, the configuration of Non-Patent Document 1 has room for improvement in terms of predicting the progress of corrosion with high accuracy according to the environment of each facility. As a result, the conventional configuration could not determine the difference in progress due to the difference in installation location of each road bridge within the same area, and it was difficult to determine an appropriate inspection interval for each facility.

[0008] An object of the present disclosure is to predict the progress of corrosion for each facility with higher accuracy.

[0009] According to this disclosure, the information processing device includes a control unit that acquires a photographic image of a structure which is a target facility, detects areas of corrosion present in the structure from the acquired photographic image, inputs the photographic image, the detected areas of corrosion, environmental data relating to environmental factors that may affect the progression of corrosion in the structure, and a desired prediction time into a trained model to acquire a prediction image showing the areas of corrosion that are predicted to be present in the structure at the prediction time, and outputs the acquired prediction image.

[0010] According to one embodiment of this disclosure, the progression of corrosion for each piece of equipment can be predicted with greater accuracy.

[0011] This is a block diagram showing an example of the hardware configuration of an information processing device according to one embodiment. This is a block diagram showing an example of the functional configuration of an information processing device according to one embodiment. This is a diagram showing an example of a captured image input to the image input unit. This is a block diagram showing an example of the functional configuration of the corrosion detection unit 22. This is a diagram illustrating the learning of the equipment detector. This is a diagram illustrating the learning of the corrosion detector. This is a block diagram showing an example of the functional configuration of the generation unit. This is a diagram illustrating the learning of the generator. This is a flowchart showing an example of the operation of an information processing device. This is a flowchart showing an example of the operation of an information processing device.

[0012] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, parts having the same configuration or function are denoted by the same reference numerals. In the description of this embodiment, redundant descriptions of the same parts may be omitted or simplified as appropriate.

[0013] <Hardware Configuration of Information Processing Device 10> Figure 1 is a block diagram showing an example of the hardware configuration of an information processing device 10 according to one embodiment. The information processing device 10 outputs the result of predicting the progression of corrosion at a desired prediction time from captured images of a structure which is the target equipment. The structure is, for example, a steel structure, but is not limited to these.

[0014] The information processing device 10 is one or a plurality of computer devices capable of communicating with each other. Specifically, the information processing device 10 may be any general-purpose electronic device such as a WS (Workstation), PC (Personal Computer), or tablet terminal, or it may be another dedicated electronic device. As shown in Figure 1, the information processing device 10 comprises a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15.

[0015] The control unit 11 includes one or more processors. In one embodiment, the "processor" is a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited to these. The control unit 11 is communicatively connected to each component constituting the information processing device 10 and controls the operation of the entire information processing device 10.

[0016] The storage unit 12 includes, for example, any storage module such as an HDD (Hard Disk Drive), SSD (Solid State Drive), ROM (Read-Only Memory), and RAM (Random Access Memory). The storage unit 12 may function as, for example, main memory, auxiliary memory, or cache memory. The storage unit 12 stores any information used in the operation of the information processing device 10. For example, the storage unit 12 may store system programs, application programs, and various information received by the communication unit 13. The storage unit 12 is not limited to those built into the information processing device 10, but may also be an external database or an external storage module.

[0017] The memory unit 12 may hold a pre-trained model that is trained to output areas of corrosion when it receives a photograph of a structure that is the target equipment. The memory unit 12 may hold a pre-trained model that is trained to output areas occupied by a structure when it receives a photograph of a structure. The memory unit 12 may hold a pre-trained model that is trained to output a prediction image showing areas of corrosion that are predicted to exist on the structure at a given time when it receives a photograph of a structure, areas occupied by corrosion on the structure, environmental data, and a desired prediction time. Here, the environmental data is arbitrary data relating to environmental factors that may affect the progression of corrosion on the structure.

[0018] The communication unit 13 includes an optional communication module that can communicate with other devices such as network storage using any communication technology. The communication unit 13 may further include a communication control module for controlling communication with other devices, and a storage module for storing communication data such as identification information necessary for communication with other devices.

[0019] The input unit 14 includes one or more input interfaces that receive user input operations and acquire input information based on user operations. For example, the input unit 14 may be, but is not limited to, physical keys, capacitive keys, a pointing device, a touchscreen integrated with the display of the output unit 15, or a microphone that accepts voice input.

[0020] The output unit 15 includes one or more output interfaces that output information to the user and notify the user. For example, the output unit 15 is a monitor (display) that outputs information as an image, or a speaker that outputs information as sound, but is not limited to these. Such a monitor may be, for example, a liquid crystal panel display or an organic EL (Electro Luminescence) display. At least one of the above-mentioned input unit 14 and output unit 15 may be configured integrally with the information processing device 10, or may be provided as a separate unit.

[0021] The information processing device 10 can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. In other words, the functions of the information processing device 10 can be realized by executing a computer program (program) according to this embodiment on the processor included in the control unit 11. To put it another way, the functions of the information processing device 10 can be realized by software. The computer program causes the computer to execute the processing steps included in the operation of the information processing device 10, thereby realizing the functions corresponding to the processing of each step on the computer. In other words, the computer program is a program that causes the computer to function as the information processing device 10 according to this embodiment.

[0022] Computer programs can be recorded on computer-readable recording media. Examples of computer-readable recording media include magnetic recording devices, optical discs, magneto-optical recording media, or semiconductor memory. Programs can be distributed, for example, by selling, transferring, or leasing portable recording media such as DVDs (Digital Versatile Discs) or USB (Universal Serial Bus) memory containing the programs. Programs may also be distributed by storing them in server storage and transferring them from the server to other computers via a network. Programs may also be provided as program products.

[0023] A computer may, for example, temporarily store a program recorded on a portable storage medium or a program transferred from a server in its main memory. The computer may then read the program stored in the main memory with its processor and execute the processing according to the read program. The computer may also directly read a program from a portable storage medium and execute the processing according to the program. The computer may sequentially execute the processing according to the received program each time a program is transferred to it from a server. Such processing may also be performed by a so-called ASP-type service that does not transfer programs from the server to the computer, but realizes its function only through execution instructions and result acquisition. "ASP" is an abbreviation for Application Service Provider. A program includes information used for processing by an electronic computer that is equivalent to a program. For example, data that is not a direct instruction to the computer but has the nature of defining the computer's processing falls under "equivalent to a program".

[0024] Some or all of the functions of the information processing device 10 may be implemented by a dedicated circuit included in the control unit 11. In other words, some or all of the functions of the information processing device 10 may be implemented by hardware. Furthermore, the information processing device 10 may be implemented by a single computer or by the cooperation of multiple computers. For example, the information processing device 10 may be implemented by a server device installed in, for example, a data center or cloud computing system, and a terminal device that accesses the server device via a network.

[0025] In the configuration described above, the information processing device 10 acquires a photograph of the structure, which is the target equipment. The information processing device 10 detects areas of corrosion present in the structure from the acquired photographic image. The information processing device 10 acquires a predicted image showing the areas of corrosion that are predicted to be present in the structure at the predicted time, based on the photographic image, the detected areas of corrosion, environmental data related to environmental factors that may affect the progression of corrosion in the structure, and a desired predicted time. The information processing device 10 outputs the acquired predicted image.

[0026] Thus, the information processing device 10 acquires a predictive image using at least the captured image, the corroded area detected from the captured image, and environmental data related to environmental factors in the structure, and can predict the progression of corrosion for each target facility in a time series with high accuracy.

[0027] <Functional Configuration of Information Processing Device 10> Figure 2 is a block diagram showing an example of the functional configuration of an information processing device 10 according to one embodiment. As shown in Figure 2, the information processing device 10 comprises functional elements of an image input unit 21, a corrosion detection unit 22, a data merging unit 23, a generation unit 24, and an image output unit 25. At least some of these functional elements may be implemented by software or by hardware.

[0028] The image input unit 21 receives an image 31 taken of a target structure using, for example, a digital camera. The target structure is, for example, a steel structure such as a bridge, road bridge, transmission tower, railway, sluice gate, storage tank, propeller tower, guardrail, and road sign, but is not limited to these. The image 31 is, for example, a color image, but may also be a monochrome image including a grayscale image. Below, as an example, the case where the image 31 is a color image will be explained. The image input unit 21 performs necessary processing, such as converting the color image 31 into a three-dimensional tensor, and outputs it to the corrosion detection unit 22 and the data merging unit 23.

[0029] The corrosion detection unit 22 detects areas of corrosion occurring on the structure from the captured image 31 received from the image input unit 21. The corrosion detection unit 22 may also use a pre-trained model to detect areas of corrosion from the captured image 31. Corrosion may include any and all deterioration of the steel parts of the structure. Corrosion may include, for example, rust on the steel used in the structure, peeling paint from the structure, cracks in the paint of the structure, mud stains attached to the structure, and moss growing on the structure. The corrosion detection unit 22 outputs a corrosion image showing the areas of corrosion present in the structure detected from the captured image 31 to the data merging unit 23.

[0030] The data merging unit 23 combines the corrosion image detected by the corrosion detection unit 22, the captured image 31 input from the image input unit 21, the year data, and the environmental data to generate combined data. The combined data may also be a numerical matrix formed by combining the corrosion image, the captured image 31, the year data, and the environmental data. The data merging unit 23 outputs the combined data to the generation unit 24.

[0031] The year data (timing data) indicates the predicted timeframe for predicting the progression of deterioration from the captured image 31 input to the image input unit 21. The year data may, for example, be information representing the elapsed time from the time the captured image 31 was taken to the predicted timeframe in years. The year data may also be generated based on the year, month, and day entered by the user as the desired prediction timeframe. The unit of the year data is years. For example, if the year data is 1, the information processing device 10 predicts the progression of corrosion one year from the time the captured image 31 input to the image input unit 21 was taken. The information processing device 10 may also accept decimal values ​​as year data. For example, if the year data is 0.1, the information processing device 10 may predict the progression of corrosion approximately 37 days (= 0.1 × 365 days) later. By allowing the input of any number of years, the information processing device 10 makes it possible to check, in chronological order, how far the corrosion will progress in a given area of ​​the structure. Therefore, the user can easily determine and set the next inspection cycle for each piece of equipment. Furthermore, the information processing device 10 may automatically determine the next inspection cycle according to the degree of corrosion progression. For example, the next inspection cycle may be determined to be the time when the area of ​​corrosion exceeds a threshold.

[0032] Environmental data is information about environmental factors that can affect the progression of corrosion in a structure. For example, environmental data includes environmental factors that affect the rate of corrosion, such as temperature, humidity, amount of airborne salt, presence or absence of salt damage, precipitation, and sunshine hours. Environmental data may also be average values ​​measured over a predetermined period, such as an annual, quarterly, or monthly period. The information processing device 10 predicts the progression of corrosion based on the environmental conditions that affect the rate of corrosion, which are provided as environmental data, thereby improving the accuracy of the prediction.

[0033] The generation unit 24 generates a predicted image 33, which is a prediction of the future progression of corrosion, from the combined data created by the data combining unit 23. The generation unit 24 may also input the combined data into a pre-trained model to obtain the predicted image 33.

[0034] The image output unit 25 outputs a prediction image 33 showing the result of predicting the progression of corrosion. For example, the image output unit 25 may output the prediction image 33 as visual information to the monitor of the output unit 15 or output the prediction image 33 as image data to the storage unit 12.

[0035] Next, we will explain the details of each functional element.

[0036] (Image Input Unit 21) The image input unit 21 receives a captured image 31 taken by a shooting device such as a digital camera, and converts the captured image 31 into a tensor of a predetermined dimension according to the type of captured image 31. If the captured image 31 is a color image, the image input unit 21 converts the captured image 31 into a three-dimensional tensor. If the captured image 31 is a monochrome image, the image input unit 21 converts the captured image 31 into a one-dimensional tensor.

[0037] The 3D tensor to which the captured image 31, which is a color image, is converted may be any color space selected from among the RGB color space, HSV color space, L*a*b* color space, etc. The color space to which the captured image 31 is converted can be appropriately selected depending on the shooting environment of the captured image 31 and the purpose of the analysis.

[0038] For example, the captured image 31 may be taken in a dark place, such as an outdoor infrastructure facility. When an image 31 is taken in such a place, the entire image will be dark. As a result, when such an image 31 is converted into a three-dimensional tensor in the RGB color space, there may be little change in R (Red), G (Green), and B (Blue). In contrast, since the HSV color space is a combination of Hue, Saturation, and Value, when the captured image 31 is converted into a three-dimensional tensor in the HSV color space, the change in Hue in the HSV color space has the effect of being useful for analysis.

[0039] Furthermore, compared to the RGB color space, the L*a*b* color space has the characteristic of being closer to human perception. Therefore, converting the captured image 31 to the L*a*b* color space has the effect of allowing analysis in a way that is closer to human perception.

[0040] As each color space has its own characteristics, the color space to which the captured image 31 is converted may be selected based on the shooting environment of the target structure and the purpose of the analysis, to obtain the one that provides the best accuracy for achieving the analysis objective. The captured image 31 may also be a video. When a video is input to the image input unit 21, the image input unit 21 converts the frames contained in the video into still images and then converts them into a three-dimensional tensor in some color space.

[0041] The image input unit 21 may also modify the size of the captured image 31 to match the size and resolution of the input image assumed by the corrosion detection unit 22 and the generation unit 24. For example, the image input unit 21 may convert the number of pixels in the captured image 31 to match the number of vertical and horizontal pixels of the input image assumed by the corrosion detection unit 22 and the generation unit 24, or it may clip a portion of the captured image 31. The captured image 31 may be an image taken from a position a predetermined distance away from the subject. The resolution per pixel of the captured image 31 (for example, the distance represented by one pixel) may be a predetermined value.

[0042] Figure 3 shows an example of a captured image 31 input to the image input unit 21. The captured image 31 includes a region 411 occupied by the target equipment structure, a region 412 occupied by equipment that is not the target equipment, and a region 413 occupied by corrosion. Region 411 is, for example, the region occupied by steel structures. Region 412 is, for example, the region occupied by equipment made of concrete or the like. The image input unit 21 converts such a captured image 31 into a color space tensor and outputs it to the corrosion detection unit 22 and the data merging unit 23.

[0043] (Corrosion detection unit 22) Figure 4 is a block diagram showing an example of the functional configuration of the corrosion detection unit 22. The corrosion detection unit 22 detects the area of ​​corrosion in the target equipment from the three-dimensional tensor of the captured image 31 input from the image input unit 21. The corrosion detection unit 22 includes an equipment detector 221 and a corrosion detector 222.

[0044] The equipment detector 221 detects a region occupied by target equipment (for example, steel material, etc.) to be inspected from the captured image 31. The region of the target equipment in the captured image 31 may be output as an image in which the equipment region is white (value 1) and the region other than the equipment is black (value 0). The corrosion detector 222 detects a region occupied by corrosion from the captured image 31. The corrosion region in the captured image 31 may be output as an image in which the corrosion region is white (value 1) and the region other than corrosion is black (value 0).

[0045] The corrosion detection unit 22 can detect the corrosion region in the target equipment in the captured image 31 by taking the logical AND of the detection results of the region of the target equipment to be inspected and the corrosion region. The corrosion region in the target equipment in the captured image 31 may be output as an image 34 (corrosion image) in which the corrosion region is white (value 1) and the region other than corrosion is black (value 0).

[0046] According to this configuration, even if various steel structures are captured in the captured image 31 when photographing infrastructure equipment, corrosion inspection can be performed only on the steel structure to be inspected, which improves inspection efficiency.

[0047] The equipment detector 221 and the corrosion detector 222 can be constructed by a learner trained in advance through machine learning.

[0048] FIG. 5 is a diagram explaining the training of the equipment detector 221. The equipment detector 221 can be constructed by the equipment trainer 223. The equipment trainer 223 is constructed by, for example, a convolutional neural network or the like. The equipment trainer 223 inputs a training image 41 obtained by photographing the target equipment and an image 42 indicating the region of the target equipment included in the image 41 as teacher data, and performs training via a model such as a convolutional neural network. The image 41 may include a target equipment region 411 and a corrosion region 412. The image 42 may be an image in which a label of value 1 (white) is assigned to a region occupied by target equipment such as a steel part to be inspected, and a label of value 0 (black) is assigned to a region other than the target equipment.

[0049] When the equipment learner 223 is constructed using a convolutional neural network, the equipment learner 223 is pre-trained using dozens or more images 41, 42. The corrosion detection unit 22 uses the equipment detector 221 (trained model) constructed by the equipment learner 223 to detect the area of ​​the target equipment structure (e.g., steel materials) from the captured image 31. The equipment detector 221 may output the area of ​​the target equipment in the captured image 31 as an image in which the area of ​​the equipment is white (value 1) and the area other than the equipment is black (value 0). Note that the target equipment to be detected can be arbitrarily set to equipment other than steel structures. For example, the target equipment to be detected may be equipment made of concrete, cement, etc. The equipment learner 223 may be implemented by software on the information processing device 10 or by another computer.

[0050] Figure 6 illustrates the learning process for the corrosion detector 222. The corrosion detector 222 can be constructed using a corrosion learner 224. The corrosion learner 224 is constructed, for example, using a convolutional neural network. The corrosion learner 224 receives training data consisting of a training image 41 of the target equipment and an image 43 showing the corroded areas included in image 41, and learns using a model such as a convolutional neural network. Image 41 may include the area 411 of the target equipment and the corroded area 412. Image 42 may be an image in which areas occupied by corrosion such as rust are labeled with a value of 1 (white), and areas other than corrosion are labeled with a value of 0 (black).

[0051] If the corrosion learner 224 is constructed using a convolutional neural network, the corrosion learner 224 is pre-trained using dozens or more images 41 and 43. The corrosion detection unit 22 uses the corrosion detector 222 (trained model) constructed by the corrosion learner 224 to detect areas of corrosion from the captured image 31. The corrosion detector 222 may output the areas of corrosion in the captured image 31 as an image in which the areas of corrosion are white (value 1) and areas other than corrosion are black (value 0). The corrosion learner 224 may be implemented by software on the information processing device 10 or by another computer.

[0052] The corrosion detection unit 22 outputs the corrosion region of the target equipment in the captured image 31 to the data merging unit 23 by taking the logical AND of the detection result of the area of ​​the target equipment output from the equipment detector 221 and the corrosion region output from the corrosion detector 222. The corrosion region of the target equipment may also be output as a two-dimensional tensor (single-layer matrix), such as an image 34 (corrosion image) where the corrosion region is white (value 1) and the non-corrosion region is black (value 0).

[0053] (Data merging unit 23) The data merging unit 23 combines the captured image 31 input from the image input unit 21, the corrosion image created by the corrosion detection unit 22, the year data, and the environmental data to generate combined data, which is a multidimensional tensor. The captured image 31 may be a three-dimensional tensor (three-layer matrix). The corrosion image may be a two-dimensional tensor (one-layer matrix). The data merging unit 23 may also convert the year data and the environmental data into two-dimensional tensors (one-layer matrices) and then combine them with the captured image 31 (three-dimensional tensor) and the corrosion image (two-dimensional tensor) to construct combined data, which is a multidimensional tensor. Here, the year data converted into a two-dimensional tensor is a two-dimensional tensor in which year data is placed for each pixel. The environmental data converted into a two-dimensional tensor is a two-dimensional tensor in which environmental data is placed for each pixel.

[0054] As mentioned above, the year data indicates the predicted timeframe for predicting the progression of deterioration based on the captured image 31 input by the image input unit 21. The predicted timeframe may be set according to user instructions. The year data may also be information representing the elapsed time in years from the time the captured image 31 was taken to the predicted timeframe. For example, if the desired predicted timeframe is 3 years from now, the year data would be 3, and if corrosion progression is to be predicted 10 years from now, the year data would be 10.

[0055] The generation unit 24 outputs a predicted image 33 showing the future corrosion area from the combined data using a pre-trained model that has been trained in advance by machine learning such as deep learning. The data combining unit 23 then creates a tensor normalized to a predetermined value such as 0-1 in accordance with the specifications of the generation unit 24. For example, the data combining unit 23 may divide the year data by a predetermined value such as 10 or 100 and create a two-dimensional tensor based on the result (quotient) of the division. That is, the data combining unit 23 may create a tensor in which the years normalized by the predetermined value are arranged as elements of a matrix of the same size as the matrix corresponding to each pixel of the captured image 31. For example, if the year data normalized between 0 and 1 is extremely small than 0.1, the accuracy of the prediction by the generation unit 24 will decrease. Therefore, the predetermined value used for the normalization division may be appropriately set so that the normalized year data does not become extremely small, depending on the prediction timing for predicting the progression of corrosion.

[0056] Environmental data can be set to any value, such as at least one of the following: temperature, humidity, amount of airborne salt, presence or absence of salt damage, precipitation, and sunshine hours (all of which may be average values ​​over a predetermined period such as a year or a month). The data merging unit 23 may also normalize such environmental data. For example, the average temperature may be a normalized value obtained by dividing it by the maximum temperature expected to be input. The presence or absence of salt damage may be binary data such as 1 or 0.

[0057] The data merging unit 23 may also create a tensor by arranging normalized values ​​of the environmental data as elements of a matrix corresponding to each pixel of the captured image 31. If the environmental data consists of multiple input values ​​(for example, temperature and humidity), the data merging unit 23 may create multiple two-dimensional tensors by normalizing each input value and arranging them in a matrix. Alternatively, the data merging unit 23 may create a single two-dimensional tensor by representing each of the multiple normalized input values ​​with a predetermined bit length and arranging the combined values ​​in a matrix, in accordance with the specifications of the generation unit 24. In this case, the data merging unit 23 may create a two-dimensional tensor by arranging data that is repeatedly concatenated by combining multiple input values, each represented with a predetermined bit length, in a matrix.

[0058] The data merging unit 23 outputs combined data, which is a multidimensional tensor generated by combining the captured image 31, corrosion image, year data, and environmental data, to the generation unit 24. Since such environmental data indicates the environmental characteristics of the equipment to be predicted, the generation unit 24 can use such environmental data to predict the progression (rate) of corrosion with higher accuracy.

[0059] (Generation Unit 24) Figure 7 is a block diagram showing an example of the functional configuration of the generation unit 24. The generation unit 24 includes a generator 241. The generator 241 generates a two-dimensional tensor from a multi-dimensional tensor output from the data merging unit 23, showing the result of predicting future corrosion progression. The two-dimensional tensor constitutes a prediction image 33 that shows the areas of corrosion predicted to exist in the structure at the prediction time. The prediction image 33 may be an image in which the areas of corrosion predicted to exist at the prediction time are white (value 1) and areas other than corrosion are black (value 0). The prediction time for which the generation unit 24 predicts the progression of corrosion can be set by the year data input to the data merging unit 23. The generator 241 in the generation unit 24 may be any model learned by a machine learning method such as deep learning in the generation learner 243.

[0060] Figure 8 illustrates the learning process of the generator 241. The generator 241 comprises a generator 241 and a discriminator 242. The generator 241 is learned by a generative learner 243. The generative learner 243 may be constructed, for example, by a deep learning method called GAN (Generative Adversarial Network). The generative learner 243 receives combined data from the generator 241 (model), which is a combination of the following four data: • A three-dimensional tensor of image 41 (first image) of the sample structure. • A two-dimensional tensor of image 43 (second image) with labels indicating the area occupied by corrosion in the first image. • A two-dimensional tensor 44 relating to year data indicating a predetermined time period. • A two-dimensional tensor 45 relating to environmental data indicating environmental factors in the sample structure (can be constructed from multiple environmental data).

[0061] The generator 241 outputs a two-dimensional tensor of image 46 (third image), labeled to indicate areas of corrosion predicted to exist on the sample structure at a predetermined time, to the classifier 242 in response to the input of combined data. The classifier 242 feeds back to the generator 241 the difference between the two-dimensional tensor of image 46 and the two-dimensional tensor of image 47 (fourth image), labeled to indicate areas of corrosion that actually existed on the sample structure at that predetermined time (ground truth data). The generator 241 adjusts its parameters so that the difference fed back from the classifier 242 becomes smaller.

[0062] In this way, the generative learner 243 is trained by repeatedly performing a process to adjust the parameters of the generator 241 so that the difference between the image 46 output by inputting a sample of the combined data into the generator 241 and the image 47 which corresponds to the ground truth data is reduced. When using a deep learning method, the generative learner 243 may be trained by preparing dozens of sets of captured images 31 and actual images 31 taken several years later.

[0063] In this embodiment, the generative learner 243 trains the generator 241 using not only the 3D tensor of image 41, the label assignment (image 43), and the 2D tensor 44 of the year data, but also environmental data. Therefore, the generator 241 can predict the progression of corrosion with high accuracy using environmental information of the equipment that affects the progression (rate) of corrosion. The type of environmental data used to train the generator 241 must be the same as the environmental data used when generating the predicted image 33 based on the captured image 31.

[0064] Furthermore, in the two-dimensional tensor 44 for the year data and the two-dimensional tensor 45 for the environmental data, the year may be normalized to a numerical value only for the matrix elements to which the corrosion region is labeled, while the other elements are set to 0. In this way, by setting the year data information only for the corrosion region to be predicted in the two-dimensional tensors 44 and 45, it is possible to suppress the noise caused by information input to regions outside the prediction target. As a result, only the corrosion region is analyzed intensively, and the accuracy of predicting the progression of corrosion can be improved.

[0065] (Image Output Unit 25) The image output unit 25 outputs a two-dimensional tensor of corrosion progression predicted by the generation unit 24 as a predicted image 33. For example, the image output unit 25 may output an image as the predicted image 33 in which the predicted corrosion progression area is painted with a predetermined color (for example, red) on the captured image 31 input to the image input unit 21. This allows the user to visually grasp the area where corrosion has progressed more intuitively. The image output unit 25 may display the predicted image 33 on the monitor of the output unit 15 or store it in the storage unit 12.

[0066] Furthermore, the image output unit 25 may use the detection results of the equipment area to display and output the percentage of the equipment area in the image where corrosion occurred during shooting, and the percentage of the equipment area in the image where corrosion occurred during prediction. The image output unit 25 may also display the captured image 31 and the predicted image 33, along with the percentage of each corrosion area in the equipment area of ​​the image, side by side. This allows the user to grasp the progression of corrosion more quantitatively.

[0067] (Example of operation) Figure 9 is a flowchart showing an example of the operation of the information processing device 10. Figure 9 shows an example of the process of generating a predicted image 33 from a captured image 31. The operation of the information processing device 10 described with reference to Figure 9 may correspond to one of the information processing methods. The operation of each step in Figure 9 may be performed based on control by the control unit 11 of the information processing device 10.

[0068] In step S1, the control unit 11 acquires a captured image 31 of the structure which is the target equipment. For example, the control unit 11 may acquire the captured image 31 by receiving it from another device via the communication unit 13 or by reading it from the storage unit 12.

[0069] In step S2, the control unit 11 detects a corrosion image (image 34) from the acquired captured image 31 that shows the area of ​​corrosion present in the structure. The control unit 11 may also detect the area of ​​corrosion included in the captured image 31 and the area of ​​the structure included in the captured image 31, and then detect the area of ​​corrosion present in the structure by performing a logical AND operation between the area of ​​corrosion included in the captured image 31 and the area of ​​the structure included in the captured image 31.

[0070] In step S3, the control unit 11 generates combined data by combining the captured image 31, the corrosion image (image 34) showing the corrosion area detected in step S2, environmental data, and timing data. The environmental data is data relating to environmental factors that may affect the progression of corrosion in the structure, and may have environmental information relating to environmental factors for each pixel. The timing data may be data that has a desired predicted timing for each pixel.

[0071] In step S4, the control unit 11 obtains a predicted image 33 from the combined data generated in step S3, showing the areas of corrosion that are predicted to exist on the structure at the predicted time. The control unit 11 may also input the combined data into a generator 241, which is a pre-trained model, to obtain the predicted image 33.

[0072] In step S5, the control unit 11 outputs the predicted image 33 acquired in step S4. Specifically, the control unit 11 may display the predicted image 33 on the monitor of the output unit 15 or store it in the storage unit 12. After completing the processing in step S5, the control unit 11 terminates the processing of the flowchart.

[0073] Figure 10 is a flowchart showing an example of the operation of the information processing device 10. Figure 10 shows an example of the process for training the generator 241. The operation of the information processing device 10 described with reference to Figure 10 may correspond to one of the information processing methods. The operation of each step in Figure 10 may be executed based on control by the control unit 11 of the information processing device 10.

[0074] In step S11, the control unit 11 inputs the input image, corrosion image, year data, and environmental data to the generator 241. The input image is a first image (image 41) of the sample structure. The corrosion image is a second image (image 42) showing the area occupied by corrosion in the first image. The year data is a two-dimensional tensor 44 indicating a predetermined period of time. The environmental data is a two-dimensional tensor 45 showing environmental information indicating environmental factors in the sample structure.

[0075] In step S12, the control unit 11 obtains the difference between the output image (image 46) of the generator 241 and the training image (image 47) using the classifier 242.

[0076] In step S13, the control unit 11 determines whether the difference satisfies a predetermined condition. For example, the control unit 11 may determine that the predetermined condition is satisfied if the difference is smaller than a predetermined threshold. Alternatively, the control unit 11 may determine that the predetermined condition is satisfied if the number of times the difference has been calculated by adjusting the parameters of the generator 241 reaches a predetermined number. If the difference satisfies the predetermined condition (YES in step S13), the control unit 11 proceeds to step S14; otherwise (NO in step S13), it proceeds to step S15.

[0077] In step S14, the control unit 11 outputs the generator 241. Then, the control unit 11 terminates the processing of the flowchart.

[0078] In step S15, the control unit 11 modifies the parameters of the generator 241 based on the difference. Then, the control unit 11 returns to step S11.

[0079] The control unit 11 may perform the processing in steps S11 to S15 for multiple input images (images 41).

[0080] As described above, the information processing device 10 acquires a captured image 31 of the target structure. The information processing device 10 detects areas of corrosion present in the structure from the acquired captured image 31. The information processing device 10 inputs the captured image 31, the detected areas of corrosion, environmental data, and a desired prediction time into a trained model to acquire a predicted image showing the areas of corrosion predicted to be present in the structure at the prediction time. Here, the environmental data is data related to environmental factors that may affect the progression of corrosion in the structure. The information processing device 10 outputs the acquired predicted image.

[0081] In this way, the information processing device 10 acquires predictive images using environmental data related to environmental factors in the structure, and can predict the progression of corrosion for each target facility in a time series with high accuracy.

[0082] The information processing device 10 may generate combined data by combining the captured image 31, a corrosion image showing the detected corrosion area, environmental data having environmental information about environmental factors for each pixel, and timing data having a desired prediction timing for each pixel. The information processing device 10 may input the generated combined data into a trained model to obtain a predicted image.

[0083] Thus, since the information processing device 10 acquires a predicted image based on the combined data, it can efficiently acquire a predicted image using a pre-trained model based on a neural network.

[0084] Furthermore, the information processing device 10 may input combined data into the model, which is obtained by combining a first image of a sample structure, a second image showing the area occupied by corrosion in the first image, environmental data showing environmental factors in the sample structure, and a predetermined time period. The information processing device 10 may train the model so that the difference between a third image showing the area of ​​corrosion predicted to exist in the sample structure at a predetermined time and a fourth image showing the area of ​​corrosion that actually existed in the sample structure at the predetermined time becomes smaller.

[0085] The information processing device 10 can use the trained model acquired through such processing to predict the progression of corrosion for each target piece of equipment in a time series with high accuracy.

[0086] Furthermore, the information processing device 10 may detect areas of corrosion included in the captured image 31 and areas of structures included in the captured image 31. The information processing device 10 may also detect areas of corrosion present in the structure by performing a logical AND operation between the detected areas of corrosion included in the captured image 31 and the areas of structures included in the captured image 31.

[0087] In this way, the information processing device 10 detects areas of corrosion present in a structure by performing a logical AND operation between the areas of corrosion included in the captured image 31 and the areas of the structure included in the captured image 31, thereby preventing the erroneous detection of corrosion in areas other than the structure. As a result, the information processing device 10 can predict the progression of corrosion for each target facility with high accuracy.

[0088] The information processing device 10 may also obtain a predicted image 33 by performing a logical AND operation between the image output from the generator 241 and the region of the structure included in the captured image 31. This prevents the device from mistakenly predicting that corrosion will progress in areas other than the structure.

[0089] This disclosure is not limited to the embodiments described above. For example, multiple blocks shown in the block diagram may be combined, or a single block may be divided. Multiple steps shown in the flowchart may be performed in parallel or in a different order, depending on the processing capacity of the device performing each step, or as necessary, instead of being performed in chronological order as described. Other modifications are possible without departing from the spirit of this disclosure.

[0090] The following additional information is disclosed regarding the embodiments described above.

[0091] [Note 1] An information processing device comprising a control unit that acquires a photograph of a structure which is the target equipment, detects areas of corrosion present in the structure from the acquired photographic image, inputs the photographic image, the detected areas of corrosion, environmental data relating to environmental factors that may affect the progression of corrosion in the structure, and a desired prediction time into a trained model to acquire a prediction image showing the areas of corrosion that are predicted to be present in the structure at the prediction time, and outputs the acquired prediction image.

[0092] [Addendum 2] The information processing apparatus according to Addendum 1, wherein the control unit generates combined data by combining the captured image, a corrosion image showing the detected area of ​​corrosion, environmental data having environmental information relating to the environmental factors for each pixel, and timing data having a desired prediction timing for each pixel, and inputs the generated combined data into the trained model to obtain the predicted image.

[0093] [Note 3] The trained model is a model trained to minimize the difference between a third image showing the predicted area of ​​corrosion present in the sample structure at the predetermined time, and a fourth image showing the area of ​​corrosion actually present in the sample structure at the predetermined time, as output from the model. This data is obtained by inputting combined data into the model, which consists of a first image of a sample structure, a second image showing the area of ​​corrosion in the first image, environmental data indicating environmental factors in the sample structure, and a predetermined time period. The information processing device is as described in Note 1 or 2.

[0094] [Appendix 4] The control unit detects the region of corrosion included in the captured image and the region of the structure included in the captured image, and detects the region of corrosion present in the structure by performing a logical AND operation between the detected region of corrosion included in the captured image and the region of the structure included in the captured image, as described in any one of the appendix items 1 to 3.

[0095] [Appendix 5] An information processing method performed by an information processing device, comprising: acquiring a photographic image of a structure that is a target facility; detecting areas of corrosion present in the structure from the acquired photographic image; inputting the photographic image, the detected areas of corrosion, environmental data relating to environmental factors that may affect the progression of corrosion in the structure, and a desired prediction time into a trained model to acquire a prediction image showing areas of corrosion that are predicted to be present in the structure at the prediction time; and outputting the acquired prediction image.

[0096] [Appendix 6] A program that causes a computer to execute the following steps: a procedure for acquiring a photographic image of a structure that is the target equipment; a procedure for detecting areas of corrosion present in the structure from the acquired photographic image; a procedure for inputting the photographic image, the detected areas of corrosion, environmental data relating to environmental factors that may affect the progression of corrosion in the structure, and a desired prediction time into a trained model to acquire a prediction image showing the areas of corrosion that are predicted to be present in the structure at the prediction time; and a procedure for outputting the acquired prediction image.

[0097] 10: Information processing unit 11: Control unit 12: Storage unit 13: Communication unit 14: Input unit 15: Output unit 21: Image input unit 22: Corrosion detection unit 221: Equipment detector 222: Corrosion detector 223: Equipment learner 224: Corrosion learner 23: Data merging unit 24: Generation unit 241: Generator 242: Identifier 243: Generation learner 25: Image output unit 31: Captured image 311, 312: Area 32: Years / environmental data 33: Predicted image 34: Image 41, 42, 43: Image 44: Years data 45: Environmental data 46, 47: Image 411, 412: Area

Claims

1. An information processing device comprising a control unit that acquires a photograph of a structure which is a target facility, detects areas of corrosion present in the structure from the acquired photographic image, inputs the photographic image, the detected areas of corrosion, environmental data relating to environmental factors that may affect the progression of corrosion in the structure, and a desired prediction time into a trained model to acquire a prediction image showing the areas of corrosion that are predicted to be present in the structure at the prediction time, and outputs the acquired prediction image.

2. The information processing apparatus according to claim 1, wherein the control unit generates combined data by combining the captured image, a corrosion image showing the detected area of ​​corrosion, environmental data having environmental information relating to the environmental factors for each pixel, and timing data having a desired prediction timing for each pixel, and inputs the generated combined data into the trained model to obtain the predicted image.

3. The information processing apparatus according to claim 1 or 2, wherein the trained model is a model trained to minimize the difference between a third image showing the predicted area of ​​corrosion present in the sample structure at the predetermined time, and a fourth image showing the area of ​​corrosion actually present in the sample structure at the predetermined time, which is output from the model.

4. The information processing apparatus according to claim 1 or 2, wherein the control unit detects a region of corrosion included in the captured image and a region of the structure included in the captured image, and detects a region of corrosion present in the structure by performing a logical AND operation between the detected region of corrosion included in the captured image and the region of the structure included in the captured image.

Images