Undersea object type-identification method and undersea object type-identification program

Abstract

An undersea object type-identification device (100): acquires a plurality of images (Dn) to be annotated that have been captured to include undersea objects and identifies the image to be annotated that has the shortest imaging distance (dm) to the objects among the imaging distances of the plurality of images (Dn) to be annotated; selects a reference image (Ir) that has a shorter imaging distance than the imaging distance (dm) of the identified image to be annotated from a pre-stored data set of a plurality of reference images (Rn) that include undersea objects; and uses the selected reference image (Ir) to identify the types of objects included in the images (Dn) to be annotated and outputs an image annotation results group (An).

Description

Method for Identifying Marine Target Species and Program for Identifying Marine Target Species

[0001] The present invention relates to a method for identifying marine target species and a program for identifying marine target species.

[0002] For solving global social issues and creating new businesses, etc., technologies for ocean digital twins have been proposed that precisely digitize all states of the ocean (on the sea surface, in the sea, and on the seabed) and formulate and pre-verify measures suitable for the purpose in the digital space. Among the application areas of ocean digital twins, in the blue carbon measures, there is an effort to credit the increased CO2 absorption amount due to the creation of seaweed beds. In order to quantify the CO2 absorption amount, an annotated dataset for training an image AI that recognizes seaweed beds is essential, and an annotation method for efficiently creating the annotated dataset is required.

[0003] Conventionally, an annotation method has been proposed in which an annotation target image is compared with a reference image, and for each pixel, it is determined whether the type of the object is included, and the determination result is output as an image annotation result for each type.

[0004] For example, there is a technology for extracting an object region from a captured image of an object, correcting the size of the object region based on the captured distance detected by a distance sensor, and predicting the production amount of the object such as agricultural crops. Also, there is a technology for correcting the color information of a captured image using a reference image based on the difference in the color of carbon-fixing organisms such as seaweed depending on the water depth and the difference in water depth for each shooting period, and improving the accuracy of estimating the type of carbon-fixing organisms. Also, there is a technology for preprocessing the spatial image of the target region to correct the pixel values considering atmospheric scattering and absorption and inputting it into a DCNN (Deep Convolutional Neural Network), and visually changing the pixel values of the spatial image to detect macroalgae, etc. Also, there is a technology for estimating the CO2 absorption amount of seaweeds by image analysis of captured image data in which seaweeds are photographed from a position different from above and the side surfaces of the seaweeds are included.

[0005] Japanese Patent Application Laid-Open No. 2022-156431 International Publication No. 2024 / 034156 US Patent Application Publication No. 2024 / 0013531 Japanese Patent Application Laid-Open No. 2024-125623

[0006] In conventional techniques, when analyzing annotated images, the accuracy of determining the type (class) of seaweed varied depending on the shooting distance to the underwater object, such as seaweed. For example, one could compare the annotated image with a reference image to determine the class of seaweed species contained in each pixel, and output the determination result as the image annotation result. However, the reference image may include images taken at long shooting distances and greatly affected by underwater turbidity, which can lead to a decrease in the accuracy of determining the type of seaweed.

[0007] In one aspect, the present invention aims to improve the accuracy of determining the type of object in the sea.

[0008] According to one embodiment, a method and program for identifying the type of object in the ocean are proposed, which include the following steps: acquiring multiple annotation target images in which images of objects in the ocean are taken; identifying the annotation target image with the shortest shooting distance among the shooting distances of the objects in each of the multiple annotation target images; selecting a reference image from a plurality of reference images in which objects in the ocean are taken that has a shorter shooting distance than the identified annotation target image; and using the selected reference image, identifying the type of object included in the annotation target image.

[0009] According to one embodiment, the accuracy of determining the type of object in the sea can be improved.

[0010] Figure 1 is an explanatory diagram of the underwater object type identification process according to the embodiment. Figure 2 is an explanatory diagram of the ocean digital twin. Figure 3 is an explanatory diagram of a processing example in which underwater object identification is applied to the ocean digital twin. Figure 4 is an explanatory diagram of the overview processing of image annotation obtained by underwater object type identification. Figure 5A is an explanatory diagram of an example of type determination result by shooting distance. (Part 1) Figure 5B is an explanatory diagram of an example of type determination result by shooting distance. (Part 2) Figure 6 is an explanatory diagram of the image dataset according to the embodiment. Figure 7 is an explanatory diagram of the type according to the embodiment. Figure 8 is a functional block diagram of the underwater object type identification device according to the embodiment. Figure 9 is a diagram showing an example of the hardware configuration of the underwater object type identification device. Figure 10 is an overall flowchart showing an example of the underwater object type identification process. Figure 11 is a flowchart showing an example of the processing of the class-specific feature construction phase. Figure 12 is a flowchart showing an example of the shortest distance identification process. Figure 13 is a flowchart showing an example of the reference image selection process. Figure 14 is a flowchart showing an example of the representative image specification process for each class. Figure 15 is a flowchart showing an example of the patch splitting process. Figure 16 is a flowchart showing an example of the feature extraction process. Figure 17 is a flowchart showing an example of the annotation phase process. Figure 18 is a flowchart showing an example of the patch splitting process. Figure 19 is a flowchart showing an example of the feature comparison process. Figure 20 is an explanatory diagram of the k-NN method.

[0011] The embodiments of the underwater object type identification method and underwater object type identification program according to the present invention will be described in detail below with reference to the drawings.

[0012] (An Example of Identifying Types of Underwater Objects According to an Embodiment) Figure 1 is an explanatory diagram of the process for identifying types of underwater objects according to an embodiment. The method for identifying types of underwater objects in this embodiment identifies and presents, for example, the types (classes) of various underwater organisms, underwater artificial structures, topography, and geology present in the sea. In this embodiment, algae (seaweed and seagrass) are used as an example of underwater objects. For example, by identifying the types of algae, the amount of CO2 absorbed by creating seaweed beds in the sea can be quantified and applied to blue carbon policies such as converting the amount of CO2 absorbed into credits.

[0013] In the embodiment of the method for identifying the type of underwater object, for example, a supervised dataset is generated for training an image AI that recognizes seaweed beds based on images taken underwater. Then, annotation processing is performed on the images to add information about the type of algae using tags, metadata, etc. Hereinafter, images taken underwater will be referred to as annotation target images.

[0014] In this embodiment, for example, a process is performed to identify the type (class) of algae contained in an annotation target image taken in the ocean. In this identification process, the shooting distance (distance from the subject) of the annotation target image is referenced to identify the shortest shooting distance, and a reference image with a shooting distance shorter than the identified shooting distance is selected from among multiple reference images. Then, the type of algae contained in the annotation target image is identified using the selected reference image. The shorter the shooting distance of the captured image, the higher the clarity of the image. In this embodiment, by using an image with a shorter shooting distance than the annotation target image as a reference image, the accuracy of the determination for identifying the type of algae is improved.

[0015] The method for identifying the type of underwater object in this embodiment involves an underwater object identification device 100, such as a server, acquiring various necessary information and executing a program to perform the process of identifying the type of algae. The underwater object identification device 100 shown in Figure 1 performs the following processing.

[0016] (1) Obtain annotation target image data D taken underwater. Annotation target image data D is a color (RGB) image and has multiple data sets Dn. Note that, for convenience, the color and gradation of annotation target image data D are not shown in Figure 1.

[0017] (2) Identify the annotated image data D that has the shortest distance data dm from the shooting distance data d associated with each image in the annotated image dataset Dn. The shooting distance data d may be, for example, the distance from the camera taking pictures underwater to the subject, or the vertical distance from the camera to the seabed.

[0018] (3) Based on the shortest distance data dm, image data with a shooting distance less than or equal to the shortest distance data dm is extracted from multiple reference image datasets Rn, and reference image Ir is selected to be used in the following processing. Figure 1 shows one reference image Ir, but multiple images may be used. In this embodiment, by using an image with a shooting distance shorter than the annotation target image data D as the reference image Ir, it is possible to improve the accuracy of the judgment by excluding images with a long shooting distance and a large influence of turbidity in the sea from the reference image Ir.

[0019] (4) From the reference image Ir, a representative image for each class (type) is specified, and a class-specific reference image Ic is created.

[0020] (5) Patch division is performed using the class-specific reference image Ic and the annotation target image dataset Dn.

[0021] (6) After batch splitting, feature extraction is performed for each class (type). Feature extraction is performed using, for example, the dimensionality reduction method using UMAP (Uniform Manifold Application and Projection) (horizontal axis: UMAP1, vertical axis: UMAP2). UMAP is one of the data analysis methods called dimensionality reduction, and by compressing ARI, high-dimensional data into a low-dimensional space, it makes it easier to visualize the structure and patterns of the data. Alternatively, it can also be done using the CNN (Convolutional Neural Network) method.

[0022] Through patch splitting and feature extraction, for example, a two-dimensional class-specific feature vector Fc is obtained, as shown in Figure 1. In Figure 1, the feature vectors of the type (class) of algae to be recognized are mapped to each class as the class-specific feature vector Fc of UMAP.

[0023] (7) Feature comparison is performed, and based on the class-specific feature quantity Fc, it is determined which class the patch features of the image data to be annotated D are closest to, for example, using a k-NN (k-Nearest Neighbors) algorithm. This determination process is performed for all patches. This results in image annotation result A, which identifies the type of subject (algae) contained in the image dataset Dn to be annotated. Image annotation result A consists of multiple images (image annotation result group An) corresponding to the image dataset Dn to be annotated, and each image annotation result A is an image with a color that can identify the type of algae (the white area in Figure 1 is the background).

[0024] (Example of application to marine digital twins) Figure 2 is an explanatory diagram of a marine digital twin. The identification of underwater object types in this embodiment is applied, for example, to the marine digital twin 200. The marine digital twin 200 enables the effective utilization of the ocean to solve global social issues and create new businesses. In the marine digital twin 200, all states of the ocean (surface, underwater, and seabed) are precisely digitized in a digital space, and measures tailored to the purpose are planned and pre-verified.

[0025] As shown in Figure 2, various measures are pre-verified by performing the following processing on the real-world underwater environment W where target objects O such as marine organisms and structures exist, using the ocean digital twin 200.

[0026] 1. Ocean data Dw is collected using underwater 3D measurement technology that captures the movement of object O in real time. Ocean data Dw is digital data that accurately reproduces the ocean by using image enhancement AI technology that converts underwater objects into highly accurate 3D data.

[0027] Then, 2. Using the collected ocean data Dw, we create an ocean model M, which is a model of the environment, organisms, and their growth of the objects that make up the ocean.

[0028] 3. Using the modeled ocean data Dw, various measures are pre-verified through simulation. These measures include blue carbon (corresponding to the processing of the embodiment), environmental assessment, offshore wind power generation maintenance, etc. 4. The prescribed measures are then implemented for the actual underwater W.

[0029] Figure 3 is an explanatory diagram illustrating an example of processing applied to the identification of underwater objects in a marine digital twin. Using Figure 3, an example of the overall system processing, including the identification of underwater object types, will be explained. A camera 302 attached to an AUV (Autonomous Underwater Vehicle) 301 captures an underwater image (the annotated image above) D containing the underwater object, which is algae. In addition, a camera 303 attached to an UAV (Unmanned Aerial Vehicle) captures a satellite / aerial image G containing the area of ​​the underwater object.

[0030] The underwater image (the image to be annotated above) D is processed through high-precision seaweed bed recognition (S301), seaweed bed area measurement (S302), and mapping to satellite / aerial image G (S303) to obtain a high-precision recognition result Da. The high-precision recognition result Da includes information on the type (class) of algae E1 in the created seaweed bed.

[0031] Furthermore, the satellite / aerial image G is processed using wide-area seaweed bed recognition (S304) to obtain a wide-area recognition result Ga. The wide-area recognition result Ga includes information on the area E2 of the created seaweed bed.

[0032] The high-precision recognition result Da and the wide-area recognition result Ga are processed in the seaweed bed recognition result integration (S305) to obtain an integrated result (wide-area and high-precision recognition result) Ta. The integrated result (wide-area and high-precision recognition result) Ta includes information indicating the distribution state E3 of the algal species (class) E2 in the created seaweed bed area E1. For example, the distribution state E3 is displayed in different colors corresponding to the algal species (class).

[0033] Then, based on the information showing the distribution state E3, the CO2 absorption amount calculation (S306) process is performed to obtain the CO2 absorption amount calculation result Tb in the created seaweed bed area E1. For example, the CO2 absorption amount calculation result Tb displays the amount of CO2 absorbed (annual CO2 absorption) by type (class) of algae E2 in the created seaweed bed area E1 in different colors.

[0034] The series of processes shown in Figure 3 implements blue carbon measures, which are a candidate application for marine digital twins. In order to convert the increased CO2 absorption amount due to seaweed bed creation into credits, a supervised dataset is essential for training an image AI that recognizes seaweed beds in order to quantify the amount of CO2 absorption. An automated annotation method is needed to efficiently create this supervised dataset. Accordingly, the process related to the identification of underwater objects performed by the embodiment (underwater object type identification device 100 in Figure 1) is included in the high-precision seaweed bed recognition process in S301. The process in S301 outputs the image annotation result A shown in Figure 1.

[0035] In the overall system processing shown in Figure 3, the underwater object type identification device 100 of the embodiment may not be limited to the process of outputting image annotation result A (image annotation result group An), but may also include and perform some or all of other processes (S301 to S306), and may perform distributed processing including multiple servers and even the cloud.

[0036] (Example of image annotation results) Figure 4 is an explanatory diagram of the overview processing of image annotation obtained by identifying the type of underwater object. The underwater object identification device 100 refers to the reference image Ir and performs automatic annotation on the acquired annotation target image data D to obtain the image annotation result A.

[0037] The image annotation result A shown in Figure 4 is an image containing the area of ​​the created seaweed bed, with the types (classes) of algae—seaweed species A (red algae), seaweed species B (green algae)—and the background colored distinctly to distinguish them. Image annotation result A allows for the identification of areas of each type of algae within the seaweed bed.

[0038] (Challenges due to background technology) If annotation processing is performed on the data to be annotated D without considering the shooting distance of the reference image Ir, the reference image Ir will include images taken at long distances and with a large influence of turbidity. In this case, the accuracy of determining the type of subject (algae) included in the image annotation result A will decrease.

[0039] In contrast, the above-described embodiment solves the above problem by performing the above-described processing (processing (1) to (7) in Figure 1). Generally, the shorter the shooting distance, the clearer the image. Therefore, as in the embodiment, by referencing an image taken at a shorter distance than the annotation target image data D as the reference image Ir, the accuracy of determining the type of algae can be improved.

[0040] Figures 5A and 5B are explanatory diagrams illustrating examples of type determination results at different shooting distances. As shown in Figure 5A(a), when the clarity of the reference image Ir is low, color distortion and blurring occur in the image, resulting in the loss of detailed information about the object (e.g., the color of the algae, the shape of the leaves, fine textures, etc.). Therefore, the distinction between types (classes) becomes ambiguous in the class-specific feature quantity Fc shown in Figure 5A(b).

[0041] On the other hand, as shown in Figure 5B(a), when the reference image Ir has high clarity, the color, edges, and texture of the algae can be utilized, making it easier to extract features specific to each type of algae. Therefore, in the determination algorithm of this embodiment, more accurate information can be used when determining the type of algae, and the class-specific feature quantity Fc shown in Figure 5B(b) can clearly distinguish between types.

[0042] (Regarding the image dataset) Figure 6 is an explanatory diagram of the image dataset according to the embodiment. Figure 6(a) shows the dataset Dn of the annotated image data D, and Figure 6(b) shows the dataset Rn of the reference image Ir.

[0043] The annotation target image data D (D1, D2, ..., DN) shown in Figure 6(a) consists of multiple images taken including an object in the sea. Each of the multiple images is assigned shooting distance data d (1 [m], 2 [m], N [m]) to the object at the time of shooting. The multiple annotation target image data D form the annotation target image dataset Dn.

[0044] Similarly, each of the reference images Ir (R1, R2, ..., RN) shown in Figure 6(b) is assigned shooting distance data d (0.5 [m], 0.3 [m], 0.1 [m]) at the time of shooting. Multiple reference images Ir form a reference image dataset Rn.

[0045] (Regarding Classification) Figure 7 is an explanatory diagram of the classification according to the embodiment. Figure 7 shows an example of displaying the class-specific feature quantity Fc by UMAP, with the vertical axis (UMAP2) showing feature quantity 1 after dimensionality reduction and the horizontal axis (UMAP1) showing feature quantity 2 after dimensionality reduction. In the embodiment, the classification (classification) refers to the category of the object to be classified. If the object is algae, it is classified into the following three classes: 1. Seaweed species A (red algae), 2. Seaweed species B (green algae), 3. Background.

[0046] (Example of Functions of Underwater Object Type Identification Device) Figure 8 is a functional block diagram of an underwater object type identification device according to an embodiment. An example of the functions of the underwater object type identification device 100 that performs the processing shown in Figure 1 will be explained. The underwater object type identification device 100 includes the functions of an acquisition unit 801, a shortest distance identification unit 802, a reference image data selection unit 803, a representative image designation unit for each class 804, a patch division unit 805, a feature extraction unit 806, a feature comparison unit 807, and an output unit 808. In the example of functions in Figure 8, the underwater object type identification device 100 holds the reference image dataset Rn in a recording unit (not shown).

[0047] The acquisition unit 801 acquires an annotation target image dataset Dn, which consists of images taken underwater. The acquisition unit 801 acquires multiple underwater images (annotation target image data D) taken by a camera 302 attached to the AUV shown in Figure 3 as a dataset Dn.

[0048] The shortest distance identification unit 802 identifies the shortest distance data dm from the shooting distance data d associated with each image in the annotation target image dataset Dn.

[0049] The shooting distance data d can be estimated from the image using methods such as DNN (Deep Neural Network). Alternatively, it can be obtained using other measuring instruments besides cameras, such as LiDAR (Light Detection and Ranging), SONAR (Sound Navigation and Ranging), and DVL (Dopple Velocity Log).

[0050] The reference image data selection unit 803 selects a reference image Ir by extracting an image whose shooting distance d is less than or equal to the shortest distance dm from among multiple reference image datasets Rn based on the shortest distance data dm. The reference image dataset Rn and the annotation target image dataset Dn may be the same dataset. If they are the same dataset, instead of the shortest distance dm, the reference image data is identified by extracting an image with a shooting distance of N or less.

[0051] The representative image designation unit 804 for each class designates a representative image for each class (type) from the reference image Ir and creates a class-specific reference image Ic. The designation method may be, for example, rectangular designation or designation by graph cut.

[0052] The patch division unit 805 performs patch division using the reference image Ir and the annotation target image dataset Dn. The patch division is performed, for example, by dividing the data into a group of 3x3 (pixel) patches.

[0053] The feature extraction unit 806 performs feature extraction for each class (type) after batch partitioning. Feature extraction is performed, for example, by dimensionality reduction using UMAP (horizontal axis: UMAP1, vertical axis: UMAP2) or by processing using CNN (Convolutional Neural Network). After patch partitioning and feature extraction, a two-dimensional class-specific feature quantity Fc is obtained.

[0054] The feature comparison unit 807 determines which class a patch of the annotation target image data D is closest to, based on the class-specific feature quantity Fc, for example, by using k-NN (k-Nearest Neighbors) processing, and performs this processing for all patches. This results in image annotation results A (image annotation result group An) for the annotation target image dataset Dn.

[0055] The output unit 808 outputs the image annotation result group An. The output unit 808 outputs the image annotation result group An to the functional unit of the later stage of the system in Figure 3 (seaweed bed area measurement S302).

[0056] (Example Hardware Configuration of Underwater Object Type Identification Device) Figure 9 shows an example hardware configuration of an underwater object type identification device. In Figure 9, the underwater object type identification device 100 can use, for example, a general-purpose server or terminal PC. The underwater object type identification device 100 includes a CPU (Central Processing Unit) 901 such as a processor, memory 902, network IF 903, recording medium IF 904, and recording medium 905. Each component is connected by a bus 900.

[0057] Here, the CPU 901 is a control unit that oversees the overall control of the underwater object type identification device 100. The CPU 901 may have multiple cores. The memory 902 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and flash ROM. Specifically, for example, the flash ROM stores the OS program, the ROM stores the application program, and the RAM is used as the work area for the CPU 901. The program stored in the memory 902 is loaded into the CPU 901, causing the CPU 901 to execute the coded processing.

[0058] The Network IF903 is connected to a network NW via a communication line and can connect to an external computer via the network NW. The Network IF903 manages the interface between the network NW and the device's internal components, controlling the input and output of data from the external computer. The Network IF903 can utilize, for example, a modem or a LAN adapter.

[0059] The recording medium IF 904 controls the reading and writing of data to the recording medium 905 according to the control of the CPU 901. The recording medium 905 stores the written data. Examples of recording media 905 include magnetic disks such as HDDs, optical disks, SSDs (Solid State Drives), and USB (Universal Serial Bus) memory.

[0060] In addition to the components described above, the underwater object type identification device 100 may also have, for example, an input device, a display, and the like.

[0061] Each function of the underwater object type identification device 100 shown in Figure 8 can be realized by programming the CPU 901 shown in Figure 9. The memory 902 and recording medium 905 shown in Figure 9 are used as storage units for the information handled by each of the functional units (acquisition unit 801 to output unit 808) in Figure 8. In addition, the recording medium 905 shown in Figure 9 may pre-record and store the reference image dataset Rn shown in Figure 8.

[0062] (Example of process for identifying the type of object underwater) Figure 10 is an overall flowchart showing an example of the process for identifying the type of object underwater. An example of the process performed by the control unit (CPU 901) of the underwater object type identification device 100 will be explained.

[0063] The underwater object type identification device 100 includes a class-specific feature construction phase (step S1001) and an annotation phase (step S1002).

[0064] In the class-specific feature construction phase (step S1001), class-specific features Fc are calculated based on the reference image dataset Rn and the annotation target image dataset Dn.

[0065] In the annotation phase (step S1002), the image annotation result set An is determined based on the image dataset Dn to be annotated and the class-specific feature vector Fc.

[0066] (Example of processing in the class-specific feature construction phase) Figure 11 is a flowchart showing an example of processing in the class-specific feature construction phase. Using Figure 11, the details of the processing in the class-specific feature construction phase (step S1001) shown in Figure 10 will be explained.

[0067] First, the underwater object type identification device 100 acquires an annotation target image dataset Dn and a reference image dataset Rn (step S1101). Next, the underwater object type identification device 100 performs a shortest distance identification process from the annotation target image dataset Dn to obtain the shortest distance data dm (step S1102).

[0068] Next, the underwater object type identification device 100 performs a reference image selection process from the shortest distance data dm and the reference image dataset Rn to obtain a reference image Ir (step S1103). Next, the underwater object type identification device 100 performs a representative image designation process for each class from the reference image Ir to obtain a class-specific reference image Ic (step S1104).

[0069] Next, the underwater object type identification device 100 performs patch division processing from the class-specific reference image Ic to obtain class-specific reference patches (step S1105). Then, it performs feature extraction processing from the class-specific reference patches to output class-specific feature quantities Fc (step S1106).

[0070] The annotation target image dataset Dn consists of, for example, 845 images of 1920 x 1080 pixels, each containing a mixture of red algae, green algae, and background. The reference image dataset Rn consists of, for example, 10 images of 1920 x 1080 pixels, each containing a mixture of red algae, green algae, and background.

[0071] The class-specific feature Fc is, for example, a table consisting of 3 classes × 100 patches × 2 feature vectors (see Figure 7). The image annotation result set An consists of, for example, 845 images of 1920 × 1080 pixels, and each image contains the type (class) identifier values ​​0, 1, and 2 for the pixels representing red algae, green algae, and background, respectively.

[0072] The shooting distance data d is, for example, the distance to the center of an image taken of the same area as the annotation target image data D or the reference image data R. Each pixel of the depth image stores a distance [m], and the shooting distance data d represents the distance [m] stored in the center of the depth image. The shortest distance data dm is the smallest distance [m] among the shooting distance data d corresponding to each image D in the annotation target image dataset Dn.

[0073] Furthermore, the reference image Ir is, for example, one image of 1920 x 1080 pixels, but there may be two or more. Also, the class-specific reference image Ic is, for example, three images of 300 x 300 pixels, consisting of, for example, an image of only red algae, an image of only green algae, and an image of only the background. Also, the class-specific reference patch is, for example, three sets of 100 patches of 3 x 3 pixels each, consisting of, for example, a set of red algae, a set of green algae, and a set of backgrounds.

[0074] Figure 12 is a flowchart showing an example of the shortest distance determination process. Figure 12 shows the details of the shortest distance determination process (step S1102 in Figure 11) performed by the shortest distance determination unit 802.

[0075] First, the underwater object type identification device 100 uses an annotation target image dataset Dn(N annotation_dataset Input the number of sheets (step S1201). Next, the underwater object type identification device 100 determines the coefficient i=1 and the shortest distance data dm(d min (Step S1202) Let ) = ∞, and take the i-th data from the annotation target image dataset Dn to obtain the annotation target image data D (Step S1203).

[0076] Next, the underwater target species identification device 100 obtains shooting distance data d corresponding to the annotation target image data D (step S1204). Then, the underwater target species identification device 100 compares the shooting distance data d with the shortest distance data dm (step S1205, d < d min ?).

[0077] As a comparison result, if d < d min (step S1205: Yes), the underwater target species identification device 100 proceeds to the process of step S1206, and if d ≥ d min (step S1205: No), it proceeds to the process of step S1207.

[0078] In step S1206, the underwater target species identification device 100 sets d min = d (step S1206) and proceeds to the process of step S1207. In step S1207, it determines whether i = N annotation_dataset (step S1207). If i ≠ N annotation_dataset (step S1207: No), the underwater target species identification device 100 increments i (i + 1) (step S1208) and returns to the process of step S1203. On the other hand, if i = N annotation_dataset (step S1207: Yes), it outputs the shortest distance data dm (d min ) (step S1209) and ends the above process.

[0079] FIG. 13 is a flowchart showing an example of the reference image selection process. FIG. 13 shows the details of the reference image selection process (step S1103 in FIG. 11) performed by the reference image data selection unit 803.

[0080] First, the underwater target species identification device 100 inputs the shortest distance data dm (d min ) and the reference image data set Rn (N ref_dataset sheets) (step S1301). Next, the underwater target species identification device 100 sets the coefficient i = 1 and the reference image Ir (I ref ) = None (initial value "none") (step S1302). Then, the underwater target species identification device 100 extracts the i-th data of the reference image data set Rn and the reference image candidate Icandidate Obtain (step S1303).

[0081] Next, the underwater object type identification device 100 is I candidate Shooting distance data d(d candidate (Step S1304) Then, the underwater object type identification device 100 determines whether the shooting distance of the shooting distance data d is shorter than the shortest distance data dm (d candidate <d min ?) is determined (step S1305).

[0082] The result of the analysis is that the shooting distance data d is shorter than the shortest distance data dm (d candidate <d min ) If (step S1305: Yes), the underwater object type identification device 100 proceeds to the process in step S1306. On the other hand, (d candidate ≥ d min If (Step S1305: No), the underwater object type identification device 100 proceeds to the process in Step S1307.

[0083] In step S1306, the underwater object type identification device 100 is I ref = I candidate d min = d candidate (Step S1306) The process then proceeds to step S1307.

[0084] In step S1307, the underwater object type identification device 100 determines whether processing has been completed for all N reference image datasets Rn (i = N ref_dataset) Determine if this is the case (step S1307). If processing of the N reference image dataset Rn is incomplete (step S1307: No, i > N). ref_dataset ), the underwater object type identification device 100 increments i (i+1) (step S1308) and returns to the process in step S1303. Meanwhile, processing is completed for all N reference image datasets Rn (i=N ref_dataset In the case of (Step S1307: Yes), the underwater object type identification device 100 uses the reference image Ir (I ref The output (step S1309) is displayed, and the above process is terminated.

[0085] Figure 14 is a flowchart showing an example of the representative image designation process for each class. Figure 14 shows the details of the representative image designation process for each class (step S1104 in Figure 11) performed by the representative image designation unit 804 for each class.

[0086] First, the underwater object type identification device 100 receives a reference image Ir (step S1401). Next, the underwater object type identification device 100 receives i=1 and class number N. class =3. Class-specific reference image I c Set to {} (step S1402), specify the region containing only class i from the reference image Ir with a rectangle, and reference image I of class i i Obtain (step S1403).

[0087] Next, the underwater object type identification device 100 is I i to I c Add to (step S1404) whether the processing for the number of classes is complete or not (i=N class) Determine this (step S1405). If processing for the number of classes is incomplete (step S1405: No, i > N). class ), the underwater object type identification device 100 increments i (i+1) (step S1406) and returns to the process in step S1403. On the other hand, if the processing for the number of classes is completed (step S1405: Yes, i=N class ), the underwater object type identification device 100 uses class-specific reference image I c Output {I1, I2, I3} (step S1407), and the above process ends.

[0088] Figure 15 is a flowchart showing an example of the patch splitting process. Figure 15 shows the details of the patch splitting process (step S1105 in Figure 11) performed by the patch splitting unit 805.

[0089] First, the underwater object type identification device 100 uses class-specific reference image I c Enter the following (step S1501). Next, the underwater object type identification device 100 determines the patch width W p = 3, height H pSet to =3 [pixels] (step S1502). Then, the underwater object type identification device 100 uses the class-specific reference image I c Each element is set to patch size [W p , H p Divide into patches (step S1503), class-specific reference patch P c The output is displayed (step S1504), and the above process is terminated.

[0090] Figure 16 is a flowchart illustrating an example of the feature extraction process. Figure 16 shows the details of the feature extraction process performed by the feature extraction unit 806 (step S1106 in Figure 11).

[0091] First, the underwater object type identification device 100 uses a class-specific reference patch P c Enter (step S1601). Next, the underwater object type identification device 100 uses the class-specific reference patch P c Each element has a size [W p , H p From the matrix of ], size W p ×H p Dimensional class-specific reference vector V c Convert to (step S1602).

[0092] Next, the underwater object type identification device 100 uses a class-specific reference vector V c Feature extraction (dimensionality reduction) is performed using UMAP, and a 2D class-specific feature quantity F is obtained. c (Step S1603) Then, the underwater object type identification device 100 obtains the class-specific feature quantity F c The output is displayed (step S1604), and the above process is terminated.

[0093] (Example of Annotation Phase Processing) Figure 17 is a flowchart showing an example of the annotation phase processing. Using Figure 17, the details of the annotation phase (step S1002) shown in Figure 10 will be explained.

[0094] The image to be annotated is, for example, a single image of 1920 x 1080 pixels, extracted from the annotated image dataset Dn. The annotated patch group described below consists of, for example, 230,400 patches of 3 x 3 pixels each. This number is obtained by dividing a 1920 x 1080 image into 3 x 3 pixels.

[0095] Furthermore, the patch to be annotated is, for example, a single 3x3 [pixels] patch, which is one patch extracted from the group of patches to be annotated. The patch annotation result is, for example, a 3x3 [pixels] patch. Each pixel in the patch has a value corresponding to the type of algae: 0 for red algae, 1 for green algae, and 2 for the background. The image annotation result is, for example, a single 1920x1080 [pixels] image, in which the pixels of red algae, green algae, and background each have type (class) identifier values ​​of 0, 1, and 2, respectively.

[0096] First, the underwater object type identification device 100 uses an annotation target image dataset Dn and class-specific feature quantities F c The data is obtained (step S1701). Next, the underwater object type identification device 100 sets the coefficient i = 1 (step S1702), retrieves the i-th data from the annotation target image dataset Dn, and obtains the annotation target image data D (step S1703).

[0097] Next, the underwater object type identification device 100 performs patch division processing from the annotation target image data D to obtain a group of annotation target patches (step S1704). Then, the underwater object type identification device 100 sets the coefficient j = 1 (step S1705), extracts the j-th data from the group of annotation target patches to obtain an annotation target patch (step S1706).

[0098] Next, the underwater object type identification device 100 identifies the annotation target patch and class-specific feature quantity F. c Then, feature comparison processing is performed to obtain patch annotation results (step S1707).

[0099] Then, the underwater object type identification device 100 determines whether the process of comparing the characteristics of the N patches is complete or not (j = N patch Determine if (step S1708). If the process of comparing the features of the number of N patches is incomplete (step S1708: No, j > N). patch ), the underwater object type identification device 100 increments j (j+1) (step S1709) and returns to the process in step S1706. On the other hand, if the process of comparing the features of the number of N patches is completed (step S1708: Yes, j=N patch ), the underwater object type identification device 100 proceeds to the process of step S1710.

[0100] In step S1710, the underwater object type identification device 100 integrates the patch annotation results obtained so far and obtains image annotation result A (step S1710).

[0101] Next, the underwater object type identification device 100 determines whether the processing of N annotation target images is complete (i = N img Determine if the processing of N images to be annotated is incomplete (Step S1711: No, i > N). img ), the underwater object type identification device 100 increments i (i+1) (step S1712) and returns to the process in step S1703. On the other hand, if the processing of N annotation target images is completed (step S1711: Yes, i=N img The underwater object type identification device 100 then integrates the image annotation results A and outputs the image annotation result group An (step S1713). The above process is then completed.

[0102] Figure 18 is a flowchart showing an example of the patch splitting process. Figure 18 shows the details of the patch splitting process (step S1704 in Figure 17) performed by the patch splitting unit 805.

[0103] First, the underwater object type identification device 100 uses the annotation target image I target Enter the following (step S1801). Next, the underwater object type identification device 100 determines the patch width Wpatch = 3, height H patch Set to =3 [pixels] (step S1802). Then, the underwater object type identification device 100 annotates the image I target Each element is set to patch size [W patch , H patch The code is divided into patches (step S1803), the patch division results are output (step S1804), and the above process is terminated.

[0104] Figure 19 is a flowchart showing an example of the feature comparison process. Figure 19 shows the details of the feature comparison process (step S1707 in Figure 17) performed by the feature comparison unit 807.

[0105] First, the underwater object type identification device 100 identifies the annotation target patch P target and class-specific features F c Enter (step S1901). Next, the underwater object type identification device 100 identifies the annotation target patch P target Each element has a size [W p , H p From the matrix of ], size W p ×H p Annotation target vector V target Convert to (step S1902).

[0106] Next, the underwater object type identification device 100 outputs the annotation target vector V target Feature extraction (dimensionality reduction) is performed using UMAP, and the 2D annotation target features F are obtained. target Obtain (step S1903). Next, the underwater object type identification device 100 identifies the annotation target feature F target Class-specific features F c Projected onto and using the k-NN method, F target Predict the class (type) and patch annotation result A patch Obtain (step S1904). Then, the underwater object type identification device 100 obtains the patch annotation result A. patch The output is displayed (step S1905), and the above process is terminated.

[0107] (About the k-NN method) Figure 20 is an explanatory diagram of the k-NN method. k-NN is an algorithm that uses majority voting and averaging to select the top k training data with the highest similarity to the input. As shown in Figure 20(a), the annotation target feature F target For each (circle in the figure), the adjacent class-specific feature F c Let's assume there is seaweed △ and a background □. In this case, as shown in Figure 20(b), when k=3, there are two seaweed △ and one background □ within region K, so the feature to be annotated F is determined by majority vote. target (〇) indicates that the class (type) is predicted to be seaweed.

[0108] The technology for identifying the type of underwater object according to the embodiment is not limited to identifying the type (class) of algae mentioned above, but can be applied to identify the type of various underwater objects. Examples of underwater objects include: 1. marine organisms, 2. underwater artificial structures, and 3. topography and geology. 1. Marine organisms include marine animals and marine plants. Examples of marine animals include corals (hard corals, soft corals, etc.), fish (clownfish, parrotfish, etc.), shellfish (scallops, abalone, etc.), crustaceans (crabs, shrimp, etc.), cephalopods (squid, octopuses, etc.), sponges (sea sponges, bone sponges, etc.), crown-of-thorns starfish, sea urchins, sea cucumbers, jellyfish, etc. Furthermore, marine plants include seaweed (wakame, garamo, kajime, red algae (tengusa, eucalyptus), green algae (anaaosa, miru), etc.), seagrasses (amagrass, sea lilies, etc.), and other organisms (wood fragments, driftwood). Also, 2. Underwater man-made structures include cracks and rusted parts of offshore wind turbines, ship wrecks, fishing nets and gear, concrete blocks, plastic waste, metal fragments, etc. 3. Topography and geology include rocks (volcanic rock, sandstone, etc.), shell deposits, sandy areas, muddy areas, seabed cracks, tunnels, etc.

[0109] In the embodiment described above, multiple annotation target images are acquired, each containing an object in the sea. The annotation target image with the shortest shooting distance among the shooting distances of the objects in each of the multiple annotation target images is identified. A reference image with a shorter shooting distance than the identified annotation target image is selected from a set of reference images containing objects in the sea that have been stored in advance. The type of object contained in the annotation target image is then identified using the selected reference image. This improves the accuracy of determining the type of object in the sea. Reference images include images with long shooting distances and are greatly affected by turbidity in the sea, so if a reference image with a long shooting distance is used, the accuracy of determining the type of object will be low. In contrast, according to this embodiment, by using a reference image with a shorter shooting distance than the identified annotation target image to determine the type of object contained in the annotation target image, it is possible to use the clearest possible reference image, and the accuracy of determining the type of object in the sea can be improved without being affected by turbidity in the sea.

[0110] Furthermore, in this embodiment, class-specific reference images are generated based on the selected reference image, indicating regions of different object types. Based on the generated class-specific reference images, class-specific features are extracted from the reference image. The process for identifying the object type involves obtaining multiple annotation target images and class-specific features from the reference image. Class-specific features are extracted from the multiple annotation target images and compared with the class-specific features from the reference image to identify the object type contained in the annotation target image. In this way, by using reference images of class-specific features, the class (type) of the object can be easily identified through a comparison process using features.

[0111] Furthermore, in this embodiment, the process of generating class-specific reference images involves specifying a representative image for each class that indicates a region containing only different types of objects in the reference image. By using reference images with specified representative images in this way, it becomes possible to easily and accurately identify the class (type) of an object using its features.

[0112] Furthermore, in this embodiment, the process of extracting class-specific features from the reference image and the annotation target image includes the process of dividing the selected reference image and the annotation target image into patches and extracting class-specific features for each patched region. This makes it possible to accurately identify the class (type) of the object at the fine patch level obtained by dividing the image.

[0113] Furthermore, in this embodiment, the process for identifying the type of object includes outputting a display image that identifies the type of object in relation to the class-specific feature quantities for the annotated image. This makes it easy to identify the class (type) of object included in the annotated image.

[0114] The method for identifying the type of underwater object described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a PC or workstation. The program for identifying the type of underwater object described in this embodiment is recorded on a computer-readable recording medium and executed by reading it from the recording medium by the computer. Furthermore, the program for identifying the type of underwater object described in this embodiment may be distributed via a network such as the Internet.

[0115] 100 Underwater object type identification device 200 Marine digital twin 302, 303 Camera 801 Acquisition unit 802 Shortest distance identification unit 803 Reference image data selection unit 804 Representative image designation unit for each class 805 Patch division unit 806 Feature extraction unit 807 Feature comparison unit 808 Output unit A (An) Annotation results for each image (group) D (Dn) Annotation target image data (dataset) Fc Class-specific features NW Network R (Rn) Reference image data (dataset) d Shooting distance (data) dm Shortest distance (data)

Claims

1. A method for identifying the type of object in the ocean, characterized in that a computer performs the following processes: acquires multiple annotation target images in which images of objects in the ocean are taken; identifies the annotation target image with the shortest shooting distance among the shooting distances of the objects in each of the multiple annotation target images; selects a reference image from a plurality of reference images in which objects in the ocean are taken that has a shooting distance shorter than the shooting distance of the identified annotation target image; and uses the selected reference image to identify the type of object included in the annotation target image.

2. The method for identifying the type of object in the sea according to claim 1, characterized in that, based on the selected reference image, class-specific reference images are generated showing regions of different types of objects, and class-specific feature quantities are extracted from the reference image based on the generated class-specific reference image to identify the type of object, the method for identifying the type of object in the sea according to claim 1, characterized in that, a plurality of annotation target images and class-specific feature quantities of the reference image are obtained, class-specific feature quantities are extracted from the plurality of annotation target images and compared with the class-specific feature quantities of the reference image to identify the type of object included in the annotation target image.

3. The method for identifying types of underwater objects according to claim 2, characterized in that the process for generating class-specific reference images includes specifying a representative image for each class that shows a region containing only different types of objects from the reference image.

4. The method for identifying the type of underwater object according to claim 2, characterized in that the process for extracting class-specific features from the reference image and the annotation target image includes the process of dividing the selected reference image and the annotation target image into patches and extracting class-specific features for each of the patched regions.

5. The method for identifying the type of object in the sea according to claim 2, characterized in that the process for identifying the type of object includes a process for outputting a display image that can identify the type of object in correspondence with the class-specific feature quantities for the annotation target image.

6. A program for identifying the type of object in the ocean, characterized by having a computer perform the following processes: acquiring multiple annotation target images that include objects in the ocean; identifying the annotation target image with the shortest shooting distance among the shooting distances of the objects in each of the multiple annotation target images; selecting a reference image from a plurality of reference images that include objects in the ocean and have a shooting distance shorter than the shooting distance of the identified annotation target image; and using the selected reference image to identify the type of object included in the annotation target image.

Images