Undersea object type-identification method and undersea object type-identification program

Abstract

An undersea object type-identification device (100) acquires an annotation target image (D) obtained by capturing an image including an undersea object, divides the annotation target image (D) into a plurality of segments (Is), each segment including pixels having similar features, and extracts a feature amount (Fs) of each of the plurality of segments (Is). The undersea object type-identification device (100) also extracts feature amounts (Fc) of a plurality of reference images (Ir) including undersea objects and held in advance, compares the segment-specific feature amounts (Fc) with the feature amounts (Fs) of the reference images, and outputs an image annotation result (As) specifying a type of an object included in the annotation target image (D).

Description

Method and program for identifying types of underwater objects

[0001] This invention relates to a method for identifying the type of object in the sea and a program for identifying the type of object in the sea.

[0002] To address global social challenges and create new businesses, a technology called the Ocean Digital Twin has been proposed, which precisely digitizes all aspects of the ocean (surface, underwater, and seabed) and allows for the planning and preliminary verification of targeted measures in a digital space. One application of the Ocean Digital Twin is in blue carbon initiatives, where the increased CO2 absorption due to seaweed bed creation is converted into credits. To quantify CO2 absorption, a supervised dataset is essential for training an image AI that recognizes seaweed beds, and an annotation method for efficiently creating such a dataset is necessary.

[0003] Conventionally, annotation methods have been proposed in which the image to be annotated is compared with a reference image for each patch division, and a class determination is made for each type of object to determine whether the object in each pixel is of a specific type, and the determination result is output as the image annotation result.

[0004] For example, there is a technique that uses a neural network to predict whether a disease is present in biological images at the patch division unit and pixel level. There is also a technique that processes medical images to be annotated using a trained lesion area detection model, frames the lesion areas in the medical images to be annotated, and annotates the lesion type of the lesion area. There is also a technique that estimates the amount of CO2 absorbed by seaweed by analyzing image data obtained by photographing seaweed from a position different from above, capturing the side view of the seaweed. Furthermore, there is a technique that crawls images of selected algal species, classifies and labels the algae within the images, and then uses a neural network to identify and classify the algae in the resulting dataset (see, for example, Patent Documents 1 to 4 below).

[0005] Japanese Patent Publication No. 2023-535338, U.S. Patent Application Publication No. 2019 / 0096060, Japanese Patent Publication No. 2024-125623, U.S. Patent Application Publication No. 2024 / 0087310

[0006] When viewing the entire image to be annotated, multiple types of algae and areas with and without algae are present, making it impossible to perform the desired annotation. For this reason, one possible approach is to divide the image to be annotated into small patches, extract features, and compare them with the features of a reference image to determine the type (class) of algae at the patch level, outputting this as the annotation result. However, when class determination is performed on a patch-by-patch basis, the accuracy of class determination decreases. If algae class determination is performed using only the features of each patch, the entire image to be annotated is not considered, and therefore, class misdetermination can occur due to the influence of noise such as water turbidity or near boundaries (the boundary between algae and the background).

[0007] In one aspect, the present invention aims to improve the accuracy of determining the type of object in the sea.

[0008] According to one embodiment, a method and program for identifying the type of object in the ocean are proposed, which include the following steps: acquiring an annotation target image by taking an image containing an object in the ocean; dividing the annotation target image into multiple segments, with pixels having similar features forming one segment; extracting feature quantities from each of the multiple segments; extracting feature quantities from multiple reference images containing objects in the ocean that have been stored in advance; comparing the feature quantities of each segment with the feature quantities of the reference images; and identifying the type of object contained in the annotation target image.

[0009] According to one embodiment, the accuracy of determining the type of object in the sea can be improved.

[0010] Figure 1 is an explanatory diagram of the underwater object type identification process according to the embodiment. Figure 2 is an explanatory diagram of the ocean digital twin. Figure 3 is an explanatory diagram of a processing example in which underwater object identification is applied to the ocean digital twin. Figure 4 is an explanatory diagram of the overview processing of image annotation obtained by underwater object type identification. Figure 5 is a diagram explaining the problems caused by the background technology. (Part 1) Figure 6 is a diagram explaining the problems caused by the background technology. (Part 2) Figure 7 is an explanatory diagram of the types according to the embodiment. Figure 8 is a functional block diagram of the underwater object type identification device according to the embodiment. Figure 9 is a diagram showing an example of the hardware configuration of the underwater object type identification device. Figure 10 is an overall flowchart showing an example of the underwater object type identification process. Figure 11 is a flowchart showing an example of the processing of the class-specific feature construction phase. Figure 12 is a flowchart showing an example of the shortest distance identification process. Figure 13 is a flowchart showing an example of the reference image selection process. Figure 14 is a flowchart showing an example of the representative image specification process for each class. Figure 15 is a flowchart showing an example of the patch splitting process. Figure 16 is a flowchart showing an example of the feature extraction process. Figure 17A is a flowchart showing an example of the annotation phase. (Part 1) Figure 17B is a flowchart showing an example of the annotation phase. (Part 2) Figure 18 is a flowchart showing an example of the segmentation process. Figure 19 is a flowchart showing an example of the patch splitting process. Figure 20 is a flowchart showing an example of the feature extraction process. Figure 21 is a flowchart showing an example of the segment representative value selection process. Figure 22 is a flowchart showing an example of the feature comparison process. Figure 23 is an explanatory diagram of the k-NN method.

[0011] The embodiments of the underwater object type identification method and underwater object type identification program according to the present invention will be described in detail below with reference to the drawings.

[0012] (An Example of Identifying Types of Underwater Objects According to an Embodiment) Figure 1 is an explanatory diagram of the process for identifying types of underwater objects according to an embodiment. The method for identifying types of underwater objects in this embodiment identifies and presents, for example, the types (classes) of various underwater organisms, underwater artificial structures, topography, and geology present in the sea. In this embodiment, algae (seaweed and seagrass) are used as an example of underwater objects. For example, by identifying the types of algae, the amount of CO2 absorbed by creating seaweed beds in the sea can be quantified and applied to blue carbon policies such as converting the amount of CO2 absorbed into credits.

[0013] In the embodiment of the method for identifying the type of underwater object, for example, a supervised dataset is generated for training an image AI that recognizes seaweed beds based on images taken underwater. Then, annotation processing is performed on the images to add information about the type of algae using tags, metadata, etc. Hereinafter, images taken underwater will be referred to as annotation target images.

[0014] In this embodiment, for example, a process is performed to identify the type (class) of algae contained in an annotation target image taken underwater. In this identification process, the annotation target image is divided into multiple segments based on image features, such as the features of pixels. A segment is a grouping of pixels that have similar features (color similarity or proximity of position) from one of the images into which the image has been divided into several chunks. Then, one feature information corresponding to each segment is selected from the features extracted from each segment, and the type (class) of algae contained in the annotation target image is identified by comparing the selected feature information with a reference image. In this embodiment, the accuracy of the determination of algae type is improved by performing algae class determination on a segment-by-segment basis.

[0015] The method for identifying the type of underwater object in this embodiment involves an underwater object identification device 100, such as a server, acquiring various necessary information and executing a program to perform the process of identifying the type of algae. The underwater object identification device 100 shown in Figure 1 performs the following: A: Class-specific feature construction phase processing and B: Annotation phase processing.

[0016] The reference image Ir is, for example, an image taken of the same region as the annotation target image D, and may have multiple datasets. In this embodiment, the reference image Ir and the annotation target image D are color (RGB) images showing various algae (red algae, green algae) and the background.

[0017] (A: Processing in the Class-Specific Feature Construction Phase) In the Class-Specific Feature Construction Phase, the class-specific features Fc obtained from the pre-prepared reference image Ir through the patch splitting A1 and feature extraction A2 processes are passed to the B: Annotation Phase.

[0018] (A1) In the patch division process, the reference image Ir is divided into, for example, a group of 3x3 [pixels] patches.

[0019] (A2) In the feature extraction process, feature extraction is performed on each of the patched reference images Ir. Feature extraction is performed using, for example, the dimensionality reduction method (horizontal axis: UMAP1, vertical axis: UMAP2) using UMAP (Uniform Manifold Application and Projection). UMAP is one of the data analysis methods called dimensionality reduction, which compresses ARI, high-dimensional data into a low-dimensional space, making it easier to visualize the structure and patterns of the data. Alternatively, it can also be performed using the CNN (Convolutional Neural Network) method.

[0020] These patch divisions and feature extraction processes yield, for example, the class-specific feature vectors Fc shown in Figure 1. In Figure 1, the feature vectors of the types (classes) of algae to be recognized are mapped to each class and shown with different shaded lines, etc., as the class-specific feature vectors Fc of UMAP. The class-specific feature vectors Fc are output to the B: annotation phase (B6: feature comparison with representative values).

[0021] (B: Processing in the annotation phase) (B1) Obtain annotation target image data D taken from underwater. The annotation target image data D has multiple n datasets Dn.

[0022] (B2) Each annotation target image data D of the annotation target image dataset Dn is subjected to segmentation processing to divide the entire image into multiple segments Is. In the segmentation processing, for example, the annotation target image data D is divided into multiple segments Is, which are formed by grouping multiple pixels with similar features into one segment, based on the features of the image (such as the proximity of pixel values ​​and positions between pixels). Figure 1 shows each segment region with different diagonal lines for each segment Is.

[0023] (B3) After segment processing, the image D to be annotated is divided into patches for each segment region. Patch division is done by dividing the image into groups of 3x3 [pixels] patches, for example.

[0024] (B4) Feature extraction is performed on each patch image of the annotation target image D after patch division. Feature extraction is performed using the same process as feature extraction on the reference image Ir, for example, using the UMAP method.

[0025] Through patch splitting and feature extraction, for example, the segment-specific feature quantities Fs shown in Figure 1 are obtained. In Figure 1, each segment-specific feature quantity Fs is indicated by different shaded lines, etc.

[0026] (B5) Select a representative value (segment-specific representative patch) Pr for each segment. For example, the centroid of each segment's feature Fs is calculated, and the sample closest to that centroid is selected as the segment's representative value Pr. In Figure 1, the representative values ​​Pr for each segment are indicated by an "x".

[0027] (B6) The representative value Pr of each segment is compared with the class-specific feature quantity Fc obtained in the processing of Phase A: Class-Specific Features. The label of the most similar class is assigned to the entire segment Is for all representative values ​​Pr, thereby generating training data for the entire image (image annotation result As). This obtains image annotation result As that identifies the types of objects (algae) contained in the annotated image dataset Dn. Image annotation result As consists of multiple images (image annotation result group Asn) corresponding to the annotated image dataset Dn, and each image annotation result As is an image with a color that can identify the type of algae (the white area in Figure 1 is the background).

[0028] (Example of application to marine digital twins) Figure 2 is an explanatory diagram of a marine digital twin. The identification of underwater object types in this embodiment is applied, for example, to the marine digital twin 200. The marine digital twin 200 enables the effective utilization of the ocean to solve global social issues and create new businesses. In the marine digital twin 200, all states of the ocean (surface, underwater, and seabed) are precisely digitized in a digital space, and measures tailored to the purpose are planned and pre-verified.

[0029] As shown in Figure 2, various measures will be pre-verified by performing the following processing on the real-world underwater environment W where target objects O such as marine organisms and structures exist, using the ocean digital twin 200.

[0030] 1. Ocean data Dw is collected using underwater 3D measurement technology that captures the movement of object O in real time. Ocean data Dw is digital data that accurately reproduces the ocean by using image enhancement AI technology that converts underwater objects into highly accurate 3D data.

[0031] Then, 2. Using the collected ocean data Dw, create an ocean model M, which is a model of the environment, organisms, and their growth of the objects that make up the ocean.

[0032] 3. Using the modeled ocean data Dw, various measures are pre-verified through simulation. These measures include blue carbon (corresponding to the processing of the embodiment), environmental assessment, offshore wind power generation maintenance, etc. 4. The prescribed measures are then implemented for the actual underwater W in the real world.

[0033] Figure 3 is an explanatory diagram illustrating an example of processing applied to the identification of underwater objects in a marine digital twin. Using Figure 3, an example of the overall system processing, including the identification of underwater object types, will be explained. A camera 302 attached to an AUV (Autonomous Underwater Vehicle) 301 captures an underwater image (the annotated image above) D containing the underwater object, which is algae. In addition, a camera 303 attached to an UAV (Unmanned Aerial Vehicle) captures a satellite / aerial image G containing the area of ​​the underwater object.

[0034] The underwater image (the image to be annotated above) D is processed through high-precision seaweed bed recognition (S301), seaweed bed area measurement (S302), and mapping to satellite / aerial image G (S303) to obtain a high-precision recognition result Da. The high-precision recognition result Da includes information on the type (class) of algae E1 in the created seaweed bed.

[0035] Furthermore, the satellite / aerial image G is processed using wide-area seaweed bed recognition (S304) to obtain a wide-area recognition result Ga. The wide-area recognition result Ga includes information on the area E2 of the created seaweed bed.

[0036] The high-precision recognition result Da and the wide-area recognition result Ga are processed in the seaweed bed recognition result integration (S305) to obtain an integrated result (wide-area and high-precision recognition result) Ta. The integrated result (wide-area and high-precision recognition result) Ta includes information indicating the distribution state E3 of the algal species (class) E2 in the created seaweed bed area E1. For example, the distribution state E3 is displayed in different colors corresponding to the algal species (class).

[0037] Then, based on the information showing the distribution state E3, the CO2 absorption amount calculation (S306) process is performed to obtain the CO2 absorption amount calculation result Tb in the created seaweed bed area E1. For example, the CO2 absorption amount calculation result Tb displays the amount of CO2 absorbed (annual CO2 absorption) by type (class) of algae E2 in the created seaweed bed area E1 in different colors.

[0038] The series of processes shown in Figure 3 implements blue carbon measures, which are a candidate application for marine digital twins. In order to convert the increased CO2 absorption amount due to seaweed bed creation into credits, a supervised dataset is essential for training an image AI that recognizes seaweed beds in order to quantify the amount of CO2 absorption. An automated annotation method is needed to efficiently create this supervised dataset. Accordingly, the process related to the identification of underwater objects performed by the embodiment (underwater object type identification device 100 in Figure 1) is included in the high-precision seaweed bed recognition process in S301. The process in S301 outputs the image annotation result As shown in Figure 1.

[0039] In the overall system processing shown in Figure 3, the underwater object type identification device 100 of the embodiment may not be limited to the process of outputting the image annotation result As (image annotation result group Asn), but may also include and perform some or all of the other processes (S301 to S306). Furthermore, distributed processing including multiple servers and even the cloud may be performed.

[0040] (Example of image annotation results) Figure 4 is an explanatory diagram of the overview processing of image annotation obtained by identifying the type of underwater object. The underwater object identification device 100 refers to the reference image Ir and performs automatic annotation on the acquired annotation target image data D to obtain the image annotation result As.

[0041] The image annotation result As shown in Figure 4 is an image containing the area of ​​the created seaweed bed, with the types (classes) of algae—seaweed species A (red algae), seaweed species B (green algae)—and the background colored distinctly. Image annotation result As allows for the identification of areas of each type of algae within the seaweed bed.

[0042] (Challenges due to background technology) Figures 5 and 6 illustrate the challenges due to background technology. As a prerequisite technology to the embodiment, the processing example shown in Figure 5 can be considered. In the processing example shown in Figure 5, it is assumed that in the processing of A: reference image Ir in the class-specific feature construction phase and B: image to be annotated D in the annotation phase, patch splitting (A1, B1) is performed in both cases. Then, after patch splitting, feature extraction (A2, B2) is performed, and annotation processing is carried out on the image data to be annotated D by referring to the class-specific feature Fc, and the image annotation result As is obtained.

[0043] In this case, the algae class is determined based only on the characteristics of each patch, and the entire image of the annotated image is not considered. As a result, as shown in Figure 6(a), it becomes difficult to recognize the turbidity of the water and the boundaries (boundaries between the algae and the background) contained in the annotated image D. In the image annotation result As, multiple classes are mixed together in the turbidity of the water and near the boundaries (boundaries between the algae and the background), resulting in a large amount of noise N and class misclassification. In this case, the accuracy of determining the type of object (algae) contained in the image annotation result As decreases.

[0044] In contrast, the above-described embodiment solves the above problem by performing the above processing (processing (A1), (A2), and especially (B1) to (B6) in Figure 1). As in the embodiment, the annotation target image data D is divided into multiple segments based on the image features. Then, one feature information corresponding to each segment is selected from the features extracted from each segment, and the selected feature information is compared with the feature information of the reference image. As a result, as shown in Figure 6(b), the image annotation result As can suppress noise such as water turbidity and noise near boundaries (boundaries between algae and background), thereby improving the accuracy of determining the type of algae.

[0045] (Regarding types (classes)) Figure 7 is an explanatory diagram of the types according to the embodiment. Figure 7 shows an example of displaying class-specific feature quantities Fc by UMAP, with the vertical axis (UMAP2) showing feature quantity 1 after dimensionality reduction and the horizontal axis (UMAP1) showing feature quantity 2 after dimensionality reduction. In the embodiment, the type (class) refers to the category of the object to be classified. If the object is algae, it is classified into the following three types (classes): 1. Seaweed species A (red algae), 2. Seaweed species B (green algae), 3. Background.

[0046] (Example of Functions of Underwater Object Type Identification Device) Figure 8 is a functional block diagram of an underwater object type identification device according to an embodiment. An example of the functions of the underwater object type identification device 100 that performs the processing shown in Figure 1 will be explained. The underwater object type identification device 100 includes the functions of an acquisition unit 801, a segmentation unit 802, a patch division unit 803, a feature extraction unit 804, a segment representative value selection unit 805, a feature comparison unit 806, and an output unit 807. Figure 8 shows each function corresponding to the processing shown in Figure 1, with the upper section showing the functions corresponding to each process in the A: Class-Specific Feature Construction Phase, and the lower section showing the functions corresponding to each process in the B: Annotation Phase.

[0047] The acquisition unit 801 acquires an annotation target image dataset Dn, which consists of images taken underwater. The acquisition unit 801 acquires multiple underwater images (annotation target image data D) taken by a camera 302 attached to the AUV shown in Figure 3 as a dataset Dn. The acquisition unit 801 also acquires a reference image Ir. In the functional example shown in Figure 8, the acquisition unit 801 acquires the reference image Ir held in the recording unit of the underwater object type identification device 100.

[0048] Furthermore, these reference image Ir and annotation target image data D include shooting distance data d. Shooting distance data d can be obtained by, for example, estimating distance information from the image using a DNN (Deep Neural Network). Alternatively, it can be obtained using measuring instruments other than cameras, such as LiDAR (Light Detection and Ranging), SONAR (Sound Navigation and Ranging), and DVL (Dopple Velocity Log).

[0049] The segmentation unit 802 divides the entire image into multiple segments Is based on the characteristics of the annotation target image data D, such as the proximity of pixel values ​​and positions between pixels. Segmentation methods include using a pre-trained model such as SAM (Segmentation Anything Model) or self-learning from multiple annotation target images.

[0050] The patch splitting unit 803 performs patch splitting on each image of the reference image Ir and the annotation target image dataset Dn. A: The patch splitting unit 803a on the class-specific feature construction phase side performs patch splitting on the reference image Ir, and B: The patch splitting unit 803b on the annotation phase side performs patch splitting on the annotation target image data D.

[0051] Here, the patch division unit 803b on the annotation phase side (B) divides the annotation target image D into patches for each segment region. For this reason, each time a patch division is performed, the patch division unit 803b acquires the annotation target image D containing the segment region from the acquisition unit 801 and divides it into patches.

[0052] In the functional example shown in Figure 8, patch splitting units 803a and 803b are placed on the A: class-specific feature construction phase side and the B: annotation phase side, respectively. However, the system is not limited to this configuration; a single patch splitting unit 803 may be used to perform patch splitting on both the A: class-specific feature construction phase side and the B: annotation phase side.

[0053] The feature extraction unit 804 performs class-specific feature extraction on the batch-divided reference image Ir and the annotation target image data D. Feature extraction is performed, for example, by dimensionality reduction using UMAP (horizontal axis: UMAP1, vertical axis: UMAP2) or by processing using CNN (Convolutional Neural Network). As a result of these patch divisions and feature extractions, segment-specific feature quantities Fs (two-dimensional class-specific feature quantities) are obtained.

[0054] In the functional example shown in Figure 8, feature extraction units 804a and 804b are located on the A: class-specific feature construction phase side and the B: annotation phase side, respectively. However, the functionality is not limited to this configuration; a single feature extraction unit 804 may perform both the feature extraction on the A: class-specific feature construction phase side and the B: annotation phase side.

[0055] The segment representative value selection unit 805 selects a representative value Pr for each segment. For example, the representative value is determined by calculating the centroid of each segment's feature quantity Fs and selecting the sample closest to that centroid as the segment's representative value (segment-specific representative patch) Pr.

[0056] The feature comparison unit 806 compares the representative value Pr of each segment with the class-specific feature quantity Fc obtained in the processing of A: Class-Specific Feature Phase A. The feature comparison unit 806 generates training data for the entire image (image annotation result As) by performing a process on all representative values ​​Pr to assign the label of the most similar class to the entire segment Is. This feature comparison is determined by a process using, for example, the k-NN (k-Nearest Neighbors) algorithm. This determination process is performed on all patches. As a result, an image annotation result As is obtained that identifies the type of object (algae) contained in the annotation target image dataset Dn.

[0057] The output unit 807 outputs the image annotation result group Asn. The output unit 807 outputs the image annotation result group Asn to the functional unit of the later stage of the system in Figure 3 (seaweed bed area measurement S302).

[0058] (Example Hardware Configuration of Underwater Object Type Identification Device) Figure 9 shows an example hardware configuration of an underwater object type identification device. In Figure 9, the underwater object type identification device 100 can use, for example, a general-purpose server or terminal PC. The underwater object type identification device 100 includes a CPU (Central Processing Unit) 901 such as a processor, memory 902, network IF 903, recording medium IF 904, and recording medium 905. Each component is connected by a bus 900.

[0059] Here, the CPU 901 is a control unit that oversees the overall control of the underwater object type identification device 100. The CPU 901 may have multiple cores. The memory 902 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and flash ROM. Specifically, for example, the flash ROM stores the OS program, the ROM stores the application program, and the RAM is used as the work area for the CPU 901. The program stored in the memory 902 is loaded into the CPU 901, causing the CPU 901 to execute the coded processing.

[0060] The Network IF903 is connected to a network NW via a communication line and can connect to an external computer via the network NW. The Network IF903 manages the interface between the network NW and the device's internal components, controlling the input and output of data from the external computer. The Network IF903 can utilize, for example, a modem or a LAN adapter.

[0061] The recording medium IF 904 controls the reading and writing of data to the recording medium 905 according to the control of the CPU 901. The recording medium 905 stores the written data. Examples of recording media 905 include magnetic disks such as HDDs, optical disks, SSDs (Solid State Drives), and USB (Universal Serial Bus) memory.

[0062] In addition to the components described above, the underwater object type identification device 100 may also have, for example, an input device, a display, and the like.

[0063] Each function of the underwater object type identification device 100 shown in Figure 8 can be realized by programming the CPU 901 shown in Figure 9. The memory 902 and recording medium 905 shown in Figure 9 are used as storage units for the information handled by each of the functional units (acquisition unit 801 to output unit 807) in Figure 8. In addition, the recording medium 905 shown in Figure 9 may pre-record and store the dataset Irn of the reference image Ir shown in Figure 8.

[0064] (Example of process for identifying the type of object underwater) Figure 10 is an overall flowchart showing an example of the process for identifying the type of object underwater. An example of the process performed by the control unit (CPU 901) of the underwater object type identification device 100 will be explained.

[0065] The underwater object type identification device 100 includes A: processing in the class-specific feature construction phase (step S1001) and B: processing in the annotation phase (step S1002).

[0066] In the class-specific feature construction phase (step S1001), class-specific features Fc are calculated based on the reference image dataset Irn and the annotation target image dataset Dn.

[0067] In the annotation phase (step S1002), the image annotation result set Asn is determined based on the image dataset Dn to be annotated and the class-specific feature vector Fc.

[0068] (Example of processing in the class-specific feature construction phase) Figure 11 is a flowchart showing an example of processing in the class-specific feature construction phase. Using Figure 11, the details of the processing in the class-specific feature construction phase (step S1001) shown in Figure 10 will be explained.

[0069] First, the underwater object type identification device 100 acquires an annotation target image dataset Dn and a reference image dataset Irn (step S1101). Next, the underwater object type identification device 100 performs a shortest distance identification process from the annotation target image dataset Dn to obtain the shortest distance data dm (step S1102).

[0070] Next, the underwater object type identification device 100 performs a reference image selection process from the shortest distance data dm and the reference image dataset Irn to obtain a reference image Ir (step S1103). Next, the underwater object type identification device 100 performs a representative image designation process for each class from the reference image Ir to obtain a class-specific reference image Ic (step S1104).

[0071] Next, the underwater object type identification device 100 performs patch division processing from the class-specific reference image Ic to obtain class-specific reference patches (step S1105). Then, it performs feature extraction processing from the class-specific reference patches to output class-specific feature quantities Fc (step S1106).

[0072] The annotation target image dataset Dn consists of, for example, 845 images of 1920 x 1080 pixels, each containing a mixture of red algae, green algae, and background. The reference image dataset Irn consists of, for example, 10 images of 1920 x 1080 pixels, each containing a mixture of red algae, green algae, and background.

[0073] The class-specific feature Fc is, for example, a table consisting of 3 classes × 100 patches × 2 feature vectors (see Figure 7). The image annotation result set Asn consists of, for example, 845 images of 1920 × 1080 pixels, and each image contains the type (class) identifier values ​​0, 1, and 2 for the pixels representing red algae, green algae, and background, respectively.

[0074] The shooting distance data d is, for example, the distance to the center of an image taken of the same area as the annotation target image data D or the reference image data R. Each pixel of the depth image stores a distance [m], and the shooting distance data d represents the distance [m] stored in the center of the depth image. The shortest distance data dm is the smallest distance [m] among the shooting distance data d corresponding to each image D in the annotation target image dataset Dn.

[0075] Furthermore, the reference image Ir is, for example, one image of 1920 x 1080 pixels, but there may be two or more. Also, the class-specific reference image Ic is, for example, three images of 300 x 300 pixels, consisting of, for example, an image of only red algae, an image of only green algae, and an image of only the background. Also, the class-specific reference patch is, for example, three sets of 100 patches of 3 x 3 pixels each, consisting of, for example, a set of red algae, a set of green algae, and a set of backgrounds.

[0076] FIG. 12 is a flowchart illustrating a processing example of the shortest distance specifying process. FIG. 12 shows details of the shortest distance specifying process (step S1102 in FIG. 11).

[0077] First, the underwater object type identification apparatus 100 generates an annotation target image dataset Dn (N annotation_dataset images) (step S1201). Next, the underwater object type identification apparatus 100 sets a coefficient i=1, and shortest distance data dm (d min )=∞ (step S1202), extracts the i-th data from the annotation target image dataset Dn, and obtains annotation target image data D (step S1203).

[0078] Next, the underwater object type identification apparatus 100 obtains shooting distance data d corresponding to the annotation target image data D (step S1204). Then, the underwater object type identification apparatus 100 compares the shooting distance data d with the shortest distance data dm (step S1205: d<d min ?).

[0079] As a comparison result, when d<d min (step S1205: Yes), the underwater object type identification apparatus 100 proceeds to the processing of step S1206, and when d≧d min (step S1205: No), the process proceeds to step S1207.

[0080] In step S1206, the underwater object type identification apparatus 100 sets d min =d (step S1206), and proceeds to the processing of step S1207. In step S1207, it is determined whether or not i=N annotation_dataset holds (step S1207). If i=N annotation_dataset does not hold (step S1207: No), the underwater object type identification apparatus 100 increments i by 1 (i+1) (step S1208), and returns to the processing of step S1203. On the other hand, if i=N annotation_dataset holds (step S1207: Yes), the shortest distance data dm (d min ) is output (step S1209), and the above processing is terminated.

[0081] Figure 13 is a flowchart showing an example of the reference image selection process. Figure 13 shows the details of the reference image selection process (step S1103 in Figure 11).

[0082] First, the underwater object type identification device 100 uses the shortest distance data dm(d min ) and the reference image dataset Irn(N ref_dataset Input (number of sheets) (step S1301). Next, the underwater object type identification device 100 determines the coefficient i = 1, reference image Ir(I ref ) = None (initial value "none") (step S1302). Then, the underwater object type identification device 100 takes out the i-th data of the reference image dataset Irn and selects the reference image candidate I candidate Obtain (step S1303).

[0083] Next, the underwater object type identification device 100 is I candidate Shooting distance data d(d candidate (Step S1304) Then, the underwater object type identification device 100 determines whether the shooting distance of the shooting distance data d is shorter than the shortest distance data dm (d candidate <d min ?) is determined (step S1305).

[0084] The result of the analysis is that the shooting distance data d is shorter than the shortest distance data dm (d candidate <d min ) If (step S1305: Yes), the underwater object type identification device 100 proceeds to the process in step S1306. On the other hand, (d candidate ≥ d min If (Step S1305: No), the underwater object type identification device 100 proceeds to the process in Step S1307.

[0085] In step S1306, the underwater object type identification device 100 is I ref = I candidate d min = d candidate (Step S1306) The process then proceeds to step S1307.

[0086] In step S1307, the underwater object type identification device 100 determines whether processing has been completed for all N reference image datasets Irn (i = N). ref_dataset) Determine if this is the case (step S1307). If processing of the N reference image dataset Irn is incomplete (step S1307: No, i > N). ref_dataset ), the underwater object type identification device 100 increments i (i+1) (step S1308) and returns to the process in step S1303. Meanwhile, processing is completed for all N reference image datasets Irn (i=N ref_dataset In the case of (Step S1307: Yes), the underwater object type identification device 100 uses the reference image Ir (I ref The output (step S1309) is displayed, and the above process is terminated.

[0087] Figure 14 is a flowchart showing an example of the representative image designation process for each class. Figure 14 shows the details of the representative image designation process for each class (step S1104 in Figure 11).

[0088] First, the underwater object type identification device 100 receives a reference image Ir (step S1401). Next, the underwater object type identification device 100 receives i=1 and class number N. class =3. Class-specific reference image I c Set to {} (step S1402), specify the region containing only class i from the reference image Ir with a rectangle, and reference image I of class i i Obtain (step S1403).

[0089] Next, the underwater object type identification device 100 is I i to I c Add to (step S1404) whether the processing for the number of classes is complete or not (i=N class) Determine this (step S1405). If processing for the number of classes is incomplete (step S1405: No, i > N). class ), the underwater object type identification device 100 increments i (i+1) (step S1406) and returns to the process in step S1403. On the other hand, if the processing for the number of classes is completed (step S1405: Yes, i=N class ), the underwater object type identification device 100 uses class-specific reference image Ic Output {I1, I2, I3} (step S1407), and the above process ends.

[0090] Figure 15 is a flowchart showing an example of the patch splitting process. Figure 15 shows the details of the patch splitting process (step S1105 in Figure 11).

[0091] First, the underwater object type identification device 100 uses class-specific reference image I c Enter the following (step S1501). Next, the underwater object type identification device 100 determines the patch width W p = 3, height H p Set to =3 [pixels] (step S1502). Then, the underwater object type identification device 100 uses the class-specific reference image I c Each element of the patch size [W p , H p Divide into patches (step S1503), class-specific reference patch P c The output is displayed (step S1504), and the above process is terminated.

[0092] Figure 16 is a flowchart illustrating an example of the feature extraction process. Figure 16 shows the details of the feature extraction process (step S1106 in Figure 11).

[0093] First, the underwater object type identification device 100 uses a class-specific reference patch P c Enter (step S1601). Next, the underwater object type identification device 100 uses the class-specific reference patch P c Each element has a size [W p , H p From the matrix of ], size W p ×H p Dimensional class-specific reference vector V c Convert to (step S1602).

[0094] Next, the underwater object type identification device 100 uses a class-specific reference vector V c Feature extraction (dimensionality reduction) is performed using UMAP, and a 2D class-specific feature quantity F is obtained. c (Step S1603) Then, the underwater object type identification device 100 obtains the class-specific feature quantity F cThe output is displayed (step S1604), and the above process is terminated.

[0095] (Example of Annotation Phase Processing) Figures 17A and 17B are flowcharts showing examples of annotation phase processing. Using Figures 17A and 17B, the details of the annotation phase (step S1002) shown in Figure 10 will be explained.

[0096] Here, the annotated image data D is, for example, a single image of 1920 x 1080 pixels, extracted from the annotated image dataset Dn. Furthermore, the segmentation result described below is, for example, a single image of 1920 x 1080 pixels with N segments, where the image contains the values ​​0, 1, ..., N-1 for pixels with segment numbers 1, 2, ..., N, respectively.

[0097] The segment-specific patch group consists of, for example, 230,400 3x3 [pixels] patches. This number is obtained by dividing a 1920x1080 [pixels] area into 3x3 [pixels] patches. The segment-specific feature quantity Fs is, for example, a table with 230,400 feature vectors × 2 elements. Each feature vector is assigned a segment number. The segment-specific representative patch group consists of N 3x3 [pixels] patches. The segment-specific representative patch (Pr) is one 3x3 [pixels] patch.

[0098] Furthermore, the segment annotation results are scalars, and for example, they contain values ​​corresponding to the type of algae: 0 for red algae, 1 for green algae, and 2 for the background. The image annotation result As is, for example, a single image of 1920 x 1080 pixels, and the image contains identifiers of type (class) 0, 1, and 2 for the pixels of red algae, green algae, and background, respectively.

[0099] First, the underwater object type identification device 100 uses an annotation target image dataset N img (Dn in Figure 1) and class-specific feature F cThe underwater object type identification device 100 then sets the coefficient i to 1 (step S1702), retrieves the i-th data from the annotation target image dataset Dn, and obtains the annotation target image D (step S1703).

[0100] Next, the underwater object type identification device 100 performs segmentation processing on the annotation target image D, and the segmentation result (number of segments N) is obtained. segment ) is obtained (step S1704).

[0101] Next, the underwater object type identification device 100 performs patch division processing from the annotation image D and segmentation results to obtain a group of patches for each segment (step S1705). Next, the underwater object type identification device 100 performs feature extraction processing from the group of patches for each segment to obtain a feature quantity Fs for each segment (step S1706). Next, the underwater object type identification device 100 performs segment representative value selection processing from the feature quantity Fs for each segment to obtain a group of representative patches for each segment (step S1707).

[0102] Then, the underwater object type identification device 100 sets the coefficient j=1 (step S1708), extracts the j-th data from the segment-specific representative patch group, and obtains the segment-specific representative patch Pr (step S1706).

[0103] Next, the underwater object type identification device 100 performs feature comparison processing from the segment-specific representative patch Pr and the class-specific feature quantity Fc to obtain the segment annotation result (step S1710).

[0104] Then, the underwater object type identification device 100 determines whether the process of comparing the features of the N segments is complete or not (j = N segment Determine if (step S1711). If the feature comparison process for N segments is incomplete (step S1711: No, j > N). segment ), the underwater object type identification device 100 increments j (j+1) (step S1712) and returns to the process in step S1709. On the other hand, if the process of comparing features for N segments is completed (step S1711: Yes, j=Nsegment The underwater object type identification device 100 then proceeds to the process of step S1713 (see Figure 17B).

[0105] In step S1713, the underwater object type identification device 100 assigns the same annotation result to patches belonging to the same segment from the segment annotation results and the segment-specific patch group, thereby obtaining the image annotation result As (step S1713).

[0106] Then, the underwater object type identification device 100 determines whether or not the processing of N annotation target images is complete (i = N img Determine if the processing of N images to be annotated is incomplete (Step S1714: No, i > N). img ), the underwater object type identification device 100 increments i (i+1) (step S1715) and returns to the process in step S1703 (see Figure 17A). On the other hand, if the processing of N annotation target images is completed (step S1714: Yes, i=N img The underwater object type identification device 100 then integrates the previous image annotation results As and outputs the image annotation result group Asn (step S1716). The above process is then completed.

[0107] Figure 18 is a flowchart illustrating an example of the segmentation process. Figure 18 shows the details of the segmentation process (step S1704 in Figure 17). In this segmentation process, the image to be annotated is segmented using a segmentation model.

[0108] First, the underwater object type identification device 100 uses the annotation target image I target And, segmentation model M segmentation Input (step S1801). Next, the underwater object type identification device 100 uses the segmentation model M segmentation Annotation target image I target Enter the following and get segmentation result I segmentation Obtain (Step S1802). Segmentation result I segmentation = Msegmentation (I target ) And the underwater object type identification device 100 is segmentation result I segmentation The output is displayed (step S1803), and the above process is terminated.

[0109] Figure 19 is a flowchart showing an example of the patch splitting process. Figure 19 shows the details of the patch splitting process (step S1705 in Figure 17).

[0110] First, the underwater object type identification device 100 uses the annotation target image I target And, segmentation result I segmentation Enter the following (step S1901). Next, the underwater object type identification device 100 determines the patch width W patch = 3, height H patch Set to =3 [pixels] (step S1902). Then, the underwater object type identification device 100 annotates the image I target Each element of the patch size [W patch , H patch Annotation target patch group P target Divide into (step S1903).

[0111] Next, the underwater object type identification device 100 identifies the annotation target patch group P. target and segmentation result I segmentation Therefore, the patch and annotation results located in the same position are linked, and the segment-specific patch group P segment Obtain (step S1904). Then, the underwater object type identification device 100 identifies the segment-specific patch group P segment The output is displayed (step S1905), and the above process is terminated.

[0112] Figure 20 is a flowchart showing an example of the feature extraction process. Figure 20 shows the details of the feature extraction process (step S1706 in Figure 17).

[0113] First, the underwater object type identification device 100 identifies segment-specific patch groups P segment Input (step S2001). Next, the underwater object type identification device 100 identifies the segment-specific patch group P segmenteach element is converted from a matrix of size W p , H p into a group of segment-specific vectors V with dimension W p ×H p V segment (step S2002).

[0114] Next, the undersea object type identification apparatus 100 performs feature extraction (dimensionality reduction) via UMAP from the segment-specific vector group V segment to obtain a two-dimensional segment-specific feature quantity F segment (Fs) (step S2003). Then, the undersea object type identification apparatus 100 outputs the segment-specific feature quantity F segment (Fs) (step S2004), and ends the above processing.

[0115] Figure 21 is a flowchart showing a processing example of the segment representative value selection process. FIG. 21 shows details of the segment representative value selection process (step S1707 in FIG. 17).

[0116] First, the undersea object type identification apparatus 100 inputs the segment-specific feature quantity F segment (step S2101). Next, the undersea object type identification apparatus 100 sets a coefficient i=1 and sets a group of segment-specific representative patches P rep =∅ (step S2102), calculates the centroid of the feature quantity F segment of the i-th segment from the segment-specific feature quantities F segment,i , to obtain F mean (step S2103).

[0117] Next, among the feature quantities F segment,i of the i-th segment, the undersea object type identification apparatus 100 takes the patch corresponding to the feature quantity closest to the centroid F mean as the representative patch P rep,i of segment i (step S2104).

[0118] Next, the undersea object type identification apparatus 100 adds the representative patch P rep,i of segment i to the group of segment-specific representative patches P rep (step S2105). Next, the undersea object type identification apparatus 100 sets the coefficient i=N segmentDetermine whether or not this is the case (step S2106). i = N segment Otherwise (step S2106: No), the underwater object type identification device 100 increments i (i+1) (step S2107) and returns to the process in step S2103. On the other hand, i = N segment If so (Step S2106: Yes), the underwater object type identification device 100 will determine the segment-specific representative patch group P rep The output is displayed (step S2108), and the above process is terminated.

[0119] Figure 22 is a flowchart showing an example of the feature comparison process. Figure 22 shows the details of the feature comparison process (step S1710 in Figure 17).

[0120] First, the underwater object type identification device 100 identifies a representative patch P for each segment. segment and class-specific features F c Enter (step S2201). Next, the underwater object type identification device 100 identifies the segment-specific representative patch P segment Each element has a size [W p , H p From the matrix of ], size W p ×H p Representative vector V for each segment segment (Step S2202)

[0121] Next, the underwater object type identification device 100 generates a segment-specific representative vector V segment Feature extraction (dimensionality reduction) is performed using UMAP, and representative features F for each 2D segment are obtained. segment (Step S2203) Next, the underwater object type identification device 100 obtains the two-dimensional segment-specific representative features F segment Class-specific features F c Projected onto the map, segment-specific features F are obtained by k-NN. segment Predict the class and segment annotation result A segment (As) is obtained (step S2204). Then, the underwater object type identification device 100 obtains segment annotation result A segment (As) is output (step S2205), and the above process is terminated.

[0122] (About the k-NN method) Figure 23 is an explanatory diagram of the k-NN method. k-NN is an algorithm that uses majority voting and averaging to select the top k training data with the highest similarity to the input. As shown in Figure 23(a), the annotation target feature F target For each (circle in the figure), the adjacent class-specific feature F c Let's assume there is seaweed △ and a background □. Here, as shown in Figure 23(b), when k=3, there are two seaweed △ and one background □ within region K, so the feature to be annotated F is determined by majority vote. target (〇) indicates that the class (type) is predicted to be seaweed.

[0123] The technology for identifying the type of underwater object according to the embodiment is not limited to identifying the type (class) of algae mentioned above, but can be applied to identify the type of various underwater objects. Examples of underwater objects include: 1. marine organisms, 2. underwater artificial structures, and 3. topography and geology. 1. Marine organisms include marine animals and marine plants. Examples of marine animals include corals (hard corals, soft corals, etc.), fish (clownfish, parrotfish, etc.), shellfish (scallops, abalone, etc.), crustaceans (crabs, shrimp, etc.), cephalopods (squid, octopuses, etc.), sponges (sea sponges, bone sponges, etc.), crown-of-thorns starfish, sea urchins, sea cucumbers, jellyfish, etc. Furthermore, marine plants include seaweed (wakame, garamo, kajime, red algae (tengusa, eucalyptus), green algae (anaaosa, miru), etc.), seagrasses (amagrass, sea lilies, etc.), and other organisms (wood fragments, driftwood). Also, 2. Underwater man-made structures include cracks and rusted parts of offshore wind turbines, ship wrecks, fishing nets and gear, concrete blocks, plastic waste, metal fragments, etc. 3. Topography and geology include rocks (volcanic rock, sandstone, etc.), shell deposits, sandy areas, muddy areas, seabed cracks, tunnels, etc.

[0124] In the embodiment described above, an annotation target image is obtained by taking an image containing an object in the sea. The annotation target image is divided into multiple segments, with pixels that have similar features forming one segment, and the feature quantities of each of the multiple segments are extracted. Then, the feature quantities of multiple reference images containing objects in the sea, which have been stored in advance, are extracted, and the feature quantities of each segment are compared with the feature quantities of the reference images to identify the type of object contained in the annotation target image. This improves the accuracy of determining the type of object in the sea. By dividing the annotation target image into segments, each segment is made up of images with similar feature quantities, and by comparing them with the feature quantities of the reference images, the type of object contained in the annotation target image can be identified with high accuracy.

[0125] Furthermore, in this embodiment, the segment division process divides the entire image to be annotated into multiple segments based on the proximity of pixel values ​​and positions between pixels in the image to be annotated. As a result, each segment consists of clusters of pixels that are close in both pixel values ​​and positions, allowing for accurate identification of the type of object by comparing their features.

[0126] Furthermore, in this embodiment, class-specific reference images are generated from the reference image to show regions of different object types, and class-specific features are extracted from the reference image based on the generated class-specific reference images. The process for identifying the object type involves comparing the segment-specific features with the class-specific features of the reference image to identify the object type included in the annotated image. This makes it easy to identify the object type by comparing the features of the segmented annotated image with the features of the class-specific reference images.

[0127] Furthermore, in this embodiment, the process of extracting features from the reference image and the image to be annotated may include the process of dividing the selected reference image and the image to be annotated into patches and extracting features from each patched region. Conventionally, feature comparison using only patch division does not use the entire image, so noise appears, for example, near water turbidity or boundaries (boundaries between algae and background), making it difficult to accurately identify the type of object. In contrast, according to this embodiment, segmentation is performed using features from the entire image, thus solving the above problem and enabling more accurate identification of the type of object by comparing features on a patch-by-patch basis.

[0128] Furthermore, in this embodiment, the process of extracting features for each segment may include a process of selecting representative values ​​of the features of multiple segments as the features for each segment. By comparing the features of the reference image with the representative values ​​of the features of each segment in this way, it becomes possible to accurately identify the type of object while reducing the processing burden.

[0129] Furthermore, in this embodiment, the process of generating class-specific reference images may include specifying a representative image for each class that indicates a region containing only different types of objects in the reference image. By using reference images with specified representative images in this way, the type of object can be easily and accurately identified using feature quantities.

[0130] Furthermore, in this embodiment, the process for identifying the type of object may include a process for outputting a display image that can identify the type of object in relation to the class-specific feature quantities for the annotated image. This makes it easy to identify the types of objects included in the annotated image.

[0131] The method for identifying the type of underwater object described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a PC or workstation. The program for identifying the type of underwater object described in this embodiment is recorded on a computer-readable recording medium and executed by reading it from the recording medium by the computer. Furthermore, the program for identifying the type of underwater object described in this embodiment may be distributed via a network such as the Internet.

[0132] 100 Underwater object type identification device 200 Ocean digital twin 302, 303 Camera 801 Acquisition unit 802 Segmentation unit 803 Patch division unit 804 Feature extraction unit 805 Segment representative value selection unit 806 Feature comparison unit 807 Output unit 901 CPU 902 Memory 903 Network IF 905 Recording medium As (Asn) Annotation results for each image (group) D (Dn) Annotation target image data (dataset) Fc Class-specific features Fs Segment-specific features NW Network Ir (Irn) Reference image data (dataset) Is Segment

Claims

1. A method for identifying the type of object in the ocean, characterized in that a computer performs the following processes:

1. Obtain an annotation target image, which is an image of an object in the ocean; 2. Divide the annotation target image into multiple segments, where pixels with similar features form a single segment; 3. Extract the feature quantities from each of the multiple segments; 4. Extract the feature quantities from multiple reference images containing objects in the ocean that have been stored in advance; 5. Compare the feature quantities of each segment with the feature quantities of the reference images to identify the type of object contained in the annotation target image.

2. The method for identifying the type of underwater object according to claim 1, characterized in that the segment division process includes a process of dividing the entire annotation target image into a plurality of segments based on the proximity of pixel values ​​and positions of pixels in the annotation target image.

3. A method for identifying the type of underwater object according to claim 1, characterized in that, based on the reference image, class-specific reference images are generated showing regions of different types of objects, class-specific feature quantities are extracted from the reference image based on the generated class-specific reference images, and the process for identifying the type of object is to compare the feature quantities of the segment with the class-specific feature quantities of the reference image to identify the type of object included in the annotation target image.

4. The method for identifying the type of underwater object according to claim 2, characterized in that the process for extracting feature quantities from the reference image and the annotation target image includes the process of dividing the selected reference image and the annotation target image into patches and extracting feature quantities from each patched region.

5. The method for identifying the type of underwater object according to claim 1, characterized in that the process for extracting feature quantities for each segment includes a process for selecting representative values ​​of the feature quantities of a plurality of segments as feature quantities for each segment.

6. The method for identifying types of underwater objects according to claim 3, characterized in that the process for generating class-specific reference images includes specifying a representative image for each class that shows a region containing only different types of objects from the reference image.

7. The method for identifying the type of object in the sea according to claim 3, characterized in that the process for identifying the type of object includes a process for outputting a display image that can identify the type of object in correspondence with the class-specific feature quantities for the annotation target image.

8. A program for identifying the type of object in the ocean, characterized by having a computer perform the following processes: acquiring an annotation target image which is an image of an object in the ocean; dividing the annotation target image into multiple segments in which pixels with similar features form one segment; extracting feature quantities from each of the multiple segments; extracting feature quantities from multiple reference images which are previously stored and contain objects in the ocean; and comparing the feature quantities of each segment with the feature quantities of the reference images to identify the type of object contained in the annotation target image.

Images