Fish school management system, fish school management method, and fish school management program

Abstract

Provided is a fish school management system comprising: a fish school estimating device that, on the basis of a captured image obtained by capturing an image of fish in a fish school within a fishing net installed in water, performs a fish school estimation including estimating the number of individual fish in the fish school; and a fish loading / unloading estimating device that, on the basis of a captured image including fish that are candidates for being loaded or unloaded, the image being captured on a water passage used for loading or unloading the fish to or from the fishing net, performs a loading / unloading estimation including estimating the number of individual fish to be loaded / unloaded.

Description

Fish school management system, fish school management method, and fish school management program

[0001] The present invention relates to a fish school management system, a fish school management method, and a fish school management program.

[0002] Non-Patent Literature 1 describes a deep learning framework for estimating crowd density from still images of dense crowds, using a combination of shallow and deep convolutional architectures to enable crowd density estimation even when the scale of people in the still images of the crowd varies. [Prior Art Documents] [Non-Patent Literature] [Non-Patent Literature 1] Lokesh Boominathan, S. Kruthiventi, R. Venkatesh Babu, "CrowdNet: A Deep Convolutional Network for Dense Crowd Counting", ACM Multimedia 22 August 2016.

[0003] In managing fish farms and similar facilities, it is sometimes necessary to count the number of fish in a school in real time. In such cases, one possible approach is to use images of the school of fish taken underwater and count the number of fish in the images. However, in the underwater environment of a fish farm, transmitting large amounts of image and video data out of the water can cause reliability problems, both with wired and wireless communication. Furthermore, conventional DNNs (Deep Neural Networks) require considerable computing resources, making real-time operation in embedded systems difficult, thus hindering implementation on edge platforms.

[0004] Even with advanced AI (Artificial Intelligence) algorithms, achieving high prediction accuracy in the unique underwater environment, where light and water conditions constantly change, has been difficult. In particular, in fish school counting, factors such as hundreds of fish densely clustered and overlapping in images, distant fish appearing small and indistinct making identification difficult, and the resulting blurring of images due to turbidity in the water during springtime in Japan (March and April), all hinder accurate counting.

[0005] Conventional approaches for solving complex problems such as fish counting used Yolo (You Only Look Once)-based object detection models, but it was difficult to accurately identify overlapping fish. Also, when preparing the learning data used for training the model, the workload of manually labeling fish for a large amount of data was significant.

[0006] In feeding, if the amount and appetite of fish are inaccurately estimated, it can lead to waste of feed. In monitoring fish cages, accurate fish density management cannot be achieved, and an increase in risks of diseases, stress, aggression, injuries, etc. due to overcrowding may hinder the growth of fish. In fish loading and unloading operations, the efficiency of the operations may decrease due to the inability to accurately manage the number of individuals and separate by size.

[0007] According to one embodiment of the present invention, a fish population management system is provided. The fish population management system may include a fish population estimation device that performs fish population estimation including estimating the number of fish in a fish population based on a captured image of the fish in the fish population within a fishing net installed in water. The fish population management system may include a loading / unloading fish estimation device that performs loading / unloading fish estimation including estimating the number of fish to be loaded / unloaded based on a captured image that includes fish candidates for loading / unloading and is captured on a waterway used for loading / unloading fish to / from the fishing net.

[0008] In the fish population management system, the loading / unloading fish estimation device may perform the loading / unloading fish estimation based on the result of the fish population estimation by the fish population estimation device.

[0009] In any of the fish population management systems, the fish population estimation device inputs a captured image of the fish in the fish population within the fishing net to a learning model that is trained using a dataset including a plurality of data in which a blurred image obtained by blurring an image including fish with the image region corresponding to the representative part of the fish in the image including fish as the center and information regarding the fish in the image including fish are associated, and outputs information regarding the fish in the image including fish as the output, thereby performing the fish population estimation.

[0010] In any of the fish school management systems, the fish school estimation device may perform fish school estimation by inputting captured images of fish in the fish school within the fishing net to a learning model that is lighter than the DNN and sized according to the device on which the fish school estimation device is implemented. This learning model is trained using a dataset that includes multiple data sets that associate blurred images obtained by blurring an image containing fish, centered on an image region corresponding to a representative part of a fish in an image containing fish, with information about the fish in the image containing fish. The learning model is trained using training data that includes multiple hyperparameters of multiple types of DNNs that take images containing fish as input and information about the fish in the image containing fish as output.

[0011] In the dataset of any of the aforementioned fish school management systems, the blurred image of the fish may be obtained by applying a filter that attenuates the intensity of the pixel regions from the pixel region corresponding to the center of the fish in the image containing the fish toward the surrounding pixel regions.

[0012] In the dataset of any of the aforementioned fish school management systems, the blurred image of the fish may be obtained by applying a Gaussian filter to the pixel region corresponding to the center of the fish in the image containing the fish.

[0013] In any of the aforementioned fish school management systems, the dataset may include a dataset generated using a simulation of a fish school enclosed by a fishing net using 3D computer graphics (Three-Dimensional Computer Graphics) technology.

[0014] Any of the fish school management systems may further include a camera system installed in the water to image fish in the fish school within the fishing net. In any of the fish school management systems, the fish school estimation device may be implemented in the camera system.

[0015] In any of the fish school management systems, the camera system may have a plurality of camera modules installed at different depths in the water. In any of the fish school management systems, a plurality of fish school estimation devices may be implemented at different depths in the water, corresponding to the plurality of camera modules. In any of the fish school management systems, information based on the fish school estimation results may be transmitted between the plurality of fish school estimation devices in an order based on the depth in the water, thereby transmitting information based on the fish school estimation results along the different depths in the water.

[0016] In any of the fish school management systems, the fish school estimation device may perform fish school estimation, which further includes estimating the size of the fish in the fish school.

[0017] Any of the fish school management systems may further include a feeding device that feeds the fish school in the fishing net based on the results of the fish school estimation by the fish school estimation device.

[0018] In any of the fish school management systems, the fish transport / inbound estimation device may perform fish transport / inbound estimation, which further includes estimating the size of the candidate fish for transport / inbound, based on captured images including the candidate fish for transport / inbound. Any of the fish school management systems may further include a sorting device that sorts the fish to be transported or inbound from the candidate fish for transport / inbound on the waterway, based on the results of the size estimation of the candidate fish for transport / inbound by the fish transport / inbound estimation device.

[0019] Any of the fish school management systems may further include a fishing net estimation device that estimates the condition of the fishing net based on an image taken in the water, including the fishing net.

[0020] According to one embodiment of the present invention, a fish school management method is provided. The fish school management method may include a fish school estimation step, which includes estimating the number of fish in a fish school based on images of fish in a fish school located in a fishing net installed underwater. The fish school management method may also include an inbound / outbound fish estimation step, which includes estimating the number of fish targeted for inbound / outbound transport, based on images of candidate fish for inbound / outbound transport acquired on a waterway used for transporting fish to and from the fishing net, and the results of the fish school estimation estimated in the fish school estimation step.

[0021] According to one embodiment of the present invention, a fish school management program is provided for causing a computer to perform any of the fish school management methods.

[0022] It should be noted that the above summary of the invention does not list all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention.

[0023] This diagram schematically shows an example of the fish school management system 90. This is an explanatory diagram explaining how to obtain a blurred image 19 by blurring a fish image 17. This diagram schematically shows an example of the fish school management system 90. This diagram schematically shows an example of the fish school estimation device 900. This is an explanatory diagram explaining the learning model. This diagram schematically shows an example of the functional configuration of the fish school estimation device 900. This diagram schematically shows an example of the functional configuration of the incoming / outgoing fish estimation device 800. This diagram schematically shows an example of the functional configuration of the fishing net estimation device 700. This diagram schematically shows an example of the processing flow by the fish school management system 90. This diagram schematically shows an example of the hardware configuration of the computer 1200 that functions as the fish school estimation device 900, the incoming / outgoing fish estimation device 800, and the fishing net estimation device 700.

[0024] The present invention will be described below through embodiments, but these embodiments are not intended to limit the scope of the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0025] The fish school management system 90 according to this embodiment has a configuration that contributes to solving the aforementioned problems. For example, the fish school management system 90 provides an efficient and highly accurate AI model that can operate on an embedded chipset near the underwater camera. This can improve, for example, the efficiency and profitability of aquaculture.

[0026] The fish school management system 90 uses a software suite that performs fish school estimation and fish ingress / degression estimation, for example, using FishNet, a density-based model that can be installed on an embedded chipset near the underwater camera. In the fish school management system 90, for example, by using a learning model with a model size limited to several tens of MB (for example, about 82 MB), the fish population estimation is performed on the processor of an edge device near the underwater camera in real time processing at 15 frames per second. This eliminates the need to transmit large amounts of image data outside the water, improving the reliability of the estimation and reducing implementation costs.

[0027] The fish school management system 90, for example, employs density heatmap prediction instead of object detection, and uses a dataset created by combining data generated using 3D CG simulation technology with digital signal processing as training data. This allows for relatively high-accuracy estimation of fish populations even under conditions such as images of densely packed and overlapping fish, images of distant fish that are unclear, and unclear images taken in murky water.

[0028] The fish school management system 90, for example, coordinates multiple fish school estimation devices 900, each implemented on multiple camera modules. This allows the fish school management system 90 to estimate the total number of individuals in a moving school of fish. The fish school management system 90 transmits the results of fish school estimation from each fish school estimation device 900 via short-range wireless communication between the multiple fish school estimation devices 900, and estimates the total number of individuals in the school based on the integrated and processed results. The fish school management system 90 enables accurate real-time counting of fish in a school, allowing for dynamic adjustment of feed amounts to reduce feed waste. Furthermore, the fish school management system 90 can reduce disease risk by maintaining an optimal fish density, maximizing fish quality and profitability. The fish school management system 90 can also contribute to human safety by replacing tasks such as inspecting fishing nets, which were previously performed by human divers.

[0029] Figure 1 schematically shows an example of a fish school management system 90. In the example shown in Figure 1, the fish school management system 90 includes a fish school estimation device 900 and a fish arrival / departure estimation device 800. In this example, a fish farm 91 is formed by installing a fishing net 92 in the water. In this example, the fishing net 92 is installed in the water by being connected to a float 95 near the water surface. In this example, a school of fish 10 is located inside the fishing net 92. In this example, the school of fish 10 includes multiple fish 100. The fish farm 91 may be a fish farm for cultivating fish. The school of fish 10 may consist of farmed fish 100.

[0030] In the example shown in Figure 1, the camera module 300 is installed inside the fish tank 91. The method of installing the camera module 300 inside the fish tank 91 is not particularly limited. For example, the camera module 300 may be installed on a float. The camera module 300 may also be installed on a support fixed to the bottom of the water. The camera module 300 may also be installed on a mobile body that can move on or underwater. The mobile body may be a ship, an underwater drone, etc.

[0031] The camera module 300 may be an optical camera. For example, the camera module 300 may be an RGB camera. The camera module 300 may be a monocular camera. The camera module 300 may be a compound camera. The camera module 300 may be a camera capable of acquiring distance information between the camera module 300 and the subject. For example, the camera module 300 is a stereo camera. The baseline of the stereo camera is not particularly limited. For example, the stereo camera may be a horizontal stereo camera, a vertical stereo camera, or a multi-baseline stereo camera using three or more cameras. The camera module 300 may be a camera capable of acquiring still images. The camera module 300 may be a camera capable of acquiring multiple still images in a time series. The camera module 300 may be a camera capable of acquiring video.

[0032] In the example shown in Figure 1, one camera module 300 is installed in the fish tank 91, but multiple camera modules 300 may be installed. In other words, a camera system including multiple camera modules 300 may be installed in the fish tank 91.

[0033] The camera module 300 images the fish 100 of the school of fish 10 and acquires captured images of the fish 100 of the school of fish 10. For example, the camera module 300 acquires captured RGB images of the school of fish 10. The camera module 300 may acquire captured stereo images of the school of fish 10. The camera module 300 may acquire captured images of stereo pairs of the school of fish 10. The camera module 300 may acquire captured images of the RGB images of the school of fish 10 with depth information of the fish 100 added.

[0034] The camera module 300 may have a light source. A separate light source may also be installed. The light source is installed in a manner suitable for imaging the fish 100 of the fish school 10 by the camera module 300. For example, the color tone of the light source is adjusted based on the color of the body surface of the fish 100 that the camera module 300 is imaging, the size of the fish 100, the type and concentration of impurities in the water in the fish tank 91, etc. The installation position and angle of the light source may be adjusted based on the installation angle of the camera module 300, etc.

[0035] The fish school management system 90 may include a camera module 300. The fish school management system 90 may include a camera system. The camera system may include one camera module 300. The camera system may include multiple camera modules 300. The camera system may include components other than cameras. For example, the camera system may include a fish school estimation device 900. The fish school management system 90 may include a light source corresponding to the camera module 300.

[0036] In the example shown in Figure 1, the fish school estimation device 900 may be implemented in correspondence with the camera module 300. In this example, the fish school estimation device 900 is implemented on the camera module 300. That is, in this example, the fish school estimation device 900 is implemented on the camera module 300 as an on-device. The fish school estimation device 900 may be implemented integrally with the camera module 300.

[0037] The fish school estimation device 900 may be mounted separately from the camera module 300. In this case, the fish school estimation device 900 may be located near the camera module 300. For example, the fish school estimation device 900 may be located at a distance such that the transmission of images captured by the camera module 300 to the fish school estimation device 900 does not pose a problem for the fish school estimation device 900's estimation of the fish school 10. The fish school estimation device 900 may also be located at a distance from the camera module 300, as long as the transmission quality can be maintained such that the transmission of images captured by the camera module 300 to the fish school estimation device 900 does not pose a problem for the fish school estimation device 900's estimation of the fish school 10.

[0038] The fish school estimation device 900 may perform fish school estimation for the fish 100 in the fish school 10. For example, the fish school estimation device 900 may perform fish school estimation including the estimation of the number of fish 100 in the fish school 10. In the example shown in Figure 1, the fish school estimation device 900 acquires the captured image of the fish school 10 taken by the camera module 300. In this example, the fish school estimation device 900 performs fish school estimation based on the acquired captured image of the fish school 10.

[0039] The fish school estimation device 900 may estimate the number of fish 100 using a density-based learning model. The fish school estimation device 900 may estimate the number of fish 100 by inputting captured images of the fish 100 in the fish school 10 into the density-based learning model. The estimation of the number of fish 100 using the density-based learning model will be described later.

[0040] The fish school estimation by the fish school estimation device 900 may include estimating the size of the fish 100. For example, the fish school estimation device 900 acquires stereo images captured by a stereo camera and estimates the size of the fish 100 based on these captured images. The fish school estimation device 900 may also use a learning model to estimate the size of the fish 100. The learning model may be a learning model that takes images of the fish school 10 as input and outputs the size of the fish 100 in the fish school 10, having been trained using multiple datasets that associate images of the fish school 10 with the sizes of the fish 100 in the images. The fish school estimation device 900 may acquire images of the fish school 10 captured by a monocular camera and estimate the size of the fish 100 by inputting these captured images into the learning model. The fish school estimation by the fish school estimation device 900 may include estimating the body length of the fish 100, the fork length of the fish 100, and the body height of the fish 100.

[0041] The fish school estimation by the fish school estimation device 900 may include estimating the weight of the fish 100. The fish school estimation device 900 may use a learning model to estimate the weight of the fish 100. The learning model may be a learning model that takes an image of the fish school 10 as input and outputs the weight of the fish 100 in the fish school 10, having been trained using multiple datasets that associate images of the fish school 10 with the weight of the fish 100 in the image. The fish school estimation device 900 may acquire an image of the fish school 10 and input the image into the learning model to estimate the weight of the fish 100.

[0042] The fish school estimation by the fish school estimation device 900 may include estimating the density of fish 100 in the fish school 10. For example, the fish school estimation device 900 may use a learning model to estimate the density of fish 100. The learning model may be a learning model that takes images of the fish school 10 as input and outputs the density of the fish school 10, having been trained using multiple datasets that associate images of the fish school 10 with the density of the fish school 10. The fish school estimation device 900 may acquire captured images of the fish school 10 and input these captured images into the learning model to estimate the density of the fish school 10. The fish school estimation device 900 may estimate the number of fish 100 in the fish school 10 and estimate the density of the fish school 10 based on the estimated number of fish and the volume of the fishing net 92. For example, the fish school estimation device 900 estimates the density of the fish school 10 by dividing the estimated number of fish by the volume of the fishing net 92. This makes it possible, for example, to divide the fish pens 91 or adjust the number of fish 100 based on the estimated density of the fish school 10, thereby managing the density of the fish school 10 to an appropriate level. For example, it becomes possible to properly maintain the stress levels and health of the fish 100, thus enabling disease prevention and increased yields.

[0043] The fish school estimation by the fish school estimation device 900 may include estimation of the fish species of the fish 100 in the fish school 10. For example, the fish school estimation device 900 estimates the fish species of the fish 100 in the fish school 10 using a learning model. The learning model may be a learning model that is trained using a plurality of datasets associating images of the fish school 10 with the fish species of the fish 100 included in the fish school 10, takes an image of the fish school 10 as an input, and outputs the fish species of the fish school 10. The fish school estimation device 900 may acquire a captured image of the fish school 10 and estimate the fish species of the fish school 10 by inputting the captured image into the learning model. The fish school estimation by the fish school estimation device 900 may include estimation regarding the mixing of other fish into the fish school 10. The fish school estimation by the fish school estimation device 900 may include estimation regarding the mixing of harmful fish into the fish school 10. The fish school estimation device 900, for example, estimates the presence or absence of sharks mixing into the fish cage 91 for cultivating sea breams.

[0044] The fish school estimation by the fish school estimation device 900 may include estimation of the sexes of the fish 100 in the fish school 10. For example, the fish school estimation device 900 estimates the sexes of the fish 100 in the fish school 10 using a learning model. The learning model may be a learning model that is trained using a plurality of datasets associating images of the fish school 10 with the sexes of the fish 100 in the fish school 10, takes an image of the fish school 10 as an input, and outputs the sexes of the fish 100 in the fish school 10. The fish school estimation device 900 may acquire a captured image of the fish school 10 and estimate the sexes of the fish 100 in the fish school 10 by inputting the captured image into the learning model.

[0045] The fish school estimation performed by the fish school estimation device 900 may include estimation of the behavior of the fish 100 in the fish school 10. For example, the fish school estimation device 900 estimates the behavior of the fish 100 in the fish school 10 using a learning model. The learning model may be a learning model that takes time-series images of the fish school 10 as input and outputs the behavior of the fish 100 in the fish school 10, and has been trained using multiple datasets that associate time-series images of the fish school 10 with the behavior of the fish 100 in the fish school 10. The fish school estimation device 900 may acquire time-series captured images of the fish school 10 and input these time-series captured images into the learning model to estimate the behavior of the fish 100 in the fish school 10. The fish school estimation device 900 may also estimate the feeding behavior of the fish 100 in the fish school 10. The fish school estimation device 900 may estimate the feeding behavior of the fish 100 in the fish school 10 based on the estimation of the behavior of the fish 100 in the fish school 10.

[0046] The fish school estimation by the fish school estimation device 900 may include estimation of the state of the fish 100 in the fish school 10. For example, the fish school estimation device 900 estimates the state of the fish 100 in the fish school 10 using a learning model. The learning model may be a learning model that takes time-series images of the fish school 10 as input and outputs the state of the fish 100 in the fish school 10, and has been trained using multiple datasets that associate time-series images of the fish school 10 with the state of the fish 100 in the fish school 10. The fish school estimation device 900 may acquire time-series captured images of the fish school 10 and estimate the state of the fish 100 in the fish school 10 by inputting these time-series captured images into the learning model. The fish school estimation device 900 may estimate the health status of the fish 100 in the fish school 10. The fish school estimation device 900 may estimate the stress status of the fish 100 in the fish school 10. The fish school estimation device 900 may estimate the stress state and health state of the fish 100 in the fish school 10 based on the estimation of the behavior of the fish 100 in the fish school 10.

[0047] The fish school estimation device 900 may estimate the abnormal state of the fish 100 in the fish school 10, the abnormal state of the body surface, the degree of body color, the degree of color gloss, the malformation rate, the maturity, etc. The fish school estimation device 900 may perform these estimations, for example, using the learning model as described above. The fish school estimation device 900 may perform these estimations based on the plurality of aforementioned estimation results. For example, the fish school estimation device 900 performs these estimations using an algorithm that combines the plurality of aforementioned estimation results. For example, the fish school estimation device 900 combines the estimation result of the number of individuals of the fish 100 in the fish school 10 and the estimation result of the contact behavior of the fish school 10 to estimate the abnormal state of the fish 100 in the fish school 10.

[0048] In the example shown in FIG. 1, there may be cases where the fish 100 is carried into the fish cage 91 or carried out from the fish cage 91 to the outside of the fish cage 91. For example, there may be cases where it is desired to add the fish 100 to the fish cage 91 or ship out the fish 100 from the fish cage 91. In this example, the case of carrying out the fish 100 from the fish cage 91 to the outside of the fish cage 91 will be described.

[0049] In the present embodiment, the "loading and unloading" of the fish 100 may refer to performing at least one of carrying the fish 100 into the fish cage 91 and carrying the fish 100 out from the fish cage 91 to the outside of the fish cage 91. In the present embodiment, the "loading and unloading" of the fish 100 includes only carrying out the fish 100 without carrying it in. In the present embodiment, the "loading and unloading" of the fish 100 includes only carrying in the fish 100 without carrying it out. [[ID=][7]]

[0050] In the example shown in FIG. 1, the fish 100 is carried out from the fish cage 91 to the fishing boat 82. In this example, using the waterway 84, the fish 100 is carried out to the fishing boat 82. The waterway 84 is sized such that fish can move within the waterway 84.

[0051] In the example shown in FIG. 1, one end of the two ends of the waterway 84 is installed on the side of the fish cage 91, and the other end is installed on the fishing boat 82. By sucking up the fish 100 together with water by a pump or the like, the fish 100 is carried out to the fishing boat 82.

[0052] In the example shown in Figure 1, the camera module 310 is installed on the waterway 84. For example, the camera module 310 may be installed so that its lens portion is located on the inner surface of the waterway 84. The camera module 310 captures images including the fish 100 being transported through the waterway 84 to the fishing boat 82.

[0053] In the example shown in Figure 1, the fish transport estimation device 800 makes estimations about the fish 100 based on the captured image including the fish 100, which is captured on the waterway 84. In this example, the fish transport estimation device 800 makes estimations about the number of fish 100 that are to be transported in and out, based on the captured image including the fish 100.

[0054] In the example shown in Figure 1, the description of the configuration of camera module 310 may be the same as the description of the configuration of camera module 300. For example, camera module 310 may be an RGB camera or a stereo camera.

[0055] In the example shown in Figure 1, the camera module 310 may have a light source. Separately from the camera module 310, the light source may be installed in a manner suitable for imaging the fish 100 of the school of fish 10 by the camera module 310. For example, the color tone of the light source may be adjusted based on the type of fish 100 that the camera module 310 is imaging, the color of their body surface, and the concentration and type of impurities in the water. The installation position and angle of the light source may be adjusted based on the installation angle of the camera module 310, etc.

[0056] In the example shown in Figure 1, the fish arrival / departure estimation device 800 may be implemented in correspondence with the camera module 310. In this example, the fish arrival / departure estimation device 800 is implemented on the camera module 310. That is, in this example, the fish arrival / departure estimation device 800 is implemented on the camera module 310 as an on-device. The fish arrival / departure estimation device 800 may be implemented integrally with the camera module 310.

[0057] The fish arrival / departure estimation device 800 may be mounted separately from the camera module 310. In this case, the fish arrival / departure estimation device 800 may be located near the camera module 310. For example, the fish arrival / departure estimation device 800 may be located at a distance such that the transmission of images captured by the camera module 310 to the fish arrival / departure estimation device 800 does not pose a problem for the fish arrival / departure estimation device 800's estimation of the fish school 10. The fish arrival / departure estimation device 800 may also be located at a distance from the camera module 310, within a range where sufficient transmission quality can be maintained so that the transmission of images captured by the camera module 310 to the fish arrival / departure estimation device 800 does not pose a problem for the fish arrival / departure estimation device 800's estimation of the fish school 10.

[0058] In the example shown in Figure 1, the explanation of the implementation of the fish arrival / departure estimation device 800 may be the same as the explanation of the implementation of the fish school estimation device 900. In the example shown in Figure 1, the configuration of the camera module 310 may be the same as the configuration of the camera module 300. For example, the camera module 310 may be an RGB camera or a stereo camera.

[0059] The fish inbound / outbound estimation device 800 may perform individual fish inbound / outbound estimation using an object detection algorithm or the like. The fish inbound / outbound estimation device 800 may perform individual fish inbound / outbound estimation using object detection with a CNN (Convolutional Neural Network). The fish inbound / outbound estimation device 800 may perform individual fish inbound / outbound estimation using YOLO-based object detection. The fish inbound / outbound estimation device 800 may perform individual fish inbound / outbound estimation using R-CNN (Region-based Convolutional Neural Network)-based object detection. The fish inbound / outbound estimation device 800 may perform individual fish inbound / outbound estimation using a density-based learning model, similar to the fish school estimation device 900. The fish school estimation device 900 may perform an estimation of the number of fish 100 by inputting captured images of the fish 100 in the fish school 10 into the density-based learning model. The estimation of the number of fish 100 using the density-based learning model will be described later.

[0060] In the example shown in Figure 1, the channel 84 may be structured such that when multiple fish 100 pass through the channel 84 at the same time, they are less likely to line up in parallel in the direction of entry and exit, and more likely to line up in a column. For example, the cross-sectional area of ​​the channel 84 perpendicular to the direction of entry and exit may be small enough that multiple fish 100 are less likely to line up in parallel. Guides made of a material that is less likely to damage the fish 100 may be installed inside the channel 84 and are arranged to make it easier for the fish 100 to line up in a column in the direction of entry and exit. This makes it less likely, for example, for the fish 100 to overlap in the depth direction of the imaging range of the camera module 310. This makes it easier to acquire images in which multiple fish 100 do not overlap, so for example, the accuracy of fish entry and exit estimation by the fish entry and exit estimation device 800 is more likely to improve.

[0061] In the example shown in Figure 1, the fish transport estimation device 800 may perform fish transport estimation based on the results of fish school estimation by the fish school estimation device 900. For example, the fish transport estimation device 800 estimates the number of fish 100 to be transported based on the number of fish 100 in the fish school 10 estimated by the fish school estimation device 900. For example, the fish transport estimation device 800 performs fish transport estimation such that the total number of fish in the fish school 10 estimated just before the start of transporting the fish school 10 corresponds to the sum of the total number of fish in the fish school 10 and the number of fish 100 to be transported after the start of transporting the fish school 10.

[0062] In the example shown in Figure 1, an example of transporting fish 100 from a fish farm 91 to a fishing boat 82 was described. However, a person skilled in the art will understand that the same method can be applied when transporting fish 100 from a fishing boat 82 to a fish farm 91. In the example shown in Figure 1, the transport of fish 100 between the fish farm 91 and the fishing boat 82 was described. However, the destination for transporting fish between the fish farm 91 and the fishing boat 82 is not particularly limited, and transport may be carried out with other destinations. For example, the same method can be used when transporting fish 100 from the fish farm 91 to a truck equipped with a tank.

[0063] Figure 2 is an explanatory diagram illustrating the acquisition of a blurred image 19 by blurring an image 17 of a fish. Blurring is a process related to the generation of training data used when generating a density-based learning model. In the example shown in Figure 2, the starting state is one in which an image 17 containing a fish is associated with information about the fish contained in the image 17 containing the fish.

[0064] The image 17 containing fish does not necessarily have to be an image of an actual fish; it may be a composite image. For example, a simulation environment in which a school of fish enclosed by a fishing net is reproduced using 3D CG technology can be used. In this case, information about the fish, such as the number of fish, size, weight, species, fish density in a specific area of ​​the fishing net, and fish density throughout the entire fishing net, is known as a simulation parameter. By using the simulation environment, it is possible to arbitrarily set the fish's arrangement, lighting conditions, water quality conditions, imaging points, etc., and virtually image and synthesize a large number of artificially generated fish images. Further digital signal processing may be applied to the synthesized fish images.

[0065] In the example shown in Figure 2, first, an image region corresponding to a representative part of the fish is identified in image 17, which contains the fish. In this example, the representative part of the fish is the center of the fish. The center of the fish may be the center of gravity of the fish. Image 18 is an image that shows the state after extracting the pixel region corresponding to the representative part of the fish. In this example, blurring is performed on image 18, and a blurred image 19 is obtained. In this example, blurring is a process that blurs the image. In this example, blurring is explained via image 18 for clarity, but blurring does not necessarily have to be explicitly done via image 18; the blurred image 19 may be obtained directly from image 17 containing the fish.

[0066] In the example shown in Figure 2, blurring is a process that applies a filter to an image 17 containing a fish, starting from the pixel region corresponding to the center of the fish and moving toward the surrounding pixel regions, causing the intensity of the pixel region to decrease. In this example, the state in which the filter is applied is conceptually represented by multiple concentric circles, with the pixel region corresponding to the center of the fish at the center, and the spacing between them increasing as you move away from the center. For example, if the blurred image 19 is represented as a heatmap, the pixel region corresponding to the center of the fish is represented by a color representing high temperature, and the surrounding pixel regions are represented by progressively lower temperatures. If the blurred image 19 is represented as a grayscale image, the pixel region corresponding to the center of the fish is represented by a color closer to black, and the surrounding pixel regions are represented by progressively whiter colors. Alternatively, the pixel region corresponding to the center of the fish may be represented by a color closer to white, and the surrounding pixel regions may be represented by progressively blacker colors.

[0067] In the example shown in Figure 2, the blurring process may be a process of applying a filter that follows a normal distribution centered on the center to the pixel region corresponding to the center of the fish in the image 17 containing the fish. In this example, the blurring process may be a process of applying a Gaussian filter to the pixel region corresponding to the center of the fish in the image 17 containing the fish.

[0068] By performing processing as shown in the example in Figure 2, it is possible to obtain training data that includes multiple datasets, each containing an image 17 with a fish, its blurred image, and information about the fish, such as the number and size of the fish. Since the data is artificially synthesized using a 3D CG simulation environment, manual annotation and labeling are not required, thus reducing the effort required to generate training data.

[0069] The processing shown in the example in Figure 2 may also be performed using an image 17 containing fish that was actually acquired by imaging. In this case, the association between the image 17 containing fish and information about the fish, and the annotation of the pixel region corresponding to the center of the fish in the image 17 containing fish are performed manually or by other means.

[0070] In the example shown in Figure 2, the dataset may include a dataset generated using a 3D CG simulation of a school of fish enclosed by a fishing net. Part of the training data used to train the density-based learning model may be created based on the aforementioned synthesized image 17 containing fish, and another part may be created based on image 17 containing fish obtained by actually taking images. In this example, the majority of the training data used to train the density-based learning model may be created based on the aforementioned synthesized image 17 containing fish, and the remaining small part may be created based on image 17 containing fish obtained by actually taking images.

[0071] The fish school estimation device 900 may perform fish school estimation by inputting captured images of fish 100 in a fish school 10 within the fishing net 92 to a learning model that takes images containing fish as input and outputs information about fish in images containing fish, which has been trained using a dataset containing multiple datasets that associate blurred images 19 with information about fish in images containing fish. In the dataset, the blurred image of the fish image may be obtained by applying a filter that attenuates the intensity of the pixel region from the pixel region corresponding to the center of the fish in the image containing fish toward the surrounding pixel region. In the dataset, the blurred image of the fish image may be obtained by applying a Gaussian filter to the pixel region corresponding to the center of the fish in the image containing fish.

[0072] Figure 3 schematically shows an example of a fish school management system 90. The example shown in Figure 3 will mainly be explained in terms of differences from the example shown in Figure 1. In the example shown in Figure 3, another fish farm 93 is located near the first fish farm 91. The other fish farm 93 is made up of fishing nets 94. In this example, the case of transporting fish 100 from fish farm 91 to fish farm 93 will be explained. In this example, a waterway 84 is used to transport the fish 100 to the other fish farm 93. The waterway 84 is large enough for the fish to move within it.

[0073] In the example shown in Figure 3, fish 100 that may be subject to removal are driven from one fish tank 91 towards another fish tank 93. For example, the fish 100 are driven by sound, light, or equipment such as a net to drive the fish. The driven fish 100 are then transported from the fish tank 91 to the other fish tank 93 via the waterway 84.

[0074] In the example shown in Figure 3, the camera module 310 is installed on the waterway 84. For example, the camera module 310 may be installed so that its lens portion is located along the path of the waterway 84. The camera module 310 captures images including the fish 100 being transported to the fishing boat 82 via the waterway 84.

[0075] In the example shown in Figure 3, a guide 85 is placed on the side of the waterway 84 that leads to the fish pond 91. The guide 85 restricts the path of the fish 100, causing the fish 100 to pass near the camera module 310 and head towards the other fish pond 93. As a result, the distance between the camera module 310 and the fish 100 stabilizes at a certain distance suitable for imaging, making it easier for the camera module 310 to image the fish 100 and improving the quality of the captured images.

[0076] In the example shown in Figure 3, as in the example shown in Figure 1, the fish arrival / departure estimation device 800 performs estimations regarding the fish 100 based on the captured images including the fish 100 captured on the waterway 84. The type of camera module 310, the light source, the implementation method of the fish arrival / departure estimation device 800, the algorithm used by the fish arrival / departure estimation device 800, the guides such as strainers in the waterway 84, and the configuration in which the fish arrival / departure estimation device 800 performs fish arrival / departure estimation based on the results of fish school estimation by the fish school estimation device 900 may be the same as in the example shown in Figure 1.

[0077] In the example shown in Figure 3, an example of transporting fish 100 from one fish farm 91 to another fish farm 93 was described. However, a person skilled in the art will understand that the same method can be applied when transporting fish 100 from another fish farm 93 to fish farm 91. In the example shown in Figure 3, the transport of fish 100 between fish farms was described, but the destination for transport between fish farm 91 and other destinations is not particularly limited, and transport may be carried out with other destinations. For example, the same method can be used when releasing fish 100 from fish farm 91 into the ocean.

[0078] Figure 4 schematically shows an example of a fish school management system 90. The example shown in Figure 4 will mainly be explained in terms of the differences from the example shown in Figure 1. In this example, the fish school management system 90 is equipped with a camera system. The camera system has multiple camera modules installed at different depths in the water. The basic configuration of each of the multiple camera modules may be the same as in the example shown in Figure 1. In this example, the camera system has camera module 301, camera module 302, ..., camera module 308, and camera module 309.

[0079] In the example shown in Figure 4, the fish school management system 90 is equipped with multiple fish school estimation devices. In this example, a fish school estimation device is implemented in each of the multiple camera modules. The basic configuration of each of the multiple fish school estimation devices may be the same as in the example shown in Figure 1. In this example, the fish school estimation device 901 is implemented in camera module 301, the fish school estimation device 902 is implemented in camera module 302, ..., the fish school estimation device 908 is implemented in camera module 308, and the fish school estimation device 909 is implemented in camera module 309.

[0080] In the example shown in Figure 4, information based on the fish school estimation results is transmitted between multiple fish school estimation devices in an order based on depth in the water. For example, fish school estimation device 909 transmits its own fish school estimation result to fish school estimation device 908. Fish school estimation device 908 transmits its own fish school estimation result and information based on the fish school estimation result from fish school estimation device 909 to other fish school estimation devices located on the shallower side of the water.

[0081] In the example shown in Figure 4, the fish school estimation device 908 transmits information that integrates its own fish school estimation result with the fish school estimation result of the fish school estimation device 909 to other fish school estimation devices located on the shallower side. For example, the fish school estimation device 908 transmits to other fish school estimation devices located on the shallower side the number of fish 100 obtained by adding the number of fish 100 estimated by itself and the number of fish 100 estimated by the fish school estimation device 909. Alternatively, the fish school estimation device 908 may transmit to other fish school estimation devices located on the shallower side the number of fish 100 obtained by subtracting the number of fish 100 that were estimated multiple times from the number of fish 100 estimated by itself and the number of fish 100 estimated by the fish school estimation device 909. For example, the fish school estimation device 908 performs the reduction process by reducing the number of fish 100 located in the region of the captured image that corresponds to the region where the imaging field of the camera module 309 and the imaging field of the camera module 308 overlap.

[0082] In the example shown in Figure 4, the fish school estimation device 908 may transmit information including its own fish school estimation result and the fish school estimation result from the fish school estimation device 909 to another fish school estimation device located on the shallower side.

[0083] In the example shown in Figure 4, the same process is repeated among multiple fish school estimation devices, moving toward the shallower water depth. In this way, information based on the fish school estimation results is transmitted among multiple fish school estimation devices in order based on the depth in the water, so that the information based on the fish school estimation results is transmitted along different depths in the water and the number of individuals can be totaled. As a result, for example, fish school estimation device 901 receives information based on the fish school estimation results from fish school estimation device 909, fish school estimation results from fish school estimation device 908, ... and fish school estimation results from fish school estimation device 902 from fish school estimation device 902. By integrating this information with its own fish school estimation results, fish school management system 90 can estimate in real time, including information about fish 100 in deep areas of the water where communication is difficult. The estimation of the total number of individuals of fish 100 may be the estimation of the maximum number of fish 100 located within the fishing net 92. The estimation of the total number of fish 100 individuals may be the estimation of the total number of fish 100 individuals located in a specific area within the fish farm 91 where multiple fish school estimation devices are installed.

[0084] In the example shown in Figure 4, the fish school estimation device 901 receives a total number of fish from the fish school estimation device 902, which is the sum of the number of fish 100 estimated by the fish school estimation device 909, the number of fish 100 estimated by the fish school estimation device 908, ..., and the number of fish 100 estimated by the fish school estimation device 902. By adding the number of fish 100 that it estimated to be fish to this total number, the fish school management system 90 can estimate the number of fish in real time, including the number of fish 100 in deep areas of the water where communication is difficult.

[0085] In the example shown in Figure 4, the transmission of information between multiple fish school estimation devices may include transmission using short-range wireless communication technology. For example, the transmission of information between multiple fish school estimation devices may include transmission via Bluetooth®. The transmission of information between multiple fish school estimation devices may be transmission using short-range wireless communication technology. Even underwater, if the distance between devices is, for example, several tens of centimeters, communication is possible using short-range wireless communication technology. Furthermore, if the data is small in amount of information, such as the estimated number of individuals, rather than large data such as captured image data, the communication load is small. By having multiple fish school estimation devices cooperate to perform fish school estimation and transmit the estimation result data, fish school estimation becomes possible over a wide range, including the deep water regions of the fish farm 91.

[0086] In the example shown in Figure 4, the transmission of information between multiple fish school estimation devices may include other transmission methods. For example, the transmission of information between multiple fish school estimation devices may be wired communication. For example, the transmission of information between multiple fish school estimation devices may include wired communication using optical fiber cables. The transmission of information between multiple fish school estimation devices may also include wired communication using optical fiber cables that serve as both a power supply and a communication line for the fish school estimation devices.

[0087] In the example shown in Figure 4, the case in which multiple fish school estimation devices are arranged in a single row along the direction of water depth has been described, but the system is not limited to this. For example, multiple rows of multiple fish school estimation devices arranged in a single row along the direction of water depth may be arranged. Multiple fish school estimation devices may be arranged in a matrix. Multiple fish school estimation devices may be tilted with respect to the direction of water depth and arranged in a single row along the tilted direction.

[0088] In the example shown in Figure 4, the fish school management system 90 includes a feeding device 96. The feeding device 96 may feed the fish school 10 in the fishing net 92 with bait 97 based on the fish school estimation results from the fish school estimation device 900. For example, the feeding device 96 feeds the fish school 10 based on the number of fish 100 in the fish school 10 estimated by the fish school estimation device 900. The feeding device 96 may feed the fish school 10 based on the contact status of the fish school 10 estimated by the fish school estimation device 900. For example, if the fish school 10 is eating the bait 97 slowly, the feeding device 96 controls the amount of bait 97 to be fed to decrease. The feeding device 96 adjusts the amount of bait 97 in real time based on the real-time number of fish and the real-time rate at which the bait 97 is being eaten, as determined by the fish school estimation device 900. The feeding device 96 may also adjust the type of bait 97.

[0089] In Figure 4, the fish management system 90 includes a sorting device 86. In this example, the sorting device 86 is installed in the waterway 84. In this example, the waterway 84 branches into a main route leading to the fishing boat 82, which is the destination for the fish, and a secondary route leading back to the fish farm 91, which is the source of the fish. In this example, the sorting device 86 is installed in the waterway 84, downstream from the location where the camera module 310 is installed, on the route through which the fish 100 are transported, and near the branching point of the main route and the secondary route. The sorting device 86 has a structure that allows the movement path of the fish 100 to be switched between the main route and the secondary route. In this example, the sorting device 86 has a flap that can open and close the main route. In this example, when the main route is open, the fish 100 are transported to the fishing boat 82 via the water flow, and when the main route is closed, the fish 100 flow into the secondary route.

[0090] As shown in Figure 4, the fish transport estimation device 800 performs fish transport estimation, including estimation of the number and size of candidate fish, based on captured images that include candidate fish for transport. In this example, the sorting device 86 sorts the fish to be transported from the candidate fish on the waterway 84 based on the size estimation results of the candidate fish for transport by the fish transport estimation device 800. For example, the sorting device 86 sorts the fish to be transported from the candidate fish for transport on the waterway 84 if the size estimated by the fish transport estimation device 800 is greater than or equal to a predetermined size threshold, and sorts them as not to be transported if the size is less than the threshold. The sorting device 86 may also sort the fish to be transported from the candidate fish for transport if the size estimated by the fish transport estimation device 800 is less than or equal to a predetermined upper size limit and greater than or equal to a predetermined lower size limit. The sorting device 86 may sort fish as oversized if the size estimated by the incoming / outgoing fish estimation device 800 exceeds the upper limit, and may sort fish as undersized if it falls below the lower limit.

[0091] In Figure 4, the fish school management system 90 includes a fishing net estimation device 700. In this example, the fishing net estimation device 700 estimates the state of the fishing net 92 based on an image of the fishing net 92 captured underwater. In this example, the fish school management system 90 includes a camera module 320. The camera module 320 may capture an image of the fishing net 92 underwater. The description of the configuration of the camera module 320 may be the same as that of the camera module 300. In this example, the configuration of the camera module 320 may be the same as that of the camera module 300. For example, the camera module 320 may be an RGB camera or a stereo camera.

[0092] In the example shown in Figure 4, the camera module 320 may have a light source, and a separate light source may be installed in addition to the camera module 320, just as with the camera module 300.

[0093] In the example shown in Figure 4, the fishing net estimation device 700 may be implemented in correspondence with the camera module 320. In this example, the fishing net estimation device 700 is implemented on the camera module 320. That is, in this example, the fishing net estimation device 700 is implemented on the camera module 320 as an on-device. The fishing net estimation device 700 may be implemented integrally with the camera module 320.

[0094] The fishing net estimation device 700 may be mounted separately from the camera module 320. In this case, the fishing net estimation device 700 may be placed in the vicinity of the camera module 320. For example, the fishing net estimation device 700 may be placed at a distance such that the transmission of images captured by the camera module 320 to the fishing net estimation device 700 does not pose a problem for the estimation of the fish school 10 by the fishing net estimation device 700. The fishing net estimation device 700 may also be placed at a distance from the camera module 320, within a range where sufficient transmission quality can be maintained so that the transmission of images captured by the camera module 320 to the fishing net estimation device 700 does not pose a problem for the estimation of the fish school 10 by the fishing net estimation device 700.

[0095] The fishing net estimation device 700 estimates the condition of the fishing net 92 based on the captured image including the fishing net 92, which is captured underwater. For example, the fishing net estimation device 700 estimates whether or not the fishing net 92 is damaged. For example, the fishing net estimation device 700 estimates whether or not the net of the fishing net 92 is torn. The fishing net estimation device 700 may also estimate the degree of deterioration of the fishing net 92.

[0096] In the example shown in Figure 4, the fishing net estimation device 700 may estimate the state of the fishing net 92 using an object detection algorithm or the like. The fishing net estimation device 700 may estimate the state of the fishing net 92 by object detection using CNN. The fishing net estimation device 700 may estimate the state of the fishing net 92 by YOLO-based object detection. The fishing net estimation device 700 may estimate the state of the fishing net 92 by R-CNN-based object detection.

[0097] Figure 5 is an explanatory diagram illustrating the learning model. Figure 5 describes the generation of a small machine learning model that can run on an edge device. Learning models that estimate the number and size of fish based on captured images are often large learning models. For example, the learning model is a large DNN. By using such a learning model, the number and size of fish can be estimated with high accuracy. However, the computing power required to run such a large learning model is considerable, so it needs to be run on a server with high computing power. Therefore, when attempting to perform real-time inference processing based on captured images taken underwater, it is difficult to use such a large DNN as is, in addition to the aforementioned problem of image transmission.

[0098] When attempting to perform real-time inference processing based on images captured underwater, it is necessary to use a small machine learning model that can operate on edge devices with limited computing power. However, it is usually difficult to perform estimation with the same high estimation accuracy as a large DNN using a small machine learning model. For example, by training with training data containing multiple hyperparameters of multiple types of DNNs that take images containing fish as input and output information about fish in the images containing fish, it is possible to generate a learning model that is lighter than a DNN and has the same estimation accuracy as a DNN. This learning model may be sized according to the device on which it is implemented. This learning model may be lighter than a larger learning model, and sized according to the device on which the fish school estimation device 900 is implemented, which is trained using a dataset containing multiple data sets that associate blurred images obtained by blurring images containing fish with information about fish in the images containing fish, centered on image regions corresponding to representative parts of fish in the images containing fish.

[0099] Figure 6 schematically shows an example of the functional configuration of the fish school estimation device 900. In the example shown in Figure 6, the fish school estimation device 900 includes an acquisition unit 910, a storage unit 920, an estimation unit 930, an output control unit 940, and a learning model generation unit 950. It is not essential that the fish school estimation device 900 includes all of these. For example, the fish school estimation device 900 does not need to include the learning model generation unit 950.

[0100] The acquisition unit 910 acquires various types of information. For example, the acquisition unit 910 acquires captured images of fish 100 in a school of fish 10 within a fishing net 92 installed underwater. The acquisition unit 910 may acquire captured images taken by the camera module 300. The acquisition unit 910 may also acquire information based on the results of fish school estimation by other fish school estimation devices transmitted from other fish school estimation devices.

[0101] The memory unit 920 stores various types of information. For example, the memory unit 920 stores captured images. The memory unit 920 may store various types of learning models. The memory unit 920 stores a density-based learning model for performing inferences about the fish 100 in the school of fish 10. For example, the memory unit 920 stores a learning model that takes an image containing fish as input and outputs information about the fish in the image containing fish. This learning model is trained using a dataset that includes multiple data sets that associate blurred images obtained by blurring an image containing fish, centered on an image region corresponding to a representative part of the fish in the image containing fish, with information about the fish in the image containing fish. The memory unit 920 stores a learning model that is lighter than the large learning model and sized according to the device on which the fish school estimation device 900 is implemented. This learning model is trained using a dataset containing multiple data sets that associate blurred images obtained by blurring an image containing fish, centered on an image region corresponding to a representative part of a fish in the image containing fish, with information about the fish in the image containing fish. The large learning model may be a DNN.

[0102] The memory unit 920 may store various types of training data. For example, the memory unit 920 stores a dataset that associates blurred images obtained by blurring an image containing fish, centered on an image region corresponding to a representative part of the fish in the image containing the fish, with information about the fish in the image containing the fish. The memory unit 920 may store training data containing multiple hyperparameters of multiple types of large learning models that take an image containing fish as input and output information about the fish in the image containing the fish, which are trained using a dataset containing multiple such datasets that associate blurred images obtained by blurring an image containing fish, centered on an image region corresponding to a representative part of the fish in the image containing the fish, with information about the fish in the image containing the fish. The large learning model may be a DNN.

[0103] The memory unit 920 may store information based on the results of fish school estimation performed by other fish school estimation devices, which may be transmitted from other fish school estimation devices.

[0104] The estimation unit 930 performs various estimations. The estimation unit 930 may also use the learning model stored in the memory unit 920 to perform these estimations. The estimation unit 930 performs fish school estimation, including the estimation of the number of fish 100 in the fish school 10. The estimation unit 930 may perform the estimation of the number of fish 100 by inputting the captured images of the fish 100 in the fish school 10 into a density-based learning model.

[0105] The estimation unit 930 may perform fish school estimation, including the estimation of the size of the fish 100. The estimation unit 930 may perform fish school estimation, including the estimation of the weight of the fish 100. The estimation unit 930 may perform fish school estimation, including the estimation of the density of fish 100 in the fish school 10. This makes it possible, for example, to divide the fish pens 91 or adjust the number of fish 100 based on the estimated density of the fish school 10, thereby managing the fish school 10 to an appropriate density. For example, it is possible to appropriately maintain the stress level and health of the fish 100, which makes it possible to prevent diseases and increase yields.

[0106] The estimation unit 930 may perform fish school estimation, including estimation of the species of fish 100 in the fish school 10. The estimation unit 930 may perform fish school estimation, including estimation of the presence of other fish in the fish school 10. The fish school estimation by the fish school estimation device 900 may include estimation of the presence of harmful fish in the fish school 10. The estimation unit 930 may perform fish school estimation, including estimation of the sex of fish 100 in the fish school 10. The estimation unit 930 may estimate the behavior of fish 100 in the fish school 10. The estimation unit 930 may estimate the feeding behavior of fish 100 in the fish school 10 based on the estimation of the behavior of fish 100 in the fish school 10.

[0107] The estimation unit 930 may perform fish school estimation, including estimation of the condition of the fish 100 in the fish school 10. The estimation unit 930 may perform fish school estimation, including estimation of the health status of the fish 100 in the fish school 10. The estimation unit 930 may perform fish school estimation, including estimation of abnormal conditions of the fish 100 in the fish school 10, abnormal conditions of the body surface, degree of body color, degree of luster, deformity rate, maturity, etc.

[0108] The output control unit 940 controls the output of various types of information. For example, the output control unit 940 outputs the estimation result from the estimation unit 930. For example, the output control unit 940 transmits the estimation result. For example, the output control unit 940 transmits the estimation result to a user terminal. For example, the output control unit 940 outputs the number of fish 100 in the school of fish 10 to a user terminal such as a smartphone via short-range wireless communication. The output control unit 940 may transmit the estimation result to other fish school estimation devices 900. The output control unit 940 outputs the estimation result to other fish school estimation devices so that information based on the fish school estimation result is transmitted between multiple fish school estimation devices in order based on depth in the water.

[0109] The output control unit 940 may be controlled to output sound based on the estimation result from the estimation unit 930. For example, the output control unit 940 may output the number of fish 100 in the school of fish 10 based on the estimation result from the estimation unit 930. The output control unit 940 may also output an alert sound based on the estimation result from the estimation unit 930.

[0110] The learning model generation unit 950 generates a learning model. The learning model generation unit 950 may generate a learning model using the learning data stored in the storage unit 920. For example, the learning model generation unit 950 generates a density-based learning model for performing inference about the fish 100 in the school of fish 10. For example, the learning model generation unit 950 generates a learning model that takes an image containing fish as input and outputs information about the fish in the image containing fish, which is trained using a dataset that includes multiple data sets that associate blurred images obtained by blurring an image containing fish, centered on an image region corresponding to a representative part of the fish in the image containing fish, with information about the fish in the image containing fish. The learning model generation unit 950 generates a learning model that is lighter than the large learning model and sized to suit the device implementing the fish school estimation device 900. This learning model is trained using a dataset containing multiple data sets that associate blurred images obtained by blurring images of fish containing fish, centered on image regions corresponding to representative parts of fish in images containing fish, with information about fish in the fish containing images. The large learning model may be a DNN.

[0111] Figure 7 schematically shows an example of the functional configuration of the fish arrival / departure estimation device 800. In the example shown in Figure 7, the fish arrival / departure estimation device 800 includes an acquisition unit 810, a storage unit 820, an estimation unit 830, an output control unit 840, and a learning model generation unit 850. It is not essential that the fish arrival / departure estimation device 800 includes all of these. For example, the fish arrival / departure estimation device 800 does not need to include the learning model generation unit 850.

[0112] The acquisition unit 810 acquires various types of information. For example, the acquisition unit 810 acquires captured images including candidate fish 100 for loading and unloading, taken over the waterway 84 used for loading and unloading fish 100 into and out of the fishing net 92. The acquisition unit 810 may acquire captured images taken by the camera module 310. The acquisition unit 810 may acquire the results of fish school estimation by the estimation unit 930.

[0113] The memory unit 820 stores various types of information. For example, the memory unit 820 stores captured images. The memory unit 820 may store various learning models. The memory unit 820 may store learning models for estimating the number of fish to be transported in and out, including estimating the number of fish individuals to be transported in and out.

[0114] The memory unit 820 may store object detection algorithms. The memory unit 820 may store object detection algorithms using CNNs. The memory unit 820 may store YOLO-based object detection algorithms. The memory unit 820 may store R-CNN-based object detection algorithms. The memory unit 820 may store density-based learning models.

[0115] The memory unit 820 may store various types of training data. For example, the memory unit 820 may store a dataset containing multiple sets of data that associate images containing fish with information about the fish in those images. The memory unit 820 may store training data containing multiple hyperparameters of multiple types of large learning models that take images containing fish as input and output information about the fish in those images, which have been trained using the dataset containing multiple sets of data that associate images containing fish with information about the fish in those images. The large learning models may be DNNs.

[0116] The memory unit 820 may store information based on the results of fish school estimation transmitted from the fish school estimation device 900.

[0117] The estimation unit 830 performs various estimations. The estimation unit 830 may also use the learning model stored in the memory unit 820 to perform these estimations. The estimation unit 830 performs fish shipment estimation, including the estimation of the number of fish targeted for shipment. The estimation unit 830 may perform the estimation of the number of fish 100 by inputting captured images containing candidate fish for shipment into the learning model stored in the memory unit 820.

[0118] The estimation unit 830 may perform fish transport estimation, including estimating the size of the candidate fish 100 for transport. The estimation unit 830 may perform fish transport estimation, including estimating the weight of the candidate fish 100 for transport.

[0119] The estimation unit 830 may perform fish transport estimation, including estimation of the species of fish 100 that are candidates for transport. The estimation unit 830 may perform fish transport estimation, including estimation of the presence of fish species not intended for transport among the candidate fish 100. The fish transport estimation by the fish transport estimation device 800 may include estimation of the presence of harmful fish. The estimation unit 830 may perform fish transport estimation, including estimation of the sex of the candidate fish 100.

[0120] The estimation unit 830 may perform fish transport estimation, including estimation of the condition of the candidate fish 100 for transport. The estimation unit 830 may perform fish transport estimation, including estimation of the health status of the candidate fish 100 for transport. The estimation unit 830 may perform fish transport estimation, including estimation of the stress status of the candidate fish 100 for transport. The estimation unit 830 may perform fish transport estimation, including estimation of abnormal conditions of the fish 100 in the fish school 10, abnormal conditions of the body surface, degree of body color, degree of luster, deformity rate, maturity, etc.

[0121] The output control unit 840 controls the output of various types of information. For example, the output control unit 840 outputs the estimation results from the estimation unit 830. For example, the output control unit 840 transmits the estimation results. For example, the output control unit 840 transmits the estimation results to a user terminal. For example, the output control unit 840 outputs the number of fish to be transported to a user terminal such as a smartphone via short-range wireless communication.

[0122] The output control unit 840 may be controlled to output sound based on the estimation result by the estimation unit 830. For example, the output control unit 840 may output the number of fish 100 in the school of fish 10 based on the estimation result by the estimation unit 830. The output control unit 840 may also output an alert sound based on the estimation result by the estimation unit 830. For example, the output control unit 840 may output an alert sound when the estimated number of fish reaches a preset number of fish to be transported.

[0123] The learning model generation unit 850 generates a learning model. The learning model generation unit 850 generates a learning model using CNNs with the learning data stored in the memory unit 820. The learning model generation unit 850 may generate a learning model using YOLO-based learning models with the learning data stored in the memory unit 820. The learning model generation unit 850 may generate an R-CNN-based learning model using the learning data stored in the memory unit 820. The learning model generation unit 850 may also generate a density-based learning model.

[0124] The learning model generation unit 850 may generate a learning model that takes an image containing a fish as input and outputs information about the fish in the image containing a fish, by training it using a dataset containing multiple data sets that associate images containing a fish with information about the fish in the image containing a fish. The learning model generation unit 850 may also generate a learning model that is lighter than the large learning model and sized according to the device on which the fish arrival / departure estimation device 800 is implemented, by training it using training data containing multiple hyperparameters of a large learning model that takes an image containing a fish as input and outputs information about the fish in the image containing a fish, by training it using a dataset containing multiple data sets that associate images containing a fish with information about the fish in the image containing a fish. The large learning model may be a DNN.

[0125] Figure 8 schematically shows an example of the functional configuration of the fishing net estimation device 700. In the example shown in Figure 8, the fishing net estimation device 700 includes an acquisition unit 710, a storage unit 720, an estimation unit 730, an output control unit 740, and a learning model generation unit 750. It is not essential that the fishing net estimation device 700 includes all of these. For example, the fishing net estimation device 700 does not need to include the learning model generation unit 750.

[0126] The acquisition unit 710 acquires various types of information. For example, the acquisition unit 710 acquires an image of the fishing net 92 that was photographed underwater. The acquisition unit 710 may also acquire an image captured by the camera module 320.

[0127] The memory unit 720 stores various types of information. For example, the memory unit 720 stores captured images. The memory unit 720 may store various learning models. The memory unit 720 may store a learning model for estimating the state of the fishing net 92.

[0128] The memory unit 720 may store object detection algorithms. The memory unit 720 may store object detection algorithms using CNNs. The memory unit 720 may store YOLO-based object detection algorithms. The memory unit 720 may store R-CNN-based object detection algorithms.

[0129] The memory unit 720 may store various types of training data. For example, the memory unit 720 may store a dataset containing multiple data sets that associate images containing fishing nets 92 with information about fishing nets, such as whether or not the fishing nets in the images containing fishing nets are damaged. The memory unit 720 may store training data containing multiple hyperparameters of multiple types of large learning models that take images containing fish as input and output information about fish in the images containing fish. These large learning models may be DNNs.

[0130] The estimation unit 730 performs various estimations. The estimation unit 730 may also use the learning model stored in the memory unit 720 to perform these estimations. The estimation unit 730 estimates the state of the fishing net 92. The estimation unit 730 may estimate the state of the fishing net 92 by inputting the captured image including the fishing net 92 into the learning model stored in the memory unit 720.

[0131] The estimation unit 730 may perform estimation of the fishing net, including estimation of the size of the holes in the fishing net 92. The estimation unit 730 may also perform estimation of the fishing net, including the degree of deterioration of the fishing net 92 over time.

[0132] The output control unit 740 controls the output of various types of information. For example, the output control unit 740 outputs the estimation result from the estimation unit 730. For example, the output control unit 740 transmits the estimation result. For example, the output control unit 740 transmits the estimation result to a user terminal. For example, the output control unit 740 outputs the estimation result to a user terminal such as a smartphone via short-range wireless communication.

[0133] The output control unit 740 may be controlled to output audio based on the estimation result from the estimation unit 730. For example, the output control unit 740 may output audio indicating whether or not the fishing net 92 is damaged, based on the estimation result from the estimation unit 730.

[0134] The learning model generation unit 750 generates a learning model. The learning model generation unit 750 may generate a learning model using CNNs with the learning data stored in the memory unit 720. The learning model generation unit 750 may generate a learning model using YOLO-based learning models with the learning data stored in the memory unit 720. The learning model generation unit 750 may generate an R-CNN-based learning model using the learning data stored in the memory unit 720.

[0135] The learning model generation unit 750 may generate a learning model that takes the estimation result as input and outputs information about fish in images containing fish, which has been trained using a dataset containing multiple data sets that associate images containing fishing nets with information about fishing nets in the images containing fishing nets. The learning model generation unit 750 may generate a learning model that is lighter than the large learning model and sized according to the device on which the fishing net estimation device 700 is implemented, which has been trained using training data containing multiple hyperparameters of a large learning model that takes images containing fishing nets as input and outputs information about fish in images containing fishing nets, which has been trained using a dataset containing multiple data sets that associate images containing fishing nets with information about fishing nets in the images containing fishing nets. The large learning model may be a DNN.

[0136] Figure 9 schematically shows an example of the processing flow by the fish school management system 90. In the example shown in Figure 9, the process is described from when the incoming / outgoing fish estimation device 800 estimates incoming / outgoing fish based on the fish school estimation results estimated by the fish school estimation device 900, until the estimation results are output.

[0137] In step 102 (sometimes abbreviated as S), the acquisition unit 910 of the fish school estimation device 900 acquires an image of the fish 100 of the fish school 10.

[0138] In S104, the estimation unit 930 of the fish school estimation device 900 performs fish school estimation, including the estimation of the number of fish 100. Here, the estimation unit 930 performs fish school estimation by inputting the captured images acquired by the acquisition unit 910 in S102 into the learning model stored in the storage unit 920.

[0139] In S106, the acquisition unit 810 of the fish transport / inbound fish estimation device 800 acquires an image containing candidate fish for transport / inbound, and the fish school estimation result estimated by the estimation unit 930 of the fish school estimation device 900 in S104.

[0140] In S108, the estimation unit 830 of the fish transport / inbound estimation device 800 performs fish transport / inbound estimation, including the estimation of the number of fish 100. Here, the estimation unit 830 performs fish transport / inbound estimation by inputting the captured images acquired by the acquisition unit 810 in S106 into the learning model stored in the storage unit 820, and comparing them with the fish school estimation results acquired by the acquisition unit 810 in S106.

[0141] In S110, the output control unit 840 controls the system to output the estimated fish shipment / inbound fish shipment result that the estimation unit 830 estimated in S108. For example, the output control unit 840 outputs the estimated fish shipment / inbound fish shipment result to the user terminal.

[0142] Figure 10 schematically shows an example of the hardware configuration of a computer 1200 that functions as a fish school estimation device 900, a fish entry / exit estimation device 800, and a fishing net estimation device 700. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the apparatus according to this embodiment, or to cause the computer 1200 to execute operations associated with the apparatus according to this embodiment or such one or more "parts", and / or to cause the computer 1200 to execute a process or a stage of such process according to this embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0143] The computer 1200 according to this embodiment includes a CPU 1212, a GPU 1213, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive and a solid-state drive, etc. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0144] The CPU 1212 operates according to the programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires the image data generated by the CPU 1212 and stores it in the frame buffer provided in the RAM 1214 or within itself, so that the image data is displayed on the display device 1218.

[0145] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0146] The ROM 1230 stores boot programs and / or hardware-dependent programs of the computer 1200, which are executed by the computer 1200 when activated. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0147] The program is provided on a computer-readable storage medium such as a DVD-ROM or IC card. The program is read from the computer-readable storage medium and installed on a storage device 1224, RAM 1214, or ROM 1230, which are examples of computer-readable storage media, and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the operation or processing of information in accordance with the use of the computer 1200.

[0148] For example, when communication is performed between a computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 1214, storage device 1224, DVD-ROM, or IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0149] Furthermore, the CPU 1212 may read all or necessary parts of a file or database stored on an external recording medium such as a storage device 1224, a DVD drive (DVD-ROM), or an IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

[0150] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to the RAM 1214. The CPU 1212 may also retrieve information in files, databases, etc., within the recording medium. For example, if a plurality of entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 may search among the plurality of entries for an entry that matches the specified condition for the attribute value of the first attribute, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0151] The program or software module described above may be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0152] In this embodiment, blocks in the flowchart and block diagram may represent a stage in a process in which an operation is performed or a "part" of a device that has the role of performing an operation. A particular stage and "part" may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, exclusive OR, negated AND, negated OR, and other logical operations, flip-flops, registers, and memory elements.

[0153] A computer-readable storage medium may include any tangible device capable of storing instructions to be executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital multipurpose disk (DVD), Blu-ray® disk, memory stick, integrated circuit card, etc.

[0154] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages.

[0155] Computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the internet to a processor or programmable circuit of a general-purpose computer, special-purpose computer, or other programmable data processing device, so that the processor or programmable circuit of the programmable data processing device, such as a computer, can execute the instructions to generate means for performing operations specified in a flowchart or block diagram. Here, the computer may be a PC (personal computer), tablet computer, smartphone, workstation, server computer, general-purpose computer, or special-purpose computer, and may also be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system and is a computer in a broad sense. In a distributed computing system, multiple computers execute a program collectively by each computer executing a part of the program and passing data during program execution between computers as needed.

[0156] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, and microcontrollers. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of the program, and the processors collectively execute the program by passing program execution data between them as needed. For example, in the execution of multitasks, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which part of a program each processor executes changes dynamically. Which part of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

[0157] Real-time fish school counting system. Background. Technical field. This invention relates to a real-time fish school counting system.

[0158] Related technologies. The non-patent document "CrowdNet: A Deep Convolutional Network for Dense Crowd Counting; Lokesh Boominathan, Srinivas SS Kruthiventi, R. Venkatesh Babu" discloses a deep convolutional network for dense crowd counting.

[0159] Detailed Patent Application for Embodiments: On-Device Fish Counting System 1. Note that each fish, labeled as 1.1 (fish), has a dot at its center. This value follows a Gaussian distribution and is propagated to neighboring pixels. In this way, a heatmap is displayed for all pixels in the image. A high heatmap value means that there are many fish near that pixel. 2. This task is then transformed into a pixel heatmap prediction problem, which is essentially a regression problem that can be predicted using a neural network. Beyond the basic algorithm, we believe there are the following innovations:

[0160] 1. Most importantly, several technological advantages allow us to obtain a highly accurate fish school counting algorithm even in blurred images. 2. We trained using a large number of blurred images. However, direct training did not work well. Instead, we processed the blurred images using a digital signal processing algorithm and trained using the generated images. 3. We jump-started training using your (SoftBank's) simulation data. This is important because it eliminates the error-prone labeling process. 4. We used only a small amount of actual fish images labeled by humans as primary training data.

[0161] Fish school counting using one or more devices within tunnels / tunnels between fish pens. 1. Synchronize the time of multiple devices (time, audio, sound, identical object video clips, etc.). 2. Detect fish with each device. 3. Send count data to multiple other devices. 4. Calculate the total count. 5. Alert fishermen when the count approaches a set number of fish (e.g., 1000). 6. The system may count down using sound, voice, or light. 7. Sound an audible signal to indicate the end of counting (reaching the target number of fish). About the algorithm: 1. Track the fish. 2. Count fish on the boundary line. 3. Decrement the number of fish that have returned to the other side of the set line.

[0162] Product Name: Watami: AI-Powered Smart Aquaculture Software Suite Product Description: Watami is an AI-powered software suite for monitoring critical aquaculture activities such as feeding, cage monitoring, and fish movement. Watami features deep neural network models for fish school counting, net damage detection, and fish size estimation. These models are enhanced by an innovative and efficient neural network architecture and deployed on an embedded chipset near the underwater camera. This eliminates the need to transmit large amounts of video data out of the water, improving reliability, reducing costs, and simplifying integration. The algorithm achieves consistently high accuracy under changing lighting, water, and weather conditions and easily adapts to various camera settings. Watami significantly improves the efficiency and profitability of aquaculture.

[0163] 1. Engineering: Automated monitoring of aquaculture farms in underwater environments is crucial in aquaculture, but remains challenging due to several factors. • Transmitting large amounts of video data out of the water presents reliability challenges, both wired and wireless. • Even advanced AI algorithms struggle to achieve high predictive accuracy under changing light and water conditions. • Many aquaculture applications require real-time operation, but running deep neural networks in embedded systems has been extremely slow. To address these challenges, we developed Watatumi, a software suite featuring a deep neural network model with the following capabilities: • Efficiency: Our efficient neural network architecture allows Watatumi to operate at the edge, eliminating the need for data transmission. For example, the fish counting model is only 82MB in size and processes 720p video in real-time at 15 frames per second on a Snapdragon 8 Gen 3 processor. • Accuracy and Robustness: Watami consistently achieves 90% to 95% accuracy in fish school counting, net damage detection, and fish size estimation in real-world underwater testing under changing conditions. This surpasses other AI-based solutions thanks to its large, high-quality training dataset consisting of field-collected and synthetically generated data. • Flexibility: Watami adapts easily to diverse use cases, accommodating various camera configurations and mounting positions.

[0164] 2. Design: Accurate, real-time monitoring of aquaculture farms significantly improves the efficiency of key aquaculture operations and reduces costs. Feeding: Fish feed accounts for 50% to 70% of production costs, and with a 30% increase in fishmeal prices since 2022, this has become a major challenge for fishermen. Conventional feeders often lead to waste due to inaccurate estimates of fish quantity and appetite. Watami measures the number of fish at different depths in real time and provides an indicator of appetite. This makes it possible to dynamically adjust feeding schedules and amounts, significantly reducing waste and costs. Cage Monitoring: Optimal fish density is crucial for efficient aquaculture. Overcrowding increases the risk of disease, stress, aggression, and injury, all of which hinder growth. Accurate fish counting helps maintain optimal density and maximize profitability and fish quality. In addition, Watami's fish net damage detection prevents fish escape and further ensures operation. Fish Migration: Watami simplifies fish migration by automating fish school counting and size estimation. These models ensure accurate population management when moving fish between pens, assist in separating fish by size, and improve efficiency during transfer from pens to vessels. System Integration: Watami's on-device deployment simplifies integration and significantly reduces system costs because it eliminates the need to transmit large amounts of video data out of the water.

[0165] 3. Innovation: Edge AI Efficiency: Watami directly performs real-time aquaculture monitoring with an embedded chipset near the underwater camera, eliminating the need to transmit large amounts of video data. Unlike traditional deep neural network methods that require considerable computing resources beyond the capabilities of the edge platform, Watami's efficient architecture processes high-resolution video at the source, simplifying integration, increasing system reliability, and reducing costs. Superior Accuracy and Robustness: Watami consistently delivers 90% to 95% accuracy under real-world underwater conditions, outperforming other AI solutions. Driven by large, high-quality training datasets, Watami's performance is robust against changing lighting, water quality, and environmental conditions. Adaptability: Watami accommodates various camera configurations and installations, easily adapting to diverse aquaculture settings. This flexibility allows fishermen to optimize feeding, monitoring, and migration processes in various environments.

[0166] 4. Human Security for All: Yes, our product supports human security. Watami contributes to food security by enhancing both food safety and affordability. By accurately monitoring fish populations, it prevents overcrowding in pens, reduces the risk of disease spread, and promotes higher quality and safer fish. Furthermore, its precise feeding application minimizes fishmeal waste and significantly reduces production costs. This efficiency could lower the overall commodity price of fish, making it more affordable and accessible to economically disadvantaged communities. Finally, automated underwater monitoring eliminates the need for professional aquaculture divers, enhancing human safety. 5. Other Technical Specifications: In actual underwater tests, Watami achieves an accuracy of 90% to 95% in fish school counting, net damage detection, and fish size estimation. When deployed on a Snapdragon 8 Gen 3 processor, this software processes 720p video in real time at 15 frames per second.

[0167] FishNet: Density-Based Fish Counting Target Automated monitoring of aquaculture farms in underwater environments is crucial in aquaculture, but remains challenging due to several factors: 1. Transmitting large amounts of video data out of the water causes reliability issues with both wired and wireless communication. 2. Even advanced AI algorithms struggle to achieve high predictive accuracy under changing light and water conditions. 3. Many aquaculture applications require real-time operation, but running deep neural networks in embedded systems has been extremely slow until now. This project is being conducted jointly by a research team at SoftBank and the Aizip Vision team. The goal is to count the number of fish in a pond that may contain hundreds of fish. Our deep neural network (DNN) model, FishNet, has the following features: 1. Efficiency: FishNet enables operation on edge devices and eliminates the need for data transmission. For example, the fish counting model is only 82MB in size and processes 720P video in real time at 15 frames per second on a Snapdragon 8 Gen3 processor. 2. Accuracy and Robustness: The model has been proven to consistently achieve 90% to 95% accuracy in fish counting under various conditions, including murky spring environments. 3. Flexibility: The model accommodates various camera settings and mounting positions and easily adapts to diverse use cases. Details of the FishNet Model Difficulties in the Problem A conventional approach to solving such problems is to use a YOLO-based object detection model [1]. However, this project has several difficulties. 1. The number of fish can be in the hundreds. In 720P images, the fish overlap terribly. Distant fish are very ambiguous and small, making it impossible to identify them together with overlapping fish. 2. The water is quite murky in March and April. It is very difficult to see the fish. We do not believe that object detection algorithms can accurately count the fish. Instead, we want to introduce a density model to solve this problem.Model Structure Description The FishNet model is based on the density model [2] rather than using a conventional object detection-based model. The main reason is that the density model can handle overlapping objects much better than conventional object detection approaches. The basic idea is as follows: Assume that at the center of each fish there is a label that is always counted as 1 (one fish). This value follows a Gaussian distribution and is propagated to neighboring pixels. In this way, a heatmap is displayed for all pixels in the image. A high heatmap value means that there are many fish near that pixel. The task of predicting the number of fish is reduced to a pixel heatmap prediction problem, which is actually a regression problem that can be predicted using a neural network. Using this basic idea, we have achieved FishNet using several new techniques: 1. A new network structure has been designed so that it can run on edge devices. 2. We trained using a large number of blurred images. However, direct training did not work well. Instead, we obtained the best results by processing the blurred images using a digital signal processing algorithm and training using the generated images. 3. We jumpstarted training using simulation data from SoftBank's Unity simulator. This is important because it eliminates the error-prone labeling process. 4. We used only a small amount of human-labeled real fish images to fine-tune the model. Model Parameters and Training Details The FishNet model size includes 82M parameters. The allowed input resolution is 1280x720, meaning it accepts 720P images as input. The model is trained using a training dataset containing approximately 20,000 images, each containing up to 400 fish. The training dataset includes: ・22% public datasets (primarily from fish5k[3]). ・65% synthetic datasets generated by SoftBank's Unity simulator[4].- The simulator can generate images with the precise location of each fish in various conditions, such as different water and sunlight conditions, increasing diversity and eliminating the need for labeling this dataset. - To address the domain gap between simulated and real images, a StyleFlow[5]-based model is used to enhance the photorealistic realism of images generated under varying conditions. This approach effectively bridges the gap, increases the diversity of the dataset, and provides a more robust foundation for models trained on it. - 5% public Coco dataset[6] - 8% public video frame test and results test datasets containing up to 4,000 images collected / labeled by the Aizip team, including the following sources: - Online video / images for various special cases - SoftBank dataset generated by Unity simulator - Videos of real fish collected by SoftBank researchers This model has been deployed on two devices: OnePlus and iPhone® 14, both using Snapdragon 8 Gen3 processors. Using a carefully designed test set, the FishNet model achieves: - 96% accuracy in clear water images - 86% accuracy in murky water images This shows how to calculate the number of fish with FishNet using a OnePlus mobile phone. This model has been tested in real life by SoftBank scientists. References: [1] Ultralytics YOLO11 [2] Lokesh Boominathan, S. Kruthiventi, R. Venkatesh Babu, "CrowdNet: A Deep Convolutional Network for Dense Crowd Counting", ACM Multimedia 2016. [3] Fish5K can be found on GitHub, but its references could no longer be found.[4] Unity - Manual [5] Fan, Weichen, Jinghuan Chen, Jiabin Ma, Jun Hou, and Shuai Yi. "Styleflow for content-fixed image to image translation." arXiv preprint arXiv:2207.01909 (2022). [6] COCO dataset [7] CES award application.

[0168] The invention according to this embodiment makes it possible, for example, to estimate the number of fish individuals in a school of fish in real time, and to estimate the number and size of fish individuals passing through a waterway in real time. This improves the production efficiency and safety of fish farming, for example. Therefore, it can contribute to achieving at least one of the Sustainable Development Goals (SDGs): Goal 2 "Zero Hunger", Goal 7 "Affordable and Clean Energy", Goal 9 "Industry, Innovation and Infrastructure", Goal 12 "Responsible Consumption and Production", and Goal 14 "Life Below Water".

[0169] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0170] It should be noted that the execution order of operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before" or "prior to," and that these can be performed in any order unless the output of a previous operation is used in a later operation. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," and "next," for convenience, this does not mean that it is mandatory to perform the operations in that order.

[0171] 10 Fish school, 17 Image, 18 Image, 19 Blurred image, 82 Fishing boat, 84 Waterway, 85 Guide, 86 Sorting device, 90 Fish school management system, 91 Fish pen, 92 Fishing net, 93 Fish pen, 94 Fishing net, 95 Float, 96 Feeding device, 97 Bait, 100 Fish, 300 Camera module, 301 Camera module, 302 Camera module, 308 Camera module, 309 Camera module, 310 Camera module, 320 Camera module, 700 Fishing net estimation device, 710 Acquisition unit, 720 Storage unit, 730 Estimation unit, 740 Output control unit, 750 Learning model generation unit, 800 Fish in / out estimation device, 810 Acquisition unit, 820 Storage unit, 830 Estimation unit, 840 Output control unit, 850 Learning model generation unit, 900 Fish school estimation device, 901 Fish school estimation device, 902 Fish school estimation device, 908 Fish school estimation device, 909 Fish school estimation device, 910 Acquisition unit, 920 Storage unit, 930 Estimation unit, 940 Output control unit, 950 Learning model generation unit, 1200 Computer, 1210 Host controller, 1212 CPU, 1213 GPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / output controller, 1222 Communication interface, 1224 Storage device, 1230 ROM, 1240 Input / output chip

Claims

1. A fish school management system comprising: a fish school estimation device that performs fish school estimation, including estimation of the number of fish in a school of fish, based on captured images of fish in a school of fish installed in a fishing net underwater; and a fish transport estimation device that performs transport fish estimation, including estimation of the number of fish targeted for transport, based on captured images taken on a waterway used for transporting fish to and from the fishing net, including candidate fish for transport.

2. The fish transport / inbound fish estimation device performs the transport / inbound fish estimation based on the results of the fish school estimation by the fish school estimation device, as described in claim 1.

3. The fish school estimation device performs fish school estimation by inputting captured images of fish in the fish school within the fishing net into a learning model that takes images of fish as input and outputs information about fish in the fish image, which is trained using a dataset that includes multiple data sets that associate blurred images obtained by blurring an image of fish with an image of fish centered on an image region corresponding to a representative part of a fish in the image of fish.

4. The fish school estimation device performs fish school estimation by inputting captured images of fish in the fishing net to a learning model that is lighter than the DNN and sized according to the device on which the fish school estimation device is implemented. This learning model is trained using a dataset that includes multiple data sets that associate blurred images obtained by blurring an image containing fish with information about the fish in the image containing fish, with information about the fish in the image containing fish, and takes an image containing fish as input and outputs information about the fish in the image containing fish as output. The learning model is sized according to the device on which the fish school estimation device is implemented.

5. The fish school management system according to claim 3 or 4, wherein in the dataset, the blurred image of the fish is obtained by applying a filter that attenuates the intensity of the pixel region from the pixel region corresponding to the center of the fish in the image containing the fish toward the surrounding pixel region.

6. The fish school management system according to claim 5, wherein, in the dataset, the blurred image of the fish is obtained by applying a Gaussian filter to the pixel region corresponding to the center of the fish in the image containing the fish.

7. The fish school management system according to any one of claims 3 to 6, wherein the dataset includes a dataset generated using a simulation of a fish school enclosed by a fishing net using 3D CG (Three-Dimensional Computer Graphics) technology.

8. The fish school management system according to any one of claims 1 to 7, further comprising a camera system installed in the water for imaging fish in a school of fish within a fishing net, wherein the fish school estimation device is implemented in the camera system.

9. The fish school management system according to claim 8, wherein the camera system has a plurality of camera modules installed at different depths in the water, a plurality of fish school estimation devices are implemented at different depths in the water corresponding to the plurality of camera modules, and information based on the fish school estimation results is transmitted between the plurality of fish school estimation devices in order based on the depth in the water, thereby transmitting information based on the fish school estimation results along the different depths in the water.

10. The fish school estimation device further performs fish school estimation, including estimation of the size of the fish in the fish school, according to any one of claims 1 to 9.

11. The fish school management system according to any one of claims 1 to 10, further comprising a feeding device for feeding the fish school in the fishing net based on the results of the fish school estimation by the fish school estimation device.

12. The fish transport / inbound fish estimation device performs transport / inbound fish estimation, which further includes estimating the size of the candidate fish for transport / inbound based on an image of the candidate fish for transport / inbound, and the fish school management system further comprises a sorting device that sorts the fish to be transported or transported from the candidate fish for transport / inbound on the waterway based on the result of the size estimation of the candidate fish for transport / inbound by the transport / inbound fish estimation device, the fish school management system according to any one of claims 1 to 11.

13. The fish school management system according to any one of claims 1 to 12, further comprising a fishing net estimation device for estimating the state of the fishing net based on an image taken in the water including the fishing net.

14. A fish school management method comprising: a fish school estimation step, which includes estimating the number of fish in a school of fish based on images of fish in a school of fish installed in a fishing net underwater; and an inbound / outbound fish estimation step, which includes estimating the number of fish targeted for inbound and outbound transport, based on images including candidate fish for inbound and outbound transport acquired on a waterway used for transporting fish to and from the fishing net, and the results of the fish school estimation estimated in the fish school estimation step.

15. A fish school management program for causing a computer to perform the fish school management method described in claim 14.

Images