Method and system for monitoring a volume of air surrounding a wind turbine

Abstract

A method, and a corresponding system, is provided for monitoring a volume (106) of air surrounding a wind turbine (102) in order to detect objects (114; 506, 507) that may require mitigation in order to avoid collision with the wind turbine. The method includes monitoring (401) the volume (106) with two cameras (104) at different positions and directed towards the wind turbine (102) such that the monitored volume (106) is defined as the volume that is observed by both cameras (104). For the respective cameras (104), at least one computer (108) is used to analyze (401) images from the camera in order to detect (402) the presence of an object of interest (114; 506, 507). If an object of interest (114; 506, 507), such as a bird, is detected in image sequences obtained simultaneously from the respective cameras (104), initiating (406) a pre-defined procedure for mitigation.

Description

METHOD AND SYSTEM FOR MONITORING A VOLUME OF AIR SURROUNDING A WIND TURBINETECHNICAL FIELD

[0001] The present invention relates to detection of birds in flight and collision mitigation efforts based on detection and position determination.BACKGROUND

[0002] Bird collisions with wind turbines pose significant risks, especially to large and migratory bird species, as turbine blades can cause fatal injuries. The importance of detecting birds approaching wind turbines lies in the opportunity it provides to mitigate these threats by implementing preventative measures, such as temporary shutdowns or adjustments in turbine operation. Technologies like radar and cameras as part of automated detection systems can identify bird movements and signal turbines to slow or halt, reducing collision rates. Through early detection and proactive responses, wind energy facilities can operate more sustainably, balancing renewable energy generation with wildlife conservation.

[0003] A combination of challenges must be overcome in order to provide efficient detection. These systems need to detect the presence of objects in the vicinity of wind turbines, avoid false positives caused by other objects or weather phenomena, differentiate between vulnerable and less vulnerable birds and preferably identify bird species, determine location and flight direction, and more. These challenges have been met by integrating multiple detection methods, typically requiring multiple types of sensors, e.g., camera, radar, thermal imaging and acoustic sensors. There is, however, a need to supplement these methods with additional options that can reduce system complexity and cost, in order to increase detection speed and reliability and reduce false positives.SUMMARY OF THE DISCLOSURE

[0004] In order to meet some of the needs outlined above, the present invention provides a method for monitoring a volume of air surrounding a wind turbine in order to detect objects that may require mitigation in order to avoid collision with the wind turbine. The method includes monitoring the volume with two cameras at different positions and directed towards the wind turbine such that the monitored volume is defined as the volume that is observed by both cameras. For the respective cameras, at least one computer is used to analyze images from the camera in order to detect the presence of an object of interest, and if an object of interest is detected in image sequences obtained simultaneously from the respective cameras, a pre-defined procedure for mitigation is initiated. The at least one computer may be one computer per camera, located locally, for example in the vicinity of the respective cameras,but the at least one computer may also be one central computer which is configured to analyze images from more than one camera.

[0005] In some embodiments of the invention the use of at least one computer to analyze images from the respective cameras includes tracking a detected object over a plurality of images, and if an object can be tracked for a predetermined period of time or over a predetermined number of images, using the at least one computer to determine if the detected object can be identified as a pre-defined object of interest. Exactly what an embodiment may be configured to determine is an object of interest may depend on requirements, but typically it may be one or more objects selected from the group consisting of: a bird, a flock of birds, a particular species of bird, a swarm of bats, and a drone.

[0006] Embodiments of the invention may identify objects of interest by providing one or more images as input to at least one of a machine learning algorithm and a decision network algorithm and receiving as output from the at least one of a machine learning algorithm and a decision network algorithm information indicating the presence or absence of an object of interest in the one or more input images.

[0007] While some embodiments may consider any detected object as an object of interest because the chance of false positives are small, in embodiments that use a machine learning algorithm or a decision network algorithm to identify objects of interest, the information indicating the presence of an object of interest may include at least one of: an identification of a species of bird, an identification of a flock of birds, an identification of a swarm of bats, a position in the image of the object of interest, and a direction of flight in the image of the object of interest.

[0008] Similarly, while some embodiments may consider simultaneous detection of an object of interest as sufficient evidence of the presence of an object of interest in the monitored volume, again because the risk of false positives is sufficiently small, some embodiments may also include using at least one computer to compare the object of interest detected in images from the first camera with the object of interest detected in images from the second camera in order to determine if the object of interest detected in images from the first camera and the object of interest detected in images from the second camera are the same object. The method may then only proceed to initiate mitigation if it is determined that the objects of interest detected in the images from both cameras are the same object. The computer performing the comparison may be a computer that also performs the image analysis described above, or it may be a separate computer.

[0009] The determination of whether an object of interest detected in images from the first camera and the object of interest detected in images from the second camera are the same object is made based on the consistency of variables determined for the object of interest detected in the images obtained from the first camera with variables determined for the object of interest detected in the images obtained from the second camera, the variablesbeing chosen from the group consisting of: identification, position, color, flight direction, wing beat frequency, size, and shape. Identification refers to determination of for example species of bird, or a more general class or type of object.

[0010] In some embodiments of the invention the method may use additional cameras with higher resolution and the ability to pan, tilt and zoom. These additional cameras may be associated with respective ones of the two cameras at different positions, and they may be used to, when an object is detected in images provided by a camera, track the object with the associated additional camera in order to obtain high resolution images of the object, and using images from the associated additional camera to determine if the detected object is an object of interest.

[0011] Some embodiments may further comprise subdividing images from the respective cameras vertically, using intersections of the subdivisions of the respective cameras to define a grid of horizontal positions in the monitored volume, obtaining coordinates representing a detected objects position in respective vertical subdivisions of images from the two cameras to look up a corresponding horizontal position in the monitored volume.

[0012] In another aspect of the invention a system is provided for monitoring a volume of air surrounding a wind turbine in order to detect objects that may require mitigation in order to avoid collision with the wind turbine. Such a system includes two cameras at different positions and directed towards the wind turbine such that the monitored volume is defined as the volume that is observed by both cameras, and at least one computer with a detection module configured to receive and analyze camera images in order to detect the presence of an object of interest in the monitored volume, and a comparison module configured to, if an object of interest is detected in images from the respective cameras, issue a mitigation instruction in order to initiate a pre-defined procedure for mitigation. It will be understood that the modules do not have to be separate modules, such that a system with a module that can perform both detection and comparison does have a module for detection and a module for comparison.

[0013] In some embodiments the at least one computer includes a tracking module configured to track a detected object over a plurality of images, and if an object can be tracked for a predetermined period of time or over a predetermined number of images, and an identification module configured to determine if a detected object can be identified as a pre-defined object of interest. As above, the modules may be distinct, but they do not have to be distinct, either from each other or from the modules mentioned above.

[0014] The identification module may be configured to identify at least one of: a bird, a flock of birds, a particular species of bird, a swarm of bats, and a drone.

[0015] The identification module may, in some embodiments, be configured to receive images as input, to identify objects of interest in received input images by performing at least one of a machine learning algorithm and a decision network algorithm, and to provide asoutput information indicating the presence or absence of an object of interest in received input images.

[0016] In some embodiments of the invention the comparison module is further configured to compare an object of interest detected in an image obtained by the first camera with an object of interest detected in an image obtained by the second camera in order to determine if the object of interest detected in an image from the first camera and the object of interest detected in an image from the second camera are the same object, and to only issue a mitigation instruction if can determine that the objects of interest detected in the images from both cameras are the same object.

[0017] The comparison module may be configured to make the comparison by comparing variables for an object of interest detected in a first image with variables for an object of interest detected in a second image, and to determine that the objects of interest detected in the images are the same object if the result of the comparison is that the variables to be consistent, the variables being chosen from the group consisting of: identification, position, color, flight direction, wing beat frequency, size, and shape. The variables may be obtained by image analysis and included in information provided by the modules performing detection, tracking or identification.

[0018] In some embodiments the system may also include additional cameras with higher resolution and the ability to pan, tilt and zoom, and associated with respective ones of the two cameras at different positions, wherein the additional cameras are configured to, when an object is detected in images provided by a camera with which it is associated, track the object in order to obtain high resolution images of the object.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The invention will now be described in further detail by way of examples and with reference to the enclosed drawings, where:

[0020] FIG. 1 illustrates some fundamental aspects of a system according to the invention where one wind turbine is monitored by two cameras;

[0021] FIG. 2 illustrates how several cameras localized with respective wind turbines may be used to monitor several wind turbines;

[0022] FIG. 3 illustrates an embodiment corresponding to the embodiment of FIG. 1, and also showing information flow between computers in the system;

[0023] FIG. 4 is a flow chart illustrating a process that is performed in accordance with the invention;

[0024] FIG. 5 shows how the invention can determine whether objects, such as birds, that are observed in images from two cameras are the same object;

[0025] FIG. 6 illustrates an embodiment where additional cameras with higher resolution image sensors and the ability to pan, tilt and zoom, can be used to track detected objects and provide higher resolution images for identification;

[0026] FIG. 7 illustrates how a horizontal grid for position determination can be defined by defining vertical sectors in the images provided from the cameras; and

[0027] FIG. 8 shows an exemplary embodiment of how modules performing different functions are distributed in a system according to the invention.DETAILED DESCRIPTION

[0028] The present invention relates generally to mitigating efforts in order to prevent or reduce occurrences where birds in flight collide with wind turbines or similar structures. The invention uses multiple cameras with overlapping fields of view and detection of a bird in more than one camera as an indication or confirmation of the bird's presence in a collision risk area. Upon such detection mitigating efforts may be initiated.

[0029] In the following description of various embodiments, reference will be made to the drawings, in which like reference numerals denote the same or corresponding elements. When letters are added to reference numerals, the letters serve to identify respective items that perform the same or similar functions, for example the respective cameras in a system. When referring to these collectively the reference number without the added letter may be used. The drawings are not necessarily to scale. Instead, certain features may be shown exaggerated in scale or in a somewhat simplified or schematic manner, wherein certain conventional elements may have been left out in the interest of exemplifying the principles of the invention rather than cluttering the drawings with details that do not contribute to the understanding of these principles.

[0030] It should be noted that, unless otherwise stated, different features or elements may be combined with each other whether or not they have been described together as part of the same embodiment below. The combination of features or elements in the exemplary embodiments are done in order to facilitate understanding of the invention rather than limit its scope to a limited set of embodiments, and to the extent that alternative elements with substantially the same functionality are shown in respective embodiments, they are intended to be interchangeable, but for the sake of brevity, no attempt has been made to disclose a complete description of all possible permutations of features.

[0031] Furthermore, those with skill in the art will understand that the invention may be practiced without many of the details included in this detailed description. Conversely, some well-known structures or functions may not be shown or described in detail, in order to avoid unnecessarily obscuring the relevant description of the various implementations. The terminology used in the description presented below is intended to be interpreted in itsbroadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific implementations of the invention.

[0032] With respect to terminology, it should particularly be noted that the terms volume, or volume of air, and area will be used interchangeably in this disclosure. Any observed area surrounding a wind turbine will be associated with a relevant height, resulting in an observed volume. However, the observed height is mostly the same for all observing cameras, while the position and direction of the cameras vary. As such, it is primarily the area around a turbine that is of interest for the purposes of the following description. However, the term area should always be understood to imply a volume, and the term volume should be understood as primarily defined as an area with a certain height. Furthermore, the term detection of an object will be used to refer to the determination of the presence of an object in one or more images, while identification of an object of interest will be used to refer to a determination of whether a detected object is a type of object that requires mitigation in order to avoid collision between the object and the wind turbine. In some cases, these categories may overlap by treating all detected objects as objects of interest.

[0033] Reference is first made to FIG. 1 which illustrates the most fundamental aspects of the invention. The drawing shows a system 100 including a wind turbine 102 and two cameras 104. Appropriate selection of camera position relative to the turbine 102, distance between the cameras 104, direction of view, and field-of-view angle, will result in a closed volume 106 containing the turbine 102 which is observed by both cameras, except for objects occluded by the turbine 102 from the point of view of one of the cameras. The field-of-view angle of a camera 104 may be changed by adjusting the zoom of the camera and this can be used to limit the monitored area to the area defined as collision risk area. Similarly, there will be a limit to how distant objects the system is able to detect. This limit, or range, may be determined by the size of the object, the resolution of the sensor, and the focal length (and / or f-number) of the camera.

[0034] Since the monitored volume is defined by the combined view of two cameras, detecting the same object in images provided by both cameras 104 means that the object is inside the volume 106, which is considered to be a collision risk area, and the system may issue a signal intended to trigger mitigation.

[0035] This setup requires only two cameras 104, while additional cameras may be added in order to increase the size of the covered area. It will be realized that increasing the size of the volume 106 by increasing its height will in most cases be unnecessary, while increasing the covered distance from the turbine 102 may be required. In the present disclosure the term area refers to the ground covered by the observed volume, while volume refers to the actual observed volume within which birds may be observed. However, for practical purposes area and volume can be considered to be synonymous.

[0036] The size of the area that can be covered with only two cameras will depend on, inter alia, field-of-view angle, focal length, and resolution, as well as the capabilities of the image processing and object recognition algorithms used to process the images in order to detect birds. In a reasonable setup with 8K cameras, two cameras should be able to cover an area with a radius of about 500 meters. This will give about 16 pixels per meter (varying with distance from camera), which means that for a bird with a wingspan of about 1 meter, there will be 16 pixels width for identification and tests have shown that this is sufficient for detection. Additional cameras may also be added in order to ensure that all areas may be viewed by at least two cameras without being occluded by the turbine 102.

[0037] The images from the cameras 104 may be delivered to a computer system 108 which performs processing in order to detect and recognize objects. Various algorithms are known in the art for detecting and recognizing objects in images. International patent application publication WO2022 / 220692 by the same applicant describes methods and systems for bird detection and species determination which may be used in conjunction with the present invention and the contents of this publication is hereby incorporated by reference in its entirety.

[0038] The computer system 108 may be connected to additional computer systems (not shown) which may be configured to control mitigating efforts upon receipt of a signal from computer 108 indicating that a bird has been detected in the monitored volume 106. In some embodiments, whether mitigating efforts are initiated may depend on the determined species of a detected bird. In some embodiments the computer 108 may handle detection and identification (e.g., species determination) as well as control of mitigating efforts.

[0039] One aspect of the invention is to take advantage of the fact that turbines in a wind farm are mounted on towers of substantially the same height and in view of each other. Mounting cameras 104 on turbine towers and directing them towards neighboring towers therefore provides an efficient way of obtaining the required camera views of respective turbines 104. This is illustrated in FIG. 2, where four turbines 102 are positioned relatively close to each other. In this example one turbine 102A is provided with two cameras, two turbines 102B, 102C are provided with three cameras, and one turbine 102D is not provided with any cameras. The cameras are not shown in this drawing, but they are implied by the indication of their respective fields of view. The fields of view are limited by field-of-view angle and a certain effective distance or range and are therefore shown as triangles. The monitored areas (volumes) 106A-C around any given turbine 102 are consequently delineated by field-of- view angle and range of the two cameras directed towards that turbine 102.

[0040] It should be noted that while the monitored area 106 in FIG. 1 is limited only by field- of-view angle, while in FIG. 2 they are also limited by range, this does not mean that this represents two distinct embodiments. Instead, various embodiments may include none, one or several cameras effectively limited by range. The limitation means that objects that are too far away from the camera to be sufficiently sharp and large (enough pixels in the image) to bedetected and identified as required by the algorithm described in further detail below. It will be realized that all cameras are ultimately limited in range (at some distance all relevant objects will be too small), but this is irrelevant in situations where the monitored area is entirely determined by the field-of-view of the cameras.

[0041] FIG. 3 illustrates the flow of information between computers or computer modules in an embodiment of the invention. In this drawing three turbines 102A-102C are shown. The volume 106 around the third turbine 102C is observed by two cameras 104A, 104B mounted on or close to the two other turbines 102A, 102B, for example on their respective towers.

[0042] The monitored volume 106 is limited by the field-of-view angles of the two cameras 104A, 104B. The respective cameras cannot determine the distance to an observed object, but images are transferred to respective local computers 108A, 108B where they are processed in order to detect the presence of an object. If an object is present in an image local computer 108 will attempt to identify the object. What the term identify actually means may vary depending on embodiments and will be described in further detail below.

[0043] If a local computer 108 identifies an object as something that may require mitigation in order to avoid or reduce the risk of collision, information to that effect is transferred to a central computer 110. The contents of this information may also vary according to embodiment and will be discussed below. If the central computer 110 receives information about an identified object from the two local computers 108A, 108B, the central computer 110 determines if the information provided from the two computers identify the same object. If so, the object is present in the monitored volume 106, which means that there is a collision risk. In this example a bird 114 is present in the monitored volume where it is observed by both cameras 104A, 104B and detected by both local computers 108A, 108B. The central computer 110 may then issue a mitigation instruction signal to a computer system 112 configured to control the turbine 102C in the observed volume 106, for example by temporarily stopping the turbine.

[0044] It will be realized that while the computers 108, 110, 112 described above are described as respective computer systems, they may also be respective modules implemented in the same computer system or distributed differently from the distribution shown in the drawing. For example, one local computer may receive images from more than one camera, or the central computer 110 and the computer system 112 controlling the turbine 102C may be implemented in a single system. Conversely, functionality may be distributed between additional computers or computer modules. For example, detection of objects in an image requires a lower number of pixels and less computational power than identifying the object as a bird of a specific species. A first module may therefore be used to detect the presence of objects that may be a bird (or any other object that may require mitigation), and a second module may perform identification when such an object has been detected. Functionality may also be replicated over several systems, for example in order to provide redundancy.

[0045] In the most straightforward embodiments of the present invention detection and identification is one and the same in the sense that detection of an object in an image means that the object is considered to be something that represents a collision risk. This may be the case for embodiments where the risk of false positives is negligible. For example, for a wind farm in an area where substantially all birds are endangered and the likelihood of detecting an object that is not such a bird is very small, detection of an object may be sufficient to require mitigation. This may also apply to the determination of the identity of the object, meaning that if two cameras observing the same volume simultaneously detect an object it is automatically assumed that the two cameras are observing the same object. Simultaneously, it should be understood as being detected in video or image sequences that overlap in time and does not imply that respective images have to be captured at the exact same time.

[0046] In more sophisticated embodiments these assumptions are not necessarily made because they would produce too many false positives. False positives could be the result of detection of objects that are actually in the monitored volume but do not require mitigation, or by simultaneous observation of two different objects that are not in the monitored volume, for example such that the first camera observes a first bird that is outside the monitored volume and the second camera observes a second bird that is outside the monitored volume. Different methods may be used or combined in order to prevent false positives.

[0047] First of all, when an object is detected, it may be determined to have certain characteristics. Exactly which characteristics the system looks for may vary depending on embodiments, but may, for example, include color, direction of flight, or flying height. In some cases observed characteristics may be used to determine whether or not the object is an object of interest (for example, the color may be inconsistent with any type of bird that requires mitigation), but characteristics may also be used to determine whether it is the same object that is observed by the two cameras. If color or flight direction is inconsistent it may be determined that it is not the same object that is being observed, and consequently that the observations do not represent a single object inside the monitored volume.

[0048] Even more sophisticated systems may be able to detect birds and perhaps also determine the species of bird. Determination of species of bird may be based on observation of several variables, and may be rules based, utilize a statistical method (e.g., a Bayesian belief network), use a trained neural network, or a combination of these. A detailed discussion of such methods can be found in the PCT patent application publication referenced above and incorporated by reference. It will be realized that variables that are consistent with identification of a particular species of bird may be inconsistent with identification of the same individual bird. For example, if the two cameras are viewing a bird that is determined to be an eagle, but the directions of flight are inconsistent, it may be concluded that the two cameras are viewing two different eagles.

[0049] It should also be noted that while the present disclosure mainly refers to detection and identification of individual birds, the detection and identification may also be configuredto identify flocks of birds, or even a cloud, flock, or swarm of bats. (In the present disclosure swarm of bats will be used to also include cloud and flock.) When identifying flocks or swarms rather than individual objects, the variables that become important for identification may change (e.g., size and shape rather than color and wing beat frequency) but the principles involved are the same. The invention may, of course, also be used to identify drones, or other flying man-made objects, and under certain conditions also swarming or migrating insects.

[0050] Turning now to FIG. 4, a method consistent with the principles of the invention will be described with reference to a flow chart. In the illustration, a first step 401 includes observation of the monitored volume 106 by two or more cameras 104 and provision of obtained images to respective local computers 108 or computer modules. In a following step 402 the local computers 108 perform detection and tracking, for example based on computer vision and machine learning algorithms. If an object is detected and can be tracked for a predetermined period of time, for example one second, or the corresponding number of image frames, the process proceeds to a next step 403 where it is determined if the detected object is an object of interest. An object may, for example, be a bird, and an object of interest may be a particular species of bird.

[0051] When an object of interest has been identified in step 403 the process moves to step 404 where identifications from two cameras are combined and compared. This step may be performed in a central computer 110, or at least in a computer module configured to receive identification information based on image data from two different cameras. In this step it is determined whether the identified objects in the images from the two cameras can be assumed to be one and the same object. This determination can be based on a selection of variables derived from the analysis of the respective images, as discussed above. If these variables are consistent, for example that they identify the same species of bird, substantially the same flight track, for example by determining that position in images with the same time stamp are consistent over time.

[0052] If the comparison of the information is that the two cameras are not tracking the same object, it is decided in step 405 that the process should return to step 401 and continue to monitor the volume 106 around the turbine 102. If, however, it is determined that the cameras are tracking the same object the process proceeds to step 406 where mitigation is initiated, for example by issuing a command to a computer system 112 configured to control the turbine in a manner determined to remove or reduce the risk of collision.

[0053] It should be realized that while the steps of this method are described as being performed in sequence, an actual system will most likely perform these steps continuously. For example, the cameras 104 will deliver a continuous stream of images to the local computers 108 and these images will be analyzed in order to detect new objects or continue to track already detected objects whether or not previously analyzed images with detected objects are being processed downstream for object identification and comparison.

[0054] Reference is now made to FIG. 5 for a discussion of how two objects, such as birds, may be detected in the field of view of two cameras and how it may be determined whether they are the same object. Two cameras 104A, 104B are located at different positions and directed towards a wind turbine (not shown). The first camera 104A obtains an image with a first image plane 501A in which two birds 506, 508 are present. The first camera may also be able to view the second camera which will be present in the image plane at location 503. Similarly, the second camera 104B obtains an image with a second image plane 501B. In the image obtained by the second camera 104B two birds 507, 509 are present as well. The task now is to determine whether the two cameras are observing the same two birds. Since the respective camera positions are known as well as the camera direction and viewing angle (zoom), it is possible to determine a line from a camera along which an observed object must be positioned. If two lines from respective cameras meet, such as in point 505 it can be determined that the two cameras are observing an objects that are in the same position, which means that it is the same object. In this case bird 506 observed by the first camera 104A and bird 507 observed by the second camera 104B is one and the same. And since the bird can be observed by both cameras it must be in the monitored area.

[0055] It will be realized that if the position 505 is determined exactly the position of the bird will be known exactly. However, this may not be necessary. It may be sufficient to determine whether the positions in the respective image planes are inconsistent with each other, for example by having clearly different altitude. An alternative or supplementary approach is to identify characteristics such as color, wing beat frequency, direction of flight, size and shape, and if there are inconsistencies it may be determined that two different birds are being observed and that they therefore are not in the monitored area.

[0056] The bird 508 which is observed by the first camera 104A is clearly flying at a different altitude than the bird 509 observed by the second camera 104B, and they are also flying in different directions. It may therefore be concluded that these are two different birds and that they are both outside the monitored area.

[0057] It is not necessary that the two cameras observe each other. However, and particularly if the cameras are capable of panning, tilting and changing zoom angle, the position of the other camera in the image plane can be used as a reference when determining directions to observed objects.

[0058] Turning now to FIG. 6 embodiments will be presented wherein the main cameras 104 are fixed and they are supplemented with additional cameras 204 with higher resolution and the ability to pan, tilt and zoom. As described above, the main cameras 104 define an area (or, more precisely, a volume) 106 that is monitored. Whenever one of the main cameras 104 detects an object 114, the associated additional camera 204 will start to track that object. The additional camera 204 will obtain a zoomed in high resolution image of the detected object and this image provides better information for identification. Furthermore, when two additional cameras 204 associated with respective main cameras 104 both track an object theidentification information from the two additional cameras 204 can be supplemented with pan and tilt angles (camera direction) and zoom factor (focal length) in order to determine whether the two sets of cameras are observing and tracking the same object. In this way, fixed low resolution low focal length cameras 104 can observe and monitor a large area / volume around the turbine 102 requiring relatively little processing power, while high quality images and more precise positioning information is provided when an object that may be a bird requiring mitigation is detected. Thus, more processing power can be made available only when it is needed. The additional cameras may, for example, be 4K PTZ (pan, tilt, zoom) cameras with a 200 millimeter optical zoom which will be able to provide a resolution of 63 pixels per meter at a distance of 1500 meters. This is sufficient for identification and is one way of providing the ability to distinguish between most bird species of interest. Other camera specifications may be chosen depending on requirements for a particular installation and future development of camera technology.

[0059] In the description above it has been explained how position in an image, and in some cases the direction of a panning and tilting additional camera, can be used to determine the position of an object such as a bird. In some embodiments this positioning is improved further by defining vertical sectors. The intersection of the sectors from two such cameras will provide a grid. FIG. 7 provides a simplified illustration of this wherein only the respective fields of view for two cameras and their subdivision into sectors. It will be seen that the result is a grid of intersections 701 and the position of each such intersection can be determined in advance. Thus, determining the position of an object, for example a bird, that is observed by both cameras will be defined by the coordinates representing the two sectors within which the object is observed. It will further be realized that the sectors can be as narrow as a single pixel, providing a very high accuracy very quickly and without the use of costly computational resources.

[0060] Some embodiments of the invention use known information about the dimensions of the turbine in view to calibrate the system. Combining this information from two cameras and treating the system as a conventional stereo vision system for this purpose, will provide high precision 3D coordinates for tracked birds.

[0061] FIG. 8 illustrates an exemplary embodiment of how software modules may be implemented in a system configured substantially as illustrated in FIG. 3. It should be noted that this represents one possible distribution of functionality over several computers and modules, while the invention encompasses other alternatives that are not illustrated in the interest of brevity.

[0062] The system includes at least two cameras 104A, 104B as described above. In order to monitor the space around more than one wind turbine, additional cameras are needed.

[0063] Associated with the respective cameras 104 are local computers 108. A local computer 108 includes a detection module 801, a tracking module 802 and an identificationmodule 803. A detection module 801 receives input from a camera 104 and processes the input in order to detect the presence of objects in the received images. If an object is detected, a tracking module 802 tracks that object and an identification module 803 attempts to identify the object. As described above, various criteria may be applied. For example, a detected object may only be considered potentially interesting after it has been tracked for a predetermined period of time, and the identification module may then determine whether the tracked object can be identified as belonging to a category or class that may require mitigation. Different embodiments of the invention may implement different criteria. For example, some embodiments may require the object to be identified as a bird, other embodiments may require identification of a specific species of bird. Other options are possible, as described above.

[0064] If the object is determined that the object is an object of interest according to the criteria defined for a specific installation, information is forwarded to a comparison module 804 which in this embodiment is part of a central computer 110 (here illustrated as a computer system including both computers 110, 112 from FIG. 3). The comparison module 804 receives information from both (or all) cameras monitoring the same space around a wind turbine and compares the received information in order to determine if the cameras have detected the same object. As explained above, two cameras may observe different objects which are not present in the monitored space, but if an object is present in the monitored space, it is visible to both cameras. (Excepting situations where the object is directly behind some obstacle as viewed from one camera, for example directly behind the wind turbine tower.) If the comparison module determines that the two cameras have detected the same object, it means that the object is present in the volume that is defined by the field of view of the two cameras, and since it was determined by the identification modules 803 that the object was an object of interest, it is concluded that mitigation is necessary. The comparison module provides this information to a mitigation module 805 that may issue an alarm, or that may be configured to control the wind turbine directly, for example by slowing the turbine down or stopping it completely.

[0065] The various modules may use different forms of computer vision and recognition. Typically, the detection module 801 will require relatively little computing power but be able to process entire images quickly. The module may, for example, include a convolutional neural network (CNN). The tracking module 802 may be an additional neural network, but in some embodiments detection and tracking is performed by the same neural network. Detection and tracking may, for example, be done using YOLOv8 (You Only Look Once), which includes various CNN and fully connected layers. The identification module 803 may be another neural network and may also combine a neural network with domain knowledge as described in the above mentioned application WO2022 / 220692 by the same applicant.

[0066] The comparison module 804 may require that the same species of bird is identified in images from both cameras. In addition, some embodiments may use additional informationsuch as flight speed and direction, and position as determined based on position in the respective images and the known position of the cameras. This has been discussed above in greater detail.

[0067] Finally, the mitigation module may be as simple as a communication module configured to issue an alarm, or it may be a sophisticated control system able to control the wind turbine that is monitored.

[0068] As mentioned, the illustrated embodiment is exemplary. Other possible configurations include use of a single computer which receives information from more than one camera, in which case there may be only one of the detection, tracking and identification modules. Other embodiments may include the comparison module 804 and the mitigation module 805, or at least the comparison module 804, in one master computer which also includes detection, tracking and identification, while an additional computer includes detection, tracking and identification, but transmits information to the master computer for comparison.

[0069] In some embodiments there may not by any identification module 803, or detection, tracking and identification may be performed by what is essentially a single module which does not require identification beyond that which is provided by detection and tracking (i.e., an object is considered to be of interest if it is detected and can be tracked, and no further identification is made).

Claims

CLAIMS1. Method for monitoring a volume (106) of air surrounding a wind turbine (102) in order to detect objects (114; 506, 507) that may require mitigation in order to avoid collision with the wind turbine, comprising: monitoring (401) the volume (106) with two cameras (104) at different positions and directed towards the wind turbine (102) such that the monitored volume (106) is defined as the volume that is observed by both cameras (104); for the respective cameras (104), using at least one computer (108) to analyze (401) images from the camera in order to detect (402) the presence of an object of interest (114; 506, 507); and if an object of interest (114; 506, 507) is detected in image sequences obtained simultaneously from the respective cameras (104), initiating (406) a pre-defined procedure for mitigation.

2. Method according to claim 1, wherein the using of at least one computer (108) to analyze (401) images from the respective cameras (104) includes tracking a detected object over a plurality of images, and if an object can be tracked for a predetermined period of time or over a predetermined number of images, using the at least one computer (108) to determine (403) if the detected object can be identified as a pre-defined object of interest.

3. Method according to claim 1 or 2, wherein an object of interest is an object selected from the group consisting of: a bird, a flock of birds, a particular species of bird, a swarm of bats, and a drone.

4. Method according to one of the previous claims, wherein an object is determined to be an object of interest by providing one or more images as input to at least one of a machine learning algorithm and a decision network algorithm, and receiving as output from the at least one of a machine learning algorithm and a decision network algorithm information indicating the presence or absence of an object of interest in the one or more input images.

5. Method according to claim 4, wherein information indicating the presence of an object of interest includes at least one of: an identification of a species of bird, an identification of a flock of birds, an identification of a swarm of bats, a position in the image of the object of interest, and a direction of flight in the image of the object of interest.

6. Method according to one of the previous claims, further comprising using at least one computer (110) to compare (404) the object of interest detected in images from the first camera with the object of interest detected in images from the second camera in order to determine (405) if the object of interest detected in images from the first camera (104A) and the object of interest detected in images from the second camera (104B) are the same object (114; 506, 507); andonly proceeding to initiate mitigation if it is determined that the objects of interest detected in the images from both cameras (104) are the same object.

7. Method according to claim 6, wherein the determination (405) of whether an object of interest (114; 505, 506) detected in images from the first camera and the object (114; 505, 506) of interest detected in images from the second camera are the same object is made based on the consistency of variables determined for the object of interest detected in the images obtained from the first camera with variables determined for the object of interest detected in the images obtained from the second camera, the variables being chosen from the group consisting of: identification, position, color, flight direction, wing beat frequency, size, and shape.

8. Method according to one of the previous claims, further comprising using additional cameras (204) with higher resolution and the ability to pan, tilt and zoom and associated with respective ones of the two cameras (104) at different positions, to, when an object is detected in images provided by a camera (104), track the object with the associated additional camera (204) in order to obtain high resolution images of the object, and using images from the associated additional camera (204) to determine if the detected object is an object of interest.

9. Method according to one of the previous claims, further comprising subdividing images from the respective cameras (104) vertically, using intersections of the subdivisions of the respective cameras to define a grid of horizontal positions in the monitored volume (106), obtaining coordinates representing a detected objects position in respective vertical subdivisions of images from the two cameras (104) to look up a corresponding horizontal position in the monitored volume (106).

10. System for monitoring a volume (106) of air surrounding a wind turbine (102) in order to detect objects (114; 506, 507) that may require mitigation in order to avoid collision with the wind turbine, comprising: two cameras (104A, 104B) at different positions and directed towards the wind turbine (102) such that the monitored volume (106) is defined as the volume that is observed by both cameras (104); at least one computer (108, 110, 112) with: a detection module (801) configured to receive and analyze (401) camera images in order to detect (402) the presence of an object of interest (114; 506, 507) in the monitored volume (106); and a comparison module (804) configured to, if an object of interest (114; 506, 507) is detected in images from the respective cameras (104), issue a mitigation instruction in order to initiate (406) a pre-defined procedure for mitigation.

11. System according to claim 10, further comprising:in the at least one computer (108, 110, 112): a tracking module (802) configured to track a detected object over a plurality of images, and if an object can be tracked for a predetermined period of time or over a predetermined number of images, and an identification module (803) configured to determine (403) if a detected object can be identified as a pre-defined object of interest.

12. System according to claim 11, wherein the identification module (803) is configured to identify at least one of: a bird, a flock of birds, a particular species of bird, a swarm of bats, and a drone.

13. System according to claim 11 or 12, wherein the identification module (803) is configured to receive images as input, to identify objects of interest in received input images by performing at least one of a machine learning algorithm and a decision network algorithm, and to provide as output information indicating the presence or absence of an object of interest in received input images.

14. System according to one of the claims 10 to 13, wherein the comparison module (804) is further configured to compare (404) an object of interest detected in an image obtained by the first camera with an object of interest detected in an image obtained by the second camera in order to determine (405) if the object of interest detected in an image from the first camera (104A) and the object of interest detected in an image from the second camera (104B) are the same object (114; 506, 507), and to only issue a mitigation instruction if can determine that the objects of interest detected in the images from both cameras (104A, 104B) are the same object.

15. System according to claim 14, wherein the comparison module (804) is further configured to make the comparison by comparing variables for an object of interest detected in a first image with variables for an object of interest detected in a second image, and to determine that the objects of interest detected in the images are the same object if the result of the comparison is that the variables to be consistent, the variables being chosen from the group consisting of: identification, position, color, flight direction, wing beat frequency, size, and shape.

16. System according to one of the claims 10 to 15, further comprising additional cameras (204) with higher resolution and the ability to pan, tilt and zoom and associated with respective ones of the two cameras (104) at different positions, wherein the additional cameras are configured to, when an object is detected in images provided by a camera (104) with which it is associated, track the object in order to obtain high resolution images of the object.

Images