Method for detecting tracks in bearing-time plots

Abstract

A method for detecting tracks (34) in bearing-time plots is disclosed, comprising the following steps: a) continuously determining a bearing angle (33) of objects (32a, 32b, 32c) by means of a passive sonar (22) in order to obtain object bearings; b) applying a model (29) trained by means of machine learning to a representation of the object bearings over time (31), wherein the model (29) is designed to detect tracks (34a, 34b, 34c) of different objects (32a, 32b, 32c).

Description

[0001] Methods for detecting traces in bearing time.

[0002] Description

[0003] The invention relates to the automated evaluation of bearing time records, for example broadband sonar data (BDT).

[0004] Bearing-time plots are typically evaluated manually by a sonar operator. Objects are represented as (primarily bright) points in a time-beam plot (called a waterfall plot). The intensity (i.e., brightness) of each point in the plot corresponds to the sound intensity of the associated time / bearing pair. That is, the higher the sound intensity, the higher, and especially brighter, the intensity of the corresponding point in the plot. The sonar operator's task is to identify traces (i.e., lines) in the plot, i.e., the bearing-time plot. Such traces represent a detected object that is making noise.

[0005] In principle, sonar operators can perform the evaluation effectively. However, it's possible that incorrect parameter choices affecting the display could render very faint (i.e., quiet) objects invisible. Therefore, the multi-hypothesis test was developed for automated display evaluation. This test can detect traces in the display. However, this only works if the sonar data is evaluated across the entire frequency range (envelope evaluation). Some objects, however, emit underwater sound primarily in specific frequency ranges. These include ships whose propellers rotate at a constant speed. Such objects can be better detected through frequency-selective evaluation followed by combining the evaluations (frequency-selective evaluation).In this case, the number of hypotheses generated by the multi-hypothesis test can no longer be meaningfully processed. The object of the present invention is therefore to create an improved concept for the automatic evaluation of bearing-time recordings.

[0006] The problem is solved by the subject matter of the independent patent claims. Further advantageous embodiments are the subject matter of the dependent patent claims.

[0007] Exemplary embodiments show a method for detecting traces of bearing-time writings, for example in passive broadband sonar data (BDT sonar data), with the following steps a) and b).

[0008] In step a), the bearing angle of objects is continuously determined using passive sonar to obtain object bearings. Such a representation of object bearings as time versus bearing is also known as a waterfall display. The bearing angles can be determined, for example, by beamforming. A classic delay-and-sum beamformer or adaptive beamforming (ABF) can be used for beamforming.

[0009] In step b), a machine learning-trained model is applied to a time-series representation of object bearings. The model is trained to detect traces of different objects. That is, the model is applied to the waterfall display to detect the object traces. This type of data processing is also known as instance segmentation. The sonar data is segmented to identify the traces. Furthermore, each trace is assigned to an instance, meaning the traces can be distinguished by the model. It has been found that such a model is also capable of reliably detecting traces in the envelope analysis.

[0010] Instantiation, i.e., the distinguishability of the tracks, enables an optional step c) involving tracking the individual tracks to store them in a machine-readable format. This allows the tracks to be used, for example, in a situational awareness display or for navigation, particularly autonomous navigation. Tracking can be performed using a Kalman filter, especially a linear one. Furthermore, a step d) can be implemented, displaying the different tracks on a screen in such a way that the tracks are immediately distinguishable to a viewer. The display of the different tracks on the screen can, for example, be color-coded, i.e., each track is assigned its own color. Thus, the tracks are visible and distinguishable to the sonar operator at a glance and without adjusting any parameters.

[0011] This means that the detection of the tracks can be performed using the model, i.e., an artificial intelligence, within the sonar data. The model can optionally comprise multiple, particularly different, sub-models. A (sub-)model describes a trained algorithm. Suitable algorithms include deep learning approaches such as artificial neural networks based on transformer technology and / or convolutional or recurrent artificial neural networks. If the model comprises several sub-models for track detection and, optionally, classification, similar algorithms (e.g., different artificial neural networks) as well as different algorithms (e.g., an artificial neural network and linear regression) can be combined.

[0012] In particular, it has been shown that no special algorithms or algorithmic structures are needed to create the model. It is sufficient to train commercially available algorithms with appropriate data. Such algorithms are also known as COTS (Components-Off-The-Shelf). It has been found that the structure—for example, the number of neurons or the number of layers in artificial neural networks—is irrelevant for training the model, as long as a minimum level of complexity is achieved. Models that perform well in public benchmarks are particularly suitable. Examples of such benchmarks include the "COCO dev-test" and the "COCO val2017." Models based on the Mask-DINO (DINO: DETR with Improved deNoising anchOr box) algorithm or the Mask R-CNN algorithm are also examples. In "K. He, G. Gkioxari, P.Dollar and R. Girshick, "Mask R-CNN," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, pp. 2980-2988, doi: 10.1109 / ICCV.2017.322. (https: / / ieeexplore.ieee.org / document / 8237584)" and "F. Li et al., "Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation," 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, pp. 3041-3050, doi: 10.1109 / CVPR52729.2023.00297.

[0013] Examples of the models are revealed in (https: / / ieeexplore.ieee.org / document / 10204168).

[0014] Preferably, at least one (partial) model is implemented as an artificial neural network. The artificial neural network can, for example, be based on Transformer technology and / or be or include a convolutional artificial neural network.

[0015] If a single model without sub-models is used, it will preferentially detect all traces and classify them. If multiple sub-models are used, one sub-model can, for example, perform trace detection, while another sub-model can perform object classification.

[0016] Real, recorded bearing-time plots can be used to train the model or sub-models. However, it is also possible to generate artificial bearing-time plots.

[0017] Artificial (i.e., synthetic) training data is suitable. Artificial data can be used particularly when the amount of real training data is insufficient. Preferably, a combination of real and artificial data can be used for training. That is, artificial data can be supplemented with real sonar data. Real sonar data can be acquired during training runs, training maneuvers, or in the field. Preferably, the real data is pre-labeled, for example, by a trained sonar operator. That is, the actual events in the sonar data are pre-defined. This is called labeling. This enables supervised learning of the model. No manual labeling is necessary for artificial sonar data, since it is known where (i.e., at which bearing and at what time) an object was simulated and where only noise is present.The labeling can therefore be automated.

[0018] The idea is therefore to use AI to detect the traces in the bearing-time graphs and thus automate the detection for the evaluation of the envelopes as well, and to eliminate human errors, for example through an incorrect choice of parameters when displaying the waterfall representation.

[0019] In exemplary embodiments, a further step (e) can be performed to classify at least one object whose track intersects with the track of another object, in order to enable the assignment of the tracks to the objects after the intersection. Advantageously, both objects of the two tracks are classified to make the assignment of the tracks to the objects after the intersection more robust. If, by chance, more than two objects intersect their paths in close proximity, the additional objects can also be classified to assign all objects to their corresponding tracks after the intersection. The classification can be performed classically, i.e., by comparing a spectrum of the underwater acoustics of the objects. However, it is also possible for the model or another (sub-)model trained using machine learning to perform the classification.

[0020] In further embodiments, step a) involves determining the bearing angles of the objects in a multitude of frequency bands to obtain the object bearings in different frequency bands. In a further step a1) following step a), the object bearings in the different frequency bands are then combined into a single map. That is, a waterfall plot is first created for each frequency band. These waterfall plots are then combined into a single waterfall plot. Before combining, the individual waterfall plots for each frequency band can be normalized, in particular normalized to the same signal-to-noise ratio. This analysis of different frequency bands is referred to as BDT2. This is advantageous because it allows objects such as, for example,Continuously rotating ship propellers exhibit high intensity in only one frequency band and can therefore be detected more effectively than when the total energy is analyzed across all frequency bands in a waterfall representation.

[0021] Furthermore, a sonar system comprising a plurality of underwater sound transducers and a signal processing unit is disclosed. Each underwater sound transducer can convert underwater sound into an underwater sound signal corresponding to the sound pressure. The underwater sound transducers of the plurality can also be referred to as an array of underwater sound transducers. The signal processing unit can then perform the aforementioned process steps using the underwater sound signals from the underwater sound transducers of the plurality.

[0022] Likewise, a computer program is disclosed, comprising instructions that, when executed by a computer, cause it to perform the aforementioned procedure using underwater sound signals. In particular, the model is implemented as a sequence of instructions in the computer program.

[0023] Preferred embodiments of the present invention are explained below with reference to the accompanying drawings. These show:

[0024] Fig. 1: a schematic representation of a sonar system;

[0025] Fig. 2: an exemplary waterfall representation, where Fig. 2a is the waterfall representation itself and Fig. 2b is the waterfall representation with traces detected by the model.

[0026] Before exemplary embodiments of the present invention are explained in detail below with reference to the drawings, it should be noted that identical, functionally equivalent or equivalent elements, objects and / or structures in the different figures are provided with the same reference numerals, so that the description of these elements shown in different exemplary embodiments is interchangeable or can be applied to one another.

[0027] Fig. 1 shows a sonar system 20 with a plurality of underwater sound transducers 22 and a signal processing unit 24. The underwater sound transducers 22 of the plurality of underwater sound transducers can each convert underwater sound 26 into an underwater sound signal 28 corresponding to the sound pressure. The signal processing unit 24 can then use the underwater sound signals 28 to determine various bearings, i.e., directions of incidence of the underwater sound, e.g., by means of direction finding. The bearings can be determined for each sample value. At some receiving angles, i.e., bearings, an object can be detected; at other bearings, only noise. This results in a three-dimensional representation of time versus bearing and the intensity of the object, which is usually represented as the brightness of the bearing / time point as the third dimension.

[0028] Based on this three-dimensional representation, a model implemented in the signal processing unit 24, i.e., an artificial intelligence, can now detect traces of the object, i.e., a bearing of the object at each displayed time point. The signal processing unit 24 can therefore output a waterfall display 30 with traces of one or more objects.

[0029] Fig. 2a shows a schematic and idealized waterfall representation. Due to the requirements of patent drawings, this can only be shown in black and white and therefore only two-dimensionally. In reality, it is also possible to include grayscale representations in the model. Fig. 2a shows a binarized representation based on a threshold value. The vertical axis represents time 31 and the horizontal axis represents bearing 33. The third dimension is usually represented as the brightness of the points representing the objects, but this is not possible in this case. Visible in Fig. 2a are three objects 32a, 32b, and 32c. The first object, 32a, moves at the same speed and in the same direction as the sonar system. Therefore, object 32a is visible at the same bearing over the entire period under consideration.The second object 32b and the third object 32c each have a different direction of travel and / or a different speed, so that the bearing changes during the period under consideration.

[0030] Fig. 2b shows the waterfall representation from Fig. 2a, but with tracks 34a, 34b, 34c detected by the model. That is, the instances, i.e., the tracks, are again individually binarized to a threshold value and combined for the representation.

[0031] The disclosed (water) sound transducers are designed for underwater use, particularly in the sea. The transducers can convert underwater sound into an electrical signal (e.g., voltage or current) corresponding to the sound pressure, the (received) underwater sound signal. The transducers can therefore be used as underwater sound receivers. The transducer material can be a piezoelectric material, such as a piezoceramic. A plurality of underwater sound transducers, or one or more underwater sound transducers in conjunction with a signal processing unit, can be referred to as a sonar system. The transducers can be used for (passive) sonar (sound navigation and ranging, dl: sound navigation and range determination). The transducers are preferably not suitable for, or are not used for, medical applications.Likewise, sound transducers are preferably not used for ultrasonic testing of materials or are not suitable for this purpose.

[0032] Although some aspects have been described in connection with a device, it is understood that these aspects also constitute a description of the corresponding process, so that a block or component of a device can also be understood as a corresponding process step or as a feature of a process step. Similarly, aspects described in connection with or as a process step also constitute a description of a corresponding block, detail, or feature of a corresponding device.

[0033] Depending on the implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example, a magnetic or optical storage medium, on which electronically readable control signals, e.g., a computer program, are stored. These signals can interact with a programmable computer system (CPU and / or GPU) in such a way that the respective method is carried out. For this reason, the digital storage medium should be computer-readable. Embodiments can therefore include a data carrier that has electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

[0034] In general, in some embodiments, the methods are performed by any hardware device. This can be a universally applicable hardware device such as a computer processor (CPU) or a graphics processing unit (GPU). A distributed execution across both CPU and GPU is also possible.

[0035] The embodiments described above merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to other people skilled in the art. Therefore, it is intended that the invention be limited only by the scope of protection of the following claims and not by the specific details presented herein by way of description and explanation of the embodiments. Reference numerals:

[0036] 20 sonar systems

[0037] 22 underwater transducers

[0038] 24 Signal processing unit 26 Underwater acoustics

[0039] 28 Underwater sound signal

[0040] 29 Model

[0041] 30 Waterfall illustration

[0042] 31 Time

[0043] 32 objects

[0044] 33 bearing

[0045] 34 tracks

Claims

Patent claims 1. Method for detecting traces (34) in bearing-time records, in particular passive broadband sonar data (BDT sonar data) by the following steps: a) Continuously determining a bearing angle (33) of objects (32a, 32b, 32c) using a passive sonar (22) to obtain object bearings; b) Applying a machine learning-trained model (29) to a time-space mapping of object bearings (31), wherein the model (29) is trained to detect traces (34a, 34b, 34c) of different objects (32a, 32b, 32c).

2. Method according to claim 1, with the following further step: c) Tracking of the individual tracks in order to store the tracks in a machine-readable format.

3. A method according to any of the preceding claims, comprising the following further step: d) Displaying the different tracks (34a, 34b, 34c) on a screen in such a way that the tracks (34a, 34b, 34c) can be immediately distinguished for a viewer.

4. Method according to any of the preceding claims, comprising the following further step: e) Classifying at least one object (32a) whose track (34a) intersects with the track (34b) of another object (32b) in order to enable the tracks (34a, 34b) to be assigned to the objects (32a, 32b) after the intersection.

5. Method according to any one of the preceding claims, - wherein in step a) the bearing angles (33) of the objects (32a, 32b, 32c) are determined in a multitude of frequency bands in order to obtain the object bearings in different frequency bands; and- wherein in a further step a1) after step a) the object bearings in the different frequency bands are combined into a common mapping.

6. Sonar system with the following features: - a plurality of underwater sound transducers (22), each configured to convert underwater sound (26) into an underwater sound signal (28) corresponding to the sound pressure; - a signal processing unit (24) configured to apply the process steps according to one of claims 1 to 5 to the underwater sound signals (28).

7. Computer program comprising instructions which, when the program is executed by a computer, cause it to execute the method according to any one of claims 1 to 5 using underwater sound signals (28).

Images