75 results about "Sound event detection" patented technology

The invention relates to the technical field of environmental sound event detection, in particular to an environmental sound event detection method based on multi-modal data fusion, which utilizes a microphone array, a camera and a sensor to synchronously acquire audio, video and environmental data to form a multi-modal data source. And the collected multi-modal data is preprocessed, so that the data quality and the accuracy of subsequent analysis are improved. And key features are extracted from the processed data, and the features are fused to form multi-modal features. And a deep learning model is constructed, and multi-modal features are used for training, so that a model capable of identifying an environmental sound event can be obtained. And performing optimization such as pruning, quantification and knowledge distillation on the trained model, and deploying the model on edge equipment to realize real-time preliminary feature extraction and event detection. The problems that an environment sound event detection method in the prior art is poor in robustness, low in real-time performance and limited in generalization ability are solved.

In general, the disclosure describes a computing system to automatically identify and classify audio input, including non-speech audio signals. The computing system may also add new classes based on only a limited number of examples of the new classes, to identify classes of sounds for which the system had not been trained.

The invention discloses an all-night snore detection method based on a frequency domain attention and self-attention pooling double-fusion mechanism, and belongs to the field of sound event detection. The method comprises the following steps: designing a waveform feature extraction module based on a frequency domain self-attention sub-module, a Mel-cepstrum feature extraction module based on a deformable convolution residual sub-module, a self-attention pooling double-fusion module and a decoder based on a multilayer perceptron; a deep learning snore event detection model based on a frequency domain attention and self-attention pooling double-fusion mechanism is constructed by using the modules. The model can detect and position snoring sound in all-night sleep sound data in a point-by-point framing detection mode by loading trained parameters, and the starting and ending time of a snoring event in all-night time and the corresponding snoring sound type are obtained. The model can rapidly and accurately detect snore segments in the whole night sleep period, and is helpful for assisting medical staff in rapidly screening patients with sleep breathing disorder in the early stage.

The invention discloses a multi-sound event detection method based on a multilayer neural network, and relates to the field of sound event detection. The invention provides a weak label multi-sound event detection model based on a fusion empirical mode decomposition algorithm, a capsule network, an attention mechanism and a bidirectional gating loop network. In the model, firstly, an AMM-EMD algorithm is used for preprocessing, sound event feature information is enriched, and a parallel convolution structure is introduced to extract advanced feature information; further processing the advanced features by using the strong generalization ability of the capsule network and the ability of capturing global dependence by an attention mechanism; and meanwhile, the BiGRU module is used for acquiring context information. According to the model provided by the invention, contrast ablation experiments are carried out on different data sets, the sound event detection effect is evaluated by utilizing F1 and ER, the model provided by the invention is improved compared with other baseline models, and F1 and ER are improved.

A method for detecting a sound field in an outside environment of a solid structure is provided. The solid structure comprises at least one sound transducing element, which is a part of the solid structure and forms a part of an outer surface of the solid structure, wherein the incident sound falls at least partly onto the sound transducing element. The method comprises obtaining sensor data from at least one vibration sensor, which is mounted onto a mounting surface of the at least one sound transducing element, wherein the at least one vibration sensor detects vibrations in the at least one sound transducing element, which are induced into the at least one sound transducing element by the incident sound field, processing the sensor data to generate output data, and providing the output data.

The invention relates to a semi-supervised underwater sound event detection method based on joint disturbance consistency constraint. The method comprises the following steps: acquiring logarithmic Mel spectrums of strong label audio data, weak label audio data and unlabeled audio data; performing weak enhancement and strong enhancement processing on the logarithmic Mel spectrum of the unlabeled audio data to obtain a corresponding enhanced spectrum; constructing a deep learning network model, and performing supervised training by using the frequency spectrums of the strong label audio data and the weak label audio data; adopting a multi-feature disturbance and strong disturbance combined semi-supervised model for the unlabeled audio data; and finally, optimizing the training process of the deep learning network model by calculating a joint loss function. And detecting an underwater sound event by using the trained deep learning network model, and outputting an event category of the audio data. The method is used for realizing effective detection of the sound event by using a small amount of strong injection data and improving the model performance.

The invention discloses a sound event detection method and system based on class distribution and time sequence context collaborative prompt, and the method comprises the steps: converting an original audio signal into a signal frame sequence, extracting the features of a Mel filter bank through a pre-training model branch, and extracting the features of a Mel spectrum through a downstream model branch; a combination of a local time sequence prompt module and a global distribution prompt module is introduced into each layer of the pre-training model, and the pre-training model outputs audio sequence features and the global distribution prompt module; performing feature fusion on the audio sequence and the output of the downstream model, calculating the frame level prediction probability of the downstream model, and realizing an audio positioning task; generating a sentence level prediction probability from the frame level prediction probability of the downstream model; for the pre-training model, processing the global distribution prompt module to obtain a sentence level prediction probability of the pre-training model; and finally, fusing the two sentence level prediction probabilities to obtain a classification result of the sound event. According to the method, the audio classification and positioning performance of sound event detection can be remarkably improved.

The invention relates to the crossing field of artificial intelligence and bird ecology, and particularly discloses a large-scale bird sound recognition method based on a deep learning technology, and the method comprises the steps: collecting a bird sound audio through a project region recording device, training a bird sound event detection model through a Bird audio detection challenge 2018 data set, carrying out the class prediction through field recording data, and carrying out the recognition of a bird sound event. Effective bird sound and background noise are separated; constructing a bird sound recognition data set in combination with a bird species catalog of the Chinese bird watching recording center and an audio file of Xen-Canto, training a bird sound recognition model by using background noise and the bird sound recognition data set, and performing verification; and finally, inputting the effective bird sound data into the model to carry out label prediction so as to identify the bird sound. The invention aims to carry out wide bird voice recognition on a project area under construction through a deep learning technology, so that the data processing cost is saved, and the recognition accuracy in a complex scene is improved.

A method for real-time sound event detection on an embedded device includes pretraining a contrastive language-audio pretraining model as an audio foundation model and preparing offline multimodal query prototypes for sound events of interest. The pretrained model and query prototypes are deployed on an embedded device. The device receives an input audio stream and extracts audio embeddings using the pretrained model. Similarity scores are calculated between the extracted audio embeddings and the prepared query prototypes. The presence of a sound event is determined based on the calculated similarity scores, and a real-time sound event detection result is output. The system includes a memory storing the pretrained model and query prototypes, an audio input interface, and a processor configured to perform the extraction, calculation, determination, and output operations. A non-transitory computer-readable medium stores instructions that, when executed, cause a processor to perform the method.

In general, the disclosure describes a computing system to automatically identify and classify audio input, including non-speech audio signals. The computing system may also add new classes based on only a limited number of examples of the new classes, to identify classes of sounds for which the system had not been trained.

The invention discloses a semantic prompt-based sound event detection data synthesis and sound event detection model training method. The sound event detection task is converted into the semantic description information, and the structured semantic prompt instruction which accurately reflects the target sound event characteristics is generated in combination with the semantic constraint rule, so that automatic mapping from semantic description to instruction generation is realized, and the manual intervention cost is reduced. And inputting the structured semantic prompt instruction into the audio generation model, and synthesizing the audio data in a large scale, thereby reducing the data acquisition cost and improving the sample diversity and expandability. The structured semantic prompt instruction can guide the model to synthesize audios of various sound event types in batches, and is automatically generated by a large language model to ensure that the synthesized audios are strictly aligned with instruction semantics. When the label is generated, the sample event type can be obtained without manual labeling, and an efficient and reliable data source is provided for sound event detection model training.

The invention provides a sound event detection method, and belongs to the technical field of sound event detection, and the method comprises the steps: obtaining a to-be-detected audio signal; preprocessing the audio signal to be detected; performing data enhancement on the audio signal after data preprocessing; performing feature extraction on the audio signal after data enhancement; performing context information extraction on the extracted features to obtain first branch features; inputting the audio signal after data enhancement into a pre-trained Transfomer-based audio teacher-student model to obtain a second branch feature; and splicing the first branch feature and the second branch feature, and inputting the spliced first branch feature and the spliced second branch feature into a classifier to obtain a sound event detection result of the to-be-detected audio signal. According to the invention, enough background information corresponding to different sound events can be obtained, and the multi-scale feature capturing capability of the model is improved.

The invention discloses a chicken flock sound event detection method based on multi-scale perception and time domain dynamic enhancement, and the method comprises the steps: carrying out the random screening and marking based on chicken flock sound data, and obtaining a standard data set with a label; preprocessing the sound signals in the standard data set to obtain a spectrogram; intercepting a spectrum picture segment containing chicken flock sound in the spectrogram, and converting the spectrum picture segment into a Mel spectrogram through a Mel filter bank; constructing a depth feature extractor, and extracting sound depth features of the Mel spectrogram by using the depth feature extractor; and constructing a frame level sound classifier, processing and analyzing sound depth features, and obtaining time domain distribution of chicken flock sound events. According to the method, the time domain distribution of the sound events of the chicken flocks can be accurately and quickly extracted in a long-time and noisy breeding environment.

Systems and methods for improving the accuracy and recall rate of prototype networks for sound event detection. A method of training a prototype network for sound event detection includes: receiving samples of an audio signal, the samples including positive samples corresponding to sound events and negative samples not corresponding to sound events; determining corresponding positive prototypes of the plurality of sound event categories based on the positive samples; determining, based on the negative samples, a respective negative prototype for a respective negative sample group, each negative prototype corresponding to a combination of a plurality of negative samples; and generating an output signal indicating whether the first sample belongs to one of the plurality of sound event categories based on a comparison between the first sample and the respective positive prototype and each negative prototype.

The invention relates to the technical field of artificial intelligence, in particular to a multi-sound event detection positioning method and device based on a neural network model. The method comprises the following steps: innovatively designing a time-frequency multi-scale residual error convolution block, forming a network model with a Conformer module and a cross-stitch unit module, extracting features in a multi-scale manner, enhancing long sequence modeling, promoting task collaborative optimization, and improving performance and accuracy; in the aspect of data processing, pre-emphasis and framing windowing improve feature quality, audio channel exchange and spectrum enhancement increase data diversity and reduce overfitting, and SALSA-Lite features are adopted to enhance feature expression. In the training strategy, a multivariate loss function is applied to accelerate convergence in consideration of task requirements, and hyper-parameters are flexibly adjusted by means of a verification set. Therefore, the method is high in training efficiency, excellent in actual performance, high in generalization ability on unknown data, capable of accurately coping with complex and changeable actual scenes and capable of effectively overcoming the defects of a traditional method.

The application discloses a sound event detection method, device and equipment and a storage medium. The method comprises the following steps: acquiring to-be-detected audio data, and extracting acoustic characteristics of the to-be-detected audio data; determining a sound event detection result of the to-be-detected audio data according to the acoustic characteristics and a pre-determined sound event relationship characteristic; wherein the sound event relationship characteristic is determined based on a pre-constructed sound event relationship graph; and the sound event relationship graph is determined according to a statistical result of sound events in an audio data set. The technical scheme solves the problems of limited application range and low accuracy of sound event detection, and can effectively expand the application range of the detection scene while improving the accuracy of sound event detection.

The invention discloses a human voice data collection method and device, equipment and a storage medium. The method comprises the following steps: inputting received audio data into a sound event detection model; a feature extraction module of the model performs feature extraction on the audio data to obtain audio features; respectively inputting the audio features into a strong label learning module, a multi-instance learning module and a CTC module of the model, correspondingly outputting a frame-level event probability, a frame-level boundary probability, an audio-level existence probability, a frame-level attention weight and a sound event prediction sequence, and fusing the output results to obtain a target detection result; and when the target detection result represents that the voice event exists in the audio data, extracting target voice data from the audio data based on the target detection result. According to the method, the audio data with different granularity labels are optimized through the multi-task learning framework of the model, different task output results are fused, the time sequence detection problem of multi-sound event overlapping is solved, and the accuracy of the target human voice data is guaranteed.

In one possible example aspect, the present disclosure provides a context-aware noise compensation system for a wearable device playing back media content to a user in a noisy environment, wherein the system is configured to: obtain a sound signal from the noisy environment; determine context information associated with the sound signal, wherein the context information comprises at least one of: vocal sound detection information indicative of presence of one or more vocal sounds of the user in the sound signal, or sound event detection information indicative of presence of one or more environmental sound events in the sound signal; and optimize, based on the context information, the media content for playback by the wearable device.

The invention discloses a small sample sound event detection method based on multi-scale feature aggregation, and relates to the technical field of audio signal processing and machine learning. The method comprises the following steps: firstly, preprocessing an input audio signal and extracting Mel spectrogram features; then, the features are input into a specially designed multi-scale feature aggregation network, the network extracts a time dimension, a frequency dimension and a local time-frequency joint feature through three parallel convolution paths, and deep fusion is carried out on multiple paths of features; thirdly, training the network by adopting a model-independent meta-learning framework, and enabling the network to learn the capability of quickly adapting to new tasks by simulating a large number of small sample classification tasks; and finally, using the trained model to quickly identify a new sound event category only containing a small number of samples. According to the method, through effective combination of multi-scale feature extraction and a meta-learning strategy, the problems of incomplete feature extraction and poor model generalization ability of a traditional method in a data scarcity scene are solved.

The application discloses a bird sound event detection method and system for monitoring state-protected birds, and comprises the following steps: collecting bird sound data and environmental noise data of state-protected birds, and constructing a bird sound spectrogram data set and a noisy bird sound data set; constructing a bird sound noise reduction model and a target detection model; training and jointly optimizing the bird sound noise reduction model and the target detection model according to the bird sound spectrogram data set and the noisy bird sound data set; constructing an image-audio mapping module for converting the image detection result of the target detection model into an audio sound event detection result; acquiring bird sound data to be detected, searching and sound source positioning the target bird sound according to the bird sound noise reduction model, the target detection model and the image-audio mapping module, and determining the species, time-frequency and direction information of the target bird sound. The application can improve the detection effect of field bird sound, and further improve the classification accuracy, and can be widely applied to the technical field of acoustic information processing.

The invention discloses a sound signal periodic feature extraction method, a network model training method, a storage medium and equipment, and belongs to the technical field of sound event detection. The objective of the invention is to solve the problems of high sensing difficulty and poor decoupling effect of overlapped acoustic events in the current acoustic detection process. The method comprises the following steps: for a sound signal i, mapping the sound signal i to a low-dimensional space through two different linear layers to obtain p and g, and respectively carrying out expansion convolution operation on p and g to obtain pconv and gconv; for p and g, feature coding is carried out based on a Fourier basis function and a gating mechanism to obtain Fourier features, for pconv and gconv, Fourier features are obtained in the same mode, and Hadamard product is carried out on the pconv and the gconv to obtain representation of periodic features. And in the training process of the corresponding model, performing reconstruction error on the sum and the original signal i, respectively calculating two norms of the sum, and adding the two obtained two norms to obtain a Fourier series regular term for training the model.

The invention relates to a sound event detection model training method and device, equipment and a storage medium. The method comprises the following steps: inputting a sample audio into a pre-trained sound event positioning model to obtain a prediction label of the sample audio; the sample audio is labeled with an initial label, and the initial label is used for representing whether the sample audio contains a sound event; the prediction label comprises probability distribution information that each audio frame in the sample audio comprises a sound event; cutting out an audio clip conforming to the target audio length from the sample audio, and marking a clip tag corresponding to the audio clip according to the initial tag and the prediction tag; the fragment label is used for representing whether the audio fragment contains a sound event; and according to the audio clip and the clip tag corresponding to the audio clip, training a to-be-trained sound event detection model to obtain a pre-trained sound event detection model. By adopting the method, the training efficiency of the sound event detection model can be improved.

The present invention discloses a method for data augmentation of audio and a real-time sound event detection system and method, including: establishing two deep learning models, where the deep learning models include a pre-awakening model and a detection model; deforming audio data through a series of audio transformations to obtain deformed audio; extracting spectral features from the deformed audio; randomly masking the spectral features to obtain data after data augmentation; using the data after data augmentation to train the deep learning models in the sound event detection method and saving the trained models; using a microphone to record an audio stream in real time, slicing the audio stream and sending it into the trained models for detection to obtain detection results. According to the present invention, the sound events to be detected can be quickly and accurately fed back, and the number of model parameters and the memory occupancy in this method are small, meeting the conditions for mobile use.

The application relates to the technical field of sound recognition, and provides a low-resource sound event detection method, device and equipment and a storage medium, wherein the method comprises the following steps: acquiring sound to be detected; inputting the sound to be detected into a trained sound group category neural network model to obtain sound group category information; inputting the sound to be detected into an encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and splicing and fusing the group category information and the fine-grained feature information to obtain fusion feature information; inputting the fusion feature information into a decoder of the sound event category judgment model, and decoding the fusion feature information based on an attention mechanism and in combination with a pre-obtained group category representation matrix to obtain a sound event judgment result. The application forms large-class group sound group category information containing rich information by using the sound to be detected, and realizes auxiliary discrimination based on large-class sound event results through a self-attention mechanism.

This disclosure relates to a model generation method, a sound event detection method, an apparatus, a medium, and a device. The method includes: extracting audio features from a first sample audio file and inputting the audio features into a sound event detection model to obtain a first detection result; determining the target loss of the model based on the first annotation result of the first sample audio file and the first detection result, the target loss including cross-entropy loss and continuity error penalty loss; and updating the model parameters based on the target loss. Since continuity detection errors are the main cause of misidentification, the continuity error penalty term is included as part of the loss function during the sound event detection model training process. The model trained using this method can maintain high accuracy while keeping the sound event recall rate unchanged. That is, it can significantly reduce misidentifications that affect user experience without introducing more missed detections, thereby improving the accuracy of sound event detection and user experience.

This invention discloses a sound event detection method based on the fusion of multi-scale spatial features and coordinate attention, belonging to the field of sound event detection technology. It aims to address the problems in existing sound event detection methods, such as the limited receptive field of the network, which makes it difficult to capture multi-scale temporal span features, and the inability to accurately focus on target sounds and suppress noise in complex time-frequency spaces. This method innovatively introduces a multi-scale coordinate attention module into the sound event feature extraction network. This module first uses parallel dilated convolutions with an increasing time dilation rate to obtain multi-scale temporal features and performs fusion and dimensionality reduction. Then, it employs a dual-axis time-frequency decoupling mechanism, performing adaptive pooling aggregation along both the time and frequency dimensions to generate directional time attention weights and frequency attention weights. Finally, the dual-axis weights are used to recalibrate the multi-scale features. This method significantly improves the accuracy and robustness of sound event detection.

The present invention provides a method, device, equipment and storage medium for locating the water level of a train water tank, comprising: obtaining an audio segment to be tested; performing feature extraction on the audio segment to be tested to obtain multiple frames of feature vectors to be tested corresponding to the audio segment to be tested; and substituting the multiple frames of feature vectors to be tested corresponding to the audio segment to be tested into a pre-established sound event detection model to obtain the water level height magnitude corresponding to the audio segment to be tested. By establishing a sound event detection model, a correspondence between the feature vector corresponding to the audio segment and the water level height magnitude is constructed. After obtaining the audio segment to be tested, its features are first extracted to obtain the feature vector to be tested, and then the feature vector to be tested is substituted into the model. This allows for rapid and accurate determination of the water level magnitude corresponding to the feature vector to be tested, i.e., the water level magnitude corresponding to the audio segment to be tested. This facilitates timely judgment of the water level condition of the train water tank during the water filling process, improves water filling efficiency and avoids waste of water resources caused by water overflow.

A method for real-time sound event detection on an embedded device includes pre-training a contrast language-audio pre-training model as an audio base model and preparing an offline multi-modal query prototype for a sound event of interest. The pre-trained model and the query prototype are deployed on an embedded device. The device receives an input audio stream and extracts an audio embedding using a pre-trained model. A similarity score between the extracted audio embedding and the prepared query prototype is calculated. The presence of the sound event is determined based on the calculated similarity score, and the real-time sound event detection result is output. The system includes a memory storing a pre-trained model and a query prototype, an audio input interface, and a processor configured to perform extraction, calculation, determination, and output operations. A non-transitory computer readable medium stores instructions that, when executed, cause a processor to implement the method.

The present application provides a method, device, equipment and readable storage medium for detecting and locating sound events, the method comprising: obtaining multiple audio signals collected by a microphone array; determining a cross-correlation function matrix between every two audio signals in the multiple audio signals; inputting each cross-correlation function matrix into a preset sound event classification model to obtain a category matrix of the sound event, the preset sound event classification model is a pre-trained neural network model; performing Fourier transform on each audio signal to obtain a spectrogram of each audio signal, and determining a sound source orientation matrix of the sound event based on the spectrum data of each audio signal; inputting the category matrix and the sound source orientation matrix into a preset matching network for matching, and obtaining the category of each sound event in the audio signal and the sound source orientation matching the category. The present application accurately and quickly realizes the detection and positioning of sound events.

The present invention discloses an audio data enhancement method, device, and medium based on the collection of sound pickup environmental factors. The method comprises obtaining a sample training set of the original audio data to be enhanced; sequentially batching, verifying, labeling, and merging the sample training set; scheduling a microphone and a speaker to mix the entire audio data of each sample batch in a real environment with the sound pickup environmental factors; sequentially segmenting and labeling the entire recording data based on the batching and labeling to obtain an enhanced sample training set for the current sample batch; obtaining the enhanced sample training set for each sample batch and splicing them together to obtain the final enhanced sample training set for the original audio data. Advantages: The present invention simultaneously takes into account environmental factors such as environmental background noise, the distance between the microphone and the sound source, and interference generated within the microphone, thereby more effectively introducing environmental information, thereby improving the accuracy of the sound event detection model in a real environment and reducing performance degradation.