Method for detecting abnormal audio, and computer device

Abstract

A method for detecting abnormal audio, and a computer device, which can be applied to the field of device detection. The method comprises: acquiring first audio data (comprising running audio and ambient audio of a device to be detected), and separately performing time domain representation and frequency domain representation of the first audio data; then calculating a correlation on the basis of obtained first time domain data and first frequency domain data, and obtaining second audio data on the basis of the correlation; extracting a target feature of the second audio data by means of a trained first model; obtaining a sparse coefficient on the basis of the target feature, and on the basis of the sparse coefficient, determining whether the second audio data is abnormal audio. Audio changes are depicted from two dimensions: the time domain and the frequency domain. In addition, a sound scene is smoothed by means of comparing the correlation between the two dimensions, to remove interference, so as to correct a start point and an end point of current audio data. In addition, key features in the second audio data are recognized using the trained first model, so that various abnormal types of sound can be recognized.

Description

A method for detecting abnormal audio and computer equipment

[0001] This application claims priority to Chinese Patent Application No. 202411764197.6, filed on November 29, 2024, entitled "A Method for Detecting Abnormal Audio and a Computer Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of equipment testing, and more particularly to a method for detecting abnormal audio and a computer device. Background Technology

[0003] With the national demand for high-quality development of new productive forces, predictive maintenance technology has become a key strategy for improving the digitalization and intelligence of industrial equipment. Sound anomaly detection, as a major technology in predictive maintenance, is increasingly widely used in actual industrial settings due to its non-invasiveness and high interpretability.

[0004] Equipment emits various sounds during operation, and these sounds contain a wealth of crucial information about the equipment's operational status. During normal operation, equipment emits normal sounds that align with human perception; however, when equipment malfunctions, it often emits sharp and piercing sounds. Against this backdrop, predictive maintenance technology based on sound anomaly detection has rapidly developed. Using sound detection methods to monitor equipment status in real time and provide early warnings of potential faults can effectively prevent unplanned downtime and avoid personnel injuries and economic losses caused by equipment failures. Due to its non-invasive nature, sound anomaly detection technology can continuously acquire the equipment's operating status in real time, enabling continuous monitoring without interfering with equipment operation. This allows for early detection of potential faults and improves equipment reliability. It's worth noting that maintenance workers in industrial settings also determine abnormal operating conditions based on the abnormal sounds emitted by the equipment. Therefore, predictive maintenance using sound anomaly detection technology is highly interpretable, and the detection results can be correlated with maintenance work orders from on-site workers.

[0005] A common sound anomaly detection scheme currently uses the calculation of the energy or root mean square (RMS) value of a specific frequency band to determine whether the sound emitted by the device under test is abnormal. If the energy (or RMS) is greater than a reference threshold, it is considered an abnormal sound; otherwise, it is considered normal. However, this method has several drawbacks: 1. Due to the high noise levels and ambient sounds in industrial environments where the device under test is located, the sound source recorded by the artificial ear may include ambient noise from other workstations, easily leading to system misjudgments. 2. Anomalies are diverse and inexhaustible; a single characteristic indicator cannot cover all anomalies (i.e., relying solely on calculating the energy or RMS of a specific frequency band to determine if a sound is abnormal) can easily result in missed anomalies and weak noise immunity. Summary of the Invention

[0006] This application provides a method and computer device for detecting abnormal audio, which is used to characterize audio changes from two dimensions: time domain and frequency domain. At the same time, it smooths the sound scene and removes interference by comparing the correlation between the two dimensions, so as to correct the start and end points of the current audio data. In addition, it uses a trained first model to identify key features in the second audio data, so that various types of abnormal sounds can be identified, thus improving efficiency.

[0007] Based on this, the embodiments of this application provide the following technical solutions:

[0008] Firstly, this application provides a method for detecting abnormal audio. This method specifically includes: First, collecting raw audio data of the device to be detected, referred to as first audio data. This first audio data includes the operating audio emitted by the device during operation and ambient audio around the device (e.g., sounds from other workstations, worker voices, object friction sounds, etc.). After acquiring the first audio data, performing time-domain representation on the first audio data to obtain corresponding time-domain data, referred to as first time-domain data; furthermore, performing frequency-domain representation on the first audio data to obtain corresponding frequency-domain data, referred to as first frequency-domain data. Then, calculating the correlation between the first time-domain data and the first frequency-domain data, and obtaining second audio data based on this correlation. This second audio data includes the operating audio emitted by the device during operation (which may be part or all, without limitation). Then, using a trained first model, extracting features from the second audio data to obtain key features, referred to as target features, which may be one or more, without limitation in this application. Finally, sparse coefficients are obtained based on the target features, and the sparse coefficients are used to determine whether the second audio data is abnormal audio.

[0009] In the above embodiments of this application, audio changes are characterized from both the time and frequency domains. Simultaneously, the correlation between these two dimensions is compared to smooth the sound scene and remove interference, thereby correcting the start and end points of the current audio data. This addresses the shortcomings of existing solutions, such as single-feature (traditional methods identify the start and end points of sound based on a single feature), weak noise resistance (traditional methods, originally applied to speech recognition, exhibit weak noise resistance when applied to noisy industrial environments, and for low signal-to-noise ratio audio streams, the time-domain feature changes are weak, making it difficult to distinguish start and end points), and overly fragmented segmentation (traditional methods identify many endpoints, dividing a complete audio stream into multiple segments, resulting in excessive dispersion and interference with business operations). Furthermore, it demonstrates strong robustness. In addition, the trained first model is used to identify key features in the second audio data, enabling the identification of various types of abnormal sounds (traditional methods lack completeness in their empirical feature indicators, resulting in a limited number and incomplete coverage, making them unsuitable for certain complex noise issues), thus improving efficiency and solving the problem of insufficient abnormal samples in industrial scenarios.

[0010] In one possible implementation of the first aspect, the trained first model is obtained by training the first model using a first training set based on a first loss function. Specifically, the first training set includes multiple first training data, which are filtered audio data. The first label is used to characterize whether the first training data is abnormal audio. For example, if the first label is label b1, it means that the corresponding first training data is abnormal audio; if the first label is label b2, it means that the corresponding first training data is normal audio.

[0011] In the above embodiments of this application, the training process of the first model is specifically described. The trained first model is used to extract key features from the audio data, avoiding the manual extraction of a large number of features, and has strong generalization and high accuracy.

[0012] In one possible implementation of the first aspect, the first model could be: multi-wavelet kernel fusion convolution.

[0013] In the above embodiments of this application, a typical network structure of the first model is provided, which is feasible.

[0014] In one possible implementation of the first aspect, after determining whether the second audio data is abnormal audio based on the sparse coefficients, the method may further include: if the second audio data is determined to be abnormal audio based on the sparse coefficients, inputting the second audio data into a trained second model to obtain a classification category for the second audio data, wherein the classification category is used to characterize the type of abnormality of the second audio data. For example, a "buzzing" sound corresponds to category 1, and category 1 represents type 1 abnormality; a "clanging" sound corresponds to category 2, and category 2 represents type 2 abnormality. This method can effectively distinguish various abnormal situations.

[0015] In the above embodiments of this application, the process of characterizing abnormal voiceprints is specifically described, that is, the abnormal audio is classified in a sound-like manner, thereby realizing the mapping of abnormal sounds to semantics. In this way, various abnormal situations can be effectively distinguished (the traditional method is that the system only outputs that there is an abnormal sound, but does not indicate what kind of abnormality caused the sound, and workers still need to review and classify it).

[0016] In one possible implementation of the first aspect, the trained second model includes a trained first sub-model and a trained second sub-model. The trained first sub-model is trained on the first sub-model using a second training set based on a second loss function. The second training set includes multiple second training data of different anomaly types, and the second training data is represented by a time-frequency graph with time on the horizontal axis and frequency on the vertical axis. The trained second sub-model is trained on the second sub-model using a third training set based on a third loss function. The third training set includes multiple third training data. In this embodiment, the third training data is the output data of the trained first sub-model, and the third training data has a second label used to represent the classification category of the third training data.

[0017] In the above embodiments of this application, it is specifically described that the second model consists of a first sub-model and a second sub-model, with the input of the second sub-model being the output of the first sub-model. Using only a small number of training samples, and based on the interpretable first and second sub-models, various abnormal situations can be effectively distinguished, demonstrating strong interpretability.

[0018] In one possible implementation of the first aspect, the first input data and the second input data are any two different training data in the second training set. The iteration termination condition for training the first sub-model includes: the difference between the first input data and the first output data corresponding to the first sub-model is lower than a first preset threshold; and / or, the difference between the first output data and the second output data of the first sub-model is higher than a second preset threshold, and the second output data corresponds to the second input data. In other words, the training objective of the first sub-model in this application is to maximize the difference between the output time-frequency graph x' (the corresponding input time-frequency graph is assumed to be x) and the time-frequency graph y' (the corresponding input time-frequency graph is assumed to be y), and to minimize the difference between the input time-frequency graph x and the output time-frequency graph x'. The function of this target model is to map out different types of abnormal components, maximizing the difference between samples of different classes.

[0019] In the above embodiments of this application, the conditions for terminating the training of the first sub-model are specifically described. These conditions enable the trained first sub-model to quickly and automatically locate abnormal components in the time-frequency graph, thereby improving the localization efficiency.

[0020] In one possible implementation of the first aspect, the first sub-model can be: a weight matrix.

[0021] In the above embodiments of this application, a typical type of the first sub-model is specifically described, which uses only a small number of samples to construct an interpretable time-frequency matrix, effectively distinguishes various abnormal situations, and completes the modeling and mapping of parameters to sound semantics.

[0022] In one possible implementation of the first aspect, the first audio data is represented in both the time domain and the frequency domain to obtain the first time domain data and the first frequency domain data, respectively. One possible implementation is as follows: For the time domain representation, the first audio data can be represented in the time domain based on time domain characteristic parameters (e.g., short-time energy, short-time zero-crossing rate, etc.) to obtain the first time domain data; for the frequency domain representation, the first audio data can be represented in the frequency domain based on frequency domain characteristic parameters (e.g., spectral centroid distance) to obtain the first frequency domain data.

[0023] In the above embodiments of this application, the changes in the first audio data can be characterized at multiple scales based on time-domain and frequency-domain features, thereby improving positioning accuracy.

[0024] In one possible implementation of the first aspect, one approach based on the correlation between the first time-domain data and the first frequency-domain data is as follows: First, calculate the correlation between the target frame signal and the preceding p frames of the target frame signal, where the target frame signal is any one frame of m segments of sub-time-domain data or n segments of sub-frequency-domain data (i.e., any one frame of the above 5 sub-data segments a-t1, a-t2, a-t3, a-p1, a-p2). Based on this, the correlation between each frame of the m segments of sub-time-domain data + n segments of sub-frequency-domain data and the preceding frames can be calculated. Then, based on the correlation between all the obtained frame signals, remove the ambient audio from the first audio data to obtain the second audio data.

[0025] In the above embodiments of this application, by introducing event correlation to calculate the correlation between each frame of signal and the previous few frames of signal, noise interference is removed to correct the start and end points of the target audio. This can effectively segment the sound signal and eliminate environmental noise, thereby solving the shortcomings of existing solutions, such as single feature (traditional methods identify the start and end points of sound based on a single feature), weak noise resistance (traditional methods were originally applied to speech recognition, but when transferred to noisy industrial sites, their noise resistance is weak, and for audio streams with low signal-to-noise ratio, the temporal feature changes are weak, making it difficult to distinguish the start and end points), and overly fragmented segmentation (traditional methods identify many endpoints, dividing a complete audio stream into multiple segments, which is too scattered and causes business interference). This method is robust.

[0026] In one possible implementation of the first aspect, one way to extract features from the second audio data using a trained first model is as follows: First, the second audio data is filtered by at least one filter to filter out one or more specific bands (e.g., high-frequency bands) to obtain filtered second audio data. Then, features are extracted from the filtered second audio data using the trained first model.

[0027] In the above embodiments of this application, filtering is used to extract features more efficiently and improve extraction efficiency.

[0028] In one possible implementation of the first aspect, the filter may specifically be a Mel filter.

[0029] The above embodiments of this application specifically describe a typical type of filter, which has wide applicability.

[0030] In one possible implementation of the first aspect, obtaining sparse coefficients based on the target features can be achieved by: first, updating a composite dictionary based on the target features. This composite dictionary is derived from a harmonic dictionary and an impulse dictionary. Specifically, the harmonic dictionary and the impulse dictionary can be fused to form the composite dictionary. For example, the matrix representing the harmonic dictionary can be concatenated with the matrix representing the impulse dictionary to obtain a new matrix, which is the composite dictionary. Then, sparse encoding is performed on the composite dictionary to obtain sparse coefficients. Finally, a decision can be made based on Support Vector Data Description (SVDD) to determine whether the second audio data is abnormal audio.

[0031] In the above embodiments of this application, the process of unsupervised abnormal sound detection using a trained first model is specifically described. Anomalies are diverse and inexhaustible; a limited number of feature indicators designed for certain anomalies are insufficient to comprehensively characterize them, easily leading to missed detections (traditional methods' empirical feature indicators lack completeness, are limited in number, and have incomplete coverage, making them unsuitable for certain complex noise types). Therefore, it is necessary to characterize key information in the time-frequency graph based on the first model, using only normal samples and existing empirical features to construct a composite dictionary for incremental learning, thereby achieving unsupervised abnormal sound detection and solving the problem of limited abnormal samples in industrial scenarios.

[0032] A second aspect of this application provides a computer device having the function of implementing the method of the first aspect or any possible implementation thereof. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0033] A third aspect of this application provides a computer device that may include a memory, a processor, and a bus system. The memory is used to store a computer program (also referred to as a program or computer-readable instructions), and the processor is used to invoke the program stored in the memory to execute the method of the first aspect of the embodiments of this application or any possible implementation of the first aspect.

[0034] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0035] The fifth aspect of this application provides a computer program or a computer program product containing instructions that, when the computer program or computer program product is run on a computer, causes the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0036] A sixth aspect of this application provides a chip including at least one processor and at least one interface circuit coupled to the processor. The interface circuit performs transceiver functions and sends instructions to the at least one processor. The at least one processor runs a computer program or instructions, having the functionality to implement the methods described in the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware, software, or a combination of hardware and software, including one or more modules corresponding to the described functions. Furthermore, the interface circuit is used to communicate with other modules outside the chip.

[0037] In some implementations of this application, some of the one or more processors may implement some steps of the above method through dedicated hardware. For example, the processing involving neural network models may be implemented by a dedicated neural network processor or graphics processor.

[0038] The method provided in this application embodiment can be implemented by a single chip or by multiple chips working together. Attached Figure Description

[0039] Figure 1 is a schematic diagram of a structural framework for an artificial intelligence main body provided in an embodiment of this application;

[0040] Figure 2 is a system architecture diagram of a task processing system provided in an embodiment of this application;

[0041] Figure 3 is a computational architecture diagram of the abnormal audio detection method provided in the embodiments of this application;

[0042] Figure 4 is a flowchart illustrating an abnormal audio detection method provided in an embodiment of this application;

[0043] Figure 5 is a schematic diagram of audio characterization based on time-domain features provided in an embodiment of this application;

[0044] Figure 6 is a schematic diagram of audio characterization based on frequency domain features provided in an embodiment of this application;

[0045] Figure 7 is a schematic diagram of the training and inference phases of the first model provided in an embodiment of this application;

[0046] Figure 8 is a schematic diagram of an abnormal audio detection method provided in an embodiment of this application;

[0047] Figure 9 is another schematic diagram of the abnormal audio detection method provided in the embodiments of this application;

[0048] Figure 10 is a test schematic diagram of a loudspeaker noise detection scenario provided in an embodiment of this application;

[0049] Figure 11 is a schematic diagram of a computer device provided in an embodiment of this application;

[0050] Figure 12 is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] This application provides a method and computer device for detecting abnormal audio, used to characterize audio changes from two dimensions: time domain and frequency domain. Simultaneously, it smooths the sound scene and removes interference by comparing the correlation between the two dimensions, thereby correcting the start and end points of the current audio data. Furthermore, it utilizes a trained first model to identify key features in the second audio data, enabling the identification of various types of abnormal sounds, demonstrating completeness and comprehensiveness. The model's feature extraction method also avoids the need for manual extraction of a large number of features, improving efficiency.

[0052] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0053] (1) Neural Network

[0054] A neural network can be composed of neural units, specifically understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Neural networks with many hidden layers are called deep neural networks (DNNs). The function of each layer in a neural network can be expressed mathematically. To describe it physically, each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations are: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... Operation 4 is completed using "+b", and operation 5 is implemented using "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight matrix of each layer of the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, as described above; that is, the W of each layer of the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0055] It should be noted that in the embodiments of this application, the models used for machine learning tasks are essentially neural networks, such as the first model, the second model, etc., as described later.

[0056] (2) Loss Function

[0057] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight matrix of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer). For example, if the network's prediction is too high, the weight matrix is ​​adjusted to predict a lower value, and this adjustment continues until the neural network can predict the target value accurately. Therefore, it's necessary to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (Loss) indicates a greater difference, so training the neural network becomes a process of minimizing this loss.

[0058] During the training of a neural network, the back propagation (BP) algorithm can be used to correct the parameters in the initial neural network model, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then propagated back to update the parameters in the initial neural network model, leading to convergence of the error loss. The back propagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0059] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0060] First, the overall workflow of an artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The framework is then elaborated on from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0061] (1) Infrastructure

[0062] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0063] (2) Data

[0064] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0065] (3) Data processing

[0066] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0067] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0068] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0069] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0070] (4) General ability

[0071] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0072] (5) Smart Products and Industry Applications

[0073] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, and smart cities.

[0074] This application describes the application of pre-trained models (such as a first model and a second model), which can be specifically applied to the field of audio data processing. For example, it can be applied to the detection of abnormal audio data and the identification of abnormal types of abnormal audio. Specifically, it can be applied to key rotating equipment in industries such as coal mining and manufacturing, as well as in industrial quality inspection scenarios. As an example, a specific application scenario could be the quality inspection of mobile phone speakers, where the device to be inspected is a mobile phone speaker, and the sensor uses the mobile phone's built-in microphone to collect raw audio data.

[0075] The model frameworks of the first and second models described in this application embodiment can be deployed on a training device (i.e., a computer device used to train the model). Please refer to Figure 2, which is a system architecture diagram of a task processing system provided in this application embodiment. In Figure 2, the task processing system 200 includes an execution device 210, a training device 220, a database 230, a client device 240, a data storage system 250, and a data acquisition device 260. The execution device 210 includes a computing module 211. The data acquisition device 260 is used to acquire a large-scale dataset (collectively referred to as a training dataset or training set) required by the user and store the training dataset in the database 230. The training device 220 trains the models (i.e., the first model 201 (e.g., multi-wavelet kernel fusion convolution) and the second model 202 described in Figure 2) based on the training dataset maintained in the database 230. It should be noted that since this application requires training more than one model, there is also more than one corresponding training dataset. Generally, one model corresponds to one training dataset. For example, in the embodiments of this application, the training dataset used to train the first model 201 can be called the first training set. However, it should be noted that in some embodiments of this application, the second model 202 may also include a first sub-model 2021 and a second sub-model 2022. Therefore, the second model 202 corresponds to multiple training datasets. The training dataset used to train the first sub-model 2021 can be called the second training set, and the training dataset used to train the second sub-model 2022 can be called the third training set. The construction of the third training dataset depends on the output data of the second sub-model 2022.

[0076] The trained model obtained from the training can be applied on the execution device 210. The execution device 210 can access data, code, etc., in the data storage system 250, and can also store data, instructions, etc., in the data storage system 250. The data storage system 250 can be located within the execution device 210, or it can be an external memory relative to the execution device 210.

[0077] The first model 201 and the second model 202 trained by the training device 220 can be used as the target model (i.e., the trained first model 201 and the trained second model 202) based on the validation dataset, with the optimal performance evaluation index selected. This target model can then be applied to different systems or devices (i.e., the execution device 210), specifically edge devices or end-side devices, such as mobile phones, tablets, laptops, monitoring systems (e.g., cameras), security systems, etc. In Figure 2, the execution device 210 is equipped with an I / O interface 212 for data interaction with external devices. The "user" can input data into the I / O interface 212 through the client device 240. For example, the client device 240 can be related devices for data acquisition, such as recording devices, monitoring modules, mobile phones, tablets, and other terminal devices deployed around the device under test. These devices can collect data to be processed in real time. After preprocessing this data, it can be used as input data to the computing module 211 of the execution device 210, where the computing module 211 performs relevant processing on the input data. Furthermore, in some embodiments of this application, the client device 240 can also be integrated into the execution device 210. For example, when the execution device 210 is a mobile phone, the data to be processed can be obtained directly through the mobile phone or the data to be processed can be received from other devices (e.g., another mobile phone), and then the computing module 211 in the mobile phone can process the data to be processed. The product form of the execution device 210 and the client device 240 is not limited here.

[0078] It is worth noting that Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of this application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210. In Figure 2, the client device 240 is an external device relative to the execution device 210. In other cases, the client device 240 can also be integrated into the execution device 210.

[0079] It should also be noted that the training of the models described in the above embodiments can all be implemented on the cloud side. For example, the training dataset can be obtained by the cloud-side training device 220 (which can be set on one or more servers or virtual machines), and the relevant models can be trained based on the training dataset. For example, the first model 201 can be trained based on the first training set, and the second model 202 can be trained based on the second and third training sets. The trained models are then sent to the execution device 210 for application. For example, the system architecture corresponding to Figure 2 is described in which the training device 220 trains the first model 201 and the second model 202. Model 202 is trained as a whole, and the trained first model 201 and second model 202 are then sent to execution device 210 for use. The training of each model described in the above embodiments can also be implemented on the terminal side, that is, the training device 220 can be located on the terminal side. For example, the training dataset (i.e., the first training set, the second training set, and the third training set) can be obtained by a terminal device (e.g., a mobile phone, tablet, etc.), a monitoring system device, etc., and each model can be trained based on the training dataset to obtain the trained model. The trained model can be used directly on the terminal device, or it can be sent by the terminal device to other devices for use. Specifically, this application embodiment does not limit on which device (cloud side or terminal side) each model is trained or applied on.

[0080] Based on the above system architecture, the core process of the abnormal audio detection method provided in this application embodiment will be described below. For details, please refer to Figure 3. Figure 3 is a computational architecture diagram of the abnormal audio detection method provided in this application embodiment. First, in the initial stage, on the central cloud side, a preset model of a specified device (i.e., the first model and the second model mentioned above) is generated based on historical audio data, and the model is sent to the edge side (here, the training process and the inference process are carried out on different sides, which is only for illustration and is not limited in practice). The edge side uses the preset model to first perform endpoint recognition to extract effective information from the sound data collected by a specific device (i.e., the device to be detected) and then performs inference to generate abnormal audio detection results and voiceprint representation information. The sound representation information of the model is compared with the sound representation information recorded by the inspection personnel, thereby providing feedback to the preset model. The feedback result is transmitted to the central cloud. The central cloud retrains the model based on the feedback result and then sends the retraining result to the edge end.

[0081] Based on the above description, the specific implementation flow of the abnormal audio detection method provided in this application embodiment will be described below. Please refer to Figure 4 for details. Figure 4 is a schematic flowchart of the abnormal audio detection method provided in this application embodiment, which may specifically include the following steps:

[0082] 401. Obtain first audio data, which includes the operating audio of the device under test and the ambient audio around the device under test.

[0083] First, the computer device collects the raw audio data of the device under test. This raw audio data can be called the first audio data. The first audio data includes the operating audio emitted by the device under test during operation and the ambient audio around the device under test (such as the sounds of other workstations, workers talking, and the sound of objects rubbing).

[0084] It should be noted that in this embodiment, a segment of raw audio data corresponds to one device to be detected. For example, assuming there are 10 devices to be detected, 10 segments of raw audio data will be acquired, which are 10 segments of first audio data. Steps 401 to 405 describe how to determine whether the first audio data of the current segment (assuming it is the first segment of these 10 segments of first audio data, corresponding to the device to be detected R1) is abnormal audio data. The remaining 9 segments of first audio data are judged based on the same steps 401 to 405, which will not be elaborated here.

[0085] 402. Perform time-domain and frequency-domain representations on the first audio data to obtain the first time-domain data and the first frequency-domain data, respectively.

[0086] After obtaining the first audio data, the first audio data is represented in the time domain to obtain the corresponding time domain data, which can be called the first time domain data, as shown in Figure 5, which represents the audio based on time domain features. In addition, the first audio data also needs to be represented in the frequency domain to obtain the corresponding frequency domain data, which can be called the first frequency domain data, as shown in Figure 6, which represents the audio based on frequency domain features.

[0087] Specifically, for time-domain representation, the first audio data can be represented in the time domain based on time-domain feature parameters (such as short-time energy, short-time zero-crossing rate, etc.) to obtain the first time-domain data; for frequency-domain representation, the first audio data can be represented in the frequency domain based on frequency-domain feature parameters (such as spectral centroid distance) to obtain the first frequency-domain data. Based on time-domain and frequency-domain features, the changes in the first audio data can be characterized at multiple scales, thereby improving positioning accuracy.

[0088] 403. Based on the correlation between the first time-domain data and the first frequency-domain data, the second audio data is obtained, and the second audio data includes the running audio.

[0089] Then, the sound scene can be smoothed based on event correlation to remove noise interference, thereby accurately identifying the start and end points of the target sound and matching scenes of different durations. Specifically, the computer device can calculate the correlation between the first time-domain data and the first frequency-domain data obtained above, and obtain the second audio data based on the correlation. The second audio data includes the running audio emitted by the device under test during operation (which may be part or all, without limitation).

[0090] Specifically, in some embodiments of this application, it is assumed that the first time-domain data includes m segments of sub-time-domain data (e.g., assuming the first audio data corresponding to the current device R1 to be detected is A, and time-domain representation of A is performed to obtain the first time-domain data At, where At contains 3 segments of sub-time-domain data a-t1, a-t2, and a-t3 respectively), and the first frequency-domain data includes n segments of sub-frequency-domain data (e.g., frequency-domain representation of A is performed to obtain the first frequency-domain data Ap, where Ap contains 2 segments of frequency-domain data a-p1 and a-p2 respectively). Then, based on the correlation between the first time-domain data and the first frequency-domain data... One implementation method is as follows: First, calculate the correlation between the target frame signal and the p frames preceding it. The target frame signal is any one frame from m segments of time-domain data or n segments of frequency-domain data (i.e., any one frame from the five segments a-t1, a-t2, a-t3, a-p1, a-p2 mentioned above). Based on this, the correlation between each frame in the m segments of time-domain data + n segments of frequency-domain data and the preceding frames can be calculated. Then, based on the correlation between all the obtained frame signals, remove the environmental audio from the first audio data to obtain the second audio data. For example, cluster analysis can be performed based on the time-domain and frequency-domain characteristics of each audio segment. Same-class segments are merged, and different-class segments are distinguished, thus obtaining the merging / distinguishing results of audio segments in different time periods, such as time period A, time period B, ... , where time period A consists entirely of audio segments of the same class, time period B consists entirely of audio segments of the same class, and so on. This will not be elaborated further here. It should be noted that, in the embodiments of this application, the correlation refers to the difference between the values ​​in the time domain and the values ​​in the frequency domain of different audio segments segmented from the same audio data. When the difference between the two is relatively small (e.g., less than a certain preset threshold), the correlation is considered high, and they are considered to correspond to the same audio segment; conversely, when the difference between the two is relatively large (e.g., greater than a certain preset threshold), the correlation is considered low, and they are considered to correspond to different audio segments.

[0091] This application divides audio signals based on multiple window lengths (hereinafter referred to as window lengths), dividing the audio signals into sub-segments of different lengths, and performing clustering based on time-domain and frequency-domain features to achieve similarity judgment. By fusing the division information of multi-scale windows, it can achieve smoothing of sound scenes based on event correlation and remove noise interference, which is more accurate than single-scale truncation.

[0092] It should be noted that since the second audio data is obtained based on m segments of sub-time domain data and n segments of sub-frequency domain data, the second audio data is also obtained by splicing multiple sub-time domain data and / or sub-frequency domain data. Therefore, in some embodiments of this application, if there is sub-frequency domain data in the data to be spliced, it can be uniformly converted into time domain data, and the final second audio data is characterized by time domain features.

[0093] In summary, steps 401 to 403 are mainly preprocessing steps for the original audio data. This process can be called "multi-scale endpoint recognition," which fully characterizes the changes in the first audio data from both the time and frequency domains. Combined with multi-scale detection with adaptive window length, it introduces event correlation matching for audio signals of different lengths, which can effectively segment the sound signal and remove environmental noise (noise removal is a prerequisite for abnormal sound detection). This solves the shortcomings of existing solutions, such as single feature (traditional methods identify the start and end points of sound based on a single feature), weak noise resistance (traditional methods were originally applied to speech recognition, but when transferred to noisy industrial sites, their noise resistance is weak, and for low signal-to-noise ratio audio streams, the time domain feature changes are weak, making it difficult to distinguish the start and end points), and overly fragmented segmentation (traditional methods identify many endpoints, dividing a complete audio stream into multiple segments, which is too scattered and causes business interference). It is also highly robust.

[0094] 404. The first trained model is used to extract features from the second audio data to obtain the target features.

[0095] Subsequently, the trained first model is used to extract features from the second audio data to obtain key features, which can be called target features. There can be one or more such features, and this application does not limit the number of features. It should be noted that in the embodiments of this application, the first model can be a multi-wavelet kernel fusion convolution or other types of neural networks, and this application does not limit the network structure of the first model.

[0096] It should be noted that, in order to perform feature extraction more efficiently, in some embodiments of this application, the second audio data can be filtered first by at least one filter, such as a Mel filter or a Mel filter bank (composed of multiple Mel filters), to filter out one or more specific bands (such as high-frequency bands) to obtain filtered second audio data. Then, the filtered second audio data is used to extract features from the first trained model to obtain the target features.

[0097] It should also be noted that, in this embodiment of the application, the trained first model is obtained by training the first model using a first training set based on a first loss function. Specifically, the first training set includes multiple first training data, which are filtered audio data. The first label is used to characterize whether the first training data is abnormal audio. For example, if the first label is label b1, it means that the corresponding first training data is abnormal audio; if the first label is label b2, it means that the corresponding first training data is normal audio.

[0098] To facilitate understanding of the training and inference processes of the first model described above, Figure 7 will be used as an example to illustrate the training and inference phases (also known as the application phases) of the first model (e.g., multi-wavelet kernel fusion convolution):

[0099] A. Training phase of the first model

[0100] Step 1: Construct the first training set. Each training sample (i.e., the first training data) is constructed based on steps 401 to 403 above. The corresponding "second audio data" is now the "first training data". The only difference between it and the "second audio data" in the inference process is that it has a label (i.e., it knows whether each second audio data is abnormal data and what type of abnormality it belongs to).

[0101] Step 2: Filter each piece of first training data to obtain filtered first training data. For example, suppose there are 100 pieces of first training data. Filter each of these 100 pieces of first training data by removing specific frequency bands (e.g., removing high frequencies) to obtain 100 filtered first training data.

[0102] Step 3: Input the filtered first training data into the first model for feature extraction, obtaining the key features of each filtered first training data (100 key features are extracted from 100 filtered first training data). Adjust the weights of the first model based on the first loss function, and iteratively train the first model until the iteration termination condition is met. As an example, the first loss function can be mean squared error. A custom surface generator is used to extract the target features, and the training objective is to minimize the mean squared error between the reconstructed training samples and the training samples input to the model.

[0103] B. Reasoning Stage of the First Model

[0104] The inference phase of the first model is actually steps 401 to 403 above. The difference from the training phase is that the second audio data is unlabeled and needs to be predicted using the trained first model.

[0105] 405. Obtain the sparse coefficients based on the target features, and determine whether the second audio data is abnormal audio based on the sparse coefficients. If yes, proceed to step 406; otherwise, proceed to step 407.

[0106] Subsequently, the computer device further obtains sparse coefficients based on the extracted target features (i.e., extracts key sparse information related to the fault from the target features), and determines whether the second audio data is abnormal audio based on the obtained sparse coefficients. If yes, step 406 is executed; otherwise, step 407 is executed.

[0107] It should be noted that in some embodiments of this application, one way to obtain sparse coefficients based on target features is as follows: First, update the composite dictionary based on the target features. This composite dictionary is obtained based on a harmonic dictionary and an impulse dictionary. Specifically, the harmonic dictionary and the impulse dictionary can be fused to form the composite dictionary. For example, the matrix representing the harmonic dictionary can be concatenated with the matrix representing the impulse dictionary to obtain a new matrix, which is the composite dictionary. Then, sparsely encode the composite dictionary to obtain sparse coefficients, such as by solving the matrix Y = DX, where Y is the input training data, D is the trained dictionary, and X is the sparse coefficient to be obtained. Finally, a decision on whether the second audio data is abnormal audio can be made based on SVDD. For example, an unsupervised anomaly detection model autoencoder can be used to determine whether the second audio data belongs to abnormal audio, as shown in Figure 8. As an example, dictionary learning can be updated by alternately optimizing the dictionary D and the sparse coefficients X. In practice, the K-SVD algorithm is usually used, where the sparse coefficients are calculated by algorithms such as matching pursuit or orthogonal matching pursuit, and the dictionary is updated by Singular Value Decomposition (SVD).

[0108] In summary, steps 404 to 405 primarily utilize the trained first model to achieve unsupervised abnormal sound detection. Anomalies are diverse and inexhaustible; a limited number of feature indicators designed for certain anomalies are insufficient to comprehensively characterize them, easily leading to missed detections (traditional methods' empirical feature indicators lack completeness, are limited in number, and have incomplete coverage, making them unsuitable for certain complex noise types). Therefore, it is necessary to characterize key information in the time-frequency map based on the first model (e.g., multi-wavelet kernel fusion convolution), using only normal samples and existing empirical features to construct a composite dictionary for incremental learning, thereby achieving unsupervised abnormal sound detection and solving the problem of limited abnormal samples in industrial scenarios.

[0109] 406. Input the second audio data into the trained second model to obtain the classification category of the second audio data. This classification category is used to characterize the anomaly type of the second audio data.

[0110] The computer device inputs the second audio data into the trained second model to obtain the classification category of the second audio data, wherein the classification category is used to characterize the anomaly type of the second audio data.

[0111] Step 406 is the process of characterizing abnormal voiceprints, that is, classifying abnormal audio in an onomatopoeic way to achieve the mapping of abnormal sounds to semantics. For example, a "buzzing" sound corresponds to category 1, and category 1 represents abnormality type 1; a "banging" sound corresponds to category 2, and category 2 represents abnormality type 2. In this way, various abnormal situations can be effectively distinguished (the traditional method is that the system only outputs that there is an abnormal sound, but does not indicate what kind of abnormality caused the sound, and workers still need to review and classify it).

[0112] Specifically, in some embodiments of this application, the trained second model includes a trained first sub-model and a trained second sub-model. The trained first sub-model is trained on the first sub-model using a second training set based on a second loss function. The second training set includes multiple second training data of different anomaly types, and the second training data is represented by a time-frequency graph with time on the horizontal axis and frequency on the vertical axis. The trained second sub-model is trained on the second sub-model using a third training set based on a third loss function. The third training set includes multiple third training data. In the embodiments of this application, the third training data is the output data of the trained first sub-model, and the third training data has a second label used to represent the classification category of the third training data.

[0113] It is important to note that the first sub-model can be any neural network, algorithm, or function with any network structure, such as a perceptron neural network, a recurrent neural network, or a modular neural network; this application does not impose any specific limitations on this. Because the network structure of the first sub-model differs, the type of the second loss function used to train it can also differ; for example, it can be mean squared error, mean absolute error, or a crossover loss function; this application does not impose any specific limitations on this. Similarly, the second sub-model can be any neural network, algorithm, or function with any network structure, such as a recurrent neural network (RNN) or a generative adversarial network (GAN); this application does not impose any specific limitations on this. Because the network structure of the second sub-model differs, the type of the third loss function used to train it can also differ; for example, it can be a multi-label classification loss or a multi-class classification hinge loss; this application does not impose any specific limitations on this.

[0114] Similarly, the training phase of this second model may include the following steps:

[0115] Step 1: Construct the second training set.

[0116] Each second training data point consists of different types of anomalous audio, and a time-frequency plot (horizontal axis for time, vertical axis for frequency) is obtained for each second training data point. For example, suppose there are 100 audio samples of different types of anomalous sounds, resulting in 100 time-frequency plots, which are 100 second training data points.

[0117] Step 2: Iterative training.

[0118] Based on the second training set constructed using the above steps, the first sub-model in the second model is iteratively trained using the second loss function. The output data of the first sub-model serves as the input data (i.e., the third training data) of the second sub-model. The multiple output data of the first sub-model constitute the third training set. Subsequently, the second sub-model in the second model is iteratively trained based on the third training set and the third loss function.

[0119] It should be noted that in some embodiments of this application, assuming that the first input data and the second input data are two arbitrarily different training data in the second training set, an iteration termination condition for training the first sub-model can be: the difference between the first input data and the first output data corresponding to the first sub-model is lower than a first preset threshold; and / or, the difference between the first output data and the second output data of the first sub-model is higher than a second preset threshold, and the second output data corresponds to the second input data. In other words, the training objective of the first sub-model in this application is to maximize the difference between the output time-frequency graph x' (the corresponding input time-frequency graph is assumed to be x) and the time-frequency graph y' (the corresponding input time-frequency graph is assumed to be y), and to minimize the difference between the input time-frequency graph x and the output time-frequency graph x'. The function of this target model is to map out different types of abnormal components, maximizing the difference between samples of different classes.

[0120] It should also be noted that, in some embodiments of this application, the first sub-model can specifically be a weight matrix (which can be represented by W). By comparing the differences between the abnormal spectra of the learning samples, the abnormal components in the time-frequency graph can be automatically located. Specifically, the SVDD method can be used to find a suitable hypersphere that maximizes the interval between normal and abnormal samples, thereby obtaining a weight matrix W with obvious periodicity and low rank.

[0121] It should also be noted that in some other embodiments of this application, the second sub-model can specifically be a single-hidden-layer neural network based on the weight matrix W. The weight coefficients in the neural network are learned through a parameter identification model, thereby achieving voiceprint representation of various abnormal sounds under limited samples. Specifically, the weight matrix is ​​used as the weight parameter w of the single-hidden-layer neural network, and the input data is a time-frequency matrix. By minimizing the training error, the weight parameters from the single hidden layer to the output layer are obtained. Then, after obtaining new data, the abnormal sound is represented through inference.

[0122] In summary, in step 406, by introducing the first and second sub-models, and using only a small number of abnormal samples, the abnormal sounds are classified using onomatopoeia (i.e., custom classification, such as what kind of abnormal sound corresponds to what type of fault). The system outputs what kind of abnormal sound ("buzzing" sound, "clanging" sound, etc.), effectively distinguishing various abnormal situations and realizing the modeling and mapping of parameters to semantics (in traditional solutions, predictive maintenance systems only output the existence of abnormal sounds, but do not indicate what kind of sound, requiring workers to review and classify, which is inefficient. There is no effective mapping relationship between abnormal sounds and onomatopoeia, making it difficult to represent abnormal voiceprints).

[0123] 407. The test result of the equipment to be tested is confirmed to be in normal operation.

[0124] The computer equipment determines that the device under test is functioning normally, so it only needs to output a result indicating that there are no abnormalities, and no further processing is required.

[0125] Based on the description of the abnormal audio detection method in the above embodiments of this application, the main differences between the embodiments of this application and traditional methods can be summarized in the following three aspects (please refer to Figure 9 for details):

[0126] (1) Multi-scale endpoint detection

[0127] By characterizing audio changes at multiple scales using time and frequency domain features and combining event correlation to smooth sound scenes, it accurately identifies the start and end points of target sounds, effectively removes environmental noise interference, and has strong noise resistance.

[0128] (2) Unsupervised abnormal sound detection

[0129] The audio signal is filtered by at least one filter (e.g., Mel frequency filter bank), and key features in the audio signal are extracted by multiple first models (e.g., wavelet kernel fusion convolution). The harmonic dictionary and impulse dictionary are combined to form a composite dictionary, thereby extracting key sparse information related to faults in the audio signal. The unsupervised SVDD scheme is adopted, which does not require exhaustive enumeration of all abnormal situations, effectively improving the accuracy of abnormal sound detection.

[0130] (3) Abnormal voiceprint characteristics

[0131] In cases with few samples, an interpretable time-frequency weight matrix is ​​introduced to mine key features of anomalies, realize the onomatopoeia of abnormal sounds, and achieve parameter-to-semantic modeling and mapping.

[0132] The differences in the above three aspects can be specifically reflected in the test results of the speaker abnormal sound detection scenario shown in Figure 10. As can be seen from Figure 10, it can ultimately achieve accurate abnormal sound detection and voiceprint representation, with strong robustness, high accuracy, and strong interpretability.

[0133] Based on the above embodiments, in order to better implement the above solutions of the embodiments of this application, related equipment for implementing the above solutions is also provided below. Referring specifically to Figure 11, which is a schematic diagram of a computer device provided in an embodiment of this application, the computer device 1100 may specifically include: an acquisition module 1101, a characterization module 1102, a correlation module 1103, a feature extraction module 1104, and an anomaly determination module 1105. The acquisition module 1101 is used to acquire first audio data, which includes the operating audio of the device under test and the ambient audio around the device under test. The characterization module 1102 is used to perform time-domain characterization and frequency-domain characterization on the first audio data to obtain first time-domain data and first frequency-domain data, respectively. The correlation module 1103 is used to obtain second audio data based on the correlation between the first time-domain data and the first frequency-domain data, which includes the operating audio. The feature extraction module 1104 is used to extract features from the second audio data using a trained first model to obtain target features. The anomaly determination module 1105 is used to obtain sparse coefficients based on the target features and determine whether the second audio data is abnormal audio based on the sparse coefficients.

[0134] In one possible design, the first model after training is obtained by training the first model on a first training set based on a first loss function; the first training set includes multiple first training data, which are filtered audio data, and the first training data has a first label, which is used to characterize whether the first training data is abnormal audio.

[0135] In one possible design, the first model includes: multi-wavelet kernel fusion convolution.

[0136] In one possible design, the anomaly determination module 1105 is specifically used to: when the second audio data is determined to be abnormal audio based on the sparse coefficient, input the second audio data into the trained second model to obtain the classification category of the second audio data, which is used to characterize the anomaly type of the second audio data.

[0137] In one possible design, the trained second model includes a trained first sub-model and a trained second sub-model. The trained first sub-model is trained on the first sub-model using a second training set based on a second loss function. The second training set includes multiple second training data of different anomaly types, represented by a time-frequency graph with time on the horizontal axis and frequency on the vertical axis. The trained second sub-model is trained on the second sub-model using a third training set based on a third loss function. The third training set includes multiple third training data, which are the output data of the trained first sub-model. The third training data has a second label, which is used to represent the classification category of the third training data.

[0138] In one possible design, the first input data and the second input data are any two different training data in the second training set. The iteration termination conditions for the training of the first sub-model include: the difference between the first input data and the first output data corresponding to the first sub-model is less than a first preset threshold; and / or, the difference between the first output data and the second output data corresponding to the first sub-model is greater than a second preset threshold, and the second output data corresponds to the second input data.

[0139] In one possible design, the first sub-model includes: a weight matrix.

[0140] In one possible design, the characterization module 1102 is specifically used to: perform time-domain characterization on the first audio data based on time-domain feature parameters to obtain first time-domain data, wherein the time-domain feature parameters include short-time energy or short-time zero-crossing rate; and perform frequency-domain characterization on the first audio data based on frequency-domain feature parameters to obtain first frequency-domain data, wherein the frequency-domain feature parameters include spectral centroid distance.

[0141] In one possible design, the first time-domain data includes m segments of sub-time-domain data, and the first frequency-domain data includes n segments of sub-frequency-domain data. The correlation module 1103 is specifically used to: calculate the correlation between the target frame signal and the previous p frames of the target frame signal, wherein the target frame signal is any one frame signal from the m segments of sub-time-domain data or the n segments of frequency-domain data; and remove the ambient audio from the first audio data based on the correlation to obtain the second audio data.

[0142] In one possible design, the feature extraction module 1104 is specifically used to: filter the second audio data through at least one filter to obtain filtered second audio data; and extract features from the filtered second audio data through a trained first model.

[0143] In one possible design, the filter includes: a Mel filter.

[0144] In one possible design, the anomaly determination module 1105 is further used to: update the composite dictionary based on the target feature, which is obtained based on the harmonic dictionary and the impulse dictionary; and perform sparse coding on the composite dictionary to obtain sparse coefficients.

[0145] It should be noted that the information interaction and execution process between the modules / units in the computer device 1100 are based on the same concept as the method embodiments described above in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0146] Next, we will introduce another computer device provided in the embodiments of this application. Please refer to Figure 12. Figure 12 is a schematic diagram of the structure of a computer device provided in the embodiments of this application. The computer device 1200 may deploy the computer device 1100 described in the embodiment corresponding to Figure 11, for implementing the functions of the computer device 1100 in the embodiment corresponding to Figure 11. Specifically, the computer device 1200 is implemented by one or more servers. The computer device 1200 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1222 and memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and storage media 1230 may be temporary storage or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the computer device 1200. Furthermore, the central processing unit 1222 can be configured to communicate with the storage medium 1230 and execute a series of instruction operations in the storage medium 1230 on the computer device 1200.

[0147] Computer device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems 1241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0148] In this embodiment, the central processing unit 1222 is used to execute the steps performed by the computer device in the above-described method embodiments of this application. For example, the central processing unit 1222 can be used to: First, collect the raw audio data (i.e., first audio data) of the device under test, which includes the operating audio emitted by the device under test during operation and the ambient audio around the device under test (e.g., sounds from other workstations, worker voices, object friction sounds, etc.). After acquiring the first audio data, perform time-domain representation on the first audio data to obtain corresponding time-domain data, which can be called first time-domain data; in addition, perform frequency-domain representation on the first audio data to obtain corresponding frequency-domain data, which can be called first frequency-domain data. Calculate the correlation between the first time-domain data and the first frequency-domain data obtained above, and obtain second audio data based on the correlation, which includes the operating audio emitted by the device under test during operation (which can be part or all, not limited). Then, perform feature extraction on the second audio data using a trained first model to obtain target features, which can be one or more, and this application does not limit this. Finally, based on the extracted target features, the sparsity coefficients are obtained (i.e., key sparse information related to the fault is extracted from the target features), and the second audio data is judged as abnormal audio based on the obtained sparsity coefficients.

[0149] It should be noted that the specific way in which the central processing unit 1222 executes the above steps is based on the same concept as the method embodiment described above in this application, and the resulting technical effects are also the same as those in the above embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, which will not be repeated here.

[0150] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0152] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0153] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for detecting abnormal audio, characterized in that, include: Acquire first audio data, which includes the operating audio of the device under test and the ambient audio around the device under test; The first audio data is represented in the time domain and the frequency domain respectively to obtain the first time domain data and the first frequency domain data respectively; Based on the correlation between the first time-domain data and the first frequency-domain data, second audio data is obtained, and the second audio data includes the running audio. The target features are obtained by extracting features from the second audio data using the trained first model. Based on the target features, sparse coefficients are obtained, and based on the sparse coefficients, it is determined whether the second audio data is abnormal audio.

2. The method according to claim 1, characterized in that, The first trained model is obtained by training the first model using the first training set based on the first loss function. The first training set includes multiple first training data, which are filtered audio data. The first training data has a first label, which is used to characterize whether the first training data is abnormal audio.

3. The method according to claim 2, characterized in that, The first model includes: Multi-wavelet kernel fusion convolution.

4. The method according to any one of claims 1-3, characterized in that, After determining whether the second audio data is abnormal audio based on the sparse coefficients, the method further includes: If the second audio data is determined to be abnormal audio based on the sparse coefficients, the second audio data is input into the trained second model to obtain the classification category of the second audio data. The classification category is used to characterize the abnormality type of the second audio data.

5. The method according to claim 4, characterized in that, The trained second model includes the trained first sub-model and the trained second sub-model; The first sub-model after training is obtained by training the first sub-model based on the second loss function and using the second training set. The second training set includes second training data of multiple different anomaly types. The second training data is represented by a time-frequency graph with time on the horizontal axis and frequency on the vertical axis. The trained second sub-model is obtained by training the second sub-model based on the third loss function and using the third training set. The third training set includes multiple third training data, which are the output data of the trained first sub-model. The third training data has a second label, which is used to characterize the classification category of the third training data.

6. The method according to claim 5, characterized in that, The first input data and the second input data are any two different training data in the second training set, and the iteration termination conditions for training the first sub-model include: The difference between the first input data and the first output data corresponding to the first sub-model is less than a first preset threshold. And / or, The difference between the first output data and the second output data output by the first sub-model is higher than a second preset threshold, and the second output data corresponds to the second input data.

7. The method according to any one of claims 5-6, characterized in that, The first sub-model includes: Weight matrix.

8. The method according to any one of claims 1-7, characterized in that, The step of performing time-domain representation and frequency-domain representation on the first audio data to obtain first time-domain data and first frequency-domain data includes: The first audio data is represented in the time domain based on time domain feature parameters to obtain first time domain data. The time domain feature parameters include short-time energy or short-time zero-crossing rate. The first audio data is characterized in the frequency domain based on frequency domain feature parameters to obtain the first frequency domain data, wherein the frequency domain feature parameters include the spectral centroid distance.

9. The method according to any one of claims 1-8, characterized in that, The first time-domain data includes m segments of sub-time-domain data, and the first frequency-domain data includes n segments of sub-frequency-domain data. The step of obtaining the second audio data based on the correlation between the first time-domain data and the first frequency-domain data includes: Calculate the correlation between the target frame signal and the p frames preceding the target frame signal, wherein the target frame signal is any one frame signal from the m segments of time-domain data or the n segments of frequency-domain data; Based on the correlation, environmental audio is removed from the first audio data to obtain the second audio data.

10. The method according to any one of claims 1-9, characterized in that, The feature extraction of the second audio data using the trained first model includes: The second audio data is filtered by at least one filter to obtain filtered second audio data; The filtered second audio data is then used to extract features using the trained first model.

11. The method according to claim 10, characterized in that, The filter includes: Mel filter.

12. The method according to any one of claims 1-11, characterized in that, The process of obtaining the sparsity coefficients based on the target features includes: The composite dictionary is updated based on the target features, and the composite dictionary is obtained based on the harmonic dictionary and the impulse dictionary. The composite dictionary is sparsely encoded to obtain sparse coefficients.

13. A computer device, characterized in that, include: The acquisition module is used to acquire first audio data, which includes the operating audio of the device under test and the ambient audio around the device under test. The characterization module is used to perform time-domain characterization and frequency-domain characterization on the first audio data respectively, to obtain first time-domain data and first frequency-domain data respectively; The correlation module is used to obtain second audio data based on the correlation between the first time-domain data and the first frequency-domain data, wherein the second audio data includes the running audio. The feature extraction module is used to extract features from the second audio data using the trained first model to obtain target features; An anomaly determination module is used to obtain sparse coefficients based on the target features, and to determine whether the second audio data is abnormal audio based on the sparse coefficients.

14. The device according to claim 13, characterized in that, The first trained model is obtained by training the first model using the first training set based on the first loss function. The first training set includes multiple first training data, which are filtered audio data. The first training data has a first label, which is used to characterize whether the first training data is abnormal audio.

15. The device according to claim 14, characterized in that, The first model includes: Multi-wavelet kernel fusion convolution.

16. The device according to any one of claims 13-15, characterized in that, The anomaly determination module is specifically used for: If the second audio data is determined to be abnormal audio based on the sparse coefficients, the second audio data is input into the trained second model to obtain the classification category of the second audio data. The classification category is used to characterize the abnormality type of the second audio data.

17. The device according to claim 16, characterized in that, The trained second model includes the trained first sub-model and the trained second sub-model; The first sub-model after training is obtained by training the first sub-model based on the second loss function and using the second training set. The second training set includes second training data of multiple different anomaly types. The second training data is represented by a time-frequency graph with time on the horizontal axis and frequency on the vertical axis. The trained second sub-model is obtained by training the second sub-model based on the third loss function and using the third training set. The third training set includes multiple third training data, which are the output data of the trained first sub-model. The third training data has a second label, which is used to characterize the classification category of the third training data.

18. The device according to claim 17, characterized in that, The first input data and the second input data are any two different training data in the second training set, and the iteration termination conditions for training the first sub-model include: The difference between the first input data and the first output data corresponding to the first sub-model is less than a first preset threshold. And / or, The difference between the first output data and the second output data output by the first sub-model is higher than a second preset threshold, and the second output data corresponds to the second input data.

19. The device according to any one of claims 17-18, characterized in that, The first sub-model includes: Weight matrix.

20. The device according to any one of claims 13-19, characterized in that, The characterization module is specifically used for: The first audio data is represented in the time domain based on time domain feature parameters to obtain first time domain data. The time domain feature parameters include short-time energy or short-time zero-crossing rate. The first audio data is characterized in the frequency domain based on frequency domain feature parameters to obtain the first frequency domain data, wherein the frequency domain feature parameters include the spectral centroid distance.

21. The device according to any one of claims 13-20, characterized in that, The first time-domain data includes m segments of sub-time-domain data, and the first frequency-domain data includes n segments of frequency-domain data. The correlation module is specifically used for: Calculate the correlation between the target frame signal and the p frames preceding the target frame signal, wherein the target frame signal is any one frame signal from the m segments of time-domain data or the n segments of frequency-domain data; Based on the correlation, environmental audio is removed from the first audio data to obtain the second audio data.

22. The device according to any one of claims 13-21, characterized in that, The feature extraction module is specifically used for: The second audio data is filtered by at least one filter to obtain filtered second audio data; The filtered second audio data is then used to extract features using the trained first model.

23. The device according to claim 22, characterized in that, The filter includes: Mel filter.

24. The device according to any one of claims 13-23, characterized in that, The anomaly determination module is further used for: The composite dictionary is updated based on the target features, and the composite dictionary is obtained based on the harmonic dictionary and the impulse dictionary. The composite dictionary is sparsely encoded to obtain sparse coefficients.

25. A computer device comprising a processor and a memory, the processor being coupled to the memory, characterized in that, The memory is used to store programs; The processor is configured to execute a program in the memory, causing the computer device to perform the method as described in any one of claims 1-12.

26. A computer storage medium, characterized in that, The device stores computer-readable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-12.

27. A computer program product, characterized in that, The computer program product includes computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-12.

28. A chip, the chip comprising a processor and a data interface, characterized in that, The processor reads instructions stored in the memory through the data interface and executes the method as described in any one of claims 1-12.

Images