In-car ambient sound enhancement method and apparatus, and electronic device and storage medium

Abstract

An in-car ambient sound enhancement method and apparatus, and an electronic device and a storage medium, which are used to solve the technical problems in the related art of low recognition accuracy of external ambient sound of a vehicle and poor environmental adaptability. The method comprises: collecting out-of-vehicle audio data and out-of-vehicle video data; on the basis of the out-of-vehicle audio data and the out-of-vehicle video data, performing scene recognition, so as to obtain a current ambient sound type and a current driving environment; extracting visual information from the out-of-vehicle video data, and extracting audio information from the out-of-vehicle audio data; on the basis of the visual information and the audio information, performing multi-modal data fusion based on comprehensive confidence, so as to obtain an effective scene determination result; and on the basis of the current driving environment, the current ambient sound type, and the effective scene determination result, enhancing out-of-vehicle ambient sound by means of an audio pass-through enhancement strategy. Therefore, on the basis of the multi-modal data fusion, the recognition accuracy of ambient sound can be improved, and in combination with the audio pass-through enhancement strategy, the ambient sound can be enhanced, thereby improving the environmental adaptability.

Description

Methods, devices, electronic equipment and storage media for enhancing ambient sound in automobiles

[0001] This application claims priority to Chinese Patent Application No. 202411984313.5, filed on December 30, 2024, entitled “Method, Apparatus, Electronic Device and Storage Medium for Enhancing Ambient Noise in Automobiles”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of automotive intelligent cockpit technology, and in particular to a method, device, electronic device, and storage medium for enhancing automotive ambient sound. Background Technology

[0003] With the development of modern automotive intelligence, vehicle sound insulation performance has been continuously improved, significantly reducing in-vehicle noise. However, during driving, the excessive quiet can make it difficult for drivers to perceive changes in the external environment, potentially causing them to miss important cues such as alarms and horns. Furthermore, prolonged driving in a very quiet environment can easily lead to fatigue and boredom. To enhance driving and passenger safety, and improve the auditory experience for in-vehicle passengers, current solutions primarily focus on strengthening or optimizing the sound environment both inside and outside the vehicle through in-car ambient sound enhancement.

[0004] Currently, most systems employ multi-microphone combinations to enhance ambient sound in automobiles. While these systems can suppress background noise (such as wind noise, tire noise, and engine noise from other vehicles) to some extent, their noise suppression effect is limited in complex environments (such as high-speed driving, multiple vehicles driving side-by-side, and heavy traffic), failing to provide high-quality audio signals and exhibiting poor environmental adaptability. Furthermore, because they rely on audio signals for sound source localization, in complex environments, relying solely on audio signals is insufficient to accurately identify and locate target sound sources, resulting in low ambient sound recognition accuracy. This leads to poor capture of relevant scene sounds (such as emergency braking sounds and alarm sounds), impacting system performance. Summary of the Invention

[0005] This invention provides a method, device, electronic device, and storage medium for enhancing automotive ambient sound, which solves or partially solves the technical problems of low accuracy in recognizing automotive external ambient sound and poor environmental adaptability in related technologies.

[0006] This invention provides a method for enhancing ambient sound in automobiles, the method comprising:

[0007] Collects external audio and video data from outside the vehicle;

[0008] Scene recognition is performed based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment.

[0009] Visual information is extracted from the external video data, and audio information is extracted from the external audio data;

[0010] By combining the visual information and the audio information, multimodal data fusion based on comprehensive confidence is performed to obtain effective scene judgment results;

[0011] Based on the current driving environment, the current ambient sound type, and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through a sound passthrough enhancement strategy.

[0012] Optionally, the step of performing scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment includes:

[0013] The external audio data and the external video data are input into a pre-trained multimodal neural network; the multimodal neural network includes a driving environment classification model and an ambient sound classification model.

[0014] The ambient sound classification model extracts audio features from the external audio data and classifies the ambient sound based on the audio features to identify the current ambient sound type of the vehicle.

[0015] The driving environment classification model extracts surrounding environment features from the external video data and classifies the driving environment based on these features to identify the vehicle's current driving environment.

[0016] Optionally, extracting visual information from the external video data includes:

[0017] The system detects and locates all faces in the external video data using a face detection algorithm, and simultaneously tracks the detected faces in real time using a face tracking algorithm, generating face detection information.

[0018] Based on the face detection information, head pose estimation is performed to obtain orientation judgment information;

[0019] Facial key point information is extracted from the face detection information. The lip region is located based on the facial key point information. The visual features of the lip region are extracted. Lip movement detection is performed based on the visual features to obtain lip movement detection information.

[0020] A face detection confidence score is generated based on the face detection information, a direction judgment confidence score is generated based on the direction judgment information, and a lip movement detection confidence score is generated based on the lip movement detection information.

[0021] Visual information is obtained by weighted fusion calculation based on the confidence scores of face detection, orientation determination, and lip movement detection.

[0022] Optionally, extracting audio information from the external audio data includes:

[0023] The external audio data is used to locate the sound source and determine the direction information of the sound source;

[0024] Perform speech recognition on the external audio data to obtain speech recognition information;

[0025] A sound source direction confidence score is generated based on the sound source direction information, and a speech recognition confidence score is generated based on the speech recognition information;

[0026] Audio information is obtained by weighted fusion calculation based on the confidence level of the sound source direction and the confidence level of the speech recognition.

[0027] Optionally, the step of combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result includes:

[0028] Multimodal data fusion is performed based on the visual information and the audio information to obtain multimodal fusion information;

[0029] Solve for the overall confidence level of the multimodal fusion information;

[0030] When the overall confidence level is greater than a preset confidence threshold, the external ambient sound scene is determined to be a valid scene.

[0031] When the overall confidence level is less than or equal to the preset confidence threshold, the external ambient sound scene is determined to be an invalid scene.

[0032] Optionally, the step of enhancing the external ambient sound based on the current driving environment, the current ambient sound type, and the effective scene judgment result, through a sound pass-through enhancement strategy, includes:

[0033] The transmission volume of the external ambient sound is dynamically adjusted based on the current ambient sound type and its intensity.

[0034] When the valid scene judgment result indicates that the external ambient sound scene is a valid scene, the transmission volume of the external ambient sound is dynamically adjusted in conjunction with the current driving environment, and / or the audio of the external ambient sound is enhanced.

[0035] Optionally, a user display interface is provided in front of the driver's cabin, the user display interface including an ambient sound type display area and a sound intensity adjustment button; the ambient sound type display area displays several adjustable ambient sound types; the vehicle ambient sound enhancement method further includes:

[0036] When an ambient sound type selection operation is detected for the ambient sound type display area, a target adjustable ambient sound type is determined from the plurality of adjustable ambient sound types.

[0037] In response to the adjustment operation of the sound intensity adjustment button, the transmission volume of the external ambient sound corresponding to the target ambient sound type is adjusted.

[0038] The present invention also provides an automotive ambient sound enhancement device, comprising:

[0039] The data acquisition unit is used to collect external audio data and external video data.

[0040] The scene recognition unit is used to perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment.

[0041] An information extraction unit is used to extract visual information from the external video data and audio information from the external audio data;

[0042] A multimodal data fusion unit is used to combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result;

[0043] The external ambient sound enhancement unit is used to enhance the external ambient sound based on the current driving environment, the current ambient sound type, and the effective scene judgment result, through a sound pass-through enhancement strategy.

[0044] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0045] The memory is used to store program code and transmit the program code to the processor;

[0046] The processor is configured to execute the automotive ambient sound enhancement method as described above, according to instructions in the program code.

[0047] The present invention also provides a computer-readable storage medium for storing program code for performing the automotive ambient sound enhancement method as described in any of the preceding claims.

[0048] As can be seen from the above technical solutions, the present invention has the following advantages:

[0049] This paper presents a method for enhancing automotive ambient sound based on multimodal data fusion. First, external audio and video data are collected. Then, scene recognition is performed based on the external audio and video data to obtain the current ambient sound type and driving environment. Visual information is extracted from the external video data, and audio information is extracted from the external audio data. Next, multimodal data fusion based on comprehensive confidence is performed, combining the visual and audio information to obtain an effective scene judgment result. Finally, based on the current driving environment, the current ambient sound type, and the effective scene judgment result, the external ambient sound is enhanced through a sound pass-through enhancement strategy. By introducing multimodal fusion processing technology and using multimodal data fusion based on comprehensive confidence for effective scene judgment, the accuracy of ambient sound recognition can be improved. Furthermore, the dynamic adjustment of ambient sound enhancement through the sound pass-through enhancement strategy improves environmental adaptability. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 is a schematic diagram of a car ambient sound enhancement system;

[0052] Figure 2 is a flowchart of a method for enhancing ambient noise in automobiles;

[0053] Figure 3 is a schematic diagram of a user interface;

[0054] Figure 4 is a schematic diagram of the overall process of a method for enhancing ambient sound in automobiles;

[0055] Figure 5 is a structural block diagram of an automotive ambient sound enhancement device. Detailed Implementation

[0056] This invention provides a method, device, electronic device, and storage medium for enhancing automotive ambient sound, which solves or partially solves the technical problems of low accuracy in recognizing automotive external ambient sound and poor environmental adaptability in related technologies.

[0057] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0058] As an example, most current solutions employ multi-microphone systems to enhance ambient sound in automobiles. While multi-microphone systems can suppress background noise (such as wind noise, tire noise, and engine noise from other vehicles) to some extent, their noise suppression effect is limited in complex environments (such as high-speed driving, multiple vehicles driving side-by-side, and heavy traffic), failing to provide high-quality audio signals and exhibiting poor environmental adaptability. Furthermore, because they rely on audio signals for sound source localization, in complex environments, it is difficult to accurately identify and locate target sound sources using only audio signals, resulting in low ambient sound recognition accuracy. This leads to poor capture of effective scene sounds (such as emergency braking sounds and alarm sounds), impacting system performance.

[0059] Further analysis reveals that current audio processing algorithms lack dynamic adjustment capabilities, failing to optimize audio processing strategies in real time based on environmental changes, resulting in significant performance variations across different environments. Furthermore, current multi-microphone systems rely solely on audio information, failing to fully utilize visual information provided by cameras. This not only hinders multimodal information fusion, limiting the system's robustness and accuracy (especially in sound recognition and capture in complex environments), but also makes it difficult to accurately identify complex driving scenarios, leading to low flexibility and intelligence in adjustment strategies. Moreover, users can only perceive the environment through sound, lacking intuitive visual feedback, resulting in a limited user experience.

[0060] Therefore, one of the core inventive points of this invention is: addressing the shortcomings of current technologies, and to solve or partially solve problems such as insufficient background noise suppression, low sound source localization accuracy, poor environmental adaptability, and insufficient multimodal information fusion in the automotive external environment, this invention provides an automotive environmental sound enhancement system and its application method based on multimodal fusion technology. By introducing multimodal fusion perception technology, combining audio and video information, and utilizing various deep learning algorithms, image data captured by the camera and audio data captured by the microphone are jointly processed to generate richer multimodal features. This enables comprehensive recognition of various environmental sounds, improves the quality of audio signals and the capture effect of effective scene sounds, and enhances the accuracy of sound source localization. Simultaneously, through a sound passthrough enhancement strategy, the audio processing algorithm is automatically adjusted according to environmental changes, suppressing background noise while enhancing the clarity of effective scene sounds. This allows for clear capture of effective scene sounds even in complex environments. Especially in high-speed driving or complex traffic environments, it can continuously provide high-quality audio capture and enhancement effects, improving driving safety and user experience.

[0061] Referring to Figure 1, a schematic diagram of the structure of an automotive ambient sound enhancement system provided in an embodiment of the present invention is shown.

[0062] Referring to Figure 1, the vehicle ambient sound enhancement system 100 mainly includes a central processing unit (CPU) 101, an external sound pickup module 102, a camera module 103, an audio playback module 104, and a user display interface 105, all of which are communicatively connected to the central processing unit 101.

[0063] Among them, the external sound pickup module 102 can be a microphone, which is mainly used to collect ambient sound outside the vehicle and transmit it as external audio data to the central processing unit 101.

[0064] The camera module 103 can be a camera (Camera, Cam), which is mainly used to collect visual information related to the outside of the vehicle and transmit it as video data to the central processing unit 101.

[0065] In practical applications, multimodal fusion can be achieved by capturing visual information about the vehicle's surroundings using a camera and combining it with audio signals from a microphone. This fusion can more accurately identify and locate external sound sources, improving the precision of sound capture. For the detection and recognition of people outside the vehicle, the camera can capture facial expressions, lip movements, and body language of speakers outside the vehicle. Combined with audio signals captured by the microphone, this allows for a more comprehensive analysis of the behavior and emotions of people outside the vehicle, improving the accuracy of speech recognition. This is especially true in noisy environments or scenarios with heavy accents, significantly enhancing the performance of the speech recognition system.

[0066] Simultaneously, the camera can monitor environmental changes in real time, such as light and background noise. Combined with the audio signal from the microphone, the audio processing strategy can be dynamically adjusted to achieve more effective noise suppression and sound source enhancement, improving the system's environmental adaptability. This makes the system more stable and reliable in different environments.

[0067] By combining a microphone and a camera, the problems of noise interference and inaccurate speech recognition inherent in single microphones are solved, significantly improving audio quality and user experience. Even if the audio signal is interfered with, video information can provide additional context to help the system better understand the audio content.

[0068] The central processing unit 101 receives data from the external sound pickup module 102 and the camera module 103, uses deep learning algorithms to process and analyze the acquired data based on ambient sound and visual information, and executes a sound pass-through enhancement strategy based on the data processing results.

[0069] The audio playback module 104 can be a speaker, mainly used to play ambient sounds.

[0070] The user display interface 105 can be a touch screen, primarily providing users with the ability to independently set and control the sound pass-through enhancement strategy. That is, it allows users to set the intensity and type of sound pass-through according to their individual needs. For example, users can choose to pass through emergency vehicle alarms, children's voices, external command and communication voices, etc., according to actual needs.

[0071] Based on the aforementioned automotive ambient sound enhancement system, and referring to Figure 2, a flowchart of a method for enhancing automotive ambient sound according to an embodiment of the present invention is shown, which specifically includes the following steps:

[0072] Step 201: Collect external audio data and external video data;

[0073] Specifically, ambient sound data (external audio data) outside the vehicle can be collected in real time through a microphone, and external video data can be collected in real time through a camera for subsequent processing.

[0074] Step 202: Perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment;

[0075] The multimodal data fusion in this embodiment of the invention can be understood as fusing the external audio data collected by the microphone and the external video data collected by the camera to achieve comprehensive judgment.

[0076] In this step, scene recognition is mainly performed using deep learning models (such as multimodal neural networks). Multimodal neural networks can simultaneously process audio and video data, performing corresponding feature extraction and classification operations.

[0077] In the specific implementation, scene recognition is performed based on external audio data and external video data to obtain the current ambient sound type and the current driving environment, which can be:

[0078] First, external audio and video data are input into a pre-trained multimodal neural network. This multimodal neural network primarily includes a driving environment classification model and an ambient sound classification model.

[0079] Driving environment classification models can specifically be deep learning models pre-trained based on driving environment classification, such as Convolutional Neural Networks (CNNs). Ambient sound classification models can specifically be audio classification models based on ambient sound classification, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs).

[0080] Next, an ambient sound classification model is used to extract audio features (such as voiceprint features and spectral features) from the external audio data, and then the ambient sound is classified based on these features to identify the current type of ambient sound in the vehicle. For example, a pre-trained audio classification model (such as a convolutional neural network (CNN) or a recurrent neural network (RNN)) can be used to classify the external audio data and identify the current type of ambient sound. Ambient sound types can include pedestrian sounds, traffic signs, vehicle alarms, background noise, etc.

[0081] Simultaneously, a driving environment classification model is used to extract surrounding environmental features (such as surrounding environment features, building features, roadside object features, etc.) from the external video data, and the driving environment is classified according to these features to identify the vehicle's current driving environment. For example, a pre-trained driving environment classification model can be used to classify the external video data and identify the current driving environment. The driving environment can include types such as urban, highway, and rural environments.

[0082] Step 203: Extract visual information from the external video data and extract audio information from the external audio data;

[0083] Effective judgment of external ambient sound scenes requires the simultaneous analysis of visual and audio information. This process can be viewed as combining video and audio data to determine correlation features. Correlation feature determination can be understood as: when it is necessary to determine whether audio data is useful information, it is necessary to combine it with video data for judgment.

[0084] For example, suppose speech recognition identifies a voice in the audio data, but video data indicates the speaker is not facing the vehicle. In this case, the ambient sound scenario (specifically referring to the voice information) is invalid. Similarly, speech recognition identifies an alarm sound in the audio data, but video data analysis shows no police cars, fire trucks, or ambulances nearby. In this case, the ambient sound scenario (specifically referring to the alarm sound) is also invalid.

[0085] For the visual information analysis process, in some embodiments, the process of extracting visual information from vehicle exterior video data includes the following sub-steps S01 to S05:

[0086] Step S01: Detect and locate all faces in the video data outside the vehicle using a face detection algorithm, and simultaneously track the detected faces in real time using a face tracking algorithm, and generate face detection information;

[0087] Step S01 mainly implements face detection and tracking. Specifically, face detection algorithms, such as Multi-Task Cascaded Convolutional Networks (MTCNN), can be used to detect all faces in the video, and tracking algorithms, such as KCF (Kernelized Correlation Filters), can be used to track the detected faces in real time.

[0088] Among them, the Multi-Task Cascaded Convolutional Neural Network (MTCNN) is an efficient face detection algorithm for face detection and facial landmark localization. It mainly consists of three stages: PNet (Proposal Network), RNet (Refine Network), and ONet (Output Network). Each stage has a specific task to progressively improve the accuracy and efficiency of detection.

[0089] The main task of PNet is to generate candidate bounding boxes, that is, to initially detect regions that may contain faces. Its main workflow is as follows: first, input image x; then extract features of x through convolutional layers; then activate the features using an activation function (such as ReLU); finally, generate candidate bounding boxes through fully connected layers.

[0090] The calculation formula corresponding to the PNet processing flow is as follows: p=σ(W p x+b p )

[0091] Where p represents the generated candidate box; W p b represents the weight matrix; p σ represents the bias vector; σ represents the activation function (such as ReLU).

[0092] The main task of RNet is to filter candidate bounding boxes and remove most non-face regions. Its main workflow is as follows: First, the candidate bounding box p generated by PNet is taken as input; then, features are further extracted from p through convolutional layers; then, activation functions (such as ReLU) are used to activate the features; finally, more accurate candidate bounding boxes are generated through fully connected layers.

[0093] The calculation formula corresponding to the RNet processing flow is as follows: r=σ(W r p+b r )

[0094] Where r represents the filtered candidate box; W r b represents the weight matrix; r σ represents the bias vector; σ represents the activation function (such as ReLU).

[0095] The main task of ONet is to accurately locate faces and generate facial landmarks. Its main workflow is as follows: First, the candidate bounding box r generated by RNet is used as input; then, features are further extracted from r through convolutional layers; then, activation functions (such as ReLU) are used to activate the features; finally, the final face bounding box and landmarks are generated through fully connected layers.

[0096] The calculation formula corresponding to the ONet processing flow is as follows: o=σ(W o r+b o )

[0097] Where o represents the final face bounding box and key points; W o b represents the weight matrix; o σ represents the bias vector; σ represents the activation function (such as ReLU).

[0098] Based on the above introduction, in the specific implementation, for face detection, firstly, continuous frames are extracted from the video data outside the vehicle; then, PNet is applied to each frame image x to generate preliminary candidate boxes p; then, RNet is applied to the candidate boxes p generated by PNet to filter out more accurate candidate boxes r; finally, ONet is applied to the candidate boxes r generated by RNet to accurately locate the face and generate key points o.

[0099] Furthermore, for face tracking, the first step is to use a multi-task cascaded convolutional neural network (MTCNN) to detect faces in the first frame of the image, generating initial face bounding boxes and key points. Then, tracking algorithms (such as KCF and DeepSort) are used to continuously track the detected faces in real time to obtain face detection information.

[0100] Step S02: Perform head pose estimation based on face detection information to obtain orientation judgment information;

[0101] Head pose estimation can be achieved primarily in two ways. One is by using face detection information as model input and employing a pose estimation model, such as OpenPose (a real-time multi-person pose estimation algorithm), to analyze the face's pose and determine whether the face is facing the vehicle. The other is by analyzing the head's pose (such as the head's rotation angle) based on face detection information to determine whether the face is facing the vehicle. Through head pose estimation, corresponding orientation information can be obtained.

[0102] Step S03: Extract facial key point information from face detection information, locate the lip region based on facial key point information, extract visual features of the lip region, and perform lip movement detection based on visual features to obtain lip movement detection information.

[0103] Lip movement detection can be seen as determining whether a person is speaking by analyzing facial expressions (such as assessing the degree of mouth opening and closing based on facial landmark information).

[0104] Specifically, firstly, facial key point information can be extracted from face detection information to locate the lip region; then, a pre-trained convolutional neural network model, such as VGG (Visual Geometry Group) or ResNet (Residual Network), is used to extract visual features of the lip region; then, the extracted visual features are input into a recurrent neural network (RNN) pre-trained based on visual feature recognition to model the temporal information of lip movements; finally, the output of the RNN passes through a fully connected layer and an activation layer (softmax function) to obtain the lip movement recognition result (whether speech has been heard), i.e., lip movement detection information.

[0105] Step S04: Generate face detection confidence based on face detection information, generate direction judgment confidence based on direction judgment information, and generate lip movement detection confidence based on lip movement detection information;

[0106] In this embodiment of the invention, the above-mentioned judgment results need to be converted into corresponding confidence levels before subsequent comprehensive analysis. Therefore, the confidence level can be calculated for each judgment result. Specifically, a face detection confidence level is generated based on the face detection information and defined as V. face The direction judgment confidence score is generated based on the direction judgment information and defined as V. direction Based on the lip movement detection information, a lip movement detection confidence score is generated and defined as V. lip .

[0107] Step S05: Perform weighted fusion calculation based on face detection confidence, orientation judgment confidence, and lip movement detection confidence to obtain visual information.

[0108] The formula for calculating visual information V is as follows: V = w1V face +w2V direction +w3V lip

[0109] Here, w1, w2, and w3 all represent weights, used to balance the importance of different information in visual information.

[0110] For the audio information analysis process, in some embodiments, the process of extracting audio information from external audio data includes the following sub-steps S11 to S14:

[0111] Step S11: Locate the sound source from the external audio data and determine the direction information of the sound source;

[0112] Microphone array technology (such as the DOA (Direction of Arrival) algorithm) can be used to locate the sound source in the external audio data, determine the direction of the sound source, and check whether the direction of the sound source is consistent with the direction of the vehicle to obtain the sound source direction information. If the direction of the sound source is consistent with the direction of the vehicle, the sound source direction information is considered valid.

[0113] Step S12: Perform speech recognition on the external audio data to obtain speech recognition information;

[0114] Speech recognition algorithms (such as DeepSpeech) can be used to identify the speech content in external audio data, determine whether the identified speech content is related to the vehicle (such as "stop" or "give way"), and generate speech recognition information. If the identified speech content is related to the vehicle, the speech recognition information is considered valid.

[0115] Step S13: Generate sound source direction confidence based on sound source direction information, and generate speech recognition confidence based on speech recognition information;

[0116] Specifically, a sound source direction confidence score is generated based on the sound source direction information, and it is defined as A. direction The speech recognition confidence score is generated based on the speech recognition information and defined as A. speech .

[0117] Step S14: Perform weighted fusion calculation based on the confidence of the sound source direction and the confidence of speech recognition to obtain audio information.

[0118] The formula for calculating audio information A is as follows: A = w4A direction +w5A speech

[0119] Among them, w4 and w5 both represent weights, which are used to balance the importance of different information in the audio information.

[0120] Step 204: Combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result;

[0121] In some embodiments, the process of combining visual and audio information to perform multimodal data fusion based on comprehensive confidence to obtain effective scene judgment results includes the following sub-steps S21 to S23:

[0122] Step S21: Perform multimodal data fusion based on visual and audio information to obtain multimodal fusion information;

[0123] The formula for calculating the multimodal fusion information F(V,A) is as follows: F(V,A)=αV+βA

[0124] For ease of calculation, V and A in the formula are normalized values, ranging from [0,1]. α and β represent weights, used to balance the importance of visual and audio information.

[0125] The weights involved in the embodiments of this invention can be adjusted according to actual applications. It is understood that this invention does not impose any limitations on this.

[0126] Step S22: Solve for the overall confidence level of the multimodal fusion information;

[0127] The overall confidence level C can be calculated using the following formula: C=σ(F(V,A))

[0128] Here, σ represents the activation function, typically the Sigmoid function, to map the output value to the range [0,1]. Assuming the input to the Sigmoid function is z, the Sigmoid function can be defined as: σ(z)=1 / (1+e^(z / z)) -z )

[0129] The overall confidence level C can also be expressed by the following formula: C=σ(F(V,A))=1 / (1+e -F(V,A) )

[0130] Step S23: When the overall confidence level is greater than the preset confidence threshold, the external ambient sound scene is determined to be a valid scene; when the overall confidence level is less than or equal to the preset confidence threshold, the external ambient sound scene is determined to be an invalid scene.

[0131] To further improve the accuracy of the judgment, a confidence threshold T is set. Only when the overall confidence level C exceeds the confidence threshold T can the external ambient sound scene be determined as a valid scene.

[0132] The specific criteria for determining valid scenarios are as follows:

[0133] For example, suppose the confidence score V for face detection is obtained by calculation. face =0.9, Confidence level V for direction judgment direction =0.85, lip movement detection confidence level V lip =0.8, Confidence level A of sound source direction direction =0.9, Speech recognition confidence level A speech =0.95.

[0134] The weights were set as w1 = 0.4, w2 = 0.3, w3 = 0.3, w4 = 0.5, w5 = 0.5, α = 0.6, and β = 0.4; the confidence threshold was set as T = 0.7.

[0135] First, calculate the visual information V: V = 0.4 × 0.9 + 0.3 × 0.85 + 0.3 × 0.8 = 0.855

[0136] Simultaneously calculate audio information A: A = 0.5 × 0.9 + 0.5 × 0.95 = 0.925

[0137] Next, the multimodal fusion information F(V,A) is calculated: F(V,A) = 0.6 × 0.855 + 0.4 × 0.925 = 0.883

[0138] Then calculate the overall confidence level C: C=σ(0.883)=1 / (1+e -0.883 )≈0.708

[0139] Finally, threshold comparison and judgment are performed:

[0140] Since C≈0.708>0.7, the current external ambient sound scene is determined to be a valid scene.

[0141] Thus, through the above calculation process, multimodal fusion and comprehensive confidence calculation can be achieved, thereby improving the accuracy and reliability of the intelligent cockpit ambient sound enhancement system.

[0142] Step 205: Based on the current driving environment, the current ambient sound type, and the effective scene judgment result, enhance the ambient sound outside the vehicle through a sound passthrough enhancement strategy.

[0143] In some embodiments, the process of enhancing the external ambient sound through a sound pass-through enhancement strategy based on the current driving environment, the current ambient sound type, and the effective scene judgment result can be as follows: dynamically adjusting the pass-through volume of the external ambient sound according to the current ambient sound type and its sound intensity; when the effective scene judgment result indicates that the external ambient sound scene is an effective scene, dynamically adjusting the pass-through volume of the external ambient sound in combination with the current driving environment, and / or enhancing the audio of the external ambient sound.

[0144] The sound passthrough enhancement strategy is mainly based on dynamic volume adjustment and volume enhancement settings.

[0145] From the perspective of effective scenarios, the sound pass-through enhancement strategy can primarily enhance ambient sounds outside the vehicle in effective scenarios (such as valid voices speaking towards the vehicle or alarm sounds) and reduce ambient sounds outside the vehicle in ineffective scenarios (such as background noise). For example, if someone is detected speaking towards the vehicle, and the microphone array determines that the sound source is from that direction, then this ambient sound scenario is considered an effective scenario, and the sound picked up by the microphone is enhanced. Similarly, if an alarm sound is detected, and video data analysis determines that there is an alarming vehicle, then this ambient sound scenario is considered an effective scenario, and the sound picked up by the microphone is enhanced.

[0146] The adjustment of ambient sound levels for the effective driving scenario will vary depending on the driving environment. For example, in an urban driving environment, the noise around the vehicle is relatively high, while in a rural driving environment, the noise around the vehicle is relatively low and quieter than in an urban environment. Therefore, the effectiveness of enhancing the intensity of ambient sound and reducing background noise will differ between the two environments. Furthermore, in an urban driving environment, if traffic control is detected, the intensity of the microphone can be increased to better capture the traffic control information. If necessary, background noise can also be reduced.

[0147] From the perspective of ambient sound itself, the sound pass-through enhancement strategy can dynamically adjust the pass-through volume based on the type and volume (intensity) of the ambient sound. This avoids the problem of poor experience for passengers inside the vehicle due to excessive volume of ambient sound outside the vehicle, and also avoids the situation where passengers inside the vehicle miss the sound due to insufficient volume of ambient sound outside the vehicle.

[0148] The dynamic volume adjustment feature can dynamically adjust the transmit volume based on the type and intensity of ambient sound. For example, the volume can be increased for emergency vehicle alarms and decreased for background noise.

[0149] Audio enhancement is primarily achieved through audio enhancement algorithms (such as noise suppression and echo cancellation) to improve the sound quality of the target environment. Specifically, deep learning models (such as deep neural networks (DNNs)) or traditional signal processing methods (such as spectral subtraction) can be used to suppress background noise, achieving the goal of noise suppression. Echo cancellation algorithms (such as acoustic echo cancellation (AEC)) can also be used to eliminate the influence of sound played from the in-vehicle speakers on the audio captured by the microphone, achieving echo cancellation and enhancing the ambient sound outside the vehicle.

[0150] In some embodiments, in conjunction with the foregoing description, the automotive ambient sound enhancement system may further include a user display interface. That is, a user display interface is provided in front of the driver's cabin inside the vehicle. For example, a schematic diagram of the user display interface is shown in Figure 3. Referring to Figure 3, the user display interface mainly includes an ambient sound type display area and sound intensity adjustment buttons.

[0151] The ambient sound type display area shows several adjustable ambient sound types, such as alarm sounds (corresponding to the ambient sounds emitted by police cars, ambulances, guide vehicles, etc.), command sounds (corresponding to the sounds of traffic control, reversing control, and other command vehicles), and external human voices (corresponding to the human voices around the vehicle, etc.).

[0152] The volume control button can be further divided into several levels, including normal, medium, and boost. Volume can be adjusted by tapping the button, or by pressing and holding the button and then sliding your finger up or down to move the volume slider.

[0153] In the specific implementation, when the system detects an ambient sound type selection operation for the ambient sound type display area, it determines the target ambient sound type from several adjustable ambient sound types; in response to the adjustment operation of the sound intensity adjustment button, it performs pass-through volume adjustment on the external ambient sound corresponding to the target ambient sound type, thereby enabling autonomous control of the user-oriented sound pass-through volume.

[0154] This invention provides an application method for an automotive ambient sound enhancement system based on multimodal fusion technology. By introducing multimodal fusion perception technology, combining audio and video information, and utilizing various deep learning algorithms, image data captured by a camera and audio data captured by a microphone are jointly processed to generate richer multimodal features. This enables comprehensive recognition of various ambient sounds, improves the quality of audio signals and the capture effect of effective scene sounds, and enhances the accuracy of sound source localization. Simultaneously, through a sound pass-through enhancement strategy, the audio processing algorithm is automatically adjusted according to environmental changes, suppressing background noise while enhancing the clarity of effective scene sounds. This allows for clear capture of effective scene sounds even in complex environments. Especially in high-speed driving or complex traffic environments, it can continuously provide high-quality audio capture and enhancement effects, improving driving safety and user experience.

[0155] For better illustration, please refer to Figure 4, which shows a schematic diagram of the overall process of a vehicle ambient sound enhancement method provided by an embodiment of the present invention. It should be noted that this embodiment only provides a brief description of the general process of vehicle ambient sound enhancement. The specific implementation process of each step can be understood by referring to the relevant content in the foregoing embodiments, and will not be elaborated here. It is understood that the present invention does not limit the scope of the invention.

[0156] Step 401: Collect external audio data and external video data;

[0157] Step 402: Input the external audio data and external video data into the pre-trained multimodal neural network for scene recognition to obtain the current ambient sound type and the current driving environment;

[0158] Step 403: Based on face detection and tracking and lip movement detection, combined with confidence-weighted fusion calculation, extract visual information from the external video data;

[0159] Step 404: Based on sound source localization and speech recognition, combined with confidence-weighted fusion calculation, extract audio information from the external audio data;

[0160] Step 405: Combine visual and audio information to perform multimodal data fusion based on comprehensive confidence to obtain effective scene judgment results;

[0161] Step 406: Based on the current driving environment, the current ambient sound type, and the effective scene judgment results, enhance the ambient sound outside the vehicle through the sound pass-through enhancement strategy;

[0162] Step 407: When an ambient sound type selection operation is detected for the user display interface, determine the target adjustable ambient sound type from several adjustable ambient sound types;

[0163] Step 408: In response to the adjustment operation of the sound intensity adjustment button, perform pass-through volume adjustment of the external ambient sound corresponding to the target ambient sound type.

[0164] Referring to Figure 5, a structural block diagram of an automotive ambient sound enhancement device provided in an embodiment of the present invention is shown, which may specifically include:

[0165] The data acquisition unit 501 is used to acquire external audio data and external video data.

[0166] The scene recognition unit 502 is used to perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment.

[0167] Information extraction unit 503 is used to extract visual information from the vehicle exterior video data and extract audio information from the vehicle exterior audio data;

[0168] The multimodal data fusion unit 504 is used to combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result;

[0169] The external ambient sound enhancement unit 505 is used to enhance the external ambient sound based on the current driving environment, the current ambient sound type, and the effective scene judgment result, through a sound pass-through enhancement strategy.

[0170] In one optional embodiment, the scene recognition unit 502 includes:

[0171] A data input unit is used to input the external audio data and the external video data into a pre-trained multimodal neural network; the multimodal neural network includes a driving environment classification model and an ambient sound classification model;

[0172] An ambient sound classification unit is used to extract audio features from the external audio data through the ambient sound classification model, and to classify the ambient sound based on the audio features to identify the current ambient sound type of the vehicle.

[0173] The driving environment classification unit is used to extract surrounding environment features from the external video data through the driving environment classification model, and to classify the driving environment based on the surrounding environment features to identify the current driving environment of the vehicle.

[0174] In one optional embodiment, the information extraction unit 503 includes:

[0175] The face detection unit is used to detect and locate all faces in the external video data through a face detection algorithm, and at the same time, it tracks the detected faces in real time based on a face tracking algorithm and generates face detection information.

[0176] The head pose estimation unit is used to estimate the head pose based on the face detection information to obtain orientation judgment information.

[0177] The lip movement detection unit is used to extract facial key point information from the face detection information, locate the lip region based on the facial key point information, extract the visual features of the lip region, and perform lip movement detection based on the visual features to obtain lip movement detection information.

[0178] A visually relevant confidence generation unit is used to generate face detection confidence based on the face detection information, generate direction judgment confidence based on the direction judgment information, and generate lip movement detection confidence based on the lip movement detection information.

[0179] The visual information generation unit is used to perform weighted fusion calculation based on the face detection confidence, the direction judgment confidence, and the lip movement detection confidence to obtain visual information.

[0180] In one optional embodiment, the information extraction unit 503 includes:

[0181] A sound source localization unit is used to locate the sound source in the external audio data and determine the direction information of the sound source.

[0182] A voice recognition unit is used to perform voice recognition on the external audio data to obtain voice recognition information;

[0183] An audio-related confidence generation unit is used to generate a sound source direction confidence score based on the sound source direction information and to generate a speech recognition confidence score based on the speech recognition information.

[0184] The audio information generation unit is used to perform weighted fusion calculation based on the confidence level of the sound source direction and the confidence level of the speech recognition to obtain audio information.

[0185] In one optional embodiment, the multimodal data fusion unit 504 includes:

[0186] A multimodal fusion information calculation unit is used to perform multimodal data fusion based on the visual information and the audio information to obtain multimodal fusion information;

[0187] The comprehensive confidence level calculation unit is used to calculate the comprehensive confidence level of the multimodal fusion information;

[0188] The effective scene determination unit is used to determine that the external ambient sound scene is an effective scene when the overall confidence level is greater than a preset confidence threshold.

[0189] The invalid scene determination unit is used to determine that the external ambient sound scene is invalid when the overall confidence level is less than or equal to a preset confidence threshold.

[0190] In one optional embodiment, the external ambient sound enhancement unit 505 is specifically used for:

[0191] The transmission volume of the external ambient sound is dynamically adjusted based on the current ambient sound type and its intensity.

[0192] When the result of the effective scene determination indicates that the external ambient sound scene is an effective scene, the transmission volume of the external ambient sound is dynamically adjusted in conjunction with the current driving environment, and / or the audio of the external ambient sound is enhanced.

[0193] In one optional embodiment, a user display interface is provided in front of the driver's cabin, the user display interface including an ambient sound type display area and a sound intensity adjustment button; the ambient sound type display area displays several adjustable ambient sound types; the vehicle ambient sound enhancement device further includes:

[0194] A target adjustable ambient sound type determination unit is used to determine a target adjustable ambient sound type from the plurality of adjustable ambient sound types when an ambient sound type selection operation for the ambient sound type display area is detected.

[0195] The transparent volume adjustment unit is used to adjust the external ambient sound volume corresponding to the target ambient sound type in response to the adjustment operation of the sound intensity adjustment button.

[0196] As the device embodiment is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment above.

[0197] This invention also provides an electronic device, which includes a processor and a memory:

[0198] The memory is used to store program code and transfer the program code to the processor;

[0199] The processor is used to execute the automotive ambient sound enhancement method of any embodiment of the present invention according to the instructions in the program code.

[0200] This invention also provides a computer-readable storage medium for storing program code for executing the automotive ambient sound enhancement method of any embodiment of this invention.

[0201] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0202] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0203] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0205] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A method of car ambient sound enhancement, characterized in that include: Collects external audio and video data from outside the vehicle; Scene recognition is performed based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment. Visual information is extracted from the external video data, and audio information is extracted from the external audio data; By combining the visual information and the audio information, multimodal data fusion based on comprehensive confidence is performed to obtain effective scene judgment results; Based on the current driving environment, the current ambient sound type, and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through a sound passthrough enhancement strategy. The method for enhancing ambient sound in automobiles according to claim 1 is characterized in that, The step of performing scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment includes: The external audio data and the external video data are input into a pre-trained multimodal neural network; the multimodal neural network includes a driving environment classification model and an ambient sound classification model. The ambient sound classification model extracts audio features from the external audio data and classifies the ambient sound based on the audio features to identify the current ambient sound type of the vehicle. The driving environment classification model extracts surrounding environment features from the external video data and classifies the driving environment based on these features to identify the vehicle's current driving environment. The method for enhancing ambient sound in automobiles according to claim 1 is characterized in that, The extraction of visual information from the external video data includes: The system detects and locates all faces in the external video data using a face detection algorithm, and simultaneously tracks the detected faces in real time using a face tracking algorithm, generating face detection information. Based on the face detection information, head pose estimation is performed to obtain orientation judgment information; Facial key point information is extracted from the face detection information. The lip region is located based on the facial key point information. The visual features of the lip region are extracted. Lip movement detection is performed based on the visual features to obtain lip movement detection information. A face detection confidence score is generated based on the face detection information, a direction judgment confidence score is generated based on the direction judgment information, and a lip movement detection confidence score is generated based on the lip movement detection information. Visual information is obtained by weighted fusion calculation based on the confidence scores of face detection, orientation determination, and lip movement detection. The method for enhancing automotive ambient sound according to claim 3 is characterized in that, The step of extracting audio information from the external audio data includes: The external audio data is used to locate the sound source and determine the direction information of the sound source; The external audio data is subjected to speech recognition to obtain speech recognition information; A sound source direction confidence score is generated based on the sound source direction information, and a speech recognition confidence score is generated based on the speech recognition information; Audio information is obtained by weighted fusion calculation based on the confidence level of the sound source direction and the confidence level of the speech recognition. The method for enhancing ambient sound in automobiles according to claim 4 is characterized in that, The step of combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result includes: Multimodal data fusion is performed based on the visual information and the audio information to obtain multimodal fusion information; Solve for the overall confidence level of the multimodal fusion information; When the overall confidence level is greater than a preset confidence threshold, the external ambient sound scene is determined to be a valid scene. When the overall confidence level is less than or equal to the preset confidence threshold, the external ambient sound scene is determined to be an invalid scene. The method for enhancing ambient sound in automobiles according to claim 1 is characterized in that, The method of enhancing external ambient sound based on the current driving environment, the current ambient sound type, and the effective scene determination result, through a sound pass-through enhancement strategy, includes: The transmission volume of the external ambient sound is dynamically adjusted based on the current ambient sound type and its intensity. When the result of the effective scene determination indicates that the external ambient sound scene is an effective scene, the transmission volume of the external ambient sound is dynamically adjusted in conjunction with the current driving environment, and / or the audio of the external ambient sound is enhanced. The method for enhancing automotive ambient sound according to any one of claims 1 to 6 is characterized in that, A user display interface is provided in front of the driver's cockpit, including an ambient sound type display area and a sound intensity adjustment button; the ambient sound type display area displays several adjustable ambient sound types; the vehicle ambient sound enhancement method further includes: When an ambient sound type selection operation is detected for the ambient sound type display area, a target adjustable ambient sound type is determined from the plurality of adjustable ambient sound types. In response to the adjustment operation of the sound intensity adjustment button, the transmission volume of the external ambient sound corresponding to the target ambient sound type is adjusted. A car ambient sound enhancement device, characterized in that, include: The data acquisition unit is used to collect external audio data and external video data. The scene recognition unit is used to perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment. An information extraction unit is used to extract visual information from the external video data and audio information from the external audio data; A multimodal data fusion unit is used to combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result; The external ambient sound enhancement unit is used to enhance the external ambient sound based on the current driving environment, the current ambient sound type, and the effective scene judgment result, through a sound pass-through enhancement strategy. An electronic device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the automotive ambient sound enhancement method according to any one of claims 1-7 according to the instructions in the program code. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for performing the automotive ambient sound enhancement method according to any one of claims 1-7.

Images