Method for simultaneous call interpretation using headset

Abstract

The present application relates to the field of call translation. Disclosed in the present application is a method for simultaneous call interpretation using a headset, which solves the problem of severe information distortion and loss of connotation often being caused during simultaneous call interpretation if the meaning of an original speech is merely rigidly conveyed in a target language, without fully capturing and expressing the connotation and context of an original text. By simulating the tone of a speaker, the present invention can significantly improve the quality of simultaneous call interpretation, enhance the expression of emotions, increase the accuracy of information, strengthen communication effects, and improve the user experience; in terms of technical implementation, speech synthesis and emotion analysis techniques can provide strong support, such that tone simulation is more natural and real; and speech data of different accents, different speakers and different background noises and speaking speeds is collected, such that a more robust and accurate speech recognition model can be trained, thereby adapting to diverse speech inputs, and improving the generalization capability of the model in different scenarios.

Description

A method for simultaneous interpretation using headphones

[0001] This application claims priority to Chinese Patent Application No. CN202411520224.5, filed on October 29, 2024, entitled "A Method for Simultaneous Interpretation Using Headphones", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of call translation, specifically to a method for simultaneous call translation using headphones. Background Technology

[0003] In today's globalized world, cross-border exchanges are becoming increasingly frequent. However, language differences often become a barrier to communication. Traditional translation methods, such as human and machine translation, while solving this problem to some extent, still suffer from inefficiency and high costs. With the development of artificial intelligence technology, especially the rapid progress in speech recognition and natural language processing, the demand for real-time simultaneous interpretation is increasing. In addition, the emergence of mixed reality technology has made it possible to provide a more intuitive and interactive translation experience. Therefore, how to achieve efficient, low-cost, multilingual, and multimodal real-time simultaneous interpretation has become an important research topic.

[0004] Application CN117979246A relates to the field of call translation, specifically to a method and system for one person to complete two-way call translation by wearing a wireless headset. Either party in the call wears a wireless headset, and the language spoken by the party wearing the wireless headset is translated into the language spoken by the other party in the call, and the translated speech is transmitted to the other party in the call. At the same time, the language spoken by the other party in the call is translated into the language spoken by the party wearing the wireless headset, and the resulting translated speech is played through the wireless headset.

[0005] In simultaneous interpreting, simply conveying the original audio in the target language without fully capturing and expressing its emotions and connotations often leads to serious distortion and a lack of depth. For native speakers of the target language, they receive only the surface-level text, failing to grasp the underlying cultural, emotional, and contextual information. This simplistic text translation can easily cause cognitive biases, resulting in comprehension that deviates significantly from, or even contradicts, the actual information conveyed. Therefore, simultaneous interpreting must prioritize a comprehensive understanding and precise expression of the original text to ensure accurate information delivery and effective communication.

[0006] Application content

[0007] To address the shortcomings of existing technologies, this application provides a method for simultaneous interpretation using headphones. This method solves the problem that in simultaneous interpretation, simply conveying the meaning of the original speech in the target language without fully capturing and expressing the emotions and connotations of the original text often leads to serious information distortion and a lack of meaning.

[0008] To achieve the above objectives, this application provides the following technical solution: a method for simultaneous interpretation using an earpiece, comprising the following steps:

[0009] S1. Retrieve speech samples of the target language online, including recordings and corresponding speech-text transcriptions, with the speech samples covering different accents, speakers, speech rates, and background noise, and establish a speech dataset; acquire user speech data and perform preprocessing.

[0010] S2. Establish a speech recognition model based on a lightweight convolutional neural network. By extracting local features from the target language and using the CTC algorithm to obtain user speech data, speech recognition is achieved, and a signal is generated and sent to the translation component; establish a user identity database.

[0011] S3. Through the translation component, analyze a type of signal to determine the target language, including text translation and sentiment translation, and generate text translation data and sentiment translation data respectively; identify the user, and make personalized adjustments to the text translation and sentiment translation based on the user's tags; generate translated voice signals through speech synthesis and send them to the target terminal.

[0012] S4. Optimize model operation, including collecting feedback data, in-depth exploration of sentiment translation, and conducting in-depth sentiment exploration on cases that performed well in the initial sentiment translation based on the user identity database.

[0013] S5. Personalized customization: By analyzing usage data, communication domains are divided according to translation results, and relevant users are matched. The commonly used vocabulary, phrases and expressions of users in that domain are automatically recorded, and this information is given priority in the text translation in step S3. At the same time, learning results are shared with other users through cloud services to form a continuously evolving translation community.

[0014] As a further solution of this application, the process of establishing the speech dataset in step S1 includes the following steps: searching for speech text of the target language online, the target language being determined according to market demand and application scenarios; classifying the speech text according to spoken language and establishing corresponding accent folders, and further classifying each accent folder according to the speaker to establish speaker folders.

[0015] The speaker folder contains recordings of the speaker speaking at different speeds under different background noise levels, along with corresponding transcripts, generating a speech dataset.

[0016] As a further aspect of this application, step S1, which involves acquiring user voice data and performing preprocessing, includes the following steps:

[0017] The user's voice signal is collected through the microphone of the headphones, and the collected voice signal is preprocessed, including noise reduction, gain control and spectrum analysis, and then converted into a digital signal.

[0018] As a further aspect of this application, step S2, establishing a speech recognition model, includes the following steps:

[0019] The system receives a speech dataset, uses Mel spectrograms to convert the speech data into spectrograms, extracts features of the target language, and uses them as input to the speech recognition model. At the same time, the system performs data augmentation on the speech data, including adding noise, speed adjustment, and pitch shifting. The augmented speech data is then segmented to generate training, validation, and test sets in a ratio of 7:1:2.

[0020] The model is trained using the training set data, and the model parameters are adjusted. The model parameters include: convolutional kernel weights, convolutional layer bias terms, weight matrix and fully connected layer bias terms, as well as the Adam optimizer and SGD optimizer.

[0021] Use validation set data to adjust model parameters to prevent overfitting;

[0022] Evaluate model performance and calculate accuracy using test set data;

[0023] The accuracy rate is calculated as follows:

[0024] Define a preset standard. If the accuracy is less than the preset standard, which is 90% in this case, the model is considered to have overfitting or underfitting problems.

[0025] Observe the loss curve, accuracy curve, and word error rate curve of the training set and validation set. If the loss and accuracy of the training set gradually decrease, and the loss and accuracy of the validation set decrease first and then increase, it is determined to be overfitting. If the loss and accuracy of the training set and validation set are both high and there is no obvious decreasing or increasing trend, it is determined to be underfitting.

[0026] To address overfitting, the diversity of training data can be increased. This can be achieved by adding noise, using speed and pitch shifting techniques to generate more training set samples and improve the model's generalization ability.

[0027] To address underfitting, increase model complexity by increasing the number of layers, neurons, and convolutional kernels to improve the model's learning ability.

[0028] The formula for calculating the convolution kernel weights is:

[0029] W new For the updated weights, W old The weights are set before the update, with an initial value of 0, and η is the learning rate, which takes a value of 0.001. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

[0030] The update formula for the convolutional layer bias term is:

[0031] b new For the updated convolutional layer bias term, b old The initial value of the convolutional layer bias term is 0; the formula for calculating the weight matrix is:

[0032] N new For the updated weights, N old The weights are set before the update, with an initial value of 0, and η is the learning rate, which is set to 0.001 here. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

[0033] The update formula for the bias term in the fully connected layer is:

[0034] a new For the updated convolutional layer bias term, a old This is the bias term of the convolutional layer before the update, with an initial value of 0;

[0035] The update formula for the Adam optimizer is:

[0036] m t and v t These are the exponentially weighted average of the gradient and the exponentially weighted average of the squared gradient, respectively. β1 and β2 are the decay rates, and ε is the smoothing term.

[0037] The formula for calculating the SGD optimizer is:

[0038] D new For the updated weights, D old The weights are set before the update, with an initial value of 0, and η is the learning rate, which takes a value of 0.001. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

[0039] As a further aspect of this application, step S2, which involves establishing a user identity database and processing voice data, includes the following steps:

[0040] The speech recognition model extracts acoustic features from user speech data, including: fundamental frequency, formants, spectrum, power spectral density, cepstral, Mel-frequency cepstral coefficients, linearly predicted cepstral coefficients, pitch, intensity, duration, zero-crossing rate, and short-time energy. Based on the extracted features, they are compared with user registration information in the identity database. If the comparison is successful, the user information, including their personal usage domain and frequently used vocabulary, is retrieved for speech data recognition and processing. If the comparison fails, the speech data is directly processed, and a new account is registered in the identity database, storing the corresponding feature information.

[0041] The voice data recognition and processing operation involves converting the format according to the language category, generating a type of information, and sending it to the translation component.

[0042] As a further aspect of this application, in step S3:

[0043] The system performs word segmentation on the text, breaking it down into words or phrases; removes punctuation and stop words; converts the text to a uniform format, such as lowercase; identifies fixed phrases in the text and converts them into corresponding phrases in the target language; identifies slang and informal expressions in the text and converts them into corresponding slang in the target language; analyzes the internal components of sentences, such as subject, predicate, and object; analyzes the logical relationships between sentences, such as contrast, cause and effect, progression, and parallelism; and generates translated text content in the target language.

[0044] By analyzing the internal components and external logical relationships of sentences, we can ensure the accuracy and fluency of the translated text while maintaining its grammatical correctness.

[0045] As a further aspect of this application,:

[0046] Based on the speech text, acoustic features of the speech, such as pitch, volume, and speech rate, are extracted. Through the emotion model, the emotions in the speech are identified, and first-level emotion information is generated and labeled. The label content is the intensity of the identified emotion, such as strong, moderate, and mild. Through speech synthesis, speech information with specific emotions is generated, thus completing the mapping of the emotion connotation of the original speech.

[0047] As a further aspect of this application, the establishment of the emotion model includes the following steps:

[0048] Feature extraction is performed on speech samples, including acoustic features of speech such as pitch, volume, and speech rate, and text features such as sentiment words and sentiment polarity; sentiment annotation is performed manually on speech and text data to label sentiment categories, a sentiment classifier is trained to identify sentiment categories in speech and text, and a sentiment generator is trained to generate speech and text with specific sentiments.

[0049] The model is evaluated by calculating the accuracy, recall, and F1 score of the large emotion model. If any performance metric falls below the preset value, the speech sample is optimized by performing text data augmentation, including synonym replacement, random insertion, random deletion, and random swapping, and speech data augmentation, including noise addition, speed adjustment, and pitch shifting. The large speech model building process is repeated for the optimized speech sample until all performance metrics are greater than or equal to the preset value.

[0050] As a further step in this application, feedback data is collected, and in-depth exploration of sentiment translation is undertaken, including the following steps:

[0051] In the user's voice data, the recordings and texts that are not repeated are marked as feedback data; using sentiment analysis tools, such as NLP models, the feedback data is further subdivided into sentiment categories, such as subdividing "happiness" into "excitement", "satisfaction" or "pleasure", etc., generating secondary sentiment information, and adding annotations, the annotation content being the intensity of the identified emotion, such as strong, moderate and mild.

[0052] This allows secondary emotional information to cover primary emotional information in the emotional translation steps within S3.

[0053] This application provides more personalized translation services by matching relevant users, prioritizing the user's commonly used vocabulary, phrases, and expressions in their communication domain, significantly improving translation accuracy, user experience, and system robustness; by sharing learning outcomes with other users through cloud services, it forms a continuously evolving translation community, promoting data sharing and collaboration, and has significant commercial value and application prospects. Attached Figure Description

[0054] Figure 1 is a schematic diagram of the method flow of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] Example 1

[0057] Please refer to Figure 1. This application provides a method for simultaneous interpretation using an earpiece, including the following steps:

[0058] S1. Retrieve speech samples of the target language online, including recordings and corresponding speech-text transcriptions, with the speech samples covering different accents, speakers, speech rates, and background noise, and establish a speech dataset; acquire user speech data and perform preprocessing.

[0059] The process of creating a speech dataset includes the following steps:

[0060] The system retrieves audio texts in the target language online, with the target language determined based on market demand and application scenarios. The audio texts are categorized according to spoken language, and corresponding accent folders are created. Within each accent folder, the audio texts are further categorized according to the speaker, creating speaker folders.

[0061] The speaker folder contains recordings of the speaker speaking at different speeds under different background noise levels, along with corresponding transcripts, generating a speech dataset.

[0062] By collecting speech data with different accents, speakers, background noise, and speech rates, a more robust and accurate speech recognition model can be trained to adapt to diverse speech inputs. Diverse speech datasets help the model learn a wider range of speech features, thereby improving its generalization ability in different scenarios and reducing dependence on specific speakers or environments.

[0063] Acquire user voice data and perform preprocessing, including the following steps:

[0064] The user's voice signal is collected through the microphone of the headphones, and the collected voice signal is preprocessed, including noise reduction, gain control and spectrum analysis, and then converted into a digital signal.

[0065] S2. Establish a speech recognition model based on a lightweight convolutional neural network. By extracting local features from the target language and using the CTC algorithm to obtain user speech data, speech recognition is achieved, and a signal is generated and sent to the translation component; establish a user identity database.

[0066] A: Establishing a speech recognition model includes the following steps:

[0067] The system receives a speech dataset, uses Mel spectrograms to convert the speech data into spectrograms, extracts features of the target language, and uses them as input to the speech recognition model. At the same time, the system performs data augmentation on the speech data, including adding noise, speed adjustment, and pitch shifting. The augmented speech data is then segmented to generate training, validation, and test sets in a ratio of 7:1:2.

[0068] The model is trained using the training set data, and the model parameters are adjusted. The model parameters include: convolutional kernel weights, convolutional layer bias terms, weight matrix and fully connected layer bias terms, as well as the Adam optimizer and SGD optimizer.

[0069] Use validation set data to adjust model parameters to prevent overfitting;

[0070] Evaluate model performance and calculate accuracy using test set data;

[0071] The accuracy rate is calculated as follows:

[0072] Define a preset standard. If the accuracy is less than the preset standard, which is 90% in this case, the model is considered to have overfitting or underfitting problems.

[0073] Observe the loss curve, accuracy curve, and word error rate curve of the training set and validation set. If the loss and accuracy of the training set gradually decrease, and the loss and accuracy of the validation set decrease first and then increase, it is determined to be overfitting. If the loss and accuracy of the training set and validation set are both high and there is no obvious decreasing or increasing trend, it is determined to be underfitting.

[0074] To address overfitting, the diversity of training data can be increased. This can be achieved by adding noise, using speed and pitch shifting techniques to generate more training set samples and improve the model's generalization ability.

[0075] To address underfitting, increase model complexity by increasing the number of layers, neurons, and convolutional kernels to improve the model's learning ability.

[0076] B: Using a speech recognition model, input user speech data and extract acoustic features, including: fundamental frequency, formants, spectrum, power spectral density, cepstral, Mel-frequency cepstral coefficients, linearly predicted cepstral coefficients, pitch, intensity, duration, zero-crossing rate, and short-time energy. Based on the extracted features, compare them with user registration information in the identity database. If the comparison is successful, retrieve the user information, including personal usage domain and commonly used vocabulary, and process the speech data for recognition. If the comparison fails, directly process the speech data for recognition and register a new account in the identity database, storing the corresponding feature information.

[0077] By selecting terms and expressions that are more suitable for the specific domain based on the user's individual usage area, more personalized speech recognition services can be provided, recognition accuracy can be improved, user data can be continuously accumulated, the model can continuously learn and optimize, improve recognition performance, and enhance the user experience.

[0078] The voice data recognition and processing operation involves converting the format according to the language category, generating a type of information, and sending it to the translation component.

[0079] For example, for users in the medical field, a glossary containing medical terminology can be generated to improve the professionalism of the translation;

[0080] The formula for calculating the convolution kernel weights is:

[0081] W new For the updated weights, W old The weights are set before the update, with an initial value of 0, and η is the learning rate, which takes a value of 0.001. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

[0082] Example: Weight update:

[0083] Assume W old It is 0.01. If W is 0.5, then new =0.01 - 0.001 * 0.5 = 0.0095;

[0084] The update formula for the convolutional layer bias term is:

[0085] b new For the updated convolutional layer bias term, b old This is the bias term of the convolutional layer before the update, with an initial value of 0;

[0086] Example: Bias term update: Assume b old =0, If it is 0.3, then b new =0 - 0.001 * 0.3 = -0.0003;

[0087] The formula for calculating the weight matrix is:

[0088] N new For the updated weights, N old The weights are set before the update, with an initial value of 0, and η is the learning rate, which is set to 0.001 here. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

[0089] Example: Weight matrix update: Assume N old It is 0.01. If N is 0.4, then Nnew =0.01 - 0.001 * 0.4 = 0.006;

[0090] The update formula for the bias term in the fully connected layer is:

[0091] a new For the updated convolutional layer bias term, a old This is the initial value of the convolutional layer bias term before the update, which is 0. Example: Update the bias term of the fully connected layer: Assume a old It is 0.03. If it is 0.2, then a new =0.03 - 0.001 * 0.2 = 0.0298;

[0092] The update formula for the Adam optimizer is:

[0093] m t and v t These are the exponentially weighted average of the gradient and the exponentially weighted average of the squared gradient, respectively. β1 and β2 are the decay rates, and ε is the smoothing term.

[0094] Example: β1 is 0.9, β2 is 0.999, ε is 1e-8, m t-1 and v t-1 The values ​​are all 0. If it is 0.3, then m t =0 + (1 - 0.9) * 0.3 = 0.03, v t =0 + (1 - 0.999) * 0.3 = 0.0003; Substituting further, we get: but

[0095] The formula for calculating the SGD optimizer is:

[0096] D new For the updated weights, D old The weights are set before the update, with an initial value of 0, and η is the learning rate, which takes a value of 0.001. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

[0097] Example: D old The value is 0, and η is 0.001. If it is -0.5, then D new =0 - 0.001 * (-0.5) = 0.0005;

[0098] S3. Through the translation component, analyze a type of signal to determine the target language, including text translation and sentiment translation, and generate text translation data and sentiment translation data respectively; identify the user, and make personalized adjustments to the text translation and sentiment translation based on the user's tags; generate translated voice signals through speech synthesis and send them to the target terminal.

[0099] Text translation refers to the textualization of the original speech data, including phrase connections and slang connections, and includes the following steps:

[0100] The system performs word segmentation on the text, breaking it down into words or phrases, removing punctuation and stop words, and converting the text to a uniform format, such as lowercase. It identifies fixed phrases in the text and converts them into corresponding phrases in the target language. It also identifies slang and informal expressions in the text and converts them into corresponding slang in the target language. Furthermore, it analyzes the internal components of sentences, such as subject, predicate, and object, and the logical relationships between sentences, such as contrast, cause and effect, progression, and parallelism. Finally, it generates translated text in the target language. By analyzing the internal components and external logical relationships of sentences, the system ensures both grammatical correctness and the accuracy and fluency of the translated text.

[0101] Emotional translation refers to the process of extracting features from the original speech data, simulating the user's tone, performing natural language processing, and building an emotional model to simulate the connotation, rhythm, intonation, and tone of the original speech. Emotional translation includes the following steps:

[0102] Based on the speech text, the acoustic features of the speech, such as pitch, volume and speech rate, are extracted. Through the emotion model, the emotion in the speech is identified, and first-level emotion information is generated and labeled. The label content is the intensity of the identified emotion, such as strong, medium and mild. Through speech synthesis, speech information with specific emotions is generated to complete the mapping of the emotion connotation of the original speech.

[0103] The establishment of an emotional model includes the following steps:

[0104] Feature extraction is performed on speech samples, including acoustic features of speech such as pitch, volume, and speech rate, and text features such as sentiment words and sentiment polarity; sentiment annotation is performed manually on speech and text data to label sentiment categories, a sentiment classifier is trained to identify sentiment categories in speech and text, and a sentiment generator is trained to generate speech and text with specific sentiments.

[0105] The model is evaluated for performance by calculating the accuracy, recall, and F1 score of the large emotion model. If any performance indicator is lower than the preset value, the speech sample is optimized by performing text data augmentation, including synonym replacement, random insertion, random deletion, and random swapping, and speech data augmentation, including noise addition, speed adjustment, and pitch shifting. The large speech model building operation is repeated for the optimized speech sample until all performance indicators are greater than or equal to the preset value.

[0106] By simulating the speaker's tone, the quality of simultaneous interpreting can be significantly improved, enhancing emotional expression, increasing information accuracy, improving communication effectiveness, and enhancing user experience. In terms of technical implementation, speech synthesis and emotion analysis technologies can provide strong support, making tone simulation more natural and realistic.

[0107] Example 2

[0108] S4. Optimize model operation, including collecting feedback data, in-depth exploration of sentiment translation, and conducting in-depth sentiment exploration on cases that performed well in the initial sentiment translation based on the user identity database.

[0109] In the user's voice data, unique recordings and texts are marked as feedback data. Sentiment analysis tools, such as NLP models, are used to further subdivide the feedback data into sentiment categories, for example, "happiness" is subdivided into "excitement," "satisfaction," or "pleasure," generating secondary sentiment information and adding annotations to indicate the intensity of the identified emotion, such as strong, moderate, and mild. The secondary sentiment information is then made to cover the primary sentiment information in the sentiment translation step within S3, thereby optimizing the sentiment translation step. This data is used to train the model and improve its sentiment translation capabilities.

[0110] By segmenting sentiment, we can more accurately capture and express the emotions in the original text, reducing the ambiguity and inaccuracy of sentiment translation; by using standard sentiment intensity, we can accurately convey the intensity of emotions, making the translation more delicate and realistic; at the same time, by using feedback data to train the model, we can continuously optimize the model's sentiment translation ability, making it more adaptable to diverse emotional expressions.

[0111] S5. Personalized customization: By analyzing usage data, communication domains are divided according to translation results, and relevant users are matched. The system automatically records the commonly used vocabulary, phrases and expressions of users in each domain and prioritizes this information in future translations. At the same time, learning outcomes are shared with other users through cloud services, forming a continuously evolving translation community.

[0112] By matching relevant users, more personalized translation services are provided, prioritizing the user's commonly used vocabulary, phrases, and expressions in their communication domain, significantly improving translation accuracy, user experience, and system robustness. Furthermore, by sharing learning outcomes with other users through cloud services, a continuously evolving translation community is formed, promoting data sharing and collaboration, and possessing significant commercial value and application prospects.

[0113] Example 3

[0114] In its specific implementation, this embodiment includes all the implementation processes of the two sets of embodiments described above.

[0115] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0116] The above embodiments are only used to illustrate the technical methods of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of this application without departing from the spirit and scope of the technical methods of this application.

Claims

1. A method for simultaneous interpretation using an earpiece, characterized in that, Includes the following steps: S1. Retrieve speech samples of the target language online, including recordings and corresponding speech-text transcriptions, with the speech samples covering different accents, speakers, speech rates, and background noise, and establish a speech dataset; acquire user speech data and perform preprocessing. S2. Establish a speech recognition model based on a lightweight convolutional neural network. By extracting local features from the target language and using the CTC algorithm to obtain user speech data, speech recognition is achieved, and a signal is generated and sent to the translation component; establish a user identity database. S3. Analyze a type of signal using the translation component to determine the target language for translation, including text translation and sentiment translation, and generate text translation data and sentiment translation data respectively. Once the user is identified, personalized adjustments are made to the text translation and emotion translation based on the user's tags. The translated voice signal is generated through speech synthesis and sent to the target terminal. S4. Optimize model operation, including collecting feedback data, in-depth exploration of sentiment translation, and conducting in-depth sentiment exploration on cases that performed well in the initial sentiment translation based on the user identity database. S5. Personalized customization: By analyzing usage data, communication domains are divided according to translation results, and relevant users are matched. The commonly used vocabulary, phrases and expressions of users in that domain are automatically recorded, and this information is given priority in the text translation in step S3. At the same time, learning results are shared with other users through cloud services to form a continuously evolving translation community.

2. The method for simultaneous interpretation using an earpiece according to claim 1, characterized in that, The process of establishing the speech dataset in step S1 includes the following steps: The system retrieves audio texts in the target language online, with the target language determined based on market demand and application scenarios. The audio texts are categorized according to spoken language, and corresponding accent folders are created. Within each accent folder, the audio texts are further categorized according to the speaker, creating speaker folders. The speaker folder contains recordings of the speaker speaking at different speeds under different background noise levels, along with corresponding transcripts, generating a speech dataset.

3. The method for simultaneous interpretation using an earpiece according to claim 1, characterized in that, In step S1, user voice data is acquired and preprocessed, including the following steps: The user's voice signal is collected through the microphone of the headphones, and the collected voice signal is preprocessed, including noise reduction, gain control and spectrum analysis, and then converted into a digital signal.

4. The method for simultaneous interpretation using an earpiece according to claim 1, characterized in that, In step S2, the speech recognition model is established, including the following steps: The system receives a speech dataset, uses Mel spectrograms to convert the speech data into spectrograms, extracts features of the target language, and uses them as input to the speech recognition model. At the same time, the system performs data augmentation on the speech data, including adding noise, speed adjustment, and pitch shifting. The augmented speech data is then segmented to generate training, validation, and test sets in a ratio of 7:1:

2. The model is trained using the training set data, and the model parameters are adjusted. The model parameters include: convolutional kernel weights, convolutional layer bias terms, weight matrix and fully connected layer bias terms, as well as the Adam optimizer and SGD optimizer. Use validation set data to adjust model parameters to prevent overfitting; Evaluate model performance and calculate accuracy using test set data; The accuracy rate is calculated as follows: Define a preset standard. If the accuracy is less than the preset standard, which is 90% in this case, the model is considered to have overfitting or underfitting problems. Observe the loss curve, accuracy curve, and word error rate curve of the training set and validation set. If the loss and accuracy of the training set gradually decrease, and the loss and accuracy of the validation set decrease first and then increase, it is determined to be overfitting. If the loss and accuracy of the training set and validation set are both high and there is no obvious decreasing or increasing trend, it is determined to be underfitting. To address overfitting, the diversity of training data can be increased. This can be achieved by adding noise, using speed and pitch shifting techniques to generate more training set samples and improve the model's generalization ability. To address underfitting, increase model complexity by increasing the number of layers, neurons, and convolutional kernels to improve the model's learning ability. The formula for calculating the convolution kernel weights is: W new For the updated weights, W old The weights are set before the update, with an initial value of 0, and η is the learning rate, which takes a value of 0.

001. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1]. The update formula for the convolutional layer bias term is: b new For the updated convolutional layer bias term, b old This is the bias term of the convolutional layer before the update, with an initial value of 0; The formula for calculating the weight matrix is: N new For the updated weights, N old The weights are set before the update, with an initial value of 0, and η is the learning rate, which is set to 0.001 here. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1]. The update formula for the bias term in the fully connected layer is: a new For the updated convolutional layer bias term, a old The initial value of the convolutional layer bias term before the update is 0; the update formula for the Adam optimizer is: m t and v t These are the exponentially weighted average of the gradient and the exponentially weighted average of the squared gradient, respectively. β1 and β2 are the decay rates, and ε is the smoothing term. The calculation formula for the SGD optimizer is: D new For the updated weights, D old The weights are set before the update, with an initial value of 0, and η is the learning rate, which takes a value of 0.

001. The gradient of the loss function with respect to the weights is calculated using the backpropagation algorithm and has a value range of [-1, 1].

5. A method for simultaneous interpretation using an earpiece according to claim 1, characterized in that, In step S2, the establishment of the user identity database and the recognition and processing of voice data include the following steps: The speech recognition model extracts acoustic features from user speech data, including: fundamental frequency, formants, spectrum, power spectral density, cepstral, Mel-frequency cepstral coefficients, linearly predicted cepstral coefficients, pitch, intensity, duration, zero-crossing rate, and short-time energy. Based on the extracted features, they are compared with user registration information in the identity database. If the comparison is successful, the user information, including their personal usage domain and frequently used vocabulary, is retrieved for speech data recognition and processing. If the comparison fails, the speech data is directly processed, and a new account is registered in the identity database, storing the corresponding feature information. The voice data recognition and processing operation involves converting the format according to the language category, generating a type of information, and sending it to the translation component.

6. The method for simultaneous interpretation using an earpiece according to claim 1, characterized in that, In step S3, the text translation includes the following steps: Perform word segmentation on the text, breaking it down into words or phrases, removing punctuation and stop words, and converting the text to a uniform format; Identify fixed phrases in the text and convert them into corresponding phrases in the target language; identify slang and informal expressions in the text and convert them into corresponding slang in the target language; It analyzes the internal components of sentences, analyzes the logical relationships between sentences, and generates text translations in the target language.

7. A method for simultaneous interpretation using an earpiece according to claim 6, characterized in that, In step S3, the emotion translation includes the following steps: Based on the speech text, the acoustic features of the speech are extracted, and the emotions in the speech are identified through the emotion model. First-level emotion information is generated and labeled with the intensity of the identified emotion. Through speech synthesis, speech information with specific emotions is generated, thus completing the mapping of the emotion connotation of the original speech.

8. A method for simultaneous interpretation using an earpiece according to claim 7, characterized in that, The establishment of the emotion model includes the following steps: Feature extraction is performed on speech samples, including acoustic features of speech such as pitch, volume, and speech rate, and text features such as sentiment words and sentiment polarity; sentiment annotation is performed manually on speech and text data to label sentiment categories, a sentiment classifier is trained to identify sentiment categories in speech and text, and a sentiment generator is trained to generate speech and text with specific sentiments. The model is evaluated by calculating the accuracy, recall, and F1 score of the large emotion model. If any performance metric falls below the preset value, the speech sample is optimized by performing text data augmentation, including synonym replacement, random insertion, random deletion, and random swapping, and speech data augmentation, including noise addition, speed adjustment, and pitch shifting. The large speech model building process is repeated for the optimized speech sample until all performance metrics are greater than or equal to the preset value.

9. A method for simultaneous interpretation using an earpiece according to claim 1, characterized in that, In step S4, feedback data is collected, and in-depth exploration of sentiment translation is conducted, including the following steps: In the user's voice data, the recordings and texts that are not repeated are marked and recorded as feedback data; using sentiment analysis tools, the feedback data are further subdivided into sentiment categories to generate secondary sentiment information, and annotations are added, with the annotation content being the intensity of the identified sentiment; This allows secondary emotional information to cover primary emotional information in the emotional translation steps within S3.

Images