745 results about "Spoken language" patented technology

An artificial intelligence system that conducts dialogs with one or more participants to achieve an educational outcome, such as teaching material with a Socratic question-and-answer method, reviewing material to reinforce ideas or to prepare a participant for future tests or events, or evaluating a participant's knowledge or communication skills. The system may use an artificial intelligence engine that can understand and generate natural language, such as a large-language model like ChatGPT. Text-to-speech and speech-to-text converters may be coupled to the AI engine to enable the system to communicate directly with participants using spoken language. The system may provide an interface for entry of contextual information for a dialog, such as background information on topics for discussion. It may provide an interface for moderation of a dialog, for example by a teacher, lawyer, thesis advisor, etc. System may evaluate the performance of the participants at the end of a dialog.

The present disclosure relates to systems and methods for enhancing the interaction between users and automated agents, such as digital assistants, by employing Large Language Models (LLMs) to infer the intent of spoken language. The invention involves continuously monitoring ambient audio, converting speech to text, and utilizing LLMs to determine whether spoken language is intended for the automated agent. A structured prompt, including the converted text and specific instructions, is sent to the LLM, which is fine-tuned to process domain-specific prompts. The LLM provides a structured output in a standardized format, indicating the user's intent. The system may involve multiple prompts to perform separate tasks, such as identifying intent and generating additional context-specific data. This approach facilitates a more natural and intuitive user experience by eliminating the need for wake words and allowing seamless conversational interaction with virtual assistants across various platforms and devices.

A method includes receiving a natural language query from a user that solicits a response from an assistant large language model (LLM), receiving a prompt composition including an instruction parameter that specifies a task for the assistant LLM to respond to user queries concisely, structuring a conciseness prompt by concatenating the prompt composition to the natural language query, and processing, using the assistant LLM, the conciseness prompt to generate a concise response to the natural language query. The method also includes providing, for output from a user device, the concise response to the natural language query.

Disclosed herein are methods and systems for a training a model for real-time patient diagnosis. A system may include a computer configured to receive audio data and video data of a clinical encounter, the audio data comprising spoken words by an entity and the video data depicting the entity; retrieve clinical data regarding the entity; execute a model using the words of the audio data and the retrieved clinical data regarding the entity as input, the execution causing the model to output a plurality of clinical diagnoses for the entity; concurrently render the corresponding video data and audio data and the plurality of clinical diagnoses via a computing device associated with a user; and store an indication of a selected clinical diagnosis of the plurality of clinical diagnoses responsive to receiving a selection of the clinical diagnosis at the computing device.

As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

A computer-implemented method, in accordance with one embodiment, includes detecting cultural context and accents of speakers portrayed in a video stream and / or an audience of the video stream. Accent tags are selected for the speakers according to the cultural context and accents of the speakers and / or the audience of the video stream. A textual representation of spoken words of the speakers is translated from a source language to a target language. The accent tags are applied to the textual representation of the spoken words in the target language according to the speakers corresponding to the textual representation of the spoken words in the target language. Speech lip movements of the speakers portrayed in the video stream are modified to match the target language and the locale accent tags. A translated video stream having the speakers appearing to speak in the target language with the modified lip movements is output.

An artificial intelligence system that conducts an oral question-and-answer session with a single respondent to achieve an educational outcome, such as reviewing material to reinforce ideas or to prepare the respondent for future tests or events, or evaluating the respondent's knowledge or communication skills. The system may use an artificial intelligence engine that can understand and generate natural language, such as a large-language model like ChatGPT™. Text-to-speech and speech-to-text converters may be coupled to the AI engine to enable the system to communicate directly with the respondent using spoken language. The system may provide an interface for entry of session information such as background information on topics and the type of questioning that should be used during the session. The respondent may be given multiple attempts to answer a question and to select the best response, and the system may provide an evaluation and feedback after each response.

According to one embodiment, a method, computer system, and computer program product for detecting, interpreting, and translating sign language is provided. The present invention may include responding to a detection of an active web conference, by analyzing a user's profile and profiles of an audience of the web conference; capturing gestures and spoken language of a presenter using one or more Internet of Things devices; processing the captured gestures and the spoken language of the presenter to interpret the gestures and the spoken language of the presenter; translating the gestures and the spoken language of the presenter into sign language; generating a digital avatar; and displaying the digital avatar along with dynamic sign language movements using the digital avatar.

A speech module is joint trained with a knowledge module by transforming a first knowledge graph into an acoustic knowledge graph. The knowledge module is trained on the acoustic knowledge graph. Then, the knowledge module is integrated with the speech module to generate an integrated knowledge-speech module. In some instances, the speech module included in the integrated knowledge-speech module is aligned with a language module to generate an optimized speech model configured to leverage acoustic information and acoustic-based knowledge information, along with language information.

An integrated development environment (IDE) leverages a generative AI model to generate industrial control code in accordance with specified functional requirements, which can be provided to the industrial IDE system as intuitive natural language spoken or written text. The industrial IDE can also analyze written code in response to natural language prompts submitted against the code, generate answers to user-submitted questions about the code, and offer recommendations for improving the code in response to specific questions or requests submitted by the user.

A method includes obtaining labeled training data including a plurality of spoken terms spoken during a conversation. For each respective spoken term, the method includes generating a corresponding sequence of intermediate audio encodings from a corresponding sequence of acoustic frames, generating a corresponding sequence of final audio encodings from the corresponding sequence of intermediate audio encodings, generating a corresponding speech recognition result, and generating a respective speaker token representing a predicted identity of a speaker for each corresponding speech recognition result. The method also includes training the joint speech recognition and speaker diarization model jointly based on a first loss derived from the generated speech recognition results and the corresponding transcriptions and a second loss derived from the generated speaker tokens and the corresponding speaker labels.

The invention belongs to the technical field of intelligent voice processing, and particularly relates to a spoken language pronunciation training and correcting system based on intelligent equipment, which acquires a user rhythm feature set including pitch change rate, accent intensity, pause duration and intonation contour by acquiring spoken language audio data sent by a user for a target text, and corrects the spoken language pronunciation training and correcting system. The method comprises the following steps: acquiring spoken language audio data, converting the spoken language audio data into a phoneme sequence aligned with target text time, generating a rhythm deviation degree report according to comparison evaluation of a user rhythm feature set and the phoneme sequence, an execution intonation mode, accent distribution, speech stream sound change and speech speed rhythm, and determining the rhythm deviation degree according to a rhythm problem type in the report in combination with user historical learning data. And forming and outputting a correction scheme including a text prompt, a targeted minimum contrast training unit and a listen-and-read simulation task, solving the problem of weak capability of correcting hyper-phoneme in a second language spoken language of an adult, and improving the authentic and fluency of the spoken language, thereby realizing efficient personalized learning.

Techniques for using multiple machine learning (ML) models, with varying compute costs, for ASR processing is described. The system may include an arbitrator component configured to determine which ML model is to be used to process an audio frame from a sequence of audio frames representing a spoken natural language input. The arbitrator component may switch between the ML models, on a frame-by-frame basis, to reduce an overall compute cost for the entire spoken natural language input. The outputs of the different ML models may be combined to determine the final output for the entire spoken natural language input.

The invention discloses a spoken language practice scene-oriented education large model evaluation method, which comprises the following steps of: obtaining voice data and a corresponding standard text, interacting with a test model through the voice data, and obtaining a recognition text, audio data and an answer text provided by the test model; the standard text and the recognition text are compared and evaluated, and the voice recognition accuracy is obtained; carrying out pronunciation accuracy, fluency and rhythm evaluation on the audio data based on multi-modal feature fusion to obtain an audio score; performing grammar accuracy evaluation on the answer text based on a dynamic punishment mechanism and the constructed prompt large model, and performing question degree evaluation on the answer text based on the constructed prompt large model to obtain an answer text score; and comprehensively summarizing the voice recognition accuracy, the audio score and the answer text score to obtain a final evaluation result.

Disclosed are apparatuses, systems, and techniques that leverage one or more language models (LMs)—such as large language models (LLMs—for efficient multi-speaker speech recognition. The techniques include processing, using a speaker diarization model, an audio feature to generate a first association of the audio feature with one or more prospective speakers, the audio feature being representative of one or more spoken words. The techniques further include providing, to an LM, a first prompt requesting the LM to identify a second association of the one or more spoken words with the one or more prospective speakers and receiving, from the LM, a first response identifying the second association of the one or more spoken words with the one or more prospective speakers. The techniques further include determining, using the first association and the second association, one or more speakers that produced the one or more spoken words.

The invention relates to the technical field of speech recognition and artificial intelligence, in particular to an automatic evaluation and correction system for spoken Chinese pronunciation based on AI speech recognition, which comprises a speech input module, a speech preprocessing module, a pronunciation analysis module, a pronunciation scoring module and a pronunciation correction module. Wherein the voice input module is used for receiving a Chinese spoken pronunciation signal of a user; the voice preprocessing module is used for preprocessing the voice data; the pronunciation analysis module is used for analyzing pronunciation of the user in real time by using an AI voice recognition technology; the pronunciation scoring module is used for calculating a score value of pronunciation of the user; and the pronunciation correction module is used for providing a targeted pronunciation correction scheme for the user. According to the invention, through the multi-dimensional pronunciation evaluation and personalized correction scheme based on the AI speech recognition technology, the pronunciation deviation of the user can be accurately recognized, real-time feedback can be provided, and the pronunciation correction accuracy and efficiency can be effectively improved.

An AI-based personality and behavior analytic and assessment system includes an input interface configured for receiving input data containing audio speech, an automatic speech recognition module configured to receive the input data, recognize the spoken words in the input data, and generate an output representing the spoken words attributed to the individual, the output including speaking moments and audio slices, a machine learning text-based feature generation pipeline configured to receive the speaking moments and generate a numerical text-based feature set, a machine learning audio-based feature generation pipeline configured to receive the audio slices and generate a numerical audio-based feature set, a machine learning inference processor configured to receive at least one of the numerical text-based feature set and the numerical audio-based feature set, develop inferences from the feature sets, and generate scores representing the probabilities for a number of personality and behavioral traits of the individual.

The invention provides a multi-modal spoken language pronunciation evaluation method based on a large language model, and belongs to the technical field of spoken language evaluation, and the method comprises the steps: obtaining the audio data of a spoken language tester; performing feature coding processing on the audio data through an acoustic encoder to obtain audio representation information, and mapping the obtained audio representation information into a public space through a modal adaptation layer; setting an audio evaluation instruction, separating the audio evaluation instruction into a plurality of phrases through a word segmentation device, converting the phrases into high-dimensional text features through a word embedding module, and mapping the high-dimensional text features into a public space; splicing the audio representation information in the public space with the high-dimensional text features, carrying out multi-modal feature combination, and constructing an evaluation sequence; and the big language model carries out evaluation prediction on the evaluation sequence to obtain a corresponding pronunciation accuracy evaluation result and a fluency evaluation result. The method does not need to depend on an automatic voice recognition system, and the evaluation efficiency of the audio data is greatly improved while the evaluation process is simplified.

The invention discloses a multi-modal emotion calculation enabling adaptive interactive language learning system and a matching method thereof, and belongs to the technical field of artificial intelligence, and the system comprises a multi-modal emotion real-time perception and quantification module MHAE-Net, a personalized adaptive dialogue and intervention strategy module ADIS-RL, and a virtual dialogue partner interactive interface module EC-NLG. The multi-modal emotion real-time perception and quantification module MHAE-Net is composed of a voice signal acquisition and processing unit, a voice emotion analysis unit, a text emotion analysis unit, a facial expression emotion analysis unit, a physiological signal emotion analysis unit and a multi-modal emotion fusion and decision-making unit. The interactive strategy is dynamically adjusted, the oral anxiety of the language is effectively relieved, the oral confidence is improved, the system adopts the multi-mode emotion calculation technology, the accuracy and robustness of emotion recognition are improved, the targeted interactive strategy is designed for different emotion states, and personalized emotion support and language practice are provided for learners.

The invention discloses a spoken English man-machine conversation interaction training method and system, and relates to the technical field of spoken English training, spoken English audio signals of a user are monitored in real time through a microphone, and the frequency spectrum information of the audio signals is refined into each frame of spectrogram and cepstrum coefficient feature vectors by using Fourier transform and feature extraction technologies; therefore, the voice signal characteristics of the user can be completely captured; three core dimensions of semantic similarity, grammatical structure similarity and fluency are incorporated into an evaluation model, a comprehensive oral English training evaluation system is formed, a text sequence is generated by analyzing a time sequence dependency relationship through a convolutional neural network, and the oral English training evaluation model of a deep learning technology is combined, so that the oral English training evaluation accuracy is improved. According to the method, the expressions of the user in the aspects of semantics, grammar and fluency can be accurately measured, the oral training level of the user can be comprehensively judged by the system by comparing the comprehensive evaluation coefficient with the preset qualified threshold value, and a personalized learning guide is provided according to a result.

The invention relates to a table question and answer method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a form file and question data uploaded by a target user; analyzing table information in the table file to obtain a candidate label set; based on the question data and the candidate tag set, determining a spoken language understanding result corresponding to the question data, wherein the spoken language understanding result comprises an intention and a keyword corresponding to the question data; performing reverse verification on the spoken language understanding result according to the question data to obtain a verification result; under the condition that the verification result indicates that the spoken language understanding result passes verification, a cue word is constructed according to table information, question data and the spoken language understanding result in the table file; calling the large language model according to the cue word for question-answering to obtain a question-answering result. By adopting the method, a large-data-volume table can be processed, and meanwhile, the accuracy and the stability of large-model table question answering can be improved.

An audio playback system is provided to facilitate adaptive output of audio content, such as spoken word content, by an audio player in response to detecting noise interferences such as external noise outside of the playback system and / or a noise indicator that external noise is about to occur. The playback system utilizes a natural skip back point found in the audio content for playback when audio content is disrupted by external noise. Content features are used to determine the skip back point in the audio content. The audio content is replayed from such a skip back point in a manner that provides a smooth transition to pre-played content and enables a listener to follow the content. Detection of multiple interfering noises within a condensed period of time are addressed to limit redundant and time consuming playbacks.

The invention relates to the technical field of visual positioning, and discloses a structured visual positioning method, system and device based on intention recognition, and the method comprises the steps: obtaining a to-be-positioned image and a current voice instruction of a user; inputting the current voice instruction into a pre-trained intention recognition model to perform semantic structuring processing so as to construct a semantic topological graph; wherein the semantic topological graph is used for describing target structured data corresponding to the current voice instruction; reasoning based on the semantic topological graph to obtain a semantic reference corresponding to the current voice instruction; and performing feature alignment on the semantic reference and the to-be-positioned image so as to position a target object matched with the semantic reference in the to-be-positioned image. According to the invention, through semantic structuring processing on the voice instruction, accurate reasoning can be carried out on the semantics containing the spoken language instruction, so that the robustness of a visual positioning system is greatly improved, and the visual positioning effect in a daily dialogue scene is improved.

To effectively resolve the issue of inappropriate handovers during communication sessions, methods, systems, and machine-readable mediums which utilize speech features captured by microphones in the original wireless peripheral device and / or the wireless peripheral device to which the communication session is to be handed over to determine if a handover should proceed or should be reversed. Speech features refer to the composite attributes of spoken language that encompass both acoustic and linguistic features. Acoustic features characterize the sound properties of speech and include, but are not limited to, timbre, pitch, intonation, speaking rate, articulation, prosody, melody, spectral features, formant frequencies, and the like. Linguistic features pertain to the actual content conveyed, comprising words, phrases, syntax, and semantics. This includes the analysis of words, phrases, syntax, and semantics to understand the context and continuity of the conversation.

Systems, methods, and non-transitory computer-readable media including instructions for detecting and utilizing facial skin micromovements are disclosed. In some non-limiting embodiments, the detection of the facial skin micromovements occurs using a speech detection system that may include a wearable housing, a light source (either a coherent light source or a non-coherent light source), a light detector, and at least one processor. One or more processors may be configured to analyze light reflections received from a facial region to determine the facial skin micromovements, and extract meaning from the determined facial skin micromovements. Examples of meaning that may be extracted from the determined facial skin micromovements may include words spoken by the individual (either silently spoken or vocally spoken), an identification of the individual, an emotional state of the individual, a heart rate of the individual, a respiration rate of the individual, or any other biometric, emotion, or speech-related indicator.

The invention relates to the technical field of artificial intelligence education, in particular to a Russian spoken language practice personalized recommendation method and system based on artificial intelligence, and the method comprises the steps: 1, outputting a phoneme sequence with a timestamp through Russian automatic voice recognition; 2, collecting an exercise interruption position and repeated read-after behavior data; 3, generating a dynamic learner portrait; 4, mapping high-frequency errors in the learner portrait into abnormal path weights of map nodes; 5, a lattice tail error option and a non-matching body verb interference item are injected; 6, when the voice fluency attenuation of the learner exceeds a dynamic threshold value, the sentence complexity is reduced; and 7, calculating an error rate descent gradient based on the exercise completion data, and dynamically adjusting the abnormal path weight of the knowledge graph. Through audio stream analysis and syntax tree construction, the system can accurately identify errors of the learner in grammar, pronunciation and other aspects, and the learning efficiency is improved.

The invention discloses a spoken language understanding joint model method based on label attention and a window mechanism. The spoken language understanding joint model method is used for improving the effects of intention recognition and slot filling. The method comprises the following steps: firstly, performing semantic coding on a statement input by a user by adopting a self-attention mechanism and Bi-LSTM coding to generate basic semantic representation; then, dynamically adjusting attention distribution of each lexical element through a tag attention mechanism, extracting sentence-level intention and slot tag semantics, and constructing an overall semantic context, so as to form an intention tag attention module and a slot tag attention module; the intention module preliminarily predicts the intention of a statement by using a block-level sliding window, and the slot module preliminarily predicts slot position information by means of a slot classifier. And then, through a graph convolution layer, carrying out adaptive fusion on the preliminarily predicted intention and slot position information, and realizing information interaction between nodes to obtain an updated label embedding representation. And finally, decoding the embedded representation through a classifier module, and generating a final intention and slot position recognition result.

At least one exemplary embodiment is directed to a method and device for voice operated control with learning. The method can include measuring a first sound received from a first microphone, measuring a second sound received from a second microphone, detecting a spoken voice based on an analysis of measurements taken at the first and second microphone, learning from the analysis when the user is speaking and a speaking level in noisy environments, training a decision unit from the learning to be robust to a detection of the spoken voice in the noisy environments, mixing the first sound and the second sound to produce a mixed signal, and controlling the production of the mixed signal based on the learning of one or more aspects of the spoken voice and ambient sounds in the noisy environments.

A method for modifying recorded audio of a video title may include: identifying a video title available for presentation; identifying a portion of audio data, of the video title, representing dialogue of an actor of the video title; identifying a first utterance of the dialogue to be replaced in the video title; selecting, based on a replaceable amount of the audio data, a second utterance to replace the utterance word in the portion of the audio data; generating, based on a voice profile of the actor, first speech signals representing the actor uttering the second utterance; removing second speech signals representing the actor uttering the first utterance from the dialogue; adding the first speech signals into the dialogue using at least a portion of the replaceable amount; and generating a modified version of the video title comprising the dialogue with the speech signals added.

Disclosed are apparatuses, systems, and techniques that may use machine learning for implementing automatic speech recognition (ASR) facilitated with a search for target words. The techniques include applying an ASR model to audio data to generate an ASR output representative of a likelihood that the audio data comprises one or more spoken speech units (SUs), generating, using the ASR output, a first score characterizing a likelihood that the audio data comprises a first word, wherein the first word comprises a dictionary word, generating, using the ASR output, a second score characterizing a likelihood that the audio data comprises a second word, wherein the second word comprises a word of a plurality of target words, wherein the plurality of target words is identified based at least on a context of the audio data, and predicting, using the first score and the second score, a spoken word associated with the audio data.