6105 results about "Voice data" patented technology

The invention discloses a multi-modal analysis method, system and device for an industrial inspection scene and a medium. The method comprises the following steps: acquiring multi-modal data of an industrial inspection area in real time, wherein the multi-modal data comprises a video image and voice data; inputting the currently collected multi-modal data into a scene type identification model, and identifying a scene type; matching a corresponding multi-modal fusion mode based on the determined scene type; according to the matched multi-modal fusion mode, performing feature extraction on the video image by using a deep learning algorithm, performing feature extraction on the text converted from the voice data by using a natural language processing technology, and performing multi-modal fusion on the extracted features; performing joint reasoning on the multi-modal fusion features by using a deep learning algorithm, and identifying potential risks or equipment faults in the industrial inspection area; and generating an inspection report and decision based on the identified potential risk or equipment fault. According to the invention, intelligent analysis of the industrial inspection scene can be realized, and the inspection efficiency and accuracy are improved.

The invention provides an intelligent sound box voice processing method and system based on artificial intelligence, and the method comprises the steps: obtaining audio data and mouth shape video data, carrying out the processing of the audio data and the mouth shape video data, and carrying out the multi-modal feature fusion, and obtaining a fusion feature; performing bimodal voice activity detection on the fusion features to obtain effective voice data; performing context sensing recognition of audio and video fusion on the effective voice data to obtain a first text; constructing a user feature model, and performing semantic understanding on the text based on the model to obtain an understanding result; performing intention recognition and slot filling based on the understanding result to obtain user intention and key information; generating a response strategy in combination with the user intention, the key information and the environment perception data; generating response voice according to the response strategy; and monitoring feedback information of the user to the response voice in real time, and updating the user feature model and the response strategy evaluation model based on feedback. According to the scheme, the voice can be recognized more accurately, and the safety and robustness of the system are enhanced.

The invention relates to the technical field of large language models, and discloses a voice data interaction feedback control processing method based on a large language model. The method comprises the following steps: acquiring an industrial voice instruction through an AMR main controller, and obtaining a standardized vector through industrial lexicon matching and intention classification; inputting an industrial large language model for reasoning processing, and generating an AMR execution scheme; carrying out distributed coordination and task allocation on the AMR cluster to form a control instruction sequence; and the motion controller executes monitoring, processes exceptions and outputs a feedback strategy. Accurate classification and semantic understanding of complex industrial instructions are achieved, the processing capacity of professional knowledge in the industrial field is improved, and meanwhile high-precision real-time state monitoring is achieved.

The invention relates to dialect speech synthesis, in particular to a semi-supervised dialect emotion speech synthesis system based on hybrid experts, which comprises a text analysis module for preprocessing an input dialect text and generating a text representation vector through feature extraction and feature fusion; the mixed expert module is used for acquiring dialect acoustic features, rhythm features, emotional features and general acoustic features; the dynamic routing module is used for realizing intelligent cooperation among experts through a task-aware soft routing algorithm; the semi-supervised learning module is used for training supervised learning by using the marked dialect emotion voice data and training self-supervised learning by using the unmarked dialect emotion voice data; the acoustic parameter generation module is used for integrating the output of each expert to generate a complete acoustic parameter set; the neural vocoder is used for converting the acoustic parameter set into final dialect emotion voice; the method can effectively overcome the defect that the dialect emotional voice is difficult to accurately synthesize under the condition of lack of sample resources.

The invention discloses a mechanical arm control method, system and equipment for realizing a multi-modal universal operation task. The method comprises the following steps: acquiring multi-modal data of text, voice and vision; performing modal alignment on the multi-modal input data, encoding the multi-modal input data into a uniform input representation vector, and splicing the uniform input representation vector into a multi-modal instruction paragraph; retrieving a user information voice database based on voiceprint features of the voice instruction data to obtain task associated voice data; instruction information is input into the strategy large model to obtain an action plan; a post-processing task action planning input controller controls a mechanical arm, and a multi-mode input instruction is updated in real time to perceive the latest task state; and performing closed-loop control based on the initial target instruction information and the environment perception information until the task is completed. The task planning and error correction capacity of the mechanical arm is improved through closed-loop control and voice data retrieval based on the large strategy model, and the dynamic adaptive capacity, the autonomous error correction capacity and the multi-mode interaction capacity of the mechanical arm are improved.

The invention provides a multi-modal intelligent question-answering and recommendation system supporting emotional speech output, and relates to the technical field of visual question-answering, and the system specifically comprises a multi-modal question-answering module, a multi-modal recommendation module, a speech recognition module, and a speech synthesis module. Wherein the multi-mode question answering module converts an image input by a user into a natural language form, and final question answering is completed in combination with a comet knowledge base COMET; the voice recognition module is used for recognizing voice data input into the multi-mode intelligent question answering and recommending system and processing the voice data into a text form; the multi-modal recommendation module identifies whether modal missing exists in input data or not, generates representation of the missing modal, obtains vector representation of the user and the article through a graph convolutional neural network and learning according to an interaction matrix of the user and the article, carries out reliability calculation, and finally sends a multi-modal recommendation result to the user and the article. Sending back to the multi-mode question and answer module; and the voice synthesis module outputs the voice to the user.

The invention discloses a large-model complaint intention recognition method based on sentiment analysis, and the method comprises the steps: obtaining call voice data of a customer, and converting the call voice data into text data; preprocessing the text data, and removing noise and marking components; performing emotion feature analysis on the text data, extracting an emotion feature value, and generating an emotion vector; the emotion vectors and the text data are input into a large model together, and the large model combines the emotion feature values and context semantics to generate intention feature vectors; constructing an emotion-intention state vector, inputting the emotion-intention state vector into an asynchronous dominant actor commentator algorithm model, and generating a corresponding complaint intention probability value; judging a complaint intention probability, and generating risk early warning; and preferentially distributing high-risk customers and responding to customer demands. Through combination of voice data preprocessing, text semantic feature extraction, emotion intensity quantitative analysis and a multi-modal fusion algorithm and efficient risk assessment based on an asynchronous dominant actor reviewer model, the early warning and response capabilities of customer complaint risks are significantly improved.

The invention discloses a large-model real-time voice text intention recognition method and system based on text prediction, and the method comprises the steps: obtaining the real-time voice data of a user, carrying out the real-time voice recognition processing through a streaming voice recognition interface, and obtaining a part of transcriptional text; inputting the partial transcription text into a mask language model for text prediction, and generating a plurality of high-credibility complete sentence candidates; based on the complete sentence candidates, the complete sentence candidates are input into a large language model in parallel for intention recognition, a corresponding intention result is obtained, and a mapping relation between the candidate sentences and the intention recognition result is established; and obtaining a sentence completely expressed by the user, calculating the similarity between the complete actual sentence and a plurality of high-credibility complete sentence candidates through a multi-level text similarity algorithm, selecting the candidate sentence with the highest similarity score, and directly obtaining a corresponding final intention recognition result based on the mapping relationship. The objective of the invention is to solve the technical problem of high response delay of an existing voice intention recognition system.

The invention provides a customer service talk skill quality inspection result determination method and device, equipment, a medium and a product, and belongs to the technical field of artificial intelligence. Determining customer service verbal skills, user verbal skills and target standard verbal skills corresponding to the customer service verbal skills according to the conversation interaction voice data, and inputting the customer service verbal skills, the user verbal skills and the target standard verbal skills into a large language model to obtain a quality inspection result; the large language model can determine an integrity index score, a key point index score, a logicality index score and an emotion index score corresponding to a single sentence verbal skill in the customer service verbal skill based on a preset instruction and a target standard verbal skill so as to output a quality inspection result. According to the method, the integrity, the key points, the logicality and the emotional expression of the dialogue are deeply analyzed by utilizing a large language model, the method is not limited to literal matching any more, and higher flexibility and accuracy are realized in customer service dialogue quality inspection.

The invention relates to the technical field of medical informatization, and discloses a method for generating a medical record report based on a doctor-patient dialogue, and the method comprises the following steps: multi-modal data collection: collecting the multi-modal data of the doctor-patient dialogue, including voice data, video data and environment data; multi-modal data preprocessing: preprocessing the collected data, including noise reduction and feature extraction of voice signals, key point extraction of video data and time synchronization of the multi-modal data; and semantic analysis: converting the voice signal into a text by using a voice recognition technology, carrying out semantic analysis on the text, and extracting key information. Through multi-modal data acquisition, semantic analysis, emotion modeling and knowledge graph reasoning technologies, the comprehensiveness, logicality and dynamic correction ability of medical record generation are improved, meanwhile, the defects of patient psychological state analysis in the prior art are overcome, information orderliness and doctor decision-making efficiency are remarkably improved through generation of the visual medical record report, and the medical record report generation method is suitable for popularization and application. And powerful support is provided for precise medical treatment.

Systems and techniques for generating and animating non-player characters (NPCs) within virtual digital environments are provided. Multimodal input data is received that comprises a plurality of input modalities for interaction with an NPC having a set of body features and a set of facial features. The multimodal input data is processed through one or more neural networks to generate animation sequences for both the body features and facial features of the NPC. Generating such animation sequences includes disentangling the multimodal input data to generate substantially disentangled latent representations, combining these representations with the multimodal input data, and using a large-language model (LLM) to generate speech data for the NPC. Further processing using reverse diffusion generates face vertex displacement data and joint trajectory data based on the combined representation and generated speech data. The face vertex displacement data, joint trajectory data, and speech data are used to produce an animated representation of the NPC, which is then provided to environment-specific adapters to animate the NPC within a virtual digital environment.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a speech recognition error correction method, which comprises the following steps of: training and generating a target field Chinese character text understanding model by collecting dialogue Chinese character texts of a target field; collecting pinyin data, and training to generate a pinyin understanding model; receiving to-be-detected voice data and converting the to-be-detected voice data into to-be-detected Chinese character texts and pinyin texts Inputting the Chinese character text and the pinyin text to corresponding understanding models to generate semantic features and pinyin features; combining the semantic features and the pinyin features to form joint features; inputting the joint features into a speech recognition error detection model, generating error probability distribution of each word, and outputting an error tag; and correcting the Chinese character text through a template error correction module based on the error tag. According to the method, more comprehensive feature representation is formed by combining semantic features and pinyin features, and the accuracy of speech recognition error detection is improved; and a template error correction module is utilized to effectively correct identification errors.

The invention discloses a speech recognition and authentication method and system based on multi-modal features and dynamic evaluation in the technical field of speech recognition and authentication, and the method comprises the steps: collecting an original speech signal of a user through a microphone, and carrying out the preprocessing of the original speech signal, and obtaining the preprocessing speech data; and extracting feature data of the preprocessed voice data by adopting a multi-dimensional feature hierarchical extraction technology, and injecting a multi-source noise sample into an acoustic feature space of a voiceprint feature model based on an initial training stage established by the voiceprint feature model to construct an anti-noise mixed voiceprint map. Through integrating voiceprint, semantics, behavior characteristics and an environment adaptation mechanism, an authentication threshold is adjusted in real time according to dynamic risk assessment, meanwhile, a risk scoring model is utilized to calculate a comprehensive risk value, authentication modes of different levels are started according to risk scenes of different degrees, and two-factor authentication is forcibly implemented for high-risk scenes. And the authentication security and reliability can be obviously enhanced.

The invention relates to the technical field of data analysis, can be applied to business scenes of medical health, financial science and technology, cultural research and the like, and discloses a multi-modal abstract generation and output method, which comprises the following steps of: acquiring target domain data and cleaning to obtain structured data; extracting entities and association relationships among the entities from the structured data, and constructing a target domain knowledge graph; generating a text abstract containing entity associated information based on the target domain knowledge graph; matching the image data and the audio data corresponding to the text abstract, and integrating the text abstract, the image data and the audio data to generate a multi-modal abstract; and constructing a user portrait according to the user behavior data, and outputting the multi-modal abstract based on the user portrait. According to the method, extraction and analysis of entities and association relationships thereof are optimized through knowledge graph construction; through multi-modal fusion, associated display of texts, images and voices is realized; and through user portrait optimization output, the efficiency and adaptability of knowledge summarization are improved.

The invention discloses a construction method of a real-time streaming voice intelligent question and answer service system. The construction method comprises the following steps: receiving input voice data; calling an independent voice recognition service to convert the voice data into an input text; the method comprises the following steps: constructing cue words of a large language model based on an input text, initiating an LLM streaming request to an independent large language model service, and obtaining streaming text answers generated by LLM in real time; segmenting the flow type text answer in real time through a dynamic sentence segmentation device to generate a plurality of clauses; calling an independent speech synthesis service in parallel for each clause, and converting the text into an audio data block; and combining the audio data blocks into streaming audio data according to the generation sequence, and returning the streaming audio data to a client for playing in real time. According to the invention, by decoupling ASR, LLM and TTS services and combining the asynchronous streaming framework and the strategy of "wanting while speaking", the real-time performance and flexibility of the voice question-answering system are significantly improved.

Systems and techniques for generating and animating non-player characters (NPCs) within virtual digital environments are provided. Multimodal input data is received that comprises a plurality of input modalities for interaction with an NPC having a set of body features and a set of facial features. The multimodal input data is processed through one or more neural networks to generate animation sequences for both the body features and facial features of the NPC. Generating such animation sequences includes disentangling the multimodal input data to generate substantially disentangled latent representations, combining these representations with the multimodal input data, and using a large-language model (LLM) to generate speech data for the NPC. Further processing using reverse diffusion generates face vertex displacement data and joint trajectory data based on the combined representation and generated speech data. The face vertex displacement data, joint trajectory data, and speech data are used to produce an animated representation of the NPC, which is then provided to environment-specific adapters to animate the NPC within a virtual digital environment.

The invention discloses a commerce and trade circulation supply chain optimization method based on multi-modal data fusion, and the method comprises the following steps: 1, multi-modal data collection: collecting the multi-modal data of the whole supply chain in real time through an Internet of Things device, an enterprise ERP system, a logistics tracking system and a third-party platform, the multi-modal data comprises text data, image data, voice data, time sequence data and spatial data. When the scheme is implemented, multi-source heterogeneous data of the Internet of Things, ERP and the like are integrated, so that the problem of an information island in a traditional supply chain is effectively solved; position coding and a cross-modal attention mechanism are introduced into the multi-modal fusion model, so that the data alignment precision is remarkably improved, for example, when'emergency order adding 'in a voice communication record is associated with a corresponding order image, the weight distribution accuracy is improved; the application of the block chain technology ensures the transparency and traceability of the supply chain decision.

The invention discloses a customer voice analysis system based on a large model, and the system comprises a voice data collection and preprocessing module which is used for collecting and preprocessing customer voice data; the voice recognition module is used for recognizing the preprocessed customer voice data; the feature fusion module is used for extracting semantic features of the text data and fusing customer voice data features to generate voice text representation; the large model fine tuning module is used for performing fine tuning on the pre-trained starry fire cognition large model; the semantic analysis and emotion recognition module is used for processing the voice text representation, extracting semantic information, recognizing an emotion state and generating an analysis result; and the dialogue context management module is used for associating customer historical voice data, establishing dialogue context information and updating a voice analysis result. According to the method, the stellate fire cognitive large model and the improved quantitative low-rank adaptation are utilized, and the method has the advantages of high precision, low calculation cost and high intelligent adaptability.

The invention discloses a multi-modal data processing method, a household appliance and a control method and system of the household appliance. The method comprises the following steps: acquiring multi-modal data; wherein the multi-modal data at least comprises at least one of text data, image data and voice data, and sensor data; the sensor data is acquired by a sensor corresponding to the household appliance; and carrying out data fusion on the multi-modal data by utilizing at least two fusion modes to obtain multi-modal fusion data with a unified semantic space. Through the above mode, effective fusion of multi-modal data in the field of household appliances can be realized, data islands can be broken, further, accurate alignment of different modal data in a unified semantic space can be realized, and the understanding ability of a large language model can be effectively improved after the multi-modal fusion data is input into the large language model.

The invention discloses a multi-modal-based emotional state evaluation method, which comprises the following steps: S1, collecting multi-modal data of a user to be tested, the multi-modal data comprising voice data, facial micro-expression data and facial heart rate data; s2, constructing a multi-modal deep neural network, inputting the multi-modal data into the multi-modal deep neural network, and dynamically adjusting the weight of the modal data by the multi-modal deep neural network by using an attention mechanism so as to output and obtain fusion features; and S3, judging the emotional state of the to-be-detected user based on the fused features. Accordingly, the invention further discloses a multi-mode-based emotional state evaluation system, and the multi-mode-based emotional state evaluation system can be used for implementing the emotional state evaluation method, so that the physiological status of the patient is more accurately and comprehensively considered, the emotional state of the patient is evaluated, and the accuracy of the emotional state evaluation is improved. Therefore, professional psychologists are assisted in diagnosing depression.

The invention discloses a data large screen visualization method and device based on a large model, equipment and a medium, and relates to the technical field of data large screens, and the method comprises the steps: obtaining externally input user voice data, carrying out the intention analysis of the user voice data, and determining a query prompt word representing the intention from the user voice data; inputting the query prompt word into a preset large model for analysis, and generating target data corresponding to the query prompt word and a target chart template matched with the query prompt word; and mapping the target data to the target chart template, determining a target chart, and visualizing the target chart. According to the method, the large-screen data display efficiency can be improved.

The invention relates to the technical field of customer services, in particular to an intelligent outbound call skill generation method, system and device using LLM and a medium, and the method comprises the steps: collecting user voice data and text dialogue data in real time, and converting the user voice data and the text dialogue data into a time sequence aligned text sequence matrix based on the user voice data and the text dialogue data; inputting the time sequence aligned text sequence matrix into a preset multi-mode sentiment analysis model, and calculating a user sentiment tendency value; extracting an intention keyword based on the text sequence matrix with the aligned time sequence, inputting the intention keyword into a preset intention recognition model, and outputting a purchase intention probability of the user; and according to the user emotion tendency value and the user purchase intention probability, calling LLM to generate a personalized outbound call, and outputting the personalized outbound call to a dialogue system. The method has the advantages that high-quality and dynamic-adaptability customer service is achieved, and user experience is remarkably improved.

The invention discloses a personalized digital human generation method based on a single video, and relates to the field of virtual digital human modeling and driving, and the method aims at the video and voice data of a target person, through introducing a multi-modal alignment constrained voice driving synchronization mechanism and combining semantic understanding and an emotion label expression generation model, a personalized digital human model is generated. High synchronization, nature and vividness of digital human facial expressions and voice contents are realized. According to the method, a self-supervised style consistency constraint is added in model training to ensure that the style of a generated character image is stable, and a 3D semantic mask is added in an image fusion stage to improve the fusion precision and realistic effect of a synthetic facial expression and a reference face. According to the method, the digital human can be rapidly cloned and driven to synthesize the expression through the voice only through a single video sample, the generation process is efficient, and the obtained digital human video has excellent sense of reality and interactivity.

The invention discloses a smart home scene understanding and interaction method and system based on multi-modal fusion, and the method comprises the steps: converting collected voice data into text data, carrying out the content understanding of collected image / video data through a multi-modal visual large model in order to more effectively understand the behaviors and potential intentions of a user, and carrying out the interaction of the collected image / video data. Then performing information extraction and reasoning on text data converted from voice data and contents understood by the multi-modal visual large model by utilizing the strong logical reasoning ability of the large language model; then, an instruction is generated based on the content of user interaction, visual cue words are generated from the instruction, real-time segmentation of the image / video content is implemented according to the visual cue words by adopting a multi-modal visual large model, and the large language model and the multi-modal visual model are fused in the intelligent home scene user interaction process through the visual cue words; and finally, performing three-dimensional processing on the segmented image, and realizing deep fusion with an AR / MR scene based on real-time segmentation of the scene.

The invention belongs to the technical field of vehicle state detection and early warning, and particularly relates to a vehicle state monitoring and early warning method and system.The monitoring and early warning method comprises the steps that a sensor obtains real-time data, an anomaly detection algorithm is applied, and an abnormal event is marked; calculating an information priority according to the abnormal severity and the driving scene, and distributing the information priority to a high-priority queue; multi-mode early warning is generated, and high-frequency sound and vibration are used during high-speed driving; extracting a voice prompt, and generating voice waveform data through a voice synthesis module; voice input of a driver is recognized, intention is analyzed, and abnormal information feedback is provided; according to the scene and the abnormal state, interactive output is optimized, and detailed information is displayed during low-speed congestion; dynamically adjusting the interface of the instrument panel, and amplifying the key area in case of abnormal severity; and integrating the image data and the voice data, generating a multi-mode signal, and transmitting the multi-mode signal after rendering processing. The vehicle abnormal information can be timely and accurately transmitted to a driver, and the driving safety is effectively improved.

A method of programming an industrial robot (100), composed of a robot manipulator (110) and a robot controller (120), comprises: recording movements of the robot manipulator during a kinesthetic programming session, for thereby obtaining a robot trajectory; capturing speech data while recording the movements of the robot manipulator; decomposing the recorded speech data into a plurality of phases; for each phase of the speech data, parsing the speech data into robot parameter values using a natural-language model (170), identifying a contemporaneous phase of the robot trajectory and annotating it with the robot parameter values; and, on the basis of the annotated robot trajectory, generating a robot program executable by the robot controller.

Machine learning systems and techniques are described herein for monitoring application servers within the computing infrastructure of a wireless network. A machine learning system may train and execute models based on performance metrics, error data, and failures from application servers configured to perform various functions relating to communication session authorization, monitoring, and charging for voice, data, and messaging communication services over the wireless network. Machine learning models may be configured to predict service degradations, detect performance anomalies, and determine root causes, and various models may output particular application servers, components, and / or predicted times of certain events. Based on the outputs of the machine learning models, the machine learning system may perform various actions for the wireless network infrastructure, including data outputting and / or executing processes on the application servers to analyze, restart, and / or repair particular components.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a service content generation method based on emotion recognition, and the method comprises the steps: collecting voice data, extracting voiceprint features, and building a user voiceprint file; extracting Mel frequency cepstrum coefficient features and Mel energy difference coefficient features, and inputting the extracted features into an emotion classification model for emotion classification recognition to obtain an emotion state of the user; the current emotional state is compared with a historical normal emotional state, emotional fluctuation is analyzed, and whether the emotional state of the user is abnormal or not is judged; and when it is detected that the emotional state is abnormal, target service content is determined from the personalized service library in combination with the emotional state and the user voiceprint file. According to the method, the voice features are extracted and the user voiceprint file is established, so that the emotion fluctuation of the user can be accurately analyzed, whether abnormity exists or not is judged, accurate recommendation service is provided in combination with the personalized features and historical behavior data of the user, and the intelligent and personalized degree of vehicle-mounted voice interaction is improved.

The invention discloses a multi-mode instruction interaction method and system based on a virtual character, computing equipment and a storage medium, and the method comprises the steps: obtaining and recognizing user posture data, and obtaining a first operation instruction corresponding to the posture data; voice data of the user are obtained and recognized to obtain intention data of the user, and a corresponding second operation instruction is obtained according to the intention data; the first operation instruction and the second operation instruction are subjected to timestamp alignment, the similarity of the first operation instruction and the second operation instruction in the semantic space is calculated, the first operation instruction and the second operation instruction are fused and dynamically adjusted through a weighted average method, and an instruction with the highest priority is obtained to serve as a final operation instruction to control an object in the virtual scene to move. According to the method, the gesture instruction and the voice instruction are deeply fused, the interaction space between the user and the virtual environment is greatly expanded, the limitation of a traditional single interaction mode is broken through, and more natural, efficient and intelligent immersive experience is provided for the user.

The invention provides a multi-mode voice interaction method based on a large model, electronic equipment and a storage medium, and the method comprises the steps: carrying out the voice recognition of the voice data of a first language inputted by a user, and obtaining a first language text; performing language translation on the first language text to translate the first language text into a second language text; performing semantic analysis on the second language text to obtain a semantic analysis result; performing multi-modal knowledge graph construction on the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multi-modal information related to the second language text is fused in the multi-modal knowledge graph; generating a dialogue strategy of the enhanced semantic understanding result; and performing voice synthesis based on the dialogue strategy to generate reply voice data expressed in the first language. According to the method and the device, multi-language interaction is realized, and meanwhile, various modal information is fused.