84 results about "Multimodal interaction" patented technology

A human-computer interface system and methods for providing intelligent, adaptive, multimodal interaction with users while accomplishing tasks on their behalf in some particular domain or combination of domains. Specifically, this system accepts user input via natural language text, mouse actions, human speech, whistles, gestures, pedal movements, facial or postural changes, and conveys results via natural language text, automatically-generated speech, and displays of graphs, tables, animation, video, and mechanical and chemical effectors that convey heat, tactile sensation, taste and smell.

A voice-enabled system for online content access and delivery provides a voice and telephony interface, as well a text and graphic interface, for browsing and accessing requested content or shopping over the Internet using a browser or a telephone. The system allows customers to access an online data application, search for desired content items, select content items, and finally pay for selected items using a credit card, over a phone line or the Internet. A telephony-Internet interface converts spoken queries into electronic commands for transmission to an online data application. Markup language-type pages transmitted to callers from the online data application are parsed to extract selected information. The selected information is then reported to the callers via audio messaging. A voice-enabled technology for mobile multi-modal interaction is also provided.

A method for managing multimodal interactions can include the step of registering a multitude of modality components with a modality component server, wherein each modality component handles an interface modality for an application. The modality component can be connected to a device. A user interaction can be conveyed from the device to the modality component for processing. Results from the user interaction can be placed on a shared memory are of the modality component server.

A system and method of presenting widgets to a user during a multi-modal interactive dialog between a user and a computer is presented. The system controls the multi-modal dialog; and when user input would help to clarify or speed up the presentation of requested information, the system presents a temporary widget to the user to elicit the user input in this regard. The system presents the widget on a display screen at a position that will not interfere with the dialog. Various types of widgets are available, such as button widgets, sliders and confirmation widgets, depending on the type of information that the system requires.

A factored multimodal interaction architecture for a distributed computing system is disclosed. The distributed computing system includes a plurality of clients and at least one application server that can interact with the clients via a plurality of interaction modalities. The factored architecture includes an interaction manager with a multimodal interface, wherein the interaction manager can receive a client request for a multimodal application in one interaction modality and transmit the client request in another modality, a browser adapter for each client browser, where each browser adapter includes the multimodal interface, and one or more pluggable synchronization modules. Each synchronization module implements one of the plurality of interaction modalities between one of the plurality of clients and the server such that the synchronization module for an interaction modality mediates communication between the multimodal interface of the client browser adapter and the multimodal interface of the interaction manager.

The ivnention discloses a multi-media real time control lessons giving system based on IP network, it is made up of lessons giving end, listening end, class service center and multipoint control unit(MCU); it realize a live telecast of teaching through multi-media transcribing the teaching spot and network transmission, through multi-media and multi-mode intercommunication network the teachers and the students natural blackboard-writing teaching by the teacher, teaching content retrieving, share of applied procedure, teaching software skimming, electronic white board, real time control teaching software and the class management function, it broads the teaching scale.

The present invention provides a method, a device and a system for multimodal interactions. The method according to the invention comprises the steps of activating a multimodal user interaction, providing at least one key input option and at least one voice input option, displaying the at least one key input option, checking if there is at least one condition affecting said voice input option, and providing voice input options and displaying indications of the provided voice input options according to the condition. The method is characterized by checking if at least one condition affecting the voice input is fulfilled and providing the at least one voice input option and displaying indications of the voice input options on the display, according to the condition.

The invention concerns a method of providing a multi-modal dialog between a multi-modal application (5) and a user (6) communicating with the multi-modal application (5) via a client (4) suited to exchange and present documents (7) encoded in standard or extended hyper text mark-up language. The invention further concerns a proxy-server (2) for executing this method. The multi-modal dialog between the multi-modal application (5) and the user (6) is established through the proxy-server (2) interacting with the client (4) via exchange of information encoded in standard or extended hyper text mark-up language. The proxy-server (2) retrieves at least one additional resource of modality requested within the multi-modal dialog. The proxy-server (2) composes a multi-modal interaction with the user (6) based on standard or extended hyper text mark-up language interactions with the client (6) and on the retrieved additional resources (71, 72, 73, 31, 32, 33).

Disclosed are methods and systems for providing a multi-mode interactive experience. In one embodiment, a method comprises hosting a virtual environment corresponding to a real-life simulation environment, on a multi-mode interactive experience server, and networking the server and a venue management system configured to control events occurring in the real-life simulation environment. The networking enables a user to interact with the multi-mode interactive experience in a real-life simulation mode or in a virtual mode. The method further comprises associating a personalized version of the multi-mode interactive experience with an identification code assigned to the user, updating the personalized version of the multi-mode interactive experience according to events occurring in the real-life simulation mode and events occurring in the virtual mode of the multi-mode interactive experience, and providing the updated personalized version of the multi-mode interactive experience in one of the real-life simulation mode or the virtual mode.

In order to enable an application to be provided with multimodal inputs, a multimodal application interface (API), which contains at least one rule for providing multimodal interaction is provided.

The invention discloses a virtual robot multimodal interaction method and system applied to a live video platform. An application of the live video platform is provided with a virtual robot for associating the live video, the virtual robot has multimodal interaction ability, and the public opinion monitoring method comprises the following steps: information collection step: collecting public opinion information of live broadcast in the current specific direct broadcasting room, wherein the public opinion information comprises viewed text feedback information; public opinion monitoring step: calling text semantic understanding ability and generating a public opinion monitoring result of the specific direct broadcasting room; and a scene event response step: judging an event characterized by the public opinion monitoring result, calling the multimodal interaction ability, and outputting multimodal response data through the virtual robot. By adoption of the virtual robot multimodal interaction method and system disclosed by the invention, the viewing feedback information of audience can be monitored and prompted in real time, the live video operation is assisted by the virtual robot to keep the viscidity with the users consistently, and thus the user experience is improved.

A System with mechanisms is provided that allows anyone or any entity to signup with basic identity information along with any service description resulting in an instant HTTP URL as a means for outside parties to contact this provider of the service and engage in multi-modal interactions involving voice, video, chat and media sharing. A single user or a group of users can signup with this system and provide optionally some contact information such as phone numbers. A HTTP URL is instantly generated that can be advertised as hyperlinks directly by the users to the outside environment such as on a website, email or any other means. The providers, who have signed-up for the service, can also remain signed-in into the system so as to engage with users wishing to contact them via the published URL. Any user with an internet connection upon clicking this URL will be directed to this proposed system and will be provided with a conversation window through which they can start engaging with the provider via chat, voice and video (webcam) and start sharing with each other any media information such as pictures and videos. The provider if not signed-in into the system can also engage with the user via SMS and email. The HTTP URL is being used as a starting point for users to engage with the providers of any service behind that URL. This URL can be advertised by the proposed system in various online marketplaces or social networks of the provider as well as on the web to be identified by search engines.

The invention provides a health information management television based on multimodal human-computer interaction. The health information management television comprises an information input module used for inputting health information of a user; an input interaction module used for acquiring image information, sound information, environment information, text input information and the like; a health monitoring module used for judging the health status of the user according to the acquired image information and sound information under a monitoring state, executing a corresponding operation according to the category of health status abnormalities, and giving corresponding hints according to the acquired environment information; and an output interaction module used for interacting with the user through images, texts and voice under an interaction state according to the acquired image information, sound information, environment information and the like, and transmitting health data to a health management center of the cloud. The health information management television based on the multimodal human-computer interaction provided by the invention can understand the behavior intention ofthe user, can realize the health information management between the television and the user through the way of multimodal interaction between the user and the television, and realize the supervision,judgment and recognition of the user behavior.

An electronic apparatus with a multi-mode interactive operation method is disclosed. The electronic apparatus includes a display unit, a selecting unit, a voice recognition unit and a control unit. The display unit displays a frame. The selecting unit selects an arbitrary area of the frame on the display unit. The voice recognition unit recognizes a voice signal as a control command. The control unit processes data according to the control command on the content of the arbitrary area selected. A multi-mode interactive operation method is disclosed herein as well.

The invention provides an interactive method based on the behavior standard of a virtual human. The virtual human starts the voice, emotion, vision and perception ability when the virtual human is inthe interactive state through the display of an intelligent device. The method comprises the following steps: acquiring the multi-modal interactive data, analyzing the multi-modal interactive data, and obtaining the interactive intention of the user; generating multi-modal response data and virtual human emotion expression data matched with multi-modal response data according to the interaction intention, wherein, the virtual human emotion expression data represent the virtual human's current emotion through the virtual human's facial expression and body movement; outputting multimodal response data with virtual human emotion expression data. The invention provides a virtual human, which can perform multimodal interaction with a user. Moreover, the invention can output the virtual human emotion expression data when outputting the multimodal response data, and express the current virtual human emotion through the virtual human emotion expression data, so that the user can enjoy the human-like interactive experience.

The invention belongs to the technical field of computer application, and particularly relates to a questionnaire test system and a questionnaire test method. A questionnaire test system based on multimodal interactions of eye movement, voice and touch screens comprises a computer (1) with built-in questionnaire test software and a touch screen input function, and is characterized by further comprising an eye movement collecting device (2) and a voice input device (5). The test device can realize questionnaire tests based on the multimodal interactions of eye movement, voice and touch screens, and adopts natural interactive modes to collect to-be-tested feedback, so that the difficulty of to-be-tested personnel for inputting information is reduced, and the effectiveness and accuracy of the questionnaire tests are improved. The test device is especially suitable for occasions that two hands of the to-be-tested personnel are occupied, and the continuity of the test process is enhanced. The test method is suitable for special populations who have disabled hands, so that the applicable scope of questionnaire test software is enlarged.

The invention discloses a vibration tactile feedback device design method based on information physical interaction, which is characterized in that some information of a virtual-real interaction system is fed back in a non-visual mode, namely a vibration tactile feedback device, and the vibration tactile feedback device is coordinated with a visual feedback system. According to the vibration tactile feedback device, on the basis of real tactile feedback of the hand and through vibration superposition of collision feedback of a real object to a virtual object, collision interaction between thevirtual object and a physical object is simulated, so that a person can sense contact collision between the real object and the virtual object in a virtual-real fusion environment, more real interaction experience is generated, and multi-mode interaction between the person and an AR environment and between the person and the virtual object is improved. The visual and tactile combined multi-mode interaction is helpful for expanding an information feedback channel, improving the coordination of people and a system, enhancing the seamless fusion among participants, a real environment and a virtual environment in an information physical fusion system, and realizing natural harmonious man-machine interaction.