145 results about "Natural interaction" patented technology

Roughly described, a natural language interpretation system that provides commands to a back-end application in response to user input is modified to separate out user interaction functions into a user interaction subsystem. The user interaction subsystem can include an interaction block that is specific to each particular I/O agency, and which converts user input received from that agency into an agency-independent form for providing to the natural language interpretation system. The user interaction subsystem also can take results from the back-end application and clarification requests and other dialoguing from the natural language interpretation system, both in device-independent form, and convert them for forwarding to the particular I/O agency.

Embodiments of the invention provide apparatus and methods for interactive reality augmentation, including a 2-dimensional camera (36) and a 3-dimensional camera (38), associated depth projector and content projector (48), and a processor (40) linked to the 3-dimensional camera and the 2-dimensional camera. A depth map of the scene is produced using an output of the 3-dimensional camera, and coordinated with a 2-dimensional image captured by the 2-dimensional camera to identify a 3-dimensional object in the scene that meets predetermined criteria for projection of images thereon. The content projector projects a content image onto the 3-dimensional object responsively to instructions of the processor, which can be mediated by automatic recognition of user gestures

The invention relates to a digital twin system of an intelligent production line, which comprises a physical space layer, an information layer and a virtual space layer. The physical space layer is composed of a physical production line, intelligent sensing devices and an industrial control network. The information layer includes a data conversion module, a data analysis module and a production line information database. The virtual space layer can adapt to a variety of platforms and environments including personal computers and handheld devices, is used for generating a virtual production line consistent with the physical production line by a 3D visual engine through online real-time and offline non-real-time rendering under the driving of the production line information database of the information layer, and has the functions of multi-angle of view visual display, natural interaction, state monitoring and so on. The real-time state information of the physical production line is collected by various intelligent sensing devices, and based on the information, a three-dimensional visual engine is driven to generate a virtual production line model consistent with the physical production line through rendering, thereby realizing the twin mirror image of the virtual production line and the physical production line.

The invention discloses a somatosensory-based natural interaction method for a virtual mine. The method comprises the steps of applying a Kinect to acquire gesture signals, depth information and bone point information of a user; carrying out smoothing filtering on images, depth information and bone information of the gesture signals; dividing gesture images by using a depth histogram, applying an eight neighborhood outline tracking algorithm to find out a gesture outline, and identifying static gestures; planning feature matching identification of dynamic gestures by improving dynamic time according to the bone information; triggering corresponding Win32 instruction information by using a gesture identification result, and transmitting the information to a virtual reality engine, respectively mapping the instruction information to the primary keyboard mouse operation of a virtual mining natural interaction system, so as to realize the somatosensory interaction control of the virtual mine. According to the method provided by the invention, the natural efficiency of man-machine interaction can be improved, and the immersion and natural infection represented by the virtual mine can be improved, and the application of the virtual reality and a somatosensory interaction technology can be effectively popularized in coal mines and other fields.

Some examples provides a set of frameworks, process and methods aimed at enabling the expression and exploration of free-form and parametric 3D shape designs enabled through natural interactions with a hand-held mobile device acting as a controller for 3D virtual objects. A reference plane in a virtual space generated by the location of the mobile device may be used to select a 3D virtual object intersected by the reference plane. Positioning of the mobile device may also be used to control a pointer in the virtual space. In an example, the orientation of the mobile device may be detected by an accelerometer or gyroscope. In example, the position of the mobile device may be detected by a position sensor.

The invention aims to provide an intelligent glasses device and method used for performing interaction based on multimodal input and capable of enabling the interaction to be closer to natural interaction of users. The method comprises the steps of obtaining multiple pieces of input information from at least one of multiple input modules; performing comprehensive logic analysis on the input information to generate an operation command, wherein the operation command has operation elements, and the operation elements at least include an operation object, an operation action and an operation parameter; and executing corresponding operation on the operation object based on the operation command. According to the intelligent glasses device and method, the input information of multiple channels is obtained through the input modules and is subjected to the comprehensive logic analysis, the operation object, the operation action and the operation element of the operation action are determined to generate the operation command, and the corresponding operation is executed based on the operation command, so that the information is subjected to fusion processing in real time, the interaction of the users is closer to an interactive mode of a natural language, and the interactive experience of the users is improved.

The invention discloses a mine virtual reality training system based on immersion type input and output equipment. With the system, a trainee can be completely immersed in the virtual scene, and carries out man-machine interaction with the virtual training scene through natural methods including gestures, walking and running, and the like, so as to complete the preset training program. The system is composed of output equipment such as a head-wearing display, an omnidirectional running machine, output equipment such as a motion capturing device, a computer, and matched training software, wherein the head-wearing display can provide complete vision immersion experiences, and the omnidirectional running machine and the motion capturing device map the movements of the trainee in the reality on a virtual object in the virtual scene, and operate the system through the natural interaction method. Compared with the traditional training method, the training system provided by the invention enables the trainee to be immersed in the virtual mine environment close to the reality at any place, and obtain the training experiences the same as those obtained in the scene, and finally, the effect comparable with the on-site training effect is obtained.

The invention discloses a mining operation multi-operation realization method based on virtual reality and augmented reality, and belongs to the technical field of the virtual reality and the augmented reality. The method comprises two patterns including the virtual reality and the augmented reality. Under a virtual reality scene, the selection and the replacement of a material and a material in the virtual reality scene can be realized, scene exploration is realized, the model can be moved and placed at will, a video is embedded, a two-dimensional code is generated, and a trigger realizes natural interaction, voice interaction and the like; under an augmented reality scene, the model can be selected, voice is played, the operation dynamics of the model can be demonstrated, and the rotation stopping, screen capture and function expansion of the model can be controlled; and under two patterns, various interaction ways of voice control, gesture control and keyboard mouse control can be realized. The method is applied to the virtual simulation application scene of the mining operation, can be used for training mine lot mining workers and students of the mining engineering specialty, reducing training capital and improving the skills of workers and provides an advanced and quick meaning for guiding production construction and science and technology studies.

The invention discloses a three-dimension gesture recognizing method based on a Kinect depth image. The three-dimension gesture recognizing method is characterized by including the following steps that (1), multiple palm feature points are detected; (2), according to the starting positions of all the palm feature points, gestures matched with the starting positions of the palm feature points are found out in a gesture library, and a gesture candidate set is constructed based on the gestures matched with the starting positions of the palm feature points; (3), motion trails of the palm feature points are tracked and recorded, and the gestures which do not meet the motion trails of the palm feature points are rejected out of the gesture candidate set; (4), according to the conditions of the gestures remaining in the gesture candidate set, a gesture recognizing result is obtained. Natural interaction with a computer is achieved, and the human-computer interaction mode is enriched. The three-dimension gesture recognizing method can be widely applied to the fields of computer game control, virtual reality, digital education and the like.

The invention discloses a natural interaction pen capable of automatically inputting written contents and a handwriting detection method. According to writing motion models of human, the interaction pen measures angular velocity information and acceleration information of the motion of a human hand and a human arm in a writing motion process by using sensors in a pen body, and output voltage signals are subjected to low-pass filtering and analogue-to-digital conversion and then are sent to a microprocessor I for calculation of writing motion characteristic quantity and recognition of writing intention and mapped into handwriting information of the written contents; the signals output by the microprocessor I are sent through a data wireless communication module I; the signals sent by the data wireless communication module I are received by a receiving end of a computer host, then processed through a microprocessor II and transmitted to the computer host through a USB interface. The natural interaction pen has the advantages of no need of handwriting panel, independent work, better accordance with the writing operation habit of people, and high portability and universality.

A new pattern and method of a virtual reality system based on mobile devices, which may allow a player to design a virtual scene structure in a physical space and allow quick generation of corresponding 3D virtual scenes by a mobile phone; real rotation of the head of the player is captured by an acceleration sensor in the mobile phone by means of a head-mounted virtual reality device to provide the player with immersive experience; and real postures of the player are tracked and identified by a motion sensing device to realize player's mobile input control on and natural interaction with the virtual scenes. The system only needs a certain physical space and simple virtual reality device to realize the immersive experience of a user, and provides both a single-player mode and a multi-player mode, wherein the multi-player mode includes a collaborative mode and a versus mode.

A wearable computer system has a user interface with at least an audio-only mode of operating, and that is natural in appearance and facilitates natural interactions with the system and the user's surroundings. The wearable computer system may retrieve information from the user's voice or surroundings using a passive user interface. The audio-only user interface for the wearable computer system may include two audio receivers and a single output device, such as a speaker, that provides audio data directly to the user. The two audio receivers may be miniature microphones that collaborate to input audio signals from the user's surroundings while also accurately inputting voice commands from the user. Additionally, the user may enter natural voice commands to the wearable computer system in a manner that blends in with the natural phrases and terminology spoken by the user.

Described herein is a method for enabling human-to-computer three-dimensional hand gesture-based natural interactions. From depth images provided by a range finding imaging system, the method enables efficient and robust detection of a particular sequence of natural gestures including a beginning (start), and an ending (stop) of a predetermined type of natural gestures for delimiting the period during which a control (interaction) gesture is operating in an environment wherein a user is freely moving his hands. The invention is more particularly, although not exclusively, concerned by detection without any false positives nor delay, of intentionally performed natural gesture subsequent to a starting finger tip or hand tip based natural gesture so as to provide efficient and robust navigation, zooming and scrolling interactions within a graphical user interface up until the ending finger tip or hand tip based natural gesture is detected.

The present invention relates to a gesture manipulation method based on three-dimensional display. The method comprises: acquiring position information of a hand; establishing a motion trajectory of the hand in a coordinate system of three-dimensional space; recognizing a gesture motion of the hand according to the motion trajectory; reading a corresponding control instruction according to the gesture motion; and controlling an operation object in a three-dimensional image according to the control instruction. By recognizing a gesture motion of an user, an operational object is controlled to execute a corresponding operation according to the gesture motion. Therefore, human-computer natural interaction can be realized with no need of being contacted with a display screen. Furthermore, a gesture manipulation system and apparatus based on the three-dimensional display are further provided.

Systems and methods for natural interaction with graphical user interfaces using gestural and vocal input in accordance with embodiments of the invention are disclosed. In one embodiment, a method for interpreting a command sequence that includes a gesture and a voice cue to issue an application command includes receiving image data, receiving an audio signal, selecting an application command from a command dictionary based upon a gesture identified using the image data, a voice cue identified using the audio signal, and metadata describing combinations of a gesture and a voice cue that form a command sequence corresponding to an application command, retrieving a list of processes running on an operating system, selecting at least one process based upon the selected application command and the metadata, where the metadata also includes information identifying at least one process targeted by the application command, and issuing an application command to the selected process.

The invention discloses a head-wearable type multi-channel interaction system and a multi-channel interaction method. The head-wearable type multi-channel interaction system comprises a head-wearable type interaction device and a data processing system. The head-wearable type interaction device comprises a sensor assembly, a signal processing device, an output device and a body. The data processing system comprises an input drive module, a multi-channel processing unit, an application system and an output drive module. The head-wearable type multi-channel interaction system and the multi-channel interaction method have the advantages that real-time fusion of multiple natural interaction modes of gestures, voice, head movement postures, heart rates, brain electricity and the like, immersive three-dimensional (3D) display and voice are integrated, the interaction experience effect of virtual reality and augmented reality is improved, and care to the health of a wearer can be achieved according to the physiological perception.

A conversational and embodied Virtual Assistant (VA) with Decision Support (DS) capabilities that can simulate and improve upon information gathering sessions between clinicians, researchers, and patients. The system incorporates a conversational and embodied VA and a DS and deploys natural interaction enabled by natural language processing, automatic speech recognition, and an animation framework capable of rendering character animation performances through generated verbal and nonverbal behaviors, all supplemented by on-screen prompts.

The invention provides a robot perception and understanding method based on man-machine collaboration. The method allows an operator to help robot to perceive and understand the environment. The method comprises the steps of firstly, audio visual mode perception natural interaction, secondly, target describing and understanding and thirdly task teaching and learning. Intelligence of the human is used, the most natural and efficient interaction manner of the human is used for man-machine communication and assistance, and the robot perception and understanding flexibility, intelligence and adaptation are improved.

The invention discloses a multi-picture display-based virtual shooting cinema system and method. The multi-picture display-based virtual shooting cinema system is simple, convenient and easy to operate, strong in immersion and the like; through such natural interaction modes as the body posture in the natural life and shooting game experience, a user plays the shooting type game in a mode of interacting with the story through the body posture and an imitation gun, the player acts a virtual leading role for controlling the game development, the user does not need additional learning and training, and the user can interact with the story through walking towards the front, walking towards the back, shooting and the like. The system romances three-dimensional game pictures in real time; the system supports double cooperative engagement through network connection; one large projection screen can provide two independent and complete game pictures for two users by means of a light splitting type double-picture principle, and accordingly the system can bring stronger and more natural immersion type experiments to the users, and the entertainment, interest and reality sense of the game in a theme cinema are reinforced.

The invention relates to a natural interactive method of a virtual learning environment based on speech emotion recognition, belonging to the field of depth learning. The method comprises the following steps: 1, collecting speech signals of students and users through kinect, resampling, adding windows by frames, and mute processing to obtain short-time single frame signals; 2, carrying out fast Fourier transform on that signal to obtain the frequency domain data, obtaining the pow spectrum thereof, and adopting a Mel filter bank to obtain a Mel spectrum diagram; 3, inputting the features of the Mel spectrum map into a convolution neural network, performing convolution operation and pooling operation, and inputting the matrix vectors of the last desample layer to the whole connecting layerto form a vector output feature; 4, compressing and inputting the output characteristic into a bi-directional long-short time memory neural network; 5, inputting the output features into a support vector machine to classify and output a classification result; 6, feeding back the classification result to the virtual learning system for virtual learning environment interaction. The invention driveslearners to adjust the learning state and enhances the practicability of the virtual learning environment.

The invention relates to the technical field of artificial intelligence, in particular to an artificial intelligence interaction method and an artificial intelligence interaction system. The artificial intelligence interaction method includes acquiring text information according to input of users; analyzing the text information to obtain reply information; synthesizing reply voice information according to the reply information; adjusting actions of virtual images in 3D (three-dimensional) models according to the reply information; displaying the virtual images by holographic imaging technologies and synchronously playing the reply voice information. The artificial intelligence interaction method and the artificial intelligence interaction system in an embodiment of the invention have the advantages that feedback in multiple dimensions such as vision and hearing can be provided for users, accordingly, natural interaction can be carried out, and interaction information is abundant.

The invention provides a three-dimensional model gesture touch browsing interaction method based on a mobile terminal. The method comprises fourteen steps. According to the three-dimensional model gesture touch browsing interaction method based on the mobile terminal, application three-dimensional figure programmatic interface OpenGL ES (OpenGL for Embedded Systems) is used for rendering a light three-dimensional model on the mobile terminal (lightweight and rendering methods are not discussed in the invention.), a novel human-computer interaction mode is provided through a gesture identification method, the defects of a traditional desktop side three-dimensional model browsing mode are effectively overcome, portability is improved, and the method is applicable to more application occasions; a natural interaction mode between fingers and the three-dimensional model is achieved, and is more flexible, more visual and more efficient compared with an interaction mode based on devices such as a mouse and a keyboard.

The invention discloses an intelligent maintenance auxiliary system for airborne equipment. The intelligent maintenance auxiliary system comprises an interaction module, an image detection and recognition module, a maintenance information management module, an analysis module and an augmented reality display module, The interaction module is used for obtaining input of a user, projection of a UI and interaction logic processing; The image detection and recognition module is used for detecting characters and specific patterns on the picture and recognizing content information; The analysis module is used for carrying out logic judgment and generating guided characters or 3D images; The maintenance information management module is used for sending the search request to the cloud database server; And the augmented reality display module is used for positioning a target position in a real environment in the field of view of the user and superposing the guidance information to the positionthrough an augmented reality technology. An augmented reality technology and intelligent maintenance assistance are organically combined, maintenance personnel are helped and guided to maintain airborne equipment in a natural interaction mode, and therefore operability and learnability of maintenance operation are improved.

The invention discloses a HoloLens-based command post cooperative work electronic sand table system. The system comprises a hardware supporting layer, a resource management layer, an application supporting layer and a service function layer; the hardware supporting layer collects interactive information of a commander through hardware in the system and transmits the interactive information to the resource management layer; the resource management layer manages resource data in the system and provides system management, resource control and data access interfaces for the application supporting layer; the application supporting layer achieves holographic image calculation and provides application development and operation environment supporting for the service function layer; the service function layer achieves holographic situation display, multi-role fight person interaction, natural interaction, cooperative plotting, discussion process control and holographic image generation. The system supports whole scene holographic display and military science element holographic display of an electronic sand table, the commander performs situation plotting operation in the cooperative work environment of the holographic electronic sand table, and role interaction between commanders inside the command post is supported; environment and situation are collected for performing role interaction with fight persons.

The invention provides an AR-oriented navigation type interactive paradigm system. The system comprises an input and perception module, a fusion module and an application expression module; the inputand sensing module obtains hand skeleton depth information through kinect to obtain a gesture depth map, and gesture recognition is completed; t experimental keywords are classified through voice keyword recognition and extraction, andthe similarity probability is obtained through similarity calculation to complete voice recognition; the fusion module classifies user intentions by calculating signal perception of different states in different modes according to gesture recognition, voice recognition and force feedback expression, determines multi-mode intentions and sets an intention expert knowledge base; and the expression module is used for carrying out voice navigation and visual presentation on the virtual chemical experiment and carrying out perceptual force feedback expression. Gesture and voice multi-modes are fused by adopting a natural interaction technology, and a natural interaction mode is explored, so that the user load is reduced, and different user experiences are brought.