79 results about "Auditory information" patented technology

The invention provides an industrial robot autonomous collaborative decision-making method and system based on multi-modal sensing and a medium, and belongs to the technical field of industrial robot intelligent control. The method comprises the steps of performing cross-modal space-time alignment to eliminate data space-time differences by collecting visual, tactile and auditory information, realizing multi-modal feature fusion in combination with a dynamic weight adjustment mechanism and an attention calculation model, and generating a joint decision strategy through a deep learning optimization model. The system dynamically allocates sensor weights according to task types, introduces a multi-objective optimization mechanism of energy consumption, precision and safety, and sets a fault-tolerant rule to automatically recover the weights or recalibrate the sensors. According to the method, the problems of rigid data fusion and single optimization dimension in traditional multi-modal decision making are solved, and the precision, the response speed and the environmental adaptability of collaborative operation of the industrial robot are remarkably improved.

The invention provides a multi-mode sensing intelligent microphone array signal processing method and system, and belongs to the technical field of signal processing, and the method comprises the steps: obtaining a visual signal through employing a multi-directional visual sensor, and obtaining a sound signal through employing a microphone array; extracting visual features and acoustic features; constructing an audio-visual topological feature space, mapping visual features and acoustic features to the space, and establishing a sound source probability distribution model; processing the sound signal by adopting a multi-dimensional discriminant adversarial generative network, and separating out a target voice signal; the acoustic environment state is evaluated in real time, and processing parameters are dynamically adjusted; quality evaluation is carried out on the separated multiple paths of target voice signals, the voice signal with the highest quality is selected as output, audio-visual multi-mode information is deeply fused and cooperatively processed, and a topology enhanced adversarial generative network architecture and an environment self-adaptive mechanism are combined, so that the voice separation effect in a complex environment is remarkably improved; and more than 85% of speech intelligibility can still be kept in a scene that six persons speak simultaneously.

The invention discloses a self-adaptive noise reduction method for a positive pressure type air respirator based on airflow spectrum characteristics, and relates to the technical field of positive pressure type air respirators.The self-adaptive noise reduction method comprises the steps that a respirator air path pressure signal and an environment noise signal are synchronously collected through an airflow sensor and a microphone array; performing time-frequency analysis on the collected signals, and extracting airflow spectrum features; and establishing a mapping relation between the working state of the breathing machine and the noise spectrum based on the historical record according to the mapping relation between the working state of the breathing machine and the noise spectrum. Through real-time acquisition and analysis of airflow frequency spectrum characteristics, accurate identification of useful airflow signals and environmental noise signals, dynamic adjustment of noise reduction parameters, effective suppression of high-frequency noise and low-frequency vibration, and improvement of noise reduction effect, compared with a traditional fixed frequency band filtering or passive sound insulation material, the method can better cope with a non-stationary noise environment, and has good application prospects. The noise reduction strategy is adjusted in real time, clear auditory information is provided in a complex noise environment, and therefore the auditory perception ability of firemen is improved.

The invention relates to the technical field of voice recognition, in particular to a campus bullying behavior detection method and system based on video phone voice recognition. By acquiring the sound signals and the continuous image frames in the monitoring area of the video phone, effective fusion of visual and auditory information is realized, so that abnormal conditions in the campus are more comprehensively captured and analyzed. The feature points of the figures in the image frames are recognized, the position changes of the figures are tracked, the aggregation feature values of the figures in the monitoring area are obtained in combination with the position distribution of the figures, and the value is of great significance to the recognition of potential bullying scenes. When the aggregated feature value is abnormal, further analyzing the sound signal, dividing sound data segments, extracting acoustic features, and calculating an abnormal factor of each sound data segment; and finally, integrating the abnormal factors and the character aggregation characteristic values of all the sound data segments to obtain an abnormal characteristic value for monitoring the campus bullying behavior, thereby reducing the risk of missing report and false report.

The invention discloses a vehicle automatic driving auditory information semantic understanding learning method and device, and the method comprises the steps: collecting sound events in a virtual environment through a virtual audio collection module, obtaining a virtual audio data set, classifying the virtual audio data in the virtual audio data set, and generating a pseudo tag; acquiring scene information of the target virtual audio data, and taking the scene information as a scenarized semantic tag of the target virtual audio data; the virtual audio data set with the label is used for deep learning model training to obtain an audio analysis model; acquiring real-time audio data, and extracting audio features of the real-time audio data through a feature extraction layer of the audio analysis model; acquiring a context feature corresponding to the audio feature in the time sequence, and generating an audio time feature according to the target audio feature and the context feature; audio time features are analyzed through an output layer of the audio analysis model to obtain classification and scenes corresponding to real-time audio data, and the perception and understanding ability of the automatic driving vehicle to auditory information is improved.

The invention discloses an audio-visual fusion nerve component and an audio-visual fusion target detection method, and belongs to the technical field of intelligent identification. The audio-visual fusion nerve component is a photoelectric detector which takes a ferroelectric semiconductor bismuth-oxygen-selenium as a channel layer and is of a back gate structure, visual information is irradiated to a channel in the form of optical power, and a photoconductive effect is generated; auditory information is converted into a voltage signal to be transmitted to the gate electrode to trigger ferroelectric polarization of the channel, channel conductance is effectively regulated and controlled in the polarization direction, photon-generated carrier generation and transmission behaviors of the channel are further regulated and controlled, and therefore nonlinear audio-visual fusion is achieved. Based on the hyper-additive property, the anti-effectiveness and the space-time fusion principle of the audio-visual fusion, reliability-enhanced audio-visual fusion target identification can be effectively carried out, especially detection of continuous fuzzy targets, including but not limited to vehicle orientation detection and vehicle type identification.

To enable installing, at appropriate positions, respective sensors for acquiring various kinds of information necessary for controlling a robot in a multimodal manner; and to acquire information from the respective sensors in an appropriate state and at appropriate timing.SOLUTION: In a hand tool 50, sensors having respective different information acquisition functions are attached to respective finger parts 22A, 22B, 22C, 22D, and 22E. In other words, sensors for acquiring various kinds of information necessary for controlling a robot are attached. A visual sensor 51A for detecting visual information is attached to a thumb, an auditory sensor 51B for detecting auditory information is attached to an index finger, an olfactory sensor 51C is attached to a middle finger, a tactile sensor 51D for detecting tactile information is attached to a ring finger, and a taste sensor 51E for detecting taste information is attached to a little finger.SELECTED DRAWING: Figure 4

The invention discloses an audiovisual navigation method for carrying out audiovisual mode fusion based on a converter model. The method comprises the following steps: collecting visual information and auditory information; the expected discount return and reward are maximized; two encoders are respectively used for carrying out feature extraction on audio-visual sense organs; performing feature mapping by using a spatial audio encoder; fusing the information of the visual mode and the auditory mode; weighting a strategy for successfully measuring the task that the sound source is a mobile sound source by using the dynamic path length; and training the network by using a near-end strategy optimization algorithm. According to the method, visual and auditory input can be better coordinated, so that audio and visual modes can be better fused, and the accuracy and precision of navigation are improved. The converter model is used for replacing a traditional gating circulation unit, and the high-difficulty navigation requirement in a noisy environment and a dynamic sound source task can be met. The near-end strategy optimization algorithm is used for training, manpower is saved, and economic benefits are improved.

To provide an information processing system, an information processing method, and a program for evaluating the degree of deterioration in cognitive function with good accuracy.SOLUTION: Provided is an information processing system in which an information processing device, a medical person terminal, and a sound collection device are communicable through a telecommunication circuit, wherein the information processing device includes a processor capable of executing a program so that each step is carried out. The processor includes: a presentation unit for presenting an indication for prompting a user to utter a prescribed speech to the user by auditory information; a reception unit for receiving speech data related to the uttered speech from the user; and an evaluation unit for evaluating the existence or degree of deterioration of a cognitive function of the user, on the basis of the speech data and a preset trained model. The trained model is a model obtained by preliminarily learning a relation between a feature amount of the prescribed uttered speech and the existence or degree of deterioration of the cognitive function of the user.SELECTED DRAWING: Figure 4

The invention provides a visual and auditory cross-modal object material retrieval method and system, and relates to the technical field of artificial intelligence. The visual and auditory cross-modal object material retrieval system comprises a data acquisition unit for acquiring visual images containing objects made of various materials and corresponding audio data; feature extraction: feature extraction is performed on the visual and auditory data, visual features include but are not limited to colors, textures and shapes, and auditory features include but are not limited to spectrum features and time domain features; feature fusion: designing a fusion algorithm, and effectively fusing visual and auditory features to form cross-modal feature representation; and retrieving a model, wherein the model can learn a mapping relationship between the cross-modal features and the material of the object. By combining visual and auditory information, the system can more comprehensively understand the characteristics of the object, the visual information can provide the appearance characteristics of the object, the auditory information can provide the dynamic characteristics about the material of the object, and the fusion is beneficial to improving the accuracy of material identification.

The invention discloses a behavior data analysis method and system based on intelligent perception. The method and system are used for improving the reliability of child behavior analysis. The method comprises the following steps: acquiring multi-dimensional monitoring data of a plurality of children in a monitoring environment, wherein the multi-dimensional monitoring data comprises visual information, auditory information and physiological information of each child; performing multi-source fusion on the visual information, the auditory information and the physiological information of each child to generate a behavior feature set; inputting the behavior feature set into a preset behavior and emotion classification model for classification to obtain a behavior mode and an emotion state of each child; on the basis of individual information of different children, the behavior modes and the emotional states in combination with scene information of the monitoring environment, knowledge graph construction on a time sequence is carried out, a children behavior graph is obtained, and the children behavior graph is used for displaying behavior events of the children in a preset time period.

The invention discloses an audio visualization method, a vehicle, a storage medium and a computer program product, and relates to the technical field of vehicles, the audio visualization method is applied to the vehicle, the vehicle comprises a first display screen, and the method specifically comprises the following steps: obtaining a plurality of real-time driving parameters of the vehicle; generating a plurality of first audio signals based on the plurality of real-time driving parameters, and determining first animation effects respectively matched with the plurality of first audio signals; first display areas, matched with the first animation effects, on the first display screen are determined; and controlling the loudspeaker to play the first audio signal, and controlling each first display area to play the corresponding first animation effect. By adopting the method and the device, a driver can more intuitively and comprehensively know the change condition generated in the driving process of the vehicle through visual and auditory information, so that the driver has more immersive driving experience.

A communication robot includes an auditory information processing portion configured to recognize a volume of voice collected by a sound collection portion and generate an auditory attention map by projecting a sound position in a three-dimensional space onto a two-dimensional attention map in which the robot is located at a center, a visual information processing portion configured to generate a visual attention map using a face detection result obtained by detecting a face of a person from an image captured by an imaging portion and a motion detection result obtained by detecting a motion of the person, an attention map generation portion configured to generate an attention map by integrating the auditory attention map and the visual attention map, and a motion processing portion configured to control eyeball movements and motions of the communication robot using the attention map.

The present invention discloses a method and device for fusing visual and auditory information in vehicle autonomous driving, including: acquiring visual information and auditory information collected by the vehicle; the auditory information includes the sound source direction; projecting the auditory information into the coordinate system of the visual information; fusing the visual information and the auditory information in the same coordinate system through an attention mechanism to obtain a fused feature; projecting the auditory information into the visual coordinate system of the visual information includes: acquiring the internal parameter matrix of the vehicle camera; obtaining the distance value from the vehicle camera to the sound source according to the sound source direction; converting the sound source direction into the coordinates in the visual coordinate system according to the internal parameter matrix and the distance value. After acquiring the visual information and the auditory information of the vehicle, by projecting the auditory information into the coordinate system of the visual information, the auditory information and the visual information can be effectively fused in the same coordinate system, so as to improve the perception accuracy and robustness of the vehicle for complex scenarios through the fused feature for target detection and positioning.

The invention discloses a brain-computer interface device and method for relieving pain based on VR music. The method comprises the steps that a user electroencephalogram signal set is acquired; the user electroencephalogram signal set comprises a user electroencephalogram signal sequence; performing classification identification processing on the user electroencephalogram signal set to obtain user feature state information; and generating corresponding auditory information and visual information according to the user feature state information. The VR technology and the brain-computer interface technology are combined, an innovative method and device are provided for pain relief, and the pain relief effect is improved; the VR music is recommended in a personalized manner according to the real-time electroencephalogram signal of the user, so that the requirements of the user can be better met, and the user experience is improved. A non-drug relieving mode is adopted, and the risk of side effects of drugs is reduced.

The application provides a multi-modal perception intelligent microphone array signal processing method and system, and belongs to the technical field of signal processing, which comprises the following steps: acquiring visual signals by using multi-directional visual sensors and acquiring sound signals by using microphone arrays; extracting visual features and acoustic features; constructing a visual-auditory topological feature space, mapping the visual features and acoustic features to the space, and establishing a sound source probability distribution model; processing the sound signals by using a multi-dimensional discriminative generative adversarial network to separate target speech signals; real-time evaluating acoustic environment states, dynamically adjusting processing parameters; evaluating the quality of the separated multi-path target speech signals, selecting the highest quality speech signal as the output, and deeply fusing and cooperatively processing multi-modal visual-auditory information, combining a topological enhanced generative adversarial network architecture and an environment adaptive mechanism, and significantly improving the speech separation effect in a complex environment, which can still maintain a speech intelligibility of more than 85% in a 6-person simultaneous speaking scene.

The invention discloses a novel fire safety evacuation guiding method. The method comprises the following steps: 1, displaying a real plane graph of a place on a display device by using visual and auditory information, and transmitting the real plane graph to a customer; 2, dividing the place into a plurality of small evacuation areas after calculation according to the number of the emergency exits on the real plane graph, the evacuation distance and the evacuation width; 3, displaying the dynamic evacuation route in each evacuation area to the customer, replacing the personnel model with a specific symbol, and moving from each evacuation area to a safe position along the demonstrated evacuation route; and 4, displaying the dynamic evacuation route in a voice prompt mode. The invention belongs to the technical field of fire-fighting evacuation, and particularly relates to a novel fire-fighting safety evacuation guiding method, which is based on a fire-fighting theory, automatically displays a real plane graph of a place on an emergency exit of electronic equipment in an image or video form, and carries out warning prompt to assist personnel to escape nearby.

The invention relates to the field of audio-visual conversion, and provides intelligent audio-visual auxiliary glasses, an audio-visual auxiliary method, electronic equipment and a storage medium, and the intelligent audio-visual auxiliary glasses comprise a glasses frame which comprises a microphone and lenses; the character conversion unit is used for receiving external sound information through the microphone, converting the sound information into characters and displaying the characters on the lens; and the voice conversion unit is used for receiving external visual information through the visual sensor, converting the visual information into voice and playing the voice. The invention provides an integrated audio-visual conversion device, which is used for solving the defect of lack of audio-visual conversion devices for visually impaired people and hearing impaired people in the prior art, can help the visually impaired people to obtain front visual information and can also help the hearing impaired people to obtain sound information, so that the audio-visual conversion effect is improved. And bidirectional enhancement of visual information and auditory information is realized.

The present application provides an autonomous collaborative decision-making method, system and medium for industrial robots based on multimodal perception, which belongs to the field of intelligent control technology for industrial robots. The method includes: collecting visual, tactile and auditory information, performing cross-modal spatiotemporal alignment to eliminate spatiotemporal differences in data, combining a dynamic weight adjustment mechanism and an attention calculation model to achieve multimodal feature fusion, and generating a joint decision-making strategy through a deep learning optimization model. The system dynamically allocates sensor weights according to the task type, and introduces a multi-objective optimization mechanism for energy consumption, accuracy and safety, while setting fault-tolerant rules to automatically restore weights or recalibrate sensors. The present invention solves the problems of rigid data fusion and single optimization dimension in traditional multimodal decision-making, and significantly improves the accuracy, response speed and environmental adaptability of collaborative operations of industrial robots.

We provide the system. [Solution] A device for analyzing visual and auditory information collected in the work environment, A device that identifies tasks that can be automated based on the aforementioned analysis, A device that proposes an optimal automation method for the aforementioned task, A system that includes this.

The invention provides a communication method and system for AR glasses with separated earphones and glasses, and a medium, and the method comprises the steps: initializing the AR glasses, and obtaining the initialization state information; scanning compatible state information between the AR glasses and the separated audio equipment based on the initialized state information; establishing communication state information between the AR glasses and the separated audio equipment based on the compatible state information; performing communication matching authentication based on the communication state information between the AR glasses and the separated audio equipment to obtain an authentication result; analyzing whether the communication is successful or not based on the authentication result, if the communication is successful, outputting audio information and instruction response, and if the communication is failed, re-pairing the separated audio equipment; communication matching is completed by analyzing the communication state of the AR glasses and the separated audio equipment, so that successful communication is realized, cooperative transmission of visual information at a glasses end and auditory information at an earphone end is realized, and synchronism of audio and pictures is ensured.

An object of a system according to an embodiment is to perform information presentation utilizing multiple senses using generated AI.SOLUTION: A system includes a visual information presentation part, an auditory information presentation part, a tactile information presentation part, a taste information presentation part, and an olfactory information presentation part. The visual information presenting unit presents visual information by using the generated AI. The audio information presenting unit presents audio information by using the generated AI. The tactile information presenting unit presents tactile information by using the generated AI. The taste information presenter presents taste information by using the generated AI. The olfactory information presenting unit presents olfactory information by using the generated AI.SELECTED DRAWING: Figure 1

This electronic device comprises: an input / output interface that receives uncompressed digital image content; a communication device that wirelessly transmits and receives data; at least one memory storing instructions; and at least one processor that obtains packet data corresponding to the digital image content and controls the communication device to transmit the packet data. The at least one processor executes the instructions to: separate video information, audio information, and meta information in the digital image content; in order to obtain encoded video information, perform encoding in units of preset blocks by using video information in units of preset lines among the video information; in order to obtain packet data, packetize each of the encoded video information, the audio information, and the meta information; and control the communication device to wirelessly transmit the packet data to a display device.

The application discloses a teaching equipment control method, a control device, a teaching system and a storage medium. The teaching equipment control method comprises the following steps: image and audio of a target in a teaching space are collected to obtain image data and audio data of the target, wherein the teaching space comprises the teaching equipment; visual information of the target is extracted by using the image data of the target, and auditory information of the target is extracted by using the audio data of the target; and the teaching equipment is controlled based on the visual information and the auditory information of the target. In the foregoing manner, the convenience of the teaching equipment control can be improved, and the accuracy is relatively high.

The present invention relates to a cloud-edge-end collaborative hearing aid inference method with controllable multi-modal perception streams. First, GRU is used for audiovisual event recognition, visual and auditory features are extracted respectively, and under the condition of controlling the visual features, a recurrent neural network is used for audiovisual event recognition. At the decision-making layer, the results of audiovisual information are fused by means of dynamic weighting. According to the influence of visual perception amount on the recognition accuracy and the influence of the change of audiovisual information weight on the final result, two parameters meeting the requirements are determined according to the actual situation. Secondly, cross-modal data generation is carried out based on DCGAN, visual and auditory features are extracted, and a part of visual features are fused with auditory features. Under the condition of controlling the visual information perception amount for fusion, a generative adversarial network is used for cross-modal data generation. According to the evaluation method of the performance of the generative adversarial network and the influence of visual information perception amount on the quality of the generated samples, parameters meeting the actual requirements are determined.

We provide the system. [Solution] A device that performs information processing to generate an individually optimized learning plan based on the learner's abilities, progress, and goals, A device that provides educational materials based on a generated learning plan, A device that analyzes learners' activities with educational materials in real time and generates evaluation results, A device that analyzes a learner's past learning history to identify learning challenges and provides additional exercises to improve those challenges, A sensor that acquires sound and video, and a device that uses the acquired data to perform interactive speech practice in real time, A device that provides immediate instruction using visual and auditory information based on speech evaluation data, A system that includes this.

A computer-implemented method of providing information to a user is provided. The method comprises receiving input of a selected operating mode and extracting, via an extraction pipeline, data from a number of data sources according to the selected operating mode. The extracted data is fed into a large language model (LLM), and the LLM generates verbal and auditory information for the user based on the data. The LLM conveys the verbal and auditory information to the user via a number of interface products according to the selected operating mode.

The hand tool 50 is equipped with sensor groups with different information acquisition functions on each of the fingers 22A, 22B, 22C, 22D, and 22E. In other words, it is equipped with sensor groups for acquiring various information required for multimodal robot control. A visual sensor 51A for detecting visual information is installed on the thumb, an auditory sensor 51B for detecting auditory information is installed on the index finger, an olfactory sensor 51C is installed on the middle finger, a tactile sensor 51D for detecting tactile information is installed on the ring finger, and a taste sensor 51E for detecting taste information is installed on the little finger.

This vehicle operation device (100) comprises: an operation panel (21) on which a plurality of operation buttons (22) are displayed; a touch detection unit (30) which detects, as a touch operation, contact with any of the plurality of operation buttons (22); an output unit (50) which can output visual information, tactile information, and auditory information; and a control unit (10) which, in response to the detection of the touch operation by the touch detection unit (30), causes the output unit to output the visual information, tactile information, and auditory information as feedback information for the touch operation. The control unit (10) causes the output unit (50) to output, for each of the operation buttons (22), feedback information in which at least one of the visual information, tactile information, and auditory information is different.

The invention discloses a vehicle automatic driving visual and auditory information fusion method and device. The method comprises the steps of obtaining visual information and auditory information collected by a vehicle; the auditory information comprises a sound source direction; projecting the auditory information into a coordinate system of the visual information; fusing the visual information and the auditory information in the same coordinate system through an attention mechanism to obtain fused features; projection of the auditory information to a visual coordinate system of the visual information includes: acquiring an internal reference matrix of a vehicle camera; obtaining a distance value from the vehicle camera to the sound source according to the sound source direction; and converting the sound source direction into coordinates in a visual coordinate system according to the internal reference matrix and the distance value. After the visual information and the auditory information of the vehicle are obtained, the auditory information is projected into the coordinate system of the visual information, so that the auditory information and the visual information can be effectively fused in the same coordinate system, target detection and positioning are performed through the fused features, and the perception precision and robustness of the vehicle to a complex scene are improved.