Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

189 results about "Visual attention" patented technology

Visual language model illusion suppression method based on adaptive dynamic attention intervention

The invention discloses a visual language model illusion suppression method based on self-adaptive dynamic attention intervention. The method is used for reducing the problem that a visual language model generates wrong associated information. The method comprises the steps that text input, visual input and historical response are acquired, and an attention accumulation vector is initialized; in the layer-by-layer calculation process of the language model, dynamically adjusting a non-normalized attention matrix, enhancing the weight of a visual sensitive attention head, and performing visual Token pruning in a deep network to optimize cross-modal information interaction; and finally, generating an output Token based on the adjusted attention mechanism, and carrying out loop iteration until a complete response is generated. According to the method, a method of combining text deviation correction through self-adaptive attention head modification and visual attention convergence-based Token pruning is adopted, so that the performance of the model in a multi-modal task is remarkably improved, and the illusion phenomenon caused by a language modal dominant reasoning process is effectively relieved.
Owner:ZHEJIANG UNIV OF TECH

Thermal runaway prevention and heat dissipation optimization method and system based on capacitor module

The invention provides a thermal runaway prevention and heat dissipation optimization method and system based on a capacitor module, and relates to the technical field of capacitor module thermal management, and the method comprises the steps: extracting the multi-source operation data features of a single capacitor through an improved layered visual attention network, carrying out the data complementation through combining a bidirectional probability diffusion model, and obtaining a multi-source operation data feature of the single capacitor; and discovering a causal relationship between network identification data by using a causal structure, generating a performance analysis report, constructing a spatio-temporal dynamic graph neural network to predict temperature field distribution, performing fault diagnosis and uncertainty quantification in combination with a multi-task learning framework, further constructing a layered early warning mechanism, and performing early warning based on a layered reinforcement learning framework. And a dynamic prediction map and a performance analysis report are combined, a heat dissipation strategy is optimized, cooperative control of the heat dissipation units is realized, an optimal heat dissipation scheme is finally obtained, thermal runaway is effectively prevented, and the safety and reliability of the capacitor module are improved.
Owner:BEIJING RUIHE DEBAO THERMAL TECH CO LTD

AI tax auditing method based on multi-modal data and attention mechanism

The invention discloses an AI tax auditing method based on multi-modal data and an attention mechanism, and relates to the technical field of tax auditing, and the method comprises the steps: carrying out the preprocessing and input of multi-modal tax data, and carrying out the unified representation of the data; identifying risks and capturing attention weights based on an artificial intelligence auditing model of an attention mechanism; mapping the captured attention weight back to a corresponding data unit of the multi-mode tax auditing original data, and generating a visual attention auditing evidence chain; and analyzing the attention weight generated in the analysis process of the artificial intelligence auditing model to identify the attention distribution abnormity existing in the process of processing the specific data area or the evidence chain link, and generating auditing blind spot early warning information based on the identified attention distribution abnormity. According to the method, the interpretability and credibility of AI auditing can be improved, the auditing efficiency and quality are improved, and a novel man-machine cooperation mode is constructed.
Owner:BEIJING HUACHEN HONGYI CONSULTING CO LTD

Immersive audio-video follow-up adjustment method and system

The invention is applicable to the field of intelligent audio adjustment, and provides an immersive audio-video follow-up adjustment method and system, and the method comprises the steps: constructing a multi-dimensional perception system, and collecting multi-source information in real time; carrying out fusion processing on the collected multi-source information based on a deep neural network model, mining a dynamic mapping relation between a user state and the video content through space-time correlation analysis, and identifying a user interaction intention and an emotional tone and a space scene attribute of the video content; according to a fusion processing result, calling a dynamic parameter adjustment engine, and generating an audio parameter adjustment scheme in real time; a user experience feedback closed loop is constructed, a visual attention area of a user is collected through eye movement tracking equipment, and personalized adjustment preference parameters are generated; according to the method and the device, the audio effect is accurately matched with the user state and the audio and video content, the naturalness and the adaptability of immersive experience are remarkably improved, universality and individual differences are considered, and the audio experience which is more suitable for scenes and needs of the user is brought to the user.
Owner:SHENZHEN ZIDOO TECH CO LTD

Image processing method and system for LED special-shaped screen

The invention discloses an image processing method and system for an LED special-shaped screen, and belongs to the technical field of image processing, and the method comprises the steps: carrying out the semantic segmentation of an input image through a convolutional neural network, and generating a feature thermodynamic diagram containing visual attention scores; generating a dynamic vector grid on the screen physical topological structure to obtain image resolution adjustment data; performing phase compensation calculation on the bent part of the edge of the dynamic vector grid to obtain phase compensation data; environment parameters are monitored in real time, a brightness-color temperature compensation matrix is dynamically generated, and environment compensation data are obtained; and rendering by using the image resolution adjustment data, the phase compensation data and the environment compensation data to obtain a final image. According to the method, semantic segmentation is performed by using the convolutional neural network, the image resolution is dynamically adjusted on the LED special-shaped screen according to the visual attention score, and image quality optimization and energy efficiency improvement are realized in combination with phase compensation and environment monitoring.
Owner:SHENZHEN ENBON OPTOELECTRONIC CO LTD

Unmanned aerial vehicle visual guidance obstacle avoidance system and method based on polarization information

The invention discloses an unmanned aerial vehicle visual guidance obstacle avoidance system and method based on polarization information, and relates to the technical field of unmanned aerial vehicle navigation and obstacle avoidance. Polarization image information in a flight scene is collected in real time through a polarization sensor, and the perception capability of a complex environment is enhanced. Polarization parameters are extracted through a compression observation method, the efficiency and accuracy of feature extraction are improved, a visual attention mechanism is combined, an obstacle background area is accurately recognized, a self-adaptive decision fusion model is constructed, the feature weight is optimized, and the reliability of target contour features is enhanced. Preliminary path search is carried out based on target contour features, and a centralized optimization sampling method is adopted, so that the probability of generating an optimal flight path in a complex environment is effectively improved. And the obstacle avoidance flight module is used for smoothing the flight path, so that the smoothness and controllability of the flight path are ensured, and the navigation and obstacle avoidance capabilities of the unmanned aerial vehicle in a complex environment are effectively improved.
Owner:JIANGSU URBAN & RURAL CONSTR VOCATIONAL COLLEGE

Method and device for generating first view angle video based on mask diffusion model and fixation point constraint

The invention discloses a first view angle video generation method and device based on a mask diffusion model and fixation point constraint, and the method comprises the steps: constructing an end-to-end deep learning frame for the demands of video filling, prediction and visual attention region control in the generation of a first view angle video; the diversified first-view-angle video generation conforming to the visual logic is realized. The method comprises the following steps: firstly, dividing an input video into a condition frame set and an unknown frame set, and adding random noise to an unknown frame by using a dynamic mask module I to generate a noisized frame; performing reverse de-noising generation on the video through a full 3D convolutional neural network module II, and guiding the network to generate video content conforming to a space-time law by taking a diffusion step and a fixation point track as conditional constraints; and finally, saliency prediction is carried out by using a fixation point positioning module III. By jointly optimizing the loss of the reverse denoising generation process and the loss of the fixation point probability graph, the reasonability and diversity of the generated video are further improved. According to the method, through joint optimization of the mask diffusion strategy and fixation point constraint, the space-time continuity, the anti-noise capability and the adaptability to the fixation point trajectory of the generated video can be effectively improved, and the method is particularly suitable for a complex multi-face expression interaction scene.
Owner:CHINA UNIV OF MINING & TECH

Multi-camera imaging method and system for head-mounted device

The invention discloses a multi-camera imaging method and system for head-mounted equipment, and relates to the technical field of multi-camera imaging, and the system comprises an image data acquisition module, a weight acquisition module, an image fusion module, a feedback control module, a visual angle movement determination module and a preloading module. The gaze weight of the image is obtained by obtaining the gaze direction of the user, an image weighted fusion mode with the current visual attention area of the user as the core is achieved, the generated main view angle image better conforms to the actual view field of the user, and immersive visual experience is enhanced; inter-frame time sequence consistency loss is introduced in the fusion process, the pixel change of the fusion image between continuous frames is restrained through time sequence smoothing, the image flicker and jump phenomena are remarkably reduced, and the visual stability in a dynamic scene is improved.
Owner:JIANGXI UNIVERSITY OF FINANCE AND ECONOMICS

Multi-target license plate recognition method based on visual attention mechanism

The invention discloses a multi-target license plate recognition method based on a visual attention mechanism, and belongs to the technical field of image processing and mode recognition, and the method comprises the steps: obtaining an input image, carrying out the multi-scale feature extraction, and generating a multi-scale feature map; performing spatial saliency calculation and normalization processing on the multi-scale feature map to generate an attention map; carrying out region division on the input image, and adopting differentiated image preprocessing strategies for different regions to generate a preprocessed image; based on the preprocessed image and the attention map, multi-target detection is carried out through a target detection network, candidate area screening is carried out, and a candidate license plate area is generated; and performing binarization processing, character segmentation and feature recognition on the candidate license plate region, and performing verification in combination with context information to generate a license plate recognition result. The attention map is generated by adopting a visual attention mechanism, differentiated image preprocessing and multi-target detection are guided according to the attention map, and multi-target license plate recognition can be completed in a complex scene.
Owner:SHENZHEN BOTE TECH CO LTD

Text-conditioned visual attention for multimodal machine learning models

The present disclosure relates to systems, non-transitory computer-readable media, and methods for conditioning images on modification texts to generate multi-modal gradient attention maps. In particular, in some embodiments, the disclosed systems generate, utilizing a vision-language neural network of an image-text comparison machine learning model, a reference text-image feature vector based on a reference image and a modification text. Additionally, in some embodiments, the disclosed systems generate, utilizing the vision-language neural network of the image-text comparison machine learning model, a target text-image feature vector based on a target image and the modification text. Moreover, in some implementations, the disclosed systems generate, from the reference text-image feature vector and the target text-image feature vector, a multi-modal gradient attention map reflecting a visual grounding of the image-text comparison machine learning model relative to the modification text.
Owner:ADOBE INC

Vehicle systems and methods for autonomous operation using driver attention

Vehicles and related systems and methods are provided for controlling a vehicle in an autonomous operating mode. One method involves a controller associated with a vehicle identifying a visual attention state associated with a driver of the vehicle based at least in part on output of an imaging device onboard the vehicle, determining, based at least in part on the visual attention state, a driver lane preference corresponding to an adjacent lane in a visual attention direction relative to a current lane of travel for the vehicle, adjusting a priority associated with the adjacent lane corresponding to the driver lane preference and autonomously operating one or more actuators onboard the vehicle to initiate maneuvering the vehicle from the current lane in a manner that is influenced by the adjusted priority associated with the adjacent lane.
Owner:GM GLOBAL TECHNOLOGY OPERATIONS LLC

Eye movement tracking-based closed cabin human eye perception evaluation method

A closed cabin human eye perception evaluation method based on eye movement tracking belongs to the technical field of computer vision, and comprises the following steps: collecting eye movement data of simulation personnel in a specific simulation environment, analyzing the eye movement data, and analyzing and calculating the reaction time and the reaction accuracy of the simulation personnel to a moving target; the visual attention distribution and information processing capability of simulation personnel in a specific task situation can be revealed, so that the decision-making efficiency and the response capability of the simulation personnel in a complex dynamic environment are reflected; the continuity of the spliced screen is calculated, and the fluency degree of the connection effect at the spliced position of the screen is reflected; the eye movement focusing degree is analyzed, the attention level of simulation personnel on a target area can be quantified, whether visual attention is concentrated in a key information area or not is helped to be judged, multiple indexes are input into a regression model, and a comprehensive human eye perception score is obtained. The method has the characteristic of human eye perception evaluation for multi-screen splicing, and a comprehensive, accurate and easy-to-operate evaluation tool is provided.
Owner:JILIN UNIVERSITY +1

Multi-resolution fusion image recognition method based on human vision mode

The invention relates to the technical field of image recognition, in particular to a multi-resolution fusion image recognition method based on a human vision mode, and the method comprises the steps: obtaining a target image through image collection equipment, carrying out the preprocessing of the image, carrying out the processing of the preprocessed image according to different resolutions, and generating images of a plurality of resolutions; the method comprises the following steps of: extracting features of images with different resolutions through a convolutional neural network, performing weighted fusion on the extracted multi-resolution image features, and then introducing a human visual attention mechanism to optimize the fused features, thereby improving the precision of image recognition. And finally, inputting the optimized multi-resolution features into a support vector machine model, performing image recognition and classification, further removing noise and enhancing classification confidence through a post-processing step, and outputting a final image recognition result. According to the method, the image recognition precision in a complex scene can be effectively improved, and the method has relatively high robustness and is suitable for various image classification tasks.
Owner:FIRST INSTITUTE OF OCEANOGRAPHY MNR +1

Cross-view image matching method and system fused with eye movement attention mechanism

The invention relates to the technical field of remote sensing image processing in computer vision, in particular to a cross-view image matching method and system fused with an eye movement attention mechanism, and the method comprises the steps: training two independent Transfomer encoder branches through a ground-remote sensing cross-view image data set; collecting eye movement track data when a human executes a cross-view image matching task, and generating a human visual attention map; performing image-level fusion on the remote sensing human visual attention map and the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder through a mixed attention module to obtain a mixed attention map; processing the original remote sensing image and the mixed attention map through an image adaptive cutting and resolution enhancement module to obtain a high-resolution remote sensing image; and inputting the high-resolution remote sensing image and the ground image into two pre-trained independent Transfomer encoder branches for secondary training. The attention of the guiding model is more focused on the image key area beneficial to matching, so that the matching precision of the ground image and the remote sensing image is improved.
Owner:Chinese People's Liberation Army Cyberspace Force Information Engineering University

Geochemical and remote sensing image fusion method, system and device and storage medium

The invention relates to the technical field of mathematical geology and geological big data, and discloses a geochemistry and remote sensing image fusion method, system and device and a storage medium, and the method comprises the steps: constructing a WVTNet model comprising a wavelet transformation module and a feature weight distribution module, and carrying out the training; wherein the wavelet transformation module is used for performing multi-scale image decomposition on geochemical data and a multispectral remote sensing image of a sample region to obtain a first image local feature and a second image local feature of different scales; the feature weight distribution module is used for acquiring key features in the plurality of first image local features and the plurality of second image local features based on a visual attention mechanism and fusing the key features to obtain a sample fusion image; and inputting the target geochemical data and the target multispectral remote sensing image of the target area into the trained WVTNet model to obtain a target fusion image. According to the method, the reliability and the information density of the fused data are improved, so that subsequent data analysis work is facilitated.
Owner:TECH & ENG CENT FOR SPACE UTILIZATION CHINESE ACAD OF SCI

Power consumption optimization control system for high-refresh-rate driving chip

The invention relates to the technical field of display system driving, and discloses a high-refresh-rate driving chip power consumption optimization control system which comprises an image content complexity modeling module for extracting region sparsity features, a visual attention prediction module for generating an attention heat map, and a display module for displaying the attention heat map. The combined graph modeling and optimization module fuses the information to construct a graph structure and predicts a refresh level, the refresh strategy generation module generates an optimal refresh strategy based on the level, the soft and hard translation execution module converts the strategy into a driving instruction and issues and executes the driving instruction, and the feedback scheduling and self-adaptive correction module dynamically adjusts the graph structure according to chip feedback. The method is used for optimizing next frame refresh control. According to the method, through region division and graph structure modeling, image region refreshing demand modeling is carried out in combination with an attention mechanism and loss feedback, differential refreshing is realized, invalid refreshing of a low-change region is effectively avoided, the content perception ability of a refreshing strategy is improved, and thus the overall power consumption is reduced.
Owner:SHENZHEN HISTONE OPTOELECTRONICS TECH CO LTD

WebM protocol low-delay video and audio translation and subtitle optimization method and system

The invention discloses a WebM protocol low-delay video and audio translation and subtitle optimization method and system, and belongs to the technical field of data processing, and the method comprises the steps: analyzing a WebM audio and video stream; establishing a video track-audio track association list and constructing a dynamic causal map; generating an initial translation text based on the audio frame sequence, the video frame sequence and the dynamic causal atlas, extracting lip movement and action semantic data of video frames through a reverse generation model to generate a completion translation text, and embedding SimpleBlock elements; defining a subtitle core gene, a non-core gene and a position gene according to the code rate data, the complete translation text and a user visual attention thermodynamic diagram, dynamically cutting the non-core gene based on code rate fluctuation, and adjusting the position gene in combination with the thermodynamic diagram to generate adaptive subtitle data; by synchronously playing and collecting feedback data, the edge weight and the subtitle position gene of the dynamic causal atlas are optimized. According to the invention, the robustness of audio and video translation and the self-adaptability of subtitle display are realized.
Owner:JIANGSU ZHIMENG INTELLIGENT TECH CO LTD

AR scenic area guide method and system based on positioning

The invention relates to the technical field of AR preview, in particular to a positioning-based AR scenic area guide method and system, and the method comprises the steps: obtaining the real-time three-dimensional position of a tourist through a Google ARCore fusion positioning engine based on a preset scenic area three-dimensional model and a space reference coordinate system; carrying out visual anchor point dynamic matching according to the real-time three-dimensional position, and identifying a current scene area corresponding to the current position of the tourist; performing virtual content scheduling and space rendering according to the current scene area, and generating space rendering content; and obtaining a visual attention behavior of a tourist on the space rendering content, screening out tourist-interested guide content according to the visual attention behavior, and generating a visual guide path according to the tourist-interested guide content. According to the method, by adjusting the loading and scheduling strategy of the virtual content, the requirements of personalized content display and smooth space guidance are met, and the experience feeling of tourists is enhanced.
Owner:湖北云雷信息技术有限公司

Enterprise multi-modal data fusion visualization method and system

The invention relates to the technical field of data processing, in particular to an enterprise multi-modal data fusion visualization method and system, and the method comprises the steps: obtaining information data of each dimension under each type of business data in enterprise multi-modal data; obtaining an association feature factor between each dimension and other dimensions under each type of business data, and integrating the called times of the information data and the time information responded by the called request to obtain a visual attention degree of each dimension under each type of business data; obtaining the degree of suction of different visualization modes in the historical record, and obtaining the degree of membership according to the information data of each dimension under each kind of business data contained in each visualization mode in the historical record; combining the visual attention degree, the membership degree and the suction degree to obtain the priority degree of the information data for each visual mode; and determining an optimal visualization mode of the information data according to the priority degree. According to the method, the visualization mode with a relatively good enterprise data visualization effect can be adaptively determined.
Owner:XINJIANG DUYI HUANQIU TECH CO LTD

Air-ground pedestrian re-identification method combining multi-frame information and prompt learning

The invention discloses an air-ground pedestrian re-identification method combining multi-frame information and prompt learning, and the method comprises the following steps: inputting a video sequence into a trained visual encoder model, mapping the same pedestrian at different visual angles into a consistent feature space, and achieving the cross-visual-angle pedestrian re-identification and tracking; according to the visual encoder model, a random rotation transformation strategy of structure perception is introduced in the input embedding stage of a visual encoder backbone network, pedestrian vector features of each frame are rotated, visual angle rotation distortion generated by aerial shooting is simulated, and an enhanced sequence is generated. Extracting global features of the enhanced sequence and the unenhanced sequence through a backbone network; inputting the global feature into an inter-frame information attention module for time dimension attention calculation to obtain an average feature of multi-frame fusion; and then inputting the multi-frame fusion average features into a prompting and guiding visual attention module to generate a text prompt so as to guide model discriminative character features.
Owner:SOUTH CHINA UNIV OF TECH

Intelligent structured medical record generation method and system based on multi-modal doctor-patient interaction

The invention provides an intelligent structured medical record generation method and system based on multi-modal doctor-patient interaction, and the method comprises the steps: collecting dialogue voice in real time, transcribing the dialogue voice into a text sequence, and recognizing a visual attention entity through monitoring the operation of a mouse in an electronic medical record system; a logic demonstration track is constructed based on a historical visual attention entity and a text sequence, and an implicit reward function is reversely derived from the logic demonstration track by using a reverse reinforcement learning algorithm. The function is used for calculating the action return value of each combination of the visual attention entity and the dialogue text, and the highest value combination is selected as an optimal alignment strategy to determine the time sequence causal relationship. And mapping the entity and the text to a medical knowledge graph, extracting a shortest semantic path as an implicit clinical reasoning chain, and generating a structured electronic medical record. According to the method, diagnosis and treatment decision logic is deduced from multi-modal behaviors of doctors through reverse reinforcement learning, and the technical problem that internal causal association between visual attention focuses of doctors and oral contents cannot be established in a traditional method is solved.
Owner:WUHAN SHENGBOHUI INFORMATION TECH CO LTD +1

Map data generation device

A point which requires visual attention is added to map data. A map data generation device acquires image data in which the outside is captured from a vehicle by an input device and a point data of the vehicle, associates both data, and generates a visual saliency map acquired by estimating fluctuation of visual saliency based on the image data by the visual saliency extraction unit. Then, whether or not a point or a section indicated by position information corresponding to the visual saliency map is a point or a section which requires visual attention is analyzed based on the visual saliency map by an analysis unit, and the point or the section which requires the visual attention is added to the map data based on an analysis result of the analysis unit by an addition device.
Owner:PIONEER IP

Real-time eye identification and visual detection process control method

The invention relates to a real-time eye identification and visual detection process control method, which comprises the following steps of: applying bitamporal edge disturbance stimulation in a visual field, inducing spontaneous dominant reaction of a dominant eye of a user, collecting perceptual response difference of a non-gazing area to the disturbance stimulation, and judging the current dominant state of the dominant eye; on the basis of micro-differential pressure distribution of different quadrant areas of eye sockets, speculating a visual attention gravity center in real time, and executing spatial geometric deformation on an ROI (Region of Interest) in a visual detection process, so that a detection area dynamically adapts to a gaze offset trend of a user; the method comprises the following steps: respectively constructing two physically independent visual detection links of a feature target strong discrimination process driven by a main visual eye and an environment contour weak confirmation process driven by an auxiliary visual eye, introducing a logic thermal impedance adjustment mechanism, and setting a buffer time window to absorb processing fluctuation caused by state switching; and dynamically matching a visual detection flow path by constructing a mapping relationship between the eye dominant frequency and the user intention type on the basis of statistical information of the dominant eye identification frequency.
Owner:AIR FORCE MEDICAL CENT PLA

Video image compression algorithm based on H.266 / VVC standard

The invention relates to the technical field of video image compression, in particular to a video image compression algorithm based on an H.266 / VVC standard, which comprises the steps of generating a visual attention thermodynamic diagram through time-space domain saliency detection, adjusting a coding tree unit QP value in a partitioned manner to realize adaptive quantization, and designing an asymmetric quantization matrix to optimize a transformation quantization effect. And the generative adversarial network is used to enhance the quality of the reconstructed frame. According to the method, the blocking effect of a complex texture region can be effectively inhibited, the quantization distortion of a human eye sensitive region is reduced, the edge preserving capability is improved, the subjective visual quality and compression efficiency of the video are remarkably improved, and the high-definition video transmission requirement is met.
Owner:HANGZHOU ZHILING TECHNOLOGY CO LTD

Complex scene indication expression understanding method based on cross-modal eye movement attention perception

The invention discloses a complex scene indication expression understanding method based on cross-modal eye movement attention perception, and belongs to the field of computer vision, machine learning and multi-modal understanding. According to the method, a dynamic deformable attention mechanism perceived by a language is designed, a human eye gaze spectrum is used as supervision information, a corresponding visual area is adaptively captured according to language features, meanwhile, a Transform decoder driven by an eye movement spectrum is designed, and the position of a target area indicated by the language is deduced by gradually fusing visual feature representation, so that the target area can be recognized. Therefore, the human eye visual attention perception area and the transfer process are displayed and simulated, and the understanding precision of complex scene indication expression is effectively improved.
Owner:UNIV OF ELECTRONICS SCI & TECH OF CHINA

Explainable visual attention for deep learning

To provide systems or techniques that facilitate explainable visual attention for deep learning.SOLUTION: A system can access a medical image generated by a medical imaging scanner, and perform, via execution of a deep learning neural network, an inferencing task on the medical image. In various embodiments, the deep learning neural network can receive as input the medical image and can produce as output both an inferencing task result and an attention map indicating on which pixels or voxels of the medical image the deep learning neural network focused in generating the inferencing task result.SELECTED DRAWING: Figure 8
Owner:GE PRECISION HEALTHCARE LLC

Multi-modal large model question and answer method, device and equipment based on attention entropy and medium

The invention discloses a multi-modal large model question and answer method, device and equipment based on attention entropy and a medium, and relates to the field of artificial intelligence and machine learning. Comprising the following steps: inputting a target image and a problem text into a multi-modal large model to obtain an image marking sequence and a text lexical element sequence; connecting an rth pruning layer in front of an rth decoding layer of the decoder, calculating an attention matrix from rth vision to text and an attention matrix from rth text to vision according to an rth image mark sequence and an rth text lexical element sequence, and determining an ith information density weight corresponding to an ith image mark in the rth image mark sequence; based on the information density weight corresponding to each image mark in the rth image mark sequence, screening out an rth reserved image mark sequence; and inputting the rth reserved image mark sequence and the rth text lexical element sequence into the rth decoding layer to obtain an answer text output by the large language model, so as to effectively balance the calculation efficiency and the semantic integrity under the condition of not needing additional training.
Owner:TSINGHUA UNIVERSITY

Remote sensing image self-learning segmentation method

The invention provides a remote sensing image self-learning segmentation method, belongs to the technical field of image processing, and aims to realize automatic, accurate, sufficient and reliable high-resolution remote sensing image segmentation. Comprising the following steps: responding to an input remote sensing image, and performing spectrum correction operation on the remote sensing image to obtain a first color image; carrying out super-pixel division on the first color image by utilizing an edge recognition and super-pixel division method for simulating a non-classical receptive field to obtain super-pixel plaques under different scales; performing clustering operation on the superpixel plaques to obtain a clustering result; training a multi-feature deformation lightweight neural network based on a visual attention mechanism by using a clustering result, and generating a super-pixel region recognition model; and performing remote sensing image segmentation by using the super-pixel region recognition model. According to the method, the self-learning ability of human eye visual perception can be simulated, then image segmentation is carried out through self-adaptive analysis, self-learning identification and self-checking correction, and a segmentation result is rapidly and accurately obtained.
Owner:NORTHWEST ENGINEERING CORPORATION LIMITED

A basketball video highlight automatic generation method and system based on multi-modal feature fusion

The application relates to a basketball video highlight automatic generation method and system based on multi-modal feature fusion, and belongs to the technical field of computer vision and multimedia intelligent analysis. The method comprises the following steps: S1, extracting and aligning a visual feature sequence and an audio feature sequence; S2, using bidirectional LSTM to perform time sequence coding on the audio-visual feature, constructing a cross-modal attention mechanism taking audio as a query and vision as a key value, and obtaining multi-modal fusion features; S3, inputting the fusion features into a 1D CNN to output a frame-by-frame highlight event probability sequence; S4, performing peak value searching based on an adaptive threshold of the probability sequence to obtain a coarse-grained timestamp; S5, taking the coarse-grained timestamp as a center to intercept local features, inputting the local features into bidirectional LSTM to predict starting and ending offsets, and obtaining accurate start and end time boundaries; and S6, clipping and splicing to generate a highlight video according to the accurate boundaries. The application realizes audio-visual semantic depth alignment through an audio-guided visual attention mechanism, and improves boundary positioning accuracy through a coarse-precision two-stage positioning architecture.
Owner:CHONGQING UNIV OF POSTS & TELECOMM

Self-adaptive closed-loop brain-computer interface neural feedback training system

The invention particularly relates to a self-adaptive closed-loop brain-computer interface neural feedback training system, and relates to the technical field of brain-computer interfaces and neural feedback. A multi-modal feature extraction module; a coefficient fusion module; and a feedback training module. In the invention, a high-precision crystal oscillator clock is adopted to realize time synchronization of electroencephalogram, eye movement and behavior signals, fusion distortion caused by signal dislocation is thoroughly eliminated, and the three types of signals respectively cover cognitive states, visual attention and motion characteristics to form complementary state evaluation dimensions; a refined quantization algorithm is designed for each mode, wherein instantaneous artifacts are eliminated through extreme value screening of the electroencephalogram coefficient, the pixel diameter is calibrated into the physical diameter through the eye movement coefficient so as to eliminate imaging interference, and the large-amplitude movement intensity and high-frequency posture micro change are considered in the behavior coefficient.
Owner:HANGZHOU BRAIN MIRACLE INTELLIGENT TECHNOLOGY CO LTD