156 results about "Video fusion" patented technology

The invention provides an intelligent sound box voice processing method and system based on artificial intelligence, and the method comprises the steps: obtaining audio data and mouth shape video data, carrying out the processing of the audio data and the mouth shape video data, and carrying out the multi-modal feature fusion, and obtaining a fusion feature; performing bimodal voice activity detection on the fusion features to obtain effective voice data; performing context sensing recognition of audio and video fusion on the effective voice data to obtain a first text; constructing a user feature model, and performing semantic understanding on the text based on the model to obtain an understanding result; performing intention recognition and slot filling based on the understanding result to obtain user intention and key information; generating a response strategy in combination with the user intention, the key information and the environment perception data; generating response voice according to the response strategy; and monitoring feedback information of the user to the response voice in real time, and updating the user feature model and the response strategy evaluation model based on feedback. According to the scheme, the voice can be recognized more accurately, and the safety and robustness of the system are enhanced.

The invention relates to the technical field of computer vision and virtual reality, and discloses a three-dimensional video fusion method based on camera self-calibration and projection texture mapping, and the method comprises the following steps: S1, obtaining video data, and collecting at least one frame of two-dimensional image in a to-be-fused video stream; optionally, the two-dimensional image is preprocessed; and S2, calibrating internal reference of the camera, detecting linear features in the two-dimensional image by using an image processing algorithm, estimating the position of a vanishing point by using a least square method or other optimization algorithms based on the detected linear segment, and calculating an internal reference matrix of the virtual camera according to the optimized vanishing point position. The perspective relation between the video image and the surface of the three-dimensional model is determined through the vanishing point detection technology, accurate fusion of the video image and the three-dimensional model is achieved, the sense of reality of a virtual scene is improved, and the tedious camera calibration process and the complex three-dimensional reconstruction process based on a calibration plate are avoided.

The embodiment of the invention provides a multi-camera video fusion method and system for a robot with a body and a storage medium, and belongs to the field of image processing. The method comprises the following steps: performing time and space alignment on images based on a vision and inertia alignment model to generate a unified aligned image sequence; extracting multi-scale image features based on the uniformly aligned image sequence, and constructing a cross-view residual image for guiding attention fusion operation to generate a fusion image; performing privacy area identification on the fused image, and performing shielding processing on a privacy area to obtain a privacy protection image; and performing semantic compression on the privacy protection image, adjusting an image frame scheduling sequence according to network link state information, and outputting a video signal for controlling a decision. The method is oriented to whole-process optimization of space-time alignment, cross-view fusion and privacy protection processing in the multi-shot image acquisition process of the robot with the body, and video signals for control are output in a self-adaptive mode based on the link state.

The invention relates to the field of artificial intelligence cross-modal retrieval, and provides a video text cross-modal retrieval method and system based on spatio-temporal feature fusion. The method comprises the following steps: carrying out key frame sampling and time sequence partitioning on an input video, extracting static visual features through a spatial feature network, and extracting motion features through a time dynamic network; a self-adaptive gating fusion module is adopted to dynamically calculate spatial-temporal feature weights and perform weighted fusion; extracting text semantic features by using a pre-training language model; constructing a double-flow projection network to map video fusion features and text features to a unified measurement space, and optimizing a feature distance by adopting a contrast loss function containing difficult negative sample mining and intra-modal constraint; and outputting a retrieval result according to the cosine similarity sequence. The system comprises four units, wherein the gating fusion module is integrated with an FPGA acceleration circuit. According to the method, mAP (at) 10 is equal to 0.78 in a UCF-101 data set, the time sequence action retrieval accuracy rate is 92.8%, and the single video retrieval delay is 23 milliseconds.

The invention provides an adaptive contrast enhancement video fusion algorithm for a complex logistics scene, and the algorithm comprises the steps: extracting environment features from an initial video data set, analyzing the inter-frame illumination change and object movement speed through employing a convolutional neural network, determining that the current scene is a peak period or a low-illumination environment, and obtaining environment change perception parameters; extracting a key frame from the second video data set, separating a foreground object and a background by adopting an image segmentation technology based on deep learning, and performing detail retention enhancement on the foreground object to obtain a third video data set; real-time verification is carried out on the final video output, if the processing delay exceeds a preset threshold value, the resource distribution proportion is adjusted through a feedback mechanism, mode switching and enhancement processing are executed again, and the updated video output is obtained.

The invention discloses a natural resource supervision method based on three-dimensional GIS scene and video fusion, and relates to the technical field of intelligent natural supervision. Comprising the steps of collecting a video stream through a preset observation point, extracting a continuous frame sequence, and calling a space parameter set; performing dynamic region detection on the video frame sequence to generate change feature data; mapping the change characteristic data into a topological structure to be updated based on the projection matrix; calling a pre-constructed three-dimensional GIS scene model, and positioning an update area; according to the type identifier of the change feature data, executing an operation of deleting the associated geometry or constructing and fusing a new geometry; adding a state attribute label to the updated geometry, and storing an update model; the resource change rate is calculated, and when the change rate exceeds a preset industry threshold value, an early warning instruction is generated and pushed to the control terminal. And the space accuracy, the model updating efficiency and the early warning reliability of resource change identification in a complex natural scene are remarkably improved.

The invention belongs to the technical field of audio and video fusion transmission, and discloses an audio and video signal fusion transmission method and device, equipment and a medium, and the method comprises the steps: extracting 512-dimensional semantic feature vectors of an audio and a video, and generating a joint feature matrix through a cross-modal attention mechanism; calculating a fusion feature flow based on a coefficient alpha output by the scene classification model; a quantum random number generator is used for generating a session master key Kmain, the session master key Kmain is packaged into an encrypted session key Kenc through a Kyber-768 algorithm, and frame-level dynamic encryption is carried out on the fusion feature flow; and selecting a GPMI Type-B wired protocol or wireless transmission based on QUIC according to the transmission environment. According to the invention, the transmission efficiency, the anti-interference capability and the anti-quantum-attack security are improved, and multi-scene requirements are met.

The invention relates to the technical field of data processing, in particular to an auxiliary driving device and system based on video image processing, and the device comprises a frame buffer queue with a multi-channel video collection module aligned with timestamps; a hierarchical calculation task decoupling module identifies task types in the frame buffer queue and marks priority labels, including a first priority label of a radar data analysis and distance superposition task, a second priority label of a vehicle profile generation task and a third priority label of a video fusion task, and outputs a data stream with priority labels; the calculation module generates a task processing result; the genetic algorithm optimization scheduling module generates a resource allocation strategy and controls the calculation module to execute task processing; and the output synchronous control module synthesizes a high-priority task result and performs frame rate synchronous output on a low-priority task. According to the method, the problem of real-time degradation caused by multi-path high-definition video stream processing is solved through task grading and dynamic resource scheduling, and the image output delay of the auxiliary driving scene is reduced.

The invention discloses a multi-source 5G message video cooperative monitoring method and system, particularly relates to the technical field of video monitoring, and is used for solving the problems of multi-camera calibration parameter misalignment and video fusion dislocation caused by thermal deformation asynchronism in the prior art. A temperature sensor is arranged on an industrial camera supporting structure, a thermal deformation three-dimensional displacement compensation vector is calculated, a preferential transmission object is marked according to the displacement gradient of an adjacent camera and a material heat conductivity coefficient threshold value, and the displacement vector is transmitted to an edge calculation node according to the priority through a 5G message. All displacement data are fused at a node end to dynamically correct space calibration parameters, a collaborative calibration matrix is generated, and finally, the matrix is utilized to perform real-time registration on multiple paths of video streams and perform workpiece defect detection, so that the geometric consistency of a multi-camera system in a thermal deformation environment is guaranteed, and the visual detection precision and stability are improved.

The invention provides a traffic data analysis method and device based on multi-channel video fusion, and relates to the technical field of traffic condition analysis, and the method comprises the steps: collecting a multi-channel traffic video stream set; standardization processing and down-sampling processing are carried out, and background identification removal is carried out on the multi-path traffic video frame set; arranging the multi-path traffic foreground image frame set according to time sequence information, carrying out traffic data analysis and abnormal target identification, and determining multi-path traffic operation parameters and abnormal traffic target information; space coding is carried out, and alignment fusion is carried out on the multiple traffic foreground sequence frame sets; and performing parallel tracking processing to obtain a target traffic data analysis result. Through the method and the device, the technical problem of low data processing efficiency caused by lack of an efficient multi-source data fusion mechanism in the prior art can be solved, and the accuracy and comprehensiveness of traffic data analysis are improved through a multi-channel video fusion technology and selection and deployment of monitoring nodes.

The invention discloses an unmanned aerial vehicle inspection system and method for an intelligent street lamp based on multi-source video fusion, and relates to the technical field of unmanned aerial vehicle inspection, and the system comprises an inspection data collection module, an inspection image analysis module, an inspection mode switching module and an inspection task dynamic evaluation module. The inspection data acquisition module, the inspection image analysis module, the inspection mode switching module and the inspection task dynamic evaluation module are in communication connection with one another; wherein the inspection data acquisition module is used for performing flight inspection on street lamps in a target inspection area one by one through an unmanned aerial vehicle with a visible light camera, and setting image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used for processing and analyzing the acquired image data; according to the invention, the problem that the inspection time consumption and the data processing burden are greatly increased in the existing multi-source image fusion processing mode is solved.

The invention discloses a speaking face generation video detection method and device based on multi-modal global and local collaboration, and the method comprises the steps: carrying out the framing and face region extraction preprocessing of a video, and enabling an input audio to be aligned with an input video frame in a time dimension; a speaking face generation video detection model is constructed and trained, an inconsistent region between video frames is focused on through a preset region attention smooth detection module and a difference capture-time frame aggregation module, and an original audio is input into an audio feature extraction module; carrying out consistency calculation on the input audio and video modal features through a preset audio and video fusion module, fusing the audio and video features so as to discriminate the forged generated video to obtain a video discrimination tag, and then calculating a loss training model; and finally, discriminating a to-be-detected video by using the trained model. According to the method, audio and video features are combined, so that the detection precision and robustness can be effectively improved.

The invention relates to the technical field of video processing, in particular to a video fusion method and system based on a large model. The method comprises the following steps: acquiring a video stream for picture segmentation, identifying a moving target for each segmented picture, predicting a moving target track, matching a minimum pixel region, and determining a key frame; extracting features, and focusing an identification target in the key area; generating a static scene, aligning the recognized target and the scene, obtaining a dynamic shielding relationship, and outputting fused video data; the system comprises a dynamic picture division module, a multi-modal feature extraction module and a dynamic fusion reconstruction module. The method comprises the following steps: identifying a moving target for each segmented image, predicting a moving target track, matching a minimum pixel region, determining a key frame, focusing an identified target in the key region, aligning the identified target and a scene, obtaining a dynamic shielding relationship, and outputting fused video data; trajectory changes of the moving target are processed, and ghosting and dislocation of the fusion result are avoided.

The invention discloses a low-illumination infrared visible light video fusion method based on an LUT and a Transform, and belongs to the field of infrared visible light fusion. Comprising dynamic object shielding extraction; static and dynamic parts are separated; dynamic target enhancement; performing time logic consistency processing; coding and decoding processing; style guidance and final fusion; and a target synthesis step. Through respective reconstruction of the dynamic part and the static part and time sequence consistency processing, the time sequence synchronization problem is effectively solved, and the reconstruction quality is improved. Particularly in multi-source video fusion, the optical flow module is used for accurately shielding the dynamic part of each video sequence, so that the influence of a dynamic background on static scene reconstruction is effectively reduced. And in combination with fine tuning of the large language model, the adaptability and fusion effect of the technology are further enhanced, and especially in a complex dynamic scene, the precision and stability of overall reconstruction are improved.

The invention belongs to the technical field of civil transportation aviation airport aircraft monitoring, and discloses a panoramic aircraft monitoring system and method based on video stitching and an electronic map. The method comprises the following steps: acquiring a vehicle position, a personnel position, aircraft information and flight dynamic information, and acquiring high-definition camera video stream data by a video gateway; performing video stitching processing on the acquired high-definition camera video stream data; the electronic map GIS server provides an electronic map service; carrying out panoramic video fusion processing on the electronic map and the spliced video, calculating the position information of the aircraft, and completing personnel management, vehicle management, flight dynamic information management and monitoring mode management; and reporting normal monitoring and exception handling conditions to the server through the client. According to the invention, a video fusion and electronic map function complementarity technology is adopted, all-weather panoramic video safety monitoring of the aircraft in different modes is realized, and invading personnel and vehicles can be actively found and alarmed.

The invention relates to the technical field of intelligent conference systems, and discloses a debugging-free audio and video fusion management and control system and method, and the method comprises the steps: carrying out the reverse deduction of a communication protocol of audio and video equipment through analyzing the rising edge features of an electrical signal and a radio frequency carrier modulation mode, and dynamically updating an adaptive rule library; the multi-modal scene perception engine fuses audio stream, video stream and environment stream data, and generates an adaptive scene strategy based on a multi-head attention mechanism; the credible recording enhancement system automatically binds the identity of a spokesman through a voiceprint clustering algorithm, and carries out block chain storage on recording contents according to speaking segments, so that plug and play of multi-brand wired / wireless equipment is realized, and different communication protocols can be automatically adapted without manual configuration; through real-time environment perception and dynamic weight distribution, the accuracy and fluency of conference scene switching are remarkably improved; and meanwhile, voiceprint dynamic registration and block chain evidence storage technologies are adopted, so that the judicial-level recording security is ensured, and meanwhile, the preparation work before meeting is greatly simplified.

The embodiment of the invention provides an audio and video fusion intelligent inspection method and system for a new energy station, and the method comprises the steps: converting the input of different modes into a discrete token sequence in a unified manner, and carrying out the deep interaction and fusion through a cross-mode Transform architecture, thereby effectively aligning features, and capturing the internal correlation between the modes. More importantly, in order to improve the recognition capability of rare faults, a fault prototype library is introduced as priori knowledge, and through an attention mechanism guided by a prototype, the model can actively compare and associate current inspection features with known typical fault modes. According to the design, the model is forced to learn more discriminative context sensing features, sparsity of training data is made up by effectively utilizing condensed fault knowledge, and fault diagnosis generalization ability and reliability of the model in a complex scene are finally improved.

The invention belongs to the technical field of intelligent traffic, and relates to a radar and video fusion dynamic space-time alignment road monitoring method, which comprises the following steps of: firstly, independently retrieving a radar monitoring target and a video monitoring target in a road monitoring area, and adopting a dynamic space alignment and multi-scale time synchronization mode to obtain a radar monitoring target and a video monitoring target; performing space-time alignment processing on the radar monitoring target and the video monitoring target, performing association matching on the radar monitoring target and the video monitoring target after space-time alignment, and judging whether association constraint conditions are met or not based on spatial position consistency, movement track continuity and appearance feature similarity; according to the method and the device, a target association result is adaptively output according to an actual monitoring scene, structured packaging is carried out on the target association result, a fusion target data packet is generated and output for monitoring feedback, and the radar and video fusion perception precision and robustness in a full scene are remarkably improved, so that the accuracy and reliability of road monitoring are effectively guaranteed.

The invention provides a video fusion method and device based on a three-dimensional scene and a storage medium, and belongs to the technical field of video surveillance, and the method comprises the steps: obtaining video frame data, including a video frame image and a corresponding timestamp, collected by a camera installed on sports equipment, obtaining space-time information of the sports equipment, comprising geographic data and corresponding timestamps; extracting image feature points of each video frame image, and constructing a mapping relation between the image feature points and the geographic data; based on the mapping relation, the pose data of the camera is iteratively solved by minimizing a re-projection error, and an objective function corresponding to the re-projection error comprises strong constraint optimization of geographic data; and determining a conversion parameter between a first coordinate system of the video frame image and a second coordinate system of a pre-constructed three-dimensional scene model based on the pose data to obtain a video fusion parameter, so as to fuse the multi-frame video frame data with the three-dimensional scene model, thereby improving the fusion effect of the three-dimensional scene and the video.

The invention relates to the technical field of AR video fusion, in particular to an AR real-time fusion method based on space coordinate transformation under the view angle of an unmanned aerial vehicle, which comprises the following steps: converting geodetic coordinate system data of the unmanned aerial vehicle and a target point into a three-dimensional Cartesian coordinate system, and calculating the linear distance between the unmanned aerial vehicle and the target point; calculating an included angle alpha of a connecting line between the unmanned aerial vehicle and the target point relative to the due north direction; according to the included angle alpha, the horizontal view field angle and geodetic coordinate system data of the unmanned aerial vehicle, calculating an included angle gamma h of a connecting line of the unmanned aerial vehicle and the target point relative to a horizontal view field reference axis of the camera; according to the linear distance, the vertical field angle and the geodetic coordinate system data, calculating an included angle gamma v of a connecting line between the unmanned aerial vehicle and the target point relative to a vertical field reference axis of the camera; based on the included angle gamma h and the included angle gamma v, analyzing pixel coordinates of the target point in the video image coordinate system; and superposing the virtual information to generate an AR fusion video stream. According to the scheme, the AR real-time fusion adaptability and precision of the unmanned aerial vehicle in a dynamic scene can be improved.

The invention discloses a high-risk rockfall monitoring method and system based on 4D millimeter-wave radar and video fusion, through fusion of the 4D millimeter-wave radar and an image acquisition technology and full utilization of the all-weather and all-time monitoring capability of the 4D millimeter-wave radar, the high-risk rockfall monitoring method and system can penetrate through severe weather conditions such as rain, fog, dust and the like; the three-dimensional position, speed and track information of the rockfall target is accurately captured, and the target identification capability is enhanced in combination with video image data, so that the false alarm rate is effectively reduced, and the monitoring precision and reliability are remarkably improved.

The invention relates to the technical field of medical image processing, in particular to a video fusion method based on an improved SURF and an optimal suture line algorithm, and the method comprises the steps: obtaining a video frame, and adaptively extracting image feature points for the video frame; constructing a neighborhood descriptor according to the image feature points; performing feature point matching on the image feature points in the adjacent video frames according to the neighborhood descriptors, and constructing matching points; and determining an optimal stitching path of the adjacent video frames according to the matching points, and then performing panoramic stitching on the video frames. In order to solve the problem that distortion is introduced due to imaging difference in a feature point identification and alignment scheme in the prior art, in the scheme, an SURF algorithm is improved, an adaptive threshold is introduced to screen image feature points, and point locations with insignificant features, which are wrongly selected due to imaging difference, are removed, so that the mismatching rate is reduced, and the accuracy of the feature point identification and alignment is improved. And the splicing quality is improved.

The invention provides a robot self-viewing angle training data generation method, device, equipment and medium, a second joint vector sequence of a target mechanical arm under a new self-viewing angle is generated based on viewing angle movement information and a first joint vector sequence corresponding to a first robot operation video, and a second joint vector sequence of the target mechanical arm under a new self-viewing angle is generated based on the viewing angle movement information. Performing view angle conversion and video rendering processing on a video background in the first robot operation video and a target mechanical arm to generate a new background video and a new mechanical arm action video, and performing video fusion on the new mechanical arm action video and the new background video to obtain a second robot operation video; and second training data of the self-view angle of the mechanical arm is constructed based on the second robot operation video and the second joint vector sequence, so that strong coupling matching of visual presentation and joint action parameters under the new self-view angle is achieved, and the strategy learning requirement of the VLA model under the self-view angle change scene is met.

The invention discloses a visual anti-occlusion tracking method based on AIS and video fusion. The method comprises the following steps: synchronously obtaining AIS data and corresponding video data; preprocessing the AIS data, predicting latitude and longitude coordinates of the AIS data, projecting the latitude and longitude coordinates to a video pixel plane, and generating a ship AIS track; meanwhile, identifying a ship bounding box in a video frame by using a video detection algorithm, and tracking by using a three-stage association anti-occlusion tracking algorithm; the similarity between the AIS trajectory and the video trajectory is calculated through a dynamic time warping algorithm, and the correlation matching of the AIS trajectory and the video trajectory is realized by using a Hungary matching algorithm, so that a stable and anti-shielding ship tracking trajectory fusing AIS and video information is obtained. According to the invention, the problem that pure visual tracking is easy to fail under the shielding condition is effectively solved.

The present application relates to the technical field of tool positioning, and discloses a tool positioning management and control system and early warning method based on RFID and video fusion, which comprises the following steps: establishing an association mapping of RFID tag unique code and tool unique identification and constructing a tool information file; configuring an electronic fence based on a management and control area space model, collecting RFID tag signals to perform area attribution inference, generating an RFID side position state and performing first stage state determination; obtaining a monitoring video stream that is spatially registered with the electronic fence space range, performing tool target detection, tool category confirmation and cross-frame target tracking on the video picture to obtain a video side tool existence state; and fusing the RFID side position state and the video side tool existence state to perform second stage determination, and starting a hierarchical early warning strategy to output alarm information according to the determination. The present application reduces the false judgment and missed judgment caused by single sensing.

The invention provides an anomaly detection method and device based on video fusion, electronic equipment and a medium, and can be applied to the technical field of video processing and digital twinning. The method comprises the steps that abnormal event detection is conducted on at least one video stream, a detection result is obtained, the video stream is obtained through at least one image collection device arranged in an inspection site, and the image collection device corresponds to a virtual image collection device in a three-dimensional virtual model for the inspection site; under the condition that the target video stream with the target abnormal event exists, generating an abnormal report according to image acquisition information of a target image acquisition device for acquiring the target video stream and the target video stream; performing image fusion on the target video stream and an adjacent video stream having an overlapped acquisition area with the target video stream to obtain a panoramic video stream; and superposing the panoramic video stream to a collection area corresponding to the panoramic video stream in the three-dimensional virtual model, and displaying the three-dimensional virtual model after superposing the panoramic video stream.

The invention discloses a video fusion method and device, a medium and a program product, and the method comprises the steps: removing pixel data corresponding to a target object in an image frame of a first video, complementing a target region in the image frame of the first video, and obtaining a second video; wherein the target area comprises pixel data of the target object; the image frame of the second video does not contain the target object; regulating and controlling the posture of a virtual object based on the posture data of the target object in the image frame of the first video, and obtaining and loading a role image set; and fusing the image frames in the second video and the role images in the role image set to obtain and load a target video. The scheme can reduce the manufacturing cost of the target video and shorten the manufacturing period of the target video.

The invention discloses a water regimen AR video fusion management system, which relates to the technical field of fusion management and comprises a hydrological data acquisition module, a video acquisition module, a geographic information acquisition module, a dynamic calibration module, an AR fusion module and an intelligent decision module. Multi-source data of the water level value H, the flow velocity value V and the rainfall value R are fused with a single risk index, and AR alarm is driven; the dynamic calibration module integrates a heterogeneous data source (hydrological sensor / video / GIS), Beidou RTK positioning data (XRTK, YRTK) and a GIS elevation model Z (X, Y) are fused into a three-dimensional space point, the intelligent decision-making module carries out graded early warning and linkage AR visual early warning based on a hydrological early warning value F and a submerging depth D (x, y), and meanwhile, a mobile terminal, AR glasses and a command center large screen are supported to synchronously display an AR fusion scene. Collaborative annotation data is encrypted and stored by adopting a block chain technology, and is shared in real time through a 5G private network, so that the timeliness of emergency command is improved.

The invention discloses a conference privacy protection system and method based on visual sound field fusion. The conference privacy protection system comprises a visual acquisition module, an audio acquisition module, an audio playing module and a processing control module, wherein the visual acquisition module transmits mouth shape area three-dimensional coordinates, ear area three-dimensional coordinates and mouth motion states of participants to the processing control module; the processing control module generates a directional pickup instruction based on the mouth shape area three-dimensional coordinates, and sends the directional pickup instruction to the audio acquisition module to control the audio acquisition module to focus mouth shape area pickup; and meanwhile, a directional playback instruction is generated based on the three-dimensional coordinates of the ear region and is sent to the audio playing module to control the audio playing module to project sound beams to ears. Through visual guidance directional pickup, ultrasonic directional playback and audio and video fusion optimization, the risk of audio diffusion is reduced, so that the conference privacy protection effect is improved.

An automatic video production device that performs fitting or fusion processing of a predetermined child video object that is not included in a parent video into a predetermined video scene of the parent video content as a video fusion source, the automatic video production device including a video content interpreting unit that interprets the entirety or a part of the video content of the parent video content, or divides the parent video content into a predetermined parent video scene to generate processing data that is predetermined parent video interpretation data obtained by converting a content interpretation result of the parent video or a division result of the parent video scene into data in a predetermined description format.