83 results about "Video fusion" patented technology

The invention relates to the field of artificial intelligence cross-modal retrieval, and provides a video text cross-modal retrieval method and system based on spatio-temporal feature fusion. The method comprises the following steps: carrying out key frame sampling and time sequence partitioning on an input video, extracting static visual features through a spatial feature network, and extracting motion features through a time dynamic network; a self-adaptive gating fusion module is adopted to dynamically calculate spatial-temporal feature weights and perform weighted fusion; extracting text semantic features by using a pre-training language model; constructing a double-flow projection network to map video fusion features and text features to a unified measurement space, and optimizing a feature distance by adopting a contrast loss function containing difficult negative sample mining and intra-modal constraint; and outputting a retrieval result according to the cosine similarity sequence. The system comprises four units, wherein the gating fusion module is integrated with an FPGA acceleration circuit. According to the method, mAP (at) 10 is equal to 0.78 in a UCF-101 data set, the time sequence action retrieval accuracy rate is 92.8%, and the single video retrieval delay is 23 milliseconds.

The embodiment of the invention provides an audio and video fusion intelligent inspection method and system for a new energy station, and the method comprises the steps: converting the input of different modes into a discrete token sequence in a unified manner, and carrying out the deep interaction and fusion through a cross-mode Transform architecture, thereby effectively aligning features, and capturing the internal correlation between the modes. More importantly, in order to improve the recognition capability of rare faults, a fault prototype library is introduced as priori knowledge, and through an attention mechanism guided by a prototype, the model can actively compare and associate current inspection features with known typical fault modes. According to the design, the model is forced to learn more discriminative context sensing features, sparsity of training data is made up by effectively utilizing condensed fault knowledge, and fault diagnosis generalization ability and reliability of the model in a complex scene are finally improved.

The invention provides a robot self-viewing angle training data generation method, device, equipment and medium, a second joint vector sequence of a target mechanical arm under a new self-viewing angle is generated based on viewing angle movement information and a first joint vector sequence corresponding to a first robot operation video, and a second joint vector sequence of the target mechanical arm under a new self-viewing angle is generated based on the viewing angle movement information. Performing view angle conversion and video rendering processing on a video background in the first robot operation video and a target mechanical arm to generate a new background video and a new mechanical arm action video, and performing video fusion on the new mechanical arm action video and the new background video to obtain a second robot operation video; and second training data of the self-view angle of the mechanical arm is constructed based on the second robot operation video and the second joint vector sequence, so that strong coupling matching of visual presentation and joint action parameters under the new self-view angle is achieved, and the strategy learning requirement of the VLA model under the self-view angle change scene is met.

The invention discloses a visual anti-occlusion tracking method based on AIS and video fusion. The method comprises the following steps: synchronously obtaining AIS data and corresponding video data; preprocessing the AIS data, predicting latitude and longitude coordinates of the AIS data, projecting the latitude and longitude coordinates to a video pixel plane, and generating a ship AIS track; meanwhile, identifying a ship bounding box in a video frame by using a video detection algorithm, and tracking by using a three-stage association anti-occlusion tracking algorithm; the similarity between the AIS trajectory and the video trajectory is calculated through a dynamic time warping algorithm, and the correlation matching of the AIS trajectory and the video trajectory is realized by using a Hungary matching algorithm, so that a stable and anti-shielding ship tracking trajectory fusing AIS and video information is obtained. According to the invention, the problem that pure visual tracking is easy to fail under the shielding condition is effectively solved.

The present application relates to the technical field of tool positioning, and discloses a tool positioning management and control system and early warning method based on RFID and video fusion, which comprises the following steps: establishing an association mapping of RFID tag unique code and tool unique identification and constructing a tool information file; configuring an electronic fence based on a management and control area space model, collecting RFID tag signals to perform area attribution inference, generating an RFID side position state and performing first stage state determination; obtaining a monitoring video stream that is spatially registered with the electronic fence space range, performing tool target detection, tool category confirmation and cross-frame target tracking on the video picture to obtain a video side tool existence state; and fusing the RFID side position state and the video side tool existence state to perform second stage determination, and starting a hierarchical early warning strategy to output alarm information according to the determination. The present application reduces the false judgment and missed judgment caused by single sensing.

The invention provides an anomaly detection method and device based on video fusion, electronic equipment and a medium, and can be applied to the technical field of video processing and digital twinning. The method comprises the steps that abnormal event detection is conducted on at least one video stream, a detection result is obtained, the video stream is obtained through at least one image collection device arranged in an inspection site, and the image collection device corresponds to a virtual image collection device in a three-dimensional virtual model for the inspection site; under the condition that the target video stream with the target abnormal event exists, generating an abnormal report according to image acquisition information of a target image acquisition device for acquiring the target video stream and the target video stream; performing image fusion on the target video stream and an adjacent video stream having an overlapped acquisition area with the target video stream to obtain a panoramic video stream; and superposing the panoramic video stream to a collection area corresponding to the panoramic video stream in the three-dimensional virtual model, and displaying the three-dimensional virtual model after superposing the panoramic video stream.

The invention discloses a video fusion method and device, a medium and a program product, and the method comprises the steps: removing pixel data corresponding to a target object in an image frame of a first video, complementing a target region in the image frame of the first video, and obtaining a second video; wherein the target area comprises pixel data of the target object; the image frame of the second video does not contain the target object; regulating and controlling the posture of a virtual object based on the posture data of the target object in the image frame of the first video, and obtaining and loading a role image set; and fusing the image frames in the second video and the role images in the role image set to obtain and load a target video. The scheme can reduce the manufacturing cost of the target video and shorten the manufacturing period of the target video.

The invention discloses a conference privacy protection system and method based on visual sound field fusion. The conference privacy protection system comprises a visual acquisition module, an audio acquisition module, an audio playing module and a processing control module, wherein the visual acquisition module transmits mouth shape area three-dimensional coordinates, ear area three-dimensional coordinates and mouth motion states of participants to the processing control module; the processing control module generates a directional pickup instruction based on the mouth shape area three-dimensional coordinates, and sends the directional pickup instruction to the audio acquisition module to control the audio acquisition module to focus mouth shape area pickup; and meanwhile, a directional playback instruction is generated based on the three-dimensional coordinates of the ear region and is sent to the audio playing module to control the audio playing module to project sound beams to ears. Through visual guidance directional pickup, ultrasonic directional playback and audio and video fusion optimization, the risk of audio diffusion is reduced, so that the conference privacy protection effect is improved.

The invention discloses an audio and video fusion method and device, equipment, a storage medium and a product, and the method comprises the steps: replacing a first audio corresponding to a role in a video with a second audio in response to a selection operation of a user on the role in the video when the video is played; wherein the second audio is generated based on the voice input of the user, or is generated based on the text or audio selected by the user in combination with the timbre of the user. Therefore, in the real-time playing process of the video, the audio of the user is fused into the video in real time, so that a role in the video is avatar as a virtual image of the user, the user obtains immersive interaction experience in the video environment, and the user is converted from an external observer to a participated interactor of a video scene.

The invention relates to a full-automatic code programming method and system based on a large model. The full-automatic code programming method comprises the steps that 1, multi-modal requirements are collected, analyzed and expressed in a standardized mode; 2, automatically arranging a development environment; step 3, code candidate generation and quality inspection; 4, compiling / constructing and automatically executing; 5, performing structured evaluation and success judgment; 6, closed-loop error correction and self-optimization control are carried out; and 7, safety and compliance control. The method supports fusion analysis of texts, voices, images and videos, and automatically extracts program specifications and acceptance criteria. According to the method, automatic generation, compiling, execution, evaluation and error correction are achieved, and manual log interpretation and prompt word rewriting are not needed.

The application provides a three-dimensional scene-based video fusion method and device and a storage medium, and belongs to the technical field of video monitoring. The method comprises the following steps: acquiring video frame data collected by a camera installed on a motion device, wherein the video frame data comprises a video frame image and a corresponding timestamp; acquiring space-time information of the motion device, wherein the space-time information comprises geographic data and a corresponding timestamp; extracting image feature points of each video frame image, and constructing a mapping relationship between the image feature points and the geographic data; based on the mapping relationship, iteratively solving pose data of the camera by minimizing a re-projection error, wherein a target function corresponding to the re-projection error comprises a strong constraint optimization of the geographic data; based on the pose data, determining conversion parameters between a first coordinate system of the video frame image and a second coordinate system of a pre-constructed three-dimensional scene model, obtaining video fusion parameters, and fusing the multiple video frame data with the three-dimensional scene model, thereby improving the fusion effect of the three-dimensional scene and the video.

This application provides a video content recognition method, apparatus, device, storage medium, and program product, belonging to the field of data processing technology. The method includes: acquiring a video to be recognized, comment data of the video to be recognized, and descriptive text, wherein the video to be recognized consists of video data and audio data; inputting the comment data into a semantic analysis model to obtain a judgment result of the background music volume; if the volume judgment result indicates a large volume, inputting the video data and audio data into an audio-video fusion extraction model to obtain fusion information text output by the audio-video fusion extraction model; using the fusion information text and the descriptive text to remove background noise from the audio data to obtain a noise-reduced frequency; and determining video content information based on the noise-reduced frequency and the video data. This method solves the problem of low accuracy in audio recognition and information extraction when background music is present.

The disclosure provides a video fusion method, and particularly relates to the technical fields of automatic driving and image processing. The specific implementation scheme is as follows: obtaining image calibration coordinates of a plurality of fisheye cameras, fusion image coordinates, and a plurality of videos captured by the plurality of fisheye cameras, wherein the fusion image coordinates are obtained by fusing all the image calibration coordinates; encoding the image calibration coordinates to obtain coordinate information encoding; obtaining weight parameters of image pixels of each image in the plurality of videos based on the fusion image coordinates; selecting image pixels of each image in the plurality of videos based on the coordinate information encoding; and calculating a fusion image corresponding to all the images in the plurality of videos based on the weight parameters of each image and the image pixels of each image. The embodiment improves the real-time performance of video fusion.

The invention discloses a room acoustic characteristic multi-parameter estimation method based on audio and video fusion so as to accurately estimate room acoustic parameters (RAPs) and improve the anti-interference capability. The method comprises the following steps: performing STFT on a single-channel voice signal by adopting an audio front end, and splicing logarithmic magnitude spectrum and inter-frame phase difference features to obtain audio preprocessing features; the front end of the video is subjected to lip ROI extraction, sequential features are extracted through a convolutional layer and ResNet-18 and then added, and video preprocessing features are obtained; an AV-MAE architecture is adopted to respectively encode audio and video features, and double-gate cross-modal fusion is introduced at a specific level; performing parallel RAPs prediction on the fused features through a plurality of parallel independent parameter estimators; a Huber loss function is adopted to train the network, and the small error precision and the large error anti-interference capability are balanced. The method has the advantages that multi-mode audio and video information is fully fused, the problems of unknown positions of the camera and the microphone and multi-task conflict are solved, and the method is adaptive to RIR parameter estimation in a complex acoustic environment.

The invention relates to the technical field of video fusion, and discloses a video fusion method and system based on a shadow map and a program product, and the method comprises the following steps: creating a virtual perspective camera in a three-dimensional scene, and generating the shadow map according to a visual cone of the virtual perspective camera; pixels in the three-dimensional scene are converted from a visual space coordinate system of a physical world camera to a virtual perspective camera coordinate system, NDC coordinates of the pixels are obtained, the NDC coordinates are aligned with texture coordinates of the shadow map, and sampling texture coordinates are obtained; judging whether the pixels in the visual range of the virtual perspective camera are in the visual area of the shadow map or not, and obtaining the color of the current pixel of the three-dimensional scene; and obtaining the color of the current pixel of the fused three-dimensional scene. According to the invention, the problems of shielding area processing errors and the like in the prior art are solved.

The application discloses a target identification method based on monitoring video and relates to the technical field of target identification. The method comprises the following steps: acquiring a plurality of regional monitoring videos, extracting a preliminary monitoring image key frame sequence by optimizing an interframe difference method, and screening an optimized monitoring image key frame sequence by a secondary clustering method; adopting a scale invariant feature transformation algorithm to perform feature matching on the optimized monitoring image key frame sequence, obtaining a monitoring image sequence with the same feature, and performing image splicing on the monitoring image sequence by a video splicing method based on a random sample consensus algorithm, thereby obtaining a spliced monitoring image sequence; performing fusion on the spliced monitoring image sequence by a video fusion algorithm based on an adaptive threshold, thereby obtaining a fused monitoring image sequence; and performing identification on the optimized key frame sequence and the fused monitoring image sequence respectively by a target identification algorithm, obtaining a first confidence degree and a second confidence degree, and obtaining a corrected confidence degree by using an adaptive weighted fusion method.

The invention discloses a radar-sonar-video fusion association method and system based on cross-modal semantic mapping. The method comprises the steps that modal specific feature vectors are extracted from three heterogeneous sensors including a radar sensor, a sonar sensor and an optical video sensor respectively; constructing a three-branch deep metric learning network, and mapping the heterogeneous features to a unified d-dimensional semantic embedding space; a tripartite graph model is constructed, and semantic similarity and kinematics consistency constraints are comprehensively considered for edge weights; an improved Kuhn-Munkres algorithm is applied to solve the global optimal matching of the tripartite graph; and outputting a fusion target state and matching confidence evaluation. Cross-physical-domain invariant feature representation is automatically extracted through deep metric learning, three-mode global optimal association is achieved in combination with a graph optimization algorithm, the technical problem that cross-physical-domain data fusion is difficult to process through a traditional method is solved, and the method has high accuracy, strong generalization ability and real-time performance in application scenes such as ocean monitoring and port security and protection and is suitable for being used in the field of ocean monitoring and port security and protection. And an effective technical scheme is provided for multi-sensor heterogeneous data fusion.

The application discloses a method and device for virtual human video fusion, which comprises the following steps: collecting a scene video based on a binocular camera, processing the scene video to obtain first depth information of at least one object in the scene, importing multiple images of virtual humans into the scene video, associating each image with second depth information of the virtual human, fusing the images of the virtual human and the scene video based on the first depth information and the second depth information to obtain a virtual human fusion video and transmitting the virtual human fusion video to a display device for displaying the virtual human fusion video on an interactive interface. The application improves the accuracy of the position display of the virtual human in the video scene.

The invention discloses a video fusion ecological environment monitoring system, and relates to the technical field of ecological environment monitoring. The system comprises a data acquisition module, a three-dimensional space registration module, a space service publishing module, a video fusion display module and a database module. The data acquisition module selects a video monitoring camera and captures a real-time monitoring video stream; the three-dimensional space registration module completes the space registration of the monitoring video and the three-dimensional scene by using a registration tool and a plurality of algorithm models; the space service publishing module is fused with the three-dimensional live-action three-dimensional model to complete service publishing; the video fusion display module realizes query display of an ecological service three-dimensional fusion scene and a monitoring video; and the database module provides data storage support. According to the invention, the problems of video picture isolation and three-dimensional scene static state in traditional monitoring are solved, and more intuitive and stereoscopic video data are provided for ecological environment management decision.

The invention discloses a road vehicle free flow identification system based on cross-system multi-source video fusion, and belongs to the technical field of Internet of Things. The system comprises a sensing layer, an edge computing layer, a cooperative computing layer and an application layer, the sensing layer is used for collecting video streams of the special identification equipment group of the toll portal and the traffic monitoring camera; the edge calculation layer is used for carrying out real-time analysis on the video stream of each identification node, generating a structured identification data packet and uploading the structured identification data packet; the cooperative calculation layer is used for carrying out environment adaptive fusion on the multi-source recognition results of the same node, carrying out real-time space-time cooperative verification on the cross-node recognition results, and constructing a vehicle track; the application layer is used for carrying out real-time path judgment and charging according to the vehicle track and detecting fake-licensed, shielding and abnormal speed behaviors; through cross-system resource integration, environment adaptive fusion, real-time multi-node cooperation and edge cloud distributed processing, a real-time and accurate technical solution is provided for free flow charging of the expressway.

The application discloses a three-dimensional video fusion calibration and real-time rendering method and device, and the method comprises the following steps: acquiring a video frame image, performing distortion correction, and saving the distortion parameters after the distortion correction; selecting a plurality of feature points in the video frame image and the picture position of a corresponding three-dimensional scene of the video frame image, respectively, and performing registration on the video frame image and the picture position of the three-dimensional scene; acquiring initial internal parameters of a camera for shooting the video, and obtaining optimal internal parameters through an optimization solving mode; determining the rotation and translation vectors of the camera relative to the origin of the world coordinate system based on the optimal internal parameters; determining the world coordinates and rotation angles of the camera in the three-dimensional scene, and projecting the image after the distortion correction onto the three-dimensional face sheet corresponding to the picture position of the three-dimensional scene in the form of texture projection. Even in the case of a large difference between internal parameter templates, the method can quickly find suitable internal parameters, and the accuracy of projection is ensured.

The invention discloses an intelligent concrete slump detection method based on multi-modal information fusion, and belongs to the technical field of concrete quality detection. The stirrer video data and the main shaft current signal are synchronously acquired. For video data, video image sequence apparent features are extracted and fused with dense optical flow field features, and a video fusion feature sequence is obtained. Specifically, the method comprises the following steps: extracting inter-frame dense fluid motion features by adopting a Farneback algorithm; and synchronously extracting image apparent visual features by using a deep network. The two are cooperated to realize cross-scale correlation of the material surface morphology and the pixel-level fluid flow state, and a video fusion feature sequence with both microscopic dynamic and global appearance is formed. And for current data, features are synchronously extracted through a deep network, a video and current parallel processing structure is integrally formed, a video fusion feature sequence and current features are subjected to feature superposition and cross attention module fusion, a multi-modal prediction model is constructed, and the slump detection accuracy is improved.

The invention relates to an intelligent replacement method for a short play video advertisement article, which relates to a video fusion technology, and comprises the following steps: carrying out multi-frame dynamic image acquisition and advertisement area three-dimensional space information extraction on a to-be-processed short play video, and establishing a physical attribute inversion and dynamic physical attribute field model; rendering parameter fitting and standardized input are carried out on the advertisement three-dimensional model according to physical parameters such as dynamic illumination, materials and roughness of an original scene, a three-dimensional space self-adaptive physical rendering engine is driven, and synchronous physical reconstruction of advertisement content and continuous frame scene illumination and material changes is achieved. In combination with multi-foreground-layer dynamic shielding and space superposition, in cooperation with the steps of boundary adaptive hybrid rendering, time sequence image quality optimization and the like, the advertisement content can be dynamically and accurately transformed according to the environment and the foreground, and finally high consistency and seamless natural fusion of the advertisement and the movie video content in physical attribute and visual perception are achieved.

The invention discloses a multi-modal video fusion method and device based on a space-time hierarchical attention network, and relates to the technical field of computer vision. The method comprises the steps of performing two-stage training on a video fusion model according to a first training data set and a second training data set based on an inter-frame consistency loss function and a bidirectional consistency loss function; obtaining a to-be-fused infrared video and a to-be-fused visible light video; based on a frame-by-frame sliding mode, performing continuous three-frame synchronous slicing on the to-be-fused infrared video and the to-be-fused visible light video to obtain a first infrared fragment set and a first visible light fragment set; adding three-dimensional position codes to the first infrared fragment set and the first visible light fragment set; and according to the second infrared fragment set and the second visible light fragment set, performing multi-modal video fusion by using the optimized video fusion model to obtain a fused video. The invention relates to an efficient and coherent multi-mode video fusion method based on time-space layered attention.

The application provides an AR video fusion superimposition method and system for ultra-low delay, which comprises the following steps: obtaining a display refresh event and a picture submission timestamp, calculating a next refresh time and calculating a remaining available time interval, and generating display synchronization constraint information; dynamically screening a synchronizable superimposition information set based on a remaining budget and an estimated execution time of virtual information; analyzing anchor points to generate geometric data, aggregating resources to a texture atlas, and calculating fusion weights combined with parameters such as urgency and stability; finally, the virtual and real scene pixels are weighted and fused by using a pixel shader, and the display is submitted. Through accurate control of the time window and intelligent screening of the content, the end-to-end delay is effectively reduced, the picture lag and trailing are avoided, the priority presentation of key information is ensured, and the smoothness, real-time performance and hardware energy efficiency of AR display are significantly improved.

The invention provides a laser point cloud and video fusion method and device based on a 3D deploy and control ball, and relates to the technical field of data processing, and the method comprises the steps: carrying out the encryption processing of laser point cloud data and video data at a data transmission stage, and completing the decryption, timestamp alignment and data format standardization at a receiving end. After standardization, the data is distributed to a plurality of computing nodes for parallel processing, image texture, edge and motion vector features are extracted on the video side, space coordinates, surface normal vectors and point density features are extracted on the point cloud side, and a fusion feature set is generated through cross-modal correlation calculation and matching. And constructing a multi-modal three-dimensional scene representation based on the set, performing automatic exploration and analysis on a working site, outputting analysis results including terrains, obstacles and the like, and further generating a working scheme with spatial geometric constraints and working safety boundaries. According to the invention, end-to-end security and time-space consistency processing of the video monitoring data and the laser point cloud data can be realized.

The weak constraint 3D WebGIS video fusion method belongs to the technical field of virtual video fusion, and is characterized in that: including the following steps: S1, three-dimensional model construction; S2, three-dimensional model optimization adjustment; S3, 3D WebGIS platform construction; S4, access to video stream data; S5, model view matrix and projection matrix calculation; S6, view cone construction and occlusion detection; S7, rendering based on WebGL fragment shader; S8, video image dynamic texture mapping; S9, video projection edge feathering; S10, dynamic parameter adjustment. The weak constraint 3D WebGIS video fusion method solves the problem of video projection occlusion to a certain extent, realizes video fusion under weak constraint, improves the video fusion effect, and has high practicability, applicability and adaptability.

The invention provides a video fusion method, system and device and a program product, and the method comprises the steps: obtaining a real-time video stream to be fused to a three-dimensional scene model, and pre-calibrating a display plane and a three-dimensional scene reference point in the three-dimensional scene model; determining a coordinate of a visual position point of a three-dimensional scene reference point on a display plane under a visual angle of a current world camera corresponding to a current frame original image in the real-time video stream; mapping the visual position point into a calibrated pixel coordinate range to obtain a coordinate of a target image reference point; calculating a coordinate transformation relation between the original image and the target image according to the coordinate of the target image reference point and the calibrated coordinate of the original image reference point; performing transformation processing on the current frame original image based on the corresponding coordinate transformation relation to obtain a corresponding target image; and rendering the target image on the display plane. According to the invention, a video fusion angle adaptive mechanism is added, and the video fusion effect is effectively improved.

The invention discloses a ship attitude monitoring method, device and system based on point cloud-video fusion. The method comprises the following steps: extracting a vertical contour of a ship body based on preprocessed video information and preprocessed three-dimensional point cloud data in combination with a ship knowledge graph; based on the extracted vertical contour of a ship body, the drifting distance of the ship body in the ship bow direction and the ship stern direction at the two attitude angles is calculated, a three-level alarm threshold system is constructed based on a knowledge graph, and an alarm threshold is dynamically adjusted through a wind wave level-ship body stability correlation model in the knowledge graph. And respectively comparing the ship body attitude angle and the drift distance in the ship bow and stern directions with the alarm threshold values of the two attitude angles and the drift distance in the ship bow and stern directions, and generating an alarm signal. Through step-by-step processing of point cloud-video fusion and knowledge graph driving, non-contact high monitoring of the ship attitude is realized, the method can be directly applied to a port automatic loading operation scene, and the operation safety and efficiency are improved.

The invention relates to a video quality prediction model training method and device, equipment and a storage medium, and the method comprises the steps: obtaining a sample original video for shooting a sample object, and constructing a sample video pair according to the sample original video and a sample synthesis video of the sample object; inputting the sample video pair into a preset model, and extracting a video frame appearance feature of each sample video in the sample video pair, a motion attribute feature of a sample object and a space attribute feature based on a feature extraction network; obtaining a sample video fusion feature corresponding to each sample video based on a feature fusion network; and according to the sample video fusion feature corresponding to each sample video and the sample video quality score label marked by each sample video, training a preset model to obtain a video quality prediction model. According to the invention, intelligent quality evaluation can be carried out on the video, and the evaluation efficiency of the video quality is improved.