2666 results about "Video sequence" patented technology

A system and method are disclosed for generating hyperspectral images from multi-modal sensor data including RGB, LiDAR, thermal, and near-infrared inputs. Training data includes hyperspectral images and corresponding multi-modal measurements. Spectral band grouping is performed based on correlation coefficients. A multi-modal decomposition network with cross-modal attention mechanisms generate reconstructed hyperspectral images by fusing complementary sensor information. A fine-tuning network creates reconstructed RGB images. A comprehensive quality assurance system analyzes spectral consistency, cross-modal coherence, and fusion artifacts to generate quality metrics. Missing data compensation strategies handle corrupted sensor inputs using information from other modalities. The system includes temporal integration for video sequences and multi-resolution processing for different sensor resolutions. Quality metrics guide network weight adjustments to improve reconstruction accuracy while maintaining robustness to sensor failures and environmental variations.

The embodiment of the invention provides a scene space three-dimensional model dynamic modeling method based on multi-modal data, and the method comprises the steps: fusing the feature information of a video sequence, point cloud data and IMU data, and building a space-time corresponding relation through employing a particle filtering algorithm. Scene semantic understanding is realized based on a Transform multi-modal scene analysis network, scene scores are calculated in combination with spatial density, depth occlusion and shape change features, and an optimal modeling algorithm is selected through a strategy network of deep reinforcement learning training. Meanwhile, scene dynamic features are analyzed by using a deep neural network and a graph neural network, adaptive grid subdivision and geometric constraint reconstruction of a dynamic region are realized, and a reconstruction threshold is dynamically adjusted according to a scene semantic type. According to the method, the three-dimensional reconstruction quality and efficiency of a large-scale complex scene are effectively improved.

The embodiment of the invention provides a scene space model adaptive modeling method based on multi-source data fusion. Complete scene information is constructed by fusing a video sequence, point cloud data and IMU (Inertial Measurement Unit) data. A deep learning network is adopted to carry out scene analysis, a scene complexity score is calculated based on a scale coefficient, a shielding degree and a geometric deformation coefficient, and intelligent matching of a modeling strategy is realized. Meanwhile, a scene dynamic area is predicted by using an optical flow algorithm, a geometric deformation field and a camera pose change, the modeling quality is monitored through reconstruction errors, and self-adaptive switching of a modeling algorithm is realized. The method effectively solves the problems of unstable precision and poor adaptability of a traditional modeling scheme in a complex dynamic scene, and improves the quality and efficiency of three-dimensional scene reconstruction.

The invention discloses a historical block scene three-dimensional reconstruction method and system based on Gaussian sputtering, and the method comprises the steps: collecting a historical block video sequence through a lightweight panorama camera, extracting a multi-frame panorama, and generating an image data set through a projection converter; monocular depth and normal estimation is carried out through a pre-training visual model, and a prior depth and normal graph data set of a historical block scene is constructed; sparse reconstruction is carried out on the multi-view image data set based on the SfM technology, initial point cloud and camera pose information are acquired, and a Gaussian ellipsoid is optimized in combination with prior depth and normal information; dynamically calculating the geometric width estimation value of the street, and guiding and adjusting the adaptive density of the Gaussian ellipsoids of the vertical surfaces on the two sides; rendering the optimized Gaussian ellipsoid through an improved rasterization renderer; and designing a multi-modal loss function and a regularization mechanism to optimize reconstruction and rendering results. According to the method, high-fidelity three-dimensional reconstruction of the historical block scene is realized, and the adaptability of the Gaussian sputtering method to the complex block scene is enhanced.

The invention is suitable for the technical field of video acquisition, and particularly relates to a video stream acquisition method based on deep learning, and the method comprises the steps: carrying out the video acquisition through a self-adaptive strategy, and obtaining real-time collected video data; performing parameter adjustment through edge calculation to obtain preprocessed video data; performing key frame selection on the preprocessed video data to obtain key frame data; performing inter-frame difference compression on the key frame data through the optimized variational auto-encoder model, and outputting video frame compression data; and when the original video needs to be recovered, analyzing the video compression data, reconstructing through the pre-trained neural network model to obtain a reconstructed video sequence, and generating a complete video according to the reconstructed video sequence. According to the method, the non-key frame is directly generated in the decoding process by using the deep learning model, so that the compression ratio can be remarkably improved, and the video playing continuity and the image quality, especially the performance in high-dynamic and complex scenes, can be improved.

The invention discloses a video super-resolution reconstruction method and system, and relates to the technical field of video restoration processing, and the method comprises the steps: collecting a to-be-reconstructed video frame sequence and a reference frame sequence; extracting a dense optical flow field between the reference frame and the target frame; performing geometric alignment on the target frame to the reference frame according to the dense optical flow field to obtain a motion consistency feature; performing feature extraction on the video frame sequence through a parameterized residual scaling module; inputting the extracted features into a bidirectional propagation module to process forward and backward image sequences respectively, extracting time sequence forward features and time sequence reverse features, and fusing the features to obtain detail features; carrying out adaptive fusion on the motion consistency features and the detail features, and sending the fused features into a reconstruction network to obtain deep visual features; pixel-level adaptive reconstruction is realized according to the deep visual features, and a super-resolution video sequence is obtained; according to the method, the feature space consistency is kept, and meanwhile, the feature matching precision in a complex motion scene is remarkably improved.

The invention discloses an autoregressive visual target tracking algorithm based on token fusion, and the algorithm specifically comprises the following steps: 1, obtaining a public visual target tracking data set, and converting the data set into a central coordinate format; 2, preprocessing an initial template of a given video sequence, an online template at the current moment and a search image to obtain visual embedding of the initial template, a dynamic template and the search image; the method comprises the following steps: preprocessing an angular coordinate sequence of a target frame in previous historical N frames of a current search image of an input video to obtain historical track word embedding of the current search image; 3, constructing an autoregressive visual target tracking model based on token fusion, and designing a loss function; step 4, obtaining a final autoregressive visual target tracking model based on token fusion; the method solves the problem of insufficient model feature extraction capability caused by insufficient utilization of spatio-temporal information and redundant background information of a search area in an existing method.

The invention discloses an AIGC multi-mode audio-visual content creation method and system, and belongs to the technical field of multi-mode audio-visual content generation and automatic creation, the AIGC multi-mode audio-visual content creation method comprises the following steps: an input module for inputting text description and materials required by creation by a user; the AIGC generation module is used for converting the text description and the material input by the user into a video sequence and a music clip; the multi-modal fusion and alignment module fuses and aligns the generated video sequence and the music clip to obtain an audio / video; the video intelligent editing module is used for splicing the segments of the audios and the videos and keeping visual coherence and smooth connection between the two audios and the videos; and the output module is used for outputting the generated audio and video as the media format required by the user. The method solves the problems that in the prior art, multi-modal audio-visual content creation lacks the ability of cross-modal cooperative work, the content generated during multi-modal information fusion and alignment is inconsistent in rhythm, semantics and emotion, the support of a later editing process is lacked, the content connection is not natural enough, and the creation speed is low.

The invention relates to the technical field of intelligent inspection, and discloses a regional abnormal condition real-time early warning method based on high-point panoramic intelligent inspection, which comprises the following steps: collecting visible light and infrared thermal imaging video streams of a target region to form a panoramic video sequence, and carrying out intelligent analysis, feature extraction and analysis, construction of a spatio-temporal topological graph and detection of an abnormal behavior mode. Predicting environmental risks, constructing a spatio-temporal evolution model, and further generating graded early warning information of regional abnormal conditions; the method effectively solves the problems of large panoramic inspection data volume, exception complexity, easy environmental influence on target detection and the like in a large-scale scene, significantly improves the accuracy and real-time performance of exception early warning, and guarantees the regional safety.

Embodiments include methods for generating advertisement videos for insertion into a video stream to promote a product, service, or brand in a manner that is consistent with the context and storyline of the video stream before and at the time of ad insertion. Methods may include capturing an image from the video stream and generating caption text using an image-to-text description model. A product, service, or brand that is consistent with the context and storyline of the captured image is selected and ad video sequence description text is generated that includes descriptions and a storyline blending descriptions of the selected product, service, or brand with the context and storyline of the primary video stream. The ad video sequence description text is used to prompt a text-to-video generation model that generates a new advertisement video clip, which is inserted into the primary video stream before distribution to video content rendering devices.

The invention relates to a character-driven lip sound synchronous digital human generation method and device, equipment and a medium, and the method comprises the steps: obtaining an input text and a single object video, and carrying out the preprocessing of the input text, and obtaining a phoneme sequence and semantic features; the method comprises the following steps: preprocessing a single-object video to obtain a circularly played video paragraph, a mask face image and a reference face image; generating voice features based on the phoneme sequence and the semantic features; generating an audio sequence based on the voice features; acquiring face deformation parameters according to the mask face image, the reference face image and the voice features, and deforming the reference face image to generate a target face image; and embedding the target face image into a video frame of the video paragraph to generate a video sequence, and splicing the video sequence with the audio sequence to obtain a target digital human video. The digital human video is generated based on text driving, so that the voice and lip-shaped video matched with the text can be generated in time, the time delay for waiting for text to be converted into voice processing is eliminated, and the digital human video production efficiency is greatly improved.

The invention provides an intelligent event identification method and system based on a high-speed camera, and the method comprises the steps: setting the collection parameters of the high-speed camera, and triggering the camera to collect a target scene video stream. And hardware acceleration decoding processing is carried out on the collected original video data stream, real-time environment illumination information of the environment illumination sensor is obtained, and dynamic brightness equalization processing is executed. And performing motion adaptive denoising processing on the video sequence. Geometric distortion correction is carried out on the image sequence through camera calibration parameters, sub-pixel-level displacement vectors and dense optical flow field data of a moving object are extracted, and multi-scale morphological features are extracted. And the features are fused to generate motion feature data, the data are processed through a spatio-temporal joint event classification model, an event identification result is output, the result is bound with a high-precision timestamp, and event identification information is output to an industrial control system display device in real time. According to the invention, the accuracy and real-time performance of event identification can be improved.

Disclosed is a nighttime unmanned aerial vehicle (UAV) object tracking method fusing a hybrid attention mechanism. According to the method, a night vision image enhancement model is employed to enhance various nighttime image frames in a night vision video sequence of UAV before performing image object tracking and recognition, to improve the accuracy of object tracking for nighttime images. With the aid of a spatial channel hybrid attention mechanism, the night vision image enhancement model can adaptively learn spatial channel features, enhancing generalization ability, and enabling the network model to focus on more effective features. Additionally, a denoising processing module is introduced to remove noise hidden in the nighttime images, improving the object positioning accuracy and tracking performance of UAV in nighttime scenarios, and reducing the failure probability of object tracking by UAV in nighttime scenarios.

The invention provides a multi-mode cooperative sensing power station high-risk operation inspection method and system, and relates to the technical field of video recognition, and the method comprises the steps: activating an unmanned plane and a quadruped robot after a power station operation task is started; starting a video acquisition unit, and establishing a synchronous video stream; carrying out fusion alignment with the global reference coordinate system through an external synchronization signal; inputting the video sequence of the fused view angle into a multi-view angle action behavior recognition network, and establishing a dangerous behavior grade score; auditory data and olfactory data of the quadruped robot are obtained, and linkage abnormity is established; and polling abnormity is reported according to linkage abnormity and dangerous behavior grade scores. Through the method and the device, the technical problem of low inspection efficiency caused by difficulty in comprehensively identifying potential risks in a dynamic environment due to limitation of a single sensing mode is solved, and the inspection efficiency of a power station is improved by fusing multi-mode data, timely finding and processing the potential risks and improving the accuracy of video and audio identification.

The invention relates to the technical field of deep learning semantic segmentation, in particular to a video sequence segmentation method based on a selective scanning visual state space model, which comprises the following steps of: inputting an image sequence into an overlapped feature block division layer, and dividing the input image sequence into an image feature block sequence; inputting the image feature block sequence into a state space model encoder based on selective scanning vision to extract thickness feature sequences on different scales; inputting the coarse and fine feature sequences on different scales into a multi-layer perceptron layer for effective fusion of multi-level features to obtain a feature sequence; inputting the feature sequence into a lightweight de-wharf prediction segmentation mask based on a convolutional neural network, and performing visual visualization to generate a semantic segmentation map; in this way, the technical problems that in the prior art, the acceptance range of a traditional convolutional neural network is limited, and a self-attention-based network has extremely high calculation complexity in the aspect of constructing long-term dependence, so that the image sequence segmentation effect is poor are solved.

Methods, systems, and bitstream syntax are described for video coding and decoding using source picture timing information which is captured by an encoder and is signaled as metadata to a decoder to assist in decoding. The proposed methods include example syntax for signaling source picture timing metadata as supplemental enhancement information (SEI) messaging for both single-layer and multi-layer video sequences.

The invention relates to a depth forgery detection method, device and equipment based on cross-modal mask modeling. The method comprises the following steps: constructing a deep forgery detection model comprising a multi-mode mask modeling module, a trans-attention fusion module and a contrast learning module; performing training optimization on the deep forgery detection model by integrating the audio and video reconstruction loss and the audio and video comparison learning loss; and inputting a to-be-detected audio and video sequence into the trained deep forgery detection model, outputting to-be-detected audio and video trans-attention coding features according to the model, carrying out audio and video matching correlation detection by inputting the to-be-detected audio and video trans-attention coding features into the classifier, and outputting a forgery detection result of the audio and video sequence. By adopting the method, the characterization of a real video can be learned by utilizing a mask modeling technology, and the characteristics of an audio-visual mode are subjected to cross fusion and comparative learning, so that the forgery detection performance is improved.

The invention discloses an academic emotion recognition method and device based on facial expressions and contexts. According to the method, accurate emotion modeling is realized by utilizing a visual language pre-training model CLIP and combining student facial expressions in a video sequence and contextual information of a learning situation. Specifically, according to the method, firstly, collaborative modeling is carried out on facial expressions and context information in a learning scene, so that the ability of recognizing the learning state (such as focusing on learning and distracting) of students is remarkably improved; and secondly, the strong language and visual feature alignment capability of the CLIP pre-training model is utilized, so that the dependence on large-scale video training data is effectively reduced, and the generalization capability and practicability of the model are improved. An accurate and feasible solution is provided for academic emotion recognition, and the method is suitable for monitoring and analysis of academic emotions of students in an education scene.

The invention discloses a plastic effect simulation display method and system based on a 3D simulation technology, and relates to the technical field of medical information, and the method comprises the steps: collecting the facial data of a patient, and constructing a multi-level facial structure initial model; constructing a hybrid deformation model; individual characteristic parameters of a patient are extracted, and an individual biomechanical parameter set is generated through the biomechanical parameter prediction network; applying the personalized biomechanical parameter set to the mixed deformation model; operation input parameters of plastic surgery are received, and the deformation result of each layer of structure is calculated and displayed in real time based on a GPU parallel computing framework; constructing a tissue healing and recovery kinetic model, and generating appearance prediction data at different postoperative time points in combination with the deformation result; and performing three-dimensional visualization processing on the appearance prediction data. According to the method, CT / MRI, 3D structured light scanning and expression dynamic video sequences are creatively integrated, the limitation of traditional single-mode data modeling is broken through, and the soft tissue boundary recognition precision is remarkably improved.

The invention relates to the technical field of artificial intelligence, and provides a video analysis method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a target video sequence and a text prompt word of a user; the target video sequence comprises a plurality of images, and each image comprises a to-be-analyzed target selected by a user; performing feature extraction on the target video sequence to obtain visual features of the target video sequence; performing feature extraction on the text cue words to obtain text features of the text cue words; and inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the to-be-analyzed target output by the large language model. Through the above mode, the analysis and description of the specific target in the video can be completed, and the interested content of the user can be focused.

The invention discloses a non-contact heart rate detection method, system and device based on visual Transform and multi-scale feature aggregation and a medium. The method comprises the steps that a face visible light video is obtained, the unified video frame rate is re-sampled through the frame rate, face key point positioning and region division are carried out on each frame of image, and a face visible light video sequence is obtained; performing time migration operation on the video sequence to obtain a difference frame sequence, performing channel fusion on the video sequence and a difference frame, and performing down-sampling on a spatial dimension to obtain a low-resolution feature tensor; constructing a non-contact heart rate extraction model, inputting the low-resolution feature tensor into the non-contact heart rate extraction model, and outputting a heart rate value of the target; the non-contact heart rate extraction model comprises a feature enhancement module, a multi-scale mask feature aggregation module, a Transform time sequence modeling module and an rPPG predictor, the system, the device and the medium are used for achieving the non-contact heart rate detection method based on visual Transform and multi-scale feature aggregation, and the precision of rPPG signal extraction is improved.

A system, a method and a processor are for AI-powered generation and delivery of video clips. The processor is configured to: load a first video file of a first video content item, the first video file comprising video frames associated with timestamps; load a first subtitle file of the first video content item, the first subtitle file comprising subtitle text associated with the timestamps; execute a natural language processing (NLP) model with the subtitle text as input, the NLP model including language pre-processing steps for classifying words, names or phrases in the subtitle text and associating initial classifiers with the subtitle text, the NLP model including one or more of a recurrent neural network (RNN), a Bidirectional Encoder Representations from Transformers (BERT) model, or a generative pre-trained transformer (GPT) model for a dialogue analysis comprising processing sequences of dialogue in the subtitle text in view of the initial classifiers to associate one or more portions of the dialogue with one or more first classifiers of first narrative elements; execute an image recognition model with at least some of the video frames as input, the image recognition model including a convolutional neural network (CNN) for an object detection analysis and a facial recognition analysis comprising processing video sequences to associate one or more of the video frames with one or more second classifiers of second narrative elements; generate a narrative map of the first video content item by temporally aligning the first narrative elements with the second narrative elements based on the timestamps associated with the video frames and the first subtitle file; and generate a video clip including at least one segment of the first video content item, the at least one segment including selected video frames associated with at least one of the first or second narrative elements identified from the narrative map and selected for inclusion in the video clip.

The invention discloses a guideboard surface deformation detection method and system based on a dynamic video sequence. The method comprises the following steps: acquiring road video data with a timestamp and position information; a traffic sign is automatically identified from the road video data, and geometric anomaly detection is carried out; a time sequence tracking sequence is established for the guideboard, and whether continuous abnormity exists in the guideboard is determined through multi-frame analysis; if yes, performing local texture consistency and local contour geometric analysis on the guideboard to obtain a fusion analysis result; performing state evaluation on supporting equipment corresponding to the guideboard to obtain a state evaluation result of the guideboard supporting equipment; and determining the final surface deformation state and the corresponding grade of the guideboard based on the fusion analysis result and the state evaluation result of the guideboard supporting equipment. By implementing the method provided by the invention, automatic detection and quantitative analysis of the bending or deformation condition of the road sign can be realized without manual intervention, and early warning information can be output in time.

A system and method for person identification from video data by disentangling biometric identity features from non-biometric appearance and activity features are disclosed. The system processes RGB video sequences depicting individuals performing various activities to extract spatio-temporal features. These features are separated into distinct biometric identity representations and non-biometric features related to appearance and performed activities. To achieve this separation and minimize appearance bias, the system utilizes an auxiliary supervisory model. At least two implementations of this supervisory model are disclosed: one using semantic supervision via structured embeddings processed through a vision-language model, and another employing silhouette-based feature distillation from a silhouette-trained neural network. Joint training for biometric identification and activity classification ensures accurate identification of individuals independently of facial visibility, clothing differences, or activity variations.

The invention belongs to the technical field of video processing, and particularly relates to a monitoring video enhancement method for a farm, which aims to solve the technical problem of low quality of an enhanced video in the prior art, and comprises the following steps: S1, processing each frame of image in a monitoring video sequence frame by frame; s2, distinguishing a target animal area from a background area, and identifying and generating an artifact mask; s3, aiming at the background area, carrying out key smoothing processing on the artifact position to inhibit the artifact; s4, for the image of the target animal area, performing adaptive nonlinear enhancement on the brightness component, and performing color correction on the chrominance component; s5, performing pixel-level fusion on the enhanced target animal area and the background area; and S6, spreading the information of the previous frame to the current frame by using the forward optical flow field, and carrying out weighted fusion on the information of the previous frame and the current frame. According to the method, the target bred animals, the background areas and the artifacts are accurately distinguished, so that refined and differentiated processing of pictures is realized.

The invention relates to the technical field of medical image processing, and discloses a craniofacial dynamic reconstruction method and system based on multi-modal data fusion, and the method comprises the steps: arranging a multi-modal data collection device in a target craniofacial region, and obtaining a static CT image, a static MRI image, a dynamic expression video sequence and a surface electromyogram signal; preprocessing the static CT image, the static MRI image, the dynamic expression video sequence and the surface electromyogram signal; inputting the preprocessed data into a multi-scale finite element model, and simulating a coupling relationship between muscle contraction force and skin deformation by adopting a biomechanical driving strategy to generate a dynamic craniofacial model; and fusing the geometric error and the motion consistency score of the real data by adopting a linear regression method, and outputting a comprehensive reconstruction quality index. According to the method, the problems of low craniofacial dynamic modeling accuracy and poor robustness in the prior art can be solved.

The invention provides an ultrasonic image discrimination perception pre-training method based on a cooperative training framework, and relates to the technical field of ultrasonic image analysis, and the method comprises the steps: obtaining an ultrasonic image sequence containing time sequence information; constructing a cooperative training architecture, wherein the two networks are both based on a visual converter and embedded with a time sequence anatomical attention module; inputting the original image into a teacher network, inputting the enhanced image into a student network, and respectively outputting global and local features; calculating an anatomical continuity measure based on the output features of the two-network time sequence anatomical attention module; constructing a total loss function; the regularization weight is dynamically adjusted according to the anatomical continuity measurement, and student network parameters are updated through gradient descent; dynamically calculating an index moving average coefficient according to the anatomical continuity measurement so as to update teacher network parameters; and iterative training is carried out until convergence. According to the method, the time sequence continuity and anatomical structure information contained in the ultrasonic video sequence are fully mined, so that the adaptive optimization of the model training process is realized.

The invention relates to the field of medical image segmentation, and discloses a precision surgery-oriented robot medical image segmentation and feature extraction method, which comprises the steps of constructing a dynamic segmentation network model of a bidirectional attention architecture, and inputting a preprocessing module for feature extraction and generating a multi-scale feature pyramid; the Transform coding branch is used for time sequence feature modeling and outputting a time sequence enhancement feature; the convolutional coding branch is used for enhancing anatomical features and surgical instrument features and outputting spatial enhancement features; the multi-stage feature fusion unit is used for performing multi-stage iterative fusion and outputting final fusion features; the decoding output module is used for decoding and generating pixel-level segmentation masks of the anatomical structure and the surgical instrument; training the dynamic segmentation network model; and performing medical image segmentation and feature extraction based on a medical image video sequence input in real time by using the trained dynamic segmentation network model. Accurate segmentation of anatomical tissues and dynamic instruments in an operation scene is realized.

The invention provides a long-term video understanding system based on adaptive sparse memory and a language model. The long-term video understanding system comprises a visual encoder used for extracting visual features from a long video; the memory bank module is used for storing and retrieving visual features of historical video contents; and the sparse adaptive module is used for dynamically managing the memory bank module, and the memory bank module interacts with the multi-modal large language model by querying a converter Q-Former and is used for incrementally processing video data and mapping visual features to a language space. By introducing an adaptive sparse memory mechanism, a long-term video sequence can be effectively processed, redundant features can be dynamically compressed, and key information can be reserved, so that efficient analysis of a long video is realized; compared with the prior art, the method has high accuracy in multiple tasks, and can dynamically manage the memory bank and reduce processing of redundant features through a sparse adaptive mechanism, so that the calculation overhead is reduced, and the overall efficiency of the system is improved.

The invention discloses a multi-modal collaborative video sequence segmentation method. The method comprises the following steps: obtaining a multi-scale local feature matrix and a multi-scale global feature matrix of an image sequence; obtaining a multi-scale text feature matrix of the text sequence; obtaining a multi-scale local-global fusion feature matrix of the multi-scale local feature matrix and the multi-scale global feature matrix; obtaining a multi-modal fusion feature matrix of the multi-scale local-global fusion feature matrix and the multi-scale text feature matrix; and utilizing a decoder of the pre-trained large model to predict and generate a segmentation mask, and outputting a semantic segmentation map. The video sequence segmentation method is stable in performance when facing complex and changeable scenes, does not need to depend on a large amount of labeled data, reduces the training cost, and is suitable for various practical application fields including intelligent monitoring, automatic driving, medical image analysis and the like.