273 results about "Depth perception" patented technology

The invention discloses a flotation froth dynamic diagnosis and self-adaptive regulation and control system based on multi-mode depth perception and time sequence prediction. The flotation froth dynamic diagnosis and self-adaptive regulation and control system aims at solving the problems that in the prior art, the flotation process is not comprehensive in monitoring perception, dynamic prediction is missing, and regulation and control self-adaptability is poor. According to the invention, by deploying a multi-source heterogeneous sensor array, multi-modal data of vision, spectrum, acoustics and the like of foam are synchronously collected; and generating comprehensive foam comprehensive state characterization by using a cross-modal attention fusion network. Modeling is carried out on dynamic evolution of foam by adopting a hierarchical time sequence prediction and anomaly detection network, the future state trend is accurately predicted, and early warning of anomaly is realized. And finally, an intelligent regulation and control agent based on deep reinforcement learning is constructed, the intelligent regulation and control agent autonomously decides optimal process parameter adjustment according to the current state and future prediction, and online learning and optimization are carried out through continuous interaction with the actual process. The beneficiation recovery rate, the grade and the stability of the production process are remarkably improved, and the operation cost is reduced.

The invention discloses a monocular depth estimation method based on hybrid CNN-Transformer, belongs to the aviation field and the computer vision field, and is applied to the field of environment depth perception in low-altitude flight of an unmanned aerial vehicle. Aiming at the problem that local-global information cannot be effectively extracted by a current depth estimation method, the design content mainly comprises the following steps: improving a visual Transform architecture based on an additive attention mechanism, optimizing feature interaction operation between image blocks, and improving the capability of capturing global semantic information; the method comprises the following steps: designing a deep network encoder mixed with a CNN-Transform module, and effectively extracting local-global feature information; a channel attention mechanism is introduced, and the feature fusion capability of the decoder is enhanced. The method is simple in process, an excellent test result is obtained through training on the flight data set Mid-Air of the low-altitude unmanned aerial vehicle, the extraction capability of local-global feature information is effectively enhanced, and the depth perception accuracy of the monocular depth estimation method in the flight scene of the low-altitude unmanned aerial vehicle is improved.

The invention provides a depth map generation method and device based on a large model, a three-dimensional reconstruction method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, can be applied to real-time road scene depth perception, environment three-dimensional reconstruction and obstacle avoidance, and can be applied to real-time road scene depth perception. And virtual and real scene fusion and other scenes can be realized. The specific implementation scheme is as follows: performing visual coding on a monocular image to obtain a coded image; inputting the coded image and the target text into a pre-trained large language model for fusion to obtain fusion features; generating global guide features based on the fusion features, wherein the global guide features comprise joint semantic information of visual features and text features; adding noise to the color image of the monocular image to obtain a noise feature sequence; de-noising the noise feature sequence under the condition of the global guide feature, and generating an implicit feature matched with the joint semantic information; a depth map is generated based on the implicit features.

The invention discloses a naked-eye 3D display optimization method based on real-time eyeball tracking, and particularly relates to the technical field of naked-eye 3D display, and the method comprises the steps: capturing eyeball movement data in real time through a visual angle tracking sensor, and collecting illumination information in combination with an ambient light sensor; depth perception parameters are calculated based on pupil diameter variation and frequency, and a time sequence prediction type dynamic compensation coefficient is generated in combination with eyeball movement acceleration; acquiring an initial fixation point coordinate by using an improved spherical projection mapping model, and performing compensation coefficient correction to obtain a real-time coordinate; and finally, according to the real-time coordinates, dynamically adjusting the refractive index distribution of the nanostructure layer, the rotation angle of the polarizer, the focal length of the optical lens and other optical modulation parameters. The real-time distance is calculated through the binocular parallax algorithm, the depth mapping value is generated by combining the focal length of the camera and the baseline distance, the method can adapt to the illumination change and the user view angle, and the 3D display effect is optimized.

A radar and camera sensor combined detection method, comprising: obtaining a plurality of image perception data groups and a plurality of raw radar perception data groups of a target object, the target object comprising a moving target object and a stationary target object (S11); according to a first data processing method, processing the plurality of raw radar perception data groups to obtain a plurality of radar sparse tensor information groups (S12); and forming a stereoscopic multi-dimensional depth perception data combination and outputting same, the stereoscopic multi-dimensional depth perception data combination being formed in a multi-dimensional matrix structure, and the stereoscopic multi-dimensional depth perception data combination comprising a plurality of radar sparse tensor information groups assigned to the multi-dimensional matrix structure and a corresponding plurality of image perception data groups (S13). The method solves the problems of how to effectively identify static objects, occluded objects, etc. and process a large amount of radar raw data.

The invention discloses an industrial robot intelligent obstacle avoidance control method and system based on visual identification, and relates to the technical field of robot obstacle avoidance control, and the method comprises the steps: carrying out the calibration of a multi-mode visual sensor, obtaining an initial depth map and point cloud information, and combining the visual identification and depth compensation technology; recognizing and positioning obstacles in the operation area, and constructing a dynamic environment map layer; and the current robot state is collected, kinematics calculation and collision distance analysis are carried out, whether obstacle avoidance operation needs to be executed or not is judged, if yes, an obstacle avoidance path is generated in combination with the dynamic environment map layer and the target point location, feasibility verification is carried out after the path is generated, and the path passing the verification serves as an execution track to be issued to the control module. According to the invention, the recognition precision and depth perception integrity of the industrial robot on obstacles in a complex environment are improved, high feasibility of path planning and high-reliability obstacle avoidance capability in a dynamic environment are realized, and the intelligent decision-making level and operation safety of the system are remarkably enhanced.

The invention discloses a human body posture estimation method, system and device and a medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a to-be-estimated picture containing a human body; the method comprises the following steps of: performing feature extraction and down-sampling on a to-be-estimated picture, performing dimension raising, depth separable convolution operation, channel aggregation operation and dimension reduction on a feature map with down-sampling resolution in an output branch after down-sampling to obtain a local feature map, performing up-sampling reconstruction on the local feature map by adopting dynamic weight interpolation, and fusing an output branch which is not down-sampled to obtain a local feature map; obtaining a first fusion feature; taking the first fusion feature as an initial feature, repeating the step of obtaining the first fusion feature, obtaining a second fusion feature, carrying out fusion to obtain a dual-scale fusion feature, extracting a depth perception feature in the dual-scale fusion feature, carrying out human body posture estimation through the depth perception feature, and obtaining a human body key point heat map. According to the invention, multi-scale and global information is obtained through a lightweight structure, and accurate key point positioning is obtained.

The invention relates to the technical field of target detection of automatic driving, in particular to a 4D multi-target sensing and tracking method and system based on sparse representation, which is an efficient 3D target detection algorithm, and through dynamic interaction of sparse 4D query vectors and multi-view and multi-scale features and in combination with a time sequence instance denoising and depth sensing enhancement module, the accuracy of target detection is improved. And automatic driving 3D perception with high precision and low calculation amount is realized. Benefited from a sparse query architecture, the method also inherits the advantage of high efficiency of a sparse method while keeping high precision. Through a series of targeted optimization, the generalization ability and robustness of the model in various special scenes are significantly enhanced.

The invention discloses a binocular vision depth estimation method and device, equipment, a medium and a product, and relates to the technical field of robot environment perception, and the method comprises the steps: obtaining a binocular image sequence which comprises a plurality of continuous target image pairs; each target image pair comprises a left eye image and a right eye image, inputting the target image pairs into the trained parallax estimation model to obtain a final prediction parallax image of the target image pairs, and calculating depth information according to the final prediction parallax image of the target image pairs and camera parameters of the binocular camera. And when the robot cannot carry a laser radar or a depth camera, depth perception based on pure vision is realized.

The invention discloses a multi-view three-dimensional reconstruction method based on depth perception, and relates to multi-view three-dimensional reconstruction. Through a multi-scale feature extraction framework, in combination with a feature pyramid and a double-branch deep fusion network, high-discrimination-force features are extracted from images of multiple visual angles. Then, utilizing microhomography transformation to map the two-dimensional features to a three-dimensional depth hypothesis plane, constructing a cost body, and emphasizing feature relevance between views through a grouping aggregation strategy; and the cost body is regularized by adopting 3D U-Net in combination with a deep collaborative attention mechanism so as to reduce mismatching and redundancy. Depth map estimation is regarded as a classification task, a depth hypothesis probability is calculated by using Softmax, an optimal depth value is selected through Argmax, and a multi-view depth map is generated. The whole process is trained by adopting a cross entropy loss supervision model, and high-precision three-dimensional reconstruction is realized. The method effectively improves the precision and efficiency of three-dimensional reconstruction, and shows higher precision and robustness in the three-dimensional reconstruction of a complex scene.

A system and method are provided for estimating a depth of an object within an image and training a depth estimation network. The depth estimation method includes: obtaining single-frame image data; obtaining scaling factor data based on the single-frame image data; generating scale-invariant depth data through inputting the single-frame image data into a depth estimation network; and generating metric depth data based on the scaling factor data and the scale-invariant depth data. The training method includes: inputting image data into a teacher machine learning (ML) model in order to generate metric depth data; inputting image data into a student ML model in order to generate scale-invariant depth data; and training a student network based on loss calculated using the metric depth data and the scale-invariant depth data.

The invention discloses a food nutrition evaluation method and system based on multi-modal fusion. The method comprises the steps of 1, extracting multi-scale features of a food RGB image and an RGB-D depth image in parallel through a multi-scale double-branch Transform encoder; 2, performing cross-modal fusion on the multi-scale features by using a depth perception enhanced attention fusion module; step 3, constructing nutrition prediction branches based on the fused features, and outputting five macro nutrient contents; and step 4, training the training data set on the model in an end-to-end mode, and inputting the RGB image and the RGB-D image of the food to be detected into the trained model to predict the nutrient content. According to the method, perception of volume information of different food types is enhanced by using the RGB-D depth mode corresponding to the food RGB image, visual information in the food RGB image and spatial physical characteristics in the RGB-D image are fused through the depth perception attention enhancement module, so that fusion characteristics with identification information are generated, and the evaluation accuracy is improved.

The invention discloses a multi-point radar camera fusion detection method based on a bird's-eye view, and the method comprises the steps: constructing an improved BEVDepth network model, employing sparse but precise radar points, introducing a radar auxiliary view conversion method RVT, converting the image features of a perspective view into a bird's-eye view BEV, making up the defects of an image in depth perception, and improving the detection precision of the bird's-eye view BEV. And more comprehensive space understanding is provided for the automatic driving system. A multi-modal deformable attention mechanism is used to further aggregate an image and a radar feature map in BEV so as to eliminate spatial dislocation between different sensor data, thereby improving perception precision. On the basis of adopting a BEVDepth model, the training of the BEVDepth model, a backbone network and the tail end of the backbone network are improved, radar data are fused, the domain invariance of the network is enhanced, and the recognition capability of the system on a 3D target is improved.

The invention relates to an intelligent adjusting system and method for the posture of an automobile seat, and belongs to the technical field of intelligent cabins. The method comprises the steps that S1, when a driver prepares to get on a vehicle, a camera which is installed on a left rearview mirror and has the three-dimensional depth perception capability is used for capturing body feature image data of the driver; s2, processing the obtained body feature image data by using a human skeleton tracking and identifying module, and extracting individual feature parameters of the driver, including three-dimensional node data of the head, the neck, the shoulders, the elbows, the wrists, the spine, the hip, the knees and the ankles; and S3, on the basis of the extracted three-dimensional node data, the optimal adjustment amount of each motor is calculated through a seat posture adjustment mathematical model, the optimal adjustment amount comprises the front-back displacement, the up-down height, the headrest height, the backrest inclination angle and the seat cushion inclination angle of the seat, and it is ensured that the driver can obtain the optimal sitting posture. Accurate and personalized adjustment of the posture of the automobile seat is achieved.

An optical display device arranged to generate in an output aperture a virtual a sky scene which is configured to make a user experience distant depth perception, the device comprising: an output light generation system for generating output light comprising a three-dimensional arrangement of a plurality of light control members, which are carried by one or more optically transparent substrates, and a light source arrange to project light to and / or from the light control members, and; an output aperture for output of the output light, wherein the light control members are configured for manipulation of the output light to create a distant object component with one or more features that appear to an observer gazing into the output aperture to be at an apparent depth ZA beyond a physical depth ZD of the device.

Systems described herein use either multiple image sensors or one image sensor and a complementary sensor to provide underwater depth perception. The systems include a computing system that provides the underwater depth perception based on data sensed by the multiple image sensors or the one image sensor and the complementary sensor. The systems can include a submersible device (such as a submersible mobile machine) that includes a holder configured to hold the one image sensor. The holder can be configured to hold the computing system in addition to the one image sensor. And, in some embodiments, the holder is configured to hold the complementary sensor in addition to the computing system and the one image sensor. Alternatively, in some embodiments, the holder is configured to hold the multiple image sensors. And, the holder can be configured to hold the computing system in addition to the multiple image sensors.

The invention discloses a 3D GS cultural relic digital reconstruction method and system based on a block chain, and the method comprises the steps: collecting the RGB image data and depth perception data of a cultural relic, eliminating the influence of different shooting conditions through an illumination separation processing technology, and building a standardized image data set; recognizing a surface area suitable for reconstruction based on image analysis, determining feature point distribution by using kernel density estimation, and generating initial three-dimensional representation through Gaussian ellipsoid fitting; performing gradient calculation and feature extraction on the depth data, and combining with Gaussian representation to form a geometric constraint mechanism; self-adaptive encryption based on visual importance is executed for a sparse region, and a layered rendering effect is achieved through opacity parameter adjustment; the rendering characteristics and the conversion relation of different view angles are analyzed, key observation points are determined through stability analysis, and a smooth multi-view-angle display sequence is constructed; and integrating multi-view rendering information to generate volumetric representation, and completing right confirmation of the high-quality three-dimensional digital model through digital signature.

The invention discloses a scene-driven high-generalization federal reinforcement learning automatic driving system and a scene-driven high-generalization federal reinforcement learning automatic driving method, which are used for simulating learning experts to learn implicit expert preferences from expert demonstration and realizing multi-angle guidance of reinforcement learning student feature extraction, reward function modeling, loss function construction and group optimization in combination with dynamic driving suggestions. The problem of reinforcement learning target alignment is solved in a targeted manner; learning experts are simulated, the relation among multiple learning tasks is enhanced through cross attention, and the aerial view reconstruction process under the embedded guidance of scene depth perception and scene semantic perception is achieved. Through cross-scene optimization, specific experience sharing among different expert data distributions is realized, and high-universality imitation learning experts are trained; motion planning, reward feedback and dynamic driving suggestions are output through a proposed reward function decoder to serve as training and optimization bases of reinforcement learning students, and learning of driving rules is achieved. And through combination of the time sequence processing module and knowledge distillation, the representation capability of original image input is improved.

The invention provides an ultrasonic transmission CT (Computed Tomography) imaging method and system, a computer and a storage medium, which can be used for quickly reconstructing an ultrasonic transmission image with high resolution and high contrast. The method comprises the following steps: in layer-by-layer scanning of an annular ultrasonic transducer array, calculating a TOF (Time of Flight) matrix containing transmission time of all ultrasonic paths after the annular ultrasonic transducer array rotates by a specific angle each time; after one layer of scanning is completed, the TOF matrixes of different angles are recombined into a three-dimensional matrix according to the spatial relation, and missing data are complemented; discretizing an area to be imaged into two-dimensional grids, making each grid correspond to a pixel point, counting transmission time and path length of all ultrasonic paths penetrating through the pixel point, calculating sound velocity contribution values of the paths, taking an average value as a sound velocity value of the pixel point, and generating a two-dimensional ultrasonic transmission image; a two-dimensional image is converted into a three-dimensional image through a ray casting algorithm, the visual influence of voxels is adjusted by introducing a weight factor based on depth, and the depth perception and semitransparent rendering effect of the image are enhanced.

The invention provides a phytoplankton chromatography sequence identification method and a phytoplankton chromatography sequence model building method, and belongs to the technical field of image enhancement identification. The method comprises the following steps: firstly, acquiring microscopic chromatography sequence data of phytoplankton, performing view field extraction and serialization recombination, and constructing a three-dimensional data set; then, constructing a three-dimensional recognition model containing physical perception and a sequence aggregation mechanism, extracting single-frame semantic features by the model by adopting a parameter-shared twin network, and introducing a physical definition prior module to calculate a space-frequency domain quality score of a slice; secondly, designing a deep perception sequence aggregation module, and adaptively aggregating key features of a high signal-to-noise ratio by taking definition scores as gating signals and combining spatial context information between slices; and finally, training and optimizing the model based on the image-level weak supervision label to obtain an optimal model. According to the method, the problems of information truncation and out-of-focus noise interference caused by extremely shallow depth of field of high-power microscopic imaging are solved, and full-depth-of-field stereoscopic perception can be realized under the condition that frame-by-frame fine labeling is not needed.

The invention belongs to the technical field of image processing and three-dimensional modeling, and particularly relates to an SLAM three-dimensional modeling method suitable for a complex dynamic scene. By constructing a multi-modal feature hierarchical processing architecture, the system realizes technical breakthrough in three dimensions: firstly, in combination with a pixel mask pattern and a multi-view geometric method, static feature points are reserved while dynamic feature points are removed as far as possible; secondly, according to the optimal flow field mapping relation of the grid nodes, static background restoration of the shielded area is achieved by establishing the geometric constraint relation of adjacent key frames; and finally, the depth perception data and the optimized feature set are fused to construct a tightly coupled three-dimensional reconstruction model, and a closed-loop optimization framework with motion robustness is formed.

The invention discloses a cross-modal fusion and multi-frequency coding view angle image generation method in an ultra-low orbit scene, and solves the problem of view angle condition generation and single image three-dimensional reconstruction method based on diffusion prior in the ultra-low orbit scene of an aircraft target. Due to insufficient condition information fusion capability, limited radiation or illumination characterization and insufficient cross-view-angle geometric constraint, a problem that a new view angle image with visual coherence and geometric consistency of texture details is difficult to obtain at the same time under large-range view angle and scale change is solved; according to the method, multi-order angle information of direction distribution is effectively captured through perceptual coding of a light direction vector, then the position of a light starting point coordinate is coded, and finally the pose of a camera pose parameter is coded, so that the influence of a relative pose on depth perception and perspective distortion in an image generation process can be depicted; the three coding results are used for a decoding stage of the potential diffusion model, and geometric consistency of detail recovery under a new view angle is effectively improved.

The invention discloses a three-dimensional light field content construction and editing method, which relates to the field of naked eye three-dimensional display, and comprises the following steps of: collecting multi-view images of a real scene through a camera, and constructing a unified three-dimensional space coordinate system; generating an initial three-dimensional scene model; performing adjustable content editing on each frame of multi-view image; updating the initial three-dimensional scene model through the edited image; the updated three-dimensional scene model is rendered through a shear type rendering method, and an image sequence with multi-view consistency is generated; and coding the generated image sequence into three-dimensional light field content by using a light field coding algorithm. According to the method, the real static scene can be efficiently converted into the three-dimensional light field content with a complete structure and a continuous visual angle through multi-visual-angle image acquisition, three-dimensional reconstruction and rendering processing, so that the method has stronger sense of space depth and reality, and the visual experience and the display effect of the three-dimensional light field content are remarkably improved.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a layout generation method, device, equipment and medium based on multi-granularity attention, and the method comprises the steps: obtaining original depth data, carrying out the standardization processing of the original depth data, and obtaining the standardized depth data; the method comprises the steps of obtaining standardized depth data, extracting a multi-scale depth perception feature set from the standardized depth data, generating a multi-granularity attention weight set based on the multi-scale depth perception feature set and the standardized depth data, performing weighted fusion processing on the multi-granularity attention weight set to generate fusion features, and generating layout data based on the fusion features. Through a depth-driven feature extraction mechanism and an attention resolution regulation and control method, granularity self-adaptive modeling of different space depth regions is realized, so that the detail retention capability of a near-end region is improved, the calculation redundancy of a far-end region is reduced, and the spatial resolution capability of layout output and the overall modeling efficiency are effectively enhanced.

The invention provides a personalized image content generation and optimization method and system fusing generative AI, and relates to the technical field of AI. The personalized image content generation and optimization method comprises the steps of analyzing a historical interaction track and a current intention expression of a target user based on a deep perception network, and constructing a multi-dimensional behavior portrait; user groups with similar generation preferences are identified by using a group feature recursive quantization technology, and a group wisdom feature map is constructed; performing multi-level mapping analysis on the user features and the group wisdom feature map, anchoring the group affiliation relationship of the target user and concretizing the guide features; constructing a multi-level guide vector according to current intention expression and guide features, performing accurate mapping on hidden space expression of an image generation model, quantizing deviation in real time and triggering intelligent calibration; and reconstructing a group distribution structure based on the evaluation data of the target user to realize dynamic evolution of the features. According to the invention, personalized demands of users can be accurately grasped, and the accuracy of image generation and the satisfaction degree of the users are improved.

The invention relates to the field of image processing and target detection, in particular to a crack detection and segmentation method based on depth perception and structural feature enhancement. The method comprises the following steps: acquiring original image data containing cracks; performing multi-scale feature extraction on the image; multi-scale feature extraction is realized in combination with a C3K2SAConv module capable of switching cavity convolution; introducing a SimAM attention mechanism to enhance crack area response; a CCM crack convolution module is designed to extract a slender and fractured crack edge structure, and detection and segmentation accuracy and boundary integrity are improved; and finally completing positioning and segmentation output of the crack region. The method improves the recognition accuracy and edge integrity of the structural crack in a complex background and low contrast environment, and is suitable for application scenes of structure monitoring, intelligent maintenance, automatic driving environment perception and the like of traffic infrastructures such as roads, bridges and the like.

The invention provides an intelligent traffic cone barrel anti-collision early warning method based on binocular vision, and aims to improve the safety of a road construction operation area. And accurate data are provided for subsequent detection and analysis through a binocular camera high-precision calibration technology based on circle detection. In combination with a data acquisition and enhancement technology, high-quality training data is provided for an improved RT-DETR algorithm, and the accuracy and efficiency of vehicle identification are significantly improved. A binocular vision depth perception and StrongSORT multi-target tracking algorithm is utilized to realize real-time speed measurement and distance measurement of a vehicle, and accurate input is provided for track prediction. Based on a GAN multi-modal trajectory prediction technology, a plurality of future trajectories are generated, and a multi-level early warning decision is realized through a comprehensive collision risk index (CRI). According to the method, the limitation of an existing prediction method in a complex road scene is effectively solved, the vehicle behavior intention is predicted and analyzed through the multi-modal trajectory, and reliable early warning information can be provided under various complex road conditions.

The invention discloses an intelligent multi-view camera tracheal intubation image processing method, device and system, and relates to the technical field of medical instruments, the method comprises the following steps: in a tracheal intubation process, a multi-view image acquisition unit collects a plurality of images and combines the images to form an image pair; and carrying out stereo matching and depth calculation on the image pairs to generate a depth map. And performing three-dimensional reconstruction by combining the depth map and the original image to generate a three-dimensional model. And finally, identifying the target in the three-dimensional model and outputting the depth of the target as an image processing result. The technical problems that an existing multi-view camera is inaccurate in depth perception and insufficient in real-time performance during tracheal intubation image processing are solved, and the technical effect of improving the depth perception precision and the real-time processing capacity of the multi-view camera during tracheal intubation image processing is achieved.

The invention relates to the technical field of robot control, in particular to a robot autonomous navigation method and device, a robot and a computer readable storage medium, and the method comprises the following steps: obtaining real-time body state information, a current position, a target position and depth perception information of an external environment of the robot; encoding the depth perception information, and splicing the encoded depth perception information with the real-time body state information to obtain unified features representing the external environment and the state of the robot; inputting the unified feature and the target position into a preset navigation decision model to generate a preliminary movement speed of the robot; the linear distance between the current position and the target position is calculated in real time, the initial movement speed is adjusted based on the linear distance, and a stable expected speed parameter is obtained; and according to the stable expected speed parameter and the real-time body state information, generating an execution instruction for driving the robot to move and controlling execution, and navigating to a target position to realize autonomous navigation of the robot in a map-free scene.

The invention discloses a voice-driven face video generation method and device based on depth perception fusion, and relates to the field of computer vision and image processing, and the method comprises the steps: S1, obtaining a face speaking video data set with an audio clip and a reference image, carrying out the preprocessing of the data set, and dividing the data set into a training data set and a test data set; s2, constructing a face video generation model; comprising an audio encoder, an image encoder, a depth encoder, a cross reference module and a cross-modal attention module. S3, training a face video generation model by using the training data set to obtain a trained face video generation model; and S4, inputting the test data set into the trained face video generation model, and outputting the generated face video combining the audio and the video. According to the method, the cross reference module and the cross-modal attention module are introduced into the face video generation model, so that the fine-grained details of motion are considered while the accuracy of the face structure of the face video is effectively improved.