The invention relates to the technical field of video generation, in particular to a space-time attention and
optical flow constrained video generation method and
system, and the
system comprises a CLIP text
encoder, a multi-frame potential
noise initialization module, space-time attention U-Net, an
optical flow prediction module, a dynamic path optimizer, a space-time
noise scheduling module and a video frame decoder. By embedding physical constraints into a
generation process, four-dimensional joint optimization of space, time,
noise and
semantics is realized, space-time attention U-Net is combined with 3D
convolution and cross-frame attention modeling space-time dependence, and an
optical flow prediction module optimizes a motion trail through residual compensation. The problems of inter-frame
jitter and long-range motion breakage are effectively solved, the PSNR fluctuation is smaller than or equal to 0.5 dB, the Warping precision reaches 0.85, the
video memory is reduced by 25%, the reasoning speed is increased by three times, 1080P and 30fps real-time generation is supported, and the
video quality and the generation efficiency are improved.