Static three-dimensional light-field content generation method and system

Abstract

Disclosed in the present invention are a static three-dimensional light-field content generation method and system. The method comprises the following steps: S1, collecting a video of static scene content, and optimally extracting a discrete disparity image sequence from the video by means of an automatic frame extraction algorithm; S2, calculating relative pose parameters of discrete disparity images; S3, on the basis of the relative pose parameters and sub-pixel arrangement rules of a light-field display, calculating a synthetic encoding parameter mapping function; and S4, on the basis of the synthetic encoding parameter mapping function, using CUDA and OpenGL rendering engines to synthesize and encode the discrete disparity image sequence, and rendering same to the light-field display for display. The present invention can quickly collect and generate high-quality static light-field content in a real scene, and can quickly adjust an actual light-field display effect on the basis of actual requirements.

Description

A method and system for generating static three-dimensional light field content Technical Field

[0001] This invention belongs to the field of three-dimensional light field display technology, specifically relating to a method and system for generating static three-dimensional light field content. Background Technology

[0002] 3D light field display technology generates stereoscopic vision through physiological or psychological factors. Common 3D display methods, such as raster 3D display, integrated imaging 3D display, and volumetric 3D display, utilize physiological and psychological factors such as binocular parallax to display parallax images on a 2D screen, allowing viewers to perceive depth information of objects on the screen and creating a sense of depth. However, acquiring high-quality light field content has always been a challenge in the industry. Common acquisition methods use sparse camera arrays to capture data, reconstruct 3D scenes through 3D reconstruction, render dense viewpoint images, and finally synthesize light field images. The 3D reconstruction methods employed here struggle to achieve high-quality reconstruction results, and the complex rendering algorithms require substantial computational resources.

[0003] Therefore, developing a simple and effective scheme for light field acquisition and high-quality light field content generation is of great significance. Summary of the Invention

[0004] To address the aforementioned shortcomings in the existing technology, the present invention provides a static three-dimensional light field content generation method and system that solves the problem of complex rendering algorithms when acquiring high-quality light field content in the existing technology.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a method for generating static three-dimensional light field content, comprising the following steps:

[0006] S1. Acquire video of static scene content, and extract discrete parallax image sequences from the video using an automatic frame-taking algorithm;

[0007] S2. Calculate the relative pose parameters of the discrete disparity image;

[0008] S3. Calculate the synthetic coding parameter mapping function based on the relative pose parameters and the sub-pixel arrangement rules of the light field display;

[0009] S4. Based on the synthesis coding parameter mapping function, the discrete parallax image sequence is synthesized and encoded using CUDA and OpenGL rendering engines, and then rendered to a light field display for display.

[0010] Further: In S1, the discrete parallax image sequence includes a still scene image sequence and a non-still scene image sequence;

[0011] The specific method for extracting image sequences of non-still objects is as follows:

[0012] Extract key information of human movements from the video, and delete frames with movement amplitude greater than the amplitude threshold using a motion detection algorithm to obtain a sequence of non-still scene images.

[0013] The specific method for extracting still life scene image sequences is as follows:

[0014] Extract any stable feature points in the scene, fit the relative position relationship curve of each frame image based on the pixel coordinates of the extracted feature points, and generate a still life scene image sequence through a frame-taking algorithm.

[0015] Furthermore: S2 specifically refers to:

[0016] The relative pose parameters of the camera corresponding to each image in the discrete parallax image sequence are calculated using a motion structure recovery algorithm, including the relative rotation matrix and the relative translation matrix.

[0017] Further: In step S3, the expression for the synthetic coding parameter mapping function F is specifically as follows:

[0018] In the formula, θ is the tilt angle of the display, and L x Where N is the number of lines on the monitor, h is the number of disparity maps, and h is the vertical resolution of the monitor. 11 h is the first parameter of the transformation matrix H(z). 12 h is the second parameter of the transformation matrix H(z). 13 h is the third parameter of the transformation matrix H(z). 21 h is the fourth parameter of the transformation matrix H(z). 22 h is the fifth parameter of the transformation matrix H(z). 23 h is the sixth parameter of the transformation matrix H(z). 31 h is the seventh parameter of the transformation matrix H(z). 32 h is the eighth parameter of the transformation matrix H(z). 33 is the ninth parameter of the transformation matrix H(z), x is the pixel x-coordinate of the corresponding target viewpoint, y is the pixel y-coordinate of the corresponding target viewpoint, f is the pixel correspondence between the synthetic coded image and the discrete disparity image sequence, z is the transformation matrix and the zero plane depth, % is the modulo operation, and k is the sub-pixel index.

[0019] Furthermore: S4 specifically refers to:

[0020] Based on the synthesis coding parameter mapping function, discrete parallax image sequences are synthesized and encoded using CUDA and OpenGL rendering engines and then rendered to a light field display for display.

[0021] A static 3D light field content generation system, the system comprising:

[0022] A video capture device placed on a guide rail is used to capture video of static scene content;

[0023] The motion structure recovery algorithm module is used to calculate the relative pose parameters of discrete parallax images, including the relative rotation matrix and the relative translation matrix;

[0024] The synthesis coding parameter mapping function calculation module is used to calculate the synthesis coding parameter mapping function based on the number of lines, tilt angle and resolution of the light field display.

[0025] The compositing encoding module is used to rasterize and render the composited image onto a light field display based on the CUDA and OpenGL rendering engines.

[0026] The beneficial effects of this invention are as follows: This invention provides a method and system for generating static three-dimensional light field content. It uses a video acquisition device placed on a horizontal guide rail to capture video of static scene content. An automatic frame-taking algorithm is used to selectively extract discrete parallax image sequences from the video. The relative pose parameters of the discrete parallax images are calculated. Based on the relative pose parameters, a zero-plane adjustable transformation matrix is ​​calculated. According to the sub-pixel arrangement rules of the light field display and the aforementioned transformation matrix, a synthesis encoding parameter mapping function is calculated. Using CUDA and OpenGL rendering engines, the parallax image sequences are synthesized, encoded, and rendered to the light field display in real time. The in-screen and out-of-screen depths are adjusted in real time based on the display effect. This invention can quickly acquire and generate high-quality static light field content in real-world scenes and can quickly adjust the actual light field display effect according to actual needs, solving the problem of complex rendering algorithms in existing technologies for acquiring high-quality light field content. Attached Figure Description

[0027] Figure 1 is a flowchart of a static three-dimensional light field content generation method according to the present invention.

[0028] Figure 2 is a schematic diagram of the structure of a static three-dimensional light field content generation system according to the present invention. Detailed Implementation

[0029] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0030] As shown in Figure 1, in one embodiment of the present invention, a method for generating static three-dimensional light field content includes the following steps:

[0031] S1. Acquire video of static scene content, and extract discrete parallax image sequences from the video using an automatic frame-taking algorithm;

[0032] S2. Calculate the relative pose parameters of the discrete disparity image;

[0033] S3. Calculate the synthetic coding parameter mapping function based on the relative pose parameters and the sub-pixel arrangement rules of the light field display;

[0034] S4. Based on the synthesis coding parameter mapping function, the discrete parallax image sequence is synthesized and encoded using CUDA and OpenGL rendering engines, and then rendered to a light field display for display.

[0035] In this embodiment, the present invention can quickly acquire real scene content and directly generate three-dimensional light field content, and the actual stereoscopic display effect can be adjusted by adjusting the position of the zero plane.

[0036] In S1, the discrete parallax image sequence includes a still scene image sequence and a non-still scene image sequence;

[0037] The specific method for extracting image sequences of non-still objects is as follows:

[0038] Extract key information of human movements from the video, and delete frames with movement amplitude greater than the amplitude threshold using a motion detection algorithm to obtain a sequence of non-still scene images.

[0039] The specific method for extracting still life scene image sequences is as follows:

[0040] Extract any stable feature points in the scene, fit the relative position relationship curve of each frame image based on the pixel coordinates of the extracted feature points, and generate a still life scene image sequence through a frame-taking algorithm.

[0041] In this embodiment, while ensuring coverage of all viewpoints as much as possible, a uniform sampling algorithm is preferentially used to select frame images from the video of static scene content as the original discrete parallax image sequence.

[0042] Specifically, S2 is:

[0043] The relative pose parameters of the camera corresponding to each image in the discrete parallax image sequence are calculated using a motion structure recovery algorithm, including the relative rotation matrix and the relative translation matrix.

[0044] In this embodiment, the motion structure recovery algorithm calculates the relative pose parameters of each image using matching points between multiple images, specifically:

[0045] S21. Use feature detection algorithms to extract key points from each image. Feature detection algorithms include SIFT (Scale-Invariant Feature Transform) or ORB (Oriented Fast and Rotated BRIEF).

[0046] S22. Find the corresponding feature point pairs between these images using feature matching algorithms, including BFMatcher (Brute-Force Matcher) or FLANN (Fast Library for Approximate Nearest Neighbors);

[0047] S23. Using the matched feature points, estimate the relative pose parameters of the camera corresponding to each disparity image by calculating the essential matrix or fundamental matrix, including the relative rotation matrix and the relative translation matrix.

[0048] In step S3, the expression for the synthetic coding parameter mapping function F is specifically as follows:

[0049] In the formula, θ is the tilt angle of the display, and L x Where N is the number of lines on the monitor, h is the number of disparity maps, and h is the vertical resolution of the monitor. 11 h is the first parameter of the transformation matrix H(z). 12 h is the second parameter of the transformation matrix H(z). 13 h is the third parameter of the transformation matrix H(z). 21 h is the fourth parameter of the transformation matrix H(z). 22 h is the fifth parameter of the transformation matrix H(z). 23 h is the sixth parameter of the transformation matrix H(z). 31 h is the seventh parameter of the transformation matrix H(z). 32 h is the eighth parameter of the transformation matrix H(z). 33 is the ninth parameter of the transformation matrix H(z), x is the pixel x-coordinate of the corresponding target viewpoint, y is the pixel y-coordinate of the corresponding target viewpoint, f is the pixel correspondence between the synthetic coded image and the discrete disparity image sequence, z is the transformation matrix and the zero plane depth, % is the modulo operation, and k is the sub-pixel index.

[0050] By adjusting the value of z, since the projection coordinates on each frame of the image are related to the zero plane depth, the final transformation matrix obtained by adjusting the value of z is related to the zero plane depth z. By adjusting the value of z, different transformation matrices can be obtained, thereby realizing the adjustment of the synthetic coding mapping function, and thus adjusting the in-screen and out-of-screen depths of the actual three-dimensional light field display.

[0051] The principle of calculating the synthetic coding parameter mapping function is as follows:

[0052] First, based on the monitor's tilt angle and line count measured using the monitor's light control element parameters and actual assembly parameters, the mapping relationship f between the synthetic coded image (x, y, k) and the discrete parallax image sequence is calculated. Its specific expression is: V index (x,y,k)=F(x,y,k)

[0053] In the formula, V index (·) is the disparity map index, and F(·) is the synthetic coding mapping function. Finally, a synthetic coding mapping function can be obtained in which the depth of the three-dimensional light field display content entering and leaving the screen can be freely adjusted.

[0054] A transformation matrix H(z) related to the position of the zero plane is calculated based on the relative pose parameters. The zero plane is the confocal surface between dense viewpoints in the 3D display and also the critical surface between the screen and the input screen in the 3D display. After the discrete disparity image sequence is transformed by this transformation matrix, an approximately off-axis disparity sequence image can be obtained. Based on the light control element parameters and the actual assembly parameters, the sub-pixel arrangement rules of the light field display are determined, the pixel correspondence f between the synthetic coding map and the discrete disparity map sequence is calculated, and the synthetic coding parameter mapping function is obtained.

[0055] The principle for calculating the transformation matrix is ​​as follows:

[0056] An initial null plane is set up with a size of H×W and a distance of Z from the acquisition device. Using the camera of the intermediate frame as the origin of the coordinate system, the coordinates of the four vertices of the null plane are P1(-W / 2,H / 2,Z), P2(W / 2,H / 2,Z), P3(-W / 2,-H / 2,Z), and P4(W / 2,-H / 2,Z). Based on the pose parameters of each frame calculated by the SFM algorithm, the four vertices of the null plane can be projected onto each frame to obtain the corresponding pixel coordinates p. i,j (u,v), and calculate the transformation matrix as the expression for the transformation matrix H(z):

[0057] By using the transformation matrix H(z), the ordinary acquisition method described above can be transformed into an approximately tangential view volume setting, achieving both out-of-screen and in-screen effects without the distortion caused by convergent settings. Furthermore, by adjusting the position of the initial zero plane and the transformation matrix, the zero plane of the light field display content can be adjusted to present different stereoscopic display effects.

[0058] Specifically, S4 is:

[0059] Based on the synthesis coding parameter mapping function, discrete parallax image sequences are synthesized and encoded using CUDA and OpenGL rendering engines and then rendered to a light field display for display.

[0060] In this embodiment, the present invention utilizes CUDA and OpenGL rendering engines to synthesize and encode the aforementioned parallax image sequence in real time and render it to a light field display for display. The in-screen and out-of-screen depths are adjusted in real time based on the display effect. During the synthesis and encoding process, the calculation of sub-pixel off-axis transformation and light field encoding are performed simultaneously. That is, a specific sub-pixel in the required discrete parallax image is located according to the mapping relationship, and the off-axis transformation is applied only to that sub-pixel, which greatly reduces computational redundancy. Furthermore, the entire process is processed at high speed in parallel within the GPU. The synthesized and encoded image is displayed using OpenGL real-time rasterization rendering, and through interactive design, the zero-plane depth is changed in real time to display different stereoscopic effects.

[0061] As shown in Figure 2, a static three-dimensional light field content generation system includes:

[0062] A video capture device placed on a guide rail is used to capture video of static scene content;

[0063] The guide rail can be either a straight or curved type, and can carry the acquisition device to acquire scene video at an approximately uniform speed and stably according to instructions. The acquisition range needs to meet the display viewing angle range of the monitor, and the position of the acquisition device is approximately at the optimal viewing position of the monitor from the scene.

[0064] The motion structure recovery algorithm module is used to calculate the relative pose parameters of discrete parallax images, including the relative rotation matrix and the relative translation matrix;

[0065] The synthesis coding parameter mapping function calculation module is used to calculate the synthesis coding parameter mapping function based on the number of lines, tilt angle and resolution of the light field display.

[0066] The compositing encoding module is used to rasterize and render the composited image onto a light field display based on the CUDA and OpenGL rendering engines.

[0067] In this embodiment, the video acquisition device is placed at a distance from the scene that is approximately the optimal viewing position of the monitor. The acquisition range can meet the display viewing angle range of the monitor. The slider on the guide rail can carry the acquisition device to acquire scene video at an approximately uniform speed and stably according to the instructions.

[0068] The beneficial effects of this invention are as follows: This invention provides a method and system for generating static three-dimensional light field content. It uses a video acquisition device placed on a horizontal guide rail to capture video of static scene content. An automatic frame-taking algorithm is used to selectively extract discrete parallax image sequences from the video. The relative pose parameters of the discrete parallax images are calculated. Based on the relative pose parameters, a zero-plane adjustable transformation matrix is ​​calculated. According to the sub-pixel arrangement rules of the light field display and the aforementioned transformation matrix, a synthesis encoding parameter mapping function is calculated. Using CUDA and OpenGL rendering engines, the parallax image sequences are synthesized, encoded, and rendered to the light field display in real time. The in-screen and out-of-screen depths are adjusted in real time based on the display effect. This invention can quickly acquire and generate high-quality static light field content in real-world scenes and can quickly adjust the actual light field display effect according to actual needs, solving the problem of complex rendering algorithms in existing technologies for acquiring high-quality light field content.

[0069] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.

Claims

1. A method for generating static three-dimensional light field content, characterized in that, Includes the following steps: S1. Acquire video of static scene content, and extract discrete parallax image sequences from the video using an automatic frame-taking algorithm; S2. Calculate the relative pose parameters of the discrete disparity image; S3. Calculate the synthetic coding parameter mapping function based on the relative pose parameters and the sub-pixel arrangement rules of the light field display; S4. Based on the synthesis coding parameter mapping function, the discrete parallax image sequence is synthesized and encoded using CUDA and OpenGL rendering engines, and then rendered to a light field display for display.

2. The static three-dimensional light field content generation method according to claim 1, characterized in that, In S1, the discrete parallax image sequence includes a still scene image sequence and a non-still scene image sequence; The specific method for extracting image sequences of non-still objects is as follows: Extract key information of human movements from the video, and delete frames with movement amplitude greater than the amplitude threshold using a motion detection algorithm to obtain a sequence of non-still scene images. The specific method for extracting still life scene image sequences is as follows: Extract any stable feature points in the scene, fit the relative position relationship curve of each frame image based on the pixel coordinates of the extracted feature points, and generate a still life scene image sequence through a frame-taking algorithm.

3. The static three-dimensional light field content generation method according to claim 2, characterized in that, Specifically, S2 is: The relative pose parameters of the camera corresponding to each image in the discrete parallax image sequence are calculated using a motion structure recovery algorithm, including the relative rotation matrix and the relative translation matrix.

4. The static three-dimensional light field content generation method according to claim 1, characterized in that, In step S3, the expression for the synthetic coding parameter mapping function F is specifically as follows: In the formula, θ is the tilt angle of the display, and L x Where N is the number of lines on the monitor, h is the number of disparity maps, and h is the vertical resolution of the monitor. 11 h is the first parameter of the transformation matrix H(z). 12 h is the second parameter of the transformation matrix H(z). 13 h is the third parameter of the transformation matrix H(z). 21 h is the fourth parameter of the transformation matrix H(z). 22 h is the fifth parameter of the transformation matrix H(z). 23 h is the sixth parameter of the transformation matrix H(z). 31 h is the seventh parameter of the transformation matrix H(z). 32 h is the eighth parameter of the transformation matrix H(z). 33 is the ninth parameter of the transformation matrix H(z), x is the pixel x-coordinate of the corresponding target viewpoint, y is the pixel y-coordinate of the corresponding target viewpoint, f is the pixel correspondence between the synthetic coded image and the discrete disparity image sequence, z is the transformation matrix and the zero plane depth, % is the modulo operation, and k is the sub-pixel index.

5. The static three-dimensional light field content generation method according to claim 4, characterized in that, Specifically, S4 is: Based on the synthesis coding parameter mapping function, discrete parallax image sequences are synthesized and encoded using CUDA and OpenGL rendering engines and then rendered to a light field display for display.

6. A static three-dimensional light field content generation system, applied to the static three-dimensional light field content generation method as described in any one of claims 1 to 5, characterized in that, The system includes: A video capture device placed on a guide rail is used to capture video of static scene content; The motion structure recovery algorithm module is used to calculate the relative pose parameters of discrete parallax images, including the relative rotation matrix and the relative translation matrix; The synthesis coding parameter mapping function calculation module is used to calculate the synthesis coding parameter mapping function based on the number of lines, tilt angle and resolution of the light field display. The compositing encoding module is used to rasterize and render the composited image onto a light field display based on the CUDA and OpenGL rendering engines.

Images