227 results about "View synthesis" patented technology

A system and method for generating multimedia artifacts with managed scene continuity in visual and multimedia using an AI-based and scene continuity aware media generation platform. The system receives a user or AI agent specification or simulation result(s), selects or trains generative models based on the specification, preprocesses relevant data, and generates scene narrative or frame-specific, sequence specific or broader continuity aware content using the selected or trained model(s). The generated content may be further enhanced using frame interpolation and view synthesis techniques to create smooth transitions or novel viewpoints or to aid in more efficient transmission or viewing or persistence of resultant content. The system enables efficient and customizable generation of high-quality scene continuity aware content for various applications in visual and multimedia production using neuro-symbolic and simulation enhanced compression, representation and generation processes.

Systems and methods for generating and providing a virtual walkthrough interface can include generating a virtual walkthrough video based on view synthesis renderings generated by neural radiance field model. The neural radiance field model can be trained based on a plurality of images of an environment and may generate the view synthesis renderings based on processing positions along a determined walkthrough path. The generated virtual walkthrough video can then be scrubbed through to provide the virtual walkthrough interface.

Systems and methods for training a neural radiance field model for noisy scenes can leverage raw noisy images in linear high dynamic range color space to train a neural radiance field model to generate view synthesis of low light and / or high contrast scenes. The trained model can then be utilized to accurately complete view rendering tasks without the preprocessing used for generating low dynamic range images. In some implementations, training on unprocessed data of a low light scene can allow for training a neural radiance field model to generate high quality view renderings of a low light scene.

The invention discloses a three-dimensional scene reconstruction method, electronic equipment, a storage medium and a program product, and relates to the technical field of data processing, and the method comprises the steps: inputting each frame of multi-view image into a depth model, and obtaining a prediction depth map of each frame of multi-view image; for any target frame image in each frame of multi-view image, optimizing a local 3D Gaussian model and camera attitude information of the target frame image based on the predicted depth map of the target frame image and the depth information of the previous frame image to obtain an updated local 3D Gaussian model and updated camera attitude information of the target frame image; and optimizing the global 3D Gaussian model to obtain an updated global 3D Gaussian model based on the updated local 3D Gaussian model and the updated camera attitude information of each frame of multi-view image, and performing three-dimensional scene reconstruction based on the updated global 3D Gaussian model. According to the invention, the accuracy of a view synthesis result based on three-dimensional scene reconstruction is improved.

Apparatuses, systems, and techniques are presented to generate images. In at least one embodiment, ray tracing is caused to be selectively performed on one or more objects within a three-dimensional (3D) environment.

The invention discloses a real-time dynamic three-dimensional reconstruction method and system based on patch matching and lightweight monocular depth estimation fusion, and the method comprises the steps: carrying out the PatchMatch matching of an input multi-view image pair, obtaining sparse depth information, and fusing the sparse depth information with a monocular depth estimation result; and the fused depth map is used for constructing a three-dimensional Gaussian point cloud, and high-quality three-dimensional scene reconstruction and new view rendering are realized. In the whole process, an end-to-end differentiable training framework is adopted, and the reconstruction precision and rendering efficiency of the 3D Gaussian point cloud in a dynamic scene are improved by optimizing a pixel-level Gaussian parameter regression network and a lightweight depth estimation model. The method can complete training and depth inference in a short time, is suitable for real-time or near-real-time application scenes, can be widely applied to the fields of virtual reality, augmented reality, film and television production and the like, and is particularly suitable for efficient three-dimensional reconstruction and new view angle synthesis of a target in a multi-view-angle dynamic scene.

The invention discloses a non-static scene reconstruction method and system based on multi-modal occlusion perception scoring. The method comprises the following steps: generating a geometric consistency distribution diagram, static feature points and geometric prior masks through three-dimensional reconstruction of a multi-view image; fusing the geometric prior mask and a semantic segmentation model to extract a semantic mask and a fused semantic feature map; guiding the image segmentation model to generate candidate masks based on static feature point positive point prompt and occlusion area negative frame prompt, and optimizing the consistency by using a grid complementary fusion method; constructing a multi-modal shielding scoring module, and fusing the multi-source features to output a binary static mask; and utilizing a static mask to constrain neural radiation field training, inhibiting dynamic interference and optimizing static scene reconstruction. According to the method, the static region is sensed cooperatively through multi-modal information, the robustness and accuracy of mask generation are improved, the interference of dynamic elements on neural radiation field modeling is effectively inhibited, and high-quality three-dimensional image reconstruction and new view synthesis of a non-static scene are realized.

Provided is a general approach to inpainting 3D content by using a 2D inpainting diffusion model trained on static scenes as a generative prior. In particular, while existing methods for 3D inpainting focus on the specific task of object removal, systems and methods of the present disclosure aim to generate realistic content in any masked 3D region while preserving the complexity of the original scene at the object and scene scale. The proposed techniques can be used in various applications such as 3D object removal, 3D inpainting and novel view synthesis.

Provided are systems and methods for synthesizing novel views of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use machine-learned models that are capable of learning from unstructured and / or unconstrained collections of imagery such as, for example, “in the wild” photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model such as one or more multilayer perceptrons (MLPs).

The invention provides a robot indoor navigation method and system, and the method comprises the steps: obtaining an RGB image frame sequence collected by a camera, and estimating a camera pose and a sparse three-dimensional landmark point of an image frame through a depth block visual odometer; based on the camera pose and the sparse three-dimensional landmark point, dense geometric enhancement is carried out on the image frame through a deep neural network of a transformer architecture to obtain dense geometric prior information; training a neural radiation field according to the initial camera pose, the dense geometric prior information and the image frame data, and performing joint optimization on neural radiation field parameters and the camera pose based on luminosity consistency, geometric consistency and camera pose constraints; based on image frames collected in real time, novel view synthesis is carried out through the optimized nerve radiation field, and robot navigation is carried out based on a novel view. According to the scheme, accurate estimation of the camera pose and high-quality reconstruction of the three-dimensional scene can be realized, the computing resource overhead is low, and the real-time performance can be guaranteed.

The invention provides a building group multi-scale high-efficiency high-precision digital twinning multi-resolution neural radiation field method. The method comprises the following steps of: 1, acquiring a multi-view apparent image of a building group and generating a multi-resolution data set; 2, establishing a multi-resolution neural network model for building group scene rendering color and volume density estimation; 3, training a building group scene rendering and three-dimensional reconstruction multi-resolution neural network model by adopting a super-resolution enhancement body rendering technology; and 4, performing multi-scale building group scene rendering and new view synthesis by adopting the trained multi-resolution neural network model. According to the method, any-view-angle image rendering, high-efficiency and high-precision three-dimensional reconstruction and digital twinning of a multi-scale large-scale building group scene are realized.

View synthesis is a computer graphics process that generates a new image of a scene from a novel (previously unseen) viewpoint of the scene. Typically, the graphics process relies on a machine learning model that has been trained with ground truth pose information. Since ground truth pose information is not readily available, some solutions rely on a Structure-from-Motion (SfM) library COLMAP to generate pose information for a given image. However, this pre-processing step is not only time-consuming but also can fail due to its sensitivity to feature extraction errors and difficulties in handling texture-less or repetitive regions. The present disclosure provides view synthesis from learned camera poses without relying on SfM pre-processing.

A scene reconstruction system renders images of a scene with high-quality geometry and appearance and supports view synthesis, relighting, and scene editing. Given a set of input images of a scene, the scene reconstruction system trains a network to learn a volume representation of the scene that includes separate geometry and reflectance parameters. Using the volume representation, the scene reconstruction system can render images of the scene under arbitrary viewing (view synthesis) and lighting (relighting) locations. Additionally, the scene reconstruction system can render images that change the reflectance of objects in the scene (scene editing).

The invention discloses a 3DGS-based integrated visual angle synthesis and sparse visual angle CT reconstruction method, which comprises the following steps: establishing an internal and external parameter matrix through X-ray scanning parameters, constructing an initial voxel space and an initialized 3DGS radiation field by combining sparse visual angle projection data and based on an ACUI strategy, modeling anisotropic radiation intensity by utilizing a radiation intensity response function, and reconstructing a 3D visual angle CT model. And generating a virtual projection by adopting volume consistency rendering. And the optimized radiation field is converted into a voxel grid, density contribution is applied to carry out voxel space reconstruction, a reconstructed voxel space is obtained, and a new 3DGS radiation field is generated. And re-rendering the virtual visual angle projection based on the new 3DGS radiation field, performing iterative optimization by using a loss function, and outputting a new visual angle synthetic image and a CT reconstruction body until a preset number of iterations is reached. And an efficient process of one-time training and dual output is realized. The efficiency and precision of X-ray imaging are improved, the radiation dose of a patient is reduced, and meanwhile, a higher-quality imaging solution is provided for medical diagnosis.

The invention discloses a layered densification Gaussian sputtering method based on visibility, and relates to the technical field of artificial intelligence and computer vision. According to the scheme, an initial three-dimensional Gaussian primitive set is generated based on sparse multi-view observation data, and initial scene representation is established by extracting spatial distribution parameters, morphological parameters and radiation parameters; performing fusion analysis on the Gaussian primitives based on the multi-dimensional observability parameter set to obtain comprehensive observability index data; executing hierarchical clustering according to the index data, and constructing a multi-layer Gaussian structure of a significant layer, a transition layer and a background layer; performing density enhancement, geometric continuity constraint and parameter update processing on different levels of Gaussian structures, and generating a rendered image of a target view angle under a volume light traveling and transparency hybrid mechanism; according to the method, continuous reconstruction of a scene structure and accurate expression of radiation characteristics can be realized under the sparse view condition, and the geometric fidelity and rendering consistency of new view angle synthesis are improved.

The invention discloses a sparse dynamic 3D Gaussian splash method based on global-local feature extraction. According to the method, a global-local feature extraction module, an inter-frame feature flow aggregation network and depth prior are combined with a 4D pseudo pose supervision mechanism for cooperative work, and high-quality dynamic scene reconstruction and rendering are realized under the condition of a sparse camera visual angle. According to the method, the multi-view image information is compressed to the feature space for processing, so that the calculation complexity is reduced, and the geometric structure and the dynamic features of the scene are effectively reserved; the fused depth prior information and the 4D pseudo pose provide accurate geometric constraints, and the inter-frame feature flow aggregation network optimizes the time consistency of the dynamic scene. The method is particularly suitable for processing a complex dynamic scene under a sparse view angle, the sense of reality and the real-time performance of virtual view synthesis are remarkably improved, and a dynamic view with a more accurate geometric structure, higher time continuity and richer details can be generated.

According to examples, a head-mounted display (HMD) headset may include at least two exterior-facing color cameras mounted on the front face of the HMD and in the same visual plane as the user's eyes. The at least two exterior-facing color cameras may collect images for stereo view synthesis. The HMD may include a processor, and a memory storing instructions, which when executed by the processor, cause the processor to perform stereo view synthesis with a machine-learning (ML) based technique including at least one of depth estimation, imaging sharpening, forward splatting, disocclusion filtering, or fusion.

The invention discloses a target-level three-dimensional point cloud cross-modal semantic retrieval method and device and electronic equipment. The target-level three-dimensional point cloud cross-modal semantic retrieval method comprises the steps that independent three-dimensional target point clouds are segmented from three-dimensional scene point clouds, and unique identifiers are given to the independent three-dimensional target point clouds; projecting each target point cloud to three orthogonal two-dimensional observation planes to generate a multi-view composite image; a pre-trained vision-language basic model is adopted to code and fuse the synthesized image, and a unified multi-modal feature vector is generated; a vector database in which the feature vectors are associated with their identifiers is constructed. And encoding a natural language query text into a text query vector by using the model, calculating the semantic similarity between the text query vector and the feature vector in the database, and positioning and returning the corresponding three-dimensional target point cloud. According to the method, accurate and efficient cross-modal retrieval from a natural language to a three-dimensional point cloud target is realized, the problem that a traditional method is difficult to support semantic fine-grained retrieval of the three-dimensional target point cloud is solved, and a key technical support is provided for training data management in the fields of intelligence and the like.

The invention discloses a camera-free pose three-dimensional scene reconstruction method based on 3DGS, and the method comprises the steps: firstly obtaining an ordered picture sequence shot by a camera and a depth map of the ordered picture sequence, constructing a picture, and pre-training a feature encoder for a data set; then initializing a camera pose of a first frame of picture in the ordered picture sequence and an empty training picture set, recovering point cloud information and initializing a 3D Gaussian model; randomly selecting a color image and a depth image of the image from the training image set as supervision information to iteratively optimize the 3D Gaussian model; iteratively optimizing the camera pose of the next frame of picture in the ordered picture sequence according to the optimized 3D Gaussian model; and finally, according to a picture sequence in the ordered picture sequence, alternately iteratively optimizing the 3D Gaussian model and optimizing the camera pose of a new frame of picture by adopting a progressive training mode to realize three-dimensional scene reconstruction. According to the method, three-dimensional reconstruction with higher quality can be realized, the quality of a picture synthesized by a new view angle is improved, and the error of camera pose estimation is reduced.

The invention discloses a three-dimensional Gaussian splashing and rasterization pipeline fused view synthesis system and method. The method comprises the following steps of: 1, acquiring two-dimensional Gaussian primitives; 2, converting the two-dimensional Gaussian primitive into two triangular grids, and converting the two triangular grids into a world coordinate system; 3, baking the spherical harmonic coefficient of the color and the opacity data in a texture form; 4, enabling each Gaussian primitive to correspond to a texture block on the texture image set, enabling a triangle to cover the texture block in a UV space, and obtaining a Gaussian weight; 5, based on the obtained Gaussian weight, simulating an alpha mixing behavior in the three-dimensional Gaussian splashing pipeline by using a time-based jitter anti-aliasing method; and 6, training a symbol vector field SDF branch while training, and mutually supervising and constraining the output of the symbol vector field and the output of the two-dimensional Gaussian. The three-dimensional Gaussian splash can be rendered in a traditional rasterization pipeline of the unreal engine 4, and the fidelity equivalent to that of a special training pipeline is kept.

Methods and systems for novel view synthesis are provided. An example method involves accessing source images of a scene, encoding each source image into a series of multiscale feature maps, defining a target view for the scene, and decoding the target view into a target image of the scene, wherein the decoding involves applying global attention across the high level features of the multiscale feature maps of the source images, and applying local attention across a limited set of the lower level features of the multiscale feature maps of the source images.

The invention discloses a small sample new view synthesis method based on reinitialization three-dimensional Gaussian splashing. The method comprises the following steps: firstly, acquiring sparse three-dimensional point cloud and camera internal and external parameters from a training view through a motion recovery structure algorithm; then, sampling points are generated in the point cloud bounding box by adopting a spatial expansion hybrid sampling strategy, and a coarse-grained Gaussian set is constructed and optimized; thirdly, obtaining a rendered image through Gaussian initialization and splash rendering, calculating pixel importance through depth errors and transmissivity, generating a fine-grained Gaussian set through back projection, and optimizing the fine-grained Gaussian set; and finally, calculating a sampling probability based on the cross-view contribution degree, screening key Gaussian distribution, and optimizing to form a final Gaussian set, thereby realizing high-quality new view synthesis. According to the method, the problem of sparsity difference is solved by eliminating extended view dependence, the multi-view consistency and local geometric details of a new view scene are improved, and high-quality new view synthesis of sparse data is realized.

Provided in the embodiments of the present application are a 3D video generation method and an electronic device. The method is applied to the electronic device. The method comprises: on the basis of captured data, acquiring 3D video material data, wherein the captured data comprises a 2D video and / or a 2D image; on the basis of the 3D video material data, analyzing a 3D video scene type; on the basis of the 3D video scene type, acquiring a new-view generation model corresponding to the 3D video scene type; and using the new-view generation model corresponding to the 3D video scene type to generate 3D video data on the basis of the 3D video material data. On the basis of the method of the embodiments of the present application, at a 3D video data generation stage, a corresponding new-view generation model is called on the basis of a 3D video scene type, and thus a more complete scene synthesis view can be acquired on the basis of a new-view synthesis algorithm that uses deep learning and artificial intelligence, such that when 3D video data is played, a user can immersively view 3D video content from different angles, thereby providing a stronger sense of 3D and richer 3D video content.

The embodiment of the invention provides a method and device for generating a space visual interaction medium based on space calculation, and relates to the field of space intelligence and space audiovisual, and the method comprises the steps: carrying out the preprocessing of an original image, and obtaining first information; performing focal plane simulation processing on the first information to obtain second information; extracting complex data of the first information, and performing wavelet transform on the complex data to obtain a feature set; performing depth estimation and view synthesis on the second information to obtain third information; performing spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; and determining a spatial video according to the first information, the feature set and the new viewpoint depth map. According to the method, the two-dimensional information is converted into the three-dimensional interaction space video, and the interaction experience of the user is improved.

The invention is suitable for the technical field of view synthesis, and provides a novel three-dimensional scene view synthesis method based on matched light rays, and the method comprises the steps: generating a plurality of 3D Gaussian ellipsoids according to a plurality of view angle images of a target scene; randomly screening out a plurality of target view angle images from the plurality of view angle images; screening out a plurality of target 3D Gaussian ellipsoids from the plurality of 3D Gaussian ellipsoids according to the plurality of target view angle images; based on the matching light, screening out a plurality of Gaussian matching pairs from the plurality of target 3D Gaussian ellipsoids; optimizing the target 3D Gaussian ellipsoids in all Gaussian matching pairs; and fusing the plurality of optimized target 3D Gaussian ellipsoids to obtain a novel view of the target scene. According to the invention, the rendering effect of the 3DGS under the sparse view angle can be improved.

Systems and methods for generating and providing a virtual walkthrough interface can include generating a virtual walkthrough video based on view synthesis renderings generated by neural radiance field model. The neural radiance field model can be trained based on a plurality of images of an environment and may generate the view synthesis renderings based on processing positions along a determined walkthrough path. The generated virtual walkthrough video can then be scrubbed through to provide the virtual walkthrough interface.

A computer-implemented methods and systems for synthesizing an image includes capturing (802) data from a scene and fusing (804) grid-based representations of the scene from a plurality of different encodings to inherit beneficial properties of the plurality of different encodings, the plurality of different encodings including a Lidar encoding and a high definition map encoding. Rays are rendered (812) from fused grid-based representations. A density is determined (814) for points in the rays, and a color is determined (816) for the points in the rays. The rays are volume rendered (814) with the density and color, and an image is synthesized (820)from volume rendered rays with the density and the color.

A method for controlling autonomous driving of a vehicle is introduced. The method may comprise, training, based on an inference depth and an inference pose, a synthetic image model for generating a synthetic image, generating, based on the synthetic image, a first virtual image to be associated with the original image, generating, based on the original image, a second virtual image, training a generative adversarial network (GAN) for determining, based on the original image, authenticity of the first virtual image and the second virtual image, training, based on the trained GAN, a depth network, wherein the trained GAN outputs a determination of the authenticity of the first virtual image, outputting, based on the trained depth network, signal, and controlling, based on the signal, autonomous driving of the vehicle.

To provide a system and a method for synthesizing novel views of complex scenes (e.g., outdoor scenes).SOLUTION: In some implementations, a system and a method can include or use a machine-learned model capable of learning from unstructured and / or unconstrained collections of imagery such as, for example, "in the wild" photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model, such as one or more multilayer perceptrons (MLPs).SELECTED DRAWING: Figure 1