50 results about "Inverse rendering" patented technology

Embodiments of the present disclosure relate to rendering and inverse rendering using one or more generative models. “Rendering” refers to the process of generating a final visual image, video frame, or animation from a 2D or 3D model. “Inverse rendering” is a process that involves deducing or estimating the properties (e.g., material maps or other properties such as geometry, lighting, and textures) of a scene from observed images or visual data. Essentially, it aims to reverse the traditional rendering process. Various aspects of the present disclosure introduce editable light and material controls into generative models to allow for artistic creation. Various embodiments integrate generative models as a renderer for classic rendering pipelines to upcycle and enhance the style of rendered content.

The invention discloses a reflecting object inverse rendering method, system and device based on two-dimensional Gaussian sputtering and multi-mode diffusion prior and a medium. The method comprises the steps that images and camera parameters of a reflecting object under multiple visual angles are acquired; calculating an observation angle according to the camera parameters; performing rasterization rendering on the images of the reflecting object under the multiple visual angles by using the trained two-dimensional Gaussian primitives under the observation visual angle to obtain a rendered image under the new visual angle; according to the method, two-dimensional Gaussian is adopted as a scene representation element, surface characteristics are better fitted, and geometric reconstruction precision can be remarkably improved; the trained two-dimensional Gaussian primitive is closer to a real three-dimensional scene, and experimental tests of multiple data sets show that the method not only can accurately reconstruct geometric and material distribution of a high-reflection area, but also can recover clear environment illumination with high-frequency details, realizes efficient and real reflection inverse rendering, and has a good application prospect. The method can be widely applied to downstream tasks such as three-dimensional reconstruction, material editing and relighting.

Methods and apparatus for relighting images. According to an example embodiment, inverse rendering is applied to an input image to extract a plurality of channels including an original lighting channel. A first neural network is used to determine a first latent feature corresponding to the input image based on a first set of channels including a shading channel generated using a replacement lighting channel. A second neural network is used to determine a second latent feature corresponding to the input image based on a different second set of channels including the replacement lighting channel. A relighted image is generated by propagating samples of a latent image map corresponding to the input image through a conditional diffusion model to which the first and second latent features are applied as first and second conditions.

The invention discloses a transparent object inverse rendering method and device based on three-dimensional Gaussian, and the method comprises the steps: firstly collecting a multi-view image in a three-dimensional scene with a transparent object, and initializing a three-dimensional Gaussian primitive; dividing the Gaussian primitive into a transparent object Gaussian primitive and an environment Gaussian primitive, baking based on an optimal projection strategy to obtain an environment Gaussian light field probe, calculating based on a bidirectional scattering distribution function carried by the transparent object Gaussian primitive to obtain a corresponding attribute map, and calculating reflected light and refracted light through an iterative probe query algorithm to obtain a target object. Transparent object reconstruction is realized through loss optimization of parameters of Gaussian primitives of the transparent object; and finally, based on the optimized transparent object Gaussian primitive, extracting a polygon mesh, recombining the polygon mesh with the environment Gaussian primitive to be recombined to obtain a recombined scene, and re-rendering the recombined scene. According to the method, performance advantages are kept, and meanwhile, the method also has advantages in inverse rendering and new view angle synthesis quality.

The invention provides a semantic segmentation method based on an HDR dynamic neural radiation field, and relates to the technical field of image segmentation. According to the semantic segmentation method based on the HDR dynamic neural radiation field, the segmentation capability of a two-dimensional visual large model on pictures and videos is utilized, the segmentation capability of the two-dimensional model is expanded to four dimensions, a hyper-parameter controlled tone mapping module is constructed, mapping from an HDR view to LDR is controlled, and the segmentation capability in an HDR scene is greatly improved; a mask deformation field network and a dynamic mask inverse rendering method are provided, expensive labeling does not need to be performed on a four-dimensional scene, and any object in the four-dimensional dynamic scene can be segmented by using a two-dimensional picture mask.

A computer-implemented process for creating 3D objects from 2D images includes receiving an input image, conditioning a generative model using image-derived, voxelized three-dimensional features, generating, by a transformer-based rectified-flow generative model parameterized as a base network optionally coupled to one or more low-rank adapter modules activatable at inference, a volumetric latent of the target object, the volumetric latent including a sparse, feature-augmented volumetric lattice obtained by transporting an initial random sample toward a learned manifold via a rectified-flow sampling process, decoding the volumetric latent by mapping the volumetric latent to a feature-bearing sparse volumetric field consistent with the volumetric lattice, decoding the field to a continuous implicit surface function, and extracting a watertight mesh by isosurface extraction, estimating camera-pose parameters by render-and-compare alignment between silhouettes rendered from the mesh and silhouettes of the input image, and, performing style-preserving inverse rendering on the mesh that updates UV-space albedo and material maps.

The invention discloses a self-supervised image relighting method based on physical consistency, and belongs to the technical field of computer vision and image processing. The method comprises the following steps: constructing a pairwise training data set containing image and environment map spherical harmonic vectors; constructing an inverse rendering neural network, and decomposing an input image into albedo and geometric visible item spherical harmonic vectors; based on a physical rendering equation, multiple illumination sample pairs in the same scene are utilized to construct a joint objective function comprising reconstruction loss, relighting loss, physical consistency loss and cyclic consistency loss to train the network. According to the method, constant material features are extracted under different illumination conditions through a physical consistency constraint enforcement network, and the robustness of the model under unknown illumination conditions is enhanced through cyclic consistency. According to the method, training can be achieved without the albedo and the geometric truth value of a real scene, the data acquisition cost is effectively reduced, the problem of generalization of synthetic data to the real scene is solved, and high-quality and high-physical-reality-sense image relighting is achieved.

The invention discloses a power transmission line tower point cloud segmentation method and device based on inverse rendering, terminal equipment and a storage medium, and belongs to the technical field of point cloud segmentation. Inputting the to-be-segmented point cloud into a point cloud prompt box generation model, so that the point cloud prompt box generation model outputs a plurality of three-dimensional prompt boxes identifying each tower on the to-be-segmented point cloud; rendering the to-be-segmented point cloud into a plurality of two-dimensional images under different visual angles, and mapping the three-dimensional prompt box into a two-dimensional prompt box; and according to the two-dimensional image and the two-dimensional prompt box under each view angle, repeatedly carrying out tower point cloud segmentation operation until a point cloud segmentation result of each tower is generated. Therefore, by implementing the method, the defects of low efficiency and low precision of the existing tower segmentation identification method based on the laser point cloud can be overcome.

The invention relates to an image processing method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: inputting a training sample into a prediction model to obtain a prediction inverse rendering result of an image sample included in the training sample; the prediction inverse rendering result is used for indicating the physical attribute of the object in the image sample; according to the predicted inverse rendering result, executing image rendering operation through a ray tracing rendering pipeline to obtain a rendered image; obtaining a first loss function value based on the difference between the image sample and the rendered image; performing parameter updating on the prediction model based on the first loss function value; wherein the trained prediction model is used for outputting an inverse rendering result of the target image included in the input data based on the input data; the inverse rendering result is used for indicating the physical attribute of the object in the target image. According to the invention, the accuracy of the inverse rendering result predicted by the prediction model can be improved.

The embodiment of the application provides a meat product quality evaluation method and system based on digital image processing, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a multi-angle / multi-channel image sequence of a meat product sample to obtain a calibration sequence for reflection component separation; performing joint demixing and inverse rendering correction of specular and diffuse reflection to obtain a net texture diffuse reflection image and a polarization map; extracting fascia rupture features based on the net texture diffuse reflection image and the polarization map, and correcting texture consistency based on the fascia rupture features to determine a tissue degradation score to form a freeze-thaw damage sensitive index; performing gated weighted fusion based on the freeze-thaw damage sensitive index to generate a freeze-thaw damage index vector, and inputting the freeze-thaw damage index vector into a texture evaluation model to output a texture evaluation result. The scheme of the application realizes adaptive fusion and quantitative evaluation of multi-source freeze-thaw damage features in the structure, moisture and optical layers, thereby significantly improving the accuracy and reliability of meat product quality evaluation.

Systems and techniques are provided for physical-based light estimation for inverse rendering of indoor scenes. For example, a computing device can obtain an estimated scene geometry based on a multi-view observation of a scene. The computing device can also obtain a light emission mask based on the multi-view observation of the scene. The computing device can also obtain an emission radiation field based on the multi-view observation of the scene. The computing device can then determine a geometry of at least one light source of the estimated scene geometry based on the light emission mask and the emission radiation field.

The application discloses a head reconstruction method based on generation and inverse rendering collaborative optimization, comprising the following steps: step 1, generating a corresponding binary mask; step 2, fitting a three-dimensional geometric grid of a face matched with an input person based on detected two-dimensional face key points to obtain a unified head coordinate system; step 3, projecting pixels of an input image to a UV space of the three-dimensional geometric grid of the face to form a defective texture with original illumination and self-occlusion; performing illumination regularization and an inverse rendering network to output a multi-channel PBR material map; step 4, generating a group of videos rotating around the head and camera internal and external parameters thereof, and cutting the videos into a multi-view hair image sequence and corresponding virtual camera parameters; step 5, performing multi-view hair reconstruction to obtain a hair geometry representation or equivalent geometry representation representing a three-dimensional structure of hair; and step 6, performing spatial alignment and combination to output a head PBR three-dimensional asset.

The present disclosure relates to an apparatus for synthesizing a 2D image and media using a reverse rendering including 3D spatially varying lighting information estimation, and the apparatus includes a target view analysis unit configured to estimate a normal map and direct lighting using a plurality of 2D images, a material estimation unit configured to reflect the normal map and direct lighting estimated from the target view analysis unit and estimate material information, a 3D lighting estimation unit configured to reflect the direct lighting estimated from the target view analysis unit and the material information estimated from the material estimation unit and estimate 3D spatially varying lighting information, and an image generation unit configured to reflect the estimated shapes, materials, and 3D spatially varying lighting information, synthesize the 2D images with 3D objects, and generate a synthesized image.

The invention discloses an assembly line reflective low-texture part pose detection method based on light field inverse rendering. Comprising the following steps: S1, equipment calibration; s2, generating an ambient light field through light field calibration; s3, collecting an image of the metal part, obtaining an actual edge through edge detection, and generating an initial prediction pose through pose estimation; s4, in the environment light field, obtaining a rendered image under the predicted pose through rendering, obtaining the gradient of the three-dimensional model of the metal part to be detected in the rendered image, and performing normalization and exponentiation processing to obtain a rendered edge; calculating a loss function value and a pose gradient according to the rendering edge and the actual edge, and updating a predicted pose according to the pose gradient; and S5, repeating the step S4 until a preset termination condition is met, and obtaining an optimal predicted pose as a pose detection result. According to the invention, after primary light field calibration is carried out, pose detection tasks of all parts on the assembly line can be completed, secondary calibration is not needed, and the requirement of efficient detection in practical application can be met.

The present application designs a method and system for estimating human geometry, illumination and material from multi-view images captured under unknown illumination based on the idea of joint optimization of neural radiance field rendering and inverse rendering, mainly divided into a geometry pre-training stage and a joint optimization stage, first pre-train the human geometry to obtain a good initialization result, then estimate the illumination and material and fine-tune the geometry initialization result. When estimating the material, the present application uses an anisotropic material model to model complex human material. When estimating the illumination, the present application does not use an additional training stage to model indirect illumination, but considers the exit radiance of indirect reflection points as indirect illumination, and estimates the illumination visibility to distinguish direct illumination and indirect illumination. The present application solves the problem of inaccurate estimation of complex human material, and can effectively decouple geometry, illumination and material from human multi-view images, and improves the realism of human relighting results.

A denoising method using a denoising system is provided. The denoising method including: receiving a target image; generating a rendered image corresponding to the target image based on pre-provided initial scene parameters; calculating denoising weights based on the target image, and generating a refined image by removing noise from at least one of the rendered image and a gradient of a loss for the rendered image based on the denoising weights; and calculating a loss between the target image and the refined image, and updating the scene parameters based on a gradient of the calculated loss.

In certain aspects, application-specific integrated circuit (ASIC) comprises a global ray-tracing scheduler (GRTS) in communication with a processing element (PE) array. The ASIC comprises a global memory access controller (GMAC) in communication with the PE array and the GRTS. The ASIC comprises a physical attributes MEM (PAMEM) in communication with the GRTS. The ASIC comprises a reconfigurable mixed-precision processing element of the PE array configured to support an inverse rendering mode for background extraction and a ray-tracing mode.

Inverse rendering is important for training neural networks for generative artificial intelligence (AI), and it involves inverting the rendering process by taking an image and converting it into scene or model parameters that can be backpropagated through a network, helping to train the network to learn to generate models, materials, textures, etc. Because of the gradients required for this backpropagation, inverse rendering requires differentiable rendering algorithms. Current differentiable renderers are based on rasterization which make it difficult to learn scene properties depending on second order effects. The present disclosure provides closest silhouette queries for computing differential visibility, which can be used for inverse rendering.

The present disclosure relates to reconstructing positron emission tomography (PET) images. The approach may involve receiving measured sinogram data from one or more scans by a PET scanner. The measured sinogram data may represent measured projections from the PET scanner. The approach may involve performing forward rendering to generate a rendered sinogram. Forward rendering may comprise sampling a number of positions corresponding to each crystal detector in a plurality of crystal detectors. The positions may define lines of response (LORs) between crystal pairs. The approach may involve performing inverse rendering based on the measured sinogram data and the rendered sinogram. Inverse rendering may comprise applying auto-differentiation for gradient-based optimization. The rendering may be performed iteratively to update pixel values of an emission image until a stopping criterion. A reconstructed PET image based on the updated emission image may be output following the stopping criterion being satisfied.

The invention provides a multi-view image-based micro-inverse rendering method, which comprises the following steps of: S1, acquiring a group of multi-view RGB (Red, Green, Blue) shot images shot for a three-dimensional object under a fixed illumination condition, acquiring initial camera parameters of the group of images and an initial geometric structure of the object based on an explicit three-dimensional reconstruction technology, and initializing a rendering parameter set, a plurality of directional light sources and a point light source are initially set in the illumination parameters; s2, constructing a differentiable rendering function which is configured to render and output a multi-view-angle linear image by taking the geometric structure, the rendering parameter set and the view angles as input, and converting the multi-view-angle linear image into a multi-view-angle RGB reconstructed image; and S3, the geometric structure, the material parameters and the illumination parameters are optimized to gradually reduce the difference between the multi-view RGB reconstructed image and the multi-view RGB shot image, and the optimized illumination parameters comprise adaptive illumination control, combination of directional light sources with similar radiation contributions, combination of directional light sources with similar irradiation directions and elimination of invalid directional light sources.

The present invention provides a semantic segmentation method based on HDR dynamic neural radiance fields, which relates to the technical field of image segmentation. A semantic segmentation method based on HDR dynamic neural radiance fields is provided. By utilizing the segmentation ability of two-dimensional vision large models on pictures and videos, the segmentation ability of two-dimensional models is extended to four dimensions, and a tone mapping module controlled by hyperparameters is constructed to control the mapping from HDR views to LDR, greatly improving the segmentation ability in HDR scenarios; a mask deformation field network and a dynamic mask inverse rendering method are proposed, which do not require expensive annotation of four-dimensional scenes, and the segmentation of any object in a four-dimensional dynamic scene can be carried out by using two-dimensional picture masks.

The present application provides an image processing method, an electronic device, and a storage medium, relating to the field of image processing. The method includes: displaying a first interface, and in response to a first operation on a first control in the first interface, acquiring a face image; processing the face image by using a trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image; and performing enhancement processing on the face image, the face normal image, the face albedo image, and the environment image by using a trained face enhancement model to obtain a face target image. In this implementation manner, the processing result of the inverse rendering model on the face image is used as face prior information, and the face enhancement model can effectively utilize the face prior information to accurately restore the face. Even under complex light and shadow conditions, it can avoid generating unnatural textures and output high-quality captured images.

Embodiments of the present disclosure relate to rendering and inverse rendering using one or more generative models. “Rendering” refers to the process of generating a final visual image or animation from a 2D or 3D model. “Inverse rendering” is a process that involves deriving or estimating the properties (e.g., material maps or other properties such as geometry, lighting, and textures) of a scene from observed images or visual data. Essentially, it aims to reverse the conventional rendering process. Various aspects of the present disclosure introduce editable lighting and material controls into generative models to enable artistic design. Several embodiments integrate generative models as renderers into traditional rendering pipelines to enhance and improve the style of rendered content.

The disclosure provides a 3D Gaussian-based inverse rendering method, device, equipment and storage medium, and relates to the technical field of artificial intelligence, specifically to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as content generation based on artificial intelligence. The specific implementation scheme is as follows: based on a plurality of two-dimensional images of different perspectives of a target scene, three-dimensional modeling is performed on the target scene to obtain scene attributes of the target scene expressed based on a plurality of 3D Gaussian points; based on the scene attributes, material attributes and lighting parameters of the target scene, color values of each 3D Gaussian point of the target scene are obtained; based on the color values of each 3D Gaussian point, the target scene is projected into a two-dimensional space according to a plurality of perspectives respectively to obtain a plurality of mapping images; and based on the loss between the plurality of mapping images and the two-dimensional images of the plurality of perspectives, the scene attributes, material attributes and lighting parameters are optimized.

A single two-dimensional (2D) image can be used as input to obtain a three-dimensional (3D) representation of the 2D image. This is done by extracting features from the 2D image by an encoder and determining a 3D representation of the 2D image utilizing a trained 2D convolutional neural network (CNN). Volumetric rendering is then run on the 3D representation to combine features within one or more viewing directions, and the combined features are provided as input to a multilayer perceptron (MLP) that predicts and outputs color (or multi-dimensional neural features) and density values for each point within the 3D representation. As a result, single-image inverse rendering may be performed using only a single 2D image as input to create a corresponding 3D representation of the scene in the single 2D image.

The invention discloses an oblique photography LoD model optimization method based on differential rendering, and belongs to the technical field of data processing. The method comprises the steps of loading a to-be-optimized model in an oblique photography LoD model into a target scene, loading a reference model into a reference scene, rendering the target scene and the reference scene through a virtual camera based on photography parameters of oblique photography to obtain a target image and a reference image, and outputting the target image and the reference image. And calculating a spatial error between the target image and the reference image, calculating a model optimization gradient corresponding to the spatial error through inverse rendering, and optimizing the to-be-optimized model based on the model optimization gradient to obtain an optimized model. In the LoD model after oblique photography modeling, based on a high-level high-precision model as a reference, optimization iteration is carried out on a low-level low-precision model, the space error between the low-level model and the high-level model is reduced, and therefore the model precision of the low-level model is improved.

The application discloses a scene reverse rendering method and device based on neural implicit expression, and the method is based on a signed distance field network, a neural radiation field network and a neural material field network to realize joint implicit expression of geometric information, illumination information and material information of a scene, and a ray tracing rendering algorithm based on Monte Carlo sampling is used to obtain a rendering result, a loss function is constructed according to the rendering result to optimize network parameters, and the signed distance field network, the neural radiation field network and the neural material field network capable of accurately predicting the geometric information, the illumination information and the material information are obtained. Meanwhile, the method also supports re-illumination editing and material editing, and achieves an augmented reality function.

The application discloses a power grid digital space modeling and inverse rendering method based on three-dimensional Gauss, and the application divides an image into transparent object Gauss and opaque environment Gauss through a finely adjusted SAM2 model, estimates the transparent object Gauss information by using a finely adjusted DiffusionRenderer model and a finely adjusted VGGT model, then jointly optimizes and reconstructs the transparent object Gauss and a traditional triangular mesh, and jointly optimizes and reconstructs the opaque environment Gauss and the transparent object Gauss. A new scene is composed of the reconstructed transparent object, the reconstructed environment and other reconstructed power grid scenes and traditional triangular meshes made by an art designer, so that virtual-real fusion and mixed real-time rendering are realized. The application guides Gauss ray tracing by using triangular mesh ray tracing, realizes precalculation and table optimization of light transmission paths by combining neural Gauss light transmission fields, and models environment light by using a neural Gauss incident light field, so that the calculation cost during rendering is greatly reduced.

The present application discloses an optimization method for an oblique photography LoD model based on differentiable rendering, belonging to the technical field of data processing. The method includes: loading the model to be optimized in the oblique photography LoD model into the target scene, and loading the reference model into the reference scene. Based on the photography parameters of the oblique photography, rendering the target scene and the reference scene through a virtual camera to obtain a target image and a reference image, calculating the spatial error between the target image and the reference image, calculating the model optimization gradient corresponding to the spatial error through inverse rendering, and optimizing the model to be optimized based on the model optimization gradient to obtain an optimized model. The present application, in the LoD model after oblique photography modeling, takes a high-level high-precision model as a reference to optimize and iterate a low-level lower-precision model, reducing the spatial error between the low-level model and the high-level model, thereby improving the model accuracy of the low-level model.

The present disclosure relates to a method, device, equipment and medium for inverse rendering an image. The method includes: inputting an image to be processed into a feature prediction model, predicting a geometric feature and a material feature of the image to be processed by the feature prediction mode to acquire a geometric feature map and a material feature map of the image to be processed, where the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map, and a metalness feature map; inputting the image to be processed, the geometric feature map and the material feature map into a lighting prediction model, and predicting a lighting value of the image to be processed pixel by pixel by the lighting prediction model to acquire a lighting feature map of the image to be processed.