Adaptive representation and dynamic drawing method and system based on deformable implicit light transport functions
By using an adaptive representation method based on deformable implicit optical transfer functions, the problem of insufficient rendering quality in complex dynamic scenes is solved, achieving efficient and high-quality rendering effects and supporting the generation of complex high-frequency effects.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2025-11-27
- Publication Date
- 2026-06-11
Smart Images
Figure CN2025138119_11062026_PF_FP_ABST
Abstract
Description
An Adaptive Representation and Dynamic Drawing Method and System Based on Deformable Implicit Optical Transfer Function Technical Field
[0001] This invention belongs to the field of rendering technology, specifically relating to an adaptive representation and dynamic rendering method and system based on deformable implicit optical transfer functions. Background Technology
[0002] Rendering technology has long been a crucial research area in computer graphics, widely applied in fields such as video games, virtual reality, and film special effects. The primary goal of rendering technology is to generate realistic images, much like the real world, through reasonable data modeling and simulation calculations. Traditional rendering techniques, especially realistic rendering techniques, require massive amounts of computation to achieve high-quality results. Taking the most widely used path tracing technology as an example, it typically takes minutes or even hours to generate a high-quality single frame, significantly increasing costs and limiting application scenarios. For instance, in game applications, developers are often required to render at a rate of at least 60 frames per second to provide players with a low-latency, high-sensitivity gaming experience. This means that the average rendering time per frame cannot exceed 16.7 milliseconds, making it difficult to meet the requirements of high-quality rendering and necessitating simplification and compromises in the rendering process and quality.
[0003] In recent years, neural network algorithms and hardware have developed rapidly, and more and more researchers have begun to apply neural networks to the field of computer graphics, achieving breakthroughs in areas such as complex lighting representation, fine material representation, and realistic appearance reconstruction. Neural networks have also demonstrated enormous potential in high-quality and efficient realistic rendering. Although existing neural network-based rendering methods can quickly and with high quality render static and simple dynamic scenes, they still struggle to achieve satisfactory rendering quality for industrial-grade dynamic scenes containing complex geometric details and a large amount of fine materials. Furthermore, existing methods have difficulty supporting the reconstruction of complex high-frequency rendering effects, such as caustics and indirect specular highlights.
[0004] Patent application CN115393498A discloses a rendering method and system based on implicit optical transfer function merging. Specifically, it includes: pre-dividing all objects in a 3D scene (excluding light sources) into multiple object groups and defining the insertion order of these groups; then inserting the divided object groups into the scene sequentially; and using implicit optical transfer function calculations based on light source sampling data, target viewpoint data, and object group data to predict the optical transfer change field of the object groups in the current 3D scene. Finally, the optical transfer change field, combined with the mask map of the object groups, is merged into the rendering result before object group insertion using matrix calculations, employing a split-and-merge approach. While this technical solution uses a split-and-merge method to render objects in the scene, it does not consider the implicit influence between objects. Summary of the Invention
[0005] In view of the above, the purpose of this invention is to provide an adaptive representation and dynamic rendering method and system based on deformable implicit optical transport functions. The method constructs deformable optical transport fields through deformable implicit optical transport functions, implicitly merges the optical transport fields of multiple objects in the scene through adaptive representation, and automatically considers the contribution of each optical transport field to the final rendering result, thereby achieving high-quality rendering of three-dimensional dynamic scenes.
[0006] To achieve the above-mentioned objectives, an embodiment provides an adaptive representation and dynamic rendering method based on a deformable implicit optical transfer function, comprising the following steps:
[0007] The scene is divided into object groups, and the implicit influence between object groups is calculated. The scene representation for each object group is calculated based on the implicit influence, where each object group contains at least one object.
[0008] Calculate the deformable implicit light transfer function of each object group based on the scene representation of each object group;
[0009] Adaptively merge deformable implicit light transfer functions of each object group and dynamically render them.
[0010] Calculate the implicit effects between groups of objects, including:
[0011] The implicit influence between objects is calculated using a first neural network based on their relative positions, relative orientations, and material parameters.
[0012] Preferably, the implicit influence-based calculation of scene representations for each object group includes:
[0013] Aggregate the implicit influences of other object groups on each object group in the scene to obtain a scene representation for each object group.
[0014] Preferably, the deformable implicit optical transfer function of each object group is calculated based on the scene representation of each object group, including:
[0015] Based on the scene representation of each object group, a second neural network is used to construct a deformable implicit light transfer function for each object group to model the influence of each object group on the rendering results. The deformable implicit light transfer function is represented as a feature field, which calculates features based on the scene representation facing the corresponding object group, the viewing direction, and the position of the visible surface.
[0016] Preferably, based on the scene representation of each object group, a deformable implicit optical transport function is constructed for each object group using a second neural network, including:
[0017] The third neural network predicts the offset based on the scene representation of each object group and obtains the offset position features based on the offset position. Then, the second neural network constructs the deformable implicit optical transport function of each object group based on the offset position features, the viewing direction, and the scene representation of each object group.
[0018] Preferably, adaptively merging the deformable implicit optical transfer functions of each object group and dynamically rendering them includes:
[0019] The deformable implicit optical transport functions of multiple object groups are implicitly merged, and the merged optical transport functions, combined with the target camera position and viewing direction, are fed into the fourth neural network to predict and render the results.
[0020] Preferably, the method of implicitly merging the deformable implicit optical transfer functions of multiple object groups includes:
[0021] Addition, multiplication, attention-based fusion, and neural network-based fusion.
[0022] To achieve the above-mentioned objectives, the embodiments also provide an adaptive representation and dynamic rendering system based on a deformable implicit optical transfer function, comprising:
[0023] The scene representation module is used to divide the scene into object groups and calculate the implicit influence between object groups. Based on the implicit influence, it calculates the scene representation for each object group, where each object group contains at least one object.
[0024] The optical transport function calculation module is used to calculate the deformable implicit optical transport function of each object group based on the scene representation of each object group.
[0025] The merge and render module is used to adaptively merge deformable implicit optical transport functions of groups of objects and render them dynamically.
[0026] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned adaptive representation and dynamic rendering method based on deformable implicit optical transfer function.
[0027] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned adaptive representation and dynamic rendering method based on deformable implicit optical transfer function.
[0028] Compared with the prior art, the beneficial effects of the present invention include at least the following:
[0029] By constructing a deformable implicit light transport function that changes according to the scene layout, we can better model complex rendering effects. Furthermore, by implicitly merging the light transport functions of multiple object groups, we can adaptively consider the influence of each object on the rendering results. This can generate higher quality rendering results, support more complex rendering effects, and achieve faster rendering speed by appropriately reducing the size of the neural network. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 is a flowchart of the adaptive representation and dynamic rendering method based on deformable implicit optical transfer function provided in the embodiment;
[0032] Figure 2 is a schematic diagram of the implicit merging method of deformable implicit optical transfer functions corresponding to multiple objects provided in the embodiment;
[0033] Figure 3 is a schematic diagram of the structure of the adaptive representation and dynamic rendering system based on the deformable implicit optical transfer function provided in the embodiment. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0035] As shown in Figure 1, the embodiment provides an adaptive representation and dynamic rendering method based on a deformable implicit optical transfer function, which includes the following steps:
[0036] S1 divides the scene into object groups and calculates the implicit influence between object groups. Based on the implicit influence, it calculates the scene representation for each object group.
[0037] In this embodiment, the 3D scene is divided into several object groups, each object group containing at least one object; that is, each object group consists of one object or a combination of multiple objects. Generally, objects grouped into the same object group are required to remain relatively stationary, meaning the relative positions and orientations between any two objects remain unchanged, but the entire object group can be dynamic. In practice, for efficiency reasons, as many relatively stationary objects as possible are placed in the same object group.
[0038] In this embodiment, when calculating the implicit influence between object groups, a first neural network is used to calculate the implicit influence between objects based on the relative positions, relative postures, and material parameters of each object group. A feasible implementation scheme is provided here, but not limited to this scheme: the implicit influence r of object group i on object group j. ij It can be represented as:
[0039] Among them, M ij This represents the transformation matrix from the object space of object group i to the object space of object group j. This matrix contains the relative position and relative pose information between object group i and object group j. i This represents the material parameters of object group i. This represents the neural network used to calculate the influence of object i.
[0040] In this embodiment, after obtaining the implicit influence between each object group, the implicit influence of other object groups on each object group in the scene is aggregated to obtain an object-oriented scene representation. Specifically, for each object group, the scene representation for that object is obtained by aggregating the influence of other object groups in the scene. A feasible implementation scheme is provided here, but not limited to this scheme: a scene representation r for object group i. i The implicit influence of all scene objects on it can be calculated by summing up the influence, i.e.: r i =∑ j r ji
[0041] S2, calculate the deformable implicit optical transfer function of each object group based on the scene representation of each object group.
[0042] In this embodiment, based on the object-oriented scene representation, a second neural network with optimizable parameters is used to represent the deformable implicit phototransmission function of each object. This deformable implicit phototransmission function can deform according to changes in the scene layout to achieve higher-quality modeling of complex rendering effects. Specifically, the deformable implicit phototransmission function of object group i can be represented as a feature field:
[0043] Among them, the feature field returns the light source observed from the direction ω. Lighting scene The features of sampling point x in the dataset. Wherein and It represents a scene; scene representation is actually an abstract representation of a scene, that is, a representation of... and The abstract result.
[0044] To enable the implicit optical transport function to better model complex rendering effects, deformable properties are further introduced. A feasible implementation scheme incorporating offset deformation is provided here, but more deformation properties, such as rotation and scaling, can be added. Specifically, a third neural network is used. Based on the object-oriented scene representation r i Predicted offset And based on the offset position x+ Obtain offset position features Then by the second neural network Hybrid coded offset position features The viewing direction ω and the scene representation r for each object group i To construct a deformable implicit light transport function for each group of objects:
[0045] S3 adaptively merges the deformable implicit light transfer functions of each object group and performs dynamic rendering.
[0046] In this embodiment, each scene contains multiple object groups, corresponding to multiple deformable implicit optical transfer functions. The deformable implicit optical transfer functions of multiple object groups are implicitly merged, specifically through an order-independent merging method for use in prediction results. The merged feature field It can be represented as:
[0047] in This indicates a merging method that maintains the order. This embodiment uses addition (as shown in Figure 2), but attention mechanisms, multiplication, and order-independent neural network structures are all possible implementation schemes.
[0048] In this embodiment, the merged optical transport function, combined with the target camera position and viewing direction, is fed into the fourth neural network to predict the rendering result. Specifically, a target viewpoint is first generated based on the target camera position and viewing direction, and geometric and material features of the visible parts of the 3D scene under the target viewpoint are generated. These features, along with the merged optical transport function, are then fed into the fourth neural network to predict the rendering result.
[0049] The fourth neural network is supervised learning using the ground truth of the rendering results of the 3D scene, which automatically considers the influence of the light transfer function of each object on the rendering results in a data-driven manner.
[0050] To train the neural network involved in this embodiment, it is necessary to obtain the ground truth of the rendering result and use the difference between the calculated result and the predicted rendering result as the loss function to jointly optimize all neural network parameters. By summing the implicit features of multiple object groups, the implicit contribution of the feature field of each object group to the final result is automatically optimized based on the training data, and then decoded by a fourth neural network.
[0051] In contrast to the implicit merging method of this invention, there is explicit merging, such as predicting the color contribution of each object and then combining the colors of each object. This method has explicit ground truth values for color contributions, which can be used for supervised training. However, this may actually add extra constraints and worsen the performance. Compared to explicit merging, the implicit merging method of this invention does not add extra constraints and achieves better rendering results.
[0052] As shown in Figure 3, the embodiment also provides an adaptive representation and dynamic rendering system 30 based on deformable implicit optical transfer functions, including a scene representation module 31, an optical transfer function calculation module 32, and a merging and rendering module 33. The scene representation module 31 is used to divide the scene into object groups and calculate the implicit influence between object groups, and calculate the scene representation for each object group based on the implicit influence. The optical transfer function calculation module 32 is used to calculate the deformable implicit optical transfer function of each object group based on the scene representation of each object group. The merging and rendering module 33 is used to adaptively merge the deformable implicit optical transfer functions of each object group and perform dynamic rendering.
[0053] It should be noted that the adaptive representation and dynamic rendering system based on deformable implicit optical transfer functions provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing adaptive representation and dynamic rendering. The functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the adaptive representation and dynamic rendering system based on deformable implicit optical transfer functions provided in the above embodiments and the method embodiments based on deformable implicit optical transfer functions belong to the same concept. For details of their specific implementation, please refer to the method embodiments based on deformable implicit optical transfer functions, which will not be repeated here.
[0054] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned adaptive representation and dynamic rendering method based on deformable implicit optical transfer function, specifically including the following steps:
[0055] S1, divide the scene into object groups, calculate the implicit influence between object groups, and calculate the scene representation for each object group based on the implicit influence;
[0056] S2, calculate the deformable implicit light transfer function of each object group based on the scene representation of each object group;
[0057] S3 adaptively merges the deformable implicit light transfer functions of each object group and performs dynamic rendering.
[0058] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the adaptive representation and dynamic rendering method based on deformable implicit optical transfer functions described in S1-S3 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0059] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described adaptive representation and dynamic rendering method based on a deformable implicit optical transfer function, specifically including the following steps:
[0060] S1, divide the scene into object groups and calculate the implicit influence between object groups, and calculate the scene representation for each object group based on the implicit influence;
[0061] S2, calculate the deformable implicit light transfer function of each object group based on the scene representation of each object group;
[0062] S3 adaptively merges the deformable implicit light transfer functions of each object group and performs dynamic rendering.
[0063] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.
[0064] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An adaptive representation and dynamic rendering method based on deformable implicit optical transfer functions, characterized in that, Includes the following steps: The scene is divided into object groups, and the implicit influence between object groups is calculated, including: using a first neural network to calculate the implicit influence between objects based on the relative position, relative pose and material parameters of each object group. The implicit influence-based computation of scene representations for each object group includes: aggregating the implicit influences of other object groups on each object group in the scene to obtain a scene representation for each object group, wherein each object group contains at least one object; The deformable implicit light transfer function of each object group is calculated based on the scene representation of each object group, including: constructing a deformable implicit light transfer function for each object group using a second neural network based on the scene representation of each object group to model the influence of each object group on the rendering results. The deformable implicit light transfer function is represented as a feature field, which calculates features based on the scene representation facing the corresponding object group, the viewing direction, and the position of the visible surface. Adaptively merge deformable implicit light transfer functions of each object group and dynamically render them.
2. The adaptive representation and dynamic rendering method based on deformable implicit optical transfer function according to claim 1, characterized in that, Based on the scene representation of each object group, a deformable implicit optical transport function is constructed for each object group using a second neural network, including: The third neural network predicts the offset based on the scene representation of each object group and obtains the offset position features based on the offset position. Then, the second neural network constructs the deformable implicit optical transport function of each object group based on the offset position features, the viewing direction, and the scene representation of each object group.
3. The adaptive representation and dynamic rendering method based on deformable implicit optical transfer function according to claim 1, characterized in that, Adaptively merge deformable implicit light transfer functions of each object group and dynamically render them, including: The deformable implicit optical transport functions of multiple object groups are implicitly merged, and the merged optical transport functions, combined with the target camera position and viewing direction, are fed into the fourth neural network to predict and render the results.
4. The adaptive representation and dynamic rendering method based on deformable implicit optical transfer function according to claim 3, characterized in that, The methods for implicitly merging deformable implicit optical transport functions of multiple object groups include: Addition, multiplication, attention-based fusion, or neural network-based fusion.
5. An adaptive representation and dynamic rendering system based on deformable implicit optical transfer functions, characterized in that, include: The scene representation module is used to divide the scene into object groups and calculate the implicit influence between object groups, including: using a first neural network to calculate the implicit influence between objects based on the relative position, relative pose and material parameters of each object group. The implicit influence-based computation of scene representations for each object group includes: aggregating the implicit influences of other object groups on each object group in the scene to obtain a scene representation for each object group, wherein each object group contains at least one object; The optical transport function calculation module is used to calculate the deformable implicit optical transport function of each object group based on the scene representation of each object group. This includes: constructing a deformable implicit optical transport function for each object group using a second neural network based on the scene representation of each object group to model the influence of each object group on the rendering results. The deformable implicit optical transport function is represented as a feature field, which calculates features based on the scene representation facing the corresponding object group, the viewing direction, and the position of the visible surface. The merge and render module is used to adaptively merge deformable implicit optical transport functions of groups of objects and render them dynamically.
6. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the adaptive representation and dynamic rendering method based on the deformable implicit optical transfer function as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the adaptive representation and dynamic rendering method based on the deformable implicit optical transfer function as described in any one of claims 1-4.