Image generation method and apparatus, device, storage medium, and product

Abstract

An image generation method, performed by an electronic device, and comprising: acquiring the vertex position, vertex feature and vertex Gaussian attribute of each graphic unit, and acquiring interpolation information of each graphic unit, each graphic unit being used for representing the surface of an object (S301); on the basis of the interpolation information, performing interpolation on the vertex position and the vertex feature in each graphic unit to obtain an interpolation point position and an interpolation feature in the graphic unit (S302); generating an interpolation Gaussian attribute of each graphic unit on the basis of the interpolation feature and information of a specified viewpoint (S303); and on the basis of the vertex position and the vertex Gaussian attribute as well as the interpolation point position and the interpolation Gaussian attribute, performing rendering processing to obtain a reconstructed image of the object at the specified viewpoint (S304).

Description

Image generation methods, apparatus, devices, storage media and products

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on February 25, 2025, with application number 202510210318.0 and entitled "Image generation method, apparatus, device, storage medium and product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, specifically to an image generation method, apparatus, device, storage medium, and product. Background Technology

[0004] With the rapid development of computer graphics technology, 3D Gaussian Splatting (3DGS) technology, with its realistic 3D scene reconstruction capabilities, has been increasingly applied in fields such as film and television special effects, virtual reality, and game development. This technology models the surfaces of objects in a 3D scene as a large number of Gaussian splatter points, using a 3D Gaussian distribution to describe local regions in 3D space. These local regions are used to approximate complex 3D scenes. During rendering, these Gaussian splatter points are projected onto a 2D image plane to obtain the reconstructed image.

[0005] However, since scene modeling relies on tens of thousands of Gaussian splash points, each of which needs to store multiple parameters such as color and position, the memory requirements for generating high-quality reconstructed images increase dramatically, severely limiting the deployment of this technology on mobile devices or resource-constrained edge devices.

[0006] Therefore, how to reduce the demand for memory resources while maintaining high-quality visual effects has become a pressing technical problem that needs to be solved. Summary of the Invention

[0007] This application provides an image generation method, apparatus, device, storage medium, and product.

[0008] In a first aspect, embodiments of this application provide an image generation method, the method comprising:

[0009] The vertex positions, vertex features, and vertex Gaussian properties of each graphics unit are obtained, and the interpolation information of each graphics unit is obtained. Each graphics unit is used to represent the surface of an object.

[0010] Based on the interpolation information, the vertex position and the vertex feature are interpolated in each graphic unit to obtain the interpolation point position and interpolation feature in each graphic unit.

[0011] Based on the interpolation features and the information from the specified viewpoint, the interpolation Gaussian properties of each graphics unit are generated;

[0012] Rendering is performed based on the vertex position and the vertex Gaussian property, as well as the interpolation point position and the interpolation Gaussian property, to obtain a reconstructed image of the object at the specified viewpoint.

[0013] Secondly, embodiments of this application provide an image generation apparatus, the apparatus comprising:

[0014] The acquisition unit is used to acquire the vertex position, vertex features and vertex Gaussian properties of each graphics unit, and to acquire the interpolation information of each graphics unit, wherein each graphics unit is used to represent the surface of an object.

[0015] The processing unit is configured to interpolate the vertex position and the vertex feature in each graphic unit based on the interpolation information, so as to obtain the interpolation point position and interpolation feature in each graphic unit.

[0016] A generation unit is used to generate the interpolation Gaussian properties of each graphics unit based on the interpolation features and information from a specified viewpoint.

[0017] The processing unit is further configured to perform rendering processing based on the vertex position and the vertex Gaussian attribute, as well as the interpolation point position and the interpolation Gaussian attribute, to obtain a reconstructed image of the object at the specified viewpoint.

[0018] Thirdly, embodiments of this application provide an electronic device, which includes one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the image generation method of the first aspect described above.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the image generation method of the first aspect described above.

[0020] Fifthly, embodiments of this application provide a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, they implement the image generation method as described in the first aspect.

[0021] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0023] Figure 1 is a schematic diagram of the principle of 3DGS and the effect of reconstructed image provided in an embodiment of this application;

[0024] Figure 2 is a schematic diagram of the architecture of an image generation system provided in an embodiment of this application;

[0025] Figure 3 is a flowchart illustrating an image generation method provided in an embodiment of this application;

[0026] Figure 4 is a schematic diagram of a 3DGS inference process provided in an embodiment of this application;

[0027] Figure 5 is another schematic flowchart of another image generation method provided in an embodiment of this application;

[0028] Figure 6 is a schematic diagram illustrating the principle of interpolation processing within a graphics unit according to an embodiment of this application;

[0029] Figure 7 is a schematic diagram of the principle of a trimming graphic unit provided in an embodiment of this application;

[0030] Figure 8 is a schematic diagram of a LOD downsampling principle provided in an embodiment of this application;

[0031] Figure 9 is a schematic diagram of a 3DGS training process provided in an embodiment of this application;

[0032] Figure 10 is a comparison diagram of the effects of another 3DGS solution provided in the embodiments of this application and the solution of this application;

[0033] Figure 11 is a schematic diagram of a quantitative evaluation at different levels of detail provided in an embodiment of this application;

[0034] Figure 12 is a schematic diagram of a qualitative assessment at different levels of detail provided in an embodiment of this application;

[0035] Figure 13 is a schematic diagram of an image generation device provided in an embodiment of this application;

[0036] Figure 14 is a schematic diagram of the structure of a computer system for an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0039] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0040] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0041] 1. 3DGS

[0042] 3DGS technology is a technique used in computer graphics and visual effects to generate high-quality renderings of three-dimensional (3D) scenes. This technique achieves efficient rendering by using a large number of Gaussian distributions to represent points or volumes in 3D space and then "splashing" these Gaussian distributions onto a two-dimensional (2D) image plane.

[0043] In 3DGS, two main concepts are involved: Gaussian distribution and splatting, where:

[0044] 1) The Gaussian distribution, also known as the normal distribution, is a continuous probability distribution in statistics. In 3DGS, each data point in 3D space can be represented by a three-dimensional Gaussian distribution. Such a three-dimensional Gaussian distribution can be called a Gaussian splat, or simply Gaussians. It can be understood that an object in 3D space is modeled as multiple Gaussian points. For a multidimensional Gaussian distribution, it can be described by the mean and covariance matrix. For example, a three-dimensional Gaussian distribution can be seen in Equation 1:

[0045] In Equation 1, G(x) represents a three-dimensional Gaussian distribution, x is a point in three-dimensional space, and μ∈R. 3 It is the center location of the Gaussian distribution, Σ∈R 3×3 Σ is the covariance matrix, describing the shape, size, and orientation of the Gaussian distribution. To more flexibly adjust the shape and orientation of the Gaussian distribution to better adapt to the geometry of a scene in 3D space, the covariance matrix Σ can be further decomposed into a scaling matrix S∈R. 3 and a rotation matrix R∈R 3×3 , such that Σ = RSS T R T .

[0046] In 3DGS, each 3D Gaussian distribution (Gaussian point) can be imagined as an "ellipsoid" in 3D space. The center of the ellipsoid corresponds to the mean μ of the Gaussian distribution, representing the position of that Gaussian point in 3D space. The size and shape of the ellipsoid are determined by the covariance matrix Σ. Unlike an ideal sphere, these ellipsoids can be stretched or compressed along different axes, exhibiting a flattened or elongated state. That is, the covariance matrix is ​​used to describe the extent of the ellipsoid's expansion in three directions. To use a 3D Gaussian distribution to represent a scene in 3D space, the size and orientation of the 3D Gaussian distribution used by 3DGS may be different; that is, the size and shape of the ellipsoid may be different. This can be described as 3DGS using anisotropic 3D Gaussian distributions distributed in 3D space to represent a scene.

[0047] To represent color when viewed from different perspectives, each 3D Gaussian distribution is further augmented with spherical harmonics (SH) and opacity to model the color. SH∈R 48 Used to represent color changes when viewed from different perspectives, it can capture complex lighting effects. Opacity α∈R can be used to represent the degree of contribution of the Gaussian distribution to the final pixel color.

[0048] Therefore, in 3DGS, a three-dimensional Gaussian distribution (Gaussian point) can be defined by the following parameters: Position, Scale, Color, Quaternion, and Opacity. The position p corresponds to the mean of the three-dimensional Gaussian distribution, i.e., the center position of the ellipsoid; the scale s can be obtained by transforming the scaling matrix of the three-dimensional Gaussian distribution; the color c can be represented by SH; the quaternion q can be obtained by transforming the rotation matrix; and the opacity α can be learned and used to control the visibility of this three-dimensional Gaussian distribution.

[0049] 2) Splatting, also known as sputtering, refers to the process of projecting a three-dimensional Gaussian distribution onto a two-dimensional image plane. Specifically, for each three-dimensional Gaussian distribution, its projection onto the two-dimensional image plane (camera viewpoint) is calculated based on its position, scale, and orientation (quaternions), and color blending is performed based on this to form the final image. In 3DGS, the splatting process can be imagined as "flattening" these ellipsoids onto the two-dimensional image plane according to their position and shape in three-dimensional space. Each ellipsoid generates a corresponding 2D projection based on its angle and distance relative to the observer. These projections are then blended together according to their color and opacity to form the final rendering result, resulting in the final image. This blending process is called α-blending, which refers to the process of calculating the cumulative color of each pixel. The formula for calculating pixel color can be found in Formula 2: C = ∑ i∈N T i c i α i Formula 2

[0050] In Formula 2, C represents pixel color, N represents the number of two-dimensional Gaussian distributions (projections of three-dimensional Gaussian distributions onto a two-dimensional image plane) ordered from near to far along the viewing direction, T represents the transmittance of the i-th two-dimensional Gaussian distribution, and c i Let α represent the color of the i-th two-dimensional Gaussian distribution. i Let represent the opacity of the i-th two-dimensional Gaussian distribution. Transmittance, representing the attenuation of light reaching the i-th two-dimensional Gaussian distribution, can be derived from... The calculated transmittance represents the product of the transparency of all preceding Gaussian distributions. In other words, during α-mixing, the image generation device traverses all Gaussian distributions from near to far based on the viewing direction, calculating the contribution c of each Gaussian distribution to the pixel color. i α i And considering the transmittance T iThe final pixel color C is obtained by summing the contributions of all Gaussian distributions.

[0051] 2. Level of Detail (LOD)

[0052] Level of Detail (LOD) is an optimization technique in computer graphics used to dynamically adjust the level of detail of objects in a scene based on their distance from the viewer or their importance, thus adjusting the rendering complexity. The main purpose of LOD is to improve rendering efficiency and performance without impacting user experience. When an object is far from the viewer or contributes little to the scene, a lower resolution or simplified model is used to represent it, reducing computation. Conversely, when an object is close to the viewer or contributes significantly to the scene, a more detailed, higher resolution model is used, which can significantly reduce computation while maintaining visual fidelity. LOD is typically implemented through a series of predefined model hierarchies, each corresponding to a different distance range or importance. Choosing an appropriate LOD level can significantly improve rendering speed, especially when dealing with objects in large-scale scenes.

[0053] In 3DGS, objects in a scene are composed of a large number of 3D Gaussian distributions (i.e., Gaussian points). The number of Gaussian points can be dynamically adjusted for electronic devices with varying computing and memory resources. For example, for electronic devices with limited computing and memory resources, the number of Gaussian points used to model objects in 3D space can be reduced to decrease the computational and storage requirements. The level of detail with the largest number of Gaussian points can be called Level of Detail 0 (LOD0), and the levels of detail with progressively fewer Gaussian points can be successively called Level of Detail 1 (LOD1), Level of Detail 2 (LOD2), Level of Detail 3 (LOD3), and so on.

[0054] In the process of image reconstruction based on 3DGS technology, it was found that modeling a 3D scene relies on tens of thousands of Gaussian points. After modeling, parameters such as position p, scale s, color c, quaternion q, and opacity α need to be stored for each Gaussian point. Since the spherical harmonic coefficients of SH involve 48 dimensions (SH∈R...),... 48 This necessitates storing 60 parameters for each Gaussian point, drastically increasing the memory requirements for generating high-quality reconstructed images and severely limiting the deployment of 3DGS technology on mobile devices or resource-constrained edge devices. Furthermore, objects in a 3D scene are modeled using multiple Gaussian points, which are irregularly distributed. Without any structural constraints, these points can be located anywhere, leading to potential visual artifacts in the rendered reconstructed image, such as floating objects, resulting in poor image quality.

[0055] Based on this, this application provides an image generation scheme by modeling objects in three-dimensional space as discrete graphic units, with Gaussian points distributed within these units. Please refer to Figure 1, which is a schematic diagram of the 3DGS principle and the effect of reconstructed images provided by this application. As shown in Figure 1, taking a forklift block as an example, the object in three-dimensional space is modeled as a set of triangles (triangle soup). As shown in the upper left corner of Figure 1, the discrete graphic unit can refer to discrete triangles, where triangles do not share sides. Each triangle contains multiple Gaussian points, which can be called a Gaussian Soup. As shown in the upper right corner of Figure 1, the vertices of the triangles and the ellipses inside represent Gaussian points.

[0056] Therefore, based on the relationship between graphics units and Gaussian points, the various attributes of Gaussian points can be generated from the information of the graphics units. This allows for the generation of reconstructed images by storing only the information of the graphics units, reducing the amount of stored data while maintaining image quality. This effectively reduces memory resource consumption and increases the possibility of deploying 3DGS technology in memory-constrained scenarios, as shown in the lower right corner of the image in Figure 1. Furthermore, the Gaussian points of objects in the modeled 3D scene contain structural constraints, resulting in better reconstructed images rendered based on these structurally constrained Gaussian points. As seen in the image in the middle of Figure 1, at LOD0, the images from this application have more detail compared to other 3DGS schemes, as shown in the lower left corner of the reconstructed image at LOD0 in Figure 1, where the peak signal-to-noise ratio (PSNR) is higher.

[0057] Furthermore, during the modeling process at different levels of detail (LOD), the irregular distribution of Gaussian points in 3D space, due to the lack of structural constraints between multiple Gaussian points, may result in random reduction of Gaussian points for LOD1, LOD2, etc., potentially leading to a loss of information about objects in the 3D scene and thus reducing image quality. For example, the lower PSNR and fewer details in the forklift area are shown in the lower left corner of the reconstructed images under LOD1 and LOD2. In this application, the number of Gaussian points is reduced at the graphics unit level, as shown in the upper right of Figure 1 by reducing the number of ellipses inside the triangles in LOD0-LOD2. This can preserve more information to a certain extent, thereby improving the image quality of the reconstructed image. As shown in the LOD chart below Figure 1, the required memory resources are lower and the image quality (higher PSNR) is better under different LOD levels.

[0058] Based on the above description, please refer to Figure 2. Figure 2 is a schematic diagram of the architecture of an image generation system provided in an embodiment of this application. As shown in Figure 2, the image generation system includes an image generation device 201 and a modeling device 202. The image generation device 201 and the modeling device 202 can be directly or indirectly connected via wired or wireless means. It should be noted that the number and form of the devices shown in Figure 2 are for illustrative purposes only and do not constitute a limitation on the embodiments of this application. In some embodiments, there can be multiple image generation devices 201 and modeling devices 202. In some embodiments, the image generation device 201 and the modeling device 202 can be the same electronic device.

[0059] The image generation device 201 can be an electronic device for generating reconstructed images. The image generation device 201 can acquire modeling information of objects in different 3D scenes from the modeling device 202, and generate reconstructed images based on the modeling information. Specifically, the image generation device 201 may include input devices, such as a touch screen, keyboard, microphone, etc., allowing the user to input information from a specified viewpoint to indicate that the generated reconstructed image is an image of the object from the specified viewpoint. The image generation device 201 may also include output devices, such as a display screen, for outputting the generated reconstructed images.

[0060] The modeling device 202 can be an electronic device that provides modeling information for objects in different 3D scenes, and can generate this modeling information based on a set of feature points representing the objects. The image generation device 201 can acquire modeling information for one or more objects from the modeling device 202, and generate reconstructed images of the objects based on the modeling information.

[0061] The image generation method provided in this application can be executed by an image generation device 201, which can be the same device as the modeling device 202. Both the image generation device 201 and the modeling device 202 can be terminal devices or servers. Terminal devices include, but are not limited to, smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (MIDs), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc. This application does not limit the specific types of devices. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This application does not limit the specific types of servers provided.

[0062] The general flow of the image generation method provided in this application is as follows:

[0063] Image generation device 201 can acquire modeling information from modeling device 202. This modeling information includes vertex positions, vertex features, vertex Gaussian properties, and interpolation information for each graphical unit representing the object's surface. Image generation device 201 can perform interpolation processing on the vertex positions and vertex features within each graphical unit based on the interpolation information, obtaining the interpolation point positions and interpolation features within each graphical unit. Based on the interpolation features and information from a specified viewpoint, it generates the interpolated Gaussian properties for each graphical unit. Image generation device 201 can perform rendering processing based on the vertex positions and vertex Gaussian properties, as well as the interpolation point positions and interpolated Gaussian properties, to obtain a reconstructed image of the object from the specified viewpoint.

[0064] In some embodiments, the modeling device 202 can acquire initial vertex positions, initial vertex Gaussian properties, initial interpolation point positions, and initial interpolation Gaussian properties for each initial graphical unit representing the object surface; this information can serve as initial modeling information. The modeling device 202 can then perform rendering processing based on the initial vertex positions and initial vertex Gaussian properties, as well as the initial interpolation point positions and initial interpolation Gaussian properties, to obtain a first training image of the object from a reference viewpoint. Subsequently, the modeling device 202 can generate first loss data based on the difference between the first training image and the reference image of the object from the reference viewpoint, and iteratively adjust the initial vertex positions, initial vertex Gaussian properties, initial interpolation point positions, and initial interpolation Gaussian properties based on the first loss data.

[0065] Therefore, the modeling device 202 can use the vertex positions, vertex Gaussian properties, interpolation point positions, and interpolation Gaussian properties of the initial graphic units obtained through iterative adjustment as modeling information to generate a reconstructed image, such as sending it as modeling information to the image generation device 201. Since the interpolation point positions of the initial graphic units obtained through iterative adjustment can be calculated based on the vertex positions and interpolation coordinates, and the interpolation Gaussian properties can be generated based on the vertex features generated from the vertex positions and the information of the specified viewpoint, only the vertex positions, vertex Gaussian properties, interpolation coordinates, vertex features, and generation information used to generate the interpolation Gaussian properties (such as the weights of the machine learning model) in the modeling information can be saved as modeling information, thereby reducing the data stream required to generate the reconstructed image and thus reducing the consumption of memory resources.

[0066] In one implementation, the aforementioned modeling information, initial modeling information, loss data, and reconstructed images can all be stored in the blockchain, preventing this information from being tampered with. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, it is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block.

[0067] It is understood that the image generation system shown in FIG2 described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0068] Based on the above image generation system, this application provides an image generation method. The image generation method described in this application can be executed by an electronic device, which can be the image generation device 201 in the image generation system shown in FIG2. Please refer to FIG3, which is a flowchart illustrating an image generation method provided in this application. The image generation method includes the following steps S301-S304:

[0069] S301. Obtain the vertex position, vertex features, and vertex Gaussian attributes of each graphic unit, and obtain the interpolation information of each graphic unit, which is used to represent the object surface.

[0070] In this embodiment, "object" refers to a 3D scene object used for modeling, such as the forklift shown in Figure 1 and the microphone shown in Figure 2. A graphical unit is a polygonal unit used for modeling, serving as the basic unit for constructing complex 3D models and representing the object's surface. In this application, a triangle is used as an example. These graphical units (such as triangles) do not share sides. A large set of graphical units is used to fit the object's surface, forming an explicit 3D structure. This embodiment uses a triangle as an example, but other shapes can also be used; this application does not limit the application to these. The graphical unit used to represent the object's surface refers to using a large set of triangles to fit the object's surface during modeling. As shown in Figures 1 and 2, the surfaces of the forklift and microphone are represented by a large set of triangles of different sizes and shapes (triangle soup). This method of representing objects with a set of triangles can be called an explicit 3D structure, i.e., structured triangular primitives.

[0071] It should be emphasized that the triangles in the triangle set are different from the triangle mesh. In the embodiments of this application, the triangles in the triangle set can be regarded as unrelated (discrete) triangles, that is, there are no common vertices and edges among the triangles.

[0072] Vertex position refers to the positional information of vertices in a graphics unit, such as the three-dimensional coordinates of the three vertices of a triangle. Vertex features are the characteristics of vertices in a graphics unit, usually represented in the form of feature vectors, and are associated with vertex positions. In this application, interpolation features of interpolation points can be obtained by interpolation calculation based on vertex features and interpolation information, and then interpolated Gaussian attributes can be generated based on the interpolation features and information from a specified viewpoint. Vertex Gaussian attributes can refer to the Gaussian distribution attributes of Gaussian splatter points (Gaussian points) set at the vertices of a graphics unit, such as scale, color, quaternion, and opacity attributes. It should be noted that in this embodiment, position is treated as a separate attribute, while scale, color, quaternion, and opacity attributes are explained as Gaussian distribution attributes.

[0073] In this embodiment, after modeling the object as discrete graphical units (such as triangles), Gaussian points are distributed within the graphical units, specifically at the vertices and inside the units. Points inside the graphical units can be considered as interpolation points obtained through vertex interpolation of the units; they can be understood as Gaussian points set at locations sampled within the units. It should be noted that in this embodiment, "vertex" refers to a Gaussian point set at the vertex of a graphical unit (such as a triangle), and "interpolation point" refers to a Gaussian point set at the location of the interpolation point inside the graphical unit (such as a triangle).

[0074] To reduce the amount of data stored for interpolation points, the image generation device can acquire interpolation information for each graphic unit. This interpolation information can be used to indicate the number of interpolation points in a graphic unit and the relationship between the interpolation points and vertices. The interpolation information for each graphic unit can be different. For example, the interpolation information can include at least one interpolation coordinate, with each interpolation coordinate corresponding to an interpolation point. Based on the interpolation information and vertex information (such as vertex position and vertex features), the image generation device can generate information related to the interpolation points (such as the position and features of the interpolation points). Therefore, after modeling, only the interpolation information needs to be stored, without needing to store the information related to the interpolation points, thus reducing memory resource consumption.

[0075] In one implementation, the image generation device can obtain modeling information (vertex positions, vertex features, vertex Gaussian properties, and interpolation information of each graphics unit) from local storage. The local storage of the image generation device can store modeling information of multiple objects, and the image generation device can obtain the modeling information of an object from local storage based on the object's identification information.

[0076] In another implementation, the image generating device can obtain modeling information of an object from the modeling device and store it in the local storage of the image generating device. For example, the modeling information can be used to generate modeling information corresponding to multiple objects, which are then stored in the modeling device. The image generating device can send a modeling information retrieval request to the modeling device to instruct it to retrieve the object's modeling information and receive the modeling information returned by the modeling device. Furthermore, the image generating device can store this information in its local storage.

[0077] In another implementation, after generating the modeling information for the object, the modeling device can progressively reduce the Gaussian points used to represent the object's surface, obtaining modeling information corresponding to multiple Levels of Detail (LODs). Specifically, after modeling is complete, the modeling device can determine the current modeling information as the modeling information corresponding to LOD0, which includes information on the most numerous Gaussian points. Furthermore, the modeling device can perform multiple downsampling processes on the interpolation points within each graphics unit to successively remove a portion of the interpolation points, and sequentially determine the modeling information obtained from each downsampling process as the modeling information corresponding to multiple levels of detail, such as LOD1, LOD2, etc. It is understandable that the interpolation information included in the modeling information at other levels of detail, such as LOD0, is different from that at LOD1 and LOD2.

[0078] Since the number of Gaussian points (interpolation points) gradually decreases under different levels of detail (LOD), the amount of data to be stored and the required memory resources under different LOD also gradually decrease. Therefore, the image generation device can obtain modeling information of an object at a specific level of detail from the modeling device based on its own memory resource constraints. For example, taking the forklift shown in Figure 1 as an example, the modeling information corresponding to LOD0 requires 3.5 Mega Bytes of memory resources, the modeling information corresponding to LOD1 requires 1.2 MB of memory resources, and the modeling information corresponding to LOD2 requires 0.7 MB of memory resources. If the image generation device is a resource-constrained edge device, it can obtain the modeling information of the object at a specific LOD based on its own available memory resources.

[0079] S302: Based on the above interpolation information, interpolate the above vertex positions and the above vertex features respectively in each graphic unit to obtain interpolation point positions and interpolation features in the above each graphic unit.

[0080] In the embodiments of the present application, the interpolation information may refer to at least one interpolation coordinate, which can be understood as the barycentric coordinate (u, v) of a graphic unit. Wherein, u and v are numbers greater than 0 and less than 1, and u+v is less than 1, that is, 0<u,v and u+v<1. It can be understood that each interpolation coordinate corresponds to one interpolation point, and the interpolation coordinates included in the interpolation information respectively corresponding to each graphic unit may be different. For example, the interpolation information of a larger-size graphic unit may include thousands of interpolation coordinates, while the interpolation information of a smaller-size graphic unit may only include 10 interpolation coordinates. Interpolation processing can be understood as densely inserting positions inside the graphic unit corresponding to the interpolation information based on the interpolation information, and setting Gaussian points at the inserted positions. It can be understood that for a given graphic unit, any point inside it can be represented by a set of weights (u, v). For example, taking a triangle as the graphic unit, the representation of the interpolation point P inside it can be shown in Formula 3: {G(P uv )|P uv =uP1+vP2+(1-u-v)P3} Formula 3

[0081] In Formula 3, G(P uv ) represents the Gaussian point set at the interpolation point P, whose interpolation coordinate is uv, P uv represents the interpolation point P, P1, P2 and P3 can represent the vertices of the graphic unit (triangle), u and v are the weights in the barycentric coordinates. For example, u can be understood as the weight of the interpolation point P relative to the vertex P1, v can be understood as the weight of the interpolation point P relative to the vertex P2, and (1-u-v) can be understood as the weight of the interpolation point P relative to the vertex P3.

[0082] In other words, interpolation of vertex positions and features is performed based on the weights indicated by the interpolation coordinates, with interpolation calculations performed separately for each vertex position and feature. The interpolation point position within each graphics unit refers to the position of each interpolation point included in each graphics unit, represented by three-dimensional position coordinates. The interpolation point position is the position obtained within a graphics unit after interpolating the vertex positions, expressed in three-dimensional position coordinates. Gaussian points are set at these positions, calculated based on vertex positions and interpolation information (such as interpolation coordinates). In some cases, position offsets can also be combined to determine the final position of the interpolation point. Interpolation features refer to the features corresponding to multiple interpolation points included in each graphics unit, i.e., feature vectors (vector representation). Interpolation features are the features corresponding to the interpolation points in each graphics unit, represented in the form of feature vectors. Obtained by interpolating the vertex features and interpolation information (such as interpolation coordinates) of the graphics unit, they can be used to generate interpolated Gaussian attributes by combining information from a specified viewpoint.

[0083] Specifically, the process by which the image generation device interpolates vertex positions and vertex features within each graphic unit based on at least one interpolation coordinate included in the interpolation information to obtain interpolation point positions and interpolation features within each graphic unit can be as follows: the image generation device performs interpolation calculation on vertex positions based on the interpolation coordinates of each graphic unit to obtain at least one interpolation point position in each graphic unit, and performs interpolation calculation on vertex features based on the interpolation coordinates of each graphic unit to obtain interpolation features of at least one interpolation point position.

[0084] The interpolation coordinates of each graphic unit are at least one interpolation coordinate included in the interpolation information of that graphic unit. The position of at least one interpolation point can refer to the position of at least one interpolation point included in each graphic unit, with each interpolation point's position corresponding to the interpolation coordinates in the interpolation information of its respective graphic unit. The interpolation feature of at least one interpolation point position refers to the feature vector of the Gaussian point set at that position; it can also be understood as the feature vector of the interpolation points included in each graphic unit, with each interpolation point's feature vector also corresponding to the interpolation coordinates in the interpolation information of its respective graphic unit.

[0085] Taking a triangle as an example, the image generation device can perform interpolation calculations on the three-dimensional position coordinates (such as P1, P2, and P3 in Formula 3) of the three vertices of the triangle based on the interpolation coordinates (u, v) in the interpolation information corresponding to each triangle, according to the calculation method shown in Formula 3. For example, after multiplying the weights (u, v, and 1-uv) with the three-dimensional coordinates (x, y, and z coordinates) of the corresponding vertex respectively, the results are summed in the x, y, and z coordinate dimensions to obtain at least one interpolation point position in each triangle.

[0086] Similarly, taking a triangle as an example, the image generation device can perform interpolation calculations on the feature vectors of the three vertices of the triangle based on the interpolation coordinates in the interpolation information corresponding to each triangle. The specific calculation method is shown in Formula 4: f uv =uf1 + vf2 + (1 - uv)f3 Formula 4

[0087] Similar to Equation 3, in Equation 4, f uv The feature vector representing the interpolation point P can be denoted by uv, and f1, f2, and f3 can represent the vertex features of the graphic unit (triangle). Therefore, the image generation device can calculate the interpolation features of at least one interpolation point position based on the calculation method in Formula 4.

[0088] In some embodiments, since interpolation points are obtained through interpolation within a graphic unit, their positions can be calculated based on their interpolation coordinates and vertex positions (e.g., the positions of the three vertices of a triangle). Consequently, the calculated interpolation points all lie within the plane containing the graphic unit (e.g., the triangle), meaning they cannot represent information outside that plane, thus limiting their expressive power in three-dimensional space. To improve the expressive power of planar triangles, a position offset (delta p) can be generated for each interpolation point. This offset can be a local micro-displacement indicated by three-dimensional coordinates, representing the offset of the interpolation point in a direction outside the plane of its graphic unit, such as a position offset perpendicular to the plane of the graphic unit.

[0089] In other words, the modeling information acquired by the image generation device can include not only the vertex positions, vertex features, vertex Gaussian properties, and interpolation information of each graphic unit, but also the position offsets of the interpolation points within each graphic unit. After calculating the position of the interpolation point using the method shown in Formula 3, the image generation device can sum this position offset to obtain the final position of the interpolation point. Specifically, the position of the interpolation point in each graphic unit can be represented as: P uv ′=P uv +ΔP uv Among them, P uv ′ represents the final position of the interpolation point P, P uv ΔP is the position of the interpolation point calculated based on Formula 3. uv This is the position offset corresponding to the interpolation point P.

[0090] S303. Based on the above interpolation features and the information of the specified viewpoint, generate the interpolation Gaussian attributes of each of the above graphic units.

[0091] In this embodiment, the interpolation feature refers to the feature vector of the interpolation point in each graphics unit, with each interpolation point corresponding to a feature vector. The specified viewpoint can be understood as the angle and distance of the object relative to the observer. Information indicating the specified viewpoint can refer to information used to indicate the viewpoint, such as the viewpoint direction parameter, which can be the viewpoint direction parameter corresponding to each interpolation point and can be calculated by providing camera position parameters. The camera position parameters can be user-inputted; that is, if the user wants to render an image of an object from a certain viewpoint, they only need to provide the camera position parameters for that viewpoint. Specifically, the calculation method for the viewpoint direction parameter is shown in Formula 5:

[0092] In Formula 5, d uv This represents the viewpoint direction parameter of interpolation point P, with its interpolation coordinates being uv. uv The position of the interpolation point P can be calculated using Formula 3 above, and X0 represents the given camera position. ||·||2 represents the Euclidean norm, also known as the L2 norm. Therefore, based on the camera position X0, the viewpoint direction parameters corresponding to each interpolation point can be calculated separately, thus obtaining the information for the specified viewpoint.

[0093] The interpolation Gaussian attribute of each graphics unit refers to the Gaussian distribution attribute of the interpolation points within that unit. Each interpolation point corresponds to a Gaussian distribution attribute, which may include scale s, color c, quaternion q, and opacity α. It can be understood that in three-dimensional space, the visual effects of lighting, color, etc., presented by an object relative to the observer differ depending on the angle and distance. This is reflected in the interpolation Gaussian attribute. Therefore, the interpolation Gaussian attribute of each interpolation point can be generated based on the interpolation characteristics and viewpoint direction parameters of each interpolation point within the graphics unit.

[0094] In one implementation, the modeling information may further include the mapping relationship between interpolation features and information from a specified viewpoint, and the interpolated Gaussian attributes, such as a functional relationship. The image generation device can determine the mapped interpolated Gaussian attributes based on the interpolation features of each interpolation point and information from a specified viewpoint (viewpoint direction parameters), namely, the scale s, color c, quaternion q, and opacity α of that interpolation point. The image generation device can then use the interpolated Gaussian attributes of the interpolation points in each graphic unit as the interpolated Gaussian attributes of the generated graphic units.

[0095] In another implementation, the modeling information can include mapping relationships between interpolation features and information from a specified viewpoint, and different attributes. For example, there could be a first mapping relationship between interpolation features and a specified viewpoint with a scale attribute, and a second mapping relationship between interpolation features and a specified viewpoint with a color attribute. The image generation device can determine the mapped scale, color, quaternion, and opacity based on the interpolation features at each interpolation point and the information from the specified viewpoint (viewpoint direction parameter), and use the scale, color, quaternion, and opacity of the interpolation points in each graphic unit as the interpolation Gaussian attributes of the generated graphic units.

[0096] Specifically, this mapping relationship can be implemented using a machine learning model. The interpolation features of the interpolation points in each graphic unit and the information of the specified viewpoint (viewpoint direction parameter) can refer to the input of the machine learning model, while the Gaussian distribution attribute of the interpolation points can refer to the output predicted by the machine learning model. This machine learning model learns this mapping relationship through training. It should be noted that the modeling information acquired by the image generation device can also include the model parameters of the machine learning model, such as the weights of each network layer in the machine learning model. Thus, the image generation device can predict the interpolation Gaussian attribute of each interpolation point based on the machine learning model, thereby obtaining the interpolation Gaussian attribute of each graphic unit.

[0097] The aforementioned machine learning model can be a Multilayer Perceptron (MLP), a feedforward artificial neural network model composed of multiple layers of nodes (or neurons), including an input layer, one or more hidden layers, and an output layer. Each neuron is connected to all neurons in the next layer via weights and applies an activation function to process its input to produce an output. The input layer receives external data as input, with each node corresponding to one dimension of the input features. The hidden layers extract features from the input and perform nonlinear transformations. The output layer generates the corresponding prediction result. For example, the MLP in this embodiment can consist of two hidden layers, each with 64 dimensions, or it can have 128 dimensions; this application does not limit this. In this application, the MLP predicts the Gaussian distribution properties of the Gaussian points set at the interpolation points; these Gaussian points can be called Neural Gaussian Splats.

[0098] In this embodiment, multiple MLPs can be used to represent the mapping relationship between interpolation features and information from a specified viewpoint and multiple attributes. These MLPs can be represented as MLPs. c MLP α MLP s MLP qThese MLPs can be called triangle MLPs. For example, the Gaussian distribution properties of the interpolation points can be calculated as shown in Equations 6-9: c uv =Sigmoid(MLP) c (f uv ,d uv Formula 6 α uv =Sigmoid(MLP) α (f uv ,d uv Formula 7 s uv =Sigmoid(MLP) s (f uv ,d uv )) Formula 8 q uv =Sigmoid(MLP) q (f uv ,d uv )) Formula 9

[0099] In formulas 6-9, c uv α uv s uv and q uv These represent the Gaussian distribution attributes of a specific interpolation point (such as interpolation point P above) within a given graphic unit. These attributes include scale, color, opacity, and quaternion values. The interpolation coordinates of this point are uv. MLP c MLP α MLP s MLP q These represent different machine learning models, each corresponding to a different color c, opacity α, scale s, and quaternion q. uv ,d uv For multiple MLP inputs, f uv The interpolation characteristics of the interpolation point P can be calculated based on Equation 4, d uv The viewpoint direction parameter of the interpolation point P can be calculated based on Formula 5. Sigmoid is an activation function. Optionally, different activation functions can be used for different properties, and their calculation method can be similar to the above formula. This application does not limit this.

[0100] Therefore, in this embodiment, by implicitly modeling the Gaussian distribution properties (such as color c, opacity α, scale s, and quaternion q) of interpolation points (Gaussian points) within graphics units using a machine learning model, redundant explicit Gaussian distribution properties, such as high-dimensional spherical harmonic coefficients, can be eliminated. Only the model parameters of the machine learning model and the vertex features of the graphics units are stored, which can reduce the memory requirements for implementing 3DGS to a certain extent. Furthermore, the input of the machine learning model includes information about the specified viewpoint (viewpoint direction parameter), making the interpolated Gaussian properties viewpoint dependent, enhancing the expressive power of the interpolation points (Gaussian points), and making the generated images more realistic.

[0101] S304. Based on the vertex position and the vertex Gaussian property, as well as the interpolation point position and the interpolation Gaussian property, a rendering process is performed to obtain the reconstructed image of the object from the specified viewpoint.

[0102] In this embodiment, vertex position and vertex Gaussian attribute, as well as interpolation point position and interpolation Gaussian attribute, can be understood as the position and Gaussian distribution attributes of vertices and interpolation points in each graphic unit (such as a triangle). In 3DGS, rendering processing can refer to the process of "splashing" Gaussian points onto a 2D image plane, which involves projection and color calculation. Reconstructed image refers to a two-dimensional image of an object from a specified viewpoint, that is, an image rendered based on the modeled Gaussian points.

[0103] Specifically, the image generation device performs rendering processing based on vertex positions and vertex Gaussian properties, as well as interpolation point positions and interpolation Gaussian properties, to obtain a reconstructed image of the object from a specified viewpoint. Specifically, the image generation device first projects the vertices of each graphics unit onto the image plane based on the vertex Gaussian properties and vertex positions to obtain the first pixel, and then projects the interpolation points within each graphics unit onto the image plane based on the interpolation Gaussian properties and interpolation point positions to obtain the second pixel. The vertex Gaussian properties include the opacity and color of the vertices, and the interpolation Gaussian properties include the opacity and color of the interpolation points. Furthermore, the image generation device can calculate the color of the first pixel based on the vertex color and opacity, and calculate the color of the second pixel based on the interpolation point color and opacity. Finally, it can generate a reconstructed image based on the colors of the first and second pixels.

[0104] Projecting vertices and interpolation points onto the image plane can refer to projecting Gaussian points (three-dimensional Gaussian distribution) onto a 2D plane, i.e., the image plane, based on the Gaussian distribution properties of the Gaussian points. This yields the projected center coordinates and 2D covariance matrix, which can be called a two-dimensional Gaussian distribution. After projection, the two-dimensional Gaussian distribution corresponds to an image region in the image plane. This can be analogized to "flattening" the Gaussian points (i.e., imagining them as "ellipsoids") onto the two-dimensional image plane according to their position and shape (position and Gaussian distribution properties), resulting in an elliptical image region. Pixels within this image region can be called the pixels covered by the Gaussian point, representing the pixels influenced by that Gaussian point.

[0105] In other words, based on the Gaussian properties of the vertices and their positions, the vertices (Gaussian points set at the vertices) of each graphics unit can be projected onto a 2D image plane. The pixels covered by this projection can be called the first pixel. Based on the Gaussian properties of the interpolation and the positions of the interpolation points, the interpolation points (Gaussian points set at the interpolation points) within each graphics unit are projected onto the 2D image plane. The pixels covered by this projection can be called the second pixel. The image plane can include both the first and second pixels. The image area covered by the projected Gaussian points can be calculated using their projected center coordinates and the 2D covariance matrix. It is understandable that a pixel can exist within the image area covered by multiple projected Gaussian points; that is, the first pixel can overlap with the second pixel.

[0106] Furthermore, taking a single pixel as an example, the image generation device can identify all Gaussian points affecting that pixel—that is, all Gaussian points in the image region covering that pixel. The image generation device can then sort all Gaussian points affecting that pixel based on the depth information before projection of each Gaussian point, for example, sorting them from front to back. This is because objects closer to the observer will occlude objects farther away, and sorting based on the depth of the Gaussian points in 3D space (i.e., their distance from the camera) can simulate this effect. It is understandable that the earlier Gaussian points are, the greater their influence on the pixel's color. Therefore, based on this sorting result, the rendering order of all Gaussian points affecting that pixel can be determined, allowing for rendering processing based on this sorting result.

[0107] Specifically, the Gaussian distribution attributes of Gaussian points include color and opacity attributes, corresponding to values ​​between 0 and 1. That is, the vertex Gaussian attribute includes the color and opacity values ​​of the vertices of each graphic unit, and the interpolation Gaussian attribute includes the color and opacity values ​​of the interpolation points within each graphic unit. The image generation device can perform alpha blending (i.e., α-blending) on ​​the Gaussian points sequentially based on the sorting results to obtain the color of that pixel. The specific method for alpha blending can be found in Equation 2.

[0108] Understandably, the color of the first pixel can be calculated based on the vertex color and opacity, and the color of the second pixel can be calculated based on the interpolation point color and opacity. The color of each pixel is calculated based on one or more Gaussian points covering it. After obtaining the colors of the first and second pixels, a reconstructed image of the object at a specified viewpoint can be generated based on these colors. This reconstructed image includes the first and second pixels, and their colors are the result of the rendering process.

[0109] Please refer to Figure 4, which is a schematic diagram of a 3DGS inference process provided in an embodiment of this application. As shown in Figure 4, taking a microphone as an example, the image generation device can obtain the vertex positions, vertex features, vertex Gaussian properties, and interpolation information corresponding to each triangle of the set of triangles representing the microphone surface. Furthermore, the image generation device can perform interpolation calculations based on the vertex positions and interpolation information to obtain the positions of the interpolation points, such as P. uv As shown. Optionally, the position P of the interpolation point can be... uv With position offset ΔP uv The sum of these values ​​serves as the final position of the interpolation point. Similarly, the image generation device can perform interpolation calculations based on vertex features and interpolation information to obtain the interpolation features of the interpolation point, such as f. uv As shown. Furthermore, the interpolation feature f uv The viewpoint orientation parameter d, calculated from the camera position X0 for a specified viewpoint, can be input into a machine learning model to predict the values ​​of the interpolation point's scale s, color c, quaternion q, and opacity α. This machine learning model can be a triangular MLP. Therefore, rendering processing (α-mixing) can be performed based on the Gaussian distribution properties of each Gaussian point (vertices and interpolation points) to obtain the reconstructed image of the microphone at the specified viewpoint.

[0110] In some embodiments of this application, when generating an image of an object from a specified viewpoint based on 3D Gaussian splashing technology, the object surface is represented by Gaussian points in the graphics unit. Compared to directly using Gaussian points, this gives the Gaussian points representing the object structured information, which can reduce visual artifacts at edges to a certain extent and improve the quality of the generated reconstructed image. Furthermore, by using the relationship between graphics unit vertices and interpolation points, and implicitly modeling the Gaussian distribution properties of the interpolation points, it is possible to avoid storing the Gaussian distribution properties of all Gaussian points. This reduces memory consumption while maintaining image quality, improving memory resource utilization efficiency and increasing the likelihood of deploying this technology in memory-constrained scenarios.

[0111] This application also provides an image generation method. The image generation method described in this application can be executed by an electronic device, which can be the modeling device 202 or the image generation device 201 shown in FIG2. Please refer to FIG5, which is another schematic flowchart of another image generation method provided in this application. This image generation method includes the following steps S501-S503:

[0112] S501. Obtain the initial vertex position, initial vertex Gaussian attribute, initial interpolation point position and initial interpolation Gaussian attribute of each initial graphic unit representing the surface of the object, and perform rendering processing based on the initial vertex position and initial vertex Gaussian attribute, as well as the initial interpolation point position and initial interpolation Gaussian attribute, to obtain the first training image of the object from the reference viewpoint.

[0113] In this embodiment, the modeling device can optimize the graphical units representing the object surface and the Gaussian points distributed within the graphical units through a two-stage training process. The first stage can be used to optimize the graphical units so that they can better fit the geometry of the object. The second stage can be used to optimize the Gaussian points, specifically to optimize the interpolation points (Gaussian points set at the interpolation points) in the graphical units, that is, to optimize the viewpoint-dependent interpolation Gaussian properties, thereby learning local details to achieve high-quality image reconstruction.

[0114] The first phase of the training process will now be described.

[0115] Specifically, the initial graphic unit can refer to the graphic unit initially used to fit the surface of the object, such as the initial triangle, whose shape and size have not yet been adjusted; for example, it could be sparse small triangles. Taking a triangle as the initial graphic unit, the initial vertex position can refer to the initial three-dimensional position coordinates of the three vertices of the triangle, and the initial vertex Gaussian attribute can refer to the Gaussian distribution attribute of the Gaussian points set at the three vertices of the triangle, such as the values ​​of scale s, color c, quaternion q, and opacity α. The initial interpolation point position can refer to the insertion position inside the triangle. The inserted position can be a Gaussian point, which can be called an interpolation point. The initial interpolation point position can include multiple positions inserted inside the triangle. The initial interpolation Gaussian attribute can refer to the Gaussian distribution attribute of the Gaussian points of the interpolation points inside the triangle, such as the values ​​of scale s, color c, quaternion q, and opacity α.

[0116] Furthermore, the modeling device can perform rendering processing based on the initial vertex positions, initial vertex Gaussian properties, initial interpolation point positions, and initial interpolation Gaussian properties to obtain the first training image of the object from the reference viewpoint. It should be noted that during the first stage of training, the reference viewpoint can be a viewpoint specified during training, and can be the same as or different from the specified viewpoint; this application does not impose any limitations on this. The first training image, obtained after rendering processing based on the initial vertex positions, initial vertex Gaussian properties, initial interpolation point positions, and initial interpolation Gaussian properties during the first stage of training, is an image of the object from the reference viewpoint. It can be analogous to the predicted value in machine learning methods and is used to compare with the reference image to generate the first loss data.

[0117] The rendering process based on the position and Gaussian distribution properties of Gaussian points can be found in the specific implementation described in step S304. Specifically, after completing the first stage of training and obtaining the vertex positions, vertex Gaussian properties, interpolation point positions, and interpolation Gaussian properties of the initial graphics units after iterative adjustments, the image generation device uses this information to generate a reconstructed image. First, the image generation device determines the specific position of each graphics unit vertex in three-dimensional space based on the vertex position. Combining the opacity and color information in the vertex Gaussian properties, and using the camera viewpoint as a reference, the vertex is projected onto a two-dimensional image plane. During the projection process, with the position of each vertex as the center, the shape and direction of the Gaussian distribution are determined according to the scale and quaternions in its Gaussian distribution properties. The three-dimensional Gaussian distribution is projected onto the two-dimensional plane to form the corresponding two-dimensional Gaussian distribution. The pixels covered after projection are the first pixels. For each first pixel, the image generation device considers that if multiple vertices project onto the same pixel, the preceding vertices will occlude the light from the following vertices. Following the order from near to far, it calculates the light attenuation based on the opacity of each vertex, and then combines this with its color to determine the pixel's color using an α-mixing method. For interpolation points, the image generation device determines their position in 3D space based on their location. Combining the opacity and color information from the interpolation Gaussian properties, and again using the camera's viewpoint as a reference, it projects the interpolation points onto the 2D image plane to obtain the second pixel covered by the projection. Similar to vertex processing, the image generation device considers the occlusion of light by preceding interpolation points or vertices during propagation, and calculates the color of the second pixel using an α-mixing method based on the interpolation point's opacity and color. Finally, the image generation device combines the colors of all first and second pixels. Since each pixel on the image plane may be affected by multiple vertices and interpolation points, it is necessary to finally determine the color of each pixel to generate a reconstructed image of the object from a specified viewpoint.

[0118] In one possible implementation, the process by which the modeling device acquires the initial vertex positions, initial vertex Gaussian properties, initial interpolation point positions, and initial interpolation Gaussian properties of each initial graphic unit representing the object surface can be as follows: First, acquire a set of feature points representing the object surface, and determine the initial vertex positions based on the positions of each feature point in the feature point set to form each initial graphic unit. Then, the modeling device can perform interpolation processing on the initial vertex positions within each initial graphic unit based on set interpolation information to obtain the initial interpolation point positions. Afterward, the modeling device can assign initial Gaussian distribution properties to the vertices and interpolation points in each initial graphic unit, respectively, to obtain the initial vertex Gaussian properties and the initial interpolation Gaussian properties.

[0119] Specifically, a set of feature points representing the surface of the object can be obtained, and each feature point is used as a vertex of a graphic unit. Random perturbations are added to the vertices of the graphic units to prevent overlapping between vertices. Based on the positions of each feature point after random perturbation, the vertex positions constituting each initial graphic unit are obtained, thereby determining the initial vertex positions to form each initial graphic unit. During interpolation processing, the interpolation information can be set to include the interpolation coordinates of K interpolation points uniformly interpolated in the initial graphic units. Based on this, according to the formula K = N... 2 The number of interpolation points is calculated using -3 (where K represents the number of interpolation points and N represents the frequency of interpolation points). Based on the positions of the three vertices of the triangle, the values ​​of the centroid coordinates u and v are determined to obtain the interpolation coordinates. The initial vertex positions are interpolated according to the formula P = uV1 + vV2 + (1 - uv)V3 (where P represents the Gaussian point set at the interpolation point P, V1, V2 and V3 represent the vertices of the graphic unit (triangle), and u and v are the weights in the centroid coordinates) to obtain the initial interpolation point positions.

[0120] The feature point set used to represent the object's surface can include multiple feature points. These can be feature points obtained through uniform sampling in three-dimensional space, or they can be a preliminary representation of a 3D scene reconstructed from multiple 2D images using Structure from Motion (SFM) technology. For example, COLMAP software can analyze feature points in 2D images using SFM technology to infer the camera's pose (position and orientation) and the 3D structure of the scene, thereby generating a sparse point cloud. The sparse point cloud includes a feature point set, where the number of feature points is relatively small and the spacing between points is relatively large. The feature points in the feature point set can be a predetermined number of feature points extracted from the sparse point cloud, such as 1000 feature points. Each feature point carries position information and may also carry color information.

[0121] Furthermore, the modeling device can use each feature point as a vertex of a graphical unit. For example, the three vertices of a triangle can overlap, which can be the position of each feature point. The modeling device can add random perturbations to the vertices of the graphical units to prevent the positions of the vertices from overlapping. Thus, based on the positions of each feature point through random perturbation, the vertex positions constituting each initial graphical unit can be obtained, resulting in the initial vertex positions. This process can be called the initialization of graphical units (initialization of triangle primitives), which can produce sparse small triangles.

[0122] The interpolation information can refer to the initial interpolation coordinates and the number of Gaussian points. For example, the interpolation information can include the interpolation coordinates of K interpolation points uniformly interpolated in the initial graphic unit. Based on this, the modeling device can perform interpolation processing on the initial vertex positions according to the interpolation information to determine the K interpolation positions and obtain the initial interpolation point positions.

[0123] Please refer to Figure 6, which is a schematic diagram illustrating the principle of interpolation processing within a graphic unit according to an embodiment of this application. As shown in Figure 6, taking a triangle as an example, the vertices of the triangle are represented by black dots, and the interpolation points inside the triangle are represented by gray dots. u and v can represent the centroid coordinates of the triangle, which are between 0 and 1, i.e., the interpolation coordinates, denoted as (u, v). Figure 6 uses K as an example for explanation. The modeling device can perform uniform interpolation processing inside the triangle, as shown in Figure 6, where the gray dots are evenly distributed inside the triangle. The formula for calculating the number of interpolation points is as shown in Formula 10:

[0124] In Formula 10, Interp.Num. represents the number of interpolation points, i.e., K and N represent the frequency of the interpolation points. For example, as shown in Figure 6, the given frequency is 4. Subtracting 3 means removing the three vertices of the triangle. If the graphic unit is a polygon, the corresponding number of vertices can be subtracted. For example, given N=4, the number of interpolation points can be calculated to be 12. Since the interpolation points are obtained by uniform interpolation, the values ​​of u and v can be determined based on the positions of the three vertices of the triangle, thus obtaining the interpolation coordinates. It can be understood that the position of each interpolation point in the above initial interpolation point positions can be set with a Gaussian point. The center position of the three-dimensional Gaussian distribution of the Gaussian point is the position obtained by uniform interpolation. The position of each interpolation point can be calculated according to Formula 3 based on the position of the vertex of its initial graphic unit and the interpolation coordinates.

[0125] After determining the interpolation points in each initial graphic unit, the modeling device can assign initial Gaussian distribution attributes to the vertices and interpolation points in each initial graphic unit. The initial Gaussian distribution attributes of the Gaussian points at the vertices in each initial graphic unit can be called initial vertex Gaussian attributes, and the initial Gaussian distribution attributes of the Gaussian points at the interpolation points in each initial graphic unit can be called initial interpolation Gaussian attributes. Specifically, the Gaussian distribution attributes can include the values ​​of scale s, color c, quaternion q, and opacity α.

[0126] In some embodiments, the initial Gaussian distribution attribute assigned to the vertices in each initial graphics unit can be a default value. When assigning the initial Gaussian distribution attribute to the interpolation points in each initial graphics unit, the value of the opacity α attribute can be calculated based on the opacity α attribute value of the vertices in the initial graphics unit. For example, it can be calculated according to Formula 3 based on the opacity values ​​of the three vertices of the triangle and the interpolation coordinates, i.e., α. uv = uα1 + vα2 + (1 - uv)α3, α uv α1, α2, and α3 are the opacity values ​​of an interpolation point in the initial graphic unit, with interpolation coordinates uv. α1, α2, and α3 are the opacity values ​​of the three vertices of the triangle.

[0127] S502. Based on the difference between the first training image and the reference image of the object under the reference viewpoint, generate first loss data.

[0128] A reference image is an image of an object from a reference perspective, analogous to the true value in machine learning methods. It is used to compare with training images (such as the first training image and the second training image) to generate corresponding loss data (such as the first loss data and the second loss data), thereby iteratively adjusting the relevant parameters.

[0129] In this embodiment, to fit the object surface, loss data can be generated using machine learning methods to iteratively optimize the graphical units, causing changes in their shape and position to represent the object's surface. The first training image can be an image generated during training, analogous to a predicted value in a machine learning method. The reference image can be analogous to the true value in a machine learning method, referring to an image of the object from a reference viewpoint, which can be preset. The first loss data can be generated based on the difference between the first training image and the reference image, used to measure the error between them. Therefore, the initial vertex position, initial vertex Gaussian attribute, initial interpolation point position, and initial interpolation Gaussian attribute can be iteratively adjusted based on the first loss data.

[0130] Specifically, the first loss data may include absolute error (L1) loss and structural similarity index Measure (SSIM) loss. L1 loss can be generated based on the sum of the absolute differences of each pixel value between the first training image and the reference image, while SSIM loss can be generated based on the differences in brightness, contrast and structural information between the first training image and the reference image, which can reflect the human visual system's perception of image quality.

[0131] In some embodiments, since the first loss data can be used to iteratively adjust the Gaussian distribution properties of each Gaussian point (vertices and interpolation points), including a scale attribute that describes the size of the Gaussian point, a scale regularization term can be included in the first loss data to penalize large-scale Gaussian points in order to prevent the Gaussian points from becoming too large and to promote them becoming larger graphical units. Specifically, the first loss data can be as shown in Equation 11: L stage1 =L1+λ ssim L SSIM +λ reg L reg Formula 11

[0132] In Formula 11, L stage1 L1 represents the training loss in the first stage, i.e., the first loss data, and L1 represents the L1 loss between the first training image and the reference image. SSIM L represents the SSIM loss between the first training image and the reference image. reg λ represents the scaling regularization term. ssim and λ reg For the weighting term, it can be set to 0.2 and 0.1 respectively, and this application does not limit it in this way. Wherein, L reg For details, please refer to Formula 12:

[0133] In Formula 12, N g s is the number of Gaussian points. n The scale of the nth Gaussian point is max(s). n ) represents the maximum scale of the nth Gaussian point. This can be understood as the scale attribute including the scale in the x, y, and z directions. max(s) n The maximum scale for each Gaussian point, for example, could be the scale in the x-direction.

[0134] S503. Based on the first loss data, the initial vertex position, the initial vertex Gaussian attribute, the initial interpolation point position, and the initial interpolation Gaussian attribute are iteratively adjusted, and the vertex position, vertex Gaussian attribute, interpolation point position, and interpolation Gaussian attribute of the initial graphic unit obtained by the iterative adjustment are used to generate the reconstructed image.

[0135] In this embodiment, the initial vertex position, initial vertex Gaussian attribute, initial interpolation point position, and initial interpolation Gaussian attribute are all learnable features. The first loss data is used to iteratively adjust the initial vertex position, initial vertex Gaussian attribute, initial interpolation point position, and initial interpolation Gaussian attribute. For example, the position gradient of each Gaussian point can be calculated based on the first loss data, and the position of each Gaussian point can be adjusted based on the position gradient to adjust the initial vertex position and initial interpolation point position. As another example, the scale gradient of each Gaussian point can be calculated based on the first loss data, and the scale value of each Gaussian point can be adjusted based on the scale gradient to adjust the initial vertex Gaussian attribute and initial interpolation Gaussian attribute. The color attribute and quaternion attribute of the Gaussian point are similarly adjusted. It is understood that after adjusting the vertex position and the interpolation point position, the interpolation coordinates of the interpolation point can be updated simultaneously.

[0136] In some embodiments, during the first stage of training, since the opacity value of the Gaussian point set at the interpolation point inside each initial graphics unit is obtained by interpolation calculation based on the opacity value of the vertex, the opacity gradient of the Gaussian point set at the vertex of each initial graphics unit can be calculated based on the first loss data to adjust the opacity value of the Gaussian point at each vertex. Then, based on the adjusted opacity value and the interpolation coordinates, the opacity value of the Gaussian point set at the interpolation point inside each initial graphics unit can be updated.

[0137] In this training phase, the number of iterations can be 3000, meaning the training can end when the iteration threshold is reached. Specifically, the modeling device can adjust the initial vertex positions, initial vertex Gaussian attributes, initial interpolation point positions, and initial interpolation Gaussian attributes based on the first loss data. It then regenerates a training image based on these adjusted initial vertex positions, Gaussian attributes, interpolation point positions, and interpolation Gaussian attributes. Furthermore, it generates another loss based on the difference between the training image and the reference image to further adjust these initial vertex positions, Gaussian attributes, interpolation point positions, and interpolation Gaussian attributes, repeating this process until the iteration threshold (such as 3000 iterations) is reached. At this point, the vertex positions, Gaussian attributes, interpolation point positions, and interpolation Gaussian attributes of the initial graphical units obtained through iterative adjustments can be used as modeling information to generate reconstructed images of the object from any viewpoint.

[0138] In one possible implementation, since it is difficult to select a suitable number of graphic units for different objects during the initialization of graphic units, the modeling device can dynamically adjust the graphic units according to the complexity of the object and the set number of iterations. Specifically, this can include the pruning and growth of graphic units. This process can be called adaptive control of graphic units (such as adaptive control of triangles).

[0139] Specifically, during the iterative adjustment process, the modeling device can acquire the Gaussian attributes and positions of the first vertex after a set number of iterations. The set number of iterations can be 100 or 500, such as adjusting a graphical unit after every 100 iterations. The Gaussian attributes and positions of the first vertex are obtained after each set number of iterations (e.g., every 100 iterations). The Gaussian attributes include the opacity of vertices in each initial graphical unit, and the positions include the positional information of the vertices in each initial graphical unit, i.e., their 3D coordinates. Furthermore, the modeling device can perform trimming processing on the initial graphical units according to trimming rules.

[0140] In one implementation, if the modeling device determines that there exists a first initial graphic unit where the opacity of all vertices is lower than a set opacity threshold, the modeling device can delete the first initial graphic unit from the initial graphic units representing the object surface. For example, when the opacity value of the Gaussian point at all vertices of a triangle is lower than the set opacity threshold τ... α If the triangle has too low opacity, it will have a negligible impact on the color of the pixels after projection. Deleting such triangles helps maintain high computational performance.

[0141] In another implementation, if the modeling device determines that the second initial graphic unit contains a specified vertex with an opacity lower than a set opacity threshold, the modeling device can update the position information of the specified vertex based on the position information of the vertices in the second initial graphic unit with opacities higher than the set opacity threshold. Taking a triangle as an example, if the second initial graphic unit contains vertices with opacities higher than the set opacity threshold τ... α The vertices also exist that are below the set opacity threshold τ. α The vertex of the triangle, which includes two cases: there exists a vertex in the triangle whose opacity is below the set opacity threshold τ. α The vertex, and the existence of two points in the triangle below the set opacity threshold τ. α The vertex.

[0142] Please also refer to Figure 7, which is a schematic diagram of the principle of a trimmed graphic unit provided in an embodiment of this application. As shown in Figure 7, taking the initial graphic unit as a triangle as an example, the dark ellipse represents the Gaussian point with high opacity, that is, the opacity is higher than the set opacity threshold τ. α The vertex, a light-colored ellipse, represents a Gaussian point with low opacity, i.e., an opacity below the set opacity threshold τ. α The vertex. As shown in Figure 7(a), there exists a vertex in the triangle that is below the set opacity threshold τ. α The vertex can then be moved to a location with an opacity higher than the set opacity threshold τ.α The vertex is moved, for example, to the midpoint of the edge between any high-opacity Gaussian point, as in (a) to the midpoint of the left edge. As shown in (b) of Figure 7, there are two points in the triangle below the set opacity threshold τ. α The vertex can then be moved to a location with an opacity higher than the set opacity threshold τ. α The vertices are moved, as in (b), to the midpoint of the edge between the high-opacity Gaussian point and the edge.

[0143] Therefore, the modeling device can update the position information of a specified vertex based on the position information of vertices in the second initial graphic unit whose opacity is higher than a set opacity threshold. It is understandable that after the position information of the specified vertex in the second initial graphic unit is updated, the position information of the interpolation point will also be updated, as it is calculated based on the position information of the three vertices of the triangle and the interpolation coordinates.

[0144] In some embodiments, after updating the position information of a specified vertex, the opacity value of that specified vertex can be updated simultaneously. For example, the modeling device can determine the average opacity value of the specified vertex (the low-opacity Gaussian point) as the updated opacity value of the specified vertex by comparing the original opacity value of the specified vertex (the low-opacity Gaussian point) with the opacity value of the high-opacity Gaussian point indicated by the movement direction. For example, as shown in Figure 7(a), the average opacity value between the original opacity values ​​of the lower left vertex of the triangle and the low-opacity Gaussian point can be calculated and used as the updated opacity value of the specified vertex. Further iterative adjustments can then be made based on the updated opacity value and the position information.

[0145] In another implementation, the Gaussian properties of the first vertex after a set number of iterations (e.g., 100) also include the scales of each vertex in the initial graphic unit in multiple directions. These multiple dimensions can refer to the scale *s* in the x, y, and z directions. The modeling device can determine the maximum scale corresponding to each vertex based on its scales in multiple directions, and determine the perimeter of each initial graphic unit based on the position information of each vertex. If the modeling device determines that the sum of the maximum scales corresponding to all vertices in the third initial graphic unit is less than the perimeter of the third initial graphic unit, then the third initial graphic unit is deleted from the initial graphic units representing the object surface.

[0146] For example, the modeling device can calculate the maximum scale corresponding to each of the three vertices of a triangle, sum them, and compare the sum with the perimeter of the triangle. If the sum of the maximum scales is less than the perimeter, the triangle is deleted. This is because the scale of the triangle is too small, that is, the size of the Gaussian point is too small, and its impact on the color of the pixels after projection is too small. Deleting such triangles can also reduce the amount of calculation and improve computational performance.

[0147] Specifically, during the iterative adjustment process, the modeling device can also acquire loss data and second vertex positions after a set number of iterations. The set number of iterations can be 100 or 500, such as adjusting the graphical unit after every 100 iterations. The loss data can be data generated based on the difference between the training image and the reference image after every 100 iterations. The second vertex position can be the same as the first vertex position, obtained after every 100 iterations. The second vertex position includes the position information of the vertices of each initial graphical unit, i.e., the three-dimensional position coordinates. Furthermore, the modeling device can perform growth processing on the initial graphical units according to growth rules.

[0148] In one implementation, the modeling device can calculate the position gradient of each vertex based on the loss data. If the modeling device determines that the position gradient of a target vertex is greater than a set gradient threshold, then an initial graphical unit is added based on the position information of the target vertex. It can be understood that if the position gradient of the target vertex... Greater than the set gradient threshold τ pos This indicates that the target vertex is in a scene region with complex high-frequency geometry. In this case, the expressive power of the graphics unit is relatively limited, so it is necessary to add more graphics units to represent the complex geometry.

[0149] Specifically, after determining the position gradient of the target vertex... Greater than the set gradient threshold τ pos Then, the modeling device can further determine whether the scale s of the target vertex is greater than the set scale threshold τ. s If it is determined that s < τ s The modeling device can then clone the target vertex at its location to generate new initial graphic units. For example, the modeling device can add three Gaussian points at the target vertex location and use these three Gaussian points as vertices of a triangle. The Gaussian distribution attributes (opacity, scale, color, and quaternions) of these three Gaussian points are identical to those of the target vertex, thus achieving "cloning" the Gaussian points. The modeling device can then randomly perturb the positions of these three Gaussian points to prevent them from overlapping, thereby forming new triangles to participate in subsequent iterative adjustments.

[0150] If it is determined that s > τ sThe modeling device can then segment the target vertex to generate new initial graphic units. For example, the modeling device can add three Gaussian points at the target vertex and use these three Gaussian points as vertices of a triangle. The opacity, color, and quaternion properties of these three Gaussian points are the same as those of the target vertex, and their scale is half that of the Gaussian point at the target vertex. The scale of the Gaussian point at the target vertex is also adjusted to half of the scale after a set number of iterations, thus achieving the "segmentation" of the Gaussian points. The modeling device can randomly perturb the positions of these three Gaussian points to prevent them from overlapping, so as to form new triangles to participate in subsequent iterative adjustments.

[0151] Therefore, the modeling device can use the vertex positions, vertex Gaussian properties, and interpolation point positions of the initial graphical units obtained through iterative adjustments during the first training phase (e.g., 3000 times) as modeling information obtained in the first training phase for the second training phase. The modeling device can also freeze the vertex positions and vertex Gaussian distributions of the initial graphical units, using them as the basis for the second training phase.

[0152] The second phase of the training process will now be described.

[0153] Specifically, the modeling device can acquire the vertex positions, vertex Gaussian properties, and interpolation point positions of the initial graphical unit obtained through iterative adjustment. Based on the iteratively adjusted vertex positions, it generates initial vertex features for the initial graphical unit and generates a second training image of the object from a reference viewpoint based on the initial vertex features and the iteratively adjusted interpolation point positions. Then, based on the difference between the second training image and the reference image, second loss data is generated, and the initial vertex features are adjusted based on the second loss data to obtain updated vertex features. The initial vertex features are the vertex features of the initial graphical unit generated based on the iteratively adjusted vertex positions, represented in the form of feature vectors, and are related to the vertex position information. During the second stage of training, the second training image can be generated based on the initial vertex features and the iteratively adjusted interpolation point positions.

[0154] Therefore, in the process of using the vertex positions, vertex Gaussian attributes, interpolation point positions, and interpolation Gaussian attributes of the initial graphic unit obtained through iterative adjustment to generate a reconstructed image, specifically, the vertex positions, vertex Gaussian attributes obtained through iterative adjustment, and updated vertex features can be used to generate the reconstructed image. Specifically, after obtaining the vertex positions, vertex Gaussian attributes, and updated vertex features obtained through iterative adjustment, the image generation device begins generating the reconstructed image. First, for interpolation calculations within the graphic unit, taking a triangle as an example, the image generation device performs linear interpolation on the updated vertex features based on the interpolation coordinates (i.e., centroid coordinates) corresponding to the triangle. Specifically, for each interpolation point within the triangle, the features of the three vertices are weighted and summed according to the weight relationship between their centroid coordinates and the features of the three vertices to obtain the interpolation features of that interpolation point. Simultaneously, the position of the interpolation point is calculated based on the same interpolation coordinates and vertex positions. Based on the three vertex positions of the triangle, linear interpolation is performed between the three vertex positions according to the weights represented by the interpolation coordinates to determine the specific three-dimensional position of the interpolation point within the triangle. Next, the image generation device generates interpolated Gaussian attributes based on interpolation features and information from the specified viewpoint. The specified viewpoint information is generated based on the camera position and the interpolation point position. The image generation device first calculates the vector difference between the camera position and the interpolation point position, then normalizes this vector to obtain a vector representing the viewpoint direction; this vector is the specified viewpoint information. Then, the interpolation features and the specified viewpoint information are input into a pre-trained Multilayer Perceptron (MLP) model. This model, based on the learned mapping relationship, outputs information such as the scale, color, quaternion, and opacity of the interpolation points; this information constitutes the interpolated Gaussian attributes. Finally, the image generation device performs rendering processing based on the iteratively adjusted vertex positions and vertex Gaussian attributes, as well as the interpolation point positions and interpolation Gaussian attributes. Using the camera viewpoint as a reference, the vertices and interpolation points are projected onto a two-dimensional image plane. For the pixels covered by the projection, the image generation device considers the occlusion of light by preceding vertices and interpolation points during propagation, based on the opacity and color of the vertices and interpolation points. Following a forward-to-backward order, it calculates the light attenuation based on the opacity of each vertex and interpolation point, and then combines this with their colors to determine the color of the pixel using an alpha-mixing method. Through the calculation and integration of the colors of all pixels, a reconstructed image of the object at a specified viewpoint is finally generated.

[0155] The vertex positions, Gaussian properties, and interpolation point positions of the initial graphic unit obtained through iterative adjustment are obtained after iterative adjustment through the first training phase (e.g., 3000 times). To implicitly model the Gaussian distribution properties of the interpolation points, the modeling device can generate feature vectors for each vertex of the initial graphic unit based on the iteratively adjusted vertex positions, thus obtaining the initial vertex features of the initial graphic unit. For example, the modeling device can generate corresponding feature vectors based on the three-dimensional position coordinates of three vertices in a triangle, obtaining f1, f2, and f3 as shown in Formula 4. It should be noted that this embodiment does not limit the generation method; the generated feature vectors only need to be related to the position information of the corresponding vertices.

[0156] Therefore, the modeling device can generate a second training image of the object from a reference viewpoint based on the initial vertex features and the interpolation point positions obtained through iterative adjustment. The vertex positions obtained through iterative adjustment include the position information of each vertex in each initial graphic unit, and the interpolation point positions obtained through iterative adjustment include the position information of each interpolation point in each initial graphic unit.

[0157] Specifically, the modeling device can perform interpolation calculations on the initial vertex features based on the position information of each interpolation point and each vertex, to obtain the initial interpolation features of each interpolation point. This interpolation calculation can include calculating interpolation coordinates based on the relationship between the position information of each interpolation point and the position information of the vertices of the graphic unit where the interpolation point is located, and then calculating the initial interpolation features of each interpolation point based on these interpolation coordinates. Specifically, the modeling device can determine the interpolation coordinates of each interpolation point within its respective initial graphic unit based on the position information of each interpolation point and the position information of the vertices of the initial graphic unit where the interpolation point is located. For example, as shown in Figure 7, the modeling device can determine the interpolation coordinates of the interpolation point, i.e., (u, v), based on the proportions of the sides of the triangles containing triangles u and v in the position information of each interpolation point. Furthermore, the modeling device can perform interpolation calculations on the initial vertex features of the initial graphic unit where each interpolation point is located, according to the calculation method shown in Formula 4, to obtain the initial interpolation features f of each interpolation point. uv .

[0158] After obtaining the initial interpolation features at each interpolation point, the modeling device can determine the predicted interpolation Gaussian attribute of each interpolation point based on the initial interpolation features and the information from the reference viewpoint. The predicted interpolation Gaussian attribute of each interpolation point can be determined based on the mapping relationship between the initial interpolation features, the information from the reference viewpoint, and the predicted interpolation Gaussian attribute. The specific determination method can be found in the implementation described in step S303, and will not be repeated here. It should be noted that this predicted interpolation Gaussian attribute is obtained by inputting the information from the reference viewpoint (viewpoint direction parameters) and the initial interpolation features into the initial machine learning model (such as a triangle MLP with unadjusted model parameters).

[0159] Therefore, the modeling device can perform rendering processing based on the vertex positions and Gaussian properties obtained through iterative adjustment, as well as the interpolation point positions and predicted interpolation Gaussian properties obtained through iterative adjustment, to obtain the second training image. The rendering process based on the Gaussian point positions and Gaussian distribution properties can be found in the specific implementation described in step S304, and will not be repeated here. It should be noted that during the second stage of training, the reference viewpoint can be the viewpoint specified during training, and can be the same as or different from the specified viewpoint; this application does not impose any limitations on this.

[0160] In the second stage of training, loss data can also be generated using machine learning methods to iteratively optimize the initial vertex features of each initial graphical unit. This allows the updated vertex features, obtained through iterative adjustment, to better fit the object surface to the interpolation points (based on the interpolated Gaussian properties generated from the updated vertex features). The second training image can be an image generated during training, referring to the object's image from a reference viewpoint. The second loss data can be generated based on the difference between the second training image and the reference image, used to measure the error between them. Therefore, the initial vertex features can be iteratively adjusted based on the second loss data. It should be noted that after the initial vertex features are adjusted to the updated vertex features, the initial interpolation features of each interpolation point are also updated to determine the updated interpolation features, which are calculated based on the updated vertex features and interpolation coordinates according to Formula 4.

[0161] In some embodiments, the second loss data is further used to update the mapping relationship between the initial interpolation features (updated interpolation features) and the information from the reference viewpoint, and the predicted interpolation Gaussian attributes. Specifically, the model parameters of the initial machine learning model (such as a triangle MLP with unadjusted model parameters) can be adjusted to obtain a trained machine learning model. Thus, in the process of using the iteratively adjusted vertex positions, the iteratively adjusted vertex Gaussian attributes, and the updated vertex features to generate the reconstructed image, the iteratively adjusted vertex positions, the iteratively adjusted vertex Gaussian attributes, the interpolation coordinates of each interpolation point, the updated vertex features, and the trained machine learning model can be used to generate the reconstructed image.

[0162] Image generation equipment can generate reconstructed images using iteratively adjusted vertex positions, iteratively adjusted vertex Gaussian attributes, interpolation coordinates of each interpolation point, updated vertex features, and a trained machine learning model. First, interpolation calculation is performed. Taking a triangle as an example, the updated vertex features are interpolated based on the interpolation coordinates of each interpolation point. The interpolation coordinates reflect the proportional relationship of the interpolation point's position relative to the three vertices within the triangle. Based on this proportional relationship, the features of the three vertices are weighted and averaged to obtain the interpolation feature of the interpolation point. Simultaneously, combining the interpolation coordinates and vertex positions, the position of the interpolation point is determined; that is, linear interpolation is performed between the three vertex positions based on the interpolation coordinates to determine the specific three-dimensional position of the interpolation point within the triangle. Next, based on the interpolation features and information from a specified viewpoint, the trained machine learning model generates interpolated Gaussian attributes. The information from the specified viewpoint is generated based on the camera position and the interpolation point positions. First, the vector difference between the camera position and the interpolation point position is calculated, and then this vector is normalized to obtain a vector representing the viewpoint direction; this vector is the information for the specified viewpoint. The interpolation features and information from a specified viewpoint are input into a trained machine learning model. Based on the learned mapping relationship, the model outputs attributes such as the scale, color, quaternion, and opacity of the interpolation points; these attributes constitute the interpolation Gaussian properties. Finally, rendering is performed based on the iteratively adjusted vertex positions and vertex Gaussian properties, as well as the interpolation point positions and interpolation Gaussian properties. Using the camera viewpoint as a reference, the vertices and interpolation points are projected onto a two-dimensional image plane. For the pixels covered by the projection, considering the occlusion of light by preceding vertices and interpolation points during propagation, the light attenuation is calculated sequentially from near to far based on the opacity of each vertex and interpolation point, and then combined with its color to determine the color of the pixel. Through the calculation and integration of the colors of all pixels, a reconstructed image of the object from the specified viewpoint is finally generated.

[0163] Specifically, the second loss data can be similar to the first loss data, and can also include absolute error (L1) loss and SSIM loss. The second loss data can also include a scale regularization term to prevent the scale of the Gaussian points from becoming too large, as shown in Equation 11.

[0164] Based on the second loss data, the model parameters of the initial machine learning model can be adjusted using the stochastic gradient descent (SGD) algorithm. Specifically, the gradient of the second loss data with respect to the model parameters is first calculated; for model parameters θ, its gradient... This represents the rate of change of the loss function L relative to θ. Then, according to the update formula... (where θ) t Here, η represents the model parameters at the t-th iteration, and η is the learning rate, used to control the step size for parameter updates. The model parameters are updated by randomly selecting a small batch of data in each iteration to calculate the gradient, repeating this process until a convergence condition is met (e.g., the change in the loss function is less than a certain threshold), thus obtaining the trained machine learning model.

[0165] In some embodiments, to further eliminate unnecessary small-scale Gaussian points and improve computational efficiency, a mask parameter can be introduced for each Gaussian point. This parameter measures the importance of the Gaussian point and can be iteratively optimized during the second-stage training process. Therefore, a mask loss can be introduced into the second loss data, with a formula similar to the scale regularization term. Specifically, the second loss data can be expressed as shown in Equation 13: L stage2 =L stage1 +λ mask L mask Formula 13

[0166] In Formula 13, L stage1 L represents the training loss in the first phase. stage2 The training loss for the second phase is represented by Equations 11 and 12, L. mask The introduced masking loss can be similar to the scaling regularization formula, or it can be constructed according to the specific implementation scenario. This application does not impose any restrictions on this, and it can be generated based on the learned masking parameters. λ mask This is a weighting term, which can be set to 0.001; this application does not impose any restrictions on this.

[0167] Masking loss is a loss term introduced into the second loss data during the second training phase to further eliminate unnecessary small-scale Gaussian points and improve computational efficiency. Its formula is similar to the scaling regularization term, and it can be generated based on the learned mask parameters. Unnecessary Gaussian points are reduced through iterative optimization of the mask parameters.

[0168] In this second training phase, the number of iterations can be 30,000. This means the modeling device can iteratively adjust the initial vertex features and the model parameters of the initial machine learning model based on the second loss data. The initial machine learning model can include multiple machine learning libraries (MLPs) corresponding to scale, color, opacity, and quaternion attributes, respectively. Optionally, it can also include an MLP for predicting positional offsets, used to generate positional offset features (delta p), which participates in the generation of training images during the second training phase. Furthermore, the modeling device can regenerate training images based on the adjusted initial vertex features and the initial machine learning model, and regenerate the loss based on the difference between the training image and the reference image, adjusting the initial vertex features and the initial machine learning model again. This process is repeated until the iteration threshold (such as 30,000 times as mentioned above) is reached, resulting in updated vertex features and a trained machine learning model.

[0169] In one possible implementation, the number of Gaussian points required to optimize the detail representation within each initial graphics unit may vary depending on the complexity of the local details. Therefore, the modeling device can dynamically adjust the number of Gaussian points set at the interpolation points in the initial graphics unit according to a set number of iterations. Specifically, this may include pruning and growing the Gaussian points set at the interpolation points. This process can be called adaptive control of Gaussian points.

[0170] Specifically, during the iterative adjustment process, the modeling device can acquire the first interpolated Gaussian attribute after a set number of iterations. The set number of iterations can be 100 or 500, and can be the same as or different from the adaptive control of the graphics unit in the first training phase; this application does not limit this, such as adjusting the Gaussian point every 500 iterations. The first interpolated Gaussian attribute can be the predicted interpolated Gaussian attribute obtained after each set number of iterations (e.g., every 500 iterations). It can be an interpolated feature calculated based on the vertex features after the set number of iterations, and input along with the reference viewpoint information into the initial machine learning model after the set number of iterations. This first interpolated Gaussian attribute can include the opacity of the Gaussian point set at the interpolation point. Furthermore, the modeling device can perform pruning processing on the Gaussian point set at the interpolation point according to pruning rules.

[0171] In one implementation, if the modeling device determines that the opacity of the first interpolated Gaussian attribute is lower than a set opacity threshold τ... α If a Gaussian point is found, the modeling device can delete the Gaussian point set at that interpolation point.

[0172] In another implementation, the modeling device can calculate the average opacity of a set Gaussian point at each interpolation point based on the opacity of the predicted interpolation Gaussian attribute obtained in each of the set number of iterations. For example, it can calculate the average opacity value during 100 iterations of adjustment. If the modeling device determines that the average opacity is below a set opacity threshold τ... α If a Gaussian point is found, the modeling device can delete the Gaussian point set at that interpolation point.

[0173] Specifically, during the iterative adjustment process, the modeling device can also acquire loss data after a set number of iterations, such as data generated based on training and reference images after every 100 iterations. Furthermore, the modeling device can calculate the position gradient of each interpolation point in the initial graphic unit based on the loss data, and perform growth processing on the initial graphic unit according to the growth rule.

[0174] In one implementation, the modeling device can determine the positional gradient of each interpolation point. Is it greater than the set gradient threshold τ? pos If the value is greater than 2, it indicates that the interpolation point is located in a scene region with complex high-frequency geometry. In this case, the representational power of the Gaussian point set at the interpolation point is relatively limited, so it is necessary to add more Gaussian points to represent the complex geometry. Since the position gradient is a three-dimensional vector, the modeling device can add Gaussian points along the gradient direction. For example, if the calculated position gradient is (2,2,2), the position of the new interpolation point can be determined based on the position of the current interpolation point and the direction and distance indicated by the position gradient, and a Gaussian point can be set at the position of the new interpolation point.

[0175] Optionally, the Gaussian distribution attribute of the newly added Gaussian point can be the same as the Gaussian distribution attribute set for the interpolation point. Since the newly added Gaussian point is also located inside the initial graphic unit where the interpolation point is located, the modeling device can determine the interpolation coordinates (u,v) of the newly added Gaussian point based on the position information of the newly added Gaussian point and the vertex position of the initial graphic unit for subsequent iterative adjustment processes.

[0176] Therefore, the modeling device can use the vertex positions, vertex Gaussian properties, and interpolation coordinates of each interpolation point obtained through iterative adjustments in the first training phase (e.g., 3000 times), along with the updated vertex features and trained machine learning model obtained through iterative adjustments in the second training phase (e.g., 30000 times), as modeling information obtained in the second training phase to construct the reconstructed image. It should be noted that the Gaussian points obtained through the first and second training phases can be used as modeling information at LOD0, and can subsequently be used to determine modeling information at LOD1, LOD2, and other levels of detail.

[0177] In one possible implementation, the modeling device can downsample the interpolation points in each initial graphic unit based on the interpolation point positions obtained through iterative adjustment, thereby removing some interpolation points from each initial graphic unit. Then, based on the vertex positions obtained through iterative adjustment and the position information of other interpolation points in each initial graphic unit (excluding some interpolation points), the interpolation coordinates of the remaining interpolation points are determined. This allows for the acquisition of modeling information at different levels of detail. Specifically, in the process of using the iteratively adjusted vertex positions, the iteratively adjusted vertex Gaussian properties, the interpolation coordinates of each interpolation point, the updated vertex features, and the trained machine learning model to generate the reconstructed image, the iteratively adjusted vertex positions, the iteratively adjusted vertex Gaussian properties, the interpolation coordinates of other interpolation points, the updated vertex features, and the trained machine learning model can be used to generate the reconstructed image.

[0178] Image generation devices can generate reconstructed images using iteratively adjusted vertex positions, iteratively adjusted vertex Gaussian attributes, interpolated coordinates of other interpolation points, updated vertex features, and a trained machine learning model. First, interpolation calculations are performed. Taking a triangle as an example, the updated vertex features are interpolated based on the interpolated coordinates of other interpolation points. The interpolated coordinates reflect the positional relationship of other interpolation points relative to the three vertices within the triangle. Based on this relationship, the features of the three vertices are weighted and summed to obtain the interpolated features of the other interpolation points. Simultaneously, combining the interpolated coordinates and vertex positions, the positions of the other interpolation points are determined; that is, linear interpolation is performed between the three vertex positions based on the interpolated coordinates to determine the specific three-dimensional position of the other interpolation points within the triangle. Next, based on the interpolated features and information from a specified viewpoint, the trained machine learning model generates interpolated Gaussian attributes. The information from the specified viewpoint is generated based on the camera position and the positions of other interpolation points. First, the vector difference between the camera position and the positions of other interpolation points is calculated, and then this vector is normalized to obtain a vector representing the viewpoint direction; this vector is the information for the specified viewpoint. The interpolation features and information from a specified viewpoint are input into a trained machine learning model. Based on the learned mapping relationship, the model outputs attributes such as scale, color, quaternion, and opacity of other interpolation points; these attributes constitute the interpolation Gaussian properties. Finally, rendering is performed based on the vertex positions and vertex Gaussian properties obtained through iterative adjustments, as well as the positions and interpolation Gaussian properties of other interpolation points. Using the camera viewpoint as a reference, vertices and other interpolation points are projected onto a 2D image plane. For pixels covered by the projection, considering the occlusion of light by preceding vertices and other interpolation points during propagation, the attenuation of light is calculated sequentially from near to far based on the opacity of each vertex and other interpolation point, and then combined with its color to determine the color of the pixel. Due to downsampling, the number of other interpolation points is reduced, which reduces computational load and memory usage, improving the efficiency of generating the reconstructed image. By calculating and integrating the colors of all pixels, a reconstructed image of the object at the specified viewpoint is finally generated.

[0179] Downsampling, also known as subsampling, refers to removing Gaussian points at some interpolation points in the initial graphic unit. The modeling device can sample Gaussian points at set intervals to remove some Gaussian points. Specifically, the sampling method can be as shown in Formula 14:

[0180] In Formula 14, G jK For the sampled Gaussian points, taking an initial graphic unit as an example, G jKThis represents the interpolation points other than some interpolation points in the initial graphic unit, where j is the index of the other interpolation points, N is the total number of Gaussian points in the initial graphic unit, and K is the set sampling interval, which can be 2. n The value of n can be set by the LOD and memory resource requirements. n is a non-negative number, and this application does not limit it.

[0181] Please refer to Figure 8, which is a schematic diagram of the LOD downsampling principle provided in an embodiment of this application. As shown in Figure 8, the modeling device can determine the high-density and low-density regions in the initial graphic unit based on the interpolation point positions obtained through iterative adjustment. The density is determined by kernel density estimation, for example, the high-density region at the top and the low-density region at the bottom inside the initial graphic unit (triangle) on the left side of Figure 8. The modeling device uses a Gaussian kernel function to perform kernel density estimation based on the interpolation point positions obtained through iterative adjustment to determine the high-density and low-density regions in the initial graphic unit. For each interpolation point x... i Its nuclear density estimation formula is: Where n is the total number of interpolation points, h is the bandwidth (controlling the influence range of the kernel function), and K is the Gaussian kernel function. Calculate the density estimate at each location x. Set a density threshold; areas with density values ​​greater than the threshold are considered high-density areas, and areas with density values ​​less than the threshold are considered low-density areas.

[0182] High-density regions are areas with high interpolation point density determined by kernel density estimation within the initial graphic unit. The modeling device, based on the interpolation point positions obtained through iterative adjustment, uses a Gaussian kernel function to estimate kernel density, setting a density threshold. Regions with density values ​​greater than this threshold are considered high-density regions. During LOD downsampling, the interpolation points in these regions are downsampled at a first predetermined ratio. Low-density regions are areas with low interpolation point density determined by kernel density estimation within the initial graphic unit. The modeling device, based on the interpolation point positions obtained through iterative adjustment, uses a Gaussian kernel function to estimate kernel density, setting a density threshold. Regions with density values ​​less than this threshold are considered low-density regions. During LOD downsampling, the interpolation points in these regions are downsampled at a second predetermined ratio.

[0183] The modeling device can then downsample interpolation points from high-density areas at a first predetermined ratio and from low-density areas at a second predetermined ratio. For example, the first predetermined ratio might be 75%, and the second predetermined ratio 25%. The specific downsampling method can be shown in Formula 14. Thus, as shown on the right side of Figure 8, compared to the number of interpolation points G0 in LOD0, half the number of interpolation points can be reduced in LOD1 (G0 / 2), and again in LOD2 (G0 / 4). The first predetermined ratio is the ratio by which interpolation points in high-density areas of the initial graphics unit are downsampled during LOD downsampling. This ratio removes some interpolation points from high-density areas to achieve modeling at different levels of detail. The second predetermined ratio is the ratio by which interpolation points in low-density areas of the initial graphics unit are downsampled during LOD downsampling. This ratio removes some interpolation points from low-density areas to achieve modeling at different levels of detail.

[0184] It should be noted that after reducing the number of interpolation points, the remaining interpolation points, if distributed according to the original Gaussian distribution, have limited expressive power. As shown in Figure 8, the Gaussian points in LOD0 are relatively small, and reducing their number by half may result in an inability to properly fit the object's surface. Therefore, a second stage of training can be performed based on the remaining interpolation points to iteratively adjust the updated vertex features and the trained machine learning model. For example, 5000 iterations can be performed. During the iterative adjustment process, adaptive control of the Gaussian points can also be implemented. The specific implementation method for the second stage of training can be found in step S503, and will not be repeated here. As can be seen, in LOD1 and LOD2, the machine learning model obtained after the second iteration can learn a better fit to the interpolation Gaussian properties of the object's surface, as shown in Figure 8, where the size of the light-colored ellipse in LOD1 and LOD2 is larger.

[0185] Please refer to Figure 9, which is a schematic diagram of a 3DGS training process provided in an embodiment of this application. As shown in Figure 9, taking a microphone as the object and a triangle as the initial graphic unit, the modeling device can acquire a set of feature points representing the microphone surface and generate sparse triangles (which can be called initial triangles) based on this. The vertex positions can be optimized during the iterative adjustment process in the first stage of training to fit the geometry of the microphone surface, presenting the triangle set in Figure 9(a) to represent the microphone. The first stage of training also includes pruning and growing the number of triangles to achieve adaptive control of the triangles. After a set number of times (e.g., 3000 times), the modeling device can freeze the Gaussian distribution attributes of the triangle vertices and the Gaussian points set at the triangle vertices to enter the second stage of training.

[0186] As shown in Figure 9(b), during the second stage of training, feature vectors (vertex features) can be generated for each vertex of the triangle, and the interpolation features f of the Gaussian points set at the interpolation points inside each triangle can be determined based on interpolation calculation. uv Furthermore, the interpolation feature f uv The viewpoint direction parameter d, calculated from the camera position X0, can be input into the initial machine learning model (triangle-MLP) to predict the values ​​of the scale s, color c, quaternion q, and opacity α attributes of the interpolation points, thereby determining the shape and size of the Gaussian points at each interpolation point, as shown in Figure 9(c). Furthermore, the modeling device can generate a reconstructed image (training image) as shown in Figure 9(d) based on the predicted Gaussian distribution attributes of the interpolation points to iteratively adjust the initial vertex features and the model parameters of the initial machine learning model.

[0187] The second stage of the training process also includes pruning and increasing the number of Gaussian points set for interpolation points inside the triangle to achieve adaptive control of the Gaussian points. After a set number of iterations (e.g., 30,000), the modeling device can use the vertex positions and Gaussian attributes obtained from the iterative adjustments in the first stage of training, as well as the interpolation coordinates of each interpolation point, the updated vertex features, and the trained machine learning model obtained in the second stage of training, for the inference process shown in Figure 4.

[0188] In some embodiments of this application, when generating an image of an object from a specified viewpoint based on 3D Gaussian splashing technology, the object surface is represented by initializing and optimizing the graphics units during the modeling process. Compared with directly using Gaussian points, this makes the Gaussian points representing the object have structured information, which can reduce visual artifacts at the edges to a certain extent and improve the effect of the generated reconstructed image.

[0189] Please refer to Figure 10. Figure 10 is a comparison diagram of the effects of another 3DGS scheme provided in this application embodiment and the scheme of this application. As shown in Figure 10, the other 3DGS scheme can be the Scaffold-GS scheme, which uses Gaussian points to represent the forklift without any structured information, resulting in isolated points at the edges, which may cause visual artifacts in the rendered reconstructed image. Furthermore, the Scaffold-GS scheme generates all Gaussian points by storing anchors, which can reduce the amount of data stored to some extent, but still requires storing 5000 anchors. In contrast, the scheme of this application only requires storing the data of 400 triangles, which is much smaller than the number of anchors required by the Scaffold-GS scheme, thereby improving memory efficiency and reducing memory resource consumption.

[0190] Furthermore, by downsampling the interpolation points in the dimension of the graphic unit, a certain amount of structured information can be preserved. Compared with randomly reducing Gaussian points, this is beneficial for preserving the information of objects in the 3D scene, so that Gaussian points at multiple levels of detail can also fit the surface of the object well, thereby improving the image quality at different levels of detail.

[0191] The following is a quantitative and qualitative evaluation of the effect of the reconstructed image generated by the image generation method provided in the embodiments of this application.

[0192] Please refer to Figure 11, which is a schematic diagram of quantitative evaluation at different levels of detail provided in an embodiment of this application. Figure 11 shows the initialization of the first stage of training by randomly selecting 1000 feature points from the feature point set, with the initial number of interpolation points K=66. Experiments were conducted after 3000 first-stage training iterations and 30000 second-stage training iterations, assuming all MLPs consist of two hidden layers, each with 64 dimensions; for several complex scenes, each hidden layer has 128 dimensions. The 3D scene data used for training and reconstruction came from the NeRF-Synthetic and NSVF datasets.

[0193] As shown in the two performance curves at the top of Figure 11, the horizontal axis represents storage size, i.e., the amount of memory resources required, expressed as a logarithm of storage size. The vertical axis represents PSNR, SSIM, and Learned Perceptual Image Patch Similarity (LPIPS), respectively. LPIPS measures the difference between two images; a smaller value indicates a smaller difference and better generation results. It is evident that, at different levels of detail (LOD), the image generation method provided in this application can reduce memory resource requirements compared to other 3DGS schemes at various LOD levels. Furthermore, PSNR and SSIM are higher than other 3DGS schemes at the same LOD, while LPIPS is lower than other 3DGS schemes at the same LOD. Therefore, the reconstructed images generated by the image generation method provided in this application have higher image quality.

[0194] As shown in the chart below Figure 11, LOD1, LOD2, and LOD3 are implemented based on sampling rates K = 1, 2, 4, and 8, respectively. An efficiency term is introduced in this chart to indicate the PSNR value corresponding to each unit of memory usage. It can be seen that the image generation method provided in this application embodiment can achieve good image quality (PSNR, SSIM) under various LODs, and under the requirement of 1MB of storage size, the image generation method provided in this application embodiment retains the most detail (efficiency) compared to other 3DGS schemes. The above quantitative evaluation shows that using a set of graphics units (triangle sets) and a machine learning model (triangle MLP) can retain the most detailed information while minimizing memory usage.

[0195] Please also refer to Figure 12, which is a schematic diagram of qualitative evaluation at different levels of detail provided in an embodiment of this application. Figure 12 also shows the initialization of the first stage of training by randomly selecting 1000 feature points from the feature point set, with the initial number of interpolation points K=66. Experiments were conducted after 3000 first-stage training iterations and 30000 second-stage training iterations, assuming all MLPs consist of two hidden layers, each with 64 dimensions; for several complex scenes, each hidden layer has 128 dimensions. The 3D scene data used for training and reconstruction came from the NeRF-Synthetic and NSVF datasets.

[0196] As shown in Figure 12, with a drum kit as an example at the top and a spaceship as an example at the bottom, at LOD0, the image generation method proposed in this embodiment (third row) can render more realistic details compared to other 3DGS schemes (first two rows), with no obvious visual artifacts on the drum surface. After downsampling (such as LOD1-LOD3), the image generation method provided in this embodiment also preserves the details of these areas, and the reconstructed image generated at LOD3 uses only 1.5MB of storage space. Similarly, similar results can be observed in the spaceship. Within the dashed box of the spaceship at the bottom of Figure 12, detail loss can be observed at LOD2 and LOD3, such as specular loss (white arrow) and overly smooth high-frequency geometry (black arrow) at LOD3. It can be seen that the image generation method proposed in this embodiment can maintain almost consistent high-detail rendering results from LOD1 to LOD3.

[0197] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0198] Please refer to Figure 13, which is a schematic diagram of an image generation apparatus provided in an embodiment of this application. The image generation apparatus shown in Figure 13 can be mounted in a computer device, specifically a server. The image generation apparatus shown in Figure 13 can be used to perform some or all of the functions described in the method embodiment of Figure 3 above. Please refer to Figure 13, the image generation apparatus 130 includes:

[0199] The acquisition unit 1301 is used to acquire the vertex position, vertex features and vertex Gaussian attributes of each graphics unit, and to acquire the interpolation information of each graphics unit, wherein each graphics unit is used to represent the surface of an object.

[0200] Processing unit 1302 is used to interpolate the vertex position and the vertex feature in each graphic unit based on the interpolation information, so as to obtain the interpolation point position and interpolation feature in each graphic unit.

[0201] The generation unit 1303 is used to generate the interpolation Gaussian attribute of each graphics unit based on the interpolation features and the information of the specified viewpoint.

[0202] The processing unit 1302 is further configured to perform rendering processing based on the vertex position and the vertex Gaussian attribute, as well as the interpolation point position and the interpolation Gaussian attribute, to obtain a reconstructed image of the object at the specified viewpoint.

[0203] In one possible implementation, the interpolation information includes at least one interpolation coordinate; the processing unit 1302 is configured to interpolate the vertex position and the vertex feature within each graphic unit based on the interpolation information, to obtain the interpolation point position and interpolation feature within each graphic unit, specifically configured to:

[0204] Based on the interpolated coordinates of each graphic unit, interpolation calculation is performed on the vertex position to obtain at least one interpolation point position in each graphic unit;

[0205] Based on the interpolation coordinates of each graphic unit, interpolation calculations are performed on the vertex features to obtain the interpolation features of the position of at least one interpolation point.

[0206] In one possible implementation, the processing unit 1302 is configured to perform rendering processing based on the vertex position and the vertex Gaussian property, and the interpolation point position and the interpolation Gaussian property, to obtain a reconstructed image of the object at the specified viewpoint, specifically for:

[0207] Based on the vertex Gaussian attribute and the vertex position, the vertices of each graphics unit are projected onto the image plane to obtain a first pixel. Based on the interpolation Gaussian attribute and the interpolation point position, the interpolation points within each graphics unit are projected onto the image plane to obtain a second pixel. The vertex Gaussian attribute includes the opacity and color of the vertex, and the interpolation Gaussian attribute includes the opacity and color of the interpolation point.

[0208] The color of the first pixel is calculated based on the color and opacity of the vertex, and the color of the second pixel is calculated based on the color and opacity of the interpolation point;

[0209] The reconstructed image is generated based on the color of the first pixel and the color of the second pixel.

[0210] In one possible implementation, the acquisition unit 1301 is further configured to acquire the initial vertex position, initial vertex Gaussian attribute, initial interpolation point position and initial interpolation Gaussian attribute of each initial graphic unit representing the surface of the object, and perform rendering processing based on the initial vertex position and the initial vertex Gaussian attribute, as well as the initial interpolation point position and the initial interpolation Gaussian attribute, to obtain the first training image of the object under the reference viewpoint;

[0211] The generation unit 1303 is further configured to generate first loss data based on the difference between the first training image and the reference image of the object under the reference view.

[0212] The adjustment unit 1304 is used to iteratively adjust the initial vertex position, the initial vertex Gaussian attribute, the initial interpolation point position, and the initial interpolation Gaussian attribute based on the first loss data, and use the vertex position, vertex Gaussian attribute, interpolation point position, and interpolation Gaussian attribute of the initial graphics unit obtained by the iterative adjustment to generate the reconstructed image.

[0213] In one possible implementation, the acquisition unit 1301 is used to acquire the initial vertex positions, initial vertex Gaussian attributes, initial interpolation point positions, and initial interpolation Gaussian attributes of each initial graphic unit representing the surface of the object, specifically for:

[0214] A set of feature points representing the surface of the object is obtained, and the initial vertex positions are determined based on the positions of each feature point in the set of feature points, so as to form each initial graphic unit;

[0215] Based on the set interpolation information, the initial vertex positions are interpolated within each initial graphic unit to obtain the initial interpolation point positions;

[0216] Initial Gaussian distribution attributes are assigned to the vertices and interpolation points in each initial graphic unit to obtain the initial vertex Gaussian attributes and the initial interpolation Gaussian attributes.

[0217] In one possible implementation, the adjustment unit 1304 is further configured to, during the iterative adjustment process, acquire a first vertex Gaussian attribute and a first vertex position after a set number of iterations, wherein the first vertex Gaussian attribute includes the opacity of each vertex in each initial graphic unit, and the first vertex position includes the position information of each vertex; if there is a first initial graphic unit in which the opacity of all vertices is lower than a set opacity threshold, then the first initial graphic unit is deleted from the initial graphic unit representing the object surface; if there is a second initial graphic unit containing a specified vertex with an opacity lower than the set opacity threshold, then the position information of the specified vertex is updated based on the position information of the vertex in the second initial graphic unit with an opacity higher than the set opacity threshold.

[0218] In one possible implementation, the first vertex Gaussian attribute further includes the scale of each vertex in multiple directions; the adjustment unit 1304 is further configured to determine the maximum scale corresponding to each vertex based on the scale of each vertex in multiple directions, and to determine the perimeter of each initial graphic unit based on the position information of each vertex; if the sum of the maximum scales corresponding to all vertices in the third initial graphic unit is less than the perimeter of the third initial graphic unit, then the third initial graphic unit is deleted from the initial graphic unit representing the object surface.

[0219] In one possible implementation, the adjustment unit 1304 is further configured to, during the iterative adjustment process, acquire loss data after a set number of iterations and a second vertex position, the second vertex position including the position information of each vertex in each initial graphics unit; calculate the position gradient of each vertex based on the loss data; and if there is a target vertex whose position gradient is greater than a set gradient threshold, add an initial graphics unit based on the position information of the target vertex.

[0220] In one possible implementation, the acquisition unit 1301 is further configured to acquire the vertex position, vertex Gaussian attribute and interpolation point position of the initial graphic unit obtained through iterative adjustment, generate the initial vertex features of the initial graphic unit based on the vertex position obtained through iterative adjustment, and generate the second training image of the object under the reference viewpoint based on the initial vertex features and the interpolation point position obtained through iterative adjustment.

[0221] The generation unit 1303 is further configured to generate second loss data based on the difference between the second training image and the reference image, and adjust the initial vertex features based on the second loss data to obtain updated vertex features;

[0222] The generation unit 1303 is used to generate the reconstructed image by using the vertex positions, vertex Gaussian attributes, interpolation point positions, and interpolation Gaussian attributes of the initial graphic units obtained through iterative adjustment, specifically for:

[0223] The vertex positions obtained through iterative adjustment, the vertex Gaussian properties obtained through iterative adjustment, and the updated vertex features are used to generate the reconstructed image.

[0224] In one possible implementation, the vertex positions obtained through iterative adjustment include the position information of each vertex in each initial graphics unit, and the interpolation point positions obtained through iterative adjustment include the position information of each interpolation point in each initial graphics unit; the generation unit 1303 is used to generate a second training image of the object under the reference viewpoint based on the initial vertex features and the interpolation point positions obtained through iterative adjustment, specifically for:

[0225] Based on the position information of each interpolation point and the position information of each vertex, interpolation calculation is performed on the initial vertex features to obtain the initial interpolation features of each interpolation point;

[0226] Based on the initial interpolation features and the information from the reference viewpoint, the predicted interpolation Gaussian attribute of each interpolation point is determined;

[0227] The second training image is obtained by rendering based on the vertex positions and Gaussian properties obtained through iterative adjustment, the interpolation point positions obtained through iterative adjustment, and the predicted interpolation Gaussian properties.

[0228] In one possible implementation, the generation unit 1303 is used to perform interpolation calculations on the initial vertex features based on the position information of each interpolation point and the position information of each vertex to obtain the initial interpolation features of each interpolation point, specifically for:

[0229] Based on the position information of each interpolation point and the position information of the vertices of the initial graphic unit where each interpolation point is located, the interpolation coordinates of each interpolation point in the initial graphic unit are determined.

[0230] Based on the interpolation coordinates of each interpolation point, the initial vertex features of the initial graphic unit where each interpolation point is located are interpolated to obtain the initial interpolation features of each interpolation point.

[0231] In one possible implementation, the predicted interpolation Gaussian attribute is obtained by inputting the information of the reference viewpoint and the initial interpolation features into an initial machine learning model; the adjustment unit 1304 is further configured to adjust the model parameters of the initial machine learning model based on the second loss data to obtain a trained machine learning model.

[0232] The generation unit 1303 is used to generate the reconstructed image by using the iteratively adjusted vertex positions, the iteratively adjusted vertex Gaussian attributes, and the updated vertex features. Specifically, it is used for:

[0233] The vertex positions obtained through iterative adjustment, the Gaussian properties of the vertex obtained through iterative adjustment, the interpolation coordinates of each interpolation point, the updated vertex features, and the trained machine learning model are used to generate the reconstructed image.

[0234] In one possible implementation, the processing unit 1302 is further configured to perform downsampling processing on the interpolation points in each initial graphic unit based on the interpolation point positions obtained by iterative adjustment, so as to remove some interpolation points in each initial graphic unit; and determine the interpolation coordinates of the other interpolation points based on the vertex positions obtained by iterative adjustment and the position information of other interpolation points in each initial graphic unit other than the some interpolation points.

[0235] The generation unit 1303 is used to generate the reconstructed image by using the iteratively adjusted vertex positions, iteratively adjusted vertex Gaussian attributes, interpolation coordinates of each interpolation point, updated vertex features, and the trained machine learning model. Specifically, it is used for:

[0236] The vertex positions obtained through iterative adjustment, the Gaussian properties of the vertex obtained through iterative adjustment, the interpolation coordinates of the other interpolation points, the updated vertex features, and the trained machine learning model are used to generate the reconstructed image.

[0237] According to one embodiment of this application, some steps involved in the image generation method shown in FIG3 can be executed by various units in the image generation apparatus shown in FIG13. For example, step S301 shown in FIG3 can be executed by the acquisition unit 1301 shown in FIG13, steps S302 and S304 can be executed by the processing unit 1302 shown in FIG13, and step S303 can be executed by the generation unit 1303 shown in FIG13. As another example, step S501 shown in FIG5 can be executed by the acquisition unit 1301 shown in FIG13, step S502 shown in FIG5 can be executed by the generation unit 1303 shown in FIG13, and step S503 shown in FIG5 can be executed by the adjustment unit 1304 shown in FIG13. The various units in the image generation apparatus shown in FIG13 can be individually or entirely combined into one or more other units, or some of the units can be further divided into two or more functionally smaller units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of this application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can be implemented by two or more units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image generation apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by two or more units.

[0238] According to another embodiment of this application, the image generation apparatus shown in FIG13, and the image generation method of the embodiment of this application, can be constructed and implemented by running a computer program (including program code) capable of performing the steps involved in the corresponding method shown in FIG3 on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0239] Based on the same inventive concept, the principle and beneficial effects of the image generation apparatus provided in the embodiments of this application are similar to the principle and beneficial effects of the image generation method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, these will not be repeated here.

[0240] Embodiments of this application also provide an electronic device, including: at least one processor; and a storage device for storing at least one computer program, which, when executed by at least one processor, causes the electronic device to implement the image generation method provided in the above embodiments.

[0241] Figure 14 shows a schematic diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. The electronic device may be the device shown in Figure 1 or Figure 2.

[0242] It should be noted that the computer system 1400 of the electronic device shown in Figure 14 is only an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0243] As shown in Figure 14, the computer system 1400 includes a Central Processing Unit (CPU) 1401, which can perform various appropriate actions and processes based on computer programs stored in Read-Only Memory (ROM) 1402 or loaded from storage portion 1408 into Random Access Memory (RAM) 1403, such as executing the image generation method described in the above embodiments. The RAM 1403 also stores various computer programs and data required for system operation. The CPU 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An Input / Output (I / O) interface 1405 is also connected to the bus 1404.

[0244] In some embodiments, the following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1410 as needed so that computer programs read from it can be installed into the storage section 1408 as needed.

[0245] In particular, according to embodiments of this application, a computer program implementing the image generation method can be carried on a computer-readable medium, which can be downloaded and installed from a network via the communication section 1409, and / or installed from a removable medium 1411.

[0246] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. For example, a computer-readable storage medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or at least two wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer program contained in the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0247] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or at least two executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0248] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0249] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor of an electronic device, causes the electronic device to implement the image generation method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0250] Another aspect of this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the image generation methods provided in the various embodiments described above. The computer program can be stored in a computer-readable storage medium. The computer program product can be a computer program as a product, such as an APP (Application), webpage, mini-program, etc.; or, the computer program product can also be a storage medium, device, terminal, virtual machine, etc., containing the computer program.

[0251] In summary, this application provides an image generation method, apparatus, device, computer-readable medium, and computer program product. The electronic device acquires the vertex positions, vertex features, vertex Gaussian attributes, and interpolation information of each graphics unit. Based on the interpolation information, it interpolates the vertex positions and vertex features within each graphics unit to obtain interpolation point positions and interpolation features. Based on the interpolation features and specified viewpoint information, it generates interpolated Gaussian attributes. Finally, it combines the vertex and interpolation point positions and Gaussian attributes to render a reconstructed image of the object from the specified viewpoint. This method structurally represents the object surface using Gaussian points in graphics units. This structural information makes the Gaussian points spatially regular, reducing visual information conflicts at edges caused by disordered Gaussian point arrangement, thereby reducing visual artifacts and improving the visual quality of the reconstructed image. Simultaneously, by using the relationship between graphics unit vertices and interpolation points and implicitly modeling interpolated Gaussian attributes, it avoids storing all Gaussian point attributes, reducing unnecessary data redundancy in the data storage structure, lowering memory resource consumption, and improving memory resource utilization, enabling the system to handle more image generation tasks with limited memory.

[0252] Furthermore, when the interpolation information contains at least one interpolation coordinate, the electronic device performs interpolation calculations on the vertex position and vertex features based on that interpolation coordinate. Mathematically, the interpolation coordinate provides a precise weight allocation basis for the generation of interpolation points. By linearly combining vertex positions and features, the continuity and logic of the geometric and attribute relationships between interpolation points and vertices are ensured, improving the distribution accuracy of Gaussian points in three-dimensional space and enabling the reconstructed image to more accurately represent the geometry of the object's surface.

[0253] Furthermore, the electronic device projects the vertices and interpolation points onto the image plane based on their Gaussian properties and positions, obtaining the first and second pixels. Then, it calculates the pixel colors based on opacity and color to generate the reconstructed image. This projection and color calculation method is based on the principles of light propagation and absorption in physical optics, considering the attenuation effect of Gaussian point opacity on light propagation and the influence of color on light reflection. This makes the pixel color calculation more consistent with physical reality, improving the color fidelity and realism of the reconstructed image.

[0254] Furthermore, the electronic device acquires the relevant attribute information of the initial graphics unit and renders the first training image. It generates first loss data based on the difference between the training image and the reference image. Based on this loss data, the initial attributes are iteratively adjusted and used to generate the reconstructed image. The iterative adjustment process is based on an optimization algorithm. In each iteration, the attributes of the graphics unit and Gaussian points are fine-tuned according to the loss data, making these attributes gradually approximate the true features of the object surface. As the number of iterations increases, the error between the reconstructed image and the reference image continuously decreases, improving the accuracy and stability of the reconstructed image.

[0255] Furthermore, the electronic device determines the initial vertex positions by acquiring a set of feature points to form initial graphic units, obtains the initial interpolation point positions based on the set interpolation information, and assigns initial Gaussian distribution attributes to the vertices and interpolation points. Constructing initial graphic units based on the set of feature points makes the initial state of the graphic units more closely resemble the actual geometric structure of the object surface, providing initial conditions closer to reality for subsequent training and adjustments, reducing convergence time during training, and improving training efficiency.

[0256] Furthermore, during iterative adjustments, the electronic device processes the graphics units based on vertex opacity, deleting those with excessively low opacity or updating the position information of low-opacity vertices. Graphical units with excessively low opacity contribute minimally to the final image during projection and color calculations; deleting these units directly reduces the amount of data and computational steps, lowering computational complexity and improving efficiency. Updating the position information of low-opacity vertices optimizes the structure of the graphics units, making the distribution of Gaussian points more reasonable and enhancing their ability to fit the object surface, thereby improving the quality of the reconstructed image.

[0257] Furthermore, when the Gaussian property of the first vertex contains multiple directional scales, the electronic device removes graphics units with excessively small scales by comparing the sum of the maximum scales of the vertices with the perimeter of the graphics unit. The impact of excessively small graphics units on pixel color after projection is negligible. Removing these graphics units avoids processing this irrelevant data, reduces waste of computational resources, improves computational efficiency, and does not significantly affect the quality of the reconstructed image.

[0258] Furthermore, during the iterative adjustment process, the electronic device adds initial graphics units based on the vertex position gradient. When the vertex position gradient exceeds a set threshold, it indicates that the geometry of the region has changed drastically, and the resolution of the current graphics unit is insufficient to accurately represent the features of the region. Adding graphics units increases the ability to represent complex geometries, improves the ability of the reconstructed image to capture the details of the object's surface, and enables the reconstructed image to more accurately restore the complex structure of the object's surface.

[0259] Furthermore, the electronic device acquires the iteratively adjusted vertex and interpolation point positions, generates initial vertex features, and obtains a second training image based on these features. Second loss data is generated by comparing these features with the reference image. The initial vertex features are then adjusted to obtain updated vertex features, which are used to generate the reconstructed image. The second stage of training and adjustment focuses on optimizing the vertex features. The updated vertex features more accurately reflect the feature information of the object's surface, and the generated interpolated Gaussian properties are more consistent with reality. This allows the interpolation points to more accurately simulate the optical and geometric properties of the object's surface in the reconstructed image, further improving the quality of the reconstructed image.

[0260] Furthermore, the electronic device calculates initial interpolation features based on the position information of each interpolation point and vertex, determines the predicted interpolated Gaussian attributes, and renders the second training image. Accurate position information provides a reliable data foundation for the calculation of interpolation features. The predicted interpolated Gaussian attributes determined based on the interpolation features and reference viewpoint information can more accurately simulate the attribute changes of Gaussian points under different viewpoints, making the second training image closer to the real scene. This provides a more accurate target for subsequent adjustments and helps improve the quality of the reconstructed image.

[0261] Furthermore, the electronic device determines the interpolation coordinates by using the interpolation points and vertex positions, and then calculates the initial interpolation features based on the interpolation coordinates. The determination of the interpolation coordinates is based on geometric relationships, ensuring a linear correlation between the calculation of the interpolation features and the vertex features. This allows the attributes of the interpolation points to accurately reflect the changes in the vertex features. Through this mathematical correlation, the expressive power of the interpolation points in three-dimensional space is improved, thereby enhancing the quality of the reconstructed image.

[0262] Furthermore, the electronic device adjusts the parameters of the initial machine learning model based on the second loss data to obtain a trained machine learning model, which is then used to generate the reconstructed image. The trained machine learning model learns the mapping relationship between interpolation features, reference viewpoint information, and interpolated Gaussian attributes. Through multiple iterations, it continuously optimizes the model parameters, enabling the model to more accurately predict interpolated Gaussian attributes. This makes the Gaussian point attributes in the reconstructed image more consistent with reality, improving the realism and quality of the reconstructed image.

[0263] Furthermore, the electronic device downsamples the interpolation points, removing some of them, redetermining the interpolation coordinates of the remaining points, and combining this with other information to generate a reconstructed image. Downsampling reduces the number of interpolation points, directly decreasing the amount of data and computational complexity, thus improving computational efficiency from a resource perspective. Simultaneously, by redetermining the interpolation coordinates and retraining, it ensures that even with fewer interpolation points, high-quality reconstructed images can still be generated by optimizing the properties of the remaining points. This enables image generation at different levels of detail, improving the flexibility and adaptability of the image generation method.

[0264] Furthermore, by using kernel density estimation to identify high-density and low-density regions within a graphic unit and downsampling them at different ratios, kernel density estimation can accurately identify the distribution density of interpolation points. It can downsample high-density regions at a larger ratio and low-density regions at a smaller ratio, thus reducing the amount of data while preserving key information about the object's surface. This further improves computational efficiency and the quality of reconstructed images at different levels of detail.

[0265] In the second stage of training, mask parameters and mask loss are introduced. Mask parameters can measure the importance of Gaussian points. By introducing mask loss into the loss function, the influence of unimportant Gaussian points will be gradually reduced during the iterative optimization process, and unnecessary small-scale Gaussian points will be eliminated in the end. This reduces the processing of irrelevant Gaussian points during the calculation process, reduces the computational complexity, improves the computational efficiency, and makes the distribution of Gaussian points more reasonable, which helps to improve the quality of the reconstructed image.

[0266] To improve the expressive power of planar triangles, position offsets are generated for interpolation points. These position offsets allow the interpolation points to be adjusted in directions outside the plane of the graphic unit, increasing the degrees of freedom of the interpolation points in three-dimensional space. This enables the interpolation points to represent more spatial information, enhances their expressive power in three-dimensional space, enriches the geometric information of the reconstructed image, and improves the accuracy of the reconstructed image in restoring the object surface.

[0267] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0268] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image generation method, performed by an electronic device, comprising: The vertex positions, vertex features, and vertex Gaussian properties of each graphics unit are obtained, and the interpolation information of each graphics unit is obtained. Each graphics unit is used to represent the surface of an object. Based on the interpolation information, the vertex position and the vertex feature are interpolated in each graphic unit to obtain the interpolation point position and interpolation feature in each graphic unit. Based on the interpolation features and information from the specified viewpoint, the interpolation Gaussian properties of each graphics unit are generated; and Rendering is performed based on the vertex position and the vertex Gaussian property, as well as the interpolation point position and the interpolation Gaussian property, to obtain a reconstructed image of the object at the specified viewpoint.

2. The method according to claim 1, wherein the interpolation information includes at least one interpolation coordinate; the step of interpolating the vertex position and the vertex feature in each graphic unit based on the interpolation information to obtain the interpolation point position and interpolation feature in each graphic unit includes: Based on the interpolated coordinates of each graphic unit, interpolation calculation is performed on the vertex position to obtain at least one interpolation point position in each graphic unit; Based on the interpolation coordinates of each graphic unit, interpolation calculations are performed on the vertex features to obtain the interpolation features of the position of at least one interpolation point.

3. The method according to claim 1 or 2, wherein the rendering process based on the vertex position and the vertex Gaussian property, and the interpolation point position and the interpolation Gaussian property, to obtain the reconstructed image of the object at the specified viewpoint, comprises: Based on the vertex Gaussian attribute and the vertex position, the vertices of each graphics unit are projected onto the image plane to obtain a first pixel. Based on the interpolation Gaussian attribute and the interpolation point position, the interpolation points within each graphics unit are projected onto the image plane to obtain a second pixel. The vertex Gaussian attribute includes the opacity and color of the vertex, and the interpolation Gaussian attribute includes the opacity and color of the interpolation point. The color of the first pixel is calculated based on the color and opacity of the vertex, and the color of the second pixel is calculated based on the color and opacity of the interpolation point; The reconstructed image is generated based on the color of the first pixel and the color of the second pixel.

4. The method according to any one of claims 1 to 3, further comprising: The initial vertex position, initial vertex Gaussian attribute, initial interpolation point position, and initial interpolation Gaussian attribute of each initial graphic unit representing the surface of the object are obtained, and rendering processing is performed based on the initial vertex position and the initial vertex Gaussian attribute, as well as the initial interpolation point position and the initial interpolation Gaussian attribute, to obtain the first training image of the object under the reference view. First loss data is generated based on the difference between the first training image and the reference image of the object under the reference view. Based on the first loss data, the initial vertex position, the initial vertex Gaussian attribute, the initial interpolation point position, and the initial interpolation Gaussian attribute are iteratively adjusted, and the vertex position, vertex Gaussian attribute, interpolation point position, and interpolation Gaussian attribute of the initial graphic unit obtained by the iterative adjustment are used to generate the reconstructed image.

5. The method according to claim 4, wherein obtaining the initial vertex position, initial vertex Gaussian attribute, initial interpolation point position, and initial interpolation Gaussian attribute of each initial graphic unit representing the surface of the object comprises: A set of feature points representing the surface of the object is obtained, and the initial vertex positions are determined based on the positions of each feature point in the set of feature points, so as to form each initial graphic unit; Based on the set interpolation information, the initial vertex positions are interpolated within each initial graphic unit to obtain the initial interpolation point positions; Initial Gaussian distribution attributes are assigned to the vertices and interpolation points in each initial graphic unit to obtain the initial vertex Gaussian attributes and the initial interpolation Gaussian attributes.

6. The method according to claim 4 or 5, further comprising: During the iterative adjustment process, the Gaussian attribute of the first vertex and the position of the first vertex after a set number of iterations are obtained. The Gaussian attribute of the first vertex includes the opacity of each vertex in each initial graphic unit, and the position of the first vertex includes the position information of each vertex. If there exists a first initial graphic unit where the opacity of all vertices is lower than a set opacity threshold, then the first initial graphic unit is deleted from the initial graphic unit representing the surface of the object. If there is a specified vertex in the second initial graphics unit whose opacity is lower than the set opacity threshold, then the position information of the specified vertex is updated based on the position information of the vertices in the second initial graphics unit whose opacity is higher than the set opacity threshold.

7. The method according to claim 6, wherein the first vertex Gaussian property further includes the scale of each vertex in multiple directions; the method further includes: Based on the scale of each vertex in multiple directions, the maximum scale corresponding to each vertex is determined, and the perimeter of each initial graphic unit is determined based on the position information of each vertex. If the sum of the maximum scales corresponding to all vertices in the third initial graphic unit is less than the perimeter of the third initial graphic unit, then the third initial graphic unit is deleted from the initial graphic unit representing the surface of the object.

8. The method according to any one of claims 4 to 7, wherein the method further comprises: During the iterative adjustment process, loss data and the second vertex position after a set number of iterations are obtained, and the second vertex position includes the position information of each vertex in each initial graphic unit; Calculate the position gradient of each vertex based on the loss data; If the position gradient of a target vertex is greater than a set gradient threshold, an initial graphics unit is added based on the position information of the target vertex.

9. The method according to any one of claims 4 to 8, wherein the method further comprises: The vertex positions, vertex Gaussian properties, and interpolation point positions of the initial graphics unit obtained through iterative adjustment are obtained. The initial vertex features of the initial graphics unit are generated based on the vertex positions obtained through iterative adjustment. The second training image of the object under the reference view is generated based on the initial vertex features and the interpolation point positions obtained through iterative adjustment. Based on the difference between the second training image and the reference image, second loss data is generated, and the initial vertex features are adjusted based on the second loss data to obtain updated vertex features; The step of using the vertex positions, vertex Gaussian properties, interpolation point positions, and interpolation Gaussian properties of the initial graphic units obtained through iterative adjustment to generate the reconstructed image includes: The vertex positions obtained through iterative adjustment, the vertex Gaussian properties obtained through iterative adjustment, and the updated vertex features are used to generate the reconstructed image.

10. The method according to claim 9, wherein the vertex positions obtained by iterative adjustment include the position information of each vertex in each initial graphics unit, and the interpolation point positions obtained by iterative adjustment include the position information of each interpolation point in each initial graphics unit; generating a second training image of the object under the reference viewpoint based on the initial vertex features and the interpolation point positions obtained by iterative adjustment includes: Based on the position information of each interpolation point and the position information of each vertex, interpolation calculation is performed on the initial vertex features to obtain the initial interpolation features of each interpolation point; Based on the initial interpolation features and the information from the reference viewpoint, the predicted interpolation Gaussian attribute of each interpolation point is determined; The second training image is obtained by rendering based on the vertex positions and Gaussian properties obtained through iterative adjustment, the interpolation point positions obtained through iterative adjustment, and the predicted interpolation Gaussian properties.

11. The method according to claim 10, wherein the step of performing interpolation calculation on the initial vertex features based on the position information of each interpolation point and the position information of each vertex to obtain the initial interpolation features of each interpolation point includes: Based on the position information of each interpolation point and the position information of the vertices of the initial graphic unit where each interpolation point is located, the interpolation coordinates of each interpolation point in the initial graphic unit are determined. Based on the interpolation coordinates of each interpolation point, the initial vertex features of the initial graphic unit where each interpolation point is located are interpolated to obtain the initial interpolation features of each interpolation point.

12. The method according to claim 11, wherein the predicted interpolation Gaussian attribute is obtained by inputting the information of the reference viewpoint and the initial interpolation features into an initial machine learning model; the method further comprises: The model parameters of the initial machine learning model are adjusted based on the second loss data to obtain the trained machine learning model; The step of using the iteratively adjusted vertex positions, iteratively adjusted vertex Gaussian properties, and updated vertex features to generate the reconstructed image includes: The vertex positions obtained through iterative adjustment, the Gaussian properties of the vertex obtained through iterative adjustment, the interpolation coordinates of each interpolation point, the updated vertex features, and the trained machine learning model are used to generate the reconstructed image.

13. The method according to claim 12, further comprising: Based on the interpolation point positions obtained through iterative adjustment, the interpolation points in each initial graphic unit are downsampled to remove some of the interpolation points in each initial graphic unit. Based on the vertex positions obtained through iterative adjustment and the position information of other interpolation points in each initial graphic unit besides the aforementioned interpolation points, the interpolation coordinates of the other interpolation points are determined; The step of using the iteratively adjusted vertex positions, iteratively adjusted vertex Gaussian attributes, interpolation coordinates of each interpolation point, updated vertex features, and trained machine learning model to generate the reconstructed image includes: The vertex positions obtained through iterative adjustment, the Gaussian properties of the vertex obtained through iterative adjustment, the interpolation coordinates of the other interpolation points, the updated vertex features, and the trained machine learning model are used to generate the reconstructed image.

14. An image generation apparatus, comprising: The acquisition unit is used to acquire the vertex position, vertex features and vertex Gaussian properties of each graphics unit, and to acquire the interpolation information of each graphics unit, wherein each graphics unit is used to represent the surface of an object. The processing unit is configured to interpolate the vertex position and the vertex feature in each graphic unit based on the interpolation information, so as to obtain the interpolation point position and interpolation feature in each graphic unit. The generation unit is used to generate the interpolation Gaussian properties of each graphics unit based on the interpolation features and information from a specified viewpoint. The processing unit is further configured to perform rendering processing based on the vertex position and the vertex Gaussian attribute, as well as the interpolation point position and the interpolation Gaussian attribute, to obtain a reconstructed image of the object at the specified viewpoint.

15. An electronic device comprising: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the electronic device to implement the image generation method according to any one of claims 1-14.

16. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method of any one of claims 1-14.

17. A computer program product comprising a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from and executes the computer program, causing the electronic device to perform the image generation method of any one of claims 1-14.

Images