Texture encoding and decoding methods and apparatus, device, storage medium, and program product

Abstract

The present application relates to the technical field of encoding and decoding, and discloses texture encoding and decoding methods and apparatus, a device, a storage medium, and a program product. The encoding method comprises: encoding a texture to obtain third feature data and a second nonlinear transformation parameter; encoding the third feature data to obtain first feature data, and encoding the second nonlinear transformation parameter to obtain a first nonlinear transformation parameter; and encoding the first feature data and the first nonlinear transformation parameter into a bit stream. The decoding method comprises: analyzing a bit stream to obtain first feature data and a first nonlinear transformation parameter; decoding the first feature data to obtain second feature data, and decoding the first nonlinear transformation parameter to obtain a second nonlinear transformation parameter; and decoding the second feature data on the basis of the second nonlinear transformation parameter to obtain a texture. According to the described method, by means of secondary compression, the texture compression rate is increased, and the occupation of network transmission bandwidth and storage space by a bit stream is reduced.

Description

Texture encoding and decoding methods, apparatus, devices, storage media, and program products

[0001] This application claims priority to Chinese Patent Application No. 202411847677.9, filed on December 13, 2024, entitled “Texture Encoding / Decoding Method, Apparatus, Device, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of encoding and decoding technology, and in particular to a method, apparatus, device, storage medium and program product for encoding and decoding textures. Background Technology

[0003] Currently, the applications of textures are becoming increasingly diverse. For example, in three-dimensional (3D) graphics and games, texture mapping can be used to render images and obtain high-quality rendering results. As the number of textures increases, the bandwidth required for network transmission and the storage space required also increase. Based on this, texture compression (also known as texture encoding / decoding) technology has emerged. The higher the texture compression ratio, the smaller the compressed bitstream, and the less bandwidth and storage space is required for network transmission. Therefore, improving texture compression ratio is a key research focus in the industry. Summary of the Invention

[0004] This application provides a texture encoding / decoding method, apparatus, device, storage medium, and program product, which can improve texture compression rate and reduce the bitstream's impact on network transmission bandwidth and storage space. The technical solution is as follows:

[0005] In a first aspect, a method for decoding a texture is provided, the method comprising: acquiring a bitstream; parsing the bitstream to obtain first feature data and a first nonlinear transformation parameter of the texture; decoding the first feature data to obtain second feature data; and decoding the first nonlinear transformation parameter to obtain a second nonlinear transformation parameter; and decoding the second feature data based on the second nonlinear transformation parameter to obtain the texture.

[0006] It can be seen that the first feature data and the first nonlinear transformation parameter in the bitstream are both data that have undergone secondary compression. The compression rate of the bitstream is very high, and the texture can be reconstructed by the decoding end through secondary decoding.

[0007] In one possible implementation, decoding the second feature data based on the second nonlinear transformation parameters to obtain the texture includes: obtaining one or more feature vectors based on the second feature data; and performing a nonlinear transformation on the one or more feature vectors according to the second nonlinear transformation parameters to obtain the texture. That is, the texture is reconstructed through feature vector construction and nonlinear transformation.

[0008] In one possible implementation, the one or more feature vectors include a second feature vector; obtaining one or more feature vectors based on the second feature data includes: obtaining a first feature vector based on the second feature data; obtaining a third feature vector, wherein the first feature vector and the third feature vector correspond to different feature maps or different feature blocks within the same feature map; and processing the first feature vector based on the third feature vector to obtain the second feature vector. That is, the first feature vector is processed using the third feature vector to assist in decoding the texture data corresponding to the first feature vector.

[0009] In one possible implementation, obtaining the third feature vector includes: obtaining auxiliary information, which indicates the third feature vector; and obtaining the third feature vector based on the auxiliary information. For example, the auxiliary information is parsed from the bitstream, and then the feature vector for auxiliary decoding is obtained according to the auxiliary information.

[0010] In one possible implementation, obtaining the third feature vector based on the auxiliary information includes: determining the feature map or feature block corresponding to the third feature vector based on the auxiliary information; and obtaining the third feature vector based on the feature map or feature block corresponding to the third feature vector.

[0011] In one possible implementation, the first feature vector corresponds to a first feature block, and the third feature vector corresponds to a second feature block. The second feature block is the feature block with the lowest complexity in the surrounding region of the first feature block, or the second feature block is a feature block in a preset region. That is, the second feature block is an auxiliary block of the first feature block. A lower-complexity auxiliary block can be used to assist in decoding the texture data corresponding to the first feature block, or an auxiliary block in a preset region can be directly used to assist in decoding the texture data corresponding to the first feature block.

[0012] In one possible implementation, before processing the first feature vector based on the third feature vector to obtain the second feature vector, the method further includes: obtaining a fourth feature vector, wherein the fourth feature vector corresponds to a different feature map or a different feature block within the same feature map as the third feature vector, and the fourth feature vector corresponds to a different feature map or a different feature block within the same feature map as the first feature vector; the processing of the first feature vector based on the third feature vector includes: processing the first feature vector based on the third feature vector and the fourth feature vector. That is, a feature block can correspond to multiple auxiliary blocks.

[0013] In one possible implementation, decoding the first feature data to obtain the second feature data includes: decoding the first feature data to obtain feature texture data; and performing texture decoding on the feature texture data to obtain the second feature data. That is, the feature texture data is first reconstructed through decoding, and then texture decoding is performed on it to reconstruct the feature map.

[0014] In one possible implementation, the feature texture data includes weight feature data and endpoint feature data corresponding to N feature maps of the texture, where N is a positive integer. That is, the feature texture data is data in a texture encoding format.

[0015] In one possible implementation, some or all of the weight feature data corresponding to the N feature maps are obtained from the weight dataset.

[0016] The weight dataset can be shared weight data, which includes weight feature data shared by N feature maps. The amount of shared weight data is less than the amount of weight feature data corresponding to the N feature maps. The shared weight data is trained during the encoding process of the texture described above. Alternatively, the weight dataset can be a preset weight dataset, which can be used for encoding and decoding one or more textures.

[0017] In one possible implementation, the first feature data includes first mapping information, which indicates the correspondence between some or all of the weight feature data corresponding to the N feature maps and the weight feature data in the weight dataset, wherein some or all of the weight feature data corresponding to the N feature maps are obtained from the weight dataset based on the first mapping information.

[0018] In one possible implementation, the weight dataset is obtained from the bitstream. For example, the first feature data includes the weight dataset, which may be shared weight data.

[0019] In one possible implementation, some or all of the endpoint feature data corresponding to the N feature maps are obtained from the endpoint dataset.

[0020] The endpoint dataset can be shared endpoint data, which includes endpoint feature data shared by N feature maps. The amount of shared endpoint data is less than the amount of endpoint feature data corresponding to the N feature maps. The shared endpoint data is trained during the encoding process of the texture described above. Alternatively, the endpoint dataset can be a pre-defined endpoint dataset, which can be used for encoding and decoding one or more textures.

[0021] In one possible implementation, the first feature data includes second mapping information, which indicates the correspondence between some or all of the endpoint feature data corresponding to the N feature maps and the endpoint feature data in the endpoint dataset. The some or all of the endpoint feature data corresponding to the N feature maps are obtained from the endpoint dataset based on the second mapping information.

[0022] In one possible implementation, the endpoint dataset is obtained from the bitstream. For example, the first feature data includes the endpoint dataset, which may be shared endpoint data.

[0023] In one possible implementation, the first feature data includes endpoint feature data and weight feature data corresponding to the first feature map of the texture; the feature texture data includes endpoint feature data and weight feature data corresponding to the second feature map of the texture, as well as endpoint feature data and weight feature data corresponding to the first feature map; the endpoint feature data of the second feature map is obtained based on the endpoint feature data corresponding to the first feature map, and / or, the weight feature data corresponding to the second feature map is obtained based on the weight feature data corresponding to the first feature map. That is, the bitstream includes relevant data from some feature maps, and the decoding end can multiplex this relevant data from some feature maps to obtain relevant data from another part of the feature maps.

[0024] Secondly, a texture encoding method is provided, the method comprising: encoding the texture to obtain third feature data and second nonlinear transformation parameters; encoding the third feature data to obtain first feature data, and encoding the second nonlinear transformation parameters to obtain first nonlinear transformation parameters; and encoding the first feature data and the first nonlinear transformation parameters into a bitstream.

[0025] In other words, the encoding end performs secondary compression on the third feature data of the texture and the second nonlinear transformation parameters, thereby improving the texture compression rate and reducing the bitstream's occupation of network transmission bandwidth and storage space.

[0026] In one possible implementation, the third feature data includes data corresponding to one or more feature maps, the one or more feature maps including a first feature map, the first feature map including a first feature block, and the second feature block being a feature block located in the same feature map as the first feature block but in a different region, or the second feature block being a feature block located in a different feature map than the first feature block. The method further includes: determining auxiliary information, the auxiliary information being used to indicate a third feature vector, the third feature vector being a feature vector corresponding to the second feature block, the third feature vector being used to construct the feature vector of the first feature block; and encoding the auxiliary information into the bitstream. That is, the encoding end can determine the second feature block as an auxiliary block corresponding to the first feature block and encode the auxiliary information into the bitstream to guide the decoding end to obtain the third feature vector to assist in decoding the texture data corresponding to the first feature block.

[0027] In one possible implementation, the auxiliary information is used to indicate the third feature vector, including: the auxiliary information is used to indicate the second feature block; and the third feature vector is obtained based on the second feature block.

[0028] In one possible implementation, determining the auxiliary information includes performing complexity detection on the texture to determine the auxiliary information. Using complexity detection allows for the allocation of more encoding bits to complex texture blocks, achieving reasonable encoding bit allocation and improving encoding quality.

[0029] In another possible implementation, both the encoder and decoder determine the auxiliary blocks according to the auxiliary block selection rules.

[0030] In one possible implementation, the second feature block is the feature block with the lowest complexity in the surrounding region of the first feature block, or the second feature block is a feature block in a preset region.

[0031] In one possible implementation, the first feature data includes first mapping information. This first mapping information indicates the correspondence between some or all of the weight feature data corresponding to the N feature maps of the texture and the weight feature data in the weight dataset. The first mapping information is used to obtain some or all of the weight feature data corresponding to the N feature maps from the weight dataset, where N is a positive integer. For a related introduction to the weight dataset, please refer to the relevant content in the first aspect.

[0032] In one possible implementation, the bitstream also includes the weight dataset.

[0033] In one possible implementation, the first feature data includes second mapping information. This second mapping information indicates the correspondence between some or all of the endpoint feature data corresponding to the N feature maps of the texture and the endpoint feature data in the endpoint dataset. The second mapping information is used to obtain some or all of the endpoint feature data corresponding to the N feature maps from the endpoint dataset, where N is a positive integer. For a related introduction to the weight dataset, please refer to the relevant content in the first aspect.

[0034] In one possible implementation, the bitstream also includes the endpoint dataset.

[0035] In one possible implementation, the first feature data includes endpoint feature data and weight feature data corresponding to a first feature map of the texture; the endpoint feature data corresponding to the first feature map is used to reconstruct the endpoint feature data corresponding to a second feature map of the texture, and / or, the weight feature data corresponding to the first feature map is used to reconstruct the weight feature data corresponding to the second feature map. That is, the encoding end encodes relevant data from some feature maps into the bitstream, further improving the texture compression rate.

[0036] Thirdly, a texture decoding apparatus is provided, which has the function of implementing the texture decoding method behavior described in the first aspect above. The decoding apparatus includes at least one module for implementing the texture decoding method provided in the first aspect above.

[0037] Fourthly, a texture encoding apparatus is provided, the apparatus having the function of implementing the texture encoding method behavior described in the second aspect above. The encoding apparatus includes at least one module for implementing the texture encoding method provided in the second aspect above.

[0038] Fifthly, a decoding device is provided, comprising a processor and a memory, the memory being used to store a computer program for executing the texture decoding method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the texture decoding method described in the first aspect.

[0039] In one possible implementation, the decoding device may further include a communication bus for establishing a connection between the processor and the memory.

[0040] In a sixth aspect, an encoding apparatus is provided, comprising a processor and a memory, the memory being used to store a computer program for executing the texture encoding method provided in the second aspect above. The processor is configured to execute the computer program stored in the memory to implement the texture encoding method described in the second aspect above.

[0041] In one possible implementation, the encoding device may further include a communication bus for establishing a connection between the processor and the memory.

[0042] In a seventh aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when the computer program is run on a computer or processor, causes the computer or processor to perform the steps of the texture decoding method described in the first aspect, and / or to perform the steps of the texture encoding method described in the second aspect.

[0043] Eighthly, a computer program product is provided, the computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the steps of the texture decoding method described in the first aspect, and / or the steps of the texture encoding method described in the second aspect. Alternatively, a computer program is provided that, when run on a computer or processor, causes the computer or processor to perform the steps of the texture decoding method described in the first aspect, and / or the steps of the texture encoding method described in the second aspect.

[0044] In a ninth aspect, an encoding and decoding system is provided, the encoding and decoding system comprising an encoding device and a decoding device, the encoding device being used to implement the steps of the texture encoding method described in the second aspect above, and the decoding device being used to implement the steps of the texture decoding method described in the first aspect above.

[0045] In a tenth aspect, an encoded bitstream is provided, the bitstream being generated according to the texture encoding method described in the second aspect above.

[0046] Eleventhly, a computer-readable storage medium is provided, the computer-readable storage medium storing a bitstream generated according to the texture encoding method described in the second aspect above.

[0047] In a twelfth aspect, an apparatus for storing a bitstream is provided, the apparatus comprising: a receiver and at least one storage medium, the receiver being configured to receive a bitstream generated according to the texture encoding method described in the second aspect above, and the at least one storage medium being configured to store the bitstream.

[0048] In a thirteenth aspect, an apparatus for transmitting a bitstream is provided, the apparatus comprising: a transmitter and a receiver, the receiver being configured to receive a bitstream generated according to the texture encoding method described in the second aspect above, and the transmitter being configured to transmit the bitstream to an end-side device via a transmission medium.

[0049] In a fourteenth aspect, an apparatus for transmitting a bitstream is provided, the apparatus comprising: a transmitter and at least one storage medium, the at least one storage medium being configured to store a bitstream generated according to the texture encoding method described in the second aspect above, the transmitter being configured to retrieve the bitstream from the storage medium and transmit the bitstream to an end-side device via the transmission medium.

[0050] In a fifteenth aspect, a system for distributing bitstreams is provided, the system comprising: at least one storage medium for storing bitstreams generated according to the texture encoding method described in the second aspect above; and a streaming media device for acquiring a target bitstream from the at least one storage medium and sending the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

[0051] The technical effects achieved by the third to fifteenth aspects mentioned above are similar to those achieved by the corresponding technical means in the first and second aspects, and will not be repeated here. Attached Figure Description

[0052] Figure 1 is a schematic diagram of a texture encoding process provided in an embodiment of this application;

[0053] Figure 2 is a schematic diagram of determining the endpoints of a texture block according to an embodiment of this application;

[0054] Figure 3 is a flowchart of a neural texture coding method provided in an embodiment of this application;

[0055] Figure 4 is a flowchart of a neural texture decoding method provided in an embodiment of this application;

[0056] Figure 5 is an architecture diagram of an image rendering system provided in an embodiment of this application;

[0057] Figure 6 is a system architecture diagram of a texture encoding and decoding scheme provided in an embodiment of this application;

[0058] Figure 7 is a flowchart illustrating a texture encoding / decoding scheme provided in an embodiment of this application;

[0059] Figure 8 is a schematic diagram of the implementation environment of a texture encoding and decoding method provided in an embodiment of this application;

[0060] Figure 9 is a schematic diagram of the structure of a client provided in an embodiment of this application;

[0061] Figure 10 is a flowchart of a texture encoding method provided in an embodiment of this application;

[0062] Figure 11 is a flowchart of another texture encoding method provided in an embodiment of this application;

[0063] Figure 12 is a flowchart of another texture encoding method provided in an embodiment of this application;

[0064] Figure 13 is a flowchart of another texture encoding method provided in an embodiment of this application;

[0065] Figure 14 is a flowchart of another texture encoding method provided in an embodiment of this application;

[0066] Figure 15 is a flowchart of another texture encoding method provided in an embodiment of this application;

[0067] Figure 16 is a flowchart of a texture decoding method provided in an embodiment of this application;

[0068] Figure 17 is a flowchart of another texture decoding method provided in an embodiment of this application;

[0069] Figure 18 is a flowchart of another texture decoding method provided in an embodiment of this application;

[0070] Figure 19 is a flowchart of another texture decoding method provided in an embodiment of this application;

[0071] Figure 20 is a flowchart of another texture decoding method provided in an embodiment of this application;

[0072] Figure 21 is a flowchart of another texture decoding method provided in an embodiment of this application;

[0073] Figure 22 is a flowchart of another texture decoding method provided in an embodiment of this application;

[0074] Figure 23 is a schematic diagram of the structure of a texture decoding device provided in an embodiment of this application;

[0075] Figure 24 is a schematic diagram of the structure of a texture encoding device provided in an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0077] To facilitate understanding, before providing a detailed explanation of the texture encoding and decoding method provided in the embodiments of this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.

[0078] First, some terms used in the embodiments of this application will be introduced.

[0079] Bitstream / File: The embodiments of this application mainly involve the encoding and decoding methods of textures (such as texture images, texture maps, etc.). The bitstream generated by encoding can be cached data stored in memory or stored as a file. There are many possible forms of existence. In the following text, only "bitstream" is used as the description object, which can be understood as "cache" or "file" and its form of existence does not affect the description scope of the embodiments of this application.

[0080] Compression: This involves utilizing information redundancy within content and removing redundant information using certain methods. In texture compression, the texture is compressed into a bitstream, the size of which is smaller than the original texture data. This bitstream can be called the original bitstream (or file; for convenience, it will be referred to as "bitstream" hereafter). In some embodiments, the original bitstream can be further compressed to obtain a new bitstream, i.e., secondary compression. Generally, the new bitstream is smaller than the original bitstream. This process of reducing the size of texture data is called "compression."

[0081] Decompression: The process of reconstructing the original texture by performing the inverse operation on the compressed bitstream is called "decompression." Specifically, for bitstreams obtained through secondary compression, performing the inverse operation on the compressed bitstream to obtain a bitstream of the same size as the original bitstream is also called "decompression." Generally, "compression" is divided into "lossless compression" and "lossy compression." Lossy compression, after decompression, reconstructs data that is completely identical to the original data, while lossy compression, after decompression, reconstructs data that is not completely identical to the original data.

[0082] Encoding: A method of expressing one set using one set by adopting certain rules. Some encoding methods can be considered as "compression".

[0083] Decoding: The inverse operation of encoding; "decompression" can be considered a form of decoding.

[0084] Texture compression is a technique used to compress textures. It can be used to store texture data in 3D computer graphics rendering systems, thereby reducing the storage space occupied by texture data. Texture compression independently encodes multiple blocks of texture (also called texture chunks) to obtain feature data in a texture encoding format. This encoding method features random access and high decoding parallelism and is supported by the vast majority of graphics cards.

[0085] A pixel is the basic unit of image display; it is a point or square that appears inseparable at any scale in an image. Each pixel can have its own color value.

[0086] A texel is short for texture element, and it is the basic unit in the texture space of computer graphics. Just as an image is composed of pixels, a texture is represented by an arrangement of texels.

[0087] Overfitting: Matching a particular dataset too closely or precisely.

[0088] Overfitting model: Compared to a limited set of data, a model with too many parameters or an overly complex structure can fit the training dataset well, but loses its generalization ability to data outside the training dataset.

[0089] Feature maps are higher-level representations extracted from raw data (such as texture). Texture feature maps characterize the texel features of the texture. Texture feature maps can be extracted using neural network models or other methods available in existing technologies.

[0090] Secondly, the application scenarios of the embodiments of this application will be introduced.

[0091] In computer graphics, texture generally refers to an image or data used to describe the details of an object's surface. This image or data can contain various information such as color, brightness, transparency, and normal vectors, which are mapped onto the surface of a 3D model (i.e., texture mapping) to simulate the realistic material and details of the object's surface.

[0092] Textures can be classified according to various factors such as their source, purpose, and storage method. They can generally include the following categories: color textures, normal textures, height textures (or displacement textures), transparency textures, gloss / reflection textures, etc.

[0093] Texture mapping is the process of applying textures to the surface of a 3D model. This typically involves associating the texture's coordinates (U, V) with the vertex coordinates (X, Y, Z) of the 3D model's surface. The U coordinates represent the texture's horizontal coordinates, usually ranging from [0, 1], indicating its horizontal position. The V coordinates represent the texture's vertical coordinates, also ranging from [0, 1], indicating its vertical position. Therefore, by specifying the U and V coordinates, an image can be precisely fitted onto the surface of a 3D object, achieving realistic material effects. Texture mapping ensures that the texture is correctly overlaid on the model's surface and maintains the correct position and proportion as the model deforms.

[0094] Therefore, textures play a crucial role in computer graphics, significantly enhancing the realism and detail of rendered scenes. Through different types of textures and texture mapping techniques, more realistic and vivid 3D models and scenes can be created.

[0095] In the field of image encoding and decoding, texture is a crucial processing object for video compression and image encoding. Effectively processing and encoding textures can improve the efficiency and quality of video compression and image encoding, thereby meeting the needs of various application scenarios. Specifically, texture encoding / decoding techniques or neural texture encoding / decoding techniques can be employed when encoding and decoding textures.

[0096] Next, we will give a brief introduction to these two encoding and decoding technologies.

[0097] 1. Texture encoding and decoding technology

[0098] Image encoding and decoding technologies (such as the Joint Photographic Experts Group (JPEG, a lossy image compression format) and Portable Network Graphics (PNG, a lossless image compression format) primarily focus on the compression and decompression of the overall image, rather than emphasizing fast random access to individual pixels. In other words, image encoding and decoding algorithms typically prioritize overall image quality, compression ratio, and decoding speed over the efficiency of accessing individual pixels. Textures, on the other hand, are usually mapped onto the surface of 3D models to simulate the details and materials of object surfaces. Therefore, during rendering, fast random access to any pixel (also called a texel in computer graphics) within the texture is frequently required. This means that encoding and decoding algorithms need to support the efficient retrieval of arbitrary texel values ​​from compressed data.

[0099] In short, compared to traditional image encoding and decoding technologies, texture encoding and decoding technologies need to support fast random access to any pixel (i.e., texel) in the texture.

[0100] To meet the requirements of fast random access, texture encoding techniques typically employ a block-based fixed-length encoding method. Referring to Figure 1, the original texture is divided into multiple fixed-size (e.g., M*N texels) texture blocks, and each block is then encoded into compressed data of a fixed number of bits to encode the original texture. Based on this, during decoding, the offset can be calculated based on the texture block number to be accessed and the compressed size of each block. This allows for quick location and decoding of the required texture block through simple offset addressing, thus achieving fast random access to any texel value of the original texture data.

[0101] For each texture block, existing texture encoding methods (such as adaptive scalable texture compression (ASTC), block compression (BC), etc.) perform intra-block encoding by storing endpoints, weights, and other encoding patterns within the texture block. Endpoints typically represent the maximum and minimum values ​​of texels within the texture block (e.g., extreme values ​​of color or brightness), while weights describe the changes in texel values ​​at different locations within the texture block (e.g., changes in color or brightness). The weights, combined with the endpoints, can recover each texel value of the original texture.

[0102] It should be noted that for each texture block, the color or brightness change of the texels within the block is usually close to a linear change. Therefore, color endpoints and interpolation weights can be recorded during encoding.

[0103] Taking texel color variation as an example, as shown in Figure 2, for each texture block, two or more key colors can be selected as endpoints based on its color variation. Then, using the endpoint colors as a reference, the weight value of each texel in the texture block is determined according to the relationship between the endpoint colors and each texel. The weight can be a scalar value, representing how close the actual texel value is to a certain endpoint, or it can be a vector value, representing the distribution of pixels across multiple color dimensions (e.g., RGB).

[0104] As an example, for a texture block containing a brick wall texture, by analyzing the color distribution within the texture block, four endpoint colors can be selected: khaki, light brown, gray, and dark brown. For each texel within the texture block, its weight value relative to these four endpoints (khaki, light brown, gray, and dark brown) is calculated. For example, a texel closer to light brown has a higher light brown weight, and a texel closer to dark brown has a higher dark brown weight.

[0105] Furthermore, the decoding process for the texture block can be as follows: First, decode the endpoint and weight information. Then, based on the weight and endpoint information, calculate the value of each texel of the texture block through interpolation. Finally, reconstruct the complete texture block.

[0106] As an example, for the texels of a texture block, the decoded texel value can be determined by the following formula (1): t=weight*endpoint1+(c-weight)*endpoint2 (1)

[0107] Where t is the decoded texel value, weight is the interpolation weight corresponding to the texel value, endpoint1 and endpoint2 are the two color endpoints of the texture block, and c is a constant or another parameter value related to the weight, such as c = 1.

[0108] Based on the above explanation, the texture encoding techniques described above have the following limitations when encoding textures:

[0109] (1) Since the interpolation encoding within a block is relatively simple, the compression rate of texture encoding technology is limited. Under the same image quality, more storage space may be needed to save the compressed texture data.

[0110] (2) The above texture encoding techniques can usually only encode each texture independently. When multiple textures need to be encoded at the same time, the encoding efficiency is relatively low. Moreover, there is redundancy and unnecessary overhead when storing or transmitting these texture data.

[0111] 2. Neural texture encoding and decoding technology

[0112] Neural texture encoding / decoding is a technique that uses neural network models to compress and decompress texture data, suitable for processing multiple textures with similar properties. Based on the idea of ​​neural network overfitting, this technique aims to overcome the shortcomings of traditional texture encoding / decoding techniques and provide a more efficient and accurate encoding / decoding solution.

[0113] The data to be encoded can be one or more textures, and the encoding result is one or more feature maps and one or more sets of neural network parameters. A texture can have one or more feature maps and one set of neural network parameters. Since the total data volume of the feature maps and neural network parameters is less than the data volume of the data to be encoded, compression of the data to be encoded is achieved through neural texture encoding.

[0114] Referring to Figure 3, the neural texture encoding process may include the following steps (11)-(13).

[0115] (11) Initialize feature data and network parameters.

[0116] Based on the encoding task settings and the structure of the neural network model, feature data and network parameters are initialized. The selection of feature data depends on the specific texture type and encoding requirements. During initialization, these feature data are typically set to random values ​​or initial values ​​based on some prior knowledge. Network parameters define the architecture of the neural network model and the parameters of the network layers. During initialization, the network layer parameters of the neural network model are typically set to small random values ​​to ensure the diversity and stability of the network during the learning process. The architecture of the neural network model (e.g., the number of network layers, the number of neurons per layer, etc.) is designed according to the specific encoding task.

[0117] (12) Neural network training.

[0118] During the training phase of a neural network model, feature data and network parameters are used as inputs. The neural network model then performs inference to obtain the inference result, which is a representation or prediction of the data to be encoded.

[0119] During inference, the neural network model performs nonlinear transformations on the input feature data through its internal connections and network parameters, progressively extracting higher-level feature representations. These representations are then transformed into the inference result in the final layer of the neural network model. The difference between the inference result and the data to be encoded is then measured using a loss function. The loss function is a mathematical expression used to calculate the error between the inference result and the actual data to be encoded; common loss functions include mean absolute error (MAE), mean squared error (MSE), and cross-entropy loss. Further, based on the calculated loss, gradient backpropagation and parameter updates are performed. That is, based on the calculated loss, the gradient of each network parameter in the neural network model is calculated using the gradient backpropagation algorithm (also known as the backpropagation algorithm). These gradients indicate how the parameters should be adjusted to reduce the loss. Then, optimization algorithms (such as stochastic gradient descent, adaptive moment estimation (Adam) algorithm, etc.) are used to update the network parameters and feature data of the neural network model to minimize the loss. The training process repeats the above steps until a certain exit condition is met. This exit condition can be that the loss is less than a certain threshold, or that the loop is repeated a certain number of times (called an epoch).

[0120] (13) Generate the bitstream.

[0121] After training, the training results are exported as the encoded bitstream. These training results include the trained feature data and network parameters. After training, the feature data already contains important information about the data to be encoded and is represented in a more compact and efficient way; the network parameters define the neural network architecture and network layer parameters used to reconstruct the data to be encoded. During the decoding phase, these parameters will be used to guide the inference process of the neural network model to recover the original texture.

[0122] It should be noted that the bitstream generated by the aforementioned neural network model contains sufficient information to allow for the recovery of all or part of the information of the data to be encoded through appropriate decoding steps during the decoding phase. The decoding process typically involves using the same neural network architecture and trained network parameters to infer the feature data in the bitstream in order to reconstruct the original texture data.

[0123] Referring to Figure 4, the neural texture decoding process may include the following steps (21)-(23).

[0124] (21) Bitstream splitting.

[0125] First, two key data components are extracted from the received bitstream (i.e., the encoded data stream): feature data and network parameters. As explained earlier, feature data is a compact representation of the texture obtained after some form of encoding. It contains enough information to recover the original texture details during decoding. Network parameters contain the structure and layer parameters of the neural network model used for decoding the feature vectors. Since the neural network model is optimized by continuously learning the texture features during the training phase, its network parameters can reflect the statistical regularities and structural characteristics of the texture data.

[0126] (22) Generate feature vectors.

[0127] During the decoding process, in order to reconstruct the texture or each texel in the texture, it is necessary to extract the corresponding feature values ​​from the feature data based on the coordinates of the sampling point (i.e., texel). These extracted feature values ​​are combined into a feature vector, which is then used as the input to the neural network model.

[0128] It should be understood that when extracting feature values ​​from feature data based on the coordinates of sampling points, a spatial mapping or indexing mechanism is involved to map the coordinates of the sampling points into the space of the feature data.

[0129] (23) Determine the decoding value of the texel through neural network model reasoning.

[0130] The decoding end obtains a neural network model identical to the neural network model trained by the encoding end according to the network parameters in the bitstream, and inputs the feature vector into the neural network model to perform inference (or backpropagation) on the input feature vector to calculate the decoded value of each texel. The decoded value represents the color, brightness or other attributes of the corresponding texel in the original texture.

[0131] In summary, neural texture encoding and decoding technology, by combining the learning capabilities of neural network models with the characteristics of texture data, can achieve efficient texture data compression and reconstruction.

[0132] As an example, suppose there is an image set containing multiple texture maps that need to be compressed and stored. Neural texture encoding / decoding techniques can be used to transform these multiple texture maps into a set of feature maps and a set of network parameters. The total data volume of these transformed data is smaller than the original data stream of the multiple texture maps. Furthermore, the encoded feature maps and neural network parameters are stored or transmitted. Due to the smaller data volume, storage space or transmission time can be saved. When needed, the encoded feature maps can be decoded based on the network parameters to reconstruct the original multiple texture maps.

[0133] Based on this, embodiments of this application provide a texture encoding and decoding method that can utilize the overfitting capability of neural network models and the data correlation within a texture or between multiple textures to encode one or more textures into a set of feature data and network parameters, and further encode the feature data and network parameters to achieve texture data compression, saving data loading bandwidth and data storage space.

[0134] It should be noted that, in addition to using a neural network model to perform nonlinear transformation on the feature vector to reconstruct the texture, other methods can also be used to perform nonlinear transformation on the feature vector, such as using a nonlinear function. The embodiments of this application do not limit the specific implementation of the nonlinear transformation.

[0135] The texture encoding / decoding scheme provided in this application can be applied to at least image rendering and image display scenarios. For ease of understanding, the following sections will describe these two application scenarios and the implementation logic of the technical solution in these scenarios.

[0136] 1. Image rendering scene

[0137] Image rendering primarily refers to the process of generating visual effects in 3D games and various graphics applications. During rendering, texture mapping technology is widely used to give objects rich surface details and realism. Texture maps act like the "skin" of an object; by mapping them onto the object's geometric surface, they can simulate the appearance of various materials (such as wood, metal, and fabric), as well as complex patterns and color variations. In other words, high-quality rendering results often rely on a large number of detailed texture maps containing rich detail information, making the final image more realistic.

[0138] Because texture maps typically contain a large amount of texel data, loading texture maps during the image rendering process requires significant data transfer bandwidth, especially in high-resolution and complex scenes, which can lead to prolonged loading times and negatively impact user experience. Furthermore, high-quality texture maps also mean larger file sizes, which undoubtedly increases the storage burden for games and applications that require installation packages. Particularly for mobile games, excessively large installation packages may reduce user download willingness and affect the product's market performance. Therefore, texture encoding methods (such as ASTC, BC, etc.) typically encode and store texture maps, i.e., storing compressed texture data.

[0139] Referring to Figure 5, during the image rendering process, the texture compression data needs to be loaded from the storage device (such as a hard disk or solid-state disk / drive, SSD) into memory, and then transferred to the graphics processing unit (GPU) for texture decoding to obtain texture data. Then, based on the geometric data and texture data, rendering is performed to obtain the rendering result, i.e., the rendered image.

[0140] Based on this, the texture encoding and decoding scheme provided in this application embodiment can be used to perform texture encoding and decoding on texture maps. Referring to Figure 6, the texture encoding and decoding system of this application embodiment includes a texture generation platform and a client. The texture generation platform can be a cloud-based server or terminal, used to generate / acquire texture maps, encode textures, and encapsulate them into a bitstream for transmission. The client is a chip with storage and processing capabilities, or a computer device, used to perform decoding tasks, rendering tasks, and display the results. This client can be a mobile phone, personal computer (PC), virtual reality (VR) glasses, augmented reality (AR) glasses, or other media products.

[0141] The texture generation platform can use any of the texture encoding methods shown in Figures 10 to 15 below to encode the texture map; at the same time, the client can use any of the texture decoding methods shown in Figures 16 to 22 below to decode the received bitstream to obtain texture data.

[0142] In one possible implementation, referring to Figure 7, the texture encoding, texture decoding, and rendering processes described above can be implemented on the client side. That is, the texture encoding system only includes the client, which independently completes the texture encoding scheme without transmitting the bitstream between the client and the texture generation platform. The implementation logic of the texture encoding, texture decoding, and rendering processes is the same as that in Figures 5 and 6, and will not be repeated here.

[0143] 2. Image display scenario

[0144] Image display refers to the process of processing image data and then transmitting it to a display screen for display. This can be applied in scenarios such as application (APP) image display, wallpaper display, video surveillance, and gaming. In the APP image display scenario, users typically view and browse images in mobile applications (such as social media and image browsers). The APP loads and decodes the image data and transmits it to the display screen to meet the user's visual needs. In the wallpaper display scenario, if a user wants to set a personalized wallpaper on their computer or mobile device desktop, the wallpaper image is decoded and processed before being transmitted to the display screen for display, providing the user with an aesthetically pleasing visual experience.

[0145] For the image encoding and decoding process described above, traditional image encoding and decoding technologies such as JPEG, PNG, and WebP (an image file format that provides both lossy and lossless / reversible compression) can typically be used to store image data. However, when these images need to be displayed in an app, since the GPU usually does not directly support hardware decoding of these compressed formats, the decoding process must be performed externally to the GPU, i.e., on the central processing unit (CPU), before the decoded image data is transmitted to the GPU for rendering and display. Therefore, traditional image encoding and decoding technologies are often computationally complex and require a long processing time, resulting in a high end-to-end latency from user request to actual display on the screen. In some extreme cases, users may see a blank screen, which significantly degrades the user experience. Moreover, since the decoding process is performed on the CPU, which is generally less efficient than the GPU in processing image data, it increases the device's power consumption. Furthermore, the need to upload the decoded image data to the GPU also consumes a significant amount of data upload bandwidth, further increasing power consumption.

[0146] To reduce CPU load and bandwidth consumption during data uploads, texture compression formats designed specifically for GPUs, such as ASTC and the BC series, can be used to encode images. These formats allow image data to be uploaded directly to the GPU in compressed form, where it is decoded and rendered without CPU assistance. While texture compression formats are more efficient for decoding and rendering on the GPU, their compression ratios are typically lower than common image compression formats like JPEG, PNG, and WebP. This means that images stored using texture compression formats will occupy more storage space, increasing the app's file size. For users, a larger file size may reduce their willingness to download and use the app. Furthermore, if image quality is compromised due to compression, it can also negatively impact the user's visual experience.

[0147] Based on this, the texture encoding and decoding scheme provided in this application embodiment can be used to perform texture encoding and decoding on the image to be displayed. The texture encoding and decoding method provided in this application embodiment can be executed on the CPU, or on the GPU, or partly on the CPU and partly on the GPU; this application embodiment does not limit this. When performing texture encoding, texture decoding, and rendering on the image to be displayed, the system architecture can refer to Figures 5-7 above, the only difference being that the encoding object "texture map" in Figures 5-7 is replaced with "the image to be displayed," the rest of the implementation logic is the same, and therefore will not be repeated here.

[0148] Finally, the implementation environment of the embodiments of this application will be described.

[0149] Referring to Figure 8, the implementation environment of the texture encoding / decoding scheme provided in this embodiment includes: a source device 10, a destination device 20, a link 30, and a storage device 40. The source device 10 can generate an encoded image, i.e., a bitstream. Therefore, the source device 10 can also be called an encoding device. The destination device 20 can decode the bitstream generated by the source device 10. Therefore, the destination device 20 can also be called a decoding device. The link 30 can receive the encoded image generated by the source device 10 and transmit the encoded image to the destination device 20. The storage device 40 can receive the encoded image generated by the source device 10 and store the encoded image. Under these conditions, the destination device 20 can directly obtain the encoded image from the storage device 40. Alternatively, the storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded image generated by the source device 10. Under these conditions, the destination device 20 can stream or download the encoded image stored in the storage device 40.

[0150] Both the source device 10 and the destination device 20 may include one or more processors and memory coupled to the one or more processors. This memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other media that can be used to store desired program code in the form of computer-accessible instructions or data structures. For example, the source device 10 may be a server cluster or distributed system composed of multiple physical servers, or it may be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud computing service center. Destination device 20 may include mobile phones, smartphones, personal digital assistants (PDAs), wearable devices, pocket PCs (PPCs), tablets, smart car systems, smart TVs, smart speakers, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, and other terminals or similar devices.

[0151] Link 30 may include one or more media or devices capable of transmitting encoded images from source device 10 to destination device 20. In one possible implementation, link 30 may include one or more communication media enabling source device 10 to directly transmit encoded images to destination device 20 in real time. In this embodiment, source device 10 may modulate the encoded image based on a communication standard, such as a wireless communication protocol, and transmit the modulated image to destination device 20. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, wide area network, or global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices facilitating communication from source device 10 to destination device 20, etc., which are not specifically limited in this embodiment.

[0152] In one possible implementation, storage device 40 can store the received encoded image sent by source device 10, and destination device 20 can directly retrieve the encoded image from storage device 40. Under such conditions, storage device 40 can include any of a variety of distributed or locally accessed data storage media. For example, any of these distributed or locally accessed data storage media can be a hard disk drive, Blu-ray disc, digital versatile disc (DVD), compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing bitstreams.

[0153] In one possible implementation, storage device 40 may correspond to a file server or another intermediate storage device that can store the bitstream generated by source device 10, and destination device 20 may stream or download the images stored on storage device 40. The file server can be any type of server capable of storing encoded images and sending them to destination device 20. In one possible implementation, the file server may include a web server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive, etc. Destination device 20 can acquire the encoded images via any standard data connection (including an Internet connection). Any standard data connection may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both suitable for acquiring encoded images stored on a file server. The transmission of encoded images from storage device 40 may be streaming, downloading, or a combination of both.

[0154] It should be noted that the implementation environment shown in Figure 8 is only one possible implementation method, and the technology of this application embodiment can be applied not only to the source device 10 that can encode images and the destination device 20 that can decode encoded images shown in Figure 8, but also to other devices that can encode images and decode bitstreams. This application embodiment does not specifically limit this.

[0155] In the implementation environment shown in Figure 8, source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, output interface 140 may include a modem / demodulator and / or a transmitter, wherein the transmitter may also be referred to as a transmitter. Data source 120 may include an image capture device (e.g., a camera, etc.), an archive containing previously captured images, a feed interface for receiving images from an image content provider, and / or a computer graphics system for generating images, or a combination of these sources of images.

[0156] In this embodiment, the data source 120 can send images to the encoder 100, which can encode the received images to obtain an encoded image. The encoder can then send the encoded image to an output interface. In some embodiments, the source device 10 directly sends the encoded image to the destination device 20 via the output interface 140. In other embodiments, the encoded image can also be stored on the storage device 40 for later retrieval by the destination device 20 for decoding and / or display.

[0157] In the implementation environment shown in Figure 8, the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive encoded images via link 30 and / or from storage device 40, and then send them to the decoder 200, which may decode the received encoded images to obtain decoded images. The decoder may send the decoded images to the display device 220. The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20. Generally, the display device 220 displays the decoded images. The display device 220 may be any type of display device, for example, a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0158] It should be understood that, although not shown in Figure 8, in some respects, encoder 100 and decoder 200 may be integrated with each other and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or separate data streams. In some embodiments, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP), if applicable.

[0159] Encoder 100 and decoder 200 may each be any of the following circuits: one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the techniques of the embodiments of this application are implemented in part in software, the apparatus may store instructions for software in a suitable non-volatile computer-readable storage medium, and one or more processors may be used to execute the instructions in hardware to implement the techniques of the embodiments of this application. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) may be considered as one or more processors. Each of encoder 100 and decoder 200 may be included in one or more encoders or decoders, and either encoder or decoder may be integrated as part of a combined encoder / decoder (encoder-decoder) in the respective apparatus.

[0160] In this application embodiment, encoder 100 may be generally referred to as an apparatus that “signals” or “sends” certain information to, for example, decoder 200. The terms “signals” or “sends” may generally refer to the transmission of syntax elements and / or other data for decoding a compressed image. This transmission may occur in real time or nearly in real time. Alternatively, this communication may occur after a period of time, for example, during encoding when syntax elements are stored in a computer-readable storage medium in a encoded bitstream, and the decoding apparatus may then retrieve the syntax elements at any time after they have been stored in this medium.

[0161] The texture encoding / decoding method provided in this application embodiment can be applied to various scenarios and system architectures. Taking the system architecture shown in Figure 5 as an example, the images to be encoded and decoded can be textures in image files or textures in video files. It should be noted that, in conjunction with the implementation environment shown in Figure 8, any texture encoding method described below can be executed by the encoder 100 in the source device 10. This encoder 100 is implemented by software, hardware, or a combination of both, becoming part or all of the texture generation platform in this application embodiment. Similarly, any texture decoding method described below can be executed by the decoder 200 in the destination device 20. This decoder is implemented by software, hardware, or a combination of both, becoming part or all of the client in this application embodiment.

[0162] For example, the texture generation platform can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud computing service center. This cloud platform can implement the texture encoding method provided in any of the embodiments shown in Figures 10 to 15 below.

[0163] Please refer to Figure 9, which is a schematic diagram of a client according to an embodiment of this application. The client includes at least one processor 901, a communication bus 902, a memory 903, and at least one communication interface 904. The terminal has a certain image rendering capability and can render scene data to obtain intermediate rendering results.

[0164] The processor 901 can be a general-purpose central processing unit (CPU), GPU, network processor (NP), microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0165] The communication bus 902 is used to transmit information between the aforementioned components. The communication bus 902 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 9, but this does not mean that there is only one bus or one type of bus.

[0166] The memory 903 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 903 may exist independently and be connected to the processor 901 via a communication bus 902. Alternatively, the memory 903 may be integrated with the processor 901.

[0167] Communication interface 904 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 904 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0168] In a specific implementation, as one example, processor 901 may include one or more CPUs, such as CPU0 and CPU1 as shown in FIG9.

[0169] In a specific implementation, as one example, the client may include multiple processors, such as processor 901 and processor 905 as shown in FIG9. Each of these processors may be a single-core processor or a multi-core processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0170] In a specific implementation, as one example, the client may further include an output device 906 and an input device 907. The output device 906 communicates with the processor 901 and can display information in various ways. For example, the output device 906 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 907 communicates with the processor 901 and can receive user input in various ways. For example, the input device 907 may be a mouse, a keyboard, a touchscreen device, or a sensing device, etc.

[0171] In some embodiments, memory 903 is used to store program code 910 for executing the scheme of this application, and processor 901 can execute the program code 910 stored in memory 903. The program code 910 may include one or more software modules, and the client can use processor 901 and program code 910 in memory 903 to implement the texture decoding method provided in any of the embodiments of FIG16 to FIG22 below, and / or implement the texture encoding method provided in any of the embodiments of FIG10 to FIG15.

[0172] The application scenarios and system architectures described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the emergence of new business scenarios, as well as the development of hardware devices and the upgrading of system architecture, the technical solutions provided in this application are also applicable to similar technical problems.

[0173] Next, the text encoding and decoding methods provided in the embodiments of this application will be explained in detail.

[0174] The encoding and decoding method provided in this application can encode the feature data and nonlinear transformation parameters of the texture into the bitstream after secondary compression, thereby improving the compression ratio, reducing the bitstream, and alleviating the occupation of network bandwidth and storage space by texture data.

[0175] Figure 10 is a flowchart of a texture encoding method provided in an embodiment of this application. The method is applied to the encoding end. Please refer to Figure 10. The method includes the following steps.

[0176] Step 1001: Encode the texture to obtain the third feature data and the second nonlinear transformation parameters.

[0177] The second nonlinear transformation parameter is used to reconstruct the texture, and the third feature data and the second nonlinear transformation parameter can be determined based on neural texture encoding and decoding techniques. For example, they can be obtained by training a neural network model. The training method is described in the section on neural texture network encoding and decoding techniques described above. For instance, the third feature data and the network parameters of the neural network model are determined through iterative updates; these network parameters are the second nonlinear transformation parameters. The third feature data and the nonlinear transformation parameter can also be determined based on other possible methods similar to neural texture encoding and decoding techniques, such as replacing the neural network model with another nonlinear function and determining the parameters of the third feature data and the nonlinear function through iterative updates; these parameters are the second nonlinear transformation parameters.

[0178] In this embodiment, the texture is encoded through iterative updates to obtain third feature data and second nonlinear transformation parameters. That is, iteratively updating the third feature data makes it more accurately represent the texture, and updating the second nonlinear transformation parameters improves the texture reconstruction effect. This will be described in detail below.

[0179] Before the first iteration, the third feature data and the second nonlinear transformation parameter are initialized, i.e., the initial values ​​of the third feature data and the second nonlinear transformation parameter are determined. The initialization method can be referred to the relevant content in Figure 3. As an example, the third feature data can be initialized as random noise. Alternatively, the third feature data can be obtained through other encoding methods, such as processing the texture using a neural network model, or obtaining the third feature data through texture compression. Similarly, the initial value of the second nonlinear transformation parameter can also be obtained through initialization, such as initializing the second nonlinear transformation parameter as random noise, or obtaining the second nonlinear transformation parameter through other methods, such as manual setting. This application does not limit the initialization method of the third feature data and the second nonlinear transformation parameter.

[0180] In the first iteration, the encoder reconstructs the texture based on the initialized third feature data and second nonlinear transformation parameters. The encoding loss for this iteration is determined based on the reconstructed texture and the original texture. In each subsequent iteration, the encoder updates the third feature data and second nonlinear transformation parameters of the texture based on the encoding loss of the previous iteration. Then, it continues to reconstruct the texture based on the current third feature data and second nonlinear transformation parameters. The encoding loss for this iteration is determined based on the reconstructed texture and the original texture. The iteration ends when the iteration exit condition is met. The third feature data and second nonlinear transformation parameters in the last iteration are the same as those in step 1001.

[0181] In one implementation, the iteration exit condition includes an iteration count threshold (referred to as the count threshold). If the number of updates to the third feature data (i.e., the number of iterations) reaches the count threshold after the current update of the third feature data and the second nonlinear transformation parameter, the iteration ends.

[0182] In another implementation, the iteration exit condition includes a loss threshold; if the encoding loss obtained during the current iteration does not exceed the loss threshold, the iteration ends.

[0183] In another implementation, the iteration exit condition includes a loss threshold and an iteration count threshold. If the number of updates to the third feature data (i.e., the number of iterations) after updating the third feature data and the second nonlinear transformation parameters has not reached the iteration count threshold, then the texture is reconstructed based on the current third feature data and the second nonlinear transformation parameters. Based on the reconstructed texture and the original texture, the encoding loss of this iteration is determined. If the encoding loss obtained in this iteration does not exceed the loss threshold, the iteration ends. That is, the iteration can be terminated early before reaching the iteration count threshold.

[0184] In addition to the iteration exit conditions exemplified above, other iteration exit conditions may also exist, and this application embodiment does not limit such conditions.

[0185] In some embodiments, a secondary compression step may be added after the iteration. Figure 11 is a flowchart of another texture encoding method provided by an embodiment of this application. Referring to Figure 11 and the relevant description of the above embodiments, in the encoding process shown in Figure 11, the texture is reconstructed based on the third feature data and the second nonlinear transformation parameter obtained in the current iteration process. This includes: decoding the third feature data in the current iteration process to obtain reconstructed feature data of the texture; constructing multiple feature vectors based on the reconstructed feature data using feature vectors; performing a nonlinear transformation on the multiple feature vectors according to the second nonlinear transformation parameter to reconstruct the texture. After the iteration, steps 1002 and 1003 are executed, namely, encoding the current third feature data to obtain first feature data, and encoding the current second nonlinear transformation parameter to obtain first nonlinear transformation parameter; and encoding the first feature data and the first nonlinear transformation parameter into the bitstream.

[0186] In other embodiments, a secondary compression step can be directly added to the above iteration process, thereby determining the reconstruction effect after secondary compression through coding loss. This will be explained in conjunction with Figure 12.

[0187] Figure 12 is a flowchart of another texture encoding method provided in an embodiment of this application. Referring to Figure 12, the encoding end reconstructs the texture based on the third feature data and the second nonlinear transformation parameter in the current iteration process, including: encoding (i.e., compressing) the third feature data and the second nonlinear transformation parameter in the current iteration process to obtain the first feature data and the first nonlinear transformation parameter in the current iteration process; decoding the first feature data and the first nonlinear transformation parameter in the current iteration process to obtain feature texture data (also called texture feature data, or texture feature map, etc.) and the reconstructed second nonlinear transformation parameter; and reconstructing the texture based on the feature texture data and the reconstructed second nonlinear transformation parameter. It should be understood that if the third feature data is regarded as feature data obtained through one encoding, then the first feature data can be regarded as feature data obtained through two encodings.

[0188] There are various ways to encode the third feature data and the second nonlinear transformation parameter, such as entropy coding, vector quantization, or one or more other methods. This application embodiment does not limit this method. The method of encoding the third feature data can be the same as or different from the method of encoding the second nonlinear transformation parameter. This application embodiment does not limit this method. The method of decoding the third feature data and the second nonlinear transformation parameter corresponds to (i.e., matches) the method of encoding them. If the encoding method is entropy coding, then the corresponding decoding method is entropy decoding. If the encoding method is vector quantization, then the corresponding decoding method is inverse vector quantization.

[0189] The process of reconstructing the texture based on the feature texture data and the reconstructed second nonlinear transformation parameters includes: decoding the feature texture data to obtain reconstructed feature data, and performing a nonlinear transformation on the reconstructed feature data according to the second nonlinear transformation parameters to reconstruct the texture. This nonlinear transformation can be implemented based on a neural network model or a nonlinear transformation function, etc.

[0190] The aforementioned encoding loss can be obtained by calculating the mean absolute error (MAE), mean square error (MSE), or cross-entropy loss between the reconstructed texture and the original texture, or by other means. This application does not limit this method.

[0191] In the embodiments of this application, the third feature data can be implemented in various ways, which will be described in detail below.

[0192] The first implementation of the third feature data includes weight feature data and endpoint feature data corresponding to each feature map of the texture.

[0193] That is, the third feature data is data in texture encoding format. Among them, the endpoint feature data represents the range of texel values ​​of the texture, and the weight feature data is used to determine the texel values ​​of the texture by combining the range of texel values.

[0194] It should be understood that the initial value of the third feature data can be determined according to the texture encoding format during initialization. The texture can have N feature maps, where N is a positive integer. The texture encoding format can be ASTC, BC, or other texture encoding formats; this application does not limit this.

[0195] The second implementation of the third feature data includes shared weight data (a weight dataset) and endpoint feature data corresponding to each feature map of the texture.

[0196] The shared weight data includes weighted feature data shared by all feature maps of the texture. This shared weight data can be viewed as a shared weight pool (or simply the weight pool). It should be understood that the shared weight data is data that is progressively optimized during the aforementioned iterative update process.

[0197] In one possible implementation, the total number of weight feature data included in the shared weight data is less than the total number of weight feature data corresponding to the N feature maps of the texture. Thus, after executing steps 1002 and 1003, the bitstream size will be smaller than the bitstream size in the first implementation, meaning the second implementation has a higher compression ratio.

[0198] As an example, the shared weight data includes T weight feature data, and each feature map in the N feature maps corresponds to B weight feature data, where T and B are both positive integers greater than 1, and T is less than N×B.

[0199] As can be seen, texture compression is further achieved by setting a shared weight pool with a smaller data volume. The shared weight pool is equivalent to directly removing some redundant weight feature data during the training process.

[0200] In one possible implementation, the third feature data also includes first mapping information, which indicates the correspondence between each weight feature data in the shared weight data and each weight feature data of the texture. That is, the correspondence between each weight feature data corresponding to the above N feature maps and each weight feature data in the shared weight data. In this way, it is convenient for the decoding end to reconstruct the weight feature data corresponding to each feature map of the texture based on the shared weight data.

[0201] Each of the N feature maps includes S feature blocks, and the first mapping information indicates the correspondence between the weight feature data corresponding to each feature block in the N feature maps and the weight feature data in the shared weight data.

[0202] Let T be the total number of weighted feature data in the shared weighted data. If a feature block has one weighted feature data, then T is less than N×S. If a feature block has g weighted feature data, where g is an integer greater than 1, then T is less than N×S×g.

[0203] The first mapping information can be in key-value form. Each key in the first mapping information represents the identifier of each weight feature data in the shared weight data, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the N feature maps.

[0204] The first mapping information can be in key-value form. Each key in the first mapping information represents the identifier of each weight feature data in the shared weight data, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the N feature maps.

[0205] As an example, taking N = 3, S = 20, g = 4, and T = 50 as an example, the N feature maps have a total of 3 × 20 × 4 = 240 weight feature data. The first mapping information can include {1: [1~24, 45, 47]; 2: [25~34, 38, 56]; 3: [79, 94~100, 240]; ...; 50: [35, 194~239]}. Among them, the serial numbers "1, 2, 3, ... 50" before the ":" are 50 keys, representing the identifiers of 50 weight feature data in the shared weight data, and the serial numbers in "[]" are values. The 50 "[]" contain a total of 240 values, representing the identifiers of the 240 weight feature data corresponding to the N feature maps. In this example, the first mapping information indicates that the first weight feature data in the shared weight data can be used as the 1st to 24th, 45th, and 47th weight feature vectors corresponding to the N feature maps; the second weight feature data in the shared weight data can be used as the 25th to 34th, 38th, and 56th weight feature vectors corresponding to the N feature maps; the third weight feature data in the shared weight data can be used as the 79th, 94th to 100th, and 240th weight feature vectors corresponding to the N feature maps; and the first weight feature data in the shared weight data can be used as the 35th, 194th to 239th weight feature vectors corresponding to the N feature maps.

[0206] Alternatively, the first mapping information can be in key-value form, where each key in the first mapping information represents the identifier of each weight feature data corresponding to N feature maps, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the shared weight data.

[0207] As an example, the first mapping information includes {[1:2];[2:25];[3:50];...;[240:48]}, where the serial numbers "1, 2, 3, ... 240" before the ":" are 240 keys, representing the identifiers of 240 weighted feature data corresponding to N feature maps, and the serial numbers after the ":" are values, with values ​​ranging from 1 to 50, representing the identifiers of 50 weighted feature data in the shared weight data.

[0208] Besides the examples above, the structure of the first mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the keys and values ​​in the above examples, another type of first mapping information can be obtained.

[0209] It should be understood that the first mapping information can be learned through training multiple times to obtain shared weight data. Alternatively, the first mapping information can be a pre-defined, fixed relationship, rather than something learned through training. In this case, the first mapping information can also be encoded into the bitstream to ensure that the decoder can obtain the correspondence between the weight feature data corresponding to each feature block in the N feature maps and the weight feature data in the shared weight data. Furthermore, the first mapping information in the third feature data can be unencoded information.

[0210] In another possible implementation, the third feature data does not include the first mapping information. The correspondence between each weight feature data in the shared weight data and each weight feature data of the texture (called the first correspondence) can be a preset fixed relationship, that is, predetermined, rather than trained. During the process of obtaining the shared weight data through multiple iterations of training at the encoder, the first correspondence remains fixed, that is, it never changes. The decoder simply decodes according to the preset first correspondence.

[0211] The weight feature data in this application embodiment is a data set, which can be in the form of a vector or other forms, and this application embodiment does not limit it.

[0212] In the second implementation of the third feature data, in addition to the shared weight data, the third feature data also includes the endpoint feature data corresponding to each feature map in the N feature maps. That is, the endpoint feature data is complete, and each feature map has its own endpoint feature data during the encoding process.

[0213] The endpoint feature data in this application embodiment can also be a data set, which can be in the form of a vector or other forms. This application embodiment does not limit this.

[0214] The third implementation of the third feature data includes shared endpoint data (which is an endpoint dataset) and weighted feature data corresponding to each feature map of the texture.

[0215] Similar to the second implementation method which sets shared weight data, the third implementation method sets shared endpoint data instead of shared weight data.

[0216] The shared endpoint data includes endpoint feature data shared by all feature maps of the texture. The shared endpoint data can be viewed as a shared endpoint pool (or simply endpoint pool). It should be understood that the shared endpoint data is data that is progressively optimized during the aforementioned iterative update process.

[0217] In one possible implementation, the total number of endpoint feature data included in the shared endpoint data is less than the total number of endpoint feature data corresponding to the N feature maps of the texture. Thus, after executing steps 1002 and 1003, the bitstream size will be smaller than the bitstream size in the first implementation, meaning the third implementation has a higher compression ratio.

[0218] As an example, the shared endpoint data includes R weighted feature data, and each of the N feature maps includes S endpoint feature data, where R and S are both positive integers greater than 1, and R is less than N×S. Here, S can also refer to the total number of feature blocks included in each feature map, meaning each feature block has one endpoint feature data.

[0219] As can be seen, texture compression is further achieved by setting a shared endpoint pool with a smaller data volume. The shared endpoint pool is equivalent to directly removing some redundant endpoint feature data during the training process.

[0220] In one possible implementation, the third feature data further includes second mapping information, which indicates the correspondence between each endpoint feature data in the shared endpoint data and each endpoint feature data of the texture, that is, the correspondence between each endpoint feature data corresponding to the aforementioned N feature maps and each endpoint feature data in the shared endpoint data. This facilitates the decoding end in reconstructing the endpoint feature data corresponding to each feature map of the texture based on the shared endpoint data.

[0221] The second mapping information can be in key-value form. Each key in the second mapping information represents the identifier of each endpoint feature data in the shared endpoint data, and the value corresponding to each key represents the identifier of the corresponding endpoint feature data in the N feature maps.

[0222] As an example, let Z be the total number of endpoint feature data in the shared endpoint data. One feature block has one endpoint feature data. Taking N = 3, S = 20, and Z = 30 as an example, the N feature maps have a total of 3 × 20 = 60 endpoint feature data. The second mapping information includes {1: [2~14, 17]; 2: [25~30, 38, 56]; 3: [33~37]; ...; 30: [48~54, 60]}. Among them, the serial numbers "1, 2, 3, ... 30" before ":" are 30 keys, representing the identifiers of the 30 endpoint feature data in the shared endpoint data. The serial numbers in "[]" are values. The 30 "[]" contain a total of 60 values, representing the identifiers of the 60 endpoint feature data corresponding to the N feature maps. In this example, the second mapping information indicates that the first endpoint feature data in the shared endpoint data can be used as the 2nd to 14th and 17th endpoint feature vectors corresponding to the N feature maps; the second endpoint feature data in the shared endpoint data can be used as the 25th to 30th, 38th and 56th endpoint feature vectors corresponding to the N feature maps; the third endpoint feature data in the shared endpoint data can be used as the 33rd to 37th endpoint feature vectors corresponding to the N feature maps; and the first endpoint feature data in the shared endpoint data can be used as the 48th to 54th and 60th endpoint feature vectors corresponding to the N feature maps.

[0223] Alternatively, the second mapping information can be in key-value form, where each key in the second mapping information represents the identifier of each endpoint feature data corresponding to N feature maps, and the value corresponding to each key represents the identifier of the corresponding endpoint feature data in the shared endpoint data.

[0224] As an example, the second mapping information includes {[1:14];[2:3];[3:20];...;[60:3]}, where the serial numbers "1, 2, 3, ... 60" before the ":" are 60 keys, representing the identifiers of the 60 endpoint feature data corresponding to the N feature maps, and the serial numbers after the ":" are values, with a value range of 1 to 30, representing the identifiers of the 30 weight feature data in the shared weight data.

[0225] Besides the examples above, the structure of the second mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the keys and values ​​in the above examples, another type of second mapping information can be obtained.

[0226] It should be understood that the second mapping information can be learned through training multiple times to obtain shared endpoint data. Alternatively, the second mapping information can be a pre-defined, fixed relationship, rather than something learned through training. In this case, the second mapping information can also be encoded into the bitstream to ensure that the decoding end can obtain the correspondence between the endpoint feature data corresponding to each feature block in the N feature maps and the endpoint feature data in the shared endpoint data. Furthermore, the second mapping information in the third feature data can be unencoded information.

[0227] In another implementation, the third feature data does not include the second mapping information. The correspondence between the endpoint feature data in the shared endpoint data and the endpoint feature data of the texture (called the second correspondence) is preset, i.e., predetermined. During the process of obtaining the shared endpoint data through multiple iterations of training at the encoding end, the second correspondence remains fixed, i.e., it never changes. The decoding end simply decodes according to the preset second correspondence.

[0228] In the third implementation of the third feature data, in addition to the shared endpoint data, the third feature data also includes the weight feature data corresponding to each feature map in the N feature maps. That is, the weight feature data is complete, and each feature map has its own weight feature data during the encoding process.

[0229] The fourth way to implement the third feature data is to include shared weight data and shared endpoint data (i.e., a weight dataset and an endpoint dataset).

[0230] In other words, a shared weight pool and a shared endpoint pool are set up for weighted feature data and endpoint feature data, respectively. The specific implementation methods can be found in the second and third implementation methods described above, and will not be repeated here. Compared to the second and third implementation methods, the fourth implementation method achieves a higher compression ratio.

[0231] In one possible implementation, the third feature data also includes first mapping information and / or second mapping information. Whether the first mapping information needs to be included can be referred to the relevant introduction in the second implementation above. Similarly, whether the second mapping information needs to be included can be referred to the relevant introduction in the third implementation above.

[0232] The fifth implementation of the third feature data includes the first mapping information and the endpoint feature data corresponding to each feature map of the texture. The first index information indicates the correspondence between the weight feature data corresponding to all feature maps of the texture and the weight feature data in the preset weight dataset (referred to as the preset weight dataset).

[0233] The preset weight dataset here differs from the shared weight data mentioned above. The preset weight dataset is a pre-defined dataset, not trained during texture encoding. This preset weight dataset can be used for encoding and decoding one or more textures, or even any texture. The preset weight dataset can be generated by AI or obtained through other means; this application does not limit this.

[0234] In the above iterative process, by continuously searching for weight feature data that can better represent the weight features of the texture from the preset weight dataset, the first mapping information is obtained. That is, the first mapping information is trained through multiple iterations, i.e., learned. Since the amount of data in the first mapping information is usually much smaller than the amount of data in all the weight feature data of the texture, and also smaller than the amount of data in the shared weight data, the fifth implementation method has a higher compression ratio.

[0235] The first mapping information can be in the form of key-value pairs. Each key in the first index information represents the identifier of each weight feature data corresponding to N feature maps, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the preset weight dataset.

[0236] As an example, taking N feature maps comprising 240 weighted feature data points, and a preset weighted dataset comprising 2000 weighted feature data points, the first index information can include: {[1:201]; [2:1632]; [3:1594]; ...; [240:44]}. Here, the serial numbers "1, 2, 3, ..., 240" before the colon represent 240 keys, indicating the identifiers of the 240 weighted feature data points corresponding to the N feature maps. The serial number after the colon is the value, ranging from 1 to 2000, representing the identifiers of the 2000 weighted feature data points in the preset weighted dataset. In this example, the first mapping information indicates that the 201st weight feature data in the preset weight dataset can be used as the 1st weight feature vector corresponding to the N feature maps, the 1632nd weight feature data in the preset weight dataset can be used as the 2nd weight feature vector corresponding to the N feature maps, the 1594th weight feature data in the preset weight dataset can be used as the 3rd weight feature vector corresponding to the N feature maps, and the 44th weight feature data in the preset weight dataset can be used as the 240th weight feature vector corresponding to the N feature maps.

[0237] Besides the examples above, the structure of the first mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the positions of the keys and values ​​in the above examples, another type of first mapping information is obtained, that is, each value in the first mapping information represents the identifier of each weight feature data corresponding to the N feature maps, and the key corresponding to each value represents the identifier of the corresponding weight feature data in the preset weight dataset.

[0238] In the fifth implementation of the third feature data, in addition to the first mapping information, the third feature data also includes the endpoint feature data corresponding to each feature map in the N feature maps. That is, the endpoint feature data is complete, and each feature map has its own endpoint feature data during the encoding process.

[0239] The sixth implementation of the third feature data includes the second mapping information and the weight feature data corresponding to each feature map of the texture. The second mapping information indicates the correspondence between the endpoint feature data corresponding to each feature map of the texture and the endpoint feature data in the preset endpoint dataset (referred to as the preset endpoint dataset).

[0240] Similar to the fifth implementation method, which sets the first mapping information based on a preset weight dataset, the third implementation method sets the second mapping information based on preset endpoint data.

[0241] The preset endpoint dataset differs from the shared endpoint data mentioned above. The preset endpoint dataset is a pre-defined dataset that can be used for encoding and decoding one or more textures, or even any texture. The preset endpoint dataset can be generated by AI or obtained through other means; this application embodiment does not limit its scope.

[0242] In the above iterative process, the second mapping information is obtained by continuously searching for endpoint feature data that can better represent the endpoint features of the texture from the preset endpoint dataset. That is, the second mapping information is trained through multiple iterations, i.e., learned. Since the amount of data in the second mapping information is usually much smaller than the amount of data in all endpoint feature data of the texture, and also smaller than the amount of data in the shared endpoint data, the compression rate of the sixth implementation method is higher.

[0243] The second mapping information can be in key-value form. Each key in the second mapping information represents the identifier of each endpoint feature data corresponding to the N feature maps, and the value corresponding to each key represents the identifier of the corresponding endpoint feature data in the preset endpoint dataset.

[0244] As an example, taking N feature maps comprising 60 endpoint feature data points and a preset endpoint dataset comprising 1000 endpoint feature data points as an example, the second mapping information may include: {[1:198]; [2:163]; [3:594]; ...; [60:413]}. Here, the serial numbers "1, 2, 3, ..., 60" before the colon represent 60 keys, indicating the identifiers of the 60 endpoint feature data points corresponding to the N feature maps, and the serial numbers after the colon represent values ​​ranging from 1 to 1000, indicating the identifiers of the 1000 endpoint feature data points in the preset endpoint dataset. In this example, the second mapping information indicates that the 198th endpoint feature data in the preset endpoint dataset can be used as the 1st endpoint feature vector corresponding to the N feature maps, the 163rd endpoint feature data in the preset endpoint dataset can be used as the 2nd endpoint feature vector corresponding to the N feature maps, the 594th endpoint feature data in the preset endpoint dataset can be used as the 3rd endpoint feature vector corresponding to the N feature maps, and the 413th endpoint feature data in the preset endpoint dataset can be used as the 240th endpoint feature vector corresponding to the N feature maps.

[0245] Besides the examples above, the structure of the second mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the positions of the keys and values ​​in the above examples, another type of second mapping information is obtained, that is, each value in the second mapping information represents the identifier of each endpoint feature data corresponding to N feature maps, and the key corresponding to each value represents the identifier of the corresponding endpoint feature data in the preset endpoint dataset.

[0246] In the sixth implementation of the third feature data, in addition to the second mapping information, the third feature data also includes the weight feature data corresponding to each feature map in the N feature maps. That is, the weight feature data is complete, and each feature map has its own weight feature data during the encoding process.

[0247] The seventh implementation of the third feature data includes first mapping information and second mapping information. The first mapping information is used to indicate the correspondence between each weight feature data corresponding to the N feature maps and the weight feature data in the preset weight dataset. The second mapping information is used to indicate the correspondence between each endpoint feature data corresponding to the N feature maps and the endpoint feature data in the preset endpoint dataset.

[0248] In other words, for weighted feature data and endpoint feature data, a first mapping information is set based on a preset weighted dataset, and a second mapping information is set based on a preset endpoint dataset. The specific implementation methods are the same as those described in the fifth and sixth implementation methods above, and will not be repeated here. Compared to the fifth and sixth implementation methods, the seventh implementation method achieves a higher compression ratio.

[0249] In the embodiment described above that uses a weighted dataset (shared weighted data or a preset weighted dataset), the weighted dataset is used for all weighted feature data corresponding to the N feature maps. In other embodiments, a weighted dataset may be used for some weighted feature data corresponding to the N feature maps, while other weighted feature data may not use a weighted dataset but be obtained through initialization and training. In this case, the aforementioned first mapping information should be used to indicate the correspondence between the partial weighted feature data corresponding to the N feature maps and the weighted feature data in the weighted dataset. Here, partial weighted feature data may refer to weighted feature data corresponding to some feature maps, weighted feature data corresponding to some feature blocks, or weighted feature data corresponding to both some feature maps and some feature blocks. This application embodiment does not limit this to any particular type.

[0250] Similarly, in the embodiment described above that uses an endpoint dataset (shared endpoint data or a preset endpoint dataset), the endpoint dataset is used for all endpoint feature data corresponding to the N feature maps. In other embodiments, the endpoint dataset may be used for some endpoint feature data corresponding to the N feature maps, while other endpoint feature data may not use the endpoint dataset but be obtained through initialization and training. In this case, the aforementioned second mapping information should be used to indicate the correspondence between the partial endpoint feature data corresponding to the N feature maps and the endpoint feature data in the endpoint dataset. Here, the partial endpoint feature data may be, for example, the endpoint feature data corresponding to some feature maps, or the endpoint feature data corresponding to some feature blocks, or the endpoint feature data corresponding to both some feature maps and some feature blocks. This application embodiment does not limit this to any particular type.

[0251] The above describes some specific implementation methods for determining the third feature data and the second nonlinear transformation parameters through iterative updates. In other embodiments, the third feature data and the second nonlinear transformation parameters can also be determined through other specific implementation methods, which are not limited in this application.

[0252] Step 1002: Encode the third feature data to obtain the first feature data, and encode the second nonlinear transformation parameter to obtain the first nonlinear transformation parameter.

[0253] There are various ways to encode the third feature data and the second nonlinear transformation parameter, such as one or more of entropy coding and vector quantization. This application embodiment does not limit this method. The method of encoding the third feature data can be the same as or different from the method of encoding the second nonlinear transformation parameter. This application embodiment does not limit this method.

[0254] As we know from the above, there are multiple ways to implement the third feature data. Therefore, there are also multiple ways to implement the corresponding first feature data, which will be introduced next.

[0255] The first implementation of the first feature data: When the third feature data includes all data corresponding to the N feature maps of the texture (i.e., the weight feature data and endpoint feature data corresponding to all feature maps), the first feature data includes all encoded data of the N feature maps, that is, it includes the encoded weight feature data (which can be called weight compressed data) and the encoded endpoint feature data (which can be called endpoint compressed data) corresponding to all feature maps. In other words, all data included in the third feature data is directly compressed. This implementation can be understood with reference to Figure 13.

[0256] A second implementation of the first feature data: When the third feature data includes all data corresponding to the N feature maps of the texture (i.e., the weight feature data and endpoint feature data corresponding to all feature maps), the first feature data includes the encoded data corresponding to those N feature maps. In other words, a portion of the data in the third feature data is compressed.

[0257] As an example, referring to Figure 14, taking N feature maps where N is an integer greater than 1 as an example, the weight feature data (referred to as weight data) corresponding to the 2nd to Nth feature maps is discarded. The weight feature data corresponding to the 1st feature map and the endpoint feature data (referred to as endpoint data) corresponding to these N feature maps are encoded to obtain the first feature data. Correspondingly, during decoding, the weight feature data corresponding to the 1st feature map can be reused to obtain the weight feature data corresponding to the 2nd to Nth feature maps. The weight feature data corresponding to each feature map is combined with the endpoint feature data to obtain all feature texture data corresponding to that feature map.

[0258] As another example, still using three feature maps, the weight feature data corresponding to some feature blocks in the second feature map is discarded, and the remaining data is encoded to obtain the first feature data. Accordingly, the decoding end can reuse the weight data corresponding to other feature blocks to reconstruct the weight feature data corresponding to the part of the feature blocks that were not encoded into the bitstream.

[0259] In this embodiment, similarity detection can be used to determine whether to discard a portion of the third feature data, or data can be discarded according to data reuse rules. This embodiment does not limit what the discarded data can be; for example, the discarded data can be part or all of the data corresponding to a feature map, or part or all of the data corresponding to a feature block.

[0260] A third implementation of the first feature data: When the third feature data includes shared weight data and endpoint feature data corresponding to N feature maps of the texture, the first feature data includes encoded shared weight data and encoded endpoint feature data corresponding to the N feature maps. That is, all data included in the third feature data is directly compressed.

[0261] In cases where the third feature data also includes first mapping information (the mapping information corresponding to the shared weight data), the first feature data also includes first mapping information, or encoded first mapping information.

[0262] A fourth implementation of the first feature data: When the third feature data includes shared endpoint data and weighted feature data corresponding to N feature maps of the texture, the first feature data includes encoded shared endpoint data and encoded weighted feature data corresponding to the N feature maps. That is, all the data in the third feature data is directly compressed.

[0263] In cases where the third feature data also includes second mapping information (the mapping information corresponding to the shared endpoint data), the first feature data also includes second mapping information, or encoded second mapping information.

[0264] The fifth implementation of the first feature data: When the third feature data includes shared weight data and shared endpoint data, the first feature data includes encoded shared weight data and encoded shared endpoint data. That is, all the data in the third feature data is directly compressed.

[0265] Where the third feature data also includes first mapping information (mapping information corresponding to the shared weight data), the first feature data also includes first mapping information, or encoded first mapping information. Where the third feature data also includes second mapping information (mapping information corresponding to the shared endpoint data), the first feature data also includes second mapping information, or encoded second mapping information.

[0266] The sixth implementation of the first feature data: When the third feature data includes the first mapping information (the mapping information corresponding to the preset weight dataset) and the endpoint feature data corresponding to the N feature maps of the texture, the first feature data includes the first mapping information (or the encoded first mapping information) and the encoded endpoint feature data corresponding to the N feature maps.

[0267] The seventh implementation of the first feature data: When the third feature data includes the second mapping information (the mapping relationship corresponding to the preset endpoint dataset) and the weight feature data corresponding to the N feature maps of the texture, the first feature data includes the second mapping information (or the encoded second mapping information) and the encoded weight feature data corresponding to the N feature maps.

[0268] The eighth implementation of the first feature data: When the third feature data includes the first mapping information (the mapping information corresponding to the preset weight dataset) and the second mapping information (the mapping information corresponding to the preset endpoint dataset), the first feature data includes the first mapping information (or the encoded first mapping information) and the second mapping information (or the encoded second mapping information).

[0269] Besides the implementation methods mentioned above, the first feature data can also be implemented in other ways. For example, based on the third implementation method of the first feature data, some endpoint feature data corresponding to the feature maps can be discarded, and the shared weight data and the endpoint feature data corresponding to the remaining feature maps can be encoded to obtain the first feature data. Similarly, based on the fourth, sixth, and seventh implementation methods of the first feature data mentioned above, some endpoint feature data or some weight feature data corresponding to the feature maps can be discarded to obtain first feature data with a smaller data volume, further improving the compression ratio. Furthermore, if the third feature data includes the weight dataset corresponding to some weight feature data of N feature maps, other weight feature data corresponding to N feature maps, and all endpoint feature data corresponding to N feature maps, then the first feature data can include the first mapping information, compressed data of other weight feature data corresponding to N feature maps, and encoded endpoint feature data corresponding to N feature maps. In other words, for any possible implementation method of the third feature data, the first feature data can also have a corresponding implementation method, which will not be listed here.

[0270] In the embodiments described above, the encoded shared weight data is incorporated into the bitstream as part of the first feature data. For example, the encoded shared weight data and the encoded endpoint feature data corresponding to the N feature maps are located in the same data segment of the bitstream, and this data segment is the data segment of the first feature data. In some other embodiments, the encoded shared weight data may not be part of the first feature data, but may be incorporated into other data segments of the bitstream. Similarly, in the embodiments described above, the encoded shared endpoint data is incorporated into the bitstream as part of the first feature data. For example, the encoded shared endpoint data and the encoded weight feature data corresponding to the N feature maps are located in the same data segment of the bitstream, and this data segment is the data segment of the first feature data. In some other embodiments, the encoded shared endpoint data may not be part of the first feature data, but may be incorporated into other data segments of the bitstream. That is to say, the embodiments of this application do not limit the position of each data in the bitstream, nor do they limit the format of the bitstream.

[0271] In this embodiment, the process of encoding the second nonlinear transformation parameter can be as follows: all of the second nonlinear transformation parameters can be encoded to obtain the first nonlinear transformation parameter. Alternatively, some parameters in the second nonlinear transformation parameter can be discarded, and the remaining parameters can be encoded to obtain the first nonlinear transformation parameter. The discarded parameters can be obtained by reusing nonlinear transformation parameters corresponding to other encoded textures.

[0272] Step 1003: Encode the first feature data and the first nonlinear transformation parameters into the code stream.

[0273] After obtaining the first feature data and the first nonlinear transformation parameters, the encoding end encodes the first feature data and the first nonlinear transformation parameters into the bitstream.

[0274] The above encoding method achieves secondary compression of texture feature data, thereby improving the compression ratio.

[0275] As mentioned above, the encoding process involves reconstructing the texture to determine whether the encoding and decoding performance of the third feature data and the second nonlinear transformation parameters after the current iteration is good. The process of reconstructing the texture involves texture decoding and feature vector construction.

[0276] First, let's introduce texture decoding.

[0277] After decoding the first feature data, the encoder obtains the reconstructed first feature data. If the first feature data includes all endpoint feature data and all weight feature data corresponding to the N feature maps of the texture, the encoder performs texture decoding on the reconstructed first feature data (called feature texture data) to obtain the second feature data. The second feature data obtained here includes all data from the N feature maps.

[0278] If the first feature data includes some endpoint feature data or some weight feature data corresponding to the N feature maps of the texture, then the encoder decodes the first feature data to reconstruct all endpoint data and all weight feature data (called feature texture data) corresponding to the N feature maps of the texture. The feature texture data is then subjected to texture decoding to obtain the second feature data. The second feature data obtained here includes all data from the N feature maps.

[0279] Here, the partial endpoint feature data or partial weight feature data corresponding to the N feature maps of the texture corresponds to various implementations of the first feature data, such as including partial endpoint feature data corresponding to the N feature maps, and / or including partial weight feature data corresponding to the N feature maps.

[0280] Corresponding to different implementation methods of the first feature data, the specific implementation methods of all endpoint feature data and all weight feature data of the N feature maps for reconstructing the texture based on the first feature data are also different.

[0281] The second implementation method corresponding to the first feature data is to reconstruct all endpoint feature data and all weight feature data corresponding to N feature maps by reusing some or all of the data in the first feature data.

[0282] The third implementation method corresponding to the first feature data is to determine the weight feature data corresponding to each feature map (which can be specific to each feature block) from the shared weight data according to the first mapping information or the first correspondence relationship determined in advance.

[0283] The fourth implementation method corresponding to the first feature data is to determine the endpoint feature data corresponding to each feature map (which can be specific to each feature block) from the shared endpoint data according to the second mapping information or the second correspondence relationship determined in advance.

[0284] The fifth implementation of the first feature data is as follows: according to the first mapping information or the first correspondence relationship determined in advance, the weight feature data corresponding to each feature map (which can be specific to each feature block) in the N feature maps is determined from the shared weight data; according to the second mapping information or the second correspondence relationship determined in advance, the endpoint feature data corresponding to each feature map (which can be specific to each feature block) in the N feature maps is determined from the shared endpoint data.

[0285] The sixth implementation method corresponding to the first feature data is to determine the weight feature data corresponding to each feature map (which can be specific to each feature block) from the preset weight dataset according to the first mapping information.

[0286] Corresponding to the seventh implementation of the first feature data, according to the second mapping information, the endpoint feature data corresponding to each feature map (which can be specific to each feature block) in the N feature maps are determined from the preset endpoint dataset.

[0287] The eighth implementation method corresponding to the first feature data is to determine the weight feature data corresponding to each feature map (which can be specific to each feature block) in the N feature maps from the preset weight dataset according to the first mapping information, and to determine the endpoint feature data corresponding to each feature map (which can be specific to each feature block) in the N feature maps from the preset endpoint dataset according to the second mapping information.

[0288] Corresponding to other possible implementations of the first feature data, according to the corresponding implementation logic, all data of N feature maps can be obtained through one or more methods such as data reuse, acquisition from the dataset, and direct decoding. This application embodiment will not elaborate on this further.

[0289] After obtaining the feature texture data, the encoding end performs texture decoding on the feature texture data to obtain the second feature data. Here, the texture decoding method matches the format of the feature texture data, and the texture decoding method can be ASTC, BC, or other texture decoding methods.

[0290] Next, we will introduce the construction of feature vectors.

[0291] After obtaining the second feature data, the encoder determines one or more feature vectors based on the second feature data, and then performs a nonlinear transformation on the one or more feature vectors according to the decoded second nonlinear transformation parameters to reconstruct the texture.

[0292] In one implementation, the encoder determines one or more feature vectors according to relevant existing technologies. For example, the encoder samples the second feature data to obtain a first feature vector for each feature block in N feature maps. The one or more feature vectors include the first feature vector of each feature block in the N feature maps, or include a feature vector obtained by concatenating the first feature vectors of all feature blocks.

[0293] Another implementation improves encoding and decoding performance by optimizing the feature vector construction. This will be discussed later.

[0294] In related technologies, a feature vector for a feature block is constructed by sampling the feature block. However, in the optimized solution of this application embodiment, a feature vector for a feature block can be constructed based on the feature vectors of other feature blocks, thereby assisting in decoding the texture data corresponding to the feature block. These other feature blocks can be referred to as auxiliary blocks corresponding to this feature block.

[0295] Based on this, in the embodiments of this application, the first feature data is used to reconstruct the feature map of the texture, and the second nonlinear transformation parameter is used to decode the feature map to reconstruct the texture. Taking the feature map of the texture as including the first feature block and the second feature block, and the second feature block as the auxiliary block corresponding to the first feature block, the encoding end also determines auxiliary information. The auxiliary information indicates the third feature vector, which is the feature vector corresponding to the second feature block. The third feature vector is used to construct the feature vector of the first feature block. The encoding end also encodes the auxiliary information into the bitstream.

[0296] The auxiliary information is used to indicate the third feature vector. For example, it can indicate that the third feature vector is used to process the first feature vector, or it can indicate that the third feature vector is used to construct a feature vector for the first feature block. In one possible implementation, the auxiliary information can indicate the location of an auxiliary sampling point, which can indicate the feature map where the auxiliary block is located, or it can indicate the auxiliary block (e.g., a second feature block). Alternatively, the auxiliary information can indicate the location of the third feature vector.

[0297] In some possible implementations, a feature block corresponds to an auxiliary block located in the same feature map as the feature block, but in a different feature map; or, the auxiliary block and the feature block are located in different feature maps. In other possible implementations, a feature block corresponds to multiple auxiliary blocks located in different feature maps. For example, these multiple auxiliary blocks may correspond to the same region of a texture, such as the same texture block. Of course, these multiple auxiliary blocks may also correspond to different regions of a texture, depending on how the auxiliary blocks are selected.

[0298] As an example, the feature map of the texture also includes a third feature block. The third feature block and the second feature block are both auxiliary blocks corresponding to the first feature block. The aforementioned auxiliary information also indicates a fourth feature vector, which is a feature vector corresponding to the third feature block. The second and third feature blocks are both used to assist in decoding the texture data corresponding to the first feature block. That is, the third feature vector and the fourth feature vector are both used to construct the feature vector of the first feature block.

[0299] Next, we will introduce how to determine the auxiliary block.

[0300] The first way to determine auxiliary blocks: complexity detection.

[0301] In some embodiments, the encoder performs complexity detection on the texture to determine auxiliary information, which also includes determining auxiliary blocks. That is, by performing complexity detection on the original texture, the auxiliary relationships between various feature blocks on the feature map are determined.

[0302] In one implementation, the texture comprises multiple texture blocks, each corresponding to an auxiliary texture block. The encoder performs complexity detection on these multiple texture blocks to determine the complexity of each texture block, and based on the complexity of these multiple texture blocks, determines the corresponding auxiliary texture block for each texture block.

[0303] As an example, the complexity of a texture block can be calculated according to the following formula (2). The complexity of a texture block can also be calculated in other ways, which are not limited in this embodiment, such as using deep learning methods for complexity detection.

[0304] In formula (2), complexity(i,j) represents the complexity at pixel / texel coordinate (i,j), which can be determined according to a certain complexity index. In formula (2), the variance of texel values ​​within a certain neighborhood of texel coordinate (i,j) is used as the complexity index, but other indices can also be used. x(i,j) represents the texel value / pixel value at texel coordinate (i,j). This represents the average texel value / pixel value at texel coordinate (i,j) and its neighborhood.

[0305] In this embodiment, the texture block with the lowest complexity in the surrounding region of each texture block can be determined as the auxiliary texture block corresponding to that texture block. The surrounding region here may include the region where the edges of the feature blocks overlap with the edges of the corresponding texture block, or it may also include the region where the vertices of the feature blocks overlap with the vertices of the corresponding texture block, or it may include distinctions determined according to other rules.

[0306] Taking Figure 1 as an example, in one possible implementation, the area surrounding texture block 1 includes texture blocks 2, 5, and 6; the area surrounding texture block 2 includes texture blocks 1, 3, and 5-7; the area surrounding texture block 3 includes texture blocks 2, 4, and 6-8; the area surrounding texture block 5 includes texture blocks 1, 2, 6, 9, and 10; the area surrounding texture block 6 includes texture blocks 1-3, 5, 7, and 9-11, ... In another possible implementation, the area surrounding texture block 1 includes texture blocks 2 and 5; the area surrounding texture block 2 includes texture blocks 1, 3, and 6; the area surrounding texture block 3 includes texture blocks 2, 4, and 7; the area surrounding texture block 5 includes texture blocks 1, 6, and 9; the area surrounding texture block 6 includes texture blocks 2, 5, 7, and 10, ...

[0307] Alternatively, for each texture block whose complexity is not the lowest, the texture block closest to it among all texture blocks with lower complexity is identified as the auxiliary texture block corresponding to that texture block. For the texture block with the lowest complexity, this texture block is designated as the auxiliary texture block corresponding to that texture block.

[0308] Alternatively, the texture blocks can be sorted by complexity from smallest to largest, and the texture block with lower complexity among two adjacent texture blocks can be designated as the auxiliary texture block corresponding to the texture block with higher complexity. For the texture block with the lowest complexity, this texture block can be designated as the auxiliary texture block corresponding to it.

[0309] Alternatively, the multiple texture blocks can be divided into two groups based on their complexity: a complex block group and a simple block group. The complexity of each texture block in the complex block group is higher than that of each texture block in the simple block group. A one-to-one correspondence is established between the texture blocks in the complex block group and the texture blocks in the simple block group, with each pair of corresponding texture blocks serving as auxiliary blocks for the other.

[0310] For example, sort the complexity of the texture block in ascending order to determine its order of complexity. The texture block at position Vi and the texture block at position Vi are auxiliary texture blocks to each other. V represents the total number of texture blocks, and the value of i ranges from... Among them, the order is in the th order. The texture blocks are texture blocks in the simple block group, and the remaining texture blocks are texture blocks in the complex block group.

[0311] As an example, with V=16, the 1st texture block and the 9th texture block are auxiliary texture blocks to each other, the 2nd texture block and the 10th texture block are auxiliary texture blocks to each other, the 3rd texture block and the 11th texture block are auxiliary texture blocks to each other, the 4th texture block and the 12th texture block are auxiliary texture blocks to each other, and so on, with the 8th texture block and the 16th texture block being auxiliary texture blocks to each other.

[0312] In addition to the methods mentioned above, the encoding end can also determine auxiliary blocks for each texture block in other possible ways, which will not be listed one by one in this application embodiment.

[0313] In another implementation, some texture blocks correspond to auxiliary texture blocks. The encoder performs complexity detection on these multiple texture blocks to determine the complexity of each texture block, and based on the complexity of these multiple texture blocks, determines the auxiliary texture blocks corresponding to some texture blocks.

[0314] The complexity of the texture block can be sorted in ascending order, with the block ranked first... The texture block at position Vi is determined as the auxiliary texture block of the texture block at position Vi. V represents the total number of texture blocks, and the value of i ranges from...

[0315] As an example, with V=16, the first texture block is the auxiliary texture block corresponding to the 9th texture block, the second texture block is the auxiliary texture block corresponding to the 10th texture block, the third texture block is the auxiliary texture block corresponding to the 11th texture block, the fourth texture block is the auxiliary texture block corresponding to the 12th texture block, and so on, with the 8th texture block being the auxiliary texture block corresponding to the 16th texture block.

[0316] Alternatively, if there are texture blocks in the surrounding area with lower complexity than this texture block, the texture block with the lowest complexity in the surrounding area is identified as the auxiliary texture block corresponding to this texture block. If there are no texture blocks in the surrounding area with lower complexity than this texture block, there is no corresponding auxiliary texture block.

[0317] Alternatively, for a texture block whose complexity exceeds a complexity threshold, select a texture block whose complexity is lower than the complexity threshold as the corresponding auxiliary texture block.

[0318] In addition to the methods mentioned above, the encoding end can also determine the auxiliary texture blocks of some texture blocks based on complexity detection in other possible ways, which will not be listed one by one in the embodiments of this application.

[0319] The second way to determine auxiliary blocks: auxiliary block selection rules.

[0320] In other words, the encoding end determines the auxiliary texture blocks corresponding to some or all of the texture blocks in the texture according to the auxiliary block selection rules. That is, the auxiliary texture blocks are texture blocks of a preset area (also known as a specified area), and correspondingly, the auxiliary blocks are also feature blocks of the preset area.

[0321] As an example, the texture block above (or below), to the left, or to the right of each texture block is identified as an auxiliary texture block for that texture block. As another example, the preceding texture block for each texture block is identified as an auxiliary texture block for that texture block. As yet another example, an auxiliary texture block can be selected from texture blocks within the same cache line as the currently encoded texture block. Alternatively, auxiliary texture blocks can be determined according to other rules, which will not be enumerated in this embodiment.

[0322] In some embodiments, there may be one auxiliary texture block; in other embodiments, there may be multiple auxiliary texture blocks, such as two. This application does not limit the scope of the embodiments. Furthermore, in addition to performing complexity detection on the original texture, auxiliary blocks can also be determined directly from the feature blocks in the feature map by performing complexity detection on the feature map.

[0323] After determining the auxiliary blocks, during the feature vector construction process, the encoder obtains the first feature vector for each feature block in the N feature maps based on the second feature data obtained from texture decoding. For the first feature block, based on the auxiliary blocks (such as the second and / or third feature blocks mentioned above), the first feature vector of the first feature block is processed to obtain the second feature vector of the first feature block. The aforementioned multiple feature vectors include the second feature vector of the first feature block. Here, the first feature block is any feature block with auxiliary blocks. Having auxiliary blocks means that its corresponding texture block has auxiliary texture blocks.

[0324] There are many ways to process the first feature vector of the first feature block to obtain the second feature vector of the first feature block based on the auxiliary block of the first feature block.

[0325] In the first implementation, the first feature vector of the first feature block is concatenated with the first feature vector of the auxiliary block (such as the third and / or fourth feature vector mentioned above) to obtain the second feature vector of the first feature block.

[0326] In this embodiment, the first feature vector of the auxiliary block can be concatenated before or after the first feature vector of the first feature block, depending on the concatenation rules. When there are multiple auxiliary blocks, this embodiment does not limit the concatenation position of the first feature vectors of these multiple auxiliary blocks; they can be concatenated according to the concatenation rules. As an example, the multiple auxiliary blocks are concatenated after the first feature vector of the first feature block in order of the size of their respective feature maps.

[0327] In this implementation, the length of the first feature vector is less than the length of the second feature vector. For each feature block without an auxiliary block, the encoder can extend the length of the first feature vector of that feature block, for example, by adding a segment of elements with a value of 0, so that the length of the feature vector of that feature block is increased to be the same as the length of the second feature vector of the first feature block. This ensures that the length of each second feature vector in the input nonlinear transformation model is the same, i.e., the dimension is the same. Of course, the method of length extension is not limited to adding a segment of elements with a value of 0; elements with other values ​​can also be added. This application embodiment does not limit this.

[0328] In the second implementation, the first feature vector of the first feature block and / or the first feature vector of the auxiliary block are transformed and then concatenated to obtain the second feature vector of the first feature block.

[0329] In the case of multiple auxiliary blocks, the encoder can transform one or more of the first feature vectors of the first feature block and the first feature vectors of the multiple auxiliary blocks according to the transformation rules before concatenating them. The transformations include one or more of translation, rotation, and mirroring. The transformation rules can specify which feature blocks(s) to transform in what way, and in what order to concatenate them.

[0330] In some embodiments, the transformation rule is predetermined, i.e., determined in advance, and both the encoder and decoder process the first feature vector according to the predetermined transformation rule. In other embodiments, the bitstream includes the transformation rule, for example, if the transformation rule is determined by the encoder, then the bitstream may include the transformation rule. Of course, even if the transformation rule is predetermined, the bitstream may still include the transformation rule to ensure that the decoder knows how to process the first feature vector.

[0331] In this implementation, the length of the first feature vector is also less than the length of the second feature vector. For each feature block that does not have an auxiliary block, the decoder can extend the length of the first feature vector of that feature block. The method of length extension can be referred to the above description, and will not be repeated here.

[0332] In the third implementation, the first feature vector of the first feature block and the first feature vector of the auxiliary block are calculated element-wise to obtain the second feature vector of the first feature block. The calculation may include one or more of addition, subtraction, multiplication, and division.

[0333] The encoding end can perform element-wise calculations on the first feature vector of the first feature block and the first feature vector of the auxiliary block according to the calculation rules. These calculation rules can specify the calculation method. When there are multiple auxiliary blocks, the calculation rules can specify the calculation method between the first feature vector of the first feature block and the first feature vectors of the multiple auxiliary blocks, such as which vector is added to which vector, or which vector is multiplied by which vector.

[0334] In some embodiments, the calculation rules are predetermined, i.e., determined in advance, and both the encoder and decoder process the first feature vector according to the predetermined calculation rules. In other embodiments, the bitstream includes the calculation rules, for example, if the calculation rules are determined by the encoder, then the bitstream may include the calculation rules. Of course, even if the calculation rules are predetermined, the bitstream may still include the calculation rules to ensure that the decoder knows how to process the first feature vector.

[0335] For the third implementation, if the length of the first eigenvector of the first feature block is the same as the length of the second eigenvector of the first feature block, then for each feature block without an auxiliary block, the first eigenvector of that feature block can be directly used as the second eigenvector of that feature block, or in other words, the first eigenvector of that feature block can be used as the eigenvector among the final multiple eigenvectors. If the length of the first eigenvector of the first feature block is less than the length of the second eigenvector, then for each feature block without an auxiliary block, the length of the first eigenvector of that feature block can be extended. The method of length extension can be referred to the above description, and will not be repeated here.

[0336] The three implementation methods described above can be used individually or in combination. For example, the first and third implementation methods can be combined. When there are multiple auxiliary blocks, these auxiliary blocks can be calculated element-wise and then concatenated with the first feature vector of the first feature block. In addition, there are many other vector processing methods, which are not limited in this embodiment.

[0337] In the above optimization scheme, by using auxiliary blocks for assisted encoding and decoding, combined with the learning ability of the neural network model, the optimization scheme can allocate more encoding bits to texture blocks with higher complexity, thereby improving encoding and decoding performance.

[0338] Figure 15 is a flowchart of another texture encoding method provided in an embodiment of this application. The above-mentioned optimization scheme based on auxiliary blocks can be understood by referring to Figure 15. Figure 15 inserts complexity detection and auxiliary block feature vectors (referred to as auxiliary feature vectors) based on the embodiments shown in Figures 12 to 14. The specific implementation process can be referred to the relevant descriptions in the embodiments above, and will not be repeated here.

[0339] Of course, the encoding method shown in Figure 11 can also be optimized by inserting complexity detection and auxiliary feature vectors into the process shown in Figure 11. The specific implementation process can be referred to the relevant description in the above embodiment, and will not be repeated here.

[0340] This application does not limit whether the multiple feature maps of a texture are encoded serially or in parallel, nor does it limit whether the multiple feature blocks of the same feature map are encoded in parallel or serially. In some embodiments, multiple feature maps can be encoded serially, and multiple feature blocks within each feature map can also be encoded serially. In other embodiments, multiple feature maps are encoded in parallel, and multiple feature blocks within each feature map are also encoded in parallel. In still other embodiments, multiple feature maps are encoded serially, and multiple feature blocks within each feature map are encoded in parallel. In yet another embodiment, some feature maps are encoded serially, while other feature maps are encoded in parallel; some feature blocks are encoded in parallel, while other feature blocks are encoded serially.

[0341] In summary, in the embodiments of this application, the encoding end can perform secondary compression on the third feature data of the texture and the second nonlinear transformation parameters, thereby improving the texture compression rate and reducing the bitstream's occupation of network transmission bandwidth and storage space.

[0342] This approach first utilizes the overfitting capability of nonlinear transformations (such as neural network models or nonlinear transformation functions) to encode the texture, obtaining the third feature data and the second nonlinear transformation parameters. The texture to be encoded can be a single texture image, such as one to be displayed, or a set of texture images, such as a set of texture maps used for rendering. In other words, this scheme can use nonlinear transformations to compress one or more texture images together to obtain the first feature data and the first nonlinear transformation parameters.

[0343] Furthermore, in some embodiments, feature vectors can be constructed based on auxiliary blocks, thereby allocating more coding bits to complex texture blocks, achieving reasonable coding bit allocation, and improving coding quality.

[0344] Figure 16 is a flowchart of a texture decoding method provided in an embodiment of this application, which is applied to the decoding end. This texture decoding method matches the texture encoding method shown in Figure 10. Referring to Figure 16, the method includes the following steps 1601 to 1603.

[0345] Step 1601: Obtain the bitstream and parse the bitstream to obtain the first feature data of the texture and the first nonlinear transformation parameters.

[0346] The first feature data is used to reconstruct the feature map of the texture, and the first nonlinear transformation parameter is used to reconstruct the texture by combining the feature map. Both the first feature data and the first nonlinear transformation parameter are compressed. For details regarding the first feature data and the first nonlinear transformation parameter, please refer to the relevant content in the encoding end embodiment; they will not be repeated here.

[0347] The bitstream in step 1601 can be a bitstream of a single texture or a bitstream of multiple textures; this embodiment of the application does not limit this.

[0348] Step 1602: Decode the first feature data to obtain the second feature data, and decode the first nonlinear transformation parameter to obtain the second nonlinear transformation parameter.

[0349] Here, the second feature data is all the data of the texture feature map, and the second nonlinear transformation parameter is the reconstructed first nonlinear transformation parameter. The process of decoding the first nonlinear transformation parameter is the decompression process. The decompression method is matched with the compression method used at the encoding end to obtain the first nonlinear transformation parameter. That is, the decompression method here can be one or more of entropy decoding, inverse vector quantization, etc.

[0350] In this embodiment of the application, the first feature data is decoded to obtain the second feature data, including the following steps 16021 and 16022.

[0351] Step 16021: Decode the first feature data to obtain feature texture data.

[0352] The process of decoding the first feature data to obtain feature texture data includes the process of decompressing the first feature data. The decompression method here corresponds to, or matches, the compression method used at the encoding end to obtain the first feature data. For example, if the first feature data at the encoding end is obtained through vector quantization, then the decompression here includes inverse vector quantization. Or, if the first feature data at the encoding end is obtained through entropy encoding, then the decompression here includes entropy decoding. Of course, the first feature data at the encoding end can also be obtained through multiple encoding operations, which may include entropy encoding, vector quantization, etc. This application embodiment does not limit this; correspondingly, the decompression here includes multiple decoding operations, which match the multiple encoding operations.

[0353] The aforementioned feature texture data includes weighted feature data and endpoint feature data corresponding to N feature maps of the texture. The endpoint feature data represents the range of texel values ​​in the texture, while the weighted feature data is used to determine the texel values ​​of the texture based on the texel value range. N is a positive integer. In other words, this feature texture data is in a texture encoding format. The process of determining the feature texture data at the decoding end is similar to that at the encoding end, and will be described below.

[0354] In this embodiment, the weighted feature data in the feature texture data is obtained based on the weighted feature data in the first feature data, or from a preset weighted dataset. The following will describe different scenarios.

[0355] The first scenario: The weighted feature data in the feature texture data is obtained based on the weighted feature data in the first feature data.

[0356] When the weighted feature data is obtained based on the weighted feature data in the first feature data, the first feature data includes partially encoded data corresponding to the N feature maps of the texture, or the first feature data includes encoded shared weight data, and the shared weight data includes weighted feature data shared by the N feature maps.

[0357] In the case where the first feature data includes encoded data corresponding to the N feature maps of the texture, the process of decoding the first feature data to obtain the feature texture data includes: decoding the first feature data to obtain the first part of the data corresponding to the N feature maps, and obtaining the feature texture data based on the first part of the data corresponding to the N feature maps.

[0358] In this embodiment, the other data (referred to as the second part of data) corresponding to the N feature maps can be determined by reusing the first part of the data corresponding to the N feature maps, thereby obtaining feature texture data. The feature texture data includes the first part of the data and the second part of the data corresponding to the N feature maps. Specifically, some or all of the data can be obtained from the first part of the data corresponding to the N feature maps to obtain the second part of the data. Which data is specifically obtained depends on what the missing data in the second part of the data is, and this will be described exemplarily below.

[0359] Based on the above description of the encoding method embodiments, the second part of the data corresponding to the N feature maps may include the weight feature data corresponding to M1 feature maps in the N feature maps, and / or the endpoint feature data corresponding to K1 feature maps in the N feature maps, where M1 and K1 are both positive integers not greater than N. Here, the M1 feature maps can be completely identical to the K1 feature maps, completely different, or partially identical.

[0360] Based on this, when the second part of the data corresponding to the N feature maps can include the weight feature data corresponding to M1 feature maps among the N feature maps, the weight feature data corresponding to the M2 feature maps (excluding the M1 feature maps) among the N feature maps can be multiplexed to obtain the weight feature data corresponding to the M1 feature maps. For example, the third feature map reuses the weight feature data corresponding to the first feature map, and the fourth feature map reuses the weight feature data corresponding to the second feature map. Which feature map(s) among the M2 feature maps is specifically reused can be determined according to the data reuse rules or based on the reuse indication information in the bitstream. The data reuse rules characterize the reuse relationship between the weight feature data corresponding to the N feature maps. For example, the reuse relationship indicates that the weight feature data corresponding to the 2nd to Nth feature maps all reuse the weight feature data corresponding to the 1st feature map. The reuse instruction information is used to indicate which feature map in the M2 feature maps the weight feature data should reuse for each of the M1 feature maps. For example, it indicates that the weight feature data corresponding to the third feature map should reuse the weight feature data corresponding to the first feature map.

[0361] If the second part of the data corresponding to the N feature maps can include the endpoint feature data corresponding to K1 feature maps among the N feature maps, then the endpoint feature data corresponding to K2 feature maps other than the K1 feature maps can be reused to obtain the endpoint feature data corresponding to the K1 feature maps. Here, the sum of K1 and K2 is N. The principle of reusing endpoint feature data is similar to that of reusing weight feature data, and will not be elaborated upon here.

[0362] As an example, a texture has a first feature map and a second feature map. The first feature data includes endpoint feature data and weight feature data (encoded) corresponding to the first feature map. The decoding end obtains the weight feature data corresponding to the second feature map based on the weight feature data corresponding to the first feature map, and / or obtains the endpoint feature data corresponding to the second feature map based on the endpoint feature data corresponding to the first feature map. The feature texture data includes the endpoint feature data and weight feature data corresponding to the first feature map, as well as the endpoint feature data and weight feature data corresponding to the second feature map.

[0363] The above describes the case where the first feature data includes the partially encoded data corresponding to the N feature maps of the texture. Next, we will describe the case where the first feature data includes encoded shared weight data.

[0364] When the first feature data includes encoded shared weight data, the encoded shared weight data is decoded to obtain shared weight data, and the weight feature data corresponding to each of the N feature maps is obtained from the shared weight data.

[0365] In one implementation, the first feature data further includes first mapping information (which may be encoded), indicating the correspondence between the weight feature data corresponding to each of the N feature maps and the weight feature data in the shared weight data. Based on this, the decoding end obtains the weight feature data corresponding to each of the N feature maps from the shared weight data according to the first mapping information.

[0366] Each of the N feature maps includes S feature blocks. The first mapping information indicates the correspondence between the weight feature data corresponding to each feature block in the N feature maps and the weight feature data in the shared weight data. The decoding end obtains the weight feature data corresponding to each feature block in the N feature maps from the shared weight data according to the first mapping information.

[0367] In this embodiment, the total number of weight feature data in the shared weight data is less than the total number of weight feature data corresponding to the N feature maps. Let T be the total number of weight feature data in the shared weight data. If a feature block has one weight feature data, then T is less than N×S. If a feature block has g weight feature data, where g is an integer greater than 1, then T is less than N×S×g.

[0368] The first mapping information can be in key-value form. Each key in the first mapping information represents the identifier of each weight feature data in the shared weight data, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the N feature maps.

[0369] As an example, taking N = 3, S = 20, g = 4, and T = 50 as an example, the N feature maps have a total of 3 × 20 × 4 = 240 weight feature data. The first mapping information can include {1: [1~24, 45, 47]; 2: [25~34, 38, 56]; 3: [79, 94~100, 240]; ...; 50: [35, 194~239]}. Among them, the serial numbers "1, 2, 3, ... 50" before the ":" are 50 keys, representing the identifiers of 50 weight feature data in the shared weight data, and the serial numbers in "[]" are values. The 50 "[]" contain a total of 240 values, representing the identifiers of the 240 weight feature data corresponding to the N feature maps. In this example, the first mapping information indicates that the first weight feature data in the shared weight data can be used as the 1st to 24th, 45th, and 47th weight feature vectors corresponding to the N feature maps; the second weight feature data in the shared weight data can be used as the 25th to 34th, 38th, and 56th weight feature vectors corresponding to the N feature maps; the third weight feature data in the shared weight data can be used as the 79th, 94th to 100th, and 240th weight feature vectors corresponding to the N feature maps; and the first weight feature data in the shared weight data can be used as the 35th, 194th to 239th weight feature vectors corresponding to the N feature maps.

[0370] Alternatively, the first mapping information can be in key-value form, where each key in the first mapping information represents the identifier of each weight feature data corresponding to N feature maps, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the shared weight data.

[0371] As an example, the first mapping information includes {[1:2];[2:25];[3:50];...;[240:48]}, where the serial numbers "1, 2, 3, ... 240" before the ":" are 240 keys, representing the identifiers of 240 weighted feature data corresponding to N feature maps, and the serial numbers after the ":" are values, with values ​​ranging from 1 to 50, representing the identifiers of 50 weighted feature data in the shared weight data.

[0372] Besides the examples above, the structure of the first mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the keys and values ​​in the above examples, another type of first mapping information can be obtained.

[0373] The weighted feature data in this embodiment is a dataset, which can be in vector form or other forms, and this embodiment does not limit this. Furthermore, the first mapping information in the first feature data can be encoded information or unencoded information, and this embodiment does not limit this.

[0374] It should be understood that the first mapping information can be learned through training multiple times to obtain shared weight data. Of course, the first mapping information can also be a pre-defined fixed relationship, rather than something learned through training. In this case, the first mapping information can also be encoded into the bitstream to ensure that the decoding end can obtain the correspondence between the weight feature data corresponding to each feature block in the N feature maps and the weight feature data in the shared weight data.

[0375] In another implementation, the first feature data does not include the first mapping information. The decoder obtains the weight feature data corresponding to each of the N feature maps from the shared weight data according to the first correspondence. Here, the first correspondence is preset, that is, predetermined. During the process of obtaining the shared weight data through multiple iterations of training, the first correspondence remains fixed, that is, it never changes.

[0376] In some embodiments, where the first feature data includes encoded shared weight data, the first feature data also includes encoded endpoint feature data corresponding to the N feature maps. That is, the endpoint feature data corresponding to the N feature maps are all encoded into the bitstream. After obtaining the weight feature data corresponding to each feature map, the decoding end combines the endpoint feature data corresponding to the feature map with the weight feature data to obtain the complete information corresponding to the feature map.

[0377] The decoding end can combine the endpoint feature data and weight feature data corresponding to each feature block in the feature map to obtain the complete information corresponding to that feature block. After combining the complete information corresponding to all feature blocks, the feature texture data is obtained.

[0378] In other embodiments, where the first feature data includes encoded shared weight data, the first feature data also includes encoded shared endpoint data. That is, the endpoint feature data can also be compressed in a similar manner to the shared weight data to reduce redundant information and decrease the bitrate. Shared endpoint data will be described in detail later.

[0379] The shared weight data described above can be viewed as a shared weight pool (or simply weight pool), where all feature maps share the weight feature data. In one possible implementation, some feature maps use shared weight data to determine their corresponding weight feature data; that is, these feature maps share a weight pool. Other feature maps do not use shared weight data but instead learn their corresponding weight feature data for each feature map through iterative training.

[0380] Similarly, the shared endpoint data described above can be viewed as a shared endpoint pool (or simply weight pool), where all feature maps share the endpoint feature data. In one possible implementation, some feature maps use shared endpoint data to determine their corresponding endpoint feature data; that is, these feature maps share a weight pool. Other feature maps do not use shared endpoint data but instead learn their corresponding endpoint feature data through iterative training.

[0381] In another possible implementation, some feature maps (or feature blocks) use shared endpoint data, and / or some feature maps (or feature blocks) use shared weight data. The feature maps using shared endpoint data may or may not overlap with the feature maps using shared weight data. In this implementation, the decoding end can obtain all the weight feature data and all the endpoint feature data of the N feature maps through one or more combinations of methods such as data reuse, acquisition from a shared dataset, and direct decoding, according to the corresponding implementation logic. These details will not be elaborated upon here.

[0382] The second scenario: The weighted feature data in the feature texture data is obtained from a preset weighted dataset.

[0383] The preset weight dataset can be shared by one or more textures. If the decoding end is a terminal, the preset weight dataset can be stored in the cloud or on the terminal. If the decoding end is in the cloud, the preset weight dataset is stored in the cloud.

[0384] When the weight feature data is obtained from a preset weight dataset, the first feature data includes first mapping information, which indicates the correspondence between the weight feature data corresponding to the N feature maps and the weight feature data in the remainder weight dataset.

[0385] The first mapping information can be in the form of key-value pairs. Each key in the first mapping information represents the identifier of each weight feature data corresponding to N feature maps, and the value corresponding to each key represents the identifier of the corresponding weight feature data in the preset weight dataset.

[0386] As an example, taking N feature maps comprising 240 weighted feature data points, and a preset weighted dataset comprising 2000 weighted feature data points, the first mapping information may include: {[1:201]; [2:1632]; [3:1594]; ...; [240:44]}. Here, the indices "1, 2, 3, ..., 240" before the colon represent 240 keys, indicating the identifiers of the 240 weighted feature data points corresponding to the N feature maps. The indices after the colon represent values ​​ranging from 1 to 2000, indicating the identifiers of the 2000 weighted feature data points in the preset weighted dataset. In this example, the first mapping information indicates that the 201st weight feature data in the preset weight dataset can be used as the 1st weight feature vector corresponding to the N feature maps, the 1632nd weight feature data in the preset weight dataset can be used as the 2nd weight feature vector corresponding to the N feature maps, the 1594th weight feature data in the preset weight dataset can be used as the 3rd weight feature vector corresponding to the N feature maps, and the 44th weight feature data in the preset weight dataset can be used as the 240th weight feature vector corresponding to the N feature maps.

[0387] Besides the examples above, the structure of the first mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the positions of the keys and values ​​in the above examples, another type of first mapping information is obtained, that is, each value in the first mapping information represents the identifier of each weight feature data corresponding to the N feature maps, and the key corresponding to each value represents the identifier of the corresponding weight feature data in the weight dataset.

[0388] It should be understood that the first mapping information can be learned during the process of encoding the texture through multiple iterations of training.

[0389] Besides the two cases mentioned above, in some other cases, some weight feature data corresponding to N feature maps are obtained from the weight dataset, while other weight feature data are obtained by decoding the first feature data.

[0390] In summary, some or all of the weighted feature data in the feature texture data are obtained from a weighted dataset (such as shared weighted data or a preset weighted dataset). This weighted dataset can be obtained from the bitstream or from a locally stored preset weighted dataset.

[0391] After introducing how to determine the weighted feature data in the feature texture data, we will now introduce how to determine the endpoint feature data in the feature texture data.

[0392] In this embodiment, the endpoint feature data in the feature texture data is obtained based on the endpoint feature data in the first feature data, or from a preset endpoint dataset. The following will describe different scenarios.

[0393] The first case: The endpoint feature data in the feature texture data is obtained based on the endpoint feature data in the first feature data.

[0394] When the endpoint feature data is obtained based on the endpoint feature data in the first feature data, the first feature data includes partially encoded data corresponding to the N feature maps of the texture, or the first feature data includes encoded shared endpoint data, and the shared endpoint data includes endpoint feature data shared by the N feature maps.

[0395] In the case where the first feature data includes encoded data corresponding to the N feature maps of the texture, the process of decoding the first feature data to obtain the feature texture data includes: decoding the first feature data to obtain the first part of the data corresponding to the N feature maps, and obtaining the feature texture data based on the first part of the data corresponding to the N feature maps. The specific implementation method can be found in the relevant introduction above, and will not be repeated here.

[0396] When the first feature data includes encoded shared endpoint data, the encoded shared endpoint data is decoded to obtain shared endpoint data, and the endpoint feature data corresponding to each of the N feature maps is obtained from the shared endpoint data.

[0397] In one implementation, the first feature data further includes second mapping information, which indicates the correspondence between the endpoint feature data corresponding to each of the N feature maps and the endpoint feature data in the shared endpoint data. Based on this, the endpoint feature data corresponding to each of the N feature maps is obtained from the shared endpoint data according to the second mapping information.

[0398] Each of the N feature maps includes S feature blocks. The second mapping information indicates the correspondence between the endpoint feature data corresponding to each feature block of the N feature maps and the endpoint feature data in the shared endpoint data. According to the second mapping information, the endpoint feature data corresponding to each feature block of the N feature maps is obtained from the shared endpoint data.

[0399] In this embodiment, the total number of endpoint feature data in the shared weight data is less than the total number of endpoint feature data corresponding to the N feature maps. The total number of endpoint feature data in the shared endpoint data is denoted as R, where one feature block has one endpoint feature data, and R is less than N×S.

[0400] The second mapping information can be in key-value form. Each key in the second mapping information represents the identifier of each endpoint feature data in the shared endpoint data, and the value corresponding to each key represents the identifier of the corresponding endpoint feature data in the N feature maps.

[0401] As an example, let Z be the total number of endpoint feature data in the shared endpoint data. One feature block has one endpoint feature data. Taking N = 3, S = 20, and Z = 30 as an example, the N feature maps have a total of 3 × 20 = 60 endpoint feature data. The second mapping information includes {1: [2~14, 17]; 2: [25~30, 38, 56]; 3: [33~37]; ...; 30: [48~54, 60]}. Among them, the serial numbers "1, 2, 3, ... 30" before ":" are 30 keys, representing the identifiers of the 30 endpoint feature data in the shared endpoint data. The serial numbers in "[]" are values. The 30 "[]" contain a total of 60 values, representing the identifiers of the 60 endpoint feature data corresponding to the N feature maps. In this example, the second mapping information indicates that the first endpoint feature data in the shared endpoint data can be used as the 2nd to 14th and 17th endpoint feature vectors corresponding to the N feature maps; the second endpoint feature data in the shared endpoint data can be used as the 25th to 30th, 38th and 56th endpoint feature vectors corresponding to the N feature maps; the third endpoint feature data in the shared endpoint data can be used as the 33rd to 37th endpoint feature vectors corresponding to the N feature maps; and the first endpoint feature data in the shared endpoint data can be used as the 48th to 54th and 60th endpoint feature vectors corresponding to the N feature maps.

[0402] Alternatively, the second mapping information can be in key-value form, where each key in the second mapping information represents the identifier of each endpoint feature data corresponding to N feature maps, and the value corresponding to each key represents the identifier of the corresponding endpoint feature data in the shared endpoint data.

[0403] As an example, the second mapping information includes {[1:14];[2:3];[3:20];...;[60:3]}, where the serial numbers "1, 2, 3, ... 60" before the ":" are 60 keys, representing the identifiers of the 60 endpoint feature data corresponding to the N feature maps, and the serial numbers after the ":" are values, with a value range of 1 to 30, representing the identifiers of the 30 weight feature data in the shared weight data.

[0404] Besides the examples above, the structure of the second mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the keys and values ​​in the above examples, another type of second mapping information can be obtained.

[0405] The endpoint feature data in this application embodiment is a data set, which can be in the form of a vector or other forms, and this application embodiment does not limit this. Furthermore, the fourth index information in the first feature data can be encoded information or unencoded information, and this application embodiment does not limit this.

[0406] It should be understood that the second mapping information can be learned through training multiple times to obtain shared endpoint data. Of course, the fourth index second mapping information can also be a pre-defined fixed relationship, rather than something learned through training. In this case, the second mapping information can also be encoded into the bitstream to ensure that the decoding end can obtain the correspondence between the endpoint feature data corresponding to each feature block in the N feature maps and the endpoint feature data in the shared endpoint data.

[0407] In another implementation, the first feature data does not include the second mapping information. The decoder obtains the endpoint feature data corresponding to each of the N feature maps from the shared endpoint data according to the second correspondence. The second correspondence is preset, i.e., predetermined, and remains fixed throughout the process of obtaining the shared endpoint data through multiple iterations of training by the encoder.

[0408] In some embodiments, where the first feature data includes encoded shared endpoint data, the first feature data also includes encoded weight feature data corresponding to the N feature maps. That is, the weight feature data corresponding to the N feature maps are all encoded into the bitstream.

[0409] The second scenario: The weighted feature data in the feature texture data is obtained from the preset endpoint dataset.

[0410] The preset endpoint dataset can be shared by one or more textures. If the decoding end is a terminal, the preset endpoint dataset can be stored in the cloud or on the terminal. If the decoding end is in the cloud, the preset endpoint dataset is stored in the cloud.

[0411] When the endpoint feature data is obtained from a preset endpoint dataset, the first feature data includes second mapping information, which indicates the correspondence between the endpoint feature data corresponding to the N feature maps and the endpoint feature data in the preset endpoint dataset.

[0412] The second mapping information can be in key-value form. Each key in the second mapping information represents the identifier of each endpoint feature data corresponding to the N feature maps, and the value corresponding to each key represents the identifier of the corresponding endpoint feature data in the preset endpoint dataset.

[0413] As an example, taking N feature maps comprising 60 endpoint feature data points and a preset endpoint dataset comprising 1000 endpoint feature data points as an example, the second mapping information may include: {[1:198]; [2:163]; [3:594]; ...; [60:413]}. Here, the serial numbers "1, 2, 3, ..., 60" before the colon represent 60 keys, indicating the identifiers of the 60 endpoint feature data points corresponding to the N feature maps, and the serial numbers after the colon represent values ​​ranging from 1 to 1000, indicating the identifiers of the 1000 endpoint feature data points in the preset endpoint dataset. In this example, the second mapping information indicates that the 198th endpoint feature data in the preset endpoint dataset can be used as the 1st endpoint feature vector corresponding to the N feature maps, the 163rd endpoint feature data in the preset endpoint dataset can be used as the 2nd endpoint feature vector corresponding to the N feature maps, the 594th endpoint feature data in the preset endpoint dataset can be used as the 3rd endpoint feature vector corresponding to the N feature maps, and the 413th endpoint feature data in the preset endpoint dataset can be used as the 240th endpoint feature vector corresponding to the N feature maps.

[0414] Besides the examples above, the structure of the second mapping information can also be other, and this application embodiment does not limit it. For example, by swapping the positions of the keys and values ​​in the above examples, another type of second mapping information is obtained, that is, each value in the second mapping information represents the identifier of each endpoint feature data corresponding to N feature maps, and the key corresponding to each value represents the identifier of the corresponding endpoint feature data in the preset endpoint dataset.

[0415] It should be understood that the second mapping information can be learned during the process of encoding the texture through multiple iterations of training.

[0416] Besides the implementation methods described above, in other possible implementations, some feature maps use weight feature data from a preset weight dataset, and / or some feature maps use endpoint feature data from a preset endpoint dataset. When both sets of feature maps exist simultaneously, they may or may not overlap. In this implementation, the decoding end can obtain all the weight feature data and all the endpoint feature data of the N feature maps through one or more combinations of methods such as data reuse, acquisition from a preset dataset, and direct decoding, according to the corresponding implementation logic. These details will not be elaborated upon here.

[0417] Besides the two cases mentioned above, in some other cases, some endpoint feature data corresponding to N feature maps are obtained from the endpoint dataset, while other endpoint feature data are obtained by decoding the first feature data.

[0418] In summary, some or all of the endpoint feature data in the feature texture data are obtained from the endpoint dataset (such as shared endpoint data or a preset endpoint dataset). The weight dataset can be obtained from the bitstream or from a locally stored preset weight dataset.

[0419] After obtaining the weight feature data and endpoint feature data corresponding to each feature map, the decoding end combines the endpoint feature data with the weight feature data to obtain the complete information of the feature map. Specifically, the decoding end can combine the endpoint feature data and weight feature data corresponding to each feature block in the feature map to obtain the complete information for that feature block. After obtaining the complete information for all feature blocks, the feature texture data is obtained.

[0420] Step 16022: Decode the feature texture data to obtain the second feature data.

[0421] After obtaining the feature texture data, the decoding end can perform texture decoding on the feature texture data to obtain the second feature data. The second feature data consists of all the data from the N feature maps.

[0422] The texture decoding method can be ASTC, BC, or other texture decoding methods, and this application embodiment does not limit this.

[0423] Step 1603: Based on the second nonlinear transformation parameters, decode the second feature data to obtain the texture.

[0424] After obtaining the second feature data, the decoding end can decode the second feature data based on the second nonlinear transformation parameters to obtain the texture, that is, reconstruct the texture.

[0425] The process of decoding the second feature data based on the second nonlinear transformation parameter to obtain the texture includes the following steps 16031 and 16032.

[0426] Step 16031: Based on the second feature data, obtain one or more feature vectors.

[0427] That is, feature vectors are constructed from the second feature data to obtain one or more feature vectors.

[0428] In this embodiment, the process of constructing feature vectors at the decoding end is the same as the process of constructing feature vectors at the encoding end during the iteration process, and also includes at least two implementation methods.

[0429] The first implementation corresponds to the method where the encoding end samples (or extracts) the first feature vector of each feature block from the feature map to obtain one or more feature vectors, and the decoding end extracts the first feature vector of each feature block from the second feature data to obtain the aforementioned one or more feature vectors. For specific implementation details, please refer to step (22) in the embodiment of Figure 4, or to related technologies. This application will not elaborate further on this aspect.

[0430] The second implementation corresponds to the method where the encoder combines the feature vector of the auxiliary block with the first feature vector of the feature block to be decoded (such as the first feature block) and the decoder also combines the feature vector of the auxiliary block with the first feature vector of the feature block to be decoded to obtain one or more of the above feature vectors.

[0431] Taking the example of one or more feature vectors including a second feature vector, the decoding end obtains one or more feature vectors based on the second feature data, including: obtaining a first feature vector based on the second feature block; obtaining a third feature vector; and processing the first feature vector based on the third feature vector to obtain the second feature vector. Here, the first feature vector and the third feature vector correspond to different feature maps or different feature blocks within the same feature map.

[0432] In this embodiment, the first feature vector corresponds to the first feature block, and the third feature vector corresponds to the second feature block. That is, the second feature block is an auxiliary block to the first feature block, used to assist in decoding the texture data corresponding to the first feature block. The first feature vector is obtained by sampling the first feature block, and the third feature vector is obtained by sampling the second feature block.

[0433] In one possible implementation, the first feature vector of the first feature block is extracted from the second feature data, and the first feature vector of the second feature block is obtained from the cache. It should be understood that in this implementation, when the first feature block is decoded, the second feature block is already a decoded feature block. Before decoding the first feature block, the first feature vector of the second feature block has already been extracted from the second feature data during the decoding process and can be retained in the cache.

[0434] In another possible implementation, the second feature block can be decoded in parallel with the first feature block, or the second feature block can be decoded after the first feature block. In this case, the first feature vector of the second feature block can be obtained from the second feature data during the decoding of the first feature block. Of course, the embodiments of this application are not limited to where the first feature vector of the second feature block is obtained during the decoding of the first feature block.

[0435] As can be seen from the examples of texture encoding methods, there are multiple ways for the encoding end to determine the auxiliary block. Similarly, there are multiple ways for the decoding end to determine the auxiliary block, which will be introduced next.

[0436] In one implementation, the bitstream includes auxiliary information, and the decoder obtains a third feature vector based on this auxiliary information. Before processing the first feature vector based on the third feature vector, the decoder also parses the bitstream to obtain the auxiliary information.

[0437] The auxiliary information is used to indicate the third feature vector. For example, it can indicate that the third feature vector is used to process the first feature vector, or that it is used to construct a feature vector from the first feature block. In one possible implementation, the auxiliary information can indicate the location of an auxiliary sampling point. The decoder determines an auxiliary feature map based on this location and samples the auxiliary feature map to obtain the third feature vector. Alternatively, the auxiliary information can indicate the location of the third feature vector, and the decoder directly obtains the third feature vector from the second feature data based on this location. A detailed description of this auxiliary information can be found in the texture encoding method embodiments, and will not be repeated here.

[0438] Based on this, the decoding end can determine the feature map or feature block corresponding to the third feature vector based on the auxiliary information, and obtain the third feature vector based on the feature map or feature block corresponding to the third feature vector.

[0439] In some embodiments, a feature block can have multiple auxiliary blocks, which come from different feature maps or different feature blocks within the same feature map. For example, the second feature data includes data from multiple feature maps of the texture, and also includes a first feature vector (called a fourth feature vector) of the third feature block. The second and third feature blocks belong to different feature maps, or to different feature blocks within the same feature map; that is, the third and fourth feature vectors correspond to different feature maps or different feature blocks within the same feature map, and the fourth feature vector also corresponds to a different feature map or a different feature block within the same feature map as the first feature vector. Processing the first feature vector based on the third feature vector includes processing the first feature vector based on both the third and fourth feature vectors. In other words, both the second and third feature blocks are auxiliary blocks of the first feature block.

[0440] In this context, the second and third feature blocks can correspond to the same region of the texture or different regions; the following descriptions will focus on the same region. That is, multiple auxiliary blocks of a feature block correspond to the same region of the texture. Therefore, the maximum number of auxiliary blocks for a feature block does not exceed the total number of feature maps N of the texture. Correspondingly, the third and fourth feature vectors correspond to the same region of the texture, such as the same pixel, or they can correspond to different regions; the following descriptions will focus on the same region. That is, the auxiliary feature vectors used to process the first feature vector correspond to the same region of the texture.

[0441] In another implementation, the decoding end determines the third feature vector (i.e., the first feature vector of the second feature block) based on the third feature vector before processing the first feature vector, according to the auxiliary block selection rules.

[0442] In the first implementation of determining the third feature vector according to the auxiliary block selection rule, the second feature block can be the feature block with the lowest complexity in the surrounding area of ​​the first feature block. That is, the decoding end determines the auxiliary block of the first feature block by itself through complexity detection according to this rule.

[0443] In the second implementation of determining the second feature block according to the auxiliary block selection rule, the second feature block is a feature block of a preset region. That is, the decoding end directly uses the feature block of the preset region as an auxiliary block of the first feature block according to the rule.

[0444] In this embodiment of the application, after determining the auxiliary block of the first feature block (such as the second feature block, or the second feature block and the third feature block), the decoding end processes the first feature vector of the first feature block based on the first feature vector of the auxiliary block to obtain the second feature vector of the first feature block.

[0445] In the first implementation, the first feature vector of the first feature block is concatenated with the first feature vector of the auxiliary block to obtain the second feature vector of the first feature block.

[0446] In this embodiment, the first feature vector of the auxiliary block can be concatenated before or after the first feature vector of the first feature block, depending on the concatenation rules. When there are multiple auxiliary blocks, this embodiment does not limit the concatenation position of the first feature vectors of these multiple auxiliary blocks; they can be concatenated according to the concatenation rules. As an example, the multiple auxiliary blocks are concatenated after the first feature vector of the first feature block in order of the size of their respective feature maps.

[0447] In this implementation, the length of the first feature vector is less than the length of the second feature vector. For each feature block without an auxiliary block, the decoder can extend the length of the first feature vector of that feature block, for example, by adding a segment of elements with a value of 0, so that the length of the feature vector of that feature block is increased to be the same as the length of the second feature vector of the first feature block. This ensures that the length of each second feature vector in the input nonlinear transformation model is the same, i.e., the dimension is the same. Of course, the method of length extension is not limited to adding a segment of elements with a value of 0; elements with other values ​​can also be added. This application embodiment does not limit this.

[0448] In the second implementation, the first feature vector of the first feature block and / or the first feature vector of the auxiliary block are transformed and then concatenated to obtain the second feature vector of the first feature block.

[0449] In the case of multiple auxiliary blocks, the decoding end can transform one or more of the first feature vectors of the first feature block and the first feature vectors of the multiple auxiliary blocks according to the transformation rules before concatenating them. The transformations include one or more of translation, rotation, and mirroring. The transformation rules can specify which feature blocks(s) to transform in what way, and in what order to concatenate them.

[0450] In some embodiments, the transformation rule is predetermined, i.e., determined in advance, and both the encoder and decoder process the first feature vector according to the predetermined transformation rule. In other embodiments, the bitstream includes the transformation rule, for example, if the transformation rule is determined by the encoder, then the bitstream may include the transformation rule. Of course, even if the transformation rule is predetermined, the bitstream may still include the transformation rule to ensure that the decoder knows how to process the first feature vector.

[0451] In this implementation, the length of the first feature vector is also less than the length of the second feature vector. For each feature block that does not have an auxiliary block, the decoder can extend the length of the first feature vector of that feature block. The method of length extension can be referred to the above description, and will not be repeated here.

[0452] In the third implementation, the first feature vector of the first feature block and the first feature vector of the auxiliary block are calculated element-wise to obtain the second feature vector of the first feature block. The calculation may include one or more of addition, subtraction, multiplication, and division.

[0453] The decoding end can perform element-wise calculations on the first feature vector of the first feature block and the first feature vector of the auxiliary block according to the calculation rules. These calculation rules can specify the calculation method. When there are multiple auxiliary blocks, the calculation rules can specify the calculation method between the first feature vector of the first feature block and the first feature vectors of multiple auxiliary blocks, such as which vector is added to which vector, or which vector is multiplied by which vector.

[0454] In some embodiments, the calculation rules are predetermined, i.e., determined in advance, and both the encoder and decoder process the first feature vector according to the predetermined calculation rules. In other embodiments, the bitstream includes the calculation rules, for example, if the calculation rules are determined by the encoder, then the bitstream may include the calculation rules. Of course, even if the calculation rules are predetermined, the bitstream may still include the calculation rules to ensure that the decoder knows how to process the first feature vector.

[0455] For the third implementation, if the length of the first eigenvector of the first feature block is the same as the length of the second eigenvector of the first feature block, then for each feature block without an auxiliary block, the first eigenvector of that feature block can be directly used as the second eigenvector of that feature block, or in other words, the first eigenvector of that feature block can be used as the eigenvector among the final multiple eigenvectors. If the length of the first eigenvector of the first feature block is less than the length of the second eigenvector, then for each feature block without an auxiliary block, the length of the first eigenvector of that feature block can be extended. The method of length extension can be referred to the above description, and will not be repeated here.

[0456] The three implementation methods described above can be used individually or in combination. For example, the first and third implementation methods can be combined. When there are multiple auxiliary blocks, these auxiliary blocks can be calculated element-wise and then concatenated with the first feature vector of the first feature block. In addition, there are many other vector processing methods, which are not limited in this embodiment.

[0457] Step 16032: Perform a nonlinear transformation on the multiple feature vectors according to the second nonlinear transformation parameters to obtain the texture.

[0458] The second nonlinear transformation parameter can be a parameter of a neural network model, a parameter of a nonlinear function, or a parameter of another nonlinear transformation model. Taking a neural network model as an example, after obtaining multiple feature vectors, the decoder can input these feature vectors into the neural network model to perform a nonlinear transformation, thus obtaining the texture output by the neural network model. Similarly, taking a nonlinear function as an example, after obtaining multiple feature vectors, the decoder can input these feature vectors into the nonlinear function to obtain the texture output by the nonlinear transformation function.

[0459] Next, the texture decoding method provided in the embodiments of this application will be described again by way of example with reference to Figures 17 to 22.

[0460] Figure 17 is a flowchart of another texture decoding method provided in an embodiment of this application. This decoding method matches the encoding method shown in Figure 11 or Figure 12. Referring to Figure 17, after the decoding end obtains the bitstream, it decodes the data in the bitstream one by one according to the coordinates to be decoded (including parsing and decoding) to obtain the second feature data (i.e., feature map) and the second nonlinear transformation parameter. Feature vectors are constructed on the feature map to obtain multiple feature vectors. Nonlinear transformation (i.e., secondary decoding) is performed on the multiple feature vectors according to the second nonlinear transformation parameter to obtain the texture (i.e., the decoding result). The texture can be sent to the rendering module for image rendering.

[0461] Figure 18 is a flowchart of another texture decoding method provided in an embodiment of this application. This decoding method matches the encoding method shown in Figure 15. Referring to Figure 18, based on Figure 17, during the feature vector construction process, the decoding end also obtains an auxiliary feature vector, processes the first feature vector based on the auxiliary feature vector to obtain a second feature vector, and performs a nonlinear transformation on the second feature vector to obtain the decoding result.

[0462] Figure 19 is a flowchart of another texture decoding method provided in an embodiment of this application. This decoding method is a further refinement of the decoding method shown in Figure 18. Referring to Figure 19, the auxiliary feature vector can be obtained from the feature texture data through texture decoding and sampling based on the coordinates of the auxiliary sampling points (coordinates in the auxiliary information).

[0463] Figure 20 is a flowchart of another texture decoding method provided in an embodiment of this application. This decoding method is a further refinement of the decoding method shown in Figure 18. Referring to Figure 19, the auxiliary feature vector can be obtained from the cached texture decoded data (called the texture cache) according to the auxiliary block selection rules.

[0464] Figure 21 is a flowchart of another texture decoding method provided in an embodiment of this application. This texture method matches the encoding method shown in Figure 14. Referring to Figure 21, based on Figure 17, during the decoding process, the weighted feature data of feature map 1 is reused to obtain the texture feature maps of feature maps 2 to N, that is, all feature texture data of feature maps 2 to N are obtained.

[0465] Figure 22 is a flowchart of another texture decoding method provided in an embodiment of this application. This texture method matches the encoding method shown in Figure 13. Referring to Figure 22, based on Figure 17, during the decoding process, the first feature data is decoded to directly obtain the weight feature data and endpoint feature data corresponding to each feature map in feature map 1 to feature map N.

[0466] This application does not limit whether the decoding of multiple feature maps of a texture is serial or parallel, nor does it limit whether the decoding of multiple feature blocks of the same feature map is parallel or serial. In some embodiments, multiple feature maps can be decoded serially, and multiple feature blocks within each feature map can also be decoded serially. In other embodiments, multiple feature maps are decoded in parallel, and multiple feature blocks within each feature map are also decoded in parallel. In still other embodiments, multiple feature maps are decoded serially, and multiple feature blocks within each feature map are decoded in parallel. In still other embodiments, some feature maps are decoded serially, while other feature maps are decoded in parallel; some feature blocks are decoded in parallel, while other feature blocks are decoded serially.

[0467] In summary, in this embodiment, both the first feature data and the first nonlinear transformation parameter in the bitstream are double-compressed data, resulting in a high compression rate. The texture can be reconstructed by the decoding end through secondary decoding. The decoding end can achieve fast decoding regardless of whether it uses a CPU, GPU, or other processor.

[0468] Figure 23 is a schematic diagram of a texture decoding device provided in an embodiment of this application. The decoding device can be implemented by software, hardware, or a combination of both as part or all of a decoding device (such as the decoder described above). Referring to Figure 23, the decoding device includes: an acquisition module 2301, a first decoding module 2302, and a second decoding module 2303.

[0469] The acquisition module 2301 is used to acquire the bitstream and parse the bitstream to obtain the first feature data and the first nonlinear transformation parameters of the texture;

[0470] The first decoding module 2302 is used to decode the first feature data to obtain the second feature data, and to decode the first nonlinear transformation parameter to obtain the second nonlinear transformation parameter;

[0471] The second decoding module 2303 is used to decode the second feature data based on the second nonlinear transformation parameters to obtain the texture.

[0472] In one possible implementation, the second decoding module 2303 includes:

[0473] Feature vector construction unit, used to obtain one or more feature vectors based on the second feature data;

[0474] A nonlinear transformation unit is used to perform a nonlinear transformation on one or more feature vectors according to a second nonlinear transformation parameter to obtain the texture.

[0475] In one possible implementation, the one or more feature vectors include a second feature vector; the feature vector building unit is specifically used for:

[0476] Based on the second feature data, the first feature vector is obtained;

[0477] Obtain the third feature vector. The first and third feature vectors correspond to different feature maps or different feature blocks in the same feature map.

[0478] Based on the third feature vector, the first feature vector is processed to obtain the second feature vector.

[0479] In one possible implementation, the feature vector construction unit is specifically used for:

[0480] Obtain auxiliary information that indicates the third feature vector;

[0481] Based on this auxiliary information, the third feature vector is obtained.

[0482] In one possible implementation, the feature vector construction unit is specifically used for:

[0483] Based on this auxiliary information, the feature map or feature block corresponding to the third feature vector is determined;

[0484] The third feature vector is obtained based on the feature map or feature block corresponding to the third feature vector.

[0485] In one possible implementation, the first feature vector corresponds to the first feature block, and the third feature vector corresponds to the second feature block. The second feature block is the feature block with the lowest complexity in the surrounding region of the first feature block, or the second feature block is a feature block in a preset region.

[0486] In one possible implementation, the feature vector building unit is also used for:

[0487] Obtain the fourth feature vector. The fourth feature vector corresponds to a different feature map or a different feature block in the same feature map as the third feature vector. The fourth feature vector also corresponds to a different feature map or a different feature block in the same feature map as the first feature vector.

[0488] The eigenvector construction unit is specifically used for:

[0489] The first feature vector is processed based on the third and fourth feature vectors.

[0490] In one possible implementation, the first decoding module 2302 includes:

[0491] The first decoding unit is used to decode the first feature data to obtain feature texture data;

[0492] The second decoding unit is used to perform texture decoding on the feature texture data to obtain the second feature data.

[0493] In one possible implementation, the feature texture data includes weight feature data and endpoint feature data corresponding to N feature maps of the texture, where N is a positive integer.

[0494] In one possible implementation, some or all of the weight feature data corresponding to the N feature maps are obtained from the weight dataset.

[0495] In one possible implementation, the first feature data includes first mapping information, which indicates the correspondence between some or all of the weight feature data corresponding to the N feature maps and the weight feature data in the weight dataset. The some or all of the weight feature data corresponding to the N feature maps are obtained from the weight dataset based on the first mapping information.

[0496] In one possible implementation, the weight dataset is obtained from the bitstream.

[0497] In one possible implementation, some or all of the endpoint feature data corresponding to the N feature maps are obtained from the endpoint dataset.

[0498] In one possible implementation, the first feature data includes second mapping information, which indicates the correspondence between some or all of the endpoint feature data corresponding to the N feature maps and the endpoint feature data in the endpoint dataset. The some or all of the endpoint feature data corresponding to the N feature maps are obtained from the endpoint dataset based on the second mapping information.

[0499] In one possible implementation, the endpoint dataset is obtained from the bitstream.

[0500] In one possible implementation, the first feature data includes endpoint feature data and weight feature data corresponding to the first feature map of the texture, and the feature texture data includes endpoint feature data and weight feature data corresponding to the second feature map of the texture, as well as endpoint feature data and weight feature data corresponding to the first feature map.

[0501] The endpoint feature data corresponding to the second feature map is obtained based on the endpoint feature data corresponding to the first feature map, and / or,

[0502] The weighted feature data corresponding to the second feature map is obtained based on the weighted feature data corresponding to the first feature map.

[0503] In this embodiment, the first feature data and the first nonlinear transformation parameter in the bitstream are both data that have undergone secondary compression. The compression rate of the bitstream is very high, and the texture can be reconstructed by the decoding end through secondary decoding.

[0504] It should be noted that the texture decoding device provided in the above embodiments is only illustrated by the division of the above functional modules when decoding the texture bitstream. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the texture decoding device and the texture decoding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0505] Figure 24 is a schematic diagram of a texture encoding device provided in an embodiment of this application. The encoding device can be implemented by software, hardware, or a combination of both as part or all of an encoding device (such as the encoder described above). Referring to Figure 24, the encoding device includes: a first encoding module 2401, a second encoding module 2402, and a third encoding module 2403.

[0506] The first encoding module 2401 is used to encode the texture to obtain the third feature data and the second nonlinear transformation parameters;

[0507] The second encoding module 2402 is used to encode the third feature data to obtain the first feature data, and to encode the second nonlinear transformation parameter to obtain the first nonlinear transformation parameter;

[0508] The third encoding module 2403 is used to encode the first feature data and the first nonlinear transformation parameters into the code stream.

[0509] In one possible implementation, the third feature data includes data corresponding to one or more feature maps, the one or more feature maps including a first feature map, the first feature map including a first feature block, and the second feature block being a feature block located in the same feature map as the first feature block but in a different region, or the second feature block being a feature block located in a different feature map than the first feature block. The encoding device further includes:

[0510] The determination module is used to determine auxiliary information, which is used to indicate the third feature vector, which is the feature vector corresponding to the second feature block, and the third feature vector is used to construct the feature vector of the first feature block;

[0511] The third encoding module 2403 is also used to encode the auxiliary information into the bitstream.

[0512] In one possible implementation, this auxiliary information is used to indicate the third feature vector, including:

[0513] Auxiliary information is used to indicate the second feature block;

[0514] The third feature vector is obtained based on the second feature block.

[0515] In one possible implementation, the determining module includes:

[0516] The texture is subjected to complexity detection to determine this auxiliary information.

[0517] In one possible implementation, the second feature block is the feature block with the lowest complexity in the region surrounding the first feature block, or the second feature block is a feature block in a preset region.

[0518] In one possible implementation, the first feature data includes first mapping information, which is used to indicate the correspondence between some or all of the weight feature data corresponding to the N feature maps of the texture and the weight feature data in the weight dataset. The first mapping information is used to obtain some or all of the weight feature data corresponding to the N feature maps from the weight dataset, where N is a positive integer.

[0519] In one possible implementation, the bitstream also includes a weighted dataset.

[0520] In one possible implementation, the first feature data includes second mapping information, which is used to indicate the correspondence between some or all of the endpoint feature data corresponding to the N feature maps of the texture and the endpoint feature data in the endpoint dataset. The second mapping information is used to obtain some or all of the endpoint feature data corresponding to the N feature maps from the endpoint dataset, where N is a positive integer.

[0521] In one possible implementation, the bitstream also includes an endpoint dataset.

[0522] In one possible implementation, the first feature data includes endpoint feature data and weight feature data corresponding to the first feature map of the texture;

[0523] The endpoint feature data corresponding to the first feature map is used to reconstruct the endpoint feature data corresponding to the second feature map of the texture, and / or,

[0524] The weighted feature data corresponding to the first feature map is used to reconstruct the weighted feature data corresponding to the second feature map.

[0525] In this embodiment, the encoding end can perform secondary compression on the third feature data of the texture and the second nonlinear transformation parameters, thereby improving the compression rate and reducing the bitstream's occupation of network transmission bandwidth and storage space.

[0526] It should be noted that the texture encoding device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the texture encoding device and texture encoding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0527] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the texture encoding method or the steps of the texture decoding method shown in the above embodiments.

[0528] This application also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the steps of the texture encoding method shown in the above embodiments, or to perform the steps of the texture decoding method shown in the above embodiments.

[0529] This application also provides a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the texture encoding method shown in the above embodiments, or to perform the steps of the texture decoding method shown in the above embodiments.

[0530] This application also provides an encoding / decoding system, which includes an encoding device and a decoding device. The encoding device is used to implement the steps of the texture encoding method shown in the above embodiments, and the decoding device is used to implement the steps of the texture decoding method shown in the above embodiments.

[0531] This application also provides an encoded bitstream, which is generated according to the texture encoding method shown in the above embodiments.

[0532] This application also provides a computer-readable storage medium storing a bitstream generated according to the texture encoding method shown in the above embodiments.

[0533] This application also provides an apparatus for storing a bitstream, the apparatus including a receiver and at least one storage medium, the receiver being used to receive a bitstream generated according to the texture encoding method shown in the above embodiments, and the at least one storage medium being used to store the bitstream.

[0534] This application also provides an apparatus for transmitting a bitstream, which includes a transmitter and a receiver. The receiver is used to receive a bitstream generated according to the texture encoding method shown in the above embodiments, and the transmitter is used to transmit the bitstream to an end-side device through a transmission medium.

[0535] This application also provides an apparatus for transmitting a bitstream, the apparatus including a transmitter and at least one storage medium, the at least one storage medium being used to store a bitstream generated according to the texture encoding method shown in the above embodiments, the transmitter being used to obtain the bitstream from the storage medium and transmit the bitstream to an end-side device through the transmission medium.

[0536] This application also provides a system for distributing bitstreams. The system includes at least one storage medium for storing bitstreams generated according to the texture encoding method shown in the above embodiments. The streaming media device is used to obtain a target bitstream from the at least one storage medium and send the target bitstream to an end-side device. The streaming media device includes a content server or a content distribution server.

[0537] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0538] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.

[0539] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the textures involved in the embodiments of this application were all obtained under full authorization.

[0540] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for decoding textures, characterized in that, The method includes: Acquire the bitstream and parse the bitstream to obtain the first feature data of the texture and the first nonlinear transformation parameters; The first feature data is decoded to obtain the second feature data, and the first nonlinear transformation parameter is decoded to obtain the second nonlinear transformation parameter; Based on the second nonlinear transformation parameters, the second feature data is decoded to obtain the texture.

2. The method as described in claim 1, characterized in that, The step of decoding the second feature data based on the second nonlinear transformation parameters to obtain the texture includes: Based on the second feature data, one or more feature vectors are obtained; The texture is obtained by performing a nonlinear transformation on the one or more feature vectors according to the second nonlinear transformation parameters.

3. The method as described in claim 2, characterized in that, The one or more feature vectors include a second feature vector; The process of obtaining one or more feature vectors based on the second feature data includes: Based on the second feature data, a first feature vector is obtained; Obtain the third feature vector, where the first feature vector and the third feature vector correspond to different feature maps or different feature blocks in the same feature map; Based on the third feature vector, the first feature vector is processed to obtain the second feature vector.

4. The method as described in claim 3, characterized in that, The process of obtaining the third feature vector includes: Obtain auxiliary information, which is used to indicate the third feature vector; Based on the auxiliary information, the third feature vector is obtained.

5. The method as described in claim 4, characterized in that, The process of obtaining the third feature vector based on the auxiliary information includes: Based on the auxiliary information, determine the feature map or feature block corresponding to the third feature vector; The third feature vector is obtained based on the feature map or feature block corresponding to the third feature vector.

6. The method according to any one of claims 3-5, characterized in that, The first feature vector corresponds to the first feature block, and the third feature vector corresponds to the second feature block. The second feature block is the feature block with the lowest complexity in the surrounding area of ​​the first feature block, or the second feature block is a feature block in a preset area.

7. The method according to any one of claims 3-6, characterized in that, Before processing the first feature vector based on the third feature vector to obtain the second feature vector, the method further includes: Obtain a fourth feature vector, wherein the fourth feature vector corresponds to a different feature map or a different feature block in the same feature map as the third feature vector, and the fourth feature vector corresponds to a different feature map or a different feature block in the same feature map as the first feature vector; The processing of the first feature vector based on the third feature vector includes: The first feature vector is processed based on the third feature vector and the fourth feature vector.

8. The method according to any one of claims 1-7, characterized in that, Decoding the first feature data to obtain the second feature data includes: Decode the first feature data to obtain feature texture data; The feature texture data is then decoded to obtain the second feature data.

9. The method as described in claim 8, characterized in that, The feature texture data includes weight feature data and endpoint feature data corresponding to N feature maps of the texture, where N is a positive integer.

10. The method as described in claim 9, characterized in that, The weight feature data corresponding to some or all of the N feature maps are obtained from the weight dataset.

11. The method as described in claim 10, characterized in that, The first feature data includes first mapping information, which is used to indicate the correspondence between some or all of the weight feature data corresponding to the N feature maps and the weight feature data in the weight dataset. The some or all of the weight feature data corresponding to the N feature maps are obtained from the weight dataset based on the first mapping information.

12. The method as described in claim 10 or 11, characterized in that, The weight dataset is obtained from the bitstream.

13. The method according to any one of claims 9-12, characterized in that, Some or all of the endpoint feature data corresponding to the N feature maps are obtained from the endpoint dataset.

14. The method as described in claim 13, characterized in that, The first feature data includes second mapping information, which is used to indicate the correspondence between some or all of the endpoint feature data corresponding to the N feature maps and the endpoint feature data in the endpoint dataset. The some or all of the endpoint feature data corresponding to the N feature maps are obtained from the endpoint dataset based on the second mapping information.

15. The method as described in claim 13 or 14, characterized in that, The endpoint dataset is obtained from the bitstream.

16. The method as described in claim 8 or 9, characterized in that, The first feature data includes endpoint feature data and weight feature data corresponding to the first feature map of the texture, and the feature texture data includes endpoint feature data and weight feature data corresponding to the second feature map of the texture, as well as endpoint feature data and weight feature data corresponding to the first feature map; The endpoint feature data corresponding to the second feature map is obtained based on the endpoint feature data corresponding to the first feature map, and / or, The weighted feature data corresponding to the second feature map is obtained based on the weighted feature data corresponding to the first feature map.

17. A texture encoding method, characterized in that, The method includes: The texture is encoded to obtain the third feature data and the second nonlinear transformation parameters; The third feature data is encoded to obtain the first feature data, and the second nonlinear transformation parameter is encoded to obtain the first nonlinear transformation parameter; The first feature data and the first nonlinear transformation parameters are encoded into the bitstream.

18. The method as described in claim 17, characterized in that, The third feature data includes data corresponding to one or more feature maps, wherein the one or more feature maps include a first feature map, the first feature map includes a first feature block, and the second feature block is a feature block located in the same feature map as the first feature block but in a different region, or the second feature block is a feature block located in a different feature map than the first feature block. The method further includes: Determine auxiliary information, which is used to indicate a third feature vector, the third feature vector being a feature vector corresponding to the second feature block, and the third feature vector being used to construct the feature vector of the first feature block; The auxiliary information is encoded into the bitstream.

19. The method as described in claim 18, characterized in that, The auxiliary information is used to indicate the third feature vector, including: The auxiliary information is used to indicate the second feature block; The third feature vector is obtained based on the second feature block.

20. The method as described in claim 18 or 19, characterized in that, The determined auxiliary information includes: Complexity detection is performed on the texture to determine the auxiliary information.

21. The method according to any one of claims 17-20, characterized in that, The second feature block is the feature block with the lowest complexity in the surrounding area of ​​the first feature block, or the second feature block is a feature block in a preset area.

22. The method according to any one of claims 17-21, characterized in that, The first feature data includes first mapping information, which is used to indicate the correspondence between some or all of the weight feature data corresponding to the N feature maps of the texture and the weight feature data in the weight dataset. The first mapping information is used to obtain some or all of the weight feature data corresponding to the N feature maps from the weight dataset, where N is a positive integer.

23. The method as described in claim 22, characterized in that, The bitstream also includes the weight dataset.

24. The method according to any one of claims 17-23, characterized in that, The first feature data includes second mapping information, which is used to indicate the correspondence between some or all of the endpoint feature data corresponding to the N feature maps of the texture and the endpoint feature data in the endpoint dataset. The second mapping information is used to obtain some or all of the endpoint feature data corresponding to the N feature maps from the endpoint dataset, where N is a positive integer.

25. The method as described in claim 24, characterized in that, The bitstream also includes the endpoint dataset.

26. The method according to any one of claims 17-21, characterized in that, The first feature data includes endpoint feature data and weight feature data corresponding to the first feature map of the texture; The endpoint feature data corresponding to the first feature map is used to reconstruct the endpoint feature data corresponding to the second feature map of the texture, and / or, The weighted feature data corresponding to the first feature map is used to reconstruct the weighted feature data corresponding to the second feature map.

27. A texture decoding device, characterized in that, The device includes: The parsing module is used to acquire the bitstream and parse the bitstream to obtain the first feature data and the first nonlinear transformation parameters of the texture; The first decoding module is used to decode the first feature data to obtain the second feature data, and to decode the first nonlinear transformation parameter to obtain the second nonlinear transformation parameter; The second decoding module is used to decode the second feature data based on the second nonlinear transformation parameters to obtain the texture.

28. A texture encoding device, characterized in that, The device includes: The first encoding module is used to encode the texture to obtain the third feature data and the second nonlinear transformation parameters; The second encoding module is used to encode the third feature data to obtain the first feature data, and to encode the second nonlinear transformation parameter to obtain the first nonlinear transformation parameter; The third encoding module is used to encode the first feature data and the first nonlinear transformation parameters into a code stream.

29. A decoding device, characterized in that, The decoding device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-16.

30. An encoding device, characterized in that, The encoding device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 17-26.

31. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the steps of the method according to any one of claims 1-16, or to perform the steps of the method according to any one of claims 17-26.

32. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method as described in any one of claims 1-16 to be performed, or the steps of the method as described in any one of claims 17-26 to be performed.

33. An encoded bitstream, characterized in that, The bitstream is generated according to the method described in any one of claims 17-26.

34. An encoded bitstream, characterized in that, The bitstream includes first feature data and first nonlinear transformation parameters. The first feature data is obtained by encoding third feature data, and the first nonlinear transformation parameters are obtained by encoding second nonlinear transformation parameters. The third feature data and the second nonlinear transformation parameters are obtained by encoding texture.

35. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores the bitstream as described in any one of claims 17-26.

36. An apparatus for storing a bitstream, characterized in that, The device includes: a receiver and at least one storage medium; The receiver is configured to receive the bitstream as described in any one of claims 17-26; The at least one storage medium is used to store the bitstream.

37. An apparatus for transmitting a code stream, characterized in that, The device includes: a transmitter and a receiver; The receiver is configured to receive the bitstream as described in any one of claims 17-26; The transmitter is used to send the bitstream to the end-side device via the transmission medium.

38. An apparatus for transmitting a code stream, characterized in that, The device includes: a transmitter and at least one storage medium; The at least one storage medium is used to store the bitstream as described in any one of claims 17-26; The transmitter is used to obtain the bitstream from the storage medium and send the bitstream to the end-side device through the transmission medium.

39. A system for distributing bitstreams, characterized in that, The system includes: At least one storage medium for storing at least one bitstream as described in any one of claims 17-26; A streaming media device is configured to acquire a target bitstream from the at least one storage medium and send the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

Images