Video data processing method and apparatus, and electronic device, computer storage medium and computer program product

Abstract

Disclosed in the present application are a video data processing method and apparatus, and an electronic device, a computer storage medium and a computer program product. The method is executed by an electronic device, and comprises: acquiring color information of an i-th video frame and an (i+1)-th video frame, and acquiring motion information of at least one pixel point in the (i+1)-th video frame, wherein the motion information is used for indicating the displacement of the pixel point between the i-th video frame and the (i+1)-th video frame; on the basis of the motion information of the at least one pixel point, determining a motion vector of at least one pixel block in the (i+1)-th video frame; on the basis of the motion vector of the at least one pixel block, separately determining from the color information of the i-th video frame and the (i+1)-th video frame color residual information respectively corresponding to the at least one pixel block; and on the basis of the color information of the i-th video frame, and the motion vector and color residual information of the at least one pixel block, constructing encoding information of video data.

Description

Methods, apparatuses, electronic devices, computer storage media, and computer program products for processing video data

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 2024117475389, filed on November 29, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of multimedia technology, and in particular to a method, apparatus, electronic device, computer storage medium, and computer program product for processing video data. Background Technology

[0004] With the rapid development of computer technology, the demand for transmitting information via video is increasing. In related technologies, video is composed of video frames, which can carry a rich variety of visual information. This information is characterized by elements such as color, outline, structure, and the spatial relationships of different objects within the video frame. However, storing and transmitting video requires a large amount of data, leading to high resource utilization rates for both electronic devices and network resources. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, computer storage medium, and computer program product for processing video data, the technical solution of which is as follows:

[0006] According to one aspect of the embodiments of this application, a method for processing video data is provided, the method being executed by an electronic device, the method comprising:

[0007] Obtain the color information of the i-th video frame and the (i+1)-th video frame, and obtain the motion information of at least one pixel in the (i+1)-th video frame;

[0008] The motion information is used to indicate the displacement of the pixel between the i-th video frame and the (i+1)-th video frame.

[0009] Based on the motion information of the at least one pixel, the motion vector of at least one pixel block in the (i+1)th video frame is determined;

[0010] Based on the motion vector of the at least one pixel block, color residual information corresponding to the at least one pixel block is determined in the color information of the i-th video frame and the (i+1)-th video frame, respectively.

[0011] The encoding information of the video data is constructed based on the color information of the i-th video frame, the motion vector of the at least one pixel block, and the color residual information.

[0012] According to another aspect of the embodiments of this application, a video data processing apparatus is provided, the apparatus comprising:

[0013] The acquisition module is configured to acquire the color information of the i-th video frame and the (i+1)-th video frame, and to acquire the motion information of at least one pixel in the (i+1)-th video frame.

[0014] The motion information is used to indicate the displacement of the pixel between the i-th video frame and the (i+1)-th video frame.

[0015] The processing module is configured to determine the motion vector of at least one pixel block in the (i+1)th video frame based on the motion information of the at least one pixel point.

[0016] The processing module is further configured to determine, based on the motion vector of the at least one pixel block, the color residual information corresponding to the at least one pixel block in the color information of the i-th video frame and the (i+1)-th video frame, respectively;

[0017] The construction module is configured to construct the encoding information of the video data based on the color information of the i-th video frame, the motion vector of the at least one pixel block, and the color residual information.

[0018] According to another aspect of the embodiments of this application, an electronic device is provided, the electronic device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the video data processing method as described above.

[0019] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program and a video stream are stored in the computer-readable storage medium, and the computer program is executed by a processor to implement the above-described video encoding method to generate the video stream.

[0020] According to another aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer instructions to implement the video data processing method described above.

[0021] This application also provides a method for generating a video stream, wherein the video stream is obtained by encoding video data using the above-described video data processing method.

[0022] The beneficial effects of the technical solutions provided in this application include at least the following:

[0023] In the video data processing method provided in this application embodiment, the electronic device can acquire the color information of the i-th video frame and the (i+1)-th video frame, and acquire the motion information of at least one pixel in the (i+1)-th video frame. The motion information is used to indicate the displacement of the pixel between the i-th video frame and the (i+1)-th video frame. Based on the motion information of at least one pixel, the motion vector of at least one pixel block in the (i+1)-th video frame is determined. Based on the motion vector of at least one pixel block, color residual information corresponding to at least one pixel block is determined in the color information of the i-th video frame and the (i+1)-th video frame, respectively. Based on the color information of the i-th video frame, and the motion vector and color residual information of at least one pixel block, the encoding information of the video data is constructed.

[0024] The video data processing method provided in this application constructs encoding information based on motion vectors, color residual information, and the color information of the i-th video frame. This is equivalent to storing only the encoding information. Compared to a scheme that stores the color information of the i-th video frame and the color information of the (i+1)-th video frame, it eliminates the need to store the color information of each video frame in the video. This reduces the storage space required for storing the video, saves computing resources, reduces the amount of data required for transmitting the video, improves the resource utilization of electronic devices, and improves the resource utilization of network resources. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 is a schematic diagram of a computer system provided in an exemplary embodiment of this application;

[0027] Figure 2 is a schematic diagram of a video data processing method provided in an exemplary embodiment of this application;

[0028] Figure 3 is a flowchart of a video data processing method provided in an exemplary embodiment of this application;

[0029] Figure 4 is a flowchart of a video data processing method provided in an exemplary embodiment of this application;

[0030] Figure 5 is a flowchart of a video data processing method provided in an exemplary embodiment of this application;

[0031] Figure 6 is a flowchart of a video data processing method provided in an exemplary embodiment of this application;

[0032] Figure 7 is a schematic diagram of video data processing provided in an exemplary embodiment of this application;

[0033] Figure 8 is a schematic diagram of video data processing provided in an exemplary embodiment of this application;

[0034] Figure 9 is a flowchart of a video data processing method provided in an exemplary embodiment of this application;

[0035] Figure 10 is a flowchart of a video data processing method provided in an exemplary embodiment of this application;

[0036] Figure 11 is a structural block diagram of a video data processing apparatus provided in an exemplary embodiment of this application;

[0037] Figure 12 is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “at least one” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the video frames, motion information and other information involved in the embodiments of this application were obtained with full authorization.

[0043] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0044] Before describing the embodiments of this application, a brief introduction to the video encoding process in related technologies will be given first.

[0045] A video signal is an image sequence consisting of multiple frames. A video signal is an electrical signal or data stream that carries the image sequence. The image sequence is arranged in chronological order. Each image in the image sequence corresponds to a video frame in the video. A video can include multiple video frames, and a video frame is an image.

[0046] A frame is a spatial sample of a video signal at a specific moment. Spatial sampling can also be called spatial representation, which is the representation of spatial information. Taking YUV mode as an example, a frame includes a luminance sample matrix (Y) and two chrominance sample matrices (Cb and Cr). For example, in YUV4:2:0, a video frame contains a luminance sample matrix Y, a chrominance sample matrix Cb, and a chrominance sample matrix Cr. The chrominance resolution can be half that of the luminance resolution.

[0047] From the perspective of video signal acquisition methods, there are two main categories: those captured by cameras and those generated by computers. Due to differences in statistical characteristics, the corresponding compression encoding methods may also differ. In other words, video frames can be captured by cameras or generated by computers. Since camera-captured and computer-generated frames differ in statistical characteristics, encoders can choose different encoding toolsets or parameter strategies to improve video compression efficiency.

[0048] Regarding video coding techniques in related technologies, these techniques can be one or more of High Efficient Video Coding (H.265 / HEVC), Versatile Video Coding (H.266 / VVC), and Audio Video Coding Standard (AVS). The AVS standard can be AVS3. A hybrid coding framework is used to perform the following operations and processing on the input raw video signal:

[0049] 1. Block Partition Structure: The input image is divided into several non-overlapping processing units, each of which undergoes a similar compression operation. This processing unit is called a CTU (Coding Tree Unit) or LCU (Large Coding Unit). The CTU can be further subdivided to obtain one or more basic coding units, called CUs (Coding Units). Each CU is the most basic element in a coding process. The following describes the various coding methods that can be used for each CU.

[0050] 2. Predictive Coding: This includes intra-frame prediction and inter-frame prediction. The original video signal is predicted from a selected reconstructed video signal to obtain a residual video signal. The encoder needs to determine the most suitable predictive coding mode from among many possible modes for the current CU and inform the decoder. Intra-frame prediction refers to predicting a signal from a region within the same image that has already been encoded and reconstructed. Inter-frame prediction refers to predicting a signal from another encoded image that is different from the current image (i.e., the current video frame).

[0051] In a predictive coding-based video compression framework, the pixel value of the current coding unit (CU) is not directly encoded. Instead, it is generated by predicting the reconstructed reference signal, and then the original pixel value is subtracted from the predicted value to obtain the residual signal. The residual signal is then transformed, quantized, and entropy-coded before being written into the bitstream.

[0052] Prediction methods can include intra-frame prediction and inter-frame prediction. For intra-frame prediction, the predicted value comes from the reconstructed adjacent regions within the current frame (which can be one or more located above and to the left of the current CU). The encoder needs to select the optimal mode from a predefined set of intra-frame prediction modes and explicitly write the index of the selected mode into the bitstream so that the decoder can reproduce the same prediction.

[0053] For inter-frame prediction, the predicted values ​​are derived from other encoded and reconstructed video frames (which can be called reference frames). Motion estimation is used to determine the best matching block of the current CU in the reference frame, and motion vectors are calculated. The encoder needs to write the selected reference frame index and motion vector information into the bitstream, and the decoder uses this information to locate the reference block as the predicted value.

[0054] 3. Transform & Quantization: The residual video signal undergoes transformation operations such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) to convert it into the transform domain, where the coefficients are called transform coefficients. In the transform domain, the signal undergoes further lossy quantization, losing some information to make the quantized signal more suitable for compression. Some video coding standards may offer more than one transform option; therefore, the encoder needs to select one transform for the current CU and inform the decoder.

[0055] The fineness of quantization is usually determined by the quantization parameter. A larger quantization parameter (QP) value means that coefficients with a wider range of values ​​will be quantized into the same output, which usually results in greater distortion and a lower bitrate. Conversely, a smaller QP value means that coefficients with a smaller range of values ​​will be quantized into the same output, which usually results in less distortion and a higher bitrate.

[0056] 4. Entropy Coding or Statistical Coding: The quantized transform domain signal is statistically compressed and encoded based on the frequency of each value, ultimately outputting a binary (0 or 1) compressed bitstream. Simultaneously, other information generated during encoding, such as the selected mode and motion vectors, also requires entropy coding to reduce the bit rate. Statistical coding is a lossless coding method that effectively reduces the bit rate required to represent the same signal. Statistical coding methods can include Variable Length Coding (VLC) or Content Adaptive Binary Arithmetic Coding (CABAC).

[0057] 5. Loop Filtering: Encoded images (i.e., video frames) undergo inverse quantization, inverse transform, and prediction compensation (the reverse operations of steps 2-4 above) to obtain a reconstructed decoded image. Compared to the original image, the reconstructed image differs in some information due to quantization, resulting in distortion. Filtering the reconstructed image, such as deblocking, Sample Adaptive Offset (SAO), or Adaptive Lattice Filter (ALF), can effectively reduce the distortion caused by quantization. Since these filtered reconstructed images will serve as a reference for subsequent encoded images to predict future signals, the above filtering operations are also called loop filtering, or filtering operations within the encoding loop.

[0058] As can be seen from the above encoding process, at the decoding end, for each CU, after obtaining the compressed bitstream, the decoder first performs entropy decoding to obtain various mode information and quantized transform coefficients. Each coefficient undergoes inverse quantization and inverse transform to obtain the residual signal. On the other hand, based on the known encoding mode information, the prediction signal corresponding to that CU can be obtained. After adding the two, the reconstructed signal is obtained. Finally, the reconstructed value of the decoded image needs to undergo a loop filtering operation to generate the final output signal.

[0059] Many mainstream video coding standards, such as HEVC, VVC, and AVS3, employ block-based hybrid coding frameworks. They divide the raw video data into a series of coded blocks and combine prediction, transform, and entropy coding methods to achieve video data compression. Motion compensation is a commonly used prediction method in video coding. Based on the redundancy characteristics of video content in the temporal or spatial domains, motion compensation derives the predicted value of the current coded block from the already coded regions. These prediction methods include inter-frame prediction, intra-frame block copy prediction, and intra-frame string copy prediction. In specific coding implementations, these prediction methods may be used individually or in combination. For coded blocks using these prediction methods, one or more two-dimensional displacement vectors are typically explicitly or implicitly encoded in the bitstream to indicate the displacement of the current coded block (or its sibling block) relative to one or more reference blocks.

[0060] Referring to Figure 1, Figure 1 illustrates a schematic diagram of a computer system provided in one embodiment of this application. This computer system can implement a system architecture for a video data processing method. The computer system may include: a terminal 100 and a server 200.

[0061] Terminal 100 can be an electronic device such as a mobile phone, tablet computer, or personal computer (PC). A client application for the target application can be installed and run on terminal 100. This target application can be a video data processing application, a model building application, or other applications that provide data processing functions; this application embodiment does not limit the specific application. Furthermore, this application embodiment does not limit the form of the target application, including but not limited to applications (Apps), mini-programs, etc., installed on terminal 100, and it can also be in web page form.

[0062] Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Server 200 can be the backend server for the aforementioned target application, providing backend services to the client of the target application. Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0063] The video data processing method provided in this application embodiment can be executed by a computer device (i.e., an electronic device), which refers to an electronic device with data computing, processing, and storage capabilities. Taking the implementation environment of the scheme shown in Figure 1 as an example, the video data processing method can be executed by the terminal 100 (e.g., by the client of the target application installed and running on the terminal 100), by the server 200, or by the interaction and cooperation between the terminal 100 and the server 200. This application embodiment does not limit this. It should be noted that the technical solution provided in this application embodiment can be applied to the H.266 / VVC standard, H.265 / HEVC standard, AVS (such as AVS3), or next-generation video encoding / decoding standards. This application embodiment does not limit this.

[0064] Furthermore, the technical solutions provided in this application can be combined with blockchain technology. For example, in the video data processing method disclosed in this application, some data involved (such as color information of video frames, encoding information of video data, etc.) can be stored on the blockchain. Terminal 100 and server 200 can communicate via a network, such as a wired or wireless network.

[0065] Next, the video data processing method in the embodiments of this application will be described: Referring to Figure 2, Figure 2 shows a schematic diagram of a video data processing method provided by an exemplary embodiment of this application. The computer device (i.e., electronic device) that implements the video data processing method in the embodiments of this application can be a terminal, a server, or a combination of the two.

[0066] In some embodiments, color information of the first video frame 302 and the second video frame 304 is obtained. The first video frame 302 and the second video frame 304 are two adjacent video frames in the video data, and the first video frame 302 is the video frame displayed before the second video frame 304. The first video frame 302 may correspond to the i-th video frame, and the second video frame 304 may correspond to the (i+1)-th video frame.

[0067] The second video frame 304 comprises a pixel block arranged in three rows and five columns. The division of the pixel block can be set according to actual usage requirements; this is only for illustration. Each pixel block includes a total of 64 pixels arranged in eight rows and eight columns. It is understood that in different embodiments, the second video frame 304 can be divided into more or fewer pixel blocks, and the number and arrangement of pixels in the pixel block can also differ from the embodiments of this application.

[0068] Taking the first pixel block 304a in the second video frame 304 as an example, the first pixel block 304a is a pixel block located in the second row and third column; the first pixel block 304a includes 64 pixels, and the motion information 306 of each pixel in the first pixel block 304a is obtained.

[0069] Motion information 306 is used to indicate the positional difference of the same virtual spatial location in the virtual environment between the first video frame 302 and the second video frame 304. The virtual environment is also known as a virtual scene, which is a virtual scene displayed (or provided) by an application when it runs on a terminal. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene.

[0070] For example, a virtual scene can include the sky, land, and ocean. The land can include environmental elements such as deserts and cities. Users can control virtual objects to perform activities within this virtual scene, including but not limited to: adjusting body posture, crawling, walking, running, riding, jumping, driving, picking up items, shooting, attacking, and throwing at least one of these activities. The virtual scene can be displayed from a first-person perspective (e.g., the user plays the role of a virtual object in the game from their own perspective); it can also be displayed from a third-person perspective (e.g., the user chases after a virtual object in the game); or it can be displayed from a bird's-eye view. Users can switch freely between these perspectives.

[0071] Virtual objects are interactive representations of people and things within a virtual scene, or movable objects within that scene. These movable objects can be virtual characters, animals, anime characters, etc., such as people, animals, plants, oil drums, walls, and stones displayed in a virtual scene. A virtual object can be a virtual avatar representing the user within that scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the space within the virtual scene.

[0072] In some embodiments, a virtual ramp exists in the first video frame 302, located at the left frame position 3020 of the first video frame 302; while in the second video frame 304, the virtual ramp is located at the right frame position 3040 of the second video frame 304; motion information 306 is used to indicate the positional difference between the left frame position 3020 and the right frame position 3040.

[0073] Motion information 306 for each pixel in the first pixel block 304a includes the virtual spatial positions corresponding to the 64 pixels in the first pixel block 304a, and the positional differences between the first video frame 302 and the second video frame 304. The direction and distance of the positional differences are indicated by directed arrows in Figure 2.

[0074] The average value of the motion information 306 of each pixel in the first pixel block 304a is determined as the motion vector of the first pixel block 304a. The motion vector is the overall representation information of the first pixel block 304a, which is statistically obtained from each pixel in the first pixel block 304a.

[0075] In some embodiments, based on the position of the first pixel block 304a in the second video frame 304, the second pixel block 302a in the first video frame 302 is obtained by translating it in the direction and distance indicated by the motion vector.

[0076] The color information of the second pixel block 302a is obtained from the color information of the first video frame 302 based on the position of the second pixel block 302a; the color information of the first pixel block 304a is obtained from the color information of the second video frame 304 based on the position of the first pixel block 304a.

[0077] Color residual information 310 is calculated based on the difference between the color information of the second pixel block 302a and the color information of the first pixel block 304a. In some embodiments, the color residual information 310 and the motion vector of the first pixel block 304a can be used to obtain the color information of the first pixel block 304a based on the color information of the first video frame 302.

[0078] Repeating the above process, the motion vectors and color residual information of each pixel block in the second video frame 304 can be obtained. This allows the color information of the second video frame 304 to be derived from the color information of the first video frame 302 based on the motion vectors and color residual information of multiple pixel blocks. By constructing the encoding information of the video data based on the color information of the first video frame 302 and the motion vectors and color residual information of the pixel blocks in the second video frame 304, the space required to store the video data can be reduced compared to directly constructing the encoding information based on the color information of the first video frame 302 and the second video frame 304.

[0079] In some embodiments, the image of the (i+1)th video frame is generated based on the color information of the i-th video frame in the encoding information, as well as the motion vector and color residual information of at least one pixel block. The encoding information in the video data can be obtained. The encoding information includes: the color information of the i-th video frame, the motion vector of at least one pixel block in the (i+1)-th video frame, and the color residual information corresponding to at least one pixel block.

[0080] The motion vector is used to indicate the displacement of the pixel block between the i-th video frame and the (i+1)-th video frame. Based on the motion vector, the pixel block corresponding to at least one pixel block is located in the color information of the i-th video frame. The corresponding pixel block is added to the color residual information to obtain the decoded color information of at least one corresponding pixel block in the (i+1)-th video frame. Based on the decoded color information, the decoded image of the (i+1)-th video frame (i.e. the picture of the (i+1)-th video frame) is generated.

[0081] The video data processing method provided in this application embodiment can obtain the color information of the (i+1)th video frame based on the color information of the i-th video frame by using the motion vector and color residual information of the pixel block. This reduces the storage space of the video data while ensuring the integrity of the color information of the (i+1)th video frame and ensuring the integrity of the image of the (i+1)th video frame.

[0082] Next, the video data processing method will be described through the following embodiments. Referring to Figure 3, Figure 3 shows a flowchart of a video data processing method provided in an exemplary embodiment of this application. This method can be executed by a computer device (i.e., an electronic device), and includes:

[0083] Step 510: Obtain the color information of the i-th video frame and the (i+1)-th video frame, and obtain the motion information of at least one pixel in the (i+1)-th video frame.

[0084] In practical applications, when users need to store or transmit video, the encoding information of the video data can be constructed. When the encoding information of the video data needs to be constructed, the i-th video frame and the (i+1)-th video frame in the video data can be obtained.

[0085] In some embodiments, the i-th video frame and the i+1-th video frame are two adjacent video frames in a video information (corresponding video data). The color information can be directly indicated by information in the form of red-green-blue (RGB) channel, or indirectly indicated by information in the form of hue-saturation-value (HSV) channel. The embodiments of this application do not limit the data format of the color information.

[0086] In some embodiments, the i-th video frame and the (i+1)-th video frame are used to display a virtual environment, which is a virtual scene; in some embodiments, the i-th video frame and the (i+1)-th video frame are two video frames in chronological order from front to back, with the i-th video frame being the first in chronological order and the (i+1)-th video frame being the second in chronological order; in some embodiments, the i-th video frame and the (i+1)-th video frame are two adjacent video frames.

[0087] Motion information is used to indicate the displacement of a pixel between adjacent video frames, that is, to indicate the displacement of a pixel between the i-th video frame and the (i+1)-th video frame. The displacement can be determined by the first position of the pixel in the i-th video frame and the second position of the pixel in the (i+1)-th video frame.

[0088] In some embodiments, motion information is used to indicate the positional change of a pixel in the (i+1)th video frame relative to the pixel in the ith video frame. At least one pixel can be a pixel at any position in the (i+1)th video frame, a pixel at the center of the (i+1)th video frame, or a pixel at the center of a region after the (i+1)th video frame has been divided into multiple regions (or pixel blocks) according to a preset size. The pixel can be one or more of the above-mentioned pixel types. This application does not limit the position of the pixel. In some embodiments, the motion information of at least one pixel is a set of information, such as a set of motion information for each pixel within the at least one pixel.

[0089] Step 520: Determine the motion vector of at least one pixel block in the (i+1)th video frame based on the motion information of at least one pixel.

[0090] In some embodiments, for motion information of at least one pixel, the at least one pixel includes a second pixel; the motion information of the second pixel is used to indicate the difference between a second position of the second pixel in the (i+1)th video frame and a first position of the first pixel in the i-th video frame. In some embodiments, the motion information of the second pixel is used to indicate the positional difference between two pixels on two video frames, and in some embodiments, the positional difference includes vector information with direction and distance on the plane of the corresponding video frames.

[0091] In some embodiments, the virtual environment can be obtained by observing a virtual environment using a virtual camera. The virtual environment can also be provided by an application installed on a computer device (i.e., an electronic device), or by an application installed on other devices. For example, the virtual environment can be provided by a game application or a simulation application.

[0092] In some embodiments, the virtual environment can be any one of a two-dimensional virtual environment, a 2.5-dimensional virtual environment, and a three-dimensional virtual environment. In some embodiments, the second pixel and the first pixel correspond to the same location in the virtual environment, and the difference between the locations in the virtual environment corresponding to the second pixel and the first pixel is less than a difference threshold, that is, the difference between the location in the virtual environment represented by the first location and the location in the virtual environment represented by the second location is less than a difference threshold. The motion information of the second pixel can indicate the changes in the viewed image of the virtual environment between two video frames.

[0093] For example, the first pixel is used to indicate the pupil of a virtual object's eye in the virtual environment, and the second pixel can also be used to indicate the pupil of a virtual object's eye in the virtual environment. The pupils of the virtual object's eye indicated by the first pixel and the second pixel are at the same position, or they can be positions with a difference of less than a difference threshold (i.e., not the same position, but close to each other).

[0094] For example, the pixel data of the second pixel is the color information of the first position in the virtual environment displayed in the (i+1)th video frame, and the pixel data of the first pixel is the color information of the first position in the virtual environment displayed in the ith video frame. The visual information presented by the pixel data of the two pixels is at the same position in the virtual environment. Taking a three-dimensional spatial environment as an example, the second pixel and the first pixel correspond to the same position in the three-dimensional spatial coordinate system describing the three-dimensional spatial environment. It can be understood that the virtual environment can also be implemented as at least one of two-dimensional virtual environments, 2.5-dimensional virtual environments, etc., and the same position is the same coordinate position in the coordinate system with the virtual environment as the reference frame.

[0095] In some embodiments, the virtual environment may contain virtual characters (the description of the virtual object can be replaced with an equivalent one), virtual items, and other virtual objects. The pixel data of the second pixel and the pixel data of the first pixel correspond to the same virtual object in the virtual environment.

[0096] In some embodiments, due to differences in one or more of the observation position and observation angle of the virtual environment in different video frames, the second position of the second pixel in the (i+1)th video frame differs from the first position of the first pixel in the i-th video frame. The motion information describes this positional difference, which, as described above, is vector information on the plane where the video frame is located, including direction and distance. In some embodiments, both the first and second positions are positions in a two-dimensional plane coordinate system of the video frame. In some embodiments, the difference between the first and second positions is a relative positional difference in a two-dimensional plane coordinate system.

[0097] In some embodiments, a pixel block includes at least two pixels in the (i+1)th video frame, and the motion vector of the pixel block is obtained based on the motion information of the pixels; in some embodiments, the motion information of the pixels is used to characterize the motion vector of the pixel block.

[0098] In some embodiments, the motion vector of a pixel block can be determined based on the motion information of one or more pixels. In some embodiments, it is not excluded that the motion information of a single pixel can be used to determine the motion vector of multiple pixel blocks; that is, the motion information of a single pixel can be used to determine the motion vector of multiple pixel blocks.

[0099] In some embodiments, when determining the motion vector of a pixel block based on the motion information of at least two pixels, the motion vector of the at least two pixels can be determined by at least one of the following methods: statistical average, median, mode, etc., that is, the motion vector of the first pixel block is determined. This application does not limit the method of determining the motion vector.

[0100] Step 530: Based on the motion vector of at least one pixel block, determine the color residual information corresponding to at least one pixel block in the color information of the i-th video frame and the (i+1)-th video frame, respectively.

[0101] In some embodiments, the motion vector of a pixel block is used to indicate the positional difference between two pixel blocks on two video frames; a first pixel may belong to a third pixel block, the third pixel block belongs to the i-th video frame, a second pixel may belong to a fourth pixel block, and the fourth pixel block belongs to the (i+1)-th video frame. The motion vector of the pixel block is used to indicate the difference between the third position of the third pixel block in the (i+1)-th video frame and the fourth position of the fourth pixel block in the i-th video frame, and the motion vector of the pixel block is used to indicate the displacement of the pixel block between the i-th video frame and the (i+1)-th video frame.

[0102] In some embodiments, the color residual information is the color difference between a pixel block in the (i+1)th video frame and its corresponding pixel block in the ith video frame. In some embodiments, the color residual information indicates the color difference between individual pixels within a pixel block.

[0103] Step 540: Construct the encoding information of the video data based on the color information of the i-th video frame, and the motion vector and color residual information of at least one pixel block.

[0104] In some embodiments, for the (i+1)th video frame, the motion vector of at least one pixel block indicates the difference from the i-th video frame in the pixel block dimension, which is a difference information in a large field of view; the color residual information is a difference information in a small field of view in the dimension of each pixel, based on the motion vector.

[0105] Here, a large field of view refers to a field of view whose area is greater than the first area threshold, and a small field of view refers to a field of view whose area is less than the second area threshold. The second area threshold can be less than or equal to the first area threshold. Field of view refers to the range of difference observation. A large field of view corresponds to the pixel block level, and a small field of view corresponds to the pixel point level.

[0106] Motion vectors and color residual information of pixel blocks can describe the differences between the i-th video frame and the (i+1)-th video frame from different dimensions. Based on the color information of the i-th video frame, the color information of the (i+1)-th video frame can be indicated, ensuring the integrity of the color information of the (i+1)-th video frame. The encoded information of the constructed video data does not need to directly carry the color information of the (i+1)-th video frame, reducing the volume of the encoded video data and improving the efficiency of one or more aspects of video data storage and transmission. In some embodiments, the encoded video data information can be transmitted between different computer devices (i.e., electronic devices) in the form of one or more encoded video streams.

[0107] In some embodiments, this application provides a method for storing video streams. A video stream can be generated by executing the video data processing method provided in this application, and the generated video stream can then be stored or transmitted. For details, please refer to the description of the video data processing method, which will not be repeated here.

[0108] In summary, the video data processing method provided in this application fully utilizes the virtual spatial positions of pixels on the video frame in the virtual environment by determining the motion vector through motion information. Encoding information is constructed based on the motion vector, color residual information, and the color information of the i-th video frame. Compared to directly storing the color information of the (i+1)-th video frame, this reduces the space required for storing video data. Through the motion vector and color residual information of the pixel blocks, the color information of the (i+1)-th video frame can be obtained based on the color information of the i-th video frame. While reducing the storage space of the video data, the integrity of the color information of the (i+1)-th video frame is ensured, guaranteeing the integrity of the image of the (i+1)-th video frame.

[0109] The video data processing method provided in this application constructs encoding information based on motion vectors, color residual information, and the color information of the i-th video frame. This is equivalent to storing only the encoding information. Compared to a scheme that stores the color information of the i-th video frame and the color information of the (i+1)-th video frame, it eliminates the need to store the color information of each video frame in the video. This reduces the storage space required for storing the video, saves computing resources, reduces the amount of data required for transmitting the video, improves the resource utilization of electronic devices, and improves the resource utilization of network resources.

[0110] Referring to Figure 4, Figure 4 illustrates a flowchart of a video data processing method provided in an exemplary embodiment of this application. This method can be executed by a computer device (i.e., an electronic device). Specifically, in the embodiment shown in Figure 3, step 520 can be implemented as step 522, meaning that step 520 shown in Figure 3 can be replaced by step 522 shown in Figure 4.

[0111] Step 522: Determine the motion vector of the first pixel block based on the motion information of the pixels in the first pixel set;

[0112] In this embodiment, the motion vector of the first pixel block in the (i+1)th video frame is used as an example. In some embodiments, motion information of at least one pixel is obtained in step 510. A subset of the at least one pixel is a first pixel set, which includes one or at least two pixels. Each pixel in the first pixel set is a pixel in the first pixel block (the first pixel block corresponds to the fourth pixel block, and the two can be equivalently replaced). The first pixel block is any one of one or more pixel blocks obtained by dividing the (i+1)th video frame according to a preset size.

[0113] In some embodiments, the motion vector of the first pixel block is determined based on the motion information of the pixels within the first pixel block. In some embodiments, the motion vector can be determined using one or at least two pixels from the first pixel set (i.e., using at least one pixel from the first pixel set).

[0114] In some embodiments, when the motion vector is determined based on the motion information of a pixel in the first set of pixels, the motion information of a pixel is determined as the motion vector of the first pixel block.

[0115] In some embodiments, when the motion vector is determined based on the motion information of at least two pixels in the first pixel set, statistical calculations are performed on the motion information of the at least two pixels to obtain the motion vector of the first pixel block. In some embodiments, performing statistical calculations on the motion information of the at least two pixels includes, but is not limited to, at least one of the following methods: statistical average, median, mode, etc. In some embodiments, step 522 can be implemented in at least one of the following ways:

[0116] The first method is to determine the motion vector of the first pixel block by averaging the motion information of at least two pixels in the first pixel set.

[0117] In some embodiments, the average value of motion information is used to indicate the average positional difference between at least two pixels in two videos; the motion vector determined based on the average value enables the sum of the color residual information of each pixel in the pixel block to approach the minimum value, thereby reducing the storage space required for the color residual information from the dimension of the sum of the color residual information of each pixel.

[0118] In some embodiments, the motion vector of the pixel block can be found in formulas (1) and (2):

[0119] Among them, MV x It is the motion vector of the x-th pixel block, where n is the number of pixels in the pixel block, mv ix It represents the motion information of the i-th pixel within the x-th pixel block; MV y mv is the motion vector of the y-th pixel block, where n is the number of pixels in the pixel block. iy It represents the motion information of the i-th pixel in the y-th pixel block.

[0120] The second method is to determine the motion vector of the first pixel block by taking the median of the motion information of at least two pixels in the first pixel set.

[0121] In some embodiments, the median of motion information indicates the motion information of a pixel segment in which the positional difference between at least two pixels in two videos is at an intermediate level. It uses the motion information of a real pixel directly as the motion vector of the pixel block. This takes into account that the sum of the color residual information of each pixel approaches the minimum value, and the motion vector is directly selected from the motion information. It can preliminarily verify whether the stored motion vector is correct by whether there is a motion vector in the motion information.

[0122] In some embodiments, the motion vector of a pixel block can be found in formulas (3) and (4).

[0123] Among them, MV med , x It is the motion vector determined by the median in the x-th pixel block, where n is the number of pixels in the pixel block. It is the xth pixel in the xth pixel block. Motion information of each pixel It is the xth pixel in the xth pixel block. Motion information of each pixel; MV med , y is the motion vector of the y-th pixel block, and n is the number of pixels in the pixel block. It is the y-th pixel in the y-th pixel block. Motion information of each pixel It is the y-th pixel in the y-th pixel block. Motion information of each pixel.

[0124] The third method is to determine the motion vector of the first pixel block by taking the modulo value of the motion information of at least two pixels in the first pixel set.

[0125] In some embodiments, the modality of motion information is the motion information of the maximum number of pixels whose positions differ equally in two videos; the motion information of a real pixel is directly used as the motion vector of the pixel block; the correctness of the stored motion vector can be directly verified by whether the motion vector is the maximum number of identical information in the motion information.

[0126] At least two of the first and third methods mentioned above can jointly determine the motion vector of the first pixel block. The following explanation is as follows: the average value of the motion information corresponds to the first weight, the median value of the motion information corresponds to the second weight, and the mode value of the motion information corresponds to the third weight.

[0127] For example, given the average, median, and mode of motion information, the average can be multiplied by a first weight to obtain a first product, the median by a second weight to obtain a second product, and the mode by a third weight to obtain a third product. The sum of these three products is then used to determine the motion vector of the first pixel block. For cases where at least two of the average, median, and mode of motion information are obtained, equivalent substitutions can be made, which will not be elaborated upon here.

[0128] In summary, the video data processing method provided in this application fully utilizes the virtual spatial positions of pixels in the video frame in the virtual environment by using motion vectors determined by motion information. The motion vector of the first pixel block is determined based on the motion information of the pixels within the first pixel block, ensuring that the motion vector of the first pixel block can characterize the motion of the pixels within the first pixel block. Encoding information is constructed based on the motion vector, color residual information, and color information of the i-th video frame. Compared to directly storing the color information of the (i+1)-th video frame, the space required for storing video data is reduced. Through the motion vector and color residual information of the pixel block, the color information of the (i+1)-th video frame can be obtained based on the color information of the i-th video frame. While reducing the storage space of video data, the integrity of the color information of the (i+1)-th video frame is ensured, thus guaranteeing the integrity of the image of the (i+1)-th video frame.

[0129] In this embodiment, the motion vector of the first pixel block can be determined by using one or more of the average value of motion information of multiple pixels, the median value of motion information, and the mode value of motion information, which can enrich the diversity of determining the motion vector of the first pixel block.

[0130] Referring to Figure 5, Figure 5 illustrates a flowchart of a video data processing method provided in an exemplary embodiment of this application. This method can be executed by a computer device (i.e., an electronic device). Specifically, in the embodiment shown in Figure 4, step 522 can be implemented as steps 522a, 522b, and 522c; that is, step 522 in Figure 4 can be replaced with steps 522a, 522b, and 522c shown in Figure 5.

[0131] Step 522a: Determine the reference vector of the first pixel block based on the motion information of the pixels in the first pixel set;

[0132] In some embodiments, the reference vector is determined based on motion information. In some embodiments, the reference vector is an intermediate parameter in the process of determining the motion vector; referring to the description in step 522, the reference information can be determined directly or statistically based on the motion information of one or more pixels within the first pixel block. The method for determining the reference vector can be referred to the method for determining the motion vector in the embodiment corresponding to Figure 4, and will not be repeated here.

[0133] Step 522b: Based on the position of the first region of the first pixel block in the (i+1)th video frame and the reference vector, determine the color sub-information of the reference pixel block in the color information of the i-th video frame;

[0134] In some embodiments, the position of the second region where the reference pixel block is located is obtained by moving the reference vector from the first region position of the first pixel block. The reference vector indicates the direction and distance of movement relative to the first region position. The color sub-information of the reference pixel block is the information located in the second region position from the color information of the i-th video frame.

[0135] Step 522c: If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block does not exceed the color threshold, the reference vector is determined as the motion vector of the first pixel block;

[0136] In some embodiments, if the difference does not exceed a color threshold, the similarity between the first pixel block and the reference pixel block meets the color threshold requirement. The color threshold is the difference threshold, and the reference vector is determined as the motion vector of the first pixel block. In some embodiments, the color threshold constrains the similarity between the reference pixel block indicated by the reference vector and the first pixel block. This avoids the problem of increased video data encoding volume due to the complex information required to be recorded in the color residual information because the difference between the first pixel block and the reference pixel block is too large.

[0137] In some embodiments, the color threshold can be a preset empirical value or determined based on the color information of the i-th video frame; the embodiments of this application do not limit the method of determining the color threshold. In an optional implementation, the color threshold is the minimum value among at least one reference difference degree.

[0138] In some embodiments, the reference difference is used to indicate the difference between the color sub-information of neighboring pixel blocks around the reference pixel block and the color sub-information of the first pixel block.

[0139] Neighboring pixel blocks have the same size as the reference pixel block and are located in the peripheral region of the reference pixel block. The size of the peripheral region is larger than that of the reference pixel block. In some embodiments, the center point of the peripheral region is the center point of the reference pixel block. There are usually overlapping pixel blocks between neighboring and reference pixel blocks, but it is not impossible for pixels in a neighboring pixel block to be completely different from pixels in the reference pixel block.

[0140] In summary, the video data processing method provided in this application fully utilizes the virtual spatial positions of pixels on the video frame in the virtual environment by determining the motion vector through motion information; it verifies the motion vector through a color threshold to ensure that the pixel block indicated by the motion vector is similar to the first pixel block, simplifying the complexity of the information in the color residual information; and it constructs encoding information based on the motion vector, color residual information, and color information of the i-th video frame; compared to directly storing the color information of the (i+1)-th video frame, it reduces the space required to store video data.

[0141] The color threshold is further described below. In some embodiments, referring to FIG6, FIG6 is a flowchart of a video data processing method provided in an exemplary embodiment of this application, which, based on FIG5, further includes steps 524 and 526 as shown in FIG6:

[0142] Step 524: If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds the color threshold, determine the correction pixel block;

[0143] In some embodiments, the difference between the color sub-information of the corrected pixel block and the color sub-information of the first pixel block is smaller than the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block.

[0144] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds the color threshold, a correction pixel block with a smaller difference from the first pixel block is determined in the i-th video frame.

[0145] Similar to the neighboring pixel blocks mentioned above, the corrected pixel block is located in the peripheral region of the reference pixel block, and the size of the peripheral region is larger than the size of the reference pixel block; see Figure 7, which shows a schematic diagram of video data processing provided by an exemplary embodiment of this application. Color information of the first video frame 302 and the second video frame 304 is obtained. The first video frame 302 and the second video frame 304 are two adjacent video frames in the video data, and the first video frame 302 is the video frame displayed before the second video frame 304.

[0146] The reference vector 305 of the first pixel block 304a points to the upper left and is obtained statistically from each pixel in the first pixel block 304a (for example, it can be the average value of the motion information of each pixel). It is the representation information of the first pixel block 304a as a whole.

[0147] Based on the position of the first pixel block 304a in the second video frame 304, a reference position is obtained by translating it in the direction and distance indicated by the reference vector 305. This reference position in the first video frame 302 is the position of the reference pixel block 302b. The peripheral region 315 of the reference pixel block 302b is centered on the reference pixel block 302b, and its area is four times the area of ​​the reference pixel block 302b. It is understandable that, in different examples, the area of ​​the peripheral region 315 and the area of ​​the reference pixel block 302b may have other different proportional relationships.

[0148] Referring to Figure 2, taking the first pixel block 304a, which comprises 64 pixels in eight rows and eight columns, as an example, the peripheral region 315 in Figure 7 comprises 256 pixels in sixteen rows and sixteen columns. Within the peripheral region 315, (16-8+1)*(16-8+1) = 81 candidate pixel blocks can be sampled, each candidate pixel block having a size of 64 pixels in eight rows and eight columns. Any two candidate pixel blocks are distinct, but some overlap is possible. With the goal of minimizing the difference between the color sub-information of the candidate pixel blocks and the color sub-information of the first pixel block 304a, a color difference search is performed to obtain the corrected pixel block 316. Based on the positional difference between the first pixel block 304a in the first region of the second video frame 304 and the corrected pixel block 316 in the second region of the first video frame 302, the motion vector 304b of the first pixel block 304a is determined.

[0149] In some embodiments, there are usually overlapping pixel blocks between the corrected pixel block and the reference pixel block, but it is not excluded that there may be cases where the pixels in the corrected pixel block are completely different from the pixels in the reference pixel block. In some embodiments, the corrected pixel block is obtained by searching in the peripheral region of the reference pixel block according to a block matching algorithm; in some embodiments, the block matching algorithm can be implemented as at least one of the following: Sum of Absolute Differences (SAD) algorithm, Sum of Squared Differences (SSD) algorithm, Zero-mean Normalized Cross-Correlation (ZNCC) algorithm, and Semi-Global Block Matching (SGBM) algorithm.

[0150] The following explanation uses the sum of absolute errors as an example. The absolute differences between the same color component matrices of the correction candidate block and the first pixel block are calculated point-by-point and then accumulated to obtain the sum of single-component absolute errors. The sums of the single-component absolute errors corresponding to Y, Cb, and Cr are then weighted (or summed) to obtain the final difference. The sum of absolute errors does not involve multiplication or division; it only involves addition, subtraction, and finding the absolute value. This saves computational resources, is sensitive to isolated bright spots and noise, and is fast. It can determine the correction candidate block corresponding to the smallest final difference value as the correction pixel block.

[0151] Step 526: Determine the motion vector of the first pixel block based on the positional difference between the first region position and the second region position of the corrected pixel block in the i-th video frame;

[0152] In some embodiments, when performing step 526, if the difference between the reference pixel block and the first pixel block is too large, the motion vector of the first pixel block is determined based on the positional difference between the correction pixel block and the first pixel block, which has a smaller difference.

[0153] In some embodiments, step 524 and step 522c in FIG. 5 are two parallel steps. As described above, step 524 is executed when the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds a color threshold; step 522c is executed when the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block does not exceed the color threshold. In some embodiments, the execution timing of step 524 in this embodiment follows step 522b above.

[0154] In summary, the video data processing method provided in this application fully utilizes the virtual spatial positions of pixels on the video frame in the virtual environment by determining the motion vector through motion information; the motion vector is verified by a color threshold, and when the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds the color threshold, a correction pixel block is re-determined to correct the motion vector of the pixel block; this ensures that the pixel block indicated by the motion vector is similar to the first pixel block, simplifying the complexity of the information in the color residual information; and the encoding information is constructed based on the motion vector, the color residual information, and the color information of the i-th video frame; compared to directly storing the color information of the (i+1)-th video frame, this reduces the space required to store the video data.

[0155] Referring to Figure 8, Figure 8 illustrates a schematic diagram of video data processing provided by an exemplary embodiment of this application. The color information of the current frame 602 (corresponding to the (i+1)th video frame) and the historical frame 604 (corresponding to the ith video frame) is obtained, and the motion vector 606 of at least one pixel block in the current frame 602 is determined. The full name of the motion vector is Motion Vector, abbreviated as MV. In some embodiments, the method for determining the motion vector 606 is described in the embodiments corresponding to Figures 3 and 4 above, and will not be repeated here.

[0156] In some embodiments, taking the first pixel block in the current frame 602 as an example, motion compensator 612 (MC) determines the associated pixel block corresponding to the first pixel block in the current frame 602 in the historical frame 604 based on motion vector 606. Motion compensator 612 obtains the color difference between the first pixel block and the associated pixel block, i.e., motion compensator 612 outputs color residual information. Switch 615 is placed in the inter-frame prediction branch and connected to motion compensator 612; transformer 622 performs a transformation based on the current frame 602 and the color residual information, converting the information to the frequency domain space. Transformer (T) is a device that transforms the information to the frequency domain space.

[0157] Quantization (Q) 624 is used to reduce the dynamic range of the transformed information. Sequencing 626 is used to rearrange the information output by quantization 624. Encoder 628 performs entropy encoding on the information output by quantization 624 to obtain encoded information 632. Encoder 628 performs entropy encoding on the information output by quantization 624 to reduce the amount of stored data. In some embodiments, the encoding method includes, but is not limited to, at least one of Shannon encoding, Huffman encoding, and arithmetic coding. In some embodiments, encoded information 632 is information obtained by processing motion vector 606 and color residual information through a transformer, quantizer, and encoder. However, it is not excluded that in some examples, the encoded information directly stores one or more of motion vector 606 and color residual information.

[0158] In some embodiments, in another example, switch 615 is placed in the intra-prediction branch and connected to intra-prediction 614; intra-prediction 614 is used to remove spatial redundancy within video frames and reduce the amount of data in video frames based on the correlation between adjacent pixels within the same video frame.

[0159] Next, the motion information will be further described. Referring to Figure 9, Figure 9 shows a flowchart of a video data processing method provided in an exemplary embodiment of this application. This method can be executed by a computer device (i.e., an electronic device). That is, based on the embodiment shown in Figure 3, it further includes step 515:

[0160] Step 515: Perform at least one of data normalization and coordinate axis transformation on the motion information of at least one pixel to obtain the corrected motion information of at least one pixel;

[0161] In some embodiments, the purpose of performing data normalization on the motion information of pixels is to constrain the motion information of pixels within a data range, such as [-1, 1] or [0, 255]. When there is motion information for at least two pixels, this eliminates the differences in dimensions and scales in the motion information of each pixel, making the motion information of different pixels comparable.

[0162] In some embodiments, the methods for performing data normalization on the motion information of at least one pixel include, but are not limited to, at least one of the following: Min-Max Normalization, Z-Score Normalization, and Log Transform.

[0163] In some embodiments, the pixel motion information is referenced to a first Cartesian coordinate system, such as one provided by the rendering engine of the virtual environment; the motion vector of the pixel block is referenced to a second Cartesian coordinate system; to avoid one or more differences in the origin position and coordinate axis direction of the first and second Cartesian coordinate systems, coordinate axis transformation is performed on the motion information of at least one pixel to prevent the motion information from failing to correctly indicate the positional difference between the pixel blocks in the i-th video frame and the i+1-th video frame.

[0164] In this embodiment, the motion vector of at least one pixel block is determined based on the corrected motion information of at least one pixel.

[0165] The data format of motion information is described below. In the embodiments of this application, the motion information of the second pixel includes distance sub-information in the first direction and the second direction;

[0166] In some embodiments, the first direction and the second direction are the directions of the two perpendicular sides of the i-th video frame, and the distance sub-information is used to indicate the components between the second position and the first position in the first and second directions. Describing the motion information of the second pixel point with distance sub-information in two mutually perpendicular directions is beneficial for determining the corresponding pixel point in a matrix of pixels in an integer manner based on the distance sub-information.

[0167] In this embodiment of the application, the motion information of the second pixel includes direction sub-information and distance sub-information;

[0168] In some embodiments, the direction sub-information is used to indicate the direction between the second position and the first position, and the distance sub-information is used to indicate the distance between the second position and the first position. This can intuitively describe the motion direction and motion distance between two pixel frames.

[0169] In summary, the method provided in this application preprocesses motion information by performing one or more of the following: data normalization and coordinate axis transformation. Data normalization ensures the elimination of dimensional and scale differences in the motion information of each pixel, while coordinate axis transformation prevents the motion information from failing to correctly indicate the positional differences between pixel blocks in the i-th and i+1-th video frames. The motion vector determined by the motion information fully utilizes the corresponding virtual spatial positions of pixels on the video frame in the virtual environment. Encoding information is constructed based on the motion vector, color residual information, and color information of the i-th video frame. Compared to directly storing the color information of the i+1-th video frame, this method reduces the space required to store video data.

[0170] Referring to Figure 10, Figure 10 illustrates a flowchart of a video data processing method provided in an exemplary embodiment of this application. This method can be executed by a computer device (i.e., an electronic device). Specifically, in the embodiment shown in Figure 3, step 510 can be implemented as steps 511 to 513; that is, step 510 shown in Figure 3 can be replaced by steps 511 to 513 shown in Figure 10; step 530 can be implemented as steps 532 and 534; that is, step 530 shown in Figure 3 can be replaced by steps 532 and 534 shown in Figure 10.

[0171] Step 511: Obtain the color information of the i-th video frame and the (i+1)-th video frame;

[0172] In some embodiments, the i-th video frame and the i+1-th video frame are two adjacent video frames in a video information. The color information can be directly indicated by information in the form of red-green-blue (RGB) channel, or indirectly indicated by information in the form of hue-saturation-value (HSV) channel. The embodiments of this application do not limit the data format of the color information.

[0173] In some embodiments, step 511 may be executed before, after, or simultaneously with any step in the first step group, which includes steps 512 and 513. This application embodiment does not limit the execution sequence described above; Figure 10 is merely an illustration. In some embodiments, step 511 may be executed before, after, or simultaneously with step 512, and steps 511 and 512 may be executed before step 513.

[0174] Step 512: Obtain control information in the virtual environment;

[0175] In some embodiments, the control information is used to control one or more of virtual objects and virtual cameras in a virtual environment; the control information is used to indicate at least one of rotating the angle of the virtual camera, moving the position of the virtual camera, moving the position of the virtual object, rotating the viewpoint of the virtual object, and controlling the virtual object to perform virtual actions.

[0176] Step 513: Based on the control information and the color information of the i-th video frame, determine the motion information of at least one pixel in the (i+1)-th video frame;

[0177] In some embodiments, the control information directly or indirectly indicates the method of moving the color information of the i-th video frame, thereby obtaining the (i+1)-th video frame. Taking the rotation of a virtual character's perspective as an example, the visual effect of rotating the virtual character's perspective is achieved by rotating and moving the pixels in the i-th video frame with the main virtual character in the video frame as the center point. Taking the movement of a virtual camera's position as an example, the visual effect of moving the virtual camera is achieved by translating the pixels in the i-th video frame.

[0178] In some embodiments, the control information, categorized by type, includes at least one of the following described in step 512: rotating the virtual camera at an angle, moving the virtual camera to a position, moving the virtual object to a position, rotating the virtual object's viewpoint, and controlling the virtual object to perform virtual actions. Different types of control information correspond to different transformation methods for the color information of the i-th video frame (such as translation, rotation, etc., as described above). Based on the mapping relationship between candidate control information and the transformation methods for the color information of the i-th video frame, the first transformation method for the color information of the i-th video frame corresponding to the control information is found; based on the scale information in the control information, the scale of the transformation performed on the color information of the i-th video frame is determined, such as the rotation angle, translation distance, and other scale information.

[0179] In some embodiments, the first transformation mode and scale information indicated by the control information are converted into motion information of the pixel; referring to the description of step 515 above, the motion information of the pixel may include distance sub-information in the first direction and the second direction, or direction sub-information and distance sub-information.

[0180] In some embodiments, taking the second pixel in the (i+1)th video frame as an example, the motion information of the second pixel is used to indicate the positional difference between the first pixel and the second pixel; the first pixel is a pixel in the i-th video frame, and the first pixel and the second pixel correspond to the same position in the virtual environment.

[0181] In some embodiments, steps 511 to 513 in this application embodiment can be combined with steps 520 to 540 shown in FIG3 to form a new embodiment and implemented separately. This application embodiment does not limit this.

[0182] Step 532: Based on the motion vector of the first pixel block, determine the color information of the associated pixel block in the color information of the i-th video frame;

[0183] In some embodiments, the first pixel block is located in a first region in the (i+1)th video frame. Based on this first region location, it is moved by a motion vector to obtain the region location (also called the second region location) of the associated pixel block in the i-th video frame. The color information of the associated pixel block is a sub-part of the color information of the i-th video frame, specifically the color information of the second region location in the i-th video frame. In some embodiments, the associated pixel block determined in the color information of the i-th video frame based on the motion vector of the first pixel block is also called the second pixel block. The second pixel block corresponds to the third pixel block, meaning the second pixel block can be equivalently replaced by the third pixel block.

[0184] In some embodiments, at least one pixel block shown in FIG3 includes a first pixel block, which is any pixel block that determines the motion vector.

[0185] Step 534: Determine the color residual information of the first pixel block based on the difference between the color information of the associated pixel block and the color information of the first pixel block;

[0186] In some embodiments, the color residual information of the first pixel block is used to indicate the difference in color dimension between the associated pixel block and the first pixel block, such as the difference in color values ​​in the RGB channel; however, it is not excluded that the difference in color dimension is recorded in the HSV channel or other ways.

[0187] It is understood that steps 532 and 534 in the embodiments of this application can be combined with steps 510, 520 and 540 shown in FIG3 to form a new embodiment and be implemented separately. The embodiments of this application do not limit this.

[0188] In summary, the embodiments of this application provide a method for processing video data. By using motion vectors determined by motion information, it fully utilizes the virtual spatial positions of pixels on the video frame in the virtual environment. Encoding information is constructed based on motion vectors, color residual information, and the color information of the i-th video frame. Compared to directly storing the color information of the (i+1)-th video frame, this reduces the space required to store video data. By using the motion vectors and color residual information of pixel blocks, the color information of the (i+1)-th video frame can be obtained based on the color information of the i-th video frame. While reducing the storage space of video data, it ensures the integrity of the color information of the (i+1)-th video frame and guarantees the integrity of the image of the (i+1)-th video frame.

[0189] In one application scenario of this application, a computer device (i.e., an electronic device) is equipped with a cloud gaming application, which provides video data processing capabilities. In some embodiments, the cloud gaming application sends encoded information to a terminal, which is equipped with the cloud gaming application and is used to display the running screen of the cloud game. It is understood that the terminal is the user's terminal, used to send control information for virtual characters in the cloud game to the computer device (i.e., the electronic device), receive encoded information, and display game frame images obtained after decoding the encoded information to the user. Accordingly,

[0190] Obtain the color information of the i-th game frame and the (i+1)-th game frame, and obtain the motion information of at least one pixel in the (i+1)-th game frame; in some embodiments, the i-th game frame and the (i+1)-th game frame are two adjacent video frames in the game video information of a cloud game.

[0191] In some embodiments, motion information is used to indicate the positional change of a pixel in the (i+1)th game frame relative to a pixel in the ith game frame. In some embodiments, the motion information of the second pixel includes distance sub-information in a first direction and a second direction;

[0192] Wherein, the first direction and the second direction are the directions of the two vertical sides of the i-th game frame, and the distance sub-information is used to indicate the components between the second position and the first position in the first direction and the second direction.

[0193] In some embodiments, the motion information of the second pixel includes direction sub-information and distance sub-information; wherein, the direction sub-information is used to indicate the direction between the second position and the first position, and the distance information is used to indicate the distance between the second position and the first position.

[0194] Based on the motion information of at least one pixel, the motion vector of at least one pixel block in the (i+1)th game frame is determined; in some embodiments, the motion information of at least one pixel includes a second pixel; the motion information of the second pixel is used to indicate the difference between a second position of the second pixel in the (i+1)th game frame and a first position of the first pixel in the ith game frame. In some embodiments, the motion information of the second pixel is used to indicate the positional difference between two pixels on two game frames.

[0195] In some embodiments, the i-th game frame and the (i+1)-th game frame are used to display a virtual environment, which is a virtual environment provided by a cloud gaming application installed on a computer device (i.e., an electronic device), providing a virtual space for virtual characters to compete in virtual games. In some embodiments, the second pixel and the first pixel correspond to the same position in the virtual environment; the motion vector of the second pixel is used to indicate changes in the viewing screen of the virtual environment.

[0196] In some embodiments, a pixel block includes at least two pixels in the (i+1)th game frame, and the motion vector of the pixel block is obtained based on the motion information of the pixels; in some embodiments, the motion information of the pixels is used to characterize the motion vector of the pixel block.

[0197] In some embodiments, the motion vector of the first pixel block is determined based on the motion information of the pixels in the first pixel set;

[0198] The first pixel set is a subset of at least one pixel, and each pixel in the first pixel set is a pixel in the first pixel block; in some embodiments, the average value, median value, or mode value of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block.

[0199] In some embodiments, a reference vector for the first pixel block is determined based on the motion information of the pixels in the first pixel set;

[0200] Based on the position of the first pixel block in the first region of the (i+1)th game frame and the reference vector, the color sub-information of the reference pixel block is determined in the color information of the i-th game frame.

[0201] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block does not exceed the color threshold, the reference vector is determined as the motion vector of the first pixel block.

[0202] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds the color threshold, a correction pixel block is determined.

[0203] The motion vector of the first pixel block is determined based on the positional difference between the first region position and the second region position of the corrected pixel block in the i-th game frame.

[0204] Specifically, the difference between the color sub-information of the corrected pixel block and the color sub-information of the first pixel block is smaller than the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block.

[0205] In some embodiments, the color threshold is the minimum of at least one reference difference degree; the reference difference degree is used to indicate the difference between the color sub-information of neighboring pixel blocks around the reference pixel block and the color sub-information of the first pixel block.

[0206] In some embodiments, the motion vector of a pixel block can be determined based on the motion information of one or at least two pixels. Based on the motion vector of at least one pixel block, color residual information corresponding to at least one pixel block is determined in the color information of the i-th game frame and the (i+1)-th game frame, respectively.

[0207] In some embodiments, the motion vector of a pixel block is used to indicate the positional difference between two pixel blocks on two game frames; in some embodiments, the color residual information is the color difference between a pixel block in the (i+1)th game frame and its corresponding pixel block in the ith game frame. In some embodiments, the color residual information indicates the color difference at each pixel point within a pixel block.

[0208] In some embodiments, based on the motion vector of the first pixel block, the color information of the associated pixel block is determined in the color information of the i-th game frame, and at least one pixel block includes the first pixel block;

[0209] Based on the difference between the color information of the associated pixel block and the color information of the first pixel block, the color residual information of the first pixel block is determined. Based on the color information of the i-th game frame, and the motion vector and color residual information of at least one pixel block, the encoding information of the video data is constructed.

[0210] In some embodiments, for the (i+1)th game frame, the motion vector of at least one pixel block indicates the difference from the i-th game frame in the pixel block dimension, which is a difference information over a large field of view; the color residual information, based on the motion vector, describes a difference information over a small field of view in the dimension of each pixel point; the differences between the i-th game frame and the (i+1)-th game frame are described from different dimensions. Based on the motion vector and the color residual information, the color information of the (i+1)-th game frame can be indicated based on the color information of the i-th game frame, ensuring the integrity of the color information of the (i+1)-th game frame; the encoded information of the constructed video data does not need to directly carry the color information of the (i+1)-th game frame, reducing the volume of the encoded information of the video data, improving the storage of video data, and improving the transmission efficiency of video data.

[0211] In one application scenario of this application, a computer device (i.e., an electronic device) is equipped with a game application or a utility application with game recording functionality, and the cloud gaming application provides video data processing. In some embodiments, the game application or the utility application with game recording functionality provides the function of recording game footage from past matches. It is understood that the game can be a single-player game or an online game requiring a network connection. In some embodiments, the application has the function of decoding encoded information and playing game footage.

[0212] Obtain the color information of the i-th game frame and the (i+1)-th game frame, and obtain the motion information of at least one pixel in the (i+1)-th game frame; in some embodiments, the i-th game frame and the (i+1)-th game frame are two adjacent video frames in the game video information of a cloud game.

[0213] In some embodiments, motion information is used to indicate the positional change of a pixel in the (i+1)th game frame relative to a pixel in the ith game frame. In some embodiments, the motion information of the second pixel includes distance sub-information in a first direction and a second direction; wherein the first direction and the second direction are the directions of the two vertical edges of the ith game frame, and the distance sub-information is used to indicate the components between the second position and the first position in the first direction and the second direction.

[0214] In some embodiments, the motion information of the second pixel includes direction sub-information and distance sub-information; wherein, the direction sub-information is used to indicate the direction between the second position and the first position, and the distance information is used to indicate the distance between the second position and the first position. Based on the motion information of at least one pixel, the motion vector of at least one pixel block in the (i+1)th game frame is determined;

[0215] In some embodiments, for motion information of at least one pixel, the at least one pixel includes a second pixel; the motion information of the second pixel is used to indicate the difference between a second position of the second pixel in the (i+1)th game frame and a first position of the first pixel in the i-th game frame. In some embodiments, the motion information of the second pixel is used to indicate the positional difference between two pixels on two game frames.

[0216] In some embodiments, the i-th game frame and the (i+1)-th game frame are used to display a virtual environment, which is a virtual environment provided by a cloud gaming application installed on a computer device (i.e., an electronic device), providing a virtual space for virtual characters to compete in virtual games. In some embodiments, the second pixel and the first pixel correspond to the same position in the virtual environment; the motion vector of the second pixel is used to indicate changes in the viewing screen of the virtual environment.

[0217] In some embodiments, a pixel block includes at least two pixels in the (i+1)th game frame, and the motion vector of the pixel block is obtained based on the motion information of the pixels; in some embodiments, the motion information of the pixels is used to characterize the motion vector of the pixel block.

[0218] In some embodiments, the motion vector of the first pixel block is determined based on the motion information of the pixels in the first pixel set; the first pixel set is a subset of at least one pixel, and each pixel in the first pixel set is a pixel in the first pixel block; in some embodiments, any one of the average, median, or mode of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block.

[0219] In some embodiments, a reference vector for a first pixel block is determined based on the motion information of pixels in the first pixel set; and color sub-information of the reference pixel block is determined in the color information of the i-th game frame based on the first region position and reference vector of the first pixel block in the i+1-th game frame.

[0220] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block does not exceed the color threshold, the reference vector is determined as the motion vector of the first pixel block.

[0221] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds a color threshold, a correction pixel block is determined; the motion vector of the first pixel block is determined based on the positional difference between the first region position and the second region position of the correction pixel block in the i-th game frame; wherein, the difference between the color sub-information of the correction pixel block and the color sub-information of the first pixel block is smaller than the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block.

[0222] In some embodiments, the color threshold is the minimum of at least one reference difference degree; the reference difference degree is used to indicate the difference between the color sub-information of neighboring pixel blocks around the reference pixel block and the color sub-information of the first pixel block.

[0223] In some embodiments, the motion vector of a pixel block can be determined based on the motion information of one or at least two pixels. Based on the motion vector of at least one pixel block, color residual information corresponding to at least one pixel block is determined in the color information of the i-th game frame and the (i+1)-th game frame, respectively.

[0224] In some embodiments, the motion vector of a pixel block is used to indicate the positional difference between two pixel blocks on two game frames; in some embodiments, the color residual information is the color difference between a pixel block in the (i+1)th game frame and the corresponding pixel block in the ith game frame. In some embodiments, the color residual information indicates the color difference at each pixel point within a pixel block.

[0225] In some embodiments, based on the motion vector of the first pixel block, the color information of the associated pixel block is determined in the color information of the i-th game frame, and at least one pixel block includes the first pixel block.

[0226] Based on the difference between the color information of the associated pixel block and the color information of the first pixel block, the color residual information of the first pixel block is determined. Based on the color information of the i-th game frame, and the motion vector and color residual information of at least one pixel block, the encoding information of the video data is constructed.

[0227] In some embodiments, for the (i+1)th game frame, the motion vector of at least one pixel block indicates the difference from the i-th game frame in the pixel block dimension, which is a difference information in a large field of view; the color residual information, based on the motion vector, describes a difference information in a small field of view in the dimension of each pixel point; the differences between the i-th game frame and the (i+1)-th game frame are described from different dimensions. Based on the motion vector and the color residual information, the color information of the (i+1)-th game frame can be indicated based on the color information of the i-th game frame, ensuring the integrity of the color information of the (i+1)-th game frame; the encoded information of the constructed video data does not need to directly carry the color information of the (i+1)-th game frame, reducing the volume of the encoded information of the video data, improving the storage of video data, and improving the transmission efficiency of video data.

[0228] Those skilled in the art will understand that the above embodiments can be implemented independently, or the above embodiments can be freely combined to create new embodiments to implement the video data processing method of the embodiments of this application.

[0229] In some embodiments, this application provides another video data processing method, which obtains the encoding information of the video data, the encoding information of the video data being constructed by the above-described video data processing method; and generates the image of the (i+1)th video frame based on the color information of the i-th video frame in the encoding information, as well as the motion vector and color residual information of at least one pixel block.

[0230] Referring to Figure 11, Figure 11 shows a structural block diagram of a video data processing apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0231] The acquisition module 810 is configured to acquire the color information of the i-th video frame and the (i+1)-th video frame, and acquire the motion information of at least one pixel in the (i+1)-th video frame, where i is a positive integer;

[0232] The motion information is used to indicate the displacement of the pixel between the i-th video frame and the (i+1)-th video frame.

[0233] The processing module 820 is configured to determine the motion vector of at least one pixel block in the (i+1)th video frame based on the motion information of the at least one pixel, wherein the at least one pixel includes a second pixel, and the motion information of the second pixel is used to indicate the difference between the second position of the second pixel in the (i+1)th video frame and the first position of the first pixel in the i-th video frame, wherein the i-th video frame and the (i+1)th video frame are used to display a virtual environment, and the pixel data of the second pixel and the pixel data of the first pixel are used to characterize the visual information of the same position in the virtual environment;

[0234] The processing module 820 is further configured to determine, based on the motion vector of the at least one pixel block, color residual information corresponding to the at least one pixel block in the color information of the i-th video frame and the (i+1)-th video frame, respectively.

[0235] The construction module 830 is configured to construct the encoding information of the video data based on the color information of the i-th video frame, the motion vector of the at least one pixel block, and the color residual information.

[0236] In an optional implementation of this application embodiment, the first pixel set is a subset of the at least one pixel, and each pixel in the first pixel set is a pixel in the first pixel block;

[0237] The processing module 820 is further configured to:

[0238] The motion vector of the first pixel block is determined based on the motion information of the pixels in the first pixel set.

[0239] In an optional implementation of this application embodiment, the processing module 820 is further configured to: determine the motion information of any pixel in the first pixel set as the motion vector of the first pixel block.

[0240] In an optional implementation of this application embodiment, the processing module 820 is further configured to perform at least one of the following:

[0241] The average value of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block;

[0242] The median of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block;

[0243] The modulo value of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block.

[0244] In an optional implementation of this application embodiment, the processing module 820 is further configured to:

[0245] Based on the motion information of the pixels in the first pixel set, the reference vector of the first pixel block is determined;

[0246] Based on the first region position of the first pixel block in the (i+1)th video frame and the reference vector, the color sub-information of the reference pixel block is determined in the color information of the i-th video frame;

[0247] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block does not exceed a color threshold, the reference vector is determined as the motion vector of the first pixel block.

[0248] In an optional implementation of this application, the color threshold is the minimum value among at least one reference difference degree; the reference difference degree is used to indicate the difference between the color sub-information of the neighboring pixel blocks around the reference pixel block and the color sub-information of the first pixel block.

[0249] In an optional implementation of this application embodiment, the processing module 820 is further configured to:

[0250] If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds a color threshold, a correction pixel block is determined.

[0251] The motion vector of the first pixel block is determined based on the positional difference between the first region position and the second region position of the corrected pixel block in the i-th video frame.

[0252] Wherein, the difference between the color sub-information of the corrected pixel block and the color sub-information of the first pixel block is smaller than the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block.

[0253] In one optional implementation of this application embodiment, the motion information of the second pixel includes distance sub-information in the first direction and the second direction;

[0254] Wherein, the first direction and the second direction are the directions of the two vertical sides of the i-th video frame, and the distance sub-information is used to indicate the components between the second position and the first position in the first direction and the second direction.

[0255] In one optional implementation of this application embodiment, the motion information of the second pixel includes direction sub-information and distance sub-information;

[0256] The direction sub-information is used to indicate the direction between the second position and the first position, and the distance sub-information is used to indicate the distance between the second position and the first position.

[0257] In an optional implementation of this application embodiment, the processing module 820 is further configured to:

[0258] Perform at least one of data normalization and coordinate axis transformation on the motion information of the at least one pixel to obtain the corrected motion information of the at least one pixel;

[0259] The motion vector of the at least one pixel block is determined based on the corrected motion information of the at least one pixel.

[0260] In an optional implementation of this application embodiment, the acquisition module 810 is further configured to:

[0261] Acquire control information in a virtual environment, the control information being used to control at least one (i.e., one or more) of virtual objects and virtual cameras in the virtual environment;

[0262] Based on the control information, the motion information of at least one pixel in the (i+1)th video frame is determined.

[0263] In an optional implementation of this application embodiment, the processing module 820 is further configured to:

[0264] Based on the motion vector of the first pixel block, the color information of the associated pixel block is determined in the color information of the i-th video frame, wherein the at least one pixel block includes the first pixel block;

[0265] The color residual information of the first pixel block is determined based on the difference between the color information of the associated pixel block and the color information of the first pixel block.

[0266] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0267] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant method; the technical effects achieved by each module performing its operation are the same as the technical effects in the embodiments of the relevant method, and will not be elaborated here.

[0268] This application also provides a computer device (i.e., an electronic device), which includes: a processor and a memory, wherein a computer program is stored in the memory; the processor is configured to execute the computer program in the memory to implement the video data processing method provided in the above-described method embodiments. In some embodiments, the computer device is a server. In some embodiments, FIG12 is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0269] Typically, server 2300 includes a processor 2301 and memory 2302. Processor 2301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2301 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 2301 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2301 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2301 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0270] The memory 2302 may include one or more computer-readable storage media, which may be non-transitory. The memory 2302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2302 are used to store at least one instruction, which is executed by the processor 2301 to implement the video data processing method provided in the method embodiments of this application.

[0271] In some embodiments, the server 2300 may optionally include an input interface 2303 and an output interface 2304. The processor 2301, memory 2302, and input interfaces 2303 and 2304 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interfaces 2303 and 2304 via a bus, signal lines, or a circuit board. The input interfaces 2303 and 2304 can be used to connect at least one input / output (I / O) related peripheral device to the processor 2301 and memory 2302. In some embodiments, the processor 2301, memory 2302, and input interfaces 2303 and 2304 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2301, memory 2302, and input interfaces 2303 and 2304 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0272] Those skilled in the art will understand that the structure shown above does not constitute a limitation on server 2300, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0273] In some embodiments, a chip is also provided, the chip including one or more of programmable logic circuits and program instructions, which, when the chip is run on a computer device (i.e. an electronic device), are used to implement the video data processing method described above.

[0274] In some embodiments, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device (i.e., an electronic device) reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the video data processing methods provided in the above-described method embodiments.

[0275] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program and a video stream. When executed by a processor, the computer program implements the aforementioned video data processing method to generate a video stream. In some embodiments, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0276] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0277] Some implementations may involve systems, methods, and / or computer-readable media at any possible level of integration technical detail. The computer-readable medium may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions on it for causing a processor to perform operations, and may also include storage of a video bitstream generated according to the video data processing method described above. When executed by a processor, the computer program / instructions may implement the steps of the video encoding method to generate the video bitstream, or implement the steps of the video decoding method to decode the video bitstream.

[0278] Any decoder technique other than parsing / entropy decoding, which can only exist in the decoder, must also exist in the corresponding encoder in essentially the same functional form. For this reason, the disclosed subject matter sometimes focuses on decoder operations, which are combined with the decoding part of the encoder. Thus, the description of encoder techniques can be simplified, since encoder techniques are inverses of the fully described decoder techniques.

[0279] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0280] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing video data, the method being executed by an electronic device, the method comprising: Obtain the color information of the i-th video frame and the (i+1)-th video frame, and obtain the motion information of at least one pixel in the (i+1)-th video frame; The motion information is used to indicate the displacement of the pixel between the i-th video frame and the (i+1)-th video frame. Based on the motion information of the at least one pixel, the motion vector of at least one pixel block in the (i+1)th video frame is determined; Based on the motion vector of the at least one pixel block, color residual information corresponding to the at least one pixel block is determined in the color information of the i-th video frame and the (i+1)-th video frame, respectively. The encoding information of the video data is constructed based on the color information of the i-th video frame, the motion vector of the at least one pixel block, and the color residual information.

2. The method according to claim 1, wherein, The first pixel set is a subset of the at least one pixel, and each pixel in the first pixel set is a pixel in the first pixel block; determining the motion vector of at least one pixel block in the (i+1)th video frame based on the motion information of the at least one pixel includes: The motion vector of the first pixel block is determined based on the motion information of the pixels in the first pixel set.

3. The method according to claim 2, wherein, Determining the motion vector of the first pixel block based on the motion information of the pixels in the first pixel set includes at least one of the following: The average value of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block; The median of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block; The modulo value of the motion information of at least two pixels in the first pixel set is determined as the motion vector of the first pixel block.

4. The method according to claim 2, wherein, Determining the motion vector of the first pixel block based on the motion information of the pixels in the first pixel set includes: The motion information of any pixel in the first set of pixels is determined as the motion vector of the first pixel block.

5. The method according to claim 2, wherein, Determining the motion vector of the first pixel block based on the motion information of the pixels in the first pixel set includes: Based on the motion information of the pixels in the first pixel set, the reference vector of the first pixel block is determined; Based on the first region position of the first pixel block in the (i+1)th video frame and the reference vector, the color sub-information of the reference pixel block is determined in the color information of the i-th video frame; If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block does not exceed a color threshold, the reference vector is determined as the motion vector of the first pixel block.

6. The method according to claim 5, wherein, The color threshold is the minimum of at least one reference difference degree; the reference difference degree is used to indicate the difference between the color sub-information of the neighboring pixel blocks around the reference pixel block and the color sub-information of the first pixel block.

7. The method according to claim 5 or 6, wherein, The method further includes: If the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block exceeds a color threshold, a correction pixel block is determined. The motion vector of the first pixel block is determined based on the positional difference between the first region position and the second region position of the corrected pixel block in the i-th video frame. Wherein, the difference between the color sub-information of the corrected pixel block and the color sub-information of the first pixel block is smaller than the difference between the color sub-information of the reference pixel block and the color sub-information of the first pixel block.

8. The method according to any one of claims 1 to 7, wherein, The motion information of the second pixel includes distance sub-information in the first direction and the second direction, and the second pixel belongs to the (i+1)th video frame. Wherein, the first direction and the second direction are the directions of the two vertical sides of the i-th video frame, and the distance sub-information is used to indicate the components between the second position and the first position in the first direction and the second direction, wherein the second position is the position of the second pixel in the (i+1)-th video frame, and the first position is the position of the first pixel in the i-th video frame.

9. The method according to any one of claims 1 to 8, wherein, The motion information of the second pixel includes direction sub-information and distance sub-information, and the second pixel belongs to the (i+1)th video frame. The direction sub-information is used to indicate the direction between the second position and the first position, and the distance sub-information is used to indicate the distance between the second position and the first position. The second position is the position of the second pixel in the (i+1)th video frame, and the first position is the position of the first pixel in the i-th video frame.

10. The method according to any one of claims 1 to 9, wherein, The method further includes: Perform at least one of data normalization and coordinate axis transformation on the motion information of the at least one pixel to obtain the corrected motion information of the at least one pixel; The motion vector of the at least one pixel block is determined based on the corrected motion information of the at least one pixel.

11. The method according to any one of claims 1 to 10, wherein, The step of obtaining motion information of at least one pixel in the (i+1)th video frame includes: Acquire control information in a virtual environment, the control information being used to control one or more virtual objects and virtual cameras in the virtual environment; Based on the control information, the motion information of at least one pixel in the (i+1)th video frame is determined.

12. The method according to any one of claims 1 to 11, wherein, Based on the motion vector of the at least one pixel block, the color residual information corresponding to the at least one pixel block is determined from the color information of the i-th video frame and the (i+1)-th video frame, respectively, including: Based on the motion vector of the first pixel block, the color information of the associated pixel block is determined in the color information of the i-th video frame, wherein the at least one pixel block includes the first pixel block; The color residual information of the first pixel block is determined based on the difference between the color information of the associated pixel block and the color information of the first pixel block.

13. A method for processing video data, the method being performed by an electronic device, the method comprising: The encoding information of the video data is obtained, and the encoding information of the video data is constructed by any one of the video data processing methods of claims 1-12; Based on the color information of the i-th video frame in the encoded information, and the motion vector and color residual information of the at least one pixel block, the image of the (i+1)-th video frame is generated.

14. A video data processing apparatus, the apparatus comprising: The acquisition module is configured to acquire the color information of the i-th video frame and the (i+1)-th video frame, and acquire the motion information of at least one pixel in the (i+1)-th video frame. The motion information is used to indicate the displacement of the pixel between the i-th video frame and the (i+1)-th video frame. The processing module is configured to determine the motion vector of at least one pixel block in the (i+1)th video frame based on the motion information of the at least one pixel point. The processing module is further configured to determine, based on the motion vector of the at least one pixel block, the color residual information corresponding to the at least one pixel block in the color information of the i-th video frame and the (i+1)-th video frame, respectively; The construction module is configured to construct the encoding information of the video data based on the color information of the i-th video frame, the motion vector of the at least one pixel block, and the color residual information.

15. The apparatus according to claim 14, wherein, The first pixel set is a subset of the at least one pixel, and each pixel in the first pixel set is a pixel in the first pixel block; the processing module is further configured to determine the motion vector of the first pixel block based on the motion information of the pixels in the first pixel set.

16. The apparatus according to claim 14 or 15, wherein, The acquisition module is further configured to acquire control information in the virtual environment, the control information being used to control one or more of virtual objects and virtual cameras in the virtual environment; and to determine the motion information of at least one pixel in the (i+1)th video frame based on the control information.

17. An electronic device, the electronic device comprising: A processor and a memory, wherein the memory stores at least one program; The processor is configured to execute the at least one program in the memory to implement the video data processing method as described in any one of claims 1 to 13.

18. A computer-readable storage medium storing a computer program and a video stream, the computer program being executed by a processor to implement the video data processing method as described in any one of claims 1 to 13 to generate the video stream.

19. A computer program product comprising computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implement the video data processing method as described in any one of claims 1 to 13.

20. A method for storing a video stream, comprising generating a video stream by performing the video data processing method according to any one of claims 1 to 12; and storing the video stream.

21. A method for transmitting a video stream, comprising generating a video stream by performing the video data processing method according to any one of claims 1 to 12; and transmitting the video stream.

Images